Server Automation with Ansible

Comprehensive Ansible playbooks for server provisioning and configuration management.

🔐 SECURITY NOTICE

This repository is designed for public sharing while keeping your actual server details private.

What's Public (✅ Safe)

• Ansible playbooks and roles
• Configuration templates
• Documentation
• Role defaults and examples

What's Protected (🔒 Git-ignored)

• Real server IP addresses (inventory/local.yml)
• SSH keys and passwords (.vault_password)
• Encrypted secrets (group_vars/vault.yml)
• Host-specific configurations (host_vars/*.yml)

Quick Setup for New Users

1. Install Python dependencies: pip install -r requirements-python.txt
2. Run the setup script: ./setup-inventory.sh
3. Edit your server details: inventory/local.yml
4. Configure secrets: ansible-vault edit group_vars/vault.yml
5. Test connectivity: ansible all -m ping

See inventory/README.md for detailed instructions.

Playbook Structure

Main Playbooks

• provision.yml - Full server provisioning (orchestrates all roles)
• deploy.yml - Service deployment and management (CI/CD pipeline)
• prod.yml - Production environment provisioning
• staging.yml - Staging environment provisioning
• dev.yml - Development environment provisioning

Playbook Reference

Infrastructure Setup

system.yml — System baseline, Docker, and security hardening. Target: dynamic (-e target_host=<host>)

ansible-playbook system.yml -e target_host=runner

user.yml — User creation, SSH keys, and dotfiles. Target: dynamic (-e target_host=<host>)

ansible-playbook user.yml -e target_host=runner

generic_setup.yml — Combined system + user setup (one-shot provisioning). Target: dynamic (-e target_host=<host>)

ansible-playbook generic_setup.yml -e target_host=runner

Service Deployments

runner.yml — Dev/media services (Frigate, Immich, Forgejo, LLM stack, ...). Two-phase: configures NAS exports on storage first, then deploys to runner.

ansible-playbook runner.yml
ansible-playbook runner.yml --tags frigate

storage.yml — Media automation (Jellyfin, Arr stack, Calibre, restic server).

ansible-playbook storage.yml
ansible-playbook storage.yml --tags jellyfin

connectivity.yml — VPN, DNS, and reverse proxy.

ansible-playbook connectivity.yml

homeassistant.yml — Home Assistant Supervised installation.

ansible-playbook homeassistant.yml

entrance.yml — Zigbee, Thread, and energy monitoring gateway.

ansible-playbook entrance.yml

doorbell.yml — IoT camera doorbell.

ansible-playbook doorbell.yml

hifi-nodes.yml — Multi-room audio clients on HiFi nodes (Snapcast or Sendspin).

ansible-playbook hifi-nodes.yml
ansible-playbook hifi-nodes.yml -e audio_client_type=sendspin

Operations

backup.yml — Backup and restore Docker services via restic.

ansible-playbook backup.yml                                          # all services
ansible-playbook backup.yml -e "backup_services=immich,frigate"      # specific services
ansible-playbook backup.yml -e "backup_mode=restore backup_services=immich"  # restore
ansible-playbook backup.yml --tags discovery                         # dry-run discovery

update-services.yml — Pull latest container images and restart.

ansible-playbook update-services.yml                   # all hosts
ansible-playbook update-services.yml --tags jellyfin   # specific service

Directory Structure

server_automation/
 ansible.cfg           # Ansible configuration
 requirements.yml      # Galaxy role dependencies
 provision.yml         # Main provisioning playbook
 prod.yml              # Production environment
 staging.yml           # Staging environment
 dev.yml               # Development environment
 role-*.yml           # Role-specific playbooks
 security-*.yml       # Security playbooks
 inventory/
    hosts            # Main inventory
    production       # Production hosts
    staging          # Staging hosts
    development      # Development hosts
    group_vars/      # Group variables
    host_vars/       # Host-specific variables
 group_vars/
    all.yml          # Global variables
    prod.yml         # Production variables
    staging.yml      # Staging variables
    dev.yml          # Development variables
    security.yml     # Security variables
 host_vars/
    example-server.yml # Host template
 roles/
     user/            # User management
     system/          # System configuration
     backup/          # Backup system
     gui-gnome/       # GNOME desktop
     audio-client/    # Multi-room audio (Snapcast/Sendspin)
     security-baseline/ # Security hardening
     firewall/        # Firewall configuration

Service Deployment Structure

The services/ directory contains service-specific deployment roles:

services/
   web/                 # Web services (Nginx, Apache)
   api/                 # API services (Node.js, Python)  
   database/            # Database services (PostgreSQL, Redis)
   monitoring/          # Monitoring stack (Prometheus, Grafana)
   cache/               # Caching services (Redis, Memcached)
   templates/           # Service templates and examples
     service-template/  # Template for creating new services

CI/CD Workflows

.gitea/workflows/
   ci.yml              # Validation workflow (lint, test, security)
   deploy.yml          # Deployment workflow (staging, production)

Geerlingguy Roles Integration

This repository integrates two key geerlingguy roles:

geerlingguy.docker (v7.0.0)

• Docker Engine installation and management
• Docker Compose installation
• User group configuration for Docker access

geerlingguy.security (v3.0.0)

• Comprehensive system security hardening
• SSH security configuration
• Fail2Ban installation and configuration
• System auditing with auditd
• Network security hardening
• Automatic security updates

See ROLES-INTEGRATION.md for detailed configuration and usage.

Usage

Installation

# 1. Install Ansible Galaxy roles
ansible-galaxy install -r requirements.yml

# 2. Install collections
ansible-galaxy collection install -r collections.yml

# 3. Set up secure inventory (first time only)
./setup-inventory.sh

# 4. Configure your servers and secrets
nano inventory/local.yml
ansible-vault edit group_vars/vault.yml

Running Playbooks

Full provisioning:

ansible-playbook provision.yml

Environment-specific (if using multiple inventories):

ansible-playbook prod.yml -i inventory/production.yml
ansible-playbook staging.yml -i inventory/staging.yml  
ansible-playbook dev.yml

Role-specific:

ansible-playbook role-system.yml
ansible-playbook role-docker.yml

Security hardening:

ansible-playbook security-baseline.yml -e "security_level=high"
ansible-playbook firewall.yml

Tags

Playbooks support granular execution using tags:

# Run only system tasks
ansible-playbook provision.yml --tags system

# Run everything except GUI
ansible-playbook provision.yml --skip-tags gui

# Run multiple tags
ansible-playbook provision.yml --tags system,user,docker

Available tags: system, user, backup, gui, audio, docker, security, firewall, wifi

Variables and Configuration

Environment Variables

• env: Environment type (prod, staging, dev)
• security_level: Security strictness (low, medium, high)

Role Configuration

Each role has its own defaults in roles/<role>/defaults/main.yml

Variable Precedence

1. Host variables (host_vars/)
2. Group variables (group_vars/)
3. Role defaults
4. Playbook variables

Security Features

• SSH hardening with fail2ban
• UFW firewall configuration
• Password policy enforcement
• Auditd logging (high security)
• Package security updates
• File permission hardening

Best Practices

• Idempotent playbooks (safe to run multiple times)
• Proper error handling with any_errors_fatal
• Environment-specific configurations
• Tag-based execution for granular control
• Variable precedence following Ansible standards
• Comprehensive logging and reporting

Dependencies

• geerlingguy.docker (Docker engine)
• community.docker (Docker modules)
• ansible.posix (POSIX utilities)
• Various security and monitoring roles

Contributing

1. Follow existing patterns and conventions
2. Maintain idempotency
3. Use appropriate tags
4. Update documentation
5. Test across environments

Ansible Vault Secrets Management

This repository includes comprehensive Ansible Vault integration for secure secrets management:

Vault Structure

vault/
  envs/           # Environment-specific secrets
    dev/
    staging/
    prod/
  secrets/        # Categorized secrets
    api_keys/
    certificates/
    databases/
    ssh_keys/
    cloud_credentials/
  passwords/      # Vault password management
  scripts/        # Helper scripts
  templates/      # Template files

Quick Start

1. Generate vault passwords:

   ./vault/scripts/generate-vault-passwords.sh
   

2. Encrypt vault files:

   ./vault/scripts/encrypt-vault.sh dev
   ./vault/scripts/encrypt-vault.sh staging  
   ./vault/scripts/encrypt-vault.sh prod
   

3. Use in playbooks:

   ANSIBLE_ENV=dev ansible-playbook playbook.yml --vault-password-file=./vault/passwords/.vault-pass.envs
   

See vault/README.md for complete documentation.

Inventory Management

The inventory system supports multiple environments and dynamic cloud integration:

Environments

• Production: High security, strict policies
• Staging: Pre-production testing
• Development: Development and experimentation

Cloud Integration

• AWS EC2 dynamic inventory
• DigitalOcean dynamic inventory
• Custom cloud provider support

See inventory/README.md for detailed usage.

Testing and Validation

Syntax Validation

# Validate playbook syntax
ansible-playbook --syntax-check provision.yml

# Validate inventory
ansible-inventory -i inventory/hosts --list

Integration Testing

# Test vault integration
./vault/scripts/test-vault-integration.sh dev

# Test role integration  
./validate-integration.sh

Dry Runs

# Check what would change
ansible-playbook provision.yml --check --diff

CI/CD Pipeline

This repository includes a comprehensive CI/CD pipeline using Forgejo/Gitea Actions with separate validation and deployment workflows.

Pipeline Overview

Code Push → CI Validation → Merge to Main → Automatic Deployment
     ↓            ↓              ↓                ↓
  Feature    Syntax Check    Staging Deploy   Production Deploy
  Branch     Security Scan   Health Checks    Manual Approval
             Integration     Rollback Ready   Full Monitoring

Workflows

1. CI Workflow (.gitea/workflows/ci.yml)

Triggers: Push to main/develop, Pull Requests Purpose: Comprehensive validation without deployment

Jobs:

• Lint & Syntax Check: YAML validation, Ansible linting, syntax verification
• Integration Tests: Custom validation, dry-run testing, role structure validation
• Documentation Check: README validation, role documentation completeness
• Security Scan: Security best practices, secrets detection, permission checks
• Summary: Consolidated results and status reporting

2. Deploy Workflow (.gitea/workflows/deploy.yml)

Triggers: Push to main (after CI success), Manual workflow dispatch Purpose: Service deployment with rollback capability

Jobs:

• Check CI Status: Ensures CI pipeline completed successfully
• Prepare Deployment: Determines target environments and services
• Deploy Staging: Automatic deployment to staging environment
• Deploy Production: Manual approval required for production
• Emergency Rollback: Automatic rollback on deployment failure

Service Deployment

Deploy Playbook (deploy.yml)

Separate from site.yml, focuses exclusively on service deployment:

# Deploy all services to staging
ansible-playbook -i inventory/staging deploy.yml

# Deploy specific services
ansible-playbook -i inventory/production deploy.yml \
  --extra-vars "target_services=web,api"

# Rollback deployment
ansible-playbook -i inventory/production deploy.yml \
  --tags rollback --extra-vars "rollback_version=v1.2.3"

Service Categories

• Web Services: Frontend applications, reverse proxies (Nginx)
• API Services: Backend APIs, microservices (Node.js, Python)
• Database Services: PostgreSQL, Redis with backup strategies
• Monitoring Services: Prometheus, Grafana, Alertmanager
• Cache Services: Redis, Memcached for performance optimization

Adding New Services

1. Create Service Role:

   cp -r services/templates/service-template services/your-service
   # Edit and customize the service configuration
   

2. Update Deploy Configuration:

   # In group_vars/all.yml or environment-specific files
   your_service_services:
     - name: your-service-app
       image: your-org/your-service
       tag: "v1.0.0"
       ports: ["8080:8080"]
       environment:
         ENV: "production"
   

3. Add Health Checks:

   your_service_health_checks:
     - name: your-service-app
       port: 8080
       path: "/health"
       expected_status: 200
   

Environment Configuration

Staging Environment

• Automatic deployment on main branch merges
• Rolling updates with 50% parallel deployment
• Quick health checks (3 retries, 15s delay)
• Limited retention (7 days backup)

Production Environment

• Manual approval required for deployment
• Blue-green strategy for zero-downtime deployment
• Comprehensive health checks (5 retries, 30s delay)
• Extended retention (30 days backup)
• Email notifications on deployment events

Self-Hosted Runners

The deployment workflow requires self-hosted runners for:

• Server Access: Direct access to deployment targets
• Network Security: Internal network connectivity
• Custom Tools: Ansible, SSH keys, monitoring tools
• Performance: Faster deployment without external dependencies

Runner Setup

# Install Forgejo Runner
sudo apt update
sudo apt install docker.io ansible python3-pip
pip3 install docker

# Configure runner with repository
./setup-runner.sh --token YOUR_RUNNER_TOKEN

Deployment Security

Secrets Management

Required secrets in Forgejo/Gitea repository settings:

• SSH_PRIVATE_KEY: SSH key for server access
• STAGING_HOST: Staging server hostname/IP
• PRODUCTION_HOST: Production server hostname/IP
• VAULT_PASSWORD: Ansible Vault password for secrets

Environment Protection

• Staging: No restrictions, automatic deployment
• Production: Manual approval required, deployment protection rules
• Rollback: Emergency rollback environment with restricted access

Monitoring and Rollback

Health Checks

All deployments include comprehensive health monitoring:

• Container Health: Docker container status and health checks
• Service Endpoints: HTTP health check endpoints
• Database Connectivity: Connection and query validation
• Resource Usage: Memory, CPU, disk space monitoring

Rollback Procedures

# Automatic rollback (triggered on deployment failure)
ansible-playbook -i inventory/production deploy.yml --tags rollback

# Manual rollback to specific version
ansible-playbook -i inventory/production deploy.yml \
  --tags rollback --extra-vars "rollback_version=20231201-123456"

Deployment Logs

• Location: /var/log/deployments/deployment.log
• Retention: Automatic cleanup with configurable retention
• Artifacts: Deployment logs uploaded as workflow artifacts
• Notifications: Email alerts on deployment success/failure

Local Testing

Pre-commit Validation

# Test the same checks that run in CI
./test-ci-locally.sh

# Run specific validation
./validate-integration.sh

# Test deployment playbook
ansible-playbook deploy.yml --check --diff -i inventory/staging

Service Testing

# Test new service locally
docker-compose -f services/your-service/docker-compose.yml up -d

# Test service health
curl -f http://localhost:8080/health

# Test with Ansible
ansible-playbook deploy.yml --tags your-service --check

Backup & Restore

The backup system uses restic with a REST server on storage.home. It auto-discovers all Docker Compose services under /docker/ on every host in the docker_servers inventory group.

Discovery (dry-run)

# See what services exist on which hosts
ansible-playbook backup.yml --tags discovery

Backup

# Backup ALL services on ALL hosts
ansible-playbook backup.yml

# Backup a single service (runs only on the host that has it)
ansible-playbook backup.yml -e "backup_services=immich"

# Backup multiple specific services
ansible-playbook backup.yml -e "backup_services=immich,frigate,forgejo"

# Backup with verbose restic output
ansible-playbook backup.yml -e "backup_services=jellyfin" -v

Restore

# Restore a single service from its latest snapshot
ansible-playbook backup.yml -e "backup_mode=restore backup_services=immich"

# Restore multiple services
ansible-playbook backup.yml -e "backup_mode=restore backup_services=immich,frigate"

Restic CLI (on any docker_server host)

# List all snapshots
restic snapshots

# List snapshots for a specific service
restic snapshots --tag immich

# Check repository integrity
restic check

# Manual prune
restic forget --keep-last 10 --prune

Restic environment variables (RESTIC_REPOSITORY, RESTIC_PASSWORD) must be set or passed for CLI usage. The backup password is stored in the Ansible vault as restic_backup_password.

Troubleshooting

Common Issues

1. Missing roles: Run ansible-galaxy install -r requirements.yml
2. Vault errors: Ensure vault passwords are set and files are encrypted
3. Permission issues: Check SSH keys and sudo privileges
4. Network issues: Verify inventory host connectivity
5. CI failures: Check .gitea/workflows/README.md for troubleshooting

Debugging

# Enable verbose output
ansible-playbook provision.yml -vvv

# Debug specific tasks
ansible-playbook provision.yml --tags system -vvv

# Debug deployment issues
ansible-playbook deploy.yml --check --diff --tags health-check

CI/CD Troubleshooting

# Check CI status
git status && git log --oneline -5

# Test CI locally
./test-ci-locally.sh

# Debug deployment
ansible-playbook deploy.yml --check --diff -vvv

# View deployment logs
tail -f /var/log/deployments/deployment.log

Best Practices

Security

• Use Ansible Vault for all secrets
• Follow principle of least privilege
• Regular security updates and auditing
• Environment-specific security levels

Maintenance

• Regular password rotation
• Backup vault passwords securely
• Test playbooks in staging before production
• Monitor for deprecated modules

Performance

• Use tags for targeted execution
• Leverage facts caching
• Optimize with strategy plugins
• Use async tasks for long-running operations

Support

For issues and questions:

1. Check the troubleshooting section
2. Review role-specific README files
3. Validate syntax and integration
4. Test in development environment first

License

This project is licensed under the MIT License - see the LICENSE file for details.