server_automation
This repo is destined for my server automations and setup.
17.1 KB•MD
README.md
17.1 KB • 608 lines • markdown
1# Server Automation with Ansible
2
3Comprehensive Ansible playbooks for server provisioning and configuration management.
4
5## ð SECURITY NOTICE
6
7**This repository is designed for public sharing while keeping your actual server details private.**
8
9### What's Public (â
Safe)
10- Ansible playbooks and roles
11- Configuration templates
12- Documentation
13- Role defaults and examples
14
15### What's Protected (ð Git-ignored)
16- Real server IP addresses (`inventory/local.yml`)
17- SSH keys and passwords (`.vault_password`)
18- Encrypted secrets (`group_vars/vault.yml`)
19- Host-specific configurations (`host_vars/*.yml`)
20
21### Quick Setup for New Users
221. Install Python dependencies: `pip install -r requirements-python.txt`
232. Run the setup script: `./setup-inventory.sh`
243. Edit your server details: `inventory/local.yml`
254. Configure secrets: `ansible-vault edit group_vars/vault.yml`
265. Test connectivity: `ansible all -m ping`
27
28See [inventory/README.md](inventory/README.md) for detailed instructions.
29
30## Playbook Structure
31
32### Main Playbooks
33- **provision.yml** - Full server provisioning (orchestrates all roles)
34- **deploy.yml** - Service deployment and management (CI/CD pipeline)
35- **prod.yml** - Production environment provisioning
36- **staging.yml** - Staging environment provisioning
37- **dev.yml** - Development environment provisioning
38
39### Role-Specific Playbooks
40- **user.yml** - User management with dynamic host targeting
41- **homeassistant.yml** - Complete Home Assistant Supervised installation
42- **role-system.yml** - System configuration only
43- **role-backup.yml** - Backup system configuration
44- **role-gui.yml** - GNOME desktop environment
45- **role-snapcast.yml** - Snapcast audio client
46- **role-docker.yml** - Docker engine installation (geerlingguy.docker)
47- **role-security.yml** - Comprehensive security hardening (geerlingguy.security)
48
49### Security Playbooks
50- **security-baseline.yml** - Fundamental security hardening
51- **firewall.yml** - UFW firewall configuration
52
53## Quick Start Examples
54
55### User Management with Dynamic Targeting
56```bash
57# Deploy user to specific server
58ansible-playbook user.yml -e target_host=192.168.1.100
59
60# Deploy user to inventory group
61ansible-playbook user.yml -e target_host=homeassistant
62
63# Custom user name
64ansible-playbook user.yml -e target_host=server1 -e user_name=admin
65```
66
67### Home Assistant Installation
68```bash
69# Complete HA installation (requires geerlingguy.docker)
70ansible-playbook homeassistant.yml
71
72# Specific server from homeassistant group
73ansible-playbook homeassistant.yml -l ha-server-01
74
75# Install only specific components
76ansible-playbook homeassistant.yml --tags "user,system,docker"
77```
78
79## Directory Structure
80
81```
82server_automation/
83 ansible.cfg # Ansible configuration
84 requirements.yml # Galaxy role dependencies
85 provision.yml # Main provisioning playbook
86 prod.yml # Production environment
87 staging.yml # Staging environment
88 dev.yml # Development environment
89 role-*.yml # Role-specific playbooks
90 security-*.yml # Security playbooks
91 inventory/
92 hosts # Main inventory
93 production # Production hosts
94 staging # Staging hosts
95 development # Development hosts
96 group_vars/ # Group variables
97 host_vars/ # Host-specific variables
98 group_vars/
99 all.yml # Global variables
100 prod.yml # Production variables
101 staging.yml # Staging variables
102 dev.yml # Development variables
103 security.yml # Security variables
104 host_vars/
105 example-server.yml # Host template
106 roles/
107 user/ # User management
108 system/ # System configuration
109 backup/ # Backup system
110 gui-gnome/ # GNOME desktop
111 snapcast-client/ # Audio streaming
112 security-baseline/ # Security hardening
113 firewall/ # Firewall configuration
114```
115
116### Service Deployment Structure
117
118The `services/` directory contains service-specific deployment roles:
119
120```
121services/
122 web/ # Web services (Nginx, Apache)
123 api/ # API services (Node.js, Python)
124 database/ # Database services (PostgreSQL, Redis)
125 monitoring/ # Monitoring stack (Prometheus, Grafana)
126 cache/ # Caching services (Redis, Memcached)
127 templates/ # Service templates and examples
128 service-template/ # Template for creating new services
129```
130
131### CI/CD Workflows
132
133```
134.gitea/workflows/
135 ci.yml # Validation workflow (lint, test, security)
136 deploy.yml # Deployment workflow (staging, production)
137```
138
139## Geerlingguy Roles Integration
140
141This repository integrates two key geerlingguy roles:
142
143### geerlingguy.docker (v7.0.0)
144- Docker Engine installation and management
145- Docker Compose installation
146- User group configuration for Docker access
147
148### geerlingguy.security (v3.0.0)
149- Comprehensive system security hardening
150- SSH security configuration
151- Fail2Ban installation and configuration
152- System auditing with auditd
153- Network security hardening
154- Automatic security updates
155
156See [ROLES-INTEGRATION.md](ROLES-INTEGRATION.md) for detailed configuration and usage.
157
158## Usage
159
160### Installation
161```bash
162# 1. Install Ansible Galaxy roles
163ansible-galaxy install -r requirements.yml
164
165# 2. Install collections
166ansible-galaxy collection install -r collections.yml
167
168# 3. Set up secure inventory (first time only)
169./setup-inventory.sh
170
171# 4. Configure your servers and secrets
172nano inventory/local.yml
173ansible-vault edit group_vars/vault.yml
174```
175
176### Running Playbooks
177
178**Full provisioning:**
179```bash
180ansible-playbook provision.yml
181```
182
183**Environment-specific (if using multiple inventories):**
184```bash
185ansible-playbook prod.yml -i inventory/production.yml
186ansible-playbook staging.yml -i inventory/staging.yml
187ansible-playbook dev.yml
188```
189
190**Role-specific:**
191```bash
192ansible-playbook role-system.yml
193ansible-playbook role-docker.yml
194```
195
196**Security hardening:**
197```bash
198ansible-playbook security-baseline.yml -e "security_level=high"
199ansible-playbook firewall.yml
200```
201
202### Tags
203
204Playbooks support granular execution using tags:
205
206```bash
207# Run only system tasks
208ansible-playbook provision.yml --tags system
209
210# Run everything except GUI
211ansible-playbook provision.yml --skip-tags gui
212
213# Run multiple tags
214ansible-playbook provision.yml --tags system,user,docker
215```
216
217Available tags: `system`, `user`, `backup`, `gui`, `snapcast`, `docker`, `security`, `firewall`
218
219## Variables and Configuration
220
221### Environment Variables
222- `env`: Environment type (prod, staging, dev)
223- `security_level`: Security strictness (low, medium, high)
224
225### Role Configuration
226Each role has its own defaults in `roles/<role>/defaults/main.yml`
227
228### Variable Precedence
2291. Host variables (`host_vars/`)
2302. Group variables (`group_vars/`)
2313. Role defaults
2324. Playbook variables
233
234## Security Features
235
236- SSH hardening with fail2ban
237- UFW firewall configuration
238- Password policy enforcement
239- Auditd logging (high security)
240- Package security updates
241- File permission hardening
242
243## Best Practices
244
245- Idempotent playbooks (safe to run multiple times)
246- Proper error handling with `any_errors_fatal`
247- Environment-specific configurations
248- Tag-based execution for granular control
249- Variable precedence following Ansible standards
250- Comprehensive logging and reporting
251
252## Dependencies
253
254- geerlingguy.docker (Docker engine)
255- community.docker (Docker modules)
256- ansible.posix (POSIX utilities)
257- Various security and monitoring roles
258
259## Contributing
260
2611. Follow existing patterns and conventions
2622. Maintain idempotency
2633. Use appropriate tags
2644. Update documentation
2655. Test across environments
266
267## Ansible Vault Secrets Management
268
269This repository includes comprehensive Ansible Vault integration for secure secrets management:
270
271### Vault Structure
272```
273vault/
274 envs/ # Environment-specific secrets
275 dev/
276 staging/
277 prod/
278 secrets/ # Categorized secrets
279 api_keys/
280 certificates/
281 databases/
282 ssh_keys/
283 cloud_credentials/
284 passwords/ # Vault password management
285 scripts/ # Helper scripts
286 templates/ # Template files
287```
288
289### Quick Start
2901. Generate vault passwords:
291 ```bash
292 ./vault/scripts/generate-vault-passwords.sh
293 ```
294
2952. Encrypt vault files:
296 ```bash
297 ./vault/scripts/encrypt-vault.sh dev
298 ./vault/scripts/encrypt-vault.sh staging
299 ./vault/scripts/encrypt-vault.sh prod
300 ```
301
3023. Use in playbooks:
303 ```bash
304 ANSIBLE_ENV=dev ansible-playbook playbook.yml --vault-password-file=./vault/passwords/.vault-pass.envs
305 ```
306
307See [vault/README.md](vault/README.md) for complete documentation.
308
309## Inventory Management
310
311The inventory system supports multiple environments and dynamic cloud integration:
312
313### Environments
314- **Production**: High security, strict policies
315- **Staging**: Pre-production testing
316- **Development**: Development and experimentation
317
318### Cloud Integration
319- AWS EC2 dynamic inventory
320- DigitalOcean dynamic inventory
321- Custom cloud provider support
322
323See [inventory/README.md](inventory/README.md) for detailed usage.
324
325## Testing and Validation
326
327### Syntax Validation
328```bash
329# Validate playbook syntax
330ansible-playbook --syntax-check provision.yml
331
332# Validate inventory
333ansible-inventory -i inventory/hosts --list
334```
335
336### Integration Testing
337```bash
338# Test vault integration
339./vault/scripts/test-vault-integration.sh dev
340
341# Test role integration
342./validate-integration.sh
343```
344
345### Dry Runs
346```bash
347# Check what would change
348ansible-playbook provision.yml --check --diff
349```
350
351## CI/CD Pipeline
352
353This repository includes a comprehensive CI/CD pipeline using Forgejo/Gitea Actions with separate validation and deployment workflows.
354
355### Pipeline Overview
356
357```
358Code Push â CI Validation â Merge to Main â Automatic Deployment
359 â â â â
360 Feature Syntax Check Staging Deploy Production Deploy
361 Branch Security Scan Health Checks Manual Approval
362 Integration Rollback Ready Full Monitoring
363```
364
365### Workflows
366
367#### 1. CI Workflow (`.gitea/workflows/ci.yml`)
368**Triggers**: Push to main/develop, Pull Requests
369**Purpose**: Comprehensive validation without deployment
370
371**Jobs**:
372- **Lint & Syntax Check**: YAML validation, Ansible linting, syntax verification
373- **Integration Tests**: Custom validation, dry-run testing, role structure validation
374- **Documentation Check**: README validation, role documentation completeness
375- **Security Scan**: Security best practices, secrets detection, permission checks
376- **Summary**: Consolidated results and status reporting
377
378#### 2. Deploy Workflow (`.gitea/workflows/deploy.yml`)
379**Triggers**: Push to main (after CI success), Manual workflow dispatch
380**Purpose**: Service deployment with rollback capability
381
382**Jobs**:
383- **Check CI Status**: Ensures CI pipeline completed successfully
384- **Prepare Deployment**: Determines target environments and services
385- **Deploy Staging**: Automatic deployment to staging environment
386- **Deploy Production**: Manual approval required for production
387- **Emergency Rollback**: Automatic rollback on deployment failure
388
389### Service Deployment
390
391#### Deploy Playbook (`deploy.yml`)
392Separate from `site.yml`, focuses exclusively on service deployment:
393
394```bash
395# Deploy all services to staging
396ansible-playbook -i inventory/staging deploy.yml
397
398# Deploy specific services
399ansible-playbook -i inventory/production deploy.yml \
400 --extra-vars "target_services=web,api"
401
402# Rollback deployment
403ansible-playbook -i inventory/production deploy.yml \
404 --tags rollback --extra-vars "rollback_version=v1.2.3"
405```
406
407#### Service Categories
408- **Web Services**: Frontend applications, reverse proxies (Nginx)
409- **API Services**: Backend APIs, microservices (Node.js, Python)
410- **Database Services**: PostgreSQL, Redis with backup strategies
411- **Monitoring Services**: Prometheus, Grafana, Alertmanager
412- **Cache Services**: Redis, Memcached for performance optimization
413
414### Adding New Services
415
4161. **Create Service Role**:
417 ```bash
418 cp -r services/templates/service-template services/your-service
419 # Edit and customize the service configuration
420 ```
421
4222. **Update Deploy Configuration**:
423 ```yaml
424 # In group_vars/all.yml or environment-specific files
425 your_service_services:
426 - name: your-service-app
427 image: your-org/your-service
428 tag: "v1.0.0"
429 ports: ["8080:8080"]
430 environment:
431 ENV: "production"
432 ```
433
4343. **Add Health Checks**:
435 ```yaml
436 your_service_health_checks:
437 - name: your-service-app
438 port: 8080
439 path: "/health"
440 expected_status: 200
441 ```
442
443### Environment Configuration
444
445#### Staging Environment
446- **Automatic deployment** on main branch merges
447- **Rolling updates** with 50% parallel deployment
448- **Quick health checks** (3 retries, 15s delay)
449- **Limited retention** (7 days backup)
450
451#### Production Environment
452- **Manual approval required** for deployment
453- **Blue-green strategy** for zero-downtime deployment
454- **Comprehensive health checks** (5 retries, 30s delay)
455- **Extended retention** (30 days backup)
456- **Email notifications** on deployment events
457
458### Self-Hosted Runners
459
460The deployment workflow requires self-hosted runners for:
461- **Server Access**: Direct access to deployment targets
462- **Network Security**: Internal network connectivity
463- **Custom Tools**: Ansible, SSH keys, monitoring tools
464- **Performance**: Faster deployment without external dependencies
465
466#### Runner Setup
467```bash
468# Install Forgejo Runner
469sudo apt update
470sudo apt install docker.io ansible python3-pip
471pip3 install docker
472
473# Configure runner with repository
474./setup-runner.sh --token YOUR_RUNNER_TOKEN
475```
476
477### Deployment Security
478
479#### Secrets Management
480Required secrets in Forgejo/Gitea repository settings:
481- `SSH_PRIVATE_KEY`: SSH key for server access
482- `STAGING_HOST`: Staging server hostname/IP
483- `PRODUCTION_HOST`: Production server hostname/IP
484- `VAULT_PASSWORD`: Ansible Vault password for secrets
485
486#### Environment Protection
487- **Staging**: No restrictions, automatic deployment
488- **Production**: Manual approval required, deployment protection rules
489- **Rollback**: Emergency rollback environment with restricted access
490
491### Monitoring and Rollback
492
493#### Health Checks
494All deployments include comprehensive health monitoring:
495- **Container Health**: Docker container status and health checks
496- **Service Endpoints**: HTTP health check endpoints
497- **Database Connectivity**: Connection and query validation
498- **Resource Usage**: Memory, CPU, disk space monitoring
499
500#### Rollback Procedures
501```bash
502# Automatic rollback (triggered on deployment failure)
503ansible-playbook -i inventory/production deploy.yml --tags rollback
504
505# Manual rollback to specific version
506ansible-playbook -i inventory/production deploy.yml \
507 --tags rollback --extra-vars "rollback_version=20231201-123456"
508```
509
510#### Deployment Logs
511- **Location**: `/var/log/deployments/deployment.log`
512- **Retention**: Automatic cleanup with configurable retention
513- **Artifacts**: Deployment logs uploaded as workflow artifacts
514- **Notifications**: Email alerts on deployment success/failure
515
516### Local Testing
517
518#### Pre-commit Validation
519```bash
520# Test the same checks that run in CI
521./test-ci-locally.sh
522
523# Run specific validation
524./validate-integration.sh
525
526# Test deployment playbook
527ansible-playbook deploy.yml --check --diff -i inventory/staging
528```
529
530#### Service Testing
531```bash
532# Test new service locally
533docker-compose -f services/your-service/docker-compose.yml up -d
534
535# Test service health
536curl -f http://localhost:8080/health
537
538# Test with Ansible
539ansible-playbook deploy.yml --tags your-service --check
540```
541
542## Troubleshooting
543
544### Common Issues
5451. **Missing roles**: Run `ansible-galaxy install -r requirements.yml`
5462. **Vault errors**: Ensure vault passwords are set and files are encrypted
5473. **Permission issues**: Check SSH keys and sudo privileges
5484. **Network issues**: Verify inventory host connectivity
5495. **CI failures**: Check `.gitea/workflows/README.md` for troubleshooting
550
551### Debugging
552```bash
553# Enable verbose output
554ansible-playbook provision.yml -vvv
555
556# Debug specific tasks
557ansible-playbook provision.yml --tags system -vvv
558
559# Debug deployment issues
560ansible-playbook deploy.yml --check --diff --tags health-check
561```
562
563### CI/CD Troubleshooting
564```bash
565# Check CI status
566git status && git log --oneline -5
567
568# Test CI locally
569./test-ci-locally.sh
570
571# Debug deployment
572ansible-playbook deploy.yml --check --diff -vvv
573
574# View deployment logs
575tail -f /var/log/deployments/deployment.log
576```
577
578## Best Practices
579
580### Security
581- Use Ansible Vault for all secrets
582- Follow principle of least privilege
583- Regular security updates and auditing
584- Environment-specific security levels
585
586### Maintenance
587- Regular password rotation
588- Backup vault passwords securely
589- Test playbooks in staging before production
590- Monitor for deprecated modules
591
592### Performance
593- Use tags for targeted execution
594- Leverage facts caching
595- Optimize with strategy plugins
596- Use async tasks for long-running operations
597
598## Support
599
600For issues and questions:
6011. Check the troubleshooting section
6022. Review role-specific README files
6033. Validate syntax and integration
6044. Test in development environment first
605
606## License
607
608This project is licensed under the MIT License - see the LICENSE file for details.