Ops — Disaster Recovery & Backups
Disaster recovery (DR) ensures your systems can recover from data loss, infrastructure failure, or regional outages. Without a tested DR plan, a single failure can cause permanent data loss, extended downtime, and business-ending consequences.
This section covers RTO/RPO, backup strategies, database backups, multi-region failover, DR testing, cloud provider tools, and a disaster recovery plan template.
Two metrics define your recovery capability: RTO (Recovery Time Objective) — how quickly you must recover — and RPO (Recovery Point Objective) — how much data loss is acceptable.
| Metric | Definition | Example |
|---|---|---|
| RTO | Maximum acceptable downtime | Must be back online within 1 hour |
| RPO | Maximum acceptable data loss | Can afford to lose at most 5 minutes of data |
| 1 | // DR strategy matrix based on RTO/RPO |
| 2 | const drStrategies = { |
| 3 | // RTO: 24h, RPO: 24h — Cheapest |
| 4 | "backup-restore": { |
| 5 | rto: "24 hours", |
| 6 | rpo: "24 hours", |
| 7 | cost: "$", |
| 8 | description: "Daily backups, restore to new infrastructure manually", |
| 9 | useCase: "Internal tools, non-critical systems", |
| 10 | }, |
| 11 | |
| 12 | // RTO: 4h, RPO: 1h — Moderate |
| 13 | "pilot-light": { |
| 14 | rto: "4 hours", |
| 15 | rpo: "1 hour", |
| 16 | cost: "$$", |
| 17 | description: "Minimal always-running standby, scale up on failover", |
| 18 | useCase: "Important but not time-critical services", |
| 19 | }, |
| 20 | |
| 21 | // RTO: minutes, RPO: near-zero — Expensive |
| 22 | "warm-standby": { |
| 23 | rto: "minutes", |
| 24 | rpo: "near-zero", |
| 25 | cost: "$$$", |
| 26 | description: "Reduced-capacity standby, scale to full on failover", |
| 27 | useCase: "User-facing production services", |
| 28 | }, |
| 29 | |
| 30 | // RTO: seconds, RPO: zero — Most expensive |
| 31 | "active-active": { |
| 32 | rto: "seconds", |
| 33 | rpo: "zero", |
| 34 | cost: "$$$$", |
| 35 | description: "Full multi-region with automatic failover", |
| 36 | useCase: "Mission-critical, zero-tolerance systems", |
| 37 | }, |
| 38 | }; |
info
Different backup types offer trade-offs between storage cost, backup speed, and restore speed. Most production environments use a combination.
| Type | What It Backs Up | Speed | Storage |
|---|---|---|---|
| Full | Entire dataset | Slowest backup, fastest restore | Highest |
| Incremental | Changes since last backup (any type) | Fastest backup, slowest restore (needs full + all incrementals) | Lowest |
| Differential | Changes since last full backup | Moderate backup, moderate restore (full + latest differential) | Moderate |
| Snapshot | Point-in-time copy of volume/database | Near-instant (copy-on-write) | Depends on change rate |
| 1 | # Typical backup schedule |
| 2 | # Sunday: Full backup |
| 3 | # Mon-Sat: Incremental backups |
| 4 | # Retain: 4 weekly fulls, 3 monthly fulls |
| 5 | |
| 6 | # PostgreSQL: Full backup with pg_dump |
| 7 | pg_dump -Fc -h localhost -U app mydb > /backups/mydb_$(date +%Y%m%d).dump |
| 8 | |
| 9 | # PostgreSQL: Continuous WAL archiving (point-in-time recovery) |
| 10 | # In postgresql.conf: |
| 11 | # wal_level = replica |
| 12 | # archive_mode = on |
| 13 | # archive_command = 'cp %p /archive/%f' |
| 14 | |
| 15 | # Restore to a specific point in time |
| 16 | pg_ctl stop -D /var/lib/postgresql/data |
| 17 | rm -rf /var/lib/postgresql/data |
| 18 | pg_basebackup -h backup-server -D /var/lib/postgresql/data -Fp -Xs -P |
| 19 | # Configure recovery.conf with restore_command and recovery_target_time |
| 20 | pg_ctl start -D /var/lib/postgresql/data |
warning
Database backups require special attention because databases have complex internal state, transactions, and consistency requirements. File-level backups of database files are usually insufficient.
| 1 | # PostgreSQL: Automated backup script |
| 2 | #!/bin/bash |
| 3 | DB_HOST="localhost" |
| 4 | DB_NAME="production" |
| 5 | BACKUP_DIR="/backups/postgres" |
| 6 | RETENTION_DAYS=30 |
| 7 | |
| 8 | # Full dump |
| 9 | pg_dump -Fc -h $DB_HOST -U postgres $DB_NAME \ |
| 10 | | gzip > "$BACKUP_DIR/full_$(date +%Y%m%d_%H%M%S).dump.gz" |
| 11 | |
| 12 | # Upload to S3 with lifecycle policy |
| 13 | aws s3 cp "$BACKUP_DIR/full_*.dump.gz" \ |
| 14 | s3://myorg-db-backups/postgres/ \ |
| 15 | --storage-class STANDARD_IA |
| 16 | |
| 17 | # Clean up old local backups |
| 18 | find $BACKUP_DIR -name "*.dump.gz" -mtime +$RETENTION_DAYS -delete |
| 19 | |
| 20 | # Verify backup integrity |
| 21 | pg_restore --list "$BACKUP_DIR/full_*.dump.gz" > /dev/null 2>&1 |
| 22 | if [ $? -eq 0 ]; then |
| 23 | echo "Backup verified successfully" |
| 24 | else |
| 25 | echo "BACKUP VERIFICATION FAILED" |
| 26 | # Send alert |
| 27 | fi |
| 1 | # MongoDB: Backup with mongodump |
| 2 | mongodump --host replicaSet/prod \ |
| 3 | --username admin \ |
| 4 | --password "$MONGO_PASSWORD" \ |
| 5 | --authenticationDatabase admin \ |
| 6 | --gzip \ |
| 7 | --archive="/backups/mongo_$(date +%Y%m%d).gz" |
| 8 | |
| 9 | # MongoDB: Restore from backup |
| 10 | mongorestore --host replicaSet/prod \ |
| 11 | --gzip \ |
| 12 | --archive="/backups/mongo_20250115.gz" \ |
| 13 | --drop # Drop existing data before restore |
best practice
Application files, uploaded content, and persistent volumes need regular backups. Use snapshot-based approaches for volumes and incremental tools for file systems.
| 1 | # AWS EBS: Snapshot-based backups |
| 2 | # Create snapshot of production volume |
| 3 | aws ec2 create-snapshot \ |
| 4 | --volume-id vol-0abc123def456 \ |
| 5 | --description "Daily backup $(date +%Y-%m-%d)" \ |
| 6 | --tag-specifications \ |
| 7 | 'ResourceType=snapshot,Tags=[{Key=Backup,Value=daily},{Key=Environment,Value=production}]' |
| 8 | |
| 9 | # AWS EBS Lifecycle Manager: Automated snapshots |
| 10 | # Policy: Snapshot every 24h, retain for 30 days |
| 11 | aws dlm create-lifecycle-policy \ |
| 12 | --description "Daily EBS snapshots" \ |
| 13 | --state ENABLED \ |
| 14 | --execution-role-arn arn:aws:iam::role/dlm-role \ |
| 15 | --policy-details '{ |
| 16 | "PolicyType": "EBS_SNAPSHOT_MANAGEMENT", |
| 17 | "ResourceTypes": ["VOLUME"], |
| 18 | "TargetTags": [{"Key": "Backup", "Value": "daily"}], |
| 19 | "Schedules": [{ |
| 20 | "Name": "daily", |
| 21 | "CreateRule": {"Interval": 24, "IntervalUnit": "HOURS"}, |
| 22 | "RetainRule": {"Count": 30} |
| 23 | }] |
| 24 | }' |
Multi-region failover provides resilience against entire region outages. The most common patterns are active-passive (one region is primary, another is standby) and active-active (both regions serve traffic simultaneously).
| 1 | // AWS Route 53: Failover routing policy |
| 2 | const failoverConfig = { |
| 3 | primary: { |
| 4 | type: "A", |
| 5 | name: "api.example.com", |
| 6 | failover: "PRIMARY", |
| 7 | region: "us-east-1", |
| 8 | setIdentifier: "us-east-1-primary", |
| 9 | healthCheckId: "us-east-1-health-check", |
| 10 | aliasTarget: { |
| 11 | dnsName: "alb-us-east-1.example.com", |
| 12 | hostedZoneId: "Z35SXDOTRQ7X7K", |
| 13 | }, |
| 14 | }, |
| 15 | secondary: { |
| 16 | type: "A", |
| 17 | name: "api.example.com", |
| 18 | failover: "SECONDARY", |
| 19 | region: "eu-west-1", |
| 20 | setIdentifier: "eu-west-1-secondary", |
| 21 | healthCheckId: "eu-west-1-health-check", |
| 22 | aliasTarget: { |
| 23 | dnsName: "alb-eu-west-1.example.com", |
| 24 | hostedZoneId: "Z3NF1Z3NOM5OY2", |
| 25 | }, |
| 26 | }, |
| 27 | }; |
| 28 | |
| 29 | // Database replication for cross-region failover |
| 30 | // PostgreSQL: Use read replicas in secondary region |
| 31 | // Promote replica to primary during failover |
| 32 | const dbFailover = { |
| 33 | primary: "postgres-primary.us-east-1.rds.amazonaws.com", |
| 34 | replica: "postgres-replica.eu-west-1.rds.amazonaws.com", |
| 35 | promotionSteps: [ |
| 36 | "1. Stop writes to primary", |
| 37 | "2. Wait for replication lag to reach zero", |
| 38 | "3. Promote replica to primary", |
| 39 | "4. Update DNS to point to new primary", |
| 40 | "5. Verify application connectivity", |
| 41 | "6. Resume writes", |
| 42 | ], |
| 43 | }; |
info
DR testing verifies that your backup and recovery procedures actually work within your target RTO and RPO. Untested DR plans are assumptions, not guarantees.
Tabletop Exercises
Walk through disaster scenarios as a team. Discuss what would happen, who does what, and identify gaps in the plan without touching production.
Backup Restore Tests
Actually restore a backup to a fresh environment. Verify data integrity, application functionality, and measure restore time against your RTO.
Chaos Engineering
Deliberately inject failures (kill instances, corrupt data, disconnect regions) and verify your DR procedures work under realistic conditions.
Cloud providers offer managed DR services that simplify backup, replication, and failover. Use them when available — they eliminate the operational burden of managing your own DR infrastructure.
| Provider | Tool | Purpose |
|---|---|---|
| AWS | Backup, DLM, Route 53 Failover | Centralized backup, automated snapshots, DNS failover |
| GCP | Backup and DR Service, Cloud DNS | Application-consistent backups, regional failover |
| Azure | Azure Backup, Traffic Manager, geo-replication | VM/database backups, geo-distributed load balancing |
Backups are high-value targets for attackers because they contain complete copies of your data. Encrypt backups at rest and in transit, and restrict access with least-privilege IAM policies.
Encrypt at rest
Use AES-256 encryption for all backups. Store encryption keys separately from the backups (use KMS, not hardcoded keys).
Encrypt in transit
Use TLS for all backup data transfer between your infrastructure and the backup storage location.
Immutability
Enable S3 Object Lock or equivalent to prevent backups from being deleted or modified — even by administrators. Protects against ransomware.
A DR plan document ensures everyone knows what to do during a disaster. Keep it accessible (not locked behind the infrastructure that is down) and review it quarterly.
| 1 | // DR Plan Structure |
| 2 | const drPlan = { |
| 3 | 1_overview: { |
| 4 | purpose: "Recovery procedures for production system failure", |
| 5 | scope: "All production services in us-east-1", |
| 6 | rto: "1 hour", |
| 7 | rpo: "5 minutes", |
| 8 | lastTested: "2025-01-10", |
| 9 | nextScheduledTest: "2025-04-10", |
| 10 | }, |
| 11 | |
| 12 | 2_team: { |
| 13 | primaryContact: "Jane Smith (jane@example.com)", |
| 14 | secondaryContact: "Bob Jones (bob@example.com)", |
| 15 | escalationPath: [ |
| 16 | "1. On-call engineer", |
| 17 | "2. Team lead", |
| 18 | "3. VP Engineering", |
| 19 | "4. CTO", |
| 20 | ], |
| 21 | }, |
| 22 | |
| 23 | 3_recoverySteps: [ |
| 24 | { |
| 25 | step: 1, |
| 26 | action: "Assess scope of disaster", |
| 27 | owner: "On-call engineer", |
| 28 | timeout: "15 minutes", |
| 29 | details: "Check monitoring dashboards, verify which services are affected", |
| 30 | }, |
| 31 | { |
| 32 | step: 2, |
| 33 | action: "Declare disaster (if region-wide)", |
| 34 | owner: "Team lead", |
| 35 | timeout: "5 minutes", |
| 36 | details: "Activate DR plan, notify stakeholders", |
| 37 | }, |
| 38 | { |
| 39 | step: 3, |
| 40 | action: "Promote database replica", |
| 41 | owner: "On-call engineer", |
| 42 | timeout: "10 minutes", |
| 43 | details: "Promote us-west-2 replica to primary", |
| 44 | }, |
| 45 | { |
| 46 | step: 4, |
| 47 | action: "Update DNS / traffic routing", |
| 48 | owner: "On-call engineer", |
| 49 | timeout: "5 minutes", |
| 50 | details: "Switch Route 53 failover to us-west-2", |
| 51 | }, |
| 52 | { |
| 53 | step: 5, |
| 54 | action: "Verify application health", |
| 55 | owner: "On-call engineer", |
| 56 | timeout: "15 minutes", |
| 57 | details: "Run smoke tests, check error rates, verify user flows", |
| 58 | }, |
| 59 | ], |
| 60 | |
| 61 | 4_communication: { |
| 62 | statusPage: "https://status.example.com", |
| 63 | internalSlack: "#incident-response", |
| 64 | customerEmail: "support@example.com", |
| 65 | }, |
| 66 | }; |
✓ Test your backups regularly
Perform full restore tests at least quarterly. A backup that cannot be restored is not a backup.
✓ Follow the 3-2-1 rule
3 copies of data, on 2 different media types, with 1 offsite (different region or provider).
✓ Automate backup verification
Automate restore-to-staging and smoke test after every backup. Do not rely on manual verification.
✓ Document and practice
Keep a written DR plan accessible offline. Conduct tabletop exercises and chaos engineering drills.