Operations Runbook: Alert Response & Disaster Recovery

Service: Personal Lifecycle Tracker
Target Environments: dev, production
Observability Suite: Prometheus (:9091), Loki (:3101), Tempo (:3201), Grafana (:3004)

1. High-Priority Alert Response Procedures

Alert: HighServerErrorRate (5xx Errors > 1%)

• Condition: 5xx HTTP response rate exceeds 1% of total traffic over a 2-minute sliding window.
• Immediate Diagnostic Steps:
  1. Open Grafana Golden Signals Dashboard at http://localhost:3004/d/lifecycle-golden-signals.
  2. Filter by Environment: production and isolate failing routes.
  3. Query Loki logs in Grafana: {service="backend"} |= "ERROR".
  4. Copy failing TraceID and paste into Tempo Trace Search.
  5. Check if PostgreSQL database is reachable: make test-integration or docker compose ps db.
• Mitigation:
  • If due to database connection exhaustion, restart container or scale pool: docker compose restart backend.
  • If due to software bug, engage the on-call engineer or trigger headless remediation loop: make on-call-verify.

Alert: HighP95Latency (Latency > 500ms)

• Condition: 95th percentile response latency exceeds 500ms over a 5-minute sliding window.
• Diagnostic Steps:
  1. Inspect PostgreSQL query durations in Prometheus: pg_stat_activity.
  2. Check for missing indexes or unpaginated list requests.
  3. Check CPU/Memory constraints: docker stats.

2. Backup & Disaster Recovery Procedures

Automated Database Backup

Run daily PostgreSQL logical backup inside container:

docker compose exec -T db pg_dump -U postgres -d lifecycle | gzip > backups/lifecycle_$(date +%Y%m%d_%H%M%S).sql.gz

Database Restore & Recovery Verification

Test backup restoration into an isolated test container:

gunzip < backups/latest.sql.gz | docker compose exec -T db psql -U postgres -d lifecycle_restore_test

3. Deployment Rollback Workflow

If a production release introduces regression:

1. Revert to previous verified container image tag in AWS SSM parameter /personal-lifecycle/production/image_tag:

   aws ssm put-parameter --name "/personal-lifecycle/production/image_tag" --value "PREVIOUS_TAG" --type String --overwrite
   

2. Re-trigger the deploy job without rebuilding:

   docker compose -f docker-compose.prod.yml pull
   docker compose -f docker-compose.prod.yml up -d --no-build
   

3. Confirm health check returns HTTP 200:

   curl -f http://localhost:8088/health