personal-lifecycle-tracker / on-call-engineer
README.md

On-Call Engineer & Autonomous Incident Response (on-call-engineer/)

This directory provides the automated alert monitoring and autonomous incident remediation pipeline for the Personal Lifecycle Tracker system.

1. Overview & Architecture

The on-call system continuously polls the observability alerting API. When Prometheus or Alertmanager emits an active firing alert, the pipeline normalizes the payload, generates an incident prompt with telemetry context, and passes it to a headless coding agent.

[Prometheus / Alertmanager] (:9091 /api/v1/alerts)
           │
           │ (every 60 seconds)
           ▼
[on-call-engineer/poll-alerts.py]
           │
     (State = firing?)
           ├──► [Normalize to payload-schema.json]
           ├──► [Archive incident to _docs/incidents/incident-<id>.json]
           ├──► [Render agent prompt to _docs/incidents/prompt-<id>.md]
           ▼
[on-call-engineer/scripts/invoke-agent]
           │
           ├──► Headless AI Coding Agent (agy / claude -p / autonomous loop)
           │      1. Collect bounded read-only evidence (scripts/collect-evidence)
           │      2. Evaluate autonomy policy (scripts/authorize-action)
           │      3. Reproduce bug with failing test inside Docker container
           │      4. Apply minimal safe correction preserving contracts
           │      5. Verify metric recovery and alert resolution
           ▼
[on-call-engineer/audit-trail.log]

2. Directory Structure

on-call-engineer/
├── poll-alerts.py          # Primary polling script (default: polls every 60s)
├── poll-alerts             # Executable bash wrapper
├── payload-schema.json     # JSON schema for normalized incident payloads
├── autonomy-policy.json    # Code-enforced autonomy permission levels
├── prompt.md               # Context-injected prompt template for headless agent
├── runbook.md              # Alert-specific triage runbook
├── audit-trail.log         # Timestamped audit log of all polling & agent actions
├── evidence.json           # Read-only bounded evidence packet (generated on incident)
├── scripts/
│   ├── poll-alerts         # Script entrypoint for polling
│   ├── invoke-agent        # Dispatches headless coding agent with incident context
│   ├── receive-alert       # Standalone receiver for webhooks or stdin JSON
│   ├── collect-evidence    # Read-only telemetry harvester (Prometheus, Loki, git)
│   ├── authorize-action    # Code-enforced policy evaluator outside the model
│   ├── trigger-test-incident # Synthetic alert trigger for drill verification
│   └── verify              # Deterministic test script proving the pipeline
└── README.md               # This documentation file

3. How to Run the Alert Poller

Continuous Polling (Default: Every 1 Minute)

# Using Python directly
python3 on-call-engineer/poll-alerts.py

# Using the bash wrapper
./on-call-engineer/poll-alerts

# Using Makefile
make on-call-poll

CLI Options

Flag Description Default
--interval <sec> Polling frequency in seconds 60 (1 minute)
--url <url> Target Alert API endpoint http://localhost:9091/api/v1/alerts
--once Run a single poll cycle and exit immediately false
--mock-file <path> Read alerts from local JSON file (for drills/tests) None
--agent-cmd <cmd> Custom command/script to invoke the headless agent on-call-engineer/scripts/invoke-agent
--retrigger Re-dispatch agent on every poll even if alert was already handled false (stateful deduplication)
--verbose, -v Enable debug logging false

Environment Variables

  • ALERT_API_URL or PROMETHEUS_URL: Override default Prometheus endpoint (default: http://localhost:9091/api/v1/alerts).
  • ALERT_POLL_INTERVAL or POLL_INTERVAL: Override default polling interval (default: 60).

Automated CI/CD Failure Detection (Dual-Channel)

The on-call system ingests CI/CD failures via two automated mechanisms without manual human intervention:

  1. Pull-Based (Continuous Poller): poll-alerts.py automatically queries GitHub Actions runs via gh CLI in every 60s cycle. When a workflow run fails on active branches, it generates _docs/incidents/incident-<id>.json and dispatches the agent immediately.
  2. Push-Based (Real-Time in CI): .github/workflows/ci.yml contains a dedicated notify-on-call job (if: failure()). Whenever ANY job in the workflow fails, it runs automatically and feeds the incident payload directly into on-call-engineer/scripts/receive-alert.

4. Headless Coding Agent Dispatch

When an alert enters the firing state (from either Prometheus or GitHub Actions CI/CD):

  1. Normalization: The raw alert labels and annotations are extracted and structured according to payload-schema.json.
  2. Persistence:
    • Incident data: _docs/incidents/incident-<timestamp>-<alert_name>.json
    • Rendered prompt: _docs/incidents/prompt-<timestamp>-<alert_name>.md
  3. Agent Invocation (scripts/invoke-agent):
    • Antigravity CLI: If agy is installed, runs agy --prompt "$(cat prompt.md)".
    • Claude Code: If claude is installed, runs claude -p "$(cat prompt.md)".
    • Autonomous Headless Responder: If running headless in container/CI, runs the autonomous triage loop: gathers evidence via collect-evidence, verifies against authorize-action, logs the session, and prepares a repro test.
  4. Resolution Tracking: When the alert recovers and is no longer returned as firing (or when the subsequent CI run passes), the poller logs ALERT_RESOLVED and clears the alert fingerprint.

5. Verification & Testing

Execute the automated end-to-end verification script:

bash on-call-engineer/scripts/verify

# Or via Makefile
make on-call-verify

The verification script confirms:

  • All required schemas, policies, and runbooks exist.
  • Alert normalization operates correctly.
  • Live Prometheus alert API is queried successfully (--once).
  • Mock firing alerts trigger incident generation and headless agent dispatch.
  • Read-only evidence collection and code-enforced autonomy policy checks succeed.

6. Local Cloud Production Simulation (SNS -> Lambda -> ECS/Fargate -> S3 & CloudWatch)

We emulate the full AWS enterprise serverless incident response architecture locally using containerized emulators without cloud spend:

[Prometheus / Alertmanager]
          │ (Alert Fires)
          ▼
[Amazon SNS / EventBridge] (Floci S3/SNS Topic)
          │
          │ (SNS Event Notification)
          ▼
[AWS Lambda Dispatcher] (cloud-simulator/lambda/dispatcher.py)
          │
          │ (Runs ephemeral task via ECS run_task)
          ▼
[AWS ECS / Fargate Ephemeral Container] (cloud-simulator/fargate-task/task-runner.sh --rm)
          │
          ├──► Headless AI Coding Agent: Gathers telemetry evidence
          ├──► Evaluates code-enforced autonomy policy
          ├──► Generates root-cause reproduction & minimal git diff patch
          ├──► Emits live logs to CloudWatch Log Stream
          │
          ▼
[Audit & Archival]
   ├──► Archives deliverables to Amazon S3: `s3://lifecycle-on-call-incidents/<incident-id>/`
   │      - `incident.json` (normalized alert payload)
   │      - `prompt.md` (agent context prompt)
   │      - `evidence.json` (bounded telemetry packet)
   │      - `diff.patch` (minimal code correction)
   │      - `transcript.log` (complete execution audit trail)
   └──► Ephemeral compute container terminates and destroys immediately (`--rm`)

Running the Cloud Simulation

# Using Makefile
make on-call-simulate-cloud

# Or direct bash script
bash on-call-engineer/cloud-simulator/scripts/run-cloud-simulation

7. Live AWS Serverless On-Call Responder (SNS -> Lambda)

For production deployments on AWS, the responder deploys as a native serverless Lambda function (on-call-engineer/lambda_handler.py) integrated with Amazon SNS and CloudWatch Alarms:

[CloudWatch Alarm / EventBridge]
           │
           │ (Alarm Firing Notification)
           ▼
[Amazon SNS Alert Topic]
           │
           │ (Event Notification)
           ▼
[AWS Lambda Responder] (on-call-engineer/lambda_handler.py)
           │
           ├──► Normalizes incoming alert payload
           ├──► Evaluates Autonomy Policy outside the model:
           │      - LEVEL_1_AUTO_EXECUTE: Autonomous operational remediation (rollback, restart)
           │      - LEVEL_2_HUMAN_APPROVAL: Pauses execution, emits patch diff, awaits operator sign-off
           │      - LEVEL_3_ESCALATE: Hard escalation to human engineer
           ├──► Verifies service health endpoint
           └──► Emits timestamped audit log to CloudWatch Logs

Running the Live AWS Level 1 Drill

make on-call-aws-drill
# or: bash on-call-engineer/scripts/run-aws-incident-drill

8. Level 2 Code-Fix, Human Approval Gate & Automated Git Push Drill

When an incident is caused by a real code defect rather than an operational blip, the responder executes the Level 2 workflow:

  1. Failure Reproduction: Runs a reproduction test proving the failure before touching any code.
  2. Autonomy Policy Enforcement (Without Approval): If human_approval=false, the responder halts immediately (PAUSED_FOR_HUMAN_APPROVAL, HTTP 202) and emits the proposed patch diff for human review.
  3. Operator Sign-Off: Once the operator reviews the proposed diff and approves (human_approval=true), the responder applies the minimal patch.
  4. Test Verification: Executes the reproduction test (verifying HTTP 201 Created) and the complete backend test suite.
  5. Security Gate & Git Audit Trail: Triggers pre-commit Semgrep SAST scan (0 findings required) and commits the verified fix to Git with incident audit references.

Running the Level 2 Code-Fix Drill

make on-call-code-fix-drill
# or: bash on-call-engineer/scripts/run-code-fix-drill
Powered by GitFront