# On-Call Incident Triage Runbook — Personal Lifecycle Tracker

This runbook guides automated and human on-call engineers triaging alerts emitted by the Personal Lifecycle Tracker observability stack.

---

## 1. General Triage Procedure

1. **Acknowledge Alert**:
   - Inspect the normalized incident payload in `_docs/incidents/incident-<id>.json`.
   - Verify impacted service (`api-backend`, `web-frontend`), environment, and deployed version.

2. **Inspect Telemetry**:
   - **Grafana Dashboard**: [http://localhost:3004/d/lifecycle-tracker-overview](http://localhost:3004/d/lifecycle-tracker-overview)
   - **Backend Logs**:
     ```bash
     rtk docker compose logs --tail=100 -f backend
     ```
   - **OTel Collector Logs**:
     ```bash
     rtk docker compose -f observability/docker-compose.yaml logs --tail=100 otel-collector
     ```
   - **Prometheus Alert API**:
     ```bash
     rtk curl http://localhost:9091/api/v1/alerts
     ```

3. **Trace & Log Correlation**:
   - In Grafana (or Tempo at `http://localhost:3201`), filter for spans with `status.code = ERROR` or HTTP status `500`.
   - In Loki (`http://localhost:3101` or via Grafana Explore), query `{service_name="api-backend"} |= "error"`.

4. **Local Defect Reproduction**:
   - Run baseline container tests to confirm current state:
     ```bash
     docker compose run --rm backend pytest tests -m "not integration"
     ```
   - Author a test reproducing the defect in `backend/tests/test_repro_<incident_id>.py`.

5. **Remediation & Verification**:
   - Implement the smallest safe correction in `backend/app/`.
   - Re-run test suite inside container: `make test-backend`.
   - Verify that the Prometheus alert changes from `firing` to `resolved`.

---

## 2. Alert-Specific Triage Guides

### `HighHttpErrorRate`
- **Severity**: Critical
- **Trigger**: 5xx HTTP errors exceed 5% over 1 minute.
- **Likely Causes**:
  - Unhandled exception in FastAPI router (`backend/app/routers/`).
  - Database connectivity loss or connection pool exhaustion in `backend/app/database.py`.
  - Malformed JWT token parsing or missing environment variables.
- **Action**:
  1. Inspect recent commits: `git log -n 5`.
  2. Inspect backend logs for tracebacks: `docker compose logs --tail=100 backend`.
  3. Verify PostgreSQL status: `docker compose exec db pg_isready -U postgres -d lifecycle`.

### `HighHttpLatencyP99`
- **Severity**: Warning
- **Trigger**: p99 HTTP latency exceeds 1000ms over 2 minutes.
- **Likely Causes**:
  - Unindexed SQL queries over `items`, `reminders`, or `groups` tables.
  - Synchronous blocking I/O inside asynchronous async def endpoints.
  - High lock contention during batch updates.
- **Action**:
  1. Inspect Tempo traces to identify slow database or processing spans.
  2. Add indexes or optimize query plan in `backend/app/store.py` / `models.py`.

### `DailyDigestFailures`
- **Severity**: Critical
- **Trigger**: Morning email digest generation failure counter increments.
- **Likely Causes**:
  - SMTP connection failure or invalid email address formatting.
  - Template rendering failure when user has zero or overdue lifecycle items.
- **Action**:
  1. Check digest logs: `docker compose logs backend | grep -i digest`.
  2. Test digest endpoint locally via `curl -X POST http://localhost:8003/api/v1/digests/test-run`.

### `RepeatedItemCreationFailures` / `LifecycleItemCreationFailures`
- **Severity**: Critical
- **Trigger**: Lifecycle item creation failure rate exceeds 5% over 5 minutes.
- **Likely Causes**:
  - Database schema constraint violation (e.g. invalid lifecycle type).
  - Pydantic validation regression on renewal date or maintenance interval.
  - Auth token expiry or missing group permission.
- **Action**:
  1. Inspect validation errors in backend logs: `docker compose logs backend | grep -E "(422|500)"`.
  2. Verify payload matches `contracts/openapi.yaml` schema for `LifecycleItemCreate`.

---

## 3. GitHub Actions CI/CD Triage Guides

### `CiCdSecurityAuditFailed`
- **Severity**: Critical
- **Trigger**: GitHub Actions `security-audit` job fails on a PR or commit.
- **Likely Causes**:
  - Semgrep SAST detected high-risk vulnerability (e.g. wildcard CORS, hardcoded secret pattern, insecure JWT algorithm).
  - `npm audit` detected high or critical CVE in frontend dependencies.
- **Action**:
  1. Reproduce deterministic SAST scan locally inside Docker:
     ```bash
     make security-audit
     ```
  2. Inspect specific file and line flagged by Semgrep.
  3. Apply minimal security remediation (e.g. restrict origins, update dependencies).
  4. Verify scan passes with 0 findings: `make security-audit`.
  5. **Negative Invariant**: Agent is **STRICTLY FORBIDDEN** from solving this by disabling Semgrep rules, adding `--no-verify`, or suppressing `--audit-level`.

### `CiCdTestJobFailed`
- **Severity**: Critical
- **Trigger**: GitHub Actions `test-backend` or `test-frontend` job fails.
- **Likely Causes**:
  - Regression in business logic, broken model validation, or broken test assertion.
  - Asynchronous timing or unhandled promise rejection in Vitest.
- **Action**:
  1. Reproduce failure locally inside container without host pollution:
     ```bash
     make test-backend
     # or: make test-frontend
     ```
  2. Write/inspect reproducing test case.
  3. Implement minimal safe correction and verify all tests pass: `make test`.

### `CiCdBuildFailed`
- **Severity**: Critical
- **Trigger**: Docker image build stage fails in CI runner.
- **Likely Causes**:
  - Outdated base image tag, missing Linux compilation dependency, or broken multi-stage target.
- **Action**:
  1. Reproduce container build locally:
     ```bash
     docker compose build --no-cache
     ```
  2. Fix Dockerfile syntax or pin dependencies in `requirements.txt` / `package.json`.

### `CiCdWorkflowSyntaxError`
- **Severity**: Warning
- **Trigger**: Invalid YAML syntax or deprecated GitHub Actions runner/action.
- **Likely Causes**:
  - Misaligned indentation, invalid `${{ ... }}` expression, or deprecated action tag.
- **Action**:
  1. Validate workflow locally via `act`:
     ```bash
     make ci-validate
     ```
  2. Correct syntax and confirm dry-run passes cleanly.