# OpenTelemetry Architecture & Telemetry Specification

**Project**: Personal Lifecycle Tracker  
**Lifecycle Stage**: Gate 14 — OpenTelemetry Full-Stack Instrumentation & Decoupled Observability

---

## 1. Golden Triangle Resource Attributes

Every telemetry signal (traces, metrics, logs) generated by any service in the Personal Lifecycle Tracker stack MUST include the standard semantic conventions:

| Resource Attribute | Semantic Key | Description | Example Values |
|---|---|---|---|
| **Service Name** | `service.name` | Logical identifier of the component | `api-backend`, `web-frontend`, `caddy-proxy` |
| **Environment** | `deployment.environment` | Deployment lifecycle stage | `development`, `staging`, `production` |
| **Deployed Version** | `service.version` | Release version or commit SHA | `dev-latest`, `20261010-123320-16fea64`, `v1.0.0` |

### Environment Variables
```ini
OTEL_SERVICE_NAME=api-backend
OTEL_RESOURCE_ATTRIBUTES=service.name=api-backend,deployment.environment=development,service.version=dev-latest
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4328
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

---

## 2. Distributed Tracing Topology (W3C Trace Context)

Distributed traces propagate correlation headers across all boundaries using the W3C Trace Context specification (`traceparent` and `tracestate`):

```
[Browser / React Web Client]
       │
       │ (W3C traceparent injected by FetchInstrumentation)
       ▼
[Production Caddy 2 Reverse Proxy]
       │
       │ (traceparent header passed through, X-Request-ID attached)
       ▼
[FastAPI Backend Application]
       │
       │ (FastAPIInstrumentor starts server span, extracts traceparent)
       ▼
[PostgreSQL Database (SQLAlchemy / psycopg2)]
         (Child DB span: query, table, operation latency)
```

---

## 3. Decoupled Observability Architecture (`observability/`)

The observability infrastructure is isolated in `observability/docker-compose.yaml` with its own lifecycle, network (`personal-lifecycle-observability-net`), and persistence:

- **OpenTelemetry Collector Contrib (`otel/opentelemetry-collector-contrib:0.95.0`)**:
  - Ingests traces, metrics, and logs via OTLP gRPC (`:4317` internal / `:4327` host) and OTLP HTTP (`:4318` internal / `:4328` host).
  - Routes traces to Tempo (`tempo:4317`).
  - Converts and exposes metrics to Prometheus (`:8889` internal / `:8899` host).
  - Routes logs to Loki (`http://loki:3100/loki/api/v1/push`).
- **Prometheus (`prom/prometheus:v2.51.0`)**:
  - Scrapes metrics on port `:9090` (host `:9091`).
- **Loki (`grafana/loki:2.9.4`)**:
  - Aggregates logs on port `:3100` (host `:3101`).
- **Tempo (`grafana/tempo:2.4.1`)**:
  - Stores distributed trace spans on port `:3200` (host `:3201`).
- **Grafana (`grafana/grafana:10.4.0`)**:
  - Serves unified visualizations on port `:3000` (host `:3004`).
  - Pre-provisioned datasources: Prometheus, Loki (with traceId derived field), Tempo (with tracesToLogsV2 and tracesToMetrics).
  - Pre-provisioned dashboard: `lifecycle-tracker-overview.json`.

---

## 4. Application Metrics & Product Indicators

All application metrics emitted by `api-backend` attach the mandatory Golden Triangle labels (`deployment_environment`, `service_version`, `service_name`).

### A. Metadata & Golden Signals
| Metric Name | Instrument | Labels | Purpose |
|---|---|---|---|
| `app_info` | Gauge | `deployment_environment`, `service_version`, `service_name` | Instantaneous metadata indicator reporting active environment and deployed version |
| `app_http_server_duration_milliseconds` | Histogram | `http_route`, `http_status_code`, `http_method`, `deployment_environment`, `service_version`, `service_name` | Latency distribution percentiles (p50, p90, p99) by normalized route |
| `app_http_requests_total` | Counter | `http_route`, `http_status_code`, `http_method`, `deployment_environment`, `service_version`, `service_name` | HTTP request throughput and error counts |

### B. Product & Domain Metrics (Personal Lifecycle Tracker)
| Metric Name | Instrument | Labels | Purpose |
|---|---|---|---|
| `app_items_created_total` | Counter | `lifecycle_type` (`renewal`, `maintenance`, `warranty`), `deployment_environment`, `service_version`, `service_name` | Tracks lifecycle item creation by category |
| `app_item_creation_failures_total` | Counter | `lifecycle_type`, `deployment_environment`, `service_version`, `service_name` | Tracks failed lifecycle item creation attempts |
| `app_items_completed_total` | Counter | `lifecycle_type`, `deployment_environment`, `service_version`, `service_name` | Tracks lifecycle completion actions |
| `app_items_deleted_total` | Counter | `lifecycle_type`, `deployment_environment`, `service_version`, `service_name` | Tracks lifecycle deletion actions |
| `app_lifecycle_items_current` | Gauge | `lifecycle_type`, `status` (`active`, `completed`, `overdue`), `deployment_environment`, `service_version`, `service_name` | Current database inventory of items |
| `app_user_signups_total` | Counter | `deployment_environment`, `service_version`, `service_name` | User account registration volume |
| `app_user_logins_total` | Counter | `status` (`success`, `failure`), `deployment_environment`, `service_version`, `service_name` | Authentication traffic and credential failure rate |
| `app_groups_created_total` | Counter | `deployment_environment`, `service_version`, `service_name` | Shared group collaboration count |
| `app_group_members_invited_total` | Counter | `deployment_environment`, `service_version`, `service_name` | Group membership invitations |
| `app_digests_generated_total` | Counter | `status` (`success`, `failure`), `deployment_environment`, `service_version`, `service_name` | Scheduled morning email digest runs |
| `app_reminders_summary_requests_total` | Counter | `deployment_environment`, `service_version`, `service_name` | In-app reminders drawer views |

---

## 5. Excluded / Filtered Routes

Health checks and internal scraping endpoints are excluded from HTTP duration and latency metrics to prevent skew and log spam:
- `/health`
- `/healthz`
- `/metrics`
- `/favicon.ico`
- `/docs`, `/redoc`, `/openapi.json`

---

## 6. Actionable Prometheus Alerting Rules (`observability/alert_rules.yml`)

Prometheus evaluates alerting rules every 5s with mandatory `service`, `environment`, `version` / `deployed_version`, `owner`, and `severity` labels:

1. **`HighHttpErrorRate`**: Triggers when 5xx errors exceed 5% of all requests over 1 minute.
2. **`HighHttpLatencyP99`**: Triggers when p99 request latency exceeds 1000ms over 2 minutes.
3. **`DailyDigestFailures`**: Triggers when daily digest email generation fails.
4. **`RepeatedItemCreationFailures`**: Triggers when lifecycle item creation failure rate exceeds 5% over 5 minutes sustained for 2 minutes (`for: 2m`). Represents direct real user impact on users attempting to save renewals, maintenance tasks, or warranties. Includes `service`, `environment`, `deployed_version`, `owner` (`backend-team`), and Grafana `dashboard_url`.
5. **`LifecycleItemCreationFailures`**: Triggers when lifecycle item creation failure rate exceeds 0.05 failures/second over 5 minutes sustained for 2 minutes (`for: 2m`).

---

## 7. Runbooks for Actionable Alerts

### `docs/telemetry.md#item-creation-failures`
- **Severity**: Critical
- **User Impact**: Users are unable to create and save lifecycle records (renewals, maintenance, warranties). Failure rate exceeds 5% sustained over 2 minutes.
- **Triage Steps**:
  1. Open the alert's `dashboard_url` to inspect current failure rates and identify the affected service version and environment.
  2. Inspect backend logs: `docker compose logs --tail=100 -f backend` or via Loki in Grafana (`{service_name="api-backend"} |= "items"`).
  3. Verify database connectivity and constraints in `backend/app/store.py` or database migrations.
  4. If a regression was introduced in the deployed version, rollback to the previous stable release or apply a hotfix.

---

## 7. Grafana Dashboard Panels & Dynamic Filtering (`observability/grafana/dashboards/lifecycle-tracker-overview.json`)

The pre-provisioned Grafana dashboard (`lifecycle-tracker-overview.json`) provides comprehensive observability into application health, HTTP traffic, and core product metrics, with top-level dynamic filtering:

### A. Dynamic Template Variables (Cascading Environment & Version Filters)
- **`$environment`**: Populated via `label_values(deployment_environment)`. Multi-select enabled, includes `All` option.
- **`$version`**: Populated via `label_values({deployment_environment=~"$environment"}, service_version)`. Dynamically cascades based on selected environment. Multi-select enabled, includes `All` option.

### B. Dashboard Panels & Metric Hierarchy
1. **Row 100: Application Health & Golden Signals**
   - **OTel Collector Status**: Collector scrape target operational status (`up{job="otel-collector"}`).
   - **Backend Service & Active Version**: FastAPI service health and active version metadata (`app_info{deployment_environment=~"$environment", service_version=~"$version"}`).
   - **Ingested Span Rate**: Span ingestion rate via Collector (`otelcol_receiver_accepted_spans`).
   - **Ingested Log Records Rate**: Log records ingestion rate (`otelcol_receiver_accepted_log_records`).
2. **Row 101: Application Traffic & HTTP Performance**
   - **HTTP Request Rate by Route**: RPS grouped by route and status code (`app_http_server_duration_milliseconds_count{deployment_environment=~"$environment", service_version=~"$version"}`).
   - **HTTP Server Latency Percentiles**: p50, p90, and p99 request latency percentiles (`app_http_server_duration_milliseconds_bucket{deployment_environment=~"$environment", service_version=~"$version"}`).
3. **Row 102: Product Metrics — Renewals, Maintenance, Warranties & Reminders**
   - **Personal Lifecycle Tracker — Product & Domain Metrics Overview**: Master KPI stat panel displaying all core product counters (`app_items_created_total`, `app_items_completed_total`, `app_items_deleted_total`, active/overdue `app_lifecycle_items_current`, `app_user_signups_total`, success/failed `app_user_logins_total`, `app_groups_created_total`, `app_group_members_invited_total`, `app_digests_generated_total`, `app_reminders_summary_requests_total`).
   - **Lifecycle Items Created by Type**: Time-series creation volume across renewals, maintenance, and warranties (`app_items_created_total`).
   - **Lifecycle Items Completed by Type**: Time-series completion actions (`app_items_completed_total`).
   - **Lifecycle Items Deleted by Type**: Time-series item deletion and churn (`app_items_deleted_total`).
   - **Daily Email Digests & In-App Reminders**: Scheduled morning email digest runs and in-app reminders drawer views (`app_digests_generated_total`, `app_reminders_summary_requests_total`).
   - **User Accounts & Collaboration Activity**: User signups, logins, groups created, and member invitations (`app_user_signups_total`, `app_user_logins_total`, `app_groups_created_total`, `app_group_members_invited_total`).
   - **Current Items by Status & Type**: Current item status distribution (`app_lifecycle_items_current`).
   - **Lifecycle Operations Rate (Created vs Completed vs Deleted)**: Real-time throughput comparing creation, completion, and deletion rates.