personal-lifecycle-tracker / _docs / telemetry.md
telemetry.md
Raw

OpenTelemetry Architecture & Telemetry Specification

Project: Personal Lifecycle Tracker
Lifecycle Stage: Gate 14 — OpenTelemetry Full-Stack Instrumentation & Decoupled Observability

1. Golden Triangle Resource Attributes

Every telemetry signal (traces, metrics, logs) generated by any service in the Personal Lifecycle Tracker stack MUST include the standard semantic conventions:

Resource Attribute Semantic Key Description Example Values
Service Name service.name Logical identifier of the component api-backend, web-frontend, caddy-proxy
Environment deployment.environment Deployment lifecycle stage development, staging, production
Deployed Version service.version Release version or commit SHA dev-latest, 20261010-123320-16fea64, v1.0.0

Environment Variables

OTEL_SERVICE_NAME=api-backend
OTEL_RESOURCE_ATTRIBUTES=service.name=api-backend,deployment.environment=development,service.version=dev-latest
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4328
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

2. Distributed Tracing Topology (W3C Trace Context)

Distributed traces propagate correlation headers across all boundaries using the W3C Trace Context specification (traceparent and tracestate):

[Browser / React Web Client]
       │
       │ (W3C traceparent injected by FetchInstrumentation)
       ▼
[Production Caddy 2 Reverse Proxy]
       │
       │ (traceparent header passed through, X-Request-ID attached)
       ▼
[FastAPI Backend Application]
       │
       │ (FastAPIInstrumentor starts server span, extracts traceparent)
       ▼
[PostgreSQL Database (SQLAlchemy / psycopg2)]
         (Child DB span: query, table, operation latency)

3. Decoupled Observability Architecture (observability/)

The observability infrastructure is isolated in observability/docker-compose.yaml with its own lifecycle, network (personal-lifecycle-observability-net), and persistence:

  • OpenTelemetry Collector Contrib (otel/opentelemetry-collector-contrib:0.95.0):
    • Ingests traces, metrics, and logs via OTLP gRPC (:4317 internal / :4327 host) and OTLP HTTP (:4318 internal / :4328 host).
    • Routes traces to Tempo (tempo:4317).
    • Converts and exposes metrics to Prometheus (:8889 internal / :8899 host).
    • Routes logs to Loki (http://loki:3100/loki/api/v1/push).
  • Prometheus (prom/prometheus:v2.51.0):
    • Scrapes metrics on port :9090 (host :9091).
  • Loki (grafana/loki:2.9.4):
    • Aggregates logs on port :3100 (host :3101).
  • Tempo (grafana/tempo:2.4.1):
    • Stores distributed trace spans on port :3200 (host :3201).
  • Grafana (grafana/grafana:10.4.0):
    • Serves unified visualizations on port :3000 (host :3004).
    • Pre-provisioned datasources: Prometheus, Loki (with traceId derived field), Tempo (with tracesToLogsV2 and tracesToMetrics).
    • Pre-provisioned dashboard: lifecycle-tracker-overview.json.

4. Application Metrics & Product Indicators

All application metrics emitted by api-backend attach the mandatory Golden Triangle labels (deployment_environment, service_version, service_name).

A. Metadata & Golden Signals

Metric Name Instrument Labels Purpose
app_info Gauge deployment_environment, service_version, service_name Instantaneous metadata indicator reporting active environment and deployed version
app_http_server_duration_milliseconds Histogram http_route, http_status_code, http_method, deployment_environment, service_version, service_name Latency distribution percentiles (p50, p90, p99) by normalized route
app_http_requests_total Counter http_route, http_status_code, http_method, deployment_environment, service_version, service_name HTTP request throughput and error counts

B. Product & Domain Metrics (Personal Lifecycle Tracker)

Metric Name Instrument Labels Purpose
app_items_created_total Counter lifecycle_type (renewal, maintenance, warranty), deployment_environment, service_version, service_name Tracks lifecycle item creation by category
app_item_creation_failures_total Counter lifecycle_type, deployment_environment, service_version, service_name Tracks failed lifecycle item creation attempts
app_items_completed_total Counter lifecycle_type, deployment_environment, service_version, service_name Tracks lifecycle completion actions
app_items_deleted_total Counter lifecycle_type, deployment_environment, service_version, service_name Tracks lifecycle deletion actions
app_lifecycle_items_current Gauge lifecycle_type, status (active, completed, overdue), deployment_environment, service_version, service_name Current database inventory of items
app_user_signups_total Counter deployment_environment, service_version, service_name User account registration volume
app_user_logins_total Counter status (success, failure), deployment_environment, service_version, service_name Authentication traffic and credential failure rate
app_groups_created_total Counter deployment_environment, service_version, service_name Shared group collaboration count
app_group_members_invited_total Counter deployment_environment, service_version, service_name Group membership invitations
app_digests_generated_total Counter status (success, failure), deployment_environment, service_version, service_name Scheduled morning email digest runs
app_reminders_summary_requests_total Counter deployment_environment, service_version, service_name In-app reminders drawer views

5. Excluded / Filtered Routes

Health checks and internal scraping endpoints are excluded from HTTP duration and latency metrics to prevent skew and log spam:

  • /health
  • /healthz
  • /metrics
  • /favicon.ico
  • /docs, /redoc, /openapi.json

6. Actionable Prometheus Alerting Rules (observability/alert_rules.yml)

Prometheus evaluates alerting rules every 5s with mandatory service, environment, version / deployed_version, owner, and severity labels:

  1. HighHttpErrorRate: Triggers when 5xx errors exceed 5% of all requests over 1 minute.
  2. HighHttpLatencyP99: Triggers when p99 request latency exceeds 1000ms over 2 minutes.
  3. DailyDigestFailures: Triggers when daily digest email generation fails.
  4. RepeatedItemCreationFailures: Triggers when lifecycle item creation failure rate exceeds 5% over 5 minutes sustained for 2 minutes (for: 2m). Represents direct real user impact on users attempting to save renewals, maintenance tasks, or warranties. Includes service, environment, deployed_version, owner (backend-team), and Grafana dashboard_url.
  5. LifecycleItemCreationFailures: Triggers when lifecycle item creation failure rate exceeds 0.05 failures/second over 5 minutes sustained for 2 minutes (for: 2m).

7. Runbooks for Actionable Alerts

docs/telemetry.md#item-creation-failures

  • Severity: Critical
  • User Impact: Users are unable to create and save lifecycle records (renewals, maintenance, warranties). Failure rate exceeds 5% sustained over 2 minutes.
  • Triage Steps:
    1. Open the alert's dashboard_url to inspect current failure rates and identify the affected service version and environment.
    2. Inspect backend logs: docker compose logs --tail=100 -f backend or via Loki in Grafana ({service_name="api-backend"} |= "items").
    3. Verify database connectivity and constraints in backend/app/store.py or database migrations.
    4. If a regression was introduced in the deployed version, rollback to the previous stable release or apply a hotfix.

7. Grafana Dashboard Panels & Dynamic Filtering (observability/grafana/dashboards/lifecycle-tracker-overview.json)

The pre-provisioned Grafana dashboard (lifecycle-tracker-overview.json) provides comprehensive observability into application health, HTTP traffic, and core product metrics, with top-level dynamic filtering:

A. Dynamic Template Variables (Cascading Environment & Version Filters)

  • $environment: Populated via label_values(deployment_environment). Multi-select enabled, includes All option.
  • $version: Populated via label_values({deployment_environment=~"$environment"}, service_version). Dynamically cascades based on selected environment. Multi-select enabled, includes All option.

B. Dashboard Panels & Metric Hierarchy

  1. Row 100: Application Health & Golden Signals
    • OTel Collector Status: Collector scrape target operational status (up{job="otel-collector"}).
    • Backend Service & Active Version: FastAPI service health and active version metadata (app_info{deployment_environment=~"$environment", service_version=~"$version"}).
    • Ingested Span Rate: Span ingestion rate via Collector (otelcol_receiver_accepted_spans).
    • Ingested Log Records Rate: Log records ingestion rate (otelcol_receiver_accepted_log_records).
  2. Row 101: Application Traffic & HTTP Performance
    • HTTP Request Rate by Route: RPS grouped by route and status code (app_http_server_duration_milliseconds_count{deployment_environment=~"$environment", service_version=~"$version"}).
    • HTTP Server Latency Percentiles: p50, p90, and p99 request latency percentiles (app_http_server_duration_milliseconds_bucket{deployment_environment=~"$environment", service_version=~"$version"}).
  3. Row 102: Product Metrics — Renewals, Maintenance, Warranties & Reminders
    • Personal Lifecycle Tracker — Product & Domain Metrics Overview: Master KPI stat panel displaying all core product counters (app_items_created_total, app_items_completed_total, app_items_deleted_total, active/overdue app_lifecycle_items_current, app_user_signups_total, success/failed app_user_logins_total, app_groups_created_total, app_group_members_invited_total, app_digests_generated_total, app_reminders_summary_requests_total).
    • Lifecycle Items Created by Type: Time-series creation volume across renewals, maintenance, and warranties (app_items_created_total).
    • Lifecycle Items Completed by Type: Time-series completion actions (app_items_completed_total).
    • Lifecycle Items Deleted by Type: Time-series item deletion and churn (app_items_deleted_total).
    • Daily Email Digests & In-App Reminders: Scheduled morning email digest runs and in-app reminders drawer views (app_digests_generated_total, app_reminders_summary_requests_total).
    • User Accounts & Collaboration Activity: User signups, logins, groups created, and member invitations (app_user_signups_total, app_user_logins_total, app_groups_created_total, app_group_members_invited_total).
    • Current Items by Status & Type: Current item status distribution (app_lifecycle_items_current).
    • Lifecycle Operations Rate (Created vs Completed vs Deleted): Real-time throughput comparing creation, completion, and deletion rates.
Powered by GitFront