Observability Infrastructure — Personal Lifecycle Tracker

A decoupled, production-grade observability stack containing OpenTelemetry Collector, Prometheus, Loki, Tempo, and Grafana (LGTM + OTel).

1. Architecture & Design Principles

                              ┌──────────────────────────────────────────┐
                              │            Application Stack             │
                              │  (FastAPI Backend / React Vite Frontend) │
                              └────────────────────┬─────────────────────┘
                                                   │
                              OTLP Traces / Metrics / Logs (HTTP 4328 / gRPC 4327)
                                                   ▼
┌────────────────────────────────────────────────────────────────────────────────────────┐
│                              Observability Compose Stack                               │
│                                                                                        │
│                         ┌────────────────────────────────────┐                         │
│                         │   OpenTelemetry Collector Contrib  │                         │
│                         │   (receivers: OTLP gRPC/HTTP)      │                         │
│                         └───────┬────────────┬───────────┬───┘                         │
│                                 │            │           │                             │
│                     Traces      │    Metrics │           │ Logs                        │
│                     (OTLP)      │   (Scrape) │           │ (Loki push)                 │
│                                 ▼            ▼           ▼                             │
│                           ┌───────────┐ ┌──────────┐ ┌──────────┐                      │
│                           │   Tempo   │ │Prometheus│ │   Loki   │                      │
│                           │  (Traces) │ │(Metrics) │ │  (Logs)  │                      │
│                           └─────┬─────┘ └────┬─────┘ └────┬─────┘                      │
│                                 │            │            │                            │
│                                 └────────────┼────────────┘                            │
│                                              ▼                                         │
│                                       ┌──────────────┐                                 │
│                                       │   Grafana    │                                 │
│                                       │ (Dashboards) │                                 │
│                                       └──────────────┘                                 │
└────────────────────────────────────────────────────────────────────────────────────────┘

Decoupled Lifecycle: The observability stack runs as an independent Docker Compose project (personal-lifecycle-observability). Application reloads, restarts, or crashes never disrupt telemetry collection or wipe dashboard history.
Cross-Pillar Correlation: Pre-provisioned Grafana datasources link traces directly to relevant logs (Tempo $\to$ Loki), logs directly to traces (Loki $\to$ Tempo via TraceID), and metrics to traces (Tempo $\to$ Prometheus).
Collision-Resistant Ports: Configured with non-colliding default ports to operate smoothly alongside other services and stacks on the machine.

2. Service Endpoints & Port Map

Component	Service Name	Container Port	Host Port (Configurable)	Description
OpenTelemetry Collector	`otel-collector`	`4317`	`4327` (`${OTEL_GRPC_PORT:-4327}`)	OTLP gRPC Receiver
OpenTelemetry Collector	`otel-collector`	`4318`	`4328` (`${OTEL_HTTP_PORT:-4328}`)	OTLP HTTP Receiver
OpenTelemetry Collector	`otel-collector`	`8889`	`8899` (`${OTEL_METRICS_PORT:-8899}`)	Collector Prometheus Metrics Exporter
OpenTelemetry Collector	`otel-collector`	`8888`	Internal	Collector Internal Health & Pipeline Telemetry
Prometheus	`prometheus`	`9090`	`9091` (`${PROMETHEUS_PORT:-9091}`)	Metrics Storage & Query Engine
Loki	`loki`	`3100`	`3101` (`${LOKI_PORT:-3101}`)	Log Aggregation Engine
Tempo	`tempo`	`3200`	`3201` (`${TEMPO_PORT:-3201}`)	Distributed Tracing Storage
Grafana	`grafana`	`3000`	`3004` (`${GRAFANA_PORT:-3004}`)	Unified Dashboard & Explorer UI

Default Grafana Credentials: admin / admin

3. Directory Layout

observability/
├── docker-compose.yaml                 # Standalone Compose project definition
├── .env.example                        # Configurable port & container name overrides
├── otel-collector-config.yaml          # Collector pipelines routing traces, metrics & logs
├── prometheus.yml                      # Prometheus scrape targets
├── loki-config.yaml                    # Loki filesystem & TSDB config
├── tempo-config.yaml                   # Tempo local trace storage & OTLP ingestion
└── grafana/
    ├── dashboards/
    │   └── lifecycle-tracker-overview.json # Golden Signals overview dashboard
    └── provisioning/
        ├── dashboards/
        │   └── dashboards.yaml         # Auto-provisioning provider for dashboards
        └── datasources/
            └── datasources.yaml        # Correlated Prometheus, Loki & Tempo datasources

4. How to Run

Start the Observability Stack

# Using root Makefile
make obs-up

# Or directly with Docker Compose
docker compose -f observability/docker-compose.yaml up -d

Check Status & Health

make obs-ps
# or
docker compose -f observability/docker-compose.yaml ps

View Live Logs

make obs-logs
# Or specific component:
docker compose -f observability/docker-compose.yaml logs -f otel-collector

Tear Down

make obs-down
# or
docker compose -f observability/docker-compose.yaml down -v

5. Verification Traffic Test

To verify that the OpenTelemetry Collector and pipelines are actively processing telemetry:

# Send test log payload to Collector via OTLP HTTP
NOW_NANO=$(date +%s000000000)
curl -i -X POST http://localhost:4328/v1/logs \
  -H "Content-Type: application/json" \
  -d '{
    "resourceLogs": [{
      "resource": {
        "attributes": [
          {"key": "service.name", "value": {"stringValue": "verification-test"}},
          {"key": "deployment.environment", "value": {"stringValue": "development"}},
          {"key": "service.version", "value": {"stringValue": "dev-latest"}}
        ]
      },
      "scopeLogs": [{
        "scope": {"name": "test-logger"},
        "logRecords": [{
          "timeUnixNano": "'"${NOW_NANO}"'",
          "severityNumber": 9,
          "severityText": "INFO",
          "body": {"stringValue": "OpenTelemetry Collector verification test log"}
        }]
      }]
    }]
  }'

Check Collector logs to confirm receipt:

docker compose -f observability/docker-compose.yaml logs otel-collector

6. Connecting Applications

From the Host (Development / Hot-Reloading)

Configure application environment variables:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4328
OTEL_SERVICE_NAME=api-backend
OTEL_RESOURCE_ATTRIBUTES=service.name=api-backend,deployment.environment=development,service.version=dev-latest

From Docker Compose Application Stack

Attach the application services to the personal-lifecycle-observability-net network:

services:
  backend:
    environment:
      OTEL_EXPORTER_OTLP_ENDPOINT: http://personal-lifecycle-otel-collector:4318
    networks:
      - default
      - observability-net

networks:
  default:
  observability-net:
    external: true
    name: personal-lifecycle-observability-net