📊 Observability Stack with OpenTelemetry, Prometheus, Grafana & FastAPI (Docker)

April 15, 2025 Pedro Martins

This project sets up a complete observability pipeline using Docker Compose, including:

OpenTelemetry Collector – Receives and processes telemetry data (both metrics and traces)
Prometheus – Scrapes and stores metrics data
Grafana – Visualizes the metrics with dashboards
FastAPI Application – An instrumented web application that sends trace data to the Collector

For additional examples and extended OpenTelemetry solutions, check out the Open-Telemetry-Solutions repository.

📁 Project Structure

The repository is organized as follows:

docker-compose.yml: Defines all four services (Collector, Prometheus, Grafana, FastAPI).
otel-collector-config.yaml: Configures the Collector to receive OTLP telemetry and export metrics and traces.
prometheus.yml: Configures Prometheus to scrape metrics from the Collector.
datasource.yml: Provisions Grafana with a Prometheus datasource.
fastapi-app/: Contains the FastAPI application that sends trace data to the Collector.

🚀 Getting Started

1. Clone the Repository

2. Run Docker Compose

Start all services with:

This command will launch:

OpenTelemetry Collector on ports 4317 (gRPC) and 4318 (HTTP)
Prometheus on port 9090
Grafana on port 3000
FastAPI Application on port 8000

🛠️ Configuration Files

docker-compose.yml

The Compose file integrates all the services together. Notice that the FastAPI service is built from the local directory and configured to send trace data to the Collector endpoint:

yaml

version: '3.8'
services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml
    ports:
      - "1888:1888"   # pprof for performance profiling
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "8890:8890"   # Prometheus metrics exported by the Collector
      - "8889:8889"   # Prometheus exporter metrics (if needed)
      - "13133:13133" # Health check endpoint

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    ports:
      - "3000:3000"

  fastapi-app:
    build: ./fastapi-app
    ports:
      - "8000:8000"
    environment:
      - OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318/v1/traces
    depends_on:
      - otel-collector

otel-collector-config.yaml

This file configures the Collector to receive OTLP data for both metrics and traces and to send metrics to Prometheus. Traces are sent to a debug exporter which logs them for easy verification.

prometheus.yml

This file instructs Prometheus to scrape metrics from the Collector’s Prometheus exporter:

grafana/provisioning/datasources/datasource.yml

Grafana is automatically provisioned with Prometheus as its default datasource:

🚀 FastAPI Application

The FastAPI service is instrumented with OpenTelemetry to send trace data to the Collector.

fastapi-app/Dockerfile

fastapi-app/requirements.txt

fastapi-app/main.py

python

from fastapi import FastAPI
from opentelemetry import trace
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

app = FastAPI()

# Configure OpenTelemetry for FastAPI
resource = Resource(attributes={SERVICE_NAME: "fastapi-service"})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

# Configure the OTLP exporter to send traces to the Collector.
otlp_exporter = OTLPSpanExporter(
    endpoint="http://otel-collector:4318/v1/traces"
)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)

# Instrument FastAPI routes automatically
FastAPIInstrumentor.instrument_app(app)

@app.get("/")
async def read_root():
    return {"message": "Hello from FastAPI"}

@app.get("/trigger")
async def trigger_trace():
    tracer = trace.get_tracer(__name__)
    with tracer.start_as_current_span("manual-span"):
        return {"message": "Manual trace span sent to the collector"}

🖥️ Access the Interfaces

Grafana: http://localhost:3000
Login: admin / admin
Prometheus: http://localhost:9090
FastAPI Application: http://localhost:8000
The / endpoint returns a welcome message, and /trigger generates a manual trace.

📡 Sending Telemetry

Your instrumented FastAPI app automatically sends trace data to the Collector at:

http://otel-collector:4318/v1/traces (within Docker networks)

From outside Docker, you can test using:

http://localhost:4318/v1/traces (if you map the port externally)

For metrics, send to:

http://localhost:4318/v1/metrics (HTTP) or
grpc://localhost:4317 (gRPC)

You can use your own instrumented applications or tools like otel-cli.

⚠️ Port 55679 Error Fix (zPages)

If you encounter this error:

Solutions:

Option 1: Remove the line mapping port 55679 from docker-compose.yml if you're not using zPages.
Option 2: Change it to a free port, for example:

yaml

- "55678:55679"

Then restart with:

🧱 Build Dashboards in Grafana

Once your metrics are coming in:

Log into Grafana at http://localhost:3000.
Create new dashboards to visualize key metrics such as:
- Request rates
- Custom app metrics
- Error counts
- And more!

📚 Additional Resources

Explore more solutions for OpenTelemetry and observability by checking out the Open-Telemetry-Solutions repository. It contains extended examples, integrations, and best practices you might find useful for your projects.