Build Daily
Tinley Park · August 23, 2026Airflow + OpenTelemetry: Trace Your Pipelines Locally
This video demonstrates how to set up local observability for Apache Airflow pipelines using OpenTelemetry, Tempo, Prometheus, and Grafana. It covers configuring tracing, instrumenting custom code with spans and metrics, and visualizing performance bottlenecks in a flame graph.
The outline
Tap a timestamp to jump there in the video.
Introduction to Pipeline Tracing
The video introduces the problem of slow data pipelines and proposes using OpenTelemetry to trace Airflow tasks locally.
Architecture Overview
The presenter outlines the local Docker stack architecture where Airflow emits traces to an OpenTelemetry collector that fans out to Tempo and Prometheus for Grafana visualization.
Demo Pipeline Structure
A simple five-task ELT pipeline is introduced, featuring tasks like API extraction, validation, transformation, loading, and metric publishing.
Instrumenting Tasks with Spans
The code for the demo pipeline is examined to show how manual spans are created around task logic and enriched with attributes like HTTP methods and row counts.
Simulating Performance Bottlenecks
Nested spans are used within a transform task to simulate a slow step, specifically an 'enrich' operation that sleeps for eight seconds to create a visible delay in the trace.
Configuring Airflow Tracing
Environment variables are configured to enable Airflow's built-in tracing and point it to the local OpenTelemetry collector using specific gRPC and HTTP ports for traces and metrics.
Custom Instrumentation Toolkit
A utility file is created to provide helpers for creating custom tracers, flushing spans before task completion, and recording SLI metrics like rows processed and data freshness.
Context Propagation Across Boundaries
Functions are implemented to inject and extract W3C trace context headers, allowing traces to propagate across network boundaries between different services or tasks.
Visualizing Traces in Grafana
The local Grafana instance is used to search for the pipeline's trace ID, revealing a detailed view of task durations and nested spans within the Airflow DAG run.
OpenTelemetry Collector Configuration
The collector's YAML configuration is reviewed to show how it receives telemetry on specific ports and routes traces to Tempo while republishing metrics for Prometheus scraping.
Tracing pipeline delays before they reach production helps a solo builder spot slow tasks early. Pairing Airflow with OpenTelemetry and Grafana makes performance bottlenecks visible without relying on external teams.