Grafana Tempo

Grafana Tempo is a distributed tracing backend from Grafana Labs. It stores the spans that instrumented services emit, so an operator can follow one request across every service it touched and see where the time went. Tempo keeps no index of spans: it writes traces as columnar blocks to object storage and reads only the columns a query needs, which keeps long retention cheap. The official docs at grafana.com/docs/tempo own the full configuration reference.

What it is

A Go server, shipped as one binary and one container image (grafana/tempo), that receives spans in OpenTelemetry (OTLP), Jaeger or Zipkin format, groups them into traces, and stores them as Apache Parquet blocks. Queries come in two shapes: a lookup by trace ID, and TraceQL, a query language that selects spans by their attributes, duration and structure and can also compute metrics from them.

Licence is AGPL-3.0. The current release line is 3.1 (v3.1.0 shipped 2026-09-29). The 3.0 line still gets patches: v3.0.4 shipped on 2026-10-16 with Go, gRPC and other dependency security updates. Grafana Cloud Traces is the hosted, paid version.

Tempo is the "T" in Grafana Labs' LGTM family: Loki for logs, Grafana for dashboards, Tempo for traces, Mimir for metrics, with Alloy as the collector. Two recent releases changed how Tempo runs:

• Tempo 3.0 (2026-05-28) replaced the write path. Ingesters and the compactor are gone. A live-store now serves recent data, and a backend scheduler and worker handle compaction and retention. In microservices mode a Kafka-compatible queue sits between the distributor and its consumers; monolithic mode needs no Kafka. TraceQL metrics became generally available. There is no in-place downgrade to 2.x.
• Tempo 3.1 (2026-09-29) writes vParquet5 blocks by default and refuses to start if configured to write vParquet3. It also added redaction jobs that select traces with a TraceQL query and arithmetic in TraceQL metrics.

The Helm charts now live in the grafana-community/helm-charts repository: tempo for monolithic mode (chart 3.2.0, appVersion 3.1.0 at the time of writing) and tempo-distributed for microservices mode.

How it works

The distributor is the entry point. It accepts spans on the OTLP gRPC (4317) and HTTP (4318) ports and on the Jaeger and Zipkin receivers, checks per-tenant limits, and shards the spans by trace ID. In monolithic mode it pushes them in-process to the live-store, which keeps recent traces in memory and in a local write-ahead log so they are searchable within seconds. The live-store then cuts Parquet blocks and flushes them to the storage backend. In microservices mode the distributor writes to Kafka instead, and separate block-builders turn the queue into blocks.

On the read path, the query frontend receives a TraceQL query or a trace-ID lookup on port 3200, splits it into jobs, and sends them to queriers. Queriers read recent data from the live-store and older data from the blocks, and the frontend merges the results. The backend scheduler creates compaction and retention jobs, and backend workers run them: they merge small blocks into larger ones and delete blocks past the retention period. An optional metrics-generator derives request-rate, error and duration metrics and service graphs from the spans and remote-writes them to Prometheus.

The design choice that defines Tempo is the missing index. A trace store that indexes every span attribute grows its index with traffic. Tempo relies on a bloom filter for trace-ID lookups and on columnar Parquet reads for searches, so storage stays cheap and the cost moves to query time. Broad TraceQL searches across long time ranges read many blocks; narrow ones with a service name and a short window stay fast.

When to use it

Reach for it when:

• Grafana is already the dashboard, and traces should link to the logs in Loki and the metrics in Prometheus for the same service and time.
• Services are instrumented with OpenTelemetry, or that work is planned. Tempo accepts OTLP directly.
• Trace volume is high or retention should be long, and object storage is the cheapest place to keep it.
• Most questions start from a trace ID found in a log line or an exemplar, or from a TraceQL search on a service and a latency threshold.

Skip it when:

• No service emits traces yet. Instrumentation is the real work, and a backend without spans adds nothing.
• The team wants a tracing UI outside Grafana. Tempo has no UI of its own.
• AGPL-3.0 is not acceptable for the use case, for example when modifying Tempo and offering it as a network service.
• A managed tracing service is already paid for and nobody wants to run another stateful service.

At a glance

Core concepts

• Trace and span: a trace is the tree of spans for one request; a span is one timed operation with attributes, events and a status.
• TraceQL: the query language. A spanset selector such as {span.http.status_code >= 500} combines with structural operators (>> descendant, > child, ~ sibling) and aggregates.
• TraceQL metrics: queries such as {resource.service.name="api"} | rate() or quantile_over_time(duration, .99) that turn spans into time series. Generally available since 3.0.
• Live-store: the 3.x component that serves recent data from memory and a WAL. It replaced the ingester.
• Block and vParquet5: a block is one Parquet file set for a tenant and time window. vParquet5 is the default write format since 3.1; vParquet4 and vParquet3 blocks stay readable.
• Metrics-generator: an optional component that builds span metrics and service graphs and remote-writes them to Prometheus.
• Tenant: Tempo is multi-tenant. With multitenancy off, every request goes to one tenant called single-tenant.

Key facts

Aspect	Value
Ingest protocols	OTLP gRPC (4317), OTLP HTTP (4318), Jaeger, Zipkin
Query API	HTTP on port 3200 (/api/traces/<id>, /api/search, /api/metrics/query_range)
Storage	S3 and S3-compatible, GCS, Azure Blob; local disk for development
Deployment modes	Monolithic (one process, no Kafka) or microservices (needs Kafka-compatible queue)
Default retention	336h (14 days) in Tempo; 24h in the monolithic Helm chart
Licence	AGPL-3.0

Compared with other trace stores

Aspect	Tempo	Jaeger	Zipkin
Index	None; bloom filter plus columnar Parquet reads	Uses the index of its storage backend	Uses the index of its storage backend
Storage	Object storage	Cassandra, Elasticsearch, OpenSearch, Badger	Cassandra, Elasticsearch, MySQL, in-memory
Query	TraceQL and TraceQL metrics	Search by service, operation, tags, duration	Search by service, span name, tags, duration
UI	Grafana	Jaeger UI (Grafana also reads it)	Zipkin UI
Licence	AGPL-3.0	Apache 2.0	Apache 2.0

How to integrate

Install the monolithic chart from its new home. This values file turns on streaming and the metrics-generator, sets an explicit retention, and keeps the WAL and blocks on a persistent volume. Swap the local backend for s3 with a bucket and endpoint for anything beyond a small setup:

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm install tempo grafana-community/tempo -n tempo --create-namespace -f values.yaml

# values.yaml
tempo:
  streamOverHttpEnabled: true   # Grafana streaming needs stream_over_http_enabled
  reportingEnabled: false
  retention: 168h               # 7 days; rendered into backend_worker.compaction.block_retention
  storage:
    trace:
      backend: local
      local:
        path: /var/tempo/traces
      wal:
        path: /var/tempo/wal
  metricsGenerator:
    enabled: true
    remoteWriteUrl: http://prometheus.monitoring:9090/api/v1/write
  overrides:
    defaults:
      metrics_generator:
        processors: [service-graphs, span-metrics]
persistence:
  enabled: true
  size: 20Gi

Point an OpenTelemetry Collector at Tempo over OTLP. The OpenTelemetry Operator manages a collector like this one as a custom resource:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: traces
spec:
  mode: deployment
  config:
    receivers:
      otlp:
        protocols:
          grpc: {endpoint: 0.0.0.0:4317}
          http: {endpoint: 0.0.0.0:4318}
    processors:
      batch: {}
    exporters:
      otlp:
        endpoint: tempo.tempo:4317
        tls: {insecure: true}
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp]

Provision the Grafana data source with links to logs and to the service graph. The datasourceUid values must match the UIDs of the Loki and Prometheus data sources:

apiVersion: 1
datasources:
  - name: Tempo
    type: tempo
    uid: tempo
    url: http://tempo.tempo:3200
    access: proxy
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        spanStartTimeShift: '-5m'
        spanEndTimeShift: '5m'
        filterByTraceID: true
        tags: [{key: 'k8s.namespace.name', value: 'namespace'}]
      serviceMap:
        datasourceUid: prometheus
      nodeGraph:
        enabled: true
      streamingEnabled:
        search: true
        metrics: true

Then try TraceQL in Explore:

{resource.service.name="api" && status=error}
{span.http.route="/checkout"} | quantile_over_time(duration, .95) by (resource.service.name)

In the GL stack

Tempo 3.1.0 runs on the management cluster as the trace store, next to Loki 3.7.8 for logs and the Prometheus v3.15.0 hub for metrics. Grafana 13.2.4 on the same cluster has Tempo as a data source alongside Prometheus and Loki, so the three signals share one query surface. Sign-in to Grafana goes through Keycloak.

Traces come from the beta cluster today. The OpenTelemetry Operator 0.161.0 runs there only, as a trial before prod. It manages the collectors and the auto-instrumentation of workloads, and the traces it collects go to Tempo on the management cluster. Logs and metrics already cover every cluster; tracing follows the same pattern of one central store with collectors on each cluster, starting with beta.

The choice matches the rest of the observability layer: one Grafana Labs store per signal on the management cluster, with Grafana as the single place to query them. Keeping the trace store next to Loki means a trace can link to the log lines of the same pod and time window without a second login or a second tool.

Gotchas

• Read the 3.0 migration guide before any 2.x upgrade. Ingesters, the compactor target and the scalable-single-binary mode are removed, and Tempo refuses to start with legacy flat overrides unless enable_legacy_overrides: true is set. tempo-cli migrate config --mode=monolithic old.yaml > new.yaml rewrites a monolithic config. There is no in-place downgrade.
• Chart retention changed in 3.2.0. In chart 3.0.0 through 3.1.0, tempo.retention only reached the scheduler, and the worker kept Tempo's default of 14 days. From 3.2.0 the chart default of 24h takes effect, and older blocks are deleted on the next retention run. Set tempo.retention or tempo.backendWorker.compaction.block_retention before upgrading.
• vParquet3 is no longer writable. Tempo 3.1 refuses to start if storage.trace.block.version is vParquet3. Change it to vParquet4 or vParquet5 first; existing vParquet3 blocks stay readable but are no longer compacted.
• Remove local-blocks from metrics-generator processors before upgrading to 3.0. The processor is gone, and the chart value tempo.metricsGenerator.traces_storage with it.
• Streaming is off in the chart. streamOverHttpEnabled defaults to false, and Grafana's streaming search and metrics need stream_over_http_enabled: true on the Tempo side.
• Recent-data queries fail when a live-store lags. Since 3.0, live_store.fail_on_high_lag defaults to true and query_end_cutoff to 30s, so Tempo returns an error instead of a partial result. Treat these errors as a signal to check the live-store, not as a bug in the query.
• Large traces get cut. Spans past the tenant's max_bytes_per_trace are discarded. A runaway loop that creates thousands of spans in one trace hits this first.
• Check the patch lines. v3.0.4 (2026-10-16) carries Go and gRPC security updates. At the time of writing no 3.1 patch release had shipped, so check the releases page for a 3.1.x before treating 3.1.0 as current on security fixes.
• The chart reports usage by default. reportingEnabled: true sends anonymous deployment-shape data to Grafana Labs. Set it to false for an air-gapped or private setup.

Risks

• Licence. AGPL-3.0 is fine for running Tempo internally. Teams that modify it and serve it to outside users must share their changes, and some legal teams block AGPL by policy.
• Project health. Strong: Grafana Labs staffs it, minor releases come every few months with patch lines for the previous minor, and the repository has about 5,500 GitHub stars.
• Churn. The 3.0 rewrite removed components, renamed flags and changed defaults, and the Helm charts moved to a new repository. Budget time to read the upgrade guide before each minor release.
• Lock-in. Low on the write side: OTLP is a standard, and a collector can send the same spans to another backend. Higher on the read side: TraceQL queries, dashboards and alert rules do not move to another engine unchanged.
• Security. Without multitenancy and an authenticating proxy, the ingest and query ports have no authentication. Span attributes can carry personal data; keep Tempo on an internal network and use the 3.1 redaction jobs or collector processors to remove sensitive fields.

Alternatives

• Jaeger: pick for a CNCF-graduated, Apache 2.0 tracing system with its own UI, on Cassandra, Elasticsearch or OpenSearch storage that a team already runs.
• Zipkin: pick for a small, mature tracer with a simple UI and wide library support, when TraceQL and object storage do not matter.
• SigNoz or other OpenTelemetry-native stacks on ClickHouse: pick when one product should hold traces, logs and metrics with SQL-style queries.
• A managed tracing service (Grafana Cloud Traces, AWS X-Ray): pick when nobody should run a stateful trace store.

Related

• Grafana: the query surface for traces, logs and metrics.
• Grafana Loki: the log store that trace views link to.
• Prometheus: receives span metrics and service graphs from the metrics-generator.
• OpenTelemetry Operator: collectors and auto-instrumentation that send traces to Tempo.
• Grafana Alloy: Grafana's collector, which can also receive and forward OTLP traces.
• Tempo documentation
• Tempo releases on GitHub
• Migrate to Tempo 3
• TraceQL reference