Grafana Loki

Grafana Loki is a log aggregation system from Grafana Labs. It collects logs from many hosts and pods into one store and lets an operator query them by the same labels that Prometheus uses for metrics (cluster, namespace, app), so a spike on a dashboard leads straight to the log lines from that service at that time. The official docs at grafana.com/docs/loki own the full configuration reference.

What it is

A Go server, shipped as one binary and one container image, that receives log lines over HTTP, groups them into streams by their label set, and writes them as compressed chunks plus a small index. The index holds only labels, never the text of the lines. Full-text work happens at query time, on the chunks that the label selector picks.

Licence is AGPL-3.0 (the project moved from Apache 2.0 in 2021). The current release line is 3.7 (3.7.0 shipped 2026-03-26; the latest patch is v3.7.8 from 2026-09-17), and the 3.6 line still gets patches (v3.6.18 on 2026-10-15). Grafana Cloud Logs is the hosted, paid version of the same code.

Loki belongs to Grafana Labs' "LGTM" open-source family: Loki for logs, Grafana for dashboards, Tempo for traces, Mimir for metrics, plus Alloy as the collector. Two recent changes matter for anyone starting today:

• Promtail is end of life as of 2026-03-02. Grafana Alloy is the supported agent, and Grafana ships a tool that converts a Promtail config to Alloy.
• The Helm chart moved. It now lives in the grafana-community/helm-charts repository with new version numbers (chart 18.x, appVersion 3.7.8 at the time of writing). The old 6.x chart from the Loki repo has a separate migration guide.

How it works

Loki is a set of components that can run as separate services or all inside one process. The write path starts at the distributor, which validates each push and hashes the stream labels to pick an ingester. The ingester keeps recent lines in memory (with a write-ahead log on disk), compresses them into chunks and flushes the chunks and index to storage. The read path starts at the query frontend and querier: the querier asks the index which chunks match the label selector, reads them, runs the LogQL filters and also asks the ingesters for lines not flushed yet. A compactor merges index files and, when enabled, deletes data past the retention period.

The design choice that defines Loki is the small index. A store that indexes every word costs more in disk and memory as volume grows. Loki pays at query time instead: a broad label selector over a long time range means many chunks to scan. Good label design keeps both sides small.

When to use it

Reach for it when:

• Grafana is already the dashboard, and logs should sit next to metrics and traces with the same labels.
• Most log questions take the form "this service, this time window, lines that contain X".
• Storage cost matters. Compressed chunks in object storage, or on one disk for a small setup, are cheap.
• Kubernetes is the platform, so pod metadata gives a clean, low-cardinality label set for free.
• One store should collect logs from several clusters, with a cluster label to tell them apart.

Skip it when:

• The main workload is full-text search or relevance ranking across all logs with no label filter. An inverted-index engine does that better.
• Logs carry high-cardinality fields (user IDs, request IDs) that must be indexed for fast lookup. Loki wants those as structured metadata or line content, not labels.
• AGPL-3.0 is not acceptable for the use case, for example when modifying Loki and offering it as a network service without sharing changes.
• A managed cloud log service is already paid for and nobody wants to run another stateful service.

At a glance

Core concepts

• Stream — all lines with the same exact label set. Every new label value makes a new stream.
• Label — an indexed key/value pair. Keep labels static and bounded: cluster, namespace, app, environment.
• Structured metadata — key/value pairs stored with each line but not indexed. The right place for trace IDs and other high-cardinality values. On by default since 3.0.
• Chunk — a compressed block of lines from one stream, flushed by the ingester.
• TSDB index + schema v13 — the current index format and storage schema. Structured metadata requires both.
• LogQL — the query language: a stream selector {app="api"}, then line filters (|=, !=, |~), parsers (| json, | logfmt) and metric functions (rate, count_over_time).
• Tenant — Loki is multi-tenant. With auth_enabled: false every request goes to one tenant called fake.

Deployment modes

Mode	What runs	Docs guidance
Monolithic (single binary)	All components in one process, -target=all	Small workloads, up to about 20 GB of logs per day. Two or more replicas need object storage.
Simple scalable	Separate read, write and backend targets	Close to 1 TB per day. Deprecated: removed in Loki 4.0.
Microservices	Each component as its own Deployment or StatefulSet	Very large clusters and fine-grained scaling. Most complex to run.

Compared with other log stores

Aspect	Loki	Elasticsearch / OpenSearch	VictoriaLogs
Index	Labels only	Full inverted index on fields and text	Indexes all fields with bloom filters
Query language	LogQL	Query DSL, KQL, SQL, PPL	LogsQL
Storage	Object store or local disk	Local disks on data nodes	Local disk
Licence	AGPL-3.0	Elastic: AGPL / SSPL / ELv2; OpenSearch: Apache 2.0	Apache 2.0
Fits best	Grafana-first stacks, label-scoped queries	Search-heavy analytics	Single-node simplicity, high-cardinality fields

How to integrate

Install the chart from its new home in monolithic mode. This values file runs one replica with object storage on an S3-compatible endpoint and turns retention on (it is off by default, and logs then live forever):

helm repo add grafana-community https://grafana-community.github.io/helm-charts
helm repo update
helm install loki grafana-community/loki -n loki --create-namespace -f values.yaml

# values.yaml
deploymentMode: Monolithic   # older charts called this SingleBinary
loki:
  auth_enabled: false
  commonConfig:
    replication_factor: 1
  schemaConfig:
    configs:
      - from: "2024-04-01"
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: loki_index_
          period: 24h        # retention needs a 24h index period
  storage:
    type: s3
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
    s3:
      endpoint: http://s3.example:8333
      region: us-east-1
      accessKeyId: CHANGE_ME      # or inject from a Secret
      secretAccessKey: CHANGE_ME
      s3ForcePathStyle: true
  limits_config:
    allow_structured_metadata: true
    retention_period: 336h   # 14 days
  compactor:
    retention_enabled: true
    delete_request_store: s3
singleBinary:
  replicas: 1
backend: {replicas: 0}
read: {replicas: 0}
write: {replicas: 0}
minio:
  enabled: false

Ship pod logs with Alloy. The cluster external label keeps logs from several clusters apart in one Loki:

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.relabel "pods" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
}

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.pods.output
  forward_to = [loki.write.central.receiver]
}

loki.write "central" {
  endpoint {
    url = "http://loki.example:3100/loki/api/v1/push"
  }
  external_labels = { cluster = "prod" }
}

Then add Loki as a Grafana data source (URL http://loki:3100 inside the cluster) and try a few LogQL queries:

{cluster="prod", namespace="web"} |= "error"
sum by (app) (rate({cluster="prod"} | json | level="error" [5m]))

In the GL stack

Loki 3.7.8 runs on the management cluster as a single binary and is the central log store for the whole homelab. It does not run on the beta or prod clusters. Instead, Grafana Alloy v1.20.1 runs as a DaemonSet on every cluster, tails the pod logs, adds a cluster label and pushes the lines to Loki on the management cluster. The same cluster label is on every Prometheus series, so a query in Grafana can move from a metric to the logs of the same cluster and namespace with no relabeling.

Grafana 13.2.4 sits on the same cluster and has Loki as a data source next to Prometheus and Tempo 3.1.0, the trace store. Sign-in to Grafana goes through Keycloak.

The trade-off of one central Loki is a known gap. If the management cluster is down, Alloy on the other clusters retries for a few minutes and then drops the batches, so logs from that window are lost. For a homelab with a small log volume, one store, one query surface and one set of labels is worth that gap. Single-binary mode keeps the footprint to one pod, well under the volume where the docs suggest a scalable mode.

Gotchas

• Retention is off by default. Without compactor.retention_enabled: true and a retention_period, logs are kept forever and storage only grows. delete_request_store is required, and the index period must be 24h.
• Label cardinality is the main failure mode. Each unique label set is a new stream. A label for request ID or user ID creates thousands of tiny streams, a big index and slow queries. The docs suggest tens of values at most for dynamic labels. Use filters or structured metadata instead.
• The limit is 15 labels per series since 3.0 (down from 30). Pushes with more labels are rejected.
• Structured metadata needs TSDB plus schema v13. An older schema with structured metadata enabled stops Loki from starting. Add a new v13 period with a future from date; do not edit old periods.
• 3.7 change: parsed labels (from | json or | logfmt) no longer override structured metadata with the same name. Check queries that relied on the old behaviour.
• No shell in the image since 3.5.8 (BusyBox was removed). kubectl exec ... sh fails; use an ephemeral debug container.
• Instant queries reject log selectors since 3.2: /api/v1/query returns 400 for a log query. Use query_range.
• The chart's MinIO subchart is deprecated, and the install docs list it for removal on 2026-10-31. Point Loki at an external S3-compatible store.
• Out-of-order and old lines can be rejected by reject_old_samples limits after an agent outage. Expect a gap, not a backfill.

Risks

• Licence. AGPL-3.0 is fine for running Loki internally. Teams that modify it and serve it to outside users must share their changes, and some legal teams block AGPL by policy.
• Project health. Strong: Grafana Labs staffs it, releases come every few weeks with two patch lines in parallel, and the repository has about 29,000 GitHub stars.
• Churn. Recent years brought schema migrations, the Promtail end of life, a deprecated deployment mode and a moved Helm chart. Budget time to read the upgrade guide before each minor release.
• Lock-in. Low for ingestion (Alloy and OpenTelemetry can send logs to other stores) but real for queries: LogQL, dashboards and alert rules do not move to another engine unchanged.
• Security. With auth_enabled: false there is no authentication on the push or query API. Keep Loki on an internal network or put an authenticating proxy in front of it.

Alternatives

• Elasticsearch or OpenSearch — pick when full-text search, field aggregations and relevance ranking are the main use. Costs more in disk, memory and operations.
• VictoriaLogs — pick for a single-node store that handles high-cardinality fields well with an Apache 2.0 licence.
• ClickHouse-based logging — pick when logs are analytics data and SQL across billions of rows is the main query style.
• A managed cloud log service (CloudWatch Logs, Grafana Cloud Logs) — pick when nobody should run a stateful log store.

Related

• Grafana Alloy — the agent that ships logs to Loki.
• Grafana — the query and dashboard surface.
• Prometheus — the metrics hub that shares the same labels.
• Tempo — the trace store next to Loki.
• SeaweedFS — an S3-compatible object store.
• Loki documentation
• Loki releases on GitHub
• Loki upgrade guide
• Label best practices