Prometheus

Prometheus is an open-source monitoring system that collects numeric metrics from services, stores them as time series and alerts when a rule matches. It pulls metrics from each target over HTTP, so a service only needs to expose a /metrics page, and the Prometheus server decides what to collect and how often. The official docs at prometheus.io/docs own the configuration reference.

What it is

A single Go binary that does service discovery, scraping, storage, querying (PromQL) and rule evaluation. It was started at SoundCloud in 2012, joined the Cloud Native Computing Foundation in 2016 and graduated in 2018. The licence is Apache 2.0, and there is no paid edition of the server itself.

The current line is v3. A new minor release cycle starts every six weeks: v3.15.0 shipped on 2026-09-25 and v3.16 is next. Non-LTS minors generally stop getting fixes once the next one ships. v3.13 is the long-term support line, supported until 2027-07-31 (v3.13.4, a security and bugfix patch, shipped on 2026-10-02). The previous LTS, v3.5, reached end of life on 2026-07-31.

The project is a family of components around the server:

• Alertmanager — receives alerts from Prometheus, groups and deduplicates them, and sends notifications.
• Exporters — node-exporter (host metrics), blackbox_exporter (probes), and many third-party exporters for databases and services.
• Client libraries — Go, Java, Python, Ruby, Rust and others, for instrumenting application code.
• Prometheus Operator (a separate prometheus-operator project) — Kubernetes custom resources (Prometheus, PrometheusAgent, ServiceMonitor, PodMonitor, PrometheusRule, Alertmanager) that render Prometheus configuration.

How it works

The server runs a loop per target: service discovery finds targets (Kubernetes pods, endpoints, nodes, or a static list), the scraper fetches each target's metrics page, and the samples go into the in-memory head block, protected by a write-ahead log (WAL). Every two hours the head is cut into an immutable block on disk. The rule engine runs recording rules (precomputed queries) and alerting rules on an interval and pushes firing alerts to Alertmanager.

Agent mode (--agent) removes the query engine, the rule engine, alerting and long-term local storage. What is left is discovery, scraping and a special WAL that deletes data as soon as it is sent. Remote write sends samples from the WAL to another system over HTTP. A full Prometheus can receive remote write at /api/v1/write when it runs with --web.enable-remote-write-receiver. Together they give a hub-and-spoke layout: agents on each cluster, one queryable server in the middle.

The important property of this layout is that the spoke clusters hold no queryable data. A dashboard, a recording rule or an alert sees a spoke only through what reached the hub. If the link fails for longer than the WAL can hold, that time window is lost for good.

When to use it

Reach for it when:

• You run Kubernetes and want metrics, dashboards and alerts with the most widely supported tooling. Most charts ship a ServiceMonitor or a metrics port.
• Your data is numeric and labelled — request rates, latencies, queue depths, resource use — and a few weeks of history is enough.
• You have several small clusters and want one place to query them. Agent mode on each cluster plus one hub keeps the spokes light.
• You want alert rules in Git as PrometheusRule objects next to the workloads they watch.

Skip it when:

• You need logs or traces. Prometheus stores metrics only; use Loki or Tempo alongside it.
• You need months or years of history at scale. Local storage is single-node and not clustered; add Thanos, Mimir or VictoriaMetrics for long-term or highly available storage.
• You need exact per-event data, such as billing. Prometheus samples on an interval and can miss data between scrapes.
• Your labels have unbounded cardinality (user IDs, request IDs). Every label combination is a new series, and memory grows with series count.

At a glance

Core concepts

• Series — a metric name plus a unique set of labels. Cost scales with active series, not with samples.
• Scrape — one HTTP pull of a target's /metrics page, on scrape_interval.
• Recording rule — a PromQL expression evaluated on an interval and stored as a new series.
• Alerting rule — a PromQL expression that fires when it returns results for the for duration.
• External labels — labels attached to every series and alert that leaves the server through remote write, federation or Alertmanager.
• ServiceMonitor / PodMonitor — Prometheus Operator resources that select Services or pods to scrape.

Key facts

Item	Value
Current release	v3.15.0 (2026-09-25); LTS v3.13 to 2027-07-31
Licence	Apache 2.0
Default retention	15 days in the binary; 10 days in the kube-prometheus-stack chart
Storage per sample	1–2 bytes on average
Block size	2 hours, compacted up to 10% of retention or 31 days
Agent WAL buffer	Up to about 2 hours when the remote end is down
Remote write memory	About 25% more server memory, per the docs
Remote write spec	1.0 stable; 2.0 experimental

Full server vs agent mode

Capability	Full server	Agent mode
Scrape and discovery	Yes	Yes
PromQL queries	Yes	No (UI shows targets and config only)
Recording and alerting rules	Yes	No
Local storage	TSDB with retention	WAL only, deleted after send
Remote write	Optional	Required to be useful
Operator resource	Prometheus (v1)	PrometheusAgent (v1alpha1)

How to integrate

The hub (kube-prometheus-stack)

The chart is published to the OCI registry and the classic Helm repo. Chart 92.10.1 is current; it requires Kubernetes 1.25 or later and defaults to Prometheus v3.15.0 and Prometheus Operator v0.94.1.

helm install kps oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack \
  --version 92.10.1 --namespace monitoring --create-namespace -f hub-values.yaml

# hub-values.yaml
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: 25GB
    enableRemoteWriteReceiver: true      # adds --web.enable-remote-write-receiver
    externalLabels:
      cluster: hub
    serviceMonitorSelectorNilUsesHelmValues: false   # pick up every ServiceMonitor
    ruleSelectorNilUsesHelmValues: false
    storageSpec:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: 30Gi

A spoke (same chart, agent mode)

prometheus.agentMode: true makes the chart render a PrometheusAgent resource instead of a Prometheus. Turn off the parts that only make sense on the hub.

# spoke-values.yaml
alertmanager:
  enabled: false
grafana:
  enabled: false
defaultRules:
  create: false            # rules run on the hub
prometheus:
  agentMode: true
  prometheusSpec:
    externalLabels:
      cluster: prod
    remoteWrite:
      - url: https://prometheus-hub.example.com/api/v1/write
        queueConfig:
          capacity: 10000          # default; 3-10x maxSamplesPerSend
          maxSamplesPerSend: 2000  # default

A recording and alerting rule

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: api-latency
  namespace: monitoring
spec:
  groups:
    - name: api
      rules:
        - record: cluster_job:http_request_duration_seconds:p95_5m
          expr: histogram_quantile(0.95, sum by (cluster, job, le) (rate(http_request_duration_seconds_bucket[5m])))
        - alert: ApiSlow
          expr: cluster_job:http_request_duration_seconds:p95_5m > 1
          for: 10m
          labels:
            severity: warning

In the GL stack

Prometheus collects the metrics for the three K3s clusters behind builddaily.io, paiddaily.io and the other products: a management cluster, a beta cluster and a prod cluster.

• The hub. The management cluster runs kube-prometheus-stack 92.8.0, which installs Prometheus v3.15.0 with Alertmanager and Grafana. The hub keeps 15 days of data, capped at 25 GB, whichever limit is reached first.
• The spokes. The beta and prod clusters run Prometheus in agent mode with node-exporter and kube-state-metrics. The agents scrape locally and remote-write everything to the hub. No queryable metrics live on the spokes.
• One label per cluster. Every series carries a cluster label, so one Grafana dashboard or one alert rule covers all three clusters and can still be filtered to one.
• What reads it. Grafana uses the hub as its metrics data source. Alertmanager receives the hub's alerts and groups them by cluster, alert name and namespace. Pyrra turns SLO objects into recording rules and burn-rate alerts on the hub, and Blackbox Exporter probe results feed those SLOs.
• Why this shape. All three clusters are small VMs on one machine. A full Prometheus with its own rules and storage on each cluster would triple the memory and the configuration to maintain. Agent mode keeps the spokes to scraping, and puts storage, rules and alert routing in one place.

Gotchas

• The chart does not upgrade CRDs. Helm 3 installs CRDs once and never updates them, and a CRD change is a major chart version bump. Read the chart's UPGRADE.md and apply the new CRDs before a major upgrade. The crds.upgradeJob value automates this but is in preview and off by default.
• ServiceMonitors silently ignored. With the default serviceMonitorSelectorNilUsesHelmValues: true, the Prometheus only selects ServiceMonitors that carry the chart's release label. Third-party ServiceMonitors without it are never scraped. The same applies to PodMonitors and rules.
• Agent outage window. The agent WAL holds unsent data for about two hours. If the hub is down longer, the WAL is truncated and the gap is permanent. One slow remote endpoint also blocks WAL reads for all shards of that queue.
• External labels do not overwrite. An external label is only applied when a series does not already have that label. A target that exposes its own cluster label keeps it.
• No rules on an agent. PrometheusRule objects on a spoke do nothing. Disable defaultRules there and write rules on the hub, aggregated by (cluster).
• v2 to v3 changes. The agent and remote-write-receiver feature flags were removed in favour of --agent and --web.enable-remote-write-receiver. Scrapes now fail on a missing or unknown Content-Type unless fallback_scrape_protocol is set. le and quantile values are normalised (le="1" becomes le="1.0"), range selectors became left-open, and the Alertmanager v1 API is gone. Check dashboards and rules that match whole-number buckets.
• Deprecations in v3.15. --log.level is deprecated in favour of runtime.log_level in the config file, and --enable-feature=xor2-encoding in favour of storage.tsdb.chunk_encoding.floats: xor2. Check that anything reading the TSDB directly, such as a Thanos sidecar, supports XOR2 before turning it on.

Risks

• Project health is strong. Prometheus is a CNCF graduated project with a regular six-week release cycle, many maintainers from different companies, and an LTS line. Apache 2.0 removes licence risk.
• Single point of storage. One hub is one disk and one process. If it is lost, history is lost and alerting stops for every cluster. Monitor the hub from outside itself (a dead-man's-switch alert such as Watchdog, or an external probe), and back up or accept the loss of the TSDB volume.
• Cardinality. One bad label can multiply series count and run the hub out of memory. Review new metrics, and use metric_relabel_configs or per-scrape sample_limit to cap them.
• Exposure. The HTTP API has no authentication by default, and the remote-write receiver accepts any writer. Keep both on private networks or behind authentication and TLS.
• Lock-in is low. OpenMetrics grew out of the Prometheus exposition format, PromQL is supported by Mimir, Thanos and VictoriaMetrics, and remote write is a published spec.

Alternatives

• VictoriaMetrics — pick it for lower memory and disk per series and a clustered mode, while keeping PromQL-compatible queries and remote write ingestion.
• Grafana Mimir or Thanos — pick them when one Prometheus server is not enough: long-term retention in object storage, high availability and global queries across many servers.
• Grafana Alloy — pick it instead of the Prometheus agent when one collector should scrape metrics and also ship logs and traces. It speaks remote write too.
• OpenTelemetry Collector — pick it when applications push OTLP and the backend may change; Prometheus can receive OTLP as well.

Related

• Alertmanager, Grafana and Pyrra — alert routing, dashboards and SLOs on top of the hub.
• Loki, Alloy, Tempo and OpenTelemetry — the logs and traces half of the observability layer.
• Blackbox Exporter and OpenCost — probes and cost allocation built on Prometheus data.
• K3s and Rancher — the clusters being monitored and the tool that manages them.
• Official: Prometheus docs, Agent mode, Remote write tuning, Release cycle, kube-prometheus-stack chart.