Alertmanager

Alertmanager takes the alerts that Prometheus rules fire and turns them into notifications people can act on. Prometheus decides that something is wrong; Alertmanager decides who is told, in which channel, how many alerts go into one message, how often a still-firing problem is repeated, and which alerts are suppressed because a bigger one already explains them. The official docs at prometheus.io/docs/alerting own the full configuration reference.

What it is

A single Go binary (alertmanager) with a small web UI, an HTTP API (/api/v2) and a companion CLI (amtool). It is part of the Prometheus project under the CNCF, licensed Apache 2.0, with no paid tier. The default ports are 9093 for the API and UI and 9094 (TCP and UDP) for cluster gossip.

The release line is still 0.x. The latest release is v0.34.1 (2026-09-17), a bug-fix release for inhibition: several issues could un-mute alerts that an inhibit rule should have kept muted. v0.34.0 (2026-08-16) added optional templatable labels on routes, server-side state filters on GET /api/v2/silences, webhook batching for the event recorder and toDate template functions. It also carries the most recent operational breaking change. The reason label on alertmanager_notifications_failed_total now separates authError (HTTP 401/403) and rateLimited (HTTP 429) from clientError. Alerts and dashboards that match reason="clientError" for those codes must change.

Earlier changes still matter on upgrade. v0.33.0 removed the alertmanager_marked_alerts metric and the no-op auto-gomaxprocs feature flag, and added an opt-in structured event recorder. Since v0.27, a new UTF-8 matcher parser runs in a fallback mode by default. The docs say UTF-8 strict mode will become the default, so configurations should be checked with amtool check-config now.

Most Kubernetes installs do not run the binary by hand. The Prometheus Operator manages it through an Alertmanager custom resource, and the kube-prometheus-stack Helm chart ships it next to Prometheus with a default set of Kubernetes alert rules.

How it works

Alertmanager is push-based and stateless in the sense that it stores no time series. Clients POST alerts, each a set of labels, annotations and a start and end time. Prometheus re-sends every firing alert on an interval with an end time in the future. If the re-sends stop, the alert resolves when its end time passes. Alerts without an end time resolve after resolve_timeout (5 minutes by default).

Inside, four parts do the work:

• The route tree. Every alert enters at the root route and walks down the child routes. The first child whose matchers fit takes it, unless continue: true lets it match later siblings too. The matching node picks the receiver and the timing.
• Aggregation groups. Alerts on the same route with equal values for the group_by labels form one group, and one group becomes one notification. group_wait (30 s default) holds the first message so related alerts can join. group_interval (5 m) sets how often changes to the group are sent. repeat_interval (4 h) sets how often an unchanged, still-firing group is sent again.
• Silences and inhibition. A silence mutes alerts that match a set of matchers for a fixed time window. An inhibit rule mutes target alerts while a source alert with equal values for the equal labels is firing. Time intervals mute or activate a route on a calendar.
• The notification log. A record of what was sent to which receiver for each group. It is what stops a repeat before repeat_interval has passed.

In a highly available setup, several replicas run side by side and gossip silences and the notification log to each other. Each Prometheus sends every alert to every replica. The replicas wait a different amount of time depending on their position in the cluster, check the shared notification log, and so only one sends each notification.

Two consequences follow from this design. The first notification for a new problem arrives group_wait after the first alert, not at once. And because repeat_interval is checked only at each group_interval, the docs note it is rounded up to a multiple of group_interval. If it is longer than --data.retention (120 h by default), the repeat happens at the end of the retention period instead.

When to use it

Reach for it when:

• Alert rules already live in Prometheus, or in a Prometheus-compatible engine, and notifications need routing by team, severity or cluster.
• A failure of one component sets off dozens of alerts, and the on-call person should get one grouped message or only the root cause.
• Maintenance windows need muting by label, set by a script or by hand, with an audit trail of who silenced what.
• Several Prometheus instances or clusters should send to one place with one routing policy.

Skip it when:

• Alerting is already done in Grafana Alerting, a hosted monitoring service or an incident platform that does its own grouping and routing.
• There are a handful of checks and one recipient. A simple uptime monitor with email is less to run.
• The alert source is not label-based. Alertmanager's whole model assumes labelled alerts.

At a glance

Core concepts

• Alert — a labelled event with annotations, a start time and an end time; identity is the label set.
• Route — a node in the routing tree with matchers, a receiver, group_by and the three timers. Children inherit unset values.
• Receiver — a named set of integrations: email, Slack, PagerDuty, Opsgenie, webhook, Microsoft Teams, Telegram, SNS, Jira, Discord, Webex, incident.io and others.
• Silence — matchers plus a time window. Not part of the config file; created through the API and stored on disk.
• Inhibit rule — source_matchers, target_matchers and an equal label list.
• Time interval — named calendar windows used by mute_time_intervals and active_time_intervals on routes. Not inherited by child routes.
• Null receiver — a receiver with no integrations, the usual target for alerts that must be accepted but never sent.

Key facts (v0.34)

Item	Value
Licence	Apache 2.0
Latest release	v0.34.1 (2026-09-17)
Ports	9093 API and UI, 9094 TCP and UDP gossip
group_wait / group_interval / repeat_interval defaults	30 s / 5 m / 4 h
resolve_timeout default	5 m
--data.retention default	120 h
Email send_resolved default	false
smtp_require_tls default	true
Matcher parser	UTF-8 fallback mode by default since v0.27
Config check	amtool check-config

Compared with Grafana Alerting

Aspect	Alertmanager	Grafana Alerting
Rule evaluation	In Prometheus (or Mimir, Thanos, VictoriaMetrics)	In Grafana, over any data source
Routing model	Label route tree, config file	Notification policies, same model, UI or provisioning
State	Silences and log on local disk, gossiped	Grafana database
Config as code	Plain YAML, amtool validation	Provisioning files or API
Fit	Prometheus-first stacks	Mixed data sources, UI-first teams

How to integrate

A minimal alertmanager.yml

global:
  smtp_smarthost: email-smtp.us-east-1.amazonaws.com:587
  smtp_from: [email protected]
  smtp_auth_username: <smtp-username>
  smtp_auth_password_file: /etc/alertmanager/secrets/alertmanager-smtp/password
  smtp_require_tls: true

route:
  receiver: email
  group_by: [cluster, alertname, namespace]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - receiver: "null"
      matchers:
        - alertname = "Watchdog"
    - receiver: "null"
      matchers:
        - severity = "info"
    - receiver: security-email
      matchers:
        - source = "falco"          # whatever label the security events carry

receivers:
  - name: "null"
  - name: email
    email_configs:
      - to: [email protected]
        send_resolved: true
  - name: security-email
    email_configs:
      - to: [email protected]

inhibit_rules:
  - source_matchers: [severity = "critical"]
    target_matchers: [severity = "warning"]
    equal: [cluster, namespace, alertname]

Matchers are double-quoted on the value side, which keeps them valid under the UTF-8 parser.

kube-prometheus-stack values

alertmanager:
  alertmanagerSpec:
    replicas: 2                     # gossip HA; Prometheus sends to both pods
    retention: 120h
    secrets: [alertmanager-smtp]    # mounted at /etc/alertmanager/secrets/alertmanager-smtp/
  config:
    # the alertmanager.yml above, inline

Check, test and silence

amtool check-config alertmanager.yml
amtool config routes test --config.file=alertmanager.yml alertname=KubeJobFailed namespace=velero
amtool --alertmanager.url=http://alertmanager:9093 silence add \
  alertname=KubeNodeNotReady --duration=2h --comment="node reboot"
amtool --alertmanager.url=http://alertmanager:9093 silence query

config routes test shows which receiver a given label set reaches without sending anything, which is the fastest check after a routing change.

In the GL stack

Alertmanager v0.34.1 runs on the management cluster and receives the alerts for all three clusters. Its one delivery channel is email, sent through Amazon SES over SMTP.

Alerts are grouped by cluster, alertname and namespace, so a problem that fires on many pods of one workload arrives as one message per cluster and namespace. The repeat interval is 12 hours, three times the 4-hour default, which keeps a known, still-firing problem to two reminders a day in an email inbox.

Two classes of alert are muted on purpose. Watchdog is the always-firing heartbeat alert that proves the alert pipeline is alive; it is routed to nothing. Info-severity alerts are muted as well. Falco runtime-security events have a route of their own, apart from the infrastructure alerts.

Backups are covered through the general job alert rather than a separate tool. KubeJobFailed fires for any failed Kubernetes Job, and the backup jobs are Jobs. A backup-retention job checks every backup set and fails when any set is older than 36 hours, so a stalled backup shows up as a failed Job and then as an email.

Gotchas

• Do not load balance Prometheus to Alertmanager. The README is explicit: point Prometheus at every replica. A Service or load balancer in between breaks the HA model.
• Gossip needs UDP and TCP. Since v0.15 the cluster port must be open for both protocols, including in container port definitions and network policies.
• A single replica still listens for peers. HA is on by default. Set --cluster.listen-address="" to turn it off for a lone instance.
• The 30-second wait is on purpose. New problems arrive after group_wait. Lowering it to near zero sends a separate message for every alert of a burst.
• repeat_interval rounds up. It is checked at each group_interval, so a value that is not a multiple is rounded up, and a value longer than --data.retention repeats at the end of retention.
• Muted alerts still need a route. Without a "null" receiver route, Watchdog falls through to the default receiver and sends forever.
• Inhibition on missing labels. If every label in equal is missing from both alerts, the rule still applies. Alerts that match both source and target sides follow a special case; keep the two sides disjoint.
• Email does not send resolves by default. send_resolved is false for email_configs, unlike most chat integrations.
• Upgrade to v0.34.1 if inhibition matters. v0.34.0 and some earlier versions could un-mute inhibited alerts.
• Watch the failure metric after v0.34.0. alertmanager_notifications_failed_total now reports authError and rateLimited reasons; an SMTP credential problem no longer counts as clientError.

Risks

• Licence and project health. Apache 2.0 under the CNCF Prometheus project, with regular releases. The 0.x version line means the project keeps the right to make breaking changes in minor releases, and the changelog marks them as [CHANGE].
• The alert path is a single chain. If Alertmanager, its SMTP relay or the credentials fail, nothing tells anyone. An external dead-man's switch on the Watchdog alert is the standard guard; a muted Watchdog gives no such signal.
• Email as the only channel. Mail delays, spam filtering or a provider sending limit delay or drop alerts with no retry visible to the receiver.
• Lock-in is low. The configuration is plain YAML and the API is open. Grafana, Mimir and other tools accept the same routing model.
• Security. The API has no built-in authentication; anyone who can reach port 9093 can create silences. Keep it behind an authenticating proxy or on an internal network, and keep SMTP credentials in a secret file.

Alternatives

• Grafana Alerting — pick it when rules span Loki, SQL and other data sources, or when alerts should be managed in a UI.
• Grafana Mimir or Cortex Alertmanager — pick it for many tenants on one shared, horizontally scaled Alertmanager.
• PagerDuty, incident.io or a similar platform — pick it for on-call schedules, escalation and phone paging. These are often fed by Alertmanager rather than replacing it.
• Uptime Kuma or another uptime checker — pick it for a small set of HTTP and port checks with no Prometheus at all.

Related

• Prometheus — evaluates the alert rules and sends the alerts.
• Grafana — dashboards and an Alertmanager data source to view alerts and silences.
• Falco — the runtime-security events with their own route.
• Velero — the backups the job and age alerts watch.
• Blackbox exporter and Pyrra — probe and SLO rules that end up as alerts here.
• Official: Alertmanager concepts, Configuration reference, Releases, High availability.