A Simple Guide to Alertmanager for Beginners

What is Alertmanager?

Alertmanager is part of the Prometheus project. It connects Prometheus alerts to real-world notifications. It helps send alerts to email, Slack, PagerDuty, or other systems that support webhooks.

Main purpose

• Receives alerts from Prometheus
• Groups and manages them
• Sends notifications to external tools

Key features

1. Aggregation: Combines the same alerts from multiple Prometheus servers and sends one notification instead of many.
2. Grouping: If the same alert repeats, Alertmanager sends it at set times instead of every second.
3. Silencing: Temporarily disables alerts during maintenance or deployments.
4. Web Interface: Runs on port 9093. You can view alerts here, but configuration is done in the alertmanager.yml file.

Basic workflow

Prometheus → Alertmanager → Email / Slack / PagerDuty / Webhook

Why We Need Alertmanager

Early problem detection

• It helps find issues before they become serious.
• For example, if an API shows many errors suddenly, I can receive an alert before it crashes completely.

Setting thresholds

• Alerts should not be too sensitive or too late.
• I can set a safe threshold just below the critical level.
• This gives me time to fix the problem before it becomes severe.

Why not only Prometheus?

• Prometheus shows alerts in its UI but does not send notifications.
• Alertmanager sends notifications to email or chat tools and avoids duplicate alerts.

Creating Alert Rules in Prometheus

Where to store alert files (Linux)

• Prometheus usually has a folder named rules.
• I save all my alert rule YAML files there.
• The folder is owned by the prometheus user.

My first alert rule

Create rules/alerts.yml:

groups:
  - name: alerts
    rules:
      - alert: IsNodeExporterUp
        expr: up{job="node_exporter"} == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Node Exporter is down"
          description: "The Node Exporter target is not responding for more than 1 minute."

What each part means

• expr: PromQL condition for the alert
• for: Duration before triggering
• labels and annotations: Add information about the alert

Link the alert file to prometheus.yml

rule_files:
  - "rules/alerts.yml"

Restart Prometheus

sudo systemctl restart prometheus

Verify

Open Prometheus → Alerts tab.

• Green = inactive
• Red = firing

Test it

Stop Node Exporter:

sudo systemctl stop node_exporter

After 1 minute, the alert fires. Start it again:

sudo systemctl start node_exporter

Improving Alerts with for and absent()

Use for to reduce false alarms

expr: up{job="node_exporter"} == 0
for: 5m

The alert fires only if the condition is true for 5 minutes.

Use absent() to detect missing metrics

Instead of:

expr: up{job="node_exporter"} == 0

Use:

expr: absent(up{job="node_exporter"})

This triggers when the target is completely missing.

Combine both

groups:
  - name: node_exporter_alerts
    rules:
      - alert: NodeExporterDown
        expr: absent(up{job="node_exporter"})
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Node Exporter down for 5 minutes"
          description: "Prometheus has not seen Node Exporter metrics for 5 minutes."

Adding Context with Labels and Annotations

Why it’s important: Alerts can go to different teams. Labels and annotations explain what happened and who should fix it.

Labels

labels:
  team: TeamAlpha
  severity: critical

Annotations

annotations:
  summary: "Node Exporter is down on {{ $labels.instance }}"
  description: "Team Alpha should check {{ $labels.instance }}. Current value: {{ $value }}"

Complete example

groups:
  - name: node_exporter_alerts
    rules:
      - alert: NodeExporterDown
        expr: up{job="node_exporter"} == 0
        for: 5m
        labels:
          team: TeamAlpha
          severity: critical
        annotations:
          summary: "Node Exporter down on {{ $labels.instance }}"
          description: "Instance {{ $labels.instance }} is not responding for more than 5 minutes. Current value: {{ $value }}"

Installing Alertmanager on Linux (Ubuntu)

Here is how I installed Alertmanager step by step.

Step 1- Prepare the system

sudo apt update && sudo apt upgrade -y
sudo groupadd prometheus 2>/dev/null || true
id -u prometheus >/dev/null 2>&1 || sudo useradd --no-create-home --shell /bin/false -g prometheus prometheus

Step 2- Download Alertmanager

Replace v0.X.Y with the version number you want.

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v0.X.Y/alertmanager-0.X.Y.linux-amd64.tar.gz
tar xvf alertmanager-0.X.Y.linux-amd64.tar.gz
cd alertmanager-0.X.Y.linux-amd64

Step 3- Move files

sudo mv alertmanager amtool /usr/local/bin/
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager
sudo mv alertmanager.yml /etc/alertmanager/
sudo chown -R prometheus:prometheus /etc/alertmanager /var/lib/alertmanager
sudo chmod -R 755 /etc/alertmanager /var/lib/alertmanager

Step 4- Create a basic configuration

/etc/alertmanager/alertmanager.yml

route:
  receiver: "default"

receivers:
  - name: "default"

Step 5- Create the systemd service

/etc/systemd/system/alertmanager.service

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager
Restart=always

[Install]
WantedBy=multi-user.target

Step 6- Start the service

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

Step 7- Access the web UI

If using UFW:

sudo ufw allow 9093/tcp

Then open in your browser:

http://<server-ip>:9093

Connecting Prometheus to Alertmanager

Edit your Prometheus configuration file:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["localhost:9093"]

Also include your alert rules:

rule_files:
  - "rules/alerts.yml"

Restart Prometheus:

sudo systemctl restart prometheus

Now Prometheus can send alerts to Alertmanager.

How Alertmanager Works

When Prometheus sends an alert:

1. Matchers check alert labels.
2. Routes decide which receiver gets the alert.
3. Receivers send notifications to destinations such as email, Slack, or PagerDuty.

Workflow

Prometheus → Alertmanager → Receiver → Email / Slack / PagerDuty / Webhook

Configuring Routes and Matchers

Why use matchers: Different teams or severities need different channels.

Example

route:
  receiver: "default"
  routes:
    - receiver: "urgent"
      matchers:
        - severity = "critical"
    - receiver: "general"
      matchers:
        - severity = "warning"

receivers:
  - name: "general"
    email_configs:
      - to: "support@example.com"
        from: "alertmanager@example.com"
        smarthost: "smtp.gmail.com:587"
        auth_username: "alertmanager@example.com"
        auth_password: "app_password"

  - name: "urgent"
    email_configs:
      - to: "oncall@example.com"
        from: "alertmanager@example.com"
        smarthost: "smtp.gmail.com:587"
        auth_username: "alertmanager@example.com"
        auth_password: "app_password"

Testing

sudo systemctl restart alertmanager
sudo systemctl restart prometheus
sudo systemctl stop node_exporter

Wait for the alert to fire and check your inbox.

Example: My Alertmanager Configuration

This is my working configuration. It includes Gmail for email alerts, Slack for chat alerts, and PagerDuty for on-call support.

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: 'yourname@gmail.com'
  smtp_auth_username: 'yourname@gmail.com'
  smtp_auth_password: 'your-app-specific-password'
  smtp_require_tls: true

route:
  receiver: 'pager_duty'
  routes:
    - receiver: 'urgent_receiver'
      matchers:
        - severity = "Critical"

receivers:
  - name: 'main_receiver'
    email_configs:
      - to: 'yourname@gmail.com'
        send_resolved: true

  - name: 'urgent_receiver'
    email_configs:
      - to: 'yourname@gmail.com'
        send_resolved: true
        headers:
          subject: 'Urgent: Critical Alert'

  - name: 'slack_receiver'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/...'
        channel: '#channelname'
        send_resolved: true

  - name: 'pager_duty'
    pagerduty_configs:
      - service_key: 'your_pagerduty_service_key'

inhibit_rules:
  - source_match:
      team: "Team reza"
    target_match:
      team: "Team reza2"
    equal: ['severity']

What this does

• Normal alerts go to email
• Critical alerts go to a different urgent email
• Alerts also go to Slack and PagerDuty
• The inhibit rule stops duplicate notifications between two teams

Restart Service

sudo systemctl restart alertmanager