{"id":47,"date":"2025-10-30T12:00:00","date_gmt":"2025-10-30T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=47"},"modified":"2026-10-30T06:40:18","modified_gmt":"2026-10-30T06:40:18","slug":"a-simple-guide-to-alertmanager-for-beginners","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=47","title":{"rendered":"A Simple Guide to Alertmanager for Beginners"},"content":{"rendered":"<h3 class=\"wp-block-heading\">What is Alertmanager?<\/h3>\n<p class=\"wp-block-paragraph\">Alertmanager is part of the Prometheus project. It connects Prometheus alerts to real-world notifications. It helps send alerts to email, Slack, PagerDuty, or other systems that support webhooks.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Main purpose<\/strong><\/p>\n<ul class=\"wp-block-list\"><li>Receives alerts from Prometheus<\/li><li>Groups and manages them<\/li><li>Sends notifications to external tools<\/li><\/ul>\n<p class=\"wp-block-paragraph\"><strong>Key features<\/strong><\/p>\n<ol class=\"wp-block-list\"><li><strong>Aggregation:<\/strong> Combines the same alerts from multiple Prometheus servers and sends one notification instead of many.<\/li><li><strong>Grouping:<\/strong> If the same alert repeats, Alertmanager sends it at set times instead of every second.<\/li><li><strong>Silencing:<\/strong> Temporarily disables alerts during maintenance or deployments.<\/li><li><strong>Web Interface:<\/strong> Runs on port 9093. You can view alerts here, but configuration is done in the alertmanager.yml file.<\/li><\/ol>\n<p class=\"wp-block-paragraph\"><strong>Basic workflow<\/strong><\/p>\n<p class=\"wp-block-paragraph\">Prometheus \u2192 Alertmanager \u2192 Email \/ Slack \/ PagerDuty \/ Webhook<\/p>\n<h3 class=\"wp-block-heading\">Why We Need Alertmanager<\/h3>\n<p class=\"wp-block-paragraph\"><strong>Early problem detection<\/strong><\/p>\n<ul class=\"wp-block-list\"><li>It helps find issues before they become serious.<\/li><li>For example, if an API shows many errors suddenly, I can receive an alert before it crashes completely.<\/li><\/ul>\n<p class=\"wp-block-paragraph\"><strong>Setting thresholds<\/strong><\/p>\n<ul class=\"wp-block-list\"><li>Alerts should not be too sensitive or too late.<\/li><li>I can set a safe threshold just below the critical level.<\/li><li>This gives me time to fix the problem before it becomes severe.<\/li><\/ul>\n<p class=\"wp-block-paragraph\"><strong>Why not only Prometheus?<\/strong><\/p>\n<ul class=\"wp-block-list\"><li>Prometheus shows alerts in its UI but does not send notifications.<\/li><li>Alertmanager sends notifications to email or chat tools and avoids duplicate alerts.<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Creating Alert Rules in Prometheus<\/h3>\n<h3 class=\"wp-block-heading\">Where to store alert files (Linux)<\/h3>\n<ul class=\"wp-block-list\"><li>Prometheus usually has a folder named rules.<\/li><li>I save all my alert rule YAML files there.<\/li><li>The folder is owned by the prometheus user.<\/li><\/ul>\n<h3 class=\"wp-block-heading\">My first alert rule<\/h3>\n<p class=\"wp-block-paragraph\">Create rules\/alerts.yml:<\/p>\n<pre class=\"wp-block-code\"><code>groups:\n  - name: alerts\n    rules:\n      - alert: IsNodeExporterUp\n        expr: up{job=\"node_exporter\"} == 0\n        for: 1m\n        labels:\n          severity: critical\n        annotations:\n          summary: \"Node Exporter is down\"\n          description: \"The Node Exporter target is not responding for more than 1 minute.\"<\/code><\/pre>\n<p class=\"wp-block-paragraph\">What each part means<\/p>\n<ul class=\"wp-block-list\"><li>expr: PromQL condition for the alert<\/li><li>for: Duration before triggering<\/li><li>labels and annotations: Add information about the alert<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Link the alert file to prometheus.yml<\/h3>\n<pre class=\"wp-block-code\"><code>rule_files:\n  - \"rules\/alerts.yml\"<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Restart Prometheus<\/h3>\n<pre class=\"wp-block-code\"><code>sudo systemctl restart prometheus<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Verify<\/h3>\n<p class=\"wp-block-paragraph\">Open Prometheus \u2192 Alerts tab.<\/p>\n<ul class=\"wp-block-list\"><li>Green = inactive<\/li><li>Red = firing<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Test it<\/h3>\n<p class=\"wp-block-paragraph\">Stop Node Exporter:<\/p>\n<pre class=\"wp-block-code\"><code>sudo systemctl stop node_exporter<\/code><\/pre>\n<p class=\"wp-block-paragraph\">After 1 minute, the alert fires. Start it again:<\/p>\n<pre class=\"wp-block-code\"><code>sudo systemctl start node_exporter<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Improving Alerts with for and absent()<\/h3>\n<h3 class=\"wp-block-heading\">Use for to reduce false alarms<\/h3>\n<pre class=\"wp-block-code\"><code>expr: up{job=\"node_exporter\"} == 0\nfor: 5m<\/code><\/pre>\n<p class=\"wp-block-paragraph\">The alert fires only if the condition is true for 5 minutes.<\/p>\n<h3 class=\"wp-block-heading\">Use absent() to detect missing metrics<\/h3>\n<p class=\"wp-block-paragraph\">Instead of:<\/p>\n<pre class=\"wp-block-code\"><code>expr: up{job=\"node_exporter\"} == 0<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Use:<\/p>\n<pre class=\"wp-block-code\"><code>expr: absent(up{job=\"node_exporter\"})<\/code><\/pre>\n<p class=\"wp-block-paragraph\">This triggers when the target is completely missing.<\/p>\n<h3 class=\"wp-block-heading\">Combine both<\/h3>\n<pre class=\"wp-block-code\"><code>groups:\n  - name: node_exporter_alerts\n    rules:\n      - alert: NodeExporterDown\n        expr: absent(up{job=\"node_exporter\"})\n        for: 5m\n        labels:\n          severity: warning\n        annotations:\n          summary: \"Node Exporter down for 5 minutes\"\n          description: \"Prometheus has not seen Node Exporter metrics for 5 minutes.\"<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Adding Context with Labels and Annotations<\/h3>\n<p class=\"wp-block-paragraph\">Why it&#8217;s important: Alerts can go to different teams. Labels and annotations explain what happened and who should fix it.<\/p>\n<p class=\"wp-block-paragraph\">Labels<\/p>\n<pre class=\"wp-block-code\"><code>labels:\n  team: TeamAlpha\n  severity: critical<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Annotations<\/p>\n<pre class=\"wp-block-code\"><code>annotations:\n  summary: \"Node Exporter is down on {{ $labels.instance }}\"\n  description: \"Team Alpha should check {{ $labels.instance }}. Current value: {{ $value }}\"<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Complete example<\/p>\n<pre class=\"wp-block-code\"><code>groups:\n  - name: node_exporter_alerts\n    rules:\n      - alert: NodeExporterDown\n        expr: up{job=\"node_exporter\"} == 0\n        for: 5m\n        labels:\n          team: TeamAlpha\n          severity: critical\n        annotations:\n          summary: \"Node Exporter down on {{ $labels.instance }}\"\n          description: \"Instance {{ $labels.instance }} is not responding for more than 5 minutes. Current value: {{ $value }}\"<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Installing Alertmanager on Linux (Ubuntu)<\/h3>\n<p class=\"wp-block-paragraph\">Here is how I installed Alertmanager step by step.<\/p>\n<h3 class=\"wp-block-heading\">Step 1- Prepare the system<\/h3>\n<pre class=\"wp-block-code\"><code>sudo apt update &amp;&amp; sudo apt upgrade -y\nsudo groupadd prometheus 2&gt;\/dev\/null || true\nid -u prometheus &gt;\/dev\/null 2&gt;&amp;1 || sudo useradd --no-create-home --shell \/bin\/false -g prometheus prometheus<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 2- Download Alertmanager<\/h3>\n<p class=\"wp-block-paragraph\">Replace v0.X.Y with the version number you want.<\/p>\n<pre class=\"wp-block-code\"><code>cd \/tmp\nwget https:\/\/github.com\/prometheus\/alertmanager\/releases\/download\/v0.X.Y\/alertmanager-0.X.Y.linux-amd64.tar.gz\ntar xvf alertmanager-0.X.Y.linux-amd64.tar.gz\ncd alertmanager-0.X.Y.linux-amd64<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 3- Move files<\/h3>\n<pre class=\"wp-block-code\"><code>sudo mv alertmanager amtool \/usr\/local\/bin\/\nsudo mkdir -p \/etc\/alertmanager \/var\/lib\/alertmanager\nsudo mv alertmanager.yml \/etc\/alertmanager\/\nsudo chown -R prometheus:prometheus \/etc\/alertmanager \/var\/lib\/alertmanager\nsudo chmod -R 755 \/etc\/alertmanager \/var\/lib\/alertmanager<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 4- Create a basic configuration<\/h3>\n<p class=\"wp-block-paragraph\">\/etc\/alertmanager\/alertmanager.yml<\/p>\n<pre class=\"wp-block-code\"><code>route:\n  receiver: \"default\"\n\nreceivers:\n  - name: \"default\"<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 5- Create the systemd service<\/h3>\n<p class=\"wp-block-paragraph\">\/etc\/systemd\/system\/alertmanager.service<\/p>\n<pre class=\"wp-block-code\"><code>[Unit]\nDescription=Prometheus Alertmanager\nAfter=network.target\n\n[Service]\nUser=prometheus\nGroup=prometheus\nType=simple\nExecStart=\/usr\/local\/bin\/alertmanager \\\n  --config.file=\/etc\/alertmanager\/alertmanager.yml \\\n  --storage.path=\/var\/lib\/alertmanager\nRestart=always\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 6- Start the service<\/h3>\n<pre class=\"wp-block-code\"><code>sudo systemctl daemon-reload\nsudo systemctl enable alertmanager\nsudo systemctl start alertmanager\nsudo systemctl status alertmanager<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Step 7- Access the web UI<\/h3>\n<p class=\"wp-block-paragraph\">If using UFW:<\/p>\n<pre class=\"wp-block-code\"><code>sudo ufw allow 9093\/tcp<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Then open in your browser:<\/p>\n<pre class=\"wp-block-code\"><code>http:\/\/&lt;server-ip&gt;:9093<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Connecting Prometheus to Alertmanager<\/h3>\n<p class=\"wp-block-paragraph\">Edit your Prometheus configuration file:<\/p>\n<pre class=\"wp-block-code\"><code>alerting:\n  alertmanagers:\n    - static_configs:\n        - targets: [\"localhost:9093\"]<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Also include your alert rules:<\/p>\n<pre class=\"wp-block-code\"><code>rule_files:\n  - \"rules\/alerts.yml\"<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Restart Prometheus:<\/p>\n<pre class=\"wp-block-code\"><code>sudo systemctl restart prometheus<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Now Prometheus can send alerts to Alertmanager.<\/p>\n<h3 class=\"wp-block-heading\">How Alertmanager Works<\/h3>\n<p class=\"wp-block-paragraph\">When Prometheus sends an alert:<\/p>\n<ol class=\"wp-block-list\"><li>Matchers check alert labels.<\/li><li>Routes decide which receiver gets the alert.<\/li><li>Receivers send notifications to destinations such as email, Slack, or PagerDuty.<\/li><\/ol>\n<p class=\"wp-block-paragraph\">Workflow<\/p>\n<p class=\"wp-block-paragraph\">Prometheus \u2192 Alertmanager \u2192 Receiver \u2192 Email \/ Slack \/ PagerDuty \/ Webhook<\/p>\n<h3 class=\"wp-block-heading\">Configuring Routes and Matchers<\/h3>\n<p class=\"wp-block-paragraph\"><strong>Why use matchers:<\/strong> Different teams or severities need different channels.<\/p>\n<p class=\"wp-block-paragraph\">Example<\/p>\n<pre class=\"wp-block-code\"><code>route:\n  receiver: \"default\"\n  routes:\n    - receiver: \"urgent\"\n      matchers:\n        - severity = \"critical\"\n    - receiver: \"general\"\n      matchers:\n        - severity = \"warning\"\n\nreceivers:\n  - name: \"general\"\n    email_configs:\n      - to: \"support@example.com\"\n        from: \"alertmanager@example.com\"\n        smarthost: \"smtp.gmail.com:587\"\n        auth_username: \"alertmanager@example.com\"\n        auth_password: \"app_password\"\n\n  - name: \"urgent\"\n    email_configs:\n      - to: \"oncall@example.com\"\n        from: \"alertmanager@example.com\"\n        smarthost: \"smtp.gmail.com:587\"\n        auth_username: \"alertmanager@example.com\"\n        auth_password: \"app_password\"<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Testing<\/h3>\n<pre class=\"wp-block-code\"><code>sudo systemctl restart alertmanager\nsudo systemctl restart prometheus\nsudo systemctl stop node_exporter<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Wait for the alert to fire and check your inbox.<\/p>\n<h3 class=\"wp-block-heading\">Example: My Alertmanager Configuration<\/h3>\n<p class=\"wp-block-paragraph\">This is my working configuration. It includes Gmail for email alerts, Slack for chat alerts, and PagerDuty for on-call support.<\/p>\n<pre class=\"wp-block-code\"><code>global:\n  smtp_smarthost: 'smtp.gmail.com:587'\n  smtp_from: 'yourname@gmail.com'\n  smtp_auth_username: 'yourname@gmail.com'\n  smtp_auth_password: 'your-app-specific-password'\n  smtp_require_tls: true\n\nroute:\n  receiver: 'pager_duty'\n  routes:\n    - receiver: 'urgent_receiver'\n      matchers:\n        - severity = \"Critical\"\n\nreceivers:\n  - name: 'main_receiver'\n    email_configs:\n      - to: 'yourname@gmail.com'\n        send_resolved: true\n\n  - name: 'urgent_receiver'\n    email_configs:\n      - to: 'yourname@gmail.com'\n        send_resolved: true\n        headers:\n          subject: 'Urgent: Critical Alert'\n\n  - name: 'slack_receiver'\n    slack_configs:\n      - api_url: 'https:\/\/hooks.slack.com\/services\/...'\n        channel: '#channelname'\n        send_resolved: true\n\n  - name: 'pager_duty'\n    pagerduty_configs:\n      - service_key: 'your_pagerduty_service_key'\n\ninhibit_rules:\n  - source_match:\n      team: \"Team reza\"\n    target_match:\n      team: \"Team reza2\"\n    equal: ['severity']<\/code><\/pre>\n<p class=\"wp-block-paragraph\">What this does<\/p>\n<ul class=\"wp-block-list\"><li>Normal alerts go to email<\/li><li>Critical alerts go to a different urgent email<\/li><li>Alerts also go to Slack and PagerDuty<\/li><li>The inhibit rule stops duplicate notifications between two teams<\/li><\/ul>\n<p class=\"wp-block-paragraph\">Restart Service<\/p>\n<pre class=\"wp-block-code\"><code>sudo systemctl restart alertmanager<\/code><\/pre>","protected":false},"excerpt":{"rendered":"<p>What is Alertmanager? Alertmanager is part of the Prometheus project. It connects Prometheus alerts to real-world notifications. It helps send alerts to&hellip;<\/p>\n","protected":false},"author":2,"featured_media":49,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"3","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/simple-guide-alertmanager-beginners-reza-chegini-ibj0c\/","footnotes":""},"categories":[10,14],"tags":[22,21,8,16],"series":[15],"class_list":["post-47","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-observability","tag-alerting","tag-alertmanager","tag-devops","tag-prometheus","series-prometheus"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/simple-guide-alertmanager-beginners-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/47","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=47"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/47\/revisions"}],"predecessor-version":[{"id":48,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/47\/revisions\/48"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/49"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=47"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=47"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=47"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=47"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}