{"id":50,"date":"2025-10-31T12:00:00","date_gmt":"2025-10-31T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=50"},"modified":"2026-10-30T06:41:07","modified_gmt":"2026-10-30T06:41:07","slug":"understanding-prometheus-service-discovery","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=50","title":{"rendered":"Understanding Prometheus Service Discovery"},"content":{"rendered":"<h3 class=\"wp-block-heading\">What Is Service Discovery<\/h3>\n<p class=\"wp-block-paragraph\">Service discovery is built into Prometheus. I configure it in the prometheus.yml file, and I do not need to install anything extra. It helps Prometheus automatically find targets like servers or applications to collect metrics from them.<\/p>\n<p class=\"wp-block-paragraph\">In traditional setups, we write server addresses manually in the configuration file. But in cloud environments, this becomes difficult because:<\/p>\n<ul class=\"wp-block-list\"><li>Servers scale up or down automatically based on load<\/li><li>New servers are created when CPU usage is high<\/li><li>Old servers are removed when they are not needed anymore<\/li><li>Prometheus needs to discover these changes without manual updates<\/li><\/ul>\n<p class=\"wp-block-paragraph\">Service discovery solves this problem by automatically finding new servers and removing old ones from monitoring.<\/p>\n<h3 class=\"wp-block-heading\">Understanding File-Based Service Discovery<\/h3>\n<p class=\"wp-block-paragraph\">File-based service discovery is one of the simplest ways to manage targets in Prometheus. Instead of hardcoding server addresses directly in prometheus.yml, I create separate YAML or JSON files that list my targets. Prometheus reads these files and automatically updates its target list when the files change.<\/p>\n<p class=\"wp-block-paragraph\">This approach is very useful when:<\/p>\n<ul class=\"wp-block-list\"><li>My cloud provider does not have built-in service discovery support<\/li><li>I have custom infrastructure or legacy systems<\/li><li>I want to integrate Prometheus with my own inventory management system<\/li><li>I need a simple way to add or remove targets without restarting Prometheus<\/li><\/ul>\n<p class=\"wp-block-paragraph\">The best part about file-based service discovery is that I can automate the file generation. I can write scripts that query my infrastructure, generate the target files, and Prometheus automatically picks up the changes. This makes it work well even in dynamic environments.<\/p>\n<h3 class=\"wp-block-heading\">File-Based Service Discovery Configuration<\/h3>\n<p class=\"wp-block-paragraph\">Let me start with the file-based service discovery example:<\/p>\n<pre class=\"wp-block-code\"><code>- job_name: \"files_sd\"\n  file_sd_configs:\n    - files:\n      - \/etc\/prometheus\/files_sd\/*.yml<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Understanding Each Line<\/h3>\n<p class=\"wp-block-paragraph\">job_name: &#8220;files_sd&#8221;<\/p>\n<p class=\"wp-block-paragraph\">This is the name of my scrape job. Prometheus uses this name to identify the group of targets. I can see this name in the Prometheus UI under Status then Targets.<\/p>\n<p class=\"wp-block-paragraph\">file_sd_configs:<\/p>\n<p class=\"wp-block-paragraph\">This tells Prometheus that I want to use file-based service discovery. The word &#8220;configs&#8221; is plural because I can have multiple file-based discovery configurations under the same job.<\/p>\n<p class=\"wp-block-paragraph\">&#8211; files:<\/p>\n<p class=\"wp-block-paragraph\">This starts the list of files that Prometheus should read. The dash means this is a list item in YAML format.<\/p>\n<p class=\"wp-block-paragraph\">&#8211; \/etc\/prometheus\/files_sd\/*.yml<\/p>\n<p class=\"wp-block-paragraph\">This is the file path where Prometheus looks for target files. Let me explain this path:<\/p>\n<ul class=\"wp-block-list\"><li>\/etc\/prometheus\/files_sd\/ is the directory path<\/li><li>*.yml is a wildcard pattern. The asterisk means &#8220;match any filename&#8221;<\/li><\/ul>\n<p class=\"wp-block-paragraph\">So Prometheus will read all files ending with .yml in that directory. For example, if I have servers.yml, apps.yml, and databases.yml in the directory, Prometheus will read all three files automatically.<\/p>\n<h3 class=\"wp-block-heading\">The Target File Format<\/h3>\n<p class=\"wp-block-paragraph\">Now let me explain the target file content:<\/p>\n<pre class=\"wp-block-code\"><code>- targets:\n  - localhost:9100\n  labels:\n    team: \"Team Reza1\"\n- targets:\n  - localhost:9200\n  labels:\n    team: \"Team Reza2\"<\/code><\/pre>\n<p class=\"wp-block-paragraph\">First Target Group:<\/p>\n<p class=\"wp-block-paragraph\">&#8211; targets:<\/p>\n<p class=\"wp-block-paragraph\">This starts a new target group. The dash means this is the first item in a list.<\/p>\n<p class=\"wp-block-paragraph\">&#8211; localhost:9100<\/p>\n<p class=\"wp-block-paragraph\">This is the actual target address. The format is hostname:port or ip:port. In this example:<\/p>\n<ul class=\"wp-block-list\"><li>localhost means the same machine where Prometheus is running<\/li><li>9100 is the port number. Port 9100 is commonly used by node_exporter.<\/li><\/ul>\n<p class=\"wp-block-paragraph\">labels:<\/p>\n<p class=\"wp-block-paragraph\">This section adds custom labels to all targets in this group.<\/p>\n<p class=\"wp-block-paragraph\">team: &#8220;Team Reza1&#8221;<\/p>\n<p class=\"wp-block-paragraph\">This creates a label called &#8220;team&#8221; with the value &#8220;Team Reza1&#8221;. When I query metrics from localhost:9100, all metrics will have this label attached.<\/p>\n<p class=\"wp-block-paragraph\">Second Target Group:<\/p>\n<p class=\"wp-block-paragraph\">&#8211; targets: &#8211; localhost:9200<\/p>\n<p class=\"wp-block-paragraph\">This is another target on port 9200. This could be a different exporter or application.<\/p>\n<p class=\"wp-block-paragraph\">labels: team: &#8220;Team Reza2&#8221;<\/p>\n<p class=\"wp-block-paragraph\">This target gets a different team label. Now I can distinguish between metrics from Team Reza1 and Team Reza2.<\/p>\n<h3 class=\"wp-block-heading\">How Prometheus Uses This Configuration<\/h3>\n<p class=\"wp-block-paragraph\">When Prometheus reads this configuration:<\/p>\n<ol class=\"wp-block-list\"><li>It looks in the directory \/etc\/prometheus\/files_sd\/<\/li><li>It finds all files ending with .yml<\/li><li>It reads the files and discovers targets<\/li><li>It scrapes metrics from each target<\/li><li>If I update the files, Prometheus automatically detects the change within the refresh interval (default is 5 minutes)<\/li><\/ol>\n<h3 class=\"wp-block-heading\">Adding Multiple Targets<\/h3>\n<p class=\"wp-block-paragraph\">I can add multiple targets in one group:<\/p>\n<pre class=\"wp-block-code\"><code>- targets:\n  - 192.168.1.10:9100\n  - 192.168.1.11:9100\n  - 192.168.1.12:9100\n  labels:\n    team: \"Team Reza1\"\n    environment: \"production\"<\/code><\/pre>\n<p class=\"wp-block-paragraph\">All three targets will get the same labels.<\/p>\n<h3 class=\"wp-block-heading\">AWS EC2 Service Discovery Configuration<\/h3>\n<p class=\"wp-block-paragraph\">Now let me explain the AWS EC2 service discovery configuration:<\/p>\n<pre class=\"wp-block-code\"><code>- job_name: \"aws_sd\"\n  ec2_sd_configs:\n    - port: 9090\n      region: us-east-1\n      access_key: ''\n      secret_key: ''\n      filters:\n        - name: tag:Name\n          values:\n            - dev-.*<\/code><\/pre>\n<h3 class=\"wp-block-heading\">Understanding Each Line<\/h3>\n<p class=\"wp-block-paragraph\">job_name: &#8220;aws_sd&#8221;<\/p>\n<p class=\"wp-block-paragraph\">This is the name of the scrape job for AWS service discovery.<\/p>\n<p class=\"wp-block-paragraph\">ec2_sd_configs:<\/p>\n<p class=\"wp-block-paragraph\">This tells Prometheus to use AWS EC2 service discovery. Prometheus will automatically query the AWS API to find EC2 instances.<\/p>\n<p class=\"wp-block-paragraph\">&#8211; port: 9090<\/p>\n<p class=\"wp-block-paragraph\">This is the port number where Prometheus expects the exporter to be running on each EC2 instance. If my instances use node_exporter on port 9100, I should change this to 9100.<\/p>\n<p class=\"wp-block-paragraph\">region: us-east-1<\/p>\n<p class=\"wp-block-paragraph\">This is the AWS region where my EC2 instances are running. Examples: us-east-1, us-west-2, eu-west-1.<\/p>\n<p class=\"wp-block-paragraph\">access_key: &#8221; secret_key: &#8221;<\/p>\n<p class=\"wp-block-paragraph\">These are AWS credentials for authentication. The empty quotes mean no key is provided. I should never write actual keys in the configuration file. Instead, I should use IAM roles or environment variables.<\/p>\n<p class=\"wp-block-paragraph\">filters:<\/p>\n<p class=\"wp-block-paragraph\">This section helps me select which EC2 instances to discover.<\/p>\n<p class=\"wp-block-paragraph\">&#8211; name: tag:Name<\/p>\n<p class=\"wp-block-paragraph\">This filters by the Name tag of EC2 instances.<\/p>\n<p class=\"wp-block-paragraph\">values: &#8211; dev-.*<\/p>\n<p class=\"wp-block-paragraph\">This is a regular expression. It matches instances with names starting with &#8220;dev-&#8221; followed by any characters. Examples:<\/p>\n<ul class=\"wp-block-list\"><li>dev-webserver (matches)<\/li><li>dev-database (matches)<\/li><li>prod-webserver (does not match)<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Relabel Configs Explanation<\/h3>\n<pre class=\"wp-block-code\"><code>relabel_configs:\n  - source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]\n    target_label: instance<\/code><\/pre>\n<h3 class=\"wp-block-heading\">First Relabel Rule<\/h3>\n<p class=\"wp-block-paragraph\">source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]<\/p>\n<p class=\"wp-block-paragraph\">This takes two pieces of information:<\/p>\n<ul class=\"wp-block-list\"><li>__meta_ec2_tag_Name &#8211; The Name tag of the EC2 instance<\/li><li>__meta_ec2_private_ip &#8211; The private IP address<\/li><\/ul>\n<p class=\"wp-block-paragraph\">target_label: instance<\/p>\n<p class=\"wp-block-paragraph\">This creates a new label called &#8220;instance&#8221; that combines both values. The default separator is semicolon, so the result looks like: &#8220;dev-webserver;10.0.1.5&#8221;<\/p>\n<h3 class=\"wp-block-heading\">Second Relabel Rule<\/h3>\n<pre class=\"wp-block-code\"><code>- source_labels: [__meta_ec2_tag_Name]\n  regex: dev-.*\n  action: drop<\/code><\/pre>\n<p class=\"wp-block-paragraph\">source_labels: [__meta_ec2_tag_Name]<\/p>\n<p class=\"wp-block-paragraph\">This takes the Name tag of the EC2 instance.<\/p>\n<p class=\"wp-block-paragraph\">regex: dev-.*<\/p>\n<p class=\"wp-block-paragraph\">This matches any name starting with &#8220;dev-&#8220;.<\/p>\n<p class=\"wp-block-paragraph\">action: drop<\/p>\n<p class=\"wp-block-paragraph\">This tells Prometheus to NOT monitor instances that match the regex. If I change &#8220;drop&#8221; to &#8220;keep&#8221;, it will monitor only instances that match and ignore everything else.<\/p>\n<h3 class=\"wp-block-heading\">Third Relabel Rule<\/h3>\n<pre class=\"wp-block-code\"><code>- source_labels: [__meta_ec2_public_ip]\n  replacement: ${1}:9090\n  target_label: __address__<\/code><\/pre>\n<p class=\"wp-block-paragraph\">source_labels: [__meta_ec2_public_ip]<\/p>\n<p class=\"wp-block-paragraph\">This takes the public IP address of the EC2 instance.<\/p>\n<p class=\"wp-block-paragraph\">replacement: ${1}:9090<\/p>\n<p class=\"wp-block-paragraph\">This adds the port number to the IP. The ${1} means &#8220;take the first source label value&#8221;. So if the public IP is 54.123.45.67, the result is 54.123.45.67:9090.<\/p>\n<p class=\"wp-block-paragraph\">target_label: address<\/p>\n<p class=\"wp-block-paragraph\">This is a special label that tells Prometheus where to scrape metrics from. By default, EC2 service discovery uses private IPs. But if I run Prometheus locally, I cannot reach private IPs. So I replace them with public IPs.<\/p>\n<h3 class=\"wp-block-heading\">Why Use Service Discovery<\/h3>\n<h3 class=\"wp-block-heading\">Static Configuration (Manual)<\/h3>\n<pre class=\"wp-block-code\"><code>- job_name: \"manual\"\n  static_configs:\n    - targets:\n      - 192.168.1.10:9100\n      - 192.168.1.11:9100<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Problems:<\/p>\n<ul class=\"wp-block-list\"><li>I must manually update prometheus.yml when servers change<\/li><li>I must restart Prometheus<\/li><li>Does not work with auto-scaling<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Service Discovery (Automatic)<\/h3>\n<pre class=\"wp-block-code\"><code>- job_name: \"automatic\"\n  file_sd_configs:\n    - files:\n      - \/etc\/prometheus\/files_sd\/*.yml<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Benefits:<\/p>\n<ul class=\"wp-block-list\"><li>Prometheus automatically detects changes<\/li><li>No restart needed<\/li><li>Works with auto-scaling<\/li><li>I can automate updates with scripts<\/li><\/ul>\n<h3 class=\"wp-block-heading\">Complete Working Example<\/h3>\n<p class=\"wp-block-paragraph\">Here is a complete configuration:<\/p>\n<pre class=\"wp-block-code\"><code>global:\n  scrape_interval: 15s\n\nscrape_configs:\n  - job_name: \"prometheus\"\n    static_configs:\n      - targets:\n        - localhost:9090\n\n  - job_name: \"files_sd\"\n    file_sd_configs:\n      - files:\n        - \/etc\/prometheus\/files_sd\/*.yml\n\n  - job_name: \"aws_sd\"\n    ec2_sd_configs:\n      - port: 9100\n        region: us-east-1\n        filters:\n          - name: tag:Monitor\n            values:\n              - \"true\"\n          - name: instance-state-name\n            values:\n              - running\n    relabel_configs:\n      - source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]\n        separator: '-'\n        target_label: instance\n      - source_labels: [__meta_ec2_public_ip]\n        replacement: ${1}:9100\n        target_label: __address__<\/code><\/pre>\n<p class=\"wp-block-paragraph\">Target file example:<\/p>\n<pre class=\"wp-block-code\"><code># File: \/etc\/prometheus\/files_sd\/servers.yml\n\n- targets:\n  - localhost:9100\n  labels:\n    team: \"Team Reza1\"\n    environment: \"production\"\n\n- targets:\n  - localhost:9200\n  labels:\n    team: \"Team Reza2\"\n    environment: \"production\"<\/code><\/pre>","protected":false},"excerpt":{"rendered":"<p>What Is Service Discovery Service discovery is built into Prometheus. I configure it in the prometheus.yml file, and I do not need&hellip;<\/p>\n","protected":false},"author":2,"featured_media":52,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"4","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/understanding-prometheus-service-discovery-reza-chegini-urhqc\/","footnotes":""},"categories":[10,14],"tags":[8,19,16,23],"series":[15],"class_list":["post-50","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-observability","tag-devops","tag-monitoring","tag-prometheus","tag-service-discovery","series-prometheus"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/understanding-prometheus-service-discovery-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/50","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=50"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/50\/revisions"}],"predecessor-version":[{"id":51,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/50\/revisions\/51"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/52"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=50"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=50"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=50"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=50"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}