Understanding Prometheus Service Discovery

What Is Service Discovery

Service discovery is built into Prometheus. I configure it in the prometheus.yml file, and I do not need to install anything extra. It helps Prometheus automatically find targets like servers or applications to collect metrics from them.

In traditional setups, we write server addresses manually in the configuration file. But in cloud environments, this becomes difficult because:

• Servers scale up or down automatically based on load
• New servers are created when CPU usage is high
• Old servers are removed when they are not needed anymore
• Prometheus needs to discover these changes without manual updates

Service discovery solves this problem by automatically finding new servers and removing old ones from monitoring.

Understanding File-Based Service Discovery

File-based service discovery is one of the simplest ways to manage targets in Prometheus. Instead of hardcoding server addresses directly in prometheus.yml, I create separate YAML or JSON files that list my targets. Prometheus reads these files and automatically updates its target list when the files change.

This approach is very useful when:

• My cloud provider does not have built-in service discovery support
• I have custom infrastructure or legacy systems
• I want to integrate Prometheus with my own inventory management system
• I need a simple way to add or remove targets without restarting Prometheus

The best part about file-based service discovery is that I can automate the file generation. I can write scripts that query my infrastructure, generate the target files, and Prometheus automatically picks up the changes. This makes it work well even in dynamic environments.

File-Based Service Discovery Configuration

Let me start with the file-based service discovery example:

- job_name: "files_sd"
  file_sd_configs:
    - files:
      - /etc/prometheus/files_sd/*.yml

Understanding Each Line

job_name: “files_sd”

This is the name of my scrape job. Prometheus uses this name to identify the group of targets. I can see this name in the Prometheus UI under Status then Targets.

file_sd_configs:

This tells Prometheus that I want to use file-based service discovery. The word “configs” is plural because I can have multiple file-based discovery configurations under the same job.

– files:

This starts the list of files that Prometheus should read. The dash means this is a list item in YAML format.

– /etc/prometheus/files_sd/*.yml

This is the file path where Prometheus looks for target files. Let me explain this path:

• /etc/prometheus/files_sd/ is the directory path
• *.yml is a wildcard pattern. The asterisk means “match any filename”

So Prometheus will read all files ending with .yml in that directory. For example, if I have servers.yml, apps.yml, and databases.yml in the directory, Prometheus will read all three files automatically.

The Target File Format

Now let me explain the target file content:

- targets:
  - localhost:9100
  labels:
    team: "Team Reza1"
- targets:
  - localhost:9200
  labels:
    team: "Team Reza2"

First Target Group:

– targets:

This starts a new target group. The dash means this is the first item in a list.

– localhost:9100

This is the actual target address. The format is hostname:port or ip:port. In this example:

• localhost means the same machine where Prometheus is running
• 9100 is the port number. Port 9100 is commonly used by node_exporter.

labels:

This section adds custom labels to all targets in this group.

team: “Team Reza1”

This creates a label called “team” with the value “Team Reza1”. When I query metrics from localhost:9100, all metrics will have this label attached.

Second Target Group:

– targets: – localhost:9200

This is another target on port 9200. This could be a different exporter or application.

labels: team: “Team Reza2”

This target gets a different team label. Now I can distinguish between metrics from Team Reza1 and Team Reza2.

How Prometheus Uses This Configuration

When Prometheus reads this configuration:

1. It looks in the directory /etc/prometheus/files_sd/
2. It finds all files ending with .yml
3. It reads the files and discovers targets
4. It scrapes metrics from each target
5. If I update the files, Prometheus automatically detects the change within the refresh interval (default is 5 minutes)

Adding Multiple Targets

I can add multiple targets in one group:

- targets:
  - 192.168.1.10:9100
  - 192.168.1.11:9100
  - 192.168.1.12:9100
  labels:
    team: "Team Reza1"
    environment: "production"

All three targets will get the same labels.

AWS EC2 Service Discovery Configuration

Now let me explain the AWS EC2 service discovery configuration:

- job_name: "aws_sd"
  ec2_sd_configs:
    - port: 9090
      region: us-east-1
      access_key: ''
      secret_key: ''
      filters:
        - name: tag:Name
          values:
            - dev-.*

Understanding Each Line

job_name: “aws_sd”

This is the name of the scrape job for AWS service discovery.

ec2_sd_configs:

This tells Prometheus to use AWS EC2 service discovery. Prometheus will automatically query the AWS API to find EC2 instances.

– port: 9090

This is the port number where Prometheus expects the exporter to be running on each EC2 instance. If my instances use node_exporter on port 9100, I should change this to 9100.

region: us-east-1

This is the AWS region where my EC2 instances are running. Examples: us-east-1, us-west-2, eu-west-1.

access_key: ” secret_key: ”

These are AWS credentials for authentication. The empty quotes mean no key is provided. I should never write actual keys in the configuration file. Instead, I should use IAM roles or environment variables.

filters:

This section helps me select which EC2 instances to discover.

– name: tag:Name

This filters by the Name tag of EC2 instances.

values: – dev-.*

This is a regular expression. It matches instances with names starting with “dev-” followed by any characters. Examples:

• dev-webserver (matches)
• dev-database (matches)
• prod-webserver (does not match)

Relabel Configs Explanation

relabel_configs:
  - source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]
    target_label: instance

First Relabel Rule

source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]

This takes two pieces of information:

• __meta_ec2_tag_Name – The Name tag of the EC2 instance
• __meta_ec2_private_ip – The private IP address

target_label: instance

This creates a new label called “instance” that combines both values. The default separator is semicolon, so the result looks like: “dev-webserver;10.0.1.5”

Second Relabel Rule

- source_labels: [__meta_ec2_tag_Name]
  regex: dev-.*
  action: drop

source_labels: [__meta_ec2_tag_Name]

This takes the Name tag of the EC2 instance.

regex: dev-.*

This matches any name starting with “dev-“.

action: drop

This tells Prometheus to NOT monitor instances that match the regex. If I change “drop” to “keep”, it will monitor only instances that match and ignore everything else.

Third Relabel Rule

- source_labels: [__meta_ec2_public_ip]
  replacement: ${1}:9090
  target_label: __address__

source_labels: [__meta_ec2_public_ip]

This takes the public IP address of the EC2 instance.

replacement: ${1}:9090

This adds the port number to the IP. The ${1} means “take the first source label value”. So if the public IP is 54.123.45.67, the result is 54.123.45.67:9090.

target_label: address

This is a special label that tells Prometheus where to scrape metrics from. By default, EC2 service discovery uses private IPs. But if I run Prometheus locally, I cannot reach private IPs. So I replace them with public IPs.

Why Use Service Discovery

Static Configuration (Manual)

- job_name: "manual"
  static_configs:
    - targets:
      - 192.168.1.10:9100
      - 192.168.1.11:9100

Problems:

• I must manually update prometheus.yml when servers change
• I must restart Prometheus
• Does not work with auto-scaling

Service Discovery (Automatic)

- job_name: "automatic"
  file_sd_configs:
    - files:
      - /etc/prometheus/files_sd/*.yml

Benefits:

• Prometheus automatically detects changes
• No restart needed
• Works with auto-scaling
• I can automate updates with scripts

Complete Working Example

Here is a complete configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets:
        - localhost:9090

  - job_name: "files_sd"
    file_sd_configs:
      - files:
        - /etc/prometheus/files_sd/*.yml

  - job_name: "aws_sd"
    ec2_sd_configs:
      - port: 9100
        region: us-east-1
        filters:
          - name: tag:Monitor
            values:
              - "true"
          - name: instance-state-name
            values:
              - running
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Name, __meta_ec2_private_ip]
        separator: '-'
        target_label: instance
      - source_labels: [__meta_ec2_public_ip]
        replacement: ${1}:9100
        target_label: __address__

Target file example:

# File: /etc/prometheus/files_sd/servers.yml

- targets:
  - localhost:9100
  labels:
    team: "Team Reza1"
    environment: "production"

- targets:
  - localhost:9200
  labels:
    team: "Team Reza2"
    environment: "production"