Getting Started with Prometheus Pushgateway

This post explains what Prometheus Pushgateway is, when to use it, how to install it, and how to push metrics from Python. It’s a simple guide for beginners in DevOps and monitoring.

1) What is Pushgateway and Why Use It

Prometheus normally pulls metrics from applications and exporters. But sometimes, scraping directly is not possible. Pushgateway is a small service that lets applications push their metrics to it. Prometheus then scrapes those metrics from Pushgateway.

You need Pushgateway when:

Your jobs are short-lived (cron jobs, backup scripts) and finish before Prometheus can scrape them.
Your application is not reachable (behind a load balancer or firewall).
You want to report results after a job finishes.

It works as a bridge between your short jobs and Prometheus.

2) When to Use Pushgateway

Use Pushgateway for:

Batch or cron jobs that run briefly and then exit.
Jobs behind restricted networks or private servers.
Scripts that finish quickly and cannot expose /metrics while running.

Avoid using it for:

Long-running applications that can be scraped directly. Direct scraping is always better for performance and reliability.

3) Installing Pushgateway

Linux (Ubuntu) – systemd setup

# Download latest version (replace <version>)
wget https://github.com/prometheus/pushgateway/releases/download/v<version>/pushgateway-<version>.linux-amd64.tar.gz
tar xvf pushgateway-<version>.linux-amd64.tar.gz
cd pushgateway-<version>.linux-amd64

# Install
sudo cp pushgateway /usr/local/bin/
sudo chown prometheus:prometheus /usr/local/bin/pushgateway

# Create service
sudo tee /etc/systemd/system/pushgateway.service >/dev/null <<'EOF'
[Unit]
Description=Prometheus Pushgateway
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/pushgateway --web.listen-address=":9091"
Restart=always

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl enable pushgateway
sudo systemctl start pushgateway
sudo systemctl status pushgateway

Open your browser at:

http://localhost:9091/metrics

You should see default metrics.

4) Basic Configuration

Default settings:

Port: 9091
Metrics path: /metrics
Same port handles both metric pushes and Prometheus scraping.

Change port example:

./pushgateway --web.listen-address=":9092"

For systemd, change the ExecStart line and restart the service:

sudo systemctl daemon-reload
sudo systemctl restart pushgateway

5) Connect Pushgateway to Prometheus

Edit your Prometheus configuration (prometheus.yml):

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['localhost:9091']

Restart Prometheus, then open Status → Targets. If you see pushgateway as UP (green), it’s working.

6) Sending Metrics from Python

Install client

python3 -m venv pgw-demo
source pgw-demo/bin/activate
pip install prometheus_client

Simple example

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
import time

# 1) Create a clean registry for this push
registry = CollectorRegistry()

# 2) Create a gauge (can go up or down)
g = Gauge(
    'python_pushgateway_sample_metric',
    'Sample metric pushed from Python to Pushgateway',
    registry=registry
)

# 3) Set a value (here: UNIX timestamp)
g.set(time.time())

# 4) Push to Pushgateway
push_to_gateway(
    'localhost:9091',
    job='python_example_job',
    grouping_key={'instance': 'server-1'},
    registry=registry
)

print("Metric pushed successfully!")

What each part does

CollectorRegistry() Makes a clean container for the metrics you want to push. This avoids sending other metrics by mistake.

Gauge(…) Creates a metric that represents a current value (can go up or down). We pass registry=registry so it belongs to our private registry.

g.set(time.time()) Sets the gauge to any numeric value. Here we use the UNIX timestamp just to have a real number.

push_to_gateway(‘localhost:9091′, job=’python_example_job’, grouping_key=…, registry=…) Sends your registry to Pushgateway.

job becomes a label in Prometheus (use a meaningful name like nightly_backup or disk_scan).
grouping_key becomes extra labels (like instance=’server-1′, env=’prod’).
Each push overwrites the previous metrics for the same job + grouping_key. (If you want to add instead of overwrite, Python also has pushadd_to_gateway.)

7) Understanding CollectorRegistry

The CollectorRegistry is a container that holds your metrics.
The default registry collects every metric your app creates.
For Pushgateway, always use a custom registry to avoid conflicts and only send what you want.

registry = CollectorRegistry()
my_gauge = Gauge('my_metric', 'Description', registry=registry)
push_to_gateway('localhost:9091', job='my_job', registry=registry)

8) Best Practices

Use Pushgateway only when needed (batch jobs, restricted apps).
Always create a custom registry for pushes.
Use clear job names like db_backup_job or file_cleaner.
Add stable labels (environment, instance).
Avoid secrets in labels or metric names.
Disable admin API in production for security:
Avoid too many unique label combinations (high cardinality).

9) Common Issues and Fixes

Metrics not visible: Check http://localhost:9091/metrics. Ensure Prometheus target is UP and using the correct port.

Connection refused: Pushgateway may not be running or port is blocked. Restart the service or allow port 9091.

Duplicate metrics: You used the default registry. Switch to a custom one (CollectorRegistry()).

Old metrics stay visible: Pushgateway keeps data until deleted. Delete with:

curl -X DELETE http://localhost:9091/metrics/job/<job-name>

Access from another machine: Run with:

./pushgateway --web.listen-address=":9091"

and open the port:

sudo ufw allow 9091

10) Summary

Pushgateway is used when Prometheus cannot scrape your app directly.
Perfect for batch jobs, cron tasks, and private apps.
Default port is 9091.
Prometheus scrapes Pushgateway just like any exporter.
Use Python client to push metrics easily.
Always use a custom registry.
Add job name and labels for better organization.
Push once at job completion to avoid missing data.