my blog

$ cat prometheus/part-02.md

Understanding Prometheus Functions

I recently started learning Prometheus, and I wanted to understand how its functions work. Prometheus functions help me change, calculate, and analyze metric data. They make it easier to build dashboards and alerts in Grafana.

Prometheus works with two main data types:

  • Instant vector: one value for each time series at one moment in time
  • Range vector: many values for each time series across a time range (for example [5m])

When a function ends with _over_time, it works with a range vector. Other functions usually work with instant vectors.

1. Existence Check Functions

absent()

Checks if a metric exists in Prometheus.

How it works

  • If the metric exists, it returns nothing (empty result).
  • If the metric does not exist, it returns 1.

Syntax

absent(v instant-vector)

Examples

absent(node_cpu_seconds_total)
absent(node_cpu_seconds_total{cpu="random"})

The first query returns empty because the metric exists. The second one returns 1 because there is no CPU labeled “random”.

Practical use

absent(up{job="node_exporter"}) == 1

This alert fires when the node exporter stops sending metrics.

absent_over_time()

Same idea, but checks data over a time range.

How it works

  • It looks for samples inside a time range (for example [5m]).
  • If there are no samples, it returns 1.
  • If there are samples, it returns nothing.

Syntax

absent_over_time(v range-vector)

Example

absent_over_time(mysql_up[5m]) == 1

This helps detect if a service did not report data during the last 5 minutes.

2. Mathematical Functions

These functions perform basic math on metric values.

abs()

Makes all numbers positive.

abs(node_filesystem_free_bytes)

Useful when I only care about the size of a change, not if it increased or decreased.

ceil()

Rounds numbers up to the next integer.

ceil(node_load1)

If the value is 1.2, it becomes 2. This helps when I want to plan resources conservatively.

floor()

Rounds numbers down to the nearest lower integer.

floor(node_memory_MemAvailable_bytes / 1024 / 1024 / 1024)

If the value is 5.9, it becomes 5. This is good for showing available resources in whole numbers.

FunctionInputOutputUse
abs()-5.75.7remove negatives
ceil()5.26round up
floor()5.95round down

3. Range Limiting Functions

These functions control or limit metric values. They are useful for keeping dashboards readable and focusing on important ranges.

clamp()

Keeps values inside a specific range.

Syntax

clamp(v instant-vector, min, max)

Example

clamp(http_request_duration_seconds, 0, 10)

Values smaller than 0 become 0. Values greater than 10 become 10.

clamp_min()

Sets a minimum value.

Syntax

clamp_min(v instant-vector, min)

Example

clamp_min(node_filesystem_size_bytes, 1000000000)

Values smaller than 1GB become 1GB.

clamp_max()

Sets a maximum value.

Syntax

clamp_max(v instant-vector, max)

Example

clamp_max(rate(node_cpu_seconds_total[5m]) * 100, 100)

Values greater than 100 are set to 100.

FunctionInputOutputUse
clamp(v,100,1000)50100keep inside range
clamp_min(v,100)50100set minimum
clamp_max(v,1000)50001000set maximum

4. Time Functions

Prometheus includes functions that work with time values. All times use the UTC timezone.

time()

Returns the current time in Unix format (seconds since 1970).

time()

Useful for comparing current time with timestamps from metrics.

timestamp()

Shows the time when each metric was collected.

timestamp(node_cpu_seconds_total)

Example

time() - timestamp(up{job="node_exporter"}) > 120

This shows if the exporter has not sent data for more than 2 minutes.

day_of_month()

Gives the day of the month (1–31).

day_of_month(timestamp(node_cpu_seconds_total))

Helps when creating monthly reports or filters.

day_of_week()

Gives the day of the week (1–7). 1 = Monday, 7 = Sunday.

day_of_week(timestamp(http_requests_total))

Helps compare weekday and weekend performance.

FunctionReturnsUse
time()current Unix timetime calculations
timestamp(v)scrape timecheck freshness
day_of_week(v)1–7weekday analysis
day_of_month(v)1–31monthly patterns

5. Sorting Functions

These functions change the order of query results. They help find the smallest or largest metric values.

sort()

Sorts results from smallest to largest.

sort(node_filesystem_avail_bytes{mountpoint="/"})

Useful to find systems with the lowest values, like least free space.

sort_desc()

Sorts results from largest to smallest.

sort_desc(rate(node_network_receive_bytes_total[5m]))

Often used with topk() to show the top N values.

topk(5, sort_desc(rate(node_network_receive_bytes_total[5m])))
FunctionOrderExampleUse
sort()ascending1, 2, 3find low values
sort_desc()descending3, 2, 1find top usage

6. Aggregation Over Time Functions

These functions calculate averages, sums, or other values over time ranges. They are very useful for creating dashboards that show trends, not just single points.

avg_over_time()

Calculates the average of all values within a specified range.

Example:

avg_over_time(http_request_duration_seconds[5m])

sum_over_time()

Returns the sum of all values within a range.

Example:

sum_over_time(increase(http_requests_total[1m])[5m:])

min_over_time()

Finds the smallest value within the time range.

Example:

min_over_time(node_filesystem_avail_bytes{mountpoint="/"}[24h])

max_over_time()

Finds the largest value within the time range.

Example:

max_over_time(node_hwmon_temp_celsius[24h])

count_over_time()

Counts the number of samples within the time range.

Example:

count_over_time(up{job="node_exporter"}[5m])

stddev_over_time()

Shows the standard deviation, helping you understand how stable or variable a metric is.

Example:

stddev_over_time(http_request_duration_seconds[10m])

stdvar_over_time()

Shows the variance, another indicator of how much values fluctuate within the range.

Example:

stdvar_over_time(node_load1[10m])

7. Practical Examples for Dashboards

Top 5 CPUs by usage

topk(5, sort_desc(rate(node_cpu_seconds_total{mode="user"}[5m])))

Filesystems with least free space

topk(10, sort(node_filesystem_avail_bytes))

Slowest HTTP responses

topk(5, sort_desc(http_request_duration_seconds))

Average CPU usage (smoothed)

avg_over_time(rate(node_cpu_seconds_total{mode="user"}[5m])[10m:1m])

Peak memory usage in 24 hours

max_over_time(node_memory_MemUsed_bytes[24h])

Detect exporter not reporting

time() - timestamp(up{job="node_exporter"}) > 120

Response time stability

stddev_over_time(http_request_duration_seconds[10m])

$ ls -t root-access/ | head -3

Hello world! uncategorized 1 min Building a Job Board API, Part 1: The Foundation job-board-api · 1/1 16 min Part 13 – Kubernetes Autoscaling in Action: A FastAPI Microservice That Scales Itself home-lab · 13/13 6 min

Leave a comment

Your email address will not be published. Required fields are marked *