{"id":165,"date":"2026-06-18T12:00:00","date_gmt":"2026-06-18T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=165"},"modified":"2026-10-30T07:17:16","modified_gmt":"2026-10-30T07:17:16","slug":"part-06-monitoring-stack-prometheus-grafana-loki-promtail-and-jaeger","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=165","title":{"rendered":"Part 06 &#8211; Monitoring Stack: Prometheus, Grafana, Loki, Promtail and Jaeger"},"content":{"rendered":"<h3 class=\"wp-block-heading\">Overview<\/h3><p class=\"wp-block-paragraph\">In Part 5 we added persistent storage with Longhorn. Now the cluster can run stateful applications without losing data. In this article I deploy the full observability stack, metrics, logs, and traces, the three pillars of modern monitoring.<\/p><p class=\"wp-block-paragraph\">By the end of this article you will have:<\/p><ul class=\"wp-block-list\"><li>Prometheus scraping metrics from every pod and node automatically<\/li><li>Grafana displaying dashboards for the entire cluster<\/li><li>Loki and Promtail collecting and aggregating logs from every pod<\/li><li>Jaeger for distributed tracing across microservices<\/li><\/ul><p class=\"wp-block-paragraph\">All accessible via grafana.rezaops.com, prometheus.rezaops.com, and so on.<\/p><h3 class=\"wp-block-heading\">The Three Pillars of Observability<\/h3><p class=\"wp-block-paragraph\">Metrics &#8211; numerical measurements over time (CPU %, memory MB, request count). Prometheus scrapes these from every pod every 15 seconds.<\/p><p class=\"wp-block-paragraph\">Logs &#8211; text output from applications (ERROR: database connection failed). Promtail ships these to Loki which stores and indexes them.<\/p><p class=\"wp-block-paragraph\">Traces &#8211; the path a single request takes through multiple services (API to auth service to database to cache). Jaeger records these end-to-end journeys.<\/p><p class=\"wp-block-paragraph\">Together they let you answer: is something broken, what exactly broke, and why did it break.<\/p><h3 class=\"wp-block-heading\">Role: monitoring<\/h3><p class=\"wp-block-paragraph\">All five tools are installed by the monitoring role running on jarvis.<\/p><h3 class=\"wp-block-heading\">Full tasks\/main.yml<\/h3><pre class=\"wp-block-code\"><code>- name: Add Helm repos for monitoring\n  shell: |\n    helm repo add prometheus-community \\\n      https:\/\/prometheus-community.github.io\/helm-charts\n    helm repo add grafana https:\/\/grafana.github.io\/helm-charts\n    helm repo add jaegertracing https:\/\/jaegertracing.github.io\/helm-charts\n    helm repo update\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Three repos added in one task. helm repo update refreshes the index so we always get the latest available versions.<\/p><pre class=\"wp-block-code\"><code>- name: Create monitoring namespace\n  shell: |\n    kubectl create namespace monitoring \\\n      --dry-run=client -o yaml | kubectl apply -f -\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">All monitoring tools share the monitoring namespace. Grouping related tools in one namespace makes it easy to see everything at once:<\/p><pre class=\"wp-block-code\"><code>kubectl get pods -n monitoring\nkubectl get svc -n monitoring<\/code><\/pre><h3 class=\"wp-block-heading\">Installing kube-prometheus-stack<\/h3><pre class=\"wp-block-code\"><code>- name: Install kube-prometheus-stack\n  shell: |\n    helm upgrade --install kube-prometheus-stack \\\n      prometheus-community\/kube-prometheus-stack \\\n      --namespace monitoring \\\n      --set grafana.adminPassword={{ grafana_admin_password }} \\\n      --set prometheus.prometheusSpec.retention={{ prometheus_retention }} \\\n      --set grafana.service.type=LoadBalancer \\\n      --set prometheus.service.type=LoadBalancer \\\n      --set alertmanager.service.type=LoadBalancer \\\n      --set prometheus.prometheusSpec.nodeSelector.\"workload\"=monitoring \\\n      --set grafana.nodeSelector.\"workload\"=monitoring \\\n      --set alertmanager.alertmanagerSpec.nodeSelector.\"workload\"=monitoring\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">kube-prometheus-stack is a single Helm chart that installs Prometheus, Grafana, Alertmanager, Node Exporter, kube-state-metrics, and the Prometheus Operator. Installing all of these separately would require managing five Helm releases and making sure they are all compatible. The stack handles this for you.<\/p><p class=\"wp-block-paragraph\">Helm values explained:<\/p><ul class=\"wp-block-list\"><li>grafana.adminPassword, homelab123, Grafana login password<\/li><li>prometheus.prometheusSpec.retention, 15d, keep 15 days of metrics data<\/li><li>grafana.service.type, LoadBalancer, Grafana gets a real IP from MetalLB<\/li><li>prometheus.service.type, LoadBalancer, Prometheus UI gets a real IP<\/li><li>alertmanager.service.type, LoadBalancer, Alertmanager UI gets a real IP<\/li><li>nodeSelector.&#8221;workload&#8221;=monitoring, pins all pods to jarvis<\/li><\/ul><p class=\"wp-block-paragraph\">Why retention=15d? Prometheus stores time-series data on disk. The longer the retention, the more disk space used. 15 days is a good balance for a home lab, enough history to spot trends without filling the disk. A 2-node cluster with around 1000 metrics generates about 1-2GB per day, so 15 days is roughly 15-30GB.<\/p><p class=\"wp-block-paragraph\">Why nodeSelector on every component? Without it, Kubernetes might schedule Grafana on thor and Prometheus on jarvis. This creates unnecessary cross-node traffic and makes debugging harder. Pinning everything to jarvis keeps monitoring self-contained.<\/p><pre class=\"wp-block-code\"><code>- name: Wait for Grafana to be ready\n  shell: |\n    kubectl wait --namespace monitoring \\\n      --for=condition=ready pod \\\n      --selector=app.kubernetes.io\/name=grafana \\\n      --timeout=120s\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml\n  retries: 3\n  delay: 20<\/code><\/pre><p class=\"wp-block-paragraph\">We specifically wait for Grafana because it is the UI we will use to verify everything else is working.<\/p><h3 class=\"wp-block-heading\">What Prometheus Scrapes Automatically<\/h3><p class=\"wp-block-paragraph\">The kube-prometheus-stack configures Prometheus to scrape metrics from:<\/p><ul class=\"wp-block-list\"><li>Kubernetes API server, cluster health metrics<\/li><li>Node Exporter on jarvis, CPU, memory, disk, network<\/li><li>Node Exporter on thor, CPU, memory, disk, network<\/li><li>kube-state-metrics, pod status and deployment health<\/li><li>Alertmanager, alert metrics<\/li><li>CoreDNS, DNS query metrics<\/li><li>Kubelet on both nodes, container metrics<\/li><\/ul><p class=\"wp-block-paragraph\">You get hundreds of metrics out of the box without any configuration. Your pods can expose additional metrics by adding a ServiceMonitor CRD, but the baseline cluster metrics are automatic.<\/p><p class=\"wp-block-paragraph\">Most useful default queries:<\/p><pre class=\"wp-block-code\"><code># Node CPU usage\n100 - (avg by(instance)(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)\n\n# Node memory usage\n(1 - node_memory_MemAvailable_bytes \/ node_memory_MemTotal_bytes) * 100\n\n# Pod restart count\nincrease(kube_pod_container_status_restarts_total[1h])\n\n# Disk usage per node\n(node_filesystem_size_bytes - node_filesystem_free_bytes) \/ node_filesystem_size_bytes * 100<\/code><\/pre><h3 class=\"wp-block-heading\">Installing Loki<\/h3><pre class=\"wp-block-code\"><code>- name: Install Loki\n  shell: |\n    helm upgrade --install loki grafana\/loki \\\n      --namespace monitoring \\\n      --set deploymentMode=SingleBinary \\\n      --set loki.auth_enabled=false \\\n      --set loki.commonConfig.replication_factor=1 \\\n      --set loki.storage.type=filesystem \\\n      --set singleBinary.replicas=1 \\\n      --set singleBinary.nodeSelector.\"workload\"=monitoring \\\n      --set read.replicas=0 \\\n      --set write.replicas=0 \\\n      --set backend.replicas=0 \\\n      --set loki.useTestSchema=true\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Helm values explained:<\/p><ul class=\"wp-block-list\"><li>deploymentMode, SingleBinary, run as one process instead of distributed components<\/li><li>loki.auth_enabled, false, no authentication needed for a home lab<\/li><li>loki.storage.type, filesystem, store logs on local disk, not S3<\/li><li>singleBinary.replicas, 1, one Loki pod<\/li><li>read\/write\/backend.replicas, 0, disable distributed components<\/li><li>loki.useTestSchema, true, use built-in schema config<\/li><\/ul><p class=\"wp-block-paragraph\">Why SingleBinary mode? The default Loki chart in newer versions expects a distributed deployment with separate read, write, and backend components, each requiring object storage like S3. This is production-grade but overkill for a home lab. SingleBinary mode runs everything in one pod and stores logs on the local filesystem.<\/p><p class=\"wp-block-paragraph\">Why useTestSchema=true? Newer Loki versions require you to define a schema config. useTestSchema=true uses a built-in default schema. In production you would define your own, but for a home lab this is the right shortcut.<\/p><h3 class=\"wp-block-heading\">Installing Promtail<\/h3><pre class=\"wp-block-code\"><code>- name: Install Promtail\n  shell: |\n    helm upgrade --install promtail grafana\/promtail \\\n      --namespace monitoring \\\n      --set loki.serviceName=loki\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Promtail is the log shipper. It runs as a DaemonSet, one pod per node, and watches \/var\/log\/pods\/ on each node, tails log files from every running container, adds labels like pod name, namespace, and container name, and ships logs to Loki.<\/p><p class=\"wp-block-paragraph\">Because it is a DaemonSet, you do not specify a nodeSelector. It automatically runs on every node in the cluster. One Promtail on jarvis, one on thor.<\/p><h3 class=\"wp-block-heading\">Installing Jaeger<\/h3><pre class=\"wp-block-code\"><code>- name: Install Jaeger\n  shell: |\n    helm upgrade --install jaeger jaegertracing\/jaeger \\\n      --namespace monitoring \\\n      --set allInOne.enabled=true \\\n      --set collector.enabled=false \\\n      --set query.enabled=false \\\n      --set agent.enabled=false \\\n      --set storage.type=memory \\\n      --set allInOne.service.type=LoadBalancer \\\n      --set allInOne.nodeSelector.\"workload\"=monitoring\n  environment:\n    KUBECONFIG: \/etc\/rancher\/k3s\/k3s.yaml<\/code><\/pre><p class=\"wp-block-paragraph\">Helm values explained:<\/p><ul class=\"wp-block-list\"><li>allInOne.enabled, true, run all Jaeger components in one pod<\/li><li>collector\/query\/agent.enabled, false, disable separate component deployments<\/li><li>storage.type, memory, store traces in memory not Elasticsearch<\/li><li>allInOne.service.type, LoadBalancer, Jaeger UI gets a real IP<\/li><\/ul><p class=\"wp-block-paragraph\">Why allInOne mode? Production Jaeger uses separate collector, query, and agent components backed by Elasticsearch or Cassandra. For a home lab, allInOne puts everything in one pod and stores traces in memory. Simple, though traces do not persist across pod restarts.<\/p><h3 class=\"wp-block-heading\">Verifying the Stack<\/h3><p class=\"wp-block-paragraph\">Get all service IPs:<\/p><pre class=\"wp-block-code\"><code>kubectl get svc -n monitoring | grep LoadBalancer\nkube-prometheus-stack-grafana        LoadBalancer  10.0.0.202   80\nkube-prometheus-stack-prometheus     LoadBalancer  10.0.0.203   9090\nkube-prometheus-stack-alertmanager   LoadBalancer  10.0.0.204   9093\njaeger-query                         LoadBalancer  10.0.0.205   16686<\/code><\/pre><p class=\"wp-block-paragraph\">Open Grafana at http:\/\/grafana.rezaops.com and login with admin and your password. Pre-built dashboards are already there:<\/p><ul class=\"wp-block-list\"><li>Kubernetes \/ Compute Resources \/ Cluster, CPU and memory across the whole cluster<\/li><li>Kubernetes \/ Compute Resources \/ Node, per-node breakdown<\/li><li>Node Exporter \/ Full, detailed OS metrics including disk I\/O, network, and filesystem<\/li><\/ul><p class=\"wp-block-paragraph\">Check Promtail is running on both nodes:<\/p><pre class=\"wp-block-code\"><code>kubectl get pods -n monitoring | grep promtail\npromtail-xxxxx   1\/1   Running   jarvis\npromtail-yyyyy   1\/1   Running   thor<\/code><\/pre><p class=\"wp-block-paragraph\">One pod per node means logs from all containers on both nodes are being collected.<\/p><p class=\"wp-block-paragraph\">In Grafana, go to Explore, change the data source to Loki, and run:<\/p><pre class=\"wp-block-code\"><code>{namespace=\"monitoring\"}<\/code><\/pre><p class=\"wp-block-paragraph\">You should see logs from all pods in the monitoring namespace streaming in real time.<\/p><h3 class=\"wp-block-heading\">Useful Queries<\/h3><p class=\"wp-block-paragraph\">Prometheus (PromQL):<\/p><pre class=\"wp-block-code\"><code># Which pods are using the most memory\ntopk(5, container_memory_usage_bytes{namespace!=\"kube-system\"})\n\n# Pod restart rate in the last hour\nrate(kube_pod_container_status_restarts_total[1h]) &gt; 0<\/code><\/pre><p class=\"wp-block-paragraph\">Loki (LogQL):<\/p><pre class=\"wp-block-code\"><code># All error logs across the cluster\n{job=\"kubernetes-pods\"} |= \"ERROR\"\n\n# Logs from a specific app\n{namespace=\"myapp\", app=\"django-app\"}\n\n# Error rate over time\nrate({job=\"kubernetes-pods\"} |= \"ERROR\" [5m])<\/code><\/pre><h3 class=\"wp-block-heading\">What&#8217;s Next<\/h3><p class=\"wp-block-paragraph\">In Part 7 I will cover the CI\/CD stack, Jenkins deployed on Kubernetes, the local container registry, and a complete Jenkinsfile pipeline that builds an image, pushes it to the registry, and triggers an ArgoCD deployment.<\/p>","protected":false},"excerpt":{"rendered":"<p>Overview In Part 5 we added persistent storage with Longhorn. Now the cluster can run stateful applications without losing data. In this&hellip;<\/p>\n","protected":false},"author":2,"featured_media":167,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"6","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-06-monitoring-stack-prometheus-grafana-loki-promtail-chegini-odixc\/","footnotes":""},"categories":[10,62,14],"tags":[8,100,66,102,101,19,16],"series":[64],"class_list":["post-165","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-home-lab","category-observability","tag-devops","tag-grafana","tag-home-lab","tag-jaeger","tag-loki","tag-monitoring","tag-prometheus","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-06-monitoring-stack-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/165","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=165"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/165\/revisions"}],"predecessor-version":[{"id":166,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/165\/revisions\/166"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/167"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=165"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=165"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=165"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=165"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}