Part 12 deployed a real application with real complexity – Django, Postgres, a Jenkins pipeline, eleven bugs. This final article goes the other direction. A deliberately minimal FastAPI service, built to do exactly one thing well: prove that Kubernetes horizontal pod autoscaling actually works, with numbers captured live from my own cluster, not a screenshot from a cloud provider’s marketing page.<\/p>
No database. No complex business logic. Two endpoints. The entire point is watching Kubernetes scale a service from one pod to five and back down again, automatically, based on real CPU pressure.<\/p>
A FastAPI microservice with:<\/p>
Live at fastapi.rezaops.com.<\/p>
The entire app is short enough to show in full:<\/p>
from fastapi import FastAPI\nfrom prometheus_fastapi_instrumentator import Instrumentator\nimport os\nimport socket\n\napp = FastAPI(title=\"FastAPI Metrics Service\")\n\nInstrumentator().instrument(app).expose(app)\n\n\n@app.get(\"\/health\")\ndef health():\n return {\n \"status\": \"healthy\",\n \"service\": \"fastapi-metrics\",\n \"version\": os.environ.get(\"APP_VERSION\", \"1.0.0\"),\n \"hostname\": socket.gethostname(),\n }<\/code><\/pre>Two lines do most of the work. Instrumentator().instrument(app).expose(app) wraps every route automatically and exposes a \/metrics endpoint in the exact format Prometheus expects – request counts, latencies, status codes, no manual instrumentation needed.<\/p>
socket.gethostname() in the health response looks unremarkable until you scale to multiple replicas. Then it becomes the easiest possible way to prove that traffic is actually being distributed across different pods rather than hammering one.<\/p>
The Part That Actually Matters – Resource Requests<\/h3>resources:\n requests:\n cpu: \"100m\"\n memory: \"64Mi\"\n limits:\n cpu: \"200m\"\n memory: \"128Mi\"<\/code><\/pre>This is not optional. HorizontalPodAutoscaler calculates CPU utilization as a percentage of the request value, not as an absolute number. Without resources.requests.cpu defined on the container, HPA has nothing to compare actual usage against and simply does not function. This is the single most common reason someone sets up HPA and watches it do nothing.<\/p>
The HPA resource itself:<\/p>
apiVersion: autoscaling\/v2\nkind: HorizontalPodAutoscaler\nspec:\n scaleTargetRef:\n apiVersion: apps\/v1\n kind: Deployment\n name: fastapi-metrics\n minReplicas: 1\n maxReplicas: 5\n metrics:\n - type: Resource\n resource:\n name: cpu\n target:\n type: Utilization\n averageUtilization: 50<\/code><\/pre>Minimum 1 replica, maximum 5, target 50% CPU. K3s ships with metrics-server pre-installed, so this worked without any extra setup – worth checking before assuming you need to install something.<\/p>
One deliberate choice in the Dockerfile: –workers 1. A single uvicorn worker per pod, instead of multiple. This keeps CPU load per replica predictable, so the relationship between traffic and pod count stays the dominant, observable effect rather than getting absorbed by in-process concurrency.<\/p>
Watching It Actually Scale<\/h3>
This is the part worth slowing down for, because the numbers below are not illustrative – they are exactly what happened on my cluster, captured live.<\/p>
Idle baseline:<\/p>
kubectl get hpa -n fastapi-metrics\nfastapi-metrics-hpa cpu: 4%\/50% 1 5 1<\/code><\/pre>Load test:<\/p>
ab -n 100000 -c 50 -H \"Host: fastapi.rezaops.com\" http:\/\/10.0.0.200\/health<\/code><\/pre>100,000 requests, 50 concurrent connections, against a single pod.<\/p>
What happened next, watched live in a second terminal:<\/p>
cpu: 4%\/50% replicas: 1 idle\ncpu: 5%\/50% replicas: 1\ncpu: 50%\/50% replicas: 1 threshold reached\ncpu: 201%\/50% replicas: 1 scale-up triggered\ncpu: 201%\/50% replicas: 5 scaled to max, immediately\ncpu: 137%\/50% replicas: 5\ncpu: 96%\/50% replicas: 5\ncpu: 5%\/50% replicas: 5 load test finished, CPU back to idle<\/code><\/pre>Four new pods went from nothing to Running in roughly two seconds:<\/p>
fastapi-metrics-7748cd54c4-hpt2t Pending -> ContainerCreating -> Running\nfastapi-metrics-7748cd54c4-w72kl Pending -> ContainerCreating -> Running\nfastapi-metrics-7748cd54c4-jbz6b Pending -> ContainerCreating -> Running\nfastapi-metrics-7748cd54c4-sjtnd Pending -> ContainerCreating -> Running<\/code><\/pre>Proving the Load Was Actually Distributed<\/h3>
Watching the replica count go up is one thing. Confirming traffic actually spreads across those replicas is another – this is where that socket.gethostname() line earns its place:<\/p>
for i in {1..10}; do curl -s -H \"Host: fastapi.rezaops.com\" http:\/\/10.0.0.200\/health | grep hostname; done\nfastapi-metrics-7748cd54c4-hpt2t\nfastapi-metrics-7748cd54c4-hpt2t\nfastapi-metrics-7748cd54c4-hpt2t\nfastapi-metrics-7748cd54c4-jbz6b\nfastapi-metrics-7748cd54c4-jbz6b\nfastapi-metrics-7748cd54c4-zpmj7\nfastapi-metrics-7748cd54c4-zpmj7\nfastapi-metrics-7748cd54c4-jbz6b\nfastapi-metrics-7748cd54c4-sjtnd\nfastapi-metrics-7748cd54c4-hpt2t<\/code><\/pre>Four distinct pods answered across ten requests. NGINX Ingress is genuinely round-robin load-balancing across every live replica, not favoring one. This single command turns an abstract architectural claim into something anyone can verify themselves in ten seconds.<\/p>
Scaling Back Down<\/h3>
After the load test ended, the replica count did not drop immediately. It held at 5 for several minutes before settling:<\/p>
kubectl get hpa -n fastapi-metrics\nfastapi-metrics-hpa cpu: 5%\/50% 1 5 1 18m<\/code><\/pre>This delay is intentional – HPA’s stabilization window, by default around five minutes, prevents the autoscaler from flapping pods up and down if traffic is bursty rather than sustained. Scaling a pod up is cheap. Scaling it down too aggressively and then immediately needing it back is expensive and slow. The stabilization window exists specifically to avoid that.<\/p>
The One Bug<\/h3>
Even a minimal project hit one issue, and it was not new – it was the same DNS problem from earlier in the series resurfacing.<\/p>
apr_sockaddr_info_get() for fastapi.rezaops.com: Temporary failure in name resolution<\/code><\/pre>thor’s system resolver was not consistently pointed at the home lab’s CoreDNS, even though querying CoreDNS directly worked fine. Rather than re-debugging the netplan configuration in the middle of a load test, I worked around it:<\/p>
ab -n 100000 -c 50 -H \"Host: fastapi.rezaops.com\" http:\/\/10.0.0.200\/health<\/code><\/pre>NGINX Ingress routes based on the Host header in the request, not on how the client resolved the IP. Hitting the Ingress IP directly with an explicit Host header works identically to hitting the domain – the domain name only needs to be correct in the header, not actually resolved by DNS. A useful pattern any time DNS is the obstacle but the underlying routing logic is what you actually need to test.<\/p>
Final Numbers<\/h3>- Baseline CPU, idle, 1 replica: 4-5%<\/li>
- CPU target threshold: 50%<\/li>
- Peak CPU during load test: 201%<\/li>
- Time to scale 1 to 5 pods: one HPA check cycle, about 15 seconds<\/li>
- Time for all 5 pods to reach Running: about 2 seconds<\/li>
- Stabilization window before scale-down: about 5 minutes<\/li>
- Final state after load test: back to 1 replica<\/li><\/ul>
Why This Matters More Than It Looks<\/h3>
This is a small project on purpose. No database, no pipeline, no eleven bugs to wade through. But it proves something the Django project could not – that the infrastructure built across this entire series actually behaves the way production Kubernetes infrastructure is supposed to behave under load, not just at rest.<\/p>
Anyone can configure an HPA resource and assume it works. Watching CPU climb from 4% to 201%, watching four pods appear in two seconds, watching traffic spread across all five, and watching it cleanly settle back down to one – that is the difference between knowing the YAML syntax and having actually seen the system do what it claims to do.<\/p>
Closing the Series<\/h3>
This is the thirteenth and final article in the Root Access Newsletter home lab series. Thirteen parts, starting from two mini PCs on a desk and ending with a Kubernetes cluster running a monitoring stack, a CI\/CD pipeline, GitOps, two deployed applications, and a self-scaling microservice with the receipts to prove it works.<\/p>
Every error along the way was real. Every fix was the actual fix, not a cleaned-up version. If you are building something similar, I hope this saved you time somewhere. If you are a hiring manager or recruiter reading this, this is what I actually do when something breaks – I find the root cause, I document it, and I fix it properly instead of working around it forever.<\/p>
Thank you for following along.<\/p>","protected":false},"excerpt":{"rendered":"
Overview Part 12 deployed a real application with real complexity – Django, Postgres, a Jenkins pipeline, eleven bugs. This final article goes…<\/p>\n","protected":false},"author":2,"featured_media":185,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"13","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/part-13-kubernetes-autoscaling-action-fastapi-scales-reza-chegini-r4ekc\/","footnotes":""},"categories":[10,62,9],"tags":[113,8,115,66,114,3],"series":[64],"class_list":["post-183","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-home-lab","category-kubernetes","tag-autoscaling","tag-devops","tag-fastapi","tag-home-lab","tag-hpa","tag-kubernetes","series-home-lab"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/10\/part-13-kubernetes-autoscaling-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/183","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=183"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/183\/revisions"}],"predecessor-version":[{"id":184,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/183\/revisions\/184"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/185"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=183"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=183"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=183"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=183"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}