A health check that lies is worse than no health check at all. This article is about making them tell the truth.<\/p>\n\n\n\n
Two questions, not one<\/h2>\n\n\n\n
It’s tempting to build one “health” endpoint that checks everything. But Kubernetes asks two different questions, and it reacts to each one differently.<\/p>\n\n\n\n
Liveness asks: is the process alive, or is it stuck? If liveness fails, Kubernetes restarts the container. That’s the right fix for a process that has frozen.<\/p>\n\n\n\n
Readiness asks: can this instance serve requests right now? If readiness fails, Kubernetes stops sending it new traffic, but doesn’t restart anything. When readiness passes again, traffic comes back.<\/p>\n\n\n\n
The most important rule of this whole part: liveness must never check dependencies. Imagine \/healthz checked the database. The database has a 30-second hiccup. Every pod’s liveness fails, so Kubernetes restarts every pod. The new pods start, check the database, which is still down, fail, and get restarted again. A short database blip has become a full outage, caused by the health check itself.<\/p>\n\n\n\n
The last box shows the real test: with Postgres stopped, \/healthz stayed 200 and \/readyz returned 503. Alive, so don’t restart it. Not ready, so stop sending it traffic.<\/p>\n\n\n\n Both live in cloud\/health.py. They are plain Django views, not DRF views. DRF would apply my global settings from Part 3: JWT authentication and IsAuthenticated. Kubernetes doesn’t log in, so the probes would get a 401.<\/p>\n\n\n\n That’s the whole thing, on purpose. If this view runs at all, the process is alive. That’s the only question liveness asks.<\/p>\n\n\n\n @never_cache adds headers that stop any proxy from caching the answer. A cached “I’m healthy” for an instance that is actually dead would be a lie.<\/p>\n\n\n\n @require_GET only allows GET. Decorators apply from the bottom up, so even a refused request gets the no-cache headers.<\/p>\n\n\n\n One simple contract: return nothing if healthy, raise if not.<\/p>\n\n\n\n SELECT 1 is the smallest real query. It proves PostgreSQL answers, without touching any table.<\/p>\n\n\n\n The cache check writes and reads back. A read of a missing key returns None without any error, but some failures only appear on a write.<\/p>\n\n\n\n The third value in each tuple is critical. Only the database is. Adding a new dependency is one line.<\/p>\n\n\n\n Every check runs, even after one fails. One response shows the whole picture.<\/p>\n\n\n\n except Exception is broad on purpose. Any failure must still produce an answer. A 500 would hide which check failed.<\/p>\n\n\n\n The log gets the full traceback. The response gets only the exception’s class name, like OperationalError or TimeoutError. Error messages can contain hostnames, ports and usernames, and a health endpoint has no authentication. Anyone could read it.<\/p>\n\n\n\n ms measures each check. A dependency that is slowly getting worse shows up here before it fails.<\/p>\n\n\n\n time.monotonic() is the right clock for measuring durations. time.time() can jump if the system clock is adjusted.<\/p>\n\n\n\n 503 Service Unavailable is the standard “I’m alive, but I can’t serve” code. Kubernetes counts any status from 200 to 399 as success.<\/p>\n\n\n\n Both go in config\/urls.py at the top level, outside \/api\/:<\/p>\n\n\n\n Notice: no trailing slash. That’s not a style choice. It’s the fix for this part’s sneakiest bug, which I’ll explain in the problems section.<\/p>\n\n\n\n I tested the checks by breaking dependencies on purpose, in two different ways:<\/p>\n\n\n\n docker compose stop makes a service unreachable. Connections are refused immediately, so failures are fast.<\/p>\n\n\n\n docker compose pause makes a service frozen. Connections are accepted, but nothing ever answers. This is much closer to a real network problem, and it’s much nastier.<\/p>\n\n\n\n Stopping Redis worked perfectly: cache: fail, ConnectionError, and \/readyz stayed 200. Then I paused LocalStack. My curl waited 90 seconds, and my timeout command had to kill it. \/readyz never answered.<\/p>\n\n\n\n The reason: none of my dependency calls had a time limit.<\/p>\n\n\n\n botocore, the library under boto3, waits up to 60 seconds to connect and 60 seconds to read, and then retries.<\/p>\n\n\n\n redis-py has no timeout at all by default. A frozen Redis would block forever.<\/p>\n\n\n\n A dependency that is down is easy. A dependency that hangs takes your app down with it.<\/p>\n\n\n\n This protects every cache call, not only the health check. redis-py’s TimeoutError is a kind of RedisError, so the fallback I built in Part 9 now kicks in after about one second. With Redis paused, \/readyz reported cache: TimeoutError after 1010.7 ms, and the job list itself still returned 200 in 1.03 seconds. Before this fix, both would have hung. One second is generous: Redis normally answers in under 5 milliseconds.<\/p>\n\n\n\n The queue needed something more interesting. My worker’s receive() uses long polling: it waits up to 20 seconds for a message, on purpose (Part 10). A health check must answer in about 1 second. One client can’t do both. So SQSQueue now has two:<\/p>\n\n\n\n The main client has read_timeout=30, which must be longer than the 20-second long poll. Shorter, and every quiet poll would fail. It retries up to 3 attempts in total, using AWS’s standard mode: retries only on errors that might go away, with a random pause that grows each time (exponential backoff with jitter).<\/p>\n\n\n\n The ping client has a one-second budget and exactly one attempt. A health check should report the first failure. Retrying only hides problems and makes the check slow.<\/p>\n\n\n\n **common unpacks a small dictionary with the endpoint and region, so they’re written once.<\/p>\n\n\n\n get_queue_url proves two things at once: SQS answers, and my queue exists.<\/p>\n\n\n\n After all that, \/readyz still hung, for the first request after a code reload. The reason: SQSQueue.__init__ looked up the queue URL with the main client, the slow one. A fresh process built the queue object during the outage, and the constructor sat there for a minute.<\/p>\n\n\n\n The fix was to make the URL lazy:<\/p>\n\n\n\n @cached_property computes the value the first time .url is read, then keeps it. And if that first attempt raises an error, nothing is stored, so the next read tries again. A lookup that fails during an outage doesn’t stay broken. Now creating the queue object makes no network call at all.<\/p>\n\n\n\n The final result, with LocalStack paused: \/readyz answered in 1.06 and 1.02 seconds, with queue: fail, ReadTimeoutError.<\/p>\n\n\n\n While I was stopping LocalStack for these tests, I noticed something worse: the worker died. One failed receive() raised an EndpointConnectionError, nothing caught it, and the container showed Exited (1).<\/p>\n\n\n\n A failed receive() is now logged, the worker waits 5 seconds, and tries again. The pause stops a tight loop from flooding the logs with errors. And a SIGTERM during the sleep still sets running = False, so the graceful shutdown from Part 10 still works.<\/p>\n\n\n\n Only AWS and network errors are caught here. A bug in my code still shows up.<\/p>\n\n\n\n A failed delete() no longer crashes the worker either. SQS will simply redeliver the message, and the idempotency from Part 12 will skip it. No second email. The two mechanisms cover for each other.<\/p>\n\n\n\n Tested: with LocalStack stopped for 15 seconds, the worker stayed Up and logged Could not receive messages twice. When LocalStack came back, it carried on.<\/p>\n\n\n\n There is one place where crashing is right: at startup.<\/p>\n\n\n\n Now that the URL is lazy, I read queue.url on purpose at startup. If the queue is missing, or SQS is unreachable when the worker starts, the worker crashes visibly. A worker that starts and quietly polls nothing looks healthy, but does nothing. That’s worse.<\/p>\n\n\n\n But a crash is only useful if something brings the worker back. So both api and worker got a restart policy in Compose:<\/p>\n\n\n\n Docker restarts a container whose process exits, with growing delays between attempts. unless-stopped means a container that I stopped on purpose stays stopped, so my tests still work. Kubernetes does the same thing, and calls it CrashLoopBackOff.<\/p>\n\n\n\n The real test: I stopped LocalStack, then restarted the worker. It crashed at startup, Docker restarted it again and again, and as soon as LocalStack came back, the log said Worker started. And the status change that had been waiting in the queue during the whole outage was processed straight away: Emailed user 5 … interview -> reviewing. Nothing was lost.<\/p>\n\n\n\n Nine problems in this step. These are the ones worth learning from.<\/p>\n\n\n\n My first URLs were “healthz\/” and “readyz\/”, with a trailing slash, like every other Django URL in the project. Every request to \/healthz returned 301, a redirect to \/healthz\/, from Django’s APPEND_SLASH setting. POST requests returned 500, because Django refuses to redirect a POST.<\/p>\n\n\n\n I found it with curl -i, which showed the 301 and an HTML content type, and with json.tool failing with Expecting value.<\/p>\n\n\n\n Here is why this matters far beyond curl. Kubernetes treats any 3xx response as success. My probes would have passed, every time, without ever reaching my view. Stopping Postgres, freezing Redis: nothing would have changed. A health check that can’t fail.<\/p>\n\n\n\n The fix: no trailing slash.<\/p>\n\n\n\n Once the redirect was fixed, stopping Redis and pausing LocalStack still changed nothing. And a frozen LocalStack “answered” in 0.013 seconds, which was impossible.<\/p>\n\n\n\n The cause: my return JsonResponse(…) was indented inside the for loop. So the function returned after the first check, every time. cache and queue never ran. Moving the return four spaces to the left fixed it.<\/p>\n\n\n\n Indentation changing the meaning of my code without any error. I’ve now met this bug in Part 4, and here.<\/p>\n\n\n\n With the two clients in place, my ping still took 2.3 to 3.0 seconds, not 1. My first guess was a cold start. That was wrong: a second, warm request was just as slow.<\/p>\n\n\n\n The real cause was my own config. In botocore’s Config(retries={…}), max_attempts counts retries, not attempts. So max_attempts: 1 meant one try plus one retry, with a random pause of up to a second between them. That matches 2.25 and 3.0 seconds exactly. The key I needed was total_max_attempts, which counts all attempts. To make it more confusing, the AWS_MAX_ATTEMPTS environment variable counts all attempts too. After the fix: 1.06 and 1.02 seconds.<\/p>\n\n\n\n I added restart: unless-stopped, ran docker compose restart, and the worker still didn’t come back after a crash.<\/p>\n\n\n\n The reason: a restart policy is part of a container’s configuration, fixed when the container is created. docker compose restart reuses the existing container, which was created three days earlier. It took docker compose up -d, which recreates containers whose configuration changed. After that, docker inspect showed restart=unless-stopped for both.<\/p>\n\n\n\n When timeout killed a hanging curl, my next command still printed a nice JSON result. It was the result from the previous test, still sitting in \/tmp\/ready.json. The identical millisecond values gave it away. My test helper now deletes the file first, and prints curl’s exit code (124 means it was killed).<\/p>\n\n\n\n This completes the first half of Phase 4’s goal: the container exposes health. The second half is metrics: a \/metrics endpoint for Prometheus, with django-prometheus. That’s where this app finally meets the monitoring stack from my home lab.<\/p>\n\n","protected":false},"excerpt":{"rendered":" Phase 4: making the app operations-ready Phase 3 is done. The app has a cache, a queue and a background worker. Now…<\/p>\n","protected":false},"author":2,"featured_media":275,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"13","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/building-job-board-api-part-13-health-checks-tell-truth-reza-chegini-kubac\/","footnotes":""},"categories":[10,116,9],"tags":[8,111,204,118,3,25,198,112,200],"series":[117],"class_list":["post-277","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-devops","category-django","category-kubernetes","tag-devops","tag-django","tag-health-checks","tag-job-board","tag-kubernetes","tag-python","tag-redis","tag-rest-api","tag-sqs","series-job-board-api"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-13-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=277"}],"version-history":[{"count":1,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/277\/revisions"}],"predecessor-version":[{"id":278,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/277\/revisions\/278"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/275"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=277"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![\"Liveness](\"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-13-diagram.png\")
<\/figure>\n\n\n\nStep 1: The two endpoints<\/h2>\n\n\n\n
\/healthz<\/h3>\n\n\n\n
@never_cache\n@require_GET\ndef healthz(request):\n return JsonResponse({\"status\": \"ok\"})<\/code><\/pre>\n\n\n\nThe checks<\/h3>\n\n\n\n
def _check_database():\n with connection.cursor() as cursor:\n cursor.execute(\"SELECT 1\")\n cursor.fetchone()\n\n\ndef _check_cache():\n cache.set(\"healthcheck\", \"ok\", timeout=5)\n assert cache.get(\"healthcheck\") == \"ok\"\n\n\ndef _check_queue():\n get_queue().ping()\n\n\nCHECKS = [\n (\"database\", _check_database, True),\n (\"cache\", _check_cache, False),\n (\"queue\", _check_queue, False),\n]<\/code><\/pre>\n\n\n\n\/readyz<\/h3>\n\n\n\n
@never_cache\n@require_GET\ndef readyz(request):\n results = {}\n overall_ok = True\n\n for name, check, critical in CHECKS:\n started = time.monotonic()\n try:\n check()\n results[name] = {\n \"status\": \"ok\",\n \"ms\": round((time.monotonic() - started) * 1000, 1),\n }\n except Exception as e:\n results[name] = {\n \"status\": \"fail\",\n \"error\": type(e).__name__,\n \"ms\": round((time.monotonic() - started) * 1000, 1),\n }\n logger.exception(\"Health check failed: %s\", name)\n if critical:\n overall_ok = False\n\n return JsonResponse(\n {\"status\": \"ready\" if overall_ok else \"not_ready\", \"checks\": results},\n status=200 if overall_ok else 503,\n )<\/code><\/pre>\n\n\n\npath(\"healthz\", healthz),\npath(\"readyz\", readyz),<\/code><\/pre>\n\n\n\nStep 2: What happens when a dependency freezes?<\/h2>\n\n\n\n
No call had a time limit<\/h3>\n\n\n\n
Redis: a one-second budget<\/h3>\n\n\n\n
CACHES = {\n \"default\": {\n \"BACKEND\": \"django_redis.cache.RedisCache\",\n \"LOCATION\": env(\"REDIS_URL\"),\n \"OPTIONS\": {\n \"CLIENT_CLASS\": \"django_redis.client.DefaultClient\",\n \"SOCKET_CONNECT_TIMEOUT\": 1, # seconds\n \"SOCKET_TIMEOUT\": 1, # seconds\n \"RETRY_ON_TIMEOUT\": True,\n # ... as in Part 9 ...\n },\n }\n}<\/code><\/pre>\n\n\n\nSQS: two clients for two different jobs<\/h3>\n\n\n\n
self.client = boto3.client(\n \"sqs\",\n config=Config(\n retries={\"mode\": \"standard\", \"max_attempts\": 3},\n connect_timeout=5,\n read_timeout=30,\n **common,\n ),\n)\nself._ping_client = boto3.client(\n \"sqs\",\n config=Config(\n retries={\"total_max_attempts\": 1},\n connect_timeout=1,\n read_timeout=1,\n **common,\n ),\n)<\/code><\/pre>\n\n\n\ndef ping(self) -> None:\n self._ping_client.get_queue_url(QueueName=self.name)<\/code><\/pre>\n\n\n\nNo network calls in a constructor<\/h3>\n\n\n\n
@cached_property\ndef url(self) -> str:\n return self.client.get_queue_url(QueueName=self.name)[\"QueueUrl\"]<\/code><\/pre>\n\n\n\nStep 3: A worker that survives outages<\/h2>\n\n\n\n
Keep polling through an outage<\/h3>\n\n\n\n
while running:\n try:\n messages = queue.receive()\n except (BotoCoreError, ClientError):\n logger.exception(\"Could not receive messages; retrying in 5 s\")\n for _ in range(50):\n if not running:\n break\n time.sleep(0.1)\n continue\n\n for message in messages:\n try:\n handle(message)\n except Exception:\n logger.exception(\"Failed to handle message %s\", message.message_id)\n continue\n\n try:\n message.delete()\n except (BotoCoreError, ClientError):\n logger.exception(\"Could not delete message %s\", message.message_id)<\/code><\/pre>\n\n\n\nFail fast at startup<\/h3>\n\n\n\n
queue = get_queue()\nlogger.info(\"Worker started, queue %s\", queue.url)<\/code><\/pre>\n\n\n\nSomething has to restart it<\/h3>\n\n\n\n
restart: unless-stopped<\/code><\/pre>\n\n\n\nProblems I hit<\/h2>\n\n\n\n
1. The redirect that passed every probe<\/h3>\n\n\n\n
2. \/readyz only checked the database<\/h3>\n\n\n\n
3. A retry setting that means the opposite of what it says<\/h3>\n\n\n\n
4. The restart policy that wasn’t there<\/h3>\n\n\n\n
5. A check that showed old results<\/h3>\n\n\n\n
Smaller ones<\/h3>\n\n\n\n
\n
The patterns<\/h3>\n\n\n\n
\n
Known gaps<\/h3>\n\n\n\n
\n
What I learned<\/h3>\n\n\n\n
\n
What comes next<\/h2>\n\n\n\n