Building a Job Board API, Part 9: A Redis Cache That Never Lies

Phase 3 starts here

Phase 2 is finished. The API has permissions, match scores, resume uploads and search. Phase 3 adds two things: Redis, and a background worker that sends emails. This part is about Redis.

In Part 8, I wrote that job search uses ILIKE ‘%…%’, which makes PostgreSQL read every row. That’s fine today, but the job list is the most-read endpoint in the whole app. Every candidate opens it, again and again, often with the same filters.

So I put a cache in front of it. A cache keeps a copy of an answer, so the next identical question is answered without asking the database again. The first request for a search runs the SQL query and stores the page in Redis for 60 seconds. Identical requests within that time come straight from Redis.

Adding a cache is easy. The hard part is that it must never lie. When a company edits a job, nobody should see the old version. There’s a well-known joke that the two hardest things in computer science are naming things and cache invalidation. In this series, I’ve already written a lot about naming things. This part is about the other one.

Step 1: Redis in Docker Compose

uv add redis
docker compose up -d --build

redis is redis-py, the Python client. Since Django 4.0, Django has a built-in Redis cache backend that uses it, so I didn’t need any extra Django package.

redis:
    image: redis:7.4-alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 10

Redis 7.4, pinned. On AWS the cache will be ElastiCache, which runs the same Redis 7 family, so the behaviour matches.
redis-cli ping answers PONG only when Redis really accepts commands. “Ready”, not just “started”, the same idea as pg_isready for Postgres. The API waits for this healthcheck before it starts.
No ports: section. Redis has no password by default. A published port would be an open door on my network. Only containers inside the Compose network can reach it.

A detail I got wrong at first, and corrected: I thought Redis would be empty after every restart, because I gave it no volume. It isn’t. The official image declares a volume at /data itself, and Redis saves a snapshot there when it shuts down cleanly. So stop and start keep the data. Only recreating the container loses it. That will matter in the next part, if Redis also becomes the queue.

Django’s cache settings

REDIS_URL=redis://redis:6379/0

redis is the Compose service name, 6379 is Redis’s default port, and /0 is the database number. One Redis server has 16 numbered databases. The cache uses database 0, so the future queue can use another one. That way, clearing the cache can never delete queued messages.

CACHES = {
    "default": {
        "BACKEND": "django.core.cache.backends.redis.RedisCache",
        "LOCATION": env("REDIS_URL"),
        "KEY_PREFIX": "jobboard",
        "TIMEOUT": 300,
    }
}

env(“REDIS_URL”) has no default, so it’s required. My env() helper from Part 1 fails loudly at startup if it’s missing.

KEY_PREFIX is added to every key. A real key looks like jobboard:1:jobs:list:version.

TIMEOUT: 300 is the default lifetime in seconds.

Why Redis, and not Django’s default cache? Without CACHES, Django uses a local-memory cache, private to each process. In production I’ll run several workers in several pods. Each one would have its own separate cache, and clearing it in one process would never reach the others. Redis is one shared cache for all of them.

Step 2: Building the cache key

Every cached page is stored under a key. The rule for a good key is simple to say and easy to get wrong:

The key must contain everything that changes the response, and nothing that doesn’t.

def job_list_cache_key(request):
    params = sorted(request.query_params.lists())
    raw = f"{request.get_host()}|{params}"
    digest = hashlib.sha256(raw.encode()).hexdigest()
    return f"jobs:list:v{_current_version()}:{digest}"

What goes in

The query parameters, sorted. ?location=vancouver&skills=python and ?skills=python&location=vancouver ask the same question. Sorting makes them produce the same key, so the second one is a cache hit. I tested it: MISS, then HIT for the two orders, and MISS for a different value.
The host. This one surprised me. The cached page contains full URLs in its next and previous pagination links. A page cached for localhost:8006 must not be served to someone using my server’s IP, or their “next page” link would point to the wrong address.
A version number, v1793504315. This is the key to invalidation, in the next section.

What stays out

The user. That’s on purpose. The job list looks exactly the same for every logged-in user, so one cached page can serve everyone. But I wrote a warning in my notes: if I ever add a field that is different for each user, the user must be added to the key. Otherwise, one person’s data would be served to another person. That is a classic, serious caching bug.

Why a hash?

hashlib.sha256(…).hexdigest() turns the parameters into 64 characters. Every key has the same length, however long the search is, and the raw search text a user typed never appears in Redis key names.

Step 3: Invalidation without deleting anything

When a company edits a job, every cached page might now be wrong: every search, every filter, every page number. How do you clear all of them?

The obvious way is to find every key that starts with jobs:list: and delete it. But that’s slow, and it races: a new page can be written while you are still deleting. So I used a different idea: versioned keys.

One number lives in Redis: the current version. It’s part of every page’s key. To invalidate everything, I just increase that number by one. Every request now builds a key with the new version, finds nothing, and runs a fresh query. The old pages are still in Redis, but nobody will ever read them again, and they expire on their own within 60 seconds.

def invalidate_job_list():
    def bump():
        try:
            cache.incr(JOB_LIST_VERSION_KEY)
        except ValueError:
            cache.add(JOB_LIST_VERSION_KEY, int(time.time()), timeout=None)
        except RedisError:
            logger.warning("Could not invalidate the job list cache", exc_info=True)

    transaction.on_commit(bump)

cache.incr increases the number inside Redis, in one atomic step. If two processes do it at the same moment, both count. No update is lost.

If the version key doesn’t exist, incr raises ValueError, so I start a new one.

If Redis is down, I log and carry on. The job was saved successfully. Failing the company’s request because of the cache would be wrong.

Why the first version is a timestamp, not 1

def _current_version():
    version = cache.get(JOB_LIST_VERSION_KEY)
    if version is None:
        cache.add(JOB_LIST_VERSION_KEY, int(time.time()), timeout=None)
        version = cache.get(JOB_LIST_VERSION_KEY)
    return version

The first version is the current Unix time, like 1793504315. Why? Imagine the version key gets lost (a recreated container, a manual delete), but some old pages are still there. If I restarted counting at 1, the new version 1 could match an old page, and stale data would come back to life. A fresh timestamp is always bigger than any version used before, so that can never happen.

Two more details:

cache.add only writes if the key doesn’t exist yet, in one atomic step. If two requests find the key missing at the same time, only one value wins, and both read that one. Using set could overwrite a version another process had just increased.
timeout=None means the version never expires. It must live longer than every page that uses it. I checked in Redis: ttl returned -1, which means no expiry.

Step 4: When to invalidate

Signals, not views

I could call invalidate_job_list() inside my views, after every create and update. But views only see changes that come through the API. A change in the Django admin, or in the shell, would never clear the cache.

So I used Django signals: built-in event hooks that fire whenever a model is saved or deleted, no matter where the change came from.

@receiver([post_save, post_delete], sender=Job)
@receiver([post_save, post_delete], sender=Company)
def job_list_changed(sender, **kwargs):
    invalidate_job_list()

post_save fires after a create or an update. post_delete fires after a delete.
sender=Job connects it only for that model. The second line does the same for Company, because the job list shows the company’s name. Renaming a company must clear the cache too.
**kwargs collects everything Django sends to the receiver. I don’t need any of it, but without **kwargs Django’s call would fail.
I invalidate on every save, even when nothing visible changed. Creating a new company at signup also clears the cache, even though it has no jobs yet. That’s an occasional extra cache miss, in exchange for code that is simple and always correct.

The receiver must be imported

Here is a trap. @receiver only connects the function when its file is imported. Nothing else in my app imports signals.py. So without one more step, the receivers would never register, and nothing would tell me.

class JobsConfig(AppConfig):
    name = "jobs"

    def ready(self):
        from . import signals  # noqa: F401

ready() runs once, after Django has loaded all the models. It’s the documented place to import signal modules. # noqa: F401 tells linters that the “imported but unused” warning is expected: importing the file is the point. I proved it worked with post_save.has_listeners(Job), which returned True.

transaction.on_commit: not too early

Look again at the last line of invalidate_job_list(): transaction.on_commit(bump). This is the subtle part of the whole step.

If I increased the version immediately, while the save was still inside an open database transaction, this could happen:

A company saves a job. The version goes up, but the transaction hasn’t committed yet.
At that moment, a candidate lists jobs. The new version means a cache miss, so the API queries the database. But the change isn’t committed, so it reads the old data.
That old data is now cached under the new version, and it is served for the next 60 seconds. Invalidation “ran”, and the cache still lies.

on_commit waits until the transaction really commits, then runs bump. And if the transaction rolls back, the callback is thrown away, so nothing is invalidated for a change that never happened. Outside a transaction (Django’s normal autocommit mode, which my API views use), on_commit runs the function immediately.

I tested both halves in the shell:

before = cache.get("jobs:list:version")
try:
    with transaction.atomic():
        Job.objects.get(id=1).save()
        raise RuntimeError("force a rollback")
except RuntimeError:
    pass
after_rollback = cache.get("jobs:list:version")

Job.objects.get(id=1).save()
after_commit = cache.get("jobs:list:version")

print(after_rollback == before, after_commit == before + 1)
# True True

Three values, not one. Before, after a rollback (must be unchanged), and after a commit (must be plus one). Comparing them proves both rules at once.

Step 5: The cached list view

I override the list() method of my JobViewSet. super().list(…) is the original: filters, pagination and the serializer, exactly as in Part 8.

def list(self, request, *args, **kwargs):
    try:
        key = job_list_cache_key(request)
        cached = cache.get(key)
    except RedisError:
        logger.warning("Job list cache unavailable", exc_info=True)
        return super().list(request, *args, **kwargs)

    if cached is not None:
        response = Response(cached)
        response["X-Cache"] = "HIT"
        return response

    response = super().list(request, *args, **kwargs)
    if response.status_code == 200:
        try:
            cache.set(key, response.data, timeout=settings.JOB_LIST_CACHE_SECONDS)
        except RedisError:
            logger.warning("Could not store job list in cache", exc_info=True)
    response["X-Cache"] = "MISS"
    return response

Hit or miss

cache.get returns None for a miss. My cached values are always page dictionaries, never None, so is not None is a safe test.
On a hit, Response(cached) renders the stored data exactly like a fresh page. I proved it: a diff of a MISS body and a HIT body printed identical.
Every response gets an X-Cache: HIT or X-Cache: MISS header. It makes the cache visible, so I can test it with curl and, later, measure it. The X- prefix is the traditional mark for a custom header.
What’s stored is response.data, the Python data before it becomes JSON. Django pickles it, which is Python’s way of turning objects into bytes.

The cache can’t skip security

An important question: if a page is in the cache, can someone get it without a token? No. DRF runs authentication and permissions before it calls list(). My cache code is inside list(), so a request without a valid token never even reaches it. I tested it: a cached query with no token returned 401 and no X-Cache header.

Invalid filters are also safe. ?job_type=freelance is rejected inside super().list(), so it returned 400 both times and was never cached.

When Redis is down

A cache is a copy. PostgreSQL is still the source of truth. So the app must work correctly even with an empty cache, or no cache at all.

Every Redis call is wrapped in try/except RedisError. RedisError is the base class of every redis-py error, so this one line covers a refused connection, a timeout, and everything else. If reading fails, I log a warning and return the normal, uncached response. If storing fails, the user still gets their fresh page.

I tested it for real:

docker compose stop redis

The job list still returned 200, with the correct jobs and no X-Cache header, which is the visible sign of “degraded mode”. The log said Job list cache unavailable. Then I started Redis again, and caching came back.

This is graceful degradation: a cache outage must never become an API outage. The site gets a bit slower, but it never breaks.

Problems I hit

1. The function nobody called

The main bug of this part. After a company edited a job title, the next job list was a HIT, with the old title. The cache was lying, which is exactly what I had worked so hard to prevent.

I narrowed it down step by step:

The version number was the same before and after the PATCH. So invalidation hadn’t happened.
The rollback test printed True False: no bump even after a commit.
I called invalidate_job_list() directly in the shell. The version was still unchanged.
post_save.has_listeners(Job) returned True. So the signals were connected. The problem was inside invalidate_job_list.
I searched the logs for my “Could not invalidate” warning. Nothing. So bump had never even run, not even to fail.

Then I printed the whole function:

sed -n '/def invalidate_job_list/,$p' jobs/cache.py

It ended right after bump’s body. The last line, transaction.on_commit(bump), was missing.

A nested function is a function defined inside another one. Defining it does nothing. Something has to call it. My code created bump and then simply threw it away. No error, no warning.

I added the missing line. Then the direct call went from 1793504315 to 1793504316, the PATCH was followed by a MISS with the new title, and the rollback test printed True True.

2. A log check that was too short

When I stopped Redis, I checked the logs with docker compose logs –tail=5 and found no warning. But the warning was there. With exc_info=True, a full traceback is printed after the warning line, and it was longer than 5 lines. So my warning was just outside the window. –tail=60 found it. Lesson: when a log check finds nothing, make sure it is looking far enough back.

3. Smaller ones

A wrong attribute in my check. I used _server, but it’s _servers. Python’s own error message said “Did you mean: ‘_servers’?” Read the whole error.
titles: command not found, twice. New terminal, lost helpers. They now live at the bottom of scripts/dev-helper.sh, together with a new xcache helper that prints the X-Cache header.
A wrong claim, corrected. I wrote in my notes that Redis would be empty after a stop and start. The version number survived two stop/start cycles, which proved me wrong. I corrected my notes, and Step 1 of this article.

The pattern

This is the same family of bug I keep meeting in this series: code that exists but isn’t connected. A hook with the wrong name. A receiver in a file nobody imports. And now, a function that is defined and never called. None of them raise an error.

The defence that worked this time was a before and after check. Not “does the output look OK?”, but “was the version N before, and N+1 after?”. Comparing two states catches what looking at one state can’t.

Known gaps

Some changes don’t send signals. QuerySet.update(), bulk operations and raw SQL skip save(), so they never clear the cache. For example, when I verified Acme with .update() back in Part 5, that would not have invalidated anything. The 60-second lifetime is the safety net for these.
No password on Redis locally. On AWS, ElastiCache will get in-transit encryption, an auth token and security groups.
No protection against a cache stampede. When a popular page expires, several requests can miss at the same moment and all query the database together. At this scale, that’s fine.

What I learned

A cache is a disposable copy. The database stays the source of truth, and the app must be correct with an empty cache or no cache at all.
Use one shared cache. A per-process cache can’t be invalidated across several workers or pods.
The key must hold everything that changes the response, and nothing else. Sorted parameters, the host, the version. And the user, the day the response depends on the user.
Invalidate with a version number. One atomic incr makes every old page unreachable, with nothing to search for or delete.
Start the version from a timestamp. A lost version key can never bring old pages back to life.
Use signals to catch every change, from the API, the admin and the shell. And make sure the signal file is imported.
Invalidate after the commit, never before. Too early and the cache can store old data under a new version.
A cache outage must not be an API outage. Catch the error, log it, serve uncached.
A function that is defined is not a function that runs. Prove it ran: compare the state before and after.

What comes next

The second half of Phase 3 is the part that makes this a real distributed system: a queue and a background worker. When a company changes an application’s status, the API will put a message on a queue, and a separate worker process will send the candidate an email. The goal of Phase 3: a status change queues a job, and the worker processes it.