root access

$ cat job-board-api/part-20.md

Building a Job Board API, Part 20: Faking AWS, the Worker and Redis

I’m sharing one of my projects here from the beginning, step by step. It’s a Job Board API built with Django, Django REST Framework, PostgreSQL and Docker Compose, with S3 and SQS (through LocalStack), Redis, and a background worker.

I’m a junior engineer looking for my first role in DevOps, SRE or Cloud. That’s why I build projects like this and write about them in public. It shows how I think, and writing it down helps me really understand it.

In Parts 18 and 19, I tested the models, the permissions and the API. But none of those tests talked to S3 or SQS, and the worker wasn’t tested at all. This part covers the code that depends on things outside the app: AWS, the worker’s loop, email, the Redis cache and the readiness checks.

The question for each one: what stays real, and what gets faked?

Only the database stays real in the tests

Only the database stays real, because the app uses PostgreSQL-only features. Everything else is replaced inside the test process, so the tests are fast, have no network, and start from an empty state every time.

This part also has something new: tests that document known gaps. Some tests don’t check that the code is right. They check exactly how it’s wrong today, so the day someone fixes it, the test changes on purpose.

moto: fake AWS inside the test process

uv add --dev "moto[s3,sqs]"

moto is a library that fakes AWS APIs. It intercepts botocore‘s HTTP layer, so every boto3 call goes to moto’s in-memory state instead of the network. The extras [s3,sqs] pull in what those two services need. It’s a dev dependency, so I rebuilt the test image with docker compose build test.

Why not use LocalStack in the tests too? LocalStack is great for running the app by hand. But for automated tests, it means shared state between tests, it’s slower, and it needs a running container. moto gives each test an empty AWS, with no network, in seconds.

The aws fixture

Every test that needs AWS asks for this fixture. Step by step:

settings.AWS_S3_ENDPOINT_URL = None
settings.AWS_S3_PUBLIC_ENDPOINT_URL = None
settings.AWS_SQS_ENDPOINT_URL = None

moto only intercepts the real AWS host names, like s3.amazonaws.com. A client created with endpoint_url=http://localstack:4566 goes around moto, and the requests really go out to the network. So the fixture clears the endpoints.

settings.STORAGES = {**settings.STORAGES}

This line looks like it does nothing: it assigns an unchanged copy. But django-storages reads the endpoint when the storage object is created, and Django keeps that object. Assigning STORAGES fires Django’s setting_changed signal, and Django then throws away the cached storage, so the next use builds a new one with the test settings.

I didn’t assume this worked. I proved it: with LocalStack stopped, the upload tests still passed, so the uploads really went to moto.

get_queue.cache_clear()
_public_s3_client.cache_clear()

Both of my clients are cached with @lru_cache, and they may have been built earlier with LocalStack endpoints. .cache_clear() makes the next call build a new client, inside the fake.

with mock_aws():
    s3 = boto3.client("s3", region_name=settings.AWS_REGION)
    s3.create_bucket(Bucket=settings.AWS_STORAGE_BUCKET_NAME)
    sqs = boto3.client("sqs", region_name=settings.AWS_REGION)
    dlq_url = sqs.create_queue(QueueName=f"{settings.SQS_QUEUE_NAME}-dlq")["QueueUrl"]
    dlq_arn = sqs.get_queue_attributes(QueueUrl=dlq_url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
    queue_url = sqs.create_queue(
        QueueName=settings.SQS_QUEUE_NAME,
        Attributes={
            "VisibilityTimeout": "30",
            "RedrivePolicy": json.dumps({"deadLetterTargetArn": dlq_arn, "maxReceiveCount": "3"}),
        },
    )["QueueUrl"]

    yield SimpleNamespace(s3=s3, sqs=sqs, queue_url=queue_url, bucket=settings.AWS_STORAGE_BUCKET_NAME)

    get_queue.cache_clear()
    _public_s3_client.cache_clear()
  • mock_aws() turns moto on until the with block ends. The state starts empty for each test.
  • The bucket uses the real name from settings, so the app code needs no changes.
  • The queue is the same as my LocalStack init script from Part 10: a DLQ first (its ARN is needed for the redrive policy), then the main queue with a 30-second visibility timeout and a redrive after 3 receives. If the script and the fixture ever disagree, the tests test a different queue from the one the app uses. They must stay in sync.
  • SQS attribute values are strings, and RedrivePolicy is a JSON string inside them.
  • It’s a yield fixture. Everything before yield is setup, the yielded value is what the test receives, and everything after is teardown, run when the test finishes.
  • The caches are cleared again at the end. This is important. The cached clients were built with no endpoint. Outside moto, a client with no endpoint talks to real AWS. Clearing them makes sure no later test can ever use one.

Testing S3

A real upload, checked three ways

def test_apply_stores_the_resume_in_s3(api_as, aws, setup):
    _, sam, job = setup
    response = apply(api_as, sam, job, resume("My CV.PDF"))
    body = response.json()

    assert response.status_code == 201
    assert body["has_resume"] is True
    assert "resume" not in body
    key = Application.objects.get(id=body["id"]).resume.name
    assert re.fullmatch(rf"resume/job-{job.id}/[0-9a-f]{{32}}\.pdf", key)
    assert bucket_keys(aws) == [key]
    assert aws.s3.get_object(Bucket=aws.bucket, Key=key)["Body"].read() == PDF

This is the first successful apply in the whole suite (201).

  • rf"..." is a raw f-string: {job.id} is filled in, and {{32}} becomes a literal {32} for the regex.
  • My CV.PDF: the upload-path rules from Part 18 applied for real. The key is random, the name is gone, and the extension is lowercased.
  • Three checks on storage: the key in the database matches the key in the bucket; it’s the only object; and the stored bytes are exactly the uploaded bytes. The last one proves the validator’s rewind from Part 18 for real. Without seek(0), the first 5 bytes would be missing.

Private, signed, and nothing stored on rejection

  • The file is private. The test reads the object’s ACL grants: exactly one, the bucket owner with FULL_CONTROL. Public access would show as a grant to a group URI (like …/global/AllUsers), so “no grant has a URI” means “not public”.
  • A rejected upload stores nothing. A cv.txt gets 400, and the bucket is empty. Validation runs before saving, and the upload happens during saving.
  • The presigned link is signed for the right file. The test splits the URL with urlparse and parse_qs and checks four things: the path is the stored key, the link expires after 300 seconds, it’s signed with SigV4 (X-Amz-Signature), and the download filename is resume-<id>.pdf, not the candidate’s original name. Actually downloading through the URL isn’t tested here. I proved that by hand against LocalStack in Part 7.

A test that documents a gap: the orphaned file

This simulates a race: two requests both pass the serializer’s duplicate check, then both try to insert, and the second hits the database’s unique constraint. Switching validate off reproduces “the check passed, but the insert failed”.

def test_duplicate_race_returns_400_but_leaves_the_file_behind(api_as, aws, setup, make_application, monkeypatch):
    _, sam, job = setup
    make_application(job, sam)
    monkeypatch.setattr(ApplicationSerializer, "validate", lambda self, attrs: attrs)

    response = apply(api_as, sam, job, resume())

    assert response.status_code == 400
    assert response.json() == {"job": "You have already applied for this job."}
    assert Application.objects.count() == 1
    assert len(bucket_keys(aws)) == 1
  • The view’s fallback works: IntegrityError becomes a ValidationError and a 400, not a 500.
  • The error is a string, not a list. It’s raised in the view, outside the serializer, so DRF doesn’t wrap it in a list. Compare the serializer’s duplicate error in Part 19, which was a list.
  • The real gap: after the request, there’s 1 object in the bucket and no new application. Django’s FileField uploads the file before the INSERT. When the INSERT fails, nothing deletes the file. It’s an orphaned resume.

This test records the current behaviour. The day I fix it (by deleting the file in the except, or with an S3 cleanup job), the last line changes, on purpose.

Testing SQS

The message lifecycle, at every stage

def test_send_receive_delete(aws):
    queue = get_queue()

    message_id = queue.send({"hello": "world"})
    assert counts(aws) == (1, 0)    # waiting

    [message] = queue.receive(wait_seconds=0)
    assert (message.id, message.body) == (message_id, {"hello": "world"})
    assert counts(aws) == (0, 1)    # in flight: hidden for the visibility timeout

    queue.delete(message)
    assert counts(aws) == (0, 0)    # gone
  • counts() returns two numbers: messages waiting, and messages in flight. They’re the same two numbers as my qdepth helper from Phase 3.
  • Why counts, not a second receive()? After a receive, the message is hidden for 30 seconds. A second receive() returns nothing whether or not the message was deleted. Only the in-flight count tells “deleted” apart from “hidden”.
  • [message] = ... is unpacking that also checks there’s exactly one message. It fails otherwise.
  • wait_seconds=0: no long polling in tests.
  • Three smaller tests: a received message is frozen (@dataclass(frozen=True)), so a handler can’t accidentally change its receipt handle; get_queue() always returns the same object (one set of clients per process); and ping() on a missing queue raises ClientError. That last one is what lets the readiness check report a deleted or misnamed queue.

Published only after commit

def test_status_change_is_published_only_after_commit(api_as, aws, application, django_capture_on_commit_callbacks):
    with django_capture_on_commit_callbacks() as callbacks:
        response = set_status(api_as, application, "interview")

        assert response.status_code == 200
        assert queued(aws) == []    # not committed yet: nothing sent

        for callback in callbacks:
            callback()    # "commit"

        [body] = queued(aws)
        assert body == {
            "type": STATUS_CHANGED,
            "event_id": body["event_id"],
            "application_id": application.id,
            "old_status": "submitted",
            "new_status": "interview",
        }
        assert uuid.UUID(body["event_id"]).version == 4
  • The problem: inside a test, the transaction never commits, so on_commit callbacks would never run.
  • django_capture_on_commit_callbacks (from pytest-django) collects them instead. Without execute=True, they’re not run, so the test can look at the queue before and after “commit”.
  • Before: empty. That’s the guarantee. The worker never sees a message while the status change could still roll back.
  • Running the callbacks by hand simulates the commit.
  • The exact body: every field the worker reads. event_id is random, so it’s compared with itself, and the next line checks it’s a real UUID version 4.

Two more tests: a no-op status change publishes nothing, and a rolled-back change publishes nothing. In the rollback test, something fails after publish_status_changed was called, inside the same transaction.atomic(). When an atomic() block rolls back, Django throws away the on_commit callbacks registered inside it. So nothing is captured, nothing is sent, and the status is unchanged. Without on_commit, the candidate would get an email about a change that never happened.

A second documented gap: the lost event

def test_queue_outage_is_logged_not_raised(api_as, application, django_capture_on_commit_callbacks, monkeypatch, caplog):
    class BrokenQueue:
        def send(self, body):
            raise EndpointConnectionError(endpoint_url="http://sqs.invalid")

    monkeypatch.setattr("jobs.events.get_queue", lambda: BrokenQueue())

    with django_capture_on_commit_callbacks(execute=True):
        response = set_status(api_as, application, "interview")

    application.refresh_from_db()
    assert response.status_code == 200
    assert application.status == "interview"
    assert "Could not queue application.status_changed" in caplog.text
  • No moto here. A tiny hand-written fake that only implements send, and fails. That’s all this test needs.
  • caplog is pytest’s log capture, with every record available as text.
  • What it documents: the request succeeds and the status is saved, but the email event is lost. It’s only logged. This is the transactional outbox gap I wrote about in Phase 3. Fixing it means storing the event in the database together with the status change, and sending it later.

Testing the worker

Handling one event

The worker tests import the module with from worker import run_worker. Importing it runs django.setup() at its top, because it’s written to run as a script. Under pytest Django is already set up, and running setup again is harmless.

  • One email, and the event claimed. The happy path: handle_status_changed(body) sends exactly one email, and the event is recorded in ProcessedEvent in the same transaction.
  • A duplicate sends nothing. SQS delivers at least once, so the same message can arrive twice, for example after a crash between sending and deleting. Calling the handler twice with the same body gives one email, one ProcessedEvent, and the log line already processed; skipping duplicate. That’s the idempotency from Part 12, now automated.
  • A deleted application is skipped but marked done. The claim happens first, and the return inside the atomic() block commits it. So a redelivery is skipped quickly, instead of failing 3 times and landing in the DLQ.
  • An unknown message type raises, so the message is retried and finally sent to the DLQ instead of being silently dropped.

The other half of idempotency: releasing the claim

def test_failed_send_releases_the_claim(application, mailoutbox, monkeypatch):
    body = event(application.id)
    real_send = run_worker.send_status_changed_email

    def mail_server_down(*args):
        raise SMTPException("mail server down")

    monkeypatch.setattr(run_worker, "send_status_changed_email", mail_server_down)
    with pytest.raises(SMTPException):
        run_worker.handle_status_changed(body)
    assert not claimed(body)    # rolled back with the failed send

    monkeypatch.setattr(run_worker, "send_status_changed_email", real_send)
    run_worker.handle_status_changed(body)    # the redelivery
    assert len(mailoutbox) == 1
    assert claimed(body)

If the email fails, the claim must be released. Otherwise the retry would see “already processed” and skip it, and the candidate would never get the email. It works because the claim and the send are in the same transaction.atomic(), so the exception rolls back both.

mailoutbox comes from pytest-django: during tests Django uses an in-memory email backend, and every “sent” email lands in this list. SMTPException is the real exception that Django’s SMTP backend raises.

The main loop, with a fake queue

To test the loop’s error handling, I needed a queue that fails exactly when I want. moto can’t make receive fail on demand, so I wrote a small fake:

class FakeQueue:
    url = "fake://job-board-status"

    def __init__(self, *batches):
        self.batches = list(batches)
        self.deleted = []

    def receive(self):
        batch = self.batches.pop(0)
        if not self.batches:
            run_worker.running = False    # last batch: let main() finish this pass and stop
        if isinstance(batch, Exception):
            raise batch
        return batch

    def delete(self, message):
        if message.id == "undeletable":
            raise EndpointConnectionError(endpoint_url="http://sqs.invalid")
        self.deleted.append(message.id)
  • Each argument is what one receive() returns: a list of messages, or an exception to raise.
  • Stopping the loop: main() runs while running:. The fake sets running = False when it hands out its last batch, so main() finishes that batch and returns. That’s exactly what SIGTERM does in production.
  • “undeletable” is a message ID that makes delete() fail.
  • The loop fixture makes main() safe to run inside a test. It sets running = True through monkeypatch (so it’s restored afterwards), turns signal.signal into a no-op (otherwise it would change how the pytest process reacts to Ctrl+C), turns close_old_connections into a no-op (more on that in the problems section), and replaces time.sleep with sleeps.append, so there’s no real 5-second wait but the test can still check the pause would have happened.

The loop tests:

  • Handled messages are deleted: [“a”, “b”] handled, [“a”, “b”] deleted.
  • A failed message is kept, and the batch continues: with a, bad, c, only a and c are deleted. bad comes back after the visibility timeout, and after 3 failures it goes to the DLQ. One poison message doesn’t block the batch.
  • A receive error waits 5 seconds and retries: sleeps == [5], then the next pass processes a. That’s the resilience fix from Part 13, now automated: a queue outage is logged, followed by a pause (so no log flood), and the loop continues.
  • A delete error doesn’t stop the worker. The message will come back, and idempotency makes that harmless.
  • Start, stop and signals: both SIGTERM and SIGINT are connected to stop, and stop(15, None) sets running to False and logs Received signal 15, finishing current batch.

Testing the emails

The email checks from Part 11 were manual. Now they’re automated:

  • The content: the subject is Your application for Junior DevOps Engineer: Interview, the email goes to the candidate, and the body says Hi sam, and New status: Interview. The status shows its label (Interview), not the stored value (interview).
  • The first name is used when there is one: Hi Sam, instead of the username.
  • Plain text is not HTML-escaped. A job title of R&D <Lead> stays R&D <Lead>, not R&amp;D. The template uses {% autoescape off %}, because it’s plain text, not HTML.
def test_newlines_cannot_reach_the_subject(application, mailoutbox):
    send_status_changed_email(application(title="Line one\nBcc: attacker@evil.test"), "offered")
    [email] = mailoutbox

    assert "\n" not in email.subject
    assert email.subject == "Your application for Line one Bcc: attacker@evil.test: Offered"
    assert email.bcc == []

This is header injection. A newline in a subject could start a new email header like Bcc:, and a company controls its own job titles. My _one_line() helper collapses all whitespace to single spaces. The text stays, but it’s inside the subject, and bcc is empty.

An unknown status raises ValueError. A bad event fails loudly, so the worker leaves it for retry and finally the DLQ.

Testing the cache

The cache tests run on the in-memory cache from Part 19. The view code is the same as with Redis.

  • The second request is a HIT: ("MISS", "HIT") from the X-Cache header, and the cached body equals the fresh one.
  • Different queries are cached separately, otherwise a search would return the unfiltered page.
  • Parameter order doesn’t matter: ?job_type=full_time&location=van and ?location=van&job_type=full_time hit the same cache entry, because the cache key sorts the query parameters.

Invalidated only after commit

def test_new_job_invalidates_the_list_only_after_commit(api_as, board, make_job, django_capture_on_commit_callbacks):
    client = api_as(board.user)
    client.get("/api/jobs/")

    with django_capture_on_commit_callbacks() as callbacks:
        make_job(board, title="SRE Intern")

    stale = client.get("/api/jobs/")
    assert stale["X-Cache"] == "HIT"
    assert "SRE Intern" not in titles(stale)

    for callback in callbacks:
        callback()

    fresh = client.get("/api/jobs/")
    assert fresh["X-Cache"] == "MISS"
    assert "SRE Intern" in titles(fresh)

The rule from Part 9: invalidate when the change is committed, not before. Before commit, other readers can’t see the new job yet. If the cache version were bumped then, a reader could fill the cache with the old list under the new version, and that stale page would live for 60 seconds.

So: before the callbacks, still a HIT without the new job. After them, a MISS with it. A similar test proves that renaming a company also invalidates the list, because the list shows company names.

When Redis fails

Three tests, one for each Redis failure, and none of them may break the request:

  • Invalidation fails: it’s logged as Could not invalidate the job list cache. The page can be stale for up to 60 seconds, which is acceptable.
  • Reading fails: the list is served straight from Postgres, with no X-Cache header at all, because the cache wasn’t involved.
  • Writing fails: a MISS is served, just not stored.

I proved all three by hand in Part 13 by pausing Redis. Now they run in a second.

Testing readiness

/readyz from Part 13 has a list of checks, health.CHECKS. Each entry is (name, function, critical). The tests replace that list with a helper:

def use_checks(monkeypatch, **replacements):
    checks = [(name, replacements.get(name, check), critical) for name, check, critical in health.CHECKS]
    monkeypatch.setattr(health, "CHECKS", checks)

Why replace the list and not the functions? The list holds references to the functions, built at import time. Patching health._check_database wouldn’t change what’s already in the list.

  • All real checks pass: a real SELECT 1, the in-memory cache and the moto queue. The test checks the whole response contract: each check’s status, which ones are critical (only the database), a timing number for each, and no-cache, so nobody caches the answer.

Database down means 503, and the message is hidden:

use_checks(monkeypatch, database=failing(OperationalError("connect failed: password=hunter2")), queue=ok)
    ...
    assert response.status_code == 503
    assert body["checks"]["database"]["error"] == "OperationalError"
    assert "hunter2" not in response.content.decode()

I put a fake secret in the exception message on purpose. Real connection errors can include host names, user names and sometimes passwords, and /readyz has no login. The response shows only the exception’s class name. The full error goes to the log.

  • Redis or SQS down: still 200, still ready. The API works around both outages, as the tests above prove. Taking the instance out of traffic would make things worse.
  • Liveness ignores every dependency: with the database, cache and queue all failing, /healthz is still 200 and /readyz is 503. That’s the core rule from Part 13. Restarting containers because the database is down only adds a restart storm to the outage.

Problems I hit

1. Tests that passed and errored at the same time

One run said 178 passed, 4 errors. Every error was at teardown, after the test itself had passed:

AttributeError: 'RedisCache' object has no attribute 'get'

(and the same for set and incr). Here is what happened:

  • My Redis-failure tests first patched django.core.cache.cache. But cache is a proxy: it forwards to whatever cache is configured right now.
  • So the patch landed on the in-memory LocMemCache.
  • At teardown, the settings fixture switched CACHES back to Redis before monkeypatch undid my patch.
  • So the undo went through the proxy to a different object, a RedisCache that had never been patched, and failed.

The fix: patch caches["default"], the real backend object, not the proxy. Then the undo goes back to the same object. I changed 4 places with sed.

The lesson: a test can pass and error, and pytest counts it in both numbers.

2. close_old_connections can’t run inside a test

The worker’s loop calls close_old_connections() on every pass, which is correct in production. But inside a test, autocommit is always off. That function sees “autocommit isn’t what the settings say” and closes the connection, which breaks the test’s transaction. So the loop fixture replaces it with a no-op. The real behaviour was already proven by hand in Phase 3.

3. Two patching traps

  • Patch the backend, not the proxy (problem 1).
  • health.CHECKS holds function references, so patching a function changes nothing. Replace the list itself.

4. My expected counts were wrong

I expected 182 tests in total and 38 in this group. The files really had 178 and 34. Reading the real output, not trusting the number I expected, is what caught it.

A side effect

To prove the AWS tests don’t use LocalStack, I stopped it. All 13 AWS tests passed. But stopping LocalStack also emptied my dev bucket again, so I had to re-upload the dev resumes afterwards.

Breaking the code on purpose

  • The SQS publish: I replaced transaction.on_commit(send) with send(), so the message is sent immediately. Exactly 2 tests failed: “published only after commit” and “rolled back publishes nothing”. Those are the two tests about the transaction, and nothing else.
  • The cache: I replaced transaction.on_commit(bump) with bump(). Exactly 1 test failed: test_new_job_invalidates_the_list_only_after_commit. That’s the cache bug from Part 9 in reverse: back then, the bump never ran at all.

After each, git checkout restored the code and everything passed.

The numbers for this part: 13 AWS tests and 34 worker, email, cache and readiness tests. queue.py and events.py are at 100%, and the total coverage is now 99%, with 178 tests.

Why I chose this way

  • moto, not LocalStack, for tests. An empty state per test, no network, and fast. LocalStack stays for running the app by hand.
  • Endpoints cleared in the fixture, because moto only intercepts real AWS host names.
  • The fixture’s queue matches the LocalStack script, so the tests test the queue the app really uses.
  • A hand-written fake for the loop, because the error paths need exact control.
  • Recorded sleeps, so the 5-second pause is checked without waiting 5 seconds.
  • Gaps documented in tests, so they stay visible and fixing them becomes a deliberate change.

Looking ahead: Azure

Phase 6 adds Azure. S3 becomes Blob Storage (also through django-storages), and its local fake is Azurite. SQS becomes Service Bus, through a second implementation of my Queue interface. There’s no in-process fake for Service Bus as convenient as moto, so the tests there will probably use interface-level fakes, plus Azurite where possible.

What I learned

  • Fake at the edges, keep the database real.
  • In-process fakes (moto) beat a local emulator (LocalStack) for tests: per-test state and no network.
  • Cached clients must be rebuilt inside the fake, and cleared again afterwards, or a fake-era client leaks into later tests.
  • django_capture_on_commit_callbacks makes on_commit testable.
  • Count messages, don’t just receive them, to tell “deleted” from “hidden”.
  • At-least-once delivery plus idempotency plus releasing the claim gives exactly-once emails.
  • Test a loop through its stop condition.
  • Patch objects, not proxies, and replace lists that hold references.
  • Never leak exception messages from an endpoint without a login.
  • A test can document a gap instead of hiding it.

What comes next

At 99%, a few lines were still missed. The last article of the testing series is about deciding what to do with each of them, and about the 80% coverage gate that makes the target enforced. It also collects every deliberate bug from Phase 5 in one place: how I know my tests actually work.

$ ls -t root-access/ | head -3

Building a Job Board API, Part 19: Who Can Do What, and What the API Really Does job-board-api · 19/20 18 min Building a Job Board API, Part 18: Tests in a Container job-board-api · 18/20 18 min Building a Job Board API, Part 17: A Production Image with gunicorn job-board-api · 17/20 16 min

Leave a comment

Your email address will not be published. Required fields are marked *