$ cat job-board-api/part-20.md
Building a Job Board API, Part 20: Faking AWS, the Worker and Redis

I’m sharing one of my projects here from the beginning, step by step. It’s a Job Board API built with Django, Django REST Framework, PostgreSQL and Docker Compose, with S3 and SQS (through LocalStack), Redis, and a background worker.
I’m a junior engineer looking for my first role in DevOps, SRE or Cloud. That’s why I build projects like this and write about them in public. It shows how I think, and writing it down helps me really understand it.
In Parts 18 and 19, I tested the models, the permissions and the API. But none of those tests talked to S3 or SQS, and the worker wasn’t tested at all. This part covers the code that depends on things outside the app: AWS, the worker’s loop, email, the Redis cache and the readiness checks.
The question for each one: what stays real, and what gets faked?

Only the database stays real, because the app uses PostgreSQL-only features. Everything else is replaced inside the test process, so the tests are fast, have no network, and start from an empty state every time.
This part also has something new: tests that document known gaps. Some tests don’t check that the code is right. They check exactly how it’s wrong today, so the day someone fixes it, the test changes on purpose.
moto: fake AWS inside the test process
uv add --dev "moto[s3,sqs]"
moto is a library that fakes AWS APIs. It intercepts botocore‘s HTTP layer, so every boto3 call goes to moto’s in-memory state instead of the network. The extras [s3,sqs] pull in what those two services need. It’s a dev dependency, so I rebuilt the test image with docker compose build test.
Why not use LocalStack in the tests too? LocalStack is great for running the app by hand. But for automated tests, it means shared state between tests, it’s slower, and it needs a running container. moto gives each test an empty AWS, with no network, in seconds.
The aws fixture
Every test that needs AWS asks for this fixture. Step by step:
settings.AWS_S3_ENDPOINT_URL = None settings.AWS_S3_PUBLIC_ENDPOINT_URL = None settings.AWS_SQS_ENDPOINT_URL = None
moto only intercepts the real AWS host names, like s3.amazonaws.com. A client created with endpoint_url=http://localstack:4566 goes around moto, and the requests really go out to the network. So the fixture clears the endpoints.
settings.STORAGES = {**settings.STORAGES}
This line looks like it does nothing: it assigns an unchanged copy. But django-storages reads the endpoint when the storage object is created, and Django keeps that object. Assigning STORAGES fires Django’s setting_changed signal, and Django then throws away the cached storage, so the next use builds a new one with the test settings.
I didn’t assume this worked. I proved it: with LocalStack stopped, the upload tests still passed, so the uploads really went to moto.
get_queue.cache_clear() _public_s3_client.cache_clear()
Both of my clients are cached with @lru_cache, and they may have been built earlier with LocalStack endpoints. .cache_clear() makes the next call build a new client, inside the fake.
with mock_aws():
s3 = boto3.client("s3", region_name=settings.AWS_REGION)
s3.create_bucket(Bucket=settings.AWS_STORAGE_BUCKET_NAME)
sqs = boto3.client("sqs", region_name=settings.AWS_REGION)
dlq_url = sqs.create_queue(QueueName=f"{settings.SQS_QUEUE_NAME}-dlq")["QueueUrl"]
dlq_arn = sqs.get_queue_attributes(QueueUrl=dlq_url, AttributeNames=["QueueArn"])["Attributes"]["QueueArn"]
queue_url = sqs.create_queue(
QueueName=settings.SQS_QUEUE_NAME,
Attributes={
"VisibilityTimeout": "30",
"RedrivePolicy": json.dumps({"deadLetterTargetArn": dlq_arn, "maxReceiveCount": "3"}),
},
)["QueueUrl"]
yield SimpleNamespace(s3=s3, sqs=sqs, queue_url=queue_url, bucket=settings.AWS_STORAGE_BUCKET_NAME)
get_queue.cache_clear()
_public_s3_client.cache_clear()
mock_aws()turns moto on until thewithblock ends. The state starts empty for each test.- The bucket uses the real name from settings, so the app code needs no changes.
- The queue is the same as my LocalStack init script from Part 10: a DLQ first (its ARN is needed for the redrive policy), then the main queue with a 30-second visibility timeout and a redrive after 3 receives. If the script and the fixture ever disagree, the tests test a different queue from the one the app uses. They must stay in sync.
- SQS attribute values are strings, and
RedrivePolicyis a JSON string inside them. - It’s a yield fixture. Everything before
yieldis setup, the yielded value is what the test receives, and everything after is teardown, run when the test finishes. - The caches are cleared again at the end. This is important. The cached clients were built with no endpoint. Outside moto, a client with no endpoint talks to real AWS. Clearing them makes sure no later test can ever use one.
Testing S3
A real upload, checked three ways
def test_apply_stores_the_resume_in_s3(api_as, aws, setup):
_, sam, job = setup
response = apply(api_as, sam, job, resume("My CV.PDF"))
body = response.json()
assert response.status_code == 201
assert body["has_resume"] is True
assert "resume" not in body
key = Application.objects.get(id=body["id"]).resume.name
assert re.fullmatch(rf"resume/job-{job.id}/[0-9a-f]{{32}}\.pdf", key)
assert bucket_keys(aws) == [key]
assert aws.s3.get_object(Bucket=aws.bucket, Key=key)["Body"].read() == PDF
This is the first successful apply in the whole suite (201).
rf"..."is a raw f-string:{job.id}is filled in, and{{32}}becomes a literal{32}for the regex.- My CV.PDF: the upload-path rules from Part 18 applied for real. The key is random, the name is gone, and the extension is lowercased.
- Three checks on storage: the key in the database matches the key in the bucket; it’s the only object; and the stored bytes are exactly the uploaded bytes. The last one proves the validator’s rewind from Part 18 for real. Without
seek(0), the first 5 bytes would be missing.
Private, signed, and nothing stored on rejection
- The file is private. The test reads the object’s ACL grants: exactly one, the bucket owner with FULL_CONTROL. Public access would show as a grant to a group URI (like …/global/AllUsers), so “no grant has a URI” means “not public”.
- A rejected upload stores nothing. A cv.txt gets 400, and the bucket is empty. Validation runs before saving, and the upload happens during saving.
- The presigned link is signed for the right file. The test splits the URL with
urlparseandparse_qsand checks four things: the path is the stored key, the link expires after 300 seconds, it’s signed with SigV4 (X-Amz-Signature), and the download filename isresume-<id>.pdf, not the candidate’s original name. Actually downloading through the URL isn’t tested here. I proved that by hand against LocalStack in Part 7.
A test that documents a gap: the orphaned file
This simulates a race: two requests both pass the serializer’s duplicate check, then both try to insert, and the second hits the database’s unique constraint. Switching validate off reproduces “the check passed, but the insert failed”.
def test_duplicate_race_returns_400_but_leaves_the_file_behind(api_as, aws, setup, make_application, monkeypatch):
_, sam, job = setup
make_application(job, sam)
monkeypatch.setattr(ApplicationSerializer, "validate", lambda self, attrs: attrs)
response = apply(api_as, sam, job, resume())
assert response.status_code == 400
assert response.json() == {"job": "You have already applied for this job."}
assert Application.objects.count() == 1
assert len(bucket_keys(aws)) == 1
- The view’s fallback works:
IntegrityErrorbecomes aValidationErrorand a 400, not a 500. - The error is a string, not a list. It’s raised in the view, outside the serializer, so DRF doesn’t wrap it in a list. Compare the serializer’s duplicate error in Part 19, which was a list.
- The real gap: after the request, there’s 1 object in the bucket and no new application. Django’s
FileFielduploads the file before the INSERT. When the INSERT fails, nothing deletes the file. It’s an orphaned resume.
This test records the current behaviour. The day I fix it (by deleting the file in the except, or with an S3 cleanup job), the last line changes, on purpose.
Testing SQS
The message lifecycle, at every stage
def test_send_receive_delete(aws):
queue = get_queue()
message_id = queue.send({"hello": "world"})
assert counts(aws) == (1, 0) # waiting
[message] = queue.receive(wait_seconds=0)
assert (message.id, message.body) == (message_id, {"hello": "world"})
assert counts(aws) == (0, 1) # in flight: hidden for the visibility timeout
queue.delete(message)
assert counts(aws) == (0, 0) # gone
counts()returns two numbers: messages waiting, and messages in flight. They’re the same two numbers as myqdepthhelper from Phase 3.- Why counts, not a second
receive()? After a receive, the message is hidden for 30 seconds. A secondreceive()returns nothing whether or not the message was deleted. Only the in-flight count tells “deleted” apart from “hidden”. [message] = ...is unpacking that also checks there’s exactly one message. It fails otherwise.wait_seconds=0: no long polling in tests.- Three smaller tests: a received message is frozen (
@dataclass(frozen=True)), so a handler can’t accidentally change its receipt handle;get_queue()always returns the same object (one set of clients per process); andping()on a missing queue raisesClientError. That last one is what lets the readiness check report a deleted or misnamed queue.
Published only after commit
def test_status_change_is_published_only_after_commit(api_as, aws, application, django_capture_on_commit_callbacks):
with django_capture_on_commit_callbacks() as callbacks:
response = set_status(api_as, application, "interview")
assert response.status_code == 200
assert queued(aws) == [] # not committed yet: nothing sent
for callback in callbacks:
callback() # "commit"
[body] = queued(aws)
assert body == {
"type": STATUS_CHANGED,
"event_id": body["event_id"],
"application_id": application.id,
"old_status": "submitted",
"new_status": "interview",
}
assert uuid.UUID(body["event_id"]).version == 4
- The problem: inside a test, the transaction never commits, so
on_commitcallbacks would never run. django_capture_on_commit_callbacks(from pytest-django) collects them instead. Withoutexecute=True, they’re not run, so the test can look at the queue before and after “commit”.- Before: empty. That’s the guarantee. The worker never sees a message while the status change could still roll back.
- Running the callbacks by hand simulates the commit.
- The exact body: every field the worker reads.
event_idis random, so it’s compared with itself, and the next line checks it’s a real UUID version 4.
Two more tests: a no-op status change publishes nothing, and a rolled-back change publishes nothing. In the rollback test, something fails after publish_status_changed was called, inside the same transaction.atomic(). When an atomic() block rolls back, Django throws away the on_commit callbacks registered inside it. So nothing is captured, nothing is sent, and the status is unchanged. Without on_commit, the candidate would get an email about a change that never happened.
A second documented gap: the lost event
def test_queue_outage_is_logged_not_raised(api_as, application, django_capture_on_commit_callbacks, monkeypatch, caplog):
class BrokenQueue:
def send(self, body):
raise EndpointConnectionError(endpoint_url="http://sqs.invalid")
monkeypatch.setattr("jobs.events.get_queue", lambda: BrokenQueue())
with django_capture_on_commit_callbacks(execute=True):
response = set_status(api_as, application, "interview")
application.refresh_from_db()
assert response.status_code == 200
assert application.status == "interview"
assert "Could not queue application.status_changed" in caplog.text
- No moto here. A tiny hand-written fake that only implements
send, and fails. That’s all this test needs. caplogis pytest’s log capture, with every record available as text.- What it documents: the request succeeds and the status is saved, but the email event is lost. It’s only logged. This is the transactional outbox gap I wrote about in Phase 3. Fixing it means storing the event in the database together with the status change, and sending it later.
Testing the worker
Handling one event
The worker tests import the module with from worker import run_worker. Importing it runs django.setup() at its top, because it’s written to run as a script. Under pytest Django is already set up, and running setup again is harmless.
- One email, and the event claimed. The happy path:
handle_status_changed(body)sends exactly one email, and the event is recorded inProcessedEventin the same transaction. - A duplicate sends nothing. SQS delivers at least once, so the same message can arrive twice, for example after a crash between sending and deleting. Calling the handler twice with the same body gives one email, one
ProcessedEvent, and the log linealready processed; skipping duplicate. That’s the idempotency from Part 12, now automated. - A deleted application is skipped but marked done. The claim happens first, and the return inside the
atomic()block commits it. So a redelivery is skipped quickly, instead of failing 3 times and landing in the DLQ. - An unknown message type raises, so the message is retried and finally sent to the DLQ instead of being silently dropped.
The other half of idempotency: releasing the claim
def test_failed_send_releases_the_claim(application, mailoutbox, monkeypatch):
body = event(application.id)
real_send = run_worker.send_status_changed_email
def mail_server_down(*args):
raise SMTPException("mail server down")
monkeypatch.setattr(run_worker, "send_status_changed_email", mail_server_down)
with pytest.raises(SMTPException):
run_worker.handle_status_changed(body)
assert not claimed(body) # rolled back with the failed send
monkeypatch.setattr(run_worker, "send_status_changed_email", real_send)
run_worker.handle_status_changed(body) # the redelivery
assert len(mailoutbox) == 1
assert claimed(body)
If the email fails, the claim must be released. Otherwise the retry would see “already processed” and skip it, and the candidate would never get the email. It works because the claim and the send are in the same transaction.atomic(), so the exception rolls back both.
mailoutbox comes from pytest-django: during tests Django uses an in-memory email backend, and every “sent” email lands in this list. SMTPException is the real exception that Django’s SMTP backend raises.
The main loop, with a fake queue
To test the loop’s error handling, I needed a queue that fails exactly when I want. moto can’t make receive fail on demand, so I wrote a small fake:
class FakeQueue:
url = "fake://job-board-status"
def __init__(self, *batches):
self.batches = list(batches)
self.deleted = []
def receive(self):
batch = self.batches.pop(0)
if not self.batches:
run_worker.running = False # last batch: let main() finish this pass and stop
if isinstance(batch, Exception):
raise batch
return batch
def delete(self, message):
if message.id == "undeletable":
raise EndpointConnectionError(endpoint_url="http://sqs.invalid")
self.deleted.append(message.id)
- Each argument is what one
receive()returns: a list of messages, or an exception to raise. - Stopping the loop:
main()runs whilerunning:. The fake setsrunning = Falsewhen it hands out its last batch, somain()finishes that batch and returns. That’s exactly what SIGTERM does in production. - “undeletable” is a message ID that makes
delete()fail. - The loop fixture makes
main()safe to run inside a test. It setsrunning = Truethroughmonkeypatch(so it’s restored afterwards), turnssignal.signalinto a no-op (otherwise it would change how the pytest process reacts to Ctrl+C), turnsclose_old_connectionsinto a no-op (more on that in the problems section), and replacestime.sleepwithsleeps.append, so there’s no real 5-second wait but the test can still check the pause would have happened.
The loop tests:
- Handled messages are deleted: [“a”, “b”] handled, [“a”, “b”] deleted.
- A failed message is kept, and the batch continues: with a, bad, c, only a and c are deleted. bad comes back after the visibility timeout, and after 3 failures it goes to the DLQ. One poison message doesn’t block the batch.
- A receive error waits 5 seconds and retries:
sleeps == [5], then the next pass processes a. That’s the resilience fix from Part 13, now automated: a queue outage is logged, followed by a pause (so no log flood), and the loop continues. - A delete error doesn’t stop the worker. The message will come back, and idempotency makes that harmless.
- Start, stop and signals: both SIGTERM and SIGINT are connected to
stop, andstop(15, None)setsrunningto False and logsReceived signal 15, finishing current batch.
Testing the emails
The email checks from Part 11 were manual. Now they’re automated:
- The content: the subject is
Your application for Junior DevOps Engineer: Interview, the email goes to the candidate, and the body says Hi sam, andNew status: Interview. The status shows its label (Interview), not the stored value (interview). - The first name is used when there is one: Hi Sam, instead of the username.
- Plain text is not HTML-escaped. A job title of R&D <Lead> stays R&D <Lead>, not R&D. The template uses
{% autoescape off %}, because it’s plain text, not HTML.
def test_newlines_cannot_reach_the_subject(application, mailoutbox):
send_status_changed_email(application(title="Line one\nBcc: attacker@evil.test"), "offered")
[email] = mailoutbox
assert "\n" not in email.subject
assert email.subject == "Your application for Line one Bcc: attacker@evil.test: Offered"
assert email.bcc == []
This is header injection. A newline in a subject could start a new email header like Bcc:, and a company controls its own job titles. My _one_line() helper collapses all whitespace to single spaces. The text stays, but it’s inside the subject, and bcc is empty.
An unknown status raises ValueError. A bad event fails loudly, so the worker leaves it for retry and finally the DLQ.
Testing the cache
The cache tests run on the in-memory cache from Part 19. The view code is the same as with Redis.
- The second request is a HIT: (
"MISS","HIT") from theX-Cacheheader, and the cached body equals the fresh one. - Different queries are cached separately, otherwise a search would return the unfiltered page.
- Parameter order doesn’t matter:
?job_type=full_time&location=vanand?location=van&job_type=full_timehit the same cache entry, because the cache key sorts the query parameters.
Invalidated only after commit
def test_new_job_invalidates_the_list_only_after_commit(api_as, board, make_job, django_capture_on_commit_callbacks):
client = api_as(board.user)
client.get("/api/jobs/")
with django_capture_on_commit_callbacks() as callbacks:
make_job(board, title="SRE Intern")
stale = client.get("/api/jobs/")
assert stale["X-Cache"] == "HIT"
assert "SRE Intern" not in titles(stale)
for callback in callbacks:
callback()
fresh = client.get("/api/jobs/")
assert fresh["X-Cache"] == "MISS"
assert "SRE Intern" in titles(fresh)
The rule from Part 9: invalidate when the change is committed, not before. Before commit, other readers can’t see the new job yet. If the cache version were bumped then, a reader could fill the cache with the old list under the new version, and that stale page would live for 60 seconds.
So: before the callbacks, still a HIT without the new job. After them, a MISS with it. A similar test proves that renaming a company also invalidates the list, because the list shows company names.
When Redis fails
Three tests, one for each Redis failure, and none of them may break the request:
- Invalidation fails: it’s logged as
Could not invalidate the job list cache. The page can be stale for up to 60 seconds, which is acceptable. - Reading fails: the list is served straight from Postgres, with no
X-Cacheheader at all, because the cache wasn’t involved. - Writing fails: a MISS is served, just not stored.
I proved all three by hand in Part 13 by pausing Redis. Now they run in a second.
Testing readiness
/readyz from Part 13 has a list of checks, health.CHECKS. Each entry is (name, function, critical). The tests replace that list with a helper:
def use_checks(monkeypatch, **replacements):
checks = [(name, replacements.get(name, check), critical) for name, check, critical in health.CHECKS]
monkeypatch.setattr(health, "CHECKS", checks)
Why replace the list and not the functions? The list holds references to the functions, built at import time. Patching health._check_database wouldn’t change what’s already in the list.
- All real checks pass: a real
SELECT 1, the in-memory cache and the moto queue. The test checks the whole response contract: each check’s status, which ones are critical (only the database), a timing number for each, and no-cache, so nobody caches the answer.
Database down means 503, and the message is hidden:
use_checks(monkeypatch, database=failing(OperationalError("connect failed: password=hunter2")), queue=ok)
...
assert response.status_code == 503
assert body["checks"]["database"]["error"] == "OperationalError"
assert "hunter2" not in response.content.decode()
I put a fake secret in the exception message on purpose. Real connection errors can include host names, user names and sometimes passwords, and /readyz has no login. The response shows only the exception’s class name. The full error goes to the log.
- Redis or SQS down: still 200, still ready. The API works around both outages, as the tests above prove. Taking the instance out of traffic would make things worse.
- Liveness ignores every dependency: with the database, cache and queue all failing,
/healthzis still 200 and/readyzis 503. That’s the core rule from Part 13. Restarting containers because the database is down only adds a restart storm to the outage.
Problems I hit
1. Tests that passed and errored at the same time
One run said 178 passed, 4 errors. Every error was at teardown, after the test itself had passed:
AttributeError: 'RedisCache' object has no attribute 'get'
(and the same for set and incr). Here is what happened:
- My Redis-failure tests first patched
django.core.cache.cache. Butcacheis a proxy: it forwards to whatever cache is configured right now. - So the patch landed on the in-memory
LocMemCache. - At teardown, the settings fixture switched
CACHESback to Redis beforemonkeypatchundid my patch. - So the undo went through the proxy to a different object, a
RedisCachethat had never been patched, and failed.
The fix: patch caches["default"], the real backend object, not the proxy. Then the undo goes back to the same object. I changed 4 places with sed.
The lesson: a test can pass and error, and pytest counts it in both numbers.
2. close_old_connections can’t run inside a test
The worker’s loop calls close_old_connections() on every pass, which is correct in production. But inside a test, autocommit is always off. That function sees “autocommit isn’t what the settings say” and closes the connection, which breaks the test’s transaction. So the loop fixture replaces it with a no-op. The real behaviour was already proven by hand in Phase 3.
3. Two patching traps
- Patch the backend, not the proxy (problem 1).
health.CHECKSholds function references, so patching a function changes nothing. Replace the list itself.
4. My expected counts were wrong
I expected 182 tests in total and 38 in this group. The files really had 178 and 34. Reading the real output, not trusting the number I expected, is what caught it.
A side effect
To prove the AWS tests don’t use LocalStack, I stopped it. All 13 AWS tests passed. But stopping LocalStack also emptied my dev bucket again, so I had to re-upload the dev resumes afterwards.
Breaking the code on purpose
- The SQS publish: I replaced
transaction.on_commit(send)withsend(), so the message is sent immediately. Exactly 2 tests failed: “published only after commit” and “rolled back publishes nothing”. Those are the two tests about the transaction, and nothing else. - The cache: I replaced
transaction.on_commit(bump)withbump(). Exactly 1 test failed:test_new_job_invalidates_the_list_only_after_commit. That’s the cache bug from Part 9 in reverse: back then, the bump never ran at all.
After each, git checkout restored the code and everything passed.
The numbers for this part: 13 AWS tests and 34 worker, email, cache and readiness tests. queue.py and events.py are at 100%, and the total coverage is now 99%, with 178 tests.
Why I chose this way
- moto, not LocalStack, for tests. An empty state per test, no network, and fast. LocalStack stays for running the app by hand.
- Endpoints cleared in the fixture, because moto only intercepts real AWS host names.
- The fixture’s queue matches the LocalStack script, so the tests test the queue the app really uses.
- A hand-written fake for the loop, because the error paths need exact control.
- Recorded sleeps, so the 5-second pause is checked without waiting 5 seconds.
- Gaps documented in tests, so they stay visible and fixing them becomes a deliberate change.
Looking ahead: Azure
Phase 6 adds Azure. S3 becomes Blob Storage (also through django-storages), and its local fake is Azurite. SQS becomes Service Bus, through a second implementation of my Queue interface. There’s no in-process fake for Service Bus as convenient as moto, so the tests there will probably use interface-level fakes, plus Azurite where possible.
What I learned
- Fake at the edges, keep the database real.
- In-process fakes (moto) beat a local emulator (LocalStack) for tests: per-test state and no network.
- Cached clients must be rebuilt inside the fake, and cleared again afterwards, or a fake-era client leaks into later tests.
django_capture_on_commit_callbacksmakeson_committestable.- Count messages, don’t just receive them, to tell “deleted” from “hidden”.
- At-least-once delivery plus idempotency plus releasing the claim gives exactly-once emails.
- Test a loop through its stop condition.
- Patch objects, not proxies, and replace lists that hold references.
- Never leak exception messages from an endpoint without a login.
- A test can document a gap instead of hiding it.
What comes next
At 99%, a few lines were still missed. The last article of the testing series is about deciding what to do with each of them, and about the 80% coverage gate that makes the target enforced. It also collects every deliberate bug from Phase 5 in one place: how I know my tests actually work.