{"id":308,"date":"2026-11-17T12:00:00","date_gmt":"2026-11-17T12:00:00","guid":{"rendered":"https:\/\/rezachegini.com\/?p=308"},"modified":"2026-11-17T19:38:07","modified_gmt":"2026-11-17T19:38:07","slug":"building-a-job-board-api-part-20-faking-aws-the-worker-and-redis","status":"publish","type":"post","link":"https:\/\/rezachegini.com\/?p=308","title":{"rendered":"Building a Job Board API, Part 20: Faking AWS, the Worker and Redis"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I&#8217;m sharing one of my projects here from the beginning, step by step. It&#8217;s a Job Board API built with Django, Django REST Framework, PostgreSQL and Docker Compose, with S3 and SQS (through LocalStack), Redis, and a background worker.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I&#8217;m a junior engineer looking for my first role in DevOps, SRE or Cloud. That&#8217;s why I build projects like this and write about them in public. It shows how I think, and writing it down helps me really understand it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In Parts 18 and 19, I tested the models, the permissions and the API. But none of those tests talked to S3 or SQS, and the worker wasn&#8217;t tested at all. This part covers the code that depends on things outside the app: AWS, the worker&#8217;s loop, email, the Redis cache and the readiness checks.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The question for each one: what stays real, and what gets faked?<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"515\" src=\"http:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-diagram-1024x515.png\" alt=\"Only the database stays real in the tests\" class=\"wp-image-312\" srcset=\"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-diagram-1024x515.png 1024w, https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-diagram-300x151.png 300w, https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-diagram-768x386.png 768w, https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-diagram.png 1347w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">Only the database stays real, because the app uses PostgreSQL-only features. Everything else is replaced inside the test process, so the tests are fast, have no network, and start from an empty state every time.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This part also has something new: tests that document known gaps. Some tests don&#8217;t check that the code is right. They check exactly how it&#8217;s wrong today, so the day someone fixes it, the test changes on purpose.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">moto: fake AWS inside the test process<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">uv add --dev \"moto[s3,sqs]\"<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">moto is a library that fakes AWS APIs. It intercepts <code>botocore<\/code>&#8216;s HTTP layer, so every <code>boto3<\/code> call goes to moto&#8217;s in-memory state instead of the network. The extras <code>[s3,sqs]<\/code> pull in what those two services need. It&#8217;s a dev dependency, so I rebuilt the test image with <code>docker compose build test<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Why not use LocalStack in the tests too? LocalStack is great for running the app by hand. But for automated tests, it means shared state between tests, it&#8217;s slower, and it needs a running container. moto gives each test an empty AWS, with no network, in seconds.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The aws fixture<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Every test that needs AWS asks for this fixture. Step by step:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">settings.AWS_S3_ENDPOINT_URL = None\nsettings.AWS_S3_PUBLIC_ENDPOINT_URL = None\nsettings.AWS_SQS_ENDPOINT_URL = None<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">moto only intercepts the real AWS host names, like <code>s3.amazonaws.com<\/code>. A client created with <code>endpoint_url=http:\/\/localstack:4566<\/code> goes around moto, and the requests really go out to the network. So the fixture clears the endpoints.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">settings.STORAGES = {**settings.STORAGES}<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This line looks like it does nothing: it assigns an unchanged copy. But django-storages reads the endpoint when the storage object is created, and Django keeps that object. Assigning <code>STORAGES<\/code> fires Django&#8217;s <code>setting_changed<\/code> signal, and Django then throws away the cached storage, so the next use builds a new one with the test settings.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I didn&#8217;t assume this worked. I proved it: with LocalStack stopped, the upload tests still passed, so the uploads really went to moto.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">get_queue.cache_clear()\n_public_s3_client.cache_clear()<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Both of my clients are cached with <code>@lru_cache<\/code>, and they may have been built earlier with LocalStack endpoints. <code>.cache_clear()<\/code> makes the next call build a new client, inside the fake.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">with mock_aws():\n    s3 = boto3.client(\"s3\", region_name=settings.AWS_REGION)\n    s3.create_bucket(Bucket=settings.AWS_STORAGE_BUCKET_NAME)\n    sqs = boto3.client(\"sqs\", region_name=settings.AWS_REGION)\n    dlq_url = sqs.create_queue(QueueName=f\"{settings.SQS_QUEUE_NAME}-dlq\")[\"QueueUrl\"]\n    dlq_arn = sqs.get_queue_attributes(QueueUrl=dlq_url, AttributeNames=[\"QueueArn\"])[\"Attributes\"][\"QueueArn\"]\n    queue_url = sqs.create_queue(\n        QueueName=settings.SQS_QUEUE_NAME,\n        Attributes={\n            \"VisibilityTimeout\": \"30\",\n            \"RedrivePolicy\": json.dumps({\"deadLetterTargetArn\": dlq_arn, \"maxReceiveCount\": \"3\"}),\n        },\n    )[\"QueueUrl\"]\n\n    yield SimpleNamespace(s3=s3, sqs=sqs, queue_url=queue_url, bucket=settings.AWS_STORAGE_BUCKET_NAME)\n\n    get_queue.cache_clear()\n    _public_s3_client.cache_clear()<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>mock_aws()<\/code> turns moto on until the <code>with<\/code> block ends. The state starts empty for each test.<\/li>\n\n\n\n<li>The bucket uses the real name from settings, so the app code needs no changes.<\/li>\n\n\n\n<li>The queue is the same as my LocalStack init script from Part 10: a DLQ first (its ARN is needed for the redrive policy), then the main queue with a 30-second visibility timeout and a redrive after 3 receives. If the script and the fixture ever disagree, the tests test a different queue from the one the app uses. They must stay in sync.<\/li>\n\n\n\n<li>SQS attribute values are strings, and <code>RedrivePolicy<\/code> is a JSON string inside them.<\/li>\n\n\n\n<li>It&#8217;s a yield fixture. Everything before <code>yield<\/code> is setup, the yielded value is what the test receives, and everything after is teardown, run when the test finishes.<\/li>\n\n\n\n<li>The caches are cleared again at the end. This is important. The cached clients were built with no endpoint. Outside moto, a client with no endpoint talks to real AWS. Clearing them makes sure no later test can ever use one.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Testing S3<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">A real upload, checked three ways<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_apply_stores_the_resume_in_s3(api_as, aws, setup):\n    _, sam, job = setup\n    response = apply(api_as, sam, job, resume(\"My CV.PDF\"))\n    body = response.json()\n\n    assert response.status_code == 201\n    assert body[\"has_resume\"] is True\n    assert \"resume\" not in body\n    key = Application.objects.get(id=body[\"id\"]).resume.name\n    assert re.fullmatch(rf\"resume\/job-{job.id}\/[0-9a-f]{{32}}\\.pdf\", key)\n    assert bucket_keys(aws) == [key]\n    assert aws.s3.get_object(Bucket=aws.bucket, Key=key)[\"Body\"].read() == PDF<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is the first successful apply in the whole suite (201).<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>rf\"...\"<\/code> is a raw f-string: <code>{job.id}<\/code> is filled in, and <code>{{32}}<\/code> becomes a literal <code>{32}<\/code> for the regex.<\/li>\n\n\n\n<li>My CV.PDF: the upload-path rules from Part 18 applied for real. The key is random, the name is gone, and the extension is lowercased.<\/li>\n\n\n\n<li>Three checks on storage: the key in the database matches the key in the bucket; it&#8217;s the only object; and the stored bytes are exactly the uploaded bytes. The last one proves the validator&#8217;s rewind from Part 18 for real. Without <code>seek(0)<\/code>, the first 5 bytes would be missing.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Private, signed, and nothing stored on rejection<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The file is private. The test reads the object&#8217;s ACL grants: exactly one, the bucket owner with FULL_CONTROL. Public access would show as a grant to a group URI (like &#8230;\/global\/AllUsers), so &#8220;no grant has a URI&#8221; means &#8220;not public&#8221;.<\/li>\n\n\n\n<li>A rejected upload stores nothing. A cv.txt gets 400, and the bucket is empty. Validation runs before saving, and the upload happens during saving.<\/li>\n\n\n\n<li>The presigned link is signed for the right file. The test splits the URL with <code>urlparse<\/code> and <code>parse_qs<\/code> and checks four things: the path is the stored key, the link expires after 300 seconds, it&#8217;s signed with SigV4 (<code>X-Amz-Signature<\/code>), and the download filename is <code>resume-&lt;id&gt;.pdf<\/code>, not the candidate&#8217;s original name. Actually downloading through the URL isn&#8217;t tested here. I proved that by hand against LocalStack in Part 7.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">A test that documents a gap: the orphaned file<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This simulates a race: two requests both pass the serializer&#8217;s duplicate check, then both try to insert, and the second hits the database&#8217;s unique constraint. Switching <code>validate<\/code> off reproduces &#8220;the check passed, but the insert failed&#8221;.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_duplicate_race_returns_400_but_leaves_the_file_behind(api_as, aws, setup, make_application, monkeypatch):\n    _, sam, job = setup\n    make_application(job, sam)\n    monkeypatch.setattr(ApplicationSerializer, \"validate\", lambda self, attrs: attrs)\n\n    response = apply(api_as, sam, job, resume())\n\n    assert response.status_code == 400\n    assert response.json() == {\"job\": \"You have already applied for this job.\"}\n    assert Application.objects.count() == 1\n    assert len(bucket_keys(aws)) == 1<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The view&#8217;s fallback works: <code>IntegrityError<\/code> becomes a <code>ValidationError<\/code> and a 400, not a 500.<\/li>\n\n\n\n<li>The error is a string, not a list. It&#8217;s raised in the view, outside the serializer, so DRF doesn&#8217;t wrap it in a list. Compare the serializer&#8217;s duplicate error in Part 19, which was a list.<\/li>\n\n\n\n<li>The real gap: after the request, there&#8217;s 1 object in the bucket and no new application. Django&#8217;s <code>FileField<\/code> uploads the file before the INSERT. When the INSERT fails, nothing deletes the file. It&#8217;s an orphaned resume.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">This test records the current behaviour. The day I fix it (by deleting the file in the <code>except<\/code>, or with an S3 cleanup job), the last line changes, on purpose.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Testing SQS<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">The message lifecycle, at every stage<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_send_receive_delete(aws):\n    queue = get_queue()\n\n    message_id = queue.send({\"hello\": \"world\"})\n    assert counts(aws) == (1, 0)    # waiting\n\n    [message] = queue.receive(wait_seconds=0)\n    assert (message.id, message.body) == (message_id, {\"hello\": \"world\"})\n    assert counts(aws) == (0, 1)    # in flight: hidden for the visibility timeout\n\n    queue.delete(message)\n    assert counts(aws) == (0, 0)    # gone<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>counts()<\/code> returns two numbers: messages waiting, and messages in flight. They&#8217;re the same two numbers as my <code>qdepth<\/code> helper from Phase 3.<\/li>\n\n\n\n<li>Why counts, not a second <code>receive()<\/code>? After a receive, the message is hidden for 30 seconds. A second <code>receive()<\/code> returns nothing whether or not the message was deleted. Only the in-flight count tells &#8220;deleted&#8221; apart from &#8220;hidden&#8221;.<\/li>\n\n\n\n<li><code>[message] = ...<\/code> is unpacking that also checks there&#8217;s exactly one message. It fails otherwise.<\/li>\n\n\n\n<li><code>wait_seconds=0<\/code>: no long polling in tests.<\/li>\n\n\n\n<li>Three smaller tests: a received message is frozen (<code>@dataclass(frozen=True)<\/code>), so a handler can&#8217;t accidentally change its receipt handle; <code>get_queue()<\/code> always returns the same object (one set of clients per process); and <code>ping()<\/code> on a missing queue raises <code>ClientError<\/code>. That last one is what lets the readiness check report a deleted or misnamed queue.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Published only after commit<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_status_change_is_published_only_after_commit(api_as, aws, application, django_capture_on_commit_callbacks):\n    with django_capture_on_commit_callbacks() as callbacks:\n        response = set_status(api_as, application, \"interview\")\n\n        assert response.status_code == 200\n        assert queued(aws) == []    # not committed yet: nothing sent\n\n        for callback in callbacks:\n            callback()    # \"commit\"\n\n        [body] = queued(aws)\n        assert body == {\n            \"type\": STATUS_CHANGED,\n            \"event_id\": body[\"event_id\"],\n            \"application_id\": application.id,\n            \"old_status\": \"submitted\",\n            \"new_status\": \"interview\",\n        }\n        assert uuid.UUID(body[\"event_id\"]).version == 4<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The problem: inside a test, the transaction never commits, so <code>on_commit<\/code> callbacks would never run.<\/li>\n\n\n\n<li><code>django_capture_on_commit_callbacks<\/code> (from pytest-django) collects them instead. Without <code>execute=True<\/code>, they&#8217;re not run, so the test can look at the queue before and after &#8220;commit&#8221;.<\/li>\n\n\n\n<li>Before: empty. That&#8217;s the guarantee. The worker never sees a message while the status change could still roll back.<\/li>\n\n\n\n<li>Running the callbacks by hand simulates the commit.<\/li>\n\n\n\n<li>The exact body: every field the worker reads. <code>event_id<\/code> is random, so it&#8217;s compared with itself, and the next line checks it&#8217;s a real UUID version 4.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Two more tests: a no-op status change publishes nothing, and a rolled-back change publishes nothing. In the rollback test, something fails after <code>publish_status_changed<\/code> was called, inside the same <code>transaction.atomic()<\/code>. When an <code>atomic()<\/code> block rolls back, Django throws away the <code>on_commit<\/code> callbacks registered inside it. So nothing is captured, nothing is sent, and the status is unchanged. Without <code>on_commit<\/code>, the candidate would get an email about a change that never happened.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">A second documented gap: the lost event<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_queue_outage_is_logged_not_raised(api_as, application, django_capture_on_commit_callbacks, monkeypatch, caplog):\n    class BrokenQueue:\n        def send(self, body):\n            raise EndpointConnectionError(endpoint_url=\"http:\/\/sqs.invalid\")\n\n    monkeypatch.setattr(\"jobs.events.get_queue\", lambda: BrokenQueue())\n\n    with django_capture_on_commit_callbacks(execute=True):\n        response = set_status(api_as, application, \"interview\")\n\n    application.refresh_from_db()\n    assert response.status_code == 200\n    assert application.status == \"interview\"\n    assert \"Could not queue application.status_changed\" in caplog.text<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>No moto here. A tiny hand-written fake that only implements <code>send<\/code>, and fails. That&#8217;s all this test needs.<\/li>\n\n\n\n<li><code>caplog<\/code> is pytest&#8217;s log capture, with every record available as text.<\/li>\n\n\n\n<li>What it documents: the request succeeds and the status is saved, but the email event is lost. It&#8217;s only logged. This is the transactional outbox gap I wrote about in Phase 3. Fixing it means storing the event in the database together with the status change, and sending it later.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Testing the worker<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">Handling one event<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The worker tests import the module with <code>from worker import run_worker<\/code>. Importing it runs <code>django.setup()<\/code> at its top, because it&#8217;s written to run as a script. Under pytest Django is already set up, and running setup again is harmless.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>One email, and the event claimed. The happy path: <code>handle_status_changed(body)<\/code> sends exactly one email, and the event is recorded in <code>ProcessedEvent<\/code> in the same transaction.<\/li>\n\n\n\n<li>A duplicate sends nothing. SQS delivers at least once, so the same message can arrive twice, for example after a crash between sending and deleting. Calling the handler twice with the same body gives one email, one <code>ProcessedEvent<\/code>, and the log line <code>already processed; skipping duplicate<\/code>. That&#8217;s the idempotency from Part 12, now automated.<\/li>\n\n\n\n<li>A deleted application is skipped but marked done. The claim happens first, and the return inside the <code>atomic()<\/code> block commits it. So a redelivery is skipped quickly, instead of failing 3 times and landing in the DLQ.<\/li>\n\n\n\n<li>An unknown message type raises, so the message is retried and finally sent to the DLQ instead of being silently dropped.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">The other half of idempotency: releasing the claim<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_failed_send_releases_the_claim(application, mailoutbox, monkeypatch):\n    body = event(application.id)\n    real_send = run_worker.send_status_changed_email\n\n    def mail_server_down(*args):\n        raise SMTPException(\"mail server down\")\n\n    monkeypatch.setattr(run_worker, \"send_status_changed_email\", mail_server_down)\n    with pytest.raises(SMTPException):\n        run_worker.handle_status_changed(body)\n    assert not claimed(body)    # rolled back with the failed send\n\n    monkeypatch.setattr(run_worker, \"send_status_changed_email\", real_send)\n    run_worker.handle_status_changed(body)    # the redelivery\n    assert len(mailoutbox) == 1\n    assert claimed(body)<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If the email fails, the claim must be released. Otherwise the retry would see &#8220;already processed&#8221; and skip it, and the candidate would never get the email. It works because the claim and the send are in the same <code>transaction.atomic()<\/code>, so the exception rolls back both.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><code>mailoutbox<\/code> comes from pytest-django: during tests Django uses an in-memory email backend, and every &#8220;sent&#8221; email lands in this list. <code>SMTPException<\/code> is the real exception that Django&#8217;s SMTP backend raises.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The main loop, with a fake queue<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To test the loop&#8217;s error handling, I needed a queue that fails exactly when I want. moto can&#8217;t make receive fail on demand, so I wrote a small fake:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">class FakeQueue:\n    url = \"fake:\/\/job-board-status\"\n\n    def __init__(self, *batches):\n        self.batches = list(batches)\n        self.deleted = []\n\n    def receive(self):\n        batch = self.batches.pop(0)\n        if not self.batches:\n            run_worker.running = False    # last batch: let main() finish this pass and stop\n        if isinstance(batch, Exception):\n            raise batch\n        return batch\n\n    def delete(self, message):\n        if message.id == \"undeletable\":\n            raise EndpointConnectionError(endpoint_url=\"http:\/\/sqs.invalid\")\n        self.deleted.append(message.id)<\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Each argument is what one <code>receive()<\/code> returns: a list of messages, or an exception to raise.<\/li>\n\n\n\n<li>Stopping the loop: <code>main()<\/code> runs while <code>running:<\/code>. The fake sets <code>running = False<\/code> when it hands out its last batch, so <code>main()<\/code> finishes that batch and returns. That&#8217;s exactly what SIGTERM does in production.<\/li>\n\n\n\n<li>&#8220;undeletable&#8221; is a message ID that makes <code>delete()<\/code> fail.<\/li>\n\n\n\n<li>The loop fixture makes <code>main()<\/code> safe to run inside a test. It sets <code>running = True<\/code> through <code>monkeypatch<\/code> (so it&#8217;s restored afterwards), turns <code>signal.signal<\/code> into a no-op (otherwise it would change how the pytest process reacts to Ctrl+C), turns <code>close_old_connections<\/code> into a no-op (more on that in the problems section), and replaces <code>time.sleep<\/code> with <code>sleeps.append<\/code>, so there&#8217;s no real 5-second wait but the test can still check the pause would have happened.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The loop tests:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Handled messages are deleted: [&#8220;a&#8221;, &#8220;b&#8221;] handled, [&#8220;a&#8221;, &#8220;b&#8221;] deleted.<\/li>\n\n\n\n<li>A failed message is kept, and the batch continues: with a, bad, c, only a and c are deleted. bad comes back after the visibility timeout, and after 3 failures it goes to the DLQ. One poison message doesn&#8217;t block the batch.<\/li>\n\n\n\n<li>A receive error waits 5 seconds and retries: <code>sleeps == [5]<\/code>, then the next pass processes a. That&#8217;s the resilience fix from Part 13, now automated: a queue outage is logged, followed by a pause (so no log flood), and the loop continues.<\/li>\n\n\n\n<li>A delete error doesn&#8217;t stop the worker. The message will come back, and idempotency makes that harmless.<\/li>\n\n\n\n<li>Start, stop and signals: both SIGTERM and SIGINT are connected to <code>stop<\/code>, and <code>stop(15, None)<\/code> sets <code>running<\/code> to False and logs <code>Received signal 15, finishing current batch<\/code>.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Testing the emails<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The email checks from Part 11 were manual. Now they&#8217;re automated:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The content: the subject is <code>Your application for Junior DevOps Engineer: Interview<\/code>, the email goes to the candidate, and the body says Hi sam, and <code>New status: Interview<\/code>. The status shows its label (Interview), not the stored value (interview).<\/li>\n\n\n\n<li>The first name is used when there is one: Hi Sam, instead of the username.<\/li>\n\n\n\n<li>Plain text is not HTML-escaped. A job title of R&amp;D &lt;Lead&gt; stays R&amp;D &lt;Lead&gt;, not R&amp;amp;D. The template uses <code>{% autoescape off %}<\/code>, because it&#8217;s plain text, not HTML.<\/li>\n<\/ul>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_newlines_cannot_reach_the_subject(application, mailoutbox):\n    send_status_changed_email(application(title=\"Line one\\nBcc: attacker@evil.test\"), \"offered\")\n    [email] = mailoutbox\n\n    assert \"\\n\" not in email.subject\n    assert email.subject == \"Your application for Line one Bcc: attacker@evil.test: Offered\"\n    assert email.bcc == []<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This is header injection. A newline in a subject could start a new email header like <code>Bcc:<\/code>, and a company controls its own job titles. My <code>_one_line()<\/code> helper collapses all whitespace to single spaces. The text stays, but it&#8217;s inside the subject, and <code>bcc<\/code> is empty.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An unknown status raises <code>ValueError<\/code>. A bad event fails loudly, so the worker leaves it for retry and finally the DLQ.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Testing the cache<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The cache tests run on the in-memory cache from Part 19. The view code is the same as with Redis.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The second request is a HIT: (<code>\"MISS\"<\/code>, <code>\"HIT\"<\/code>) from the <code>X-Cache<\/code> header, and the cached body equals the fresh one.<\/li>\n\n\n\n<li>Different queries are cached separately, otherwise a search would return the unfiltered page.<\/li>\n\n\n\n<li>Parameter order doesn&#8217;t matter: <code>?job_type=full_time&amp;location=van<\/code> and <code>?location=van&amp;job_type=full_time<\/code> hit the same cache entry, because the cache key sorts the query parameters.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Invalidated only after commit<\/h3>\n\n\n\n<pre class=\"wp-block-preformatted\">def test_new_job_invalidates_the_list_only_after_commit(api_as, board, make_job, django_capture_on_commit_callbacks):\n    client = api_as(board.user)\n    client.get(\"\/api\/jobs\/\")\n\n    with django_capture_on_commit_callbacks() as callbacks:\n        make_job(board, title=\"SRE Intern\")\n\n    stale = client.get(\"\/api\/jobs\/\")\n    assert stale[\"X-Cache\"] == \"HIT\"\n    assert \"SRE Intern\" not in titles(stale)\n\n    for callback in callbacks:\n        callback()\n\n    fresh = client.get(\"\/api\/jobs\/\")\n    assert fresh[\"X-Cache\"] == \"MISS\"\n    assert \"SRE Intern\" in titles(fresh)<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The rule from Part 9: invalidate when the change is committed, not before. Before commit, other readers can&#8217;t see the new job yet. If the cache version were bumped then, a reader could fill the cache with the old list under the new version, and that stale page would live for 60 seconds.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">So: before the callbacks, still a HIT without the new job. After them, a MISS with it. A similar test proves that renaming a company also invalidates the list, because the list shows company names.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">When Redis fails<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Three tests, one for each Redis failure, and none of them may break the request:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Invalidation fails: it&#8217;s logged as <code>Could not invalidate the job list cache<\/code>. The page can be stale for up to 60 seconds, which is acceptable.<\/li>\n\n\n\n<li>Reading fails: the list is served straight from Postgres, with no <code>X-Cache<\/code> header at all, because the cache wasn&#8217;t involved.<\/li>\n\n\n\n<li>Writing fails: a MISS is served, just not stored.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">I proved all three by hand in Part 13 by pausing Redis. Now they run in a second.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Testing readiness<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><code>\/readyz<\/code> from Part 13 has a list of checks, <code>health.CHECKS<\/code>. Each entry is (<code>name<\/code>, <code>function<\/code>, <code>critical<\/code>). The tests replace that list with a helper:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">def use_checks(monkeypatch, **replacements):\n    checks = [(name, replacements.get(name, check), critical) for name, check, critical in health.CHECKS]\n    monkeypatch.setattr(health, \"CHECKS\", checks)<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Why replace the list and not the functions? The list holds references to the functions, built at import time. Patching <code>health._check_database<\/code> wouldn&#8217;t change what&#8217;s already in the list.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>All real checks pass: a real <code>SELECT 1<\/code>, the in-memory cache and the moto queue. The test checks the whole response contract: each check&#8217;s status, which ones are critical (only the database), a timing number for each, and no-cache, so nobody caches the answer.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Database down means 503, and the message is hidden:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">use_checks(monkeypatch, database=failing(OperationalError(\"connect failed: password=hunter2\")), queue=ok)\n    ...\n    assert response.status_code == 503\n    assert body[\"checks\"][\"database\"][\"error\"] == \"OperationalError\"\n    assert \"hunter2\" not in response.content.decode()<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">I put a fake secret in the exception message on purpose. Real connection errors can include host names, user names and sometimes passwords, and <code>\/readyz<\/code> has no login. The response shows only the exception&#8217;s class name. The full error goes to the log.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Redis or SQS down: still 200, still ready. The API works around both outages, as the tests above prove. Taking the instance out of traffic would make things worse.<\/li>\n\n\n\n<li>Liveness ignores every dependency: with the database, cache and queue all failing, <code>\/healthz<\/code> is still 200 and <code>\/readyz<\/code> is 503. That&#8217;s the core rule from Part 13. Restarting containers because the database is down only adds a restart storm to the outage.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Problems I hit<\/h3>\n\n\n\n<h3 class=\"wp-block-heading\">1. Tests that passed and errored at the same time<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">One run said 178 passed, 4 errors. Every error was at teardown, after the test itself had passed:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">AttributeError: 'RedisCache' object has no attribute 'get'<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">(and the same for <code>set<\/code> and <code>incr<\/code>). Here is what happened:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>My Redis-failure tests first patched <code>django.core.cache.cache<\/code>. But <code>cache<\/code> is a proxy: it forwards to whatever cache is configured right now.<\/li>\n\n\n\n<li>So the patch landed on the in-memory <code>LocMemCache<\/code>.<\/li>\n\n\n\n<li>At teardown, the settings fixture switched <code>CACHES<\/code> back to Redis before <code>monkeypatch<\/code> undid my patch.<\/li>\n\n\n\n<li>So the undo went through the proxy to a different object, a <code>RedisCache<\/code> that had never been patched, and failed.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">The fix: patch <code>caches[\"default\"]<\/code>, the real backend object, not the proxy. Then the undo goes back to the same object. I changed 4 places with <code>sed<\/code>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The lesson: a test can pass and error, and pytest counts it in both numbers.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. close_old_connections can&#8217;t run inside a test<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The worker&#8217;s loop calls <code>close_old_connections()<\/code> on every pass, which is correct in production. But inside a test, autocommit is always off. That function sees &#8220;autocommit isn&#8217;t what the settings say&#8221; and closes the connection, which breaks the test&#8217;s transaction. So the loop fixture replaces it with a no-op. The real behaviour was already proven by hand in Phase 3.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Two patching traps<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Patch the backend, not the proxy (problem 1).<\/li>\n\n\n\n<li><code>health.CHECKS<\/code> holds function references, so patching a function changes nothing. Replace the list itself.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">4. My expected counts were wrong<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">I expected 182 tests in total and 38 in this group. The files really had 178 and 34. Reading the real output, not trusting the number I expected, is what caught it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">A side effect<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">To prove the AWS tests don&#8217;t use LocalStack, I stopped it. All 13 AWS tests passed. But stopping LocalStack also emptied my dev bucket again, so I had to re-upload the dev resumes afterwards.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Breaking the code on purpose<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The SQS publish: I replaced <code>transaction.on_commit(send)<\/code> with <code>send()<\/code>, so the message is sent immediately. Exactly 2 tests failed: &#8220;published only after commit&#8221; and &#8220;rolled back publishes nothing&#8221;. Those are the two tests about the transaction, and nothing else.<\/li>\n\n\n\n<li>The cache: I replaced <code>transaction.on_commit(bump)<\/code> with <code>bump()<\/code>. Exactly 1 test failed: <code>test_new_job_invalidates_the_list_only_after_commit<\/code>. That&#8217;s the cache bug from Part 9 in reverse: back then, the bump never ran at all.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">After each, <code>git checkout<\/code> restored the code and everything passed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The numbers for this part: 13 AWS tests and 34 worker, email, cache and readiness tests. <code>queue.py<\/code> and <code>events.py<\/code> are at 100%, and the total coverage is now 99%, with 178 tests.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Why I chose this way<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>moto, not LocalStack, for tests. An empty state per test, no network, and fast. LocalStack stays for running the app by hand.<\/li>\n\n\n\n<li>Endpoints cleared in the fixture, because moto only intercepts real AWS host names.<\/li>\n\n\n\n<li>The fixture&#8217;s queue matches the LocalStack script, so the tests test the queue the app really uses.<\/li>\n\n\n\n<li>A hand-written fake for the loop, because the error paths need exact control.<\/li>\n\n\n\n<li>Recorded sleeps, so the 5-second pause is checked without waiting 5 seconds.<\/li>\n\n\n\n<li>Gaps documented in tests, so they stay visible and fixing them becomes a deliberate change.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Looking ahead: Azure<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Phase 6 adds Azure. S3 becomes Blob Storage (also through django-storages), and its local fake is Azurite. SQS becomes Service Bus, through a second implementation of my Queue interface. There&#8217;s no in-process fake for Service Bus as convenient as moto, so the tests there will probably use interface-level fakes, plus Azurite where possible.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What I learned<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Fake at the edges, keep the database real.<\/li>\n\n\n\n<li>In-process fakes (moto) beat a local emulator (LocalStack) for tests: per-test state and no network.<\/li>\n\n\n\n<li>Cached clients must be rebuilt inside the fake, and cleared again afterwards, or a fake-era client leaks into later tests.<\/li>\n\n\n\n<li><code>django_capture_on_commit_callbacks<\/code> makes <code>on_commit<\/code> testable.<\/li>\n\n\n\n<li>Count messages, don&#8217;t just receive them, to tell &#8220;deleted&#8221; from &#8220;hidden&#8221;.<\/li>\n\n\n\n<li>At-least-once delivery plus idempotency plus releasing the claim gives exactly-once emails.<\/li>\n\n\n\n<li>Test a loop through its stop condition.<\/li>\n\n\n\n<li>Patch objects, not proxies, and replace lists that hold references.<\/li>\n\n\n\n<li>Never leak exception messages from an endpoint without a login.<\/li>\n\n\n\n<li>A test can document a gap instead of hiding it.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">What comes next<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">At 99%, a few lines were still missed. The last article of the testing series is about deciding what to do with each of them, and about the 80% coverage gate that makes the target enforced. It also collects every deliberate bug from Phase 5 in one place: how I know my tests actually work.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;m sharing one of my projects here from the beginning, step by step. It&#8217;s a Job Board API built with Django, Django&hellip;<\/p>\n","protected":false},"author":2,"featured_media":310,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"20","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/building-job-board-api-part-20-faking-aws-worker-redis-reza-chegini-wlrpc\/","footnotes":""},"categories":[85,10,116],"tags":[60,214,8,111,191,118,215,212,25,198,112,200,213],"series":[117],"class_list":["post-308","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-devops","category-django","tag-aws","tag-coverage","tag-devops","tag-django","tag-drf","tag-job-board","tag-moto","tag-pytest","tag-python","tag-redis","tag-rest-api","tag-sqs","tag-testing","series-job-board-api"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-20-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/308","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=308"}],"version-history":[{"count":2,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/308\/revisions"}],"predecessor-version":[{"id":313,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/308\/revisions\/313"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/310"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=308"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}