- \n
- SQS, not Redis. My plan said “Redis locally, SQS on AWS”. But that means two different queue backends and two code paths. Instead, I use SQS through LocalStack everywhere, the same fake AWS I set up for S3 in Part 7. Same code locally and on AWS. And, importantly, LocalStack gives me SQS’s real retry behaviour, which I need to test properly.<\/li>\n\n\n\n
- The folder is called cloud\/, not platform\/. My plan put cloud code in a platform\/ package. But Python has a built-in module called platform, and my folder would hide it. Any library doing import platform would get my code instead. So: cloud\/queue.py.<\/li>\n<\/ol>\n\n\n\n
How SQS delivers a message<\/h2>\n\n\n\n
One idea surprised me more than anything else in this step: receiving a message does not remove it.<\/p>\n\n\n\n
![\"How](\"http:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-10-sqs-diagram-1024x726.png\")
<\/figure>\n\n\n\nWhen the worker receives a message, SQS only hides it for 30 seconds. This is called the visibility timeout. If the worker finishes the job and deletes the message, it’s gone for good. If the worker crashes, or the job fails, there is no delete. After 30 seconds the message becomes visible again, and the worker gets another try.<\/p>\n\n\n\n
That’s how retries work, for free. But what about a message that can never succeed, for example one with a broken body? It would fail forever. That’s what the dead-letter queue (DLQ) is for. After 3 receives without a delete, SQS moves the message to a separate queue, where it is parked for a human to inspect.<\/p>\n\n\n\n
This model is called at-least-once delivery. Every message is delivered at least once, but sometimes more than once. So whatever the worker does must be safe to repeat. Handling that properly is a later step, but I designed for it from the start, as you’ll see.<\/p>\n\n\n\n
Step 1: Creating the queues<\/h2>\n\n\n\n
LocalStack runs every script in its init\/ready.d\/ folder when it starts. In Part 7, one script created the S3 bucket. Now a second one creates two queues:<\/p>\n\n\n\n
#!\/bin\/bash\nset -euo pipefail\n\ndlq_url=$(awslocal sqs create-queue --queue-name \"${SQS_QUEUE_NAME}-dlq\" --query QueueUrl --output text)\ndlq_arn=$(awslocal sqs get-queue-attributes --queue-url \"$dlq_url\" --attribute-names QueueArn --query Attributes.QueueArn --output text)\n\ncat > \/tmp\/queue-attributes.json <<EOF\n{\n \"VisibilityTimeout\": \"30\",\n \"RedrivePolicy\": \"{\\\"deadLetterTargetArn\\\":\\\"${dlq_arn}\\\",\\\"maxReceiveCount\\\":\\\"3\\\"}\"\n}\nEOF\n\nawslocal sqs create-queue --queue-name \"${SQS_QUEUE_NAME}\" --attributes file:\/\/\/tmp\/queue-attributes.json<\/code><\/pre>\n\n\n\n- \n
- The DLQ comes first. The main queue’s settings point at the DLQ by its ARN (Amazon Resource Name), so I need the ARN before I can create the main queue. –query picks it out of the get-queue-attributes response.<\/li>\n\n\n\n
- –query … –output text picks one field from the JSON response and prints it without quotes, so $(…) captures a clean value.<\/li>\n\n\n\n
- VisibilityTimeout: 30 must be longer than one message takes to process. If it were shorter, a slow worker would see its own message come back while it was still working on it.<\/li>\n\n\n\n
- RedrivePolicy says: after 3 receives without a delete, move the message to the DLQ.<\/li>\n<\/ul>\n\n\n\n
Why write the settings to a file? RedrivePolicy is a JSON document stored as a string inside another JSON document. Getting the quotes right on one command line is almost impossible. A file is much easier. The heredoc (<<EOF) is unquoted on purpose, so bash fills in the real ${dlq_arn}.<\/p>\n\n\n\n
In Compose, I added SQS to LocalStack (SERVICES=s3,sqs), and the healthcheck now checks the bucket and the queue:<\/p>\n\n\n\n
test: [\"CMD-SHELL\", \"awslocal s3api head-bucket --bucket $${AWS_STORAGE_BUCKET_NAME} && awslocal sqs get-queue-url --queue-name $${SQS_QUEUE_NAME}\"]<\/code><\/pre>\n\n\n\n&& runs the second check only if the first passes. The main queue is created after the DLQ, so if the main queue exists, both do. The API and the worker wait for this healthcheck, so neither starts before its queue is ready.<\/p>\n\n\n\n
One side effect I expected: changing LocalStack’s settings means recreating the container, and that empties S3 again (no persistence, as in Part 7). So I re-uploaded my test resume afterwards.<\/p>\n\n\n\n
Step 2: One interface for any cloud<\/h2>\n\n\n\n
In the Azure phase of this project, the queue will be Azure Service Bus, not SQS. I don’t want to rewrite the API and the worker when that happens. So they never talk to SQS directly. They talk to an interface.<\/p>\n\n\n\n
The message<\/h3>\n\n\n\n
@dataclass(frozen=True)\nclass QueueMessage:\n id: str\n body: dict\n receipt_handle: str<\/code><\/pre>\n\n\n\n- \n
- @dataclass writes the boring parts of a class for me: the constructor, a readable print-out, and comparison.<\/li>\n\n\n\n
- frozen=True makes it immutable. A received message can’t be changed by accident.<\/li>\n\n\n\n
- id is SQS’s message ID. It stays the same on every redelivery.<\/li>\n\n\n\n
- receipt_handle is a token for this one receive. SQS needs it to delete the message, and it changes on every redelivery. I tested both: same id: True, new receipt handle: True.<\/li>\n<\/ul>\n\n\n\n
The interface<\/h3>\n\n\n\n
class Queue(ABC):\n @abstractmethod\n def send(self, body: dict) -> str: ...\n\n @abstractmethod\n def receive(self, max_messages: int = 10, wait_seconds: int = 20) -> list[QueueMessage]: ...\n\n @abstractmethod\n def delete(self, message: QueueMessage) -> None: ...<\/code><\/pre>\n\n\n\nAn abstract base class (ABC) lists methods without implementing them. With @abstractmethod, a backend that forgets one of the three can’t even be created: Python raises a TypeError immediately, not halfway through processing a message. delete takes the whole message, so callers never need to know that SQS uses receipt handles. Service Bus does it differently.<\/p>\n\n\n\n
The SQS implementation<\/h3>\n\n\n\n
def receive(self, max_messages: int = 10, wait_seconds: int = 20) -> list[QueueMessage]:\n response = self.client.receive_message(\n QueueUrl=self.url,\n MaxNumberOfMessages=max_messages,\n WaitTimeSeconds=wait_seconds,\n )\n return [\n QueueMessage(id=m[\"MessageId\"], body=json.loads(m[\"Body\"]), receipt_handle=m[\"ReceiptHandle\"])\n for m in response.get(\"Messages\", [])\n ]<\/code><\/pre>\n\n\n\n- \n
- WaitTimeSeconds=20 is long polling. SQS holds the request open until a message arrives, or for up to 20 seconds. An idle worker makes about 3 calls a minute, instead of hammering SQS in a tight loop. That matters, because SQS bills per request.<\/li>\n\n\n\n
- MaxNumberOfMessages=10 takes up to 10 messages per call, SQS’s maximum, so there are fewer calls.<\/li>\n\n\n\n
- response.get(“Messages”, []): when nothing arrives, SQS leaves out the Messages key completely. Writing response[“Messages”] would crash with a KeyError on every empty poll.<\/li>\n\n\n\n
- Bodies are JSON, not pickle: any language can read them, and they’re safe to inspect by hand. That paid off later, when I read a bad message straight out of the DLQ.<\/li>\n<\/ul>\n\n\n\n
@lru_cache(maxsize=1)\ndef get_queue() -> Queue:\n return SQSQueue(settings.SQS_QUEUE_NAME)<\/code><\/pre>\n\n\n\nThe single entry point. It returns the interface type. In the Azure phase, this is the only function that changes.<\/p>\n\n\n\n
Step 3: Publishing the event<\/h2>\n\n\n\n
STATUS_CHANGED = \"application.status_changed\"\n\ndef publish_status_changed(application, old_status):\n body = {\n \"type\": STATUS_CHANGED,\n \"event_id\": str(uuid.uuid4()),\n \"application_id\": application.id,\n \"old_status\": old_status,\n \"new_status\": application.status,\n }\n\n def send():\n try:\n message_id = get_queue().send(body)\n logger.info(\"Queued %s for application %s (message %s)\", STATUS_CHANGED, application.id, message_id)\n except (BotoCoreError, ClientError):\n logger.exception(\"Could not queue %s for application %s\", STATUS_CHANGED, application.id)\n\n transaction.on_commit(send)<\/code><\/pre>\n\n\n\nIDs, not data<\/h3>\n\n\n\n
The message holds no email address and no name. Only the application’s ID and the two statuses. The worker loads the current data from the database itself. That keeps messages small, keeps personal data out of the queue, and means the worker always uses fresh data, not a copy that may be out of date.<\/p>\n\n\n\n
One name, defined once<\/h3>\n\n\n\n
STATUS_CHANGED is defined in one place, and both the API (which sends) and the worker (which receives) import it. If I mistype it, I get an ImportError at startup, not a quiet mismatch where the worker never recognises the message. After everything this series has taught me about silent name bugs, that was an easy choice.<\/p>\n\n\n\n
Two different IDs<\/h3>\n\n\n\n
event_id identifies the change. SQS’s own MessageId identifies the delivery. If the same change were ever sent twice, the two messages would have different message IDs, but the same event ID. That difference is what will let the worker recognise duplicates later.<\/p>\n\n\n\n
on_commit, again<\/h3>\n\n\n\n
The same pattern as the cache in Part 9, for two reasons:<\/p>\n\n\n\n
- \n
- If the message were sent before the commit, a fast worker could read the application and still see the old status.<\/li>\n\n\n\n
- If the change rolls back, no message should exist at all.<\/li>\n<\/ol>\n\n\n\n
And yes, send is a nested function. Without the last line, it would never run, and nothing would tell me. That was exactly my bug in Part 9. This time the line was there from the start.<\/p>\n\n\n\n
If SQS is down<\/h3>\n\n\n\n
The error is logged, not raised. By the time send runs, the status change is already saved. Returning a 500 would tell the company that it failed, when it didn’t, and they might try again and change the status twice.<\/p>\n\n\n\n
(BotoCoreError, ClientError) covers every boto3 failure: BotoCoreError for network and client-side problems, ClientError for errors AWS sends back.<\/p>\n\n\n\n
Only on a real change<\/h3>\n\n\n\n
In the view:<\/p>\n\n\n\n
old_status = application.status\nserializer.save()\nif application.status != old_status:\n publish_status_changed(application, old_status)<\/code><\/pre>\n\n\n\nold_status must be read before save(), because the serializer updates the same object in place. And if a company sets a status to the value it already has, no message is sent, so the candidate never gets a pointless duplicate email.<\/p>\n\n\n\n
Step 4: The worker<\/h2>\n\n\n\n
Booting Django by hand<\/h3>\n\n\n\n
The worker is a plain Python script, not a web request, and it doesn’t run through manage.py. So it has to start Django itself:<\/p>\n\n\n\n
os.environ.setdefault(\"DJANGO_SETTINGS_MODULE\", \"config.settings\")\ndjango.setup()\n\nfrom django.db import close_old_connections # noqa: E402\nfrom cloud.queue import get_queue # noqa: E402\nfrom jobs.events import STATUS_CHANGED # noqa: E402<\/code><\/pre>\n\n\n\nThe order matters. My project imports must come after django.setup(). Importing jobs.events first would load the models before Django is ready, and crash with AppRegistryNotReady. # noqa: E402 tells the linter that “import not at the top of the file” is intended here.<\/p>\n\n\n\n
Handling one message<\/h3>\n\n\n\n
def handle(message):\n body = message.body\n if body.get(\"type\") == STATUS_CHANGED:\n logger.info(\"Application %s: %s -> %s (event %s)\",\n body[\"application_id\"], body[\"old_status\"], body[\"new_status\"], body[\"event_id\"])\n return\n raise ValueError(f\"Unknown message type: {body.get('type')!r}\")<\/code><\/pre>\n\n\n\nAn unknown message type raises an error. It is not skipped. If I skipped it, the worker would delete it, and the evidence would be gone. By raising, the message isn’t deleted, it’s retried, and after 3 tries it lands in the DLQ, where I can read it. A message that can never succeed is called a poison message.<\/p>\n\n\n\n
The loop<\/h3>\n\n\n\n
while running:\n close_old_connections()\n for message in queue.receive():\n try:\n handle(message)\n except Exception:\n logger.exception(\"Failed to handle message %s; it will be retried\", message.id)\n continue\n queue.delete(message)<\/code><\/pre>\n\n\n\n- \n
- Delete only after success. On an error, continue skips the delete, and SQS brings the message back after 30 seconds.<\/li>\n\n\n\n
- except Exception is broad on purpose: one bad message must never kill the worker. It still lets Ctrl+C and shutdown signals through, because those don’t count as an Exception in Python.<\/li>\n\n\n\n
- close_old_connections(): Django normally closes database connections at the end of each request. A worker has no requests, so a connection could go stale (for example after a Postgres restart) and fail later. This line throws away broken connections, and the next query opens a fresh one.<\/li>\n<\/ul>\n\n\n\n
Stopping cleanly<\/h3>\n\n\n\n
When Docker stops a container, it sends a SIGTERM signal, waits, and then kills the process with SIGKILL. I don’t want the worker killed halfway through a message.<\/p>\n\n\n\n
running = True\n\ndef stop(signum, frame):\n global running\n logger.info(\"Received signal %s, finishing current batch\", signum)\n running = False\n\nsignal.signal(signal.SIGTERM, stop)\nsignal.signal(signal.SIGINT, stop)<\/code><\/pre>\n\n\n\nThe handler only sets a flag. The loop finishes its current work, sees running is false, and exits normally. global running is required, because without it, running = False would create a new local variable, and the loop would never stop.<\/p>\n\n\n\n
Same image, different command<\/h3>\n\n\n\n
worker:\n build: .\n command: [\"python\", \"-m\", \"worker.run_worker\"]\n env_file: .env\n environment:\n - POSTGRES_HOST=db\n stop_grace_period: 30s<\/code><\/pre>\n\n\n\n- \n
- One image, two process types. The API and the worker share the same code and dependencies. In Kubernetes, the worker will be a second Deployment from the same image, scaled separately.<\/li>\n\n\n\n
- python -m worker.run_worker, not python worker\/run_worker.py. The -m form puts \/app on Python’s import path, so cloud and jobs can be imported.<\/li>\n\n\n\n
- No ports. The worker only makes outgoing connections.<\/li>\n\n\n\n
- stop_grace_period: 30s. Docker’s default is 10 seconds, but a long poll can take 20. I measured a real stop: 11.9 seconds from the signal to “Worker stopped”. The default would have killed it.<\/li>\n<\/ul>\n\n\n\n
Step 5: Testing it<\/h2>\n\n\n\n
I used a small helper, qdepth, that prints how many messages are waiting in a queue and how many are in flight (received, but not deleted yet).<\/p>\n\n\n\n
The happy path<\/h3>\n\n\n\n
Acme moved application 3 from “reviewing” to “interview”. The worker logged:<\/p>\n\n\n\n
Application 3: reviewing -> interview (event 7618c3e2-...)<\/code><\/pre>\n\n\n\nAnd the queue showed 0 waiting, 0 in flight: the message was received, handled and deleted.<\/p>\n\n\n\n
Setting the same status again returned 200, and the worker saw nothing. No change, no message.<\/p>\n\n\n\n
The worker is down<\/h3>\n\n\n\n
This is the test that shows why queues exist. I stopped the worker, then changed a status:<\/p>\n\n\n\n
- \n
- The API still answered 200. It doesn’t need the worker at all.<\/li>\n\n\n\n
- The queue showed 1 message waiting.<\/li>\n\n\n\n
- I started the worker. Its first lines were Worker started, then Application 3: interview -> reviewing. The queue went back to 0.<\/li>\n<\/ul>\n\n\n\n
The message simply waited. That is decoupling: the API and the worker only know about the queue, never about each other, so either one can be down.<\/p>\n\n\n\n
A rollback<\/h3>\n\n\n\n
I changed a status inside a transaction and forced a rollback. The status in the database stayed “reviewing”, the worker saw nothing, and the queue stayed at 0. on_commit threw the message away together with the rollback.<\/p>\n\n\n\n
A poison message<\/h3>\n\n\n\n
I sent a message with {“type”: “bogus”} by hand and waited. The worker log showed three failures, each with ValueError: Unknown message type: ‘bogus’, and the worker stayed Up the whole time. Then I read the DLQ and found the message, with its original body, {“type”: “bogus”}. Nothing was lost, and one bad message didn’t take the worker down.<\/p>\n\n\n\n
A clean shutdown<\/h3>\n\n\n\n
time docker compose stop worker<\/code><\/pre>\n\n\n\nThe log showed Received signal 15, finishing current batch, then Worker stopped. Total: 11.9 seconds, inside the 30-second grace period. No SIGKILL.<\/p>\n\n\n\n
With these results, the goal of Phase 3 is demonstrated: a status change queues a job, and the worker processes it.<\/p>\n\n\n\n
Problems I hit<\/h2>\n\n\n\n
1. A log line that could never appear<\/h3>\n\n\n\n
I expected the API to log Queued application.status_changed for application 3. It never did. But the worker did process the message, so it was definitely sent.<\/p>\n\n\n\n
The reason: Django has no logging configuration in my project yet. Without one, Python only prints WARNING and above. My logger.info(…) line was simply dropped. The worker doesn’t have this problem, because it calls logging.basicConfig(level=logging.INFO) itself.<\/p>\n\n\n\n
This had a nasty side effect on one of my tests. To check that “same status sends no message”, I counted Queued lines in the API log and got 0. But that count could never have been anything else, because the API can’t print INFO at all. The meaningful result was the worker’s count of 0.<\/p>\n\n\n\n
This is the same lesson as the “negative test that lied” in Part 7: a check that says “nothing happened” is only proof if it could have seen something. Proper logging comes in Phase 4.<\/p>\n\n\n\n
2. Counting one failure instead of three<\/h3>\n\n\n\n
In the poison message test, I first counted only one failure, not three. Two things went wrong in my check, not in the code. I pressed Ctrl+C during a sleep 100, which only ended the sleep, so the next commands ran early. And my log window, –since 2m, started after a recent worker restart. The message was already in the DLQ, which by itself proves 3 receives. With –since 15m, I counted 3.<\/p>\n\n\n\n
3. LocalStack emptied S3 again<\/h3>\n\n\n\n
Expected, and the same as in Part 7. Recreating LocalStack wiped the bucket, so I re-uploaded my test resume.<\/p>\n\n\n\n
Known gaps<\/h2>\n\n\n\n
- \n
- A message can be lost. If SQS is unreachable at the moment of the commit, the status is saved but the message is gone, with only an error in the log. The proper fix is called a transactional outbox: save the event in a database table in the same transaction as the status change, and let a separate process send it to the queue. I haven’t built it in this phase, but I want to name it.<\/li>\n\n\n\n
- The API logs nothing at INFO. Fixed in Phase 4.<\/li>\n\n\n\n
- Duplicates on redelivery. At-least-once delivery means the same message can arrive twice. The event_id is ready for it. The fix is a later step.<\/li>\n\n\n\n
- No auto-reload for the worker. After a code change: docker compose restart worker.<\/li>\n<\/ul>\n\n\n\n
What I learned<\/h2>\n\n\n\n
- \n
- Slow work doesn’t belong inside a request. Put a note on a queue and answer straight away.<\/li>\n\n\n\n
- Receiving is not deleting. A message is only gone when the worker deletes it after success. That’s how retries work.<\/li>\n\n\n\n
- A dead-letter queue keeps the evidence. Bad messages are parked, not lost, and they can’t loop forever.<\/li>\n\n\n\n
- Long polling is cheaper and faster than asking an empty queue again and again.<\/li>\n\n\n\n
- Code against an interface. The API and worker don’t know SQS exists, so Azure Service Bus can slot in later.<\/li>\n\n\n\n
- Send IDs, not data. Smaller messages, no personal data in the queue, always-fresh data.<\/li>\n\n\n\n
- Publish after the commit. Never for a rolled-back change, and the worker never reads the old state.<\/li>\n\n\n\n
- One bad message must never kill the worker.<\/li>\n\n\n\n
- Shut down gracefully. Set a flag on SIGTERM, finish the current work, and give Docker enough time to wait.<\/li>\n\n\n\n
- A “nothing happened” check needs proof it could have seen something.<\/li>\n<\/ul>\n\n\n\n
What comes next<\/h2>\n\n\n\n
The worker can receive messages, but it still only logs them. Next, it sends the real email to the candidate. After that comes the hardest part of Phase 3: idempotency, so that a message delivered twice never sends two emails.<\/p>\n","protected":false},"excerpt":{"rendered":"
Why not just send the email? When a company moves an application from “reviewing” to “interview”, the candidate should get an email.…<\/p>\n","protected":false},"author":2,"featured_media":259,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_rezaops_part":"10","_rezaops_linkedin_url":"https:\/\/www.linkedin.com\/pulse\/building-job-board-api-part-10-queue-background-worker-reza-chegini-0rgkc\/","footnotes":""},"categories":[85,10,116],"tags":[60,8,111,191,118,194,25,112,200],"series":[117],"class_list":["post-255","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-aws","category-devops","category-django","tag-aws","tag-devops","tag-django","tag-drf","tag-job-board","tag-localstack","tag-python","tag-rest-api","tag-sqs","series-job-board-api"],"jetpack_featured_media_url":"https:\/\/rezachegini.com\/wp-content\/uploads\/2026\/11\/job-board-api-part-10-cover.png","_links":{"self":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=255"}],"version-history":[{"count":3,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/255\/revisions"}],"predecessor-version":[{"id":262,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/posts\/255\/revisions\/262"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=\/wp\/v2\/media\/259"}],"wp:attachment":[{"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=255"},{"taxonomy":"series","embeddable":true,"href":"https:\/\/rezachegini.com\/index.php?rest_route=%2Fwp%2Fv2%2Fseries&post=255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}