Queues, jobs, and async work
Make consumers safe to retry, give failures somewhere to land, and cap the backoff.
Async work fails quietly. The queue keeps accepting while the consumer keeps retrying, and nobody looks until the backlog is days deep.
Fixing that means consumers that are safe to retry and a dead-letter queue that somebody actually owns.
Add a dead-letter queue for messages that will never succeed
A poison message lands in the letter queue instead of looping forever
Alert on queue depth and consumer lag
Alert fires against a replayed backlog
Make a consumer idempotent so a redelivery stops causing a second effect
Replay test proving one effect from two deliveries
Make a scheduled job safe to run twice
Concurrent-run test proving a single outcome
Move a slow synchronous call into a background job
Request timing before and after, the job covered by its own tests
Put a limit on retries that never stop
Failure path tested, the ceiling documented in the description
Stop a stuck consumer from holding work forever
A hung message returns to the queue instead of stalling the worker
Drain and reprocess a dead-letter backlog nobody has opened
Every message classified, reprocessed or discarded with a recorded reason
Stop a saved record and its message from diverging
Failure-injection test proving the two cannot diverge
Put the agent team to work on your backlog
Connect a repository and assign SHIP your first issue. Your first missions are free.