Distributed Task Queue in Kotlin using Coroutines

Project roadmap

1. 01

   Design the queue

   ~2h

   Write DESIGN.md: a task is JSON (id, type, payload, attempts, max attempts, created at); the Redis layout: a pending list, a processing sorted set scored by lease expiry, a delayed sorted set for retries, a dead list, and a hash of task status and results; how a worker takes a task (atomically moving it from pending to processing with a Lua script) and what happens on success, failure and crash. Done when: DESIGN.md explains every key and state change.

2. 02

   Build the worker pool

   ~3h

   Write a worker using a Redis client (such as Lettuce with its coroutine API). A fetcher coroutine takes tasks atomically and sends them into a Channel; N processor coroutines read from it and run the handler for each task type. Use a supervisor scope so one failing task doesn't cancel the others. On success, remove the task from processing and store the result. Done when: two worker processes share 1,000 tasks, and each task runs once.

3. 03

   Add retries and the dead-letter queue

   ~2h

   On failure, record the error, increase attempts, and move the task to the delayed set with exponential backoff and jitter; a mover coroutine puts due tasks back on pending. After max attempts, move the task to the dead list. A reaper moves tasks whose lease expired (from a crashed worker) back to pending. Use Lua scripts so no step is ever half done. Done when: a failing task ends in the dead list, and a task from a killed worker is completed by another.

4. 04

   Add the submission API

   ~2h

   Build a Ktor server with POST /tasks (validate the type and payload, return the task id, and accept an idempotency key so the same task isn't submitted twice), GET /tasks/{id} (status, attempts, result or error), GET /dead and POST /dead/{id}/retry. Use kotlinx.serialization. Done when: tasks can be submitted and tracked through the API.

5. 05

   Add monitoring and graceful shutdown

   ~2h

   Expose GET /metrics with the pending, processing, delayed and dead counts, and tasks completed and failed per minute. On shutdown, stop fetching, let processors finish their current tasks up to a timeout, and return unfinished tasks to pending. Log each task with its id, type and duration. Done when: stopping a worker mid-load finishes or returns every task, and the metrics are accurate.

6. 06

   Test it

   ~2.5h

   Write tests against a real Redis started with Testcontainers: tasks processed once with several workers, retries with backoff (using short delays in tests), dead-lettering, lease expiry recovery, idempotent submission, and graceful shutdown. Use kotlinx-coroutines-test where timing matters. Done when: all tests pass reliably.

Resources

• DocsKotlin coroutines guide
• DocsKotlin coroutines: channels
• DocsLettuce Redis client
• DocsRedis: reliable queue pattern (LMOVE)
• DocsKtor server documentation