Distributed Task Queue

Project roadmap

1. 01

   Design the schema

   ~2h

   Write migrations for a tasks table: id, queue name, type, payload (jsonb), status (queued, leased, succeeded, failed or dead), attempts, max attempts, run_after, lease owner and lease expiry, idempotency key (unique per queue), last error and timestamps. Add a partial index on (queue, run_after) for queued tasks, so finding the next task stays fast. Use pgx for database access. Done when: the migrations create the table and indexes on an empty database.

2. 02

   Define the gRPC API

   ~2h

   Write queue.proto with a TaskQueue service: Enqueue (queue, type, payload, idempotency key, max attempts, optional delay), Lease (a worker id, queues and a lease duration, returning a task or nothing), Heartbeat (extend a lease), Complete, Fail (with an error and whether to retry) and GetTask. Generate the Go code and implement Enqueue with INSERT ... ON CONFLICT on the idempotency key, returning the existing task on a duplicate. Done when: enqueuing the same idempotency key twice returns the same task id.

3. 03

   Lease tasks safely

   ~2.5h

   Implement Lease in one transaction: select the oldest queued task whose run_after has passed, with FOR UPDATE SKIP LOCKED, and mark it leased with the worker id and an expiry time. SKIP LOCKED lets many workers lease at the same time without blocking each other or getting the same task. Heartbeat extends the expiry only if the caller still owns the lease. Done when: 10 concurrent workers never receive the same task.

4. 04

   Complete, fail and retry

   ~2h

   Complete marks a task succeeded only if the caller still owns an unexpired lease, since otherwise another worker may already be running it. Fail increases attempts and, if attempts remain, puts the task back to queued with run_after = now + base × 2^attempts plus random jitter. Otherwise it moves the task to dead with the error. Done when: a task that fails twice then succeeds ends as succeeded, and one that always fails ends as dead.

5. 05

   Recover expired leases

   ~1.5h

   Run a reaper every few seconds, in the server, that puts tasks whose lease has expired back to queued (counting it as an attempt), so a crashed worker's tasks aren't lost. Make sure a slow worker that finishes after its lease expired can't overwrite the new result: its Complete call must be rejected. Done when: killing a worker in the middle of a task leads another worker to finish it.

6. 06

   Write a worker library and dead-letter tools

   ~2.5h

   Write a Go worker package: register a handler per task type, poll Lease with backoff when the queue is empty, send heartbeats while a handler runs, and report Complete or Fail. Handlers must be idempotent, since a task can run more than once. Add a CLI to list dead tasks, show their errors and requeue them. Done when: a sample worker processes tasks end to end, and dead tasks can be requeued with the CLI.

7. 07

   Test under concurrency and failure

   ~2.5h

   Write integration tests with a real PostgreSQL (testcontainers-go): no task is leased twice by 20 concurrent workers, idempotent enqueue, retries and the dead-letter queue, lease expiry after a simulated crash, and a late Complete being rejected. Run them with -race. Done when: go test -race ./... passes reliably when run several times.

Resources

• DocsPostgreSQL: SELECT ... FOR UPDATE SKIP LOCKED
• DocsgRPC Go basics tutorial
• Docspgx PostgreSQL driver
• DocsTestcontainers for Go
• ArticleExponential backoff and jitter (AWS)