Distributed Job Worker

Project roadmap

1. 01

   Design the protocol

   ~2.5h

   Write DESIGN.md: a job (id, kind, payload as JSON, status, attempts, max attempts, run after, lease owner, lease expiry and a lease token that increases each time the job is leased), and the API: POST /jobs (with an idempotency key), POST /lease (returns a job and its token, or nothing), POST /jobs/{id}/renew, POST /jobs/{id}/complete and /fail (each with the token). Explain why a stale token must be rejected. Done when: DESIGN.md covers every state and failure case.

2. 02

   Build the coordinator

   ~3h

   Build the coordinator with Tokio, axum and SQLite (through sqlx), with Serde types for every request and response. Leasing picks the oldest available job in a single transaction (an UPDATE ... RETURNING on one row), increasing its token and setting the expiry. Enqueue with an existing idempotency key returns the existing job. Done when: leasing from many concurrent requests never gives the same job to two workers.

3. 03

   Renew, complete and reject stale workers

   ~2h

   Renew extends the lease only when the token matches and the lease hasn't expired. Complete and fail only succeed with the current token, so a worker whose lease expired and was taken over can't overwrite the new result (answer 409). Completing twice with the same token is accepted without changing anything, so retried requests are safe. Done when: tests show stale tokens rejected, and duplicate completions ignored.

4. 04

   Retry and recover

   ~2h

   On failure, schedule the job again with exponential backoff and jitter (setting run after) until max attempts, then move it to dead. A background task in the coordinator makes jobs with expired leases available again, counting it as an attempt. Add endpoints to list and requeue dead jobs. Done when: a job that fails twice then succeeds completes, and a job whose worker vanished is leased again.

5. 05

   Write the worker

   ~2.5h

   Write a worker binary that runs N concurrent job loops with Tokio: lease a job, run its handler (chosen by kind) while a separate task renews the lease every third of its duration, then complete or fail with the token. If renewal fails, cancel the handler. Retry calls to the coordinator on network errors, and shut down cleanly on Ctrl+C. Done when: several workers process 1,000 jobs with each finished exactly once.

6. 06

   Add observability

   ~1.5h

   Add structured logs with tracing (job id, kind, attempt and worker) and a /stats endpoint on the coordinator with jobs by status, average time waiting and running, and lease expirations. Done when: /stats and the logs show what the system is doing.

7. 07

   Test failures on purpose

   ~2.5h

   Write integration tests that start the coordinator on a random port with a temporary database: concurrent leasing, idempotent enqueue, a worker killed mid-job (its lease expires and another finishes it), a slow worker whose late completion is rejected, retries into the dead list, and duplicate completion requests. Done when: cargo test passes reliably when run several times.

Resources

• DocsTokio tutorial
• DocsSerde documentation
• Docssqlx
• ArticleMartin Kleppmann: how to do distributed locking (fencing tokens)
• Docsaxum documentation