Distributed Task Queue with Celery and Redis

Project roadmap

1. 01

   Set up Celery with Redis

   ~1.5h

   Run Redis with Docker. Create a Celery app configured with Redis as the broker and result backend, and settings such as task_acks_late and a visibility timeout, explaining in comments what each changes. Define a simple add task, start a worker, and call it from a Python shell with .delay() and .get(). Done when: a task sent from the shell runs on the worker and returns its result.

2. 02

   Write real tasks

   ~2h

   Implement two tasks that take real time: resize an image into several sizes with Pillow, and generate a CSV or PDF report from sample data. Pass only small arguments (file paths or ids, never file contents), and make each task idempotent, so running it twice gives the same result without duplicates (for example writing to a temporary file then renaming it). Done when: both tasks produce correct output when run through workers.

3. 03

   Add retries and failure handling

   ~2h

   Configure retries for temporary errors (network errors, or a simulated flaky service) with autoretry_for, retry_backoff, retry_jitter and max_retries, and don't retry permanent errors such as an invalid input file. Record tasks that fail for good in a failed jobs table or file with the error and arguments, and add a script to re-run them. Set time limits per task. Done when: a flaky task succeeds after retries, and a permanently failing one is recorded with its error.

4. 04

   Schedule periodic tasks

   ~1h

   Use Celery beat to run a cleanup task every hour (deleting results and files older than a day) and a daily summary report. Make sure a periodic task can't run twice at once if it takes long (for example with a Redis lock). Done when: beat starts the scheduled tasks on time, and overlapping runs are prevented.

5. 05

   Build the Flask API

   ~2h

   Build a Flask app with POST /jobs/resize (upload an image, validated for type and size, saved to disk, then queued) and POST /jobs/report, each returning 202 with the job id, and GET /jobs/<id> returning the state (pending, started, retrying, success or failure), progress if available, and the result or error. Use update_state to report progress from long tasks. Done when: jobs can be submitted and followed through the API until they finish.

6. 06

   Scale and monitor

   ~1.5h

   Run everything with Docker Compose (Redis, the API, beat and several workers), send 100 jobs, and compare how long they take with 1 and 4 workers. Set up Flower to watch workers and tasks. Record the results in README.md. Done when: README.md shows the effect of adding workers.

7. 07

   Test it

   ~2h

   Write pytest tests: task logic called directly as functions, retries with a flaky fake (using task_always_eager in tests, or a real worker in an integration test), idempotency when a task runs twice, the API with Flask's test client, and the cleanup task. Done when: pytest passes.

Resources

• DocsCelery: first steps
• DocsCelery: tasks (retries and idempotency)
• DocsCelery: periodic tasks (beat)
• DocsFlask documentation
• DocsFlower: Celery monitoring