Background Job Dashboard
Build a small background job system in Ruby with Redis, and a Rails dashboard to watch it, similar to what Sidekiq offers. Applications push jobs (a class name and arguments) onto queues in Redis; worker processes take them, run them, and retry failures with increasing waits, moving jobs that keep failing to a dead set. Workers report a heartbeat, so the dashboard shows which are alive and what each is doing. The dashboard shows queue sizes, running, scheduled, retrying and dead jobs, durations and error messages, and lets you retry or delete jobs. You'll learn Redis data structures (lists, sorted sets and hashes), atomic operations, reliable queues that survive worker crashes, retries and backoff, and building an operations dashboard with Rails.
- Estimate
- ~17h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
Rails · RubyGems · Redis
Project roadmap
- 01
Design the job system
~2.5hWrite DESIGN.md: jobs are JSON (id, class, args, queue, attempts, enqueued at); queues are Redis lists; a job being processed is moved atomically to a per-worker processing list (with LMOVE), so it isn't lost if the worker crashes; scheduled and retry jobs are sorted sets scored by run time; dead jobs are a capped sorted set; statistics and worker heartbeats are hashes with expiry. Done when: DESIGN.md explains every key and each job's lifecycle.
- 02
Build the client and worker
~3hWrite a gem-style library with a Job module (perform_async and perform_in) and a worker process that takes jobs with BLMOVE, finds the class, calls perform with the arguments, records the duration, and removes the job from its processing list when done. Run several threads per worker process with a configurable count. Done when: jobs pushed from a Rails console are run by the worker.
- 03
Add retries, scheduling and dead jobs
~2.5hOn failure, store the error and backtrace, increase attempts, and add the job to the retry set with exponential backoff. A poller moves due scheduled and retry jobs back to their queues, using a Lua script so two pollers never move the same job. After max attempts, move the job to the dead set. Done when: a job failing 3 times then succeeding finishes, and one that always fails ends up dead.
- 04
Recover from crashed workers
~2hEach worker writes a heartbeat every 5 seconds with an expiry of 30 seconds, including its host, pid and current jobs. A recovery task finds processing lists of workers without a heartbeat and moves their jobs back to the queue. Document that jobs may run more than once, so they must be idempotent. Done when: killing a worker with kill -9 mid-job results in the job being run by another worker.
- 05
Build the dashboard
~3hBuild a Rails app (or a mountable engine) reading from Redis: an overview with processed and failed counts and queue sizes; pages for each queue, scheduled, retries and dead jobs with pagination; job details with arguments, error and backtrace; and live workers with their current job. Add actions to retry, delete, and retry all dead jobs, protected with authentication. Done when: the dashboard shows accurate live data and its actions work.
- 06
Add limits and metrics
~1.5hAdd per-queue concurrency limits and queue priorities (workers check higher-priority queues first), a maximum job payload size, and history charts of processed and failed jobs per minute for the last hour, stored in Redis with expiry. Done when: the charts show recent activity, and priorities are respected.
- 07
Test it
~2.5hWrite RSpec tests against a real Redis (flushed between tests): enqueueing and processing, retries with backoff, scheduled jobs, the dead set, recovery of an abandoned processing list, the Lua poller not duplicating jobs with two pollers, and dashboard actions with request specs. Done when: all tests pass.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack