Distributed Job Scheduler
Build a job scheduler in C# that runs background tasks on a schedule (every hour, every night at 2:00, or once at a given time) across several worker machines, making sure each run happens exactly once even when workers crash or several try at the same moment. Schedules and runs are stored in PostgreSQL; workers claim due runs with row locks, keep a lease alive while running, and failed runs are retried with increasing waits. An ASP.NET Core API lets you create schedules, trigger jobs manually, and see the history of every run with its duration and errors. You'll learn cron expressions, safe work claiming with SELECT ... FOR UPDATE SKIP LOCKED, leases and heartbeats, retries, idempotency, background services in .NET, and testing distributed behavior.
- Estimate
- ~16h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
ASP.NET Core · System.Text.Json · PostgreSQL
Project roadmap
- 01
Design the scheduler
~2.5hWrite DESIGN.md: a job has a name, a type (which code runs), a JSON payload, a schedule (a cron expression, or a single run time), a time zone, max attempts and a timeout. Each planned execution is a run with a status (scheduled, running, succeeded, failed or dead), attempts, lease owner and expiry, and error. Describe how runs are created from schedules without duplicates, and what happens when a worker dies. Done when: DESIGN.md covers every state and failure.
- 02
Store jobs and plan runs
~2.5hCreate the PostgreSQL tables with a migration, including a unique index on (job id, scheduled time) so the same run can never be planned twice. Write a planner, run by every worker, that computes upcoming run times from cron expressions with the Cronos library in the job's time zone and inserts them with ON CONFLICT DO NOTHING. Done when: running the planner on several workers at once creates each run exactly once.
- 03
Claim and execute runs
~3hWrite a BackgroundService worker that claims due runs with SELECT ... FOR UPDATE SKIP LOCKED in a short transaction, setting the lease owner and expiry. Execute the job's handler (resolved from the type through dependency injection) with a CancellationToken for the timeout, renewing the lease while it runs, then record success or failure. Done when: several workers share the runs, and no run executes twice.
- 04
Retry and recover
~2hOn failure, schedule another attempt with exponential backoff and jitter, until max attempts, then mark the run dead. A reaper returns runs with an expired lease (from a crashed worker) to scheduled. A worker whose lease expired must not record a result. Document that handlers must be idempotent. Done when: killing a worker mid-run leads another to complete it, and a failing job ends as dead after its attempts.
- 05
Build the management API
~2hAdd ASP.NET Core endpoints to create, update, pause and delete jobs (validating cron expressions), trigger a run now, list runs with filters (job, status and date range) and pagination, and requeue dead runs. Use System.Text.Json with enums as strings. Done when: jobs can be managed and their run history inspected through the API.
- 06
Add observability
~1.5hLog each run with the job, run id, worker, attempt and duration. Expose metrics: runs by status, queue delay (how late runs start), run duration and active workers. Add health checks that report when due runs are waiting too long. Done when: the metrics and health checks show the scheduler's state.
- 07
Test it
~2.5hWrite integration tests with Testcontainers PostgreSQL: planning without duplicates across workers, 5 workers claiming 200 runs with each run executed once, retries and dead runs, lease expiry after a simulated crash, a late result being rejected, and API validation. Done when: dotnet test passes reliably when run several times.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack