Workflow Orchestrator

Project roadmap

1. 01

   Set up the project and schema

   ~2h

   Create a Spring Boot 3 project (Java 17 or newer) with Web, Data JDBC or JPA, Validation, Flyway and the PostgreSQL driver. Write Flyway migrations for workflow_definitions (name, JSON definition as jsonb), workflow_runs (status, timestamps) and job_runs (run id, job name, status, attempt, next_attempt_at, locked_by, locked_until, last_error). Done when: the app starts and Flyway creates the schema in PostgreSQL.

2. 02

   Define and validate workflows

   ~2.5h

   A workflow definition is JSON parsed with Jackson: a name and a list of jobs, each with a name, a type (for example http, shell or sleep, run by handler classes you write), parameters, dependsOn and maxAttempts. Add POST /workflows that rejects unknown job types, dependencies on jobs that don't exist, and cycles, detected with a topological sort, returning 400 with every problem listed. Done when: valid workflows are saved, and invalid ones are rejected with clear errors.

3. 03

   Start runs and schedule jobs

   ~2h

   Add POST /workflows/{name}/runs, which creates a workflow run and one job_run per job, all with status PENDING. A job becomes READY when all the jobs it depends on have SUCCEEDED. Write this rule as a single, well-tested method that updates statuses inside a transaction. Done when: starting a run marks exactly the jobs without dependencies as READY.

4. 04

   Execute jobs with workers

   ~3h

   Run a pool of worker threads (a scheduled task or an ExecutorService). Each worker claims a READY job whose next_attempt_at has passed using SELECT ... FOR UPDATE SKIP LOCKED, so two workers never take the same job, sets it to RUNNING with a lease (locked_until = now + 60 seconds), runs its handler, and records SUCCEEDED or FAILED. After each result, update which jobs are now READY. Done when: a workflow with parallel branches runs every job once, in a valid order.

5. 05

   Retry with backoff

   ~2h

   When a job fails, and it has attempts left, set it back to READY with next_attempt_at = now + base × 2^attempt plus some random jitter, and save the error message. When it runs out of attempts, mark it FAILED, mark the jobs that depend on it as SKIPPED, and mark the run FAILED once nothing else can progress. Done when: a job that fails twice then succeeds completes the run, and a job that always fails ends the run as FAILED.

6. 06

   Recover after a crash

   ~2.5h

   On startup, and every minute, find RUNNING jobs whose lease has expired, which means their worker died, and put them back to READY, counting it as an attempt. Make handlers safe to repeat, and document which ones are. Add GET /runs/{id} showing every job's status, attempts and last error. Done when: killing the app in the middle of a run and starting it again completes the run, without re-running jobs that already succeeded.

7. 07

   Test against a real database

   ~2.5h

   Write tests with JUnit 5 and Testcontainers PostgreSQL: cycle detection, dependency ordering, parallel jobs, retries with a fake handler that fails a set number of times, SKIPPED dependents, and crash recovery with an expired lease. Also test that two workers claiming at the same time never get the same job. Done when: ./mvnw test (or ./gradlew test) passes.

Resources

• DocsSpring Boot reference documentation
• DocsPostgreSQL: SELECT ... FOR UPDATE SKIP LOCKED
• DocsFlyway documentation
• DocsTestcontainers for Java
• ArticleTopological sorting (Wikipedia)