Parallel Task Runner with Worker Pool

Project roadmap

1. 01

   Design the job queue file format

   ~1h

   Define a simple line-based format for job commands and their IDs, read from a queue file.

2. 02

   Build a FIFO-based semaphore for concurrency limiting

   ~3h

   Use mkfifo and read/write tokens to implement an N-worker semaphore controlling how many jobs run at once.

3. 03

   Dispatch jobs as background processes

   ~2.5h

   Loop over the queue, acquiring a semaphore slot before backgrounding each job with proper output redirection.

4. 04

   Collect exit codes and per-job logs

   ~2h

   Capture each job's exit status into a results file and store its stdout/stderr in per-job log files.

5. 05

   Add progress reporting and summary

   ~1.5h

   Print live progress (completed/total) and a final summary table of successes and failures.

6. 06

   Add flock-based locking for shared state

   ~1.5h

   Use flock to protect the shared results file from concurrent writes across workers.

Resources

• DocsBash Reference Manual