Parallel Task Runner with Worker Pool
Build a Bash script that runs a list of shell commands in parallel, but never more than N at a time, similar to a simple version of GNU parallel or xargs -P. You give it a file with one job per line (an id and a command) and a number of workers; it runs the jobs, saves each job's output to its own log file, records every exit code, shows live progress, and ends with a summary of what succeeded and failed. The worker limit is built with a named pipe (FIFO) used as a token bucket, and shared files are protected with flock so parallel jobs don't corrupt them. You'll learn background jobs and wait, file descriptors and FIFOs, flock, signal handling with trap, and writing robust Bash.
- Estimate
- ~12.5h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
mkfifo · wait · flock
Project roadmap
- 01
Define the job file and options
~1.5hCreate runjobs.sh with set -euo pipefail. The job file has one job per line as id<TAB>command; blank lines and lines starting with # are ignored, and ids must be unique. Parse options with getopts: -j number of workers (default: the number of CPUs from nproc), -o log folder, -t timeout per job in seconds. Validate everything and print usage on errors. Done when: the script validates a sample job file and prints its jobs.
- 02
Limit concurrency with a FIFO
~2.5hCreate a FIFO with mkfifo in a temporary folder, open it for reading and writing on a spare file descriptor (exec 3<>"$fifo") and remove the file. Write N tokens (one per line) into it. Before starting a job, read one token (blocking when none is left); when a job finishes, write its token back. Done when: with -j 3 and 10 jobs running sleep 2, at most 3 run at the same time.
- 03
Run jobs in the background
~2hStart each job in a background subshell running bash -c with the command, sending its output and errors to log-folder/id.log, with an optional timeout using the timeout command. The subshell returns its token even when the job fails, using a trap on EXIT. Collect the process ids and wait for all of them at the end. Done when: all jobs run, each with its own log file.
- 04
Record results safely
~1.5hEach job appends id, exit code, start and end times, and duration to a shared results.tsv. Because several jobs finish at once, wrap each append in flock on a lock file so lines never get mixed together. Record a timeout (exit code 124) separately from a failure. Done when: running 50 fast jobs with -j 10 gives exactly 50 intact lines in results.tsv.
- 05
Show progress and a summary
~1.5hPrint progress lines like [12/50] done: build-docs (ok, 3.2s) as jobs finish. At the end, print a summary table with the counts of succeeded, failed and timed-out jobs, the failed ids with the path to their log, and the total time. Exit with code 1 if any job failed. Done when: the summary matches results.tsv, and the exit code reflects failures.
- 06
Handle Ctrl+C cleanly
~1.5hTrap INT and TERM: stop starting new jobs, send TERM to running jobs (using their process ids), wait for them, mark them as cancelled in the results, remove the temporary folder, and still print the summary. Done when: pressing Ctrl+C during a run leaves no jobs running and no temporary files behind.
- 07
Test and lint
~2hWrite tests with bats-core: the concurrency limit is respected (record start and end times and check the overlap), failures and timeouts are recorded, output goes to the right log, duplicate ids are rejected, and results.tsv stays intact under load. Run shellcheck on the script. Done when: the tests pass, and shellcheck reports no warnings.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack