Parallel Image Pipeline

Project roadmap

1. 01

   Design the pipeline

   ~2h

   Write DESIGN.md: the stages (list files, decode, filter, encode and write), the threads per stage, the bounded queues between stages (and why their size limits memory), how errors are handled per image (logged and skipped), how the pipeline shuts down (end markers flowing through the queues), and how resume works (skipping outputs that already exist and are complete). Set up a CMake project with OpenCV and Boost (for Program_options). Done when: DESIGN.md explains the stages, queues, errors and shutdown.

2. 02

   Build a bounded queue

   ~2h

   Implement a thread-safe bounded queue with std::mutex and two std::condition_variable (not full and not empty), with push (waiting when full), pop (waiting when empty), and close (waking all waiting threads so they can stop). Test it with several producers and consumers. Done when: a stress test passes millions of items through without loss or duplicates, with no ThreadSanitizer errors.

3. 03

   Implement the stages

   ~2.5h

   Write each stage as a function run by N threads: decoding with cv::imread (or imdecode from file bytes), filters built from the command line (such as --resize 800 --blur 3 --edges), and encoding and writing with cv::imwrite to a temporary name, then renaming. Catch errors for each image, record them, and continue. Done when: a folder of images is processed with the chosen filters, and corrupt files are reported without stopping.

4. 04

   Connect the pipeline and shut down cleanly

   ~2h

   Connect the stages with queues, start the threads, and shut down in order: when listing finishes, close the first queue; each stage closes its output queue after all its threads finish. Handle Ctrl+C by stopping the listing and letting images already in progress finish. Done when: the pipeline processes everything and exits cleanly, and Ctrl+C stops it quickly without corrupt outputs.

5. 05

   Resume and report

   ~1.5h

   Skip images whose output already exists and is newer than the input, so an interrupted run can simply be started again. Print progress, and at the end a report: images processed, skipped and failed, the time, and throughput and average queue fill for each stage, to show the bottleneck. Done when: re-running after an interruption skips finished images, and the report shows per-stage numbers.

6. 06

   Tune and measure

   ~2h

   Run the pipeline on a few thousand images with different thread counts per stage and queue sizes, measuring total time and peak memory, and compare with a single-threaded version. Record the results in README.md and explain which stage limited speed and why. Done when: README.md shows the measurements and conclusions.

7. 07

   Test it

   ~2h

   Write tests (with GoogleTest or Catch2) for the queue, each filter on small generated images (checking output sizes and simple pixel properties), error handling with a corrupt file, resume behavior, and a full pipeline run on a small folder. Run under ThreadSanitizer. Done when: all tests pass with no sanitizer errors.

Resources

• DocsOpenCV: image file reading and writing
• DocsOpenCV: image filtering
• Docsstd::condition_variable (cppreference)
• DocsBoost.Program_options tutorial
• DocsThreadSanitizer documentation