Parallel File Hasher

Project roadmap

1. 01

   Set up the CLI

   ~1.5h

   Create bin/phash with use strict and use warnings, and Getopt::Long options: one or more paths, --workers N (default: the number of CPU cores), --output FILE, --cache FILE, --verify URL_OR_FILE and --exclude PATTERN. List Digest::SHA, DBI, DBD::SQLite and LWP::UserAgent in a cpanfile. Validate every option and print usage on error. Done when: bin/phash --help prints all options.

2. 02

   Collect the file list

   ~1.5h

   Walk the given paths with File::Find, keeping regular files that don't match --exclude, and record each file's relative path, size and modification time. Sort the list by path using byte order so it's the same on every system. Report unreadable directories and continue. Done when: the tool prints a stable, sorted file count and total size for a test folder.

3. 03

   Hash in parallel worker processes

   ~3h

   Fork N workers, each connected to the parent with a pipe. Split the files into small jobs and send them to idle workers, so a few huge files don't leave the others waiting. Each worker hashes files with Digest::SHA's addfile (which streams the file) and writes back path, hash and status. The parent reaps workers with waitpid and reports any worker that dies. Done when: hashing a large test folder uses all workers, and the results match sha256sum.

4. 04

   Write a reproducible manifest

   ~1.5h

   After all results arrive, sort them by path and write a manifest in the standard sha256sum format (hash, two spaces, path), so it can also be checked with sha256sum -c. Write it to a temporary file and rename it into place, so an interrupted run never leaves a half-written manifest. Done when: running with 1 worker and with 8 workers produces byte-identical manifests.

5. 05

   Cache unchanged files with DBI

   ~2h

   Store each file's path, size, modification time and hash in a SQLite database through DBI. On the next run, reuse the cached hash when the size and modification time are unchanged, and only hash new or modified files. Add --no-cache to force a full run, and remove cache entries for files that no longer exist. Done when: a second run on unchanged files finishes much faster, and editing one file re-hashes only that file.

6. 06

   Verify against a remote manifest

   ~2h

   With --verify, load an existing manifest from a file, or download it with LWP::UserAgent when it's a URL (with a timeout, and failing on non-2xx responses). Compare it with the local files and report files that are missing, extra or have a different checksum, exiting with code 1 on any mismatch. Done when: verifying against a manifest with one wrong hash reports that exact file.

7. 07

   Test it

   ~2h

   Write tests in t/ with Test::More: hashing matches known values, the manifest is identical for different worker counts, the cache skips unchanged files and catches modified ones, verification finds missing and changed files, and file names with spaces work. Serve a test manifest with a small local HTTP server to test URL verification. Done when: prove -l t/ passes.

Resources

• DocsDigest::SHA on MetaCPAN
• Docsperlipc: Perl interprocess communication
• DocsDBI on MetaCPAN
• DocsLWP::UserAgent on MetaCPAN
• Docssha256sum manual page