Concurrent File Deduplicator using Ruby Threads

Project roadmap

1. 01

   Set up the CLI

   ~1h

   Create a Ruby project with a Gemfile and bin/dedupe. Parse options with OptionParser: a directory (required), --threads N (default: the number of CPU cores from Etc.nprocessors), --delete and --yes, and --json. Print usage and exit with code 2 for a missing or invalid directory. Done when: bin/dedupe --help prints the options.

2. 02

   Walk the folder and group by size

   ~1.5h

   Collect every regular file under the directory with Find.find, skipping symlinks and empty files, and recording each file's size. Group files by size with group_by and keep only sizes that appear more than once, since a file with a unique size can't have a duplicate. Print how many files were found and how many need hashing. Done when: the counts are correct for a test folder.

3. 03

   Hash files with a thread pool

   ~2.5h

   Put the files to hash into a Queue and start N worker threads. Each worker takes a path, computes its SHA-256 by reading the file in 1 MB chunks with Digest::SHA256 (so large files never load fully into memory), and stores the result in a shared hash protected by a Mutex. Report unreadable files instead of crashing. Done when: the hashes match the output of sha256sum for test files.

4. 04

   Report duplicates

   ~1.5h

   Group paths by hash; each hash with more than one path is a duplicate group. Print the groups sorted by wasted space (size × (copies − 1)), largest first, then the total reclaimable space in human-readable units (KB, MB, GB). With --json, print the same data as JSON. Done when: a test folder with known duplicates reports exactly the expected groups.

5. 05

   Delete extra copies safely

   ~1.5h

   With --delete, keep the oldest file in each group (by modification time) and list what would be deleted. Delete the files only when --yes is also given. Print what was deleted and how much space was freed. Never delete the last copy, and report any file that fails to delete. Done when: --delete alone deletes nothing, and --delete --yes removes exactly the extra copies.

6. 06

   Benchmark and explain

   ~2h

   Create a large test folder (for example 2,000 files of various sizes), then time the hashing with 1, 2, 4 and 8 threads using Benchmark.realtime, running each several times. Write the results and your explanation in BENCHMARK.md: where threads helped, where they didn't, and how the GVL and disk caching affected the numbers. Done when: BENCHMARK.md has a results table and an explanation.

7. 07

   Write tests

   ~1.5h

   Write minitest or RSpec tests that create temporary folders with duplicates, same-size files with different content, empty files and nested folders. Test grouping, hashing, the order of deleted files, and that no deletion happens without --yes. Done when: the tests pass.

Resources

• DocsRuby Thread class
• DocsRuby Thread::Queue
• DocsRuby Digest library
• DocsOptionParser
• DocsRuby Benchmark module