Directory File Deduplicator by Content Hash

Project roadmap

1. 01

   Set up the CLI with clap

   ~1h

   Create a binary crate and add clap (with the derive feature), walkdir, sha2, hex and anyhow. Define the arguments: a directory, --min-size (default 1 byte), --delete, --yes and --json. Return an error with a clear message if the path isn't a directory. Done when: dedupe --help shows every option and a bad path prints an error.

2. 02

   Walk the directory tree

   ~1.5h

   Use walkdir::WalkDir to visit every entry, keeping only regular files (skip directories and symlinks) at least --min-size bytes big, and store each path with its size. When an entry can't be read (for example, permission denied), print a warning and continue instead of stopping. Done when: the tool prints the number of files found and their total size for a test folder.

3. 03

   Group by size, then hash

   ~2h

   Group files in a HashMap<u64, Vec<PathBuf>> by size and drop sizes with only one file. For the rest, compute a SHA-256 hash with sha2::Sha256 by reading each file through a BufReader in 64 KB chunks, so large files are never loaded fully into memory. Done when: the hash of a test file matches sha256sum, and files with unique sizes are never hashed.

4. 04

   Report duplicate groups

   ~1.5h

   Group the hashed files in a HashMap from hash to paths; any hash with two or more paths is a duplicate group. Print the groups sorted by wasted space, largest first, followed by the total space that could be reclaimed, in human-readable units. With --json, print the same data as JSON using serde_json. Done when: a test folder with known duplicates produces exactly the expected groups.

5. 05

   Delete extra copies safely

   ~1.5h

   With --delete, keep the first path of each group (sorted alphabetically) and list the files that would be removed. Ask for confirmation on stdin unless --yes is given, then delete with std::fs::remove_file, reporting any failure and continuing. Never delete the last copy. Done when: declining the prompt deletes nothing, and confirming removes exactly the extra copies.

6. 06

   Show progress

   ~1h

   While hashing, print a progress line to stderr showing files hashed out of the total and the MB processed, updating in place with \r. Keep stdout clean so the report can be piped to a file. Done when: progress is visible during a large scan, and redirecting stdout to a file contains only the report.

7. 07

   Write tests

   ~2h

   Write integration tests using the tempfile crate to create folders with duplicates, same-size files with different content, nested folders and an empty file. Test the grouping, the hashing, and that delete mode keeps exactly one file per group. Done when: cargo test passes.

Resources

• Docswalkdir crate
• Docssha2 crate
• Docsclap documentation
• Docsanyhow crate
• DocsThe Rust Programming Language (book)