File Deduplication Tool

Project roadmap

1. 01

   Set up the project

   ~1.5h

   Create a CMake project using C++17 or newer that links OpenSSL (find_package(OpenSSL REQUIRED) and target_link_libraries with OpenSSL::Crypto). The program takes a directory path and two flags: --report (print only, the default) and --interactive. Print usage and exit with code 2 if the path is missing or isn't a directory. Done when: the project builds with CMake and prints usage for bad input.

2. 02

   List all files

   ~1.5h

   Use std::filesystem::recursive_directory_iterator with the skip_permission_denied option to collect every regular file under the root, along with its size. Skip symlinks so the same file isn't counted twice, and ignore empty files. Done when: the program prints the number of files found and their total size.

3. 03

   Group files by size first

   ~1h

   Put the files in a std::unordered_map from size to a list of paths. Only groups with two or more files can contain duplicates, so drop every size that appears once. Print how many files still need to be hashed. Done when: files with a unique size are excluded before any hashing happens.

4. 04

   Hash file contents with SHA-256

   ~2.5h

   Write a function that computes a file's SHA-256 with OpenSSL's EVP interface (EVP_MD_CTX_new, EVP_DigestInit_ex with EVP_sha256, EVP_DigestUpdate, EVP_DigestFinal_ex), reading the file in 64 KB chunks so large files never load fully into memory. Return the hash as a hex string, and report unreadable files instead of crashing. Done when: the hash of a test file matches the output of sha256sum.

5. 05

   Find and report duplicate groups

   ~2h

   Within each same-size group, map each hash to its list of paths; any hash with two or more paths is a duplicate group. Print every group with the size and all paths, sorted by wasted space (size × (copies − 1)), largest first, then a total of reclaimable space in MB. Done when: a test folder with known duplicates produces exactly the expected groups.

6. 06

   Delete duplicates interactively

   ~2h

   With --interactive, go through each group, list its files with numbers, and ask which one to keep. Delete the others with std::filesystem::remove only after the user confirms. Allow skipping a group, never delete the last copy, and print a summary of files deleted and space freed. Done when: in a test folder, choosing a file to keep deletes exactly the other copies.

7. 07

   Test it

   ~2h

   Write tests (with Catch2, GoogleTest or a small test executable registered with CTest) that build a temporary folder with duplicates, unique files, same-size files with different content, and an empty file. Check the grouping and the hashing, and that report mode deletes nothing. Done when: ctest passes.

Resources

• Docsstd::filesystem (cppreference)
• DocsOpenSSL EVP digest functions
• DocsCMake tutorial
• DocsCatch2 on GitHub