File Duplicate Scanner
Build a C++ command-line tool that finds duplicate files in a folder and its subfolders, such as the same photo saved three times under different names, and shows them grouped together with the space you could free. It works in stages so it stays fast: it first groups files by size (files of different sizes can't be identical), then compares a quick hash of the first few kilobytes, and only then a full hash of the whole file. It never deletes anything itself; it just prints a clear report, or saves it as a file. You'll learn walking folders with std::filesystem, reading files in chunks, hashing, choosing the right containers (maps and vectors), handling errors such as unreadable files, and reading command-line options with Boost.Program_options.
- Estimate
- ~8.5h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
CMake · std::filesystem · Boost.Program_options
Project roadmap
- 01
Define the tool
~1hWrite README.md: the command takes one or more folders and options (--min-size to skip tiny files, --include-hidden, --output report.txt), and prints groups of identical files with their size and the total space wasted. Set up a CMake project with Boost.Program_options for the options and --help. Done when: the program builds and prints its help.
- 02
List files and group by size
~1.5hWalk the folders with std::filesystem::recursive_directory_iterator, skipping symbolic links, folders you can't enter (use the skip_permission_denied option) and files below the minimum size. Store each file's path and size in a struct, and group them in a std::unordered_map from size to a vector of paths. Keep only groups with 2 or more files. Done when: the tool prints how many files it found and how many share a size with another file.
- 03
Compare contents with hashes
~2hFor files in the same size group, compute a quick hash of the first 4 KB and regroup by it, then compute a full hash of the whole file, reading it in chunks of 64 KB, only for files whose quick hashes match. Use a simple, well-known hash such as FNV-1a 64-bit or implement one (or use a small SHA-256 library). Files with the same size and full hash are duplicates. Done when: on a test folder, exactly the identical files are grouped.
- 04
Print the report
~1hPrint each group of duplicates with its size and paths, sorted with the largest wasted space first, and a summary: the number of groups, duplicate files and the total space that could be freed (in KB, MB or GB). Write the same report to a file with --output. Done when: the report is easy to read, and the totals are correct.
- 05
Handle problems
~1.5hHandle a folder that doesn't exist, a file that disappears or can't be read during the scan (report it and continue), empty files (group them separately or skip them), and the same folder given twice. Catch std::filesystem::filesystem_error so one bad file never stops the scan. Done when: a scan with unreadable files completes and lists what it skipped.
- 06
Test it
~1.5hWrite tests with a small framework such as Catch2 or GoogleTest that create a temporary folder with identical files, files of the same size but different content, files with the same start but different ends, and empty files, then check the groups. Done when: the tests pass.
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack