Large File Sync Engine

Project roadmap

1. 01

   Design the sync protocol

   ~2.5h

   Write DESIGN.md: how files are chunked (content-defined chunking with a rolling hash such as FastCDC, with minimum, average and maximum chunk sizes), chunks identified by their SHA-256 hash, a file described by its ordered list of chunk hashes (a manifest), the gRPC service (which chunks are missing, upload chunks as a stream, commit a manifest, and download a file), how resuming works, and what the PostgreSQL tables hold. Done when: DESIGN.md covers chunking, the protocol and recovery.

2. 02

   Chunk files

   ~2.5h

   Implement content-defined chunking (your own implementation of FastCDC or Rabin fingerprinting, or a library, explaining how it works), streaming through files without loading them into memory. Show that inserting a few bytes at the start of a large file changes only one or two chunks, while fixed-size chunking would change all of them. Done when: tests confirm chunk boundaries are stable after insertions.

3. 03

   Build the gRPC service

   ~3h

   Define the service in a .proto file and generate the Go code. The server stores chunks as files named by their hash (writing to a temporary name, verifying the hash, then renaming), records chunks and file manifests in PostgreSQL (with pgx), answers which of a list of chunks are missing, and commits a manifest only when all its chunks exist. Limit message sizes and concurrent streams. Done when: a client can upload a file, and the server stores its chunks and manifest.

4. 04

   Sync only what changed

   ~2h

   Write the client's sync command: chunk the local file, ask which chunks are missing, upload only those (several at a time), then commit the manifest. Print how many chunks and bytes were sent compared with the file size. Add a download command that rebuilds a file from its manifest and verifies the whole file's hash. Done when: syncing a large file after a small edit uploads only a few chunks.

5. 05

   Resume interrupted transfers

   ~2h

   Because chunks are stored independently and committed only when complete, an interrupted sync simply asks again which chunks are missing and continues. Handle a chunk upload cut off halfway (the partial temporary file is discarded), retries with backoff on network errors, and deadlines on every call. Clean up chunks no manifest uses after a grace period. Done when: killing the client mid-sync and running it again completes the sync without re-sending finished chunks.

6. 06

   Add observability

   ~1.5h

   Log each sync with bytes sent, chunks reused and duration, expose server metrics (chunks stored, deduplication ratio, bytes received and errors), and add a verify command that checks every stored chunk's hash on the server. Done when: the metrics show the deduplication savings, and verify finds a chunk corrupted on purpose.

7. 07

   Test it

   ~2.5h

   Write tests with a server running in the test (and PostgreSQL in Testcontainers): upload and download round trips for small, empty and multi-gigabyte generated files, edits at the start, middle and end, interrupted uploads resumed, corrupted chunks rejected, and concurrent syncs of the same file. Run with -race. Done when: go test -race ./... passes.

Resources

• ArticleFastCDC paper (USENIX ATC 2016)
• ArticleRolling hash (Wikipedia)
• DocsgRPC Go basics tutorial (streaming)
• Docspgx PostgreSQL driver
• Articlersync and its delta algorithm (Wikipedia)