Concurrent Image Downloader
Build a Go command-line tool that downloads a list of images from a manifest file, several at a time but never more than a set limit. Each image is checked before and after downloading: it must really be an image, must not exceed a maximum size, and must match its expected checksum when one is given. Temporary failures are retried with increasing waits, finished files are never downloaded again, and an interrupted run can simply be started again and picks up where it stopped. At the end, a report shows what succeeded, what was skipped and what failed. You'll learn net/http with timeouts and contexts, goroutines with a worker pool, limiting size while streaming, verifying checksums, safe file writes, and testing with httptest.
- Estimate
- ~10.5h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
net/http · goroutines · crypto/sha256
Project roadmap
- 01
Define the manifest and behavior
~1.5hWrite README.md: the manifest is a CSV or JSON list of entries with a URL, an output file name and an optional SHA-256 checksum. Options: -concurrency (default 4), -max-size (default 10 MB), -retries (default 3), -out folder and -timeout. List what happens for every outcome: success, already downloaded, too large, not an image, wrong checksum, and HTTP or network error. Done when: README.md covers every option and outcome.
- 02
Download one file safely
~2hWrite a download function using an http.Client with a timeout and a context. Check the status code, reject files whose Content-Length exceeds the limit, and stream the body through io.LimitReader so a server that lies about the size can't fill your disk. Write to a temporary file and rename it only after everything succeeded. Done when: a normal image downloads, and an oversized one is rejected without leaving a file behind.
- 03
Check the content
~1.5hCheck that the file is an image by looking at its first bytes with http.DetectContentType, and optionally that it decodes with image.DecodeConfig. Calculate SHA-256 while writing (with io.MultiWriter), and compare it with the manifest when a checksum is given. Reject mismatches and delete the temporary file. Done when: a text file renamed .jpg and a file with the wrong checksum are both rejected.
- 04
Download in parallel
~2hStart a fixed number of worker goroutines reading entries from a channel, and collect the results through another channel. Cancel everything cleanly on Ctrl+C using signal.NotifyContext. Skip entries whose output file already exists and matches the checksum (or simply exists when there's no checksum), so re-running continues where it stopped. Done when: 100 downloads run with at most the chosen number at once, and a second run skips them all.
- 05
Retry and report
~1.5hRetry timeouts, network errors, 429 and 5xx responses with exponential backoff and jitter, but never 404 or validation failures. Print progress as files finish, and at the end a summary with counts per outcome and the list of failures with their reasons. Write the full report as JSON with -report. Exit with a non-zero code if anything failed. Done when: the summary matches what happened, and failures show clear reasons.
- 06
Test with a fake server
~2hWrite tests using httptest.Server that serves a real image, a fake image, an oversized file, a slow response, a 500 that succeeds on retry, and a 404. Test the concurrency limit (count simultaneous requests on the server), skipping existing files, and checksum checks. Run with -race. Done when: go test -race ./... passes.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack