Concurrent Web Crawler with Worker Pools
Build a web crawler in Go that starts from one page, follows its links to a chosen depth, and records every page's title, status code and links, fetching many pages in parallel with a fixed pool of worker goroutines. It stays on the starting site (or a list of allowed domains), never visits the same page twice, respects robots.txt, limits how fast it requests pages from each site, and can be stopped cleanly at any time, still writing what it found. The result is a CSV report, including broken links. Only crawl sites you own or have permission to crawl. You'll learn goroutines, channels and sync.WaitGroup, bounded worker pools, HTML parsing with goquery, URL handling, rate limiting and context cancellation.
- Estimate
- ~13h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
goroutines · channels · goquery
Project roadmap
- 01
Design the crawler
~1.5hWrite README.md: the options (start URL, -depth, -workers, -rate per domain, -allowed-domains, -max-pages, -timeout and -out), what counts as the same page (normalized URLs: lowercase host, no fragment, and removed default ports), and the output columns (URL, depth, status, title, number of links, error). Sketch how jobs flow between the frontier, workers and results. Done when: README.md explains the options and flow.
- 02
Fetch and parse a page
~2hWrite a fetch function with an http.Client that has a timeout, sends a descriptive user agent, only reads HTML responses, and limits the body size. Parse it with goquery to get the title and every a[href], resolving relative links against the page URL and keeping only http and https links. Done when: fetching a test page returns its title and absolute links.
- 03
Build the worker pool
~2.5hStart N workers reading jobs (URL and depth) from a channel and sending results to another channel. A single coordinator goroutine owns the visited set and the frontier: it adds new links from results (if under the depth limit, allowed and not yet seen) and knows when the crawl is finished by counting pending jobs, so no mutex is needed. Done when: crawling a local test site visits every page once and then stops.
- 04
Be polite
~2hFetch and respect each site's robots.txt (for example with the temoto/robotstxt package), and limit requests per domain with golang.org/x/time/rate, one limiter per domain. Handle 429 and 503 responses by backing off. Done when: request times in the log respect the per-domain limit even with many workers.
- 05
Write the report
~1.5hWrite the results to a CSV file with encoding/csv as they arrive. At the end, print a summary: pages crawled, errors by type, broken links (4xx and 5xx) with the pages that link to them, and the time taken. Done when: the CSV opens correctly in a spreadsheet, and broken links are listed with their sources.
- 06
Stop cleanly
~1.5hUse signal.NotifyContext so Ctrl+C cancels a shared context: workers stop taking new jobs, in-flight requests are cancelled, and the coordinator writes the results collected so far and the summary. Also stop at -max-pages or an overall -timeout. Done when: pressing Ctrl+C mid-crawl exits within a second with a complete, valid CSV.
- 07
Test it
~2hWrite tests with httptest.Server serving a small site with loops, relative links, broken links, a robots.txt that disallows a path, and a slow page. Check depth limits, no page visited twice, disallowed pages skipped, cancellation, and the per-domain rate. Run with -race. Done when: go test -race ./... passes.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack