Concurrent Web Crawler with Worker Pools

Project roadmap

1. 01

   Design the worker pool

   ~2h

   Design a job queue and a bounded worker pool architecture using channels and sync.WaitGroup.

2. 02

   Implement the fetch worker

   ~2h

   Write the worker function that fetches a page, parses links, and extracts the page title.

3. 03

   Track depth and visited URLs

   ~1.5h

   Add depth tracking and a visited-URL set to prevent infinite crawling cycles.

4. 04

   Add per-domain rate limiting

   ~2h

   Throttle requests per domain so the crawler doesn't overwhelm any single site.

5. 05

   Generate a report

   ~1.5h

   Aggregate crawl results and write a summary report to a CSV file.

6. 06

   Add graceful shutdown

   ~2h

   Support context cancellation and timeouts so the crawler shuts down cleanly.

Resources

• DocsGo Documentation