Concurrent Web Crawler with Cats Effect Fibers

Project roadmap

1. 01

   Set up an HTTP client

   ~1.5h

   Configure the sttp client as a Cats Effect resource for safe acquisition and release.

2. 02

   Implement single-page fetch and link extraction

   ~2h

   Fetch a page and parse outbound links from its HTML.

3. 03

   Implement concurrent fiber-based crawling

   ~3h

   Spawn fibers to crawl multiple pages concurrently up to a configured depth and concurrency limit.

4. 04

   Add URL deduplication and rate limiting

   ~2.5h

   Track visited URLs with a concurrent Ref and throttle requests per domain.

5. 05

   Add graceful shutdown and error handling

   ~2h

   Ensure in-flight fibers are cancelled cleanly on shutdown and individual failures don't crash the crawl.

6. 06

   Write tests

   ~2h

   Test link extraction, deduplication, and concurrency limits against a mock server.

Resources

• DocsScala Documentation