Concurrent Web Crawler with Cats Effect Fibers

Project roadmap

1. 01

   Set up the HTTP client

   ~1.5h

   Create an sbt project with Cats Effect, fs2, sttp (with a Cats Effect backend such as the fs2 or http4s backend) and jsoup for HTML parsing. Create the sttp backend as a Resource in an IOApp, with connection and read timeouts and a descriptive user agent, and fetch one page to check it works. Done when: the app fetches a page and prints its status, and the client is closed on exit.

2. 02

   Fetch a page and extract links

   ~1.5h

   Write a function that fetches a URL, accepts only HTML responses up to a size limit, parses it with jsoup, and returns the title and links resolved to absolute URLs (dropping fragments, mailto links and non-http schemes). Normalize URLs so the same page always has the same form. Done when: unit tests extract the right links from saved HTML.

3. 03

   Crawl concurrently with fibers

   ~3h

   Crawl breadth-first: process each depth level with parTraverseN (or a Semaphore limiting how many fetches run at once), collect the new links, and continue to the next level until the depth limit or a maximum number of pages. Only follow links on the starting domain (or an allowed list). Done when: a local test site is crawled completely with at most N requests in flight.

4. 04

   Deduplicate and rate limit

   ~2h

   Keep visited URLs in a Ref[IO, Set[String]], adding each URL atomically with modify so two fibers can never both claim it. Limit requests per domain with a small per-domain rate limiter (for example a Semaphore plus a delay, or a token bucket in a Ref), and respect robots.txt. Done when: no page is fetched twice, and requests to one site are spaced out.

5. 05

   Handle errors and shut down cleanly

   ~1.5h

   Catch errors per page (timeouts, bad status codes and parse errors) with handleErrorWith and record them as results, so one failure never stops the crawl. Stream results with fs2 to a JSON Lines file as they arrive. On Ctrl+C, IOApp cancels the main fiber: make sure in-flight requests are cancelled, the output file is flushed and closed, and a summary is printed. Done when: interrupting a crawl leaves a complete output file and a summary.

6. 06

   Write tests

   ~2h

   Write munit-cats-effect tests against a fake server (an http4s server in the test or sttp's stub backend) serving a site with loops, broken links, slow pages and external links: link extraction, deduplication under concurrency, depth limits, the concurrency limit (counting simultaneous requests on the server), error handling, and cancellation. Done when: sbt test passes reliably.

Resources

• DocsCats Effect: concepts (fibers and cancellation)
• DocsCats Effect: Resource
• Docssttp client documentation
• Docsfs2 guide
• Docsjsoup: HTML parser