Async Web Crawler with Tokio

Project roadmap

1. 01

   Fetch and parse one page

   ~2h

   Create a project with tokio, reqwest, scraper, url and clap. Write an async fetch_page function using one shared reqwest::Client with a timeout and a descriptive user agent, that only accepts HTML responses up to a size limit, and returns the title and every a[href] link resolved against the page URL (with Url::join), keeping only http and https links and dropping #fragments. Done when: fetching a test page returns its title and absolute links.

2. 02

   Crawl with a concurrency limit

   ~2.5h

   Crawl breadth-first: keep a queue of (URL, depth), and spawn a Tokio task per page, each acquiring a permit from an Arc<Semaphore> before fetching, so at most N requests run at once. Collect results through an mpsc channel into a single coordinator task that adds new links to the queue. Stop when the queue is empty and no tasks are running. Done when: a local test site is crawled completely with at most N requests at a time.

3. 03

   Visit each page once

   ~1.5h

   Normalize URLs before comparing them (lowercase host, remove default ports and fragments, and sort or remove query parameters if you choose), and keep a HashSet of visited URLs owned by the coordinator, so no Mutex is needed. Mark a URL as visited when it's queued, not when it's fetched, so it's never queued twice. Done when: a site with many links pointing to the same pages is crawled with each page fetched once.

4. 04

   Stay on the site and be polite

   ~2h

   Only follow links on the starting domain (with an option to allow others), stop at the maximum depth and a maximum number of pages, respect robots.txt (fetched once, for example with the texting_robots crate), and wait a minimum delay between requests. Back off on 429 and 503 responses. Done when: disallowed paths and other domains are skipped, and requests are spaced out.

5. 05

   Save results and stop cleanly

   ~2h

   Write each result (URL, depth, status, title, number of links, and error if any) as a JSON line as soon as it arrives. On Ctrl+C (tokio::signal::ctrl_c), stop queuing new pages, let running requests finish or cancel them after a short wait, flush the output file, and print a summary. Done when: pressing Ctrl+C mid-crawl leaves a complete, valid output file.

6. 06

   Test it

   ~2h

   Write tests with a local test server (for example with the wiremock crate or a small axum app) serving a site with loops, broken links, a disallowed path, a slow page and links to another domain. Check the depth limit, no duplicate fetches, the concurrency limit, and that robots.txt is respected. Done when: cargo test passes.

Resources

• DocsTokio tutorial
• Docstokio::sync::Semaphore
• Docsreqwest documentation
• Docsscraper crate documentation
• DocsGoogle: introduction to robots.txt