Async Web Crawler

Project roadmap

1. 01

   Set up the project

   ~1.5h

   Create a Python 3.11+ project with httpx, beautifulsoup4 (or selectolax), fastapi, uvicorn, pandas and pytest with pytest-asyncio, listed in pyproject.toml or requirements.txt. Define a CrawlConfig dataclass: seed URLs, allowed hosts, maximum pages, maximum depth, global concurrency and per-host delay. Done when: the package imports and a config can be created from a small YAML or JSON file.

2. 02

   Normalize and filter URLs

   ~2h

   Write normalize(url, base) that resolves relative links, lowercases the scheme and host, removes fragments (#...), removes default ports and sorts query parameters, so the same page always gets the same key. Write is_allowed(url) that accepts only http and https URLs on the allowed hosts and skips files such as images and PDFs. Done when: unit tests show that equivalent URLs normalize to the same string.

3. 03

   Crawl with bounded concurrency

   ~3h

   Build the crawler with an asyncio.Queue of (url, depth) and a fixed number of worker tasks sharing one httpx.AsyncClient with timeouts. A global asyncio.Semaphore caps concurrent requests, and each host has its own lock and last-request time to enforce the per-host delay. A seen set prevents visiting a URL twice. Stop at the page or depth limit. Done when: crawling a test site visits each page exactly once and never exceeds the limits.

4. 04

   Respect robots.txt

   ~1.5h

   Before crawling a host, download its robots.txt once, parse it with urllib.robotparser, and skip disallowed URLs. Use Crawl-delay when the site specifies one and it's larger than your per-host delay. Send a descriptive User-Agent that includes a contact URL. Done when: a disallowed path in a test robots.txt is never requested.

5. 05

   Store results and resume

   ~2.5h

   Save every fetched page to SQLite: url, status, content type, size, response time, title, depth, fetched time and outgoing links. Also save the queue of pending URLs. On restart, load the pending queue and the already-seen URLs so the crawl continues instead of starting over. Retry network errors and 5xx responses up to 3 times with backoff. Done when: stopping a crawl midway and starting it again finishes without refetching any page.

6. 06

   Add a FastAPI control service

   ~2h

   Create a FastAPI app with POST /crawls (start a crawl in the background from a config), GET /crawls/{id} (status, pages fetched, errors, pages per second) and POST /crawls/{id}/stop. Run the crawler as an asyncio task inside the app. Done when: a crawl can be started, followed and stopped through the API.

7. 07

   Report with pandas and test

   ~2h

   Add GET /crawls/{id}/report, which loads the results with pandas and returns: pages per status code, the slowest pages, broken links (and the pages linking to them) and pages per depth, plus a CSV export. Test the crawler against a local test site served with pytest fixtures, including robots.txt, a broken link and a slow page. Done when: pytest passes without touching the internet.

Resources

• Docsasyncio documentation
• DocsHTTPX async client
• Docsurllib.robotparser
• DocsFastAPI documentation
• Docspandas getting started