Multi-threaded Web Crawler with Rate Limiting

Project roadmap

1. 01

   Fetch and parse one page

   ~1.5h

   Create a project with requests, beautifulsoup4 and pytest. Write fetch_page(url) that downloads a page with a requests.Session (10-second timeout, a descriptive User-Agent), and returns the status code, response time, title, word count of the visible text, and all links resolved to absolute URLs with urljoin. Only parse HTML responses. Done when: calling it on a real page prints its details and links.

2. 02

   Normalize URLs and stay in scope

   ~1.5h

   Write normalize(url) that removes #fragments, lowercases the scheme and host, removes default ports and trailing slashes consistently, and sorts query parameters. Write in_scope(url) that accepts only http and https links on the starting domain (and its subdomains with --include-subdomains), skipping links to files such as .pdf, .zip and images. Done when: unit tests show that equivalent URLs normalize to the same string.

3. 03

   Crawl with a thread pool

   ~2.5h

   Crawl breadth-first: keep a frontier of (url, depth), a visited set protected by a threading.Lock, and a ThreadPoolExecutor with --workers threads (default 5). Submit fetch jobs, and as they complete with as_completed, add new in-scope, unvisited links to the frontier. Stop at --max-pages or --max-depth. Done when: crawling a small site visits each page once and respects both limits.

4. 04

   Respect robots.txt and rate limits

   ~2h

   Load robots.txt once per host with urllib.robotparser and skip disallowed URLs. Enforce a minimum delay between requests to the same host (--delay, default 1 second, or the site's Crawl-delay if larger) using a lock and the time of the last request per host, so threads take turns. Retry 429 and 5xx responses up to 2 times with backoff. Done when: a disallowed path is never fetched, and requests to one host are never closer together than the delay.

5. 05

   Save results to SQLite and resume

   ~2h

   Store each page (url, status, response time, title, word count, depth, fetched time) and each link (from, to) in SQLite. Since sqlite3 connections shouldn't be shared between threads, send results to the main thread, which does all the writing. With --resume, reload visited URLs and unfinished frontier entries from the database and continue. Done when: stopping a crawl with Ctrl+C and running again with --resume continues without refetching pages.

6. 06

   Report broken links and slow pages

   ~1.5h

   At the end, print a summary: pages crawled, pages by status code, the average and slowest response times, broken links (4xx and 5xx) with the pages that link to them, and pages with very little text. Add --report report.json to save it. Done when: crawling a test site with a known broken link lists it along with the page that contains it.

7. 07

   Test against a local site

   ~2h

   Write pytest tests that serve a small fake website locally (with http.server in a thread, or the responses library) with several pages, a loop of links, a broken link, a robots.txt that disallows one path, and a slow page. Test the page limit, the depth limit, deduplication, robots.txt and the broken link report. Done when: pytest passes without using the internet.

Resources

• Docsconcurrent.futures documentation
• DocsRequests: advanced usage (sessions)
• DocsBeautiful Soup documentation
• Docsurllib.robotparser
• Docssqlite3 module