Multi-threaded Web Crawler with Rate Limiting

Project roadmap

1. 01

   Build the single-threaded crawler core

   ~2h

   Write a function that fetches a page, extracts links and metadata with BeautifulSoup, and stays within a given domain.

2. 02

   Add a thread pool for concurrency

   ~2.5h

   Use concurrent.futures.ThreadPoolExecutor with a shared queue to crawl multiple pages in parallel.

3. 03

   Respect robots.txt and rate limits

   ~2h

   Parse robots.txt rules and add per-domain delay/throttling to avoid overloading target sites.

4. 04

   Deduplicate and store results

   ~1.5h

   Track visited URLs to avoid re-crawling and persist extracted metadata to a JSON or SQLite output.

5. 05

   Add CLI options and reporting

   ~1.5h

   Expose crawl depth, max pages, and delay as CLI flags, and print a summary report when finished.

Resources

• DocsPython Documentation