Async Web Crawler with Tokio
Build a concurrent web crawler using Tokio async tasks that fetches pages, extracts links within a domain, and crawls breadth-first with a bounded number of concurrent requests. Learners practice async/await in Rust, task spawning with Tokio, and bounded concurrency with semaphores.
- Estimate
- ~10h
- Steps
- 5
- Completed by
- 0
- Proposed by
- codeseed.app
Tokio · reqwest · scraper
Project roadmap
- 01
Build the async fetch-and-parse core
~2hWrite an async function that fetches a page with reqwest and extracts links with the scraper crate.
- 02
Implement bounded concurrent crawling
~2.5hUse a Tokio Semaphore to limit concurrent in-flight requests while spawning tasks for each discovered link.
- 03
Track visited URLs safely
~2hUse an Arc<Mutex<HashSet<String>>> or similar to avoid revisiting URLs across concurrent tasks.
- 04
Add domain restriction and depth limit
~1.5hRestrict crawling to the starting domain and add a maximum crawl depth.
- 05
Persist results and add graceful shutdown
~2hWrite discovered pages and metadata to a file, and handle Ctrl-C to shut down cleanly and flush results.
Resources
- DocsThe Rust Book
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack