codeseed
Sign in
Explore Projects
PerladvancedbackendAI generated

Multi-threaded Web Scraper and Data Aggregator

Build a concurrent web scraper that crawls a set of product pages across multiple threads, extracts pricing data with regex/HTML parsing, and aggregates results into a consolidated CSV report while respecting rate limits. Learners practice Perl threading, concurrency-safe data structures, and polite scraping practices.

Estimate
~9.5h
Steps
5
Completed by
0
Proposed by
codeseed.app

threads · LWP::UserAgent · HTML::TreeBuilder

Sign in to claim this project

Project roadmap

  1. 01

    Build the single-threaded scraper

    ~2h

    Implement page fetching and HTML::TreeBuilder-based extraction for one product page.

  2. 02

    Add a thread pool

    ~2.5h

    Use Perl's threads module with a shared work queue to fetch multiple pages concurrently.

  3. 03

    Add thread-safe aggregation

    ~2h

    Use threads::shared structures to safely collect results from worker threads.

  4. 04

    Implement rate limiting

    ~1.5h

    Add per-domain request throttling to avoid overloading target servers.

  5. 05

    Export consolidated results

    ~1.5h

    Write aggregated data to a CSV report and log any failed fetches for retry.

Resources

Ready to build this?

Get a GitHub repo and start building. Your AI reviewer checks each step as you go.

~9.5h · 5 steps

Tech stack

threadsLWP::UserAgentHTML::TreeBuilder