Multi-threaded Web Scraper and Data Aggregator

Project roadmap

1. 01

   Scrape one page

   ~2h

   Set up a cpanfile with LWP::UserAgent, HTML::TreeBuilder, WWW::RobotRules and Text::CSV. Write a function that downloads one product page (with a timeout and a descriptive user agent) and extracts the name, price, currency and availability using HTML::TreeBuilder's look_down, returning a hash. Save a few pages as test files and make the extraction work on them first. Done when: the function returns correct data for the saved test pages.

2. 02

   Add a thread pool

   ~2.5h

   Read a list of URLs from a file. Start N worker threads (a -threads option) that take URLs from a Thread::Queue, scrape them, and put results on a results queue. Finish cleanly by sending one end marker per worker, then joining the threads. Done when: 50 URLs are scraped with 4 threads, and every URL produces a result or an error.

3. 03

   Collect results safely

   ~2h

   Have one collector (the main thread) read the results queue and store the rows, so workers never write shared files. Keep shared counters (done, failed) as threads::shared variables and lock them when updating. Print progress as results come in. Done when: the counts are exact every run, and no result is lost or duplicated.

4. 04

   Be polite

   ~2h

   Fetch and respect robots.txt for each site with WWW::RobotRules, skipping disallowed pages. Enforce a minimum delay between requests to the same site across all threads, using a shared hash of last request times protected with lock. Back off and retry on 429 and 503 responses, honoring Retry-After. Done when: the logged request times show the delay is respected per site, even with 8 threads.

5. 05

   Export the report and failures

   ~1.5h

   Write all results to a CSV with Text::CSV, sorted by URL, with the columns URL, name, price, currency, availability and scraped at. Write failed URLs with their error to failed.txt, which can be used as the input for a retry run. Print a summary at the end. Done when: the CSV opens correctly in a spreadsheet, and a retry run from failed.txt works.

6. 06

   Handle bad pages

   ~1.5h

   Make extraction robust: pages where the price is missing or written differently (for example with a comma for decimals), products out of stock, redirects, and non-HTML responses. Record a clear reason for each failure instead of dying, and never let one bad page stop a worker thread. Done when: a list with broken and unusual pages completes, with every problem recorded.

7. 07

   Test it

   ~2h

   Write tests with Test::More: extraction on saved pages (normal, missing price, out of stock), price parsing, and the per-site delay logic. Add an integration test using a small local HTTP server (for example HTTP::Daemon) serving test pages, checking the scraper with several threads. Done when: prove -l t/ passes.

Resources

• Docsperlthrtut: Perl threads tutorial
• DocsThread::Queue on MetaCPAN
• DocsHTML::TreeBuilder on MetaCPAN
• DocsLWP::UserAgent on MetaCPAN
• DocsWWW::RobotRules on MetaCPAN
• DocsBooks to Scrape (a practice site)