Price Watch Scraper

Project roadmap

1. 01

   Set up the project

   ~1h

   Create a virtual environment and install requests, beautifulsoup4, pandas and Pillow, listing them in requirements.txt. Create products.txt with 5 product URLs from books.toscrape.com, one per line, and a script price_watch.py with the subcommands check, history and chart (use argparse). Done when: python price_watch.py --help lists the three subcommands.

2. 02

   Download and parse a product page

   ~2h

   Write fetch_product(url) that downloads the page with requests (10-second timeout, a descriptive User-Agent) and uses BeautifulSoup to extract the product name and price text. Keep the CSS selectors in constants at the top of the file, so they're easy to update when the site changes. Raise a clear error when the price isn't found. Done when: fetching one product URL returns its name and price text.

3. 03

   Turn price text into numbers

   ~1.5h

   Write parse_price(text) that turns strings like £51.77, $1,299.00 or 12,50 € into a Decimal amount and a currency code. Return an error for text with no number in it. Write unit tests for at least 6 different formats. Done when: the tests for parse_price pass.

4. 04

   Record price changes

   ~2h

   The check command fetches every URL in products.txt, waiting 1 second between requests, and appends a row (timestamp, url, name, price, currency) to prices.csv only when the price differs from the last recorded price for that URL. Load and save the file with pandas. Print a line for each product: unchanged, new, or changed with the old and new price. Done when: running check twice without changes adds no new rows.

5. 05

   Handle failures

   ~1.5h

   A product that fails (timeout, 404, missing price) shouldn't stop the others: retry network errors twice with a short pause, then report the failure and continue. Never record a price when parsing failed. Exit with code 1 if any product failed, and print a summary at the end. Done when: a list containing a broken URL still checks the other products and reports the failure.

6. 06

   Show the history and a chart

   ~2h

   The history command prints each product's price history with pandas: first and latest price, lowest and highest price, and the change in percent. The chart command draws a simple line chart of one product's price over time as a PNG with Pillow's ImageDraw: axes, the price line, and the min and max values as labels. Done when: history prints the table and chart creates a readable PNG.

7. 07

   Test without the internet

   ~1.5h

   Save two copies of a product page as HTML fixtures with different prices. Write pytest tests that parse the fixtures (no network), check change detection with a temporary prices.csv, and mock requests.get to simulate a timeout. Done when: pytest passes with the network turned off.

Resources

• DocsBooks to Scrape (practice site)
• DocsRequests documentation
• DocsBeautiful Soup documentation
• Docspandas getting started
• DocsPillow ImageDraw module