Web Scraper for Book Prices with Nokogiri
Build a Ruby scraper that collects book titles, prices and availability from a book listing website, follows the pages of results, and saves each run to a CSV file with the time it ran, so it can report which books changed price since last time. It's built politely: it waits between requests, identifies itself, respects robots.txt, and keeps going when a page is missing an element. Practice on books.toscrape.com, a site made for learning scraping, and only scrape other sites if their terms allow it. You'll learn HTTP requests with HTTParty, parsing HTML with Nokogiri and CSS selectors, pagination, handling messy data, saving and comparing results with CSV, and testing with saved pages.
- Estimate
- ~8.5h
- Steps
- 6
- Completed by
- 0
- Proposed by
- codeseed.app
Nokogiri · HTTParty · CSV
Project roadmap
- 01
Fetch and parse one page
~1.5hCreate a project with a Gemfile (httparty, nokogiri and csv). Fetch the first listing page of books.toscrape.com with HTTParty, with a timeout and a descriptive User-Agent header, check the response code, and parse it with Nokogiri. Use CSS selectors to extract each book's title, price (parsed into a number, with its currency) and availability. Save a copy of the page for tests. Done when: the script prints the 20 books on the first page with correct prices.
- 02
Follow the pages
~1.5hFind the next page link, resolve it against the current URL, and keep going until there's no next link or a --max-pages limit is reached. Wait 1 to 2 seconds between requests. Collect all books in one list, removing duplicates by URL. Done when: all 1,000 books on the practice site are collected across 50 pages.
- 03
Save runs to CSV
~1hAppend each run's results to books.csv with the columns scraped at, title, URL, price, currency and availability, writing the header only when creating the file. Use the csv library so titles with commas are quoted correctly. Done when: each run adds its rows, and the file opens correctly in a spreadsheet.
- 04
Detect price changes
~1.5hLoad the previous run's rows from the CSV, match books by URL, and print a report of books whose price changed (old price, new price and the difference), books that appeared or disappeared, and changes in availability. Done when: editing a price in the CSV by hand and running again reports that change.
- 05
Be polite and robust
~1.5hRead robots.txt before scraping and skip disallowed paths, retry failed requests once after a pause, and back off on 429 or 503 responses. When an element is missing from a book, record it as blank with a warning instead of crashing, and summarize warnings at the end. Done when: a page with missing elements and a failed request are handled without stopping the run.
- 06
Test it
~1.5hWrite RSpec or Minitest tests using the saved pages (and WebMock to fake requests): extracting books, price parsing, following pages, the missing-element case, and the price change report from two small CSV files. Done when: the tests pass without network access.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack