Scheduled Webpage Change Monitor
Build a Go tool that watches web pages for changes, such as a product's price, a job listing or a documentation page, and notifies you by email or webhook when the part you care about changes. On a schedule, it downloads each page, extracts only the relevant section with a CSS selector using goquery (so ads and timestamps don't trigger false alarms), compares it with the previous snapshot, and sends a readable diff when there's a meaningful change. Each page has its own interval, and checks run concurrently. Only watch pages whose terms allow automated access, and keep intervals reasonable. You'll learn HTTP requests with timeouts, HTML parsing with CSS selectors, text diffing, time.Ticker and goroutines, sending email with net/smtp, and testing with a fake server.
- Estimate
- ~10h
- Steps
- 7
- Completed by
- 0
- Proposed by
- codeseed.app
goquery · time · SMTP/webhook
Project roadmap
- 01
Fetch and extract content
~1.5hWrite a fetcher using an http.Client with a timeout and a descriptive user agent, and use goquery to select the configured part of the page (such as #price or .job-list), turning it into normalized text (collapsed whitespace, and optionally ignoring elements matched by an exclude selector, such as timestamps). Report when the selector matches nothing, which often means the page layout changed. Done when: extracting a section of a test page gives clean, stable text.
- 02
Store snapshots
~1hSave each extracted text as a snapshot file in a folder per watched page (named from a hash of the URL and selector), with a timestamp, keeping the last N snapshots. Write snapshots to a temporary file and rename it. Done when: repeated checks create snapshots, and old ones are removed.
- 03
Diff the snapshots
~2hCompare the new text with the previous snapshot line by line using a diff library (such as github.com/sergi/go-diff or a simple longest-common-subsequence diff you write), ignoring changes smaller than a configurable threshold. Produce a readable summary of added and removed lines. Done when: a change in the test page produces a clear diff, and identical content produces none.
- 04
Schedule checks
~1.5hRun one goroutine per watched page with its own time.Ticker (plus a small random offset so checks don't all happen at once), limit how many fetches run at the same time with a semaphore, and stop everything cleanly on Ctrl+C with a context. Log each check's result. Done when: several pages are checked on their own intervals, and the tool exits cleanly.
- 05
Send notifications
~1.5hWhen a change is detected, send an email with net/smtp (with TLS and credentials from environment variables) or a POST to a webhook with the page name, URL, time and diff. Retry a failed notification once and log failures without stopping the checks. Also notify once when a selector stops matching. Done when: a change sends one notification with the diff.
- 06
Support many pages
~1hRead the watched pages from a YAML config (name, URL, selector, exclude selector, interval with a minimum such as 5 minutes, and notification channel), validating it at startup, and reload it on SIGHUP. Done when: pages can be added or changed in the config without restarting.
- 07
Test it
~1.5hWrite tests with httptest.Server serving pages whose content you change between requests: extraction and normalization, ignoring excluded elements, diffs and thresholds, a missing selector, and notifications sent to a fake webhook. Run with -race. Done when: go test -race ./... passes.
Resources
Ready to build this?
Get a GitHub repo and start building. Your AI reviewer checks each step as you go.
Tech stack