Website Change Sentinel

Project roadmap

1. 01

   Define what's watched

   ~1h

   Write README.md and a config format: one page per line with a name, URL, optional start and end markers (text that surrounds the part you care about), and optional ignore patterns (regular expressions for dates, counters and similar noise). Describe the snapshot folder and the report format. Done when: README.md describes the config and outputs with an example.

2. 02

   Download and extract text

   ~2h

   Download each page with curl (--fail, a timeout, a descriptive user agent, and following redirects), then turn the HTML into text: remove script and style blocks and tags with awk and sed, decode common entities (&, < and  ), keep only the part between the markers, and drop lines matching the ignore patterns. Collapse whitespace so formatting changes don't count. Done when: extracting a test page gives clean, stable text.

3. 03

   Keep snapshots

   ~1h

   Store the latest text for each page in a snapshot folder (named from the page name), and keep the last few previous versions with timestamps. On the first run, save the snapshot without reporting a change. Write snapshots to a temporary file and rename it. Done when: each run updates the snapshots, and old versions are kept up to the limit.

4. 04

   Report meaningful changes

   ~1.5h

   Compare the new text with the previous snapshot using diff -u, and when they differ, report the page name, URL, time and the added and removed lines. Ignore changes smaller than a configurable number of lines. Handle download failures separately (reporting that the page couldn't be fetched, without overwriting the snapshot). Done when: a change in a test page is reported with a clear diff, and failures don't create false changes.

5. 05

   Schedule it

   ~1.5h

   Add a crontab example, and package the tool as a small Docker image (for example based on alpine with curl, gawk and bash) that runs the checks in a loop at an interval, with the config and snapshot folder mounted as volumes; build and run it with the Docker CLI. Optionally send reports to a webhook with curl. Done when: the container checks pages on schedule and keeps its snapshots between restarts.

6. 06

   Test it

   ~1.5h

   Serve test pages from a local folder (for example with python3 -m http.server) and change them between runs. Write bats-core tests for text extraction, markers, ignore patterns, the first-run behavior, change detection and download failures. Run shellcheck. Done when: the tests pass, and shellcheck reports no warnings.

Resources

• Docscurl man page
• DocsGNU Awk user's guide
• DocsGNU diffutils manual
• DocsDocker: writing a Dockerfile
• Docsbats-core documentation