Log Snapshot Diff

Project roadmap

1. 01

   Set up the script and sample logs

   ~1h

   Create logdiff.sh with #!/usr/bin/env bash and set -euo pipefail, and make it executable. Add samples/before.log and samples/after.log, each with about 50 lines in a format like 2024-05-01 10:00:01 ERROR Database timeout, where after.log has a few new messages, some removed and some more frequent. The script takes two arguments (the old and new snapshot) and prints usage if either is missing. Done when: ./logdiff.sh without arguments prints the usage.

2. 02

   Normalize the lines with awk

   ~1.5h

   Write a normalize function that uses awk to remove the date and time at the start of each line, and replaces numbers with N and hexadecimal ids with ID, so similar messages look identical (Timeout after 30s and Timeout after 45s both become Timeout after Ns). Save the normalized versions of both files to temporary files created with mktemp, and delete them with a trap on exit. Done when: normalizing the sample logs produces lines without timestamps or changing numbers.

3. 03

   Find new and removed lines

   ~1.5h

   Sort the unique normalized lines of each file with sort -u, then use comm to find lines only in the new snapshot (new messages) and only in the old one (messages that disappeared). Print them under the headings NEW and REMOVED, with a count for each. Done when: the report lists exactly the messages you added to and removed from the sample files.

4. 04

   Compare how often messages appear

   ~1.5h

   Count each normalized message in both files with sort | uniq -c, then use awk to join the two counts by message and print the messages whose count changed, as old count -> new count with the difference, sorted by the biggest increase first. Done when: a message that went from 2 to 10 appears at the top of the CHANGED section.

5. 05

   Fetch a snapshot from a URL

   ~1h

   If either argument starts with http:// or https://, download it first with curl -sS --fail --max-time 30 into a temporary file, and print a clear error if the download fails. Add a --level ERROR option that compares only lines of that level. Done when: comparing a local file with the same file served over HTTP (for example with python3 -m http.server) reports no differences.

6. 06

   Check it and test edge cases

   ~1h

   Run shellcheck and fix every warning. Test and document in the README: identical files (report says no changes), an empty file, a missing file (clear error) and a failed download. Exit with code 1 when there are differences and 0 when there aren't. Done when: shellcheck is clean, and each case behaves as documented.

Resources

• Docscomm(1) manual page
• Docsuniq(1) manual page
• DocsGNU Awk user's guide
• Docscurl manual
• DocsShellCheck