Service Health Loop

Project roadmap

1. 01

   Define the checks and the log

   ~1.5h

   Write README.md: the targets file format (one per line: a name, a type http, docker or ssh, the target such as a URL, container name or host and command, and an optional expected status and timeout), the states (up, down and degraded for slow responses), how many failures in a row mean down, and the incident log format (time, name, old state, new state, reason, and the duration of the incident when it ends). Done when: README.md defines targets, states and the log format with examples.

2. 02

   Run the checks

   ~2h

   Write health.sh with set -euo pipefail. For http targets, use curl with --max-time, capturing the status code and total time with -w. For docker targets, use docker inspect to read the container's state and health status. For ssh targets, run a command with ssh -o BatchMode=yes -o ConnectTimeout=5 and check its exit code. Each check returns a state and a reason. Done when: running the script once prints the current state of every target.

3. 03

   Record state changes

   ~2h

   Store each target's current state, failure count and since when in a state file between runs. Only change a target to down after N failed checks in a row, and back to up after one success. When the state changes, append a line to the incident log, including the incident duration on recovery. Protect the state file with flock so overlapping runs can't corrupt it. Done when: stopping a test service creates one down entry after N checks, and restarting it creates one recovery entry with the duration.

4. 04

   Handle failures of the checker itself

   ~1.5h

   Handle a malformed targets file (report the line and skip it), a missing Docker daemon or ssh command, timeouts, and a corrupted state file (start fresh with a warning). Run checks in parallel with background jobs and wait, so one slow target doesn't delay the rest. Done when: none of these problems stops the other checks.

5. 05

   Report with awk

   ~1.5h

   Add a report command that uses awk on the incident log to show, for a chosen period, each target's number of incidents, total downtime, longest incident and availability percentage, as an aligned table. Done when: the report's numbers match the incident log.

6. 06

   Schedule it

   ~1h

   Add a loop mode (run every N seconds until stopped) and document a cron or systemd timer setup in README.md. Optionally send a notification to a webhook on state changes with curl. Done when: the checker runs unattended and logs incidents over time.

7. 07

   Test it

   ~2h

   Write tests with bats-core using fake targets: a local HTTP server (for example python3 -m http.server) you can stop and start, a test container, and a fake ssh command on the PATH. Check the down threshold, recovery with duration, parallel checks, report numbers and bad targets lines. Run shellcheck. Done when: the tests pass, and shellcheck reports no warnings.

Resources

• Docscurl man page (--max-time and --write-out)
• DocsGNU Awk user's guide
• Docsdocker inspect reference
• Docsssh(1) man page
• Docsbats-core documentation