Process Resource Sentinel

Project roadmap

1. 01

   Design the sentinel

   ~2h

   Write DESIGN.md: the watch file format (one target per line: a name, where it runs (local, docker or ssh:host), how to find it (a process name, a pid file or a container name), and CPU and memory limits), the sample interval, how many samples in a row make a breach, the state file, and the breach log format (start, end, peak value and duration). Done when: DESIGN.md defines targets, sampling, breaches and logs with examples.

2. 02

   Sample local processes

   ~2h

   Write a function that finds processes by name with pgrep (summing all matching processes) and reads their CPU and memory with ps -o pid,pcpu,rss, using awk to add them up and convert memory to MB. Report a clear state when the process isn't running. Done when: sampling a running test process prints its CPU and memory, and a stopped one is reported as missing.

3. 03

   Sample containers and remote hosts

   ~2h

   For Docker targets, read CPU and memory with docker stats --no-stream --format and parse the output with awk (converting MiB and GiB). For remote targets, run the same ps command over ssh -o BatchMode=yes -o ConnectTimeout=5, sampling all targets on the same host in one connection. Treat a failed connection as unknown, not as a breach. Done when: container and remote targets report their usage.

4. 04

   Detect and record breaches

   ~2.5h

   Keep each target's state (normal or breaching, consecutive samples over the limit, breach start and peak) in a state file protected with flock. Start a breach only after N samples in a row over the limit, record its peak while it lasts, and write a log line when it ends with its duration. A process that disappears during a breach ends it with that reason. Done when: a test process using high CPU for a while creates one breach entry with the right duration and peak.

5. 05

   Keep it safe and light

   ~1.5h

   Run the sampling loop with a fixed interval that accounts for how long sampling took, and sample remote hosts in parallel with background jobs. Handle a corrupted state file (start fresh with a warning) and Ctrl+C (finish the current sample, save state and exit). Check the sentinel's own CPU use stays small. Done when: the sentinel runs for an hour with steady intervals and low overhead.

6. 06

   Report with awk

   ~1.5h

   Add a report command that uses awk on the breach log to show, per target, the number of breaches, total and longest breach time, and the highest peak, for a chosen period, plus the currently ongoing breaches. Optionally send breach start and end messages to a webhook with curl. Done when: the report matches the log.

7. 07

   Test it

   ~2h

   Write tests with bats-core using fake ps, docker and ssh commands placed first on the PATH that return prepared output, so breaches can be simulated precisely: short spikes ignored, breaches started and ended, missing processes, unreachable hosts, and report numbers. Run shellcheck. Done when: the tests pass, and shellcheck reports no warnings.

Resources

• Docsps(1) man page
• Docsproc(5) man page
• DocsGNU Awk user's guide
• Docsdocker stats reference
• Docsbats-core documentation