Server Health Monitoring Script with Alerts

Project roadmap

1. 01

   Collect system metrics

   ~1.5h

   Create a project with psutil and requests. Write a function that returns a snapshot: overall CPU percentage (measured over one second), memory used and available as a percentage, and disk usage for each mounted partition you choose (psutil.disk_partitions and disk_usage), plus the time. Write each snapshot as a line to a log file using logging with a RotatingFileHandler. Done when: running the script logs a correct snapshot compared with top and df.

2. 02

   Define thresholds

   ~1.5h

   Read limits from a config file (TOML with tomllib, or YAML): per metric, the warning limit and how many consecutive readings must be over it (for example CPU above 90% for 5 minutes). Keep each metric's state (normal or alerting, and the count of readings over the limit) in a small state file, so it survives restarts. Done when: a breach is only declared after the configured number of consecutive readings, and short spikes are ignored.

3. 03

   Send notifications

   ~2h

   Send an alert when a metric starts alerting and a recovery message when it returns to normal: by email with smtplib and email.message (with TLS, and the password from an environment variable) and/or to a Slack incoming webhook with requests (with a timeout). Include the host name, metric, value, limit and duration. Don't repeat an alert for the same breach, and log (but don't crash on) notification failures. Done when: a simulated breach sends one alert and one recovery message.

4. 04

   Add scheduling

   ~1h

   Support two ways to run: --loop, which checks every N seconds until stopped (handling Ctrl+C and SIGTERM cleanly), and --once for use from cron. Document a crontab line and a systemd service file in README.md. Done when: both modes work, and the loop stops cleanly.

5. 05

   Add a status command

   ~1h

   Add --status, which prints a snapshot as an aligned table with each metric's value, limit and state (OK or ALERTING since a given time), and --json for other tools. Done when: the status command shows current readings and states.

6. 06

   Test it

   ~1.5h

   Write pytest tests with psutil and the notification senders replaced by mocks: breach detection with consecutive readings, no alert for spikes, a single alert per breach, recovery messages, state surviving a restart, and config validation. Done when: pytest passes.

Resources

• Docspsutil documentation
• DocsPython smtplib
• DocsPython logging: RotatingFileHandler
• DocsSlack: sending messages using incoming webhooks
• DocsRequests: quickstart