Container Health-Check Daemon

Project roadmap

1. 01

   Write the polling loop

   ~2h

   Write healthd.sh with set -euo pipefail that every N seconds lists the containers to watch (those with a label such as healthd.enable=true, found with docker ps --filter), runs docker inspect on them, and uses jq to extract each one's name, state, health status (starting, healthy or unhealthy) and the last health check output. Handle containers without a health check. Done when: the loop prints each watched container's health every interval.

2. 02

   Apply the restart policy

   ~2h

   Read the policy from a config file or container labels: restart after the container has been unhealthy for M checks in a row, wait a backoff delay that doubles after each restart, and allow at most K restarts per hour. Keep each container's restart times and state in memory (and in the state file). Restart with docker restart, logging the reason and the last health output. Done when: a test container that turns unhealthy is restarted according to the policy, and stops being restarted after its budget is used up.

3. 03

   Handle signals and shut down cleanly

   ~1h

   Trap SIGTERM and SIGINT to finish the current check cycle, write the final status file, and exit; make the sleep between cycles interruptible (sleep in the background and wait on it) so shutdown is immediate. Trap SIGHUP to reload the config. Done when: stopping the daemon exits within a second, and SIGHUP reloads the config.

4. 04

   Write a status file

   ~1h

   After each cycle, write a JSON status file (built with jq so it's always valid) with each container's health, consecutive failures, restarts in the last hour, next allowed restart time and whether it's given up, plus the daemon's last cycle time. Write it to a temporary file and rename it, so readers never see half a file. Done when: the status file is always valid JSON and reflects the current state.

5. 05

   Run as a systemd service

   ~1h

   Write a systemd unit file with Restart=on-failure, a dependency on docker.service, and output going to the journal, and document installing, starting, and reading logs with journalctl -u healthd. Run it as a user in the docker group rather than root, and explain the security meaning of that. Done when: the daemon starts at boot and its logs appear in the journal.

6. 06

   Alert on repeated failures

   ~1h

   When a container uses up its restart budget, send one alert (a webhook with curl, or an email) with the container, the number of restarts and the last health check output, and a recovery message when it becomes healthy again. Don't repeat the alert while it stays in the same state. Done when: a container that keeps failing sends one alert, and one recovery message when fixed.

7. 07

   Test it

   ~2h

   Create test containers whose health check you can make fail or pass on demand (for example by creating or deleting a file inside the container). Write bats-core tests for the policy logic with a fake docker command, and an end-to-end test with real containers checking restarts, backoff, the budget, the status file and alerts. Run shellcheck. Done when: the tests pass, and shellcheck reports no warnings.

Resources

• DocsDockerfile reference: HEALTHCHECK
• Docsdocker inspect reference
• Docsjq manual
• Docssystemd.service man page
• Docsbats-core documentation