Multi-Host Deploy Runner

Project roadmap

1. 01

   Design the deployment

   ~2h

   Write DESIGN.md: the inventory file (hosts with groups and optional user and port), the plan file (ordered steps, each a command with options: run on all hosts, once, or only on a group; a timeout; and whether a failure is fatal), the wave size, the health check URL per host, and what rollback does. Describe what happens when a step or health check fails in the middle of a wave. Done when: DESIGN.md defines the files, waves and failure rules.

2. 02

   Run steps on one host

   ~2h

   Write a function that runs a plan's steps on one host in order with ssh -o BatchMode=yes -o ConnectTimeout=10, applying each step's timeout, writing the output of each step with timestamps to logs/RUN_ID/HOST.log, and stopping at the first fatal failure. Record the result of each step (ok, failed or skipped, with its duration) in a results file. Done when: running a plan against one test host logs each step and stops on failure.

3. 03

   Deploy in waves

   ~2.5h

   Split the hosts into waves (such as 1 canary host first, then groups of N), run hosts in a wave in parallel as background jobs, wait for them all, and continue to the next wave only if every host succeeded. Run once steps on the first host only, before the rest. Stop the whole rollout on the first failed wave. Done when: a failure on the canary host stops the deployment before any other host is touched.

4. 04

   Check health

   ~1.5h

   After each host's steps, poll its health URL with curl --fail --max-time until it answers correctly or a deadline passes, and treat a failed health check as a failed host. Optionally check the version reported by the service matches the one being deployed. Done when: a host whose service doesn't come back is marked failed and stops the rollout.

5. 05

   Roll back

   ~2h

   Add a rollback plan that runs on hosts that were already updated in the failed run (read from the results file), newest first, and a --rollback RUN_ID command to run it later. Keep a lock (flock) so two deployments can never run at the same time, and a --dry-run that prints what would happen on each host. Done when: a failed deployment can be rolled back on exactly the hosts that changed.

6. 06

   Summarize the run

   ~1h

   At the end, use awk on the results file to print a summary table: each host's steps, status and duration, the wave it was in, and the total time, plus the paths to failing hosts' logs. Exit with a non-zero code when anything failed. Done when: the summary makes the result of the run clear at a glance.

7. 07

   Test it

   ~2.5h

   Test with several SSH servers in Docker containers (for example linuxserver/openssh-server) running a small fake service with a health endpoint, including one that fails on purpose. Check waves, the canary stop, once steps, health failures, rollback and dry runs. Add bats-core tests for the plan and inventory parsing, and run shellcheck. Done when: the tests pass, and shellcheck reports no warnings.

Resources

• Docsssh(1) man page
• Docscurl man page
• DocsGNU Awk user's guide
• ArticleMartin Fowler: canary release
• Docsbats-core documentation