amadan
Region HA: a failed drain probe in a two-box region leaves the host offline until an operator acts open
Paul accepted this behaviour for the first version on 2026-10-08, with a follow-up. This discussion is that follow-up.
What happens today (region HA plan 1b-next, ruling 26):
- In a two-box region, a planned drain moves a host from box A to box B.
- If B fails the readiness probe, there is no third box to try.
- A has already released, so the host is not served anywhere. The operation goes
stuck, and the host stays offline until an operator either stops B and replays the operation, or returns the host to A.
Why it is fail-safe: the source has released and its data is flushed. Nothing is lost, only availability.
Options for the follow-up:
- Return to the source automatically. If the probe fails and the source box is still healthy and eligible, run the rollback path back onto A without an operator. This needs a ruling on whether the source's released state can be re-admitted without an operator.
- Probe before release. Warm B and probe it while A is still serving, and release A only after B passes. That costs a second copy during the drain, but it removes this window for planned drains.
- Scale out for the drain. Have the autoscaler add a temporary third box for a drain in a two-box region, then remove it.
Option 2 is probably the cleanest for planned drains. Failover after a box dies cannot use it, because the source is already gone.
Spec: docs/superpowers/specs/2026-10-04-region-ha-box-placement-design.md. Plan: docs/superpowers/plans/2026-10-07-region-ha-1b-next-relocation.md (ruling 26).