carlos / platform Public

Filed at Paul's request, 2026-10-07, from the region HA work (platform-99). It is deliberately not started; it would be region HA's spec 5, after specs 2 to 4.

Today

Region HA spec 1 (merged 3e12133) recovers within a region only. When every box in a region dies:

1. The Auto Scaling group launches a replacement.
2. A box in another region confirms the newcomer's group membership (H17).
3. The newcomer fences the dead boxes on a verified terminated read (H19).
4. It relocates the instances onto itself, restoring each from that region's data bucket.

Recovery takes minutes. Two boxes per region is the recommended minimum, so that a survivor exists and recovery skips the launch.

None of that helps when the AWS region itself is down. Its bucket is unreachable, and no other region holds a copy of the instance's data.

What cross-region failover would need

• Per-app opt-in. Paul's residency rulings (nearest-region R1 and R2) make an instance's region part of its record. Copying tenant data out of that region is the owner's choice, and the console must say what it means.
• Continuous replication into a second region's bucket. That covers the database (Litestream), the files manifest and objects, and the wake state (timers, NextDue, Replication). The read replicas (#162) and moves' cross-region staging are the nearest existing machinery.
• A promotion step. It is a move whose source cannot respond. It needs the claim's lost-region evidence (what proves a whole region is gone, given that terminated reads from that region are unavailable), and fencing so the old region cannot resume writing when it comes back. That last part is the hard part.
• Routing. Steering has to move the app's hosts to the second region (latency steering, #169), and the return path has to be considered.
• A stated loss window. Writes not yet replicated across regions are lost, as H3 says for in-region failover, but the window is wider.

Open questions for Paul when this starts

• Automatic or operator-triggered promotion? Region-wide outages are rare, so a person may be the right gate.
• Which region is the standby: chosen per app, or the nearest eligible region?
• Is this a paid-plan feature (Carloku), like H6's file durability?