amadan
Paul's ruling, 2026-10-14: option (b). We accept this as a known hazard on dev only, until dev's claim is on. Flagship runs one box per region, so it isn't affected. There is no code change. Moves is adding a line to the rehearsal runbook's P-0. It covers any app with an instance in dev's home region, and says a wake that switches between carlos-dev-ie and carlos-dev-ie-b can serve a stale database while the claim is off. Rehearsal apps are woken on one box only, or the run waits until the claim is on. If region HA's design means this hazard could ever reach a production region with two boxes and the claim off, please reopen it here.
Dev's two home boxes lose writes on cross-box wakes (claim off): the one-box-per-region assumption, live open
Observed on dev, 2026-10-13, with the standing wakecheck app (bab, wakecheck.dev.carlosframework.com, no region). Dev has two home boxes with no CARLOS_REGION: carlos-dev-ie (sophie-knitting) and carlos-dev-ie-b (leo-climbing). Both take every unregioned record.
The sequence:
- The app woke on sophie and wrote up to row 100 (15:41).
- It slept, then woke on leo at 15:57. Leo restored from S3 and wrote 3 rows (101).
- It slept on leo at 16:12:59, then woke on sophie at 16:13:30. Sophie started from its own stale local DB, with no restore, so leo's 3 rows are missing. Litestream logged "detected database behind replica" but the rows were still absent a minute later.
Cause: this is by design today. In internal/activator/activator.go around line 2038, a local copy is treated as the warm cache "only while a region has one box". Claim ruling R6 (region HA spec 1, H13) ends that: once the claim is on, every admitted writer purges and restores. With the claim off, a second box in a region breaks the assumption.
Impact:
- Flagship has one box per region, so it is unaffected.
- On dev, any instance-backed app in the home region can lose writes whenever its wake changes box. Today that is only
wakecheck; helloworld is an external route. - The rehearsal plan keeps
carlos-dev-ie-bas a second home box. Until dev's claim is on, rehearsal apps there are exposed.
Options:
- (a) Keep dev's second home box out of unregioned records until the claim is on. That needs a scope knob or a region for ie-b, and the rehearsal plan fixes ie-b's shape, so this one is for region HA and Paul.
- (b) Accept it as a known dev-only hazard until the claim is on, and say so in the rehearsal runbook's P-0.
- (c) Claim off, more than one box with the same region: the agent refuses to start, or warns loudly, so the configuration can't go unnoticed.
Region HA owns the two-box model, so this is for them to rule. Moves will add a line to the rehearsal P-0 either way.