| 1 | # Region HA, spec 1: placing an instance on a box |
| 2 | |
| 3 | Design, 2026-10-06, **rev 5.4**. DRAFT. Plan 1a is built and merged |
| 4 | (aadb860); nothing else here is built. |
| 5 | |
| 6 | **Rev 5.4** answers Astra's fourth review of rev 5 |
| 7 | (`CODEX-FINDINGS-REGION-HA-REV5-4.md`, **B4-n**), which found no |
| 8 | two-writer or data-loss path that depends on disk evidence. It removes |
| 9 | the last runtime dependency on volume recording: the multi-box gate no |
| 10 | longer checks it at box start, and recorder failure only alerts and |
| 11 | keeps a dead box's chores open (B4-1). It also brings three summaries |
| 12 | into line with H19 (B4-2). §23 maps both. |
| 13 | |
| 14 | **Rev 5.3** records Paul's ruling **H19** (2026-10-06), which supersedes |
| 15 | H15 and the volume half of H10, and answers Astra's third review of rev |
| 16 | 5 (`CODEX-FINDINGS-REGION-HA-REV5-3.md`, **B3-n**) in §22. Rounds 1 to 3 |
| 17 | kept finding new ways a disk could outlive its box behind the volume |
| 18 | guard. H19 takes the disk out of the failover path: a terminated |
| 19 | instance can never run again, so the execution fence rests on the |
| 20 | verified `terminated` read alone. Disk evidence decides only whether |
| 21 | the dead box's `park-local` chores are discharged automatically; when |
| 22 | it cannot be proven, they stay open for an operator, and recovery waits |
| 23 | on nothing. The SCP, holds, the change role, the guard attestation and |
| 24 | the service-role audit are removed. Most of round 3 is moot under H19; |
| 25 | §22 says which, and why. |
| 26 | |
| 27 | **Rev 5.2** answers Astra's second review (`CODEX-FINDINGS-REGION-HA-REV5-2.md`, |
| 28 | cited **B2-n**): four P1 and two P2, all accepted; §21 maps each, with |
| 29 | the round 1 items it marked partly resolved. The volume guard now |
| 30 | covers root-volume replacement, snapshots and images (B2-1), and Auto |
| 31 | Scaling's own service-linked role, through the APIs that start its work |
| 32 | (B2-3). A hold is closed by revoking the change role's sessions and |
| 33 | waiting for in-flight disk tasks before the validating record (B2-2). |
| 34 | The files manifest is pinned from the newest admitted generation, not |
| 35 | from the database baseline's (B2-4). The local install receipt is bound |
| 36 | to the installation stamp, and a box that has admitted a writer never |
| 37 | reinstalls (B2-5). H17's confirmer may treat an enrolled box as gone on |
| 38 | a positive `terminated` read (B2-6). |
| 39 | |
| 40 | **Rev 5.1** answers Astra's first review of rev 5 |
| 41 | (`CODEX-FINDINGS-REGION-HA-REV5-1.md`, cited **B1-n**): five P1, three P2 |
| 42 | and one P3, all accepted; §20 maps each. A restarted probe now restores |
| 43 | from its real predecessor, and `released` coverage is bound to one |
| 44 | session (B1-1). A replacement target must make its own local install |
| 45 | before probing (B1-2). An operator's volume change first suspends |
| 46 | automatic disk evidence, and a volume that leaves a running box keeps it |
| 47 | suspended until accounted for (B1-3). The volume guard protects its own |
| 48 | tag, needs an attested Organizations policy, and without one there is no |
| 49 | automatic fence (B1-4). A lost or unreadable execution-lease state keeps |
| 50 | updates out (B1-5). A source lost in `starting` turns the drain into a |
| 51 | failover (B1-6). A console bound to a fenced box leaves the census |
| 52 | (B1-7). Code citations are refreshed (B1-9). The two points this raised |
| 53 | for Paul were ruled the same day: an enrolled box in another region |
| 54 | confirms the first box of an emptied region (H17), and a box-started |
| 55 | relocation accepts a console's newest report when its box's update |
| 56 | guard is armed (H18). §11 is empty again. |
| 57 | |
| 58 | **Rev 5** restates the design against the host lifecycle claim as it now |
| 59 | stands, rev 6.25 (413b12cb on `design-replica-ownership`), with its |
| 60 | library on main (`internal/hostclaim`, `internal/hostguard`, |
| 61 | `internal/stopevidence`, `internal/park`, `internal/genselect`), and |
| 62 | records Paul's rulings H10 to H18 of 2026-10-06. Rev 4.2 was written |
| 63 | against claim rev 6.3 and asked for changes K1 to K11. Those sessions |
| 64 | never saw the request, and the claim moved on: box references are |
| 65 | opaque, `control/host-actors.json` is the one roster, every writer is |
| 66 | admitted by `AdmitWriter`, every admission is cold until certificates |
| 67 | work, a prepared generation is opened once, and regional work is done |
| 68 | by any enrolled box in the region. So rev 5 drops spec 1's own box |
| 69 | records as authority (H11), the trusted local copy (H13), and its own |
| 70 | `holder.generation` install CAS. Relocation (§5.8) keeps every property |
| 71 | it had, and installs through the claim's prepared-generation and |
| 72 | `AdmitWriter` rules instead. The termination fence stays automatic, now |
| 73 | also requiring every EBS volume to be deleted with the instance (H10). |
| 74 | A dev test showed a terminated instance's describe lists no volumes, so |
| 75 | that proof comes from a record taken while the box runs, kept true by |
| 76 | IAM (H15). A relocation is started without a console: the boxes, not a |
| 77 | roster reservation, keep it apart from executable updates (H14, §5.10). |
| 78 | §9's K-table is replaced by requests L1 to L20 against claim rev 6.25 |
| 79 | and moves rev 7.17. §19 says what of plan 1a is superseded. The three |
| 80 | points rev 5 first left open were ruled the same day as H14 to H16 |
| 81 | (§11). |
| 82 | |
| 83 | Rev 4.2 answered Astra round 5 (`CODEX-FINDINGS-REGION-HA-5.md`, A5-1), |
| 84 | and Astra round 6 (`CODEX-FINDINGS-REGION-HA-6.md`) found it ready for |
| 85 | plan. Rev 4.1 answers Astra round 4 (`CODEX-FINDINGS-REGION-HA-4.md`, |
| 86 | **A4-n**). Rev 4 answers Astra round 3 (`CODEX-FINDINGS-REGION-HA-3.md`, **A3-n**, |
| 87 | plus the A2-5, A2-7 and A1-1 residuals). Rev 3 answered Astra round 2 (`CODEX-FINDINGS-REGION-HA-2.md`, cited |
| 88 | **A2-n**, plus the round 1 findings it marked partly resolved). Rev 2 |
| 89 | answered round 1 (`CODEX-FINDINGS-REGION-HA-1.md`, **A1-n**). Every |
| 90 | finding is accepted; §13 maps each to its section. Where rev 5 changes a |
| 91 | mechanism a disposition names, the disposition's property still holds, |
| 92 | and the section it points to says how. |
| 93 | |
| 94 | This is the first of four specs that give each region high availability |
| 95 | and autoscaling: |
| 96 | |
| 97 | 1. **Box placement** (this spec). With several boxes in a region, the |
| 98 | host lifecycle claim names the one box that runs each instance, and |
| 99 | every box, edge and activator follows the claim. |
| 100 | 2. **Placement policy.** Which eligible box a new or displaced instance |
| 101 | goes to. |
| 102 | 3. **Failover.** A dead box's instances are re-placed onto the region's |
| 103 | survivors. |
| 104 | 4. **Autoscaling.** A region's boxes are an Auto Scaling group that grows |
| 105 | on demand and shrinks by draining. |
| 106 | |
| 107 | It builds on two specs still in review, and asks them for changes (§9): |
| 108 | |
| 109 | - the host lifecycle claim, **rev 6.25** (`design-replica-ownership`, |
| 110 | 413b12cb), cited as **claim §n**. Its sections marked PROPOSED (§5.9 |
| 111 | P5-D1 to P5-D6, §6.6 P6-D1 to P6-D8, §7.3 D2 to D6, §10.1 D1 and D7) |
| 112 | are cited as proposals, not as settled text. Rev 6.26 (98b45332) |
| 113 | changes only the staging scope of a delete's park (R17-2) and touches |
| 114 | nothing here; |
| 115 | - move-any-app (`design-moves-any-region`), **rev 7.17** (39904c84), |
| 116 | cited as **moves §n**. Rev 7.18 (a4711728) changes no rule. |
| 117 | |
| 118 | It also builds on nearest-region (on main since 3f4a9bfd), whose box |
| 119 | evidence in the edges file is how a console knows which enrolled boxes |
| 120 | are alive, and on #212 option B for box references. |
| 121 | |
| 122 | ## 0. Rulings (Paul, 2026-10-04 and 2026-10-06) |
| 123 | |
| 124 | | # | Ruling | |
| 125 | |---|---| |
| 126 | | H1 | **The proof that a box is dead is that AWS terminated it.** Each region's boxes run in an Auto Scaling group with health checks. When the group terminates an unhealthy box, a region placer that has only read-only AWS rights sees `terminated` and re-places the box's instances. The console still holds no EC2 rights. This replaces M16 / claim §7.2's operator-only declare-lost **for terminated boxes only**; a `stopped` box stays manual. It also replaces fleets D7's "no automatic reaper" for platform boxes. Lease expiry is still never proof (CC5). | |
| 127 | | H2 | **Shape the claim now.** The HA specs cite the claim rather than invent a second primitive, and send the claim and moves sessions the exact changes they need (§9) before those specs are finalised on a one-box-per-region assumption. | |
| 128 | | H3 | **About one second of data loss, written down.** Litestream keeps shipping asynchronously. Writes that had not reached S3 when a box died are lost, and the docs and console say so. | |
| 129 | | H4 | **Placement policy: capacity plus soft spread.** Among eligible boxes, choose the one with the most headroom. Prefer a box that runs no other instance of the same app, but never refuse placement for that reason. If no box is eligible, the instance waits as `unplaced` and an operator is alerted. (Spec 2.) | |
| 130 | | H5 | **Each region's minimum is configurable: 1 by default, 2 where HA is sold.** Scale-out is automatic on low headroom. Scale-in is automatic too, but only after the box has been drained into the region's other boxes. (Spec 4.) | |
| 131 | | H6 | **File durability follows the paid plan.** Continuous sync on the top plan, periodic snapshots on the bottom one. The platform exposes this as a per-app setting (§5.6) and Carloku maps its plans to it. Platform-managed files (`instance.key`, `report.token`, `<host>.env`) sync continuously on every plan. | |
| 132 | | H7 | **Ticks are at-least-once.** A delivery that ran on a dead box but whose bookkeeping did not reach S3 may run again. Documented; apps that need exactly-once guard it themselves. | |
| 133 | | H8 | **`files_durability` defaults to `snapshot`** for apps with no setting. | |
| 134 | | H9 | **The reconcile sweep is 10 minutes** (§5.3). | |
| 135 | | H10 | **(2026-10-06) Termination fences stay automatic, and the disk must die with the box.** H1 stands against claim §7.3's operator-only declare-lost. A box AWS reports `terminated` is fenced automatically by a region placer: a box with read-only EC2 rights. The console holds none. The fence additionally requires every EBS volume the instance had attached to be `DeleteOnTermination: true`, read from the same `DescribeInstances` reply's `BlockDeviceMappings`. Otherwise the operator path applies. Stopped, unclear or vanished boxes stay operator-only (claim §7.3 D2 as written). Absence is never evidence. | |
| 136 | | H11 | **(2026-10-06) The roster is the claim's `control/host-actors.json`** (claim §10.1 D7). Region HA drops its own `control/boxes/<id>.json` authority. `internal/boxid`'s AWS-signed IMDS identity verification becomes the evidence required to enroll a box. Enrollment records the proven account, region and instance id in the roster entry's `cloud` field, and the termination fence binds to that entry. Box references are opaque (claim §2, rev 6.16) and follow #212 option B: the label is the key, the instance id is private. | |
| 137 | | H12 | **(2026-10-06) Executor choice at create** is spec 1's `choose()`: prefer a box running no instance of the app; then the most headroom (until spec 2, the fewest claimed hosts); then the claim's P5-D2 SHA-256 of `host + "\x00" + box reference` as the final tie-break. It is recorded once, in the create intent. Recovery never substitutes. | |
| 138 | | H13 | **(2026-10-06) Claim ruling R6 supersedes HA3 and §5.5.** A box never reuses its local database. Every admission is cold until claim P6-D4 lifts. HA3 and §5.5 are deleted, and the activator half of plan 1a's install marker is not used. | |
| 139 | | H14 | **(2026-10-06) Relocation starts need no console reservation (L6).** A relocation start requires that no executable update is reserved for `from` or `to` in `control/host-actors.json`, which the worker reads and never writes. Root on each box refuses to start an executable update while that box is a participant in a running claim operation. The update side's refusal is what closes the race with a reservation made after the start's read. (§5.10.) | |
| 140 | | H15 | **(2026-10-06) H10's volume check rests on a running-time record, not the terminated read.** Tested on dev 2026-10-06: while `shutting-down`, `DescribeInstances` still listed both volumes of a test instance with their `DeleteOnTermination` flags; once `terminated`, `BlockDeviceMappings` was empty, the `true` root volume was already gone (`InvalidVolume.NotFound`) and the `false` volume survived as `available`. So the fence rests on (1) a record of the box's block device mappings taken by regional workers while it runs, refreshed on change, kept with its roster-bound evidence, showing every attached EBS volume `DeleteOnTermination: true`; (2) IAM denying `ec2:ModifyInstanceAttribute` and `ec2:AttachVolume` on platform boxes to every principal except operator roles, so the record cannot go stale behind the platform's back; (3) the terminated read (describe, event, later the group's history). A `shutting-down` read that still lists mappings may refresh the record but is never itself evidence. A missing record, one older than the stated bound, or one with any `false` volume sends the box to the operator path. (§5.9, §6.) The controller added `ec2:DetachVolume` to the deny in (2), as within the ruling's intent. | |
| 141 | | H16 | **(2026-10-06) Auto Scaling boxes are enrolled automatically, with evidence (spec 4, L19).** The console enrolls an Auto Scaling box itself when its `boxid` evidence verifies (the deployment's account, the region's AWS region) and a regional worker confirms, with read-only Auto Scaling rights, that the instance is a current member of that region's group. Anything else still needs an operator's approval. | |
| 142 | | H17 | **(2026-10-06) The first box of an emptied region is confirmed from another region.** Any enrolled box in another region holds cross-region read-only Auto Scaling and EC2 describe rights and may confirm a new box's group membership (H16) for a region that has no enrolled box. That is its only cross-region use. One-box regions therefore recover automatically. (§5.1, §6, L19.) | |
| 143 | | H18 | **(2026-10-06) A stale console report does not hold up a box-started relocation.** For box-started relocations only, the census accepts a console's newest report however old, provided that report met the protocol floor and the console's box has the claim's D1 root update guard armed, so it cannot be swapped for an older binary underneath. Everything else in claim D7 is unchanged. (§5.9, L20.) | |
| 144 | | H19 | **(2026-10-06) The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores.** Supersedes H15 and the volume half of H10; H10's automatic failover stands. The execution fence is a verified `terminated` read, bound to the roster's `cloud` identity, from the event or a describe (later the group's history); absence is never evidence. This is the claim's own lost-box rule for execution (§7.2), which needs no volume condition: a terminated instance can never run again, a surviving disk cannot write to storage by itself, and a disk attached to another instance is a different box with a different enrollment, never admitted as the old executor. The disk question belongs only to claim D5's destroyed-local exception: whether the dead box's `park-local` chores can be discharged automatically. That needs the volume evidence (the box's launch-time mappings all `DeleteOnTermination`, recorded while running); if it cannot be proven, the chores stay open for an operator (claim D2 and D5) and nothing about recovery waits on them. A surviving copy of a disk is a data-retention and cleanup concern for the operator path, not a two-writer risk. Tofu keeps `delete_on_termination = true`. (§5.9, §6, L2, L3.) | |
| 145 | |
| 146 | H10 to H19 supersede only what they name: H10 adds the volume condition |
| 147 | to H1 and H15 says where it is read from, H11 replaces §5.1's box records |
| 148 | as authority, H13 replaces HA3 and §5.5, H14 replaces claim D7's console |
| 149 | reservation for relocation starts, H16 and H17 settle enrollment for |
| 150 | spec 4, H18 narrows claim D7's console census for box-started |
| 151 | relocations, and H19 supersedes H15 and the volume half of H10 (the |
| 152 | fence needs no disk evidence; only automatic discharge of local chores |
| 153 | does). |
| 154 | Everything else in H1 to H9 stands. |
| 155 | |
| 156 | Standing rulings this spec keeps: |
| 157 | |
| 158 | - One writer: a home-run instance runs on exactly one box (claim C4, |
| 159 | multi-region ingress, sidecar members). |
| 160 | - Placement targets a region and pool, never a named box (fleets D5). |
| 161 | An operator or a customer still cannot pin an instance to a box; only |
| 162 | the placer chooses. |
| 163 | - The console is the single control-plane writer for operator and |
| 164 | customer requests, and holds no EC2 rights (claim §7.2). Boxes write |
| 165 | claims too, as the claim allows them: writer admission, publication, |
| 166 | their own chores and, under this spec, relocation. |
| 167 | - A box must hold the deployment's secrets key to receive instances (R9), |
| 168 | and every box reconcile would admit in a region must qualify |
| 169 | (nearest-region F2). |
| 170 | - Unreplicated state on a dead box is lost, and we say so. |
| 171 | - **Claim R6:** a box never reuses its local database; every writer start |
| 172 | restores from the certified remote (claim §6.4a). Under P6-D4, every |
| 173 | admission is cold. |
| 174 | |
| 175 | ## 1. Why |
| 176 | |
| 177 | Today a region is one box. Placement is per region: a bucket record has |
| 178 | `Region` and `Pool` and no box (`internal/provision/provision.go:49`). |
| 179 | Every box whose `CARLOS_REGION` matches mints the route |
| 180 | (`inScope`, `internal/provision/reconcile.go:348`, called at `:820`). Nothing |
| 181 | enforces "exactly one box": `reconcile.go:192` states it as intent only. |
| 182 | |
| 183 | A second box in a region would break three things: |
| 184 | |
| 185 | 1. **Both boxes run the instance's route.** The S3 lease (`lock.json`, |
| 186 | 15-minute TTL, `internal/lease/lease.go:37`) stops them running at |
| 187 | once, but whichever box a request reaches wins. |
| 188 | 2. **Both boxes claim the host to the edges.** The home map broke the |
| 189 | tie by sorted box name, with no liveness (`foldHomes`, |
| 190 | `internal/edge/homemap.go`), so a dead box that sorts first kept |
| 191 | winning. Plan 1a settles it by installation stamp (§5.4). |
| 192 | 3. **A box can serve a stale local copy.** The activator trusts a local |
| 193 | database as the warm cache because "on one box … the local copy can |
| 194 | only be what we ourselves checkpointed" (`internal/activator/activator.go:1164-1168`). |
| 195 | With two boxes taking turns, that is false. Claim R6 removes the |
| 196 | warm cache altogether (H13). |
| 197 | |
| 198 | And when a box dies, its instances stay down. EC2 auto-recover brings |
| 199 | back the same machine, if it can (`host_auto_recover`, |
| 200 | `infra/modules/region-host/main.tf:521`). |
| 201 | |
| 202 | ## 2. Words |
| 203 | |
| 204 | - **Box**: one platform host machine. Not a customer fleet box; fleets |
| 205 | keep their own mechanism and are out of scope. |
| 206 | - **Box reference**: the name the claim uses for a box in `op.exec`, |
| 207 | `participants`, `touched.boxes`, `inventory`, `lost`, owners and |
| 208 | execution-lease keys. It is opaque and compared by exact equality |
| 209 | (claim §2, rev 6.16). Under #212 option B it is the box's label key. |
| 210 | The instance id is private: it lives in the box's roster entry, under |
| 211 | `cloud` (H11). Spec 1 never parses a reference. The edges file keeps |
| 212 | its field name `box_id` for it; plan 1a already publishes the label |
| 213 | there. |
| 214 | - **Roster**: `control/host-actors.json` (claim §10.1 D7, proposed, |
| 215 | accepted for region HA by H11). Console-authored, root-signed, |
| 216 | conditionally replaced. Each box entry binds a reference to its |
| 217 | `enrollment`, `region`, `pool`, `state` (`active`, `quarantined`, |
| 218 | `retired`) and `cloud: {provider, account, region, instance_id}`. |
| 219 | - **Executor**: the one box the claim's `op.exec` names for a host. |
| 220 | - **Relocatable host**: an exec-backed instance or a sidecar, whose whole |
| 221 | start state is in S3 or rendered from its record (HA5). Unit- and |
| 222 | socket-backed routes (the console, platform services) are **not** |
| 223 | relocatable: their unit definitions live outside the record. They stay |
| 224 | on their box, as today (A1-8). |
| 225 | - **Eligible box**: a box whose roster entry is `active`, whose `cloud` |
| 226 | was proven by `internal/boxid` at enrollment (§5.1), and whose enrolled |
| 227 | region and `cloud.region` are the placement's; whose edges file is fresh |
| 228 | (nearest-region F1) and names the same reference; that is reachable, |
| 229 | hosting, serves the instance's pool, accepts inbound, holds the secrets |
| 230 | key (R9), publishes the HA capability (§8), and is not draining. For a |
| 231 | given host it must also not be on that claim's `lost` list and not owe |
| 232 | that host a `park-local` chore. |
| 233 | - **Region placer**: a **regional worker** in claim P6-D5's sense: any |
| 234 | active enrolled box in the region with access to the region's data |
| 235 | bucket. It runs relocation recovery, failover (spec 3) and drain (spec |
| 236 | 4). There is no separate leader roster. Correctness never depends on |
| 237 | which worker acts: every step is a claim CAS conditional on `op.id`, |
| 238 | the phase and the ETag (claim C2). Spec 3 may still use a lease to |
| 239 | decide who **starts** relocations, for efficiency only. |
| 240 | - **Prepared generation**: a generation built before any writer, from a |
| 241 | sealed or pinned source, with its snapshot, level-0 anchor and install |
| 242 | manifest, and opened at most once by a cold `AdmitWriter` (claim §6.4a |
| 243 | item 3, P6-D3; moves §7.6). |
| 244 | - **Relocation**: changing a host's executor to another box **in the |
| 245 | same region** (§5.8). A move changes region; relocation does not. |
| 246 | |
| 247 | ## 3. Invariants |
| 248 | |
| 249 | **HA1. The claim names the executor.** A relocatable host runs only on |
| 250 | the box its claim's `op.exec` names. This is claim C4 unchanged. What |
| 251 | changes is that `op.exec` may now name any eligible box in the record's |
| 252 | region, not "the region's box". |
| 253 | |
| 254 | **HA2. Region equality never admits.** The activator, config sync, |
| 255 | release adoption and timers act for a host only on its executor, or on a |
| 256 | participant in the phase that permits it (§5.8). Being in the right |
| 257 | region is necessary, never sufficient (claim C4 says this already; |
| 258 | today's code does not do it). |
| 259 | |
| 260 | **HA3. Withdrawn (H13).** Claim R6 governs instead: no box skips restore. |
| 261 | Every admitted writer purges its local database and restores from the |
| 262 | certified remote (claim §6.4a), and under P6-D4 every admission is cold. |
| 263 | |
| 264 | **HA4. Edges follow the newest installation.** When two edges files claim |
| 265 | one host, the edge believes the one carrying the higher `holder.stamp`. |
| 266 | Stamps only increase (claim `gen`), so a dead box's stale file always |
| 267 | loses to its successor's, with no liveness rule. |
| 268 | |
| 269 | **HA5. Everything a relocatable host needs to start is in S3.** The |
| 270 | database, files directory, env files and timer state are under the |
| 271 | incarnation's `holder.location` in the current generation, or are |
| 272 | rendered from the record (config, release). Nothing needed to start it |
| 273 | lives only on the box running it. |
| 274 | |
| 275 | **HA6. The executor changes only by a claim operation.** Create, a move, |
| 276 | or a relocation (§5.8). Each changes `op.exec` in a CAS that cites stop |
| 277 | evidence (claim §6.1) or a lost-box fence, the termination fence |
| 278 | included (§5.9), and writes `holder.stamp` in the same CAS (claim R6-4). |
| 279 | A relocated host's data becomes current only through a cold |
| 280 | `AdmitWriter` into a fully built prepared generation (claim §6.4a). |
| 281 | |
| 282 | **HA7. Termination evidence is bound to the roster.** A fence counts only |
| 283 | if the terminated instance is the one the box's roster entry records |
| 284 | under `cloud`, in the deployment's own account and the region's own AWS |
| 285 | region (A1-1, H11). No disk condition applies to the fence (H19): disk |
| 286 | evidence decides only whether the box's local chores are discharged |
| 287 | automatically (§5.9). |
| 288 | |
| 289 | ## 4. What stays the same |
| 290 | |
| 291 | - A region with one box behaves exactly as today, and every new rule |
| 292 | holds trivially there. |
| 293 | - Static sites and public addr routes still run on every box |
| 294 | (`everywhereClass`, `reconcile.go:215`). Only relocatable hosts gain an |
| 295 | executor that can change; unit- and socket-backed routes keep theirs. |
| 296 | - Customer fleets keep their binding stamps (`internal/host/fleetbind.go`) |
| 297 | and per-request fallback (`fleetChannelFor`, `internal/edge/edge.go:2466`). |
| 298 | - The ingress model: any edge serves any app and backhauls home-run |
| 299 | hosts to their box (multi-region ingress). This spec changes which box |
| 300 | is "home", not how backhaul works. |
| 301 | - Latency steering and ingress for boxes that come and go are spec 4. |
| 302 | |
| 303 | ## 5. Design |
| 304 | |
| 305 | ### 5.1 Box identity, the roster and evidence |
| 306 | |
| 307 | **Rev 5 (H11).** Spec 1's own `control/boxes/<id>.json` is no longer an |
| 308 | authority. The roster is. What plan 1a built in `internal/boxid` becomes |
| 309 | the evidence a box must present to be **enrolled**. |
| 310 | |
| 311 | **Enrollment.** A box asks to be enrolled by publishing its identity |
| 312 | evidence, the record plan 1a defined: |
| 313 | |
| 314 | control/boxes/<box reference>.json |
| 315 | |
| 316 | ```json |
| 317 | { |
| 318 | "v": 1, |
| 319 | "box": "<box reference>", |
| 320 | "identity_document": "<base64 IMDS instance identity document>", |
| 321 | "identity_signature": "<base64 RSA-SHA256 signature from IMDS>", |
| 322 | "written_at": "<RFC3339>" |
| 323 | } |
| 324 | ``` |
| 325 | |
| 326 | The object keeps plan 1a's storage rules: create-only, never deleted |
| 327 | (§6). It grants nothing by itself. It is the evidence enrollment reads. |
| 328 | |
| 329 | Enrollment is the claim's (D7, L7). For region HA it must: |
| 330 | |
| 331 | - verify the record with `boxid.Verifier`: the signature under the |
| 332 | embedded certificate for the document's region, `accountId` equal to |
| 333 | the deployment's account, and the document's region equal to the AWS |
| 334 | region the box is being enrolled into; |
| 335 | - record the proven `accountId`, `region` and `instanceId` in the entry's |
| 336 | `cloud: {provider: "aws", account, region, instance_id}`, and the |
| 337 | record's SHA-256 beside it; |
| 338 | - refuse a reference already enrolled to a different instance (#212's |
| 339 | refusal to publish under a label another machine holds); |
| 340 | - for a reference that already appears in claims (`touched.boxes`, |
| 341 | `inventory`) or edges files, as a box predating region HA does, also |
| 342 | check that the signed `instanceId` equals the region's tofu |
| 343 | `region-host` module `instance_id` output. This is the check rev 4's |
| 344 | operator bind made (A1-1 residual, A3-3). It now happens once, inside |
| 345 | the operator-approved enrollment (claim §10.1 P3), so the separate bind |
| 346 | object is not used. |
| 347 | |
| 348 | **Auto Scaling boxes (H16, spec 4).** The console enrolls a box launched |
| 349 | by the region's Auto Scaling group itself, with no operator, when two |
| 350 | pieces of evidence agree: its `boxid` record verifies with the |
| 351 | deployment's account and the region's AWS region, and a regional worker |
| 352 | has written a create-only membership confirmation, |
| 353 | `control/box-enrollment/<box reference>/<instance id>.json`, from a fresh |
| 354 | read-only `autoscaling:DescribeAutoScalingInstances` showing the instance |
| 355 | a current member (`Pending` or `InService`) of that region's group, with |
| 356 | the group name from the deployment's configuration. Anything else, a |
| 357 | fixed box included, is enrolled only with an operator's approval. Spec 4 |
| 358 | builds this; the claim's D7 needs to allow it (L19). |
| 359 | |
| 360 | **A region with no enrolled box (H17).** When the target region has no |
| 361 | **live** enrolled box, the confirmer may be any `active` enrolled box of |
| 362 | another region, acting as a regional worker of its own |
| 363 | region and checking its own entry first. This is the only thing it ever |
| 364 | does across regions. Its confirmation binds to the target region's |
| 365 | group, not to anything the new box says: |
| 366 | |
| 367 | - it reads the target region's group name and ARN from the roster's |
| 368 | `regions` entry for that placement (claim P5-D2's extension), never |
| 369 | from the new box or an edges file; |
| 370 | - it calls `DescribeAutoScalingInstances` for the instance id in the |
| 371 | target region's AWS region, and requires the reply to name exactly that |
| 372 | group, lifecycle state `Pending` or `InService`, and an availability |
| 373 | zone in that AWS region; then `DescribeInstances` there, requiring the |
| 374 | instance `pending` or `running` with the deployment's `OwnerId`; |
| 375 | - **the region is empty (B2-6).** The roster entry of a dead box stays |
| 376 | `active` until a fence retires it, and only a regional worker can write |
| 377 | the fence, so the roster alone would never show the region empty. The |
| 378 | confirmer therefore describes, in the same region and the same pass, |
| 379 | every other enrollment the roster lists there as `active` or |
| 380 | `quarantined`, and requires each to read **`terminated`**, positively, |
| 381 | for its roster `cloud`. `InvalidInstanceID.NotFound`, any other state, |
| 382 | an error or a partial reply refuses the confirmation (absence is never |
| 383 | evidence; a box that has left `DescribeInstances` goes to an operator). |
| 384 | The confirmation lists those reads (`terminated_enrollments: [{box, |
| 385 | enrollment, instance_id, observed_at, request_id}]`). They only show |
| 386 | the region has no live worker. They are not a termination fence, they |
| 387 | discharge nothing, and they never retire a roster entry; |
| 388 | - it writes the same create-only object, |
| 389 | `control/box-enrollment/<box reference>/<instance id>.json`, with |
| 390 | `{target_placement, aws_region, group_arn, instance_id, |
| 391 | lifecycle_state, observed_at, request_ids, confirmer: {box, |
| 392 | enrollment, region}}`. Two confirmers race by the same conditional |
| 393 | write: one creates it, the other gets a refusal, reads it, and stops. |
| 394 | |
| 395 | The console accepts a cross-region confirmation only if, at enrollment, |
| 396 | every enrollment the roster lists as `active` or `quarantined` in the |
| 397 | target region appears in the confirmation's `terminated_enrollments` |
| 398 | (any new one refuses), the confirmer was `active`, the group ARN equals the roster's for that |
| 399 | placement, and the `boxid` record's region equals `aws_region`. Once |
| 400 | enrolled, the new box is the region's worker: it writes its own volume |
| 401 | records, fences its predecessor through the ordinary §5.9 path (its own |
| 402 | terminated read), and recovers |
| 403 | the hosts (§5.8). So a one-box region recovers automatically (§7). |
| 404 | |
| 405 | A box whose entry is not `active` with a proven `cloud` is never |
| 406 | eligible, and a claim naming one is never relocated automatically: an |
| 407 | operator resolves it. |
| 408 | |
| 409 | **Edges file fields.** Each box's edges file |
| 410 | (`internal/host/edgesfile.go`) carries, as plan 1a built: |
| 411 | |
| 412 | ```json |
| 413 | { |
| 414 | "box_id": "<box reference>", |
| 415 | "capabilities": ["region-ha-1"], |
| 416 | "draining": false, |
| 417 | "headroom": { "mem_free_mib": 2210, "disk_free_mib": 31200, "awake": 14 } |
| 418 | } |
| 419 | ``` |
| 420 | |
| 421 | - `box_id` must equal an `active` roster reference; a file naming an |
| 422 | unknown or inactive reference makes nothing eligible. The edges file is |
| 423 | evidence of liveness and configuration, never of identity or |
| 424 | membership (claim D7: heartbeats describe members, they do not create |
| 425 | them). |
| 426 | - `capabilities` gates multi-box admission (§8). |
| 427 | - `draining` is set through the box by the placer (spec 4). |
| 428 | - `headroom` is the input to spec 2's choice. Spec 1 publishes it only. |
| 429 | |
| 430 | `eligibleBoxes(snapshot, roster, region, pool)` returns the boxes, |
| 431 | applying the predicate one box at a time (nearest-region |
| 432 | `TestEligibilityNeverCombinesBoxes`). The expected AWS region comes from |
| 433 | the roster entry's enrolled region and `cloud.region`, never from a vote |
| 434 | of edges files (§18's second correction, now answered). |
| 435 | |
| 436 | ### 5.2 Create names a box |
| 437 | |
| 438 | Claim §5.1's create intent names its executor (P5-D6 |
| 439 | `selected_executor`, `executor_enrollment`), written in the step 2 CAS |
| 440 | before any effect. For a relocatable host it is |
| 441 | `choose(eligibleBoxes(region, pool), app)` (H12, L8). Step 4 |
| 442 | (`CompleteCreate`) sets `op.exec` to exactly that box. |
| 443 | |
| 444 | `choose`, until spec 2 (H12): |
| 445 | |
| 446 | 1. prefer a box that runs no instance of the app (H4's soft spread: it |
| 447 | never refuses); |
| 448 | 2. then the most headroom; until spec 2 that is the fewest claimed hosts, |
| 449 | counted from the console's claim-recovery listing (claim §5.1 step 5). |
| 450 | A stale count only affects balance, never safety; |
| 451 | 3. then the greatest hexadecimal SHA-256 of `host + "\x00" + box |
| 452 | reference` (claim P5-D2), and the smallest reference if two hashes |
| 453 | are equal. |
| 454 | |
| 455 | It is a pure function over the roster, the edges snapshot and the |
| 456 | counts, and it is unit-tested alone. |
| 457 | |
| 458 | - The choice is made once. Recovery never substitutes a newly available |
| 459 | box (P5-D2). If the chosen box cannot complete the create (fenced, |
| 460 | quarantined, or it never acknowledges), recovery cancels the create |
| 461 | (claim §5.1 step 5, P5-D6), and the caller may create again. |
| 462 | - No eligible box: the create fails with today's "region has no box |
| 463 | that can take this" refusal (nearest-region). Spec 2 adds the |
| 464 | `unplaced` state for re-placement, where refusing is not possible. |
| 465 | - The choice is recorded only in the claim. The record keeps `Region` |
| 466 | and gains no box field, so a record still describes residency, not |
| 467 | placement. |
| 468 | - A move into a multi-box region chooses its target box the same way, |
| 469 | in its start CAS (L15). |
| 470 | |
| 471 | ### 5.3 Reconcile: executing rows, held rows, former rows |
| 472 | |
| 473 | Reconcile distinguishes these cases for a relocatable host in its region |
| 474 | (A1-6). Each is decided by a fresh claim read. |
| 475 | |
| 476 | | The claim names this box as | Row | |
| 477 | |---|---| |
| 478 | | `op.exec`, op `serving` | mint or keep `ready` | |
| 479 | | `from` of a `relocating` op in `starting`, or in `releasing` until its admission fence closes (A2-3) | keep `ready` | |
| 480 | | any other participant role of a running operation (§5.8 table; moves §9.6) | mint or keep held, with `hold_op` = the op id (moves §5.2), never `ready` | |
| 481 | | the owner of an open `park-local` chore for the host | keep held until the chore's retirement removes the row | |
| 482 | | none of these | mint nothing. An existing row is held, never served, and alerted as unexplained | |
| 483 | |
| 484 | - **Former executors (rev 5).** Rev 4.2 had reconcile drop a former |
| 485 | executor's row as a registry change. Under claim P6-D2 a host's route |
| 486 | and timer rows are typed inventory entries, and `park-local`'s |
| 487 | retirement journal archives and removes them, conditional on their |
| 488 | recorded ownership. So reconcile never deletes a relocatable host's |
| 489 | row. It holds it, and the chore retires it under the host guard. A row |
| 490 | is never removed by hostname alone, which could remove a successor's. |
| 491 | - The stale-hold rule (claim §8, CC10) is unchanged: it reopens a hold |
| 492 | only on the current executor of a `serving` host. |
| 493 | - **When reconcile reads claims.** The safety gate is the activator's |
| 494 | fresh claim read at every spawn (claim C4), not reconcile. So |
| 495 | reconcile reads a host's claim when it would mint a row, and re-checks |
| 496 | the rows it keeps on a slow sweep, every 10 minutes (H9). This is a |
| 497 | stated exception to C4's "reconcile's decision to keep a row" (L12): a |
| 498 | kept row that is stale for up to 10 minutes can route a request to |
| 499 | this box, where the activator refuses to spawn. Reading every claim on |
| 500 | every pass would cost about $2.30 a day per box at 2,000 hosts on |
| 501 | 30-second passes. |
| 502 | - **Prompt changes.** The placer bumps the region's control serial after |
| 503 | each executor CAS, so the region's boxes reconcile at once and re-read |
| 504 | the claims of the hosts they hold or would mint. |
| 505 | - A claim read error keeps an existing row and mints none (claim C6). |
| 506 | |
| 507 | ### 5.4 Edges and the home map |
| 508 | |
| 509 | - The edges-file pass (`edgesFilePass`, `edgesfile.go`) publishes a host |
| 510 | as `home` only when this box is its executor **and** its row is |
| 511 | `ready`, and adds the installation's stamp: |
| 512 | `"hosts": {"shop.bab.oncarlos.com": {"role": "home", "stamp": 41}}`. |
| 513 | A relocation's `to` therefore publishes nothing while it installs and |
| 514 | probes, and becomes home at `serving`. |
| 515 | - The home map (`foldHomes`, `homemap.go`) resolves a host claimed by two |
| 516 | files to the higher stamp (HA4). Plan 1a built this. |
| 517 | - **Unstamped entries.** Before §8's gate opens, every region has one |
| 518 | box, so there is no conflict to resolve. After it opens, every box |
| 519 | publishes stamps for relocatable hosts, and an unstamped entry for a |
| 520 | host that some file carries stamped is ignored. §18's correction |
| 521 | stands: two unstamped entries are normal for everywhere-class routes, |
| 522 | so the edge reports only two boxes at the same positive stamp. |
| 523 | - The edge never reads claims on the request path. |
| 524 | |
| 525 | ### 5.5 The activator's local copy: withdrawn (H13) |
| 526 | |
| 527 | Rev 4.2's installation-complete marker and its warm-cache check (HA3) are |
| 528 | deleted. Claim R6 replaces them: every admitted writer purges any local |
| 529 | database with the error-returning purge and restores from the certified |
| 530 | remote (claim §6.4a, "The baseline is always the certified remote"). |
| 531 | Under P6-D4 every admission is cold. The activator's one-box comment at |
| 532 | `activator.go:1164-1168` is replaced by a reference to claim R6, not to HA3. |
| 533 | |
| 534 | Plan 1a's `activator.WarmCopyTrusted` is not used, and nothing reads the |
| 535 | install marker to skip a restore (§19). A1-5's concern, that authority is |
| 536 | not evidence of a complete install, is answered more strongly: no local |
| 537 | copy is ever trusted, complete or not. |
| 538 | |
| 539 | ### 5.6 Host files reach S3, in order |
| 540 | |
| 541 | Moves carry the files directory (F4) and `<host>.env` (F6) as a sealed |
| 542 | archive taken from the live source (moves §6.1, §8). A dead source can't |
| 543 | produce one, so HA5 needs them in S3 already. |
| 544 | |
| 545 | **The setting.** An app's manifest gains `files_durability`: |
| 546 | |
| 547 | - `continuous`: a changed file is published within about one second |
| 548 | of its **first** unpublished modification, whether or not the app has |
| 549 | closed it (A2-5): |
| 550 | - **Watches.** inotify `IN_MODIFY`, `IN_CLOSE_WRITE`, `IN_MOVED_*`, |
| 551 | `IN_CREATE` and `IN_DELETE` on every directory of the tree, added |
| 552 | recursively as directories appear. |
| 553 | - **Deadline.** The first modification of a file arms a one-second |
| 554 | deadline. Later writes do not push it back. A file still being |
| 555 | written is published at the deadline, then again at the next one. |
| 556 | - **Overflow.** `IN_Q_OVERFLOW`, or any watch that can't be added, |
| 557 | triggers an immediate full scan, comparing size, mtime and inode with |
| 558 | the last published state. |
| 559 | - **A consistent image.** The uploader reads the file, then checks that |
| 560 | size, mtime and inode are unchanged. If they changed, it retries |
| 561 | within the deadline. A file that never holds still for a read is |
| 562 | still published **at every deadline**, as the last image read, and |
| 563 | flagged as possibly torn. It never waits for a scan (A2-5 residual). |
| 564 | - **Missing coverage.** While any directory has no working watch |
| 565 | (overflow, a watch that can't be added, the watch limit reached), the |
| 566 | publisher **polls** that subtree once a second, comparing size, mtime |
| 567 | and inode, until the watch is restored. Polling is reported as a |
| 568 | box metric, and alerts if it lasts an hour. The files directory is |
| 569 | not a database: apps that update a file in place must expect a torn |
| 570 | image after a failover, and the docs say so. The instance database is |
| 571 | Litestream's. |
| 572 | - **Repair.** A full scan every 10 minutes catches anything a watch |
| 573 | missed. It is repair, never the path a change waits on; |
| 574 | - `snapshot`: the whole directory is published every |
| 575 | `files_snapshot_interval`, default 15 minutes. |
| 576 | |
| 577 | Carloku maps its plans to this (H6). Apps with no setting get |
| 578 | `snapshot` (H8). |
| 579 | |
| 580 | **Always continuous, whatever the setting:** `instance.key`, |
| 581 | `report.token`, `<host>.env`, `<host>-sidecar.env`, and the host's wake |
| 582 | state (§5.7: timer rows, the route's `NextDue` and `Replication`). They |
| 583 | are small, and losing them is what kept kpmg down for 1h54m (moves §1.2). |
| 584 | |
| 585 | **Ordering (A1-4).** Object versions do not say which update wins, so |
| 586 | nothing is ever overwritten and one manifest decides. §18 records how |
| 587 | plan 1a built it: content is identified by SHA-256, and `snapshot` is a |
| 588 | periodic full publish with no `base`. |
| 589 | |
| 590 | - **Objects are immutable.** Each upload goes to a new key, |
| 591 | `<holder.location>/files/<generation>/objects/<seq>-<sha256>`, written |
| 592 | create-only. A stalled upload that completes late creates an object no |
| 593 | manifest names. |
| 594 | - **One sequence per host.** The executor numbers every local mutation |
| 595 | (write, rename, delete) with a monotonically increasing `seq`, kept |
| 596 | under the host guard and recovered from the manifest on restart. |
| 597 | - **The manifest** is one object per generation, |
| 598 | `<holder.location>/files/<generation>/manifest.json`, replaced only |
| 599 | with `If-Match`. A delete is a tombstone entry, never an S3 delete. The |
| 600 | manifest is published only after every object it names is durable, and |
| 601 | its `seq` only increases, so a late manifest write fails its |
| 602 | `If-Match`. |
| 603 | - **Restore** reads the manifest once, then fetches the objects it names |
| 604 | and checks each hash (§18). Nothing else in the prefix is read. |
| 605 | - **Release flush.** A box giving up a host publishes every pending |
| 606 | mutation, **whatever the app's setting**, before its stop evidence |
| 607 | (§5.8's release sequence). |
| 608 | |
| 609 | **Under the claim (rev 5).** Three rules follow from the claim as it now |
| 610 | stands. |
| 611 | |
| 612 | - **The publisher is a generation-bound writer.** It belongs to the |
| 613 | writer session, like the replicator: it is started for the `{location, |
| 614 | generation, session}` the admission named, refuses any key outside |
| 615 | `<location>/files/<generation>/`, and stops with the session. It is a |
| 616 | producer for stop evidence (claim P6-D1) and is joined before the |
| 617 | receipt. Its PUTs are a second ordinary-replication exception to claim |
| 618 | C3 (L13). |
| 619 | - **Each cold admission starts a new manifest.** Under P6-D4 every wake |
| 620 | is a cold admission into a new generation. Before the app starts, the |
| 621 | admitting box writes `files/<new generation>/manifest.json` |
| 622 | create-only, as a copy of the **files pin**: the newest manifest in the |
| 623 | admitted chain, read by exact version (B2-4). That is the manifest of |
| 624 | `holder.generation` as observed, or, if that generation has none (its |
| 625 | copy never completed, so its app never started), of the nearest |
| 626 | predecessor in `holder.admissions` that has one. It is chosen |
| 627 | independently of the database baseline: the files publisher and the |
| 628 | replicator make progress separately, so a generation can hold |
| 629 | published files before its first database upload. Its entries keep |
| 630 | naming the predecessor's objects where they are, so an unchanged file |
| 631 | is never uploaded again. A late write from the predecessor's publisher |
| 632 | lands in the predecessor's prefix, which nothing admitted reads again. |
| 633 | - **Old objects are retired by the claim, never reaped.** Rev 4.2's "reaped |
| 634 | after a day" is withdrawn (plan 1a never built it). A dead generation's |
| 635 | `files/<generation>/` prefix is retired with that generation, by the |
| 636 | `park-generation` chore `PublishGeneration` queues, except objects that |
| 637 | a manifest in the live set still names (L13). The **files live set** is |
| 638 | every manifest from the files pin through `holder.generation`, plus any |
| 639 | manifest a running operation has journaled in its pin, and every |
| 640 | object they name. A park that runs while an operation is in progress |
| 641 | keeps all of them. The at-rest sweep treats |
| 642 | `files/` as part of the location, never as residue to classify |
| 643 | blindly (claim P6-D3). |
| 644 | |
| 645 | A `snapshot` app's files can be up to the interval older than its |
| 646 | database after a failover, and the console says so on the app's |
| 647 | Overview. |
| 648 | |
| 649 | **Moves.** A move keeps its sealed archive at the cut (moves §8), and |
| 650 | installs a fresh manifest in its new generation (L17). |
| 651 | |
| 652 | ### 5.7 What a new executor rebuilds |
| 653 | |
| 654 | | State | How the new box gets it | |
| 655 | |---|---| |
| 656 | | Database | built from the pinned recovery set into a prepared generation, then restored from it by the probe's cold `AdmitWriter` (§5.8) | |
| 657 | | Files directory, env files | the pinned manifest (§5.6), copied into the prepared generation | |
| 658 | | Wake state | the host's timer rows, as `Registry.ReplaceTimers` rows (moves §9.4), and the route's `NextDue` and `Replication` (moves §6.2), published continuously as one entry under the manifest (A1-7) | |
| 659 | | `<host>.config.env` | rendered for the held row (`RenderHeld`, claim §6.1 CC13, P6-D3), needs the secrets key (R9) | |
| 660 | | Release | materialized for the held row (`MaterializeHeld`) | |
| 661 | | Certificates | the cert mirror's boot refresh (on main) | |
| 662 | | Registry row | reconcile, held while it installs, `ready` at `serving` (§5.3) | |
| 663 | | Inventory | written by the new box before it first materializes state (claim §6.3, P6-D2) | |
| 664 | |
| 665 | **Wake state (A1-7).** The executor publishes the host's wake state as |
| 666 | an always-continuous entry in the manifest whenever any of it changes: |
| 667 | - its timer rows (active rows and their bookkeeping, moves §9.4's |
| 668 | canonical digest), on a registration, a delivery or a rearm; |
| 669 | - the route's `NextDue`, the app's own next wake |
| 670 | (`activator.go:2204`), whenever the activator sets it; |
| 671 | - the route's `Replication` mode. |
| 672 | |
| 673 | The new executor sets `NextDue` and `Replication` from it, as a move's |
| 674 | landing does (moves §7.6). A once-timer the app registered is |
| 675 | therefore in S3 within about a second of the app being told it |
| 676 | succeeded. On install the new box imports the timer rows with |
| 677 | `ReplaceTimers`, preserving `next_at`, `due_at`, `attempts` and |
| 678 | `registration`. Timers on held rows stay frozen (moves §9.4, claim |
| 679 | P6-D8) until the row is `ready`. |
| 680 | |
| 681 | - An occurrence that fell due while the host had no executor fires once |
| 682 | when the new executor imports it: the converger catches up from the |
| 683 | imported `next_at`, not from `Next(now)`. |
| 684 | - A delivery that ran on the dead box but whose bookkeeping did not reach |
| 685 | S3 runs again. Ticks are documented as **at-least-once** (H7). |
| 686 | |
| 687 | ### 5.8 Relocation |
| 688 | |
| 689 | A new claim op kind, `relocating` (L1), from `serving` only, for a |
| 690 | relocatable host, within one region. It is the stateful, same-region |
| 691 | executor change the claim's `placing` refuses (P5-D5). |
| 692 | |
| 693 | **The start CAS** writes `op.kind`, `op.id` and the intent before any |
| 694 | other effect (claim C2): |
| 695 | |
| 696 | {from, to, to_enrollment, region, reason, stamp_from, |
| 697 | generation_from, fence?, directory_revision} |
| 698 | |
| 699 | `reason` is `drain`, `rebalance` or `failover`. For failover, `fence` |
| 700 | names the termination fence record (§5.9) by key and SHA-256. The start |
| 701 | is refused by the claim's start guard (claim §4, CC11, as amended by |
| 702 | L11), when `to` is not eligible for the host (§2), and when the roster |
| 703 | shows an executable update reserved for `from` or `to` (§5.10, H14). |
| 704 | `directory_revision` is the roster revision that check read. No console |
| 705 | reservation is taken. Participants are `from` and `to`, or `to` alone |
| 706 | for a failover. |
| 707 | |
| 708 | **Who acts.** Every phase is a claim CAS conditional on `op.id`, the |
| 709 | phase and the ETag. The recovery owner for every phase is the region's |
| 710 | regional workers (§2, claim P6-D5), which replay from the claim alone |
| 711 | (claim C2). Each worker first checks its own roster entry is `active` in |
| 712 | this region. Two workers racing make the same CAS, and one fails. `from` |
| 713 | and `to` hold execution leases (claim §7.1, D6) for as long as they are |
| 714 | participants. |
| 715 | |
| 716 | **Phases (A1-3).** In the "admits" column, "writer" means a writer |
| 717 | admission (`AdmitWriter`) is legal for that box; nothing else may open |
| 718 | replication. |
| 719 | |
| 720 | | Phase | Who acts | `op.exec` | `holder.stamp` | Rows (§5.3) | Admits | |
| 721 | |---|---|---|---|---|---| |
| 722 | | `starting` | a worker writes the intent; `from` (if alive) and `to` write execution leases and `exec_ack` (D6). The barrier needs only `to` when the intent cites a fence, or once a source-loss CAS (below) has removed `from` | `from`, or `none` after a source-loss CAS | `stamp_from` | `from` ready; `to` held | `from`, as for `serving` | |
| 723 | | `releasing` | `from` runs the **release sequence** (below) and publishes its stop receipt; for failover the fence is already in the intent and this phase is skipped | `from` until evidence | unchanged | `from` ready until its admission fence closes, then held; `to` held | `from`, until its admission fence closes | |
| 724 | | `pinning` | a worker CAS citing the receipt or the fence: `op.exec: none`, a non-installing transition that keeps the stamp (claim R6-4, D5's form). Then it selects and journals the **pinned recovery set** (below) | `none` | unchanged | `from` held (if alive); `to` held | nobody | |
| 725 | | `installing` | a worker CAS that journals the pin and a fresh prepared-generation id `P`, hands `op.exec` to `to`, and writes `holder.stamp` = this CAS's `gen` (R6-4: a new installation), adding `to` to `touched.boxes`. `to` activates that stamp under the host guard (claim §6.4), writes its inventory entry, **builds `P`** and makes its **local install** (below) | `to` | `S'` | `to` held; `from` held | nobody: `to` writes local state under the guard, starts no app and admits no writer | |
| 726 | | `installed` | a worker CAS once `P`'s install manifest is complete and verified, journaling `P`'s prepared authority (claim P6-D3). The executor then journals its **local install receipt** by CAS, if this CAS did not already carry it (a replacement makes its own, below) | `to` | `S'` | as `installing` | nobody: the executor may make its local install, nothing else | |
| 727 | | `probing` | `to` enters it by CAS, which requires a local install receipt for exactly `op.exec`, its enrollment and `op.journal.install_stamp`; then each probe attempt is one cold `AdmitWriter` (below), then the probe (moves §7.7, same region), within the probe deadline | `to` | `S'`, kept by the first admission | as `installing` | `to`, writer, probe only | |
| 728 | | `probe-failed` | `to` runs the release sequence for its own session | `to` until evidence | unchanged | as `installing` | nobody | |
| 729 | | `serving` | a worker CAS back to `serving` with `participants: [to]`; `from` gets a `park-local` chore unless it is discharged under H10 (§5.9) | `to` | `S'` | `to` ready; `from` held until its chore retires the row | `to` | |
| 730 | |
| 731 | **The release sequence (A3-2).** Whenever a live box gives up a host, |
| 732 | the source in `releasing` or a failed target in `probe-failed`, it does |
| 733 | these in order, journaling each. It runs inside claim P6-D1's stop |
| 734 | collection: root's collector has recorded its durable fence, and the |
| 735 | agent does steps 1 to 3 before it closes the producers and returns its |
| 736 | drain acknowledgement. |
| 737 | |
| 738 | 1. **Stop every writer (A4-1).** Close admission, drain spawn attempts, |
| 739 | drain in-flight timer deliveries (moves §9.4), then stop the host's |
| 740 | process tree and wait for it to be empty: the per-host cgroup's |
| 741 | `populated 0` and no member processes. The database, WAL and SHM |
| 742 | files may still be held, but **only** by the box's own replicator for |
| 743 | this host, identified by the agent's pid and the replicator's own |
| 744 | descriptors (A5-1). Any other holder fails the step. From here nothing |
| 745 | but the replicator can touch the database, and nothing at all can |
| 746 | change files or wake state. `internal/stopevidence` has no check of |
| 747 | this form today: `Collect` requires the replicators closed. Rev 5 |
| 748 | requests a writers-stopped check that allows exactly these |
| 749 | descriptors (L10). |
| 750 | 2. **Flush the database:** final checkpoint and `SyncAndWait`, then |
| 751 | record the **final TXID** `N`: the stopped database's own position. |
| 752 | The step passes only once a listing shows the newest level-0 file's |
| 753 | `MaxTXID` equal to `N` (moves §5.1 step 1). A sync error is not |
| 754 | swallowed (claim P6-D4's warning about `Replicator.Close`); it retries, |
| 755 | and a flush that never reaches `N` goes to `stuck`. |
| 756 | 3. **Publish every pending file and wake-state mutation**, whatever the |
| 757 | app's `files_durability`, and record the manifest's final `seq`. |
| 758 | 4. **Finish stop evidence:** close the replicator (`DB.Close`) and the |
| 759 | file publisher, then root completes claim §6.1's **unrestricted** |
| 760 | check (`stopevidence.Collect`, P6-D1): no holder at all of the |
| 761 | database, WAL or SHM files in `/proc/*/fd`, and every other |
| 762 | condition. The receipt is published (P5-D3) with its `evidence_at`. |
| 763 | Only now is the release complete. |
| 764 | |
| 765 | Because step 1 empties the process tree first, the positions recorded in |
| 766 | steps 2 and 3 really are final. The session ends uncertified, as every |
| 767 | session does under P6-D4; coverage (below) is what keeps its writes. |
| 768 | |
| 769 | The journal entry is `{"released_by": "<box>", "generation": "<G>", |
| 770 | "session": S, "final_txid": N, "manifest_seq": M, "receipt": "<key>", |
| 771 | "receipt_sha256": "…"}`, and `pinning` refuses to pin less than it. |
| 772 | |
| 773 | **The pinned recovery set (A1-2).** In `pinning`, a worker runs |
| 774 | `genselect.Select` against the claim observation of its `pinning` CAS: |
| 775 | `holder.generation`, the unfinished run in `holder.admissions` and the |
| 776 | marker, all in this region's bucket (claim §6.2a, §6.4a item 3). It |
| 777 | journals the plan's exact object versions (the snapshot and every LTX |
| 778 | file after it, contiguous), and, chosen separately, the exact version of |
| 779 | the **files pin** (§5.6: the newest manifest in the admitted chain, not |
| 780 | the database baseline's generation), with files and timers included. |
| 781 | The two can come from different generations; each is the newest |
| 782 | durable state of its own publisher (B2-4). |
| 783 | |
| 784 | - **After a release sequence** (drain, rebalance, probe failure): the set |
| 785 | must reach the journaled `final_txid` and `manifest_seq`. If the listing |
| 786 | has not caught up, `pinning` waits; it never pins less than what the |
| 787 | releasing box flushed. |
| 788 | - **Failover:** the worker pins after the fence's `observed_at` plus 30 |
| 789 | seconds, so requests S3 had already accepted from the terminated box |
| 790 | have completed. Whatever did not land is the H3 loss window. This is a |
| 791 | loss bound, not a safety bound: the pin names exact versions, and a |
| 792 | later PUT can only add a version nothing reads. |
| 793 | |
| 794 | **Building the prepared generation (A1-2, A2-1).** In `installing`, `to` |
| 795 | builds `P` exactly as a move's landing builds its generation (moves §7.6 |
| 796 | `building` and `published`, L16), with the pinned versions as its source |
| 797 | in place of a staged replica: |
| 798 | |
| 799 | - under the replica lease on `holder.location` (claim C3), restore the |
| 800 | pinned versions with the planner, verifying each file's checksum, |
| 801 | contiguity, the final position `N` and `PRAGMA integrity_check`; |
| 802 | - write the build intent, then the full **snapshot** `[1, N]`, then the |
| 803 | **level-0 anchor** `[N, N]`, each acknowledged, then the completed |
| 804 | **install manifest** with each `versionId`, `N`, checksums and the |
| 805 | database's SHA-256. The pinned Litestream derives a box's replication |
| 806 | position from the newest remote level-0 file. A generation holding a |
| 807 | snapshot and nothing at level 0 restarts at TXID 1, and a cold restore |
| 808 | from the snapshot then skips every later write (moves R5-1); |
| 809 | - write `P`'s files manifest, create-only, as the pinned manifest's |
| 810 | entries (objects named where they are, §5.6), with the files, env and |
| 811 | wake state. |
| 812 | |
| 813 | That is the **remote build**: immutable, built once, and complete when |
| 814 | a fresh read verifies all three objects. `installed` records it. |
| 815 | `generation.json` is not written here: the probe's admission publishes |
| 816 | it (moves §7.14). |
| 817 | |
| 818 | **The local install (B1-2)** is separate, because it belongs to one box. |
| 819 | The box that is `op.exec`, under the host guard with the stamp it |
| 820 | activated: |
| 821 | |
| 822 | - applies the files and env from `P`'s files manifest to the local |
| 823 | directory, `ReplaceTimers` the wake state, and sets `NextDue` and |
| 824 | `Replication`; |
| 825 | - runs `RenderHeld` and `MaterializeHeld` with current inputs, recorded |
| 826 | as the relocation's inputs (P6-D3); |
| 827 | - journals each step locally, and then, last, records its **local |
| 828 | install receipt** in the op journal by CAS: |
| 829 | `local_install: {box, enrollment, install_stamp, |
| 830 | files_manifest_version, journal_sha256}`. |
| 831 | |
| 832 | **`install_stamp`** is the stamp of the CAS that handed this box |
| 833 | `op.exec` (`installing`, or a replacement's), journaled by that CAS as |
| 834 | `op.journal.install_stamp`. It is the installation's provenance, not the |
| 835 | current writer's stamp: later probe admissions take new stamps (§6.4a |
| 836 | item 4) and leave it alone (B2-5). A receipt counts only when its box, |
| 837 | enrollment and `install_stamp` equal `op.exec`, its enrollment and |
| 838 | `op.journal.install_stamp`, so a replacement executor never inherits its |
| 839 | predecessor's. Remaking the local install before the first admission is |
| 840 | not a reinstall: `P` is not touched. |
| 841 | |
| 842 | **After the first admission, the local install is never remade |
| 843 | (B2-5).** Once any admission entry names `P` (or a later generation of |
| 844 | this op), the box's local files, env and wake state are the live state |
| 845 | of that writer lineage. A new agent on the same box stops the app and |
| 846 | admits a new session, as for any cold wake; it never reapplies `P`'s |
| 847 | files or timers. A box that finds its receipt missing after that point |
| 848 | has lost local state it cannot rebuild from `P`: the op goes to `stuck`, |
| 849 | and the operator resolves it (or a fence makes it a nested failover |
| 850 | from `holder.generation` and the files pin). |
| 851 | |
| 852 | **The probe's admission.** Each probe attempt is one `AdmitWriter` |
| 853 | (claim §6.4a), made by `to` as `op.exec`, bound to the observed `op.id` |
| 854 | and phase `probing`: |
| 855 | |
| 856 | - **mode** cold, always (P6-D4, H13); |
| 857 | - **generation and baseline (B1-1):** the first attempt opens `P`, |
| 858 | which no admission has named (P6-D3's opened-once evidence), and |
| 859 | restores it from its pinned snapshot and anchor |
| 860 | (`genselect.SelectPrepared`). Every later attempt is cold into a fresh |
| 861 | generation whose baseline is chosen as for any cold admission: |
| 862 | `genselect.Select` from the predecessor, `holder.generation` as |
| 863 | observed, through the unfinished run (claim §6.4a item 3; moves §7.14's |
| 864 | restore row). So attempt three restores attempt two's generation with |
| 865 | everything it replicated, not `P`. Prepared authority is used once; |
| 866 | - **`KeepStamp`** on the first attempt, because no writer has been |
| 867 | admitted since the `installing` CAS issued `S'` (`holder.stamp > |
| 868 | holder.session`). Later attempts take a new stamp; |
| 869 | - **coverage, per admission (B1-1):** decided for the session the |
| 870 | admission's predecessor actually is, the last entry of |
| 871 | `holder.admissions`, never for the operation as a whole: |
| 872 | - `released` (L9) only on the admission that opens `P`, and only when |
| 873 | that predecessor is exactly the `{box, session}` whose release |
| 874 | receipt this op journaled and whose `final_txid` the pin reached. |
| 875 | That is a drain, a rebalance, or the nested relocation after a |
| 876 | `probe-failed`. Until L9 is accepted, relocation by drain is not |
| 877 | enabled (§8), because coverage `""` would declare a false loss on |
| 878 | every drain; |
| 879 | - `""` everywhere else: a failover (the admission records |
| 880 | `writes_lost` for `from`'s last session, `remote_max_txid` the pin's |
| 881 | `N`, H3's written-down loss), and every later probe attempt, whose |
| 882 | predecessor is the previous attempt's uncertified session. `to` |
| 883 | raises one alert per declaration; |
| 884 | - then, as for every cold admission: purge, restore the chosen baseline, |
| 885 | start, readiness, `generation.json`, then `PublishGeneration`, which |
| 886 | queues `park-generation` for the dead generations of this region in |
| 887 | the chain (`generation_from` at the first publication; an earlier |
| 888 | attempt's generation at a later one) (claim §6.4a item 5). A queued |
| 889 | park never touches a generation a later attempt still needs: the |
| 890 | baseline is always the newest generation, protected as the unfinished |
| 891 | run until its successor publishes (claim §6.2a); |
| 892 | - **limits:** at most three attempts within the probe deadline, as a |
| 893 | landing's (claim §6.4a item 7). A probe never marks the host failed; |
| 894 | running out of attempts is `probe-failed`. |
| 895 | |
| 896 | **A host with nothing replicated.** If the pin's `N` is 0 (the app never |
| 897 | committed a transaction), there is no prepared generation: claim |
| 898 | P6-D3 forbids a `[1, 0]` snapshot or a `[0, 0]` anchor. `installing` |
| 899 | then builds only the files manifest and local state, and the first |
| 900 | probe attempt is an ordinary cold admission into a fresh, empty |
| 901 | generation (claim §6.4a, "Empty bootstrap"). |
| 902 | |
| 903 | The test is the moves one, run for relocation: relocate, write, |
| 904 | replicate, terminate the new executor, cold-restore, and check that every |
| 905 | replicated write is present (`internal/sim/litestreamfacts`). Late writes |
| 906 | from the old installation land in `generation_from`, which nothing |
| 907 | admitted reads again, and which publication retires. |
| 908 | |
| 909 | **The no-reinstall boundary (A1-3).** Once the `installed` CAS succeeds: |
| 910 | |
| 911 | - nobody builds `P` again: replay from `installed` or `probing` goes |
| 912 | forward (the local install only if the executor has no receipt and no |
| 913 | admission has named `P`; then the probe, then `serving`), never back to |
| 914 | `installing`; |
| 915 | - `P` is opened at most once. After its first admission, every restart |
| 916 | takes a fresh generation from its predecessor; |
| 917 | - a worker or `to` that paused and resumes revalidates the phase under |
| 918 | the host guard, with the stamp it captured, and stops. |
| 919 | |
| 920 | Replay from `installing` (before `installed`) parks `to`'s partial local |
| 921 | state with `retire-artifacts` (claim §6.4), and rebuilds `P` from the |
| 922 | same pin. The build intent makes the bytes identical (moves §7.6). |
| 923 | Nothing has opened `P` yet, so nothing is lost. |
| 924 | |
| 925 | **Exclusion.** Only `op.exec` installs or admits, and only `to` is |
| 926 | `op.exec` from `installing` on. Every CAS is conditional on `op.id` and |
| 927 | the phase. `to` holds the replica lease while it builds. |
| 928 | |
| 929 | **Replacing a participant (A2-2, A2-7).** The claim allows a new |
| 930 | operation only from `serving`, so relocation defines its own exits (L1). |
| 931 | Each is one CAS, conditional on `op.id` and the phase, that cites its |
| 932 | evidence and names the replacement `to'`. A CAS that moves `op.exec` |
| 933 | writes the stamp under R6-4. Every rule that applied to the phase still |
| 934 | applies, and `to'` acknowledges its execution lease before the op leaves |
| 935 | that phase. |
| 936 | |
| 937 | | `to` stops being usable | Evidence | The op resumes at | |
| 938 | |---|---|---| |
| 939 | | terminated in `starting` or `releasing` | `to`'s termination fence | the **same** phase. The source's release requirements are unchanged: a drain still needs `from`'s stop evidence | |
| 940 | | terminated in `pinning` | `to`'s termination fence | `pinning` | |
| 941 | | terminated in `installing` | `to`'s termination fence | `installing`, with `op.exec: to'`, a new stamp and a fresh prepared id `P'`, from the journaled pin. The old `P` was never opened; the at-rest sweep parks it | |
| 942 | | terminated at `installed`, or in `probing` before its first `AdmitWriter` succeeded (no admission entry names `P`) | `to`'s termination fence | `installed`, with `op.exec: to'`, a new stamp, and no local install receipt. `to'` activates the stamp, makes its own local install from `P`'s files manifest and records its receipt (B1-2), then enters `probing`. `P` was never opened, so `to'` opens it | |
| 943 | | terminated after its first `AdmitWriter` | `to`'s termination fence | a nested failover from `pinning`, pinning from `holder.generation` as observed (`P` or a later fresh generation, with `to`'s replicated writes). Coverage `""` declares `to`'s sessions | |
| 944 | | alive, but its probe fails or misses the **probe deadline** (10 minutes) | `to`'s release sequence, in `probe-failed` | a nested relocation from `pinning`, pinned to at least `to`'s journaled `final_txid` and `manifest_seq`, so writes made during the probe are kept. `to` is excluded from the choice for this host | |
| 945 | |
| 946 | **Losing the source (B1-6).** A drain or rebalance whose `from` is |
| 947 | terminated before its stop receipt becomes a failover, in one CAS |
| 948 | conditional on `op.id` and the phase, citing `from`'s fence: |
| 949 | |
| 950 | | `from` terminated | The CAS | The op resumes at | |
| 951 | |---|---|---| |
| 952 | | in `starting`, acknowledged or not | journals `failover_fence` (key, SHA-256) and `reason: failover`, keeping the intent and every acknowledgement; removes `from` from `participants` and from the barrier; sets `op.exec: none`, keeping the stamp (claim D5's non-installing form); adds `from` to `lost`, and discharges its `park-local` chores only if its volume records qualify, otherwise leaves them open for an operator (§5.9, H19) | `starting`, waiting only for `to`'s acknowledgement; the next CAS goes to `pinning` with the fence as its evidence, skipping `releasing` | |
| 953 | | in `releasing`, before its receipt | the same | `pinning`, with the fence as evidence | |
| 954 | | after its receipt (`pinning` on) | `from` only goes on `lost`, and its chores are discharged if its volume records qualify, otherwise left open; nothing else changes | the same phase | |
| 955 | |
| 956 | The pin for a converted op follows the failover rule (fence plus 30 |
| 957 | seconds), and the first admission's coverage is `""`. |
| 958 | - **The bound.** After two replacements of `to` in one op, the placer |
| 959 | stops at the current phase's safe point (never between `installed` |
| 960 | and a probe) and marks the op `stuck`. It alerts, and the op waits for |
| 961 | `carlos ops host resolve <host>`. The host stays down, and the console |
| 962 | says why. |
| 963 | - A `probe-failed` target that cannot finish its release sequence (its |
| 964 | cgroup won't drain, claim §6.1, or its flush fails) also goes to |
| 965 | `stuck`. Nothing moves away from a live box without its stop evidence. |
| 966 | |
| 967 | **What a relocation leaves behind.** `generation_from` and the |
| 968 | predecessor run are retired by `PublishGeneration`'s `park-generation` |
| 969 | chores in this region. An abandoned `P` is parked by the at-rest sweep. |
| 970 | `from`'s local state is retired by its `park-local` chore, or discharged |
| 971 | with the fence (§5.9). Relocation installs into the region's host |
| 972 | namespace, so it waits for `quiet_after[region]` like a push (claim |
| 973 | §6.2a); for a live location that time has normally passed. |
| 974 | |
| 975 | **Cost of a drain:** one cut of a few seconds, like a move's (M2), with |
| 976 | no cross-region copy, plus one snapshot and anchor of the database in |
| 977 | the region (moves §7.3's T11 arithmetic, same region). Failover's gap is |
| 978 | detection plus restore (spec 3). |
| 979 | |
| 980 | ### 5.9 The termination fence |
| 981 | |
| 982 | **What counts (A1-1, A1-11, H10, H19).** A box is fenced by termination |
| 983 | when a regional worker, with its own read-only rights in the box's AWS |
| 984 | region, observes state exactly **`terminated`** for the instance id, |
| 985 | account (`OwnerId`) and region in the box's roster entry's `cloud` |
| 986 | (H11, HA7), by one of the three sources below. Nothing else is needed |
| 987 | (H19). This is claim §7.2's rule for execution, that the instance is |
| 988 | fenced by the cloud, without the operator: a terminated instance can |
| 989 | never run again; a disk that survives it cannot write to storage by |
| 990 | itself; and a disk attached to another instance is a different box, |
| 991 | with a different instance id and enrollment, never admitted as this |
| 992 | box's executor (claim C4, HA7). |
| 993 | |
| 994 | **Disk evidence, for discharging local chores only (H19, claim D5).** |
| 995 | Whether the dead box's `park-local` chores can be discharged without |
| 996 | retirement is a separate question, about cleanup, not about who may |
| 997 | write. A worker discharges them automatically, in the fence CAS, only |
| 998 | when the box's **volume records** (below) show its block device |
| 999 | mappings, from the first record taken at enrollment (the launch-time |
| 1000 | mappings) to the last taken while it ran, with every attached EBS volume |
| 1001 | `DeleteOnTermination: true` and no volume ever removed. Otherwise the |
| 1002 | chores stay open for an operator's destroyed-local declaration or the |
| 1003 | returning-box retirement (claim D2, D5). Nothing about recovery waits on |
| 1004 | them: the fence, the relocation and the census do not read volume |
| 1005 | records (§5.8, L5). |
| 1006 | |
| 1007 | This is not proof that no copy of the disk exists. A volume kept by a |
| 1008 | Recycle Bin rule, copied, snapshotted or imaged, or detached by an |
| 1009 | operator, can survive. That is a data-retention concern, handled by the |
| 1010 | operator path and the deployment's own retention settings, never a |
| 1011 | second writer: no such disk can act as the executor (above). The dev |
| 1012 | test of 2026-10-06 (§8 step 5) showed why the evidence must be recorded |
| 1013 | while the box runs: once `terminated`, `BlockDeviceMappings` was empty, |
| 1014 | the `DeleteOnTermination: true` root volume was already gone |
| 1015 | (`InvalidVolume.NotFound`), and the `false` volume survived as |
| 1016 | `available`. |
| 1017 | |
| 1018 | **The volume record.** Regional workers describe each enrolled box in |
| 1019 | their region while it runs: at enrollment, on every 10-minute sweep, and |
| 1020 | at once when a describe shows `shutting-down`. A worker writes a new |
| 1021 | record, create-only, whenever the mapping differs from the newest |
| 1022 | record, and otherwise at least once an hour: |
| 1023 | |
| 1024 | control/box-volumes/<enrollment>/<observed_at>.json |
| 1025 | |
| 1026 | ```json |
| 1027 | {"v": 1, "box": "<box reference>", "enrollment": "<enrollment>", |
| 1028 | "cloud": {"provider": "aws", "account": "123…", |
| 1029 | "region": "eu-west-1", "instance_id": "i-0abc…"}, |
| 1030 | "instance_state": "running", |
| 1031 | "block_devices": [{"device": "/dev/xvda", "volume_id": "vol-…", |
| 1032 | "delete_on_termination": true, "status": "attached"}], |
| 1033 | "removed_volumes": [], |
| 1034 | "observed_at": "<RFC3339>", "request_id": "<EC2 request id>", |
| 1035 | "written_by": {"box": "<reference>", "enrollment": "<enrollment>"}} |
| 1036 | ``` |
| 1037 | |
| 1038 | - A record is bound to the roster entry: its `cloud` must equal the |
| 1039 | entry's, and a record for another instance is ignored. |
| 1040 | - `removed_volumes` lists any volume the previous record had and this |
| 1041 | one lacks (a detach, or a replaced root volume). Any removal, or any |
| 1042 | `false` flag in any record, rules out automatic discharge for that |
| 1043 | enrollment for good, and alerts once so an operator knows. |
| 1044 | - A `shutting-down` read may add a record; it is never termination |
| 1045 | evidence. |
| 1046 | |
| 1047 | **What never counts.** |
| 1048 | |
| 1049 | - `stopped` and `stopping` do not count, because a stopped box can start |
| 1050 | again with its disk. A missing instance, `InvalidInstanceID.NotFound`, |
| 1051 | an API error, a partial reply, or a lookup in another account or |
| 1052 | region is **never** evidence. AWS keeps a terminated instance visible |
| 1053 | in `DescribeInstances` for typically under an hour, so absence proves |
| 1054 | nothing. All of those go to the operator: `carlos ops host resolve |
| 1055 | --declare-lost`, with claim §7.3 D2's binding, attestation and |
| 1056 | challenge as written (H10). |
| 1057 | - The worker reads the state itself. It never trusts a box's own report, |
| 1058 | an edges file, or anything a box publishes about itself. A volume |
| 1059 | record is a cloud read, not such a report: any regional worker may |
| 1060 | write one, the described box included, under the claim's trusted |
| 1061 | box-writer boundary (P5-D3). It decides only cleanup (above), so a |
| 1062 | wrong one can leave a disk unretired, never admit a writer. |
| 1063 | |
| 1064 | **The fence record.** The worker writes one record per terminated |
| 1065 | enrollment, create-only (L4): |
| 1066 | |
| 1067 | control/box-fences/<enrollment>.json |
| 1068 | |
| 1069 | ```json |
| 1070 | {"v": 1, "box": "<box reference>", "enrollment": "<enrollment>", |
| 1071 | "cloud": {"provider": "aws", "account": "123…", |
| 1072 | "region": "eu-west-1", "instance_id": "i-0abc…"}, |
| 1073 | "state": "terminated", |
| 1074 | "observed_at": "<RFC3339>", |
| 1075 | "via": "describe" | "event" | "activity", |
| 1076 | "request_id": "<EC2 request id, when via=describe>", |
| 1077 | "event_id": "<EventBridge event id, when via=event>", |
| 1078 | "written_by": {"box": "<reference>", "enrollment": "<enrollment>"}} |
| 1079 | ``` |
| 1080 | |
| 1081 | It is not AWS-signed: like claim D2's operator observation, it is the |
| 1082 | worker's authenticated report of a cloud read, made with rights the |
| 1083 | console does not hold. Every relocation and claim CAS that relies on it |
| 1084 | cites its key and SHA-256 (`evidence_refs`, claim P5-D3). |
| 1085 | |
| 1086 | **Three ways to observe `terminated`.** Any of these is the terminated |
| 1087 | read: |
| 1088 | |
| 1089 | 1. **The durable event.** An EventBridge rule sends EC2 "Instance |
| 1090 | State-change Notification" events with `state: terminated` for the |
| 1091 | deployment's account to an SQS queue (14-day retention). Plan 1a |
| 1092 | built one queue per box. The queue's policy accepts messages only from |
| 1093 | that rule's ARN. An event whose account, region and instance id match |
| 1094 | a roster entry's `cloud` is the read, `via: "event"`. |
| 1095 | 2. **A direct read.** `DescribeInstances` for the instance id, in the |
| 1096 | region, returning `terminated` with the deployment's `OwnerId`, |
| 1097 | `via: "describe"`. Workers make it on their sweep and whenever an |
| 1098 | event or the group's history suggests a termination. |
| 1099 | 3. **The group's own history.** For a box launched by an Auto Scaling |
| 1100 | group, `DescribeScalingActivities` for the region's group: an activity |
| 1101 | terminating that instance id with status `Successful`, |
| 1102 | `via: "activity"`. Spec 4 verifies on dev that `Successful` is |
| 1103 | recorded only once the instance has reached `terminated`; until then |
| 1104 | this source is off. |
| 1105 | |
| 1106 | **What is not guaranteed (A2-4).** EC2 delivers state-change events to |
| 1107 | EventBridge on a best-effort basis, and a terminated instance leaves |
| 1108 | `DescribeInstances` after about an hour. When no source shows |
| 1109 | `terminated`, there is **no automatic recovery**. The placer marks the box's |
| 1110 | hosts `awaiting-evidence`, alerts at once, and keeps retrying. The only |
| 1111 | way out is the operator's `--declare-lost` (claim §7.3 D2). A CloudTrail |
| 1112 | `TerminateInstances` record is **not** evidence (A3-1): it proves a |
| 1113 | request was made, and an instance can stay `shutting-down` after a |
| 1114 | successful one. The console may show it to help an operator find the |
| 1115 | instance, nothing more. If no terminal state can be read, the hosts stay |
| 1116 | `awaiting-evidence`. That is a liveness gap, accepted. Absence is never |
| 1117 | evidence, at any point. Spec 3 sets how long the placer retries before |
| 1118 | alerting, and owns the loop that reads the queue and history. |
| 1119 | |
| 1120 | **What the fence does (L2 to L5).** |
| 1121 | |
| 1122 | - **It is a lost-box fence** (claim C5, §4) for every host whose claim |
| 1123 | names the box. A relocation started on it may move `op.exec` off the |
| 1124 | box. |
| 1125 | - **The box goes on each affected claim's `lost` list**, with the fence's |
| 1126 | reference. A terminated instance cannot run again, so it needs no |
| 1127 | quarantine and no `--readmit` (H1). Its reference never names a |
| 1128 | working box again. |
| 1129 | - **Its local chores are discharged, if the disk evidence allows (H19).** |
| 1130 | When the volume records qualify (above), its `park-local` chores are |
| 1131 | discharged in the same CAS, by `CompleteObligation` semantics, without |
| 1132 | retirement: claim D5's destroyed-local exception (L3). The fence CAS |
| 1133 | cites the records it relied on. When they do not qualify, the chores |
| 1134 | stay open for an operator, and the relocation proceeds regardless: |
| 1135 | a dead box's `park-local` names only that box, which no relocation |
| 1136 | writes (claim CC11). Regional chores |
| 1137 | are reassigned to `region:<r>`, keeping their ids, operations and |
| 1138 | scopes (D5). Remote scopes it justifies take the fence's `observed_at` |
| 1139 | as their original evidence time (claim P5-D3, D5). |
| 1140 | - **The roster follows later.** The console moves the box's roster entry |
| 1141 | to `retired`, citing the record, when it next runs. Until then the |
| 1142 | record alone excludes the box from the census for lifecycle starts |
| 1143 | (L5), so failover never waits for a console. |
| 1144 | - **A console that ran on the box leaves the census too (B1-7).** Claim |
| 1145 | D7 requires a fresh report from every console able to mutate the |
| 1146 | namespace. A console enrollment gains `compute: {box, enrollment}`, |
| 1147 | the platform box its process runs on, set at console enrollment and |
| 1148 | checked against every console report, which that box's trusted writer |
| 1149 | publishes with its own `boot_id`. A console whose `compute` box has a |
| 1150 | valid fence record is excluded from the census exactly as the box is |
| 1151 | (L5). A console with no `compute` binding is never excluded this way. |
| 1152 | - **A stale console report (H18).** For a relocation a box starts, and |
| 1153 | nothing else, the census accepts a console's **newest** report however |
| 1154 | old, when that report met every D7 floor check (protocol minimum, |
| 1155 | authenticated release minimum, matching floor digest) and the |
| 1156 | console's `compute` box has reported `floor_guard_ready` with the same |
| 1157 | floor digest. Claim D1 makes an armed guard permanent (it never |
| 1158 | expires and cannot be lowered offline), so that box cannot run an |
| 1159 | older console binary underneath. A console with no `compute` binding, |
| 1160 | or whose box never reported the guard armed, still needs a fresh |
| 1161 | report. Every other D7 rule, and every console-started operation, is |
| 1162 | unchanged (L20). |
| 1163 | - **The loss is recorded twice.** The admission that takes over records |
| 1164 | `writes_lost` (§5.8), and the console copies the fence into claim D4's |
| 1165 | durable loss record when it next runs, with timer knowledge `partial`: |
| 1166 | wake state was published continuously, up to H3's window. |
| 1167 | |
| 1168 | ### 5.10 Relocation starts and executable updates (H14) |
| 1169 | |
| 1170 | Claim D7 serializes a lifecycle start against executable updates with a |
| 1171 | reservation the console writes into the roster. A relocation must start |
| 1172 | when the console is down, so H14 moves the serialization to the boxes. |
| 1173 | Each side records its own intent durably first, then checks the |
| 1174 | other's, so of two that race at least one sees the other. |
| 1175 | |
| 1176 | **The worker, at the start.** Before the start CAS, the worker reads |
| 1177 | `control/host-actors.json` as an exact object and verifies its root |
| 1178 | signature (an unreadable or unverifiable roster refuses the start). It |
| 1179 | requires `to`, and `from` unless the start cites a fence, to be |
| 1180 | `active`, and requires that no entry in `activations` names `from`'s or |
| 1181 | `to`'s enrollment. It records the revision it read as |
| 1182 | `directory_revision` in the intent. It writes nothing to the roster. |
| 1183 | |
| 1184 | **A participant, before it acknowledges.** A box becomes a participant |
| 1185 | in practice when it acknowledges the operation (claim D6). Before its |
| 1186 | first execution-lease PUT, its controller: |
| 1187 | |
| 1188 | 1. writes its `.hostexec/<host>.json` state as `pending`, durably (claim |
| 1189 | D6's existing state and write protocol); |
| 1190 | 2. then reads root's update state: the system-update journal and root's |
| 1191 | **update-begun** marker (below). If an update is begun or in progress |
| 1192 | for this box, it refuses to acknowledge, clears its `pending` state, |
| 1193 | and retries after the update finishes. The operation cannot leave |
| 1194 | `starting` without `to`'s acknowledgement, so nothing has moved; the |
| 1195 | worker waits, or replaces `to` (§5.8). |
| 1196 | |
| 1197 | **Root, before it starts an update.** Every path that begins an |
| 1198 | executable change (system apply, explicit rollback, a guarded |
| 1199 | activation under claim D1, the console's activation reservation being |
| 1200 | acted on), under root's existing system-update lock: |
| 1201 | |
| 1202 | 1. writes an **update-begun** marker, durably, beside the update journal |
| 1203 | (`/opt/carlos/system/guard/update-begun.json`, naming the update); |
| 1204 | 2. then scans `<DataDir>/.hostexec/` completely. **Any** controller |
| 1205 | record there, in any state (`pending`, `active` or `lost`), and any |
| 1206 | record root cannot read or parse, or a directory it cannot list, means |
| 1207 | the box may still be a participant: root removes its marker, refuses |
| 1208 | the update, and leaves the console's activation reservation held, to |
| 1209 | retry later (B1-5); |
| 1210 | 3. otherwise it proceeds. The marker stays until the update finishes or |
| 1211 | is recovered. |
| 1212 | |
| 1213 | Boot recovery of an update already begun before any participation needs |
| 1214 | no new check: no participant acknowledged meanwhile, because step 2 of |
| 1215 | the participant's side saw the marker. |
| 1216 | |
| 1217 | **When exclusion ends (B1-5).** A controller record is removed only by |
| 1218 | claim D6's clearing: a fresh claim read showing `serving`, `free`, or |
| 1219 | that the box no longer participates, **and** any cleanup the record |
| 1220 | requires complete. A `lost` record whose cleanup-only termination failed |
| 1221 | is never cleared: claim D6 keeps that deny, and root's refusal stays |
| 1222 | with it. A box whose processes may still run under an operation never |
| 1223 | takes an executable update. An operator who needs to update such a box |
| 1224 | first resolves the host (`carlos ops host resolve`). |
| 1225 | |
| 1226 | **The race.** If the console reserves an update for `to` after the |
| 1227 | worker's roster read, the start proceeds, and the reservation is |
| 1228 | harmless by itself. Whichever of `to`'s acknowledgement and root's |
| 1229 | begin writes its marker first, the other sees it and refuses: either |
| 1230 | the update waits until the operation returns to `serving` and D6 clears |
| 1231 | the controller's state, or the acknowledgement waits until the update |
| 1232 | finishes. Root's refusal is what closes the race (H14). A box stays a |
| 1233 | participant from its `pending` state until D6 clears its record, through |
| 1234 | `active` and `lost` alike, so an update never runs under a relocation's |
| 1235 | `installing`, `probing` or release, nor while a lost controller's |
| 1236 | cleanup is unfinished. |
| 1237 | |
| 1238 | This holds for every claim operation a box takes part in, not only |
| 1239 | relocations. L6 asks the claim to make it D7's rule for box-started |
| 1240 | operations. |
| 1241 | |
| 1242 | ## 6. Infrastructure and IAM (A1-12) |
| 1243 | |
| 1244 | - **Box role** keeps plan 1a's `ec2:DescribeInstances` with a condition |
| 1245 | on `ec2:Region`, and `autoscaling:DescribeScalingActivities` with a |
| 1246 | condition on `aws:RequestedRegion`, both equal to the box's region, |
| 1247 | plus SQS `ReceiveMessage`/`DeleteMessage` on its termination queue. |
| 1248 | Describe actions do not support resource-tag conditions, so deployment |
| 1249 | membership is **not** an IAM property: the verifier checks `OwnerId`, |
| 1250 | the region and the roster entry (§5.9). No `ec2:Stop*`, `Terminate*` or |
| 1251 | `Modify*`, and no Auto Scaling write rights. It also needs the claim's |
| 1252 | own grants: `s3:ListBucketVersions`, `s3:GetObjectVersion` and |
| 1253 | `s3:DeleteObjectVersion` on its region's data bucket (claim §6.2a), |
| 1254 | which `genselect` and the parker use. |
| 1255 | - **Disks (H19).** Nothing in IAM guards a box's disks for the fence's |
| 1256 | sake: rev 5.2's SCP, holds, volume-change role, guard attestation and |
| 1257 | service-role audit are withdrawn. The launch template and the |
| 1258 | `region-host` module keep `delete_on_termination = true` on every |
| 1259 | volume, which is cheap and makes automatic discharge of local chores |
| 1260 | the usual case (§5.9). The deployment should configure no Recycle Bin |
| 1261 | rule, Data Lifecycle Manager or AWS Backup policy that selects |
| 1262 | platform box volumes; one that does is a data-retention choice, and |
| 1263 | the operator path then owns their cleanup. The deployment accounts |
| 1264 | are Organizations members (organization `o-sgnzdj0z0c`, management |
| 1265 | account 665106592142; flagship 271376211898, dev 640012953448, checked |
| 1266 | 2026-10-06), which no longer matters to the fence. |
| 1267 | - **Auto Scaling enrollment (H16, H17, spec 4).** The box role gains one |
| 1268 | separate statement: `autoscaling:DescribeAutoScalingInstances` and |
| 1269 | `ec2:DescribeInstances`, read-only, conditioned on |
| 1270 | `aws:RequestedRegion` being one of the deployment's AWS regions (any of |
| 1271 | them, not only the box's own). Nothing else is granted across regions: |
| 1272 | every other statement keeps its own-region condition, and no write, |
| 1273 | queue or S3 right crosses regions. The code calls another region only |
| 1274 | to confirm membership for a region with no `active` box (§5.1), and |
| 1275 | the fence, volume records and relocation refuse any observation not |
| 1276 | made in the box's own region. |
| 1277 | - **Console role:** unchanged, no EC2 rights. |
| 1278 | - **S3 (A2-6).** Two kinds of write, kept apart: |
| 1279 | - **Create-only** (`If-None-Match: *`): content objects, |
| 1280 | `instances/<host>/*/files/*/objects/`, in the region's data bucket; |
| 1281 | box identity records, `control/boxes/`, volume records, |
| 1282 | `control/box-volumes/`, membership confirmations, |
| 1283 | `control/box-enrollment/`, and fence records, `control/box-fences/`, |
| 1284 | in the control bucket. |
| 1285 | - **Conditional replace** (`If-Match`, or `If-None-Match: *` for the |
| 1286 | first write): `instances/<host>/*/files/*/manifest.json` only. |
| 1287 | |
| 1288 | S3 enforces both in policy (A3-3): the data bucket denies `PutObject` |
| 1289 | on `…/objects/*` without `s3:if-none-match: *`, and on |
| 1290 | `…/manifest.json` without either `s3:if-match` or |
| 1291 | `s3:if-none-match: *`. Each Deny carries |
| 1292 | `s3:ObjectCreationOperation: true`, so it applies to `PutObject`, |
| 1293 | `CopyObject` and `CompleteMultipartUpload`, never to |
| 1294 | `CreateMultipartUpload` or `UploadPart`, which can't carry the headers |
| 1295 | (A4-2). The one writer, `hostfiles.Publish`, sends the headers and is |
| 1296 | tested against both refusals. The control bucket denies `PutObject` |
| 1297 | without `s3:if-none-match: *`, and `DeleteObject` and |
| 1298 | `DeleteObjectVersion`, on `control/boxes/*` (plan 1a's policy) and on |
| 1299 | `control/box-volumes/*`, `control/box-enrollment/*` and |
| 1300 | `control/box-fences/*`. Each Deny is scoped to its prefix only: a |
| 1301 | broad Deny on this bucket once broke every lease release. The |
| 1302 | `*.bind.json` statement plan 1a added is no longer needed (§19). Data |
| 1303 | retirement under `files/` is by exact version, through the parker, |
| 1304 | like every other retirement (claim C3). |
| 1305 | - **Tofu:** the `region-host` module has `delete_on_termination = true` |
| 1306 | on the root volume (plan 1a), and should have it on every other EBS |
| 1307 | volume a box attaches. It never affects the fence (H19): it only lets |
| 1308 | a dead box's `park-local` chores be discharged automatically (§5.9). Moving to an Auto Scaling group is spec 4; spec 1 runs on |
| 1309 | today's fixed boxes and on a second fixed box added by hand. |
| 1310 | |
| 1311 | ## 7. Failure modes |
| 1312 | |
| 1313 | | What happens | Result | |
| 1314 | |---|---| |
| 1315 | | Executor box dies, region has another box | spec 3 relocates its hosts after the termination fence. Until then: requests 503 with Retry-After. | |
| 1316 | | Executor dies, region had one box | the Auto Scaling group launches a replacement; an enrolled box in another region confirms its group membership (H17) and the console enrolls it. It is then the region's worker: it reads the termination event, writes the fence from that `terminated` read alone, and relocates every host to itself. The dead box's chores are discharged only if its volume records qualify (H19). | |
| 1317 | | A console's box is alive but its report is stale | a box-started relocation proceeds on the console's newest report if that met the floor and the box's update guard is armed (H18); otherwise it waits and alerts. | |
| 1318 | | The console's box is the one that died | failover does not need it: the fence, the census exclusion and every relocation CAS are a regional worker's (§5.9, L5, L6). | |
| 1319 | | Box partitioned from the control bucket | it keeps serving what it serves (no execution lease while `serving`, claim §7.1). It can't take new work. The group's health check decides whether it is terminated. | |
| 1320 | | Box alive but failing its health check | the group terminates it, then as above. Instances that were healthy lose up to H3's window, declared by `writes_lost`. | |
| 1321 | | A terminated box had a volume without `DeleteOnTermination`, or one was removed | fenced and relocated as usual; its `park-local` chores stay open for an operator (§5.9, H19). | |
| 1322 | | A copy of a dead box's disk survives (Recycle Bin, a copy, a snapshot) | a retention and cleanup matter for the operator; it can never act as the executor (§5.9, H19). | |
| 1323 | | A region's only box dies and its roster entry is still `active` | the cross-region confirmer reads it `terminated` and confirms the replacement; the replacement fences it (§5.1, B2-6). | |
| 1324 | | A participant's execution lease is lost and its cleanup fails | its controller record stays; root refuses executable updates on that box until the host is resolved (§5.10, B1-5). | |
| 1325 | | The source dies while a drain is in `starting` | the op becomes a failover in place (§5.8, B1-6). | |
| 1326 | | An update is reserved for `to` while a relocation starts | root refuses to begin it while `to` is a participant, or `to` refuses to acknowledge while it runs; never both running (§5.10). | |
| 1327 | | A forged or confused edges file names a terminated instance | ignored: identity comes only from the roster's `cloud`, proven at enrollment (HA7). | |
| 1328 | | Placer dies mid-relocation | another regional worker replays from the claim (§5.8). | |
| 1329 | | Target dies after its probe admitted | a nested failover from `holder.generation` (§5.8). | |
| 1330 | | Target alive but its probe keeps failing | stop evidence, then a nested relocation to another box; after two replacements, `stuck` and an alert (§5.8). | |
| 1331 | | No termination evidence can be found | `awaiting-evidence`, an alert, and the operator's `--declare-lost` (§5.9). | |
| 1332 | | Two files claim a host | the higher stamp wins (HA4). Two at the same stamp are reported (§18). | |
| 1333 | | Claim read fails on reconcile | keep existing rows, mint none (C6). | |
| 1334 | | `snapshot` app fails over | files lose up to the interval; the database loses about a second. Stated on Overview. | |
| 1335 | | A unit- or socket-backed route's box dies | not relocated (A1-8). It comes back with its box, or by the HA ladder's restore runbook, as today. | |
| 1336 | |
| 1337 | ## 8. Rollout (A1-9) |
| 1338 | |
| 1339 | 1. **Prerequisite:** the claim and its floor are live (claim §10), the |
| 1340 | roster exists with every box enrolled from `internal/boxid` evidence |
| 1341 | (§5.1), and every host in the region has a claim (adopted under claim |
| 1342 | §9 and P5-D2, which requires exactly one proven installation owner). |
| 1343 | 2. Ship §5 dormant behind `CARLOS_REGION_HA`, with capability |
| 1344 | `region-ha-1` published in each box's edges file once enabled. With one |
| 1345 | box per region, every rule is a no-op; ship it to all four regions and |
| 1346 | watch. |
| 1347 | 3. **The multi-box gate.** A second box may join a region only when: |
| 1348 | - every relocatable host there has a claim; |
| 1349 | - every box in the region, and every edge in the deployment, publishes |
| 1350 | `region-ha-1` and runs at or above the HA floor; |
| 1351 | - every box in the region is `active` in the roster with a proven |
| 1352 | `cloud`; |
| 1353 | - the region's termination queues exist. |
| 1354 | |
| 1355 | **Volume recording is never part of this gate (B4-1, H19).** Spec 1 |
| 1356 | ships the recorder with the rest of §5, and rollout checks once that |
| 1357 | it runs, but neither the gate, nor a box's start, nor row minting, |
| 1358 | nor relocation, nor recovery ever waits on a volume record. A |
| 1359 | recorder that cannot write raises an alert, and its only effect is |
| 1360 | that a box that dies meanwhile keeps its `park-local` chores open for |
| 1361 | an operator (§5.9). |
| 1362 | |
| 1363 | The HA floor joins the protocol floor (claim §10, D1): every path that |
| 1364 | restores an executable (system apply, rollback, boot recovery from |
| 1365 | `carlos.prev`) refuses a binary below it once a region has two boxes. |
| 1366 | The gate is checked in `carlos system` and in the console's box |
| 1367 | list. Each box re-checks at start only its runtime half: the |
| 1368 | capability and floor of the region's boxes and edges, and their |
| 1369 | roster entries. A box that finds a second box in its region without |
| 1370 | that half satisfied refuses to mint relocatable rows and alerts. The |
| 1371 | termination queues are a rollout check only: a missing queue loses |
| 1372 | one trigger, and the describe still fences (§5.9). |
| 1373 | 4. **Relocation by drain** stays off until the claim accepts L9; failover |
| 1374 | relocation needs L1 to L6. |
| 1375 | 5. Rehearse on dev with two fixed boxes: create spreads across both; |
| 1376 | drain one (§5.8) with `moveprobe` shaped like ips (moves §14.4), |
| 1377 | including timers and a `continuous` files app; terminate one by hand |
| 1378 | and run the spec 3 placer manually. **Done on 2026-10-06 (H15):** a |
| 1379 | t4g.nano launched with a `DeleteOnTermination: true` root and a |
| 1380 | `DeleteOnTermination: false` extra volume, then terminated. While |
| 1381 | `shutting-down`, `DescribeInstances` still listed both volumes, with |
| 1382 | their flags and status `attached`. Once `terminated`, |
| 1383 | `BlockDeviceMappings` was `[]`, the root volume was already gone |
| 1384 | (`InvalidVolume.NotFound`), and the `false` volume survived as |
| 1385 | `available`. So disk evidence comes from records taken while the box |
| 1386 | runs, and decides only the discharge of local chores (H19, §5.9). The |
| 1387 | rehearsal terminates a box with an all-`true` mapping (chores |
| 1388 | discharged) and one with a `false` volume (fenced and relocated |
| 1389 | alike, chores left open). |
| 1390 | The rehearsal also enrolls a box into a dev region with no `active` |
| 1391 | box, confirmed from another region (H17), and checks that the |
| 1392 | cross-region statement allows only the two describe calls. |
| 1393 | 6. Then specs 2, 3, 4. |
| 1394 | |
| 1395 | ## 9. Changes requested of the claim and moves specs |
| 1396 | |
| 1397 | For platform-1e (claim rev 6.25, 413b12cb) and platform-d8 (moves rev |
| 1398 | 7.17, 39904c84). They replace rev 4.2's K1 to K11, which those sessions |
| 1399 | never received. "Ruled" means Paul's rulings of 2026-10-06 (H10 to H19, |
| 1400 | and amadan #236) decide the direction; the owning session still writes |
| 1401 | the text. |
| 1402 | |
| 1403 | | # | Amends | Change | Status | |
| 1404 | |---|---|---|---| |
| 1405 | | L1 | claim §4, §8, §6.4a (which kinds and phases admit), §7.3 D5 and D6 | New op kind `relocating` (§5.8): same region, stateful, from `serving`; its phase table with `op.exec`, stamp, rows and admission per phase; the `installed` no-reinstall boundary for the remote build, with a per-box local install receipt, bound to the stamp of the CAS that handed the box `op.exec`, required before `probing` and never remade after the first admission; probe attempts after the first restoring from their predecessor; the replacement table; the source-loss conversion to failover from `starting` or `releasing`; two replacements then `stuck`; the release sequence. Recovery owner `region:<r>`, any regional worker (P6-D5). The D6 barrier needs only `to` when the start cites a termination fence or after a source-loss CAS. Excludes moves, placements and pushes. | Ruled in direction (H10; #236: "a new op kind, or an equivalent that the claim prefers") | |
| 1406 | | L2 | claim §4, C5, §7.2, §7.3 D2 | A termination fence (§5.9) is a lost-box fence for execution, written by a regional worker with no person in the loop: a verified `terminated` read (event, describe or, later, the group's history), bound to the roster entry's `cloud`. No disk condition. A `shutting-down` read is never evidence. Stopped, unclear and vanished boxes keep D2's operator path unchanged. | Ruled (H10, H19) | |
| 1407 | | L3 | claim §7.3 D5 | A fenced terminated box's `park-local` chores are discharged automatically in the fence CAS only when its volume records, from enrollment to its last running record, show every attached EBS volume `DeleteOnTermination` and none removed; that is D5's destroyed-local proof for discharge. Otherwise they stay open for D2/D5's operator path, and no recovery step waits on them. Regional chores move to `region:<r>` as D5 says. | Ruled (H19) | |
| 1408 | | L4 | claim §7.3 D3 and D4, §5.9 P5-D3 | The fence record `control/box-fences/<enrollment>.json`, create-only, box-written, cited by `evidence_refs`. A terminated box goes on `lost` with that reference and no quarantine revision; it needs no readmission. The console later copies it into a D4 loss record and retires the roster entry. | Ruled in direction (H10, H1); record format for the claim session | |
| 1409 | | L5 | claim §10.1 D7 (required actors, console enrollments) | An enrolled box with a valid fence record is excluded from the census without waiting for the console to retire it. A console enrollment gains `compute: {box, enrollment}`, checked against its reports; a console whose `compute` box is fenced is excluded the same way (B1-7). | Ruled (H10, H14; #236: failover works when the console's box died) | |
| 1410 | | L6 | claim §10.1 D7 (lifecycle reservations, `actors_hold`), §7.3 D6 | For operations a box starts, replace the console reservation with §5.10's box-side check: the worker reads (never writes) the roster and refuses if an update is reserved for `from` or `to`; a participant writes its `pending` controller state, then refuses to acknowledge while root has an update begun; root writes its update-begun marker, then refuses an update while any controller record exists in any state, `lost` included, or cannot be read. A record is removed only by D6's clearing with its cleanup complete (B1-5). | Ruled (H14) | |
| 1411 | | L7 | claim §10.1 D7, §7.3 D2, §5.9 P5-D2, §6.6 P6-D2 | Enrollment of a platform box requires `internal/boxid` evidence; it records `cloud` from it, and checks the tofu `instance_id` for a reference already in claims or edges files. No other roster, and no `control/boxes` authority. | Ruled (H11) | |
| 1412 | | L8 | claim §5.1 steps 2 and 4, §5.9 P5-D2 and P5-D6 | The executor in the create intent is `choose()` (§5.2), with P5-D2's SHA-256 as the final tie-break; chosen once, never substituted; a create whose box can't complete is cancelled. | Ruled (H12) | |
| 1413 | | L9 | claim §6.4a item 2, `hostclaim.AdmitWriter` | A coverage value `released`, legal only on the admission that opens a prepared generation built from a pin reaching the `final_txid` a released box recorded from its stopped database, and only when the admission's predecessor (the last `holder.admissions` entry) is exactly that box and session, under its stop receipt. It declares nothing. Without it every drain declares a false loss (B1-1). | Requested | |
| 1414 | | L10 | claim §6.1, §6.6 P6-D1, `internal/stopevidence` | A writers-stopped check for release step 1: cgroup empty, no member processes, and database holders limited to the agent's own identified replicator descriptors. `Collect` stays unchanged for step 4. | Requested | |
| 1415 | | L11 | claim §4 (CC11), `hostclaim.StartGuard` | A `park-generation` chore in the region does not block a `relocating` start there: relocation writes only its own new prepared generation, never one being retired. Under P6-D4 nearly every recently woken host has one, which would hold failover up to the park horizon. | Requested | |
| 1416 | | L12 | claim C4, §8 (CC10), §6.6 P6-D2; moves §9.6 | Reconcile's row cases for `relocating` (§5.3): held participant rows; a former executor's row kept held until its `park-local` retires it; the 10-minute sweep as a stated exception to C4 for kept rows (H9). | Requested | |
| 1417 | | L13 | claim C3, §6.2a, §6.6 P6-D3 | The files publisher is a second ordinary-replication exception, bound to `{location, generation, session}`. `files/<generation>/` belongs to that generation for the classifier; `park-generation` retires it except objects named by the files live set: every manifest from the files pin through `holder.generation`, and any manifest a running operation's pin journals (B2-4). | Requested | |
| 1418 | | L14 | moves §7.10 | A target terminated mid-move may be taken off the move by the termination fence, as M16's evidence, automatically, under the same before-commit rules. Moves decides what the move does then; the placer only supplies the fence. | Ruled (H10) | |
| 1419 | | L15 | moves §7.1, §7.3, §19.1 | A move into a multi-box region names its target **box** in the start CAS, chosen by `choose()`. Readiness, evidence, install and recovery bind to that box; replacing it is an explicit fenced transition (A1-10). | Ruled (H12) | |
| 1420 | | L16 | moves §5.1, §7.6, P6-D3's provider | Relocation's install reuses the landing's `building` and `published` phases and the prepared-generation authority, with a pinned set of remote versions as the source in place of a staged replica. One shared routine, so both get the same anchor and manifest rules. | Requested | |
| 1421 | | L17 | moves §6.1, §8, §9.4 | Every installation, a move's landing included, writes a files manifest in its new generation (§5.6), so unchanged files stay recoverable after the next failure; the archive at the cut stays. Timer freezing and draining apply to `relocating` holds. | Requested | |
| 1422 | | L18 | joint model, moves §14.2 | `internal/sim/hostlife` extends to two boxes per region: create spread, relocation by drain, termination mid-serve and in every relocation phase, nested failover, a stale edges file, an unenrolled reference, a volume without `DeleteOnTermination`, a crash mid-install, late file uploads and late PUTs after termination. | Requested | |
| 1423 | | L19 | claim §10.1 D7 (enrollment), with P3 | An Auto Scaling box is enrolled by the console with no operator when its `boxid` evidence verifies (deployment account, the region's AWS region) and a regional worker's create-only confirmation shows it a current member of the region's group (§5.1). Every other enrollment stays operator-approved. Spec 4 builds it. For a region with no live enrolled box (every enrollment there read `terminated`, positively, in the same pass, B2-6), any `active` box of another region confirms, by the same create-only write, bound to the group ARN the roster records for that placement; its cross-region rights are the two read-only describes and nothing else (H17). | Ruled (H16, H17) | |
| 1424 | | L20 | claim §10.1 D7 (required actors, console reports) | For box-started relocations only, a console's newest report counts however old if it met every floor check and the console's `compute` box has reported its D1 root update guard armed with the same floor digest. Every other census rule is unchanged. | Ruled (H18) | |
| 1425 | |
| 1426 | **Rev 4.2's requests, mapped.** K1 is L8. K2 is L1, with L9 to L11. K3 |
| 1427 | is L2 to L5. K7 is L14. K8 is L16 and L17. K9 is L18. K10 is L15. K11 is |
| 1428 | L12. Three are dropped because the claim now satisfies them: |
| 1429 | |
| 1430 | - **K4** (claim CAS rights for placers): boxes already write claims |
| 1431 | (`AdmitWriter`, `PublishGeneration`, chores), and P6-D5 gives |
| 1432 | `region:<r>` work to any enrolled worker. L1 names the owner. |
| 1433 | - **K5** (adoption only while a region has one box): P5-D2 requires |
| 1434 | exactly one proven installation owner and refuses zero or several, |
| 1435 | which is the property K5 wanted, whatever the box count. |
| 1436 | - **K6** (box names bound by `control/boxes`): claim §2 rev 6.16 makes |
| 1437 | references opaque, and H11 makes the roster the binding (L7). |
| 1438 | |
| 1439 | The disposition tables (§13 to §17) still cite K numbers. They refer to |
| 1440 | rev 4.2's table; this mapping says where each now lives. |
| 1441 | |
| 1442 | ## 10. Testing |
| 1443 | |
| 1444 | - **Unit:** enrollment evidence (signature, account, region, the tofu |
| 1445 | check for an existing reference, a reference held by another |
| 1446 | instance); `eligibleBoxes` (one box at a time, staleness, draining, |
| 1447 | roster state, `lost`, an owed `park-local`, capability, the expected |
| 1448 | region from the roster); `choose` (spread, fewest hosts, the SHA-256 |
| 1449 | tie-break); home-map stamp resolution; the files manifest (immutable |
| 1450 | objects, tombstones, late uploads, `If-Match` races, a cold admission's |
| 1451 | copied manifest naming the predecessor's objects); timer catch-up; |
| 1452 | termination-fence acceptance (NotFound, wrong account, wrong region, |
| 1453 | stopped, all sources empty, an event alone, a describe alone, a |
| 1454 | `shutting-down` read offered as evidence, and the fence accepted with |
| 1455 | no volume record at all); chore discharge (all records `true`: |
| 1456 | discharged; a `false` flag, a removed volume or no record: left open, |
| 1457 | with the relocation unaffected); a box starting while |
| 1458 | `control/box-volumes/` cannot be written: it mints its relocatable |
| 1459 | rows, takes part in relocation and recovers a dead peer's hosts, |
| 1460 | and only alerts (B4-1); the volume recorder (writes on change, |
| 1461 | at least hourly, `removed_volumes` when a volume leaves the mapping); |
| 1462 | cross-region confirmation (accepted only when the target region has no |
| 1463 | `active` box, refused for another group, an instance not |
| 1464 | `Pending`/`InService`, a confirmer not `active`, a region mismatch, a |
| 1465 | second confirmer losing the create-only race, and an emptied region |
| 1466 | whose dead box is still `active`: confirmed on a `terminated` read, |
| 1467 | refused on NotFound or `stopped`, B2-6); the console census for a box-started |
| 1468 | relocation (a stale report accepted with an armed guard, refused with |
| 1469 | no `compute` binding, no armed-guard report or a floor mismatch, and |
| 1470 | unchanged for a console-started operation); continuous files with a |
| 1471 | file held open (published at the deadline), inotify overflow, a file |
| 1472 | that changes during its read; the create-only and replace split in |
| 1473 | `hostfiles.Publish`. |
| 1474 | - **Relocation:** every row of the replacement table, the |
| 1475 | two-replacement bound, `probe-failed` without stop evidence, both |
| 1476 | participants' rows through every phase, the first probe opening `P` |
| 1477 | with `KeepStamp` and a restarted probe taking a fresh generation, and |
| 1478 | relocate → write → replicate → terminate → cold restore with the |
| 1479 | level-0 anchor; a release with a live embedded replicator (step 1 |
| 1480 | passes with only its descriptors, step 4 passes only after `DB.Close`, |
| 1481 | any other holder fails step 1); a failover admission declaring |
| 1482 | `writes_lost` for `from`'s session and a drain declaring nothing; the |
| 1483 | start refused when the roster reserves an update for `from` or `to`; |
| 1484 | a files update published by a probe session before its first database |
| 1485 | upload, then termination: the files pin keeps it while the database |
| 1486 | baseline falls back (B2-4); a crash during `probing` after the second |
| 1487 | admission: no reinstall, and the receipt still valid by |
| 1488 | `install_stamp` (B2-5); probe attempts one, two and three, with attempt three restoring |
| 1489 | attempt two's replicated writes and declaring attempt two's session |
| 1490 | (B1-1); a target terminated at `installed` replaced by one that makes |
| 1491 | its own local install before probing, and a `probing` CAS refused with |
| 1492 | another box's receipt (B1-2); `from` terminated in `starting` before |
| 1493 | and after acknowledging (B1-6). |
| 1494 | - **Starts and updates (§5.10):** root's begin and a participant's |
| 1495 | acknowledgement interleaved at every point, with a crash between each |
| 1496 | marker and its check: never an update and an acknowledged |
| 1497 | participation at once, and a crash leaves at most a refusal, cleared |
| 1498 | by retry; a lease lost during `probing` with cleanup failing, and an |
| 1499 | unreadable controller record, both refusing the update (B1-5). |
| 1500 | - **Integration:** two agents against one fake region, one control |
| 1501 | bucket: create lands on the chosen box only; the other box mints no |
| 1502 | row; the edge backhauls to the executor; relocate by drain through |
| 1503 | every phase with a crash injected at each; the old box's local copy is |
| 1504 | never served after it becomes executor again (claim R6). |
| 1505 | - **Joint model (L18)** decides readiness, as it does for the claim and |
| 1506 | moves. |
| 1507 | - **Rehearsal** (§8 step 5). |
| 1508 | |
| 1509 | ## 11. Open questions |
| 1510 | |
| 1511 | None. |
| 1512 | |
| 1513 | Rev 4.2's three were ruled as H7, H8 and H9 (2026-10-04). Rev 5's first |
| 1514 | three were ruled on 2026-10-06 as H14 (the box-side reservation check, |
| 1515 | §5.10), H15 (the running-time volume record, since superseded by H19) |
| 1516 | and H16 (automatic enrollment, §5.1, L19). |
| 1517 | |
| 1518 | Astra's rev 5 review raised two more, ruled the same day as H17 (an |
| 1519 | enrolled box in another region confirms the first box of an emptied |
| 1520 | region, §5.1, §6, L19) and H18 (a box-started relocation accepts a |
| 1521 | console's newest report when its box's update guard is armed, §5.9, |
| 1522 | L20). Astra's rev 5 round 3 found more ways around the volume guard; |
| 1523 | Paul answered with H19, which takes disk evidence out of the failover |
| 1524 | path (§5.9, §22). |
| 1525 | |
| 1526 | ## 12. Out of scope, for the later specs |
| 1527 | |
| 1528 | - **Spec 2:** `headroom` thresholds, soft spread across an app's |
| 1529 | instances, the `unplaced` state and its alert, rebalancing. |
| 1530 | - **Spec 3:** the placer loop: reading the termination queue, relocating |
| 1531 | every host off a terminated box in bounded parallel, waking always-on |
| 1532 | and scheduled instances, the failover gap target. |
| 1533 | - **Spec 4:** the Auto Scaling group per region, launch template and |
| 1534 | join without SSM hand steps, automatic enrollment of new boxes (H16, |
| 1535 | L19), |
| 1536 | ingress for boxes that come and go (a per-region NLB with fixed EIPs, |
| 1537 | or an EIP pool claimed at boot), HTTP-01 certificates, steering, |
| 1538 | scale-out on headroom, scale-in by drain (§5.8), per-region minimum |
| 1539 | (H5). |
| 1540 | - **Not planned:** relocating unit- and socket-backed routes (the |
| 1541 | console and platform services). Their HA is the HA ladder's, separately. |
| 1542 | |
| 1543 | ## 13. Astra round 1 dispositions (rev 2) |
| 1544 | |
| 1545 | | Finding | Disposition | |
| 1546 | |---|---| |
| 1547 | | A1-1 P1 termination evidence not bound to identity | accepted: AWS-signed box records (§5.1), HA7, fence fields (§5.9), K6 | |
| 1548 | | A1-2 P1 new generation never populated | accepted: pinned recovery set and `installing` builds `G'` (§5.8), K8 | |
| 1549 | | A1-3 P1 replay overwrites an installation that has written | accepted: phase table, `installed` boundary, participant-terminated exits (§5.8), K2 | |
| 1550 | | A1-4 P1 file update and tombstone order | accepted: immutable objects, per-host `seq`, `If-Match` manifest, snapshot boundary (§5.6) | |
| 1551 | | A1-5 P2 `active_stamp` is not install evidence | accepted: installation-complete marker (§5.5), HA3 | |
| 1552 | | A1-6 P2 participant rows and former executors | accepted: three row cases (§5.3), K11 | |
| 1553 | | A1-7 P2 timers not reconstructible | accepted: timer state replicated continuously, imported with `ReplaceTimers`, catch-up (§5.7) | |
| 1554 | | A1-8 P2 unit and socket hosts | accepted: not relocatable (§2, §7, §12) | |
| 1555 | | A1-9 P2 adoption alone does not admit a second box | accepted: capability, HA floor and multi-box gate (§8), unstamped entries (§5.4) | |
| 1556 | | A1-10 P2 moves target box | accepted: K10 | |
| 1557 | | A1-11 P2 termination evidence can vanish | accepted: EventBridge to SQS with 14-day retention, NotFound never evidence (§5.9) | |
| 1558 | | A1-12 P2 Describe permissions by tag | accepted: `ec2:Region` condition, membership checked by the verifier (§6) | |
| 1559 | | A1-13 P3 citations | accepted: `lease.go:37`, `reconcile.go:202`, `edgesfile.go:272`/`:326` | |
| 1560 | |
| 1561 | ## 14. Astra round 2 dispositions |
| 1562 | |
| 1563 | | Finding | Disposition | |
| 1564 | |---|---| |
| 1565 | | A1-1 residual: a legacy label's first registration | accepted: an operator's bind checked against the region's tofu `instance_id` output and the claims' existing label (§5.1) | |
| 1566 | | A1-7 residual: `routes.next_due` | accepted: wake state carries `NextDue` and `Replication` with the timer rows (§5.6, §5.7) | |
| 1567 | | A2-1 P1 snapshot-only baseline loses the replication position | accepted: moves rev 7.6's replication baseline with the level-0 anchor, verified before `installed`, and its test (§5.8), K8 | |
| 1568 | | A2-2 P1 early target replacement skips release and pinning | accepted: phase-specific replacement table (§5.8), K2 | |
| 1569 | | A2-3 P2 reconcile drops the relocating source | accepted: per-phase rows for both participants (§5.3, §5.8 table) | |
| 1570 | | A2-4 P2 event delivery is best-effort | accepted: a third method (Auto Scaling activity history, verified in spec 4 first), `awaiting-evidence` with an immediate alert, and operator resolution via CloudTrail when all are empty (§5.9) | |
| 1571 | | A2-5 P2 close and rename don't bound continuous sync | accepted: `IN_MODIFY` with a fixed one-second deadline from the first modification, recursive watches, overflow scans, a stable-read check (§5.6) | |
| 1572 | | A2-6 P2 create-only conflicts with manifest replacement | accepted: create-only and conditional-replace writes separated, enforced in one writer (§6) | |
| 1573 | | A2-7 P2 a live target whose probe never passes | accepted: a probe deadline, `probe-failed` with stop evidence, a nested relocation that keeps probe writes, and `stuck` after two replacements (§5.8), K2 | |
| 1574 | |
| 1575 | ## 15. Astra round 3 dispositions |
| 1576 | |
| 1577 | | Finding | Disposition | |
| 1578 | |---|---| |
| 1579 | | A2-5 residual: an unstable file waits for the scan; a failed watch | accepted: published at every deadline as the last image, flagged; per-second polling while coverage is missing (§5.6) | |
| 1580 | | A2-7 residual and A3-2 P1: probe failure skips the target's durability barrier | accepted: one release sequence for the source and a failed target, journaling `final_txid` and `manifest_seq` before stop evidence, all files flushed whatever the plan; pinning must reach them (§5.8), K2, K8 | |
| 1581 | | A1-1 residual and A3-3 P1: the bind isn't operator-authenticated | accepted: binds name the exact record version; the control bucket's policy limits binds to operator roles and denies replace and delete under `control/boxes/` (§5.1) | |
| 1582 | | A3-1 P1: a CloudTrail request is not compute fencing | accepted: claim §7.2's terminal-state rule unchanged; CloudTrail is a finding aid only; otherwise `awaiting-evidence` (§5.9), K3 | |
| 1583 | | A3-3 (part): S3 can enforce conditional headers | accepted: `s3:if-none-match` and `s3:if-match` policy conditions, scoped per prefix (§6) | |
| 1584 | | A3-4 P3: citations | accepted: `internal/host/fleetbind.go`, `edge.go:2432`, `homemap.go:538`/`:582`, `reconcile.go:335`/`:799` | |
| 1585 | |
| 1586 | ## 16. Astra round 4 dispositions |
| 1587 | |
| 1588 | | Finding | Disposition | |
| 1589 | |---|---| |
| 1590 | | A4-1 P1: the release sequence flushes before stopping writers | accepted: stop and empty the process tree and timer deliveries first, then flush and journal, then finish stop evidence (§5.8), K2 | |
| 1591 | | A4-2 P2: the conditional Deny blocks multipart preparation | accepted: `s3:ObjectCreationOperation: true` on every header Deny; multipart completion carries the header; tested (§6) | |
| 1592 | | A4-3 P3: 29245f69 is not moves rev 7.6 | accepted: rev 7.6 named as an uncommitted draft, rev 7.5 at cdc98845 as the last committed; the plan pins the commit (header) | |
| 1593 | |
| 1594 | ## 17. Astra round 5 dispositions |
| 1595 | |
| 1596 | | Finding | Disposition | |
| 1597 | |---|---| |
| 1598 | | A5-1 P1: step 1 waits for the replicator's handle, closed only in step 4 | accepted: step 1 allows only the box's own replicator descriptors; step 4 closes it and runs the unrestricted check; tested with a live replicator (§5.8), K2 | |
| 1599 | |
| 1600 | ## 18. Notes from plan 1a (2026-10-05) |
| 1601 | |
| 1602 | Plan `docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md` builds |
| 1603 | the parts of this spec that need no claim. It keeps every property above |
| 1604 | and changes three mechanisms: |
| 1605 | |
| 1606 | 1. **Exact content by hash, not by S3 version id (§5.6, §5.1).** The store |
| 1607 | abstraction has no version-id read and the box role has no |
| 1608 | `s3:GetObjectVersion`. Objects are immutable, their key contains their |
| 1609 | SHA-256, the manifest records it, and a reader verifies it. A bind |
| 1610 | names its record by `record_sha256` in place of `record_version_id`. |
| 1611 | 2. **The identity signature (§5.1).** The record carries IMDS's |
| 1612 | base64 RSA-SHA256 signature (`/latest/dynamic/instance-identity/signature`), |
| 1613 | verified with the regional certificate, in place of the PKCS7 form. |
| 1614 | Same document, same signer, no new dependency. |
| 1615 | 3. **`snapshot` is a periodic full publish, not a sealed archive (§5.6).** |
| 1616 | `hostfiles.SyncDir` stages every difference between the directory and |
| 1617 | the manifest. Run on the snapshot interval it gives the `snapshot` |
| 1618 | setting; run every ten minutes it is the continuous setting's repair |
| 1619 | scan; run before stop evidence it is the release flush. The manifest |
| 1620 | therefore has no `base` or `base_seq`, and restore reads one manifest |
| 1621 | and the objects it names. |
| 1622 | |
| 1623 | Two corrections to the spec, found while planning: |
| 1624 | |
| 1625 | - **§5.4's alert for two unstamped entries is dropped.** Static sites and |
| 1626 | public address routes run on every box, so every box publishes them as |
| 1627 | `home`, unstamped: that is normal, and the edge cannot tell them from |
| 1628 | relocatable hosts. The edge reports only two boxes at the same positive |
| 1629 | stamp. Plan 1b decides whether the publisher marks relocatable entries |
| 1630 | so the edge can alert on those alone. |
| 1631 | - **§5.1's region check has two halves.** `boxid.Verify` proves which |
| 1632 | region a box is in. Whoever uses that compares it with the region it |
| 1633 | expects: `eligibleBoxes` takes the placement's AWS region and refuses a |
| 1634 | box whose signed region differs, and `termfence.Verify` compares the |
| 1635 | observation's region with the signed one. Plan 1b must supply the |
| 1636 | expected region from something better than a majority vote of the |
| 1637 | unsigned edges files. |
| 1638 | - Build tags (Task 6): the descriptor-based directory code is `//go:build unix`, not `linux`, so the CLI's macOS build uses it too; Windows gets a stub that publishes nothing. |
| 1639 | |
| 1640 | Two limits to lift in plan 1b: a file over 64 MiB is recorded in the |
| 1641 | manifest as skipped and is not published (the store takes a whole body in |
| 1642 | memory), and the third termination-evidence source (Auto Scaling activity |
| 1643 | history) is not built until spec 4 verifies it. |
| 1644 | |
| 1645 | ## 19. What rev 5 means for plan 1a (2026-10-06) |
| 1646 | |
| 1647 | Plan 1a (aadb860) was built against rev 4.2. Rev 5 keeps most of it and |
| 1648 | changes how some of it is used. Plan 1b is written against rev 5. |
| 1649 | |
| 1650 | **Superseded.** |
| 1651 | |
| 1652 | - **`control/boxes/<id>.json` as authority (H11).** `boxid.Load` and |
| 1653 | `boxid.Verifier` are now used once, at enrollment, and their result is |
| 1654 | stored in the roster entry's `cloud`. `eligibleBoxes` takes its |
| 1655 | verified identities from the roster, not from loading records. The |
| 1656 | record object stays as the enrollment evidence, create-only, and grants |
| 1657 | nothing. |
| 1658 | - **The operator bind** (`Bind`, `control/boxes/<label>.bind.json`, `carlos ops |
| 1659 | box bind`, plan 1b item 13). Its tofu check moves into enrollment |
| 1660 | (§5.1). The bucket policy's bind statement is harmless and can go at |
| 1661 | the next apply. |
| 1662 | - **`WarmCopyTrusted` and the activator's warm path (H13).** Nothing skips |
| 1663 | a restore, so plan 1b item 4 is dropped. The `InstallMarker` type may |
| 1664 | serve as relocation's last local install journal entry, as moves' |
| 1665 | `.files-gen` does, or be removed; nothing reads it to trust a local |
| 1666 | database. |
| 1667 | - **`chooseBox`'s last tie-break.** The lowest box id becomes P5-D2's |
| 1668 | greatest SHA-256 of `host + "\x00" + reference` (H12). The first two |
| 1669 | rules stand. |
| 1670 | - **`termfence`'s binding and inputs.** `Verify` binds to a |
| 1671 | `boxid.Identity`; it must bind to the roster entry's `cloud` and |
| 1672 | produce §5.9's record. It takes no volume input (H19); a separate |
| 1673 | function decides chore discharge from the volume records. `FromDescribe` and `FromEvent` stay as the two terminated |
| 1674 | reads; a `shutting-down` observation is never accepted. Plan 1b adds |
| 1675 | the volume recorder, the discharge check, §5.10's two checks and, for |
| 1676 | spec 4, the membership confirmation (H16). |
| 1677 | |
| 1678 | **Stays as built.** |
| 1679 | |
| 1680 | - `internal/boxid`'s record, signature check and embedded regional |
| 1681 | certificates (verified on the dev box), as enrollment evidence. |
| 1682 | - The edges file's `box_id` (the label, which is the reference under |
| 1683 | #212 option B), `capabilities`, `draining` and `headroom`. |
| 1684 | - The home map settling two claims by installation stamp, and reporting |
| 1685 | equal stamps. |
| 1686 | - `eligibleBoxes`' one-box-at-a-time predicate, with the roster as its |
| 1687 | source. |
| 1688 | - `internal/hostfiles`: the ordered manifest, immutable hashed objects, |
| 1689 | `SyncDir`, the continuous publisher, the watcher, restore and wake |
| 1690 | state. Plan 1b adds the generation binding, the copied manifest at |
| 1691 | each cold admission, and retirement by the claim (§5.6). |
| 1692 | - `registry.ReplaceTimers`. |
| 1693 | - The infrastructure: one termination queue per box, root volumes deleted |
| 1694 | on termination, the create-only `control/boxes/` policy. The fence |
| 1695 | prefix `control/box-fences/` needs the same create-only, never-deleted |
| 1696 | statements (§6). |
| 1697 | |
| 1698 | **Plan 1b's list, revised.** Items 1, 3, 5, 9 and 10 stand, read against |
| 1699 | rev 5 (item 10's expected region now comes from the roster). Item 2 |
| 1700 | follows §5.3's revised table. Item 4 is dropped. Item 6 becomes |
| 1701 | publishing the enrollment evidence at start. Item 7 is §5.8 as rev 5 |
| 1702 | states it, and waits for L1 and L9 to L11. Item 8 adds the roster check |
| 1703 | to the gate; volume recording is not part of it (B4-1). |
| 1704 | |
| 1705 | Rev 5.1 adds to plan 1b: the removed-volume rule (§5.9), the local |
| 1706 | install receipt (§5.8), and root's refusal on any controller record |
| 1707 | (§5.10). Rev 5.1 and 5.2's hold, release, guard attestation and SCP are |
| 1708 | withdrawn by H19 and are not built. |
| 1709 | |
| 1710 | ## 20. Astra rev 5 round 1 dispositions |
| 1711 | |
| 1712 | Reviewed rev 5 at 2a9165d4 (`CODEX-FINDINGS-REGION-HA-REV5-1.md`). Each |
| 1713 | citation was checked against claim 98b45332, moves a4711728 and main |
| 1714 | before acceptance. |
| 1715 | |
| 1716 | | Finding | Disposition | |
| 1717 | |---|---| |
| 1718 | | B1-1 P1 probe retries restore `P` again and lose replicated writes; `released` suppresses loss too widely | accepted: claim §6.4a item 3 and moves §7.14's restore row choose a fresh admission's baseline from its predecessor. Only the first attempt opens `P`; later attempts restore through `genselect.Select`; coverage is decided per admission, `released` only on `P`'s opening with a predecessor matching the journaled release (§5.8, L9) | |
| 1719 | | B1-2 P1 a replacement at `installed` never makes its local install | accepted: the remote build and the per-box local install are separate; `probing` requires a local install receipt bound to `op.exec`, its enrollment and `holder.stamp`; a replacement makes its own from `P`'s files manifest, and `P` is still built once and opened once (§5.8, L1) | |
| 1720 | | B1-3 P1 an operator's volume change before a refresh, or a detach, allows a false destroyed-disk fence | accepted: a hold written before the change suspends disk evidence; a volume leaving a running box's mapping suspends it too; release needs a newer all-true record and an operator's attestation for every removed volume (§5.9, §6, §7) | |
| 1721 | | B1-4 P1 the guard rests on an unprotected tag; the non-Organizations fallback still fences | accepted: the SCP also denies changing `carlos:role` except at launch; exemption narrowed to one hold-gated role; a root-signed guard attestation is required, and without one there is no automatic fence. Stricter than H15, never weaker (§5.9, §6, §8) | |
| 1722 | | B1-5 P1 a `lost` controller stops excluding updates | accepted: claim D6 does define `lost` and keeps its deny on failed cleanup. Root now refuses on any controller record in any state or any unreadable one, until D6's clearing with cleanup complete (§5.10, L6) | |
| 1723 | | B1-6 P2 the source lost in `starting` has no transition | accepted: a source-loss CAS converts the op to failover from `starting` or `releasing`, keeping the intent and acknowledgements, removing `from` from the barrier (§5.8, L1) | |
| 1724 | | B1-7 P2 a dead console stays in the D7 census | accepted: claim D7 does require every console. Console enrollments gain a `compute` binding; a console on a fenced box is excluded. A live console with a stale report was ruled as H18 (§5.9, L5, L20) | |
| 1725 | | B1-8 P2 automatic enrollment cannot bootstrap an emptied region | accepted: §2 makes a confirmer an active enrolled box in the region, so none is left. ruled as H17: an enrolled box in another region confirms, with read-only cross-region describes only, so one-box regions recover automatically (§5.1, §6, §7, L19) | |
| 1726 | | B1-9 P3 stale code citations | accepted, checked on main: `provision.go:49`, `reconcile.go:348`/`:820`, `:192`, `:215`, `activator.go:1164-1168`, `:2204`, `edge.go:2466`, `main.tf:521`. Disposition tables §13 to §17 keep their historical anchors | |
| 1727 | |
| 1728 | ## 21. Astra rev 5 round 2 dispositions |
| 1729 | |
| 1730 | Reviewed rev 5.1 at fdc0fe07 (`CODEX-FINDINGS-REGION-HA-REV5-2.md`). |
| 1731 | Each finding was checked against claim 98b45332, moves a4711728, main |
| 1732 | and AWS's documented API and policy contracts before acceptance. |
| 1733 | |
| 1734 | | Finding | Disposition | |
| 1735 | |---|---| |
| 1736 | | B1-2 partly resolved: the receipt dies with the next probe stamp | answered by B2-5 | |
| 1737 | | B1-3 partly resolved: release can precede authorized mutations | answered by B2-2 | |
| 1738 | | B1-4 partly resolved: root replacement unguarded; the attestation conflicts with Auto Scaling | answered by B2-1 and B2-3 | |
| 1739 | | B1-8 partly resolved: H17's emptiness test is circular | answered by B2-6 | |
| 1740 | | B2-1 P1 `CreateReplaceRootVolumeTask` bypasses the guard | accepted: added to the deny, with snapshots and images (which also leave a copy of the disk), the hold workflow and the rehearsal; a replaced root volume shows as removed (§5.9, §6, §8, L2) | |
| 1741 | | B2-2 P1 a release does not close the change's authority | accepted: the hold names its instance and the role's exemption is limited to it; release revokes the role's earlier sessions, waits for no attaching, detaching, replace-root, snapshot or image task and for the signature-age bound, then needs a validating record after that barrier. Session tags are acknowledged as the caller's word; the hold-first order is the operator trust boundary, as in claim D2 (§5.9) | |
| 1742 | | B2-3 P1 Auto Scaling's service-linked role defeats the attestation | accepted: the attestation no longer asserts that no service role can touch a disk. It reads every service-linked role's managed policy at its default version and requires each disk or tag action to map to an initiating API the SCP denies; Auto Scaling's (refresh, group create and update, warm pool, attach) are denied but for the hold-gated role; a policy version change invalidates it (§6, §7) | |
| 1743 | | B2-4 P1 the files pin follows the database baseline and can drop published files | accepted: the files pin is the newest manifest in the admitted chain, chosen independently; the files live set covers it through `holder.generation` and any running op's pin (§5.6, §5.8, L13) | |
| 1744 | | B2-5 P2 later probe stamps invalidate the receipt and invite a reinstall | accepted: the receipt binds to `op.journal.install_stamp`, the stamp of the CAS that handed `op.exec`; after the first admission the local install is never remade, and a lost receipt is `stuck` (§5.8, L1) | |
| 1745 | | B2-6 P2 the dead box stays `active`, so H17 never fires | accepted: the confirmer requires a positive `terminated` read for every `active` or `quarantined` enrollment in the region, in the same pass, as part of confirming membership; it is not a fence and retires nothing. The new box then fences through the ordinary path (§5.1, §7, L19) | |
| 1746 | |
| 1747 | ## 22. Astra rev 5 round 3 dispositions, and H19 |
| 1748 | |
| 1749 | Reviewed rev 5.2 at 075d709f (`CODEX-FINDINGS-REGION-HA-REV5-3.md`). |
| 1750 | Each finding was checked against AWS's documented contracts. All five |
| 1751 | are about the volume guard. Paul's ruling H19 (2026-10-06) took the disk |
| 1752 | out of the failover path, so they no longer bear on safety. |
| 1753 | |
| 1754 | | Finding | Disposition | |
| 1755 | |---|---| |
| 1756 | | B3-1 P1 Recycle Bin keeps volumes deleted on termination | correct about AWS, and moot for the fence under H19: a retained volume cannot run or write, and is never admitted as the executor. For chore discharge it is a retention concern for the operator path; §6 asks the deployment to configure no retention rule over platform volumes (§5.9, §6) | |
| 1757 | | B3-2 P1 `CopyVolumes` bypasses the copy guard | correct, and moot under H19 for the same reason: a copy is a data-retention concern, never a second writer (§5.9) | |
| 1758 | | B3-3 P1 another Auto Scaling group can overwrite the protected tag | moot: H19 removes the tag-conditioned guard; nothing in the fence or discharge reads the `carlos:role` tag any more (§5.1, §6) | |
| 1759 | | B3-4 P1 concurrent releases move the revocation cutoff back | moot: holds, releases and the change role are withdrawn (§5.9, §6) | |
| 1760 | | B3-5 P2 the release barrier omits Auto Scaling workflows | moot: there is no release barrier. A root replacement under any workflow shows as a removed volume, which only keeps the chores open (§5.9) | |
| 1761 | | B2-1, B2-2, B2-3 (partly resolved in round 3) | withdrawn with the mechanisms they patched; the property they protected is no longer needed for execution (H19) | |
| 1762 | |
| 1763 | **Earlier volume-guard findings, revisited.** B1-3 (operator changes |
| 1764 | before a refresh), B1-4 (an unprotected tag, the non-Organizations |
| 1765 | fallback), B2-1 (root replacement, snapshots, images), B2-2 (release |
| 1766 | authority) and B2-3 (service-linked roles) all showed ways a disk could |
| 1767 | outlive its box despite the records. Under H19 each such disk is a |
| 1768 | cleanup concern: the box is fenced by its `terminated` read alone, and |
| 1769 | a surviving disk can never become the executor, because admission needs |
| 1770 | `op.exec` to name an enrolled, active box (claim C4, HA7), and a new |
| 1771 | instance is a new enrollment. What remains of them is one cheap rule: |
| 1772 | any `false` flag or removed volume in any record keeps the dead box's |
| 1773 | chores open for an operator. Their dispositions in §20 and §21 stand as |
| 1774 | history. |
| 1775 | |
| 1776 | **Checked after removing the guard.** H17's confirmer no longer reads |
| 1777 | the `carlos:role` tag, and needs nothing else from the guard. The |
| 1778 | volume recorder stays, for discharge only. §5.10, the census (L5, L20), |
| 1779 | relocation (§5.8) and the fence read no disk evidence. CC11 lets a |
| 1780 | relocation start while a dead box's `park-local` is open, since it |
| 1781 | names only that box. A surviving disk attached to a new instance brings |
| 1782 | host artifacts the new enrollment does not own; claim P6-D2 refuses to |
| 1783 | initialize or materialize over ambiguous ownership, and every writer |
| 1784 | restores from the certified remote anyway (claim R6), so the old |
| 1785 | database is never served. |
| 1786 | |
| 1787 | ## 23. Astra rev 5 round 4 dispositions |
| 1788 | |
| 1789 | Reviewed rev 5.3 at 27058fe1 (`CODEX-FINDINGS-REGION-HA-REV5-4.md`). |
| 1790 | Astra agreed B3-1 to B3-5 are moot under H19, and found no two-writer |
| 1791 | or data-loss path that depends on disk evidence. |
| 1792 | |
| 1793 | | Finding | Disposition | |
| 1794 | |---|---| |
| 1795 | | B4-1 P2 volume recording is still a runtime admission prerequisite through §8's gate | accepted: under H19, volume recording never gates execution, admission, relocation or recovery. The gate no longer checks it; a box's start re-checks only capability, floor and roster; recorder failure alerts and keeps a dead box's chores open; tested through start, row minting and recovery (§8, §10, §19) | |
| 1796 | | B4-2 P3 summaries still describe the disk-dependent fence | accepted: the source-loss table discharges chores only on qualifying records (§5.8); the tofu note says `delete_on_termination` affects only discharge (§6); the one-box row fences on the `terminated` read alone (§7) | |
| 1797 | |
| 1798 | ## 24. Notes from plan 1b (2026-10-06) |
| 1799 | |
| 1800 | Plan `docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md` |
| 1801 | builds the parts of rev 5.4 that need none of L1 to L20: enrollment |
| 1802 | evidence, §19's corrections, the agent publishing its identity record, |
| 1803 | the fence bound to the roster's `cloud`, volume records and the discharge |
| 1804 | check, the edges file following `op.exec`, and root's half of §5.10. All |
| 1805 | of it is dormant behind `CARLOS_REGION_HA`. It keeps every property above |
| 1806 | and settles five details: |
| 1807 | |
| 1808 | 1. **Enrollment refuses one instance under two references (§5.1).** The |
| 1809 | record's `box` field is not covered by AWS's signature, so box A's |
| 1810 | record copied to B's key with that field rewritten still verifies. |
| 1811 | `boxid.Verifier.Enroll` refuses an instance the roster holds under another |
| 1812 | reference, as well as a reference the roster holds for another |
| 1813 | instance, counting entries in every state. |
| 1814 | 2. **A positive stamp is the relocatable mark (§18's open question).** |
| 1815 | Only relocatable hosts with a claim, and their aliases and served |
| 1816 | catch-alls, carry a stamp, so the edge's report |
| 1817 | of two boxes at one positive stamp already concerns them alone. No |
| 1818 | separate marker. |
| 1819 | 3. **The edges file's claim reads (§5.4).** A relocatable host with no |
| 1820 | claim is published as before, unstamped: §8's gate admits a second box |
| 1821 | only once every host has a claim. A read that fails keeps the last |
| 1822 | decision. Each read stands 10 minutes (H9), at most 256 reads a pass. |
| 1823 | 4. **"The first record taken at enrollment" (§5.9)** is the earliest |
| 1824 | volume record for the enrollment, observed within one hour of the |
| 1825 | roster's enrollment time. A later first record cannot speak for a |
| 1826 | volume attached and removed before it, so the chores stay open. And |
| 1827 | (controller ruling, 2026-10-06) the newest record must be no more |
| 1828 | than two hours older than the termination observation; otherwise the |
| 1829 | chores stay open for the operator path. |
| 1830 | 5. **No volume record from a `terminated` read.** Its mappings are empty |
| 1831 | (§8 step 5's dev test), and recording it would list every volume as |
| 1832 | removed. |
| 1833 | |
| 1834 | It also removes plan 1a's `Bind`, `BindKey`, `Verifier.Verify` and |
| 1835 | `activator.WarmCopyTrusted`, and makes every box-evidence prefix |
| 1836 | create-only and never deleted in the deployment bucket's policy. |
| 1837 | |
| 1838 | Controller rulings of 2026-10-06, on Paul's behalf: the edges file |
| 1839 | withholds a relocatable host when `op.exec` names another box or none |
| 1840 | (§5.4 as written); root's §5.10 refusal stays behind `CARLOS_REGION_HA` |
| 1841 | until claim D6 lands, then becomes unconditional; the `*.bind.json` |
| 1842 | statement and `operator_role_arns` stay until no root sets the variable. |
| 1843 | |