carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Region HA, spec 1: placing an instance on a box
2
3Design, 2026-10-06, **rev 5.4**. DRAFT. Plan 1a is built and merged
4(aadb860); nothing else here is built.
5
6**Rev 5.4** answers Astra's fourth review of rev 5
7(`CODEX-FINDINGS-REGION-HA-REV5-4.md`, **B4-n**), which found no
8two-writer or data-loss path that depends on disk evidence. It removes
9the last runtime dependency on volume recording: the multi-box gate no
10longer checks it at box start, and recorder failure only alerts and
11keeps a dead box's chores open (B4-1). It also brings three summaries
12into line with H19 (B4-2). §23 maps both.
13
14**Rev 5.3** records Paul's ruling **H19** (2026-10-06), which supersedes
15H15 and the volume half of H10, and answers Astra's third review of rev
165 (`CODEX-FINDINGS-REGION-HA-REV5-3.md`, **B3-n**) in §22. Rounds 1 to 3
17kept finding new ways a disk could outlive its box behind the volume
18guard. H19 takes the disk out of the failover path: a terminated
19instance can never run again, so the execution fence rests on the
20verified `terminated` read alone. Disk evidence decides only whether
21the dead box's `park-local` chores are discharged automatically; when
22it cannot be proven, they stay open for an operator, and recovery waits
23on nothing. The SCP, holds, the change role, the guard attestation and
24the service-role audit are removed. Most of round 3 is moot under H19;
25§22 says which, and why.
26
27**Rev 5.2** answers Astra's second review (`CODEX-FINDINGS-REGION-HA-REV5-2.md`,
28cited **B2-n**): four P1 and two P2, all accepted; §21 maps each, with
29the round 1 items it marked partly resolved. The volume guard now
30covers root-volume replacement, snapshots and images (B2-1), and Auto
31Scaling's own service-linked role, through the APIs that start its work
32(B2-3). A hold is closed by revoking the change role's sessions and
33waiting for in-flight disk tasks before the validating record (B2-2).
34The files manifest is pinned from the newest admitted generation, not
35from the database baseline's (B2-4). The local install receipt is bound
36to the installation stamp, and a box that has admitted a writer never
37reinstalls (B2-5). H17's confirmer may treat an enrolled box as gone on
38a positive `terminated` read (B2-6).
39
40**Rev 5.1** answers Astra's first review of rev 5
41(`CODEX-FINDINGS-REGION-HA-REV5-1.md`, cited **B1-n**): five P1, three P2
42and one P3, all accepted; §20 maps each. A restarted probe now restores
43from its real predecessor, and `released` coverage is bound to one
44session (B1-1). A replacement target must make its own local install
45before probing (B1-2). An operator's volume change first suspends
46automatic disk evidence, and a volume that leaves a running box keeps it
47suspended until accounted for (B1-3). The volume guard protects its own
48tag, needs an attested Organizations policy, and without one there is no
49automatic fence (B1-4). A lost or unreadable execution-lease state keeps
50updates out (B1-5). A source lost in `starting` turns the drain into a
51failover (B1-6). A console bound to a fenced box leaves the census
52(B1-7). Code citations are refreshed (B1-9). The two points this raised
53for Paul were ruled the same day: an enrolled box in another region
54confirms the first box of an emptied region (H17), and a box-started
55relocation accepts a console's newest report when its box's update
56guard is armed (H18). §11 is empty again.
57
58**Rev 5** restates the design against the host lifecycle claim as it now
59stands, rev 6.25 (413b12cb on `design-replica-ownership`), with its
60library on main (`internal/hostclaim`, `internal/hostguard`,
61`internal/stopevidence`, `internal/park`, `internal/genselect`), and
62records Paul's rulings H10 to H18 of 2026-10-06. Rev 4.2 was written
63against claim rev 6.3 and asked for changes K1 to K11. Those sessions
64never saw the request, and the claim moved on: box references are
65opaque, `control/host-actors.json` is the one roster, every writer is
66admitted by `AdmitWriter`, every admission is cold until certificates
67work, a prepared generation is opened once, and regional work is done
68by any enrolled box in the region. So rev 5 drops spec 1's own box
69records as authority (H11), the trusted local copy (H13), and its own
70`holder.generation` install CAS. Relocation (§5.8) keeps every property
71it had, and installs through the claim's prepared-generation and
72`AdmitWriter` rules instead. The termination fence stays automatic, now
73also requiring every EBS volume to be deleted with the instance (H10).
74A dev test showed a terminated instance's describe lists no volumes, so
75that proof comes from a record taken while the box runs, kept true by
76IAM (H15). A relocation is started without a console: the boxes, not a
77roster reservation, keep it apart from executable updates (H14, §5.10).
78§9's K-table is replaced by requests L1 to L20 against claim rev 6.25
79and moves rev 7.17. §19 says what of plan 1a is superseded. The three
80points rev 5 first left open were ruled the same day as H14 to H16
81(§11).
82
83Rev 4.2 answered Astra round 5 (`CODEX-FINDINGS-REGION-HA-5.md`, A5-1),
84and Astra round 6 (`CODEX-FINDINGS-REGION-HA-6.md`) found it ready for
85plan. Rev 4.1 answers Astra round 4 (`CODEX-FINDINGS-REGION-HA-4.md`,
86**A4-n**). Rev 4 answers Astra round 3 (`CODEX-FINDINGS-REGION-HA-3.md`, **A3-n**,
87plus the A2-5, A2-7 and A1-1 residuals). Rev 3 answered Astra round 2 (`CODEX-FINDINGS-REGION-HA-2.md`, cited
88**A2-n**, plus the round 1 findings it marked partly resolved). Rev 2
89answered round 1 (`CODEX-FINDINGS-REGION-HA-1.md`, **A1-n**). Every
90finding is accepted; §13 maps each to its section. Where rev 5 changes a
91mechanism a disposition names, the disposition's property still holds,
92and the section it points to says how.
93
94This is the first of four specs that give each region high availability
95and autoscaling:
96
971. **Box placement** (this spec). With several boxes in a region, the
98 host lifecycle claim names the one box that runs each instance, and
99 every box, edge and activator follows the claim.
1002. **Placement policy.** Which eligible box a new or displaced instance
101 goes to.
1023. **Failover.** A dead box's instances are re-placed onto the region's
103 survivors.
1044. **Autoscaling.** A region's boxes are an Auto Scaling group that grows
105 on demand and shrinks by draining.
106
107It builds on two specs still in review, and asks them for changes (§9):
108
109- the host lifecycle claim, **rev 6.25** (`design-replica-ownership`,
110 413b12cb), cited as **claim §n**. Its sections marked PROPOSED (§5.9
111 P5-D1 to P5-D6, §6.6 P6-D1 to P6-D8, §7.3 D2 to D6, §10.1 D1 and D7)
112 are cited as proposals, not as settled text. Rev 6.26 (98b45332)
113 changes only the staging scope of a delete's park (R17-2) and touches
114 nothing here;
115- move-any-app (`design-moves-any-region`), **rev 7.17** (39904c84),
116 cited as **moves §n**. Rev 7.18 (a4711728) changes no rule.
117
118It also builds on nearest-region (on main since 3f4a9bfd), whose box
119evidence in the edges file is how a console knows which enrolled boxes
120are alive, and on #212 option B for box references.
121
122## 0. Rulings (Paul, 2026-10-04 and 2026-10-06)
123
124| # | Ruling |
125|---|---|
126| H1 | **The proof that a box is dead is that AWS terminated it.** Each region's boxes run in an Auto Scaling group with health checks. When the group terminates an unhealthy box, a region placer that has only read-only AWS rights sees `terminated` and re-places the box's instances. The console still holds no EC2 rights. This replaces M16 / claim §7.2's operator-only declare-lost **for terminated boxes only**; a `stopped` box stays manual. It also replaces fleets D7's "no automatic reaper" for platform boxes. Lease expiry is still never proof (CC5). |
127| H2 | **Shape the claim now.** The HA specs cite the claim rather than invent a second primitive, and send the claim and moves sessions the exact changes they need (§9) before those specs are finalised on a one-box-per-region assumption. |
128| H3 | **About one second of data loss, written down.** Litestream keeps shipping asynchronously. Writes that had not reached S3 when a box died are lost, and the docs and console say so. |
129| H4 | **Placement policy: capacity plus soft spread.** Among eligible boxes, choose the one with the most headroom. Prefer a box that runs no other instance of the same app, but never refuse placement for that reason. If no box is eligible, the instance waits as `unplaced` and an operator is alerted. (Spec 2.) |
130| H5 | **Each region's minimum is configurable: 1 by default, 2 where HA is sold.** Scale-out is automatic on low headroom. Scale-in is automatic too, but only after the box has been drained into the region's other boxes. (Spec 4.) |
131| H6 | **File durability follows the paid plan.** Continuous sync on the top plan, periodic snapshots on the bottom one. The platform exposes this as a per-app setting (§5.6) and Carloku maps its plans to it. Platform-managed files (`instance.key`, `report.token`, `<host>.env`) sync continuously on every plan. |
132| H7 | **Ticks are at-least-once.** A delivery that ran on a dead box but whose bookkeeping did not reach S3 may run again. Documented; apps that need exactly-once guard it themselves. |
133| H8 | **`files_durability` defaults to `snapshot`** for apps with no setting. |
134| H9 | **The reconcile sweep is 10 minutes** (§5.3). |
135| H10 | **(2026-10-06) Termination fences stay automatic, and the disk must die with the box.** H1 stands against claim §7.3's operator-only declare-lost. A box AWS reports `terminated` is fenced automatically by a region placer: a box with read-only EC2 rights. The console holds none. The fence additionally requires every EBS volume the instance had attached to be `DeleteOnTermination: true`, read from the same `DescribeInstances` reply's `BlockDeviceMappings`. Otherwise the operator path applies. Stopped, unclear or vanished boxes stay operator-only (claim §7.3 D2 as written). Absence is never evidence. |
136| H11 | **(2026-10-06) The roster is the claim's `control/host-actors.json`** (claim §10.1 D7). Region HA drops its own `control/boxes/<id>.json` authority. `internal/boxid`'s AWS-signed IMDS identity verification becomes the evidence required to enroll a box. Enrollment records the proven account, region and instance id in the roster entry's `cloud` field, and the termination fence binds to that entry. Box references are opaque (claim §2, rev 6.16) and follow #212 option B: the label is the key, the instance id is private. |
137| H12 | **(2026-10-06) Executor choice at create** is spec 1's `choose()`: prefer a box running no instance of the app; then the most headroom (until spec 2, the fewest claimed hosts); then the claim's P5-D2 SHA-256 of `host + "\x00" + box reference` as the final tie-break. It is recorded once, in the create intent. Recovery never substitutes. |
138| H13 | **(2026-10-06) Claim ruling R6 supersedes HA3 and §5.5.** A box never reuses its local database. Every admission is cold until claim P6-D4 lifts. HA3 and §5.5 are deleted, and the activator half of plan 1a's install marker is not used. |
139| H14 | **(2026-10-06) Relocation starts need no console reservation (L6).** A relocation start requires that no executable update is reserved for `from` or `to` in `control/host-actors.json`, which the worker reads and never writes. Root on each box refuses to start an executable update while that box is a participant in a running claim operation. The update side's refusal is what closes the race with a reservation made after the start's read. (§5.10.) |
140| H15 | **(2026-10-06) H10's volume check rests on a running-time record, not the terminated read.** Tested on dev 2026-10-06: while `shutting-down`, `DescribeInstances` still listed both volumes of a test instance with their `DeleteOnTermination` flags; once `terminated`, `BlockDeviceMappings` was empty, the `true` root volume was already gone (`InvalidVolume.NotFound`) and the `false` volume survived as `available`. So the fence rests on (1) a record of the box's block device mappings taken by regional workers while it runs, refreshed on change, kept with its roster-bound evidence, showing every attached EBS volume `DeleteOnTermination: true`; (2) IAM denying `ec2:ModifyInstanceAttribute` and `ec2:AttachVolume` on platform boxes to every principal except operator roles, so the record cannot go stale behind the platform's back; (3) the terminated read (describe, event, later the group's history). A `shutting-down` read that still lists mappings may refresh the record but is never itself evidence. A missing record, one older than the stated bound, or one with any `false` volume sends the box to the operator path. (§5.9, §6.) The controller added `ec2:DetachVolume` to the deny in (2), as within the ruling's intent. |
141| H16 | **(2026-10-06) Auto Scaling boxes are enrolled automatically, with evidence (spec 4, L19).** The console enrolls an Auto Scaling box itself when its `boxid` evidence verifies (the deployment's account, the region's AWS region) and a regional worker confirms, with read-only Auto Scaling rights, that the instance is a current member of that region's group. Anything else still needs an operator's approval. |
142| H17 | **(2026-10-06) The first box of an emptied region is confirmed from another region.** Any enrolled box in another region holds cross-region read-only Auto Scaling and EC2 describe rights and may confirm a new box's group membership (H16) for a region that has no enrolled box. That is its only cross-region use. One-box regions therefore recover automatically. (§5.1, §6, L19.) |
143| H18 | **(2026-10-06) A stale console report does not hold up a box-started relocation.** For box-started relocations only, the census accepts a console's newest report however old, provided that report met the protocol floor and the console's box has the claim's D1 root update guard armed, so it cannot be swapped for an older binary underneath. Everything else in claim D7 is unchanged. (§5.9, L20.) |
144| H19 | **(2026-10-06) The failover fence rests on the `terminated` read alone; disk evidence only decides discharge of local chores.** Supersedes H15 and the volume half of H10; H10's automatic failover stands. The execution fence is a verified `terminated` read, bound to the roster's `cloud` identity, from the event or a describe (later the group's history); absence is never evidence. This is the claim's own lost-box rule for execution (§7.2), which needs no volume condition: a terminated instance can never run again, a surviving disk cannot write to storage by itself, and a disk attached to another instance is a different box with a different enrollment, never admitted as the old executor. The disk question belongs only to claim D5's destroyed-local exception: whether the dead box's `park-local` chores can be discharged automatically. That needs the volume evidence (the box's launch-time mappings all `DeleteOnTermination`, recorded while running); if it cannot be proven, the chores stay open for an operator (claim D2 and D5) and nothing about recovery waits on them. A surviving copy of a disk is a data-retention and cleanup concern for the operator path, not a two-writer risk. Tofu keeps `delete_on_termination = true`. (§5.9, §6, L2, L3.) |
145
146H10 to H19 supersede only what they name: H10 adds the volume condition
147to H1 and H15 says where it is read from, H11 replaces §5.1's box records
148as authority, H13 replaces HA3 and §5.5, H14 replaces claim D7's console
149reservation for relocation starts, H16 and H17 settle enrollment for
150spec 4, H18 narrows claim D7's console census for box-started
151relocations, and H19 supersedes H15 and the volume half of H10 (the
152fence needs no disk evidence; only automatic discharge of local chores
153does).
154Everything else in H1 to H9 stands.
155
156Standing rulings this spec keeps:
157
158- One writer: a home-run instance runs on exactly one box (claim C4,
159 multi-region ingress, sidecar members).
160- Placement targets a region and pool, never a named box (fleets D5).
161 An operator or a customer still cannot pin an instance to a box; only
162 the placer chooses.
163- The console is the single control-plane writer for operator and
164 customer requests, and holds no EC2 rights (claim §7.2). Boxes write
165 claims too, as the claim allows them: writer admission, publication,
166 their own chores and, under this spec, relocation.
167- A box must hold the deployment's secrets key to receive instances (R9),
168 and every box reconcile would admit in a region must qualify
169 (nearest-region F2).
170- Unreplicated state on a dead box is lost, and we say so.
171- **Claim R6:** a box never reuses its local database; every writer start
172 restores from the certified remote (claim §6.4a). Under P6-D4, every
173 admission is cold.
174
175## 1. Why
176
177Today a region is one box. Placement is per region: a bucket record has
178`Region` and `Pool` and no box (`internal/provision/provision.go:49`).
179Every box whose `CARLOS_REGION` matches mints the route
180(`inScope`, `internal/provision/reconcile.go:348`, called at `:820`). Nothing
181enforces "exactly one box": `reconcile.go:192` states it as intent only.
182
183A second box in a region would break three things:
184
1851. **Both boxes run the instance's route.** The S3 lease (`lock.json`,
186 15-minute TTL, `internal/lease/lease.go:37`) stops them running at
187 once, but whichever box a request reaches wins.
1882. **Both boxes claim the host to the edges.** The home map broke the
189 tie by sorted box name, with no liveness (`foldHomes`,
190 `internal/edge/homemap.go`), so a dead box that sorts first kept
191 winning. Plan 1a settles it by installation stamp (§5.4).
1923. **A box can serve a stale local copy.** The activator trusts a local
193 database as the warm cache because "on one box … the local copy can
194 only be what we ourselves checkpointed" (`internal/activator/activator.go:1164-1168`).
195 With two boxes taking turns, that is false. Claim R6 removes the
196 warm cache altogether (H13).
197
198And when a box dies, its instances stay down. EC2 auto-recover brings
199back the same machine, if it can (`host_auto_recover`,
200`infra/modules/region-host/main.tf:521`).
201
202## 2. Words
203
204- **Box**: one platform host machine. Not a customer fleet box; fleets
205 keep their own mechanism and are out of scope.
206- **Box reference**: the name the claim uses for a box in `op.exec`,
207 `participants`, `touched.boxes`, `inventory`, `lost`, owners and
208 execution-lease keys. It is opaque and compared by exact equality
209 (claim §2, rev 6.16). Under #212 option B it is the box's label key.
210 The instance id is private: it lives in the box's roster entry, under
211 `cloud` (H11). Spec 1 never parses a reference. The edges file keeps
212 its field name `box_id` for it; plan 1a already publishes the label
213 there.
214- **Roster**: `control/host-actors.json` (claim §10.1 D7, proposed,
215 accepted for region HA by H11). Console-authored, root-signed,
216 conditionally replaced. Each box entry binds a reference to its
217 `enrollment`, `region`, `pool`, `state` (`active`, `quarantined`,
218 `retired`) and `cloud: {provider, account, region, instance_id}`.
219- **Executor**: the one box the claim's `op.exec` names for a host.
220- **Relocatable host**: an exec-backed instance or a sidecar, whose whole
221 start state is in S3 or rendered from its record (HA5). Unit- and
222 socket-backed routes (the console, platform services) are **not**
223 relocatable: their unit definitions live outside the record. They stay
224 on their box, as today (A1-8).
225- **Eligible box**: a box whose roster entry is `active`, whose `cloud`
226 was proven by `internal/boxid` at enrollment (§5.1), and whose enrolled
227 region and `cloud.region` are the placement's; whose edges file is fresh
228 (nearest-region F1) and names the same reference; that is reachable,
229 hosting, serves the instance's pool, accepts inbound, holds the secrets
230 key (R9), publishes the HA capability (§8), and is not draining. For a
231 given host it must also not be on that claim's `lost` list and not owe
232 that host a `park-local` chore.
233- **Region placer**: a **regional worker** in claim P6-D5's sense: any
234 active enrolled box in the region with access to the region's data
235 bucket. It runs relocation recovery, failover (spec 3) and drain (spec
236 4). There is no separate leader roster. Correctness never depends on
237 which worker acts: every step is a claim CAS conditional on `op.id`,
238 the phase and the ETag (claim C2). Spec 3 may still use a lease to
239 decide who **starts** relocations, for efficiency only.
240- **Prepared generation**: a generation built before any writer, from a
241 sealed or pinned source, with its snapshot, level-0 anchor and install
242 manifest, and opened at most once by a cold `AdmitWriter` (claim §6.4a
243 item 3, P6-D3; moves §7.6).
244- **Relocation**: changing a host's executor to another box **in the
245 same region** (§5.8). A move changes region; relocation does not.
246
247## 3. Invariants
248
249**HA1. The claim names the executor.** A relocatable host runs only on
250the box its claim's `op.exec` names. This is claim C4 unchanged. What
251changes is that `op.exec` may now name any eligible box in the record's
252region, not "the region's box".
253
254**HA2. Region equality never admits.** The activator, config sync,
255release adoption and timers act for a host only on its executor, or on a
256participant in the phase that permits it (§5.8). Being in the right
257region is necessary, never sufficient (claim C4 says this already;
258today's code does not do it).
259
260**HA3. Withdrawn (H13).** Claim R6 governs instead: no box skips restore.
261Every admitted writer purges its local database and restores from the
262certified remote (claim §6.4a), and under P6-D4 every admission is cold.
263
264**HA4. Edges follow the newest installation.** When two edges files claim
265one host, the edge believes the one carrying the higher `holder.stamp`.
266Stamps only increase (claim `gen`), so a dead box's stale file always
267loses to its successor's, with no liveness rule.
268
269**HA5. Everything a relocatable host needs to start is in S3.** The
270database, files directory, env files and timer state are under the
271incarnation's `holder.location` in the current generation, or are
272rendered from the record (config, release). Nothing needed to start it
273lives only on the box running it.
274
275**HA6. The executor changes only by a claim operation.** Create, a move,
276or a relocation (§5.8). Each changes `op.exec` in a CAS that cites stop
277evidence (claim §6.1) or a lost-box fence, the termination fence
278included (§5.9), and writes `holder.stamp` in the same CAS (claim R6-4).
279A relocated host's data becomes current only through a cold
280`AdmitWriter` into a fully built prepared generation (claim §6.4a).
281
282**HA7. Termination evidence is bound to the roster.** A fence counts only
283if the terminated instance is the one the box's roster entry records
284under `cloud`, in the deployment's own account and the region's own AWS
285region (A1-1, H11). No disk condition applies to the fence (H19): disk
286evidence decides only whether the box's local chores are discharged
287automatically (§5.9).
288
289## 4. What stays the same
290
291- A region with one box behaves exactly as today, and every new rule
292 holds trivially there.
293- Static sites and public addr routes still run on every box
294 (`everywhereClass`, `reconcile.go:215`). Only relocatable hosts gain an
295 executor that can change; unit- and socket-backed routes keep theirs.
296- Customer fleets keep their binding stamps (`internal/host/fleetbind.go`)
297 and per-request fallback (`fleetChannelFor`, `internal/edge/edge.go:2466`).
298- The ingress model: any edge serves any app and backhauls home-run
299 hosts to their box (multi-region ingress). This spec changes which box
300 is "home", not how backhaul works.
301- Latency steering and ingress for boxes that come and go are spec 4.
302
303## 5. Design
304
305### 5.1 Box identity, the roster and evidence
306
307**Rev 5 (H11).** Spec 1's own `control/boxes/<id>.json` is no longer an
308authority. The roster is. What plan 1a built in `internal/boxid` becomes
309the evidence a box must present to be **enrolled**.
310
311**Enrollment.** A box asks to be enrolled by publishing its identity
312evidence, the record plan 1a defined:
313
314 control/boxes/<box reference>.json
315
316```json
317{
318 "v": 1,
319 "box": "<box reference>",
320 "identity_document": "<base64 IMDS instance identity document>",
321 "identity_signature": "<base64 RSA-SHA256 signature from IMDS>",
322 "written_at": "<RFC3339>"
323}
324```
325
326The object keeps plan 1a's storage rules: create-only, never deleted
327(§6). It grants nothing by itself. It is the evidence enrollment reads.
328
329Enrollment is the claim's (D7, L7). For region HA it must:
330
331- verify the record with `boxid.Verifier`: the signature under the
332 embedded certificate for the document's region, `accountId` equal to
333 the deployment's account, and the document's region equal to the AWS
334 region the box is being enrolled into;
335- record the proven `accountId`, `region` and `instanceId` in the entry's
336 `cloud: {provider: "aws", account, region, instance_id}`, and the
337 record's SHA-256 beside it;
338- refuse a reference already enrolled to a different instance (#212's
339 refusal to publish under a label another machine holds);
340- for a reference that already appears in claims (`touched.boxes`,
341 `inventory`) or edges files, as a box predating region HA does, also
342 check that the signed `instanceId` equals the region's tofu
343 `region-host` module `instance_id` output. This is the check rev 4's
344 operator bind made (A1-1 residual, A3-3). It now happens once, inside
345 the operator-approved enrollment (claim §10.1 P3), so the separate bind
346 object is not used.
347
348**Auto Scaling boxes (H16, spec 4).** The console enrolls a box launched
349by the region's Auto Scaling group itself, with no operator, when two
350pieces of evidence agree: its `boxid` record verifies with the
351deployment's account and the region's AWS region, and a regional worker
352has written a create-only membership confirmation,
353`control/box-enrollment/<box reference>/<instance id>.json`, from a fresh
354read-only `autoscaling:DescribeAutoScalingInstances` showing the instance
355a current member (`Pending` or `InService`) of that region's group, with
356the group name from the deployment's configuration. Anything else, a
357fixed box included, is enrolled only with an operator's approval. Spec 4
358builds this; the claim's D7 needs to allow it (L19).
359
360**A region with no enrolled box (H17).** When the target region has no
361**live** enrolled box, the confirmer may be any `active` enrolled box of
362another region, acting as a regional worker of its own
363region and checking its own entry first. This is the only thing it ever
364does across regions. Its confirmation binds to the target region's
365group, not to anything the new box says:
366
367- it reads the target region's group name and ARN from the roster's
368 `regions` entry for that placement (claim P5-D2's extension), never
369 from the new box or an edges file;
370- it calls `DescribeAutoScalingInstances` for the instance id in the
371 target region's AWS region, and requires the reply to name exactly that
372 group, lifecycle state `Pending` or `InService`, and an availability
373 zone in that AWS region; then `DescribeInstances` there, requiring the
374 instance `pending` or `running` with the deployment's `OwnerId`;
375- **the region is empty (B2-6).** The roster entry of a dead box stays
376 `active` until a fence retires it, and only a regional worker can write
377 the fence, so the roster alone would never show the region empty. The
378 confirmer therefore describes, in the same region and the same pass,
379 every other enrollment the roster lists there as `active` or
380 `quarantined`, and requires each to read **`terminated`**, positively,
381 for its roster `cloud`. `InvalidInstanceID.NotFound`, any other state,
382 an error or a partial reply refuses the confirmation (absence is never
383 evidence; a box that has left `DescribeInstances` goes to an operator).
384 The confirmation lists those reads (`terminated_enrollments: [{box,
385 enrollment, instance_id, observed_at, request_id}]`). They only show
386 the region has no live worker. They are not a termination fence, they
387 discharge nothing, and they never retire a roster entry;
388- it writes the same create-only object,
389 `control/box-enrollment/<box reference>/<instance id>.json`, with
390 `{target_placement, aws_region, group_arn, instance_id,
391 lifecycle_state, observed_at, request_ids, confirmer: {box,
392 enrollment, region}}`. Two confirmers race by the same conditional
393 write: one creates it, the other gets a refusal, reads it, and stops.
394
395The console accepts a cross-region confirmation only if, at enrollment,
396every enrollment the roster lists as `active` or `quarantined` in the
397target region appears in the confirmation's `terminated_enrollments`
398(any new one refuses), the confirmer was `active`, the group ARN equals the roster's for that
399placement, and the `boxid` record's region equals `aws_region`. Once
400enrolled, the new box is the region's worker: it writes its own volume
401records, fences its predecessor through the ordinary §5.9 path (its own
402terminated read), and recovers
403the hosts (§5.8). So a one-box region recovers automatically (§7).
404
405A box whose entry is not `active` with a proven `cloud` is never
406eligible, and a claim naming one is never relocated automatically: an
407operator resolves it.
408
409**Edges file fields.** Each box's edges file
410(`internal/host/edgesfile.go`) carries, as plan 1a built:
411
412```json
413{
414 "box_id": "<box reference>",
415 "capabilities": ["region-ha-1"],
416 "draining": false,
417 "headroom": { "mem_free_mib": 2210, "disk_free_mib": 31200, "awake": 14 }
418}
419```
420
421- `box_id` must equal an `active` roster reference; a file naming an
422 unknown or inactive reference makes nothing eligible. The edges file is
423 evidence of liveness and configuration, never of identity or
424 membership (claim D7: heartbeats describe members, they do not create
425 them).
426- `capabilities` gates multi-box admission (§8).
427- `draining` is set through the box by the placer (spec 4).
428- `headroom` is the input to spec 2's choice. Spec 1 publishes it only.
429
430`eligibleBoxes(snapshot, roster, region, pool)` returns the boxes,
431applying the predicate one box at a time (nearest-region
432`TestEligibilityNeverCombinesBoxes`). The expected AWS region comes from
433the roster entry's enrolled region and `cloud.region`, never from a vote
434of edges files (§18's second correction, now answered).
435
436### 5.2 Create names a box
437
438Claim §5.1's create intent names its executor (P5-D6
439`selected_executor`, `executor_enrollment`), written in the step 2 CAS
440before any effect. For a relocatable host it is
441`choose(eligibleBoxes(region, pool), app)` (H12, L8). Step 4
442(`CompleteCreate`) sets `op.exec` to exactly that box.
443
444`choose`, until spec 2 (H12):
445
4461. prefer a box that runs no instance of the app (H4's soft spread: it
447 never refuses);
4482. then the most headroom; until spec 2 that is the fewest claimed hosts,
449 counted from the console's claim-recovery listing (claim §5.1 step 5).
450 A stale count only affects balance, never safety;
4513. then the greatest hexadecimal SHA-256 of `host + "\x00" + box
452 reference` (claim P5-D2), and the smallest reference if two hashes
453 are equal.
454
455It is a pure function over the roster, the edges snapshot and the
456counts, and it is unit-tested alone.
457
458- The choice is made once. Recovery never substitutes a newly available
459 box (P5-D2). If the chosen box cannot complete the create (fenced,
460 quarantined, or it never acknowledges), recovery cancels the create
461 (claim §5.1 step 5, P5-D6), and the caller may create again.
462- No eligible box: the create fails with today's "region has no box
463 that can take this" refusal (nearest-region). Spec 2 adds the
464 `unplaced` state for re-placement, where refusing is not possible.
465- The choice is recorded only in the claim. The record keeps `Region`
466 and gains no box field, so a record still describes residency, not
467 placement.
468- A move into a multi-box region chooses its target box the same way,
469 in its start CAS (L15).
470
471### 5.3 Reconcile: executing rows, held rows, former rows
472
473Reconcile distinguishes these cases for a relocatable host in its region
474(A1-6). Each is decided by a fresh claim read.
475
476| The claim names this box as | Row |
477|---|---|
478| `op.exec`, op `serving` | mint or keep `ready` |
479| `from` of a `relocating` op in `starting`, or in `releasing` until its admission fence closes (A2-3) | keep `ready` |
480| any other participant role of a running operation (§5.8 table; moves §9.6) | mint or keep held, with `hold_op` = the op id (moves §5.2), never `ready` |
481| the owner of an open `park-local` chore for the host | keep held until the chore's retirement removes the row |
482| none of these | mint nothing. An existing row is held, never served, and alerted as unexplained |
483
484- **Former executors (rev 5).** Rev 4.2 had reconcile drop a former
485 executor's row as a registry change. Under claim P6-D2 a host's route
486 and timer rows are typed inventory entries, and `park-local`'s
487 retirement journal archives and removes them, conditional on their
488 recorded ownership. So reconcile never deletes a relocatable host's
489 row. It holds it, and the chore retires it under the host guard. A row
490 is never removed by hostname alone, which could remove a successor's.
491- The stale-hold rule (claim §8, CC10) is unchanged: it reopens a hold
492 only on the current executor of a `serving` host.
493- **When reconcile reads claims.** The safety gate is the activator's
494 fresh claim read at every spawn (claim C4), not reconcile. So
495 reconcile reads a host's claim when it would mint a row, and re-checks
496 the rows it keeps on a slow sweep, every 10 minutes (H9). This is a
497 stated exception to C4's "reconcile's decision to keep a row" (L12): a
498 kept row that is stale for up to 10 minutes can route a request to
499 this box, where the activator refuses to spawn. Reading every claim on
500 every pass would cost about $2.30 a day per box at 2,000 hosts on
501 30-second passes.
502- **Prompt changes.** The placer bumps the region's control serial after
503 each executor CAS, so the region's boxes reconcile at once and re-read
504 the claims of the hosts they hold or would mint.
505- A claim read error keeps an existing row and mints none (claim C6).
506
507### 5.4 Edges and the home map
508
509- The edges-file pass (`edgesFilePass`, `edgesfile.go`) publishes a host
510 as `home` only when this box is its executor **and** its row is
511 `ready`, and adds the installation's stamp:
512 `"hosts": {"shop.bab.oncarlos.com": {"role": "home", "stamp": 41}}`.
513 A relocation's `to` therefore publishes nothing while it installs and
514 probes, and becomes home at `serving`.
515- The home map (`foldHomes`, `homemap.go`) resolves a host claimed by two
516 files to the higher stamp (HA4). Plan 1a built this.
517- **Unstamped entries.** Before §8's gate opens, every region has one
518 box, so there is no conflict to resolve. After it opens, every box
519 publishes stamps for relocatable hosts, and an unstamped entry for a
520 host that some file carries stamped is ignored. §18's correction
521 stands: two unstamped entries are normal for everywhere-class routes,
522 so the edge reports only two boxes at the same positive stamp.
523- The edge never reads claims on the request path.
524
525### 5.5 The activator's local copy: withdrawn (H13)
526
527Rev 4.2's installation-complete marker and its warm-cache check (HA3) are
528deleted. Claim R6 replaces them: every admitted writer purges any local
529database with the error-returning purge and restores from the certified
530remote (claim §6.4a, "The baseline is always the certified remote").
531Under P6-D4 every admission is cold. The activator's one-box comment at
532`activator.go:1164-1168` is replaced by a reference to claim R6, not to HA3.
533
534Plan 1a's `activator.WarmCopyTrusted` is not used, and nothing reads the
535install marker to skip a restore (§19). A1-5's concern, that authority is
536not evidence of a complete install, is answered more strongly: no local
537copy is ever trusted, complete or not.
538
539### 5.6 Host files reach S3, in order
540
541Moves carry the files directory (F4) and `<host>.env` (F6) as a sealed
542archive taken from the live source (moves §6.1, §8). A dead source can't
543produce one, so HA5 needs them in S3 already.
544
545**The setting.** An app's manifest gains `files_durability`:
546
547- `continuous`: a changed file is published within about one second
548 of its **first** unpublished modification, whether or not the app has
549 closed it (A2-5):
550 - **Watches.** inotify `IN_MODIFY`, `IN_CLOSE_WRITE`, `IN_MOVED_*`,
551 `IN_CREATE` and `IN_DELETE` on every directory of the tree, added
552 recursively as directories appear.
553 - **Deadline.** The first modification of a file arms a one-second
554 deadline. Later writes do not push it back. A file still being
555 written is published at the deadline, then again at the next one.
556 - **Overflow.** `IN_Q_OVERFLOW`, or any watch that can't be added,
557 triggers an immediate full scan, comparing size, mtime and inode with
558 the last published state.
559 - **A consistent image.** The uploader reads the file, then checks that
560 size, mtime and inode are unchanged. If they changed, it retries
561 within the deadline. A file that never holds still for a read is
562 still published **at every deadline**, as the last image read, and
563 flagged as possibly torn. It never waits for a scan (A2-5 residual).
564 - **Missing coverage.** While any directory has no working watch
565 (overflow, a watch that can't be added, the watch limit reached), the
566 publisher **polls** that subtree once a second, comparing size, mtime
567 and inode, until the watch is restored. Polling is reported as a
568 box metric, and alerts if it lasts an hour. The files directory is
569 not a database: apps that update a file in place must expect a torn
570 image after a failover, and the docs say so. The instance database is
571 Litestream's.
572 - **Repair.** A full scan every 10 minutes catches anything a watch
573 missed. It is repair, never the path a change waits on;
574- `snapshot`: the whole directory is published every
575 `files_snapshot_interval`, default 15 minutes.
576
577Carloku maps its plans to this (H6). Apps with no setting get
578`snapshot` (H8).
579
580**Always continuous, whatever the setting:** `instance.key`,
581`report.token`, `<host>.env`, `<host>-sidecar.env`, and the host's wake
582state (§5.7: timer rows, the route's `NextDue` and `Replication`). They
583are small, and losing them is what kept kpmg down for 1h54m (moves §1.2).
584
585**Ordering (A1-4).** Object versions do not say which update wins, so
586nothing is ever overwritten and one manifest decides. §18 records how
587plan 1a built it: content is identified by SHA-256, and `snapshot` is a
588periodic full publish with no `base`.
589
590- **Objects are immutable.** Each upload goes to a new key,
591 `<holder.location>/files/<generation>/objects/<seq>-<sha256>`, written
592 create-only. A stalled upload that completes late creates an object no
593 manifest names.
594- **One sequence per host.** The executor numbers every local mutation
595 (write, rename, delete) with a monotonically increasing `seq`, kept
596 under the host guard and recovered from the manifest on restart.
597- **The manifest** is one object per generation,
598 `<holder.location>/files/<generation>/manifest.json`, replaced only
599 with `If-Match`. A delete is a tombstone entry, never an S3 delete. The
600 manifest is published only after every object it names is durable, and
601 its `seq` only increases, so a late manifest write fails its
602 `If-Match`.
603- **Restore** reads the manifest once, then fetches the objects it names
604 and checks each hash (§18). Nothing else in the prefix is read.
605- **Release flush.** A box giving up a host publishes every pending
606 mutation, **whatever the app's setting**, before its stop evidence
607 (§5.8's release sequence).
608
609**Under the claim (rev 5).** Three rules follow from the claim as it now
610stands.
611
612- **The publisher is a generation-bound writer.** It belongs to the
613 writer session, like the replicator: it is started for the `{location,
614 generation, session}` the admission named, refuses any key outside
615 `<location>/files/<generation>/`, and stops with the session. It is a
616 producer for stop evidence (claim P6-D1) and is joined before the
617 receipt. Its PUTs are a second ordinary-replication exception to claim
618 C3 (L13).
619- **Each cold admission starts a new manifest.** Under P6-D4 every wake
620 is a cold admission into a new generation. Before the app starts, the
621 admitting box writes `files/<new generation>/manifest.json`
622 create-only, as a copy of the **files pin**: the newest manifest in the
623 admitted chain, read by exact version (B2-4). That is the manifest of
624 `holder.generation` as observed, or, if that generation has none (its
625 copy never completed, so its app never started), of the nearest
626 predecessor in `holder.admissions` that has one. It is chosen
627 independently of the database baseline: the files publisher and the
628 replicator make progress separately, so a generation can hold
629 published files before its first database upload. Its entries keep
630 naming the predecessor's objects where they are, so an unchanged file
631 is never uploaded again. A late write from the predecessor's publisher
632 lands in the predecessor's prefix, which nothing admitted reads again.
633- **Old objects are retired by the claim, never reaped.** Rev 4.2's "reaped
634 after a day" is withdrawn (plan 1a never built it). A dead generation's
635 `files/<generation>/` prefix is retired with that generation, by the
636 `park-generation` chore `PublishGeneration` queues, except objects that
637 a manifest in the live set still names (L13). The **files live set** is
638 every manifest from the files pin through `holder.generation`, plus any
639 manifest a running operation has journaled in its pin, and every
640 object they name. A park that runs while an operation is in progress
641 keeps all of them. The at-rest sweep treats
642 `files/` as part of the location, never as residue to classify
643 blindly (claim P6-D3).
644
645A `snapshot` app's files can be up to the interval older than its
646database after a failover, and the console says so on the app's
647Overview.
648
649**Moves.** A move keeps its sealed archive at the cut (moves §8), and
650installs a fresh manifest in its new generation (L17).
651
652### 5.7 What a new executor rebuilds
653
654| State | How the new box gets it |
655|---|---|
656| Database | built from the pinned recovery set into a prepared generation, then restored from it by the probe's cold `AdmitWriter` (§5.8) |
657| Files directory, env files | the pinned manifest (§5.6), copied into the prepared generation |
658| Wake state | the host's timer rows, as `Registry.ReplaceTimers` rows (moves §9.4), and the route's `NextDue` and `Replication` (moves §6.2), published continuously as one entry under the manifest (A1-7) |
659| `<host>.config.env` | rendered for the held row (`RenderHeld`, claim §6.1 CC13, P6-D3), needs the secrets key (R9) |
660| Release | materialized for the held row (`MaterializeHeld`) |
661| Certificates | the cert mirror's boot refresh (on main) |
662| Registry row | reconcile, held while it installs, `ready` at `serving` (§5.3) |
663| Inventory | written by the new box before it first materializes state (claim §6.3, P6-D2) |
664
665**Wake state (A1-7).** The executor publishes the host's wake state as
666an always-continuous entry in the manifest whenever any of it changes:
667- its timer rows (active rows and their bookkeeping, moves §9.4's
668 canonical digest), on a registration, a delivery or a rearm;
669- the route's `NextDue`, the app's own next wake
670 (`activator.go:2204`), whenever the activator sets it;
671- the route's `Replication` mode.
672
673The new executor sets `NextDue` and `Replication` from it, as a move's
674landing does (moves §7.6). A once-timer the app registered is
675therefore in S3 within about a second of the app being told it
676succeeded. On install the new box imports the timer rows with
677`ReplaceTimers`, preserving `next_at`, `due_at`, `attempts` and
678`registration`. Timers on held rows stay frozen (moves §9.4, claim
679P6-D8) until the row is `ready`.
680
681- An occurrence that fell due while the host had no executor fires once
682 when the new executor imports it: the converger catches up from the
683 imported `next_at`, not from `Next(now)`.
684- A delivery that ran on the dead box but whose bookkeeping did not reach
685 S3 runs again. Ticks are documented as **at-least-once** (H7).
686
687### 5.8 Relocation
688
689A new claim op kind, `relocating` (L1), from `serving` only, for a
690relocatable host, within one region. It is the stateful, same-region
691executor change the claim's `placing` refuses (P5-D5).
692
693**The start CAS** writes `op.kind`, `op.id` and the intent before any
694other effect (claim C2):
695
696 {from, to, to_enrollment, region, reason, stamp_from,
697 generation_from, fence?, directory_revision}
698
699`reason` is `drain`, `rebalance` or `failover`. For failover, `fence`
700names the termination fence record (§5.9) by key and SHA-256. The start
701is refused by the claim's start guard (claim §4, CC11, as amended by
702L11), when `to` is not eligible for the host (§2), and when the roster
703shows an executable update reserved for `from` or `to` (§5.10, H14).
704`directory_revision` is the roster revision that check read. No console
705reservation is taken. Participants are `from` and `to`, or `to` alone
706for a failover.
707
708**Who acts.** Every phase is a claim CAS conditional on `op.id`, the
709phase and the ETag. The recovery owner for every phase is the region's
710regional workers (§2, claim P6-D5), which replay from the claim alone
711(claim C2). Each worker first checks its own roster entry is `active` in
712this region. Two workers racing make the same CAS, and one fails. `from`
713and `to` hold execution leases (claim §7.1, D6) for as long as they are
714participants.
715
716**Phases (A1-3).** In the "admits" column, "writer" means a writer
717admission (`AdmitWriter`) is legal for that box; nothing else may open
718replication.
719
720| Phase | Who acts | `op.exec` | `holder.stamp` | Rows (§5.3) | Admits |
721|---|---|---|---|---|---|
722| `starting` | a worker writes the intent; `from` (if alive) and `to` write execution leases and `exec_ack` (D6). The barrier needs only `to` when the intent cites a fence, or once a source-loss CAS (below) has removed `from` | `from`, or `none` after a source-loss CAS | `stamp_from` | `from` ready; `to` held | `from`, as for `serving` |
723| `releasing` | `from` runs the **release sequence** (below) and publishes its stop receipt; for failover the fence is already in the intent and this phase is skipped | `from` until evidence | unchanged | `from` ready until its admission fence closes, then held; `to` held | `from`, until its admission fence closes |
724| `pinning` | a worker CAS citing the receipt or the fence: `op.exec: none`, a non-installing transition that keeps the stamp (claim R6-4, D5's form). Then it selects and journals the **pinned recovery set** (below) | `none` | unchanged | `from` held (if alive); `to` held | nobody |
725| `installing` | a worker CAS that journals the pin and a fresh prepared-generation id `P`, hands `op.exec` to `to`, and writes `holder.stamp` = this CAS's `gen` (R6-4: a new installation), adding `to` to `touched.boxes`. `to` activates that stamp under the host guard (claim §6.4), writes its inventory entry, **builds `P`** and makes its **local install** (below) | `to` | `S'` | `to` held; `from` held | nobody: `to` writes local state under the guard, starts no app and admits no writer |
726| `installed` | a worker CAS once `P`'s install manifest is complete and verified, journaling `P`'s prepared authority (claim P6-D3). The executor then journals its **local install receipt** by CAS, if this CAS did not already carry it (a replacement makes its own, below) | `to` | `S'` | as `installing` | nobody: the executor may make its local install, nothing else |
727| `probing` | `to` enters it by CAS, which requires a local install receipt for exactly `op.exec`, its enrollment and `op.journal.install_stamp`; then each probe attempt is one cold `AdmitWriter` (below), then the probe (moves §7.7, same region), within the probe deadline | `to` | `S'`, kept by the first admission | as `installing` | `to`, writer, probe only |
728| `probe-failed` | `to` runs the release sequence for its own session | `to` until evidence | unchanged | as `installing` | nobody |
729| `serving` | a worker CAS back to `serving` with `participants: [to]`; `from` gets a `park-local` chore unless it is discharged under H10 (§5.9) | `to` | `S'` | `to` ready; `from` held until its chore retires the row | `to` |
730
731**The release sequence (A3-2).** Whenever a live box gives up a host,
732the source in `releasing` or a failed target in `probe-failed`, it does
733these in order, journaling each. It runs inside claim P6-D1's stop
734collection: root's collector has recorded its durable fence, and the
735agent does steps 1 to 3 before it closes the producers and returns its
736drain acknowledgement.
737
7381. **Stop every writer (A4-1).** Close admission, drain spawn attempts,
739 drain in-flight timer deliveries (moves §9.4), then stop the host's
740 process tree and wait for it to be empty: the per-host cgroup's
741 `populated 0` and no member processes. The database, WAL and SHM
742 files may still be held, but **only** by the box's own replicator for
743 this host, identified by the agent's pid and the replicator's own
744 descriptors (A5-1). Any other holder fails the step. From here nothing
745 but the replicator can touch the database, and nothing at all can
746 change files or wake state. `internal/stopevidence` has no check of
747 this form today: `Collect` requires the replicators closed. Rev 5
748 requests a writers-stopped check that allows exactly these
749 descriptors (L10).
7502. **Flush the database:** final checkpoint and `SyncAndWait`, then
751 record the **final TXID** `N`: the stopped database's own position.
752 The step passes only once a listing shows the newest level-0 file's
753 `MaxTXID` equal to `N` (moves §5.1 step 1). A sync error is not
754 swallowed (claim P6-D4's warning about `Replicator.Close`); it retries,
755 and a flush that never reaches `N` goes to `stuck`.
7563. **Publish every pending file and wake-state mutation**, whatever the
757 app's `files_durability`, and record the manifest's final `seq`.
7584. **Finish stop evidence:** close the replicator (`DB.Close`) and the
759 file publisher, then root completes claim §6.1's **unrestricted**
760 check (`stopevidence.Collect`, P6-D1): no holder at all of the
761 database, WAL or SHM files in `/proc/*/fd`, and every other
762 condition. The receipt is published (P5-D3) with its `evidence_at`.
763 Only now is the release complete.
764
765Because step 1 empties the process tree first, the positions recorded in
766steps 2 and 3 really are final. The session ends uncertified, as every
767session does under P6-D4; coverage (below) is what keeps its writes.
768
769The journal entry is `{"released_by": "<box>", "generation": "<G>",
770"session": S, "final_txid": N, "manifest_seq": M, "receipt": "<key>",
771"receipt_sha256": "…"}`, and `pinning` refuses to pin less than it.
772
773**The pinned recovery set (A1-2).** In `pinning`, a worker runs
774`genselect.Select` against the claim observation of its `pinning` CAS:
775`holder.generation`, the unfinished run in `holder.admissions` and the
776marker, all in this region's bucket (claim §6.2a, §6.4a item 3). It
777journals the plan's exact object versions (the snapshot and every LTX
778file after it, contiguous), and, chosen separately, the exact version of
779the **files pin** (§5.6: the newest manifest in the admitted chain, not
780the database baseline's generation), with files and timers included.
781The two can come from different generations; each is the newest
782durable state of its own publisher (B2-4).
783
784- **After a release sequence** (drain, rebalance, probe failure): the set
785 must reach the journaled `final_txid` and `manifest_seq`. If the listing
786 has not caught up, `pinning` waits; it never pins less than what the
787 releasing box flushed.
788- **Failover:** the worker pins after the fence's `observed_at` plus 30
789 seconds, so requests S3 had already accepted from the terminated box
790 have completed. Whatever did not land is the H3 loss window. This is a
791 loss bound, not a safety bound: the pin names exact versions, and a
792 later PUT can only add a version nothing reads.
793
794**Building the prepared generation (A1-2, A2-1).** In `installing`, `to`
795builds `P` exactly as a move's landing builds its generation (moves §7.6
796`building` and `published`, L16), with the pinned versions as its source
797in place of a staged replica:
798
799- under the replica lease on `holder.location` (claim C3), restore the
800 pinned versions with the planner, verifying each file's checksum,
801 contiguity, the final position `N` and `PRAGMA integrity_check`;
802- write the build intent, then the full **snapshot** `[1, N]`, then the
803 **level-0 anchor** `[N, N]`, each acknowledged, then the completed
804 **install manifest** with each `versionId`, `N`, checksums and the
805 database's SHA-256. The pinned Litestream derives a box's replication
806 position from the newest remote level-0 file. A generation holding a
807 snapshot and nothing at level 0 restarts at TXID 1, and a cold restore
808 from the snapshot then skips every later write (moves R5-1);
809- write `P`'s files manifest, create-only, as the pinned manifest's
810 entries (objects named where they are, §5.6), with the files, env and
811 wake state.
812
813That is the **remote build**: immutable, built once, and complete when
814a fresh read verifies all three objects. `installed` records it.
815`generation.json` is not written here: the probe's admission publishes
816it (moves §7.14).
817
818**The local install (B1-2)** is separate, because it belongs to one box.
819The box that is `op.exec`, under the host guard with the stamp it
820activated:
821
822- applies the files and env from `P`'s files manifest to the local
823 directory, `ReplaceTimers` the wake state, and sets `NextDue` and
824 `Replication`;
825- runs `RenderHeld` and `MaterializeHeld` with current inputs, recorded
826 as the relocation's inputs (P6-D3);
827- journals each step locally, and then, last, records its **local
828 install receipt** in the op journal by CAS:
829 `local_install: {box, enrollment, install_stamp,
830 files_manifest_version, journal_sha256}`.
831
832**`install_stamp`** is the stamp of the CAS that handed this box
833`op.exec` (`installing`, or a replacement's), journaled by that CAS as
834`op.journal.install_stamp`. It is the installation's provenance, not the
835current writer's stamp: later probe admissions take new stamps (§6.4a
836item 4) and leave it alone (B2-5). A receipt counts only when its box,
837enrollment and `install_stamp` equal `op.exec`, its enrollment and
838`op.journal.install_stamp`, so a replacement executor never inherits its
839predecessor's. Remaking the local install before the first admission is
840not a reinstall: `P` is not touched.
841
842**After the first admission, the local install is never remade
843(B2-5).** Once any admission entry names `P` (or a later generation of
844this op), the box's local files, env and wake state are the live state
845of that writer lineage. A new agent on the same box stops the app and
846admits a new session, as for any cold wake; it never reapplies `P`'s
847files or timers. A box that finds its receipt missing after that point
848has lost local state it cannot rebuild from `P`: the op goes to `stuck`,
849and the operator resolves it (or a fence makes it a nested failover
850from `holder.generation` and the files pin).
851
852**The probe's admission.** Each probe attempt is one `AdmitWriter`
853(claim §6.4a), made by `to` as `op.exec`, bound to the observed `op.id`
854and phase `probing`:
855
856- **mode** cold, always (P6-D4, H13);
857- **generation and baseline (B1-1):** the first attempt opens `P`,
858 which no admission has named (P6-D3's opened-once evidence), and
859 restores it from its pinned snapshot and anchor
860 (`genselect.SelectPrepared`). Every later attempt is cold into a fresh
861 generation whose baseline is chosen as for any cold admission:
862 `genselect.Select` from the predecessor, `holder.generation` as
863 observed, through the unfinished run (claim §6.4a item 3; moves §7.14's
864 restore row). So attempt three restores attempt two's generation with
865 everything it replicated, not `P`. Prepared authority is used once;
866- **`KeepStamp`** on the first attempt, because no writer has been
867 admitted since the `installing` CAS issued `S'` (`holder.stamp >
868 holder.session`). Later attempts take a new stamp;
869- **coverage, per admission (B1-1):** decided for the session the
870 admission's predecessor actually is, the last entry of
871 `holder.admissions`, never for the operation as a whole:
872 - `released` (L9) only on the admission that opens `P`, and only when
873 that predecessor is exactly the `{box, session}` whose release
874 receipt this op journaled and whose `final_txid` the pin reached.
875 That is a drain, a rebalance, or the nested relocation after a
876 `probe-failed`. Until L9 is accepted, relocation by drain is not
877 enabled (§8), because coverage `""` would declare a false loss on
878 every drain;
879 - `""` everywhere else: a failover (the admission records
880 `writes_lost` for `from`'s last session, `remote_max_txid` the pin's
881 `N`, H3's written-down loss), and every later probe attempt, whose
882 predecessor is the previous attempt's uncertified session. `to`
883 raises one alert per declaration;
884- then, as for every cold admission: purge, restore the chosen baseline,
885 start, readiness, `generation.json`, then `PublishGeneration`, which
886 queues `park-generation` for the dead generations of this region in
887 the chain (`generation_from` at the first publication; an earlier
888 attempt's generation at a later one) (claim §6.4a item 5). A queued
889 park never touches a generation a later attempt still needs: the
890 baseline is always the newest generation, protected as the unfinished
891 run until its successor publishes (claim §6.2a);
892- **limits:** at most three attempts within the probe deadline, as a
893 landing's (claim §6.4a item 7). A probe never marks the host failed;
894 running out of attempts is `probe-failed`.
895
896**A host with nothing replicated.** If the pin's `N` is 0 (the app never
897committed a transaction), there is no prepared generation: claim
898P6-D3 forbids a `[1, 0]` snapshot or a `[0, 0]` anchor. `installing`
899then builds only the files manifest and local state, and the first
900probe attempt is an ordinary cold admission into a fresh, empty
901generation (claim §6.4a, "Empty bootstrap").
902
903The test is the moves one, run for relocation: relocate, write,
904replicate, terminate the new executor, cold-restore, and check that every
905replicated write is present (`internal/sim/litestreamfacts`). Late writes
906from the old installation land in `generation_from`, which nothing
907admitted reads again, and which publication retires.
908
909**The no-reinstall boundary (A1-3).** Once the `installed` CAS succeeds:
910
911- nobody builds `P` again: replay from `installed` or `probing` goes
912 forward (the local install only if the executor has no receipt and no
913 admission has named `P`; then the probe, then `serving`), never back to
914 `installing`;
915- `P` is opened at most once. After its first admission, every restart
916 takes a fresh generation from its predecessor;
917- a worker or `to` that paused and resumes revalidates the phase under
918 the host guard, with the stamp it captured, and stops.
919
920Replay from `installing` (before `installed`) parks `to`'s partial local
921state with `retire-artifacts` (claim §6.4), and rebuilds `P` from the
922same pin. The build intent makes the bytes identical (moves §7.6).
923Nothing has opened `P` yet, so nothing is lost.
924
925**Exclusion.** Only `op.exec` installs or admits, and only `to` is
926`op.exec` from `installing` on. Every CAS is conditional on `op.id` and
927the phase. `to` holds the replica lease while it builds.
928
929**Replacing a participant (A2-2, A2-7).** The claim allows a new
930operation only from `serving`, so relocation defines its own exits (L1).
931Each is one CAS, conditional on `op.id` and the phase, that cites its
932evidence and names the replacement `to'`. A CAS that moves `op.exec`
933writes the stamp under R6-4. Every rule that applied to the phase still
934applies, and `to'` acknowledges its execution lease before the op leaves
935that phase.
936
937| `to` stops being usable | Evidence | The op resumes at |
938|---|---|---|
939| terminated in `starting` or `releasing` | `to`'s termination fence | the **same** phase. The source's release requirements are unchanged: a drain still needs `from`'s stop evidence |
940| terminated in `pinning` | `to`'s termination fence | `pinning` |
941| terminated in `installing` | `to`'s termination fence | `installing`, with `op.exec: to'`, a new stamp and a fresh prepared id `P'`, from the journaled pin. The old `P` was never opened; the at-rest sweep parks it |
942| terminated at `installed`, or in `probing` before its first `AdmitWriter` succeeded (no admission entry names `P`) | `to`'s termination fence | `installed`, with `op.exec: to'`, a new stamp, and no local install receipt. `to'` activates the stamp, makes its own local install from `P`'s files manifest and records its receipt (B1-2), then enters `probing`. `P` was never opened, so `to'` opens it |
943| terminated after its first `AdmitWriter` | `to`'s termination fence | a nested failover from `pinning`, pinning from `holder.generation` as observed (`P` or a later fresh generation, with `to`'s replicated writes). Coverage `""` declares `to`'s sessions |
944| alive, but its probe fails or misses the **probe deadline** (10 minutes) | `to`'s release sequence, in `probe-failed` | a nested relocation from `pinning`, pinned to at least `to`'s journaled `final_txid` and `manifest_seq`, so writes made during the probe are kept. `to` is excluded from the choice for this host |
945
946**Losing the source (B1-6).** A drain or rebalance whose `from` is
947terminated before its stop receipt becomes a failover, in one CAS
948conditional on `op.id` and the phase, citing `from`'s fence:
949
950| `from` terminated | The CAS | The op resumes at |
951|---|---|---|
952| in `starting`, acknowledged or not | journals `failover_fence` (key, SHA-256) and `reason: failover`, keeping the intent and every acknowledgement; removes `from` from `participants` and from the barrier; sets `op.exec: none`, keeping the stamp (claim D5's non-installing form); adds `from` to `lost`, and discharges its `park-local` chores only if its volume records qualify, otherwise leaves them open for an operator (§5.9, H19) | `starting`, waiting only for `to`'s acknowledgement; the next CAS goes to `pinning` with the fence as its evidence, skipping `releasing` |
953| in `releasing`, before its receipt | the same | `pinning`, with the fence as evidence |
954| after its receipt (`pinning` on) | `from` only goes on `lost`, and its chores are discharged if its volume records qualify, otherwise left open; nothing else changes | the same phase |
955
956The pin for a converted op follows the failover rule (fence plus 30
957seconds), and the first admission's coverage is `""`.
958- **The bound.** After two replacements of `to` in one op, the placer
959 stops at the current phase's safe point (never between `installed`
960 and a probe) and marks the op `stuck`. It alerts, and the op waits for
961 `carlos ops host resolve <host>`. The host stays down, and the console
962 says why.
963- A `probe-failed` target that cannot finish its release sequence (its
964 cgroup won't drain, claim §6.1, or its flush fails) also goes to
965 `stuck`. Nothing moves away from a live box without its stop evidence.
966
967**What a relocation leaves behind.** `generation_from` and the
968predecessor run are retired by `PublishGeneration`'s `park-generation`
969chores in this region. An abandoned `P` is parked by the at-rest sweep.
970`from`'s local state is retired by its `park-local` chore, or discharged
971with the fence (§5.9). Relocation installs into the region's host
972namespace, so it waits for `quiet_after[region]` like a push (claim
973§6.2a); for a live location that time has normally passed.
974
975**Cost of a drain:** one cut of a few seconds, like a move's (M2), with
976no cross-region copy, plus one snapshot and anchor of the database in
977the region (moves §7.3's T11 arithmetic, same region). Failover's gap is
978detection plus restore (spec 3).
979
980### 5.9 The termination fence
981
982**What counts (A1-1, A1-11, H10, H19).** A box is fenced by termination
983when a regional worker, with its own read-only rights in the box's AWS
984region, observes state exactly **`terminated`** for the instance id,
985account (`OwnerId`) and region in the box's roster entry's `cloud`
986(H11, HA7), by one of the three sources below. Nothing else is needed
987(H19). This is claim §7.2's rule for execution, that the instance is
988fenced by the cloud, without the operator: a terminated instance can
989never run again; a disk that survives it cannot write to storage by
990itself; and a disk attached to another instance is a different box,
991with a different instance id and enrollment, never admitted as this
992box's executor (claim C4, HA7).
993
994**Disk evidence, for discharging local chores only (H19, claim D5).**
995Whether the dead box's `park-local` chores can be discharged without
996retirement is a separate question, about cleanup, not about who may
997write. A worker discharges them automatically, in the fence CAS, only
998when the box's **volume records** (below) show its block device
999mappings, from the first record taken at enrollment (the launch-time
1000mappings) to the last taken while it ran, with every attached EBS volume
1001`DeleteOnTermination: true` and no volume ever removed. Otherwise the
1002chores stay open for an operator's destroyed-local declaration or the
1003returning-box retirement (claim D2, D5). Nothing about recovery waits on
1004them: the fence, the relocation and the census do not read volume
1005records (§5.8, L5).
1006
1007This is not proof that no copy of the disk exists. A volume kept by a
1008Recycle Bin rule, copied, snapshotted or imaged, or detached by an
1009operator, can survive. That is a data-retention concern, handled by the
1010operator path and the deployment's own retention settings, never a
1011second writer: no such disk can act as the executor (above). The dev
1012test of 2026-10-06 (§8 step 5) showed why the evidence must be recorded
1013while the box runs: once `terminated`, `BlockDeviceMappings` was empty,
1014the `DeleteOnTermination: true` root volume was already gone
1015(`InvalidVolume.NotFound`), and the `false` volume survived as
1016`available`.
1017
1018**The volume record.** Regional workers describe each enrolled box in
1019their region while it runs: at enrollment, on every 10-minute sweep, and
1020at once when a describe shows `shutting-down`. A worker writes a new
1021record, create-only, whenever the mapping differs from the newest
1022record, and otherwise at least once an hour:
1023
1024 control/box-volumes/<enrollment>/<observed_at>.json
1025
1026```json
1027{"v": 1, "box": "<box reference>", "enrollment": "<enrollment>",
1028 "cloud": {"provider": "aws", "account": "123…",
1029 "region": "eu-west-1", "instance_id": "i-0abc…"},
1030 "instance_state": "running",
1031 "block_devices": [{"device": "/dev/xvda", "volume_id": "vol-…",
1032 "delete_on_termination": true, "status": "attached"}],
1033 "removed_volumes": [],
1034 "observed_at": "<RFC3339>", "request_id": "<EC2 request id>",
1035 "written_by": {"box": "<reference>", "enrollment": "<enrollment>"}}
1036```
1037
1038- A record is bound to the roster entry: its `cloud` must equal the
1039 entry's, and a record for another instance is ignored.
1040- `removed_volumes` lists any volume the previous record had and this
1041 one lacks (a detach, or a replaced root volume). Any removal, or any
1042 `false` flag in any record, rules out automatic discharge for that
1043 enrollment for good, and alerts once so an operator knows.
1044- A `shutting-down` read may add a record; it is never termination
1045 evidence.
1046
1047**What never counts.**
1048
1049- `stopped` and `stopping` do not count, because a stopped box can start
1050 again with its disk. A missing instance, `InvalidInstanceID.NotFound`,
1051 an API error, a partial reply, or a lookup in another account or
1052 region is **never** evidence. AWS keeps a terminated instance visible
1053 in `DescribeInstances` for typically under an hour, so absence proves
1054 nothing. All of those go to the operator: `carlos ops host resolve
1055 --declare-lost`, with claim §7.3 D2's binding, attestation and
1056 challenge as written (H10).
1057- The worker reads the state itself. It never trusts a box's own report,
1058 an edges file, or anything a box publishes about itself. A volume
1059 record is a cloud read, not such a report: any regional worker may
1060 write one, the described box included, under the claim's trusted
1061 box-writer boundary (P5-D3). It decides only cleanup (above), so a
1062 wrong one can leave a disk unretired, never admit a writer.
1063
1064**The fence record.** The worker writes one record per terminated
1065enrollment, create-only (L4):
1066
1067 control/box-fences/<enrollment>.json
1068
1069```json
1070{"v": 1, "box": "<box reference>", "enrollment": "<enrollment>",
1071 "cloud": {"provider": "aws", "account": "123…",
1072 "region": "eu-west-1", "instance_id": "i-0abc…"},
1073 "state": "terminated",
1074 "observed_at": "<RFC3339>",
1075 "via": "describe" | "event" | "activity",
1076 "request_id": "<EC2 request id, when via=describe>",
1077 "event_id": "<EventBridge event id, when via=event>",
1078 "written_by": {"box": "<reference>", "enrollment": "<enrollment>"}}
1079```
1080
1081It is not AWS-signed: like claim D2's operator observation, it is the
1082worker's authenticated report of a cloud read, made with rights the
1083console does not hold. Every relocation and claim CAS that relies on it
1084cites its key and SHA-256 (`evidence_refs`, claim P5-D3).
1085
1086**Three ways to observe `terminated`.** Any of these is the terminated
1087read:
1088
10891. **The durable event.** An EventBridge rule sends EC2 "Instance
1090 State-change Notification" events with `state: terminated` for the
1091 deployment's account to an SQS queue (14-day retention). Plan 1a
1092 built one queue per box. The queue's policy accepts messages only from
1093 that rule's ARN. An event whose account, region and instance id match
1094 a roster entry's `cloud` is the read, `via: "event"`.
10952. **A direct read.** `DescribeInstances` for the instance id, in the
1096 region, returning `terminated` with the deployment's `OwnerId`,
1097 `via: "describe"`. Workers make it on their sweep and whenever an
1098 event or the group's history suggests a termination.
10993. **The group's own history.** For a box launched by an Auto Scaling
1100 group, `DescribeScalingActivities` for the region's group: an activity
1101 terminating that instance id with status `Successful`,
1102 `via: "activity"`. Spec 4 verifies on dev that `Successful` is
1103 recorded only once the instance has reached `terminated`; until then
1104 this source is off.
1105
1106**What is not guaranteed (A2-4).** EC2 delivers state-change events to
1107EventBridge on a best-effort basis, and a terminated instance leaves
1108`DescribeInstances` after about an hour. When no source shows
1109`terminated`, there is **no automatic recovery**. The placer marks the box's
1110hosts `awaiting-evidence`, alerts at once, and keeps retrying. The only
1111way out is the operator's `--declare-lost` (claim §7.3 D2). A CloudTrail
1112`TerminateInstances` record is **not** evidence (A3-1): it proves a
1113request was made, and an instance can stay `shutting-down` after a
1114successful one. The console may show it to help an operator find the
1115instance, nothing more. If no terminal state can be read, the hosts stay
1116`awaiting-evidence`. That is a liveness gap, accepted. Absence is never
1117evidence, at any point. Spec 3 sets how long the placer retries before
1118alerting, and owns the loop that reads the queue and history.
1119
1120**What the fence does (L2 to L5).**
1121
1122- **It is a lost-box fence** (claim C5, §4) for every host whose claim
1123 names the box. A relocation started on it may move `op.exec` off the
1124 box.
1125- **The box goes on each affected claim's `lost` list**, with the fence's
1126 reference. A terminated instance cannot run again, so it needs no
1127 quarantine and no `--readmit` (H1). Its reference never names a
1128 working box again.
1129- **Its local chores are discharged, if the disk evidence allows (H19).**
1130 When the volume records qualify (above), its `park-local` chores are
1131 discharged in the same CAS, by `CompleteObligation` semantics, without
1132 retirement: claim D5's destroyed-local exception (L3). The fence CAS
1133 cites the records it relied on. When they do not qualify, the chores
1134 stay open for an operator, and the relocation proceeds regardless:
1135 a dead box's `park-local` names only that box, which no relocation
1136 writes (claim CC11). Regional chores
1137 are reassigned to `region:<r>`, keeping their ids, operations and
1138 scopes (D5). Remote scopes it justifies take the fence's `observed_at`
1139 as their original evidence time (claim P5-D3, D5).
1140- **The roster follows later.** The console moves the box's roster entry
1141 to `retired`, citing the record, when it next runs. Until then the
1142 record alone excludes the box from the census for lifecycle starts
1143 (L5), so failover never waits for a console.
1144- **A console that ran on the box leaves the census too (B1-7).** Claim
1145 D7 requires a fresh report from every console able to mutate the
1146 namespace. A console enrollment gains `compute: {box, enrollment}`,
1147 the platform box its process runs on, set at console enrollment and
1148 checked against every console report, which that box's trusted writer
1149 publishes with its own `boot_id`. A console whose `compute` box has a
1150 valid fence record is excluded from the census exactly as the box is
1151 (L5). A console with no `compute` binding is never excluded this way.
1152- **A stale console report (H18).** For a relocation a box starts, and
1153 nothing else, the census accepts a console's **newest** report however
1154 old, when that report met every D7 floor check (protocol minimum,
1155 authenticated release minimum, matching floor digest) and the
1156 console's `compute` box has reported `floor_guard_ready` with the same
1157 floor digest. Claim D1 makes an armed guard permanent (it never
1158 expires and cannot be lowered offline), so that box cannot run an
1159 older console binary underneath. A console with no `compute` binding,
1160 or whose box never reported the guard armed, still needs a fresh
1161 report. Every other D7 rule, and every console-started operation, is
1162 unchanged (L20).
1163- **The loss is recorded twice.** The admission that takes over records
1164 `writes_lost` (§5.8), and the console copies the fence into claim D4's
1165 durable loss record when it next runs, with timer knowledge `partial`:
1166 wake state was published continuously, up to H3's window.
1167
1168### 5.10 Relocation starts and executable updates (H14)
1169
1170Claim D7 serializes a lifecycle start against executable updates with a
1171reservation the console writes into the roster. A relocation must start
1172when the console is down, so H14 moves the serialization to the boxes.
1173Each side records its own intent durably first, then checks the
1174other's, so of two that race at least one sees the other.
1175
1176**The worker, at the start.** Before the start CAS, the worker reads
1177`control/host-actors.json` as an exact object and verifies its root
1178signature (an unreadable or unverifiable roster refuses the start). It
1179requires `to`, and `from` unless the start cites a fence, to be
1180`active`, and requires that no entry in `activations` names `from`'s or
1181`to`'s enrollment. It records the revision it read as
1182`directory_revision` in the intent. It writes nothing to the roster.
1183
1184**A participant, before it acknowledges.** A box becomes a participant
1185in practice when it acknowledges the operation (claim D6). Before its
1186first execution-lease PUT, its controller:
1187
11881. writes its `.hostexec/<host>.json` state as `pending`, durably (claim
1189 D6's existing state and write protocol);
11902. then reads root's update state: the system-update journal and root's
1191 **update-begun** marker (below). If an update is begun or in progress
1192 for this box, it refuses to acknowledge, clears its `pending` state,
1193 and retries after the update finishes. The operation cannot leave
1194 `starting` without `to`'s acknowledgement, so nothing has moved; the
1195 worker waits, or replaces `to` (§5.8).
1196
1197**Root, before it starts an update.** Every path that begins an
1198executable change (system apply, explicit rollback, a guarded
1199activation under claim D1, the console's activation reservation being
1200acted on), under root's existing system-update lock:
1201
12021. writes an **update-begun** marker, durably, beside the update journal
1203 (`/opt/carlos/system/guard/update-begun.json`, naming the update);
12042. then scans `<DataDir>/.hostexec/` completely. **Any** controller
1205 record there, in any state (`pending`, `active` or `lost`), and any
1206 record root cannot read or parse, or a directory it cannot list, means
1207 the box may still be a participant: root removes its marker, refuses
1208 the update, and leaves the console's activation reservation held, to
1209 retry later (B1-5);
12103. otherwise it proceeds. The marker stays until the update finishes or
1211 is recovered.
1212
1213Boot recovery of an update already begun before any participation needs
1214no new check: no participant acknowledged meanwhile, because step 2 of
1215the participant's side saw the marker.
1216
1217**When exclusion ends (B1-5).** A controller record is removed only by
1218claim D6's clearing: a fresh claim read showing `serving`, `free`, or
1219that the box no longer participates, **and** any cleanup the record
1220requires complete. A `lost` record whose cleanup-only termination failed
1221is never cleared: claim D6 keeps that deny, and root's refusal stays
1222with it. A box whose processes may still run under an operation never
1223takes an executable update. An operator who needs to update such a box
1224first resolves the host (`carlos ops host resolve`).
1225
1226**The race.** If the console reserves an update for `to` after the
1227worker's roster read, the start proceeds, and the reservation is
1228harmless by itself. Whichever of `to`'s acknowledgement and root's
1229begin writes its marker first, the other sees it and refuses: either
1230the update waits until the operation returns to `serving` and D6 clears
1231the controller's state, or the acknowledgement waits until the update
1232finishes. Root's refusal is what closes the race (H14). A box stays a
1233participant from its `pending` state until D6 clears its record, through
1234`active` and `lost` alike, so an update never runs under a relocation's
1235`installing`, `probing` or release, nor while a lost controller's
1236cleanup is unfinished.
1237
1238This holds for every claim operation a box takes part in, not only
1239relocations. L6 asks the claim to make it D7's rule for box-started
1240operations.
1241
1242## 6. Infrastructure and IAM (A1-12)
1243
1244- **Box role** keeps plan 1a's `ec2:DescribeInstances` with a condition
1245 on `ec2:Region`, and `autoscaling:DescribeScalingActivities` with a
1246 condition on `aws:RequestedRegion`, both equal to the box's region,
1247 plus SQS `ReceiveMessage`/`DeleteMessage` on its termination queue.
1248 Describe actions do not support resource-tag conditions, so deployment
1249 membership is **not** an IAM property: the verifier checks `OwnerId`,
1250 the region and the roster entry (§5.9). No `ec2:Stop*`, `Terminate*` or
1251 `Modify*`, and no Auto Scaling write rights. It also needs the claim's
1252 own grants: `s3:ListBucketVersions`, `s3:GetObjectVersion` and
1253 `s3:DeleteObjectVersion` on its region's data bucket (claim §6.2a),
1254 which `genselect` and the parker use.
1255- **Disks (H19).** Nothing in IAM guards a box's disks for the fence's
1256 sake: rev 5.2's SCP, holds, volume-change role, guard attestation and
1257 service-role audit are withdrawn. The launch template and the
1258 `region-host` module keep `delete_on_termination = true` on every
1259 volume, which is cheap and makes automatic discharge of local chores
1260 the usual case (§5.9). The deployment should configure no Recycle Bin
1261 rule, Data Lifecycle Manager or AWS Backup policy that selects
1262 platform box volumes; one that does is a data-retention choice, and
1263 the operator path then owns their cleanup. The deployment accounts
1264 are Organizations members (organization `o-sgnzdj0z0c`, management
1265 account 665106592142; flagship 271376211898, dev 640012953448, checked
1266 2026-10-06), which no longer matters to the fence.
1267- **Auto Scaling enrollment (H16, H17, spec 4).** The box role gains one
1268 separate statement: `autoscaling:DescribeAutoScalingInstances` and
1269 `ec2:DescribeInstances`, read-only, conditioned on
1270 `aws:RequestedRegion` being one of the deployment's AWS regions (any of
1271 them, not only the box's own). Nothing else is granted across regions:
1272 every other statement keeps its own-region condition, and no write,
1273 queue or S3 right crosses regions. The code calls another region only
1274 to confirm membership for a region with no `active` box (§5.1), and
1275 the fence, volume records and relocation refuse any observation not
1276 made in the box's own region.
1277- **Console role:** unchanged, no EC2 rights.
1278- **S3 (A2-6).** Two kinds of write, kept apart:
1279 - **Create-only** (`If-None-Match: *`): content objects,
1280 `instances/<host>/*/files/*/objects/`, in the region's data bucket;
1281 box identity records, `control/boxes/`, volume records,
1282 `control/box-volumes/`, membership confirmations,
1283 `control/box-enrollment/`, and fence records, `control/box-fences/`,
1284 in the control bucket.
1285 - **Conditional replace** (`If-Match`, or `If-None-Match: *` for the
1286 first write): `instances/<host>/*/files/*/manifest.json` only.
1287
1288 S3 enforces both in policy (A3-3): the data bucket denies `PutObject`
1289 on `…/objects/*` without `s3:if-none-match: *`, and on
1290 `…/manifest.json` without either `s3:if-match` or
1291 `s3:if-none-match: *`. Each Deny carries
1292 `s3:ObjectCreationOperation: true`, so it applies to `PutObject`,
1293 `CopyObject` and `CompleteMultipartUpload`, never to
1294 `CreateMultipartUpload` or `UploadPart`, which can't carry the headers
1295 (A4-2). The one writer, `hostfiles.Publish`, sends the headers and is
1296 tested against both refusals. The control bucket denies `PutObject`
1297 without `s3:if-none-match: *`, and `DeleteObject` and
1298 `DeleteObjectVersion`, on `control/boxes/*` (plan 1a's policy) and on
1299 `control/box-volumes/*`, `control/box-enrollment/*` and
1300 `control/box-fences/*`. Each Deny is scoped to its prefix only: a
1301 broad Deny on this bucket once broke every lease release. The
1302 `*.bind.json` statement plan 1a added is no longer needed (§19). Data
1303 retirement under `files/` is by exact version, through the parker,
1304 like every other retirement (claim C3).
1305- **Tofu:** the `region-host` module has `delete_on_termination = true`
1306 on the root volume (plan 1a), and should have it on every other EBS
1307 volume a box attaches. It never affects the fence (H19): it only lets
1308 a dead box's `park-local` chores be discharged automatically (§5.9). Moving to an Auto Scaling group is spec 4; spec 1 runs on
1309 today's fixed boxes and on a second fixed box added by hand.
1310
1311## 7. Failure modes
1312
1313| What happens | Result |
1314|---|---|
1315| Executor box dies, region has another box | spec 3 relocates its hosts after the termination fence. Until then: requests 503 with Retry-After. |
1316| Executor dies, region had one box | the Auto Scaling group launches a replacement; an enrolled box in another region confirms its group membership (H17) and the console enrolls it. It is then the region's worker: it reads the termination event, writes the fence from that `terminated` read alone, and relocates every host to itself. The dead box's chores are discharged only if its volume records qualify (H19). |
1317| A console's box is alive but its report is stale | a box-started relocation proceeds on the console's newest report if that met the floor and the box's update guard is armed (H18); otherwise it waits and alerts. |
1318| The console's box is the one that died | failover does not need it: the fence, the census exclusion and every relocation CAS are a regional worker's (§5.9, L5, L6). |
1319| Box partitioned from the control bucket | it keeps serving what it serves (no execution lease while `serving`, claim §7.1). It can't take new work. The group's health check decides whether it is terminated. |
1320| Box alive but failing its health check | the group terminates it, then as above. Instances that were healthy lose up to H3's window, declared by `writes_lost`. |
1321| A terminated box had a volume without `DeleteOnTermination`, or one was removed | fenced and relocated as usual; its `park-local` chores stay open for an operator (§5.9, H19). |
1322| A copy of a dead box's disk survives (Recycle Bin, a copy, a snapshot) | a retention and cleanup matter for the operator; it can never act as the executor (§5.9, H19). |
1323| A region's only box dies and its roster entry is still `active` | the cross-region confirmer reads it `terminated` and confirms the replacement; the replacement fences it (§5.1, B2-6). |
1324| A participant's execution lease is lost and its cleanup fails | its controller record stays; root refuses executable updates on that box until the host is resolved (§5.10, B1-5). |
1325| The source dies while a drain is in `starting` | the op becomes a failover in place (§5.8, B1-6). |
1326| An update is reserved for `to` while a relocation starts | root refuses to begin it while `to` is a participant, or `to` refuses to acknowledge while it runs; never both running (§5.10). |
1327| A forged or confused edges file names a terminated instance | ignored: identity comes only from the roster's `cloud`, proven at enrollment (HA7). |
1328| Placer dies mid-relocation | another regional worker replays from the claim (§5.8). |
1329| Target dies after its probe admitted | a nested failover from `holder.generation` (§5.8). |
1330| Target alive but its probe keeps failing | stop evidence, then a nested relocation to another box; after two replacements, `stuck` and an alert (§5.8). |
1331| No termination evidence can be found | `awaiting-evidence`, an alert, and the operator's `--declare-lost` (§5.9). |
1332| Two files claim a host | the higher stamp wins (HA4). Two at the same stamp are reported (§18). |
1333| Claim read fails on reconcile | keep existing rows, mint none (C6). |
1334| `snapshot` app fails over | files lose up to the interval; the database loses about a second. Stated on Overview. |
1335| A unit- or socket-backed route's box dies | not relocated (A1-8). It comes back with its box, or by the HA ladder's restore runbook, as today. |
1336
1337## 8. Rollout (A1-9)
1338
13391. **Prerequisite:** the claim and its floor are live (claim §10), the
1340 roster exists with every box enrolled from `internal/boxid` evidence
1341 (§5.1), and every host in the region has a claim (adopted under claim
1342 §9 and P5-D2, which requires exactly one proven installation owner).
13432. Ship §5 dormant behind `CARLOS_REGION_HA`, with capability
1344 `region-ha-1` published in each box's edges file once enabled. With one
1345 box per region, every rule is a no-op; ship it to all four regions and
1346 watch.
13473. **The multi-box gate.** A second box may join a region only when:
1348 - every relocatable host there has a claim;
1349 - every box in the region, and every edge in the deployment, publishes
1350 `region-ha-1` and runs at or above the HA floor;
1351 - every box in the region is `active` in the roster with a proven
1352 `cloud`;
1353 - the region's termination queues exist.
1354
1355 **Volume recording is never part of this gate (B4-1, H19).** Spec 1
1356 ships the recorder with the rest of §5, and rollout checks once that
1357 it runs, but neither the gate, nor a box's start, nor row minting,
1358 nor relocation, nor recovery ever waits on a volume record. A
1359 recorder that cannot write raises an alert, and its only effect is
1360 that a box that dies meanwhile keeps its `park-local` chores open for
1361 an operator (§5.9).
1362
1363 The HA floor joins the protocol floor (claim §10, D1): every path that
1364 restores an executable (system apply, rollback, boot recovery from
1365 `carlos.prev`) refuses a binary below it once a region has two boxes.
1366 The gate is checked in `carlos system` and in the console's box
1367 list. Each box re-checks at start only its runtime half: the
1368 capability and floor of the region's boxes and edges, and their
1369 roster entries. A box that finds a second box in its region without
1370 that half satisfied refuses to mint relocatable rows and alerts. The
1371 termination queues are a rollout check only: a missing queue loses
1372 one trigger, and the describe still fences (§5.9).
13734. **Relocation by drain** stays off until the claim accepts L9; failover
1374 relocation needs L1 to L6.
13755. Rehearse on dev with two fixed boxes: create spreads across both;
1376 drain one (§5.8) with `moveprobe` shaped like ips (moves §14.4),
1377 including timers and a `continuous` files app; terminate one by hand
1378 and run the spec 3 placer manually. **Done on 2026-10-06 (H15):** a
1379 t4g.nano launched with a `DeleteOnTermination: true` root and a
1380 `DeleteOnTermination: false` extra volume, then terminated. While
1381 `shutting-down`, `DescribeInstances` still listed both volumes, with
1382 their flags and status `attached`. Once `terminated`,
1383 `BlockDeviceMappings` was `[]`, the root volume was already gone
1384 (`InvalidVolume.NotFound`), and the `false` volume survived as
1385 `available`. So disk evidence comes from records taken while the box
1386 runs, and decides only the discharge of local chores (H19, §5.9). The
1387 rehearsal terminates a box with an all-`true` mapping (chores
1388 discharged) and one with a `false` volume (fenced and relocated
1389 alike, chores left open).
1390 The rehearsal also enrolls a box into a dev region with no `active`
1391 box, confirmed from another region (H17), and checks that the
1392 cross-region statement allows only the two describe calls.
13936. Then specs 2, 3, 4.
1394
1395## 9. Changes requested of the claim and moves specs
1396
1397For platform-1e (claim rev 6.25, 413b12cb) and platform-d8 (moves rev
13987.17, 39904c84). They replace rev 4.2's K1 to K11, which those sessions
1399never received. "Ruled" means Paul's rulings of 2026-10-06 (H10 to H19,
1400and amadan #236) decide the direction; the owning session still writes
1401the text.
1402
1403| # | Amends | Change | Status |
1404|---|---|---|---|
1405| L1 | claim §4, §8, §6.4a (which kinds and phases admit), §7.3 D5 and D6 | New op kind `relocating` (§5.8): same region, stateful, from `serving`; its phase table with `op.exec`, stamp, rows and admission per phase; the `installed` no-reinstall boundary for the remote build, with a per-box local install receipt, bound to the stamp of the CAS that handed the box `op.exec`, required before `probing` and never remade after the first admission; probe attempts after the first restoring from their predecessor; the replacement table; the source-loss conversion to failover from `starting` or `releasing`; two replacements then `stuck`; the release sequence. Recovery owner `region:<r>`, any regional worker (P6-D5). The D6 barrier needs only `to` when the start cites a termination fence or after a source-loss CAS. Excludes moves, placements and pushes. | Ruled in direction (H10; #236: "a new op kind, or an equivalent that the claim prefers") |
1406| L2 | claim §4, C5, §7.2, §7.3 D2 | A termination fence (§5.9) is a lost-box fence for execution, written by a regional worker with no person in the loop: a verified `terminated` read (event, describe or, later, the group's history), bound to the roster entry's `cloud`. No disk condition. A `shutting-down` read is never evidence. Stopped, unclear and vanished boxes keep D2's operator path unchanged. | Ruled (H10, H19) |
1407| L3 | claim §7.3 D5 | A fenced terminated box's `park-local` chores are discharged automatically in the fence CAS only when its volume records, from enrollment to its last running record, show every attached EBS volume `DeleteOnTermination` and none removed; that is D5's destroyed-local proof for discharge. Otherwise they stay open for D2/D5's operator path, and no recovery step waits on them. Regional chores move to `region:<r>` as D5 says. | Ruled (H19) |
1408| L4 | claim §7.3 D3 and D4, §5.9 P5-D3 | The fence record `control/box-fences/<enrollment>.json`, create-only, box-written, cited by `evidence_refs`. A terminated box goes on `lost` with that reference and no quarantine revision; it needs no readmission. The console later copies it into a D4 loss record and retires the roster entry. | Ruled in direction (H10, H1); record format for the claim session |
1409| L5 | claim §10.1 D7 (required actors, console enrollments) | An enrolled box with a valid fence record is excluded from the census without waiting for the console to retire it. A console enrollment gains `compute: {box, enrollment}`, checked against its reports; a console whose `compute` box is fenced is excluded the same way (B1-7). | Ruled (H10, H14; #236: failover works when the console's box died) |
1410| L6 | claim §10.1 D7 (lifecycle reservations, `actors_hold`), §7.3 D6 | For operations a box starts, replace the console reservation with §5.10's box-side check: the worker reads (never writes) the roster and refuses if an update is reserved for `from` or `to`; a participant writes its `pending` controller state, then refuses to acknowledge while root has an update begun; root writes its update-begun marker, then refuses an update while any controller record exists in any state, `lost` included, or cannot be read. A record is removed only by D6's clearing with its cleanup complete (B1-5). | Ruled (H14) |
1411| L7 | claim §10.1 D7, §7.3 D2, §5.9 P5-D2, §6.6 P6-D2 | Enrollment of a platform box requires `internal/boxid` evidence; it records `cloud` from it, and checks the tofu `instance_id` for a reference already in claims or edges files. No other roster, and no `control/boxes` authority. | Ruled (H11) |
1412| L8 | claim §5.1 steps 2 and 4, §5.9 P5-D2 and P5-D6 | The executor in the create intent is `choose()` (§5.2), with P5-D2's SHA-256 as the final tie-break; chosen once, never substituted; a create whose box can't complete is cancelled. | Ruled (H12) |
1413| L9 | claim §6.4a item 2, `hostclaim.AdmitWriter` | A coverage value `released`, legal only on the admission that opens a prepared generation built from a pin reaching the `final_txid` a released box recorded from its stopped database, and only when the admission's predecessor (the last `holder.admissions` entry) is exactly that box and session, under its stop receipt. It declares nothing. Without it every drain declares a false loss (B1-1). | Requested |
1414| L10 | claim §6.1, §6.6 P6-D1, `internal/stopevidence` | A writers-stopped check for release step 1: cgroup empty, no member processes, and database holders limited to the agent's own identified replicator descriptors. `Collect` stays unchanged for step 4. | Requested |
1415| L11 | claim §4 (CC11), `hostclaim.StartGuard` | A `park-generation` chore in the region does not block a `relocating` start there: relocation writes only its own new prepared generation, never one being retired. Under P6-D4 nearly every recently woken host has one, which would hold failover up to the park horizon. | Requested |
1416| L12 | claim C4, §8 (CC10), §6.6 P6-D2; moves §9.6 | Reconcile's row cases for `relocating` (§5.3): held participant rows; a former executor's row kept held until its `park-local` retires it; the 10-minute sweep as a stated exception to C4 for kept rows (H9). | Requested |
1417| L13 | claim C3, §6.2a, §6.6 P6-D3 | The files publisher is a second ordinary-replication exception, bound to `{location, generation, session}`. `files/<generation>/` belongs to that generation for the classifier; `park-generation` retires it except objects named by the files live set: every manifest from the files pin through `holder.generation`, and any manifest a running operation's pin journals (B2-4). | Requested |
1418| L14 | moves §7.10 | A target terminated mid-move may be taken off the move by the termination fence, as M16's evidence, automatically, under the same before-commit rules. Moves decides what the move does then; the placer only supplies the fence. | Ruled (H10) |
1419| L15 | moves §7.1, §7.3, §19.1 | A move into a multi-box region names its target **box** in the start CAS, chosen by `choose()`. Readiness, evidence, install and recovery bind to that box; replacing it is an explicit fenced transition (A1-10). | Ruled (H12) |
1420| L16 | moves §5.1, §7.6, P6-D3's provider | Relocation's install reuses the landing's `building` and `published` phases and the prepared-generation authority, with a pinned set of remote versions as the source in place of a staged replica. One shared routine, so both get the same anchor and manifest rules. | Requested |
1421| L17 | moves §6.1, §8, §9.4 | Every installation, a move's landing included, writes a files manifest in its new generation (§5.6), so unchanged files stay recoverable after the next failure; the archive at the cut stays. Timer freezing and draining apply to `relocating` holds. | Requested |
1422| L18 | joint model, moves §14.2 | `internal/sim/hostlife` extends to two boxes per region: create spread, relocation by drain, termination mid-serve and in every relocation phase, nested failover, a stale edges file, an unenrolled reference, a volume without `DeleteOnTermination`, a crash mid-install, late file uploads and late PUTs after termination. | Requested |
1423| L19 | claim §10.1 D7 (enrollment), with P3 | An Auto Scaling box is enrolled by the console with no operator when its `boxid` evidence verifies (deployment account, the region's AWS region) and a regional worker's create-only confirmation shows it a current member of the region's group (§5.1). Every other enrollment stays operator-approved. Spec 4 builds it. For a region with no live enrolled box (every enrollment there read `terminated`, positively, in the same pass, B2-6), any `active` box of another region confirms, by the same create-only write, bound to the group ARN the roster records for that placement; its cross-region rights are the two read-only describes and nothing else (H17). | Ruled (H16, H17) |
1424| L20 | claim §10.1 D7 (required actors, console reports) | For box-started relocations only, a console's newest report counts however old if it met every floor check and the console's `compute` box has reported its D1 root update guard armed with the same floor digest. Every other census rule is unchanged. | Ruled (H18) |
1425
1426**Rev 4.2's requests, mapped.** K1 is L8. K2 is L1, with L9 to L11. K3
1427is L2 to L5. K7 is L14. K8 is L16 and L17. K9 is L18. K10 is L15. K11 is
1428L12. Three are dropped because the claim now satisfies them:
1429
1430- **K4** (claim CAS rights for placers): boxes already write claims
1431 (`AdmitWriter`, `PublishGeneration`, chores), and P6-D5 gives
1432 `region:<r>` work to any enrolled worker. L1 names the owner.
1433- **K5** (adoption only while a region has one box): P5-D2 requires
1434 exactly one proven installation owner and refuses zero or several,
1435 which is the property K5 wanted, whatever the box count.
1436- **K6** (box names bound by `control/boxes`): claim §2 rev 6.16 makes
1437 references opaque, and H11 makes the roster the binding (L7).
1438
1439The disposition tables (§13 to §17) still cite K numbers. They refer to
1440rev 4.2's table; this mapping says where each now lives.
1441
1442## 10. Testing
1443
1444- **Unit:** enrollment evidence (signature, account, region, the tofu
1445 check for an existing reference, a reference held by another
1446 instance); `eligibleBoxes` (one box at a time, staleness, draining,
1447 roster state, `lost`, an owed `park-local`, capability, the expected
1448 region from the roster); `choose` (spread, fewest hosts, the SHA-256
1449 tie-break); home-map stamp resolution; the files manifest (immutable
1450 objects, tombstones, late uploads, `If-Match` races, a cold admission's
1451 copied manifest naming the predecessor's objects); timer catch-up;
1452 termination-fence acceptance (NotFound, wrong account, wrong region,
1453 stopped, all sources empty, an event alone, a describe alone, a
1454 `shutting-down` read offered as evidence, and the fence accepted with
1455 no volume record at all); chore discharge (all records `true`:
1456 discharged; a `false` flag, a removed volume or no record: left open,
1457 with the relocation unaffected); a box starting while
1458 `control/box-volumes/` cannot be written: it mints its relocatable
1459 rows, takes part in relocation and recovers a dead peer's hosts,
1460 and only alerts (B4-1); the volume recorder (writes on change,
1461 at least hourly, `removed_volumes` when a volume leaves the mapping);
1462 cross-region confirmation (accepted only when the target region has no
1463 `active` box, refused for another group, an instance not
1464 `Pending`/`InService`, a confirmer not `active`, a region mismatch, a
1465 second confirmer losing the create-only race, and an emptied region
1466 whose dead box is still `active`: confirmed on a `terminated` read,
1467 refused on NotFound or `stopped`, B2-6); the console census for a box-started
1468 relocation (a stale report accepted with an armed guard, refused with
1469 no `compute` binding, no armed-guard report or a floor mismatch, and
1470 unchanged for a console-started operation); continuous files with a
1471 file held open (published at the deadline), inotify overflow, a file
1472 that changes during its read; the create-only and replace split in
1473 `hostfiles.Publish`.
1474- **Relocation:** every row of the replacement table, the
1475 two-replacement bound, `probe-failed` without stop evidence, both
1476 participants' rows through every phase, the first probe opening `P`
1477 with `KeepStamp` and a restarted probe taking a fresh generation, and
1478 relocate → write → replicate → terminate → cold restore with the
1479 level-0 anchor; a release with a live embedded replicator (step 1
1480 passes with only its descriptors, step 4 passes only after `DB.Close`,
1481 any other holder fails step 1); a failover admission declaring
1482 `writes_lost` for `from`'s session and a drain declaring nothing; the
1483 start refused when the roster reserves an update for `from` or `to`;
1484 a files update published by a probe session before its first database
1485 upload, then termination: the files pin keeps it while the database
1486 baseline falls back (B2-4); a crash during `probing` after the second
1487 admission: no reinstall, and the receipt still valid by
1488 `install_stamp` (B2-5); probe attempts one, two and three, with attempt three restoring
1489 attempt two's replicated writes and declaring attempt two's session
1490 (B1-1); a target terminated at `installed` replaced by one that makes
1491 its own local install before probing, and a `probing` CAS refused with
1492 another box's receipt (B1-2); `from` terminated in `starting` before
1493 and after acknowledging (B1-6).
1494- **Starts and updates (§5.10):** root's begin and a participant's
1495 acknowledgement interleaved at every point, with a crash between each
1496 marker and its check: never an update and an acknowledged
1497 participation at once, and a crash leaves at most a refusal, cleared
1498 by retry; a lease lost during `probing` with cleanup failing, and an
1499 unreadable controller record, both refusing the update (B1-5).
1500- **Integration:** two agents against one fake region, one control
1501 bucket: create lands on the chosen box only; the other box mints no
1502 row; the edge backhauls to the executor; relocate by drain through
1503 every phase with a crash injected at each; the old box's local copy is
1504 never served after it becomes executor again (claim R6).
1505- **Joint model (L18)** decides readiness, as it does for the claim and
1506 moves.
1507- **Rehearsal** (§8 step 5).
1508
1509## 11. Open questions
1510
1511None.
1512
1513Rev 4.2's three were ruled as H7, H8 and H9 (2026-10-04). Rev 5's first
1514three were ruled on 2026-10-06 as H14 (the box-side reservation check,
1515§5.10), H15 (the running-time volume record, since superseded by H19)
1516and H16 (automatic enrollment, §5.1, L19).
1517
1518Astra's rev 5 review raised two more, ruled the same day as H17 (an
1519enrolled box in another region confirms the first box of an emptied
1520region, §5.1, §6, L19) and H18 (a box-started relocation accepts a
1521console's newest report when its box's update guard is armed, §5.9,
1522L20). Astra's rev 5 round 3 found more ways around the volume guard;
1523Paul answered with H19, which takes disk evidence out of the failover
1524path (§5.9, §22).
1525
1526## 12. Out of scope, for the later specs
1527
1528- **Spec 2:** `headroom` thresholds, soft spread across an app's
1529 instances, the `unplaced` state and its alert, rebalancing.
1530- **Spec 3:** the placer loop: reading the termination queue, relocating
1531 every host off a terminated box in bounded parallel, waking always-on
1532 and scheduled instances, the failover gap target.
1533- **Spec 4:** the Auto Scaling group per region, launch template and
1534 join without SSM hand steps, automatic enrollment of new boxes (H16,
1535 L19),
1536 ingress for boxes that come and go (a per-region NLB with fixed EIPs,
1537 or an EIP pool claimed at boot), HTTP-01 certificates, steering,
1538 scale-out on headroom, scale-in by drain (§5.8), per-region minimum
1539 (H5).
1540- **Not planned:** relocating unit- and socket-backed routes (the
1541 console and platform services). Their HA is the HA ladder's, separately.
1542
1543## 13. Astra round 1 dispositions (rev 2)
1544
1545| Finding | Disposition |
1546|---|---|
1547| A1-1 P1 termination evidence not bound to identity | accepted: AWS-signed box records (§5.1), HA7, fence fields (§5.9), K6 |
1548| A1-2 P1 new generation never populated | accepted: pinned recovery set and `installing` builds `G'` (§5.8), K8 |
1549| A1-3 P1 replay overwrites an installation that has written | accepted: phase table, `installed` boundary, participant-terminated exits (§5.8), K2 |
1550| A1-4 P1 file update and tombstone order | accepted: immutable objects, per-host `seq`, `If-Match` manifest, snapshot boundary (§5.6) |
1551| A1-5 P2 `active_stamp` is not install evidence | accepted: installation-complete marker (§5.5), HA3 |
1552| A1-6 P2 participant rows and former executors | accepted: three row cases (§5.3), K11 |
1553| A1-7 P2 timers not reconstructible | accepted: timer state replicated continuously, imported with `ReplaceTimers`, catch-up (§5.7) |
1554| A1-8 P2 unit and socket hosts | accepted: not relocatable (§2, §7, §12) |
1555| A1-9 P2 adoption alone does not admit a second box | accepted: capability, HA floor and multi-box gate (§8), unstamped entries (§5.4) |
1556| A1-10 P2 moves target box | accepted: K10 |
1557| A1-11 P2 termination evidence can vanish | accepted: EventBridge to SQS with 14-day retention, NotFound never evidence (§5.9) |
1558| A1-12 P2 Describe permissions by tag | accepted: `ec2:Region` condition, membership checked by the verifier (§6) |
1559| A1-13 P3 citations | accepted: `lease.go:37`, `reconcile.go:202`, `edgesfile.go:272`/`:326` |
1560
1561## 14. Astra round 2 dispositions
1562
1563| Finding | Disposition |
1564|---|---|
1565| A1-1 residual: a legacy label's first registration | accepted: an operator's bind checked against the region's tofu `instance_id` output and the claims' existing label (§5.1) |
1566| A1-7 residual: `routes.next_due` | accepted: wake state carries `NextDue` and `Replication` with the timer rows (§5.6, §5.7) |
1567| A2-1 P1 snapshot-only baseline loses the replication position | accepted: moves rev 7.6's replication baseline with the level-0 anchor, verified before `installed`, and its test (§5.8), K8 |
1568| A2-2 P1 early target replacement skips release and pinning | accepted: phase-specific replacement table (§5.8), K2 |
1569| A2-3 P2 reconcile drops the relocating source | accepted: per-phase rows for both participants (§5.3, §5.8 table) |
1570| A2-4 P2 event delivery is best-effort | accepted: a third method (Auto Scaling activity history, verified in spec 4 first), `awaiting-evidence` with an immediate alert, and operator resolution via CloudTrail when all are empty (§5.9) |
1571| A2-5 P2 close and rename don't bound continuous sync | accepted: `IN_MODIFY` with a fixed one-second deadline from the first modification, recursive watches, overflow scans, a stable-read check (§5.6) |
1572| A2-6 P2 create-only conflicts with manifest replacement | accepted: create-only and conditional-replace writes separated, enforced in one writer (§6) |
1573| A2-7 P2 a live target whose probe never passes | accepted: a probe deadline, `probe-failed` with stop evidence, a nested relocation that keeps probe writes, and `stuck` after two replacements (§5.8), K2 |
1574
1575## 15. Astra round 3 dispositions
1576
1577| Finding | Disposition |
1578|---|---|
1579| A2-5 residual: an unstable file waits for the scan; a failed watch | accepted: published at every deadline as the last image, flagged; per-second polling while coverage is missing (§5.6) |
1580| A2-7 residual and A3-2 P1: probe failure skips the target's durability barrier | accepted: one release sequence for the source and a failed target, journaling `final_txid` and `manifest_seq` before stop evidence, all files flushed whatever the plan; pinning must reach them (§5.8), K2, K8 |
1581| A1-1 residual and A3-3 P1: the bind isn't operator-authenticated | accepted: binds name the exact record version; the control bucket's policy limits binds to operator roles and denies replace and delete under `control/boxes/` (§5.1) |
1582| A3-1 P1: a CloudTrail request is not compute fencing | accepted: claim §7.2's terminal-state rule unchanged; CloudTrail is a finding aid only; otherwise `awaiting-evidence` (§5.9), K3 |
1583| A3-3 (part): S3 can enforce conditional headers | accepted: `s3:if-none-match` and `s3:if-match` policy conditions, scoped per prefix (§6) |
1584| A3-4 P3: citations | accepted: `internal/host/fleetbind.go`, `edge.go:2432`, `homemap.go:538`/`:582`, `reconcile.go:335`/`:799` |
1585
1586## 16. Astra round 4 dispositions
1587
1588| Finding | Disposition |
1589|---|---|
1590| A4-1 P1: the release sequence flushes before stopping writers | accepted: stop and empty the process tree and timer deliveries first, then flush and journal, then finish stop evidence (§5.8), K2 |
1591| A4-2 P2: the conditional Deny blocks multipart preparation | accepted: `s3:ObjectCreationOperation: true` on every header Deny; multipart completion carries the header; tested (§6) |
1592| A4-3 P3: 29245f69 is not moves rev 7.6 | accepted: rev 7.6 named as an uncommitted draft, rev 7.5 at cdc98845 as the last committed; the plan pins the commit (header) |
1593
1594## 17. Astra round 5 dispositions
1595
1596| Finding | Disposition |
1597|---|---|
1598| A5-1 P1: step 1 waits for the replicator's handle, closed only in step 4 | accepted: step 1 allows only the box's own replicator descriptors; step 4 closes it and runs the unrestricted check; tested with a live replicator (§5.8), K2 |
1599
1600## 18. Notes from plan 1a (2026-10-05)
1601
1602Plan `docs/superpowers/plans/2026-10-05-region-ha-1a-foundations.md` builds
1603the parts of this spec that need no claim. It keeps every property above
1604and changes three mechanisms:
1605
16061. **Exact content by hash, not by S3 version id (§5.6, §5.1).** The store
1607 abstraction has no version-id read and the box role has no
1608 `s3:GetObjectVersion`. Objects are immutable, their key contains their
1609 SHA-256, the manifest records it, and a reader verifies it. A bind
1610 names its record by `record_sha256` in place of `record_version_id`.
16112. **The identity signature (§5.1).** The record carries IMDS's
1612 base64 RSA-SHA256 signature (`/latest/dynamic/instance-identity/signature`),
1613 verified with the regional certificate, in place of the PKCS7 form.
1614 Same document, same signer, no new dependency.
16153. **`snapshot` is a periodic full publish, not a sealed archive (§5.6).**
1616 `hostfiles.SyncDir` stages every difference between the directory and
1617 the manifest. Run on the snapshot interval it gives the `snapshot`
1618 setting; run every ten minutes it is the continuous setting's repair
1619 scan; run before stop evidence it is the release flush. The manifest
1620 therefore has no `base` or `base_seq`, and restore reads one manifest
1621 and the objects it names.
1622
1623Two corrections to the spec, found while planning:
1624
1625- **§5.4's alert for two unstamped entries is dropped.** Static sites and
1626 public address routes run on every box, so every box publishes them as
1627 `home`, unstamped: that is normal, and the edge cannot tell them from
1628 relocatable hosts. The edge reports only two boxes at the same positive
1629 stamp. Plan 1b decides whether the publisher marks relocatable entries
1630 so the edge can alert on those alone.
1631- **§5.1's region check has two halves.** `boxid.Verify` proves which
1632 region a box is in. Whoever uses that compares it with the region it
1633 expects: `eligibleBoxes` takes the placement's AWS region and refuses a
1634 box whose signed region differs, and `termfence.Verify` compares the
1635 observation's region with the signed one. Plan 1b must supply the
1636 expected region from something better than a majority vote of the
1637 unsigned edges files.
1638- Build tags (Task 6): the descriptor-based directory code is `//go:build unix`, not `linux`, so the CLI's macOS build uses it too; Windows gets a stub that publishes nothing.
1639
1640Two limits to lift in plan 1b: a file over 64 MiB is recorded in the
1641manifest as skipped and is not published (the store takes a whole body in
1642memory), and the third termination-evidence source (Auto Scaling activity
1643history) is not built until spec 4 verifies it.
1644
1645## 19. What rev 5 means for plan 1a (2026-10-06)
1646
1647Plan 1a (aadb860) was built against rev 4.2. Rev 5 keeps most of it and
1648changes how some of it is used. Plan 1b is written against rev 5.
1649
1650**Superseded.**
1651
1652- **`control/boxes/<id>.json` as authority (H11).** `boxid.Load` and
1653 `boxid.Verifier` are now used once, at enrollment, and their result is
1654 stored in the roster entry's `cloud`. `eligibleBoxes` takes its
1655 verified identities from the roster, not from loading records. The
1656 record object stays as the enrollment evidence, create-only, and grants
1657 nothing.
1658- **The operator bind** (`Bind`, `control/boxes/<label>.bind.json`, `carlos ops
1659 box bind`, plan 1b item 13). Its tofu check moves into enrollment
1660 (§5.1). The bucket policy's bind statement is harmless and can go at
1661 the next apply.
1662- **`WarmCopyTrusted` and the activator's warm path (H13).** Nothing skips
1663 a restore, so plan 1b item 4 is dropped. The `InstallMarker` type may
1664 serve as relocation's last local install journal entry, as moves'
1665 `.files-gen` does, or be removed; nothing reads it to trust a local
1666 database.
1667- **`chooseBox`'s last tie-break.** The lowest box id becomes P5-D2's
1668 greatest SHA-256 of `host + "\x00" + reference` (H12). The first two
1669 rules stand.
1670- **`termfence`'s binding and inputs.** `Verify` binds to a
1671 `boxid.Identity`; it must bind to the roster entry's `cloud` and
1672 produce §5.9's record. It takes no volume input (H19); a separate
1673 function decides chore discharge from the volume records. `FromDescribe` and `FromEvent` stay as the two terminated
1674 reads; a `shutting-down` observation is never accepted. Plan 1b adds
1675 the volume recorder, the discharge check, §5.10's two checks and, for
1676 spec 4, the membership confirmation (H16).
1677
1678**Stays as built.**
1679
1680- `internal/boxid`'s record, signature check and embedded regional
1681 certificates (verified on the dev box), as enrollment evidence.
1682- The edges file's `box_id` (the label, which is the reference under
1683 #212 option B), `capabilities`, `draining` and `headroom`.
1684- The home map settling two claims by installation stamp, and reporting
1685 equal stamps.
1686- `eligibleBoxes`' one-box-at-a-time predicate, with the roster as its
1687 source.
1688- `internal/hostfiles`: the ordered manifest, immutable hashed objects,
1689 `SyncDir`, the continuous publisher, the watcher, restore and wake
1690 state. Plan 1b adds the generation binding, the copied manifest at
1691 each cold admission, and retirement by the claim (§5.6).
1692- `registry.ReplaceTimers`.
1693- The infrastructure: one termination queue per box, root volumes deleted
1694 on termination, the create-only `control/boxes/` policy. The fence
1695 prefix `control/box-fences/` needs the same create-only, never-deleted
1696 statements (§6).
1697
1698**Plan 1b's list, revised.** Items 1, 3, 5, 9 and 10 stand, read against
1699rev 5 (item 10's expected region now comes from the roster). Item 2
1700follows §5.3's revised table. Item 4 is dropped. Item 6 becomes
1701publishing the enrollment evidence at start. Item 7 is §5.8 as rev 5
1702states it, and waits for L1 and L9 to L11. Item 8 adds the roster check
1703to the gate; volume recording is not part of it (B4-1).
1704
1705Rev 5.1 adds to plan 1b: the removed-volume rule (§5.9), the local
1706install receipt (§5.8), and root's refusal on any controller record
1707(§5.10). Rev 5.1 and 5.2's hold, release, guard attestation and SCP are
1708withdrawn by H19 and are not built.
1709
1710## 20. Astra rev 5 round 1 dispositions
1711
1712Reviewed rev 5 at 2a9165d4 (`CODEX-FINDINGS-REGION-HA-REV5-1.md`). Each
1713citation was checked against claim 98b45332, moves a4711728 and main
1714before acceptance.
1715
1716| Finding | Disposition |
1717|---|---|
1718| B1-1 P1 probe retries restore `P` again and lose replicated writes; `released` suppresses loss too widely | accepted: claim §6.4a item 3 and moves §7.14's restore row choose a fresh admission's baseline from its predecessor. Only the first attempt opens `P`; later attempts restore through `genselect.Select`; coverage is decided per admission, `released` only on `P`'s opening with a predecessor matching the journaled release (§5.8, L9) |
1719| B1-2 P1 a replacement at `installed` never makes its local install | accepted: the remote build and the per-box local install are separate; `probing` requires a local install receipt bound to `op.exec`, its enrollment and `holder.stamp`; a replacement makes its own from `P`'s files manifest, and `P` is still built once and opened once (§5.8, L1) |
1720| B1-3 P1 an operator's volume change before a refresh, or a detach, allows a false destroyed-disk fence | accepted: a hold written before the change suspends disk evidence; a volume leaving a running box's mapping suspends it too; release needs a newer all-true record and an operator's attestation for every removed volume (§5.9, §6, §7) |
1721| B1-4 P1 the guard rests on an unprotected tag; the non-Organizations fallback still fences | accepted: the SCP also denies changing `carlos:role` except at launch; exemption narrowed to one hold-gated role; a root-signed guard attestation is required, and without one there is no automatic fence. Stricter than H15, never weaker (§5.9, §6, §8) |
1722| B1-5 P1 a `lost` controller stops excluding updates | accepted: claim D6 does define `lost` and keeps its deny on failed cleanup. Root now refuses on any controller record in any state or any unreadable one, until D6's clearing with cleanup complete (§5.10, L6) |
1723| B1-6 P2 the source lost in `starting` has no transition | accepted: a source-loss CAS converts the op to failover from `starting` or `releasing`, keeping the intent and acknowledgements, removing `from` from the barrier (§5.8, L1) |
1724| B1-7 P2 a dead console stays in the D7 census | accepted: claim D7 does require every console. Console enrollments gain a `compute` binding; a console on a fenced box is excluded. A live console with a stale report was ruled as H18 (§5.9, L5, L20) |
1725| B1-8 P2 automatic enrollment cannot bootstrap an emptied region | accepted: §2 makes a confirmer an active enrolled box in the region, so none is left. ruled as H17: an enrolled box in another region confirms, with read-only cross-region describes only, so one-box regions recover automatically (§5.1, §6, §7, L19) |
1726| B1-9 P3 stale code citations | accepted, checked on main: `provision.go:49`, `reconcile.go:348`/`:820`, `:192`, `:215`, `activator.go:1164-1168`, `:2204`, `edge.go:2466`, `main.tf:521`. Disposition tables §13 to §17 keep their historical anchors |
1727
1728## 21. Astra rev 5 round 2 dispositions
1729
1730Reviewed rev 5.1 at fdc0fe07 (`CODEX-FINDINGS-REGION-HA-REV5-2.md`).
1731Each finding was checked against claim 98b45332, moves a4711728, main
1732and AWS's documented API and policy contracts before acceptance.
1733
1734| Finding | Disposition |
1735|---|---|
1736| B1-2 partly resolved: the receipt dies with the next probe stamp | answered by B2-5 |
1737| B1-3 partly resolved: release can precede authorized mutations | answered by B2-2 |
1738| B1-4 partly resolved: root replacement unguarded; the attestation conflicts with Auto Scaling | answered by B2-1 and B2-3 |
1739| B1-8 partly resolved: H17's emptiness test is circular | answered by B2-6 |
1740| B2-1 P1 `CreateReplaceRootVolumeTask` bypasses the guard | accepted: added to the deny, with snapshots and images (which also leave a copy of the disk), the hold workflow and the rehearsal; a replaced root volume shows as removed (§5.9, §6, §8, L2) |
1741| B2-2 P1 a release does not close the change's authority | accepted: the hold names its instance and the role's exemption is limited to it; release revokes the role's earlier sessions, waits for no attaching, detaching, replace-root, snapshot or image task and for the signature-age bound, then needs a validating record after that barrier. Session tags are acknowledged as the caller's word; the hold-first order is the operator trust boundary, as in claim D2 (§5.9) |
1742| B2-3 P1 Auto Scaling's service-linked role defeats the attestation | accepted: the attestation no longer asserts that no service role can touch a disk. It reads every service-linked role's managed policy at its default version and requires each disk or tag action to map to an initiating API the SCP denies; Auto Scaling's (refresh, group create and update, warm pool, attach) are denied but for the hold-gated role; a policy version change invalidates it (§6, §7) |
1743| B2-4 P1 the files pin follows the database baseline and can drop published files | accepted: the files pin is the newest manifest in the admitted chain, chosen independently; the files live set covers it through `holder.generation` and any running op's pin (§5.6, §5.8, L13) |
1744| B2-5 P2 later probe stamps invalidate the receipt and invite a reinstall | accepted: the receipt binds to `op.journal.install_stamp`, the stamp of the CAS that handed `op.exec`; after the first admission the local install is never remade, and a lost receipt is `stuck` (§5.8, L1) |
1745| B2-6 P2 the dead box stays `active`, so H17 never fires | accepted: the confirmer requires a positive `terminated` read for every `active` or `quarantined` enrollment in the region, in the same pass, as part of confirming membership; it is not a fence and retires nothing. The new box then fences through the ordinary path (§5.1, §7, L19) |
1746
1747## 22. Astra rev 5 round 3 dispositions, and H19
1748
1749Reviewed rev 5.2 at 075d709f (`CODEX-FINDINGS-REGION-HA-REV5-3.md`).
1750Each finding was checked against AWS's documented contracts. All five
1751are about the volume guard. Paul's ruling H19 (2026-10-06) took the disk
1752out of the failover path, so they no longer bear on safety.
1753
1754| Finding | Disposition |
1755|---|---|
1756| B3-1 P1 Recycle Bin keeps volumes deleted on termination | correct about AWS, and moot for the fence under H19: a retained volume cannot run or write, and is never admitted as the executor. For chore discharge it is a retention concern for the operator path; §6 asks the deployment to configure no retention rule over platform volumes (§5.9, §6) |
1757| B3-2 P1 `CopyVolumes` bypasses the copy guard | correct, and moot under H19 for the same reason: a copy is a data-retention concern, never a second writer (§5.9) |
1758| B3-3 P1 another Auto Scaling group can overwrite the protected tag | moot: H19 removes the tag-conditioned guard; nothing in the fence or discharge reads the `carlos:role` tag any more (§5.1, §6) |
1759| B3-4 P1 concurrent releases move the revocation cutoff back | moot: holds, releases and the change role are withdrawn (§5.9, §6) |
1760| B3-5 P2 the release barrier omits Auto Scaling workflows | moot: there is no release barrier. A root replacement under any workflow shows as a removed volume, which only keeps the chores open (§5.9) |
1761| B2-1, B2-2, B2-3 (partly resolved in round 3) | withdrawn with the mechanisms they patched; the property they protected is no longer needed for execution (H19) |
1762
1763**Earlier volume-guard findings, revisited.** B1-3 (operator changes
1764before a refresh), B1-4 (an unprotected tag, the non-Organizations
1765fallback), B2-1 (root replacement, snapshots, images), B2-2 (release
1766authority) and B2-3 (service-linked roles) all showed ways a disk could
1767outlive its box despite the records. Under H19 each such disk is a
1768cleanup concern: the box is fenced by its `terminated` read alone, and
1769a surviving disk can never become the executor, because admission needs
1770`op.exec` to name an enrolled, active box (claim C4, HA7), and a new
1771instance is a new enrollment. What remains of them is one cheap rule:
1772any `false` flag or removed volume in any record keeps the dead box's
1773chores open for an operator. Their dispositions in §20 and §21 stand as
1774history.
1775
1776**Checked after removing the guard.** H17's confirmer no longer reads
1777the `carlos:role` tag, and needs nothing else from the guard. The
1778volume recorder stays, for discharge only. §5.10, the census (L5, L20),
1779relocation (§5.8) and the fence read no disk evidence. CC11 lets a
1780relocation start while a dead box's `park-local` is open, since it
1781names only that box. A surviving disk attached to a new instance brings
1782host artifacts the new enrollment does not own; claim P6-D2 refuses to
1783initialize or materialize over ambiguous ownership, and every writer
1784restores from the certified remote anyway (claim R6), so the old
1785database is never served.
1786
1787## 23. Astra rev 5 round 4 dispositions
1788
1789Reviewed rev 5.3 at 27058fe1 (`CODEX-FINDINGS-REGION-HA-REV5-4.md`).
1790Astra agreed B3-1 to B3-5 are moot under H19, and found no two-writer
1791or data-loss path that depends on disk evidence.
1792
1793| Finding | Disposition |
1794|---|---|
1795| B4-1 P2 volume recording is still a runtime admission prerequisite through §8's gate | accepted: under H19, volume recording never gates execution, admission, relocation or recovery. The gate no longer checks it; a box's start re-checks only capability, floor and roster; recorder failure alerts and keeps a dead box's chores open; tested through start, row minting and recovery (§8, §10, §19) |
1796| B4-2 P3 summaries still describe the disk-dependent fence | accepted: the source-loss table discharges chores only on qualifying records (§5.8); the tofu note says `delete_on_termination` affects only discharge (§6); the one-box row fences on the `terminated` read alone (§7) |
1797
1798## 24. Notes from plan 1b (2026-10-06)
1799
1800Plan `docs/superpowers/plans/2026-10-06-region-ha-1b-claim-readers.md`
1801builds the parts of rev 5.4 that need none of L1 to L20: enrollment
1802evidence, §19's corrections, the agent publishing its identity record,
1803the fence bound to the roster's `cloud`, volume records and the discharge
1804check, the edges file following `op.exec`, and root's half of §5.10. All
1805of it is dormant behind `CARLOS_REGION_HA`. It keeps every property above
1806and settles five details:
1807
18081. **Enrollment refuses one instance under two references (§5.1).** The
1809 record's `box` field is not covered by AWS's signature, so box A's
1810 record copied to B's key with that field rewritten still verifies.
1811 `boxid.Verifier.Enroll` refuses an instance the roster holds under another
1812 reference, as well as a reference the roster holds for another
1813 instance, counting entries in every state.
18142. **A positive stamp is the relocatable mark (§18's open question).**
1815 Only relocatable hosts with a claim, and their aliases and served
1816 catch-alls, carry a stamp, so the edge's report
1817 of two boxes at one positive stamp already concerns them alone. No
1818 separate marker.
18193. **The edges file's claim reads (§5.4).** A relocatable host with no
1820 claim is published as before, unstamped: §8's gate admits a second box
1821 only once every host has a claim. A read that fails keeps the last
1822 decision. Each read stands 10 minutes (H9), at most 256 reads a pass.
18234. **"The first record taken at enrollment" (§5.9)** is the earliest
1824 volume record for the enrollment, observed within one hour of the
1825 roster's enrollment time. A later first record cannot speak for a
1826 volume attached and removed before it, so the chores stay open. And
1827 (controller ruling, 2026-10-06) the newest record must be no more
1828 than two hours older than the termination observation; otherwise the
1829 chores stay open for the operator path.
18305. **No volume record from a `terminated` read.** Its mappings are empty
1831 (§8 step 5's dev test), and recording it would list every volume as
1832 removed.
1833
1834It also removes plan 1a's `Bind`, `BindKey`, `Verifier.Verify` and
1835`activator.WarmCopyTrusted`, and makes every box-evidence prefix
1836create-only and never deleted in the deployment bucket's policy.
1837
1838Controller rulings of 2026-10-06, on Paul's behalf: the edges file
1839withholds a relocatable host when `op.exec` names another box or none
1840(§5.4 as written); root's §5.10 refusal stays behind `CARLOS_REGION_HA`
1841until claim D6 lands, then becomes unconditional; the `*.bind.json`
1842statement and `operator_role_arns` stay until no root sets the variable.
1843