amadan

carlos / platform Public

region HA 2b: plan and spec rev 21 for the final review

D1: the plan's host.env Optional snippet is the built one, without
CARLOS_DNSIMPLE_TOKEN, the three status destination overrides or
CARLOS_ALERT_WEBHOOK; Global Constraints says why once.
D2: CheckConfirmed gives way to the built shape: the unexported
checkConfirmed returns its confirmation, the pass binds the
invitation's deadline to it, and an invited box is found with the
read-only enrollmentNonce. Census overlay comment lines left as the
overlay has them.
D3: 2c's seams: the bootstrap binary sits outside both protected
slots; no launch lifecycle hook on the group; control/box-enrollment
leaves infra/modules/deployment-buckets; host-env write needs
--aws-region or AWS_REGION.
D4: activation gates 3 (guard-arm, recover, guard-exec across the
joining-to-hosting restart on dev) and 4 (carloku.com's host.env files
against the allowlist), in the plan and spec rev 21 (section 33).
D5: the copy record and gate need no change; the plan says so.

Astra's final review (CODEX-FINAL-REVIEW-REGION-HA-2B.md) committed as
written.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by claude 2fe0238c9a945e7ceb26235b82f2f7e755c305d1 parent 18ed4a9
3 files changed, +186 −80
  • docs/superpowers/plans/2026-10-17-region-ha-2b-joining.md +123 −76
  • docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md +47 −0
  • docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md +16 −4
diff --git a/docs/superpowers/plans/2026-10-17-region-ha-2b-joining.md b/docs/superpowers/plans/2026-10-17-region-ha-2b-joining.md
index c3af46094..83c2d824d 100644
--- a/docs/superpowers/plans/2026-10-17-region-ha-2b-joining.md
+++ b/docs/superpowers/plans/2026-10-17-region-ha-2b-joining.md
@@ -4,11 +4,11 @@
**Goal:** Build spec 2 §5's box side and console side of joining, dormant behind `CARLOS_REGION_HA` and claim mode: the agent chooses hosting or joining at every start, a joining box publishes its identity record and waits behind a 503 health answer, the console confirms the box's group membership itself and then invites and enrolls it, root arms the box's guard, a fresh box installs its channel's release into both protected slots, and `host.env` is assembled from SSM parameters.
-**Architecture:** The mode is chosen once, in `RunAgent`, before anything else starts (`hostingGate`). Joining is its own small run (`internal/host/joining.go`): the identity record loop, a 503-only health listener, the guard-ready report, and a marker that asks root (`carlos system join`, behind a path unit) to arm the guard with the invitation's enrollment. Hosting is main's claim agent, which reports hosting to the health answer only while its claim worker runs. The console is the only membership confirmer (H28): its enrolment pass, `hostactors.AutoEnroller` driven by `internal/console/enrollpass.go`, asks the cloud through `hostactors.MembershipReader`, its own read-only client, and `EnrollAutomatic` and `CheckConfirmed` make that read themselves and build the confirmation inside `hostactors` where no caller can supply one. The write each read allows runs on a store bound to the read's deadline (`writeBound`, the single enforcement point), so no store call of it is issued 60 seconds or more after the read's anchor, the describe's start, and the console waits at most 15 seconds for any call it issues. An outcome the console did not see is unknown, and a later pass settles it the ordinary way, by reading the roster. A cancelled write can still commit later, on the roster revision it was built against. They stand on main's `BeginEnrollmentAttempt`, `Invite`, `Replace` and `membership.go`, unchanged. A box its region's group launched is referenced by its group reference, `g-` and 20 hex digits of its account and instance's digest (`GroupBoxReference`): it publishes under it, `statusLoop` included (and publishes nothing while the reference cannot be read), and the console derives it again from the instance it proved, never trusting the key the box chose. The status reader drops reports no box has rewritten in a day before its key cap, so replaced boxes' history never hides the running ones. Each AWS part is an adapter named for AWS: the SSM parameter store (`internal/hostenv/awsssm.go`) and the Auto Scaling and EC2 describe calls (`internal/hostactors/awsmembership`, opened by the console in `internal/console/awsmembership.go`).
+**Architecture:** The mode is chosen once, in `RunAgent`, before anything else starts (`hostingGate`). Joining is its own small run (`internal/host/joining.go`): the identity record loop, a 503-only health listener, the guard-ready report, and a marker that asks root (`carlos system join`, behind a path unit) to arm the guard with the invitation's enrollment. Hosting is main's claim agent, which reports hosting to the health answer only while its claim worker runs. The console is the only membership confirmer (H28): its enrolment pass, `hostactors.AutoEnroller` driven by `internal/console/enrollpass.go`, asks the cloud through `hostactors.MembershipReader`, its own read-only client, and `EnrollAutomatic` and the unexported `checkConfirmed` make that read themselves and build the confirmation inside `hostactors` where no caller can supply one. The pass binds the invitation's deadline to the confirmation `checkConfirmed` returns, and finds an invited box with the read-only `enrollmentNonce`, never `Invite`. The write each read allows runs on a store bound to the read's deadline (`writeBound`, the single enforcement point), so no store call of it is issued 60 seconds or more after the read's anchor, the describe's start, and the console waits at most 15 seconds for any call it issues. An outcome the console did not see is unknown, and a later pass settles it the ordinary way, by reading the roster. A cancelled write can still commit later, on the roster revision it was built against. They stand on main's `BeginEnrollmentAttempt`, `Invite`, `Replace` and `membership.go`, unchanged. A box its region's group launched is referenced by its group reference, `g-` and 20 hex digits of its account and instance's digest (`GroupBoxReference`): it publishes under it, `statusLoop` included (and publishes nothing while the reference cannot be read), and the console derives it again from the instance it proved, never trusting the key the box chose. The status reader drops reports no box has rewritten in a day before its key cap, so replaced boxes' history never hides the running ones. Each AWS part is an adapter named for AWS: the SSM parameter store (`internal/hostenv/awsssm.go`) and the Auto Scaling and EC2 describe calls (`internal/hostactors/awsmembership`, opened by the console in `internal/console/awsmembership.go`).
**Tech Stack:** Go 1.26, module `github.com/carlosframework/platform`; `internal/hostenv` (new), `internal/host`, `internal/hostactors`, `internal/console`, the `carlos` CLI (repo root); AWS SDK v2 SSM and Auto Scaling (new, v1.74.0, the versions built against the repo's `aws-sdk-go-v2 v1.44.0`), EC2 (already a dependency).
-**Spec:** `docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md`, **rev 20**: §5 (boot and enrolment, host.env's fixed path and checked write, the write deadline and its per-call timeout, unknown outcomes settled by the roster, group references, the status listing and the activation gates), §12 (spec 2's hand-off to spec 3), §4 (the SSM layout and IAM, for 2c's seams), §25 (the provider boundary), rulings H1 to H28 (H28: the console is the only membership confirmer), and platform-1e's census ruling region-ha-2b (RULED by the Plan 14 controller 2026-10-16, with its conditions). Plan 2a (`2026-10-16-region-ha-2a-core.md`, built and merged as 6834659c) for format and the seams it left.
+**Spec:** `docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md`, **rev 21** (rev 21 records the final review below): §5 (boot and enrolment, host.env's fixed path and checked write, the write deadline and its per-call timeout, unknown outcomes settled by the roster, group references, the status listing and the activation gates), §12 (spec 2's hand-off to spec 3), §4 (the SSM layout and IAM, for 2c's seams), §25 (the provider boundary), rulings H1 to H28 (H28: the console is the only membership confirmer), and platform-1e's census ruling region-ha-2b (RULED by the Plan 14 controller 2026-10-16, with its conditions). Plan 2a (`2026-10-16-region-ha-2a-core.md`, built and merged as 6834659c) for format and the seams it left.
**Review round 1.** Astra's review of the first draft (`CODEX-REVIEW-REGION-HA-2B-1.md`) found eight things; this revision answers all of them, with the claim owner's conditions on the first (C1 to C4):
@@ -68,6 +68,19 @@
|---|---|
| P3: the SDK test misses successful writes through the wrapper stack; W1 (platform-1e) | `TestTheBoundReachesTheSDKRequest`'s fake bucket now honours S3's conditional headers and is seeded with a root-signed roster, so the conditional roster `PUT` succeeds through the deadline store, the mutation boundary, the conditional record store and the S3 store, and the test asserts that `PUT`'s deadline and its `If-Match`. Dropping the context where any wrapper forwards `Put` fails it (Task 4) |
+**Final review of the build.** Astra's whole-branch review of the build (`CODEX-FINAL-REVIEW-REGION-HA-2B.md`) found two P2s and a P3, and plan text the build had overtaken. Answered in one fix wave, under the controller's rulings (binding):
+
+| Finding | What changed |
+|---|---|
+| P2: a stalled membership read stops enrolment across all regions | `confirmMembership` gives each `Member` call its own `storeCallTimeout` (15 s) context; a read cut off by it, or by the pass's context, is `ErrMembershipUnknown`: the attempt stands, its wait does not grow, and the pass goes on to the next box. `TestAStalledMembershipReadIsCutOffAndThePassGoesOn` (Tasks 4, 5) |
+| P2: accepted status overrides break the group-report contract | `CARLOS_STATUS_BUCKET`, `CARLOS_STATUS_PREFIX` and `CARLOS_STATUS_REGION` leave `hostenv.Optional`, refused up front ("Global Constraints", host.env, says why) (Task 2) |
+| P3: the allowlist admits credentials embedded in webhook URLs | `CARLOS_ALERT_WEBHOOK` leaves `hostenv.Optional`, on the DNSimple token's rule. `TestStatusOverridesAndTheAlertWebhookAreRefusedByNameNotValue` refuses all four by name in `Assemble` and `Seed`, never printing a value (Task 2) |
+| Plan text: the DNSimple token, `CheckConfirmed` | The `Optional` snippet is the built one, without `CARLOS_DNSIMPLE_TOKEN`. The admission check is the unexported `checkConfirmed`, which returns its confirmation; the pass binds the invitation's deadline to it, and finds an invited box with the read-only `enrollmentNonce` (Tasks 4, 5). The census overlay's comment lines are left as the overlay has them |
+| Plan text: the bootstrap binary and 2c | 2c's seams: the bootstrap binary sits outside both protected slots; no launch lifecycle hook on the group; `control/box-enrollment/` leaves `infra/modules/deployment-buckets`; `host-env write` needs `--aws-region` or `AWS_REGION` |
+| Before activation | "Activation gates" 3 (the guard's restart transition on dev) and 4 (carloku.com's host.env files against the allowlist); gate 2, stop/start versus reboot, stays |
+
+The copy record and its gate need no change: no user-facing string changed. The new cut-off's text is in the pass's log line, which is not a row.
+
## Scope
Spec 2 is four plans. This is **2b**. What 2a left it, and what it leaves the others:
@@ -89,13 +102,13 @@ Spec 2 is four plans. This is **2b**. What 2a left it, and what it leaves the ot
- **A stopped group box that is started again** (C3) never resumes hosting on its old enrollment. EC2 sets the identity document's `pendingTime` at every start, not at a reboot; when the current document's `pendingTime` differs from the one in the identity record the roster entry binds (by digest), the box is held, answers 503, and its group replaces it. It cannot prove membership again under the same reference while its roster entry stands; its replacement enrolls under its own reference.
- **Box references (S3; census ruling region-ha-2b).** A box its region's group launched is referenced by its group reference, `GroupBoxReference(account, instance)`: `g-` and the first 20 hex digits of `sha256("carlos-box-ref-v1|" + account + "|" + instance)`. Instance ids never repeat in an account, and the reference is a pseudonym, so the public `status/fleet.json` shows it and never an instance id.
- **The box** takes it when its host.env says `CARLOS_BOX_REFERENCE=group`, which only `host-env write` adds: `metricsBoxLabel` answers it, so the identity record, mode selection, root's join and the edges file use it; `statusLoop`'s tick (`groupStatusTick`, one call, re-sealed) publishes under it; the joining report is written under it. An identity IMDS has not given is no reference, never a label in its place: such a box publishes no status report at all, never under `status.Label`, which may be another box's key (`WriteBox` replaces what is there), and asks IMDS again on the next tick. That holds whatever the claim and region HA switches say.
- - **The console** never trusts the key a box chose. The pass derives the reference from the instance the identity record proves and refuses a record under any other key; `CheckConfirmed` and `EnrollAutomatic` refuse a request whose reference is not the instance's; `PrivateBoxReport` reads a group box's report under the reference derived from its pinned instance (the attempt's before enrolment, the roster's after), so every `VerifyReport` input for a group box comes from the console's own derivation, and a report another box wrote under that key fails on its instance and enrollment.
+ - **The console** never trusts the key a box chose. The pass derives the reference from the instance the identity record proves and refuses a record under any other key; `checkConfirmed` and `EnrollAutomatic` refuse a request whose reference is not the instance's; `PrivateBoxReport` reads a group box's report under the reference derived from its pinned instance (the attempt's before enrolment, the roster's after), so every `VerifyReport` input for a group box comes from the console's own derivation, and a report another box wrote under that key fails on its instance and enrollment.
- **Every other box**, fleet boxes included, keeps exactly today's label and status key.
- **A joining box** publishes its signed identity record, publishes no edges file (so it is never referenced, `references.go:63`), opens no registry, admits and serves nothing, and answers `/.carlos/lb-health` with 503. Only its guard-ready report and its identity record leave the box.
- **The health answer's hosting condition** is the gate's decision, set before the claim worker starts and cleared when it ends (2a deferred this).
-- **Enrolment (A2-1, H28).** The pass begins an attempt for each published identity record whose box the roster does not list. It invites only after its own membership read of the attempt's instance (`CheckConfirmed`), and enrolls only when the box's guard-ready report passes `hostactors.VerifyReport` (the check `membership.go:40` makes), through `EnrollAutomatic`, which reads membership again itself. The attempt is single-use (`Consumed`, unchanged), so a replay is refused. The box takes its group's role in the enrolling write (2a). The pass enrolls into the console's own namespace and the `default` pool, and only in a deployment whose host floor is published.
+- **Enrolment (A2-1, H28).** The pass begins an attempt for each published identity record whose box the roster does not list. It invites only after its own membership read of the attempt's instance (`checkConfirmed`, unexported), on the store bound to the confirmation that read returns, and enrolls only when the box's guard-ready report passes `hostactors.VerifyReport` (the check `membership.go:40` makes), through `EnrollAutomatic`, which reads membership again itself. The attempt is single-use (`Consumed`, unchanged), so a replay is refused. An invited box is found with `enrollmentNonce`, which only reads and verifies the invitation; `Invite`, which is get-or-create, runs only on the bound store after a membership read. The box takes its group's role in the enrolling write (2a). The pass enrolls into the console's own namespace and the `default` pool, and only in a deployment whose host floor is published.
- **Membership (H16, H28; C1, C2).** The console is the only membership confirmer. No box writes or contributes membership evidence, and no box role can describe the group or create under `control/box-enrollment/`.
- - `EnrollAutomatic` and `CheckConfirmed` take a `MembershipReader` (the console's AWS client) and call `Member` themselves. The confirmation is built from that answer alone, by the unexported `confirmMembership`, and never leaves `hostactors`; no exported function takes a confirmation or a `Membership`. Tests use a fake reader.
+ - `EnrollAutomatic` and `checkConfirmed` (unexported, `hostactors`' own) take a `MembershipReader` (the console's AWS client) and call `Member` themselves, each call cut off at `storeCallTimeout` (15 s) as unknown. The confirmation is built from that answer alone, by the unexported `confirmMembership`, and never leaves `hostactors`; no exported function takes a confirmation or a `Membership`. Tests use a fake reader.
- A member is in the region's group, `Pending` or `InService`, `pending` or `running`, owned by the region's account, in its AWS region, with a request id from each of the two calls.
- **Freshness bound: 60 seconds, enforced at the write (S1, T3, U2).** A fresh describe runs immediately before each write, the invitation and the enrolment. **The write deadline is anchored at the describe's start, which is never later than its completion, and a write 60 s or more after that anchor is refused. Each store call issued before it gets a hard timeout of 15 s (`storeCallTimeout`).** What that guarantees (V1): every store call of the invite and the enrolling write is issued before the deadline (60 s from the start of the membership describe); the console waits at most 15 s for any call; a write whose result the console did not see is unknown and is settled on a later pass by reading the roster. A cancelled write can still commit later; such an enrolment rests on a membership read that was true when the write was issued, and is the same exposure as an instance detached after enrolment, which spec 3's worker reconciles. `writeBound` is the single enforcement point. It gives the sealed writers (`Invite`, and `Replace` with the `checkMembership` it makes) a copy of the directory whose store refuses every call at or after the deadline, by the read's clock or the context's, and a context that ends at the deadline. What it guarantees: no store call, the roster `Put` and the invitation's `Put` included, is issued at or after the deadline; a call in flight then is cancelled; a call the store has not answered 15 s after it was issued is cut off by the wrapper itself (`deadlineStore.call` returns on its own and cancels the call's context, which the AWS SDK enforces at the HTTP layer); and a call that answers only after the deadline, even with success, is `ErrWriteDeadline`, unknown. The store is wrapped, not trusted: `release.S3Store` passes its context to the AWS SDK, which honours it, but `release.DirStore` ignores its context. The pass's S3 store is the console's claim data boundary, whose client the console shares; rather than a second client and a second boundary, the wrapper gives each call its own 15-second context, the same hard limit at the HTTP layer, and does not wait past it whatever the store does. Every write in the invitation and enrolment paths is conditional (V4; lines are main's, `6834659c`): the roster write is `Directory.Replace`'s compare-and-swap, `d.Store.Put(ctx, Key, wire, before.ETag)` (`internal/hostactors/roster.go:281`), which refuses an observation with no ETag (`roster.go:216`), and every roster write the pass makes goes through it (`enroll.go:80`, the enrolment `EnrollAutomatic` makes through `enroll`, and `enrollment_confirmation.go:158`, `BeginEnrollmentAttempt`'s); the invitation is created only if absent, `d.Store.Put(ctx, key, raw, "")` (`internal/hostactors/invitation.go:78`). `release.Store`'s contract makes `""` create-only (`If-None-Match: *`) and an ETag replace-only (`If-Match`) (`internal/release/store.go:40`), as `S3Store.Put` sends them (`store.go:795` to `:797`) and `DirStore.Put` checks them (`store.go:430` to `:440`). The pass's store keeps that: the observed mutation boundary re-reads the object and refuses a Put whose ETag no longer matches before forwarding the same condition (`internal/hostops/mutation_boundary.go:147` to `:162`), and the conditional record store passes it through for every key that is not an instance or sidecar record (`internal/hostops/record_writes.go:86` to `:88`), over the console's raw store (`internal/console/claim_runtime.go:42`, `:46`, `:60`). So a write that commits late lands only on the exact roster revision it was built against, or creates an invitation that did not exist; it never overwrites a newer revision or another enrolment. A missed deadline keeps the attempt and its wait. A describe that itself took 60 seconds or more is `ErrMembershipStale`, checked once in `checkConfirmed` after the describe, and counts against the attempt. Nothing else checks freshness: no callback in `enroll` and no check before `Invite`, which would only repeat the bound; between the describe and `Invite` sits nothing that waits.
- **Unknown outcomes are settled by the roster (U1).** An outcome the bound cannot confirm (`ErrWriteDeadline`) is resolved the ordinary way on the next pass, which reads the signed roster as every pass does: a box the roster lists is enrolled, and the pass has nothing more to do for it; one it does not list goes on with the same attempt while it is still valid, and its wait does not grow. There is no read-back, no memory of unknown outcomes and no page. An instance that left its group but still runs after its enrolment landed keeps that enrolment until an operator removes it or spec 3's regional worker reconciles it ("Activation gates", note); a terminated one is kept from hosting and from capacity by main's termination fence (`internal/hostactors/fence_authority.go:14`, honoured at `internal/hostops/executor.go:204`), and a stopped one by the stale-edges rule (`internal/boxchoice/boxchoice.go:131`) and C3's restart hold (Task 6's `relaunched`).
@@ -103,7 +116,7 @@ Spec 2 is four plans. This is **2b**. What 2a left it, and what it leaves the ot
- **Unknown is not invalid (#7).** A read that fails, or is not an answer (EC2 does not list the instance, a missing request id), is `ErrMembershipUnknown`: the pass asks again next time with the same attempt, which it neither supersedes nor counts. Only the cloud's positive answer that the instance is not a current member (`ErrNotMember`), or a read that went stale before its write (`ErrMembershipStale`), counts against the attempt: the pass keeps it, asking nothing more, for `Patience`, doubling with each refusal up to 32 times, then begins another. A record whose instance is gone costs one roster write and one describe an hour.
- **Predecessors until spec 3 (C3).** 2a's readers already exclude fenced and terminated boxes: every one goes through `ExecutorChooser.eligible` (`internal/hostops/executor.go:172`), which keeps only `active` roster entries (`:201`) and drops a box whose termination fence validates (`:204`, `TerminationFence`), and `boxchoice.Eligible` then requires a fresh edges file (`internal/boxchoice/boxchoice.go:131`) and drops the claim's lost boxes (`:143`), and `boxchoice.Placeable` drops headroom older than 15 minutes (`internal/boxchoice/policy.go:69`). `PlaceableBoxes` (`executor.go:103`, the create, move and no-room doors: `internal/console/apiinstances.go:1267`, `boxselect.go:204`, `nocapacity.go:171`), `Choose` and `Permits` (`executor.go:61`, `:85`) and `Capacity` (`executor.go:122`, the `EligibleBoxes` metric through `internal/console/regioncapacity.go:108`) all read that set. A terminated box is out as soon as its fence is written, and in any case once its edges file is 15 minutes old; a stopped box drops out by the same freshness rule. 2b adds no exclusion.
- **The channel install (A1-6, A2-2).** `carlos system install-channel` verifies the signed pointer (`verifyPointer`), refuses a release that does not pass the published host floor exactly as guard-arm will check it (`hostfloor.CheckImage`), fetches against the signed sha256, stages through `stageCandidate`, and puts the artifact in both `carlos` and `carlos.prev`. It keeps the release identity for guard-arm, and writes `applied.json` and `seen.json` as an apply would. On every run, each directory on the agent's cache path (`system`, `system/router` and the version's own) is created as bootstrap creates directories (`bootstrapMakeDir`, no symlink followed) and handed, alone, to the service user (`ownByService`: bootstrap's descriptor-anchored open, then `Fchown`), whether this run or an earlier failed one created it; a symlinked parent is refused; nothing else is chowned. The agent's next update can then download and ask. It refuses an armed guard, a box that already runs another release, a missing floor, and any read that fails.
-- **host.env (§5 step 2).** SecureString parameters named for their variables, under `/carlos/<deployment>/<region>/` and, for the five shared variables, `/carlos/<deployment>/shared/`. Every variable §5 step 2 names is required. The file is written 0600 by root, sorted `KEY=value` lines, with values restricted to letters, digits and `_ . / : , @ + = % -` so systemd and a shell read it alike; anything else is refused, naming the variable and never the value. A variable in both scopes is refused. Only allowlisted names are accepted (`Required` and `Optional`, census condition 2): any other name, under either scope, is refused by name by `write` and by `seed`, and keys that would select a box's authority from the store alone (fleet or tunnel identity, address overrides, pools, directories, relay settings, tokens) are not on the list. The path is fixed, `/etc/carlos/host.env`; `write` takes no flag to move it (U3). It is written only through `/etc` and `/etc/carlos` held by descriptor: each opened `O_NOFOLLOW|O_DIRECTORY` and refused unless `fstat` shows a directory root owns that no group or other can write (`/etc/carlos` is created 0755 first if missing, then opened and checked the same way); the new file is created 0600 inside the held directory, fsynced, renamed over `host.env` within it, and the directory fsynced, so a reader sees the old file or the new one. That code is Linux's (`write_linux.go`); elsewhere `write` refuses. `write` adds `CARLOS_BOX_REFERENCE=group`, which is never a parameter. The marker and `CARLOS_REGION`, which do select authority, are cross-checked against the signed roster at the agent's use (condition 3, Task 6), not at write: at boot the box has no roster entry to check against, and before enrolment neither grants anything, since the console derives a joining box's reference and region from its proven instance. `seed` writes a region's own values with overwrite and the shared ones create-only (`SetNew`, SSM `Overwrite: false`); when another seed got there first, it reads the value back and goes on only if it is the value it would have written, otherwise it refuses, naming the variable.
+- **host.env (§5 step 2).** SecureString parameters named for their variables, under `/carlos/<deployment>/<region>/` and, for the five shared variables, `/carlos/<deployment>/shared/`. Every variable §5 step 2 names is required. The file is written 0600 by root, sorted `KEY=value` lines, with values restricted to letters, digits and `_ . / : , @ + = % -` so systemd and a shell read it alike; anything else is refused, naming the variable and never the value. A variable in both scopes is refused. Only allowlisted names are accepted (`Required` and `Optional`, census condition 2): any other name, under either scope, is refused by name by `write` and by `seed`, and keys that would select a box's authority from the store alone (fleet or tunnel identity, address overrides, pools, directories, relay settings, tokens) are not on the list. Two more kinds are left off (final review). `CARLOS_DNSIMPLE_TOKEN` and `CARLOS_ALERT_WEBHOOK` are credentials, or can carry one: a webhook URL can embed its token. The status destination overrides, `CARLOS_STATUS_BUCKET`, `CARLOS_STATUS_PREFIX` and `CARLOS_STATUS_REGION`, would split a group box's reports from the console's reads: a group box reports to the deployment bucket, unprefixed, where the console reads its guard-ready report, while joining and once hosting. Each is refused by name, never by value, like any other name off the list. The path is fixed, `/etc/carlos/host.env`; `write` takes no flag to move it (U3). It is written only through `/etc` and `/etc/carlos` held by descriptor: each opened `O_NOFOLLOW|O_DIRECTORY` and refused unless `fstat` shows a directory root owns that no group or other can write (`/etc/carlos` is created 0755 first if missing, then opened and checked the same way); the new file is created 0600 inside the held directory, fsynced, renamed over `host.env` within it, and the directory fsynced, so a reader sees the old file or the new one. That code is Linux's (`write_linux.go`); elsewhere `write` refuses. `write` adds `CARLOS_BOX_REFERENCE=group`, which is never a parameter. The marker and `CARLOS_REGION`, which do select authority, are cross-checked against the signed roster at the agent's use (condition 3, Task 6), not at write: at boot the box has no roster entry to check against, and before enrolment neither grants anything, since the console derives a joining box's reference and region from its proven instance. `seed` writes a region's own values with overwrite and the shared ones create-only (`SetNew`, SSM `Overwrite: false`); when another seed got there first, it reads the value back and goes on only if it is the value it would have written, otherwise it refuses, naming the variable.
- **No roster field is added.** The roster wire is unchanged (`EnrollmentAttempts` and the region's group fields are main's), so 2b has no reader-ordering constraint. The only new objects are box-local (`/opt/carlos/system/join/`, `/opt/carlos/system/router/channel-identity.json`, `/data/system-join-requested`) or main's keys (invitations, status reports, identity records). Nothing is written under `control/box-enrollment/`: main's `ConfirmationKey` and the stored confirmation (`MembershipConfirmation`, `TerminatedEnrollment`) had no reader but `EnrollAutomatic` and its tests, so Task 4 removes them.
- **The provider boundary (§25).** Spec 2's core files import no `github.com/aws/...` package; AWS parts are adapters named for AWS; `region_ha_boundary_test.go` holds both. 2b builds no S3 client.
- **The status listing (T2).** Every replaced group box leaves a status report under its own reference, so `status/boxes/` keeps growing. `loadBoxesReport` (the reader behind `status.LoadBoxes` and `LoadBoxesComplete`) lists the prefix with each object's write time where the store gives one (`release.ModLister`: both real stores do), and drops reports no box has rewritten in a day before any read and before its 200-key cap, so the cap bounds current reports only. It is an optimisation, never authority: every reader still judges each report's own `GeneratedAt` as before (`BoxStale`, the console's `VerifyReport`), and a day keeps every box an operator would still look for as stale. A store without write times (a fleet's prefix store) is listed whole, as before. 2c adds a 30-day expiry on `status/boxes/` as hygiene; nothing depends on it, so a self-hosted store without lifecycle rules works the same.
@@ -113,10 +126,12 @@ Spec 2 is four plans. This is **2b**. What 2a left it, and what it leaves the ot
## Activation gates
-2b ships dormant. Automatic enrolment must not be switched on for any region until both hold:
+2b ships dormant. Automatic enrolment must not be switched on for any region until all four hold:
1. **Group references have shipped** (Astra round 2, #3). A release with all of 2b, Tasks 2, 4, 5, 6 and 8 in particular (the references, the status tick that publishes only under them, and the status listing that keeps replaced boxes' history from hiding the running ones), runs on every box of the region and on the console. Without them labels collide, and a region in its first build has no healthy target, so its group never replaces the box that cannot publish (spec §5's first-build exception); an alarm reports that stall and does not repair it. They ship in 2b, so this gate is that release being rolled.
2. **`pendingTime` is verified on dev** (Open question 7): verify on dev before the first region that a group box stopped and started is held and replaced, and that one rebooted hosts again.
+3. **The guard's restart transition is seen on dev** (final review). On dev, a fresh group box goes guard-arm, then carlos-system-recover, then guard-exec across the joining-to-hosting restart, and comes up hosting. After the restart the guard drop-in requires `carlos-system-recover.service`, and a recover that fails leaves the edge down, since `Restart=always` does not fire again on a failed dependency. Static inspection cannot establish that systemd transition.
+4. **carloku.com's host.env files are checked against the allowlist** (final review), which now excludes the status destination overrides and the alert webhook ("Global Constraints", host.env), before the first seed. A box that carries an excluded name has to drop it, or its region cannot seed.
**Activation note: detached instances are spec 3's.** 2b removes nothing from the roster. An instance detached from its group but still running keeps its enrolment until an operator removes it. Spec 3's requirements carry it: **the regional worker reconciles roster membership against Auto Scaling group membership, including detached-but-running instances** (spec §12, Open question 9).
@@ -145,7 +160,7 @@ Spec 2 is four plans. This is **2b**. What 2a left it, and what it leaves the ot
|---|---|---|---|
| 2 | `internal/hostenv` (`Fetch`, `Assemble`, `Seed`, `sameShared`, `Allowed`, `notAllowed`; `write_other.go:writeHostEnv`) | The group-box marker; create-only shared values; the allowlist; off Linux, `write` refuses | No root-read site; `SetNew` and `Set` are not `Put`. Round 4 removed `ownedByEUID` with its files |
| 3 | `channelInstall.cacheDir`, `ownByService`, `ownDir` | Every cache parent made and handed to the service user, each run; symlinks refused | They call `bootstrapMakeDir`, `bootstrapOpenDir` and `bootstrapOwner` (main's, unchanged) |
-| 4 | `Directory.EnrollAutomatic`, `CheckConfirmed`, `checkConfirmed`, `confirmMembership`, `writeBound`, `deadlineStore.call`, `deadlineStore`'s other methods | The console's own read; the write bound, the single enforcement point, with each call's 15-second hard timeout; the group-reference check | Not root-read units; `deadlineStore.Put` is the B8 row above. `Directory.enroll` is main's again (round 3) |
+| 4 | `Directory.EnrollAutomatic`, `checkConfirmed`, `confirmMembership`, `writeBound`, `deadlineStore.call`, `deadlineStore`'s other methods | The console's own read; the write bound, the single enforcement point, with each call's 15-second hard timeout; the group-reference check | Not root-read units; `deadlineStore.Put` is the B8 row above. `Directory.enroll` is main's again (round 3) |
| 4 | `Directory.PrivateBoxReport` (`private.go`) | A group box's report is read under the reference derived from its pinned instance | Not a census unit. It feeds `VerifyReport` (unchanged) for `checkMembership`, `Census` and activation; non-group boxes unchanged |
| 4 | `GroupBoxReference` (`groupref.go`) | New: the reference function | No I/O |
| 5 | `AutoEnroller.EnrollPass`, `step`, `confirmAndWrite`, `logRead`, `logOutcome` | The pass, its write bound, its reference check and its log lines (round 4 removed `readBack` and `pageNotMember`) | Not root-read units; its construction and invocation are the claim-authority sites above |
@@ -503,7 +518,7 @@ Shared values are written create-only (R4). Two seeds can both read a shared var
**The ruling's conditions on `runHostEnv` (census ruling region-ha-2b).**
1. No box writes the parameters it reads: the box role has no `ssm:PutParameter` or `ssm:DeleteParameter` on either path; only the operator's (and the console's) principal writes. That is 2c's to grant and to prove with a negative check (2c's seam 3).
2. `write` takes only allowlisted names (`Required` and `Optional`; `Allowed`): any other name, under either scope, is refused by name, and `seed` refuses a file that carries one. Values are restricted to the safe set, so a newline, a carriage return, a NUL, whitespace, quotes or a `$` (anything that could start a second key or change how systemd reads the line) is refused, naming the variable and never the value. The path is fixed at `/etc/carlos/host.env` (U3): `write` has no `--out`, and tests reach the code through the unexported `hostEnvRoot`, never a flag. `writeHostEnv` holds `/etc` and then `/etc/carlos` by descriptor, each opened `O_NOFOLLOW|O_DIRECTORY` under the one before (`checkedDir`), so a symlink at either is refused and neither is looked up by name again; each must be a directory, owned by root (`write` already refuses any other user), and writable by no group or other, or `write` refuses it (RHB-C67). A missing `/etc/carlos` is created 0755, then opened and checked like the rest. The file is created 0600, `O_EXCL|O_NOFOLLOW`, under a random name inside the held directory (`openat`), written and fsynced, renamed over `host.env` within the same directory, and the directory fsynced: a reader sees the old file or the new one, never part of either, and no other user can swap the file or the directory under it. The descriptor calls are Linux's (`//go:build linux`); elsewhere `writeHostEnv` refuses (`write_other.go`, RHB-C68), so darwin and windows build.
-3. The values that select authority, the group marker and `CARLOS_REGION`, are cross-checked against the signed roster at the agent's use, in mode selection (Task 6), not here: at boot there is no roster entry for the box to check against, and before enrolment neither value grants anything. Keys that would select authority with no roster to check them by are kept off the allowlist (fleet and tunnel identity, address overrides, pools, directories, relay settings, tokens).
+3. The values that select authority, the group marker and `CARLOS_REGION`, are cross-checked against the signed roster at the agent's use, in mode selection (Task 6), not here: at boot there is no roster entry for the box to check against, and before enrolment neither value grants anything. Keys that would select authority with no roster to check them by are kept off the allowlist (fleet and tunnel identity, address overrides, pools, directories, relay settings, tokens), and so are the alert webhook and the status destination overrides ("Global Constraints", host.env, says why).
- [ ] **Step 1: Write the failing tests**
@@ -583,7 +598,7 @@ func today() map[string]string {
"CARLOS_RELEASE_PUBKEY": "q3J0bXlwdWJsaWNrZXk+/=",
"CARLOS_SECRETS_KEY": "c2VjcmV0c2tleQ==",
"CARLOS_SYSTEM_CHANNEL": "stable",
- "CARLOS_STATUS_PREFIX": "fleet",
+ "CARLOS_HIBERNATE_PREFIX": "fleet",
}
}
@@ -852,7 +867,7 @@ func TestWriteTakesOnlyAllowedNamesAndSafeValues(t *testing.T) {
// Values that would break a line, end the file early or change how
// systemd reads it.
for _, v := range []string{"a\nCARLOS_REGION=uk", "a\x00b", "a\rb", "$(id)", "a b", "\"quoted\""} {
- p.vals["/carlos/flagship/ie/CARLOS_STATUS_PREFIX"] = v
+ p.vals["/carlos/flagship/ie/CARLOS_HIBERNATE_PREFIX"] = v
if _, err := Fetch(ctx, p, "flagship", "ie"); !errors.Is(err, ErrValue) || strings.Contains(err.Error(), v) {
t.Fatalf("%q: %v", v, err)
}
@@ -1118,24 +1133,30 @@ var Required = []string{
// carry, from the agent's config surface (README's host.env list and the
// variables the agent reads). Together with Required they are the
// allowlist: any other name, under either scope, is refused by name,
-// never by value, by write (Assemble) and by seed. Keys that would choose
-// a box's authority from the store alone are deliberately not on it: a
-// fleet's or tunnel's identity, an address override, a pool, the data,
-// opt and deployment directories, the relay's settings and any token or
-// credential. CARLOS_REGION and the group marker, which do select, are
-// cross-checked against the signed roster where the agent uses them
-// (internal/host's mode selection). Adding a variable is a code change,
-// reviewed.
+// never by value, by write (Assemble) and by seed. The list leaves out
+// the keys that select a box's authority (a fleet's or tunnel's identity,
+// an address override, a pool, the data, opt and deployment directories,
+// the relay's settings) and every token or API credential, except the
+// deployment's own required secrets, CARLOS_SECRETS_KEY and
+// CARLOS_RELEASE_PUBKEY. CARLOS_ALERT_WEBHOOK is left out on the same
+// rule: a webhook URL can carry a credential. So are the status
+// destination overrides, CARLOS_STATUS_BUCKET, CARLOS_STATUS_PREFIX and
+// CARLOS_STATUS_REGION: a group box reports to the deployment bucket,
+// unprefixed, where the console reads its guard-ready report, while it is
+// joining and once it is hosting. An endpoint override such as
+// CARLOS_S3_ENDPOINT stays, because it selects no box authority and only
+// operators can write the parameter paths. CARLOS_REGION and the group
+// marker, which do select, are cross-checked against the signed roster
+// where the agent uses them (internal/host's mode selection). Adding a
+// variable is a code change, reviewed.
var Optional = []string{
"CARLOS_ACME_DELEGATION_ZONE",
"CARLOS_ACME_DIRECTORY",
"CARLOS_ALERT_CPU_PCT",
"CARLOS_ALERT_DISK_PCT",
"CARLOS_ALERT_MEM_PCT",
- "CARLOS_ALERT_WEBHOOK",
"CARLOS_AWS_REGION",
"CARLOS_BACKHAUL_MAX_INFLIGHT",
- "CARLOS_DNSIMPLE_TOKEN",
"CARLOS_DNS_PROVIDER",
"CARLOS_EDGE_GZIP",
"CARLOS_HIBERNATE_IDLE",
@@ -1148,9 +1169,6 @@ var Optional = []string{
"CARLOS_REQUIRE_POINTER_DIGEST",
"CARLOS_S3_ENDPOINT",
"CARLOS_S3_QUIRKS",
- "CARLOS_STATUS_BUCKET",
- "CARLOS_STATUS_PREFIX",
- "CARLOS_STATUS_REGION",
"CARLOS_STEERING_ZONE_ID",
"CARLOS_SWAP",
"CARLOS_UNIT_SETTLE",
@@ -2705,7 +2723,7 @@ git commit -m "region HA 2b: carlos system install-channel"
**Files:**
- Create: `internal/hostactors/membershipread.go`, `internal/hostactors/groupref.go`, `internal/hostactors/membershipread_test.go`, `internal/hostactors/export_test.go`, `internal/hostactors/boundsdk_test.go`
-- Modify: `internal/hostactors/enrollment_confirmation.go` (`EnrollAutomatic` and `CheckConfirmed` read membership themselves; the stored confirmation is removed), `internal/hostactors/private.go` (a group box's report under its derived reference)
+- Modify: `internal/hostactors/enrollment_confirmation.go` (`EnrollAutomatic` and the unexported `checkConfirmed` read membership themselves; the stored confirmation is removed), `internal/hostactors/private.go` (a group box's report under its derived reference)
- Modify: `internal/hostactors/attempt_regression_test.go`, `internal/hostactors/roles_test.go` (main's tests, onto a fake reader)
- Create: `internal/hostactors/awsmembership/awsmembership.go`, `awsmembership_test.go`
- Modify: `go.mod`, `go.sum` (`github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0`)
@@ -2713,11 +2731,11 @@ git commit -m "region HA 2b: carlos system install-channel"
**Interfaces:**
- Consumes: `validateScope`, `segment`, `token`, `EnrollmentAttempt`, `Directory.enroll`, `Replace` and `checkMembership` (all main's, unchanged, run on the bound store), `release.Store`.
-- Produces (`hostactors`): `type MembershipReader interface { Member(ctx, awsRegion, instance string) (Membership, error) }`; `Membership`; `MembershipEvidence{RequestIDs []string; ObservedAt time.Time}`; `ErrMembershipUnknown`, `ErrNotMember`, `ErrMembershipStale`, `ErrWriteDeadline`; `GroupBoxReference(account, instance string) string`; `func (d Directory) CheckConfirmed(ctx, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (EnrollmentRequest, MembershipEvidence, error)`; `func (d Directory) EnrollAutomatic(ctx, sg release.Signer, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (Observed, MembershipEvidence, error)`. Unexported: `membershipConfirmation` (with `fresh`, `deadline`, `atWrite`), `confirmMembership`, `writeBound`, `deadlineStore` (with `call`), `membershipFreshness = time.Minute`, and `var storeCallTimeout = 15 * time.Second` (a var so tests can shorten it). Changed: `Directory.PrivateBoxReport` reads a group box (`Box.Group` set) under `GroupBoxReference` of its pinned account and instance.
+- Produces (`hostactors`): `type MembershipReader interface { Member(ctx, awsRegion, instance string) (Membership, error) }`; `Membership`; `MembershipEvidence{RequestIDs []string; ObservedAt time.Time}`; `ErrMembershipUnknown`, `ErrNotMember`, `ErrMembershipStale`, `ErrWriteDeadline`; `GroupBoxReference(account, instance string) string`; `func (d Directory) EnrollAutomatic(ctx, sg release.Signer, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (Observed, MembershipEvidence, error)`. Unexported: `func (d Directory) checkConfirmed(ctx, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (EnrollmentRequest, membershipConfirmation, error)`, `membershipConfirmation` (with `fresh`, `deadline`, `atWrite`, `evidence`), `confirmMembership` (each `Member` call cut off at `storeCallTimeout`, as unknown), `writeBound`, `deadlineStore` (with `call`), `membershipFreshness = time.Minute`, and `var storeCallTimeout = 15 * time.Second` (a var so tests can shorten it). Changed: `Directory.PrivateBoxReport` reads a group box (`Box.Group` set) under `GroupBoxReference` of its pinned account and instance.
- Removes (`hostactors`): `ConfirmationKey`, `MembershipConfirmation`, `TerminatedEnrollment`, `freshConfirmation`. Nothing on main but `EnrollAutomatic` and its two tests used them.
- Produces (`hostactors/awsmembership`): `New(ctx) (*Reader, error)` over the console's credentials; `(*Reader).Member` (`DescribeAutoScalingInstances`, then `DescribeInstances` by the `instance-id` filter).
-H28: the console is the only membership confirmer, and its read cannot be forged (C1). `EnrollAutomatic` no longer reads a confirmation object from the store: it takes the console's `MembershipReader`, calls `Member` itself, and builds the confirmation from that answer through the unexported `confirmMembership`. The confirmation type is unexported and never leaves the package, and no exported function takes one, or a `Membership`; `TestNoExportedPathTakesAConfirmation` holds that. `CheckConfirmed` is the same check without the write, which the console's pass makes immediately before it invites (Task 5).
+H28: the console is the only membership confirmer, and its read cannot be forged (C1). `EnrollAutomatic` no longer reads a confirmation object from the store: it takes the console's `MembershipReader`, calls `Member` itself, and builds the confirmation from that answer through the unexported `confirmMembership`. The confirmation type is unexported and never leaves the package, and no exported function takes one, or a `Membership`; `TestNoExportedPathTakesAConfirmation` holds that. `checkConfirmed`, unexported, is the same check without the write, which the console's pass makes immediately before it invites (Task 5); it returns the confirmation itself, and the pass binds the invitation to it with `writeBound`, so no exported path carries one.
Every other admission check stands, unchanged: the attempt in the roster matches the request, the record (by digest), the instance, the region's group (name and ARN), account and AWS region; it is not consumed; the two attempt ids are tokens and differ. The checks that existed only for box confirmers go: the confirmer's roster entry and `TerminationFence`, and the cross-region `TerminatedEnrollments` (H17 is withdrawn: the console reads any region, so an emptied region recovers the same way). `BeginEnrollmentAttempt` and the single-use `Consumed` are main's, unchanged.
@@ -2848,7 +2866,7 @@ func TestEnrollAutomaticReadsMembershipItself(t *testing.T) {
if _, _, err := w.enroll(); !errors.Is(err, ErrRoster) || len(w.cloud.calls) != 1 {
t.Fatalf("replay: %v, calls %v", err, w.cloud.calls)
}
- if _, _, err := w.d.CheckConfirmed(context.Background(), next, w.v, w.in, w.raw, "i-new", w.cloud, w.now); !errors.Is(err, ErrRoster) {
+ if _, _, err := w.d.checkConfirmed(context.Background(), next, w.v, w.in, w.raw, "i-new", w.cloud, w.now); !errors.Is(err, ErrRoster) {
t.Fatalf("replayed check: %v", err)
}
}
@@ -2867,7 +2885,7 @@ func TestEnrollAutomaticRefusesANonMember(t *testing.T) {
} {
t.Run(name, func(t *testing.T) {
w := newMemberWorld(t, m)
- if _, _, err := w.d.CheckConfirmed(context.Background(), w.o, w.v, w.in, w.raw, "i-new", w.cloud, w.now); !errors.Is(err, ErrNotMember) {
+ if _, _, err := w.d.checkConfirmed(context.Background(), w.o, w.v, w.in, w.raw, "i-new", w.cloud, w.now); !errors.Is(err, ErrNotMember) {
t.Fatalf("check: %v", err)
}
if _, _, err := w.enroll(); !errors.Is(err, ErrNotMember) {
@@ -2946,11 +2964,17 @@ func TestEnrollAutomaticRefusesAReadOlderThanAMinuteAtTheWrite(t *testing.T) {
// No exported identifier of this package carries a membership
// confirmation: the only one is built inside it, from a reader's answer.
func TestNoExportedPathTakesAConfirmation(t *testing.T) {
- for _, name := range []string{"EnrollAutomatic", "CheckConfirmed"} {
- m, ok := reflect.TypeOf(Directory{}).MethodByName(name)
- if !ok {
- t.Fatalf("no %s", name)
- }
+ // Every exported method of Directory, EnrollAutomatic among them: the
+ // admission check itself (checkConfirmed) is not exported.
+ if _, ok := reflect.TypeOf(Directory{}).MethodByName("EnrollAutomatic"); !ok {
+ t.Fatal("no EnrollAutomatic")
+ }
+ if _, ok := reflect.TypeOf(Directory{}).MethodByName("CheckConfirmed"); ok {
+ t.Fatal("the admission check is exported again")
+ }
+ for j := 0; j < reflect.TypeOf(Directory{}).NumMethod(); j++ {
+ m := reflect.TypeOf(Directory{}).Method(j)
+ name := m.Name
for i := 0; i < m.Type.NumIn(); i++ {
in := m.Type.In(i)
if strings.Contains(strings.ToLower(in.Name()), "confirmation") || in == reflect.TypeOf(MembershipEvidence{}) || in == reflect.TypeOf(Membership{}) {
@@ -3643,16 +3667,41 @@ type membershipConfirmation struct {
// region's account, in its AWS region, with a request id for each call.
// The read's anchor, its time, is taken before the call: never later than
// the describe's completion, so its age is never understated.
+//
+// The call gets at most storeCallTimeout, as each bound store call does:
+// neither describe sets a deadline of its own, and a connection that
+// stalls would otherwise hold the serial pass, and every box behind this
+// one, for as long as it stalls. A call cut off, by the timeout or by the
+// pass's own context, is unknown whatever it returned: the attempt stands
+// and its wait does not grow.
func confirmMembership(ctx context.Context, r MembershipReader, region Region, instance string, now func() time.Time) (membershipConfirmation, error) {
if r == nil || now == nil {
return membershipConfirmation{}, fmt.Errorf("%w: no reader", ErrMembershipUnknown)
}
at := now().UTC()
- m, err := r.Member(ctx, region.AWSRegion, instance)
+ rctx, cancel := context.WithTimeout(ctx, storeCallTimeout)
+ m, err := r.Member(rctx, region.AWSRegion, instance)
+ cut := rctx.Err()
+ cancel()
+ // The ids the cloud gave are kept whatever came of the read, failed
+ // calls included, for the console's log.
+ c := membershipConfirmation{requestIDs: append([]string(nil), m.RequestIDs...), observedAt: at}
+ if cut != nil {
+ return c, fmt.Errorf("%w: the read did not finish in time: %v", ErrMembershipUnknown, cut)
+ }
if err != nil {
- return membershipConfirmation{observedAt: at}, fmt.Errorf("%w: %v", ErrMembershipUnknown, err)
+ return c, fmt.Errorf("%w: %v", ErrMembershipUnknown, err)
+ }
+ // A reply without an id from each call is not an answer, whatever it
+ // claims, so it is never counted against the attempt.
+ if len(m.RequestIDs) < 2 {
+ return c, fmt.Errorf("%w: the cloud gave no request ids", ErrMembershipUnknown)
+ }
+ for _, id := range m.RequestIDs {
+ if id == "" || id == NoRequestID {
+ return c, fmt.Errorf("%w: the cloud gave a blank request id", ErrMembershipUnknown)
+ }
}
- c := membershipConfirmation{requestIDs: append([]string(nil), m.RequestIDs...), observedAt: at}
switch {
case m.GroupName != region.GroupName:
return c, fmt.Errorf("%w: it is in group %q", ErrNotMember, m.GroupName)
@@ -3666,13 +3715,6 @@ func confirmMembership(ctx context.Context, r MembershipReader, region Region, i
return c, fmt.Errorf("%w: it is %q", ErrNotMember, m.InstanceState)
case m.Owner != region.Account || m.AvailabilityZoneRegion != region.AWSRegion:
return c, fmt.Errorf("%w: it is in account %q, region %q", ErrNotMember, m.Owner, m.AvailabilityZoneRegion)
- case len(m.RequestIDs) < 2:
- return c, fmt.Errorf("%w: the cloud gave no request ids", ErrMembershipUnknown)
- }
- for _, id := range m.RequestIDs {
- if id == "" {
- return c, fmt.Errorf("%w: the cloud gave a blank request id", ErrMembershipUnknown)
- }
}
return c, nil
}
@@ -3729,7 +3771,8 @@ func writeBound(ctx context.Context, d Directory, c membershipConfirmation, now
// context is cancelled, so an S3 request is abandoned at the HTTP layer.
// It bounds how long the console waits, not when a write commits: a
// call cut off may still commit later, and is settled from the roster on
-// a later pass. A var so tests can shorten it.
+// a later pass. confirmMembership gives each membership read the same
+// limit. A var so tests can shorten it.
var storeCallTimeout = 15 * time.Second
// deadlineStore refuses every call once check says the deadline has been
@@ -3938,22 +3981,15 @@ func GroupBoxReference(account, instance string) string {
-// Confirmers write this create-only evidence with read-only ASG/EC2 results.
-// This consumer gives H16/H17 their admission boundary; spec 4 owns producing it.
-func (d Directory) EnrollAutomatic(ctx context.Context, sg release.Signer, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, now time.Time) (Observed, error) {
-+// CheckConfirmed is EnrollAutomatic's admission check, made without
++// checkConfirmed is EnrollAutomatic's admission check, made without
+// writing: the attempt in the roster matches the request and the record,
+// the request's reference is the instance's group reference
+// (GroupBoxReference), and a describe made now confirms the attempt's
+// instance is a current member of the region's group. It returns in
-+// pinned to that instance and what the read saw. The console's enrollment
-+// pass asks it immediately before it invites a box, and makes the
-+// invitation under the read's deadline (autoenroll.go).
-+func (d Directory) CheckConfirmed(ctx context.Context, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (EnrollmentRequest, MembershipEvidence, error) {
-+ pinned, c, err := d.checkConfirmed(ctx, o, v, in, raw, instance, r, now)
-+ if err != nil {
-+ return EnrollmentRequest{}, c.evidence(), err
-+ }
-+ return pinned, c.evidence(), nil
-+}
-+
++// pinned to that instance and the confirmation the read built, which
++// never leaves this package. The console's enrollment pass asks it
++// immediately before it invites a box, and makes the invitation on the
++// store writeBound binds to that confirmation (autoenroll.go).
+func (d Directory) checkConfirmed(ctx context.Context, o Observed, v boxid.Verifier, in EnrollmentRequest, raw []byte, instance string, r MembershipReader, now func() time.Time) (EnrollmentRequest, membershipConfirmation, error) {
if err := validateScope(o.Roster, Scope{Namespace: in.Namespace, Regions: []string{in.Region}, Pools: []string{in.Pool}}); err != nil {
- return Observed{}, err
@@ -4268,11 +4304,11 @@ git commit -m "region HA 2b: group membership, read by the console"
- Modify: `region_ha_boundary_test.go`, the overlay, `internal/provision/plan14_root_reads_test.go` (the seal)
**Interfaces:**
-- Consumes: `Directory.Read`, `Invite`, `BeginEnrollmentAttempt`, `CheckConfirmed`, `EnrollAutomatic`, `GroupBoxReference`, `writeBound` and `ErrWriteDeadline` (Task 4), `MembershipReader`, `VerifyReport`, `Evidence.BoxReport`, `RoleName`; `awsmembership.New` (Task 4); `boxid.Load`, `Verifier`, `EmbeddedCerts`; `hostfloor.Read`, `Digest`; the console's `ClaimAuthority` (`Directory`, `Signer`, `Evidence`, `Namespace`).
+- Consumes: `Directory.Read`, `Invite`, `enrollmentNonce`, `BeginEnrollmentAttempt`, `checkConfirmed`, `EnrollAutomatic`, `GroupBoxReference`, `writeBound` and `ErrWriteDeadline` (Task 4), `MembershipReader`, `VerifyReport`, `Evidence.BoxReport`, `RoleName`; `awsmembership.New` (Task 4); `boxid.Load`, `Verifier`, `EmbeddedCerts`; `hostfloor.Read`, `Digest`; the console's `ClaimAuthority` (`Directory`, `Signer`, `Evidence`, `Namespace`).
- Produces (`hostactors`): `type AutoEnroller struct { Directory; Signer; Evidence; Members MembershipReader; Certs; Namespace, Pool string; Now func() time.Time; Patience time.Duration; Logf func(string, ...any) }`; `func (e *AutoEnroller) EnrollPass(ctx) ([]EnrollStep, error)`; `EnrollStep{Box, Step string}`; `ErrNoFloor`.
- Produces (`console`): `newAutoEnroller(*hostops.ProductionAuthority, hostactors.MembershipReader) (*hostactors.AutoEnroller, error)`; `runEnrollPass(ctx, e, every, logf)`; `(*Server).startEnrollPass(ctx)`; `consoleMembershipReader(ctx)` (`awsmembership.go`, the adapter); `enrollPassInterval = 10s`, `enrollPatience = 2m`.
-For each box whose identity record is published and which the roster does not list, the pass takes the next step its state allows (§5 step 5, A2-1, H28): with no usable attempt, begin one; with one, read the instance's membership itself (`CheckConfirmed`) and invite on the directory bound to that read's deadline (`writeBound`), so `Invite`'s own read, its signature's `Put` and any read-back of its own are refused at or after the deadline, a call in flight then is cancelled, and one still unanswered 15 seconds after it began is cut off; once invited, wait for the box's guard-ready report to pass `VerifyReport` (the check `membership.go:40` makes at admission), and then enroll through `EnrollAutomatic`, which reads membership again itself and writes on its own bound directory. The same attempt serves the invitation and the enrolment: the read, not the attempt, carries the freshness, and the attempt's `Consumed` refuses a replay. Round 3 removed the check the pass made just before `Invite`: the bound store is the single enforcement point, and nothing that waits sits between the describe and the `Invite` call.
+For each box whose identity record is published and which the roster does not list, the pass takes the next step its state allows (§5 step 5, A2-1, H28): with no usable attempt, begin one; with one, read the instance's membership itself (`checkConfirmed`) and invite on the directory bound to the confirmation that read returns (`writeBound`), so `Invite`'s own read, its signature's `Put` and any read-back of its own are refused at or after the deadline, a call in flight then is cancelled, and one still unanswered 15 seconds after it began is cut off; once invited, wait for the box's guard-ready report to pass `VerifyReport` (the check `membership.go:40` makes at admission), and then enroll through `EnrollAutomatic`, which reads membership again itself and writes on its own bound directory. The same attempt serves the invitation and the enrolment: the read, not the attempt, carries the freshness, and the attempt's `Consumed` refuses a replay. An invited box is found with `enrollmentNonce`, which reads and verifies the invitation and writes nothing; `Invite` is get-or-create, so it runs only on the bound store after a membership read, and an invitation that vanished between two reads is never minted again without one. Round 3 removed the check the pass made just before `Invite`: the bound store is the single enforcement point, and nothing that waits sits between the describe and the `Invite` call.
A read that fails is unknown: the step names it, and the next pass asks again with the same attempt, whose wait does not grow (#7). So is a write the bound could not confirm (`ErrWriteDeadline`, S1, U2): nothing was issued at or after the deadline, but a call cut off in flight, or one that answered only after the deadline, may have committed, or may still commit later (V1). `TestAWriteTheConsoleGaveUpOnCanCommitLater` documents that exposure: the enrolling `Put` blocks, the pass returns unknown, the instance is detached, the next pass finds the box absent and its fresh describe refuses without writing, then the delayed `Put` commits on the unchanged roster revision, and the pass after that sees the box enrolled and does nothing more. Such an enrolment rests on a membership read that was true when its write was issued: the exposure of an instance detached after enrolment, which spec 3's worker reconciles. An invitation that landed is simply found and the box goes on to its guard-ready report: an invitation grants nothing.
@@ -4513,7 +4549,7 @@ func TestAMembershipReadThatFailsKeepsTheAttempt(t *testing.T) {
// C2: the invitation is written only while the console's read is under a
// minute old. Here every look at the clock takes 31 seconds: the read is
-// fresh when CheckConfirmed has it, and over a minute old by the time the
+// fresh when checkConfirmed has it, and over a minute old by the time the
// invitation would be written. That is unknown: the attempt and its wait
// stand.
func TestAnInvitationNeedsAReadUnderAMinuteOldAtTheWrite(t *testing.T) {
@@ -5057,7 +5093,7 @@ import (
//
// 1. no invitation: begin an enrollment attempt (BeginEnrollmentAttempt),
// then, on the next pass, confirm the instance's group membership with
-// the console's own read (CheckConfirmed, Members) and invite the box
+// the console's own read (checkConfirmed, Members) and invite the box
// (Invite) while that read is under a minute old;
// 2. invited: wait for the box to arm its guard and publish its
// guard-ready report (VerifyReport, the same check membership.go makes
@@ -5232,23 +5268,28 @@ func (e *AutoEnroller) step(ctx context.Context, o Observed, floor hostfloor.Ver
}
in := EnrollmentRequest{Reference: ref, Namespace: e.Namespace, Label: ref, Region: name, Pool: e.Pool, AWSRegion: region.AWSRegion}
- _, _, err = d.Store.Get(ctx, "control/host-actor-invitations/"+ref+".json")
+ // The invitation is only read here, once, and verified for this
+ // record (enrollmentNonce writes nothing). Invite is get-or-create, so
+ // it runs only on the bound store after a membership read
+ // (confirmAndWrite): an invitation that vanished between two reads is
+ // never minted again without one.
+ nonce, err := d.enrollmentNonce(ctx, in, raw)
invited := err == nil
- if err != nil && !errors.Is(err, os.ErrNotExist) {
+ switch {
+ case err == nil, errors.Is(err, os.ErrNotExist):
+ case errors.Is(err, ErrRoster):
+ return Observed{}, fmt.Sprintf("its invitation is not for this record: %v", err)
+ default:
return Observed{}, fmt.Sprintf("its invitation could not be read: %v", err)
}
if invited {
- inv, err := d.Invite(ctx, e.Signer, ref, raw)
- if err != nil {
- return Observed{}, fmt.Sprintf("its invitation is not for this record: %v", err)
- }
// The report is read under the reference derived from the
// attempt's proven instance (PrivateBoxReport, Group set).
rep, err := e.Evidence.BoxReport(ctx, ref, Box{Namespace: e.Namespace, Label: ref, Group: region.GroupName, Cloud: ev.Cloud})
if err != nil {
return Observed{}, "waiting for its guard-ready report"
}
- if err := VerifyReport(ctx, e.Evidence, d.PublicKey, floor, rep, inv.Enrollment, instance, now); err != nil {
+ if err := VerifyReport(ctx, e.Evidence, d.PublicKey, floor, rep, nonce, instance, now); err != nil {
return Observed{}, "waiting for its guard-ready report"
}
}
@@ -5316,7 +5357,10 @@ func (e *AutoEnroller) confirmAndWrite(ctx context.Context, o Observed, v boxid.
}
return next, fmt.Sprintf("enrolled in %s with the group's role, %s", name, RoleName(region.Role)), nil
}
- _, ev, err := d.CheckConfirmed(ctx, o, v, in, raw, instance, e.Members, e.Now)
+ // checkConfirmed returns the confirmation its read built, and the
+ // write is bound to that, never to one rebuilt from exported evidence.
+ _, c, err := d.checkConfirmed(ctx, o, v, in, raw, instance, e.Members, e.Now)
+ ev := c.evidence()
if err != nil {
e.logRead(in, instance, ev, "", err)
return Observed{}, fmt.Sprintf("its membership could not be confirmed: %v", err), err
@@ -5329,7 +5373,7 @@ func (e *AutoEnroller) confirmAndWrite(ctx context.Context, o Observed, v boxid.
// answers only after the deadline. The next pass settles it from the
// bucket.
// Nothing that waits sits between the read and this call.
- bd, bctx, cancel := writeBound(ctx, d, membershipConfirmation{observedAt: ev.ObservedAt}, e.Now)
+ bd, bctx, cancel := writeBound(ctx, d, c, e.Now)
defer cancel()
_, err = bd.Invite(bctx, e.Signer, in.Reference, raw)
e.logRead(in, instance, ev, "invited", err)
@@ -8400,26 +8444,29 @@ What 2c builds, and exactly what 2b needs from it:
1. **The start-up script's order** (§5, R5). The launch template's user data names the deployment, its bucket, the region's placement code and its AWS region (tofu knows all four), so the script needs no carlos binary to begin:
1. Harden.
- 2. Fetch `s3://<deployment bucket>/bootstrap/carlos` with the AWS CLI (`aws s3 cp`).
+ 2. Fetch `s3://<deployment bucket>/bootstrap/carlos` with the AWS CLI (`aws s3 cp`) to a path of its own, outside both protected slots (below).
3. Read `/carlos/<deployment>/shared/bootstrap-sha256` with the AWS CLI (`aws ssm get-parameter --name /carlos/<deployment>/shared/bootstrap-sha256 --with-decryption --query Parameter.Value --output text`) and check the file against it (`sha256sum`). A mismatch, or a read that fails, stops the script.
- 4. `<bootstrap binary> system host-env write --deployment <name> --region <code> --aws-region <aws region>` as root. It needs only the box role's SSM and KMS read, and creates `/etc/carlos`.
+ 4. `<bootstrap binary> system host-env write --deployment <name> --region <code> --aws-region <aws region>` as root. It needs only the box role's SSM and KMS read, and creates `/etc/carlos`. It needs the AWS region from `--aws-region` or `AWS_REGION`: the SDK does not read it from IMDS, so the script passes the flag.
5. `<bootstrap binary> bootstrap`: users, directories (with `/data` handed to the service user), units, including `carlos-system-join.path`.
6. With host.env's variables in the environment, `<bootstrap binary> system install-channel` as root. It hands the directories it creates under `/data` to the service user.
- 7. Delete the bootstrap binary, and start `carlos-edge.service`. The agent joins on its own from there.
+ 7. Delete the bootstrap binary by its own path, and start `carlos-edge.service`. The agent joins on its own from there.
**The bootstrap object must be a release with 2b in it**: it runs `system host-env write` and `system install-channel`, and its `bootstrap` installs the join units.
+
+ **The bootstrap binary sits outside both protected slots**, `/opt/carlos/carlos` and `/opt/carlos/carlos.prev`, for example at `/root/carlos-bootstrap` (final review). `install-channel` refuses a slot that holds a build it did not install, so a bootstrap binary fetched into a slot stops the install. And step 7's delete must never touch an installed binary: the bootstrap file is neither a slot nor a link to one.
2. **The parameter layout.** One SecureString per variable, named for it, under `/carlos/<deployment>/<region>/`, and under `/carlos/<deployment>/shared/` for `CARLOS_DEPLOYMENT_BUCKET`, `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS`, `CARLOS_RELEASE_PUBKEY` and `CARLOS_SECRETS_KEY`. The required set is `hostenv.Required`. `bootstrap-sha256` under shared is the script's, and `host-env write` skips it (not a variable name). Deployment and region names are lower-case labels. An operator seeds a region with `carlos system host-env seed --deployment <name> --region <code> --kms-key <the configuration key>` from one of that region's existing boxes; shared values are written create-only, so two operators seeding at once cannot leave different shared values behind.
3. **IAM, box role** (§4):
- `ssm:GetParametersByPath` on exactly `/carlos/<deployment>/<region>` and `/carlos/<deployment>/shared`, and `ssm:GetParameter` on `/carlos/<deployment>/shared/bootstrap-sha256` for the script; `kms:Decrypt` on the configuration key, conditioned on `kms:ViaService = ssm.<region>.amazonaws.com` and the `PARAMETER_ARN` encryption context; the explicit SSM Deny kept for everything outside the two paths.
- **No** Auto Scaling or EC2 describe, and **no** write under `control/box-enrollment/` (H28, C4). §4's rev 15 box grant of `autoscaling:DescribeAutoScalingInstances` and `DescribeAutoScalingGroups` is withdrawn; 2c must not add it.
- **No** `ssm:PutParameter` and **no** `ssm:DeleteParameter` on either parameter path (census condition 1 on `runHostEnv`): a box reads the parameters it is configured from and never writes them; only the operator's principal (seam 4), and the console's if 2c gives it one, write there.
- S3 in the deployment bucket: `GetObject` on `control/host-actor-invitations/*`, `control/boxes/*`, `control/host-actors.json`, `control/moves-protocol.json`, `control/moves-protocol-auth.json`, `control/system/router/*` (the channel pointer and release identities), `bootstrap/router/*` (artifacts), `bootstrap/carlos` (the script's fetch), `control/box-fences/*`, and `status/boxes/*`; `PutObject` on the box's own `status/boxes/<label>.json` (as today) and `control/boxes/<label>.json` (create-only, as today). Most of these the box role already has on main; 2c should check the list against the module rather than add duplicates.
- - **The negative check, 2c's infrastructure test (C4).** With the box role's credentials, both of these must be denied: creating an object under `control/box-enrollment/` (`aws s3api put-object --bucket <deployment bucket> --key control/box-enrollment/probe/probe/probe.json --if-none-match '*'`), and `aws autoscaling describe-auto-scaling-instances --instance-ids <a group instance>` in the region's AWS region. `aws ec2 describe-instances` should be checked the same way. Add, for census condition 1: with the same credentials, `aws ssm put-parameter --name /carlos/<deployment>/<region>/CARLOS_STATUS_PREFIX --type SecureString --value probe --overwrite` and `aws ssm delete-parameter --name /carlos/<deployment>/<region>/CARLOS_STATUS_PREFIX` are both denied, and the same for one name under `/carlos/<deployment>/shared/`.
+ - **The negative check, 2c's infrastructure test (C4).** With the box role's credentials, both of these must be denied: creating an object under `control/box-enrollment/` (`aws s3api put-object --bucket <deployment bucket> --key control/box-enrollment/probe/probe/probe.json --if-none-match '*'`), and `aws autoscaling describe-auto-scaling-instances --instance-ids <a group instance>` in the region's AWS region. `aws ec2 describe-instances` should be checked the same way. Add, for census condition 1: with the same credentials, `aws ssm put-parameter --name /carlos/<deployment>/<region>/CARLOS_SWAP --type SecureString --value probe --overwrite` and `aws ssm delete-parameter --name /carlos/<deployment>/<region>/CARLOS_SWAP` are both denied, and the same for one name under `/carlos/<deployment>/shared/`.
4. **IAM, operator seeding** (S4, Astra round 2 new #2): `ssm:PutParameter` (with and without overwrite) and `ssm:GetParametersByPath` on the two paths; `kms:Encrypt` on the configuration key; and `kms:Decrypt` on it, conditioned on `kms:ViaService = ssm.<region>.amazonaws.com` and the `PARAMETER_ARN` encryption context matching the two paths. `seed` reads the shared values decrypted (`ByPath` asks with `WithDecryption`) to compare them, and reads one back after a lost create race, so without `kms:Decrypt` seeding a second region, or resolving a race, fails. RHB-C61 names all four.
5. **IAM, console role** (H28, C4): `autoscaling:DescribeAutoScalingInstances` and `ec2:DescribeInstances`, read-only, conditioned on `aws:RequestedRegion` being one of the deployment's AWS regions (every region the roster's `Regions` name, so an emptied region recovers the same way). Neither action takes a resource ARN, so the region condition is the limit. No other new AWS call. The enrolment pass reads and writes the deployment bucket as the console already does: `control/host-actors.json` (CAS replace), `control/host-actor-invitations/*` (create-only), and reads `control/boxes/*`, `status/boxes/*`, `control/moves-protocol*.json`, `control/system/router/releases/*`. It no longer reads `control/box-enrollment/*`. 2c should confirm the console role's bucket statement covers the invitation prefix, and add the two describes.
-6. **The health check and the grace period.** A joining box answers `/.carlos/lb-health` on port 80 with 503 until it restarts as the claim agent. The grace period must cover the boot, the install, and joining: the console's pass every 10 seconds (an attempt, then its membership read and the invitation on the next pass), root's arm on the path unit, the report within a minute, the console's second read and the enrolment, and the restart. A few minutes on dev; §4's 20 minutes is ample, and dev should measure it. Dev should also stop and start one enrolled group box once and see it held at 503 and replaced, and reboot one and see it host again (Open question 7).
+6. **The health check and the grace period.** A joining box answers `/.carlos/lb-health` on port 80 with 503 until it restarts as the claim agent. The grace period must cover the boot, the install, and joining: the console's pass every 10 seconds (an attempt, then its membership read and the invitation on the next pass), root's arm on the path unit, the report within a minute, the console's second read and the enrolment, and the restart. A few minutes on dev; §4's 20 minutes is ample, and dev should measure it. **No launch lifecycle hook on the group**: a box held in `Pending:Wait` reads to the console as not a member (only `Pending` and `InService` are), so its attempt is refused and waits out the backoff, and joining stalls behind the hook. Dev should also stop and start one enrolled group box once and see it held at 503 and replaced, and reboot one and see it host again (Open question 7).
7. **Status report expiry (T2).** A lifecycle rule on the deployment bucket expiring objects under `status/boxes/` 30 days after they were last written (beside the existing rules in `infra/modules/deployment-buckets/main.tf`). It is hygiene only: the status reader already drops reports no box has rewritten in a day before its key cap, so nothing depends on the rule, and a self-hosted store without lifecycle rules works the same. Nothing reads old reports for audit or history: every reader of the prefix wants current reports (Task 6), so the expiry loses nothing.
8. **The roster.** Each region with a group needs, in the signed roster, `Region{Account, AWSRegion, GroupName, GroupARN, Namespaces (the console's namespace), Pools ("default")}`, and the deployment's host floor published. 2b reads these; the `region-group` phase writes the group's fields (Open question 1).
+9. **`control/box-enrollment/` leaves the deployment bucket's module.** Remove it from `box_evidence_prefixes` in `infra/modules/deployment-buckets/main.tf`, with its comment and the rows in `tests/evidence.tftest.hcl`. Nothing writes membership confirmations there any more, and the console no longer reads them (H28). Seam 3's negative check still runs: the box role is denied a create there whatever the bucket policy lists.
## Open questions
@@ -8430,5 +8477,5 @@ What 2c builds, and exactly what 2b needs from it:
5. **`DescribeAutoScalingGroups`.** Not needed: the console's read binds to the group name and ARN the signed roster names. Spec rev 16 drops it from §4.
6. **The invitation's confirmation.** Resolved by H28: the console reads membership itself immediately before the invitation and again before the enrolment, so the same attempt serves both and no fresh attempt is begun after the guard-ready report. Spec rev 16's §5 says so.
7. **`pendingTime` on a start: an activation gate.** The held-after-restart rule (C3) rests on EC2 setting the identity document's `pendingTime` at every start and never at a reboot. AWS documents the field as the launch time, but not that exact invariant (Astra round 2). Verify on dev before the first region: stop and start a group box (expect held, 503, replaced) and reboot one (expect hosting). Until then automatic enrolment stays off ("Activation gates", 2).
-8. **The allowlist's completeness.** `hostenv.Optional` was drawn from README's host.env surface and the variables the agent reads. A deployment whose boxes carry another variable will find `seed` refusing it by name; adding it is a code change and a review, which is the point. Check carloku.com's regions' host.env files against the list before the first seed.
+8. **The allowlist's completeness.** `hostenv.Optional` was drawn from README's host.env surface and the variables the agent reads. A deployment whose boxes carry another variable will find `seed` refusing it by name; adding it is a code change and a review, which is the point. Check carloku.com's regions' host.env files against the list before the first seed: a box that carries `CARLOS_DNSIMPLE_TOKEN`, `CARLOS_ALERT_WEBHOOK` or a status destination override has to drop it, or the region cannot seed ("Activation gates", 4).
9. **Detached but running instances: spec 3's requirement (U1, platform-1e).** 2b removes nothing from the roster and pages no one: an instance detached from its group that keeps running keeps its enrolment until an operator removes it. Spec 3's requirements list carries it, and spec 2's hand-off to spec 3 (§12) says so: **the regional worker reconciles roster membership against Auto Scaling group membership, including detached-but-running instances.**
diff --git a/docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md b/docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md
new file mode 100644
index 000000000..85638618f
--- /dev/null
+++ b/docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md
@@ -0,0 +1,47 @@
+Reviewed `26065378..ac45a709` at HEAD `4433a788`. **Two P2 findings and one P3.** No builds or tests run; worktree unchanged.
+
+1. **P2 — A stalled membership read stops enrollment across all regions.**
+ [membershipread.go:106](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha-2b-build/internal/hostactors/membershipread.go:106) calls `Member` with the console worker’s unbounded context. Neither AWS describe adds a deadline, and the SDK client has no overall request timeout. A connection that stalls after connecting can block the serial enrollment pass indefinitely. `writeBound` starts only after this read returns.
+ **Fix:** bound the membership read, classify timeout as unknown, and retain the attempt without increasing backoff. Add a test with a reader that waits for cancellation and verify subsequent boxes still progress.
+
+2. **P2 — Accepted status overrides break the group-report contract.**
+ [hostenv.go:86](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha-2b-build/internal/hostenv/hostenv.go:86) permits `CARLOS_STATUS_BUCKET` and `CARLOS_STATUS_PREFIX`, but [PrivateBoxReport](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha-2b-build/internal/hostactors/private.go:20) reads the deployment bucket’s unprefixed key. A different status bucket leaves the console waiting forever for guard readiness. With only a prefix, joining writes the expected unprefixed report, but hosting switches to prefixed reports; the console’s original report then expires.
+ **Fix:** reject incompatible overrides for group boxes, or make joining, hosting and console readers share one supported destination. Test the production report path through the joining-to-hosting transition.
+
+3. **P3 — The allowlist still admits credentials embedded in webhook URLs.**
+ [hostenv.go:71](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha-2b-build/internal/hostenv/hostenv.go:71) accepts `CARLOS_ALERT_WEBHOOK`, including URLs containing authentication tokens. Its characters pass `Render`. This contradicts the binding credential exclusion used to remove `CARLOS_DNSIMPLE_TOKEN`.
+ **Fix:** remove it from the group allowlist and test refusal without exposing its value, unless the census owner explicitly approves an exception.
+
+Deferred-minor triage:
+
+| Disposition | Items |
+|---|---|
+| **Before merge** | Fix the status mismatch and webhook exclusion above. Update the plan’s obsolete DNSimple-token allowance and `CheckConfirmed` references. Explicitly require the bootstrap binary to sit outside both protected slots: otherwise installation refuses it, or the subsequent bootstrap-file deletion can remove the installed binary. |
+| **Closed by inspection** | Missed write deadlines do not become `ErrMembershipStale` or increase backoff. Empty-reference errors are retried as unknown. Invited, unarmed boxes re-touch the marker each minute. |
+| **Before activation** | Keep the stop/start versus reboot verification. Add the fresh-box guard-arm → recovery → guard-exec restart rehearsal as an explicit activation gate. Static inspection cannot establish that systemd transition. |
+| **Nonblocking follow-ups** | Error wrapping and wording; cleanup logging; ownership, zero-timestamp and gate-test gaps; SDK-client reuse; ignored flags; host-env timeout, reseeding and durability details; joining cancellation, flag parsing, log deduplication and response-body cleanup. |
+| **No change required** | Command syntax is exempt from copy review; approved CLI prose need not gain backticks. The alleged mixed shared-key race is not supported by this implementation: all shared keys are required, processed in sorted order, and a loser stops at its first conflicting key. Partial progress on errors still warrants narrowing the comment. |
+
+The [build ledger records a separate merge hold](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/region-ha-2b-build/.superpowers/sdd/2026-10-17-region-ha-2b-joining/progress.md:109) until platform-1e posts “published” on discussion #288. Its current external status was not checked.
+
+**Verdict: not ready to merge.**
+
+---
+
+**FIRST PROMPT:**
+Final whole-branch review of region-ha-2b-build at `4433a788`, read-only, without heavy tests.
+
+**WHAT:**
+Review the implementation and triage deferred minors.
+
+**STATUS:**
+Review complete: two P2 findings, one P3; no files changed.
+
+**NEEDED:**
+Nothing for this review.
+
+**RECOMMENDED:**
+Fix the findings and plan discrepancies, add focused regressions, then verify the publication hold is cleared.
+
+**SESSION:**
+Safe to close — no processes started or scratch files created.
\ No newline at end of file
diff --git a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md
index e35f5ae17..81159877e 100644
--- a/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md
+++ b/docs/superpowers/specs/2026-10-15-region-ha-two-box-regions-design.md
@@ -1,6 +1,6 @@
# Region HA, spec 2: two-box regions
-Design, 2026-10-16, **rev 20**. Rev 15 was approved by Paul on 2026-10-16; rev 16's ruling H28 was approved in direction the same day, and platform-1e's census ruling region-ha-2b was RULED by the Plan 14 controller the same evening, with conditions. Nothing here is built.
+Design, 2026-10-17, **rev 21**. Rev 15 was approved by Paul on 2026-10-16; rev 16's ruling H28 was approved in direction the same day, and platform-1e's census ruling region-ha-2b was RULED by the Plan 14 controller the same evening, with conditions. Nothing here is built.
**Rev 18** answers Astra's third review of plan 2b (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-3.md`, cited **B3-n**) and the census ruling's conditions where the spec changes: the write deadline's anchor, edge and single enforcement point, and the read-back of an unknown enrolment (§5 step 5); a group box publishes no status until its reference resolves, and the status reader's listing (§5 step 5); host.env's allowlist and cross-check (§5 step 2). §30 maps them.
@@ -8,6 +8,8 @@ Design, 2026-10-16, **rev 20**. Rev 15 was approved by Paul on 2026-10-16; rev 1
**Rev 20** answers Astra's fifth review of plan 2b (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-5.md`, cited **B5-n**) and platform-1e's acceptance of revision 4: the 15-second timeout bounds how long the console waits, not when a write commits, and rev 19's 75-second commit ceiling is withdrawn; every write in the invitation and enrolment paths is conditional, so a late commit lands only where it was aimed (§5 step 5). §32 maps them.
+**Rev 21** answers Astra's final review of the plan 2b build (`docs/superpowers/plans/CODEX-FINAL-REVIEW-REGION-HA-2B.md`, cited **BF-n**): each membership describe is cut off at 15 seconds as unknown (§5 step 5); host.env's allowlist leaves out the alert webhook and the status destination overrides (§5 step 2); and automatic enrolment gains two activation gates (§5). §33 maps them.
+
**Rev 17** answers Astra's second review of plan 2b (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-2.md`, cited **B2-n**) where the spec changes: the membership read's 60 seconds are a deadline on the write itself, not a check before it (§5 step 5); a box its region's group launched is referenced by a group reference, not a status label (§5); operator seeding needs `kms:Decrypt` (§5 step 2); and automatic enrolment has two activation gates (§5). §29 maps them.
**Rev 16** records Paul's ruling **H28**: the console is the only membership confirmer. It confirms a group box's membership itself, with its own read-only Auto Scaling and EC2 describes, immediately before it invites and before it enrolls; no box writes or contributes membership evidence, and H17's cross-region box confirmer is withdrawn. It answers Astra's review of plan 2b (`docs/superpowers/plans/CODEX-REVIEW-REGION-HA-2B-1.md`, cited **B1-n**) where the spec changes: §4's IAM, §5's boot order and enrolment, §10, §11 and §25. §28 maps them.
@@ -189,7 +191,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita
- the alert topic;
- `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS` and `CARLOS_REGION_HA`.
- The file is written mode 0600, owned by root. A missing required parameter stops the boot. Configuration comes before the install, because the install verifies the release with the key it holds. Shared values are seeded create-only, so two seeds can't leave different shared values behind. Only an allowlist of variable names is taken from the parameters (any other is refused by name), values are limited to a safe set, and the file is replaced whole. Its path is fixed, `/etc/carlos/host.env`, with no flag to move it (B4-4). `/etc` and `/etc/carlos` are each opened by descriptor, without following a symlink, and refused unless they are directories root owns that no group or other can write (a missing `/etc/carlos` is created 0755, then opened and checked the same way); the new file is created 0600 inside the held directory, fsynced, renamed over the old one within it, and the directory fsynced. Off Linux the write refuses. The two values that select a box's authority, the group marker below and `CARLOS_REGION`, are checked against the signed roster before a group box hosts, not at write, since at boot the box has no roster entry yet. Box roles never write the parameters. Seeding reads the shared values back decrypted, so the operator needs `ssm:GetParametersByPath` and `ssm:PutParameter` on the two paths, `kms:Encrypt` on the configuration key, and `kms:Decrypt` on it conditioned on `kms:ViaService = ssm.<region>.amazonaws.com` and the `PARAMETER_ARN` encryption context of the two paths (B2-new-2). `host-env write` also adds `CARLOS_BOX_REFERENCE=group`, which marks the box as one its group launched; it is never a parameter.
+ The file is written mode 0600, owned by root. A missing required parameter stops the boot. Configuration comes before the install, because the install verifies the release with the key it holds. Shared values are seeded create-only, so two seeds can't leave different shared values behind. Only an allowlist of variable names is taken from the parameters (any other is refused by name), values are limited to a safe set, and the file is replaced whole. The allowlist leaves out credentials, `CARLOS_ALERT_WEBHOOK` among them, since a webhook URL can carry one (BF-3), and the status destination overrides, `CARLOS_STATUS_BUCKET`, `CARLOS_STATUS_PREFIX` and `CARLOS_STATUS_REGION`: a group box reports to the deployment bucket, unprefixed, where the console reads its guard-ready report, while joining and once hosting (BF-2). Its path is fixed, `/etc/carlos/host.env`, with no flag to move it (B4-4). `/etc` and `/etc/carlos` are each opened by descriptor, without following a symlink, and refused unless they are directories root owns that no group or other can write (a missing `/etc/carlos` is created 0755, then opened and checked the same way); the new file is created 0600 inside the held directory, fsynced, renamed over the old one within it, and the directory fsynced. Off Linux the write refuses. The two values that select a box's authority, the group marker below and `CARLOS_REGION`, are checked against the signed roster before a group box hosts, not at write, since at boot the box has no roster entry yet. Box roles never write the parameters. Seeding reads the shared values back decrypted, so the operator needs `ssm:GetParametersByPath` and `ssm:PutParameter` on the two paths, `kms:Encrypt` on the configuration key, and `kms:Decrypt` on it conditioned on `kms:ViaService = ssm.<region>.amazonaws.com` and the `PARAMETER_ARN` encryption context of the two paths (B2-new-2). `host-env write` also adds `CARLOS_BOX_REFERENCE=group`, which marks the box as one its group launched; it is never a parameter.
3. **Install the signed channel release** (A1-6).
- Run the bootstrap binary's `carlos bootstrap` for users, directories and units (`internal/host/bootstrap.go`). It doesn't install a release.
- Then install the release named by the signed channel pointer, verified with `CARLOS_RELEASE_PUBKEY`, through the same staging and verification path `carlos system update` uses (`internal/host/systemleg.go`, `internal/release/systempointer.go`): `carlos system install-channel`. The directories it creates under `/data` are handed to the service user, as bootstrap's own are, so the agent's later updates can download there.
@@ -207,7 +209,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita
5. **Hosting mode.** The agent's next start check finds the roster entry and the armed guard, and it restarts as the claim-mode agent. Only now does it publish edges files and answer health 200 (§6), and only then is it eligible for placement (§7).
**The rules of the read.**
- - **Freshness: 60 seconds, enforced at the write, and a 15-second hard timeout per call** (B2-new-1, B3, B4-1). A fresh describe runs immediately before each write, the invitation and the enrolment. **The write deadline is anchored at the completion of the membership describe or earlier, never later (2b anchors it at the describe's start), and a write 60 s or more after the anchor is refused: the write is issued no later than 60 s after the membership read. Each store call issued before the deadline has a hard timeout of 15 s.** What that guarantees (B5-1): every store call of the invite and the enrolling write is issued before the deadline (60 s from the start of the membership describe); the console waits at most 15 s for any call; a write whose result the console did not see is unknown and is settled on a later pass by reading the roster. A cancelled write can still commit later; such an enrolment rests on a membership read that was true when the write was issued, and is the same exposure as an instance detached after enrolment, which spec 3's worker reconciles. The invitation and the enrolling write, with every bucket read and signature they make on the way (the sealed `Invite`, `checkMembership` and `Replace` unchanged), run on a store bound to the deadline, the single enforcement point. It guarantees that no store call, the roster `Put` included, is issued at or after the deadline; its context cancels a call in flight at the deadline; a call not answered 15 s after it was issued is cut off by the wrapper itself, whether or not the store honours its context; and a call that answers only after the deadline, even with success, is unknown. So a write whose commit can't be confirmed before the deadline is unknown. The hard timeout is the call's own context, which the S3 client enforces at the HTTP layer; the console's shared S3 client is not changed and no second client is built. Every write on these paths is conditional: the roster write is a compare-and-swap on the ETag the pass read, and an invitation is created only if absent, through every wrapper of the console's store. So a write that commits late lands only on the exact roster revision it was built against, and never overwrites a newer revision or another enrolment. A missed deadline keeps the attempt and its backoff; a describe that itself took 60 seconds or more counts against it.
+ - **Freshness: 60 seconds, enforced at the write, and a 15-second hard timeout per call** (B2-new-1, B3, B4-1). A fresh describe runs immediately before each write, the invitation and the enrolment. **The write deadline is anchored at the completion of the membership describe or earlier, never later (2b anchors it at the describe's start), and a write 60 s or more after the anchor is refused: the write is issued no later than 60 s after the membership read. Each store call issued before the deadline has a hard timeout of 15 s.** What that guarantees (B5-1): every store call of the invite and the enrolling write is issued before the deadline (60 s from the start of the membership describe); the console waits at most 15 s for any call; a write whose result the console did not see is unknown and is settled on a later pass by reading the roster. A cancelled write can still commit later; such an enrolment rests on a membership read that was true when the write was issued, and is the same exposure as an instance detached after enrolment, which spec 3's worker reconciles. The invitation and the enrolling write, with every bucket read and signature they make on the way (the sealed `Invite`, `checkMembership` and `Replace` unchanged), run on a store bound to the deadline, the single enforcement point. It guarantees that no store call, the roster `Put` included, is issued at or after the deadline; its context cancels a call in flight at the deadline; a call not answered 15 s after it was issued is cut off by the wrapper itself, whether or not the store honours its context; and a call that answers only after the deadline, even with success, is unknown. So a write whose commit can't be confirmed before the deadline is unknown. The hard timeout is the call's own context, which the S3 client enforces at the HTTP layer; the console's shared S3 client is not changed and no second client is built. Every write on these paths is conditional: the roster write is a compare-and-swap on the ETag the pass read, and an invitation is created only if absent, through every wrapper of the console's store. So a write that commits late lands only on the exact roster revision it was built against, and never overwrites a newer revision or another enrolment. A missed deadline keeps the attempt and its backoff; a describe that itself took 60 seconds or more counts against it. The describe has the same 15-second cut-off (BF-1): neither AWS call sets a deadline of its own, and a stalled connection would otherwise hold the serial pass and every box behind it. A describe cut off is unknown, its attempt kept and its backoff unchanged, and the pass goes on to the next box.
- **Unknown outcomes are settled by the roster** (B4-2, B4-3). An outcome the bound can't confirm is resolved the ordinary way, by the next pass's read of the signed roster: a box the roster lists is enrolled and done; one it doesn't list goes on with the same attempt while that attempt is still valid, under the same rules, and an unknown doesn't grow its backoff. Nothing is read back, remembered or paged. A terminated instance is kept out of hosting and capacity by its termination fence, a stopped one by the stale-edges rule and the restart hold. One that left its group but still runs keeps its enrolment until an operator removes it; reconciling that is spec 3's (§12).
- **Unforgeable.** `EnrollAutomatic` and its check take the console's reader and make the describe themselves; the confirmation is built inside `hostactors` from that answer and never leaves it. No exported path takes a confirmation, so nothing a box writes can stand in for one.
- **Unknown is not invalid** (B1-7). A read that fails, or isn't an answer, is unknown: the pass asks again with the same attempt, and its backoff doesn't grow. Only the cloud's positive answer that the instance isn't a current member, or a read that went stale before its write, counts against the attempt. The pass then keeps the attempt, without describing again, for a wait that doubles with each refusal up to 32 times its base, before it begins another.
@@ -217,7 +219,7 @@ Anything else is joining mode. Joining resumes from whatever it finds: no invita
**The status listing** (B3-2). Each replaced group box leaves a report under its own reference. The status reader drops reports no box has rewritten in a day, from the listing's write times, before any read and before its key cap, so the cap bounds current reports only. It's an optimisation, not authority: every reader still judges a report's own age as before. The deployment bucket also expires `status/boxes/` objects after 30 days, as hygiene; nothing reads old reports for history, and nothing depends on the rule, so a store without lifecycle rules works the same.
- **Activation gates.** Automatic enrolment isn't switched on for any region until (1) a release with group references runs on every box of that region and on the console: without them, a region in its first build, whose group never replaces an unhealthy box, could stall on a label collision that an alarm reports but nothing repairs (B2-3); and (2) the `pendingTime` behaviour behind the stopped-box rule has been seen on dev, before the first region: a group box stopped and started is held and replaced, one rebooted hosts again.
+ **Activation gates.** Automatic enrolment isn't switched on for any region until (1) a release with group references runs on every box of that region and on the console: without them, a region in its first build, whose group never replaces an unhealthy box, could stall on a label collision that an alarm reports but nothing repairs (B2-3); (2) the `pendingTime` behaviour behind the stopped-box rule has been seen on dev, before the first region: a group box stopped and started is held and replaced, one rebooted hosts again; (3) on dev, a fresh group box goes guard-arm, then carlos-system-recover, then guard-exec across the joining-to-hosting restart, and comes up hosting, since static inspection can't establish that systemd transition (BF); and (4) carloku.com's host.env files have been checked against the allowlist, which now excludes the status destination overrides and the alert webhook, before the first seed (BF-2, BF-3).
**What spec 2 builds here:**
- the start-up script, and the SSM parameter layout with a `carlos` command that writes it from today's `host.env`;
@@ -658,3 +660,13 @@ Findings from Astra's second review of plan 2b that change this spec. B2's partl
| platform-1e: the per-call context reaches the SDK | Accepted: plan 2b tests it through the console's production stores, down to the HTTP request |
| platform-1e: late commits land only where they were aimed | Confirmed from main's code: the roster write is a compare-and-swap and an invitation is create-only, through every wrapper of the console's store (§5 step 5) |
| platform-1e: the two host.env units | Ruled A,J4 under census ruling region-ha-2b |
+
+## 33. Astra's final review of the plan 2b build (rev 21)
+
+| Finding | Disposition |
+|---|---|
+| BF-1 P2: a stalled membership read stops enrolment across all regions | Accepted: each describe gets the 15-second cut-off each bound store call has; a describe cut off is unknown, the attempt kept, its backoff unchanged, and the pass goes on to the next box (§5 step 5) |
+| BF-2 P2: accepted status overrides break the group-report contract | Accepted, refused up front: `CARLOS_STATUS_BUCKET`, `CARLOS_STATUS_PREFIX` and `CARLOS_STATUS_REGION` leave host.env's allowlist. A group box reports to the deployment bucket, unprefixed, where the console reads. A deployment that needs a separate status bucket can't use group boxes until joining, hosting and the console share that destination (§5 step 2) |
+| BF-3 P3: the allowlist admits credentials embedded in webhook URLs | Accepted: `CARLOS_ALERT_WEBHOOK` leaves the allowlist, on the rule that keeps the DNSimple token off it. Group boxes alert through SNS until a reviewed exception or a credential store exists (§5 step 2) |
+| BF, before activation: the guard's restart transition | Accepted as activation gate 3: on dev, a fresh group box goes guard-arm, then carlos-system-recover, then guard-exec across the joining-to-hosting restart, and comes up hosting. The stop/start and reboot check stays gate 2 (§5) |
+| BF, before activation: the allowlist against real boxes | Activation gate 4: carloku.com's host.env files checked against the allowlist before the first seed (§5) |