amadan
runbook: delta v2 for Option B (backfill-done, B1/B2/B4/B5, gates, trial assumptions)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+500
−265
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+500 −265
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index 4be8d6d04..113639580 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -1,6 +1,10 @@ |
| # Host claim activation, dev cohort 1: mechanics trial runbook |
| -> **PAUSED 2026-10-19: design first (Paul).** The trial holds at the end of step 7: both boxes armed, floor v0.28.4, roster rev 9, home `ie`, leo's key and fence credentials installed, claims off on leo, everything serving. Do NOT write `control/hosts-backfill.json`, do NOT run 8.1b, and do NOT apply the traffic barrier. |
| +> **PAUSED 2026-10-19: design first (Paul). Resumes under Option B, pending Paul's decisions D2 to D11.** The trial holds at the end of step 7: both boxes armed, floor v0.28.4, roster rev 9, home `ie`, leo's key and fence credentials installed, claims off on leo, everything serving. Do NOT write `control/hosts-backfill.json`, do NOT run 8.1b, and do NOT apply the traffic barrier until Paul's go for the window. |
| +> |
| +> **Option B (2026-10-20).** Fable's plan review (`after236/fable-plan-review.md`, sections 3B, 4 and 6) chose "empty leo, console off, straight to done". The dev console stays claims-off through the window. `claimtrial` is deleted before it (block B1), leo's leftovers are moved aside (block B2), and the window's 8.2b is the new signer phase `backfill-done`, which takes the plan from `collecting` straight to `done`. There is no adoption, no 8.2c and no files seed (8.3 is gone). After step 9, block B4 turns the console's claims on and acceptance runs on a new app, `claimtrial2`. The code is platform-1e's branch `claim-empty-cohort-done`, not yet built, reviewed or released. Nothing here runs before Paul has decided D2 to D11 on [amadan #250](https://amadan.net/carlos/platform/-/discussion/250). The gates, trial assumptions and blocks are in "Window package (for Paul)" below. |
| +> |
| +> What follows is the 2026-10-19 pause note, kept for the record. |
| > |
| > The step 8 builder (branch `claim-step8-readiness`, pushed and PARKED, not merged) fixed S1 (hosts-backfill signer phase), S2 (census classes for leo's real tree), the journal-migrate umask, and a fifth fault (leo had no `/data/.hostclaims`). Running its witness against leo's real tree then found three design gaps: |
| > - **S3:** `backfill-adopt` needs a census report from every roster box, and sophie (the console box, which stays claims-off) can't produce one. |
| @@ -46,7 +50,7 @@ On Paul's go, given directly to the controller and specified by platform-1e: the |
| - The bucket is unchanged. The roster is sha256 `20ebe512…` (revision 9, etag `a69f6d93…`, 19:35:16Z). The floor `moves-protocol.json` is `1a6a7e50…` and `moves-protocol-auth.json` is `3bbd48a8…` (19:33:45Z). |
| - AccessDenied since 22:50Z: sophie 2, both `ce:GetCostAndUsage`; leo 0. |
| -**Resume.** Do this only on Paul's go, after the design note. Do it as 7.4, with the files put back first. From DOCS-T1, run `box_inspect "$SOPHIE"` (the box helpers) so `roster.json` and `rev` are current. Then on sophie, as root: |
| +**Resume.** Do this only on Paul's go, after the design note. Under Option B this is the first part of block B4, after step 9; the `console-report` it ends with is B4's timed one. Do it as 7.4, with the files put back first. From DOCS-T1, run `box_inspect "$SOPHIE"` (the box helpers) so `roster.json` and `rev` are current. Then on sophie, as root: |
| ```bash |
| set -euo pipefail |
| @@ -70,13 +74,82 @@ systemctl restart console.service |
| systemctl is-active console.service |
| ``` |
| -The `cmp` line checks that the drop-in is still the saved bytes. It was never removed, so nothing needs restoring there. If the drop-in differs, or any hash line fails, that is a STOP. `console-config` rewrites the same `=1` bytes and reloads systemd. Then run 7.4's checks. `claim-console.env` is `5df17bac`, 73 bytes, root `600`. The console is `active`, running `/opt/carlos/system/console/console` as `carlos` with `CARLOS_HOST_CLAIM=1`, and its journal shows the run sweep and seeds pass lines again. Wait for `status/consoles/dev-console.json` to move, run `box_inspect "$SOPHIE"`, then run `console-report`'s dry run and apply as in 7.4. Finish with `/`, `/healthz` (`f3bf1214`), wakecheck and claimtrial at 200, and edge and instance-control on both boxes unrestarted. |
| +The `cmp` line checks that the drop-in is still the saved bytes. It was never removed, so nothing needs restoring there. If the drop-in differs, or any hash line fails, that is a STOP. `console-config` rewrites the same `=1` bytes and reloads systemd. Then run 7.4's checks. `claim-console.env` is `5df17bac`, 73 bytes, root `600`. The console is `active`, running `/opt/carlos/system/console/console` as `carlos` with `CARLOS_HOST_CLAIM=1`, and its journal shows the run sweep and seeds pass lines again. Wait for `status/consoles/dev-console.json` to move, run `box_inspect "$SOPHIE"`, then run `console-report`'s dry run and apply as in 7.4. Finish with `/`, `/healthz` (the console's current build), and edge and instance-control on both boxes unrestarted. Under Option B, `claimtrial`'s and wakecheck's records are deleted in block B1, so neither is checked here. |
| + |
| +**Pause again (the pause recipe, as commands).** Block B4 uses this if `instances create` fails three times, and D9 uses it after acceptance. It puts back 6.3's off form, the exact 72 bytes `console-config` writes without the flag, and changes nothing else. On sophie, as root: |
| + |
| +```bash |
| +set -euo pipefail |
| +umask 077 |
| +[ "$(sha256sum /etc/carlos/claim-console.env | cut -c1-64)" = 5df17bac42a91a02d90584edc5476c045400af57e467b2ea0815ba50b7e31d03 ] |
| +printf 'CARLOS_CONSOLE_ID=dev-console\nCONSOLE_HOME_REGION=ie\nCARLOS_HOST_CLAIM=\n' > /etc/carlos/.claim-console.env.pause |
| +[ "$(sha256sum /etc/carlos/.claim-console.env.pause | cut -c1-64)" = c1ae2900db31295090d2f4f2f9b5f6cdcca07e74545a6b1a9da039c3abf5c412 ] |
| +mv -T /etc/carlos/.claim-console.env.pause /etc/carlos/claim-console.env |
| +systemctl restart console.service |
| +systemctl is-active console.service |
| +tr '\0' '\n' < "/proc/$(systemctl show -p MainPID --value console.service)/environ" | grep '^CARLOS_HOST_CLAIM=' |
| +``` |
| + |
| +Expected: `active`, then `CARLOS_HOST_CLAIM=` (empty). `/healthz` answers 200 within a few seconds, and the journal since the restart has no `off on a host-claim console` line. If the first hash line fails, the file is not the `=1` form `console-config` wrote: stop and report, change nothing. |
| Draft, first written 2026-10-16, revised 2026-10-17 against main at 26065378. Step 5.0a and step 6 were revised on 2026-10-18 against platform-1e's branch `claim-step6-artifacts` at `c38a4ca9`, and re-checked against main at `7c1beb2d` after it merged (published (v0.28.2, tag on 7c1beb2d)). Revised again on 2026-10-18 for the second signing session on v0.28.3, against main at `50f5b268` (`install-channel --policy`, from branch `claim-prev-rehearsal`) and the root-runner rehearsal of steps 6 and 7. Moved to v0.28.4 the same day, against main at `f3bf1214` (`install-channel --policy --identity`, from branch `claim-untagged-pointer`). It authorizes nothing. Every change below waits for Paul's go at its own checkpoint. Neither the approved copy quoted in step 8 nor the activation tools' tests authorize running anything. |
| ## Window package (for Paul): step 8, leo's attended offline window |
| +### Option B: window ID, gates and trial assumptions (delta v2, 2026-10-20) |
| + |
| +This part replaces the 2026-10-19 plan wherever they differ. Its authority is Fable's plan review (`after236/fable-plan-review.md`), sections 3B, 4, 5, 6 and 10, and platform-1e's rulings relayed by the coordinator on 2026-10-20. Names used below: |
| +- **Blocks B1, B2, B4 and B5** are this delta's operator blocks. They are not section 5's rollback cases, which are always written "case B1" and "case B2". |
| +- **Gates G0 to G11** are this checklist's. Section 3's older gates are written "3.G1" to "3.G7". |
| + |
| +**The window ID.** One variable, set at Paul's go. Every other window value derives from it: the window directory, the backfill ID and the deadline watcher's times (section 1). Paul has not fixed the time. |
| + |
| +```bash |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go, for example 20261021T0900Z; T0 is this time, in UTC |
| +WINDOW="/etc/carlos/claim-window-cohort1-$WINDOW_ID" |
| +BACKFILL_ID="cohort1-${WINDOW_ID%%T*}" |
| +``` |
| + |
| +`BACKFILL_ID` is one lower-case path segment, `cohort1-` and the window's date. If the window moves, both move with it, and a window that has started needs a new ID (8.1's `mkdir` refuses an existing `$WINDOW`). |
| + |
| +**The order, in the morning, each step behind its own gate:** copy approval and merge (G1); tag and publish v0.28.5; block B5 (G2); signing session 6 opened, with the P-B2 grants and `publish-identity` for v0.28.5 (G3); leo rolled to v0.28.5 (G4); blocks B1 and B2 (G5); the window: 8.1, 8.1b, 8.2, 8.2b, 8.4 (G6 to G8), then the signing session is closed; step 9 (G9); block B4 (G10); acceptance 1 to 4 on `claimtrial2` (G11); Paul's gos for 10.5 and 10.6. |
| + |
| +#### Gates G0 to G11 (the operator ticks each) |
| + |
| +Each failure action is the whole action. Nothing is retried in place unless the gate says so. |
| + |
| +- [ ] **G0 (night): branch built.** Pass: platform-1e's `claim-empty-cohort-done` is built, its tests are green, the root-runner witness passes and Astra has nothing above P3. Then continue to the morning package. On failure: deliver Fable's review and this delta, make no release; Paul decides between finishing Option B and holding. |
| +- [ ] **G1: copy, merge, CI.** Pass: Paul approves the strings (D7), the branch is merged, and CI is green on the squashed tip. Then tag and publish v0.28.5, dev only (D3). On failure: hold. No release, and the window does not open. |
| +- [ ] **G2: prechecks (block B5).** Pass: `apps/bab//` has no current object; no backfill plan; no claims; no `admission.json`; the roster's `regions.ie.pools` contains `cohort1`; sophie and dennis have no `cohort1` row and no `CARLOS_POOLS`; the three stopped instances carry no box role; the dev console is claims-off. Then open the signing session. On failure: fix the one item if it is the known cleanup (with Paul's go), otherwise hold. |
| + - **Known blocker, found 2026-10-20 and resolved by block B1: "G2 FAIL: records without acq".** Tonight's read-only run (`after236/precheck-20261020.md`, check 2b) found three instance records, `claimtrial`, `wakecheck` and `rollcheck-0261`, none with an `acq` field. `backfill-collect` refuses a record without one (`claimactivation/backfill.go:95`). platform-1e ruled a zero-host plan: block B1 deletes all three records (and the offline-move rehearsal app's), so the plan holds no hosts and no record needs an `acq`. Fable's "two records with acquisitions" is replaced by "no instance records" (B5's re-run after B1). |
| +- [ ] **G3: signing session.** Pass: the signer role has the P-B2 grants (each phase's dry run passes), and `publish-identity` for v0.28.5 wrote its identity create-only. Then roll leo. On failure: tear down, run the CloudTrail check, hold. |
| +- [ ] **G4: leo on v0.28.5.** Pass: `carlos system update` completes, the guard is active, `carlos.prev` is v0.28.4, and leo's box report shows v0.28.5 and verifies (`status`). Then block B1. On failure: leo is claims-off, so this is the ordinary failed-roll path: fix forward only (H11). Hold the window. |
| +- [ ] **G5: blocks B1 and B2.** Pass: no instance record in the bucket; leo's registry has zero rows; sophie's has no `wakecheck` row; nothing left under `/data/.instances`, `/data/instances`, `/data/programs` or `/opt/carlos/live` for any B2 host, on any box. Then the window. On failure: hold. Nothing claim-mode has run. |
| +- [ ] **G6: 8.1 and 8.1b.** Pass: 8.1's masks and `install-control --dry-run`; 8.1b's `backfill-collect`: zero hosts (`hosts` `{}`), phase `collecting`, one create-only write. Then 8.2. On failure: case B1 undo (ruling R1): restore units, enablement and barrier. The plan, if written, stays, and is harmless with the console off. |
| +- [ ] **G7: 8.2.** Pass: `host.env`, `install-control`, `journal-migrate --window`, the mode checks, instance-control active, and leo's census report lists zero installations. Then 8.2b. On failure: stop and hold from the `host.env` edit (R1). The console stays off, so dev carries on. |
| +- [ ] **G8: 8.2b `backfill-done`.** Pass: phase `done`, one If-Match write, and sophie (the roster box with no census key) listed as unreported. Then 8.4. On failure: hold. The plan stays `collecting`; with the console off nothing is frozen. |
| +- [ ] **G9: 8.4, the go for 9, step 9.** Pass: 8.4's restore; Paul's go for step 9; the edge active through `guard-exec`, one `REFUSING CARLOS_SWAP=overlap (host-claims-on)`, and a fresh verified box report with `host_claim_enabled`. Then block B4. On failure: stop and hold ("Stop and hold", "Steps 9 and 10"). |
| +- [ ] **G10: block B4.** Pass: the console's claims on (the "Paused state" resume block), `status/consoles/dev-console.json` moved, `console-report` applied; then `apps create claimtrial2`, `ship wc4`, `promote`, `instances enable --pool cohort1`, and `instances create` succeeding within one agent tick of `console-report`, in at most three attempts. Then acceptance. On failure: a refusal is fail-closed; capture the door's log line. After a third failed attempt, switch the console off again ("Paused state", the pause recipe) and hold leo. |
| +- [ ] **G11: acceptance 1 to 4** on `claimtrial2` (launch, a confirmed stop, two cold generations with rising `hits=`, isolation). Then Paul's gos for 10.5 and 10.6. On failure: stop and hold; post the evidence on #250. |
| + |
| +#### Trial assumptions (Option B): PROPOSED, for Paul to confirm in the morning (D5) |
| + |
| +Option B gives these up on purpose. Fable recommends accepting all four. None is accepted until Paul says so. |
| + |
| +| # | The trial accepts | Instead of | Status | |
| +|---|---|---|---| |
| +| 1 | Other boxes are shown to be outside the cohort, and to stay so, by a read-only precheck (block B5) and a no-touch rule on sophie and dennis for claim state. Their only changes are ordinary ones under D2 and D10: B1's record deletes retire their rows on the next pass (sophie's stale `wakecheck` row, one of its four routes, all pool-less, goes this way), B2 moves leftovers aside, and dennis takes the `stable` roll. | Machine-verified exclusion of the other boxes | PROPOSED | |
| +| 2 | #312 (the console's evidence lasts one agent tick) is met by ordering and at most three attempts (block B4) | The #312 fix | PROPOSED | |
| +| 3 | While the dev console is in claim mode, pool-less creates on dev are refused (409) | Pool-less creates working throughout | PROPOSED | |
| +| 4 | `claimtrial2` cannot be deleted until sophie's cohort | A deletable acceptance app | PROPOSED | |
| + |
| +Two notes for D2, from the coordinator (2026-10-20): |
| +- **wakecheck** is the standing dev wake check. B1 deletes its record. Recreate it after the trial, once the dev console is claims-off again (D9), with the same name and build `wc4`, and update the dev wake check recipe. Until then nothing on dev answers the wake check. |
| +- **rollcheck-0261** is a static roll-check app in Paul's own account `bab`, created on 2026-10-13 for the v0.26.1 roll. Losing its record needs Paul's OK at D2. |
| + |
| +### The 2026-10-19 package |
| + |
| Prepared 2026-10-19 for platform-1e, who takes it to Paul. Every check below was read-only, from DOCS-T1 (profile `kass`, `user/remote-agent`, account `640012953448`) and over SSM on leo, between 20:30Z and 20:46Z. Nothing was changed on any box, bucket or security group, and nothing was created. Code was read at `f3bf1214` (v0.28.4, what leo runs). Main has one newer commit, `96db0efe` (#317, `guard-arm` directory modes), which step 8 does not run. |
| **Verdict: not ready for a go.** Two findings in the code would stop the window at 8.2 ("STOPs before the go", below). Each needs platform-1e's answer before Paul is asked. The barrier, the deadline, the gates and the umask changes are ready. |
| @@ -85,6 +158,8 @@ Prepared 2026-10-19 for platform-1e, who takes it to Paul. Every check below was |
| ### STOPs before the go |
| +Under Option B: S1 is met by `backfill-collect` (8.1b) and `backfill-done` (8.2b), with a zero-host plan. S2 is met by v0.28.5's census classes and `journal-migrate --window` (platform-1e's `claim-step8-readiness`, carried into `claim-empty-cohort-done`) and by block B2, which moves the hosts' leftovers off leo. Both are checked again against v0.28.5 before the go. |
| + |
| **S1. `journal-migrate` needs a signed backfill plan that nothing ships to write.** `carlos system journal-migrate` always runs the census path (`RunSystemJournalMigrate` passes `census=true`, `internal/host/claimjournal_migration.go:61`). Its census collector starts by reading `control/hosts-backfill.json` (`hostops.BackfillKey`) and refuses unless that object is root-signed with the release key, at `phase` `collecting`, with a `hosts` map that freezes every instance record in the deployment by key and etag, and `directory_revision` equal to the roster's (`rootMigrationCollector.collect`, `internal/host/rootmigration_census.go:37-75`). On dev the key is 404 (read 20:40Z). At `f3bf1214` nothing outside tests writes a `collecting` plan: the only non-test uses of `"collecting"` are the two readers, the root tests sign the plan with a fixture key (`rootmigration_bootstrap_root_linux_test.go:165-171`), and `claims run` has no such phase (its phases are `initialize`, `configure`, `home`, `invite-box`, `enroll-box`, `invite-console`, `enroll-console`, `bind-key`, `publish-floor`, `sign-policy` and `publish-identity`). The next stage is missing too: `Backfill.Publish`, which moves the plan to `adopting`, has no production caller (its comment names "the #232 coordinator"). The console adopts claims only while the plan reads `adopting` (`ReconcileClaims`, `internal/hostops/backfill_controller.go`), and `control/hosts/` on dev is empty. So as tooled, 8.2's `journal-migrate` fails in its first observation, before it writes `migration.json`. And even with the plan in place, nothing would give `claimtrial` the claim the seed's cold admission needs. The plan would have to freeze all three records in the bucket: `claimtrial`, `rollcheck-0261` and `wakecheck`. platform-1e: is a backfill writer and adopter owed (a `claims run` phase is a signing session, with the release key on a host again), or does the trial reach the census another way? |
| **S2. The migration census walks the box's real trees, and leo's would not pass as written.** In its `ready` step, `journal-migrate` calls `hostbox.CheckMigrationRoots` over `/data`, `/opt/carlos` and `/etc/systemd/system` (`rootmigration_production_linux.go:156`). Any non-directory that is not in an installation's inventory and not in a shared rule is `ErrTypedInventory` (`internal/hostbox/migration_census_linux.go`; the rules are `claimGlobalSharedPaths`, `internal/host/claimcensus.go`). By that reading, three things on leo fail: |
| @@ -98,43 +173,48 @@ A refusal there leaves `migration.json` at phase `recovered`, an unfinished migr |
| ### 1. Window ID and deadline |
| -Proposed, for the first weekday after S1 and S2 are cleared and Paul's go is given, and no earlier than **2026-10-21**: |
| +Option B (2026-10-20): the window time is `WINDOW_ID`, set at Paul's go (top of this package). Every time below is relative to T0, the time `WINDOW_ID` names, which falls after v0.28.5 is on leo and blocks B1, B2 and B5 have passed. There is no files seed, so the 2026-10-19 plan's seed cut-off and its gate 3.G1 (claimtrial drained and replicated) are gone: leo has no routes once block B1 has run. |
| -| | Time (UTC) | What | |
| +| | Time | What | |
| |---|---|---| |
| -| Window ID | `20261021T0900Z` | `WINDOW='/etc/carlos/claim-window-cohort1-20261021T0900Z'`. If the date moves, the ID moves with it | |
| -| T−45 | 08:15Z | Second root session opened and the deadline watcher started (below); the gates re-run read-only (section 3); a fresh check that `sg-054aa165db842b6ba` is still on leo's ENI alone | |
| -| T−30 | 08:30Z | Barrier on (section 2), checked. From now on claimtrial gets no requests, so it hibernates within about 16 minutes (idle 15m plus the activator's tick) | |
| -| T−5 | 08:55Z | Gate G1: claimtrial hibernated and replicated. If it is not, wait. If it still is not by 09:15Z, abort before 8.1: lift the barrier, nothing else | |
| -| T0 | 09:00Z | 8.1. Budget 15 minutes | |
| -| | 09:15Z | 8.2. Budget 20 minutes | |
| -| Seed cut-off | 10:05Z | The last time 8.3 may start. 8.3 (override, seed and capture) has a 30-minute budget and 8.4 a 20-minute one. The data is one 24 KB database with about 2 KB of replica: no rate measured on leo exists, and these budgets are many times what that needs | |
| -| Soft end | 10:30Z | No new block starts except 8.4 or a hold | |
| -| **Hard end** | **11:00Z** | The deadline. Two hours from T0 | |
| +| Window ID | set at go | `WINDOW_ID`, and from it `WINDOW` and `BACKFILL_ID` (top of this package). If the time moves, all three move | |
| +| T−45 | | Second root session opened and the deadline watcher started (below); block B5 and block B2's final checks re-run read-only; a fresh check that `sg-054aa165db842b6ba` is still on leo's ENI alone | |
| +| T−30 | | Barrier on (section 2), checked | |
| +| T−5 | | Gate G5 re-read: leo's registry has zero rows and no `carlos-instance-gen@` or `carlos-instance@` unit is listed. If not, abort before 8.1: lift the barrier, nothing else | |
| +| T0 | | 8.1, then 8.1b on the signer. Budget 20 minutes | |
| +| | T0+20m | 8.2. Budget 20 minutes | |
| +| | T0+40m | 8.2b on the signer, then 8.4. Budget 20 minutes | |
| +| Soft end | T0+90m | No new block starts except 8.4 or a hold | |
| +| **Hard end** | **T0+120m** | The deadline. Two hours from T0 | |
| **Who enforces it.** A second root session on leo, held by someone other than the operator running the 8.x blocks. Proposed: platform-1e, with the controller as stand-in; Paul names the holder at the go. It is a separate `aws --profile kass --region eu-west-1 ssm start-session --target i-046c7fe43620944bc`, then `sudo -i`, opened at T−45 and kept until leo is masked and held, or step 9's go. The holder runs this watcher at T−45 (`date -d` is GNU date, which leo has): |
| ```bash |
| -WINDOW='/etc/carlos/claim-window-cohort1-20261021T0900Z' |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go: the same value as the window package's |
| +[[ $WINDOW_ID =~ ^[0-9]{8}T[0-9]{4}Z$ ]] |
| +WINDOW="/etc/carlos/claim-window-cohort1-$WINDOW_ID" |
| SEED=/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| umask 077 |
| -until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T10:05:00Z +%s)" ]; do sleep 15; done; echo "SEED CUT-OFF $(date -u +%T)" |
| -until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T10:30:00Z +%s)" ]; do sleep 15; done; echo "SOFT END $(date -u +%T)" |
| -until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T11:00:00Z +%s)" ]; do sleep 15; done; echo "DEADLINE $(date -u +%T)" |
| +T0=$(date -u -d "${WINDOW_ID:0:4}-${WINDOW_ID:4:2}-${WINDOW_ID:6:2} ${WINDOW_ID:9:2}:${WINDOW_ID:11:2}:00Z" +%s) |
| +date -u -d "@$T0" |
| +until [ "$(date -u +%s)" -ge $((T0 + 5400)) ]; do sleep 15; done; echo "SOFT END $(date -u +%T)" |
| +until [ "$(date -u +%s)" -ge $((T0 + 7200)) ]; do sleep 15; done; echo "DEADLINE $(date -u +%T)" |
| if [ -e "$SEED" ]; then |
| systemctl show carlos-edge.service -p InvocationID --value > "$WINDOW/deadline-invocation" |
| systemctl stop carlos-edge.service |
| systemctl show carlos-edge.service -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/deadline-result" |
| journalctl "_SYSTEMD_INVOCATION_ID=$(cat "$WINDOW/deadline-invocation")" --no-pager > "$WINDOW/deadline-journal" |
| fi |
| -pgrep -a -f '^/opt/carlos/carlos system journal-migrate$' || echo "no journal-migrate" |
| +pgrep -a -f '^/opt/carlos/carlos system journal-migrate( |$)' || echo "no journal-migrate" |
| systemctl show carlos-edge.service carlos-instance-control.service -p Id -p LoadState -p ActiveState -p SubState |
| ``` |
| +The watcher's first test checks the ID's shape; a wrong one ends the shell before any wait starts, so start it again with the right value. `date -u -d "@$T0"` prints T0 for the holder to compare with the go. The `pgrep` pattern allows arguments, because 8.2 now runs `journal-migrate --window "$WINDOW"`. Under Option B nothing creates the seed override, so the `$SEED` branch only acts if one is ever found. |
| + |
| **At the deadline, exactly this happens:** |
| -1. If the seed override is in place, the watcher captures the invocation ID first, then stops the one-shot and saves its result and journal under `$WINDOW` (`[runbook.cancellation]`). Otherwise it stops nothing. |
| +1. If a seed override is in place (under Option B none should be), the watcher captures the invocation ID first, then stops the one-shot and saves its result and journal under `$WINDOW` (`[runbook.cancellation]`). Otherwise it stops nothing. |
| 2. If `journal-migrate` is still running, the primary operator interrupts it (Ctrl-C, its documented cancel: SIGTERM or interrupt cancel the import). If the primary session is gone, the holder sends `kill -TERM <pid>` to the PID the watcher printed. Never `pkill -f`. |
| -3. The primary runs 8.4's first block (remask the edge, remove the override, put the guard drop-in back), or, if 8.3 had not started, nothing. If the primary session is gone, the holder runs that block, with `INVOCATION=$(cat "$WINDOW/deadline-invocation")` and the same `WINDOW`, `SEED` and `GUARD` values. |
| +3. The edge has stayed masked since 8.1 (there is no 8.3). If a seed override was found in step 1, the primary follows section 5's case C (capture and report first); otherwise nothing. If the primary session is gone, the holder runs it, with `INVOCATION=$(cat "$WINDOW/deadline-invocation")` and the same `WINDOW`, `SEED` and `GUARD` values. |
| 4. Leo is then in "Stop and hold" with the edge persistently masked: step 9's hold state, which survives a reboot. The barrier stays on (section 5). Evidence is captured and posted on #250. No block starts after the deadline. Anything further needs a new go. The exception is a window that never reached 8.2's `host.env` edit: that one is undone (section 5, case B1, ruling R1), not held. |
| ### 2. External traffic barrier |
| @@ -184,12 +264,14 @@ Expected: the two inbound rules back, with no description (new rule IDs), plus t |
| **SSM keeps working.** The SSM agent connects outwards on TCP 443 to the regional SSM endpoints (there are no VPC endpoints in this VPC). The egress rule is untouched, the subnet's network ACL allows all traffic both ways, and with no inbound rule left, replies to outbound connections are still admitted, because security groups are stateful. The same goes for S3, KMS and STS from the box, and for the instance metadata service, which is link-local. Agent `3.3.4851.0` was `Online` at 20:30:34Z. |
| -**The one way the barrier can disappear.** The `region-host` module declares these ingress rules inline, so any `tofu apply` in `platform-infrastructure/dev` that touches `module.host_ie_b.aws_security_group.host`, including a whole-root apply, puts them back. Rule: no dev apply that includes `module.host_ie_b` while the barrier is on. Dev takes targeted plans only for the trial already. The primary re-runs the `describe-security-group-rules` line before 8.3 and before step 9. |
| +**The one way the barrier can disappear.** The `region-host` module declares these ingress rules inline, so any `tofu apply` in `platform-infrastructure/dev` that touches `module.host_ie_b.aws_security_group.host`, including a whole-root apply, puts them back. Rule: no dev apply that includes `module.host_ie_b` while the barrier is on. Dev takes targeted plans only for the trial already. The primary re-runs the `describe-security-group-rules` line before 8.2 and before step 9. |
| **Alternatives, not proposed.** A dedicated barrier group swapped onto leo's ENI does the same, but creates a group and drifts `vpc_security_group_ids` instead. It would matter only if leo's group were shared, and it is not. Disassociating the Elastic IP would leave the VPC path open, and would change leo's public address. |
| ### 3. Pre-window gates (read-only evidence, 2026-10-19) |
| +These are the 2026-10-19 gates, read as "3.G1" to "3.G7" to keep them apart from the Option B checklist at the top of this package. Under Option B: 3.G1 is replaced by blocks B1 and B2 and checklist gate G5 (no `claimtrial` to drain); 3.G2's leftovers are moved by block B2; 3.G4's bucket item is checklist gate G2 (block B5); 3.G7's census STOP is S1 and S2, met as "STOPs before the go" says. 3.G3, 3.G5 and 3.G6 are re-read at T−45. |
| + |
| | Gate | Status | Evidence | |
| |---|---|---| |
| | G1 `claimtrial`: a drained, replicated stop | PASS today; re-run at T−5 | See below | |
| @@ -257,7 +339,7 @@ Today: |
| ### 4. Umask audit |
| -The 8.x blocks run in one attended root shell that 8.1 starts with `umask 077`. Platform binaries those blocks run straight from that shell: `carlos system claims install-control --dry-run` (8.1, 8.2, 8.3, 8.4), `carlos system claims install-control` (8.2) and `carlos system journal-migrate` (8.2). The files seed is not one: it runs inside `carlos-edge.service` as `carlos`, under the unit's umask. No unit sets `UMask=` (`git grep UMask` at `f3bf1214` finds only `internal/edge/purge_listen_unix.go`), so that is systemd's `0022`. It reaches root only over the control socket. |
| +The 8.x blocks run in one attended root shell that 8.1 starts with `umask 077`. Platform binaries those blocks run straight from that shell: `carlos system claims install-control --dry-run` (8.1, 8.2, 8.4), `carlos system claims install-control` (8.2) and `carlos system journal-migrate --window "$WINDOW"` (8.2). (Under Option B there is no 8.3, and 8.1b and 8.2b run on the signer, not on leo.) The files seed is not one: it runs inside `carlos-edge.service` as `carlos`, under the unit's umask. No unit sets `UMask=` (`git grep UMask` at `f3bf1214` finds only `internal/edge/purge_listen_unix.go`), so that is systemd's `0022`. It reaches root only over the control socket. |
| What each one creates (`f3bf1214`): |
| - **`install-control --dry-run`** creates nothing. It returns before the lock and before any directory (`internal/host/activation_install_linux.go:260-262`). Its `verifyOfflineRootMigration` only reads `systemctl show` and `/proc`. |
| @@ -266,19 +348,19 @@ What each one creates (`f3bf1214`): |
| - **Written by the shell itself, not a binary:** `$WINDOW` and its copies (`0700`/`0600` on purpose, root-only), and the seed override and its directory under `/run/systemd/system/carlos-edge.service.d` (`0600`/`0700`, read only by systemd as root). Both are safe as they are. |
| Changes now written into the blocks: |
| -- **8.1** defines `pub() ( umask 022; "$@" )` straight after `umask 077`, and every platform binary call in 8.1 to 8.4 goes through it: the four `install-control --dry-run` lines and `install-control` in 8.2. |
| -- **8.2**'s `journal-migrate` subshell starts with `umask 022`. It is followed by a mode check in the style of the guard-directory check: `stat` and `test` for `/var/lib/carlos-claims` `700 root:root`; for `/opt/carlos/system`, `guard`, `guard/moves` and `guard/seeds`, `755 root:root` plus `sudo -u carlos test -r` and `test -x`; `guard/carlos` executable by `carlos`; and `/var/lib/carlos-swap-status` `755 root:root` if it exists. Any mismatch is a STOP: no chmod by hand, no rerun, hold. |
| -- **8.3** notes why its override needs no change. |
| +- **8.1** defines `pub() ( umask 022; "$@" )` straight after `umask 077`, and every platform binary call in 8.1 to 8.4 goes through it: the `install-control --dry-run` lines and `install-control` in 8.2. |
| +- **8.2**'s `journal-migrate` subshell starts with `umask 022`. It is followed by a mode check in the style of the guard-directory check: `stat` and `test` for `/var/lib/carlos-claims` `700 root:root`; for `/opt/carlos/system`, `guard`, `guard/moves` and `guard/seeds`, `755 root:root` plus `sudo -u carlos test -r` and `test -x`; `guard/carlos` executable by `carlos`; and `/var/lib/carlos-swap-status` `755 root:root` if it exists. Any mismatch is a STOP: no chmod by hand, no rerun, hold. platform-1e's R3 (`claim-step8-readiness`) makes `journal-migrate` create the two directories `0755` whatever the umask; the subshell and the mode check stay, as a second check. |
| +- Under Option B no seed override is written (8.3 is gone). Block B2's aside directory is written by the shell under `umask 077`, root `0700`, on purpose. |
| `umask 022` loosens nothing private here: every root-only path above sets its mode explicitly. The one exception is `journals.db`, which becomes `0644` inside the `0700` `/var/lib/carlos-claims`, so it stays unreachable. Aside, not umask: `journal-migrate` resets `/data/.hostexec` from `guard-arm`'s `2770` to `0700 carlos:carlos`. Both work for the agent; the two pieces of code disagree. |
| ### 5. Rollback: undo before 8.2's `host.env` edit, stop and hold from it, as commands |
| -Paul's ruling R1 (2026-10-19) splits the window in two. A stop before 8.2's `host.env` edit is fully undone (case B1): units and edge restored, enablement restored, barrier lifted. From that edit onwards a stop is step 9's stop-and-hold path ("Stop and hold"), cases B2 to E. Stopping is always allowed. Run it in the primary root session on leo, which keeps `WINDOW`, `SEED`, `GUARD`, `INVOCATION`, `UNITS` and `TRIGGERS` from 8.1 and 8.3. If that session is gone, set them again in the second session from the window ID and 8.1 and 8.3's literal values, with `set -eu`, `umask 077` and `pub() ( umask 022; "$@" )` first. |
| +Paul's ruling R1 (2026-10-19) splits the window in two. A stop before 8.2's `host.env` edit is fully undone (case B1): units and edge restored, enablement restored, barrier lifted. From that edit onwards a stop is step 9's stop-and-hold path ("Stop and hold"), cases B2 to E. Stopping is always allowed. Run it in the primary root session on leo, which keeps `WINDOW`, `SEED`, `GUARD`, `UNITS` and `TRIGGERS` from 8.1 (`INVOCATION` is set only if a seed override is ever found, case C). If that session is gone, set them again in the second session from `WINDOW_ID` and 8.1's literal values, with `set -eu`, `umask 077` and `pub() ( umask 022; "$@" )` first. |
| **A. Barrier on, 8.1 not started (abort).** Leo is unchanged. Lift the barrier (section 2, "Reverse"). Done. |
| -**B1. From 8.1's first `systemctl stop` until 8.2 changes `host.env`: undo (R1).** That is all of 8.1, including a mask loop that stopped part-way and 8.1's closing `install-control --dry-run`, which creates nothing, up to the moment 8.2's `host.env` edit begins. 8.2's own dry run comes after that edit, so it falls in B2. Claims are still off and nothing claim-mode has run. |
| +**B1. From 8.1's first `systemctl stop` until 8.2 changes `host.env`: undo (R1).** That is all of 8.1, including a mask loop that stopped part-way and 8.1's closing `install-control --dry-run`, which creates nothing, and 8.1b on the signer, up to the moment 8.2's `host.env` edit begins. 8.2's own dry run comes after that edit, so it falls in B2. Claims are still off and nothing claim-mode has run. A collecting plan 8.1b wrote stays where it is: with the dev console claims-off nothing reads it (gate G6), and a later window needs a new `WINDOW_ID`, so a new `BACKFILL_ID`. |
| First prove that 8.2's edit has not happened and that nothing claim-mode has written anything. If either check fails, this is B2, not B1: hold. |
| @@ -350,7 +432,7 @@ Leave `$WINDOW` in place as the window's record. `held/` is empty now, and nothi |
| If any `cmp` or `test` in B1 fails, stop there. Move nothing more, start nothing, and report on #250 with the evidence. A unit already moved back stays back, and one still held stays masked. |
| -**B2. From 8.2's `host.env` edit until the seed override exists (8.2): hold.** |
| +**B2. From 8.2's `host.env` edit until 8.4 (8.2 and 8.2b): hold.** |
| - If the mask loop somehow had not finished (it must have, before 8.2 starts), finish that loop first. The triggers are still enabled, so without their masks a reboot would start them: |
| ```bash |
| @@ -365,8 +447,9 @@ If any `cmp` or `test` in B1 fails, stop there. Move nothing more, start nothing |
| ``` |
| - Then leave every mask, `$WINDOW`, `host.env` (edited or not) and `carlos-instance-control.service` as they are. Do not put `host.env.before` back by hand: once 8.2 has touched `host.env`, R1's undo no longer applies. |
| +- A refused 8.2b leaves the plan `collecting` on the signer side. Leave it: with the dev console claims-off, nothing is frozen (gate G8). Close the signing session. |
| -**C. The seed override is in place (8.3).** This is the fragile state (H6): |
| +**C. A seed override is in place.** Under Option B nothing writes one: there is no 8.3, and the edge stays masked from 8.1 until step 9, with its guard drop-in never moved. If `$SEED` is ever found, stop: start nothing, capture `systemctl cat carlos-edge.service` and `ls -la /etc/systemd/system/carlos-edge.service.d /run/systemd/system/carlos-edge.service.d "$WINDOW/held"`, and report on #250. The commands below are the 2026-10-19 plan's, for the state 8.3 left (the edge unit file back in `/etc`, its guard drop-in in `$WINDOW/held`). Run them only if that is what the capture shows, and only with platform-1e's say. This is the fragile state (H6): |
| ```bash |
| INVOCATION=${INVOCATION:-$(systemctl show carlos-edge.service -p InvocationID --value)} |
| @@ -389,11 +472,11 @@ systemctl daemon-reload |
| pub /opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| -That is 8.4's first block, with the invocation captured before the stop. Do not run 8.4's second block: services and triggers stay masked. If a `cmp` fails, the file changed after the window: stop, move nothing, report. If leo rebooted while the override was in place, the override is gone and the edge unit is disabled and unguarded. Do not start anything: check `systemctl cat carlos-edge.service` and `ls -la /etc/systemd/system/carlos-edge.service.d "$WINDOW/held"`, then restore the edge mask and the guard drop-in as above (minus the stop and the override removal), and report. |
| +That is the 2026-10-19 plan's 8.4 first block, with the invocation captured before the stop. Do not run 8.4's second block: services and triggers stay masked. If a `cmp` fails, the file changed after the window: stop, move nothing, report. If leo rebooted while the override was in place, the override is gone and the edge unit is disabled and unguarded. Do not start anything: check `systemctl cat carlos-edge.service` and `ls -la /etc/systemd/system/carlos-edge.service.d "$WINDOW/held"`, then restore the edge mask and the guard drop-in as above (minus the stop and the override removal), and report. |
| **D. After 8.4 (edge masked, waiting for step 9).** Nothing to stop. |
| -**E. Steps 9 and 10.** As "Stop and hold" says: `systemctl stop carlos-edge.service`, stop the five triggers, then `cmp`, move the edge unit to `$WINDOW/held`, link `/dev/null`, `daemon-reload` and check `LoadState=masked`. |
| +**E. Step 9, block B4 and step 10.** If block B4 failed, first switch the console off again ("Paused state", the pause recipe). Then, as "Stop and hold" says: `systemctl stop carlos-edge.service`, stop the five triggers, then `cmp`, move the edge unit to `$WINDOW/held`, link `/dev/null`, `daemon-reload` and check `LoadState=masked`. |
| **Restoring the edge.** B1 restores the edge and its enablement itself, above. In every hold case (B2 to E) the hold leaves the edge persistently masked: a `/dev/null` link at `/etc/systemd/system/carlos-edge.service`, the original in `$WINDOW/held`, its guard drop-in back at `90-host-floor.conf`, and the seed override gone. That state survives a reboot. Bringing the edge back to serving is step 9's block, and it needs its own go. |
| @@ -413,12 +496,13 @@ cat /proc/sys/kernel/random/boot_id |
| - `sg-054aa165db842b6ba` changes only through section 2's two commands; |
| - no dev `tofu apply` that includes `module.host_ie_b`; |
| - leave the Elastic IP association, sophie's group `sg-088e3aa23d6f2feba` and sophie herself alone; |
| -- do not delete the window's deadline files (`$WINDOW/deadline-*`, `hold-invocation`). |
| +- do not delete the window's deadline files (`$WINDOW/deadline-*`, `hold-invocation`); |
| +- leave block B2's aside directories (`/var/lib/carlos-aside-*`) on every box as they are. |
| ### Notes found on the way (not blockers) |
| - **Leo's swap is zram.** `swapon --show` lists `/dev/zram0`, compressed RAM, not a disk device. "Owed before production cohorts" says leo has swap and that a tmpfs page could reach disk. With zram it stays in memory, so that note may be overstated for leo. It still holds for any box with disk swap. |
| -- **Drop-ins the masks also hide.** `carlos-system-update.service` and `carlos-system-recover.service` carry `guard-arm`'s `90-host-floor.conf` drop-ins as well as the edge. 8.1's masks hide them, and 8.4 brings them back with their units. Only the edge's is moved, as 8.3 says. |
| +- **Drop-ins the masks also hide.** `carlos-system-update.service` and `carlos-system-recover.service` carry `guard-arm`'s `90-host-floor.conf` drop-ins as well as the edge. 8.1's masks hide them, and 8.4 brings them back with their units. Under Option B none of them is moved (the 2026-10-19 plan's 8.3 moved only the edge's). |
| - **Leo has other ACME certificates.** Leo holds `acme/` certificates for `platform.dev`, `proof*`, `tunnel`, `website` and `helloworld`. They are harmless, but they are part of S2's `/data` list. |
| ## Go package (for Paul): the v0.28.4 session |
| @@ -710,7 +794,7 @@ Re-run P9 right before step 6 (step 6.3), inside the signing session and before |
| ## Blockers |
| 1. **Release.** Resolved: v0.28.0, cut from 26065378 (Plan 14 Parts 2 and 3, the activation tools, e8e961dd), is fully rolled on dev (2026-10-18). Step 1. Two dev-only releases follow, each rolled to dev with Paul's go: v0.28.1 (the box-record verb), rolled 2026-10-18 and used by step 5.0 (done), and v0.28.2 (gap 1 and 2 fixes), which everything past 5.0 waits for. The v0.28.2 dev roll moves the boxes only: leave out step 1's console item, because step 5.0a ships and promotes the dev console's v0.28.2 build after pointer digests are on. v0.28.2 was published on 2026-10-18 (tag on `7c1beb2d`) and rolled to leo, sophie and dennis (current, 0 refusing; each reads `carlos v0.28.2`, `.prev` v0.28.1). The dev console is held at `3b69c2bd` on purpose until 5.0a. |
| -2. **Empty app segment.** 267 objects under `apps/bab//` (P5). Source: leo's app-less `proof*` routes, removed (2026-10-16). Cleanup waits for the e8e961dd release to reach leo and for Paul's approval. Step 2. Still open on 2026-10-18: 267 objects under `apps/bab//`. |
| +2. **Empty app segment.** 267 objects under `apps/bab//` (P5). Source: leo's app-less `proof*` routes, removed (2026-10-16). Cleanup waits for the e8e961dd release to reach leo and for Paul's approval. Step 2. Still open on 2026-10-18: 267 objects under `apps/bab//`. Read 2026-10-20 (block B5, `after236/precheck-20261020.md` check 1): `0` current objects and no `PRE /`; 1044 noncurrent versions under 1044 delete markers, the newest marker 2026-10-18T09:48:17Z, which is the plain `--apply` result step 2 predicts. `hostops.Apps` lists current objects only, so this reads as cleared; D11 is moot unless B5 finds otherwise. |
| 3. **IAM.** Leo's object-version actions were applied on 2026-10-18 (targeted, `0 to add, 1 to change, 0 to destroy`): the claim's data store lists, reads and deletes object versions in `carlos-dev-ie-640012953448` on the seed and cold-wake paths. The fence-writer role (`carlos-dev-ie-b-fence-writer`, bootstrap user `carlos-dev-ie-b-fence-writer-root`, no access key yet) and the deployment bucket policy were applied on 2026-10-18 from the saved targeted plan (`5 added, 0 changed, 0 destroyed`) and merged as platform-infrastructure#46 (`beb346b`). The bucket policy names that role as the only fence writer. After the apply, wakecheck still served on sophie (200) and leo answered `waking elsewhere` (sophie held the lease), with no AccessDenied in either box's journals. Step 3. |
| 4. **Box identity records.** Resolved 2026-10-18 (step 5.0 done: `control/boxes/leo-climbing.json` and `control/boxes/sophie-knitting.json`). Step 5's enrollment needs each box's signed AWS identity record at `control/boxes/<box>.json`. None exists, and none has ever been written: the publisher (`internal/host/boxrecord.go`, `boxRecordLoop`) runs only with `CARLOS_REGION_HA` on, which is off on every dev box. Leo's and sophie's records must be published before step 5. Paul ruled a reviewed one-off publish, with region HA staying off. |
| @@ -2417,19 +2501,22 @@ State now: leo holds the only copy of the fence-writer credentials, root `600` a |
| ## Step 8. Leo's attended offline window |
| -What: Plan 9 milestone 4 on leo, with the files seed inside the same window. The commands are quoted from platform-1e's run 11 report ("Leo's attended offline window"); the quoted text in each `[...]` block is the files seed copy Paul approved on 2026-10-15, verbatim. This is an attended procedure: there is no automatic transaction and no resume after a reboot. |
| +What: Plan 9 milestone 4 on leo. The commands are quoted from platform-1e's run 11 report ("Leo's attended offline window"); the quoted text in each `[...]` block is the files seed copy Paul approved on 2026-10-15, verbatim. This is an attended procedure: there is no automatic transaction and no resume after a reboot. |
| -Paul's go: for the window, with a written deadline. Step 9 needs its own go. |
| +**Option B (2026-10-20).** Leo enters the window empty: blocks B1 and B2 below delete every instance record and move the hosts' leftovers aside first, with the dev console claims-off. The window is 8.1, 8.1b (`backfill-collect`, a zero-host plan), 8.2 (`journal-migrate --window`), 8.2b (`backfill-done`) and 8.4. There is no adoption, so no 8.2c, and no files seed, so no 8.3: `[runbook.classification]` below says a proven new empty host needs no files seed, and `claimtrial2` is made new and empty after step 9 (block B4). The seed strings stay quoted because 8.4 and the deadline still cite them. |
| + |
| +Paul's go: for the window, with a written deadline. Step 9 needs its own go, and block B4 with acceptance 1 to 4 another (D6). |
| Before the window: |
| - blockers 1 to 3 closed and steps 4 to 7a done; |
| -- a confirmed drained and replicated stop is possible for `claimtrial`, and nothing else is on leo; |
| +- gates G0 to G4 ticked (top of the window package): v0.28.5 published, block B5 passed, signing session 6 open, leo on v0.28.5; |
| +- blocks B1 and B2 done (gate G5): no instance record in the bucket, and nothing on leo; |
| - an external traffic barrier that survives a reboot is in place; |
| - a second root session is open to enforce the deadline; |
| - any update or recovery on leo is settled; genuine pending update evidence is kept; |
| - the named-unit preflight still holds: leo has no named-unit or `backing=unit` route (P5). |
| -The window ID, deadline, barrier, gate evidence, umask audit and hold commands are in "Window package (for Paul)" at the top of this file. Its STOPs S1 and S2 must be cleared before the go. platform-1e is fixing S1, S2 and `journal-migrate`'s chmod for v0.28.5, and the window waits for that release and Paul's go. |
| +The window ID, deadline, barrier, gate evidence, umask audit and hold commands are in "Window package (for Paul)" at the top of this file. Its STOPs S1 and S2 are met under Option B as that package's "STOPs before the go" says, once v0.28.5 carries platform-1e's branch. |
| Rollback (Paul's ruling R1, 2026-10-19): a window that stops before 8.2's `host.env` edit is fully undone. The units and the edge are restored, enablement is restored and the barrier is lifted (window package, section 5, case B1). From 8.2's `host.env` edit onwards a stop is stop-and-hold ("Stop and hold", and section 5's cases B2 to E). |
| @@ -2465,6 +2552,179 @@ The approved description of the step: |
| On leo, "the rollout-owned edge mask" and "runtime mask" in these strings mean the persistent mask made in 8.1 (the unit file moved aside and `/dev/null` linked in its place). A runtime mask in `/run` does not work on leo (P8). |
| +### Before the window (Option B): P-B2, B5, B1 and B2 |
| + |
| +In this order: P-B2 is approved before signing session 6 opens; B5 runs before the session opens (gate G2); B1 and B2 run after leo is on v0.28.5 (gate G4), and B5's record and registry checks run again after B2 (gate G5). |
| + |
| +#### P-B2. The signer's grants (Paul's approval, D4) |
| + |
| +Signing session 6 opens as `signing-host.md` "Create" says, with the release key on a host again. Its role adds, by exact key or prefix only, to what the fifth session had: |
| +- write on exactly `control/system/router/releases/v0.28.5.json`, for `publish-identity`; |
| +- `s3:GetObject` and `s3:PutObject` on `control/hosts-backfill.json`; |
| +- `s3:GetObject` on `control/host-census/<BACKFILL_ID>/*` and `control/hosts-quarantine.json`; |
| +- `s3:GetObject` on `apps/*/*/instances/*.json`, and `s3:ListBucket` with the prefixes `apps/` and `control/hosts/`. |
| + |
| +The list is the step 8 delta's, plus the `control/hosts/` listing `backfill-done` makes (Fable section 4, T1). Confirm it against platform-1e's branch before Paul approves it. The session stays open from `publish-identity` until 8.2b passes or the window stops, then closes as "Closing the signing session" says. |
| + |
| +#### B5. Prechecks (gate G2, read-only) |
| + |
| +From DOCS-T1, profile `kass`, with the `ssm.sh` helper ("How commands are run"). Run the `curl` outside the sandbox. Nothing here writes anything. |
| + |
| +```bash |
| +R="--profile kass --region eu-west-1" |
| +B=carlos-dev-640012953448 |
| +S="$TMPDIR/cohort1/ssm.sh" |
| +# 1. The empty app segment: no current object, no PRE / |
| +aws $R s3 ls --recursive "s3://$B/apps/bab//" | wc -l |
| +aws $R s3 ls "s3://$B/apps/bab/" |
| +# 2. Instance records, and each one's acq (backfill-collect refuses a record without one) |
| +aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -E '/(instances|sidecars)/[^/]+\.json$' > "$TMPDIR/cohort1/records" || true |
| +cat "$TMPDIR/cohort1/records" |
| +while read -r key; do aws $R s3 cp "s3://$B/$key" - | jq -r --arg k "$key" '[$k, (.acq // "MISSING")] | @tsv'; done < "$TMPDIR/cohort1/records" |
| +# 3. No backfill plan, no claims, no admission |
| +aws $R s3api head-object --bucket "$B" --key control/hosts-backfill.json 2>&1 | grep -c 'Not Found' |
| +aws $R s3api list-objects-v2 --bucket "$B" --prefix control/hosts/ --query 'KeyCount' |
| +aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -cE '/admission\.json$' || true |
| +# 4. The roster's pools, from a signature-verified inspect on sophie (stdout only, nothing staged) |
| +"$S" eu-west-1 i-05498eada6f0afd23 'cd /root/claim-trial && /opt/carlos/carlos system claims inspect --bucket carlos-dev-640012953448 --aws-region eu-west-1 --public-key "$(cat release.pub)" | python3 -c "import json,sys; d=json.load(sys.stdin); print(d[\"roster\"][\"revision\"], d[\"roster\"][\"regions\"][\"ie\"][\"pools\"])"' |
| +# 5. Sophie and dennis: no cohort1 row, no CARLOS_POOLS |
| +"$S" eu-west-1 i-05498eada6f0afd23 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes where pool=\x27cohort1\x27\").fetchone()[0])"; grep -c "^CARLOS_POOLS=" /etc/carlos/host.env || true' |
| +"$S" eu-west-2 i-009d593f451cfd562 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes where pool=\x27cohort1\x27\").fetchone()[0])"; grep -c "^CARLOS_POOLS=" /etc/carlos/host.env || true' |
| +# 6. The three stopped instances carry no box role |
| +aws $R ec2 describe-instances --instance-ids i-03845a1ad8db16bbf i-00225aeea39e09fa5 i-0ec5ed933bf6539d3 \ |
| + --query 'Reservations[].Instances[].[InstanceId,State.Name,IamInstanceProfile.Arn]' --output text |
| +# 7. The dev console is claims-off |
| +curl -s -o /dev/null -w '%{http_code}\n' https://platform.dev.carlosframework.com/healthz |
| +"$S" eu-west-1 i-05498eada6f0afd23 'sha256sum /etc/carlos/claim-console.env | cut -c1-64; tr "\0" "\n" < /proc/$(systemctl show -p MainPID --value console.service)/environ | grep "^CARLOS_HOST_CLAIM="; journalctl -u console.service --since "$(systemctl show -p ActiveEnterTimestamp --value console.service)" --no-pager | grep -c "off on a host-claim console" || true' |
| +``` |
| + |
| +Expected, line by line: |
| +1. `0`; the listing has no `PRE /`. |
| +2. Before block B1: the three records `apps/bab/claimtrial/…`, `apps/bab/rollcheck-0261/…` and `apps/bab/wakecheck/…` (plus the offline-move rehearsal app's, if that rehearsal has run), each `MISSING`. That is the known blocker under gate G2, resolved by B1. After B1: no record at all, and the `while` loop prints nothing. |
| +3. `1` (the plan is 404); `0`; `0`. |
| +4. `Success`, then `9 ['cohort1', 'console']` (the revision, then `cohort1` among the pools). A later revision is fine if the signing session moved it; read it from the session's record. |
| +5. `Success`, `0`, `0` for each box. |
| +6. Each `stopped`, with profile `kass-prod-profile`, `kass-sidecar-profile` or `kass-devbox-profile`; none `carlos-dev-host*`. |
| +7. `200`; then `c1ae2900…c412` (the off form), `CARLOS_HOST_CLAIM=` (empty) and `0`. |
| + |
| +Any other answer fails G2: fix the one item if it is the known cleanup (with Paul's go), otherwise hold. |
| + |
| +**Tonight's read-only run, 2026-10-20** (`after236/precheck-20261020.md`, run from DOCS-T1 with the same reads): every item PASS except item 2. Item 1: `0` current objects and no `PRE /` (1044 noncurrent versions under delete markers, as step 2 predicts). Item 2: three records, `claimtrial`, `wakecheck` and `rollcheck-0261`, none with `acq`: the known blocker. Item 3: 404, `0` claims ever written, `0` admissions. Item 4: revision 9, etag `a69f6d93…`, pools `cohort1` and `console`. Item 5: `0` and `0` on both; sophie has four routes, all pool-less (the console, `helloworld`, `website` and a stale `wakecheck` row); dennis has none. Item 6: as expected. Item 7: `200` at `f3bf1214`, `c1ae2900…`, the empty flag, no claim-runtime line, `NRestarts=0` since the pause. |
| + |
| +#### B1. Delete every instance record through the ordinary door (gate G5, decision D2) |
| + |
| +The dev console must be claims-off (B5 item 7) when this runs: a claims-off console retires a record directly, with no claim operation. platform-1e's ruling (2026-10-20) is a zero-host plan, so B1 deletes every instance record in the bucket: |
| +- `bab/claimtrial`: `claimtrial.dev.carlosframework.com`, the 2026-10-18 trial app; |
| +- `bab/wakecheck`: `wakecheck.dev.carlosframework.com`, the standing dev wake check (recreated after the trial, see "Trial assumptions"); |
| +- `bab/rollcheck-0261`: `rollcheck-0261.dev.carlosframework.com`, static, Paul's roll-check app from 2026-10-13 (his OK at D2); |
| +- the offline-move rehearsal app (name set at the rehearsal; see the moves-offline runbook). The rehearsal runs before the window: its app is created claims-off on sophie and moved to dennis. |
| + |
| +From DOCS-T1, in B5's shell (`R`, `B` and `S` set), signed in to the dev console as the operator (this machine's default console is flagship, so every command names the dev console). `CARLOS_DEPLOYMENT_BUCKET` must be unset. The two rehearsal values come from the moves-offline runbook's record; the `:?` lines refuse to go on while either is empty: |
| + |
| +```bash |
| +DEV=https://platform.dev.carlosframework.com |
| +REHEARSAL_APP='' # set at the rehearsal |
| +REHEARSAL_HOST='' # set at the rehearsal |
| +: "${REHEARSAL_APP:?set it from the rehearsal record}" |
| +: "${REHEARSAL_HOST:?set it from the rehearsal record}" |
| +carlos instances delete --app claimtrial --account bab --host claimtrial.dev.carlosframework.com --console $DEV --no-prompt |
| +carlos instances delete --app wakecheck --account bab --host wakecheck.dev.carlosframework.com --console $DEV --no-prompt |
| +carlos instances delete --app rollcheck-0261 --account bab --host rollcheck-0261.dev.carlosframework.com --console $DEV --no-prompt |
| +carlos instances delete --app "$REHEARSAL_APP" --account bab --host "$REHEARSAL_HOST" --console $DEV --no-prompt |
| +carlos apps delete --account bab --console $DEV claimtrial |
| +``` |
| + |
| +Each `instances delete` prints `deleted instance <host>` and says the route retires on the next reconcile pass. The last line moves the `claimtrial` app to the Trash (`apps/bab/claimtrial/deleted.json`), which also stops it minting again; `wakecheck`, `rollcheck-0261` and the rehearsal app keep their apps, so wakecheck can be recreated under its own name. If the rehearsal app is in another account, use that account's sqid in its line. |
| + |
| +Then check the bucket, from DOCS-T1: |
| + |
| +```bash |
| +aws $R s3 ls --recursive "s3://$B/apps/" | awk '{print $4}' | grep -cE '/(instances|sidecars)/[^/]+\.json$' || true |
| +aws $R s3 ls "s3://$B/apps/bab/claimtrial/deleted.json" |
| +``` |
| + |
| +Expected: `0`, then the trash marker. The console announces each delete, so the boxes retire the rows within seconds; the hourly floor and the 5-minute pass are the fallback. Then the registries, read-only (P4's query): |
| + |
| +```bash |
| +leo 'python3 -c " |
| +import sqlite3 |
| +c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| +print(c.execute(\"select count(*) from routes\").fetchone()[0]) |
| +for r in c.execute(\"select host,app,pool from routes order by host\"): print(r)" |
| +systemctl list-units --all --plain --no-legend "carlos-instance-gen@*" "carlos-instance@*" |
| +journalctl -u carlos-edge.service --since -30min --no-pager | grep "removed retired"' |
| +"$S" eu-west-1 i-05498eada6f0afd23 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); [print(r) for r in c.execute(\"select host,app,pool from routes order by host\")]"' |
| +"$S" eu-west-2 i-009d593f451cfd562 'python3 -c "import sqlite3; c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True); print(c.execute(\"select count(*) from routes\").fetchone()[0])"' |
| +``` |
| + |
| +Expected: |
| +- **leo: zero rows** (`0`, no route listed), no instance unit, and a `provision: removed retired instance route claimtrial.dev.carlosframework.com (record gone)` line; |
| +- sophie: three rows, the console (`platform.dev.carlosframework.com`), `helloworld.dev.oncarlos.com` and `website.dev.oncarlos.com`; the stale `wakecheck` row is gone; |
| +- dennis: `0` (the rehearsal app's row, if it was there, is gone). |
| + |
| +If a row stays after 10 minutes, read that box's edge journal for the reason and hold. Do not remove a row by hand. |
| + |
| +#### B2. Move the leftovers aside, never delete them (gate G5) |
| + |
| +A claims-off box retires only the registry row. The host's files stay: its data directory, its program history, its live link, its key and token directory, and its rendered config. The migration census refuses unclassified leftovers on leo, and every leftover left anywhere is a STOP in that box's activation window later (Fable section 9, item 4). Certificates under `/data/acme` stay: the certificate cache is box state. |
| + |
| +Run it on leo first, then on sophie and dennis, each in an attended root shell, after B1's registry checks pass on that box. The same block runs on every box; a path that is not there is reported `absent` and left. `/data/instances/<host>` is the bind-mounted view of `/data/.instances/<host>` (`data-instances.mount` binds the whole directory), so one move covers both. A nested host's `<host>.config.env` and `<host>.env` sit inside that directory; a flat host's sit in `/data` and are moved by name. |
| + |
| +```bash |
| +set -eu |
| +umask 077 |
| +STAMP=$(date -u +%Y%m%dT%H%MZ) |
| +ASIDE="/var/lib/carlos-aside-$STAMP" |
| +REHEARSAL_HOST='' # set at the rehearsal: the same value as B1's |
| +: "${REHEARSAL_HOST:?set it from the rehearsal record}" |
| +HOSTS=(claimtrial.dev.carlosframework.com wakecheck.dev.carlosframework.com rollcheck-0261.dev.carlosframework.com "$REHEARSAL_HOST") |
| +test "$(stat -c %d /data)" = "$(stat -c %d /var/lib)" |
| +test "$(stat -c %d /opt/carlos/live)" = "$(stat -c %d /var/lib)" |
| +mkdir -m 0700 -- "$ASIDE" |
| +test "$(stat -c '%a %U:%G' "$ASIDE")" = '700 root:root' |
| +aside() { |
| + if [ -L "$1" ] || [ -e "$1" ]; then |
| + mv -T -- "$1" "$ASIDE/$H/$2" |
| + echo "moved $1" |
| + else |
| + echo "absent $1" |
| + fi |
| +} |
| +for H in "${HOSTS[@]}"; do |
| + test "$(python3 -c "import sqlite3,sys; c=sqlite3.connect('file:/data/registry.db?mode=ro',uri=True); print(c.execute('select count(*) from routes where host=?',(sys.argv[1],)).fetchone()[0])" "$H")" = 0 |
| + test -z "$(systemctl list-units --all --plain --no-legend "carlos-instance-gen@$H:*" "carlos-instance@$H.service")" |
| + mkdir -m 0700 -- "$ASIDE/$H" |
| + if [ -L "/opt/carlos/live/$H" ]; then readlink -- "/opt/carlos/live/$H"; fi |
| + aside "/data/.instances/$H" instance |
| + aside "/data/programs/$H.json" program.json |
| + aside "/opt/carlos/live/$H" live-link |
| + aside "/data/$H" key-dir |
| + aside "/data/$H.config.env" config.env |
| + aside "/data/$H.env" env |
| +done |
| +``` |
| + |
| +The two device tests make every move a rename on the root filesystem (on leo, `/`, `/data`, `/opt` and `/var/lib` are one ext4 filesystem, 3.G5); if either fails on a box, stop on that box and report. The registry test refuses a host that still has a row on this box, and the unit test one with an instance unit loaded: go back to B1's checks. Then check, with exact paths, that nothing is left (gate G5). One test per line, so `set -e` stops on the first that fails: |
| + |
| +```bash |
| +for H in "${HOSTS[@]}"; do |
| + test ! -e "/data/.instances/$H" |
| + test ! -L "/data/.instances/$H" |
| + test ! -e "/data/instances/$H" |
| + test ! -L "/data/instances/$H" |
| + test ! -e "/data/programs/$H.json" |
| + test ! -e "/opt/carlos/live/$H" |
| + test ! -L "/opt/carlos/live/$H" |
| + test ! -e "/data/$H" |
| + test ! -e "/data/$H.config.env" |
| + test ! -e "/data/$H.env" |
| +done |
| +ls -la /data/instances /data/.instances /data/programs /opt/carlos/live |
| +ls -laR "$ASIDE" |
| +``` |
| + |
| +Expected: no failed test. On leo, `/data/instances` and `/data/.instances` each hold only `_probe`, and `/data/programs` and `/opt/carlos/live` are empty; on sophie and dennis, none of the four lists has an entry for any of the four hosts (sophie's other hosts stay); the aside directory holds what was moved, root `0700`. On leo the moves are `claimtrial`'s and wakecheck's instance directories, program files and live links (S2's list); `rollcheck-0261` and the rehearsal host are `absent` there. Record each box's `$ASIDE` path and listing on #250. Never delete from it: it is the hosts' retained data until a separately scoped clean-up. `/run/carlos/.instances/<host>` is tmpfs, outside the census roots, and goes at the next reboot; it is not moved. |
| + |
| ### 8.1 Record, drain and mask |
| Choose a new window directory outside systemd's unit search path. The commands assume the twelve canonical unit files P8 recorded are root-owned regular files in `/etc/systemd/system`. Inspect their parents, drop-ins, effective units, metadata and enablement first. If a path is a symlink, a pre-existing mask or an unexpected unit, stop and resolve it with its owner; do not overwrite it. Before stopping anything, require the recorded states: edge and all five triggers loaded and active; all six migration services loaded and inactive. |
| @@ -2477,13 +2737,23 @@ umask 077 |
| # umask 022, so a directory the binary means to be public is not made 0700 |
| # (#317; the window package's umask audit). |
| pub() ( umask 022; "$@" ) |
| -WINDOW='/etc/carlos/claim-window-cohort1-<approved-window-id>' |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go: the window package's value |
| +[[ $WINDOW_ID =~ ^[0-9]{8}T[0-9]{4}Z$ ]] |
| +WINDOW="/etc/carlos/claim-window-cohort1-$WINDOW_ID" |
| +GUARD=/etc/systemd/system/carlos-edge.service.d/90-host-floor.conf |
| +SEED=/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| +test -f "$GUARD" |
| +test ! -L "$GUARD" |
| +test ! -e "$SEED" |
| +test ! -L "$SEED" |
| TRIGGERS=(carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path) |
| SERVICES=(carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service) |
| UNITS=("${TRIGGERS[@]}" "${SERVICES[@]}" carlos-edge.service) |
| mkdir -m 0700 -- "$WINDOW" |
| mkdir -m 0700 -- "$WINDOW/held" |
| cp -a -- /etc/carlos/host.env "$WINDOW/host.env.before" |
| +stat -- "$GUARD" > "$WINDOW/edge-guard.stat" |
| +cp -a -- "$GUARD" "$WINDOW/edge-guard.before" |
| for unit in "${UNITS[@]}"; do |
| path="/etc/systemd/system/$unit" |
| test -f "$path" |
| @@ -2504,7 +2774,9 @@ systemctl list-units --all --plain 'carlos-instance@*' 'carlos-instance-gen@*' |
| pgrep -a -u carlos || test "$?" = 1 |
| ``` |
| -Require no tenant writer, no remaining carlos-user process, completed replication and no active root mutator. Stop if a unit is failed or in transition rather than assuming it drained. Disabling the edge keeps it from starting at boot while the seed override exists; the saved enablement is restored in step 9. Keep the external traffic barrier throughout. |
| +`GUARD` and `SEED` are set here, not in a seed block, so section 5's cases and 8.4 have them. Their four tests check that the edge's guard drop-in is in place and that no seed override exists; under Option B none is ever written. The guard's bytes and metadata are saved as `edge-guard.before` and `edge-guard.stat` for 8.4's check. Leo has no routes after block B1, so the `list-units` lines print nothing. |
| + |
| +Require no tenant writer, no remaining carlos-user process, completed replication and no active root mutator. Stop if a unit is failed or in transition rather than assuming it drained. Disabling the edge keeps it from starting at boot; the saved enablement is restored in step 9. Keep the external traffic barrier throughout. |
| ```bash |
| for unit in "${UNITS[@]}"; do |
| @@ -2518,6 +2790,32 @@ pub /opt/carlos/carlos system claims install-control --dry-run |
| Expected: every unit `LoadState=masked` and inactive. `install-control --dry-run` calls main's unchanged `verifyOfflineRootMigration`: all six migration services must be `LoadState=masked` and inactive or failed, and no root `adopt`, `system apply` or `guard-arm` process may remain. The operator checks the triggers, the edge and the tenant drain. Do not relax the verifier or read it as proof of replication. |
| +### 8.1b Collect the backfill: a zero-host plan (signer, at the end of 8.1) |
| + |
| +`journal-migrate`'s census reads a signed `collecting` plan, so it must exist before 8.2. It runs in signing session 6 on the trusted signer, as one SSM command from DOCS-T1 that sources `helpers.sh` (5.1's first helper block, with `CARLOS=/usr/local/bin/carlos`, `KEY=/run/claim-signer/release.key` and no `AWS_PROFILE`). The dev console is claims-off, so the plan freezes nothing on dev (Fable section 1). It still comes at the end of 8.1, with leo's agent stopped, so nothing on leo moves a record between collect and the census. |
| + |
| +```bash |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go: the window package's value |
| +BACKFILL_ID="cohort1-${WINDOW_ID%%T*}" |
| +refresh |
| +jq -n --arg b "$B" --argjson rev "$REV" --arg id "$BACKFILL_ID" \ |
| + '{phase:"backfill-collect",bucket:$b,expected_revision:$rev,backfill:{id:$id}}' > plan.json |
| +"$CARLOS" system claims run --plan /root/claim-trial/plan.json --signing-key "$KEY" --public-key "$PUB" --aws-region "$R" --dry-run > backfill-collect-dry.json |
| +jq '{writes: [.writes[] | {key, before_etag}], backfill}' backfill-collect-dry.json |
| +"$CARLOS" system claims run --plan /root/claim-trial/plan.json --signing-key "$KEY" --public-key "$PUB" --aws-region "$R" > backfill-collect.json |
| +jq -e --arg id "$BACKFILL_ID" '.backfill.id == $id' backfill-collect.json |
| +jq -e '.backfill.phase == "collecting"' backfill-collect.json |
| +jq -e --argjson rev "$REV" '.backfill.directory_revision == $rev' backfill-collect.json |
| +jq -e '.backfill.hosts == {}' backfill-collect.json |
| +jq -e '(.backfill.hosts | length) == 0' backfill-collect.json |
| +jq -e '(.writes | length) == 1' backfill-collect.json |
| +jq -e '.writes[0].key == "control/hosts-backfill.json"' backfill-collect.json |
| +jq -e '.writes[0].before_etag == ""' backfill-collect.json |
| +refresh |
| +``` |
| + |
| +Expected: the dry run prints one create-only write of `control/hosts-backfill.json` and no hosts; the apply prints the same plan as read back; every `jq -e` prints `true`; the roster revision is unchanged. `hosts` is the empty map `{}`, not `null`: platform-1e's ruling (2026-10-20) is that collect writes an empty, non-nil host map, pinned by a test on their branch. Refusals, from `claim-step8-readiness`: `backfill refused: the roster must be published with a home region and a verified floor`, `backfill refused: a backfill plan already exists`, `backfill refused: every instance record must carry an acquisition and stay unchanged while the plan is written`. The last means a record appeared after block B1. Any refusal is a STOP before 8.2: case B1 undo (gate G6). A plan already written stays, and with the console off it is harmless. |
| + |
| ### 8.2 Configure, migrate and start root |
| Update the protected `/etc/carlos/host.env` through the approved configuration procedure: exactly one `CARLOS_HOST_CLAIM=1`, exactly one `CARLOS_HOST_CLAIM_ORIGINS=1`, no `CARLOS_MOVES_PHASE1`, no `CARLOS_REGION` (never a literal `CARLOS_REGION=ie` on a home box), and `CARLOS_POOLS=cohort1` kept from step 4. Keep the saved original (`$WINDOW/host.env.before`). Then: |
| @@ -2531,7 +2829,7 @@ systemctl cat carlos-instance-control.service |
| set -a |
| . /etc/carlos/host.env |
| set +a |
| - /opt/carlos/carlos system journal-migrate |
| + /opt/carlos/carlos system journal-migrate --window "$WINDOW" |
| ) |
| # Mode check after journal-migrate (window package, umask audit). Its census |
| # step creates /opt/carlos/system/guard/moves and .../seeds as public |
| @@ -2559,6 +2857,19 @@ systemctl is-active carlos-instance-control.service |
| journalctl -u carlos-instance-control.service --since '-5min' --no-pager |
| ``` |
| +`--window "$WINDOW"` is v0.28.5's (platform-1e's `claim-step8-readiness`): the census accepts the window's `/dev/null` masks only when `$WINDOW` is root's own `0700` directory outside the unit and data paths and `$WINDOW/held` holds each masked unit's original, which 8.1 made. Before writing anything it refuses otherwise, with `the window directory must be root's own private directory outside the unit and data paths, with every masked unit's original in its held directory`. Any other entry it cannot class is named: `migration census: <root> entry "<path>" is not classified`. That is a STOP (case B2 hold). |
| + |
| +Then leo's signed census report, read from DOCS-T1: |
| + |
| +```bash |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go: the window package's value |
| +BACKFILL_ID="cohort1-${WINDOW_ID%%T*}" |
| +aws --profile kass --region eu-west-1 s3 cp "s3://carlos-dev-640012953448/control/host-census/$BACKFILL_ID/leo-climbing.json" "$TMPDIR/cohort1/leo-census.json" |
| +jq -e '(.report.installations | length) == 0' "$TMPDIR/cohort1/leo-census.json" |
| +``` |
| + |
| +Expected: `true`. Leo's report lists zero installations, which is what 8.2b's `backfill-done` requires of every report (gate G7). |
| + |
| Run `install-control` once: after host.env carries `CARLOS_HOST_CLAIM=1` and `CARLOS_HOST_CLAIM_ORIGINS=1`, and before `journal-migrate`. The rendered unit depends only on the host.env settings (`readClaimUnitSettings`: enabled, the data and opt paths, the root retry directory), not on the census, so it is not run again afterwards (platform-1e, 2026-10-17). |
| Expected: |
| @@ -2570,195 +2881,32 @@ Expected: |
| Keep every migration and root record. A failure leaves serving held. |
| -### 8.3 Run the approved files seed |
| - |
| -[runbook.budget] |
| - |
| -> Files seed processes one host at a time. Budget a database restore and upload using measured rehearsal rates, and record a deadline before starting. Capacity checks enforce the start floor, not a reserved restore budget. At the deadline, stop carlos-edge.service through the approved activation workflow. Cleanup has a 30-second budget; the temporary unit allows 45 seconds before systemd ends the process. Keep interrupted evidence so the next rerun can settle the attempt. |
| - |
| -[runbook.run] |
| - |
| -> Run the initial files seed in the delegated edge unit during the offline activation window. Each eligible host receives an ordinary uncertified cold admission, retirement of its old local database, restoration of the remote baseline and publication into the admitted files prefix. Writes beyond that remote baseline are declared lost, which is why the drained, replicated stop must come first. The command never launches a tenant. For a later rerun, invoke the same command as the service user with the protected box configuration; it sends a synchronous request to the authenticated running agent, which excludes competing wakes through settlement. |
| - |
| -[runbook.run_command] |
| +### 8.2b Done: `backfill-done` (signer, after 8.2's checks) |
| -``` |
| -carlos system files seed |
| -``` |
| - |
| -[cli.usage] |
| - |
| -``` |
| -Usage: carlos system files seed [--host <host>] |
| -``` |
| - |
| -[runbook.delegation] |
| - |
| -> Through the approved activation workflow, create the new temporary override at the path below. Refuse an existing file or symlink there. Save the original effective unit and masks; preserve the service user, protected environment, delegation and hardening. Reload systemd and inspect the effective command before removing only the rollout-owned runtime mask. Keep the traffic barrier in place while starting the seed one-shot. Capture its InvocationID, Result, ExecMainStatus and journal before stopping it; RemainAfterExit keeps a successful completed invocation available. Save failed results before any reset-failed. After stopping, restore the rollout-owned runtime mask, remove only the recorded override, reload and inspect the normal unit. Resolve pre-existing operator masks through their owner. Rerun a crashed one-shot in this same delegated unit within the approved offline workflow. |
| - |
| -[runbook.override_path] |
| - |
| -``` |
| -/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| -``` |
| - |
| -[runbook.override] |
| - |
| -``` |
| -[Service] |
| -Type=oneshot |
| -RemainAfterExit=yes |
| -ExecStart= |
| -ExecStart=/opt/carlos/carlos system files seed |
| -ExecReload= |
| -Restart=no |
| -TimeoutStartSec=0 |
| -TimeoutStopSec=45s |
| -``` |
| +The new signer phase from platform-1e's branch `claim-empty-cohort-done` (Fable section 4, T1; the brief calls it `carlos claims backfill-done`). It takes the plan from `collecting` straight to `done`. It refuses unless: the plan's etag is the collecting one; the plan verifies; the phase is `collecting`; the live record listing equals the plan's hosts (none); `control/hosts/` lists nothing; no `apps/*/*/admission.json` exists; and every roster box with a census key has a report for this plan, captured after the plan was made, listing zero installations. Boxes without a key are skipped and listed as unreported: on dev that is sophie. It then writes the plan as `done` with If-Match and reads it back. There is no adoption: 8.2c is gone, and the files seed (the 2026-10-19 plan's 8.3) with it. |
| -[runbook.reload_command] |
| - |
| -``` |
| -systemctl daemon-reload |
| -``` |
| - |
| -[runbook.unit_verify] |
| - |
| -``` |
| -systemctl cat carlos-edge.service |
| -``` |
| - |
| -[runbook.unmask_command] |
| - |
| -``` |
| -systemctl unmask --runtime carlos-edge.service |
| -``` |
| - |
| -On leo this unmask is replaced by putting the edge unit file back from `$WINDOW/held`, as in the block below. |
| - |
| -[runbook.start_command] |
| - |
| -``` |
| -systemctl start carlos-edge.service |
| -``` |
| - |
| -[runbook.unit_status_command] |
| - |
| -``` |
| -systemctl show carlos-edge.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| -``` |
| - |
| -[runbook.journal_command] |
| - |
| -``` |
| -journalctl _SYSTEMD_INVOCATION_ID=<invocation-id> --no-pager |
| -``` |
| - |
| -[cli.exit_status] |
| - |
| -> Exit 0 means the selected run finished without a reported error; it can include skipped hosts and does not prove that every host on the box was seeded. Exit 2 means the executor collected host failures or noticed cancellation between hosts. Exit 1 covers setup, validation, transport and other errors outside that collected result. Capture the service’s ExecMainStatus and Result; systemctl start has its own exit status. |
| - |
| -[cli.forwarded_errors] |
| - |
| -> Failure lines contain the underlying provider or operating-system error, sometimes across several lines. Save the full output. A shared “selected authority changed” error does not identify one cause by itself; use the root service journal and the host’s retained evidence. |
| - |
| -The leo commands. The edge's `90-host-floor.conf` (installed by `guard-arm` in step 6) requires `carlos-system-recover.service` and replaces `ExecStart` with `guard-exec`. While recovery is masked, that dependency keeps the seed from starting, and the file name also sorts after `90-files-seed-activation.conf`. Inspect and save its exact bytes and metadata. Only if it is the expected `guard-arm` drop-in, move it out of the search path for the seed, then restore that same file after remasking the edge. Do not remove unrelated dependencies, hardening or service settings. Confirm the approved binary and armed floor before moving it. |
| +Section 4 fixes the phase, not its plan input or its output fields, so those lines carry a marker until they are checked against platform-1e's branch. In the same signing session, as 8.1b: |
| ```bash |
| -GUARD=/etc/systemd/system/carlos-edge.service.d/90-host-floor.conf |
| -SEED=/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| -test -f "$GUARD" |
| -test ! -L "$GUARD" |
| -stat -- "$GUARD" > "$WINDOW/edge-guard.stat" |
| -cp -a -- "$GUARD" "$WINDOW/edge-guard.before" |
| -cat "$GUARD" |
| -test ! -e "$SEED" |
| -test ! -L "$SEED" |
| -mkdir -p -- /run/systemd/system/carlos-edge.service.d |
| -(set -C; cat > "$SEED" <<'SEED_OVERRIDE' |
| -[Service] |
| -Type=oneshot |
| -RemainAfterExit=yes |
| -ExecStart= |
| -ExecStart=/opt/carlos/carlos system files seed |
| -ExecReload= |
| -Restart=no |
| -TimeoutStartSec=0 |
| -TimeoutStopSec=45s |
| -SEED_OVERRIDE |
| -) |
| -cp -a -- "$SEED" "$WINDOW/seed-override.before" |
| -mv -T -- "$GUARD" "$WINDOW/held/90-host-floor.conf" |
| -test "$(readlink /etc/systemd/system/carlos-edge.service)" = /dev/null |
| -rm -- /etc/systemd/system/carlos-edge.service |
| -mv -T -- "$WINDOW/held/carlos-edge.service" /etc/systemd/system/carlos-edge.service |
| -systemctl daemon-reload |
| -systemctl cat carlos-edge.service |
| -systemctl show carlos-edge.service -p ExecStart -p Requires -p After -p User -p Group -p Delegate -p EnvironmentFiles |
| -pub /opt/carlos/carlos system claims install-control --dry-run |
| -``` |
| - |
| -The seed override and its directory are written by this shell, under `umask 077`, so they are `600` and `700` root. Only systemd reads them, as root, so that is intended and needs no check. The seed itself runs inside `carlos-edge.service` as `carlos`, with the unit's umask (no unit sets `UMask=`, so systemd's `0022`), not this shell's. |
| - |
| -Before starting, require the exact seed `ExecStart`, the original service user, protected environment, delegation and hardening, no dependency on a masked service, and an active claim-on root service. The normal edge stays stopped. The six migration masks and all five trigger masks stay in place. |
| - |
| -```bash |
| -if systemctl start carlos-edge.service; then START_RC=0; else START_RC=$?; fi |
| -printf '%s\n' "$START_RC" > "$WINDOW/seed-systemctl-status" |
| -systemctl show carlos-edge.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/seed-result" |
| -INVOCATION=$(systemctl show carlos-edge.service -p InvocationID --value) |
| -test -n "$INVOCATION" |
| -printf '%s\n' "$INVOCATION" > "$WINDOW/seed-invocation" |
| -journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > "$WINDOW/seed-journal" |
| -``` |
| - |
| -The one-shot has no start timeout, so the second root session enforces the deadline. Capture the service's `Result` and `ExecMainStatus`, not only the status of `systemctl start`. |
| - |
| -Expected: `Host "claimtrial.dev.carlosframework.com": seeded.` and its admitted prefix line, `Result=success`, `ExecMainStatus=0`. Review every host, skipped path, admitted prefix, retry state and blocked-host result against the activation inventory and incarnation. A successful seed does not prove a healthy tenant. |
| - |
| -[runbook.skips] |
| - |
| -> Review every skipped-file line. Oversized and non-regular files are omitted. Unreadable or changing files fail that host. |
| - |
| -[runbook.committed] |
| - |
| -> A committed manifest permits serving through the ordinary origin, restore and readiness checks. An existing manifest is never seeded over. Full verification and repair are deferred, and moves must remain disabled until their separate gates pass. Missing or corrupt evidence needs investigation; do not remove a manifest to make a host eligible for seeding. |
| - |
| -[runbook.stragglers] |
| - |
| -> Record every failed, skipped or deferred host and the reason. A serving host without committed files evidence refuses before admission and spends no new wake retry; other hosts may serve after their activation gates pass. Rerun a blocked host through the running agent with both claim flags set and moves off, or repeat the delegated one-shot inside an approved offline window. Keep any existing retry debt. For claims assigned elsewhere, check that box’s activation results. |
| - |
| -[runbook.retries] |
| - |
| -> Each new preparation spends one wake retry attempt. Under the host’s wake exclusion, a rerun settles a crashed attempt and rereads the claim before proceeding. It never refunds that attempt. The command prints the current retry state and waits for the retry-after time before preparing again; the old storage lease must also have expired or been released through its owner checks. An established manifest causes no new preparation. Capacity and evidence refusals precede reservation. If stop proof is unavailable, keep the evidence and investigate. Exhausted episodes need an operator-authorized console reset. A successful seed does not report a healthy tenant. |
| - |
| -[runbook.rerun_command] |
| - |
| -``` |
| -carlos system files seed --host <host> |
| +WINDOW_ID=YYYYMMDDTHHMMZ # set at go: the window package's value |
| +BACKFILL_ID="cohort1-${WINDOW_ID%%T*}" |
| +refresh |
| +jq -n --arg b "$B" --argjson rev "$REV" --arg id "$BACKFILL_ID" \ |
| + '{phase:"backfill-done",bucket:$b,expected_revision:$rev,backfill:{id:$id}}' > plan.json # FLAGS: confirm against platform-1e's branch |
| +"$CARLOS" system claims run --plan /root/claim-trial/plan.json --signing-key "$KEY" --public-key "$PUB" --aws-region "$R" --dry-run > backfill-done-dry.json |
| +jq '{writes: [.writes[] | {key, before_etag}], backfill}' backfill-done-dry.json |
| +"$CARLOS" system claims run --plan /root/claim-trial/plan.json --signing-key "$KEY" --public-key "$PUB" --aws-region "$R" > backfill-done.json |
| +jq -e '.backfill.phase == "done"' backfill-done.json # FLAGS: confirm against platform-1e's branch |
| +jq -e '(.writes | length) == 1' backfill-done.json |
| +jq -e '.writes[0].key == "control/hosts-backfill.json"' backfill-done.json |
| +jq -e '(.writes[0].before_etag | length) > 0' backfill-done.json |
| +jq -e '.backfill.unreported == ["sophie-knitting"]' backfill-done.json # FLAGS: confirm against platform-1e's branch |
| +refresh |
| ``` |
| -[runbook.reset] |
| - |
| -> For an exhausted episode, use an authorized operator session to POST /api/cli/ops/hosts/<host>/resolve with action="reset-wake-retry", the current incarnation and episode, and a new request_id. The console must prove any active attempt stopped before resetting it. Confirm the new episode through the approved operator workflow before rerunning. Do not give this authority to the service user. |
| - |
| -[runbook.cleanup] |
| - |
| -> Save every admitted-prefix line with the activation record, including lines printed before a host fails. The command does not print a separate orphan-prefix inventory and deletes no orphan objects. If interruption happens before a prefix is printed, keep the durable claim and root records for investigation. Any later deletion needs separately scoped checks of references, leases and stop evidence; never expire a whole generation by age alone. |
| - |
| -[runbook.retention] |
| +Expected: the dry run prints one If-Match write of `control/hosts-backfill.json` (its `before_etag` the collecting plan's) and writes nothing; the apply prints the plan read back at `done`; every `jq -e` prints `true`; sophie is the one unreported box; the roster revision is unchanged. The one new refusal is `backfill refused: a census report lists installations; this phase is for an empty cohort` (Fable's suggested text, waiting for Paul's copy review, D7). The others reuse `ErrBackfillPlan` and `ErrBackfillRecords`. Any refusal is gate G8's failure: hold (case B2). The plan stays `collecting`, and with the dev console claims-off nothing is frozen. |
| -> Each claimed cold wake retires the previous local database on the same instance mount, including its generation marker and any WAL or SHM files. The first seed parks the legacy database once, with a seven-day deadline. A generation that never launched is discarded and cannot replace the retained park. A later generation whose tenant started may replace that slot. Root checks expiry and disk pressure on startup and once a minute while running, and may reap verified completed parks early, oldest first. A retained park therefore has no guaranteed minimum lifetime. Incomplete, unknown or identity-mismatched state is kept for recovery or investigation. Never delete parked paths by hand. |
| - |
| -[runbook.park_status] |
| - |
| -> Read the root service journal for recovery and cleanup failures. Rich park listings and automatic rehabilitation of ambiguous directories are deferred. Keep unknown or identity-mismatched data for a separately scoped investigation. Do not treat a quarantine name as deletion authority. |
| - |
| -[runbook.park_journal_command] |
| - |
| -``` |
| -journalctl -u carlos-instance-control.service --no-pager |
| -``` |
| +Then close signing session 6 ("Closing the signing session"), with its CloudTrail check. Nothing after 8.2b needs the signer. |
| ### 8.4 Remask, restore and hold for the serving go |
| @@ -2790,28 +2938,23 @@ On leo this mask is replaced by moving the edge unit file back into `$WINDOW/hel |
| rm -- /run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| ``` |
| -The leo commands. Whether the seed succeeded or failed, stop the one-shot and save its final result. Restore the persistent edge mask before removing the recorded seed override or restoring the guard. Check that the unit and drop-in still match the saved files before moving or deleting them; stop on foreign changes. |
| +These approved strings describe the seed one-shot. Under Option B there is no seed (8.3 is gone): the edge has stayed masked since 8.1, its guard drop-in never moved, and no override was written. So 8.4's first block is a check that this is still so, not a remask. The 2026-10-19 plan's first block is kept in section 5, case C, for a seed override that should never exist. |
| + |
| +The leo commands. Check that the edge is still this window's mask, that the held original and the guard drop-in still match the copies 8.1 saved, and that no override exists. Stop on any foreign change: |
| ```bash |
| -systemctl stop carlos-edge.service |
| -systemctl show carlos-edge.service -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/seed-result-final" |
| -journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > "$WINDOW/seed-journal-final" |
| -cmp -- /etc/systemd/system/carlos-edge.service "$WINDOW/carlos-edge.service.before" |
| -mv -T -- /etc/systemd/system/carlos-edge.service "$WINDOW/held/carlos-edge.service" |
| -ln -s -- /dev/null /etc/systemd/system/carlos-edge.service |
| -systemctl daemon-reload |
| -systemctl show carlos-edge.service -p LoadState -p ActiveState |
| -cmp -- "$SEED" "$WINDOW/seed-override.before" |
| -rm -- "$SEED" |
| -test ! -e "$GUARD" |
| +test "$(readlink /etc/systemd/system/carlos-edge.service)" = /dev/null |
| +cmp -- "$WINDOW/held/carlos-edge.service" "$WINDOW/carlos-edge.service.before" |
| +test -f "$GUARD" |
| test ! -L "$GUARD" |
| -cmp -- "$WINDOW/held/90-host-floor.conf" "$WINDOW/edge-guard.before" |
| -mv -T -- "$WINDOW/held/90-host-floor.conf" "$GUARD" |
| -systemctl daemon-reload |
| +cmp -- "$GUARD" "$WINDOW/edge-guard.before" |
| +test ! -e "$SEED" |
| +test ! -L "$SEED" |
| +systemctl show carlos-edge.service -p LoadState -p ActiveState |
| pub /opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| -After every seed and migration check passes, restore only this window's recorded originals for the six services, then the five triggers. For each, verify the saved bytes and metadata, require its current path to be this window's `/dev/null` link, remove the link and move the original back. Do not start the triggers yet. |
| +After every migration check and 8.2b pass, restore only this window's recorded originals for the six services, then the five triggers. For each, verify the saved bytes and metadata, require its current path to be this window's `/dev/null` link, remove the link and move the original back. Do not start the triggers yet. |
| ```bash |
| for unit in "${SERVICES[@]}" "${TRIGGERS[@]}"; do |
| @@ -2828,7 +2971,7 @@ Expected: the edge `masked` and inactive; the services and triggers loaded and i |
| ## Step 9. Serving under the claim |
| -Paul's go: required, after step 8's migration, seed and retained-state checks pass. Keep the external traffic barrier until then. |
| +Paul's go: required, after step 8's migration, 8.2b and retained-state checks pass (gate G8). Keep the external traffic barrier until then. The dev console stays claims-off through step 9; block B4 turns it on afterwards. |
| In the same attended root shell on leo: restore the edge original, reload, inspect the effective normal guarded agent command and its recovery dependency, restore its recorded enablement, start it last among services, then restore the triggers that were active before (all five on the recorded baseline): |
| @@ -2855,18 +2998,113 @@ Expected: |
| - the edge `active` with its `guard-exec` command; the twelve units match the baseline in P8; |
| - `REFUSING CARLOS_SWAP=overlap (host-claims-on)` once in the edge journal; |
| - no "signed roster has no home region" line; |
| -- `carlos system status --console https://platform.dev.carlosframework.com` shows leo current. |
| +- `carlos system status --console https://platform.dev.carlosframework.com` shows leo current; |
| +- leo's next box report (`status/boxes/leo-climbing.json`) is fresh, verifies, and carries `host_claim_enabled` (gate G9). |
| On failure: stop and hold. |
| +## B4. Console claims on, then `claimtrial2` (after step 9) |
| + |
| +Paul's go: one go for block B4 and acceptance 1 to 4 (D6), after step 9 has passed (gate G9) and the barrier is lifted. Fable section 3B names this the one untested dev path: claim-mode `apps create`, `ship`, `promote`, `instances enable --pool` and `instances create`. A refusal there comes before any claim write, and the console can be switched off again, so the blast radius is `claimtrial2` alone. |
| + |
| +**Why the timing matters (H10, #312).** `instances create` on a claim-mode console needs the console's compute evidence, which `console-report` puts into sophie's box report. The agent rewrites that report whole at its next status tick, about five minutes, and the entry is gone. So `instances create` must succeed within one agent tick of `console-report`: five minutes, not six. Everything from `console-report` to `instances create` runs from one shell, in one go, with `date -u` stamped between commands, and the create gets at most three attempts. |
| + |
| +**1. Before the clock starts (DOCS-T1).** Save 5.1's second helper block (`ssm_run`, `box_get`, `box_put`, `box_inspect`, with `LEO` and `SOPHIE`) as `$TMPDIR/cohort1/box-helpers.sh`, preceded by `R=eu-west-1` and `export AWS_PROFILE=kass`. Fetch and check the build, as step 4 did: |
| + |
| +```bash |
| +aws --profile kass s3 cp s3://carlos-dev-640012953448/apps/bab/wakecheck/releases/wc4/linux-arm64 "$TMPDIR/cohort1/linux-arm64" |
| +sha256sum "$TMPDIR/cohort1/linux-arm64" # ee276c083a04b6de11fc81b3a219f587394efd9600b46d21185398e44a4abcbb |
| +aws --profile kass s3 ls s3://carlos-dev-640012953448/apps/bab/claimtrial2/ |
| +``` |
| + |
| +Expected: the sha256 above, and nothing under `apps/bab/claimtrial2/` (the name is new). |
| + |
| +**2. Console claims on.** On sophie, the "Paused state" resume block, with `box_inspect "$SOPHIE"` from DOCS-T1 first, then 7.4's checks: `claim-console.env` is `5df17bac`, the console `active` with `CARLOS_HOST_CLAIM=1`, and the run sweep and seeds pass lines back in its journal. Wait for `status/consoles/dev-console.json` to move. Stop before the resume block's own `console-report`: step 3 runs it. |
| + |
| +**3. The timed run (DOCS-T1, one shell).** Wait until sophie's box report (`status/boxes/sophie-knitting.json`) has just been rewritten, for the widest window, as 7.2 does. Then run this as one script, outside the sandbox, signed in to the dev console, with `CARLOS_DEPLOYMENT_BUCKET` unset except on the `instances enable` line: |
| + |
| +```bash |
| +cat > "$TMPDIR/cohort1/b4.sh" <<'B4' |
| +set -euo pipefail |
| +. "$TMPDIR/cohort1/box-helpers.sh" |
| +cd "$TMPDIR/cohort1" |
| +DEV=https://platform.dev.carlosframework.com |
| +H2=claimtrial2.dev.carlosframework.com |
| +CR='cd /root/claim-trial && /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 --expect-revision "$(cat rev)" --console-id "$(cat console-id)" --public-key "$(cat release.pub)"' |
| +date -u +%FT%TZ |
| +box_inspect "$SOPHIE" |
| +date -u +%FT%TZ |
| +ssm_run "$SOPHIE" "$CR --dry-run" |
| +date -u +%FT%TZ |
| +ssm_run "$SOPHIE" "$CR" |
| +date -u +%FT%TZ |
| +carlos apps create --app claimtrial2 --account bab --console $DEV |
| +date -u +%FT%TZ |
| +carlos ship --app claimtrial2 --account bab --console $DEV --version wc4 \ |
| + --label "dev claim trial tenant 2, wakecheck wc4 build" --no-prompt "$TMPDIR/cohort1/linux-arm64" |
| +date -u +%FT%TZ |
| +carlos promote --app claimtrial2 --account bab --console $DEV --no-prompt wc4 edge |
| +date -u +%FT%TZ |
| +AWS_PROFILE=kass CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 \ |
| + carlos instances enable --app claimtrial2 --account bab --domain dev.carlosframework.com --pool cohort1 --channel edge |
| +date -u +%FT%TZ |
| +carlos instances create --app claimtrial2 --account bab --host $H2 --console $DEV --no-prompt |
| +date -u +%FT%TZ |
| +B4 |
| +TMPDIR="$TMPDIR" bash "$TMPDIR/cohort1/b4.sh" 2>&1 | tee "$TMPDIR/cohort1/b4-attempt1.log" |
| +``` |
| + |
| +Expected, in order: `console-report` publishes; `apps create` gives the single `edge` channel; `ship` prints `shipped claimtrial2 wc4`; `promote` prints `promoted claimtrial2 wc4 -> edge`; `instances enable` writes `instances.json` with pool `cohort1` and channel `edge`; `instances create` prints `created instance claimtrial2.dev.carlosframework.com`. The stamp after `instances create` is less than five minutes after the stamp before the applied `console-report`. That is attempt 1. |
| + |
| +**If `instances create` is refused** (the console's evidence has gone, or a claim-mode refusal): capture the refusal and the console's journal line for it (`journalctl -u console.service --since -10min` on sophie, read-only). Then attempt 2, and if needed attempt 3, each a fresh report and an immediate create, from the same kind of script: |
| + |
| +```bash |
| +cat > "$TMPDIR/cohort1/b4-retry.sh" <<'B4R' |
| +set -euo pipefail |
| +. "$TMPDIR/cohort1/box-helpers.sh" |
| +cd "$TMPDIR/cohort1" |
| +DEV=https://platform.dev.carlosframework.com |
| +H2=claimtrial2.dev.carlosframework.com |
| +CR='cd /root/claim-trial && /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 --expect-revision "$(cat rev)" --console-id "$(cat console-id)" --public-key "$(cat release.pub)"' |
| +date -u +%FT%TZ |
| +box_inspect "$SOPHIE" |
| +date -u +%FT%TZ |
| +ssm_run "$SOPHIE" "$CR" |
| +date -u +%FT%TZ |
| +carlos instances create --app claimtrial2 --account bab --host $H2 --console $DEV --no-prompt |
| +date -u +%FT%TZ |
| +B4R |
| +TMPDIR="$TMPDIR" bash "$TMPDIR/cohort1/b4-retry.sh" 2>&1 | tee "$TMPDIR/cohort1/b4-attempt2.log" |
| +``` |
| + |
| +Run it at most twice (logs `b4-attempt2.log` and `b4-attempt3.log`). Never change an identifier between attempts, and never loop it. A refusal in any step before `instances create` is not retried: capture it and go to the third-failure action. |
| + |
| +**On a third failed attempt:** switch the console off again ("Paused state", the pause recipe), confirm `/healthz` 200 and the empty `CARLOS_HOST_CLAIM=`, and hold leo ("Stop and hold"). Post the three logs and the console's journal lines on #250. Whatever B4 wrote stays (`claimtrial2`'s app and releases); do not delete it. |
| + |
| +**4. After the create (no clock).** Read the claim and the route, and get the host a certificate as step 4.5 did. Only the housekeeper-lease holder orders certificates, and only when a TLS hello for the name reaches it. Outside the sandbox: |
| + |
| +```bash |
| +aws --profile kass s3 cp s3://carlos-dev-640012953448/control/hosts/claimtrial2.dev.carlosframework.com.json - \ |
| + | jq '{kind: .op.kind, exec: .op.exec, writer: .holder.writer_box}' |
| +aws --profile kass s3 cp s3://carlos-dev-640012953448/control/housekeeper-lease/lock.json - | jq -r '.owner' |
| +openssl s_client -connect 99.80.78.4:443 -servername claimtrial2.dev.carlosframework.com </dev/null |
| +aws --profile kass s3 ls s3://carlos-dev-640012953448/control/certs/claimtrial2.dev.carlosframework.com |
| +leo 'python3 -c " |
| +import sqlite3 |
| +c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| +for r in c.execute(\"select host,app,pool,hibernate from routes order by host\"): print(r)"' |
| +``` |
| + |
| +Expected: a claim serving on `leo-climbing`, written by `leo-climbing` (the field names are from the step 8 delta's adoption wait, `after236/step8-readiness-runbook-delta.md`; confirm them against the branch); the lease owner is sophie (`ip-172-31-14-16`), so the hello goes to sophie's address; the certificate object appears; leo's registry has one row, `claimtrial2.dev.carlosframework.com`, pool `cohort1`; sophie's registry has none. Then acceptance, step 10. |
| + |
| ## Step 10. Acceptance |
| -Paul's go: for the reboot and the signed update. |
| +Paul's go: one go for acceptance 1 to 4, with block B4 (D6); one each for 10.5, the reboot, and 10.6, the signed update. |
| -The acceptance host is the disposable app, `HOST=claimtrial.dev.carlosframework.com`. Never run these checks on wakecheck or a real app. Every curl runs outside the sandbox. |
| +The acceptance host is the disposable app made in block B4, `HOST=claimtrial2.dev.carlosframework.com`. Never run these checks on wakecheck or a real app. Every curl runs outside the sandbox. Acceptance 1 to 4 are gate G11. |
| -1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180` and `hits=4` (the three seeded rows survived), and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| -2. First cold wake. Wait for it to sleep: `carlos logs --app claimtrial --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before. |
| +1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180` and a `hits=` value, and on leo a `carlos-instance-gen@$HOST:<n>` unit active. The first value is not 1: the app's first wake writes rows of its own (probe requests), as step 4 found. Record it. |
| +2. First cold wake. Wait for it to sleep: `carlos logs --app claimtrial2 --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before. |
| 3. Second cold wake. Repeat. Expect 200 and `hits=` one higher again. |
| 4. Isolation. While it is awake: |
| @@ -2876,11 +3114,11 @@ The acceptance host is the disposable app, `HOST=claimtrial.dev.carlosframework. |
| Expect only the host's own directory, and two different mount namespaces. `carlos status --layout` shows the host `nested`. |
| 5. Reboot. `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| -6. Signed repair or update. Apply the approved signed update to leo with `carlos system update`; confirm the box comes back and `claimtrial` wakes. |
| +6. Signed repair or update. Apply the approved signed update to leo with `carlos system update`; confirm the box comes back and `claimtrial2` wakes. |
| -Pass: all six. Record each response and the journal lines on #250. |
| +Pass: all six. Record each response and the journal lines on #250. After acceptance, D9: Fable recommends switching the dev console off again (the pause recipe) so dev's pool-less creates work, then recreating wakecheck (see "Trial assumptions"). |
| -If `claimtrial`'s wake-retry budget is exhausted, the reset goes through the console API (the reset CLI does not expose it yet). Stage a protected curl config with the operator's bearer header at `/root/claim-trial/operator-curl.conf` (never in shell arguments or logs), `reset-input.json` with the exact `incarnation`, `episode` and a unique retained `request_id`, and `acceptance-host` with the hostname. On the trusted operator host: |
| +If `claimtrial2`'s wake-retry budget is exhausted, the reset goes through the console API (the reset CLI does not expose it yet). Stage a protected curl config with the operator's bearer header at `/root/claim-trial/operator-curl.conf` (never in shell arguments or logs), `reset-input.json` with the exact `incarnation`, `episode` and a unique retained `request_id`, and `acceptance-host` with the hostname. On the trusted operator host: |
| ```bash |
| set -euo pipefail |
| @@ -2901,26 +3139,23 @@ A 200 acknowledges the request; it does not prove a wake. Check the new retry ep |
| **Rollback slot after session 2 (2026-10-19).** 6.1a's `install-channel --policy --identity` put the v0.28.4 bytes (`736826e4`) into both `/opt/carlos/carlos` and `/opt/carlos/carlos.prev` on leo and sophie. Neither box can roll back to v0.28.3 from its own slot. To recover v0.28.3 on a box, fetch it again from the channel. Promote v0.28.3 on that box's system channel, `stable` for leo or `canary` for sophie, with `carlos system update`. The box's update path then downloads and verifies the v0.28.3 router from the bucket (`bootstrap/router/v0.28.3/carlos-linux-arm64`, sha256 `6154afb4…`) and stages it. Do not copy a binary in by hand. |
| -This replaces a claims-off rollback for the trial. If anything from step 5 on fails, or acceptance fails, stop and leave leo as it is for diagnosis. The one exception is step 8 stopped before 8.2's `host.env` edit, which is undone (ruling R1, below). Nothing on leo is undone, repaired or retried until platform-1e has looked and Paul has decided what comes next. Do not unset the claim flags, unbind the census, delete records or guards, or attempt a data-preserving rollback. Only `claimtrial` is affected, and it holds nothing of value. |
| +This replaces a claims-off rollback for the trial. If anything from step 5 on fails, or acceptance fails, stop and leave leo as it is for diagnosis. The one exception is step 8 stopped before 8.2's `host.env` edit, which is undone (ruling R1, below). Nothing on leo is undone, repaired or retried until platform-1e has looked and Paul has decided what comes next. Do not unset the claim flags, unbind the census, delete records or guards, or attempt a data-preserving rollback. Under Option B leo holds no host until block B4, then only `claimtrial2`, which holds nothing of value. |
| + |
| +**Before the window (blocks B5, B1 and B2).** Nothing claim-mode has run; leo and the console are claims-off. A failed B1 or B2 check is a hold (gate G5): change nothing more, and leave what B1 deleted and what B2 moved aside as it is. Never put a moved path back by hand. |
| Paul's go: not needed to stop. Stopping is always allowed. Anything after it needs a go. |
| **Inside the signing session (5.1 to 7).** Nothing claim-mode runs on leo yet. A stop ends the session: close it at once ("Closing the signing session": shred the key, terminate the signer, delete its role, profile and security group, run the CloudTrail check). Leave leo, sophie, the roster, the floor and the bucket as they are; leo keeps serving claims off. Resuming needs Paul's go for a new signing session, which would put the release key on a host a second time. |
| -**Step 8, before 8.2's `host.env` edit (8.1): undo.** Paul's ruling R1 (2026-10-19). Claims are still off and nothing claim-mode has run, so the window is fully undone, not held. Check that `host.env` still matches `$WINDOW/host.env.before` and that `/var/lib/carlos-claims` does not exist. Then put the twelve unit files back from `$WINDOW/held` and remove the `/dev/null` links, `daemon-reload`, restore the edge's enablement, start the edge and the five triggers, and lift the barrier by re-authorizing ports 80 and 443 on `sg-054aa165db842b6ba`. The exact commands are in the window package, section 5, case B1. If either check fails, it is the next case: hold. |
| - |
| -**Step 8, from 8.2's `host.env` edit until the seed starts (8.2): hold.** The six services, five triggers and the edge are already persistently masked. Leave every mask, `$WINDOW` and `host.env` as they are. Leave instance-control as it is (running if 8.2 started it). |
| +**Step 8, before 8.2's `host.env` edit (8.1 and 8.1b): undo.** Paul's ruling R1 (2026-10-19). A collecting plan 8.1b wrote stays; with the console claims-off it is harmless. Claims are still off and nothing claim-mode has run, so the window is fully undone, not held. Check that `host.env` still matches `$WINDOW/host.env.before` and that `/var/lib/carlos-claims` does not exist. Then put the twelve unit files back from `$WINDOW/held` and remove the `/dev/null` links, `daemon-reload`, restore the edge's enablement, start the edge and the five triggers, and lift the barrier by re-authorizing ports 80 and 443 on `sg-054aa165db842b6ba`. The exact commands are in the window package, section 5, case B1. If either check fails, it is the next case: hold. |
| -**Step 8, while the seed override is in place (8.3).** This is the one fragile state: the edge unit file is back in `/etc`, its guard drop-in is in `$WINDOW/held`, and the seed override lives in `/run`, which a reboot clears. Return to the masked state: |
| -1. At the deadline or on failure, capture `InvocationID` before stopping, as `[runbook.cancellation]` says. |
| -2. Run the first block of 8.4 (stop the one-shot, save the final result and journal, remask the edge, remove the seed override, put the guard drop-in back). |
| -3. Do not run the second block of 8.4: the services and triggers stay masked. |
| +**Step 8, from 8.2's `host.env` edit until 8.4 (8.2 and 8.2b): hold.** The six services, five triggers and the edge are already persistently masked. Leave every mask, `$WINDOW` and `host.env` as they are. Leave instance-control as it is (running if 8.2 started it). A refused 8.2b leaves the plan `collecting`; leave it, and close the signing session. |
| -If leo reboots or the operator session is lost while the override is in place, re-establish the masks and review the actual effective units before anything else. |
| +**Step 8, a seed override found.** Under Option B there is no 8.3 and nothing writes a seed override. If one is ever found at `$SEED`, start nothing, capture the effective units, and report: the window package, section 5, case C. The 2026-10-19 plan's steps for that state are kept there. |
| **Step 8, after 8.4.** The edge is masked; nothing more to stop. |
| -**Steps 9 and 10.** Keep the traffic and admission barrier, and stop the edge, which stops admission and every tenant. Then stop the triggers and put the persistent edge mask back with a fresh copy, so the hold survives a reboot: |
| +**Step 9, block B4 and step 10.** If block B4 failed three times, or the console's claims are on when the hold starts, first switch the console off again ("Paused state", the pause recipe) unless platform-1e asks to keep it on for diagnosis. Keep the traffic and admission barrier, and stop the edge, which stops admission and every tenant. Then stop the triggers and put the persistent edge mask back with a fresh copy, so the hold survives a reboot: |
| ```bash |
| systemctl stop carlos-edge.service |
| @@ -2947,7 +3182,7 @@ cat /proc/sys/kernel/random/boot_id |
| > Before disabling claims, record outstanding park failures and retained disk space. A box that has enrolled a census still maintains its retained root state during update-time recovery, even when claims are off. The no-census case remains separate. Do not widen unit permissions or delete retained paths by hand; use an approved recovery procedure. |
| -What "hold" leaves running: the operating system (systemd, journald, sshd, the SSM agent), so the box stays reachable for read-only diagnosis, and `carlos-instance-control.service`, the root service that keeps recovery and retained state consistent. Stop instance-control too (`systemctl stop carlos-instance-control.socket carlos-instance-control.service`) only if it is itself failing or platform-1e asks. Sophie, its console and dennis carry on as normal; sophie keeps serving wakecheck. |
| +What "hold" leaves running: the operating system (systemd, journald, sshd, the SSM agent), so the box stays reachable for read-only diagnosis, and `carlos-instance-control.service`, the root service that keeps recovery and retained state consistent. Stop instance-control too (`systemctl stop carlos-instance-control.socket carlos-instance-control.service`) only if it is itself failing or platform-1e asks. Sophie, its console and dennis carry on as normal. (Under Option B wakecheck's record is deleted in block B1, so nothing serves it until it is recreated after the trial.) |
| What must not be touched while leo is held: |
| - `$WINDOW` (`/etc/carlos/claim-window-cohort1-*`) and everything in `$WINDOW/held`, including the original unit files and `host.env.before`; |
| @@ -2956,7 +3191,7 @@ What must not be touched while leo is held: |
| - `/var/lib/carlos-claims`, including migration state, journals and parks; |
| - `/etc/carlos/host-actor.json`, `/etc/carlos/host-evidence.key`, `/etc/carlos/fence-writer/` and `/opt/carlos/system/guard`; |
| - `/opt/carlos/carlos`, `/opt/carlos/carlos.prev` and anything staged: no `carlos system update`, rollback or old-road roll, and no `stable`-channel roll while leo is held, because it is on `stable` with dennis; |
| -- `/data`, including `registry.db`, `/data/instances` and any parked database; |
| +- `/data`, including `registry.db`, `/data/instances` and any parked database, and block B2's `/var/lib/carlos-aside-*` directories; |
| - sophie's protected console (`/opt/carlos/system/console/console` and `console.prev`), every published identity (`control/system/router/releases/v0.28.2.json`, unused; `control/system/router/releases/v0.28.4.json`; and `v0.28.3.json` if one ever appears, unused), every staged `policy*.json` and `identity*.json`, `/opt/carlos/system/router/channel-identity.json`, and the `bab` pointer-digest marker; |
| - the roster, floor and enrollments, and the claim's S3 objects: `control/hosts/`, `control/host-actors.json`, `control/host-origins/`, admitted prefixes, and `instances/` in `carlos-dev-ie-640012953448`; |
| - no `systemctl reset-failed` before the failed results are recorded, no reboot unless the edge is persistently masked, no rerun of `journal-migrate`, `install-control` or the seed, and no console `reset-wake-retry`. |
| @@ -2965,7 +3200,7 @@ Leaving hold is a separate decision: a fix and resume, or a claims-off teardown |
| ## Known hazards |
| -**H1. wakecheck (#286).** wakecheck leaves leo through `CARLOS_POOLS=cohort1` (step 4) and stays on sophie, claims off, as today. Its record is unregioned, so both home boxes mint it. With leo claimed and sophie not, the same host would have two writers on different lineages. DNS for `*.dev.carlosframework.com` points at sophie and bots wake wakecheck there, so it cannot be kept to leo, and a wake on sophie during the window would break the drained, replicated stop the seed needs. |
| +**H1. wakecheck (#286).** wakecheck leaves leo through `CARLOS_POOLS=cohort1` (step 4) and stays on sophie, claims off, as today. Its record is unregioned, so both home boxes mint it. With leo claimed and sophie not, the same host would have two writers on different lineages. DNS for `*.dev.carlosframework.com` points at sophie and bots wake wakecheck there, so it cannot be kept to leo, and a wake on sophie during the window would break the drained, replicated stop the seed needs. Under Option B this hazard is removed for the trial: block B1 deletes wakecheck's record, sophie retires its row, and wakecheck is recreated only after the trial, with the console claims-off again (D9). |
| **H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Tell them before step 4. After step 9 leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo for the rest of the trial, including any hold. Their test apps would also need `--pool cohort1`. |
| @@ -2975,15 +3210,15 @@ Leaving hold is a separate decision: a fix and resume, or a claims-off teardown |
| **H5. Runtime masks lose to `/etc`.** Never use `systemctl mask --runtime` or `unmask --runtime` on leo. Use only the moved-file masks of step 8 and check `LoadState=masked`. |
| -**H6. The seed override window.** Between the two halves of 8.3 and 8.4 the edge is unmasked, its guard drop-in is out, and the seed override sits in `/run`. A reboot then removes the override and leaves an unguarded, disabled edge unit. Keep this stretch short, attended and inside the deadline. |
| +**H6. The seed override window.** In the 2026-10-19 plan, between the two halves of 8.3 and 8.4 the edge was unmasked, its guard drop-in out, and the seed override in `/run`. A reboot then removes the override and leaves an unguarded, disabled edge unit. Under Option B there is no seed, so this stretch never happens; the edge stays masked from 8.1 to step 9. An override found anyway is section 5's case C. |
| -**H7. The dev console as an actor.** Step 6 arms the guard on sophie with `--console`, and steps 6 and 7 restart `console.service`, so the dev console is briefly down each time. Sophie's own host claims stay off. The enrolled console is what allows an operator `reset-wake-retry` for `claimtrial`. |
| +**H7. The dev console as an actor.** Step 6 arms the guard on sophie with `--console`, and steps 6 and 7 restart `console.service`, so the dev console is briefly down each time. Sophie's own host claims stay off. The enrolled console is what allows an operator `reset-wake-retry` for `claimtrial2`. |
| **H8. Memory.** Leo has 927 MB. Claim mode raises instance-control's cap to 256M. Watch `systemctl show carlos-instance-control.service -p MemoryCurrent` during step 10. |
| **H9. No new tools on the boxes.** `jq` is not installed on leo, sophie or dennis (2026-10-17), and nothing is installed for the trial (platform-1e). Every `jq` line runs on the trusted signing host, which fetches box JSON over SSM and stages parsed results back (steps 5 to 7). Staged values are checked by sha256, and the box commands verify the roster's signature and revision. |
| -**H10. The console's evidence is short-lived (#312).** The agent writes sophie's box report whole at each status tick (5 minutes plus jitter), so the console process entry `console-report` adds is gone at the next tick. `publish-floor` must follow `console-report` at once (step 7.2). Anything later that reads the console's compute evidence sees it vanish within about 6 minutes. |
| +**H10. The console's evidence is short-lived (#312).** The agent writes sophie's box report whole at each status tick (5 minutes plus jitter), so the console process entry `console-report` adds is gone at the next tick. `publish-floor` must follow `console-report` at once (step 7.2). Anything later that reads the console's compute evidence sees it vanish at the next tick. Plan for five minutes, not six (Fable section 11): block B4's `instances create` must succeed within one tick of its `console-report`, in at most three attempts. |
| **H11. `install-channel --policy --identity` drops the rollback.** Step 6.1a replaces `carlos.prev` (v0.28.3) with v0.28.4 on leo and sophie. Afterwards neither box can roll back to v0.28.3, and once armed, the guard refuses anything below the v0.28.4 floor anyway. A broken v0.28.4 on those boxes is fixed forward with a new release, never by a rollback. |