docs: cohort-1 trial runbook refreshed against main 26065378
- Merged origin/main. The runbook now points at main's carlos system claims (inspect, run, evidence-key, fence-credentials, console-config, console-report, install-control); flags and help checked against activation_command*.go and the runner's phases. - The automated offline window is gone. Step 8 is the attended manual window from activation-tools-report run 11: persistent masks by moving the twelve /etc unit files aside and linking /dev/null, install-control --dry-run as the verifyOfflineRootMigration check, journal-migrate, root start, the seed one-shot with the guard drop-in handled, remask and restore. The approved files seed strings are back in the window steps, verbatim; the two runtime-mask commands are marked as replaced. - Order: release and roll claims off (dev first), apps/bab// cleanup, IAM, approved pool + disposable app, roster/enrollment, guards and console, readers/floor/home/key/fence credentials, window, serving go, acceptance. Stop and hold rewritten for the manual window. - Prechecks re-run read-only on 2026-10-17: leo has only wakecheck (proof routes gone); all boxes still v0.27.3, which lacks 26065378 and e8e961dd; release-key explicitDeny holds; leo's role still lacks version actions; 267 objects remain under apps/bab//, none written since 2026-10-16 15:46; the twelve window units are root-owned files in the expected states; jq is absent on all dev boxes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+595
−441
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+595 −441
CI cancelled — superseded by a newer push — run details
CI log
=== gate === cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.011s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index cea57f158..d20fb3b67 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -1,37 +1,43 @@ |
| -# Host claim activation, dev cohort 1: runbook draft |
| +# Host claim activation, dev cohort 1: mechanics trial runbook |
| -Draft, 2026-10-16, revised the same day for Paul's decisions. It authorizes nothing. Every change below waits for Paul's go at its own checkpoint, and the copy-approved strings quoted in step 5 are not an approval to run them. |
| +Draft, first written 2026-10-16, revised 2026-10-17 against main at 26065378. It authorizes nothing. Every change below waits for Paul's go at its own checkpoint. Neither the approved copy quoted in step 8 nor the activation tools' tests authorize running anything. |
| Dev cohort 1 is a mechanics trial (Paul, 2026-10-16, relayed by platform-1e): |
| -- The scope is one box, `carlos-dev-ie-b` (leo-climbing). `carlos-dev-uk` is not part of this trial; it stays a later option (see the end of this document). |
| +- The scope is one box, `carlos-dev-ie-b` (leo-climbing). `carlos-dev-uk` is not part of this trial; it stays a later option. |
| - Acceptance runs on a disposable app made for the trial, never on wakecheck or any real app. |
| -- On failure the box is stopped and held for diagnosis, as described in "Stop and hold". There is no claims-off rollback in this trial. |
| +- On failure the box is stopped and held for diagnosis ("Stop and hold"). There is no claims-off rollback in this trial. |
| - Plan 15's independent claim writer and key rotation are deferred for the trial only. They are accepted trial gaps, not blockers, and must be closed before any non-trial cohort. |
| -The dev console box, `carlos-dev-ie` (sophie-knitting), is not activated. It carries the `platform.dev.carlosframework.com` route with `unit=console.service`, and Plan 14 Part 3's activation preflight refuses any box with a named-unit or `backing=unit` route. |
| +The dev console box, `carlos-dev-ie` (sophie-knitting), is not activated. It carries the `platform.dev.carlosframework.com` route with `unit=console.service`, and the named-unit activation preflight refuses any box with a named-unit or `backing=unit` route until Plan 14 item 4a lands. Sophie takes part only as the console's compute host, with its host claims off. |
| + |
| +What changed on 2026-10-17: |
| +- Plan 14 Parts 2 and 3 and the activation tools are on main (26065378). Claims stay off everywhere. |
| +- `carlos system claims` on main has `inspect`, `run`, `evidence-key`, `fence-credentials`, `console-config`, `console-report` and `install-control`. The automated offline window (`offline-begin`, `offline-seed`, `offline-restore`) was removed. Leo's window is the files seed's approved runbook again, with manual masks that persist, and `verifyOfflineRootMigration` as the strict check. |
| +- The `proof*` routes are gone from leo, and the empty-app writer fix (e8e961dd) is on main. |
| Sources: |
| -- the binding scope, `activation-scope-2026-10-16.md` (posted on amadan #250 and #280; the file itself lives in platform-1e's scratch, not in the repo); |
| +- the binding scope, `activation-scope-2026-10-16.md` (posted on amadan #250 and #280); |
| - Plan 9 Task C2, milestones 2 to 4 (`docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md` on `design-replica-ownership`); |
| - `docs/host-claim-restore-runbook.md` on main, including "Home-region activation order"; |
| -- the files seed, on main as ee35dee2, and its runbook strings, which Paul approved unchanged on 2026-10-15 (copy review `files-seed-copy-review/result.json` in platform-1e's scratch, action `approve`); |
| -- Plan 14 Parts 2 and 3, unmerged, on `feat/claim-root-trusted-state-p3` at 305866ad: `claimactivation_preflight.go`, `rootmigration*.go` and `claimjournal_migration.go`; |
| -- platform-1e's activation tools, unmerged, on `feat/claim-activation-tools` at 0e81652a, and their report `activation-tools-report.md` (run 2) in platform-1e's scratch; |
| +- main's `internal/host/activation_command.go` and `activation_command_linux.go` (command flags and help), `internal/claimactivation/runner.go` (plan phases), `internal/host/rootmigration_production_linux.go` (`verifyOfflineRootMigration`, `rootMigrationOfflineUnits`) and `docs/site/cli.md` "carlos system claims"; |
| +- platform-1e's `activation-tools-report.md`: run 11 (2026-10-17) for leo's attended window, and run 2 for the operator sequence of the retained commands, which run 11 states are byte-identical on main; |
| +- the files seed runbook strings, which Paul approved unchanged on 2026-10-15 (copy review `files-seed-copy-review/result.json`, action `approve`); |
| +- `docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash` on main; |
| - amadan #250, #272, #280 and #286. |
| ## Boxes |
| -| Name | Box | Instance | Region | Role | Today | |
| +| Name | Box | Instance | Region | Role | In the trial | |
| |---|---|---|---|---|---| |
| | `carlos-dev-ie-b` | leo-climbing | i-046c7fe43620944bc | eu-west-1, home, no `CARLOS_REGION` | `carlos-dev-host-b` | the trial box | |
| -| `carlos-dev-uk` | dennis-stenciling | i-009d593f451cfd562 | eu-west-2, `CARLOS_REGION=uk` | `carlos-dev-host-uk` | not in this trial; later option | |
| -| `carlos-dev-ie` | sophie-knitting | i-05498eada6f0afd23 | eu-west-1, home | `carlos-dev-host` | not activated (console box); its console is enrolled as an actor | |
| +| `carlos-dev-ie` | sophie-knitting | i-05498eada6f0afd23 | eu-west-1, home | `carlos-dev-host` | console actor only; host claims off | |
| +| `carlos-dev-uk` | dennis-stenciling | i-009d593f451cfd562 | eu-west-2, `CARLOS_REGION=uk` | `carlos-dev-host-uk` | no; later option | |
| -Dev account 640012953448. AWS profile `kass` (a plain IAM user, so it works inside the sandbox). Deployment bucket `carlos-dev-640012953448`. Data buckets `carlos-dev-ie-640012953448` and `carlos-dev-uk-640012953448`. Console `https://platform.dev.carlosframework.com`. |
| +Dev account 640012953448. AWS profile `kass` (a plain IAM user, so it works inside the sandbox). Deployment bucket `carlos-dev-640012953448`. Data bucket for leo `carlos-dev-ie-640012953448`. Console `https://platform.dev.carlosframework.com`. Home roster name `ie`; AWS region `eu-west-1`; home boxes leave `CARLOS_REGION` unset. |
| ## How commands are run |
| -Box commands run as root over SSM. Save this helper as `$TMPDIR/cohort1/ssm.sh`: |
| +Read-only checks and single commands run as root over SSM. Save this helper as `$TMPDIR/cohort1/ssm.sh`: |
| ```bash |
| #!/bin/bash |
| @@ -50,11 +56,13 @@ aws --profile kass --region "$region" ssm get-command-invocation --command-id "$ |
| `leo` below means `$TMPDIR/cohort1/ssm.sh eu-west-1 i-046c7fe43620944bc`. Delete `$TMPDIR/cohort1` by exact path when the work is done. |
| +The multi-block procedures in steps 5 to 8 (`/root/claim-trial` work and the offline window) need an attended root shell on the box, because later blocks reuse variables from earlier ones. Open it through the approved administration transport, for example `aws --profile kass --region eu-west-1 ssm start-session --target i-046c7fe43620944bc`, then a root shell. Check the output between blocks; never paste a whole section as an unattended script. |
| + |
| Never print a secret. Key checks print a presence flag and the first 8 hex digits of a SHA-256, never the value. |
| ## Prechecks |
| -All of these are read-only. The results recorded here were taken on 2026-10-16 between 07:20Z and 07:45Z. |
| +All read-only. First run 2026-10-16 07:20Z to 07:45Z; the checks that could have changed were re-run on 2026-10-17 at 20:10Z. Each result below says which. |
| ### P1. Release public key on the box |
| @@ -64,43 +72,34 @@ leo 'grep -c "^CARLOS_RELEASE_PUBKEY=" /etc/carlos/host.env; grep "^CARLOS_RELEA |
| Pass: `1`, and the same fingerprint on every dev box. |
| -Today: PASS. Present once on leo, sophie and dennis, fingerprint `960d03ff` on all three. Step 2 still proves that it is the key that signs dev releases and the roster. |
| +2026-10-17: PASS. Present once on leo, sophie and dennis, fingerprint `960d03ff` on all three. Step 1's signed update proves it is the key that signs dev releases; the same key signs the roster. |
| ### P2. IAM and actor certification |
| ```bash |
| -aws --profile kass iam list-role-policies --role-name carlos-dev-host-b |
| aws --profile kass iam get-role-policy --role-name carlos-dev-host-b --policy-name carlos-dev-host-b-buckets \ |
| --query PolicyDocument --output json | grep -c 'GetObjectVersion\|DeleteObjectVersion\|ListBucketVersions' |
| aws --profile kass s3api get-bucket-policy --bucket carlos-dev-640012953448 --query Policy --output text \ |
| | python3 -c 'import json,sys;print([s.get("Sid") for s in json.load(sys.stdin)["Statement"]])' |
| aws --profile kass iam list-roles --query 'Roles[?contains(RoleName, `fence`)].RoleName' --output text |
| -for b in carlos-dev-640012953448 carlos-dev-ie-640012953448 carlos-dev-uk-640012953448; do |
| - aws --profile kass s3api get-bucket-versioning --bucket $b --output text; done |
| +for b in carlos-dev-640012953448 carlos-dev-ie-640012953448; do aws --profile kass s3api get-bucket-versioning --bucket $b --output text; done |
| +for r in carlos-dev-host-b carlos-dev-host; do aws --profile kass iam simulate-principal-policy \ |
| + --policy-source-arn arn:aws:iam::640012953448:role/$r --action-names ssm:GetParameter \ |
| + --resource-arns arn:aws:ssm:eu-west-1:640012953448:parameter/carlos/release-key \ |
| + --query 'EvaluationResults[0].EvalDecision' --output text; done |
| aws --profile kass s3api head-object --bucket carlos-dev-640012953448 --key control/host-actors.json |
| -leo 'for p in /etc/carlos/host-actor.json /etc/carlos/host-evidence.key /etc/carlos/fence-writer/credentials.json /opt/carlos/system/guard; do [ -e $p ] && echo "present $p $(stat -c "%U %a %h" $p)" || echo "absent $p"; done' |
| -``` |
| - |
| -Pass, once step 3 is done: |
| -- `carlos-dev-host-b-buckets` grants `s3:GetObjectVersion`, `s3:DeleteObjectVersion` and `s3:ListBucketVersions` on the IE data bucket (count 3 or more); |
| -- the deployment bucket policy has `BoxRecordsAreCreateOnly`, `BoxRecordsAreNeverDeleted` and `OnlyBoxRolesWriteTerminationFences`, the last naming a dev fence-writer role; |
| -- a dedicated fence-writer role exists for leo; |
| -- all three buckets report `Enabled`; |
| -- `control/host-actors.json` exists, is signed by the release key and has leo's row; |
| -- on leo, `/etc/carlos/host-actor.json` is present, `/etc/carlos/host-evidence.key` is `root 600 1`, the fence-writer credentials are `root 400 1`, and `/opt/carlos/system/guard` exists. |
| - |
| -Today: FAIL, which is expected before step 3 apart from the first two items. |
| -- `carlos-dev-host-b-buckets` has no version actions (count 0). Leo's role predates the Plan 2 module statements. The roles `carlos-dev-host` (sophie) and `carlos-dev-host-uk` (dennis) already have them. |
| -- The deployment bucket has no bucket policy (`NoSuchBucketPolicy`), so the region HA evidence rules and M-2's fence-writer rule are absent. |
| -- No fence-writer role exists. |
| -- Versioning is `Enabled` on all three buckets. PASS. |
| -- `control/host-actors.json` does not exist (404). Dev has no signed roster. |
| -- None of the four local files exists on leo. |
| -- RESOLVED 2026-10-16 (platform-infrastructure#44, Paul's approval): box roles can no longer read `/carlos/release-key`, the deployment root signing key that signs the roster and the floor. Re-checked read-only the same day by reading the role policies and simulating `ssm:GetParameter` (`aws iam simulate-principal-policy`), with no parameter read and nothing decrypted: |
| - - `carlos-dev-host-b` (leo) and `carlos-dev-host` (sophie): `explicitDeny` on `arn:aws:ssm:eu-west-1:640012953448:parameter/carlos/release-key`, from the new inline policy `deny-release-key-reads` (it also denies `GetParameters`, `GetParameterHistory`, and `GetParametersByPath` on `/carlos`). `/carlos/secrets-key` stays `allowed`. |
| - - `carlos-dev-host-uk` (not in the trial): `explicitDeny` on the eu-west-1 key through its region scope. The simulator says `allowed` for a `/carlos/release-key` in eu-west-2, but no such parameter exists there (`describe-parameters`, metadata only). Do not copy the key into eu-west-2. |
| - - The flagship half (carloku-infrastructure#11) was not checked from here, because this task does not touch flagship. |
| -- Accepted trial gap: leo's role may `s3:PutObject` anywhere in the deployment bucket, including `control/hosts/`, so Plan 15's independent claim-writer boundary is not in place on dev. Deferred for the trial (see "Accepted trial gaps"). |
| +``` |
| + |
| +Pass, once steps 3 and 5 to 7 are done: version actions count 3 or more; the bucket policy has `BoxRecordsAreCreateOnly`, `BoxRecordsAreNeverDeleted` and `OnlyBoxRolesWriteTerminationFences` naming leo's fence-writer role; a fence-writer role exists for leo; both buckets `Enabled`; both roles `explicitDeny` on the release key; `control/host-actors.json` exists. |
| + |
| +2026-10-17: |
| +- Release key: PASS, re-run today. `carlos-dev-host-b` and `carlos-dev-host` both `explicitDeny`, from the inline policy `deny-release-key-reads` (platform-infrastructure#44, Paul's approval). Checked by simulation only; no parameter was read or decrypted. On 2026-10-16 `carlos-dev-host-uk` was also denied for the eu-west-1 key, and no copy of the key exists in eu-west-2; do not copy it there. The flagship half (carloku-infrastructure#11) was not checked from here. |
| +- Version actions on leo's role: FAIL, re-run today, count 0. Sophie's and dennis's roles have them. |
| +- Deployment bucket policy: FAIL, re-run today, `NoSuchBucketPolicy`. |
| +- Fence-writer role: FAIL, re-run today, none. |
| +- Versioning: PASS (2026-10-16), `Enabled`. |
| +- Roster: none yet, re-run today, 404. Expected before step 5. |
| +- Accepted trial gap: leo's role may `s3:PutObject` anywhere in the deployment bucket, including `control/hosts/`. |
| ### P3. Every roster reader's version |
| @@ -111,9 +110,9 @@ leo '/opt/carlos/carlos version; [ -e /opt/carlos/carlos.prev ] && /opt/carlos/c |
| carlos version |
| ``` |
| -Run the box line on sophie and dennis too. Pass: every box, console, `.prev` image and operator CLI is on a build containing Plan 13 and its follow-ups (2ad5412c, in every tag since v0.26.1). Before step 4b, every one of them is on the cohort release. |
| +Run the box line on sophie and dennis too. Pass: before step 7, every box, `.prev` image, console and operator CLI is on the trial release. |
| -Today: PASS for Plan 13. Leo v0.27.3 with `.prev` v0.27.2; sophie v0.27.3 with `.prev` v0.27.2; dennis v0.27.3 with no `.prev`; console `e15ad6be` (v0.27.3); local CLI v0.27.2. `system status`: dennis and leo on stable, sophie on canary, all `current`. The cohort release does not exist yet: v0.27.3 has neither the files seed (ee35dee2) nor Plan 14 Parts 2 and 3. |
| +2026-10-17, re-run today: leo v0.27.3 (`.prev` v0.27.2), sophie v0.27.3 (`.prev` v0.27.2), dennis v0.27.3 (no `.prev`), console `e15ad6be` (v0.27.3), local CLI v0.27.2. `system status`: leo and dennis on `stable`, sophie on `canary`, all `current`. v0.27.3 is still the latest release, and it contains neither 26065378 (Plan 14 Parts 2 and 3 and the activation tools) nor e8e961dd (the empty-app writer fix). Step 1 fixes that. Every one of these builds already contains Plan 13, so the home designation order is safe once they are all on the trial release. |
| ### P4. Leo is empty or quiet |
| @@ -122,18 +121,14 @@ leo 'python3 -c " |
| import sqlite3 |
| c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| for r in c.execute(\"select host,app,kind,unit,backing,addr,hibernate from routes order by host\"): print(r)" |
| -for h in proof.dev.carlosframework.com proof.dev.oncarlos.com proof2.dev.oncarlos.com wakecheck.dev.carlosframework.com; do |
| - echo "$h last=$(journalctl -u carlos-edge.service --no-pager | grep "$h" | tail -1 | cut -c1-15)"; done |
| -ss -ltn | grep -c ":8099 "; systemctl list-units --no-legend --plain "carlos-instance-gen@*" "carlos-instance@*"' |
| +systemctl list-units --no-legend --plain "carlos-instance-gen@*" "carlos-instance@*"; pgrep -c -u carlos' |
| ``` |
| -Pass: only known routes; no instance unit running; no recent request lines. |
| +Pass: only known routes; no instance unit running. |
| -Today: PASS. Four routes: |
| -- `proof.dev.carlosframework.com`, `proof.dev.oncarlos.com` and `proof2.dev.oncarlos.com`: kind `instance`, empty app, address-routed to `127.0.0.1:8099`. Nothing listens there, and the edge journal last names them on Aug 28. |
| -- `wakecheck.dev.carlosframework.com`: app `wakecheck`, `hibernate=1`, nested layout. Asleep: no instance unit runs, and its only recent journal lines are the five-minute provision passes. |
| +2026-10-17, re-run today: PASS. One route, `wakecheck.dev.carlosframework.com` (app `wakecheck`, `hibernate=1`, asleep, no instance unit). The three `proof*` routes are gone. One process runs as the carlos user (the edge agent). |
| -### P5. No named-unit or unit-backed work on leo |
| +### P5. No named-unit or unit-backed work, and a clean bucket app tree |
| ```bash |
| leo 'python3 -c " |
| @@ -144,83 +139,75 @@ find /etc/carlos/apps -mindepth 1 | wc -l |
| find /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system -name "carlos-app@*" | grep -vcE "/carlos-app@\.(service|socket)$"' |
| aws --profile kass s3 ls --recursive s3://carlos-dev-640012953448/apps/ | awk '{print $4}' | grep -E '/(instances|sidecars)/[^/]+\.json$' |
| aws --profile kass s3 ls s3://carlos-dev-640012953448/apps/bab/ |
| +aws --profile kass s3 ls --recursive 's3://carlos-dev-640012953448/apps/bab//' | wc -l |
| ``` |
| -These mirror Part 3's `claimActivationPreflight` (registry), `claimActivationVerified` (bucket records for leo's region and pools) and `claimActivationLocalUnits` (unit files). Pass: `0`, `0`, `0`; every listed record has an empty `unit`; and every `PRE` entry under `apps/bab/` is a valid app name. |
| +These mirror main's `claimActivationPreflight` (registry), `claimActivationVerified` (bucket records for leo's region and pools) and `claimActivationLocalUnits` (unit files). Pass: `0`, `0`, `0`; no listed record has a `unit`; no `PRE /` under `apps/bab/`; `0` objects under `apps/bab//`. |
| -Today: FAIL. |
| -- Registry, local units and records: PASS. No named-unit or `backing=unit` route, `/etc/carlos/apps` is empty, and there is no `carlos-app@<name>` unit. The unit files that mention `/opt/carlos/live/` or `User=carlos` all carry embedded unit names. The bucket holds two records, `bab/wakecheck` and `bab/rollcheck-0261` (static), neither with a unit. |
| -- Bucket app tree: FAIL, a blocker. `apps/bab/` lists `PRE /`, an empty app segment. It holds 286 objects under `apps/bab//errors/leo-climbing/`, the newest written at 06:46Z today. `hostops.Apps` lists `apps/bab/` with a `/` delimiter, meets the empty name and returns `ErrIdentity`. On Part 3 that makes `claimActivationVerified` and the migration census refuse, on every box that uses this deployment bucket. The console's claim runtime and purge passes call the same function. The source is leo's three app-less `proof*` routes. The writer fix, `fix-empty-app-key`, is merging now (blocker 2 below). |
| +2026-10-17: |
| +- Registry: PASS, re-run today, `0` on leo. (Sophie still has 1, its console route, which is why sophie is not activated.) |
| +- Unit files and `/etc/carlos/apps`: PASS (2026-10-16). |
| +- Bucket records: PASS, re-run today. Only `bab/wakecheck` and `bab/rollcheck-0261` (static), neither with a unit. |
| +- Bucket app tree: FAIL, re-run today. `apps/bab/` still lists `PRE /`, with 267 objects under `apps/bab//errors/leo-climbing/`. The newest is from 2026-10-16 15:46:30, so nothing new has been written since the `proof*` routes went. `hostops.Apps` on main still returns `ErrIdentity` on the empty segment, so `claimActivationVerified` and the migration census would refuse. Step 2 cleans it up. |
| ### P6. No claim state yet |
| ```bash |
| -leo 'for p in /var/lib/carlos-claims /data/.hostclaims; do [ -e $p ] && echo "present $p" || echo "absent $p"; done; ls /data/hostguard | wc -l' |
| +leo 'for p in /var/lib/carlos-claims /data/.hostclaims /etc/carlos/host-actor.json /etc/carlos/host-evidence.key /etc/carlos/fence-writer /opt/carlos/system/guard /etc/systemd/system/carlos-edge.service.d; do [ -e $p ] && echo "present $p" || echo "absent $p"; done; ls /data/hostguard | wc -l' |
| ``` |
| -Pass: both absent, `0`. |
| +Pass: all absent, `0`. |
| -Today: PASS on leo, sophie and dennis. With no `/data/.hostclaims` there are no legacy cold journals, so the files seed authority review's L4 check (old serving cold journals) passes too. |
| +2026-10-17: PASS, re-run today. With no `/data/.hostclaims` there are no legacy cold journals to migrate. |
| ### P7. Switches |
| ```bash |
| leo 'for k in CARLOS_SWAP CARLOS_HOST_CLAIM CARLOS_HOST_CLAIM_ORIGINS CARLOS_MOVES_PHASE1 CARLOS_REGION CARLOS_REGION_HA CARLOS_SYSTEM_CHANNEL CARLOS_POOLS; do |
| - grep -qE "^$k=" /etc/carlos/host.env && echo "$k=$(grep -E "^$k=" /etc/carlos/host.env | tail -1 | cut -d= -f2-)" || echo "$k unset"; done |
| -grep -qE "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env && echo "CARLOS_SECRETS_KEY present" || echo "CARLOS_SECRETS_KEY absent"' |
| + grep -qE "^$k=" /etc/carlos/host.env && echo "$k=$(grep -E "^$k=" /etc/carlos/host.env | tail -1 | cut -d= -f2-)" || echo "$k unset"; done' |
| ``` |
| -Pass: `CARLOS_SWAP=overlap`; every claim, moves and region HA switch unset. |
| +Pass: `CARLOS_SWAP=overlap`; everything else unset. |
| -Today: PASS. Leo has `CARLOS_SWAP=overlap`, and `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS`, `CARLOS_MOVES_PHASE1`, `CARLOS_REGION`, `CARLOS_REGION_HA`, `CARLOS_SYSTEM_CHANNEL` and `CARLOS_POOLS` unset. `CARLOS_SECRETS_KEY` is absent on leo, which the narrowed scope allows because moves stay off. |
| +2026-10-17: PASS, re-run today. |
| -### P8. Other box facts the later steps rely on |
| +### P8. The twelve window units, and box facts |
| ```bash |
| -leo 'uname -r; findmnt -no FSTYPE --target /data; df -h /data | sed 1d; free -m | sed -n 2p |
| -stat -c "%U:%G %a %n" / /var /var/lib |
| -find / -xdev -maxdepth 4 -name proof-policy.json | wc -l |
| -systemctl --version | head -1 |
| -for u in carlos-edge carlos-adopt carlos-stage carlos-config-sync carlos-restart carlos-system-update carlos-system-recover carlos-instance-control; do |
| - echo "$u $(systemctl show $u.service -p FragmentPath --value)"; done' |
| +leo 'for u in carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service carlos-edge.service; do |
| + p=/etc/systemd/system/$u; if [ -f $p ] && [ ! -L $p ]; then f="file $(stat -c "%U %a" $p)"; else f=NOT-A-REGULAR-FILE; fi |
| + echo "$u $f $(systemctl show $u -p LoadState -p ActiveState -p UnitFileState --value | tr "\n" " ")"; done |
| +command -v jq || echo "jq absent"; uname -r; df -h /data | sed 1d; free -m | sed -n 2p; systemctl --version | head -1' |
| ``` |
| -Today: |
| -- Kernel 6.12.105, `/data` on ext4, 18G free, 927 MB of memory. PASS. Flag-on instance-control takes `MemoryMax=256M`, a large share of this box; watch it in step 7. |
| -- `/`, `/var` and `/var/lib` are `root:root 755`, and no `proof-policy.json` exists. PASS for the Plan 14 Part 1 pre-roll checks. |
| -- systemd 257. Every unit the offline window masks has its file in `/etc/systemd/system`. FAIL, a blocker. A runtime mask is a link in `/run/systemd/system`, and systemd reads `/etc/systemd/system` first, so the `/etc` unit file wins. #272's round-3 dev check saw exactly this: "runtime masks lose to the /etc unit files". Part 3's `verifyOfflineRootMigration` requires `LoadState=masked` on six units, and the seed runbook relies on a runtime mask on the edge. platform-1e's persistent offline transaction (`system claims offline-begin`, `offline-seed`, `offline-restore`) replaces runtime masks for the trial (blocker 3 below). |
| +Pass: all twelve are root-owned regular files in `/etc/systemd/system`; the edge and the five triggers are `loaded active`; the six services are `loaded inactive`. This is the baseline step 8's window requires. |
| + |
| +2026-10-17: |
| +- Units: PASS, re-run today. All twelve are `root 644` regular files. Edge and the five triggers `loaded active enabled`; the six services `loaded inactive` (`carlos-system-recover.service` enabled, the others static). The edge has no drop-in directory yet; `90-host-floor.conf` appears when step 6 arms the guard. |
| +- These files live in `/etc/systemd/system`, so a runtime mask in `/run/systemd/system` loses to them (#272 round 3 saw this). Step 8 masks by moving each file aside and linking `/dev/null` in its place. |
| +- `jq`: absent on leo, sophie and dennis (re-run today). Steps 5 to 7 use `jq` on the boxes. See H9. |
| +- Kernel 6.12.105, `/data` on ext4, 17G free, 927 MB of memory, systemd 257 (2026-10-16 and today). |
| ## Blockers |
| -1. **Activation runner: built, under authority review.** `carlos system claims ...` on `feat/claim-activation-tools` at 0e81652a (unmerged). It creates and signs the roster, invites and enrolls leo, sophie and the dev console, publishes the floor and `home_region`, and installs the evidence key, the fence-writer credentials and the claim-on instance-control unit. Steps 3 to 7 quote its commands. It must be reviewed, merged and released (step 1); its copy and census rows are still pending review. |
| -2. **Empty app segment.** `apps/bab//` in the deployment bucket (P5) makes the activation preflight and the migration census refuse. Source: leo's three app-less `proof*` routes, removing (2026-10-16), by another agent on Paul's answer. The writer fix is `fix-empty-app-key`. The one-off cleanup of `apps/bab//` waits for the e8e961dd release and Paul's approval. |
| -3. **Masks: built, under authority review.** Runtime masks do not mask units installed in `/etc/systemd/system` (P8). The tools' persistent offline transaction (`system claims offline-begin`, `offline-seed`, `offline-restore`, same branch) replaces them in step 5. |
| -4. **IAM.** Leo's IAM role lacks the version actions, and dev has neither the deployment bucket policy nor a fence-writer role (P2). Step 3a fixes this. |
| -5. **Release.** Plan 14 Parts 2 and 3 and the activation tools are not merged, and no release contains them with the files seed. Step 1 fixes this. |
| +1. **Release.** No release contains 26065378 (Plan 14 Parts 2 and 3, the activation tools) or e8e961dd (the empty-app writer fix). Step 1. |
| +2. **Empty app segment.** 267 objects under `apps/bab//` (P5). Source: leo's app-less `proof*` routes, removed (2026-10-16). Cleanup waits for the e8e961dd release to reach leo and for Paul's approval. Step 2. |
| +3. **IAM.** Leo's role lacks the version actions, and dev has neither the deployment bucket policy nor a fence-writer role (P2). Step 3. |
| ## Accepted trial gaps |
| Paul deferred these for this trial only. They are not blockers here, and they must be closed before any non-trial cohort: |
| - **Plan 15's independent claim writer.** Leo's instance role can write `control/hosts/`, so the box's own credentials could rewrite its claims. |
| -- **Key rotation.** The release signing key is not rotated for the trial, and no rotation schedule is set for the fence-writer bootstrap keys. (The separate exposure, box roles reading `/carlos/release-key`, is resolved: see P2.) |
| - |
| -## Step 0. Coordinate |
| - |
| -Paul's go: none needed to talk, but nothing below starts until these are done. |
| - |
| -- Tell platform-d8 that leo is about to change hands, and wait for its zero-downtime dev checks there to finish. Once claims are on, leo declines `CARLOS_SWAP=overlap` (#272 ruling, reason `host-claims-on`), so those checks cannot run on leo for the rest of the trial. |
| -- Tell the region HA and moves sessions. #286's ruling (b) says rehearsal apps wake on one dev box only until dev's claim is on. |
| -- Post the planned window on #250. |
| +- **Key rotation.** The release signing key is not rotated for the trial, and no rotation schedule is set for the fence-writer bootstrap keys. (The separate exposure, box roles reading `/carlos/release-key`, is resolved; see P2.) |
| ## Step 1. Release and roll, claims off |
| -What: cut a release containing Plan 14 Parts 2 and 3, the files seed and the activation tools, and roll it with every claim switch unset. Dev goes first. Flagship does not need this release before dev's window (Paul, 2026-10-16); its roll is separate. |
| +What: cut a release from main containing 26065378 and e8e961dd, and roll it with every claim switch unset. Dev goes first. Flagship does not need this release before dev's window (Paul, 2026-10-16); its roll is separate. |
| Paul's go: to cut the release, then to roll dev. |
| -1. Merge Parts 2 and 3 and `feat/claim-activation-tools` after their reviews, re-merge main, wait for CI. |
| -2. Cut the release by the README section "Releasing the platform". Check `gh release list -R carlosframework/releases` and `git ls-remote --tags origin` before choosing the number. |
| -3. Roll the dev boxes on the new road only: |
| +1. Cut the release by the README section "Releasing the platform". Check `gh release list -R carlosframework/releases` and `git ls-remote --tags origin` before choosing the number. |
| +2. Roll the dev boxes on the new road only: |
| ```bash |
| carlos system update --console https://platform.dev.carlosframework.com --channel stable --version vX.Y.Z --yes |
| @@ -228,54 +215,39 @@ Paul's go: to cut the release, then to roll dev. |
| carlos system status --console https://platform.dev.carlosframework.com |
| ``` |
| - `stable` moves leo and dennis together; `canary` moves sophie. Dennis is not in the trial, but it rolls with leo here because both are on `stable`. |
| -4. Roll the dev console direct-bucket with `ship-app.sh` (profile `kass`, bucket `carlos-dev-640012953448`). |
| -5. Check dev end to end: wake `wakecheck` on sophie through DNS and on leo with `curl --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210` (outside the sandbox). |
| + `stable` moves leo and dennis together; `canary` moves sophie. |
| +3. Roll the dev console direct-bucket with `ship-app.sh` (profile `kass`, bucket `carlos-dev-640012953448`). |
| +4. Check dev end to end: wake `wakecheck` on sophie through DNS (claims are still off, and wakecheck is still on both boxes). |
| Expected: |
| - `system status` shows all three dev boxes `current` at vX.Y.Z; |
| - `curl -sI https://platform.dev.carlosframework.com/ | grep -i x-carlos-version` shows the release commit; |
| - on leo, `systemctl show carlos-instance-control.service -p ActiveEnterTimestamp` is later than the update, and `/opt/carlos/carlos version` prints vX.Y.Z; |
| -- both wakes answer 200, leo's with `host=ip-172-31-19-180`. |
| - |
| -Hazards: |
| -- The old-updater instance-control restart. Dev's updater is v0.27.3, which restarts instance-control before the edge. If a wake is refused with "running an older build", instance-control was not restarted: restart it with `systemctl restart carlos-instance-control.service` (Paul's go). |
| -- #272: do not use the old road (`update-platform-binary.sh`) on leo. After an old-road roll, `result.json` "applied" is terminal, so the system leg never returns the box to its channel pointer, and it must be restored by hand. |
| - |
| -On failure: `carlos system rollback --console https://platform.dev.carlosframework.com --channel stable` (and `canary`). This is an ordinary claims-off release rollback. It works only while no actor is armed. After step 3, never run below the floor. |
| - |
| -Failure handling from here on: steps 1, 2 and 3a change nothing claim-related, so their own undo applies. From step 3b on, a failure means stop and hold. Before step 5 there are no claim-mode units, so "hold" means: stop the runbook, change nothing more, and leave leo serving claims off as it is. From step 5 on, follow "Stop and hold". |
| - |
| -## Step 2. Release public key |
| - |
| -What: confirm `CARLOS_RELEASE_PUBKEY` in leo's host.env is the key that signs dev releases, the roster and the floor. Root's held release and its signed launch reservations read it in claim mode. |
| +- `/opt/carlos/carlos system claims help` on leo prints main's usage, listing `inspect`, `run`, `evidence-key`, `fence-credentials`, `console-config`, `console-report` and `install-control`; |
| +- the wake answers 200. |
| -Paul's go: only if it must be changed. |
| +Hazards: H3 (#272, never the old road on leo) and H4 (the old-updater instance-control restart). |
| -Check: P1, plus step 1's update. A box verifies the channel pointer with this key, so a successful `carlos system update` to vX.Y.Z proves the match. |
| +On failure: `carlos system rollback --console https://platform.dev.carlosframework.com --channel stable` (and `canary`). This is an ordinary claims-off release rollback, and it works only while no actor is armed. From step 6 on, never run below the floor. |
| -Expected: P1 passes and step 1 applied on leo. Today the key is already present (`960d03ff` on all three boxes), so no change is planned. |
| +Failure handling for the rest of the runbook: steps 1 to 4 change nothing claim-related, so their own undo applies. From step 5 on, a failure means stop and hold. Before step 8 there are no claim-mode units, so "hold" means: stop the runbook, change nothing more, and leave leo serving claims off as it is. From step 8 on, follow "Stop and hold". |
| -Rollback: none; nothing changes. |
| +## Step 2. Clean up the empty app segment |
| -## The activation tools |
| +Paul's go: required. Paul runs it himself, from a laptop, after the step 1 release is on leo. |
| -Steps 3 to 7 use platform-1e's activation tools, `carlos system claims ...`. They are on the unmerged branch `feat/claim-activation-tools` at 0e81652a ("Claim activation tools, run 2 (not for merge)"), under authority review. Their copy is pending Paul's review and their census rows are pending controller review. The binary that runs them must come from a release that contains them (step 1). Every command in steps 3 to 7 is quoted exactly from platform-1e's report, `activation-tools-report.md` ("Exact command sequence for the Leo dev mechanics trial", sections A to F). Where this runbook and the report differ, the report wins. |
| +Run `docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash` (on main since e8e961dd). Its bucket, prefix, profile and region are fixed in the script; every key is checked to start with exactly `apps/bab//`, and each call names one key. |
| -The tools replace the earlier draft's pending pieces: |
| -- `system claims inspect`, `run`, `console-config`, `console-report`, `evidence-key` and `fence-credentials` create and sign the roster, enroll leo and the dev console, publish the floor and `home_region`, and install the protected keys (formerly blocker 1); |
| -- `system claims offline-begin`, `offline-seed` and `offline-restore` are a persistent mask transaction (formerly blocker 3's runtime-mask problem). `offline-seed` performs the files seed's delegated one-shot, its override and the edge mask handling; |
| -- `system claims install-control` installs the claim-on instance-control unit. |
| - |
| -Where the commands run: |
| -- **The trusted signing host:** a root shell holding `/root/claim-trial` with `release.pub`, `release.key`, `regions.json`, `leo.json`, `sophie.json`, `console.json` and `floor.json`. The release signing key never goes on either box. |
| -- **Leo and sophie:** root shells through the approved administration transport (the SSM helper above). Inputs are staged at the exact paths shown, root-owned, private files `0600`, no symlinks or extra hard links. Staging is a checkpoint, not permission to make up signatures. |
| +```bash |
| +docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash |
| +docs/superpowers/notes/2026-10-16-empty-app-key-cleanup.bash --apply |
| +``` |
| -The controller supplies, and Paul approves, the concrete inputs: the signed release, floor and guard artifacts; namespace, pool and label inputs (leo's pool is `cohort1`, sophie's matches its real placement); the console identity; the fence role and credentials; and the reviewed `/root/claim-trial/leo-host.env`. Revalidate both instance IDs before the trial. |
| +The first form is a dry run that lists and counts. `--apply` deletes each current object by exact key, leaving delete markers; that is reversible and enough for `hostops.Apps`, which lists current objects only. `--apply --versions` is permanent and is not needed for the trial. |
| -## Step 3. Actor and IAM certification |
| +Expected: P5's bucket lines show no `PRE /` and `0` objects. |
| -### 3a. IAM |
| +## Step 3. IAM |
| Paul's go: to apply the dev root. |
| @@ -284,19 +256,81 @@ Paul's go: to apply the dev root. |
| - `carlos-dev-host-b-buckets` gains `PullDoorObjectVersions`, the `s3:DeleteObjectVersion` statement and `PullDoorVersionList` on the IE data bucket; |
| - a new `deployment_box_records` bucket policy on `carlos-dev-640012953448`; |
| - a new fence-writer role and bootstrap user for leo; |
| + - no change to the `deny-release-key-reads` policies; |
| - no `user_data` change. A `user_data` diff restarts boxes, so stop and ask. |
| 3. Apply. Re-run P2's IAM lines. |
| 4. Denied-operation witnesses, through a reviewed runner: leo's instance role is refused a put to `control/box-fences/`, the fence-writer role may create there with `If-None-Match`, and nobody may delete there. |
| -The activation tools do not provision IAM. The fence-writer credentials that step 5 installs come from this step. |
| +The activation tools do not provision IAM. The fence-writer credentials step 7 installs come from here. |
| + |
| +Expected: P2's IAM items pass, apart from the roster. |
| + |
| +On failure: revert the dev root to its previous module wiring and apply, but only before any fence object is written. |
| + |
| +## Step 4. Trial pool and disposable app, claims still off |
| + |
| +Paul's go: APPROVED by Paul on 2026-10-16. Apply it when the trial nears, and only after coordinating with platform-d8, because it stops their test apps landing on leo. This is a host.env change on leo. |
| + |
| +Why a pool: dev's two home boxes both mint every unregioned, pool-less record. A disposable app created without a pool would land on sophie too (#286). `CARLOS_POOLS=cohort1` takes wakecheck off leo and is the only setting that puts the disposable app on leo alone. |
| + |
| +1. Tell platform-d8, and wait for a clear answer (H2). |
| +2. Set the pool and restart the edge, which reads host.env at start: |
| + |
| + ```bash |
| + leo 'cp -p /etc/carlos/host.env /etc/carlos/host.env.bak-pool-$(date -u +%Y%m%d) |
| + printf "CARLOS_POOLS=cohort1\n" >> /etc/carlos/host.env |
| + systemctl restart carlos-edge.service && systemctl is-active carlos-edge.service' |
| + ``` |
| + |
| +3. After the next provision pass (up to 5 minutes), run P4. Expected: wakecheck is gone from leo's registry. Sophie still serves it through DNS. |
| +4. Create the disposable app `claimtrial` in account `bab`, from wakecheck's own build (version `wc4`), so each request writes one SQLite row and replies with the host and the row count (`hits=`). Ship and promote it the same way wakecheck was, in store mode, then: |
| + |
| + ```bash |
| + export AWS_PROFILE=kass CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 |
| + carlos instances enable --app claimtrial --account bab --domain dev.carlosframework.com --pool cohort1 |
| + carlos instances create --app claimtrial --account bab --host claimtrial.dev.carlosframework.com |
| + ``` |
| -Expected: P2's IAM items pass. |
| +5. After the next provision pass, the route is on leo only. Write three rows, outside the sandbox: |
| -On failure: revert the dev root to its previous module wiring and apply, but only before any fence object is written. Evidence objects already written stay. |
| + ```bash |
| + for i in 1 2 3; do curl -s --resolve claimtrial.dev.carlosframework.com:443:18.200.150.210 https://claimtrial.dev.carlosframework.com/; echo; done |
| + ``` |
| -### 3b. Roster, enrollment and console enrollment (report section A) |
| + Expected: three 200 answers with `host=ip-172-31-19-180` and `hits=1`, `2`, `3`. Sophie's registry has no `claimtrial` route. |
| -Paul's go: once for 3b. |
| +`claimtrial` is the only host the files seed and acceptance touch. It holds nothing anyone needs. |
| + |
| +## Steps 5 to 7: the activation tools |
| + |
| +Steps 5 to 7 use `carlos system claims` from main. Its help text on main: |
| + |
| +``` |
| +usage: carlos system claims <command> [flags] |
| + |
| + inspect --bucket B --aws-region R --public-key K |
| + run --plan FILE --signing-key FILE --public-key K --aws-region R |
| + evidence-key --roster FILE --expect-revision N --bucket B --box BOX --enrollment E --public-key K |
| + fence-credentials --source FILE --bucket B --aws-region R --role ARN |
| + console-config --roster FILE --expect-revision N --bucket B --console-id ID --public-key K [--enable-console-runtime] |
| + console-report --bucket B --aws-region R --expect-revision N --console-id ID --public-key K |
| + install-control |
| + |
| +For commands that write, add --dry-run to validate and print the planned operation without writing. |
| +Run signing phases on the trusted operator host. Run box installers as root. |
| +``` |
| + |
| +`run` takes a JSON plan whose `phase` is one of `initialize`, `configure`, `invite-box`, `enroll-box`, `invite-console`, `enroll-console`, `publish-floor`, `home` or `bind-key`, with `bucket` and `expected_revision`. The command blocks below are quoted exactly from platform-1e's report (run 2, sections A to C, whose command branches run 11 confirms are byte-identical on main), and every flag was checked against main's code. |
| + |
| +Where they run: |
| +- **The trusted signing host:** a root shell holding `/root/claim-trial` with `release.pub`, `release.key`, `regions.json`, `leo.json`, `sophie.json`, `console.json` and `floor.json`. The release signing key never goes on either box. |
| +- **Leo and sophie:** attended root shells. Inputs are staged at the exact paths shown, root-owned, private files `0600`, no symlinks or extra hard links. Staging is a checkpoint, not permission to make up signatures. |
| + |
| +The controller supplies, and Paul approves, the concrete inputs: the signed release, floor and guard artifacts; namespace, pool and label inputs (leo's pool `cohort1`, sophie's matching its real placement); the console identity; the fence role and credentials from step 3; and leo's reviewed host.env changes. Revalidate both instance IDs first. |
| + |
| +## Step 5. Roster, enrollment and console enrollment |
| + |
| +Paul's go: once for step 5. |
| Each box's signed AWS identity record must already be published by the existing box identity mechanism. A missing record is a stop, not permission to make one up. |
| @@ -330,7 +364,7 @@ phase() { |
| } |
| ``` |
| -For a new roster only (dev has none today, P2), initialize, configure, then invite and enroll both computes and the console. On resume, run `refresh` and inspect the actual state instead. Never turn a read error into permission to initialize or overwrite. |
| +For a new roster only (dev has none, P2): initialize, configure, then invite and enroll both computes and the console. On resume, run `refresh` and inspect the actual state instead. Never turn a read error into permission to initialize or overwrite. |
| ```bash |
| jq -n --arg b "$B" '{phase:"initialize",bucket:$b,expected_revision:0}' > plan.json |
| @@ -355,17 +389,17 @@ for box in leo-climbing sophie-knitting; do |
| done |
| ``` |
| -Expected: every dry run, then its apply, succeeds; `inspect.json` lists `leo-climbing` (pool `cohort1`, instance `i-046c7fe43620944bc`), `sophie-knitting` and the console. Sophie is enrolled only as the console's compute host; its host claims stay off. |
| +Expected: every dry run, then its apply, succeeds; `inspect.json` lists `leo-climbing` (pool `cohort1`, instance `i-046c7fe43620944bc`), `sophie-knitting` and the console. |
| Then transfer `roster.json`, `release.pub`, the matching enrollment JSON, the approved armed `policy.json` and the signed `identity.json` to `/root/claim-trial` on each box. No private signing key is transferred. |
| On failure: hold. A partly applied phase is resumed from the latest authenticated revision, never recreated. |
| -### 3c. Arm leo's guard and start the guarded console (report section B) |
| +## Step 6. Arm leo's guard and start the guarded console |
| -Paul's go: once for 3c. It includes a `console.service` restart on sophie, which briefly interrupts the dev console. |
| +Paul's go: once for step 6. It restarts `console.service` on sophie, which briefly interrupts the dev console. |
| -On leo, after the approved release and protected current and previous binaries are installed: |
| +On leo, after the release and protected current and previous binaries are installed: |
| ```bash |
| set -euo pipefail |
| @@ -407,364 +441,223 @@ systemctl is-active console.service |
| --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" |
| ``` |
| -Expected: `guard-arm` succeeds on both; `console.service` is `active`; the report publishes. If reporting refuses because a just-started console or box has no fresh telemetry yet, hold, read the service journal, wait for its normal report and rerun the reporting pair. Never refresh timestamps by hand. Leo's host claims stay off until the offline window. Get real fresh box evidence from both enrolled computes before step 4. |
| - |
| -On failure: hold. Arming is one-way: keep the local armed policy, and never delete the enrollment or the guard. Restore only an image at or above the floor. |
| - |
| -## Step 4. Plan 13 order: readers, then floor and home, then enable |
| - |
| -### 4a. Readers |
| - |
| -Re-run P3. Dennis is not in the trial, but its agent reads the roster, so it counts. Expected: leo, sophie, dennis, their `.prev` images, any staged recovery image, the dev console and the operator CLI are all on the trial release (at minimum, on a Plan 13 build). The signed fleet data does not record every reader's build, so this is the operator's check. |
| - |
| -### 4b. Publish the floor and home, bind leo's key, enable the console runtime (report section C) |
| - |
| -Paul's go: once for 4b. `home_region` becomes write-once as soon as claims are on. The home name is `ie` (the roster and home designation use `ie`, AWS uses `eu-west-1`, and home boxes leave `CARLOS_REGION` unset). |
| - |
| -On the trusted signing host, immediately after sophie's successful report: |
| - |
| -```bash |
| -jq -n --slurpfile x floor.json '{floor:$x[0]}' > floor-phase.json |
| -phase publish-floor floor-phase.json |
| -printf '%s\n' '{"home_region":"ie"}' > home-phase.json |
| -phase home home-phase.json |
| -refresh |
| -``` |
| - |
| -On leo, read the latest signed roster and generate the protected evidence key. Keep the applied response, not the dry-run one: |
| - |
| -```bash |
| -set -euo pipefail |
| -umask 077 |
| -cd /root/claim-trial |
| -PUB=$(cat release.pub) |
| -/opt/carlos/carlos system claims inspect --bucket carlos-dev-640012953448 --aws-region eu-west-1 --public-key "$PUB" > inspect.json |
| -REV=$(jq -er '.roster.revision' inspect.json) |
| -LE=$(jq -er '.roster.boxes["leo-climbing"].enrollment' inspect.json) |
| -jq '.signed_roster' inspect.json > roster.json |
| -/opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| - --bucket carlos-dev-640012953448 --box leo-climbing --enrollment "$LE" --public-key "$PUB" --dry-run |
| -/opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| - --bucket carlos-dev-640012953448 --box leo-climbing --enrollment "$LE" --public-key "$PUB" > key-proof.json |
| -``` |
| - |
| -Transfer only `key-proof.json` to the trusted signing host. It holds the public binding and proof, never the private key. There: |
| - |
| -```bash |
| -jq '{binding:.key_proof.binding,proof:.key_proof.proof}' key-proof.json > bind-phase.json |
| -phase bind-key bind-phase.json |
| -refresh |
| -jq -e '.roster.home_region == "ie" and .roster.publication == "published" and .roster.request_limit_ns == 120000000000' inspect.json |
| -``` |
| - |
| -On sophie, enable only the console runtime: |
| - |
| -```bash |
| -set -euo pipefail |
| -umask 077 |
| -cd /root/claim-trial |
| -PUB=$(cat release.pub) |
| -CID=$(cat console-id) |
| -/opt/carlos/carlos system claims inspect --bucket carlos-dev-640012953448 --aws-region eu-west-1 --public-key "$PUB" > inspect.json |
| -REV=$(jq -er '.roster.revision' inspect.json) |
| -jq '.signed_roster' inspect.json > roster.json |
| -/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| - --bucket carlos-dev-640012953448 --console-id "$CID" --public-key "$PUB" --enable-console-runtime --dry-run |
| -/opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| - --bucket carlos-dev-640012953448 --console-id "$CID" --public-key "$PUB" --enable-console-runtime |
| -systemctl restart console.service |
| -systemctl is-active console.service |
| -/opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 \ |
| - --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" --dry-run |
| -/opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 \ |
| - --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" |
| -``` |
| - |
| -Expected: the final `jq -e` prints `true`; `/etc/carlos/host-evidence.key` exists on leo (`root`, `600`, one link); `console.service` is `active` and its report publishes. If publication loses a race with the agent's own report, inspect again, report again and retry with the current revision. |
| - |
| -On failure: hold. A torn publish is resumed with identical bytes. Never erase the floor or substitute another digest. The claim flag on leo stays off. |
| - |
| -## Step 5. Offline window: migration and files seed |
| - |
| -What: Plan 9 milestone 4 on leo, with the files seed inside the same window, through the persistent offline transaction. Tenants stay stopped from the drain until step 6. |
| - |
| -Paul's go: for the window, with a written deadline. Step 6 needs its own go. |
| +Expected: |
| +- `guard-arm` succeeds on both. On leo it writes `/etc/carlos/host-actor.json` and installs `/etc/systemd/system/carlos-edge.service.d/90-host-floor.conf`, which step 8 has to handle. |
| +- `console-config` writes `/etc/carlos/claim-console.env` and `/etc/systemd/system/console.service.d/91-claim-runtime.conf`, reloads systemd and does not start anything. |
| +- `console.service` is `active` and the report publishes. If reporting refuses because a just-started console or box has no fresh telemetry yet, hold, read the journal, wait for the normal report and rerun the reporting pair. Never refresh timestamps by hand. |
| -Before the window: blockers 2 and 5 are closed, the activation tools are merged and released, 5.0 is done, and platform-d8 has confirmed leo is free. |
| +Leo's host claims stay off until step 8. Get real fresh box evidence from both enrolled computes before step 7. |
| -### 5.0 Trial pool and disposable app, claims still off |
| +On failure: hold. Arming is one-way: keep the local armed policy, and never delete the enrollment or the guard. Restore only an image at or above the floor. |
| -Paul's go: APPROVED by Paul on 2026-10-16. Apply it when the trial nears, and only after coordinating with platform-d8, because it stops their test apps landing on leo. This is a host.env change on leo. `leo-host.env` in 5.1 keeps `CARLOS_POOLS=cohort1`. |
| +## Step 7. Readers, floor, home, leo's key and fence credentials |
| -Choice: keep the `CARLOS_POOLS` plan. The disposable app does not make it unnecessary. Dev's two home boxes both mint every unregioned, pool-less record, so a disposable app created without a pool would land on sophie too, which is the #286 problem moved to a new app. The pool does two jobs at once: it takes wakecheck off leo (H1), and it is the only knob that puts the disposable app on leo alone. |
| +Paul's go: once for step 7. `home_region` becomes write-once as soon as claims are on. |
| -1. Set the pool and restart the edge, which reads host.env at start: |
| +1. **Readers.** Re-run P3. Dennis is not in the trial, but its agent reads the roster, so it counts. Every reader must be on the trial release before the home designation is published. |
| +2. **Floor and home.** On the trusted signing host, immediately after sophie's successful report: |
| ```bash |
| - leo 'cp -p /etc/carlos/host.env /etc/carlos/host.env.bak-pool-$(date -u +%Y%m%d) |
| - printf "CARLOS_POOLS=cohort1\n" >> /etc/carlos/host.env |
| - systemctl restart carlos-edge.service && systemctl is-active carlos-edge.service' |
| + jq -n --slurpfile x floor.json '{floor:$x[0]}' > floor-phase.json |
| + phase publish-floor floor-phase.json |
| + printf '%s\n' '{"home_region":"ie"}' > home-phase.json |
| + phase home home-phase.json |
| + refresh |
| ``` |
| -2. After the next provision pass (up to 5 minutes), check leo's routes with P4's registry query. Expected: wakecheck is gone from leo's registry; the three `proof*` hand routes are still there (their removal is a question for Paul). Sophie still serves wakecheck through DNS. |
| -3. Create the disposable app `claimtrial` in account `bab`, from wakecheck's own build (version `wc4`), so each request writes one SQLite row and replies with the host and the row count (`hits=`). Ship and promote it the same way wakecheck was, in store mode, then: |
| +3. **Leo's evidence key.** On leo, keeping the applied response, not the dry-run one: |
| ```bash |
| - export AWS_PROFILE=kass CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 |
| - carlos instances enable --app claimtrial --account bab --domain dev.carlosframework.com --pool cohort1 |
| - carlos instances create --app claimtrial --account bab --host claimtrial.dev.carlosframework.com |
| + set -euo pipefail |
| + umask 077 |
| + cd /root/claim-trial |
| + PUB=$(cat release.pub) |
| + /opt/carlos/carlos system claims inspect --bucket carlos-dev-640012953448 --aws-region eu-west-1 --public-key "$PUB" > inspect.json |
| + REV=$(jq -er '.roster.revision' inspect.json) |
| + LE=$(jq -er '.roster.boxes["leo-climbing"].enrollment' inspect.json) |
| + jq '.signed_roster' inspect.json > roster.json |
| + /opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket carlos-dev-640012953448 --box leo-climbing --enrollment "$LE" --public-key "$PUB" --dry-run |
| + /opt/carlos/carlos system claims evidence-key --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket carlos-dev-640012953448 --box leo-climbing --enrollment "$LE" --public-key "$PUB" > key-proof.json |
| ``` |
| -4. After the next provision pass, the route is on leo only. Write three rows, outside the sandbox: |
| + Transfer only `key-proof.json` to the trusted signing host. It holds the public binding and proof, never the private key. There: |
| ```bash |
| - for i in 1 2 3; do curl -s --resolve claimtrial.dev.carlosframework.com:443:18.200.150.210 https://claimtrial.dev.carlosframework.com/; echo; done |
| + jq '{binding:.key_proof.binding,proof:.key_proof.proof}' key-proof.json > bind-phase.json |
| + phase bind-key bind-phase.json |
| + refresh |
| + jq -e '.roster.home_region == "ie" and .roster.publication == "published" and .roster.request_limit_ns == 120000000000' inspect.json |
| ``` |
| - Expected: three 200 answers with `host=ip-172-31-19-180` and `hits=1`, `2`, `3`. Sophie's registry has no `claimtrial` route. |
| - |
| -`claimtrial` is the only host the files seed and acceptance should touch. It holds nothing anyone needs, so a held or broken trial costs no data. |
| - |
| -On failure: hold. Leave the pool and the app as they are and report. |
| - |
| -### 5.1 Fence credentials, offline transaction, migration, root startup and seed (report section D) |
| +4. **Console runtime.** On sophie: |
| -Before entry: hold the admission and traffic barrier, drain and replicate `claimtrial`, confirm P5 still passes, and verify the independent preflight and retained-state gates from the binding scope. |
| - |
| -The reviewed `/root/claim-trial/leo-host.env` keeps every existing setting and secret, sets `CARLOS_HOST_CLAIM=1`, `CARLOS_HOST_CLAIM_ORIGINS=1` and `CARLOS_POOLS=cohort1`, and leaves `CARLOS_REGION` and `CARLOS_MOVES_PHASE1` absent or empty. Never add a literal `CARLOS_REGION=ie` on a home box. Stage `/root/claim-trial/fence-credentials.json` and `/root/claim-trial/fence-role-arn` from step 3a's provisioning. |
| - |
| -On leo, as root: |
| + ```bash |
| + set -euo pipefail |
| + umask 077 |
| + cd /root/claim-trial |
| + PUB=$(cat release.pub) |
| + CID=$(cat console-id) |
| + /opt/carlos/carlos system claims inspect --bucket carlos-dev-640012953448 --aws-region eu-west-1 --public-key "$PUB" > inspect.json |
| + REV=$(jq -er '.roster.revision' inspect.json) |
| + jq '.signed_roster' inspect.json > roster.json |
| + /opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket carlos-dev-640012953448 --console-id "$CID" --public-key "$PUB" --enable-console-runtime --dry-run |
| + /opt/carlos/carlos system claims console-config --roster /root/claim-trial/roster.json --expect-revision "$REV" \ |
| + --bucket carlos-dev-640012953448 --console-id "$CID" --public-key "$PUB" --enable-console-runtime |
| + systemctl restart console.service |
| + systemctl is-active console.service |
| + /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" --dry-run |
| + /opt/carlos/carlos system claims console-report --bucket carlos-dev-640012953448 --aws-region eu-west-1 \ |
| + --expect-revision "$REV" --console-id "$CID" --public-key "$PUB" |
| + ``` |
| -```bash |
| -set -euo pipefail |
| -umask 077 |
| -cd /root/claim-trial |
| -FENCE_ROLE=$(cat fence-role-arn) |
| -/opt/carlos/carlos system claims fence-credentials --source /root/claim-trial/fence-credentials.json \ |
| - --bucket carlos-dev-640012953448 --aws-region eu-west-1 --role "$FENCE_ROLE" --dry-run |
| -/opt/carlos/carlos system claims fence-credentials --source /root/claim-trial/fence-credentials.json \ |
| - --bucket carlos-dev-640012953448 --aws-region eu-west-1 --role "$FENCE_ROLE" |
| -/opt/carlos/carlos system claims offline-begin --dry-run |
| -/opt/carlos/carlos system claims offline-begin |
| -if ! test -e /root/claim-trial/host.env.before; then |
| - cp --preserve=all /etc/carlos/host.env /root/claim-trial/host.env.before |
| -fi |
| -install -o root -g root -m 0600 /root/claim-trial/leo-host.env /etc/carlos/host.env |
| -/opt/carlos/carlos system claims install-control --dry-run |
| -/opt/carlos/carlos system claims install-control |
| -set -a |
| -. /etc/carlos/host.env |
| -set +a |
| -unset CARLOS_REGION CARLOS_MOVES_PHASE1 |
| -/opt/carlos/carlos system journal-migrate |
| -/opt/carlos/carlos system claims install-control --dry-run |
| -/opt/carlos/carlos system claims install-control |
| -systemctl add-wants multi-user.target carlos-instance-control.service |
| -systemctl restart carlos-instance-control.service |
| -systemctl is-active carlos-instance-control.service |
| -/opt/carlos/carlos system claims offline-seed --dry-run |
| -/opt/carlos/carlos system claims offline-seed |
| -cp /etc/carlos/claim-activation/offline/seed-result.txt /root/claim-trial/seed-result.txt |
| -INVOCATION=$(sed -n 's/^InvocationID=//p' /root/claim-trial/seed-result.txt) |
| -test -n "$INVOCATION" |
| -journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > /root/claim-trial/seed-journal.txt |
| -cat /root/claim-trial/seed-result.txt |
| -cat /root/claim-trial/seed-journal.txt |
| -systemctl show carlos-edge.service --property=LoadState,ActiveState |
| -``` |
| +5. **Fence credentials.** On leo, with `/root/claim-trial/fence-credentials.json` and `/root/claim-trial/fence-role-arn` staged from step 3: |
| -`install-control` runs twice on purpose: the second run, after `journal-migrate`, adds the census-dependent root-retry configuration. |
| + ```bash |
| + set -euo pipefail |
| + umask 077 |
| + cd /root/claim-trial |
| + FENCE_ROLE=$(cat fence-role-arn) |
| + /opt/carlos/carlos system claims fence-credentials --source /root/claim-trial/fence-credentials.json \ |
| + --bucket carlos-dev-640012953448 --aws-region eu-west-1 --role "$FENCE_ROLE" --dry-run |
| + /opt/carlos/carlos system claims fence-credentials --source /root/claim-trial/fence-credentials.json \ |
| + --bucket carlos-dev-640012953448 --aws-region eu-west-1 --role "$FENCE_ROLE" |
| + ``` |
| Expected: |
| -- `/etc/carlos/fence-writer/credentials.json` installed (`root`, `400`); |
| -- `journal-migrate` exits 0, and `/var/lib/carlos-claims/authority/migration/migration.json` exists; |
| -- `carlos-instance-control.service` is `active`; |
| -- the seed result shows `Host "claimtrial.dev.carlosframework.com": seeded.` and its admitted prefix line, with no retained-state refusal (wakecheck is not on leo); |
| -- the last line shows the edge `LoadState=masked` and `ActiveState=inactive`. |
| +- the `jq -e` in 3 prints `true`; |
| +- `/etc/carlos/host-evidence.key` exists on leo (`root`, `600`, one link); |
| +- `console.service` is `active` and its report publishes; if publication loses a race with the agent's own report, inspect again, report again and retry with the current revision; |
| +- `/etc/carlos/fence-writer/credentials.json` exists on leo (`root`, `600`, as the command prints). |
| -Archive the result and journal even if a command fails. With `set -e`, run the capture commands in a separate root shell after a failure. Before any retry, save that invocation's evidence under a new filename, because the next invocation replaces `seed-result.txt`. Resume an interrupted entry with `offline-begin` and an interrupted seed with `offline-seed`. Neither authorizes serving. A completed offline transaction is not reusable as a new window; keep its record. |
| +On failure: hold. A torn publish is resumed with identical bytes. Never erase the floor or substitute another digest. Leo's claim flag stays off. |
| -The approved files seed runbook strings, quoted verbatim in the appendix, describe what the seed does and how to read its output. Their manual override, mask and unmask commands are not run: `offline-seed` performs them. |
| +## Step 8. Leo's attended offline window |
| -On failure: stop and hold. Leave the transaction's persistent mask and override in place. |
| +What: Plan 9 milestone 4 on leo, with the files seed inside the same window. The commands are quoted from platform-1e's run 11 report ("Leo's attended offline window"); the quoted text in each `[...]` block is the files seed copy Paul approved on 2026-10-15, verbatim. This is an attended procedure: there is no automatic transaction and no resume after a reboot. |
| -## Step 6. Serving under the claim (report section E) |
| +Paul's go: for the window, with a written deadline. Step 9 needs its own go. |
| -Paul's go: required, after step 5's migration, seed and retained-state checks pass. Keep the external traffic barrier until then. |
| +Before the window: |
| +- blockers 1 to 3 closed and steps 4 to 7 done; |
| +- a confirmed drained and replicated stop is possible for `claimtrial`, and nothing else is on leo; |
| +- an external traffic barrier that survives a reboot is in place; |
| +- a second root session is open to enforce the deadline; |
| +- any update or recovery on leo is settled; genuine pending update evidence is kept; |
| +- the named-unit preflight still holds: leo has no named-unit or `backing=unit` route (P5). |
| -On leo: |
| +The approved description of the step: |
| -```bash |
| -/opt/carlos/carlos system claims offline-restore --dry-run |
| -/opt/carlos/carlos system claims offline-restore |
| -systemctl is-active carlos-instance-control.service carlos-edge.service |
| -journalctl -u carlos-edge.service -u carlos-instance-control.service --since '-5min' --no-pager |
| -``` |
| - |
| -`offline-restore` keeps any operator masks that existed before, restores only this transaction's replacements and the previous trigger states, and starts a previously active edge last. |
| - |
| -Expected: |
| -- both units `active`; |
| -- `REFUSING CARLOS_SWAP=overlap (host-claims-on)` once in the edge journal; |
| -- no "signed roster has no home region" line; |
| -- `carlos system status --console https://platform.dev.carlosframework.com` shows leo current. |
| - |
| -On failure: stop and hold. |
| +[runbook.heading] |
| -## Step 7. Acceptance |
| +> Files seed during claim activation |
| -Paul's go: for the reboot only. |
| +[runbook.current_status] |
| -The acceptance host is the disposable app, `HOST=claimtrial.dev.carlosframework.com`. Never run these checks on wakecheck or a real app. Every curl runs outside the sandbox. |
| +> Use this step only within an approved claim activation, with an approved binary and census revision. Copy approval does not authorize a rollout. Keep moves disabled; the independent claim-writer, census and deployment gates remain separate. |
| -1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180` and `hits=4` (the three seeded rows survived), and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| -2. First cold wake. Wait for it to sleep: `carlos logs --app claimtrial --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before: the row written at launch survived the restore. |
| -3. Second cold wake. Repeat. Expect 200 and `hits=` one higher again. |
| -4. Isolation. While it is awake: |
| +[runbook.placement] |
| - ```bash |
| - leo 'pid=$(systemctl show -p MainPID --value "carlos-instance-gen@'"$HOST"':*"); ls /proc/$pid/root/data/instances/; readlink /proc/$pid/ns/mnt /proc/1/ns/mnt' |
| - ``` |
| +> Insert this step after milestone 4’s claim-on root startup and journal recovery, before normal edge serving. Keep CARLOS_MOVES_PHASE1 unset. Before rollout, inspect the dev cold-journal state required by the authority review, and record the approved claim-writer and census revisions. A serving activation does not approve moves. |
| - Expect only the host's own directory, and two different mount namespaces. `carlos status --layout` shows the host `nested`. |
| -5. Reboot. `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| -6. Signed repair or update. Apply the approved signed update to leo and confirm the box comes back and `claimtrial` wakes. The exact release input belongs to the approved trial record. |
| +[runbook.gates] |
| -Pass: all six. Record each response and the journal lines on #250. |
| +> Complete the claim-adoption, independent claim-writer and census activation gates. Require serving with moves off, publication and readiness, same-mount database retirement, bounded retention, pressure and expiry reaping, normal reboot recovery, early disk and evidence refusals, crashed-seed settlement, delegated controller authentication and concurrent-wake exclusion to pass. Require a real launch and two cold wakes with real origin evidence. Skipped privileged tests do not pass. Every claimed host must use nested layout, with no unfinished migration. Keep moves disabled; its verification and repair gates are separate. |
| -### Resetting an exhausted wake retry (report section F) |
| +[runbook.preflight] |
| -For `claimtrial` only, if its wake-retry budget is exhausted. The authorized operator takes the current holder incarnation and retry episode from the trial's claim evidence. The reset CLI does not expose this yet, so it goes through the console API. Stage a protected curl config with the operator's bearer header at `/root/claim-trial/operator-curl.conf` (never in shell arguments or logs), `reset-input.json` with the exact `incarnation`, `episode` and a unique retained `request_id`, and `acceptance-host` with the disposable hostname. On the trusted operator host: |
| +> No automated diagnostic preflight, exact restore sizing or space reservations ship with this step. Complete the approved activation checks and record measured disk headroom before the offline window. Files seed checks the existing start floor before reserving an attempt; it does not guarantee enough space for the whole restore or upload. |
| -```bash |
| -set -euo pipefail |
| -umask 077 |
| -cd /root/claim-trial |
| -HOST=$(cat acceptance-host) |
| -jq '{action:"reset-wake-retry",incarnation:.incarnation,episode:.episode,request_id:.request_id}' \ |
| - reset-input.json > reset.json |
| -curl --fail-with-body --silent --show-error --config /root/claim-trial/operator-curl.conf \ |
| - --header 'Content-Type: application/json' --data-binary @/root/claim-trial/reset.json \ |
| - "https://platform.dev.carlosframework.com/api/cli/ops/hosts/$HOST/resolve" |
| -``` |
| +[runbook.classification] |
| -A 200 acknowledges the request; it does not prove a wake. Check the new retry episode and zero budget in the trial evidence, with holder, session and stamp unchanged, then check a real wake. A 409 is a stale or refused reset: read the current evidence and investigate. Do not rewrite identifiers or loop. |
| +> Review each skipped or blocked host separately. Seeded-instance routes, backing routes, named units, sidecars, tunnels and replicas cannot use this serving preparation. A proven new empty host needs no files seed. Fix missing builds, flat or unfinished layouts and database path mismatches before rerunning. A missing files directory is a refusal, not an empty source. Hibernating eligible hosts retain their files and timer state. |
| -## Stop and hold |
| +[runbook.window] |
| -This replaces a claims-off rollback for the trial. If anything from 5.0 on fails, or acceptance fails, stop and leave leo as it is for diagnosis. Nothing on leo is undone, repaired or retried until platform-1e has looked and Paul has decided what comes next. Do not unset the claim flags, unbind the census, delete records or guards, or attempt a data-preserving rollback. Only `claimtrial` is affected, and it holds nothing of value. |
| +> During milestone 4, drain requests, flush database replication and stop tenant writers through the approved activation workflow. Keep the rollout-owned edge mask in place until the delegation step. Set CARLOS_HOST_CLAIM and CARLOS_HOST_CLAIM_ORIGINS, leave CARLOS_MOVES_PHASE1 unset, start the claim-on root service, and verify journal migration and root recovery before seeding. Claimed serving and C8 must work with moves off. |
| -Paul's go: not needed to stop. Stopping is always allowed. Anything after it needs a go. |
| +On leo, "the rollout-owned edge mask" and "runtime mask" in these strings mean the persistent mask made in 8.1 (the unit file moved aside and `/dev/null` linked in its place). A runtime mask in `/run` does not work on leo (P8). |
| -**During the offline transaction (step 5, before `offline-restore`).** The edge is already persistently masked and stopped. Leave the transaction's mask and any seed override in place. Recover only through the owned transaction (`offline-begin` or `offline-seed`), and only with a go. Save `seed-result.txt` and the invocation journal under new filenames. |
| +### 8.1 Record, drain and mask |
| -**After `offline-restore` (steps 6 and 7).** Keep the traffic and admission barrier, and stop leo's edge, which stops admission and every tenant: |
| +Choose a new window directory outside systemd's unit search path. The commands assume the twelve canonical unit files P8 recorded are root-owned regular files in `/etc/systemd/system`. Inspect their parents, drop-ins, effective units, metadata and enablement first. If a path is a symlink, a pre-existing mask or an unexpected unit, stop and resolve it with its owner; do not overwrite it. Before stopping anything, require the recorded states: edge and all five triggers loaded and active; all six migration services loaded and inactive. |
| ```bash |
| +set -eu |
| +umask 077 |
| +WINDOW='/etc/carlos/claim-window-cohort1-<approved-window-id>' |
| +TRIGGERS=(carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path) |
| +SERVICES=(carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service) |
| +UNITS=("${TRIGGERS[@]}" "${SERVICES[@]}" carlos-edge.service) |
| +mkdir -m 0700 -- "$WINDOW" |
| +mkdir -m 0700 -- "$WINDOW/held" |
| +cp -a -- /etc/carlos/host.env "$WINDOW/host.env.before" |
| +for unit in "${UNITS[@]}"; do |
| + path="/etc/systemd/system/$unit" |
| + test -f "$path" && test ! -L "$path" |
| + test "$(stat -c %u "$path")" = 0 |
| + stat -- "$path" > "$WINDOW/$unit.stat" |
| + sha256sum -- "$path" > "$WINDOW/$unit.sha256" |
| + cp -a -- "$path" "$WINDOW/$unit.before" |
| + systemctl cat "$unit" > "$WINDOW/$unit.effective" |
| + systemctl show "$unit" -p LoadState -p ActiveState -p UnitFileState > "$WINDOW/$unit.state" |
| +done |
| +systemctl list-units --all --plain 'carlos-instance@*' 'carlos-instance-gen@*' |
| +systemctl stop "${TRIGGERS[@]}" |
| +systemctl stop "${SERVICES[@]}" |
| systemctl stop carlos-edge.service |
| -systemctl is-active carlos-edge.service || test "$(systemctl show -p ActiveState --value carlos-edge.service)" = inactive |
| +systemctl disable carlos-edge.service |
| +systemctl list-units --all --plain 'carlos-instance@*' 'carlos-instance-gen@*' |
| +pgrep -a -u carlos || test "$?" = 1 |
| ``` |
| -Then stop the triggers that could change the box under you, and capture evidence read-only: |
| +Require no tenant writer, no remaining carlos-user process, completed replication and no active root mutator. Stop if a unit is failed or in transition rather than assuming it drained. Disabling the edge keeps it from starting at boot while the seed override exists; the saved enablement is restored in step 9. Keep the external traffic barrier throughout. |
| ```bash |
| -leo 'systemctl stop carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path |
| -systemctl show carlos-edge.service carlos-instance-control.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| -journalctl -u carlos-edge.service -u carlos-instance-control.service --since "-2h" --no-pager | tail -400 |
| -ls -la /var/lib/carlos-claims /etc/carlos/claim-activation 2>&1; cat /proc/sys/kernel/random/boot_id' |
| +for unit in "${UNITS[@]}"; do |
| + mv -T -- "/etc/systemd/system/$unit" "$WINDOW/held/$unit" |
| + ln -s -- /dev/null "/etc/systemd/system/$unit" |
| +done |
| +systemctl daemon-reload |
| +systemctl show "${UNITS[@]}" -p Id -p LoadState -p ActiveState |
| +/opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| -Post a summary on #250 and tell platform-1e. |
| - |
| -What "hold" leaves running: the operating system (systemd, journald, sshd, the SSM agent), so the box stays reachable for read-only diagnosis, and `carlos-instance-control.service`, the root service that keeps root recovery and retained state consistent. Stop instance-control too (`systemctl stop carlos-instance-control.socket carlos-instance-control.service`) only if it is itself failing or platform-1e asks. Sophie, its console and dennis carry on as normal; sophie keeps serving wakecheck. |
| - |
| -A stopped edge is not a persistent mask once the transaction has completed. Do not reboot or start leo while it is held. A reboot needs a new persistent hold procedure, approved by the controller first. |
| - |
| -What must not be touched while leo is held: |
| -- `/etc/carlos/host.env` (claim flags and `CARLOS_POOLS` stay set) and `/root/claim-trial`, including `host.env.before`; |
| -- `/var/lib/carlos-claims` and `/etc/carlos/claim-activation`, including migration state, journals, parks and the offline transaction record; |
| -- `/etc/carlos/host-actor.json`, `/etc/carlos/host-evidence.key`, `/etc/carlos/fence-writer/` and `/opt/carlos/system/guard`; |
| -- `/opt/carlos/carlos`, `/opt/carlos/carlos.prev` and anything staged: no `carlos system update`, rollback or old-road roll, and no `stable`-channel roll while leo is held, because it is on `stable` with dennis; |
| -- `/data`, including `registry.db`, `/data/instances` and any parked database; |
| -- unit files, drop-ins and masks, as found; |
| -- the roster, floor and enrollments, and the claim's S3 objects: `control/hosts/`, `control/host-actors.json`, `control/host-origins/`, admitted prefixes, and `instances/` in `carlos-dev-ie-640012953448`; |
| -- no `systemctl reset-failed`, no rerun of `journal-migrate`, `install-control` or the seed, no `offline-restore`, and no console `reset-wake-retry`. |
| - |
| -Leaving hold is a separate decision: a fix and resume, or a claims-off teardown under its own reviewed procedure. This runbook does not cover either. Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`, so a later teardown has to say how it treats that (open question 2). |
| - |
| -## Known hazards |
| - |
| -**H1. wakecheck (#286).** Chosen: wakecheck leaves leo through `CARLOS_POOLS=cohort1` (step 5.0) and stays on sophie, claims off, as today. Acceptance uses the disposable `claimtrial` app in pool `cohort1`. |
| - |
| -Why: wakecheck's record is unregioned, so sophie and leo both mint it. With leo claimed and sophie not, the same host would have two writers on different lineages. Leo's claimed generations restore from the claim location; sophie writes its legacy replica and also starts from its own stale local database (#286). DNS for `*.dev.carlosframework.com` points at sophie, and bots wake wakecheck there, so it cannot be kept to leo. A wake on sophie during the window would also break the drained, replicated stop the seed depends on. The disposable app needs the pool for the same reason: without one, sophie would mint it too. |
| - |
| -A box with `CARLOS_POOLS` set serves only the listed pools, not the shared `""` pool, so leo's reconcile pass removes wakecheck. The activation preflight skips records in other pools. |
| - |
| -**H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Notify them before step 5.0 and wait for a clear answer. After step 6, leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo for the rest of the trial, including any hold. The pool change also means their test apps would need `--pool cohort1`. |
| - |
| -**H3. #272.** A system pointer whose `result.json` says "applied" is terminal: after an old-road roll the system leg never returns the box to its channel pointer. Roll leo only with `carlos system update`. Separately, #272's stale-artefact refusal is deferred past activation: once a host is retired on leo, installing it again on leo refuses. That fails closed. Never remove `consumed/`, lock or record files by hand. |
| +Expected: every unit `LoadState=masked` and inactive. `install-control --dry-run` calls main's unchanged `verifyOfflineRootMigration`: all six migration services must be `LoadState=masked` and inactive or failed, and no root `adopt`, `system apply` or `guard-arm` process may remain. The operator checks the triggers, the edge and the tenant drain. Do not relax the verifier or read it as proof of replication. |
| -**H4. The old-updater instance-control restart.** An updater older than v0.26.0 does not restart `carlos-instance-control`, and the new agent then refuses isolated wakes. Dev's boxes run v0.27.3, whose updater does restart it. Check `ActiveEnterTimestamp` after every roll anyway. Changing host.env never restarts instance-control on its own; step 5.1 restarts it explicitly. |
| +### 8.2 Configure, migrate and start root |
| -**H5. Runtime masks lose to `/etc`.** Blocker 3. Never use `systemctl mask --runtime` or `unmask --runtime` on leo. Use only the offline transaction, and check `systemctl show carlos-edge.service --property=LoadState,ActiveState` reads `masked` and `inactive` before migrating. |
| +Update the protected `/etc/carlos/host.env` through the approved configuration procedure: exactly one `CARLOS_HOST_CLAIM=1`, exactly one `CARLOS_HOST_CLAIM_ORIGINS=1`, no `CARLOS_MOVES_PHASE1`, no `CARLOS_REGION` (never a literal `CARLOS_REGION=ie` on a home box), and `CARLOS_POOLS=cohort1` kept from step 4. Keep the saved original (`$WINDOW/host.env.before`). Then: |
| -**H6. Empty app segment.** Blocker 2. Leo's three app-less `proof*` routes wrote error objects under `apps/bab//errors/leo-climbing/` (newest 06:46Z on 2026-10-16); they are being removed. Clean up only after the e8e961dd release is on leo and Paul approves; before that, new objects may reappear. |
| - |
| -**H7. The dev console as an actor.** Step 3c arms the guard on sophie with `--console` and restarts `console.service` twice (3c and 4b), so the dev console is briefly down each time. Sophie's own host claims stay off. Without the enrolled console, an exhausted wake-retry episode on leo could not be reset. The reset door was proven on platform-1e's root runner (401, 404, 200 and 409 cases). |
| - |
| -**H8. Memory.** Leo has 927 MB. Claim mode raises instance-control's cap to 256M. Watch `systemctl show carlos-instance-control.service -p MemoryCurrent` during step 7. |
| - |
| -## Later option: carlos-dev-uk |
| - |
| -Not part of this trial (Paul, 2026-10-16). Kept here for a later cohort. dennis-stenciling is placement `uk`, serves nothing, and already has its own `CARLOS_SECRETS_KEY` in host.env (fingerprint `3fe29008`, the same as sophie's). Adding it later means running steps 2 to 7 on it as well (SSM region eu-west-2, instance i-009d593f451cfd562), with these differences: |
| - |
| -- IAM: `carlos-dev-host-uk` already has `s3:GetObjectVersion` and `s3:DeleteObjectVersion` on the UK data bucket. It still needs its own fence-writer role, and the deployment bucket policy covers both boxes. |
| -- Roster: its row has region `uk`. The home designation stays the IE name. |
| -- Files seed: it has no hosts, so `carlos system files seed` reports nothing to seed. Acceptance needs an app created with `--region uk`. |
| -- Guard: `/opt/carlos/carlos` on dennis is owned `carlos:carlos`, not root, and it has no `carlos.prev`. Check with platform-1e whether `guard-arm` accepts that before step 3b on dennis. |
| -- Its `carlos-instance-control.service` is idle and socket-activated today. Step 5.1's explicit restart is still required. |
| -- It rolls on the `stable` channel with leo. |
| - |
| -## Open questions |
| - |
| -Answered on 2026-10-16: dev-uk is out of this trial; the dev console is enrolled by the activation tools; the independent writer and key rotation are accepted trial gaps; box roles no longer read `/carlos/release-key`; the activation tools (runner, offline transaction, `install-control`, console enrollment) are built on `feat/claim-activation-tools` at 0e81652a; the home name is `ie`; the `proof*` routes are being removed; step 5.0 is approved; flagship does not need the release before dev's window. |
| - |
| -For platform-1e: |
| -1. Should `hostops.Apps` skip an empty app segment instead of refusing, so one stray object cannot stop activation on a whole deployment? |
| -2. For a later teardown out of hold: how do writes made under the claim reach the claims-off restore path? |
| -3. Before any reboot while held: the persistent hold procedure the tools' report asks the controller to approve. |
| - |
| -## Appendix: approved files seed runbook strings |
| - |
| -Quoted verbatim from the files seed copy Paul approved on 2026-10-15. The headings in brackets are the string identifiers. In this trial `system claims offline-seed` (step 5.1) performs the delegation, override, mask and unmask commands below; they are kept here as the approved description, not as steps to run. |
| - |
| -[runbook.heading] |
| - |
| -> Files seed during claim activation |
| - |
| -[runbook.current_status] |
| - |
| -> Use this step only within an approved claim activation, with an approved binary and census revision. Copy approval does not authorize a rollout. Keep moves disabled; the independent claim-writer, census and deployment gates remain separate. |
| - |
| -[runbook.placement] |
| - |
| -> Insert this step after milestone 4’s claim-on root startup and journal recovery, before normal edge serving. Keep CARLOS_MOVES_PHASE1 unset. Before rollout, inspect the dev cold-journal state required by the authority review, and record the approved claim-writer and census revisions. A serving activation does not approve moves. |
| - |
| -[runbook.gates] |
| - |
| -> Complete the claim-adoption, independent claim-writer and census activation gates. Require serving with moves off, publication and readiness, same-mount database retirement, bounded retention, pressure and expiry reaping, normal reboot recovery, early disk and evidence refusals, crashed-seed settlement, delegated controller authentication and concurrent-wake exclusion to pass. Require a real launch and two cold wakes with real origin evidence. Skipped privileged tests do not pass. Every claimed host must use nested layout, with no unfinished migration. Keep moves disabled; its verification and repair gates are separate. |
| - |
| -[runbook.preflight] |
| - |
| -> No automated diagnostic preflight, exact restore sizing or space reservations ship with this step. Complete the approved activation checks and record measured disk headroom before the offline window. Files seed checks the existing start floor before reserving an attempt; it does not guarantee enough space for the whole restore or upload. |
| - |
| -[runbook.classification] |
| +```bash |
| +/opt/carlos/carlos system claims install-control --dry-run |
| +/opt/carlos/carlos system claims install-control |
| +systemctl cat carlos-instance-control.service |
| +( |
| + set -a |
| + . /etc/carlos/host.env |
| + set +a |
| + /opt/carlos/carlos system journal-migrate |
| +) |
| +systemctl daemon-reload |
| +systemctl add-wants multi-user.target carlos-instance-control.service |
| +systemctl restart carlos-instance-control.service |
| +systemctl is-active carlos-instance-control.service |
| +journalctl -u carlos-instance-control.service --since '-5min' --no-pager |
| +``` |
| -> Review each skipped or blocked host separately. Seeded-instance routes, backing routes, named units, sidecars, tunnels and replicas cannot use this serving preparation. A proven new empty host needs no files seed. Fix missing builds, flat or unfinished layouts and database path mismatches before rerunning. A missing files directory is a refusal, not an empty source. Hibernating eligible hosts retain their files and timer state. |
| +Expected: |
| +- `install-control` prints `"target": "/etc/systemd/system/carlos-instance-control.service"`, `"reload": true` and `"start": false`; |
| +- the unit keeps its protected environment, `StateDirectory=carlos-claims` and `MemoryMax=256M`; |
| +- `journal-migrate` succeeds and publishes the census; `/var/lib/carlos-claims/authority/migration/migration.json` exists; |
| +- `active`, and no migration, recovery or floor error in the journal. |
| -[runbook.window] |
| +Keep every migration and root record. A failure leaves serving held. |
| -> During milestone 4, drain requests, flush database replication and stop tenant writers through the approved activation workflow. Keep the rollout-owned edge mask in place until the delegation step. Set CARLOS_HOST_CLAIM and CARLOS_HOST_CLAIM_ORIGINS, leave CARLOS_MOVES_PHASE1 unset, start the claim-on root service, and verify journal migration and root recovery before seeding. Claimed serving and C8 must work with moves off. |
| +### 8.3 Run the approved files seed |
| [runbook.budget] |
| @@ -786,14 +679,6 @@ carlos system files seed |
| Usage: carlos system files seed [--host <host>] |
| ``` |
| -[cli.exit_status] |
| - |
| -> Exit 0 means the selected run finished without a reported error; it can include skipped hosts and does not prove that every host on the box was seeded. Exit 2 means the executor collected host failures or noticed cancellation between hosts. Exit 1 covers setup, validation, transport and other errors outside that collected result. Capture the service’s ExecMainStatus and Result; systemctl start has its own exit status. |
| - |
| -[cli.forwarded_errors] |
| - |
| -> Failure lines contain the underlying provider or operating-system error, sometimes across several lines. Save the full output. A shared “selected authority changed” error does not identify one cause by itself; use the root service journal and the host’s retained evidence. |
| - |
| [runbook.delegation] |
| > Through the approved activation workflow, create the new temporary override at the path below. Refuse an existing file or symlink there. Save the original effective unit and masks; preserve the service user, protected environment, delegation and hardening. Reload systemd and inspect the effective command before removing only the rollout-owned runtime mask. Keep the traffic barrier in place while starting the seed one-shot. Capture its InvocationID, Result, ExecMainStatus and journal before stopping it; RemainAfterExit keeps a successful completed invocation available. Save failed results before any reset-failed. After stopping, restore the rollout-owned runtime mask, remove only the recorded override, reload and inspect the normal unit. Resolve pre-existing operator masks through their owner. Rerun a crashed one-shot in this same delegated unit within the approved offline workflow. |
| @@ -836,6 +721,8 @@ systemctl cat carlos-edge.service |
| systemctl unmask --runtime carlos-edge.service |
| ``` |
| +On leo this unmask is replaced by putting the edge unit file back from `$WINDOW/held`, as in the block below. |
| + |
| [runbook.start_command] |
| ``` |
| @@ -854,6 +741,64 @@ systemctl show carlos-edge.service -p InvocationID -p Result -p ExecMainStatus - |
| journalctl _SYSTEMD_INVOCATION_ID=<invocation-id> --no-pager |
| ``` |
| +[cli.exit_status] |
| + |
| +> Exit 0 means the selected run finished without a reported error; it can include skipped hosts and does not prove that every host on the box was seeded. Exit 2 means the executor collected host failures or noticed cancellation between hosts. Exit 1 covers setup, validation, transport and other errors outside that collected result. Capture the service’s ExecMainStatus and Result; systemctl start has its own exit status. |
| + |
| +[cli.forwarded_errors] |
| + |
| +> Failure lines contain the underlying provider or operating-system error, sometimes across several lines. Save the full output. A shared “selected authority changed” error does not identify one cause by itself; use the root service journal and the host’s retained evidence. |
| + |
| +The leo commands. The edge's `90-host-floor.conf` (installed by `guard-arm` in step 6) requires `carlos-system-recover.service` and replaces `ExecStart` with `guard-exec`. While recovery is masked, that dependency keeps the seed from starting, and the file name also sorts after `90-files-seed-activation.conf`. Inspect and save its exact bytes and metadata. Only if it is the expected `guard-arm` drop-in, move it out of the search path for the seed, then restore that same file after remasking the edge. Do not remove unrelated dependencies, hardening or service settings. Confirm the approved binary and armed floor before moving it. |
| + |
| +```bash |
| +GUARD=/etc/systemd/system/carlos-edge.service.d/90-host-floor.conf |
| +SEED=/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| +test -f "$GUARD" && test ! -L "$GUARD" |
| +stat -- "$GUARD" > "$WINDOW/edge-guard.stat" |
| +cp -a -- "$GUARD" "$WINDOW/edge-guard.before" |
| +cat "$GUARD" |
| +test ! -e "$SEED" && test ! -L "$SEED" |
| +mkdir -p -- /run/systemd/system/carlos-edge.service.d |
| +(set -C; cat > "$SEED" <<'SEED_OVERRIDE' |
| +[Service] |
| +Type=oneshot |
| +RemainAfterExit=yes |
| +ExecStart= |
| +ExecStart=/opt/carlos/carlos system files seed |
| +ExecReload= |
| +Restart=no |
| +TimeoutStartSec=0 |
| +TimeoutStopSec=45s |
| +SEED_OVERRIDE |
| +) |
| +cp -a -- "$SEED" "$WINDOW/seed-override.before" |
| +mv -T -- "$GUARD" "$WINDOW/held/90-host-floor.conf" |
| +test "$(readlink /etc/systemd/system/carlos-edge.service)" = /dev/null |
| +rm -- /etc/systemd/system/carlos-edge.service |
| +mv -T -- "$WINDOW/held/carlos-edge.service" /etc/systemd/system/carlos-edge.service |
| +systemctl daemon-reload |
| +systemctl cat carlos-edge.service |
| +systemctl show carlos-edge.service -p ExecStart -p Requires -p After -p User -p Group -p Delegate -p EnvironmentFiles |
| +/opt/carlos/carlos system claims install-control --dry-run |
| +``` |
| + |
| +Before starting, require the exact seed `ExecStart`, the original service user, protected environment, delegation and hardening, no dependency on a masked service, and an active claim-on root service. The normal edge stays stopped. The six migration masks and all five trigger masks stay in place. |
| + |
| +```bash |
| +if systemctl start carlos-edge.service; then START_RC=0; else START_RC=$?; fi |
| +printf '%s\n' "$START_RC" > "$WINDOW/seed-systemctl-status" |
| +systemctl show carlos-edge.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/seed-result" |
| +INVOCATION=$(systemctl show carlos-edge.service -p InvocationID --value) |
| +test -n "$INVOCATION" |
| +printf '%s\n' "$INVOCATION" > "$WINDOW/seed-invocation" |
| +journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > "$WINDOW/seed-journal" |
| +``` |
| + |
| +The one-shot has no start timeout, so the second root session enforces the deadline. Capture the service's `Result` and `ExecMainStatus`, not only the status of `systemctl start`. |
| + |
| +Expected: `Host "claimtrial.dev.carlosframework.com": seeded.` and its admitted prefix line, `Result=success`, `ExecMainStatus=0`. Review every host, skipped path, admitted prefix, retry state and blocked-host result against the activation inventory and incarnation. A successful seed does not prove a healthy tenant. |
| + |
| [runbook.skips] |
| > Review every skipped-file line. Oversized and non-regular files are omitted. Unreadable or changing files fail that host. |
| @@ -898,6 +843,8 @@ carlos system files seed --host <host> |
| journalctl -u carlos-instance-control.service --no-pager |
| ``` |
| +### 8.4 Remask, restore and hold for the serving go |
| + |
| [runbook.finish] |
| > Save the invocation ID, service result, exit status, complete output, each reported admitted prefix, retry state, skipped paths and blocked-host list. Match hosts and incarnations to the activation inventory; the seed output does not print an incarnation field. Stop the one-shot, restore this rollout’s runtime mask, remove only its override and reload systemd. Verify that the effective unit again starts the normal agent with its original restart policy and hardening. Keep the traffic barrier until Plan 9 allows serving, then remove only the rollout-owned mask and start the normal agent. Complete the live cold-wake and readiness checks before declaring activation complete. |
| @@ -918,12 +865,219 @@ systemctl stop carlos-edge.service |
| systemctl mask --runtime carlos-edge.service |
| ``` |
| +On leo this mask is replaced by moving the edge unit file back into `$WINDOW/held` and linking `/dev/null`, as in the block below. |
| + |
| [runbook.remove_override_command] |
| ``` |
| rm -- /run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| ``` |
| +The leo commands. Whether the seed succeeded or failed, stop the one-shot and save its final result. Restore the persistent edge mask before removing the recorded seed override or restoring the guard. Check that the unit and drop-in still match the saved files before moving or deleting them; stop on foreign changes. |
| + |
| +```bash |
| +systemctl stop carlos-edge.service |
| +systemctl show carlos-edge.service -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/seed-result-final" |
| +journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > "$WINDOW/seed-journal-final" |
| +cmp -- /etc/systemd/system/carlos-edge.service "$WINDOW/carlos-edge.service.before" |
| +mv -T -- /etc/systemd/system/carlos-edge.service "$WINDOW/held/carlos-edge.service" |
| +ln -s -- /dev/null /etc/systemd/system/carlos-edge.service |
| +systemctl daemon-reload |
| +systemctl show carlos-edge.service -p LoadState -p ActiveState |
| +cmp -- "$SEED" "$WINDOW/seed-override.before" |
| +rm -- "$SEED" |
| +test ! -e "$GUARD" && test ! -L "$GUARD" |
| +cmp -- "$WINDOW/held/90-host-floor.conf" "$WINDOW/edge-guard.before" |
| +mv -T -- "$WINDOW/held/90-host-floor.conf" "$GUARD" |
| +systemctl daemon-reload |
| +/opt/carlos/carlos system claims install-control --dry-run |
| +``` |
| + |
| +After every seed and migration check passes, restore only this window's recorded originals for the six services, then the five triggers. For each, verify the saved bytes and metadata, require its current path to be this window's `/dev/null` link, remove the link and move the original back. Do not start the triggers yet. |
| + |
| +```bash |
| +for unit in "${SERVICES[@]}" "${TRIGGERS[@]}"; do |
| + cmp -- "$WINDOW/held/$unit" "$WINDOW/$unit.before" |
| + test "$(readlink "/etc/systemd/system/$unit")" = /dev/null |
| + rm -- "/etc/systemd/system/$unit" |
| + mv -T -- "$WINDOW/held/$unit" "/etc/systemd/system/$unit" |
| +done |
| +systemctl daemon-reload |
| +systemctl show "${SERVICES[@]}" "${TRIGGERS[@]}" -p Id -p LoadState -p ActiveState |
| +``` |
| + |
| +Expected: the edge `masked` and inactive; the services and triggers loaded and inactive. Leo now waits, masked, for step 9. |
| + |
| +## Step 9. Serving under the claim |
| + |
| +Paul's go: required, after step 8's migration, seed and retained-state checks pass. Keep the external traffic barrier until then. |
| + |
| +In the same attended root shell on leo: restore the edge original, reload, inspect the effective normal guarded agent command and its recovery dependency, restore its recorded enablement, start it last among services, then restore the triggers that were active before (all five on the recorded baseline): |
| + |
| +```bash |
| +cmp -- "$WINDOW/held/carlos-edge.service" "$WINDOW/carlos-edge.service.before" |
| +test "$(readlink /etc/systemd/system/carlos-edge.service)" = /dev/null |
| +rm -- /etc/systemd/system/carlos-edge.service |
| +mv -T -- "$WINDOW/held/carlos-edge.service" /etc/systemd/system/carlos-edge.service |
| +systemctl daemon-reload |
| +systemctl cat carlos-edge.service |
| +systemctl show carlos-edge.service -p ExecStart -p Requires -p After |
| +if grep -qx 'UnitFileState=enabled' "$WINDOW/carlos-edge.service.state"; then |
| + systemctl enable carlos-edge.service |
| +fi |
| +systemctl start carlos-edge.service |
| +systemctl is-active carlos-edge.service |
| +systemctl start "${TRIGGERS[@]}" |
| +systemctl show "${UNITS[@]}" -p Id -p LoadState -p ActiveState -p UnitFileState |
| +``` |
| + |
| +Compare the final units, drop-ins, metadata, enablement and activity with the saved baseline, keeping the intended claim-on root unit and configuration. Release the external traffic barrier only after the serving check allows it. |
| + |
| +Expected: |
| +- the edge `active` with its `guard-exec` command; the twelve units match the baseline in P8; |
| +- `REFUSING CARLOS_SWAP=overlap (host-claims-on)` once in the edge journal; |
| +- no "signed roster has no home region" line; |
| +- `carlos system status --console https://platform.dev.carlosframework.com` shows leo current. |
| + |
| +On failure: stop and hold. |
| + |
| +## Step 10. Acceptance |
| + |
| +Paul's go: for the reboot and the signed update. |
| + |
| +The acceptance host is the disposable app, `HOST=claimtrial.dev.carlosframework.com`. Never run these checks on wakecheck or a real app. Every curl runs outside the sandbox. |
| + |
| +1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180` and `hits=4` (the three seeded rows survived), and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| +2. First cold wake. Wait for it to sleep: `carlos logs --app claimtrial --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before. |
| +3. Second cold wake. Repeat. Expect 200 and `hits=` one higher again. |
| +4. Isolation. While it is awake: |
| + |
| + ```bash |
| + leo 'pid=$(systemctl show -p MainPID --value "carlos-instance-gen@'"$HOST"':*"); ls /proc/$pid/root/data/instances/; readlink /proc/$pid/ns/mnt /proc/1/ns/mnt' |
| + ``` |
| + |
| + Expect only the host's own directory, and two different mount namespaces. `carlos status --layout` shows the host `nested`. |
| +5. Reboot. `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| +6. Signed repair or update. Apply the approved signed update to leo with `carlos system update`; confirm the box comes back and `claimtrial` wakes. |
| + |
| +Pass: all six. Record each response and the journal lines on #250. |
| + |
| +If `claimtrial`'s wake-retry budget is exhausted, the reset goes through the console API (the reset CLI does not expose it yet). Stage a protected curl config with the operator's bearer header at `/root/claim-trial/operator-curl.conf` (never in shell arguments or logs), `reset-input.json` with the exact `incarnation`, `episode` and a unique retained `request_id`, and `acceptance-host` with the hostname. On the trusted operator host: |
| + |
| +```bash |
| +set -euo pipefail |
| +umask 077 |
| +cd /root/claim-trial |
| +HOST=$(cat acceptance-host) |
| +jq '{action:"reset-wake-retry",incarnation:.incarnation,episode:.episode,request_id:.request_id}' \ |
| + reset-input.json > reset.json |
| +curl --fail-with-body --silent --show-error --config /root/claim-trial/operator-curl.conf \ |
| + --header 'Content-Type: application/json' --data-binary @/root/claim-trial/reset.json \ |
| + "https://platform.dev.carlosframework.com/api/cli/ops/hosts/$HOST/resolve" |
| +``` |
| + |
| +A 200 acknowledges the request; it does not prove a wake. Check the new retry episode and zero budget in the trial evidence, with holder, session and stamp unchanged, then check a real wake. A 409 is a stale or refused reset: read the current evidence and investigate. Do not rewrite identifiers or loop. |
| + |
| +## Stop and hold |
| + |
| +This replaces a claims-off rollback for the trial. If anything from step 5 on fails, or acceptance fails, stop and leave leo as it is for diagnosis. Nothing on leo is undone, repaired or retried until platform-1e has looked and Paul has decided what comes next. Do not unset the claim flags, unbind the census, delete records or guards, or attempt a data-preserving rollback. Only `claimtrial` is affected, and it holds nothing of value. |
| + |
| +Paul's go: not needed to stop. Stopping is always allowed. Anything after it needs a go. |
| + |
| +**Steps 5 to 7.** Nothing claim-mode runs yet. Stop the runbook and change nothing more; leo keeps serving claims off. |
| + |
| +**Step 8, before the seed starts (8.1, 8.2).** The six services, five triggers and the edge are already persistently masked. Leave every mask and `$WINDOW` as they are. Leave instance-control as it is (running if 8.2 started it). |
| + |
| +**Step 8, while the seed override is in place (8.3).** This is the one fragile state: the edge unit file is back in `/etc`, its guard drop-in is in `$WINDOW/held`, and the seed override lives in `/run`, which a reboot clears. Return to the masked state: |
| +1. At the deadline or on failure, capture `InvocationID` before stopping, as `[runbook.cancellation]` says. |
| +2. Run the first block of 8.4 (stop the one-shot, save the final result and journal, remask the edge, remove the seed override, put the guard drop-in back). |
| +3. Do not run the second block of 8.4: the services and triggers stay masked. |
| + |
| +If leo reboots or the operator session is lost while the override is in place, re-establish the masks and review the actual effective units before anything else. |
| + |
| +**Step 8, after 8.4.** The edge is masked; nothing more to stop. |
| + |
| +**Steps 9 and 10.** Keep the traffic and admission barrier, and stop the edge, which stops admission and every tenant. Then stop the triggers and put the persistent edge mask back with a fresh copy, so the hold survives a reboot: |
| + |
| +```bash |
| +systemctl stop carlos-edge.service |
| +systemctl stop carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path |
| +cmp -- /etc/systemd/system/carlos-edge.service "$WINDOW/carlos-edge.service.before" |
| +mv -T -- /etc/systemd/system/carlos-edge.service "$WINDOW/held/carlos-edge.service" |
| +ln -s -- /dev/null /etc/systemd/system/carlos-edge.service |
| +systemctl daemon-reload |
| +systemctl show carlos-edge.service -p LoadState -p ActiveState |
| +``` |
| + |
| +If the `cmp` fails, the unit changed after the window: stop, do not move it, and report. |
| + |
| +In every case, capture evidence read-only and post a summary on #250: |
| + |
| +```bash |
| +systemctl show carlos-edge.service carlos-instance-control.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| +journalctl -u carlos-edge.service -u carlos-instance-control.service --since "-2h" --no-pager | tail -400 |
| +ls -la "$WINDOW" "$WINDOW/held" /var/lib/carlos-claims /etc/systemd/system/carlos-edge.service.d /run/systemd/system/carlos-edge.service.d 2>&1 |
| +cat /proc/sys/kernel/random/boot_id |
| +``` |
| + |
| [runbook.rollback] |
| > Before disabling claims, record outstanding park failures and retained disk space. A box that has enrolled a census still maintains its retained root state during update-time recovery, even when claims are off. The no-census case remains separate. Do not widen unit permissions or delete retained paths by hand; use an approved recovery procedure. |
| + |
| +What "hold" leaves running: the operating system (systemd, journald, sshd, the SSM agent), so the box stays reachable for read-only diagnosis, and `carlos-instance-control.service`, the root service that keeps recovery and retained state consistent. Stop instance-control too (`systemctl stop carlos-instance-control.socket carlos-instance-control.service`) only if it is itself failing or platform-1e asks. Sophie, its console and dennis carry on as normal; sophie keeps serving wakecheck. |
| + |
| +What must not be touched while leo is held: |
| +- `$WINDOW` (`/etc/carlos/claim-window-cohort1-*`) and everything in `$WINDOW/held`, including the original unit files and `host.env.before`; |
| +- `/etc/carlos/host.env` (claim flags and `CARLOS_POOLS` stay set) and `/root/claim-trial`; |
| +- the `/dev/null` mask links in `/etc/systemd/system`, the unit files, drop-ins (including `90-host-floor.conf` and `91-claim-runtime.conf` on sophie) and any leftover seed override; |
| +- `/var/lib/carlos-claims`, including migration state, journals and parks; |
| +- `/etc/carlos/host-actor.json`, `/etc/carlos/host-evidence.key`, `/etc/carlos/fence-writer/` and `/opt/carlos/system/guard`; |
| +- `/opt/carlos/carlos`, `/opt/carlos/carlos.prev` and anything staged: no `carlos system update`, rollback or old-road roll, and no `stable`-channel roll while leo is held, because it is on `stable` with dennis; |
| +- `/data`, including `registry.db`, `/data/instances` and any parked database; |
| +- the roster, floor and enrollments, and the claim's S3 objects: `control/hosts/`, `control/host-actors.json`, `control/host-origins/`, admitted prefixes, and `instances/` in `carlos-dev-ie-640012953448`; |
| +- no `systemctl reset-failed` before the failed results are recorded, no reboot unless the edge is persistently masked, no rerun of `journal-migrate`, `install-control` or the seed, and no console `reset-wake-retry`. |
| + |
| +Leaving hold is a separate decision: a fix and resume, or a claims-off teardown under its own reviewed procedure. This runbook does not cover either. Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`, so a later teardown has to say how it treats that (open question 2). |
| + |
| +## Known hazards |
| + |
| +**H1. wakecheck (#286).** wakecheck leaves leo through `CARLOS_POOLS=cohort1` (step 4) and stays on sophie, claims off, as today. Its record is unregioned, so both home boxes mint it. With leo claimed and sophie not, the same host would have two writers on different lineages. DNS for `*.dev.carlosframework.com` points at sophie and bots wake wakecheck there, so it cannot be kept to leo, and a wake on sophie during the window would break the drained, replicated stop the seed needs. |
| + |
| +**H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Tell them before step 4. After step 9 leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo for the rest of the trial, including any hold. Their test apps would also need `--pool cohort1`. |
| + |
| +**H3. #272.** A system pointer whose `result.json` says "applied" is terminal: after an old-road roll the system leg never returns the box to its channel pointer. Roll leo only with `carlos system update`. Separately, #272's stale-artefact refusal is deferred past activation: once a host is retired on leo, installing it again on leo refuses. That fails closed. Never remove `consumed/`, lock or record files by hand. |
| + |
| +**H4. The old-updater instance-control restart.** An updater older than v0.26.0 does not restart `carlos-instance-control`, and the new agent then refuses isolated wakes. Dev's boxes run v0.27.3, whose updater does restart it. Check `ActiveEnterTimestamp` after every roll anyway. Changing host.env never restarts instance-control; step 8.2 restarts it explicitly. |
| + |
| +**H5. Runtime masks lose to `/etc`.** Never use `systemctl mask --runtime` or `unmask --runtime` on leo. Use only the moved-file masks of step 8 and check `LoadState=masked`. |
| + |
| +**H6. The seed override window.** Between the two halves of 8.3 and 8.4 the edge is unmasked, its guard drop-in is out, and the seed override sits in `/run`. A reboot then removes the override and leaves an unguarded, disabled edge unit. Keep this stretch short, attended and inside the deadline. |
| + |
| +**H7. The dev console as an actor.** Step 6 arms the guard on sophie with `--console`, and steps 6 and 7 restart `console.service`, so the dev console is briefly down each time. Sophie's own host claims stay off. The enrolled console is what allows an operator `reset-wake-retry` for `claimtrial`. |
| + |
| +**H8. Memory.** Leo has 927 MB. Claim mode raises instance-control's cap to 256M. Watch `systemctl show carlos-instance-control.service -p MemoryCurrent` during step 10. |
| + |
| +**H9. `jq` is not installed on the boxes.** Steps 5 to 7 use `jq` on leo and sophie (`inspect.json`, `roster.json`, the enrollment and key files), and it is absent on all three dev boxes (2026-10-17). Either install it on leo and sophie under that step's go (a package change on the box), or run the `jq` lines on the trusted host and stage their outputs at the same paths. Open question 4. |
| + |
| +## Later option: carlos-dev-uk |
| + |
| +Not part of this trial (Paul, 2026-10-16). dennis-stenciling is placement `uk`, serves nothing, and already has its own `CARLOS_SECRETS_KEY` (fingerprint `3fe29008`, the same as sophie's). Adding it later means steps 3 to 10 on it as well (SSM region eu-west-2, instance i-009d593f451cfd562), with these differences: |
| +- IAM: `carlos-dev-host-uk` already has `s3:GetObjectVersion` and `s3:DeleteObjectVersion` on the UK data bucket; it still needs its own fence-writer role. |
| +- Roster: its row has region `uk`. The home designation stays `ie`. |
| +- Files seed: it has no hosts. Acceptance needs an app created with `--region uk`. |
| +- Guard: `/opt/carlos/carlos` on dennis is owned `carlos:carlos`, not root, and it has no `carlos.prev`. Check whether `guard-arm` accepts that. |
| +- Its `carlos-instance-control.service` is idle and socket-activated. Step 8.2's explicit restart is still required. |
| +- It rolls on the `stable` channel with leo. |
| + |
| +## Open questions |
| + |
| +Answered: dev-uk is out of this trial; the console is enrolled by the activation tools; the independent writer and key rotation are accepted trial gaps; box roles no longer read `/carlos/release-key`; the home name is `ie`; the `proof*` routes are removed; step 4 is approved; flagship does not need the release before dev's window; the automated offline window is replaced by the manual one. |
| + |
| +For platform-1e: |
| +1. Should `hostops.Apps` skip an empty app segment instead of refusing, so one stray object cannot stop activation on a whole deployment? |
| +2. For a later teardown out of hold: how do writes made under the claim reach the claims-off restore path? |
| +3. The run 2 report said to run `install-control` again after `journal-migrate`, so the census-dependent root-retry configuration is included. Run 11's window installs it once, before `journal-migrate`. Is a second install needed in 8.2? |
| +4. `jq` on leo and sophie (H9): install it, or run those lines on the trusted host? |
| + |
| +For Paul: |
| +5. Run 11 asks Paul to confirm three minimal usage and docs edits for the removed commands (`activation-tools-run-11-strings.json`). Main already carries the shortened usage. |