amadan
docs: cohort-1 runbook becomes a leo-only mechanics trial (Paul, 2026-10-16)
Applies Paul's decisions relayed by platform-1e: - scope is carlos-dev-ie-b (leo) only; dev-uk becomes a later option; - acceptance runs on a disposable app, claimtrial, pinned to leo by CARLOS_POOLS=cohort1 (new step 5.0). The pool stays because without it sophie would also mint the disposable app; it also takes wakecheck off leo; - the claims-off rollback is replaced by "Stop and hold", which lists what keeps running and what must not be touched; - Plan 15's independent writer and key rotation are accepted trial gaps, to be closed before any non-trial cohort. Blocker updates: steps point at platform-1e's activation runner and persistent-mask installer (both pending), which also cover console enrollment; apps/bab// has a writer fix (fix-empty-app-key), with the cleanup after it reaches leo; removing the proof* routes is a question for Paul. Open questions trimmed to what is still open.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+154
−85
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+154 −85
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 309.384s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.749s
ok github.com/carlosframework/platform/cmd/moveprobe 92.269s
ok github.com/carlosframework/platform/cmd/zddprobe 0.709s
ok github.com/carlosframework/platform/internal/activator 249.978s
ok github.com/carlosframework/platform/internal/agentsock 0.018s
ok github.com/carlosframework/platform/internal/alerting 0.005s
ok github.com/carlosframework/platform/internal/analytics 0.018s
ok github.com/carlosframework/platform/internal/appconfig 2.479s
ok github.com/carlosframework/platform/internal/apperr 0.072s
ok github.com/carlosframework/platform/internal/applog 0.125s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 1.077s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.014s
ok github.com/carlosframework/platform/internal/certwarm 0.002s
ok github.com/carlosframework/platform/internal/cgroup 0.698s
ok github.com/carlosframework/platform/internal/changelog 0.304s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 399.406s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 5.969s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.351s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.004s
ok github.com/carlosframework/platform/internal/edge 26.719s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.137s
ok github.com/carlosframework/platform/internal/gate 1.655s
ok github.com/carlosframework/platform/internal/genselect 0.016s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.030s
ok github.com/carlosframework/platform/internal/host 166.542s
ok github.com/carlosframework/platform/internal/hostactors 0.553s
ok github.com/carlosframework/platform/internal/hostbox 2.580s
ok github.com/carlosframework/platform/internal/hostclaim 0.508s
ok github.com/carlosframework/platform/internal/hostevidence 0.011s
ok github.com/carlosframework/platform/internal/hostexec 5.039s
ok github.com/carlosframework/platform/internal/hostfence 0.030s
ok github.com/carlosframework/platform/internal/hostfiles 10.493s
ok github.com/carlosframework/platform/internal/hostfloor 0.014s
ok github.com/carlosframework/platform/internal/hostguard 1.011s
ok github.com/carlosframework/platform/internal/hostops 1.633s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.010s
ok github.com/carlosframework/platform/internal/hostquarantine 0.010s
ok github.com/carlosframework/platform/internal/hostrelocate 0.462s
ok github.com/carlosframework/platform/internal/hostresolve 0.147s
ok github.com/carlosframework/platform/internal/hostwrite 0.010s
ok github.com/carlosframework/platform/internal/infraview 0.011s
ok github.com/carlosframework/platform/internal/instancectl 4.180s
ok github.com/carlosframework/platform/internal/instanceexec 2.722s
ok github.com/carlosframework/platform/internal/instancefs 0.015s
ok github.com/carlosframework/platform/internal/instancekey 0.014s
ok github.com/carlosframework/platform/internal/instanceupdates 0.017s
ok github.com/carlosframework/platform/internal/layout 1.535s
ok github.com/carlosframework/platform/internal/lease 9.812s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.006s
ok github.com/carlosframework/platform/internal/ledger 0.015s
ok github.com/carlosframework/platform/internal/mail 0.210s
ok github.com/carlosframework/platform/internal/maintenance 0.081s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.524s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.066s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 21.036s
ok github.com/carlosframework/platform/internal/moves/archive 0.824s
ok github.com/carlosframework/platform/internal/objstore 0.188s
ok github.com/carlosframework/platform/internal/ops 0.175s
ok github.com/carlosframework/platform/internal/park 0.209s
ok github.com/carlosframework/platform/internal/privatefile 0.003s
ok github.com/carlosframework/platform/internal/provision 17.663s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 4.948s
ok github.com/carlosframework/platform/internal/relay 7.230s
ok github.com/carlosframework/platform/internal/release 20.087s
ok github.com/carlosframework/platform/internal/rollout 3.841s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.010s
ok github.com/carlosframework/platform/internal/rootstate 0.025s
ok github.com/carlosframework/platform/internal/runs 0.573s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.499s
ok github.com/carlosframework/platform/internal/seeds 0.010s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.967s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.031s
ok github.com/carlosframework/platform/internal/status 0.038s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.210s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.008s
ok github.com/carlosframework/platform/internal/termfence 0.009s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.593s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 47120546 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/16 13:44:35 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DF0674C694B14A, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index 0c28ef68a..18c6f0429 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -1,8 +1,14 @@ |
| # Host claim activation, dev cohort 1: runbook draft |
| -Draft, 2026-10-16. It authorizes nothing. Every change below waits for Paul's go at its own checkpoint, and the copy-approved strings quoted in step 5 are not an approval to run them. |
| +Draft, 2026-10-16, revised the same day for Paul's decisions. It authorizes nothing. Every change below waits for Paul's go at its own checkpoint, and the copy-approved strings quoted in step 5 are not an approval to run them. |
| -Cohort 1 is one dev box, `carlos-dev-ie-b`, with `carlos-dev-uk` as an optional second region. The dev console box, `carlos-dev-ie` (sophie-knitting), is excluded. It carries the `platform.dev.carlosframework.com` route with `unit=console.service`, and Plan 14 Part 3's activation preflight refuses any box with a named-unit or `backing=unit` route. |
| +Dev cohort 1 is a mechanics trial (Paul, 2026-10-16, relayed by platform-1e): |
| +- The scope is one box, `carlos-dev-ie-b` (leo-climbing). `carlos-dev-uk` is not part of this trial; it stays a later option (see the end of this document). |
| +- Acceptance runs on a disposable app made for the trial, never on wakecheck or any real app. |
| +- On failure the box is stopped and held for diagnosis, as described in "Stop and hold". There is no claims-off rollback in this trial. |
| +- Plan 15's independent claim writer and key rotation are deferred for the trial only. They are accepted trial gaps, not blockers, and must be closed before any non-trial cohort. |
| + |
| +The dev console box, `carlos-dev-ie` (sophie-knitting), is not activated. It carries the `platform.dev.carlosframework.com` route with `unit=console.service`, and Plan 14 Part 3's activation preflight refuses any box with a named-unit or `backing=unit` route. |
| Sources: |
| - the binding scope, `activation-scope-2026-10-16.md` (posted on amadan #250 and #280; the file itself lives in platform-1e's scratch, not in the repo); |
| @@ -16,9 +22,9 @@ Sources: |
| | Name | Box | Instance | Region | Role | Today | |
| |---|---|---|---|---|---| |
| -| `carlos-dev-ie-b` | leo-climbing | i-046c7fe43620944bc | eu-west-1, home, no `CARLOS_REGION` | `carlos-dev-host-b` | cohort 1 | |
| -| `carlos-dev-uk` | dennis-stenciling | i-009d593f451cfd562 | eu-west-2, `CARLOS_REGION=uk` | `carlos-dev-host-uk` | optional, empty | |
| -| `carlos-dev-ie` | sophie-knitting | i-05498eada6f0afd23 | eu-west-1, home | `carlos-dev-host` | excluded (console box) | |
| +| `carlos-dev-ie-b` | leo-climbing | i-046c7fe43620944bc | eu-west-1, home, no `CARLOS_REGION` | `carlos-dev-host-b` | the trial box | |
| +| `carlos-dev-uk` | dennis-stenciling | i-009d593f451cfd562 | eu-west-2, `CARLOS_REGION=uk` | `carlos-dev-host-uk` | not in this trial; later option | |
| +| `carlos-dev-ie` | sophie-knitting | i-05498eada6f0afd23 | eu-west-1, home | `carlos-dev-host` | not activated (console box); its console is enrolled as an actor | |
| Dev account 640012953448. AWS profile `kass` (a plain IAM user, so it works inside the sandbox). Deployment bucket `carlos-dev-640012953448`. Data buckets `carlos-dev-ie-640012953448` and `carlos-dev-uk-640012953448`. Console `https://platform.dev.carlosframework.com`. |
| @@ -41,7 +47,7 @@ aws --profile kass --region "$region" ssm get-command-invocation --command-id "$ |
| --query '[Status,StandardOutputContent,StandardErrorContent]' --output text |
| ``` |
| -`leo` below means `$TMPDIR/cohort1/ssm.sh eu-west-1 i-046c7fe43620944bc`, and `dennis` means `$TMPDIR/cohort1/ssm.sh eu-west-2 i-009d593f451cfd562`. Delete `$TMPDIR/cohort1` by exact path when the work is done. |
| +`leo` below means `$TMPDIR/cohort1/ssm.sh eu-west-1 i-046c7fe43620944bc`. Delete `$TMPDIR/cohort1` by exact path when the work is done. |
| Never print a secret. Key checks print a presence flag and the first 8 hex digits of a SHA-256, never the value. |
| @@ -77,7 +83,7 @@ leo 'for p in /etc/carlos/host-actor.json /etc/carlos/host-evidence.key /etc/car |
| Pass, once step 3 is done: |
| - `carlos-dev-host-b-buckets` grants `s3:GetObjectVersion`, `s3:DeleteObjectVersion` and `s3:ListBucketVersions` on the IE data bucket (count 3 or more); |
| - the deployment bucket policy has `BoxRecordsAreCreateOnly`, `BoxRecordsAreNeverDeleted` and `OnlyBoxRolesWriteTerminationFences`, the last naming a dev fence-writer role; |
| -- a dedicated fence-writer role exists for each cohort box; |
| +- a dedicated fence-writer role exists for leo; |
| - all three buckets report `Enabled`; |
| - `control/host-actors.json` exists, is signed by the release key and has leo's row; |
| - on leo, `/etc/carlos/host-actor.json` is present, `/etc/carlos/host-evidence.key` is `root 600 1`, the fence-writer credentials are `root 400 1`, and `/opt/carlos/system/guard` exists. |
| @@ -89,8 +95,8 @@ Today: FAIL, which is expected before step 3 apart from the first two items. |
| - Versioning is `Enabled` on all three buckets. PASS. |
| - `control/host-actors.json` does not exist (404). Dev has no signed roster. |
| - None of the four local files exists on leo. |
| -- Leo's role may `ssm:GetParameter` on `parameter/carlos/*` in eu-west-1, which holds `/carlos/release-key`, the deployment root signing key. The same key signs the roster and the floor. See open question 11. |
| -- Leo's role may `s3:PutObject` anywhere in the deployment bucket, including `control/hosts/`. Plan 15's independent claim-writer boundary is therefore not in place on dev. See open question 10. |
| +- Accepted trial gap: leo's role may `ssm:GetParameter` on `parameter/carlos/*` in eu-west-1, which holds `/carlos/release-key`, the deployment root signing key. The same key signs the roster and the floor. Key rotation is deferred for the trial (see "Accepted trial gaps"). |
| +- Accepted trial gap: leo's role may `s3:PutObject` anywhere in the deployment bucket, including `control/hosts/`, so Plan 15's independent claim-writer boundary is not in place on dev. Deferred for the trial (see "Accepted trial gaps"). |
| ### P3. Every roster reader's version |
| @@ -140,7 +146,7 @@ These mirror Part 3's `claimActivationPreflight` (registry), `claimActivationVer |
| Today: FAIL. |
| - Registry, local units and records: PASS. No named-unit or `backing=unit` route, `/etc/carlos/apps` is empty, and there is no `carlos-app@<name>` unit. The unit files that mention `/opt/carlos/live/` or `User=carlos` all carry embedded unit names. The bucket holds two records, `bab/wakecheck` and `bab/rollcheck-0261` (static), neither with a unit. |
| -- Bucket app tree: FAIL, a blocker. `apps/bab/` lists `PRE /`, an empty app segment. It holds 286 objects under `apps/bab//errors/leo-climbing/`, the newest written at 06:46Z today. `hostops.Apps` lists `apps/bab/` with a `/` delimiter, meets the empty name and returns `ErrIdentity`. On Part 3 that makes `claimActivationVerified` and the migration census refuse, on every box that uses this deployment bucket. The console's claim runtime and purge passes call the same function. See open question 9. |
| +- Bucket app tree: FAIL, a blocker. `apps/bab/` lists `PRE /`, an empty app segment. It holds 286 objects under `apps/bab//errors/leo-climbing/`, the newest written at 06:46Z today. `hostops.Apps` lists `apps/bab/` with a `/` delimiter, meets the empty name and returns `ErrIdentity`. On Part 3 that makes `claimActivationVerified` and the migration census refuse, on every box that uses this deployment bucket. The console's claim runtime and purge passes call the same function. The source is leo's three app-less `proof*` routes. The writer fix, `fix-empty-app-key`, is merging now (blocker 2 below). |
| ### P6. No claim state yet |
| @@ -178,21 +184,27 @@ for u in carlos-edge carlos-adopt carlos-stage carlos-config-sync carlos-restart |
| Today: |
| - Kernel 6.12.105, `/data` on ext4, 18G free, 927 MB of memory. PASS. Flag-on instance-control takes `MemoryMax=256M`, a large share of this box; watch it in step 7. |
| - `/`, `/var` and `/var/lib` are `root:root 755`, and no `proof-policy.json` exists. PASS for the Plan 14 Part 1 pre-roll checks. |
| -- systemd 257. Every unit the offline window masks has its file in `/etc/systemd/system`. FAIL, a blocker. A runtime mask is a link in `/run/systemd/system`, and systemd reads `/etc/systemd/system` first, so the `/etc` unit file wins. #272's round-3 dev check saw exactly this: "runtime masks lose to the /etc unit files". Part 3's `verifyOfflineRootMigration` requires `LoadState=masked` on six units, and the seed runbook relies on a runtime mask on the edge. See open question 5. |
| +- systemd 257. Every unit the offline window masks has its file in `/etc/systemd/system`. FAIL, a blocker. A runtime mask is a link in `/run/systemd/system`, and systemd reads `/etc/systemd/system` first, so the `/etc` unit file wins. #272's round-3 dev check saw exactly this: "runtime masks lose to the /etc unit files". Part 3's `verifyOfflineRootMigration` requires `LoadState=masked` on six units, and the seed runbook relies on a runtime mask on the edge. platform-1e is building a persistent-mask installer (blocker 3 below). |
| + |
| +## Blockers |
| -## Blockers found today |
| +1. **Activation runner, pending (platform-1e is building it).** Nothing on main or the Plan 14 branches creates the signed roster (`hostactors.Directory.Initialize`), invites and enrolls a box or the console, installs `/etc/carlos/host-evidence.key`, publishes the floor (`Directory.PublishFloor`) or publishes `home_region`. Steps 3b, 3c and 4b use platform-1e's activation runner once it is reviewed. The same work enrolls the dev console as an actor. |
| +2. **Empty app segment, fix merging.** `apps/bab//` in the deployment bucket (P5) makes the activation preflight and the migration census refuse. The writers are leo's three app-less `proof*` routes. The writer fix merges as `fix-empty-app-key`. After that fix reaches leo, a one-off cleanup removes the objects under `apps/bab//`, with Paul's approval. Whether the `proof*` routes themselves are removed is a pending question for Paul. |
| +3. **Persistent masks, pending (platform-1e is building the installer).** Runtime masks do not mask units installed in `/etc/systemd/system` (P8). Step 5 uses platform-1e's persistent-mask installer once it is reviewed. |
| +4. **IAM.** Leo's IAM role lacks the version actions, and dev has neither the deployment bucket policy nor a fence-writer role (P2). Step 3a fixes this. |
| +5. **Release.** Plan 14 Parts 2 and 3 are not merged, and no release contains them with the files seed. Step 1 fixes this. |
| -1. No runner exists on main or on the Plan 14 branches for: creating the signed roster (`hostactors.Directory.Initialize`), inviting and enrolling a box, installing `/etc/carlos/host-evidence.key`, publishing the floor (`Directory.PublishFloor`) or publishing `home_region`. Plan 9 C2 marks these "confirm at approval". Steps 3 and 4 cannot run until one is reviewed. |
| -2. The empty app segment `apps/bab//` in the deployment bucket (P5) makes the activation preflight and the migration census refuse. |
| -3. Runtime masks do not mask units installed in `/etc/systemd/system` (P8), so the offline window's masks cannot be made to hold as written. |
| -4. Leo's IAM role lacks the version actions, and dev has neither the deployment bucket policy nor a fence-writer role (P2). |
| -5. Plan 14 Parts 2 and 3 are not merged, and no release contains them with the files seed. |
| +## Accepted trial gaps |
| + |
| +Paul deferred these for this trial only. They are not blockers here, and they must be closed before any non-trial cohort: |
| +- **Plan 15's independent claim writer.** Leo's instance role can write `control/hosts/`, so the box's own credentials could rewrite its claims. |
| +- **Key rotation.** Box roles in eu-west-1 can read `/carlos/release-key` from SSM. No key is rotated or moved for the trial, and no rotation schedule is set for the fence-writer bootstrap keys. |
| ## Step 0. Coordinate |
| Paul's go: none needed to talk, but nothing below starts until these are done. |
| -- Tell platform-d8 that leo is about to change hands, and wait for its zero-downtime dev checks there to finish. Once claims are on, leo declines `CARLOS_SWAP=overlap` (#272 ruling, reason `host-claims-on`), so those checks cannot run on leo until a full rollback. |
| +- Tell platform-d8 that leo is about to change hands, and wait for its zero-downtime dev checks there to finish. Once claims are on, leo declines `CARLOS_SWAP=overlap` (#272 ruling, reason `host-claims-on`), so those checks cannot run on leo for the rest of the trial. |
| - Tell the region HA and moves sessions. #286's ruling (b) says rehearsal apps wake on one dev box only until dev's claim is on. |
| - Post the planned window on #250. |
| @@ -212,7 +224,7 @@ Paul's go: to cut the release, then to roll dev, then (separately, not from this |
| carlos system status --console https://platform.dev.carlosframework.com |
| ``` |
| - `stable` moves leo and dennis together; `canary` moves sophie. |
| + `stable` moves leo and dennis together; `canary` moves sophie. Dennis is not in the trial, but it rolls with leo here because both are on `stable`. |
| 4. Roll the dev console direct-bucket with `ship-app.sh` (profile `kass`, bucket `carlos-dev-640012953448`). |
| 5. Check dev end to end: wake `wakecheck` on sophie through DNS and on leo with `curl --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210` (outside the sandbox). |
| 6. Flagship boxes, then the flagship console, by the release recipe, under their own approval. |
| @@ -227,7 +239,9 @@ Hazards: |
| - The old-updater instance-control restart. Dev's updater is v0.27.3, which restarts instance-control before the edge. If a wake is refused with "running an older build", instance-control was not restarted: restart it with `systemctl restart carlos-instance-control.service` (Paul's go). |
| - #272: do not use the old road (`update-platform-binary.sh`) on leo. After an old-road roll, `result.json` "applied" is terminal, so the system leg never returns the box to its channel pointer, and it must be restored by hand. |
| -Rollback: `carlos system rollback --console https://platform.dev.carlosframework.com --channel stable` (and `canary`). This works only while no actor is armed. After step 3, never run below the floor. |
| +On failure: `carlos system rollback --console https://platform.dev.carlosframework.com --channel stable` (and `canary`). This is an ordinary claims-off release rollback. It works only while no actor is armed. After step 3, never run below the floor. |
| + |
| +Failure handling from here on: steps 1, 2 and 3a change nothing claim-related, so their own undo applies. From step 3b on, a failure means stop and hold. Before step 5 there are no claim-mode units, so "hold" means: stop the runbook, change nothing more, and leave leo serving claims off as it is. From step 5 on, follow "Stop and hold". |
| ## Step 2. Release public key |
| @@ -250,25 +264,25 @@ What: put the IAM shape from main's modules on dev, enroll leo as a claim actor, |
| Paul's go: to apply the dev root. |
| 1. Push the GitHub mirror forward to the release tag (fast-forward check first), because the dev root pulls modules from GitHub. |
| -2. In `platform-infrastructure/dev`, wire one `box-fence-writer` module per cohort box and pass their role ARNs to `module.buckets.box_fence_writer_role_arns`. Run `tofu init -upgrade && tofu plan -out=cohort1.plan` and read the whole plan. Expect: |
| +2. In `platform-infrastructure/dev`, wire one `box-fence-writer` module for leo and pass its role ARN to `module.buckets.box_fence_writer_role_arns`. Run `tofu init -upgrade && tofu plan -out=cohort1.plan` and read the whole plan. Expect: |
| - `carlos-dev-host-b-buckets` gains `PullDoorObjectVersions`, the `s3:DeleteObjectVersion` statement and `PullDoorVersionList` on the IE data bucket; |
| - a new `deployment_box_records` bucket policy on `carlos-dev-640012953448`; |
| - - new fence-writer roles and bootstrap users; |
| + - a new fence-writer role and bootstrap user for leo; |
| - no `user_data` change. A `user_data` diff restarts boxes, so stop and ask. |
| 3. Apply. Re-run P2's IAM lines. |
| 4. Denied-operation witnesses, through a reviewed runner: leo's instance role is refused a put to `control/box-fences/`, the fence-writer role may create there with `If-None-Match`, and nobody may delete there. |
| Expected: P2's IAM items pass. |
| -Rollback: revert the dev root to its previous module wiring and apply, but only before any fence object is written. Evidence objects already written stay. |
| +On failure: revert the dev root to its previous module wiring and apply, but only before any fence object is written. Evidence objects already written stay. |
| -### 3b. Enroll leo and arm its guard |
| +### 3b. Enroll leo and the dev console, and arm leo's guard |
| Paul's go: once for the whole of 3b. |
| -Blocked: there is no reviewed runner for the roster, the invitation, the enrollment or the evidence key (blocker 1). With one, the order is: |
| +Run with platform-1e's activation runner (pending, blocker 1). The order it follows: |
| -1. Create the signed roster, `control/host-actors.json`, and a root-signed invitation `control/host-actor-invitations/leo-climbing.json`. |
| +1. Create the signed roster, `control/host-actors.json`, and a root-signed invitation `control/host-actor-invitations/leo-climbing.json`. Enroll the dev console as an actor in the same roster; what that installs on sophie is part of the runner's review. |
| 2. Install, root-owned and outside any agent-writable path: |
| - `/etc/carlos/host-evidence.key`, `root`, mode `600`, one link; |
| - `/etc/carlos/fence-writer/credentials.json`, `root`, mode `400`. |
| @@ -284,20 +298,20 @@ Blocked: there is no reviewed runner for the roster, the invitation, the enrollm |
| Expected: |
| - `/etc/carlos/host-actor.json` written by `guard-arm`; |
| - `/opt/carlos/system/guard/policy.json` present; |
| -- the roster's `boxes.leo-climbing` row has leo's enrollment and `cloud.instance_id` `i-046c7fe43620944bc`; |
| +- the roster's `boxes.leo-climbing` row has leo's enrollment and `cloud.instance_id` `i-046c7fe43620944bc`, and the roster lists the dev console; |
| - `carlos system status --console https://platform.dev.carlosframework.com` shows leo ready, with the floor digest. |
| -Rollback: stop here and leave every flag unset. Arming is one-way: keep the local armed policy, and never delete the enrollment or the guard. Restore only an image at or above the floor. |
| +On failure: hold. Leave every flag unset and change nothing more. Arming is one-way: keep the local armed policy, and never delete the enrollment or the guard. Restore only an image at or above the floor. |
| ### 3c. Publish the floor |
| Paul's go: separate from 3b. |
| -Blocked on blocker 1. Re-read every report, the roster revision and the candidate digest just before publishing. Publish through the reviewed runner, then read back the roster. |
| +Run with platform-1e's activation runner (pending, blocker 1). Re-read every report, the roster revision and the candidate digest just before publishing. Publish, then read back the roster. |
| Expected: the roster has `publication: "published"` and a `floor_sha256` equal to the digest in leo's readiness report. |
| -Rollback: a torn publish is resumed with identical bytes. Never erase the floor or substitute another digest. If a check after publishing fails, the claim flag stays off. |
| +On failure: hold. A torn publish is resumed with identical bytes. Never erase the floor or substitute another digest. If a check after publishing fails, the claim flag stays off. |
| ## Step 4. Plan 13 order: reader, then home, then enable |
| @@ -305,15 +319,15 @@ Paul's go: before 4b. The designation becomes write-once as soon as claims are o |
| ### 4a. Readers |
| -Re-run P3. Expected: leo, sophie, dennis, their `.prev` images, any staged recovery image, the dev console and the operator CLI are all on vX.Y.Z (at minimum, on a Plan 13 build). The signed fleet data does not record every reader's build, so this is the operator's check. |
| +Re-run P3. Dennis is not in the trial, but its agent reads the roster, so it counts. Expected: leo, sophie, dennis, their `.prev` images, any staged recovery image, the dev console and the operator CLI are all on vX.Y.Z (at minimum, on a Plan 13 build). The signed fleet data does not record every reader's build, so this is the operator's check. |
| ### 4b. Home designation |
| -Publish `home_region` as dev's home roster name. The proposed name is `ie` (open question 8). Blocked on blocker 1. |
| +Publish `home_region` as dev's home roster name, with platform-1e's activation runner (pending, blocker 1). The proposed name is `ie` (open question 5). |
| Expected: the roster read back has `"home_region": "ie"`, and after step 6 leo's agent does not log "host claim: signed roster has no home region". |
| -Rollback: before claims are on, a new signed roster revision can correct the name. After claims are on, the name cannot change. |
| +On failure: before claims are on, a new signed roster revision can correct the name. After claims are on, the name cannot change. |
| ### 4c. Enable |
| @@ -325,23 +339,63 @@ What: Plan 9 milestone 4 on leo, with the files seed inside the same window. Ten |
| Paul's go: for the window, with a written deadline. A second go is needed in step 6 before serving resumes. |
| -Before the window: blockers 2 and 3 are fixed, the wakecheck decision (hazard H1) is applied, and platform-d8 has confirmed leo is free. |
| +Before the window: blockers 1 to 3 are closed (the `apps/bab//` cleanup done after `fix-empty-app-key` reached leo), 5.0 is done, and platform-d8 has confirmed leo is free. |
| + |
| +### 5.0 Trial pool and disposable app, claims still off |
| + |
| +Paul's go: separate, before the window. This is a host.env change on leo. |
| + |
| +Choice: keep the `CARLOS_POOLS` plan. The disposable app does not make it unnecessary. Dev's two home boxes both mint every unregioned, pool-less record, so a disposable app created without a pool would land on sophie too, which is the #286 problem moved to a new app. The pool does two jobs at once: it takes wakecheck off leo (H1), and it is the only knob that puts the disposable app on leo alone. |
| + |
| +1. Set the pool and restart the edge, which reads host.env at start: |
| + |
| + ```bash |
| + leo 'cp -p /etc/carlos/host.env /etc/carlos/host.env.bak-pool-$(date -u +%Y%m%d) |
| + printf "CARLOS_POOLS=cohort1\n" >> /etc/carlos/host.env |
| + systemctl restart carlos-edge.service && systemctl is-active carlos-edge.service' |
| + ``` |
| + |
| +2. After the next provision pass (up to 5 minutes), check leo's routes with P4's registry query. Expected: wakecheck is gone from leo's registry; the three `proof*` hand routes are still there (their removal is a question for Paul). Sophie still serves wakecheck through DNS. |
| +3. Create the disposable app `claimtrial` in account `bab`, from wakecheck's own build (version `wc4`), so each request writes one SQLite row and replies with the host and the row count (`hits=`). Ship and promote it the same way wakecheck was, in store mode, then: |
| + |
| + ```bash |
| + export AWS_PROFILE=kass CARLOS_DEPLOYMENT_BUCKET=carlos-dev-640012953448 |
| + carlos instances enable --app claimtrial --account bab --domain dev.carlosframework.com --pool cohort1 |
| + carlos instances create --app claimtrial --account bab --host claimtrial.dev.carlosframework.com |
| + ``` |
| + |
| +4. After the next provision pass, the route is on leo only. Write three rows, outside the sandbox: |
| + |
| + ```bash |
| + for i in 1 2 3; do curl -s --resolve claimtrial.dev.carlosframework.com:443:18.200.150.210 https://claimtrial.dev.carlosframework.com/; echo; done |
| + ``` |
| + |
| + Expected: three 200 answers with `host=ip-172-31-19-180` and `hits=1`, `2`, `3`. Sophie's registry has no `claimtrial` route. |
| + |
| +`claimtrial` is the only host the files seed and acceptance should touch. It holds nothing anyone needs, so a held or broken trial costs no data. |
| + |
| +On failure: hold. Leave the pool and the app as they are and report. |
| ### 5.1 Drain and quiesce |
| +Mask the six offline units and the edge with platform-1e's persistent-mask installer (pending, blocker 3). A runtime mask (`systemctl mask --runtime`) does not work on these boxes, because the unit files live in `/etc/systemd/system`. |
| + |
| ```bash |
| leo 'systemctl list-units --no-legend --plain "carlos-instance-gen@*" "carlos-instance@*"' |
| -leo 'systemctl stop carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path |
| -systemctl mask --runtime carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service |
| -for u in carlos-adopt carlos-stage carlos-config-sync carlos-restart carlos-system-update carlos-system-recover; do echo "$u $(systemctl show $u.service -p LoadState -p ActiveState --value | tr "\n" " ")"; done' |
| -leo 'systemctl stop carlos-edge.service && systemctl mask --runtime carlos-edge.service && systemctl show carlos-edge.service -p LoadState -p ActiveState --value; pgrep -u carlos | wc -l' |
| +leo 'systemctl stop carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path' |
| +# platform-1e's persistent-mask installer masks: carlos-adopt.service carlos-stage.service |
| +# carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service |
| +leo 'for u in carlos-adopt carlos-stage carlos-config-sync carlos-restart carlos-system-update carlos-system-recover; do echo "$u $(systemctl show $u.service -p LoadState -p ActiveState --value | tr "\n" " ")"; done' |
| +leo 'systemctl stop carlos-edge.service' |
| +# then the persistent-mask installer masks carlos-edge.service |
| +leo 'systemctl show carlos-edge.service -p LoadState -p ActiveState --value; pgrep -u carlos | wc -l' |
| ``` |
| -Expected: no instance unit before the stop; each of the six units `masked inactive`; the edge `masked inactive`; `0` processes under the carlos user. Stopping the edge drains requests and flushes replication for any awake instance. |
| +Expected: before the stop, no instance unit other than claimtrial's; each of the six units `masked inactive`; the edge `masked inactive`; `0` processes under the carlos user. Stopping the edge drains requests and flushes replication for any awake instance. |
| The adopt pass must be stopped first. It re-renders the instance-control unit from host.env and restarts it, so with `CARLOS_HOST_CLAIM` set it would start claim mode at an unplanned moment. |
| -Today this step cannot pass: see blocker 3. The masks above land in `/run` and lose to the `/etc` unit files. |
| +Where the seed strings in 5.6 say "runtime mask", use the installer's mask instead. Their `systemctl unmask --runtime` and `systemctl mask --runtime` commands are replaced by the installer's unmask and mask. |
| ### 5.2 Set the flags |
| @@ -355,7 +409,7 @@ Expected: `2`, then `0`. Any non-empty value of `CARLOS_HOST_CLAIM`, including ` |
| ### 5.3 Render the claim-on root unit |
| -The reviewed installer is not named yet (open question 6). After it runs: |
| +The installer that renders this unit while the adopt pass is masked is not named yet (open question 4; it may be part of platform-1e's activation runner). After it runs: |
| ```bash |
| leo 'systemctl cat carlos-instance-control.service | grep -E "^StateDirectory=carlos-claims|^EnvironmentFile=-/etc/carlos/host.env|^MemoryMax=256M"' |
| @@ -581,21 +635,17 @@ rm -- /run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| > Before disabling claims, record outstanding park failures and retained disk space. A box that has enrolled a census still maintains its retained root state during update-time recovery, even when claims are off. The no-census case remains separate. Do not widen unit permissions or delete retained paths by hand; use an approved recovery procedure. |
| -Expected for leo: one result line per selected host. If wakecheck stays on leo, `Host "wakecheck.dev.carlosframework.com": seeded.` and its admitted prefix line. The `proof*` routes are address-routed and have no app; record whatever is printed for them. Save the full output with the activation record. |
| +Expected for leo: `Host "claimtrial.dev.carlosframework.com": seeded.` and its admitted prefix line. wakecheck is no longer on leo. The `proof*` routes, if Paul keeps them, are address-routed and have no app; record whatever is printed for them. Save the full output with the activation record. |
| ### 5.7 Leave the window, but keep serving off |
| -Follow `[runbook.finish]` above up to "Keep the traffic barrier until Plan 9 allows serving". Then put the update and adopt units back: |
| +Follow `[runbook.finish]` above up to "Keep the traffic barrier until Plan 9 allows serving". Then unmask the six update and adopt units with the persistent-mask installer, and start their timers and paths: |
| ```bash |
| -leo 'systemctl unmask --runtime carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service |
| -systemctl start carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path' |
| +leo 'systemctl start carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path' |
| ``` |
| -Rollback for step 5: |
| -- Before 5.2: unmask the six units and the edge, start the timers and paths, start `carlos-edge.service`. |
| -- After 5.2, before 5.4: restore `/etc/carlos/host.env` from the backup, render the flag-off unit with the same installer, `systemctl daemon-reload && systemctl restart carlos-instance-control.service`, then unmask and start as above. |
| -- After 5.4: use the full rollback below. Keep `/var/lib/carlos-claims` and every record. |
| +On failure anywhere in step 5 (after 5.0): stop and hold. Do not restore host.env, re-render the flag-off unit or unmask anything. |
| ## Step 6. Claims on |
| @@ -604,7 +654,8 @@ What: lift the traffic barrier so leo serves under the claim. |
| Paul's go: required, after 5.5 and 5.6 pass. |
| ```bash |
| -leo 'systemctl unmask --runtime carlos-edge.service && systemctl start carlos-edge.service && systemctl is-active carlos-edge.service |
| +# first unmask carlos-edge.service with the persistent-mask installer |
| +leo 'systemctl start carlos-edge.service && systemctl is-active carlos-edge.service |
| journalctl -u carlos-edge.service --since "-5min" --no-pager | grep -iE "host claim|REFUSING CARLOS_SWAP|seed" | tail -20' |
| ``` |
| @@ -614,16 +665,16 @@ Expected: |
| - no "signed roster has no home region" line; |
| - `carlos system status --console https://platform.dev.carlosframework.com` shows leo current. |
| -Rollback: stop the edge (`systemctl stop carlos-edge.service`) to stop admission, then decide between holding and the full rollback. |
| +On failure: stop and hold. |
| ## Step 7. Acceptance |
| Paul's go: for the reboot only. |
| -Use the acceptance host chosen under hazard H1. Call it `$HOST`. Every curl runs outside the sandbox. |
| +The acceptance host is the disposable app, `HOST=claimtrial.dev.carlosframework.com`. Never run these checks on wakecheck or a real app. Every curl runs outside the sandbox. |
| -1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180`, and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| -2. First cold wake. Wait for it to sleep: `carlos logs --app <app> --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before: the row written at launch survived the restore. |
| +1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180` and `hits=4` (the three seeded rows survived), and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| +2. First cold wake. Wait for it to sleep: `carlos logs --app claimtrial --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before: the row written at launch survived the restore. |
| 3. Second cold wake. Repeat. Expect 200 and `hits=` one higher again. |
| 4. Isolation. While it is awake: |
| @@ -636,51 +687,72 @@ Use the acceptance host chosen under hazard H1. Call it `$HOST`. Every curl runs |
| Pass: all five. Record each response and the journal lines on #250. |
| -## Full rollback to claims off |
| +## Stop and hold |
| -Paul's go: required. |
| +This replaces a claims-off rollback for the trial. If any step from 5.0 on fails, or acceptance fails, stop the claim-mode units and leave leo exactly as it is for diagnosis. Nothing on leo is undone, repaired or retried until platform-1e has looked and Paul has decided what comes next. Only claimtrial is affected, and it holds nothing of value. |
| -1. Stop admission and drain: `systemctl stop carlos-edge.service`, then wait for every instance unit to stop. |
| -2. Record outstanding park failures and retained disk space, as `[runbook.rollback]` says. |
| -3. Remove both flags: |
| +Paul's go: not needed to stop. Stopping is always allowed. Anything after it needs a go. |
| + |
| +1. Save the failing step's output, its InvocationID if it was the seed one-shot, and the time. |
| +2. Stop admission and every tenant, then the root service and its socket: |
| ```bash |
| - leo 'sed -i.bak-claimsoff-$(date -u +%Y%m%d) "/^CARLOS_HOST_CLAIM=/d;/^CARLOS_HOST_CLAIM_ORIGINS=/d" /etc/carlos/host.env; grep -c "^CARLOS_HOST_CLAIM" /etc/carlos/host.env' |
| + leo 'systemctl stop carlos-edge.service |
| + systemctl stop carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path |
| + systemctl stop carlos-instance-control.socket carlos-instance-control.service |
| + systemctl list-units --no-legend --plain "carlos-*" --state=active' |
| ``` |
| - Expect `0`. |
| -4. Render the flag-off instance-control unit (the adopt pass does this on its next run; or run the reviewed installer), then `systemctl daemon-reload && systemctl restart carlos-instance-control.service`. `systemctl cat carlos-instance-control.service | grep -c StateDirectory=carlos-claims` prints `0`. |
| -5. `systemctl start carlos-edge.service`. Wake the acceptance host. Expect 200. |
| + Expected: no `carlos-edge`, `carlos-instance-control` or `carlos-instance-gen@` unit active, and no active `carlos-*` timer or path. |
| +3. Capture evidence, read-only: |
| + |
| + ```bash |
| + leo 'systemctl show carlos-edge.service carlos-instance-control.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| + journalctl -u carlos-edge.service -u carlos-instance-control.service --since "-2h" --no-pager | tail -400 |
| + ls -la /var/lib/carlos-claims /run/systemd/system/carlos-edge.service.d 2>&1; cat /proc/sys/kernel/random/boot_id' |
| + ``` |
| -Keep, and never delete by hand: `/var/lib/carlos-claims`, the guard, the floor, the roster enrollment, parked databases and every claim or root record. Never roll below the floor. Leo's census stays enrolled, so root keeps maintaining retained state during update-time recovery with claims off. |
| + Post a summary on #250 and tell platform-1e. |
| -Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`. A claims-off wake restores from the legacy replica. Confirm with platform-1e how claim-era writes reach the claims-off path before rolling back a host that matters (open question 12). |
| +What "hold" leaves running: the instance itself and the operating system, including systemd, journald, sshd and the SSM agent, so the box stays reachable for read-only diagnosis. Its disk and every file stay as they were at the stop. Sophie and dennis carry on as normal; sophie keeps serving wakecheck. `system status` shows leo as not reporting. |
| + |
| +What must not be touched while leo is held: |
| +- `/etc/carlos/host.env`: the claim flags and `CARLOS_POOLS` stay set; |
| +- `/var/lib/carlos-claims` and everything under it, including migration state, journals and parks; |
| +- `/etc/carlos/host-actor.json`, `/etc/carlos/host-evidence.key`, `/etc/carlos/fence-writer/` and `/opt/carlos/system/guard`; |
| +- `/opt/carlos/carlos`, `/opt/carlos/carlos.prev` and anything staged: no `carlos system update`, rollback or old-road roll, and no `stable`-channel roll while leo is held, because it is on `stable` with dennis; |
| +- `/data`, including `registry.db`, `/data/instances` and any parked database; |
| +- unit files, drop-ins (including a leftover `90-files-seed-activation.conf`) and masks, as found; |
| +- the claim's S3 objects: `control/hosts/`, `control/host-actors.json`, `control/host-origins/`, admitted prefixes, and `instances/` in `carlos-dev-ie-640012953448`; |
| +- no `systemctl reset-failed`, no reboot (the boot ID and boot-local device numbers are evidence, and boot runs recovery), no rerun of `journal-migrate` or the seed, and no console `reset-wake-retry`. |
| + |
| +Leaving hold is a separate decision: a fix and resume, or a claims-off teardown under its own reviewed procedure. This runbook does not cover either. Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`, so a later teardown has to say how it treats that (open question 7). |
| ## Known hazards |
| -**H1. wakecheck (#286).** Decision proposed: wakecheck stays on sophie and leaves leo for cohort 1. Acceptance uses a new leo-only app. |
| +**H1. wakecheck (#286).** Chosen: wakecheck leaves leo through `CARLOS_POOLS=cohort1` (step 5.0) and stays on sophie, claims off, as today. Acceptance uses the disposable `claimtrial` app in pool `cohort1`. |
| -Why: wakecheck's record is unregioned, so sophie and leo both mint it. With leo claimed and sophie not, the same host has two writers on different lineages. Leo's claimed generations restore from the claim location; sophie writes its legacy replica and also starts from its own stale local database (#286). DNS for `*.dev.carlosframework.com` points at sophie, and bots wake wakecheck there, so "woken on leo only" cannot be guaranteed. A wake on sophie during the window would also break the drained, replicated stop the seed depends on. |
| +Why: wakecheck's record is unregioned, so sophie and leo both mint it. With leo claimed and sophie not, the same host would have two writers on different lineages. Leo's claimed generations restore from the claim location; sophie writes its legacy replica and also starts from its own stale local database (#286). DNS for `*.dev.carlosframework.com` points at sophie, and bots wake wakecheck there, so it cannot be kept to leo. A wake on sophie during the window would also break the drained, replicated stop the seed depends on. The disposable app needs the pool for the same reason: without one, sophie would mint it too. |
| -How: set `CARLOS_POOLS=cohort1` in leo's host.env (a host.env change, under step 5's go). A box with `CARLOS_POOLS` set serves only the listed pools, not the shared `""` pool, so leo's reconcile pass removes wakecheck. Create the acceptance app with `carlos instances create --pool cohort1` (store mode), so only leo mints it. The activation preflight skips records in other pools. Alternative, if Paul rejects the pool change: keep wakecheck on both boxes and accept that acceptance on leo is not proof of isolation from sophie. Not recommended. |
| +A box with `CARLOS_POOLS` set serves only the listed pools, not the shared `""` pool, so leo's reconcile pass removes wakecheck. The activation preflight skips records in other pools. |
| -**H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Notify them before step 5 and wait for a clear answer. After step 6, leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo until a full rollback. A `CARLOS_POOLS` change also means their test apps need `--pool cohort1`. |
| +**H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Notify them before step 5.0 and wait for a clear answer. After step 6, leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo for the rest of the trial, including any hold. The pool change also means their test apps would need `--pool cohort1`. |
| **H3. #272.** A system pointer whose `result.json` says "applied" is terminal: after an old-road roll the system leg never returns the box to its channel pointer. Roll leo only with `carlos system update`. Separately, #272's stale-artefact refusal is deferred past activation: once a host is retired on leo, installing it again on leo refuses. That fails closed. Never remove `consumed/`, lock or record files by hand. |
| **H4. The old-updater instance-control restart.** An updater older than v0.26.0 does not restart `carlos-instance-control`, and the new agent then refuses isolated wakes. Dev's boxes run v0.27.3, whose updater does restart it. Check `ActiveEnterTimestamp` after every roll anyway. Changing host.env never restarts instance-control on its own; step 5.5 restarts it explicitly. |
| -**H5. Runtime masks lose to `/etc`.** Blocker 3. Do not start the window until platform-1e names a masking method that shows `LoadState=masked` on these boxes. |
| +**H5. Runtime masks lose to `/etc`.** Blocker 3. Do not start the window until platform-1e's persistent-mask installer is reviewed and shows `LoadState=masked` on leo. |
| -**H6. Empty app segment.** Blocker 2. Something on leo still writes error objects under `apps/bab//errors/leo-climbing/` (newest 06:46Z today). Deleting them is a bucket change for Paul's go, and the writer needs a fix or new objects will reappear. |
| +**H6. Empty app segment.** Blocker 2. Leo's three app-less `proof*` routes write error objects under `apps/bab//errors/leo-climbing/` (newest 06:46Z on 2026-10-16). Clean up only after `fix-empty-app-key` is on leo; before that, new objects reappear. |
| -**H7. The dev console is not a claim actor.** With sophie excluded, nothing on dev can perform an operator `reset-wake-retry`, so an exhausted wake-retry episode on leo stays blocked until it is reset. Enrolling the console would arm the guard on sophie, which is otherwise excluded (open question 7). |
| +**H7. The dev console as an actor.** platform-1e is building console enrollment, so the dev console can perform an operator `reset-wake-retry` for leo's hosts. Until it is enrolled, an exhausted wake-retry episode on leo stays blocked. What enrollment arms on sophie is part of reviewing platform-1e's runner. |
| **H8. Memory.** Leo has 927 MB. Claim mode raises instance-control's cap to 256M. Watch `systemctl show carlos-instance-control.service -p MemoryCurrent` during step 7. |
| -## Optional second region: carlos-dev-uk |
| +## Later option: carlos-dev-uk |
| -dennis-stenciling is placement `uk`, serves nothing, and already has its own `CARLOS_SECRETS_KEY` in host.env (fingerprint `3fe29008`, the same as sophie's). Adding it means running steps 2 to 7 on `dennis` as well, with these differences: |
| +Not part of this trial (Paul, 2026-10-16). Kept here for a later cohort. dennis-stenciling is placement `uk`, serves nothing, and already has its own `CARLOS_SECRETS_KEY` in host.env (fingerprint `3fe29008`, the same as sophie's). Adding it later means running steps 2 to 7 on it as well (SSM region eu-west-2, instance i-009d593f451cfd562), with these differences: |
| - IAM: `carlos-dev-host-uk` already has `s3:GetObjectVersion` and `s3:DeleteObjectVersion` on the UK data bucket. It still needs its own fence-writer role, and the deployment bucket policy covers both boxes. |
| - Roster: its row has region `uk`. The home designation stays the IE name. |
| @@ -691,18 +763,15 @@ dennis-stenciling is placement `uk`, serves nothing, and already has its own `CA |
| ## Open questions |
| +Answered on 2026-10-16: dev-uk is out of this trial; the dev console is enrolled by platform-1e's runner; the independent writer and key rotation are accepted trial gaps; the activation runner and persistent-mask installer are being built by platform-1e; the `apps/bab//` writer fix is `fix-empty-app-key`. |
| + |
| For Paul: |
| -1. wakecheck (H1): approve `CARLOS_POOLS=cohort1` on leo and a leo-only acceptance app, or choose otherwise. |
| -2. Include dev-uk in cohort 1? |
| +1. Remove leo's three app-less `proof*` routes (`proof.dev.carlosframework.com`, `proof.dev.oncarlos.com`, `proof2.dev.oncarlos.com`)? They are the source of `apps/bab//`, and nothing has used them since Aug 28. |
| +2. Approve `CARLOS_POOLS=cohort1` on leo (step 5.0), which takes wakecheck off leo and pins `claimtrial` to it. |
| 3. platform-1e's step 1 rolls the release to flagship after dev, claims off. Must flagship be rolled before dev's offline window, or only before a flagship cohort? |
| For platform-1e: |
| -4. Which reviewed runner creates the roster, invites and enrolls leo, installs `host-evidence.key` and the fence-writer credentials, publishes the floor and publishes `home_region` (blocker 1)? |
| -5. How should the six offline units and the edge be masked when their unit files are in `/etc/systemd/system` (blocker 3)? `verifyOfflineRootMigration` needs `LoadState=masked`. |
| -6. Which installer renders the claim-on instance-control unit while the adopt pass is masked (5.3)? |
| -7. Is the dev console a claim actor for cohort 1 (H7)? |
| -8. Dev's home roster name for `home_region`: `ie`? |
| -9. The empty `apps/bab//` prefix (blocker 2): who removes those objects, and which code writes them? Should `hostops.Apps` skip an empty segment instead of refusing? |
| -10. Plan 15's independent claim writer: what is the dev boundary, given leo's role can put to `control/hosts/`? |
| -11. Box roles in eu-west-1 can read `/carlos/release-key` from SSM. Does certification require moving that key or narrowing the grant? |
| -12. After a full rollback, how do writes made under the claim reach the claims-off restore path? |
| +4. Does the activation runner also render the claim-on instance-control unit while the adopt pass is masked (5.3), or is that a separate installer? |
| +5. Dev's home roster name for `home_region`: `ie`? |
| +6. Should `hostops.Apps` skip an empty app segment instead of refusing, so one stray object cannot stop activation on a whole deployment? |
| +7. For a later teardown out of hold: how do writes made under the claim reach the claims-off restore path? |