amadan
runbook: harness corrections C1-C3 (pause race, console before publish-identity, the 8.2 host.env edit)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
1 file changed,
+17
−5
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+17 −5
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 301.906s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.943s
ok github.com/carlosframework/platform/cmd/moveprobe 93.190s
ok github.com/carlosframework/platform/cmd/zddprobe 0.699s
ok github.com/carlosframework/platform/internal/activator 251.443s
ok github.com/carlosframework/platform/internal/agentsock 0.016s
ok github.com/carlosframework/platform/internal/alerting 0.004s
ok github.com/carlosframework/platform/internal/analytics 0.018s
ok github.com/carlosframework/platform/internal/appconfig 2.487s
ok github.com/carlosframework/platform/internal/apperr 0.072s
ok github.com/carlosframework/platform/internal/applog 0.136s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.005s
ok github.com/carlosframework/platform/internal/boxid 0.561s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.008s
ok github.com/carlosframework/platform/internal/caps 0.012s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.692s
ok github.com/carlosframework/platform/internal/changelog 0.303s
ok github.com/carlosframework/platform/internal/claimactivation 0.025s
ok github.com/carlosframework/platform/internal/clamd 0.058s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 403.235s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.047s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.008s
ok github.com/carlosframework/platform/internal/dbverify 0.004s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.004s
ok github.com/carlosframework/platform/internal/domains 2.417s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.003s
ok github.com/carlosframework/platform/internal/edge 44.766s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.131s
ok github.com/carlosframework/platform/internal/gate 1.656s
ok github.com/carlosframework/platform/internal/genselect 0.014s
ok github.com/carlosframework/platform/internal/geo 0.016s
ok github.com/carlosframework/platform/internal/geoip 0.031s
ok github.com/carlosframework/platform/internal/historyreadback 0.155s
ok github.com/carlosframework/platform/internal/host 174.765s
ok github.com/carlosframework/platform/internal/hostactors 0.712s
ok github.com/carlosframework/platform/internal/hostbox 2.645s
ok github.com/carlosframework/platform/internal/hostclaim 0.544s
ok github.com/carlosframework/platform/internal/hostevidence 0.013s
ok github.com/carlosframework/platform/internal/hostexec 4.735s
ok github.com/carlosframework/platform/internal/hostfence 0.034s
ok github.com/carlosframework/platform/internal/hostfiles 10.376s
ok github.com/carlosframework/platform/internal/hostfloor 0.012s
ok github.com/carlosframework/platform/internal/hostguard 1.033s
ok github.com/carlosframework/platform/internal/hostops 1.605s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.013s
ok github.com/carlosframework/platform/internal/hostquarantine 0.012s
ok github.com/carlosframework/platform/internal/hostrelocate 0.457s
ok github.com/carlosframework/platform/internal/hostresolve 0.147s
ok github.com/carlosframework/platform/internal/hostwrite 0.012s
ok github.com/carlosframework/platform/internal/infraview 0.014s
ok github.com/carlosframework/platform/internal/instancectl 5.211s
ok github.com/carlosframework/platform/internal/instanceexec 2.731s
ok github.com/carlosframework/platform/internal/instancefs 0.017s
ok github.com/carlosframework/platform/internal/instancekey 0.014s
ok github.com/carlosframework/platform/internal/instanceupdates 0.007s
ok github.com/carlosframework/platform/internal/layout 1.554s
ok github.com/carlosframework/platform/internal/lbhealth 0.002s
ok github.com/carlosframework/platform/internal/lease 9.792s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.006s
ok github.com/carlosframework/platform/internal/ledger 0.012s
ok github.com/carlosframework/platform/internal/mail 0.159s
ok github.com/carlosframework/platform/internal/maintenance 0.076s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.493s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.082s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 20.928s
ok github.com/carlosframework/platform/internal/moves/archive 0.829s
ok github.com/carlosframework/platform/internal/objstore 0.173s
ok github.com/carlosframework/platform/internal/ops 0.174s
ok github.com/carlosframework/platform/internal/park 0.202s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 17.690s
ok github.com/carlosframework/platform/internal/publicrelease 0.006s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.398s
ok github.com/carlosframework/platform/internal/relay 7.227s
ok github.com/carlosframework/platform/internal/release 21.747s
ok github.com/carlosframework/platform/internal/rollout 3.616s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.006s
ok github.com/carlosframework/platform/internal/rootparticipation 0.057s
ok github.com/carlosframework/platform/internal/rootstate 0.026s
ok github.com/carlosframework/platform/internal/runs 0.569s
ok github.com/carlosframework/platform/internal/s3compat 0.004s
ok github.com/carlosframework/platform/internal/safefile 0.006s
ok github.com/carlosframework/platform/internal/schedule 2.473s
ok github.com/carlosframework/platform/internal/seeds 0.010s
ok github.com/carlosframework/platform/internal/sim/hostlife 8.445s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.368s
ok github.com/carlosframework/platform/internal/status 0.039s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.244s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.003s
ok github.com/carlosframework/platform/internal/termfence 0.003s
ok github.com/carlosframework/platform/internal/tunnelclient 0.008s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.001s
ok github.com/carlosframework/platform/internal/vet 1.545s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48824482 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/20 14:03:33 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E041D0093A0BB4, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index eb8387e9b..15b7c785d 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -87,7 +87,8 @@ printf 'CARLOS_CONSOLE_ID=dev-console\nCONSOLE_HOME_REGION=ie\nCARLOS_HOST_CLAIM |
| mv -T /etc/carlos/.claim-console.env.pause /etc/carlos/claim-console.env |
| systemctl restart console.service |
| systemctl is-active console.service |
| -tr '\0' '\n' < "/proc/$(systemctl show -p MainPID --value console.service)/environ" | grep '^CARLOS_HOST_CLAIM=' |
| +for i in $(seq 1 40); do p=$(systemctl show -p MainPID --value console.service); [ "$(readlink /proc/$p/exe)" = /opt/carlos/system/console/console ] && break; sleep 0.25; done |
| +tr '\0' '\n' < "/proc/$p/environ" | grep '^CARLOS_HOST_CLAIM=' |
| ``` |
| Expected: `active`, then `CARLOS_HOST_CLAIM=` (empty). `/healthz` answers 200 within a few seconds, and the journal since the restart has no `off on a host-claim console` line. If the first hash line fails, the file is not the `=1` form `console-config` wrote: stop and report, change nothing. |
| @@ -113,7 +114,7 @@ BACKFILL_ID="cohort1-${WINDOW_ID%%T*}" |
| `BACKFILL_ID` is one lower-case path segment, `cohort1-` and the window's date. If the window moves, both move with it, and a window that has started needs a new ID (8.1's `mkdir` refuses an existing `$WINDOW`). |
| -**The order, in the morning, each step behind its own gate:** copy approval and merge (G1); tag and publish v0.28.5; block B5 (G2); signing session 6 opened, with the P-B2 grants and `publish-identity` for v0.28.5 (G3); leo rolled to v0.28.5 (G4); blocks B1 and B2 (G5); the window: 8.1, 8.1b, 8.2, 8.2b, 8.4 (G6 to G8), then the signing session is closed; step 9 (G9); block B4 (G10); acceptance 1 to 4 on `claimtrial2` (G11); Paul's gos for 10.5 and 10.6. |
| +**The order, in the morning, each step behind its own gate:** copy approval and merge (G1); tag and publish v0.28.5; 5.0a item 4 for v0.28.5: ship and promote the prebuilt v0.28.5 console to `canary/console` (`CV` = v0.28.5's commit short sha), as 5.0a did for v0.28.4; block B5 (G2); signing session 6 opened, with the P-B2 grants and `publish-identity` for v0.28.5, using that console version (`CV`) (G3); leo rolled to v0.28.5 (G4); blocks B1 and B2 (G5); the window: 8.1, 8.1b, 8.2, 8.2b, 8.4 (G6 to G8), then the signing session is closed; step 9 (G9); block B4 (G10); acceptance 1 to 4 on `claimtrial2` (G11); Paul's gos for 10.5 and 10.6. |
| **Two ordering rules (platform-1c, confirmed in the harness, 2026-10-20):** |
| - **The identity before the roll.** Once a floor is published, every console-driven `system update`, promote or rollback reserves the update in the signed roster (`reserveHostUpdate`, `internal/console/hostactivation.go:31`), and it resolves the release's published identity by tag. So `publish-identity` for v0.28.5 (G3) must come before leo's roll (G4), as above. |
| @@ -128,7 +129,7 @@ Each failure action is the whole action. Nothing is retried in place unless the |
| - [ ] **G1: copy, merge, CI.** Pass: Paul approves the strings (D7), the branch is merged, and CI is green on the squashed tip. Then tag and publish v0.28.5, dev only (D3). On failure: hold. No release, and the window does not open. |
| - [ ] **G2: prechecks (block B5).** Pass: `apps/bab//` has no current object; no backfill plan; no claims; no `admission.json`; the roster's `regions.ie.pools` contains `cohort1`; sophie and dennis have no `cohort1` row and no `CARLOS_POOLS`; the three stopped instances carry no box role; the dev console is claims-off. Then open the signing session. On failure: fix the one item if it is the known cleanup (with Paul's go), otherwise hold. |
| - **Known blocker, found 2026-10-20 and resolved by block B1: "G2 FAIL: records without acq".** Tonight's read-only run (`after236/precheck-20261020.md`, check 2b) found three instance records, `claimtrial`, `wakecheck` and `rollcheck-0261`, none with an `acq` field. `backfill-collect` refuses a record without one (`claimactivation/backfill.go:95`). platform-1e ruled a zero-host plan: block B1 deletes all three records (and the offline-move rehearsal app's), so the plan holds no hosts and no record needs an `acq`. Fable's "two records with acquisitions" is replaced by "no instance records" (B5's re-run after B1). |
| -- [ ] **G3: signing session.** Pass: the signer role has the P-B2 grants (each phase's dry run passes), and `publish-identity` for v0.28.5 wrote its identity create-only. Then roll leo. On failure: tear down, run the CloudTrail check, hold. |
| +- [ ] **G3: signing session.** Pass: v0.28.5's console is shipped and promoted to `canary/console` (5.0a item 4, `CV` = v0.28.5's commit short sha), the signer role has the P-B2 grants (each phase's dry run passes), and `publish-identity` for v0.28.5, with `identity.console.version` set to that `CV`, wrote its identity create-only. Then roll leo. On failure: tear down, run the CloudTrail check, hold. |
| - [ ] **G4: leo on v0.28.5, by re-arm in place** (ruled by platform-1c under Paul's re-arm-in-place choice, 2026-10-20). |
| - **Why not `system update`:** leo's installed v0.28.4 updater can never take a release on an armed box. The guard's `Disk.Run` reads combined output, so the old binaries' stderr breaks the version and plan parses (harness Run 1 and Run 4). The release carries the stdout-only fix (8a313548), but it only helps once it is installed. This road is only for a guard-armed box whose installed updater predates 8a313548. Today that is leo and sophie on v0.28.4, and sophie is not in this window. |
| - **Precondition:** the harness has passed this recovery AND a normal follow-up `system update` on the exact release commit. Harness run ID: `__________` (from harness-build's RUNLOG). |
| @@ -2583,7 +2584,7 @@ In this order: P-B2 is approved before signing session 6 opens; B5 runs before t |
| #### P-B2. The signer's grants (Paul's approval, D4) |
| Signing session 6 opens as `signing-host.md` "Create" says, with the release key on a host again. Its role adds, by exact key or prefix only, to what the fifth session had: |
| -- write on exactly `control/system/router/releases/v0.28.5.json`, for `publish-identity`; |
| +- write on exactly `control/system/router/releases/v0.28.5.json`, for `publish-identity` (its `identity.console.version` is the `CV` of the v0.28.5 console already promoted to `canary/console`, as the morning order says); |
| - `s3:GetObject` and `s3:PutObject` on `control/hosts-backfill.json`; |
| - `s3:GetObject` on `control/host-census/<BACKFILL_ID>/*` and `control/hosts-quarantine.json`; |
| - `s3:GetObject` on `apps/*/*/instances/*.json`, and `s3:ListBucket` with the prefixes `apps/` and `control/hosts/`. |
| @@ -2868,9 +2869,20 @@ Expected: the dry run prints one create-only write of `control/hosts-backfill.js |
| ### 8.2 Configure, migrate and start root |
| -Update the protected `/etc/carlos/host.env` through the approved configuration procedure: exactly one `CARLOS_HOST_CLAIM=1`, exactly one `CARLOS_HOST_CLAIM_ORIGINS=1`, no `CARLOS_MOVES_PHASE1`, no `CARLOS_REGION` (never a literal `CARLOS_REGION=ie` on a home box), and `CARLOS_POOLS=cohort1` kept from step 4. Keep the saved original (`$WINDOW/host.env.before`). Then: |
| +Update the protected `/etc/carlos/host.env` through the approved configuration procedure: exactly one `CARLOS_HOST_CLAIM=1`, exactly one `CARLOS_HOST_CLAIM_ORIGINS=1`, no `CARLOS_MOVES_PHASE1`, no `CARLOS_REGION` (never a literal `CARLOS_REGION=ie` on a home box), and `CARLOS_POOLS=cohort1` kept from step 4. Keep the saved original (`$WINDOW/host.env.before`). The edit itself, as commands, comes first in the block: |
| ```bash |
| +H=/etc/carlos/host.env |
| +cmp -- "$H" "$WINDOW/host.env.before" |
| +test "$(grep -c '^CARLOS_HOST_CLAIM=' "$H")" = 0 |
| +test "$(grep -c '^CARLOS_HOST_CLAIM_ORIGINS=' "$H")" = 0 |
| +printf 'CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n' >> "$H" |
| +test "$(grep -cx 'CARLOS_HOST_CLAIM=1' "$H")" = 1 |
| +test "$(grep -cx 'CARLOS_HOST_CLAIM_ORIGINS=1' "$H")" = 1 |
| +test "$(grep -c '^CARLOS_MOVES_PHASE1' "$H")" = 0 |
| +test "$(grep -c '^CARLOS_REGION' "$H")" = 0 |
| +test "$(grep -cx 'CARLOS_POOLS=cohort1' "$H")" = 1 |
| +stat -c '%n %a %U:%G %s' "$H"; diff "$WINDOW/host.env.before" "$H" || true |
| pub /opt/carlos/carlos system claims install-control --dry-run |
| pub /opt/carlos/carlos system claims install-control |
| systemctl cat carlos-instance-control.service |