carlos / platform Public

runbook: G4 is the ruled re-arm-in-place recovery for leo (old updater can't take a release on an armed box); #322

claude 4 days ago cbf495a2c7cdf5fff6e668ca98bda14d5bcfe0c1 parent 6700ec3

1 file changed, +18 −3

docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md +18 −3

CI log

=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok  	github.com/carlosframework/platform	314.083s
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.071s
ok  	github.com/carlosframework/platform/cmd/moveprobe	92.201s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.702s
ok  	github.com/carlosframework/platform/internal/activator	251.708s
ok  	github.com/carlosframework/platform/internal/agentsock	0.014s
ok  	github.com/carlosframework/platform/internal/alerting	0.003s
ok  	github.com/carlosframework/platform/internal/analytics	0.019s
ok  	github.com/carlosframework/platform/internal/appconfig	2.499s
ok  	github.com/carlosframework/platform/internal/apperr	0.071s
ok  	github.com/carlosframework/platform/internal/applog	0.124s
ok  	github.com/carlosframework/platform/internal/awsinv	0.004s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.004s
ok  	github.com/carlosframework/platform/internal/boxid	0.889s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.011s
ok  	github.com/carlosframework/platform/internal/caps	0.012s
ok  	github.com/carlosframework/platform/internal/certwarm	0.002s
ok  	github.com/carlosframework/platform/internal/cgroup	0.695s
ok  	github.com/carlosframework/platform/internal/changelog	0.303s
ok  	github.com/carlosframework/platform/internal/claimactivation	0.025s
ok  	github.com/carlosframework/platform/internal/clamd	0.054s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
ok  	github.com/carlosframework/platform/internal/console	410.698s
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	6.131s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.009s
ok  	github.com/carlosframework/platform/internal/dbverify	0.004s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.002s
ok  	github.com/carlosframework/platform/internal/docsite	0.004s
ok  	github.com/carlosframework/platform/internal/domains	2.402s
ok  	github.com/carlosframework/platform/internal/doorkey	0.006s
ok  	github.com/carlosframework/platform/internal/economics	0.004s
ok  	github.com/carlosframework/platform/internal/edge	44.981s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.008s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.133s
ok  	github.com/carlosframework/platform/internal/gate	1.655s
ok  	github.com/carlosframework/platform/internal/genselect	0.015s
ok  	github.com/carlosframework/platform/internal/geo	0.017s
ok  	github.com/carlosframework/platform/internal/geoip	0.032s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.156s
ok  	github.com/carlosframework/platform/internal/host	175.522s
ok  	github.com/carlosframework/platform/internal/hostactors	0.517s
ok  	github.com/carlosframework/platform/internal/hostbox	2.620s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.558s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.015s
ok  	github.com/carlosframework/platform/internal/hostexec	4.735s
ok  	github.com/carlosframework/platform/internal/hostfence	0.033s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.418s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.015s
ok  	github.com/carlosframework/platform/internal/hostguard	1.026s
ok  	github.com/carlosframework/platform/internal/hostops	1.652s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.014s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.015s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.441s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.145s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.014s
ok  	github.com/carlosframework/platform/internal/infraview	0.015s
ok  	github.com/carlosframework/platform/internal/instancectl	5.223s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.738s
ok  	github.com/carlosframework/platform/internal/instancefs	0.018s
ok  	github.com/carlosframework/platform/internal/instancekey	0.015s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.006s
ok  	github.com/carlosframework/platform/internal/layout	1.568s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.003s
ok  	github.com/carlosframework/platform/internal/lease	9.793s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.005s
ok  	github.com/carlosframework/platform/internal/ledger	0.012s
ok  	github.com/carlosframework/platform/internal/mail	0.181s
ok  	github.com/carlosframework/platform/internal/maintenance	0.102s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.475s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.077s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	21.145s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.798s
ok  	github.com/carlosframework/platform/internal/objstore	0.183s
ok  	github.com/carlosframework/platform/internal/ops	0.179s
ok  	github.com/carlosframework/platform/internal/park	0.199s
ok  	github.com/carlosframework/platform/internal/privatefile	0.003s
ok  	github.com/carlosframework/platform/internal/provision	17.944s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.006s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.415s
ok  	github.com/carlosframework/platform/internal/relay	7.230s
ok  	github.com/carlosframework/platform/internal/release	21.738s
ok  	github.com/carlosframework/platform/internal/rollout	3.659s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.006s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.058s
ok  	github.com/carlosframework/platform/internal/rootstate	0.026s
ok  	github.com/carlosframework/platform/internal/runs	0.572s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.497s
ok  	github.com/carlosframework/platform/internal/seeds	0.010s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	8.473s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.436s
ok  	github.com/carlosframework/platform/internal/status	0.038s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.210s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.004s
ok  	github.com/carlosframework/platform/internal/termfence	0.004s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.003s
ok  	github.com/carlosframework/platform/internal/updatewords	0.004s
ok  	github.com/carlosframework/platform/internal/usage	0.005s
ok  	github.com/carlosframework/platform/internal/version	0.001s
ok  	github.com/carlosframework/platform/internal/vet	1.549s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48824482 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
  minio: cached, digest verified
  mc: cached, digest verified
  minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
  buckets and scoped user ready
=== RUN   TestDeliveredContractMinio
=== RUN   TestDeliveredContractMinio/put
=== RUN   TestDeliveredContractMinio/get_round-trips_the_body
=== RUN   TestDeliveredContractMinio/list_shows_the_key
=== RUN   TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN   TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN   TestDeliveredContractMinio/delete_removes_it
=== RUN   TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
    --- PASS: TestDeliveredContractMinio/put (0.00s)
    --- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
    --- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
    --- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
    --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
        --- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
    --- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
    --- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok  	github.com/carlosframework/platform/internal/objstore	0.025s
=== RUN   TestT4StorageProbeMinio
2026/10/20 08:55:05 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E030FADC7B102B, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok  	github.com/carlosframework/platform/internal/host	0.021s
minio contract: PASS

diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md

index 8c0476b8b..37ab6e580 100644

--- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md

+++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md

@@ -129,9 +129,24 @@ Each failure action is the whole action. Nothing is retried in place unless the

- [ ] **G2: prechecks (block B5).** Pass: `apps/bab//` has no current object; no backfill plan; no claims; no `admission.json`; the roster's `regions.ie.pools` contains `cohort1`; sophie and dennis have no `cohort1` row and no `CARLOS_POOLS`; the three stopped instances carry no box role; the dev console is claims-off. Then open the signing session. On failure: fix the one item if it is the known cleanup (with Paul's go), otherwise hold.

- **Known blocker, found 2026-10-20 and resolved by block B1: "G2 FAIL: records without acq".** Tonight's read-only run (`after236/precheck-20261020.md`, check 2b) found three instance records, `claimtrial`, `wakecheck` and `rollcheck-0261`, none with an `acq` field. `backfill-collect` refuses a record without one (`claimactivation/backfill.go:95`). platform-1e ruled a zero-host plan: block B1 deletes all three records (and the offline-move rehearsal app's), so the plan holds no hosts and no record needs an `acq`. Fable's "two records with acquisitions" is replaced by "no instance records" (B5's re-run after B1).

- [ ] **G3: signing session.** Pass: the signer role has the P-B2 grants (each phase's dry run passes), and `publish-identity` for v0.28.5 wrote its identity create-only. Then roll leo. On failure: tear down, run the CloudTrail check, hold.

-- [ ] **G4: leo on v0.28.5.**

- - **Precondition:** the harness has passed G4 on the exact release commit, through an armed box with the old updater. Harness run ID: `__________` (from harness-build's RUNLOG).

- - **Pass:** `carlos system update` completes, the guard is active, `carlos.prev` is v0.28.4, and leo's box report shows v0.28.5 and verifies (`status`). Then block B1.

+- [ ] **G4: leo on v0.28.5, by re-arm in place** (ruled by platform-1c under Paul's re-arm-in-place choice, 2026-10-20).

+ - **Why not `system update`:** leo's installed v0.28.4 updater can never take a release on an armed box. The guard's `Disk.Run` reads combined output, so the old binaries' stderr breaks the version and plan parses (harness Run 1 and Run 4). The release carries the stdout-only fix (8a313548), but it only helps once it is installed. This road is only for a guard-armed box whose installed updater predates 8a313548. Today that is leo and sophie on v0.28.4, and sophie is not in this window.

+ - **Precondition:** the harness has passed this recovery AND a normal follow-up `system update` on the exact release commit. Harness run ID: `__________` (from harness-build's RUNLOG).

+ - **The recovery, one box at a time, attended:**

+ 1. On leo: `systemctl stop carlos-system-update.path`. This comes BEFORE the promote, so the old updater never attempts the release and never marks it refused.

+ 2. Promote v0.28.5 to router/stable, then `refresh` (the roster revision moves).

+ 3. Wait for leo's agent to fetch the cache, `/data/system/router/v0.28.5/carlos`. Check its sha256 against the signed pointer. Stage the cache by hand (carlos-owned, same sha) ONLY if it is missing, and record which happened.

+ 4. Stage the release's linux-arm64 binary at `/root/carlos-v0.28.5`, root 0700. Its sha256 must match BOTH the signed `SHA256SUMS` and the published identity's router sha, or STOP.

+ 5. `systemd-run --wait --pipe -p EnvironmentFile=-/etc/carlos/host.env -p UMask=0022 /root/carlos-v0.28.5 system apply`, with nothing added.

+ 6. Evidence:

+ - `applied.json` and `result.json` read `applied`;

+ - the guard identities include v0.28.5;

+ - `/proc/<edge pid>/exe` hashes to the release, and `NRestarts` is 0;

+ - leo's next report shows `activation_complete` true;

+ - the roster's `activations` is empty;

+ - `carlos system status` shows leo current.

+ 7. `systemctl start carlos-system-update.path`, then `shred -u /root/carlos-v0.28.5`, and record it in the window log.

+ - **Pass:** step 6's evidence, the guard active, and `carlos.prev` v0.28.4. Then block B1.

- **On failure:** STOP. The router/stable activation stays held. Promote and rollback of stable are refused until a recovery ruling, so don't attempt a rollback. Leo and dennis keep serving on v0.28.4. Report the activation ID and leo's refusal line. (A refused roll on an armed box records no completion. `settleGuard` withdraws root's marker and the agent retries the same refused candidate, `cancel-activation` refuses a published activation, and only completion or a quarantine or fence releases it: harness Run 1. Dennis shares stable, so his updates freeze too; that's accepted for dev.)

- [ ] **G5: blocks B1 and B2.** Pass: no instance record in the bucket; leo's registry has zero rows of any kind, counted over every table, apex and domain rows included (B1's check prints `total 0`); sophie's has no `wakecheck` row; nothing left under `/data/.instances`, `/data/instances`, `/data/programs` or `/opt/carlos/live` for any B2 host, on any box. Then the window. On failure: hold. Nothing claim-mode has run.

- [ ] **G6: 8.1 and 8.1b.** Pass: 8.1's masks and `install-control --dry-run`; 8.1b's `backfill-collect`: zero hosts (`hosts` empty, not `null`), phase `collecting`, one create-only write. Then 8.2. On failure: case B1 undo (ruling R1): restore units, enablement and barrier. The plan, if written, stays, and is harmless with the console off.