amadan
docs: cohort-1 runbook: step 8 window package for Paul (not ready: S1 backfill plan, S2 migration census); umask 022 for 8.x binaries
Adds "Window package (for Paul)" at the top: window 20261021T0900Z to 11:00Z with a second-session deadline watcher; the barrier (revoke the two inbound rules on leo's own SG sg-054aa165db842b6ba, reverse, reboot and SSM reasoning); gate evidence read 2026-10-19 20:30Z to 20:46Z; the umask audit; and stop-and-hold as commands. Two STOPs found reading f3bf1214: journal-migrate needs a root-signed control/hosts-backfill.json at phase collecting that no shipped tool writes (404 on dev), and CheckMigrationRoots would refuse leo's real trees (the window's /dev/null masks, /data box files, wakecheck leftovers). 8.1 to 8.4 now run every platform binary through pub (umask 022), and 8.2 checks the guard/moves and guard/seeds modes after journal-migrate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+306
−5
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+306 −5
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 315.350s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.152s
ok github.com/carlosframework/platform/cmd/moveprobe 92.303s
ok github.com/carlosframework/platform/cmd/zddprobe 0.717s
ok github.com/carlosframework/platform/internal/activator 253.362s
ok github.com/carlosframework/platform/internal/agentsock 0.018s
ok github.com/carlosframework/platform/internal/alerting 0.010s
ok github.com/carlosframework/platform/internal/analytics 0.021s
ok github.com/carlosframework/platform/internal/appconfig 2.553s
ok github.com/carlosframework/platform/internal/apperr 0.106s
ok github.com/carlosframework/platform/internal/applog 0.178s
ok github.com/carlosframework/platform/internal/awsinv 0.006s
ok github.com/carlosframework/platform/internal/boxchoice 0.006s
ok github.com/carlosframework/platform/internal/boxid 0.655s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.014s
ok github.com/carlosframework/platform/internal/caps 0.015s
ok github.com/carlosframework/platform/internal/certwarm 0.003s
ok github.com/carlosframework/platform/internal/cgroup 0.736s
ok github.com/carlosframework/platform/internal/changelog 0.340s
ok github.com/carlosframework/platform/internal/claimactivation 0.034s
ok github.com/carlosframework/platform/internal/clamd 0.062s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 409.733s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.363s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.011s
ok github.com/carlosframework/platform/internal/dbverify 0.014s
ok github.com/carlosframework/platform/internal/dnsrec 0.003s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.470s
ok github.com/carlosframework/platform/internal/doorkey 0.009s
ok github.com/carlosframework/platform/internal/economics 0.006s
ok github.com/carlosframework/platform/internal/edge 45.005s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.009s
ok github.com/carlosframework/platform/internal/fleetapi 0.153s
ok github.com/carlosframework/platform/internal/gate 1.660s
ok github.com/carlosframework/platform/internal/genselect 0.026s
ok github.com/carlosframework/platform/internal/geo 0.022s
ok github.com/carlosframework/platform/internal/geoip 0.039s
ok github.com/carlosframework/platform/internal/historyreadback 0.180s
ok github.com/carlosframework/platform/internal/host 178.613s
ok github.com/carlosframework/platform/internal/hostactors 0.591s
ok github.com/carlosframework/platform/internal/hostbox 2.825s
ok github.com/carlosframework/platform/internal/hostclaim 0.581s
ok github.com/carlosframework/platform/internal/hostevidence 0.022s
ok github.com/carlosframework/platform/internal/hostexec 4.899s
ok github.com/carlosframework/platform/internal/hostfence 0.030s
ok github.com/carlosframework/platform/internal/hostfiles 10.373s
ok github.com/carlosframework/platform/internal/hostfloor 0.016s
ok github.com/carlosframework/platform/internal/hostguard 1.053s
ok github.com/carlosframework/platform/internal/hostops 1.676s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.015s
ok github.com/carlosframework/platform/internal/hostquarantine 0.016s
ok github.com/carlosframework/platform/internal/hostrelocate 0.494s
ok github.com/carlosframework/platform/internal/hostresolve 0.138s
ok github.com/carlosframework/platform/internal/hostwrite 0.015s
ok github.com/carlosframework/platform/internal/infraview 0.017s
ok github.com/carlosframework/platform/internal/instancectl 5.218s
ok github.com/carlosframework/platform/internal/instanceexec 2.747s
ok github.com/carlosframework/platform/internal/instancefs 0.006s
ok github.com/carlosframework/platform/internal/instancekey 0.003s
ok github.com/carlosframework/platform/internal/instanceupdates 0.007s
ok github.com/carlosframework/platform/internal/layout 1.560s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 9.812s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.007s
ok github.com/carlosframework/platform/internal/ledger 0.014s
ok github.com/carlosframework/platform/internal/mail 0.180s
ok github.com/carlosframework/platform/internal/maintenance 0.085s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.515s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.071s
ok github.com/carlosframework/platform/internal/metrics 0.007s
ok github.com/carlosframework/platform/internal/moves 21.061s
ok github.com/carlosframework/platform/internal/moves/archive 0.798s
ok github.com/carlosframework/platform/internal/objstore 0.178s
ok github.com/carlosframework/platform/internal/ops 0.167s
ok github.com/carlosframework/platform/internal/park 0.227s
ok github.com/carlosframework/platform/internal/privatefile 0.003s
ok github.com/carlosframework/platform/internal/provision 17.398s
ok github.com/carlosframework/platform/internal/publicrelease 0.007s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.463s
ok github.com/carlosframework/platform/internal/relay 7.229s
ok github.com/carlosframework/platform/internal/release 22.039s
ok github.com/carlosframework/platform/internal/rollout 3.682s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.061s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.586s
ok github.com/carlosframework/platform/internal/s3compat 0.005s
ok github.com/carlosframework/platform/internal/safefile 0.004s
ok github.com/carlosframework/platform/internal/schedule 2.502s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.065s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.983s
ok github.com/carlosframework/platform/internal/status 0.037s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.199s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.579s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 48824482 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.024s
=== RUN TestT4StorageProbeMinio
2026/10/19 21:05:36 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E00A4368E0B92A, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.021s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| index fbf16843d..fe7dbf274 100644 |
| --- a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -2,6 +2,277 @@ |
| Draft, first written 2026-10-16, revised 2026-10-17 against main at 26065378. Step 5.0a and step 6 were revised on 2026-10-18 against platform-1e's branch `claim-step6-artifacts` at `c38a4ca9`, and re-checked against main at `7c1beb2d` after it merged (published (v0.28.2, tag on 7c1beb2d)). Revised again on 2026-10-18 for the second signing session on v0.28.3, against main at `50f5b268` (`install-channel --policy`, from branch `claim-prev-rehearsal`) and the root-runner rehearsal of steps 6 and 7. Moved to v0.28.4 the same day, against main at `f3bf1214` (`install-channel --policy --identity`, from branch `claim-untagged-pointer`). It authorizes nothing. Every change below waits for Paul's go at its own checkpoint. Neither the approved copy quoted in step 8 nor the activation tools' tests authorize running anything. |
| +## Window package (for Paul): step 8, leo's attended offline window |
| + |
| +Prepared 2026-10-19 for platform-1e, who takes it to Paul. Every check below was read-only, from DOCS-T1 (profile `kass`, `user/remote-agent`, account `640012953448`) and over SSM on leo, between 20:30Z and 20:46Z. Nothing was changed on any box, bucket or security group, and nothing was created. Code was read at `f3bf1214` (v0.28.4, what leo runs). Main has one newer commit, `96db0efe` (#317, `guard-arm` directory modes), which step 8 does not run. |
| + |
| +**Verdict: not ready for a go.** Two findings in the code would stop the window at 8.2 ("STOPs before the go", below). Each needs platform-1e's answer before Paul is asked. The barrier, the deadline, the gates and the umask changes are ready. |
| + |
| +### STOPs before the go |
| + |
| +**S1. `journal-migrate` needs a signed backfill plan that nothing ships to write.** `carlos system journal-migrate` always runs the census path (`RunSystemJournalMigrate` passes `census=true`, `internal/host/claimjournal_migration.go:61`). Its census collector starts by reading `control/hosts-backfill.json` (`hostops.BackfillKey`) and refuses unless that object is root-signed with the release key, at `phase` `collecting`, with a `hosts` map that freezes every instance record in the deployment by key and etag, and `directory_revision` equal to the roster's (`rootMigrationCollector.collect`, `internal/host/rootmigration_census.go:37-75`). On dev the key is 404 (read 20:40Z). At `f3bf1214` nothing outside tests writes a `collecting` plan: the only non-test uses of `"collecting"` are the two readers, the root tests sign the plan with a fixture key (`rootmigration_bootstrap_root_linux_test.go:165-171`), and `claims run` has no such phase (its phases are `initialize`, `configure`, `home`, `invite-box`, `enroll-box`, `invite-console`, `enroll-console`, `bind-key`, `publish-floor`, `sign-policy` and `publish-identity`). The next stage is missing too: `Backfill.Publish`, which moves the plan to `adopting`, has no production caller (its comment names "the #232 coordinator"). The console adopts claims only while the plan reads `adopting` (`ReconcileClaims`, `internal/hostops/backfill_controller.go`), and `control/hosts/` on dev is empty. So as tooled, 8.2's `journal-migrate` fails in its first observation, before it writes `migration.json`. And even with the plan in place, nothing would give `claimtrial` the claim the seed's cold admission needs. The plan would have to freeze all three records in the bucket: `claimtrial`, `rollcheck-0261` and `wakecheck`. platform-1e: is a backfill writer and adopter owed (a `claims run` phase is a signing session, with the release key on a host again), or does the trial reach the census another way? |
| + |
| +**S2. The migration census walks the box's real trees, and leo's would not pass as written.** In its `ready` step, `journal-migrate` calls `hostbox.CheckMigrationRoots` over `/data`, `/opt/carlos` and `/etc/systemd/system` (`rootmigration_production_linux.go:156`). Any non-directory that is not in an installation's inventory and not in a shared rule is `ErrTypedInventory` (`internal/hostbox/migration_census_linux.go`; the rules are `claimGlobalSharedPaths`, `internal/host/claimcensus.go`). By that reading, three things on leo fail: |
| +- **8.1's own masks.** Every one of the twelve window units is an embedded unit, which the census classes as `CensusFile`: a regular file only (`censusRule.check`, `internal/hostbox/empty_census.go:304`). 8.1 puts a `/dev/null` symlink at each of those paths, inside the census root, and `journal-migrate` runs while they are there. (The design's mask was a runtime mask in `/run`, outside the root. P8 showed that one cannot work on leo.) |
| +- **`/data`.** 61 non-directories sit outside the shared classes (`.hostclaims`, `.hostexec`, `hostguard`, `relocate`, `moves/held-stage[d]`, `.moves` and the registry files). By top directory: `acme` 17 (including certificates for hosts long gone: `proof*`, `tunnel`, `website`, `helloworld`), `.instances` 15 and `instances` 15 (`claimtrial`'s, plus the stale `wakecheck` database step 4 left), `programs` 2, `zdd` 2, `monitor`, `carlos/geoip/current`, `system/router/requested.json`, `adopt-pass.lock`, `channel-first-seen.json`, `restart-marks.json` and its lock, `stage-requested`, `swap-failures.json` and `system-update-requested`. `claimtrial`'s own files might be covered by its installation's inventory once it is adopted. The box's own files and `wakecheck`'s leftovers belong to no installation. |
| +- **`/opt/carlos/live`.** It holds `claimtrial.dev.carlosframework.com` and `wakecheck.dev.carlosframework.com`, both links into `versions/wc4`, and the code says each link "belongs to a host and is unclassified". `wakecheck` has no route on leo. |
| + |
| +A refusal there leaves `migration.json` at phase `recovered`, an unfinished migration, with leo held and masked. No witness covers this: the migration root tests (`TestRootRebootPlan14Migration*`, `TestRootRebootPlan14MigrationInstalled*`) use scratch roots, never a box-shaped tree with the window's masks. platform-1e: confirm or correct this reading, ideally with one root-runner run of 8.1 then `journal-migrate` on a copy of leo's tree. Then rule on the fix: census rules, a different mask, and whether the `wakecheck` leftovers on leo (`/data/instances/wakecheck…`, `/data/.instances/wakecheck…`, `/opt/carlos/live/wakecheck…`, `/data/programs/wakecheck…json`) are cleaned up before the window, and by whom. |
| + |
| +**Ruling also wanted (R1).** May a window that stops before 8.2's `host.env` edit be undone, rather than held? Until then claims are still off and nothing claim-mode has run: put the twelve unit files back from `$WINDOW/held`, re-enable and start the edge and the triggers, and lift the barrier. The runbook says hold from 8.1 on. Given S1 and S2, an 8.1-only undo is worth deciding before the go. |
| + |
| +### 1. Window ID and deadline |
| + |
| +Proposed, for the first weekday after S1 and S2 are cleared and Paul's go is given, and no earlier than **2026-10-21**: |
| + |
| +| | Time (UTC) | What | |
| +|---|---|---| |
| +| Window ID | `20261021T0900Z` | `WINDOW='/etc/carlos/claim-window-cohort1-20261021T0900Z'`. If the date moves, the ID moves with it | |
| +| T−45 | 08:15Z | Second root session opened and the deadline watcher started (below); the gates re-run read-only (section 3); a fresh check that `sg-054aa165db842b6ba` is still on leo's ENI alone | |
| +| T−30 | 08:30Z | Barrier on (section 2), checked. From now on claimtrial gets no requests, so it hibernates within about 16 minutes (idle 15m plus the activator's tick) | |
| +| T−5 | 08:55Z | Gate G1: claimtrial hibernated and replicated. If it is not, wait. If it still is not by 09:15Z, abort before 8.1: lift the barrier, nothing else | |
| +| T0 | 09:00Z | 8.1. Budget 15 minutes | |
| +| | 09:15Z | 8.2. Budget 20 minutes | |
| +| Seed cut-off | 10:05Z | The last time 8.3 may start. 8.3 (override, seed and capture) has a 30-minute budget and 8.4 a 20-minute one. The data is one 24 KB database with about 2 KB of replica: no rate measured on leo exists, and these budgets are many times what that needs | |
| +| Soft end | 10:30Z | No new block starts except 8.4 or a hold | |
| +| **Hard end** | **11:00Z** | The deadline. Two hours from T0 | |
| + |
| +**Who enforces it.** A second root session on leo, held by someone other than the operator running the 8.x blocks. Proposed: platform-1e, with the controller as stand-in; Paul names the holder at the go. It is a separate `aws --profile kass --region eu-west-1 ssm start-session --target i-046c7fe43620944bc`, then `sudo -i`, opened at T−45 and kept until leo is masked and held, or step 9's go. The holder runs this watcher at T−45 (`date -d` is GNU date, which leo has): |
| + |
| +```bash |
| +WINDOW='/etc/carlos/claim-window-cohort1-20261021T0900Z' |
| +SEED=/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| +umask 077 |
| +until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T10:05:00Z +%s)" ]; do sleep 15; done; echo "SEED CUT-OFF $(date -u +%T)" |
| +until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T10:30:00Z +%s)" ]; do sleep 15; done; echo "SOFT END $(date -u +%T)" |
| +until [ "$(date -u +%s)" -ge "$(date -u -d 2026-10-21T11:00:00Z +%s)" ]; do sleep 15; done; echo "DEADLINE $(date -u +%T)" |
| +if [ -e "$SEED" ]; then |
| + systemctl show carlos-edge.service -p InvocationID --value > "$WINDOW/deadline-invocation" |
| + systemctl stop carlos-edge.service |
| + systemctl show carlos-edge.service -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/deadline-result" |
| + journalctl "_SYSTEMD_INVOCATION_ID=$(cat "$WINDOW/deadline-invocation")" --no-pager > "$WINDOW/deadline-journal" |
| +fi |
| +pgrep -a -f '^/opt/carlos/carlos system journal-migrate$' || echo "no journal-migrate" |
| +systemctl show carlos-edge.service carlos-instance-control.service -p Id -p LoadState -p ActiveState -p SubState |
| +``` |
| + |
| +**At the deadline, exactly this happens:** |
| +1. If the seed override is in place, the watcher captures the invocation ID first, then stops the one-shot and saves its result and journal under `$WINDOW` (`[runbook.cancellation]`). Otherwise it stops nothing. |
| +2. If `journal-migrate` is still running, the primary operator interrupts it (Ctrl-C, its documented cancel: SIGTERM or interrupt cancel the import). If the primary session is gone, the holder sends `kill -TERM <pid>` to the PID the watcher printed. Never `pkill -f`. |
| +3. The primary runs 8.4's first block (remask the edge, remove the override, put the guard drop-in back), or, if 8.3 had not started, nothing. If the primary session is gone, the holder runs that block, with `INVOCATION=$(cat "$WINDOW/deadline-invocation")` and the same `WINDOW`, `SEED` and `GUARD` values. |
| +4. Leo is then in "Stop and hold" with the edge persistently masked: step 9's hold state, which survives a reboot. The barrier stays on (section 5). Evidence is captured and posted on #250. No block starts after the deadline. Anything further needs a new go. |
| + |
| +### 2. External traffic barrier |
| + |
| +**Leo's security group is its own.** Read 20:30Z. Leo (`i-046c7fe43620944bc`) has one ENI, `eni-0a49b9ce1d65584d2`, with one security group, `sg-054aa165db842b6ba` (`carlos-dev-ie-b`, "carlos dev IE host B (backhaul proving ground)"). `describe-network-interfaces` filtered on that group returns leo's ENI only. No other security group in the account refers to it, and `describe-security-group-references` is empty. Its rules: inbound `sgr-0d3da6656deec14b0` (TCP 443 from `0.0.0.0/0`) and `sgr-032478c46083d5074` (TCP 80 from `0.0.0.0/0`), no IPv6, no SSH; outbound `sgr-03fb2a58524aa13ba` (all traffic to `0.0.0.0/0`). The VPC's other groups each serve one ENI of their own (`carlos-dev-ie` is sophie's), and `carlos-root-runner` and `default` serve none. The public address is Elastic IP `eipalloc-0cbb760a6cd15d71d` (18.200.150.210). On leo the edge listens on `*:80` and `*:443`, sshd on `:22` (no rule admits it), and there is no Tailscale. |
| + |
| +**Why not DNS.** Dev's DNS is a DNSimple wildcard (no Route 53 zone in the account). Every `*.dev.carlosframework.com` name, `claimtrial` included, resolves to sophie's 99.80.78.4, never to leo. So there is no DNS record to change, and DNS was never the path to leo anyway. Traffic reaches leo by its IP: leo mirrors `claimtrial`'s Let's Encrypt certificate from `control/certs/`, so a direct-IP request with that SNI is served with a valid certificate (step 4's `--resolve` checks do exactly that), and scanners hit the IP without SNI. VPC-internal traffic reaches 172.31.19.180 the same way. Only the security group stops all of it. |
| + |
| +**Mechanism: revoke the two inbound rules on leo's own group.** This creates nothing and touches no other box. |
| + |
| +Apply, from DOCS-T1, at T−30, after re-checking that the group is on leo's ENI alone: |
| + |
| +```bash |
| +R="--profile kass --region eu-west-1" |
| +aws $R ec2 describe-network-interfaces --filters Name=group-id,Values=sg-054aa165db842b6ba \ |
| + --query 'NetworkInterfaces[].[NetworkInterfaceId,Attachment.InstanceId]' --output text # exactly: eni-0a49b9ce1d65584d2 i-046c7fe43620944bc |
| +aws $R ec2 revoke-security-group-ingress --group-id sg-054aa165db842b6ba \ |
| + --security-group-rule-ids sgr-0d3da6656deec14b0 sgr-032478c46083d5074 |
| +aws $R ec2 describe-security-group-rules --filters Name=group-id,Values=sg-054aa165db842b6ba \ |
| + --query 'SecurityGroupRules[].[SecurityGroupRuleId,IsEgress,IpProtocol,FromPort,ToPort,CidrIpv4]' --output text |
| +``` |
| + |
| +Expected: `true` from the revoke, then exactly one rule, `sgr-03fb2a58524aa13ba True -1 -1 -1 0.0.0.0/0`. |
| + |
| +Check it, outside the sandbox: a bare TCP connect, so no TLS and no request, and nothing that could wake claimtrial: |
| + |
| +```bash |
| +for p in 80 443; do timeout 10 bash -c "exec 3<>/dev/tcp/18.200.150.210/$p" 2>/dev/null && echo "$p OPEN: STOP" || echo "$p closed"; done |
| +aws $R ssm describe-instance-information --filters Key=InstanceIds,Values=i-046c7fe43620944bc --query 'InstanceInformationList[].PingStatus' --output text |
| +``` |
| + |
| +Expected: `80 closed`, `443 closed`, `Online`. Then one read-only SSM command (`date -u`) returns `Success`. |
| + |
| +Reverse, only as section 5 says: |
| + |
| +```bash |
| +aws $R ec2 authorize-security-group-ingress --group-id sg-054aa165db842b6ba --ip-permissions \ |
| + 'IpProtocol=tcp,FromPort=80,ToPort=80,IpRanges=[{CidrIp=0.0.0.0/0}]' \ |
| + 'IpProtocol=tcp,FromPort=443,ToPort=443,IpRanges=[{CidrIp=0.0.0.0/0}]' |
| +aws $R ec2 describe-security-group-rules --filters Name=group-id,Values=sg-054aa165db842b6ba \ |
| + --query 'SecurityGroupRules[].[IsEgress,IpProtocol,FromPort,ToPort,CidrIpv4,Description]' --output text |
| +``` |
| + |
| +Expected: the two inbound rules back, with no description (new rule IDs), plus the egress rule. That is the shape the `region-host` module declares (inline `ingress` blocks at `infra/modules/region-host/main.tf:97-108`), so the dev root shows no drift for it afterwards. |
| + |
| +**Why it survives a reboot.** The rules live in AWS, on the security group attached to leo's ENI. Nothing on the box holds them, so a reboot, a crash or a stop and start of the instance leaves the barrier in place (the ENI, its group and the Elastic IP all stay attached). |
| + |
| +**SSM keeps working.** The SSM agent connects outwards on TCP 443 to the regional SSM endpoints (there are no VPC endpoints in this VPC). The egress rule is untouched, the subnet's network ACL allows all traffic both ways, and with no inbound rule left, replies to outbound connections are still admitted, because security groups are stateful. The same goes for S3, KMS and STS from the box, and for the instance metadata service, which is link-local. Agent `3.3.4851.0` was `Online` at 20:30:34Z. |
| + |
| +**The one way the barrier can disappear.** The `region-host` module declares these ingress rules inline, so any `tofu apply` in `platform-infrastructure/dev` that touches `module.host_ie_b.aws_security_group.host`, including a whole-root apply, puts them back. Rule: no dev apply that includes `module.host_ie_b` while the barrier is on. Dev takes targeted plans only for the trial already. The primary re-runs the `describe-security-group-rules` line before 8.3 and before step 9. |
| + |
| +**Alternatives, not proposed.** A dedicated barrier group swapped onto leo's ENI does the same, but creates a group and drifts `vpc_security_group_ids` instead. It would matter only if leo's group were shared, and it is not. Disassociating the Elastic IP would leave the VPC path open, and would change leo's public address. |
| + |
| +### 3. Pre-window gates (read-only evidence, 2026-10-19) |
| + |
| +| Gate | Status | Evidence | |
| +|---|---|---| |
| +| G1 `claimtrial`: a drained, replicated stop | PASS today; re-run at T−5 | See below | |
| +| G2 Nothing else on leo | Registry and units PASS; `/data` has leftovers (S2) | See below | |
| +| G3 No update or recovery pending | PASS | See below | |
| +| G4 P5 named-unit preflight | PASS | See below | |
| +| G5 Disk headroom | Measured | See below | |
| +| G6 Nested layout, no unfinished migration | PASS | See below | |
| +| G7 Census and claim-writer | Census: STOP (S1, S2). Claim writer: accepted trial gap | See below | |
| +| Second root session | Proposed (section 1) | none yet | |
| +| Barrier | Ready (section 2) | not applied | |
| + |
| +**G1, claimtrial: a drained, replicated stop.** How it is checked, at T−5 with the barrier already on: |
| +- the bare TCP connects show `closed`, so nothing reaches leo; |
| +- leo's edge journal has `activator: claimtrial.dev.carlosframework.com hibernated` after the barrier went on, and `systemctl list-units --all 'carlos-instance-gen@*'` lists nothing; |
| +- the last `replica sync` line before that has `txid.db` equal to `txid.replica`; |
| +- the newest level-0 object under `s3://carlos-dev-ie-640012953448/instances/claimtrial.dev.carlosframework.com/0000/` ends in that same txid; |
| +- `lock.json` beside it is gone (released). |
| + |
| +Commands: `journalctl -u carlos-edge.service --since -30min --no-pager | grep -E 'claimtrial.*(hibernat|started)|replica sync' | tail -5` on leo, then `aws --profile kass s3 ls <prefix>/0000/` and `s3 ls <prefix>/lock.json` from DOCS-T1. With claims off there is no `closed.json` marker. The litestream equality, the matching S3 object and the released lease are the evidence the trial has. |
| + |
| +Today: |
| +- At 20:32:51Z claimtrial was awake: `carlos-instance-gen@claimtrial.dev.carlosframework.com:13.service` `active running`, started 20:21:49Z by 7a.2's 20:21:48Z serving check. |
| +- From 20:32:26Z to 20:33:42Z every 2-second `replica sync` line read `txid.replica=0000000000000076 txid.db=0000000000000076`. S3's newest level-0 object was `0000000000000076-0000000000000076.ltx` (uploaded 20:22:00Z), and the L1 compaction covered `0x72` to `0x75`. |
| +- It hibernated at 20:39:53Z to 20:39:54Z (`idle 15m28s — hibernating`, then `hibernated`). The last sync line, at 20:39:54Z, read `0x76 = 0x76`. |
| +- At 20:44:21Z: `lock.json` 404, newest level-0 object still `…0076-…0076`. Replica position equals local: `0x76` both. |
| +- The previous cycle agrees: hibernated 19:53:23Z to 19:53:24Z, last sync `0x74 = 0x74`, and no sync lines between then and the next wake at 20:21:49Z. |
| + |
| +**G2, nothing else on leo** (20:32:51Z to 20:37:29Z): |
| +- Registry: one route, `claimtrial.dev.carlosframework.com` (`claimtrial`, `instance`, no unit, no backing, `hibernate=1`, provisioned, pool `cohort1`, no region, channel `edge`). |
| +- Units: no `carlos-app@` instance units and no `carlos-instance@*`. The only instance unit was `claimtrial`'s generation 13 (since gone). |
| +- `/etc/carlos/apps` empty. Processes as `carlos`: the agent (PID 495234) and claimtrial's two while it was awake. |
| +- `/data`: leftovers beyond claimtrial. The stale `wakecheck` database under `instances/` and `.instances/` (step 4 left it on purpose), `programs/wakecheck…json`, the `/opt/carlos/live/wakecheck…` link, and certificates of old hosts under `acme/`. None of them serves anything. They matter only to the census (S2). |
| +- `/data/instances/_probe` is the platform's reserved probe directory (`instancectl.ProbeDirName`). |
| + |
| +**G3, no update or recovery pending** (20:33:06Z, 20:44Z): |
| +- `applied.json`: `v0.28.4`, `stable`, 07:52:09Z. `result.json`: `outcome` `applied`, the same version, `promoted_at` 07:50:55Z. `seen.json`: `stable` 07:50:55Z. |
| +- The `stable` pointer read from the bucket: `current.version` `v0.28.4`, `linux-arm64` `736826e4`, `released_at` 07:50:55Z, untagged, which is expected. |
| +- `/data/system/router/requested.json` names that same release (`seen_at` 07:51:51Z). `/data/system-update-requested` (07:51:51Z) and `/data/stage-requested` (2026-10-18 10:03:49Z) are the agent's request markers for passes that have completed. `/data/swap-failures.json` has `records: null`. |
| +- Units: `carlos-system-update.service` and `carlos-system-recover.service` `loaded inactive`, result `success`. Recover's last run exited 0, with no entries this boot. No failed units, no queued jobs. |
| +- Every slot is `736826e4`, `carlos v0.28.4`. There is no genuine pending update evidence to keep. |
| + |
| +**G4, P5 named-unit preflight** (20:32:51Z, 20:44:21Z): registry count of `unit!=''` or `backing='unit'` routes `0`; `/etc/carlos/apps` `0`; `carlos-app@` instance units `0`. Bucket records: `bab/claimtrial`, `bab/rollcheck-0261` and `bab/wakecheck`; `claimtrial`'s has pool `cohort1`, no region, no `unit`, no `backing`. `apps/bab/` has no `PRE /`, and `apps/bab//` holds `0` objects. |
| + |
| +**G5, disk headroom** (20:33:19Z): |
| +- `/` (ext4 on `nvme0n1p1`, which holds `/data`, `/opt` and `/var/lib`): 20G, 1.6G used, 17G free (9%); inodes 4% used. |
| +- `/run` tmpfs 93M, 488K used. claimtrial's local data: 116K. Its replica: about 2 KB of LTX at levels 0 and 1, plus older compactions. |
| +- Memory: 927 MB total, 615 MB available. Swap is `/dev/zram0`, 463.5M, 6M used. |
| +- `carlos-instance-control.service` has `MemoryMax` 64M today (claims off). 8.2's unit sets 256M (H8). |
| + |
| +**G6, nested layout, no unfinished migration** (about 20:45Z, and leo's journal): |
| +- `carlos status --layout --console https://platform.dev.carlosframework.com` (the verified v0.28.4 amd64 CLI, `2d53162f`) prints `leo-climbing (nested by default) claimtrial.dev.carlosframework.com nested —`. |
| +- Provision logs `row is on the nested layout` for claimtrial on every pass (latest 20:32:58Z), and its database is at `/data/instances/claimtrial.dev.carlosframework.com/`. |
| +- No claim migration has started: `/var/lib/carlos-claims` is absent and `control/hosts/` is empty. `/var/lib/carlos-swap-status` is absent. `/var/lib/carlos-release` is `755 root:root`. |
| + |
| +**G7, census and claim writer.** |
| +- The census gate is a STOP (S1, S2). |
| +- The independent claim writer (Plan 15) is an accepted trial gap: Paul, 2026-10-16, recorded under "Scope" and "Accepted trial gaps" in this runbook, from `activation-scope-2026-10-16.md` as posted on #250 and #280. |
| +- Privileged tests that stand in for the rest, from the root runner (`carlos-root-runner`, throwaway arm64 EC2), on 2026-10-18 with tags `rootrunner,activationconsole`: |
| + - claim-step6-artifacts: `20261018T070118-3fa2a1` and `20261018T073324-a80825`, `TestRootActivationConsoleInstall` and `TestRootActivationConsoleRuntimePayload` PASS (`20261018T065544-81a49a` failed first, on a console image fixture). |
| + - claim-prev-rehearsal: `20261018T140405-72da7b` (both PASS, after two failed runs, one on the agent's nil-config panic and one on an assertion), `20261018T143015-99978e` (both PASS; the new `TestRootInstallChannelPolicyCensusBox` failed on a pre-layout fixture), `20261018T143955-9974bd` (`TestRootInstallChannelPolicyCensusBox` PASS), and `20261018T145201-fa9c03` (all three PASS). |
| + - claim-untagged-pointer: `20261018T171446-cadbbc` and `20261018T172526-d4ef51` (committed tree `255b0b3c`), all three PASS. |
| + - Before those, the integrated Plan 14 and tools root suite at `43ff0259` (2026-10-17): all 120 plain root tests and 15 reboot pairs PASS. That covers the migration pairs `Migration` and `MigrationInstalled`, `FilesSeedTwoHost`, `AcceptanceSeedBatch`, the lifecycle end-to-end pair and the `ServingPark` pairs. |
| +- What none of them covers: `journal-migrate` on a box-shaped tree under the window's `/dev/null` masks, a backfill plan made by a shipped tool, or the seed after a real migration (S1, S2). #250 itself could not be re-read: the hub did not answer at 20:47Z to 20:50Z. The names above come from platform-1e's reports in the claim notes. |
| + |
| +### 4. Umask audit |
| + |
| +The 8.x blocks run in one attended root shell that 8.1 starts with `umask 077`. Platform binaries those blocks run straight from that shell: `carlos system claims install-control --dry-run` (8.1, 8.2, 8.3, 8.4), `carlos system claims install-control` (8.2) and `carlos system journal-migrate` (8.2). The files seed is not one: it runs inside `carlos-edge.service` as `carlos`, under the unit's umask. No unit sets `UMask=` (`git grep UMask` at `f3bf1214` finds only `internal/edge/purge_listen_unix.go`), so that is systemd's `0022`. It reaches root only over the control socket. |
| + |
| +What each one creates (`f3bf1214`): |
| +- **`install-control --dry-run`** creates nothing. It returns before the lock and before any directory (`internal/host/activation_install_linux.go:260-262`). Its `verifyOfflineRootMigration` only reads `systemctl show` and `/proc`. |
| +- **`install-control`** writes three paths, none of them umask-dependent: the lock `/etc/systemd/system/.claims-activation.lock` (`0600`, checked), the unit `/etc/systemd/system/carlos-instance-control.service` (`safefile.WriteAt`, fchmod `0644`) and a `0700` staging directory it removes. The fourth path does depend on the umask: `/var/lib/carlos-swap-status`, made by `rootstate.Open(…, create, public)` (`internal/host/rootupdates_runtime.go:286`), and the agent (as `carlos`) reads it (`rootRetryStatus`, `relayRootRetry`). But it is made only once the census exists, which on the runbook's order (`install-control` before `journal-migrate`) it does not. **At risk only on a re-run.** |
| +- **`journal-migrate`** writes root-private state under `/var/lib/carlos-claims` (`0700`, which the claim unit's `StateDirectoryMode=0700` also expects; files fchmod `0600`), `/var/lib/carlos-release/private`, and `/data/.hostexec` (fchown `carlos`, fchmod `0700`). None of those is umask-dependent. **At risk:** `/opt/carlos/system/guard/moves` and `/opt/carlos/system/guard/seeds`, which its `ready` step creates as public rootstate directories (`internal/host/rootmovesfloor.go:58-62`, from `rootmigration_production_linux.go:171-177`). `rootstate` makes them with `Mkdirat(0755)`, which umask `077` turns into `0700`. It then refuses unless group and other have `r-x` (`internal/rootstate/state_linux.go:66-94`), and never chmods. So under `umask 077` the migration fails at `ready`: the census is written as pending, `migration.json` is left at `recovered`, and an empty `0700` `moves/` stays behind. Every rerun then fails on that directory, even under `umask 022`. If the directory were somehow left `0700`, the carlos agent could not read the floor projections there (`rootmovesfloor_source.go:158-166`, `rootseedsfloor.go:143-160`). Today neither directory exists on leo; `guard` holds `carlos`, `identities/` and `policy.json`. |
| +- **Written by the shell itself, not a binary:** `$WINDOW` and its copies (`0700`/`0600` on purpose, root-only), and the seed override and its directory under `/run/systemd/system/carlos-edge.service.d` (`0600`/`0700`, read only by systemd as root). Both are safe as they are. |
| + |
| +Changes now written into the blocks: |
| +- **8.1** defines `pub() ( umask 022; "$@" )` straight after `umask 077`, and every platform binary call in 8.1 to 8.4 goes through it: the four `install-control --dry-run` lines and `install-control` in 8.2. |
| +- **8.2**'s `journal-migrate` subshell starts with `umask 022`. It is followed by a mode check in the style of the guard-directory check: `stat` and `test` for `/var/lib/carlos-claims` `700 root:root`; for `/opt/carlos/system`, `guard`, `guard/moves` and `guard/seeds`, `755 root:root` plus `sudo -u carlos test -r` and `test -x`; `guard/carlos` executable by `carlos`; and `/var/lib/carlos-swap-status` `755 root:root` if it exists. Any mismatch is a STOP: no chmod by hand, no rerun, hold. |
| +- **8.3** notes why its override needs no change. |
| + |
| +`umask 022` loosens nothing private here: every root-only path above sets its mode explicitly. The one exception is `journals.db`, which becomes `0644` inside the `0700` `/var/lib/carlos-claims`, so it stays unreachable. Aside, not umask: `journal-migrate` resets `/data/.hostexec` from `guard-arm`'s `2770` to `0700 carlos:carlos`. Both work for the agent; the two pieces of code disagree. |
| + |
| +### 5. Rollback: stop and hold, as commands |
| + |
| +This is step 9's stop-and-hold path ("Stop and hold"), laid out by where the window stopped. Stopping is always allowed. Run it in the primary root session on leo, which keeps `WINDOW`, `SEED`, `GUARD`, `INVOCATION`, `UNITS` and `TRIGGERS` from 8.1 and 8.3. If that session is gone, set them again in the second session from the window ID and 8.1 and 8.3's literal values, with `set -eu`, `umask 077` and `pub() ( umask 022; "$@" )` first. |
| + |
| +**A. Barrier on, 8.1 not started (abort).** Leo is unchanged. Lift the barrier (section 2, "Reverse"). Done. |
| + |
| +**B. From 8.1's first `systemctl stop` until the seed override exists (8.1, 8.2).** |
| +- If the stop came before 8.1's mask loop finished, finish that loop first. The triggers are still enabled, so without their masks a reboot would start them: |
| + |
| + ```bash |
| + for unit in "${UNITS[@]}"; do |
| + if [ -L "/etc/systemd/system/$unit" ]; then continue; fi |
| + cmp -- "/etc/systemd/system/$unit" "$WINDOW/$unit.before" |
| + mv -T -- "/etc/systemd/system/$unit" "$WINDOW/held/$unit" |
| + ln -s -- /dev/null "/etc/systemd/system/$unit" |
| + done |
| + systemctl daemon-reload |
| + systemctl show "${UNITS[@]}" -p Id -p LoadState -p ActiveState |
| + ``` |
| + |
| +- Then leave every mask, `$WINDOW` and `carlos-instance-control.service` as they are (subject to ruling R1). |
| + |
| +**C. The seed override is in place (8.3).** This is the fragile state (H6): |
| + |
| +```bash |
| +INVOCATION=${INVOCATION:-$(systemctl show carlos-edge.service -p InvocationID --value)} |
| +printf '%s\n' "$INVOCATION" > "$WINDOW/hold-invocation" |
| +systemctl stop carlos-edge.service |
| +systemctl show carlos-edge.service -p Result -p ExecMainStatus -p ActiveState -p SubState > "$WINDOW/seed-result-final" |
| +journalctl "_SYSTEMD_INVOCATION_ID=$INVOCATION" --no-pager > "$WINDOW/seed-journal-final" |
| +cmp -- /etc/systemd/system/carlos-edge.service "$WINDOW/carlos-edge.service.before" |
| +mv -T -- /etc/systemd/system/carlos-edge.service "$WINDOW/held/carlos-edge.service" |
| +ln -s -- /dev/null /etc/systemd/system/carlos-edge.service |
| +systemctl daemon-reload |
| +systemctl show carlos-edge.service -p LoadState -p ActiveState |
| +cmp -- "$SEED" "$WINDOW/seed-override.before" |
| +rm -- "$SEED" |
| +test ! -e "$GUARD" && test ! -L "$GUARD" |
| +cmp -- "$WINDOW/held/90-host-floor.conf" "$WINDOW/edge-guard.before" |
| +mv -T -- "$WINDOW/held/90-host-floor.conf" "$GUARD" |
| +systemctl daemon-reload |
| +pub /opt/carlos/carlos system claims install-control --dry-run |
| +``` |
| + |
| +That is 8.4's first block, with the invocation captured before the stop. Do not run 8.4's second block: services and triggers stay masked. If a `cmp` fails, the file changed after the window: stop, move nothing, report. If leo rebooted while the override was in place, the override is gone and the edge unit is disabled and unguarded. Do not start anything: check `systemctl cat carlos-edge.service` and `ls -la /etc/systemd/system/carlos-edge.service.d "$WINDOW/held"`, then restore the edge mask and the guard drop-in as above (minus the stop and the override removal), and report. |
| + |
| +**D. After 8.4 (edge masked, waiting for step 9).** Nothing to stop. |
| + |
| +**E. Steps 9 and 10.** As "Stop and hold" says: `systemctl stop carlos-edge.service`, stop the five triggers, then `cmp`, move the edge unit to `$WINDOW/held`, link `/dev/null`, `daemon-reload` and check `LoadState=masked`. |
| + |
| +**Restoring the edge.** In every case the hold leaves the edge persistently masked: a `/dev/null` link at `/etc/systemd/system/carlos-edge.service`, the original in `$WINDOW/held`, its guard drop-in back at `90-host-floor.conf`, and the seed override gone. That state survives a reboot. Bringing the edge back to serving is step 9's block, and it needs its own go. |
| + |
| +**Evidence, every case.** On leo, read-only (the "Stop and hold" capture), then post the summary on #250: |
| + |
| +```bash |
| +systemctl show carlos-edge.service carlos-instance-control.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| +journalctl -u carlos-edge.service -u carlos-instance-control.service --since "-2h" --no-pager | tail -400 |
| +ls -la "$WINDOW" "$WINDOW/held" /var/lib/carlos-claims /etc/systemd/system/carlos-edge.service.d /run/systemd/system/carlos-edge.service.d 2>&1 |
| +cat /var/lib/carlos-claims/authority/migration/migration.json 2>&1 | head -c 400 |
| +cat /proc/sys/kernel/random/boot_id |
| +``` |
| + |
| +**Lifting the barrier.** The runbook keeps it through any hold, and that stays the default: the barrier comes off only in step 9, after its serving check, with step 9's go. If Paul wants leo's group back to its declared state while leo is held, the precondition, read on leo, is `systemctl show carlos-edge.service -p LoadState --value` printing `masked`, and `ss -ltn '( sport = :80 or sport = :443 )' | sed 1d | wc -l` printing `0`. Then run section 2's "Reverse" and check that the two inbound rules are back. With the edge masked nothing listens on 80 or 443, so lifting it then exposes nothing. |
| + |
| +**What must not be touched while leo is held.** Everything in "Stop and hold"'s list: `$WINDOW` and `held/`, `host.env`, `/root/claim-trial`, the mask links, unit files and drop-ins, `/var/lib/carlos-claims`, `host-actor.json`, `host-evidence.key`, `fence-writer/`, the guard, both `carlos` slots, `/data`, sophie's protected console, the identities, the roster, floor and enrollments, and the claim's S3 objects. No `reset-failed` before results are saved, no reboot unless the edge is persistently masked, and no rerun of `journal-migrate`, `install-control` or the seed. Added for the window: |
| +- `sg-054aa165db842b6ba` changes only through section 2's two commands; |
| +- no dev `tofu apply` that includes `module.host_ie_b`; |
| +- leave the Elastic IP association, sophie's group `sg-088e3aa23d6f2feba` and sophie herself alone; |
| +- do not delete the window's deadline files (`$WINDOW/deadline-*`, `hold-invocation`). |
| + |
| +### Notes found on the way (not blockers) |
| + |
| +- **Leo's swap is zram.** `swapon --show` lists `/dev/zram0`, compressed RAM, not a disk device. "Owed before production cohorts" says leo has swap and that a tmpfs page could reach disk. With zram it stays in memory, so that note may be overstated for leo. It still holds for any box with disk swap. |
| +- **Drop-ins the masks also hide.** `carlos-system-update.service` and `carlos-system-recover.service` carry `guard-arm`'s `90-host-floor.conf` drop-ins as well as the edge. 8.1's masks hide them, and 8.4 brings them back with their units. Only the edge's is moved, as 8.3 says. |
| +- **Leo has other ACME certificates.** Leo holds `acme/` certificates for `platform.dev`, `proof*`, `tunnel`, `website` and `helloworld`. They are harmless, but they are part of S2's `/data` list. |
| + |
| ## Go package (for Paul): the v0.28.4 session |
| Final, 2026-10-19. Every value below was read or verified read-only on DOCS-T1 between 07:50Z and 07:55Z; nothing was changed on any box or bucket. Earlier packages are in this file's history: v0.28.3 at `ccaea17f`, v0.28.2 at `72b5ab3e`. |
| @@ -2010,6 +2281,8 @@ Before the window: |
| - any update or recovery on leo is settled; genuine pending update evidence is kept; |
| - the named-unit preflight still holds: leo has no named-unit or `backing=unit` route (P5). |
| +The window ID, deadline, barrier, gate evidence, umask audit and hold commands are in "Window package (for Paul)" at the top of this file. Its STOPs S1 and S2 must be cleared before the go. |
| + |
| The approved description of the step: |
| [runbook.heading] |
| @@ -2049,6 +2322,11 @@ Choose a new window directory outside systemd's unit search path. The commands a |
| ```bash |
| set -eu |
| umask 077 |
| +# umask 077 is for this shell's own files ($WINDOW and the copies in it). |
| +# Every platform binary in 8.1 to 8.4 runs through pub, in a subshell with |
| +# umask 022, so a directory the binary means to be public is not made 0700 |
| +# (#317; the window package's umask audit). |
| +pub() ( umask 022; "$@" ) |
| WINDOW='/etc/carlos/claim-window-cohort1-<approved-window-id>' |
| TRIGGERS=(carlos-adopt.timer carlos-adopt.path carlos-stage.path carlos-restart.path carlos-system-update.path) |
| SERVICES=(carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service) |
| @@ -2084,7 +2362,7 @@ for unit in "${UNITS[@]}"; do |
| done |
| systemctl daemon-reload |
| systemctl show "${UNITS[@]}" -p Id -p LoadState -p ActiveState |
| -/opt/carlos/carlos system claims install-control --dry-run |
| +pub /opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| Expected: every unit `LoadState=masked` and inactive. `install-control --dry-run` calls main's unchanged `verifyOfflineRootMigration`: all six migration services must be `LoadState=masked` and inactive or failed, and no root `adopt`, `system apply` or `guard-arm` process may remain. The operator checks the triggers, the edge and the tenant drain. Do not relax the verifier or read it as proof of replication. |
| @@ -2094,15 +2372,35 @@ Expected: every unit `LoadState=masked` and inactive. `install-control --dry-run |
| Update the protected `/etc/carlos/host.env` through the approved configuration procedure: exactly one `CARLOS_HOST_CLAIM=1`, exactly one `CARLOS_HOST_CLAIM_ORIGINS=1`, no `CARLOS_MOVES_PHASE1`, no `CARLOS_REGION` (never a literal `CARLOS_REGION=ie` on a home box), and `CARLOS_POOLS=cohort1` kept from step 4. Keep the saved original (`$WINDOW/host.env.before`). Then: |
| ```bash |
| -/opt/carlos/carlos system claims install-control --dry-run |
| -/opt/carlos/carlos system claims install-control |
| +pub /opt/carlos/carlos system claims install-control --dry-run |
| +pub /opt/carlos/carlos system claims install-control |
| systemctl cat carlos-instance-control.service |
| ( |
| + umask 022 |
| set -a |
| . /etc/carlos/host.env |
| set +a |
| /opt/carlos/carlos system journal-migrate |
| ) |
| +# Mode check after journal-migrate (window package, umask audit). Its census |
| +# step creates /opt/carlos/system/guard/moves and .../seeds as public |
| +# directories (rootstate.Open(dir, true, true): Mkdirat 0755, then it |
| +# refuses unless group and other have r-x, and it never chmods). The agent, |
| +# running as carlos, reads the floor projections there. Any mismatch is a |
| +# STOP: do not chmod by hand, do not rerun, hold. |
| +stat -c '%n %a %U:%G' /var/lib/carlos-claims /opt/carlos/system /opt/carlos/system/guard \ |
| + /opt/carlos/system/guard/moves /opt/carlos/system/guard/seeds |
| +test "$(stat -c '%a %U:%G' /var/lib/carlos-claims)" = '700 root:root' |
| +for d in /opt/carlos/system /opt/carlos/system/guard /opt/carlos/system/guard/moves /opt/carlos/system/guard/seeds; do |
| + test "$(stat -c '%a %U:%G' "$d")" = '755 root:root' |
| + sudo -u carlos test -r "$d" |
| + sudo -u carlos test -x "$d" |
| +done |
| +sudo -u carlos test -x /opt/carlos/system/guard/carlos |
| +if [ -e /var/lib/carlos-swap-status ]; then |
| + test "$(stat -c '%a %U:%G' /var/lib/carlos-swap-status)" = '755 root:root' |
| + sudo -u carlos test -x /var/lib/carlos-swap-status |
| +fi |
| systemctl daemon-reload |
| systemctl add-wants multi-user.target carlos-instance-control.service |
| systemctl restart carlos-instance-control.service |
| @@ -2116,6 +2414,7 @@ Expected: |
| - `install-control` prints `"target": "/etc/systemd/system/carlos-instance-control.service"`, `"reload": true` and `"start": false`; |
| - the unit keeps its protected environment, `StateDirectory=carlos-claims` and `MemoryMax=256M`; |
| - `journal-migrate` succeeds and publishes the census; `/var/lib/carlos-claims/authority/migration/migration.json` exists; |
| +- the mode check after it: `/var/lib/carlos-claims` `700 root:root`; `/opt/carlos/system`, `guard`, `guard/moves` and `guard/seeds` each `755 root:root` and readable and searchable by `carlos`; `guard/carlos` executable by `carlos`; `/var/lib/carlos-swap-status`, if it exists, `755 root:root`. Under `set -e` a failed `test` ends the block: that is a STOP; |
| - `active`, and no migration, recovery or floor error in the journal. |
| Keep every migration and root record. A failure leaves serving held. |
| @@ -2243,9 +2542,11 @@ mv -T -- "$WINDOW/held/carlos-edge.service" /etc/systemd/system/carlos-edge.serv |
| systemctl daemon-reload |
| systemctl cat carlos-edge.service |
| systemctl show carlos-edge.service -p ExecStart -p Requires -p After -p User -p Group -p Delegate -p EnvironmentFiles |
| -/opt/carlos/carlos system claims install-control --dry-run |
| +pub /opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| +The seed override and its directory are written by this shell, under `umask 077`, so they are `600` and `700` root. Only systemd reads them, as root, so that is intended and needs no check. The seed itself runs inside `carlos-edge.service` as `carlos`, with the unit's umask (no unit sets `UMask=`, so systemd's `0022`), not this shell's. |
| + |
| Before starting, require the exact seed `ExecStart`, the original service user, protected environment, delegation and hardening, no dependency on a masked service, and an active claim-on root service. The normal edge stays stopped. The six migration masks and all five trigger masks stay in place. |
| ```bash |
| @@ -2353,7 +2654,7 @@ test ! -e "$GUARD" && test ! -L "$GUARD" |
| cmp -- "$WINDOW/held/90-host-floor.conf" "$WINDOW/edge-guard.before" |
| mv -T -- "$WINDOW/held/90-host-floor.conf" "$GUARD" |
| systemctl daemon-reload |
| -/opt/carlos/carlos system claims install-control --dry-run |
| +pub /opt/carlos/carlos system claims install-control --dry-run |
| ``` |
| After every seed and migration check passes, restore only this window's recorded originals for the six services, then the five triggers. For each, verify the saved bytes and metadata, require its current path to be this window's `/dev/null` link, remove the link and move the original back. Do not start the triggers yet. |