amadan
docs: dev cohort-1 host-claim activation runbook (draft)
Draft runbook for turning host claims on for carlos-dev-ie-b (leo-climbing), with carlos-dev-uk as an optional second region. Read-only prechecks with today's results, steps 0 to 7 with a Paul's-go checkpoint before every change, acceptance checks, per-step and full rollback, known hazards and open questions. The files seed strings are quoted verbatim from Paul's approved copy review (2026-10-15). Nothing was changed on dev. Five blockers are listed: no runner for the roster, enrollment, floor or home designation; an empty app segment apps/bab// in the dev deployment bucket that makes the activation preflight refuse; runtime masks that lose to /etc unit files; leo's IAM role and the dev bucket policy behind main's modules; Plan 14 Parts 2 and 3 unmerged and unreleased.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+708
−0
docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md+708 −0
CI passed — run details
CI log
=== gate ===
cd launchctx && go vet ./... && go test ./... -count=1
ok amadan.net/carlos/platform.git/launchctx 0.011s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
go vet ./...
gofmt -l .: clean
go test ./... -count=1 -timeout 30m
ok github.com/carlosframework/platform 288.682s
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 2.724s
ok github.com/carlosframework/platform/cmd/moveprobe 93.240s
ok github.com/carlosframework/platform/cmd/zddprobe 0.715s
ok github.com/carlosframework/platform/internal/activator 249.259s
ok github.com/carlosframework/platform/internal/agentsock 0.019s
ok github.com/carlosframework/platform/internal/alerting 0.005s
ok github.com/carlosframework/platform/internal/analytics 0.033s
ok github.com/carlosframework/platform/internal/appconfig 2.595s
ok github.com/carlosframework/platform/internal/apperr 0.090s
ok github.com/carlosframework/platform/internal/applog 0.137s
ok github.com/carlosframework/platform/internal/awsinv 0.004s
ok github.com/carlosframework/platform/internal/boxchoice 0.004s
ok github.com/carlosframework/platform/internal/boxid 1.285s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.021s
ok github.com/carlosframework/platform/internal/caps 0.018s
ok github.com/carlosframework/platform/internal/certwarm 0.004s
ok github.com/carlosframework/platform/internal/cgroup 0.718s
ok github.com/carlosframework/platform/internal/changelog 0.321s
ok github.com/carlosframework/platform/internal/clamd 0.055s
? github.com/carlosframework/platform/internal/cliflag [no test files]
ok github.com/carlosframework/platform/internal/console 388.015s
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 6.329s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.009s
ok github.com/carlosframework/platform/internal/dbverify 0.005s
ok github.com/carlosframework/platform/internal/dnsrec 0.005s
ok github.com/carlosframework/platform/internal/docsite 0.005s
ok github.com/carlosframework/platform/internal/domains 2.345s
ok github.com/carlosframework/platform/internal/doorkey 0.006s
ok github.com/carlosframework/platform/internal/economics 0.005s
ok github.com/carlosframework/platform/internal/edge 26.705s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.007s
ok github.com/carlosframework/platform/internal/fleetapi 0.131s
ok github.com/carlosframework/platform/internal/gate 1.663s
ok github.com/carlosframework/platform/internal/genselect 0.014s
ok github.com/carlosframework/platform/internal/geo 0.009s
ok github.com/carlosframework/platform/internal/geoip 0.031s
ok github.com/carlosframework/platform/internal/host 162.813s
ok github.com/carlosframework/platform/internal/hostactors 0.331s
ok github.com/carlosframework/platform/internal/hostbox 2.512s
ok github.com/carlosframework/platform/internal/hostclaim 0.474s
ok github.com/carlosframework/platform/internal/hostevidence 0.015s
ok github.com/carlosframework/platform/internal/hostexec 4.933s
ok github.com/carlosframework/platform/internal/hostfence 0.033s
ok github.com/carlosframework/platform/internal/hostfiles 10.504s
ok github.com/carlosframework/platform/internal/hostfloor 0.015s
ok github.com/carlosframework/platform/internal/hostguard 1.012s
ok github.com/carlosframework/platform/internal/hostops 1.587s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.015s
ok github.com/carlosframework/platform/internal/hostquarantine 0.015s
ok github.com/carlosframework/platform/internal/hostrelocate 0.422s
ok github.com/carlosframework/platform/internal/hostresolve 0.132s
ok github.com/carlosframework/platform/internal/hostwrite 0.015s
ok github.com/carlosframework/platform/internal/infraview 0.015s
ok github.com/carlosframework/platform/internal/instancectl 4.178s
ok github.com/carlosframework/platform/internal/instanceexec 2.724s
ok github.com/carlosframework/platform/internal/instancefs 0.019s
ok github.com/carlosframework/platform/internal/instancekey 0.015s
ok github.com/carlosframework/platform/internal/instanceupdates 0.020s
ok github.com/carlosframework/platform/internal/layout 1.497s
ok github.com/carlosframework/platform/internal/lease 9.812s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.019s
ok github.com/carlosframework/platform/internal/ledger 0.011s
ok github.com/carlosframework/platform/internal/mail 0.157s
ok github.com/carlosframework/platform/internal/maintenance 0.080s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.496s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.073s
ok github.com/carlosframework/platform/internal/metrics 0.004s
ok github.com/carlosframework/platform/internal/moves 20.936s
ok github.com/carlosframework/platform/internal/moves/archive 0.778s
ok github.com/carlosframework/platform/internal/objstore 0.167s
ok github.com/carlosframework/platform/internal/ops 0.174s
ok github.com/carlosframework/platform/internal/park 0.200s
ok github.com/carlosframework/platform/internal/privatefile 0.007s
ok github.com/carlosframework/platform/internal/provision 16.984s
ok github.com/carlosframework/platform/internal/publicrelease 0.009s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 4.949s
ok github.com/carlosframework/platform/internal/relay 7.240s
ok github.com/carlosframework/platform/internal/release 20.090s
ok github.com/carlosframework/platform/internal/rollout 3.801s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.006s
ok github.com/carlosframework/platform/internal/rootparticipation 0.008s
ok github.com/carlosframework/platform/internal/rootstate 0.025s
ok github.com/carlosframework/platform/internal/runs 0.576s
ok github.com/carlosframework/platform/internal/s3compat 0.004s
ok github.com/carlosframework/platform/internal/safefile 0.003s
ok github.com/carlosframework/platform/internal/schedule 2.500s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/hostlife 9.595s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.039s
ok github.com/carlosframework/platform/internal/status 0.038s
ok github.com/carlosframework/platform/internal/steering 0.005s
ok github.com/carlosframework/platform/internal/stopevidence 0.235s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.002s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.004s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.566s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 47120546 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.022s
=== RUN TestT4StorageProbeMinio
2026/10/16 07:53:13 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18DEF34835B69B5F, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.020s
minio contract: PASS
diff --git a/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md| new file mode 100644 |
| index 000000000..0c28ef68a |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-16-claim-activation-dev-cohort1.md |
| @@ -0,0 +1,708 @@ |
| +# Host claim activation, dev cohort 1: runbook draft |
| + |
| +Draft, 2026-10-16. It authorizes nothing. Every change below waits for Paul's go at its own checkpoint, and the copy-approved strings quoted in step 5 are not an approval to run them. |
| + |
| +Cohort 1 is one dev box, `carlos-dev-ie-b`, with `carlos-dev-uk` as an optional second region. The dev console box, `carlos-dev-ie` (sophie-knitting), is excluded. It carries the `platform.dev.carlosframework.com` route with `unit=console.service`, and Plan 14 Part 3's activation preflight refuses any box with a named-unit or `backing=unit` route. |
| + |
| +Sources: |
| +- the binding scope, `activation-scope-2026-10-16.md` (posted on amadan #250 and #280; the file itself lives in platform-1e's scratch, not in the repo); |
| +- Plan 9 Task C2, milestones 2 to 4 (`docs/superpowers/plans/2026-10-08-host-lifecycle-claim-9-activation-readiness.md` on `design-replica-ownership`); |
| +- `docs/host-claim-restore-runbook.md` on main, including "Home-region activation order"; |
| +- the files seed, on main as ee35dee2, and its runbook strings, which Paul approved unchanged on 2026-10-15 (copy review `files-seed-copy-review/result.json` in platform-1e's scratch, action `approve`); |
| +- Plan 14 Parts 2 and 3, unmerged, on `feat/claim-root-trusted-state-p3` at 305866ad: `claimactivation_preflight.go`, `rootmigration*.go` and `claimjournal_migration.go`; |
| +- amadan #250, #272, #280 and #286. |
| + |
| +## Boxes |
| + |
| +| Name | Box | Instance | Region | Role | Today | |
| +|---|---|---|---|---|---| |
| +| `carlos-dev-ie-b` | leo-climbing | i-046c7fe43620944bc | eu-west-1, home, no `CARLOS_REGION` | `carlos-dev-host-b` | cohort 1 | |
| +| `carlos-dev-uk` | dennis-stenciling | i-009d593f451cfd562 | eu-west-2, `CARLOS_REGION=uk` | `carlos-dev-host-uk` | optional, empty | |
| +| `carlos-dev-ie` | sophie-knitting | i-05498eada6f0afd23 | eu-west-1, home | `carlos-dev-host` | excluded (console box) | |
| + |
| +Dev account 640012953448. AWS profile `kass` (a plain IAM user, so it works inside the sandbox). Deployment bucket `carlos-dev-640012953448`. Data buckets `carlos-dev-ie-640012953448` and `carlos-dev-uk-640012953448`. Console `https://platform.dev.carlosframework.com`. |
| + |
| +## How commands are run |
| + |
| +Box commands run as root over SSM. Save this helper as `$TMPDIR/cohort1/ssm.sh`: |
| + |
| +```bash |
| +#!/bin/bash |
| +# usage: ssm.sh <region> <instance-id> '<shell command>' |
| +set -euo pipefail |
| +region=$1 inst=$2 cmd=$3 |
| +b64=$(printf '%s' "$cmd" | base64 -w0) |
| +params=$(python3 -c 'import json,sys;print(json.dumps({"commands":["echo %s | base64 -d | bash" % sys.argv[1]]}))' "$b64") |
| +id=$(aws --profile kass --region "$region" ssm send-command --instance-ids "$inst" \ |
| + --document-name AWS-RunShellScript --parameters "$params" --query Command.CommandId --output text) |
| +until s=$(aws --profile kass --region "$region" ssm get-command-invocation --command-id "$id" \ |
| + --instance-id "$inst" --query Status --output text 2>/dev/null) && [ "$s" != InProgress ] && [ "$s" != Pending ]; do sleep 3; done |
| +aws --profile kass --region "$region" ssm get-command-invocation --command-id "$id" --instance-id "$inst" \ |
| + --query '[Status,StandardOutputContent,StandardErrorContent]' --output text |
| +``` |
| + |
| +`leo` below means `$TMPDIR/cohort1/ssm.sh eu-west-1 i-046c7fe43620944bc`, and `dennis` means `$TMPDIR/cohort1/ssm.sh eu-west-2 i-009d593f451cfd562`. Delete `$TMPDIR/cohort1` by exact path when the work is done. |
| + |
| +Never print a secret. Key checks print a presence flag and the first 8 hex digits of a SHA-256, never the value. |
| + |
| +## Prechecks |
| + |
| +All of these are read-only. The results recorded here were taken on 2026-10-16 between 07:20Z and 07:45Z. |
| + |
| +### P1. Release public key on the box |
| + |
| +```bash |
| +leo 'grep -c "^CARLOS_RELEASE_PUBKEY=" /etc/carlos/host.env; grep "^CARLOS_RELEASE_PUBKEY=" /etc/carlos/host.env | cut -d= -f2- | tr -d "\"" | sha256sum | cut -c1-8' |
| +``` |
| + |
| +Pass: `1`, and the same fingerprint on every dev box. |
| + |
| +Today: PASS. Present once on leo, sophie and dennis, fingerprint `960d03ff` on all three. Step 2 still proves that it is the key that signs dev releases and the roster. |
| + |
| +### P2. IAM and actor certification |
| + |
| +```bash |
| +aws --profile kass iam list-role-policies --role-name carlos-dev-host-b |
| +aws --profile kass iam get-role-policy --role-name carlos-dev-host-b --policy-name carlos-dev-host-b-buckets \ |
| + --query PolicyDocument --output json | grep -c 'GetObjectVersion\|DeleteObjectVersion\|ListBucketVersions' |
| +aws --profile kass s3api get-bucket-policy --bucket carlos-dev-640012953448 --query Policy --output text \ |
| + | python3 -c 'import json,sys;print([s.get("Sid") for s in json.load(sys.stdin)["Statement"]])' |
| +aws --profile kass iam list-roles --query 'Roles[?contains(RoleName, `fence`)].RoleName' --output text |
| +for b in carlos-dev-640012953448 carlos-dev-ie-640012953448 carlos-dev-uk-640012953448; do |
| + aws --profile kass s3api get-bucket-versioning --bucket $b --output text; done |
| +aws --profile kass s3api head-object --bucket carlos-dev-640012953448 --key control/host-actors.json |
| +leo 'for p in /etc/carlos/host-actor.json /etc/carlos/host-evidence.key /etc/carlos/fence-writer/credentials.json /opt/carlos/system/guard; do [ -e $p ] && echo "present $p $(stat -c "%U %a %h" $p)" || echo "absent $p"; done' |
| +``` |
| + |
| +Pass, once step 3 is done: |
| +- `carlos-dev-host-b-buckets` grants `s3:GetObjectVersion`, `s3:DeleteObjectVersion` and `s3:ListBucketVersions` on the IE data bucket (count 3 or more); |
| +- the deployment bucket policy has `BoxRecordsAreCreateOnly`, `BoxRecordsAreNeverDeleted` and `OnlyBoxRolesWriteTerminationFences`, the last naming a dev fence-writer role; |
| +- a dedicated fence-writer role exists for each cohort box; |
| +- all three buckets report `Enabled`; |
| +- `control/host-actors.json` exists, is signed by the release key and has leo's row; |
| +- on leo, `/etc/carlos/host-actor.json` is present, `/etc/carlos/host-evidence.key` is `root 600 1`, the fence-writer credentials are `root 400 1`, and `/opt/carlos/system/guard` exists. |
| + |
| +Today: FAIL, which is expected before step 3 apart from the first two items. |
| +- `carlos-dev-host-b-buckets` has no version actions (count 0). Leo's role predates the Plan 2 module statements. The roles `carlos-dev-host` (sophie) and `carlos-dev-host-uk` (dennis) already have them. |
| +- The deployment bucket has no bucket policy (`NoSuchBucketPolicy`), so the region HA evidence rules and M-2's fence-writer rule are absent. |
| +- No fence-writer role exists. |
| +- Versioning is `Enabled` on all three buckets. PASS. |
| +- `control/host-actors.json` does not exist (404). Dev has no signed roster. |
| +- None of the four local files exists on leo. |
| +- Leo's role may `ssm:GetParameter` on `parameter/carlos/*` in eu-west-1, which holds `/carlos/release-key`, the deployment root signing key. The same key signs the roster and the floor. See open question 11. |
| +- Leo's role may `s3:PutObject` anywhere in the deployment bucket, including `control/hosts/`. Plan 15's independent claim-writer boundary is therefore not in place on dev. See open question 10. |
| + |
| +### P3. Every roster reader's version |
| + |
| +```bash |
| +carlos system status --console https://platform.dev.carlosframework.com |
| +curl -sI https://platform.dev.carlosframework.com/ | grep -i x-carlos-version |
| +leo '/opt/carlos/carlos version; [ -e /opt/carlos/carlos.prev ] && /opt/carlos/carlos.prev version' |
| +carlos version |
| +``` |
| + |
| +Run the box line on sophie and dennis too. Pass: every box, console, `.prev` image and operator CLI is on a build containing Plan 13 and its follow-ups (2ad5412c, in every tag since v0.26.1). Before step 4b, every one of them is on the cohort release. |
| + |
| +Today: PASS for Plan 13. Leo v0.27.3 with `.prev` v0.27.2; sophie v0.27.3 with `.prev` v0.27.2; dennis v0.27.3 with no `.prev`; console `e15ad6be` (v0.27.3); local CLI v0.27.2. `system status`: dennis and leo on stable, sophie on canary, all `current`. The cohort release does not exist yet: v0.27.3 has neither the files seed (ee35dee2) nor Plan 14 Parts 2 and 3. |
| + |
| +### P4. Leo is empty or quiet |
| + |
| +```bash |
| +leo 'python3 -c " |
| +import sqlite3 |
| +c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| +for r in c.execute(\"select host,app,kind,unit,backing,addr,hibernate from routes order by host\"): print(r)" |
| +for h in proof.dev.carlosframework.com proof.dev.oncarlos.com proof2.dev.oncarlos.com wakecheck.dev.carlosframework.com; do |
| + echo "$h last=$(journalctl -u carlos-edge.service --no-pager | grep "$h" | tail -1 | cut -c1-15)"; done |
| +ss -ltn | grep -c ":8099 "; systemctl list-units --no-legend --plain "carlos-instance-gen@*" "carlos-instance@*"' |
| +``` |
| + |
| +Pass: only known routes; no instance unit running; no recent request lines. |
| + |
| +Today: PASS. Four routes: |
| +- `proof.dev.carlosframework.com`, `proof.dev.oncarlos.com` and `proof2.dev.oncarlos.com`: kind `instance`, empty app, address-routed to `127.0.0.1:8099`. Nothing listens there, and the edge journal last names them on Aug 28. |
| +- `wakecheck.dev.carlosframework.com`: app `wakecheck`, `hibernate=1`, nested layout. Asleep: no instance unit runs, and its only recent journal lines are the five-minute provision passes. |
| + |
| +### P5. No named-unit or unit-backed work on leo |
| + |
| +```bash |
| +leo 'python3 -c " |
| +import sqlite3 |
| +c=sqlite3.connect(\"file:/data/registry.db?mode=ro\",uri=True) |
| +print(c.execute(\"select count(*) from routes where unit!=\x27\x27 or backing=\x27unit\x27\").fetchone()[0])" |
| +find /etc/carlos/apps -mindepth 1 | wc -l |
| +find /etc/systemd/system /run/systemd/system /usr/lib/systemd/system /lib/systemd/system -name "carlos-app@*" | grep -vcE "/carlos-app@\.(service|socket)$"' |
| +aws --profile kass s3 ls --recursive s3://carlos-dev-640012953448/apps/ | awk '{print $4}' | grep -E '/(instances|sidecars)/[^/]+\.json$' |
| +aws --profile kass s3 ls s3://carlos-dev-640012953448/apps/bab/ |
| +``` |
| + |
| +These mirror Part 3's `claimActivationPreflight` (registry), `claimActivationVerified` (bucket records for leo's region and pools) and `claimActivationLocalUnits` (unit files). Pass: `0`, `0`, `0`; every listed record has an empty `unit`; and every `PRE` entry under `apps/bab/` is a valid app name. |
| + |
| +Today: FAIL. |
| +- Registry, local units and records: PASS. No named-unit or `backing=unit` route, `/etc/carlos/apps` is empty, and there is no `carlos-app@<name>` unit. The unit files that mention `/opt/carlos/live/` or `User=carlos` all carry embedded unit names. The bucket holds two records, `bab/wakecheck` and `bab/rollcheck-0261` (static), neither with a unit. |
| +- Bucket app tree: FAIL, a blocker. `apps/bab/` lists `PRE /`, an empty app segment. It holds 286 objects under `apps/bab//errors/leo-climbing/`, the newest written at 06:46Z today. `hostops.Apps` lists `apps/bab/` with a `/` delimiter, meets the empty name and returns `ErrIdentity`. On Part 3 that makes `claimActivationVerified` and the migration census refuse, on every box that uses this deployment bucket. The console's claim runtime and purge passes call the same function. See open question 9. |
| + |
| +### P6. No claim state yet |
| + |
| +```bash |
| +leo 'for p in /var/lib/carlos-claims /data/.hostclaims; do [ -e $p ] && echo "present $p" || echo "absent $p"; done; ls /data/hostguard | wc -l' |
| +``` |
| + |
| +Pass: both absent, `0`. |
| + |
| +Today: PASS on leo, sophie and dennis. With no `/data/.hostclaims` there are no legacy cold journals, so the files seed authority review's L4 check (old serving cold journals) passes too. |
| + |
| +### P7. Switches |
| + |
| +```bash |
| +leo 'for k in CARLOS_SWAP CARLOS_HOST_CLAIM CARLOS_HOST_CLAIM_ORIGINS CARLOS_MOVES_PHASE1 CARLOS_REGION CARLOS_REGION_HA CARLOS_SYSTEM_CHANNEL CARLOS_POOLS; do |
| + grep -qE "^$k=" /etc/carlos/host.env && echo "$k=$(grep -E "^$k=" /etc/carlos/host.env | tail -1 | cut -d= -f2-)" || echo "$k unset"; done |
| +grep -qE "^CARLOS_SECRETS_KEY=" /etc/carlos/host.env && echo "CARLOS_SECRETS_KEY present" || echo "CARLOS_SECRETS_KEY absent"' |
| +``` |
| + |
| +Pass: `CARLOS_SWAP=overlap`; every claim, moves and region HA switch unset. |
| + |
| +Today: PASS. Leo has `CARLOS_SWAP=overlap`, and `CARLOS_HOST_CLAIM`, `CARLOS_HOST_CLAIM_ORIGINS`, `CARLOS_MOVES_PHASE1`, `CARLOS_REGION`, `CARLOS_REGION_HA`, `CARLOS_SYSTEM_CHANNEL` and `CARLOS_POOLS` unset. `CARLOS_SECRETS_KEY` is absent on leo, which the narrowed scope allows because moves stay off. |
| + |
| +### P8. Other box facts the later steps rely on |
| + |
| +```bash |
| +leo 'uname -r; findmnt -no FSTYPE --target /data; df -h /data | sed 1d; free -m | sed -n 2p |
| +stat -c "%U:%G %a %n" / /var /var/lib |
| +find / -xdev -maxdepth 4 -name proof-policy.json | wc -l |
| +systemctl --version | head -1 |
| +for u in carlos-edge carlos-adopt carlos-stage carlos-config-sync carlos-restart carlos-system-update carlos-system-recover carlos-instance-control; do |
| + echo "$u $(systemctl show $u.service -p FragmentPath --value)"; done' |
| +``` |
| + |
| +Today: |
| +- Kernel 6.12.105, `/data` on ext4, 18G free, 927 MB of memory. PASS. Flag-on instance-control takes `MemoryMax=256M`, a large share of this box; watch it in step 7. |
| +- `/`, `/var` and `/var/lib` are `root:root 755`, and no `proof-policy.json` exists. PASS for the Plan 14 Part 1 pre-roll checks. |
| +- systemd 257. Every unit the offline window masks has its file in `/etc/systemd/system`. FAIL, a blocker. A runtime mask is a link in `/run/systemd/system`, and systemd reads `/etc/systemd/system` first, so the `/etc` unit file wins. #272's round-3 dev check saw exactly this: "runtime masks lose to the /etc unit files". Part 3's `verifyOfflineRootMigration` requires `LoadState=masked` on six units, and the seed runbook relies on a runtime mask on the edge. See open question 5. |
| + |
| +## Blockers found today |
| + |
| +1. No runner exists on main or on the Plan 14 branches for: creating the signed roster (`hostactors.Directory.Initialize`), inviting and enrolling a box, installing `/etc/carlos/host-evidence.key`, publishing the floor (`Directory.PublishFloor`) or publishing `home_region`. Plan 9 C2 marks these "confirm at approval". Steps 3 and 4 cannot run until one is reviewed. |
| +2. The empty app segment `apps/bab//` in the deployment bucket (P5) makes the activation preflight and the migration census refuse. |
| +3. Runtime masks do not mask units installed in `/etc/systemd/system` (P8), so the offline window's masks cannot be made to hold as written. |
| +4. Leo's IAM role lacks the version actions, and dev has neither the deployment bucket policy nor a fence-writer role (P2). |
| +5. Plan 14 Parts 2 and 3 are not merged, and no release contains them with the files seed. |
| + |
| +## Step 0. Coordinate |
| + |
| +Paul's go: none needed to talk, but nothing below starts until these are done. |
| + |
| +- Tell platform-d8 that leo is about to change hands, and wait for its zero-downtime dev checks there to finish. Once claims are on, leo declines `CARLOS_SWAP=overlap` (#272 ruling, reason `host-claims-on`), so those checks cannot run on leo until a full rollback. |
| +- Tell the region HA and moves sessions. #286's ruling (b) says rehearsal apps wake on one dev box only until dev's claim is on. |
| +- Post the planned window on #250. |
| + |
| +## Step 1. Release and roll, claims off |
| + |
| +What: cut a release containing Plan 14 Parts 2 and 3 and the files seed, and roll it with every claim switch unset. Dev goes first, then flagship. |
| + |
| +Paul's go: to cut the release, then to roll dev, then (separately, not from this runbook) to roll flagship. |
| + |
| +1. Merge Parts 2 and 3, re-merge main, wait for CI. |
| +2. Cut the release by the README section "Releasing the platform". Check `gh release list -R carlosframework/releases` and `git ls-remote --tags origin` before choosing the number. |
| +3. Roll the dev boxes on the new road only: |
| + |
| + ```bash |
| + carlos system update --console https://platform.dev.carlosframework.com --channel stable --version vX.Y.Z --yes |
| + carlos system update --console https://platform.dev.carlosframework.com --channel canary --version vX.Y.Z --yes |
| + carlos system status --console https://platform.dev.carlosframework.com |
| + ``` |
| + |
| + `stable` moves leo and dennis together; `canary` moves sophie. |
| +4. Roll the dev console direct-bucket with `ship-app.sh` (profile `kass`, bucket `carlos-dev-640012953448`). |
| +5. Check dev end to end: wake `wakecheck` on sophie through DNS and on leo with `curl --resolve wakecheck.dev.carlosframework.com:443:18.200.150.210` (outside the sandbox). |
| +6. Flagship boxes, then the flagship console, by the release recipe, under their own approval. |
| + |
| +Expected: |
| +- `system status` shows all three dev boxes `current` at vX.Y.Z; |
| +- `curl -sI https://platform.dev.carlosframework.com/ | grep -i x-carlos-version` shows the release commit; |
| +- on leo, `systemctl show carlos-instance-control.service -p ActiveEnterTimestamp` is later than the update, and `/opt/carlos/carlos version` prints vX.Y.Z; |
| +- both wakes answer 200, leo's with `host=ip-172-31-19-180`. |
| + |
| +Hazards: |
| +- The old-updater instance-control restart. Dev's updater is v0.27.3, which restarts instance-control before the edge. If a wake is refused with "running an older build", instance-control was not restarted: restart it with `systemctl restart carlos-instance-control.service` (Paul's go). |
| +- #272: do not use the old road (`update-platform-binary.sh`) on leo. After an old-road roll, `result.json` "applied" is terminal, so the system leg never returns the box to its channel pointer, and it must be restored by hand. |
| + |
| +Rollback: `carlos system rollback --console https://platform.dev.carlosframework.com --channel stable` (and `canary`). This works only while no actor is armed. After step 3, never run below the floor. |
| + |
| +## Step 2. Release public key |
| + |
| +What: confirm `CARLOS_RELEASE_PUBKEY` in leo's host.env is the key that signs dev releases, the roster and the floor. Root's held release and its signed launch reservations read it in claim mode. |
| + |
| +Paul's go: only if it must be changed. |
| + |
| +Check: P1, plus step 1's update. A box verifies the channel pointer with this key, so a successful `carlos system update` to vX.Y.Z proves the match. |
| + |
| +Expected: P1 passes and step 1 applied on leo. Today the key is already present (`960d03ff` on all three boxes), so no change is planned. |
| + |
| +Rollback: none; nothing changes. |
| + |
| +## Step 3. Actor and IAM certification |
| + |
| +What: put the IAM shape from main's modules on dev, enroll leo as a claim actor, arm its guard and publish the floor. Plan 9 C2 order: arm every enrolled actor, publish only once every actor reports ready. |
| + |
| +### 3a. IAM |
| + |
| +Paul's go: to apply the dev root. |
| + |
| +1. Push the GitHub mirror forward to the release tag (fast-forward check first), because the dev root pulls modules from GitHub. |
| +2. In `platform-infrastructure/dev`, wire one `box-fence-writer` module per cohort box and pass their role ARNs to `module.buckets.box_fence_writer_role_arns`. Run `tofu init -upgrade && tofu plan -out=cohort1.plan` and read the whole plan. Expect: |
| + - `carlos-dev-host-b-buckets` gains `PullDoorObjectVersions`, the `s3:DeleteObjectVersion` statement and `PullDoorVersionList` on the IE data bucket; |
| + - a new `deployment_box_records` bucket policy on `carlos-dev-640012953448`; |
| + - new fence-writer roles and bootstrap users; |
| + - no `user_data` change. A `user_data` diff restarts boxes, so stop and ask. |
| +3. Apply. Re-run P2's IAM lines. |
| +4. Denied-operation witnesses, through a reviewed runner: leo's instance role is refused a put to `control/box-fences/`, the fence-writer role may create there with `If-None-Match`, and nobody may delete there. |
| + |
| +Expected: P2's IAM items pass. |
| + |
| +Rollback: revert the dev root to its previous module wiring and apply, but only before any fence object is written. Evidence objects already written stay. |
| + |
| +### 3b. Enroll leo and arm its guard |
| + |
| +Paul's go: once for the whole of 3b. |
| + |
| +Blocked: there is no reviewed runner for the roster, the invitation, the enrollment or the evidence key (blocker 1). With one, the order is: |
| + |
| +1. Create the signed roster, `control/host-actors.json`, and a root-signed invitation `control/host-actor-invitations/leo-climbing.json`. |
| +2. Install, root-owned and outside any agent-writable path: |
| + - `/etc/carlos/host-evidence.key`, `root`, mode `600`, one link; |
| + - `/etc/carlos/fence-writer/credentials.json`, `root`, mode `400`. |
| +3. Arm the guard: |
| + |
| + ```bash |
| + /opt/carlos/carlos system guard-arm --policy "$APPROVED_POLICY" --identity "$SIGNED_IDENTITY" --enrollment "$SIGNED_ENROLLMENT" |
| + ``` |
| + |
| + These are approved, protected files, not guessed strings. |
| +4. Complete leo's enrollment into the roster. |
| + |
| +Expected: |
| +- `/etc/carlos/host-actor.json` written by `guard-arm`; |
| +- `/opt/carlos/system/guard/policy.json` present; |
| +- the roster's `boxes.leo-climbing` row has leo's enrollment and `cloud.instance_id` `i-046c7fe43620944bc`; |
| +- `carlos system status --console https://platform.dev.carlosframework.com` shows leo ready, with the floor digest. |
| + |
| +Rollback: stop here and leave every flag unset. Arming is one-way: keep the local armed policy, and never delete the enrollment or the guard. Restore only an image at or above the floor. |
| + |
| +### 3c. Publish the floor |
| + |
| +Paul's go: separate from 3b. |
| + |
| +Blocked on blocker 1. Re-read every report, the roster revision and the candidate digest just before publishing. Publish through the reviewed runner, then read back the roster. |
| + |
| +Expected: the roster has `publication: "published"` and a `floor_sha256` equal to the digest in leo's readiness report. |
| + |
| +Rollback: a torn publish is resumed with identical bytes. Never erase the floor or substitute another digest. If a check after publishing fails, the claim flag stays off. |
| + |
| +## Step 4. Plan 13 order: reader, then home, then enable |
| + |
| +Paul's go: before 4b. The designation becomes write-once as soon as claims are on. |
| + |
| +### 4a. Readers |
| + |
| +Re-run P3. Expected: leo, sophie, dennis, their `.prev` images, any staged recovery image, the dev console and the operator CLI are all on vX.Y.Z (at minimum, on a Plan 13 build). The signed fleet data does not record every reader's build, so this is the operator's check. |
| + |
| +### 4b. Home designation |
| + |
| +Publish `home_region` as dev's home roster name. The proposed name is `ie` (open question 8). Blocked on blocker 1. |
| + |
| +Expected: the roster read back has `"home_region": "ie"`, and after step 6 leo's agent does not log "host claim: signed roster has no home region". |
| + |
| +Rollback: before claims are on, a new signed roster revision can correct the name. After claims are on, the name cannot change. |
| + |
| +### 4c. Enable |
| + |
| +Enabling happens in steps 5 and 6. |
| + |
| +## Step 5. Offline window: migration and files seed |
| + |
| +What: Plan 9 milestone 4 on leo, with the files seed inside the same window. Tenants stay stopped from the drain until serving resumes. |
| + |
| +Paul's go: for the window, with a written deadline. A second go is needed in step 6 before serving resumes. |
| + |
| +Before the window: blockers 2 and 3 are fixed, the wakecheck decision (hazard H1) is applied, and platform-d8 has confirmed leo is free. |
| + |
| +### 5.1 Drain and quiesce |
| + |
| +```bash |
| +leo 'systemctl list-units --no-legend --plain "carlos-instance-gen@*" "carlos-instance@*"' |
| +leo 'systemctl stop carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path |
| +systemctl mask --runtime carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service |
| +for u in carlos-adopt carlos-stage carlos-config-sync carlos-restart carlos-system-update carlos-system-recover; do echo "$u $(systemctl show $u.service -p LoadState -p ActiveState --value | tr "\n" " ")"; done' |
| +leo 'systemctl stop carlos-edge.service && systemctl mask --runtime carlos-edge.service && systemctl show carlos-edge.service -p LoadState -p ActiveState --value; pgrep -u carlos | wc -l' |
| +``` |
| + |
| +Expected: no instance unit before the stop; each of the six units `masked inactive`; the edge `masked inactive`; `0` processes under the carlos user. Stopping the edge drains requests and flushes replication for any awake instance. |
| + |
| +The adopt pass must be stopped first. It re-renders the instance-control unit from host.env and restarts it, so with `CARLOS_HOST_CLAIM` set it would start claim mode at an unplanned moment. |
| + |
| +Today this step cannot pass: see blocker 3. The masks above land in `/run` and lose to the `/etc` unit files. |
| + |
| +### 5.2 Set the flags |
| + |
| +```bash |
| +leo 'cp -p /etc/carlos/host.env /etc/carlos/host.env.bak-cohort1-$(date -u +%Y%m%d) |
| +printf "CARLOS_HOST_CLAIM=1\nCARLOS_HOST_CLAIM_ORIGINS=1\n" >> /etc/carlos/host.env |
| +grep -cE "^CARLOS_HOST_CLAIM=1$|^CARLOS_HOST_CLAIM_ORIGINS=1$" /etc/carlos/host.env; grep -c "^CARLOS_MOVES_PHASE1=" /etc/carlos/host.env' |
| +``` |
| + |
| +Expected: `2`, then `0`. Any non-empty value of `CARLOS_HOST_CLAIM`, including `0`, turns it on. The files seed requires `CARLOS_HOST_CLAIM_ORIGINS` to be exactly `1`. |
| + |
| +### 5.3 Render the claim-on root unit |
| + |
| +The reviewed installer is not named yet (open question 6). After it runs: |
| + |
| +```bash |
| +leo 'systemctl cat carlos-instance-control.service | grep -E "^StateDirectory=carlos-claims|^EnvironmentFile=-/etc/carlos/host.env|^MemoryMax=256M"' |
| +``` |
| + |
| +Expected: all three lines. |
| + |
| +### 5.4 Migrate |
| + |
| +```bash |
| +leo 'set -a; . /etc/carlos/host.env; set +a; /opt/carlos/carlos system journal-migrate; echo exit=$?' |
| +``` |
| + |
| +On Part 3 this runs the activation preflight, checks the six units are masked and inactive, refuses while any `carlos adopt`, `system apply` or `guard-arm` process runs, then performs the offline root migration and publishes the census report. |
| + |
| +Expected: `exit=0`; `/var/lib/carlos-claims/authority/migration/migration.json` exists (root-owned). No legacy journals exist on leo, so nothing is imported. |
| + |
| +### 5.5 Root startup |
| + |
| +Plan 9 milestone 4's commands: |
| + |
| +```bash |
| +leo 'systemctl daemon-reload |
| +systemctl add-wants multi-user.target carlos-instance-control.service |
| +systemctl restart carlos-instance-control.service |
| +systemctl is-active carlos-instance-control.service |
| +journalctl -u carlos-instance-control.service --since "-5min" --no-pager | tail -30' |
| +``` |
| + |
| +Expected: `active`, and no migration, recovery or floor error in the journal. A migration or evidence failure leaves the edge stopped. |
| + |
| +### 5.6 Files seed |
| + |
| +Quoted verbatim from the approved copy (2026-10-15). The headings in brackets are the string identifiers. |
| + |
| +[runbook.heading] |
| + |
| +> Files seed during claim activation |
| + |
| +[runbook.current_status] |
| + |
| +> Use this step only within an approved claim activation, with an approved binary and census revision. Copy approval does not authorize a rollout. Keep moves disabled; the independent claim-writer, census and deployment gates remain separate. |
| + |
| +[runbook.placement] |
| + |
| +> Insert this step after milestone 4’s claim-on root startup and journal recovery, before normal edge serving. Keep CARLOS_MOVES_PHASE1 unset. Before rollout, inspect the dev cold-journal state required by the authority review, and record the approved claim-writer and census revisions. A serving activation does not approve moves. |
| + |
| +[runbook.gates] |
| + |
| +> Complete the claim-adoption, independent claim-writer and census activation gates. Require serving with moves off, publication and readiness, same-mount database retirement, bounded retention, pressure and expiry reaping, normal reboot recovery, early disk and evidence refusals, crashed-seed settlement, delegated controller authentication and concurrent-wake exclusion to pass. Require a real launch and two cold wakes with real origin evidence. Skipped privileged tests do not pass. Every claimed host must use nested layout, with no unfinished migration. Keep moves disabled; its verification and repair gates are separate. |
| + |
| +[runbook.preflight] |
| + |
| +> No automated diagnostic preflight, exact restore sizing or space reservations ship with this step. Complete the approved activation checks and record measured disk headroom before the offline window. Files seed checks the existing start floor before reserving an attempt; it does not guarantee enough space for the whole restore or upload. |
| + |
| +[runbook.classification] |
| + |
| +> Review each skipped or blocked host separately. Seeded-instance routes, backing routes, named units, sidecars, tunnels and replicas cannot use this serving preparation. A proven new empty host needs no files seed. Fix missing builds, flat or unfinished layouts and database path mismatches before rerunning. A missing files directory is a refusal, not an empty source. Hibernating eligible hosts retain their files and timer state. |
| + |
| +[runbook.window] |
| + |
| +> During milestone 4, drain requests, flush database replication and stop tenant writers through the approved activation workflow. Keep the rollout-owned edge mask in place until the delegation step. Set CARLOS_HOST_CLAIM and CARLOS_HOST_CLAIM_ORIGINS, leave CARLOS_MOVES_PHASE1 unset, start the claim-on root service, and verify journal migration and root recovery before seeding. Claimed serving and C8 must work with moves off. |
| + |
| +[runbook.budget] |
| + |
| +> Files seed processes one host at a time. Budget a database restore and upload using measured rehearsal rates, and record a deadline before starting. Capacity checks enforce the start floor, not a reserved restore budget. At the deadline, stop carlos-edge.service through the approved activation workflow. Cleanup has a 30-second budget; the temporary unit allows 45 seconds before systemd ends the process. Keep interrupted evidence so the next rerun can settle the attempt. |
| + |
| +[runbook.run] |
| + |
| +> Run the initial files seed in the delegated edge unit during the offline activation window. Each eligible host receives an ordinary uncertified cold admission, retirement of its old local database, restoration of the remote baseline and publication into the admitted files prefix. Writes beyond that remote baseline are declared lost, which is why the drained, replicated stop must come first. The command never launches a tenant. For a later rerun, invoke the same command as the service user with the protected box configuration; it sends a synchronous request to the authenticated running agent, which excludes competing wakes through settlement. |
| + |
| +[runbook.run_command] |
| + |
| +``` |
| +carlos system files seed |
| +``` |
| + |
| +[cli.usage] |
| + |
| +``` |
| +Usage: carlos system files seed [--host <host>] |
| +``` |
| + |
| +[cli.exit_status] |
| + |
| +> Exit 0 means the selected run finished without a reported error; it can include skipped hosts and does not prove that every host on the box was seeded. Exit 2 means the executor collected host failures or noticed cancellation between hosts. Exit 1 covers setup, validation, transport and other errors outside that collected result. Capture the service’s ExecMainStatus and Result; systemctl start has its own exit status. |
| + |
| +[cli.forwarded_errors] |
| + |
| +> Failure lines contain the underlying provider or operating-system error, sometimes across several lines. Save the full output. A shared “selected authority changed” error does not identify one cause by itself; use the root service journal and the host’s retained evidence. |
| + |
| +[runbook.delegation] |
| + |
| +> Through the approved activation workflow, create the new temporary override at the path below. Refuse an existing file or symlink there. Save the original effective unit and masks; preserve the service user, protected environment, delegation and hardening. Reload systemd and inspect the effective command before removing only the rollout-owned runtime mask. Keep the traffic barrier in place while starting the seed one-shot. Capture its InvocationID, Result, ExecMainStatus and journal before stopping it; RemainAfterExit keeps a successful completed invocation available. Save failed results before any reset-failed. After stopping, restore the rollout-owned runtime mask, remove only the recorded override, reload and inspect the normal unit. Resolve pre-existing operator masks through their owner. Rerun a crashed one-shot in this same delegated unit within the approved offline workflow. |
| + |
| +[runbook.override_path] |
| + |
| +``` |
| +/run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| +``` |
| + |
| +[runbook.override] |
| + |
| +``` |
| +[Service] |
| +Type=oneshot |
| +RemainAfterExit=yes |
| +ExecStart= |
| +ExecStart=/opt/carlos/carlos system files seed |
| +ExecReload= |
| +Restart=no |
| +TimeoutStartSec=0 |
| +TimeoutStopSec=45s |
| +``` |
| + |
| +[runbook.reload_command] |
| + |
| +``` |
| +systemctl daemon-reload |
| +``` |
| + |
| +[runbook.unit_verify] |
| + |
| +``` |
| +systemctl cat carlos-edge.service |
| +``` |
| + |
| +[runbook.unmask_command] |
| + |
| +``` |
| +systemctl unmask --runtime carlos-edge.service |
| +``` |
| + |
| +[runbook.start_command] |
| + |
| +``` |
| +systemctl start carlos-edge.service |
| +``` |
| + |
| +[runbook.unit_status_command] |
| + |
| +``` |
| +systemctl show carlos-edge.service -p InvocationID -p Result -p ExecMainStatus -p ActiveState -p SubState |
| +``` |
| + |
| +[runbook.journal_command] |
| + |
| +``` |
| +journalctl _SYSTEMD_INVOCATION_ID=<invocation-id> --no-pager |
| +``` |
| + |
| +[runbook.skips] |
| + |
| +> Review every skipped-file line. Oversized and non-regular files are omitted. Unreadable or changing files fail that host. |
| + |
| +[runbook.committed] |
| + |
| +> A committed manifest permits serving through the ordinary origin, restore and readiness checks. An existing manifest is never seeded over. Full verification and repair are deferred, and moves must remain disabled until their separate gates pass. Missing or corrupt evidence needs investigation; do not remove a manifest to make a host eligible for seeding. |
| + |
| +[runbook.stragglers] |
| + |
| +> Record every failed, skipped or deferred host and the reason. A serving host without committed files evidence refuses before admission and spends no new wake retry; other hosts may serve after their activation gates pass. Rerun a blocked host through the running agent with both claim flags set and moves off, or repeat the delegated one-shot inside an approved offline window. Keep any existing retry debt. For claims assigned elsewhere, check that box’s activation results. |
| + |
| +[runbook.retries] |
| + |
| +> Each new preparation spends one wake retry attempt. Under the host’s wake exclusion, a rerun settles a crashed attempt and rereads the claim before proceeding. It never refunds that attempt. The command prints the current retry state and waits for the retry-after time before preparing again; the old storage lease must also have expired or been released through its owner checks. An established manifest causes no new preparation. Capacity and evidence refusals precede reservation. If stop proof is unavailable, keep the evidence and investigate. Exhausted episodes need an operator-authorized console reset. A successful seed does not report a healthy tenant. |
| + |
| +[runbook.rerun_command] |
| + |
| +``` |
| +carlos system files seed --host <host> |
| +``` |
| + |
| +[runbook.reset] |
| + |
| +> For an exhausted episode, use an authorized operator session to POST /api/cli/ops/hosts/<host>/resolve with action="reset-wake-retry", the current incarnation and episode, and a new request_id. The console must prove any active attempt stopped before resetting it. Confirm the new episode through the approved operator workflow before rerunning. Do not give this authority to the service user. |
| + |
| +[runbook.cleanup] |
| + |
| +> Save every admitted-prefix line with the activation record, including lines printed before a host fails. The command does not print a separate orphan-prefix inventory and deletes no orphan objects. If interruption happens before a prefix is printed, keep the durable claim and root records for investigation. Any later deletion needs separately scoped checks of references, leases and stop evidence; never expire a whole generation by age alone. |
| + |
| +[runbook.retention] |
| + |
| +> Each claimed cold wake retires the previous local database on the same instance mount, including its generation marker and any WAL or SHM files. The first seed parks the legacy database once, with a seven-day deadline. A generation that never launched is discarded and cannot replace the retained park. A later generation whose tenant started may replace that slot. Root checks expiry and disk pressure on startup and once a minute while running, and may reap verified completed parks early, oldest first. A retained park therefore has no guaranteed minimum lifetime. Incomplete, unknown or identity-mismatched state is kept for recovery or investigation. Never delete parked paths by hand. |
| + |
| +[runbook.park_status] |
| + |
| +> Read the root service journal for recovery and cleanup failures. Rich park listings and automatic rehabilitation of ambiguous directories are deferred. Keep unknown or identity-mismatched data for a separately scoped investigation. Do not treat a quarantine name as deletion authority. |
| + |
| +[runbook.park_journal_command] |
| + |
| +``` |
| +journalctl -u carlos-instance-control.service --no-pager |
| +``` |
| + |
| +[runbook.finish] |
| + |
| +> Save the invocation ID, service result, exit status, complete output, each reported admitted prefix, retry state, skipped paths and blocked-host list. Match hosts and incarnations to the activation inventory; the seed output does not print an incarnation field. Stop the one-shot, restore this rollout’s runtime mask, remove only its override and reload systemd. Verify that the effective unit again starts the normal agent with its original restart policy and hardening. Keep the traffic barrier until Plan 9 allows serving, then remove only the rollout-owned mask and start the normal agent. Complete the live cold-wake and readiness checks before declaring activation complete. |
| + |
| +[runbook.stop_command] |
| + |
| +``` |
| +systemctl stop carlos-edge.service |
| +``` |
| + |
| +[runbook.cancellation] |
| + |
| +> If the deadline arrives, capture the current InvocationID before stopping the service. Save that invocation’s journal and the final Result and ExecMainStatus afterward. A stopped unit may no longer expose its InvocationID. Keep all durable claim and root evidence for the next rerun. |
| + |
| +[runbook.restore_mask_command] |
| + |
| +``` |
| +systemctl mask --runtime carlos-edge.service |
| +``` |
| + |
| +[runbook.remove_override_command] |
| + |
| +``` |
| +rm -- /run/systemd/system/carlos-edge.service.d/90-files-seed-activation.conf |
| +``` |
| + |
| +[runbook.rollback] |
| + |
| +> Before disabling claims, record outstanding park failures and retained disk space. A box that has enrolled a census still maintains its retained root state during update-time recovery, even when claims are off. The no-census case remains separate. Do not widen unit permissions or delete retained paths by hand; use an approved recovery procedure. |
| + |
| +Expected for leo: one result line per selected host. If wakecheck stays on leo, `Host "wakecheck.dev.carlosframework.com": seeded.` and its admitted prefix line. The `proof*` routes are address-routed and have no app; record whatever is printed for them. Save the full output with the activation record. |
| + |
| +### 5.7 Leave the window, but keep serving off |
| + |
| +Follow `[runbook.finish]` above up to "Keep the traffic barrier until Plan 9 allows serving". Then put the update and adopt units back: |
| + |
| +```bash |
| +leo 'systemctl unmask --runtime carlos-adopt.service carlos-stage.service carlos-config-sync.service carlos-restart.service carlos-system-update.service carlos-system-recover.service |
| +systemctl start carlos-adopt.timer carlos-system-update.path carlos-stage.path carlos-restart.path' |
| +``` |
| + |
| +Rollback for step 5: |
| +- Before 5.2: unmask the six units and the edge, start the timers and paths, start `carlos-edge.service`. |
| +- After 5.2, before 5.4: restore `/etc/carlos/host.env` from the backup, render the flag-off unit with the same installer, `systemctl daemon-reload && systemctl restart carlos-instance-control.service`, then unmask and start as above. |
| +- After 5.4: use the full rollback below. Keep `/var/lib/carlos-claims` and every record. |
| + |
| +## Step 6. Claims on |
| + |
| +What: lift the traffic barrier so leo serves under the claim. |
| + |
| +Paul's go: required, after 5.5 and 5.6 pass. |
| + |
| +```bash |
| +leo 'systemctl unmask --runtime carlos-edge.service && systemctl start carlos-edge.service && systemctl is-active carlos-edge.service |
| +journalctl -u carlos-edge.service --since "-5min" --no-pager | grep -iE "host claim|REFUSING CARLOS_SWAP|seed" | tail -20' |
| +``` |
| + |
| +Expected: |
| +- `active`; |
| +- `REFUSING CARLOS_SWAP=overlap (host-claims-on)` once; |
| +- no "signed roster has no home region" line; |
| +- `carlos system status --console https://platform.dev.carlosframework.com` shows leo current. |
| + |
| +Rollback: stop the edge (`systemctl stop carlos-edge.service`) to stop admission, then decide between holding and the full rollback. |
| + |
| +## Step 7. Acceptance |
| + |
| +Paul's go: for the reboot only. |
| + |
| +Use the acceptance host chosen under hazard H1. Call it `$HOST`. Every curl runs outside the sandbox. |
| + |
| +1. Launch. `curl -s --resolve $HOST:443:18.200.150.210 https://$HOST/`. Expect 200 with `host=ip-172-31-19-180`, and on leo a `carlos-instance-gen@$HOST:<n>` unit active. |
| +2. First cold wake. Wait for it to sleep: `carlos logs --app <app> --console https://platform.dev.carlosframework.com --since 30m` ends in "instance exited" (idle is 15 minutes). Wake it again. Expect 200, and `hits=` one higher than before: the row written at launch survived the restore. |
| +3. Second cold wake. Repeat. Expect 200 and `hits=` one higher again. |
| +4. Isolation. While it is awake: |
| + |
| + ```bash |
| + leo 'pid=$(systemctl show -p MainPID --value "carlos-instance-gen@'"$HOST"':*"); ls /proc/$pid/root/data/instances/; readlink /proc/$pid/ns/mnt /proc/1/ns/mnt' |
| + ``` |
| + |
| + Expect only the host's own directory, and two different mount namespaces. `carlos status --layout` shows the host `nested`. |
| +5. Reboot. `aws --profile kass --region eu-west-1 ec2 reboot-instances --instance-ids i-046c7fe43620944bc`. After it returns: `carlos-instance-control.service` and `carlos-edge.service` active; `/var/lib/carlos-claims` intact; no recovery or park error in `journalctl -u carlos-instance-control.service -b --no-pager`; a wake answers 200 with `hits=` continuing. |
| + |
| +Pass: all five. Record each response and the journal lines on #250. |
| + |
| +## Full rollback to claims off |
| + |
| +Paul's go: required. |
| + |
| +1. Stop admission and drain: `systemctl stop carlos-edge.service`, then wait for every instance unit to stop. |
| +2. Record outstanding park failures and retained disk space, as `[runbook.rollback]` says. |
| +3. Remove both flags: |
| + |
| + ```bash |
| + leo 'sed -i.bak-claimsoff-$(date -u +%Y%m%d) "/^CARLOS_HOST_CLAIM=/d;/^CARLOS_HOST_CLAIM_ORIGINS=/d" /etc/carlos/host.env; grep -c "^CARLOS_HOST_CLAIM" /etc/carlos/host.env' |
| + ``` |
| + |
| + Expect `0`. |
| +4. Render the flag-off instance-control unit (the adopt pass does this on its next run; or run the reviewed installer), then `systemctl daemon-reload && systemctl restart carlos-instance-control.service`. `systemctl cat carlos-instance-control.service | grep -c StateDirectory=carlos-claims` prints `0`. |
| +5. `systemctl start carlos-edge.service`. Wake the acceptance host. Expect 200. |
| + |
| +Keep, and never delete by hand: `/var/lib/carlos-claims`, the guard, the floor, the roster enrollment, parked databases and every claim or root record. Never roll below the floor. Leo's census stays enrolled, so root keeps maintaining retained state during update-time recovery with claims off. |
| + |
| +Data written under the claim went to the host's claim location, `instances/<host>/<incarnation>`. A claims-off wake restores from the legacy replica. Confirm with platform-1e how claim-era writes reach the claims-off path before rolling back a host that matters (open question 12). |
| + |
| +## Known hazards |
| + |
| +**H1. wakecheck (#286).** Decision proposed: wakecheck stays on sophie and leaves leo for cohort 1. Acceptance uses a new leo-only app. |
| + |
| +Why: wakecheck's record is unregioned, so sophie and leo both mint it. With leo claimed and sophie not, the same host has two writers on different lineages. Leo's claimed generations restore from the claim location; sophie writes its legacy replica and also starts from its own stale local database (#286). DNS for `*.dev.carlosframework.com` points at sophie, and bots wake wakecheck there, so "woken on leo only" cannot be guaranteed. A wake on sophie during the window would also break the drained, replicated stop the seed depends on. |
| + |
| +How: set `CARLOS_POOLS=cohort1` in leo's host.env (a host.env change, under step 5's go). A box with `CARLOS_POOLS` set serves only the listed pools, not the shared `""` pool, so leo's reconcile pass removes wakecheck. Create the acceptance app with `carlos instances create --pool cohort1` (store mode), so only leo mints it. The activation preflight skips records in other pools. Alternative, if Paul rejects the pool change: keep wakecheck on both boxes and accept that acceptance on leo is not proof of isolation from sophie. Not recommended. |
| + |
| +**H2. Shared use of leo.** platform-d8 runs zero-downtime dev checks on leo. Notify them before step 5 and wait for a clear answer. After step 6, leo declines `CARLOS_SWAP=overlap` with `host-claims-on`, so their checks cannot run on leo until a full rollback. A `CARLOS_POOLS` change also means their test apps need `--pool cohort1`. |
| + |
| +**H3. #272.** A system pointer whose `result.json` says "applied" is terminal: after an old-road roll the system leg never returns the box to its channel pointer. Roll leo only with `carlos system update`. Separately, #272's stale-artefact refusal is deferred past activation: once a host is retired on leo, installing it again on leo refuses. That fails closed. Never remove `consumed/`, lock or record files by hand. |
| + |
| +**H4. The old-updater instance-control restart.** An updater older than v0.26.0 does not restart `carlos-instance-control`, and the new agent then refuses isolated wakes. Dev's boxes run v0.27.3, whose updater does restart it. Check `ActiveEnterTimestamp` after every roll anyway. Changing host.env never restarts instance-control on its own; step 5.5 restarts it explicitly. |
| + |
| +**H5. Runtime masks lose to `/etc`.** Blocker 3. Do not start the window until platform-1e names a masking method that shows `LoadState=masked` on these boxes. |
| + |
| +**H6. Empty app segment.** Blocker 2. Something on leo still writes error objects under `apps/bab//errors/leo-climbing/` (newest 06:46Z today). Deleting them is a bucket change for Paul's go, and the writer needs a fix or new objects will reappear. |
| + |
| +**H7. The dev console is not a claim actor.** With sophie excluded, nothing on dev can perform an operator `reset-wake-retry`, so an exhausted wake-retry episode on leo stays blocked until it is reset. Enrolling the console would arm the guard on sophie, which is otherwise excluded (open question 7). |
| + |
| +**H8. Memory.** Leo has 927 MB. Claim mode raises instance-control's cap to 256M. Watch `systemctl show carlos-instance-control.service -p MemoryCurrent` during step 7. |
| + |
| +## Optional second region: carlos-dev-uk |
| + |
| +dennis-stenciling is placement `uk`, serves nothing, and already has its own `CARLOS_SECRETS_KEY` in host.env (fingerprint `3fe29008`, the same as sophie's). Adding it means running steps 2 to 7 on `dennis` as well, with these differences: |
| + |
| +- IAM: `carlos-dev-host-uk` already has `s3:GetObjectVersion` and `s3:DeleteObjectVersion` on the UK data bucket. It still needs its own fence-writer role, and the deployment bucket policy covers both boxes. |
| +- Roster: its row has region `uk`. The home designation stays the IE name. |
| +- Files seed: it has no hosts, so `carlos system files seed` reports nothing to seed. Acceptance needs an app created with `--region uk`. |
| +- Guard: `/opt/carlos/carlos` on dennis is owned `carlos:carlos`, not root, and it has no `carlos.prev`. Check with platform-1e whether `guard-arm` accepts that before step 3b on dennis. |
| +- Its `carlos-instance-control.service` is idle and socket-activated today. Step 5.5's explicit restart is still required. |
| +- It rolls on the `stable` channel with leo. |
| + |
| +## Open questions |
| + |
| +For Paul: |
| +1. wakecheck (H1): approve `CARLOS_POOLS=cohort1` on leo and a leo-only acceptance app, or choose otherwise. |
| +2. Include dev-uk in cohort 1? |
| +3. platform-1e's step 1 rolls the release to flagship after dev, claims off. Must flagship be rolled before dev's offline window, or only before a flagship cohort? |
| + |
| +For platform-1e: |
| +4. Which reviewed runner creates the roster, invites and enrolls leo, installs `host-evidence.key` and the fence-writer credentials, publishes the floor and publishes `home_region` (blocker 1)? |
| +5. How should the six offline units and the edge be masked when their unit files are in `/etc/systemd/system` (blocker 3)? `verifyOfflineRootMigration` needs `LoadState=masked`. |
| +6. Which installer renders the claim-on instance-control unit while the adopt pass is masked (5.3)? |
| +7. Is the dev console a claim actor for cohort 1 (H7)? |
| +8. Dev's home roster name for `home_region`: `ie`? |
| +9. The empty `apps/bab//` prefix (blocker 2): who removes those objects, and which code writes them? Should `hostops.Apps` skip an empty segment instead of refusing? |
| +10. Plan 15's independent claim writer: what is the dev boundary, given leo's role can put to `control/hosts/`? |
| +11. Box roles in eu-west-1 can read `/carlos/release-key` from SSM. Does certification require moving that key or narrowing the grant? |
| +12. After a full rollback, how do writes made under the claim reach the claims-off restore path? |