carlos / platform Public

Download this file

1	# Host claim restore runbook
2
3	Before enabling `CARLOS_HOST_CLAIM` anywhere, rehearse and approve the restore procedure below. Obtain scoped approval before changing production. The local tests do not authorize activation.
4
5	## Group C floor and prerequisites
6
7	**C1: release floor.** Every writer and reader, including offline and recovery binaries, must preserve `record_seq` and understand SQL receipt identity, `sql_recover_after`, root-only journals and their migration. Older binaries drop `record_seq`. Once claims are on, never downgrade below the certified floor, even after turning the flag off. Select the exact certified release at rollout.
8
9	**C2: console filesystem.** Claim-mode SQL requires `FS_IOC_GETFSUUID` on the database's filesystem. Linux 6.5 or later supports this call; ext4, xfs and btrfs implement it, while tmpfs does not. Check Carloku's console kernel and its actual database filesystem before enabling the flag. In a controlled rehearsal, use the built release to verify that `OpenGated` succeeds on persistent scratch space on that filesystem. Checking the filesystem type alone is insufficient. There is no member CLI command for this probe; the rollout must supply a reviewed runner.
10
11	The root enrollment file `/etc/carlos/host-actor.json` must contain the console host's EC2 instance ID. SQL receipt identity binds that ID, the machine identity, filesystem UUID, inode and database nonce. A database restored onto a different instance opens under a new database ID. A file-level restore onto a new inode also gets a new identity. In both cases, copied receipts cannot settle effects bound to the original database. A block-level restore on the same instance, such as an EBS snapshot or a root-volume replacement, keeps the original database ID even though receipts written after the snapshot are missing. Never start the console on one without the restore break-glass below. The enrolled ID must match the live instance ID from Nitro DMI or IMDSv2. A copied enrollment file alone is refused; an off-EC2 console cannot open a gated database.
12
13	Issued or uncertain effects bound to an unavailable or replaced database remain unsettled, including a database replaced under a running console. After `sql_recover_after`, recovery reports an error and alerts operators. Expiry, missing receipts and registration timeouts do not prove an outcome.
14
15	## Restore break-glass
16
17	1. Freeze new work for the affected apps within an explicitly approved maintenance scope. Positively stop or fence the original and replacement console writers; heartbeat expiry is insufficient.
18	2. Preserve the original database, its write-ahead log and shared-memory files, the restored copy, database identities, admission records, effect tuples and logs.
19	3. After a block-level restore on the same instance, keep the console stopped. Copy the restored database file and its write-ahead log to new files, move them into place and remove the old shared-memory file, so the console opens the copy on a new inode under a new database ID. Then treat the database from before the restore as unavailable (step 5).
20	4. If the original authoritative database remains available with its matching identity, use the reviewed recovery runner to call `RecoverSQLTransaction` against it. `FenceSQLOutcome` reports an existing committed receipt or commits a fence that prevents a delayed transaction from starting. Never rewrite identities to make a copy look original.
21	5. If the original database is unavailable, retain the uncertain effects and keep admission held. There is no automatic settlement across databases or member CLI for this repair. Resuming requires a separately reviewed and approved procedure with positive evidence for each effect. Without that evidence, the affected apps remain blocked.
22
23	Before restoring traffic, verify that matching effects have settled as `sql-committed` or `sql-fenced`, that mismatched effects remain uncertain and alerted, and that every affected registration and exclusive operation has settled. Preserve the audit evidence. Rehearse a restore onto a replacement instance, a same-instance block-level restore, database replacement and recovery on the certified binary before enabling the flag.
24
25	## Journal rollout
26
27	Run the same certified build in the agent and instance-control. All journal mutations use protocol v2, with a 128-bit caller nonce and an issue time. Retries retain the exact request; rebuilding it from its payload creates a different call. A new request has a 15-minute execution window. Committed receipts are retained for 24 hours and pruned in bounded batches; older requests without positive commit evidence stay uncertain. A durable rejection boundary keeps pruned requests uncertain across clock corrections.
28
29	Retirement journals live under `/var/lib/carlos-claims/retire`, with cold-handover journals in its `cold` directory. Root accepts agent-written inventory only for the route's own data namespace and registry rows. Agent-written inventories cannot authorize moving platform binaries or units. Agent-writable journal directories and symlinks are refused.
30
31	Before the first start on a box with legacy journals, stop the edge and old agent writers and run `carlos system journal-migrate` with the certified binary. Migration requires signed root stop evidence and leaves the originals in place. Once the migration marker exists, later legacy files are ignored with an alert and do not block root startup. Do not delete the marker.
32
33	Flag-on instance-control uses `MemoryMax=256M` and `GOMEMLIMIT=192MiB`; flag-off limits are unchanged. This is a sizing estimate backed by bounded migration inputs, not a production memory measurement. Measure peak memory during the rollout rehearsal.
34
35	## Home-region activation order
36
37	Upgrade every reader of `control/host-actors.json` to a build containing Plan 13 before publishing `home_region`. This includes consoles, agents, root services, update and rollback tools, and offline or recovery binaries. Old roster documents still parse on the new build; pre-Plan-13 binaries reject a signed roster containing `home_region`.
38
39	After verifying every reader's build, publish the home designation, then enable `CARLOS_HOST_CLAIM`. The designation is write-once: choose the deployment's home roster name before publishing it. Without a designation, a box runs claims under its roster name with no `@home` alias and logs that state once per process.
40
41	The signed fleet data does not identify the running build of every roster reader, so this ordering is an operator prerequisite rather than an automated publication preflight. Do not publish the designation until you have verified the remaining readers outside that data. Publishing it does not authorize a production rollout; the scoped approval requirement above still applies.
42