Host claim restore runbook

Before enabling CARLOS_HOST_CLAIM anywhere, rehearse and approve the restore procedure below. Obtain scoped approval before changing production. The local tests do not authorize activation.

Group C floor and prerequisites

C1: release floor. Every writer and reader, including offline and recovery binaries, must preserve record_seq and understand SQL receipt identity, sql_recover_after, root-only journals and their migration. Older binaries drop record_seq. Once claims are on, never downgrade below the certified floor, even after turning the flag off. Select the exact certified release at rollout.

C2: console filesystem. Claim-mode SQL requires FS_IOC_GETFSUUID on the database's filesystem. Linux 6.5 or later supports this call; ext4, xfs and btrfs implement it, while tmpfs does not. Check Carloku's console kernel and its actual database filesystem before enabling the flag. In a controlled rehearsal, use the built release to verify that OpenGated succeeds on persistent scratch space on that filesystem. Checking the filesystem type alone is insufficient. There is no member CLI command for this probe; the rollout must supply a reviewed runner.

The root enrollment file /etc/carlos/host-actor.json must contain the console host's EC2 instance ID. SQL receipt identity binds that ID, the machine identity, filesystem UUID, inode and database nonce. A database restored onto a different instance opens under a new database ID. A file-level restore onto a new inode also gets a new identity. In both cases, copied receipts cannot settle effects bound to the original database. A block-level restore on the same instance, such as an EBS snapshot or a root-volume replacement, keeps the original database ID even though receipts written after the snapshot are missing. Never start the console on one without the restore break-glass below. The enrolled ID must match the live instance ID from Nitro DMI or IMDSv2. A copied enrollment file alone is refused; an off-EC2 console cannot open a gated database.

Issued or uncertain effects bound to an unavailable or replaced database remain unsettled, including a database replaced under a running console. After sql_recover_after, recovery reports an error and alerts operators. Expiry, missing receipts and registration timeouts do not prove an outcome.

Restore break-glass

1. Freeze new work for the affected apps within an explicitly approved maintenance scope. Positively stop or fence the original and replacement console writers; heartbeat expiry is insufficient.
2. Preserve the original database, its write-ahead log and shared-memory files, the restored copy, database identities, admission records, effect tuples and logs.
3. After a block-level restore on the same instance, keep the console stopped. Copy the restored database file and its write-ahead log to new files, move them into place and remove the old shared-memory file, so the console opens the copy on a new inode under a new database ID. Then treat the database from before the restore as unavailable (step 5).
4. If the original authoritative database remains available with its matching identity, use the reviewed recovery runner to call RecoverSQLTransaction against it. FenceSQLOutcome reports an existing committed receipt or commits a fence that prevents a delayed transaction from starting. Never rewrite identities to make a copy look original.
5. If the original database is unavailable, retain the uncertain effects and keep admission held. There is no automatic settlement across databases or member CLI for this repair. Resuming requires a separately reviewed and approved procedure with positive evidence for each effect. Without that evidence, the affected apps remain blocked.

Before restoring traffic, verify that matching effects have settled as sql-committed or sql-fenced, that mismatched effects remain uncertain and alerted, and that every affected registration and exclusive operation has settled. Preserve the audit evidence. Rehearse a restore onto a replacement instance, a same-instance block-level restore, database replacement and recovery on the certified binary before enabling the flag.

Journal rollout

Run the same certified build in the agent and instance-control. All journal mutations use protocol v2, with a 128-bit caller nonce and an issue time. Retries retain the exact request; rebuilding it from its payload creates a different call. A new request has a 15-minute execution window. Committed receipts are retained for 24 hours and pruned in bounded batches; older requests without positive commit evidence stay uncertain. A durable rejection boundary keeps pruned requests uncertain across clock corrections.

Retirement journals live under /var/lib/carlos-claims/retire, with cold-handover journals in its cold directory. Root accepts agent-written inventory only for the route's own data namespace and registry rows. Agent-written inventories cannot authorize moving platform binaries or units. Agent-writable journal directories and symlinks are refused.

Before the first start on a box with legacy journals, stop the edge and old agent writers and run carlos system journal-migrate with the certified binary. Migration requires signed root stop evidence and leaves the originals in place. Once the migration marker exists, later legacy files are ignored with an alert and do not block root startup. Do not delete the marker.

Flag-on instance-control uses MemoryMax=256M and GOMEMLIMIT=192MiB; flag-off limits are unchanged. This is a sizing estimate backed by bounded migration inputs, not a production memory measurement. Measure peak memory during the rollout rehearsal.

Home-region activation order

Upgrade every reader of control/host-actors.json to a build containing Plan 13 before publishing home_region. This includes consoles, agents, root services, update and rollback tools, and offline or recovery binaries. Old roster documents still parse on the new build; pre-Plan-13 binaries reject a signed roster containing home_region.

After verifying every reader's build, publish the home designation, then enable CARLOS_HOST_CLAIM. The designation is write-once: choose the deployment's home roster name before publishing it. Without a designation, a box runs claims under its roster name with no @home alias and logs that state once per process.

The signed fleet data does not identify the running build of every roster reader, so this ordering is an operator prerequisite rather than an automated publication preflight. Do not publish the designation until you have verified the remaining readers outside that data. Publishing it does not authorize a production rollout; the scoped approval requirement above still applies.