amadan
Add the layout-migration build plan (flat->nested engineering sequence)
Summary
Turns the migration runbook's prerequisite build (isolated-exec plans 4-5) into fleet-safe landable increments A-F: nested-path primitives + stop drift (A), carlos status --layout red gate (B, the safe first step), the converter package + agent wiring (C), downgrade guards (D), the operator migrate/revert verb (E), and mint-nested-by-default last (F). Settles nested (User=carlos) over unit-backed DynamicUser to avoid a hibernation/replication data-loss rewrite.
Agent notes
Implementer code-state notes in the doc; one to double-check: it claims console lives under cmd/console/ not internal/console/ — internal/console/ does exist in this tree, so verify before remapping. Increment B is the minimal first branch and is being implemented separately.
1 file changed,
+662
−0
docs/superpowers/plans/2026-10-06-layout-migration-build-plan.md+662 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-10-06-layout-migration-build-plan.md b/docs/superpowers/plans/2026-10-06-layout-migration-build-plan.md| new file mode 100644 |
| index 000000000..c43381365 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-06-layout-migration-build-plan.md |
| @@ -0,0 +1,662 @@ |
| +# Flat -> nested layout migration: the build plan |
| + |
| +- **Date:** 2026-10-06 |
| +- **Driver:** finding #1 of `docs/security/2026-10-06-access-assessment.md` |
| + ("Flat instance layout provides no isolation"), confirmed live on |
| + `carlos.exciting.io`. Every managed instance runs flat: a plain child of |
| + `carlos-edge.service`, uid `carlos`, with a full view of `/data` and |
| + `/etc/carlos`. Any one hostile tenant reads every sibling's database and |
| + admin token and the control-plane credentials. Attack chain A: full takeover. |
| +- **Companion:** `docs/superpowers/plans/2026-10-06-flat-to-nested-migration.md` |
| + is the RUNBOOK (what an operator runs, host by host, once the machinery |
| + exists). This document is the ENGINEERING PLAN that builds that machinery: |
| + it turns §7 (the prerequisite build) and §8 (the four guardrails) of the |
| + runbook into a sequenced set of landable branches. It changes no Go source. |
| +- **Source of the design:** the isolated-exec series already specified the |
| + converge/journal/console mechanics in full. This plan is a re-sequencing of |
| + `2026-09-12-isolated-exec-4-layout.md` (plan 4) and |
| + `2026-09-12-isolated-exec-5-console.md` (plan 5), checked against the design |
| + spec `2026-09-11-isolated-exec-design.md` (revision 15) and against the live |
| + code on `land-ica`. Where this plan deviates from plans 4/5 it says so and |
| + why; otherwise plans 4/5 are the task-by-task authority and this plan points |
| + at them rather than restating their code. |
| + |
| +--- |
| + |
| +## 1. End state and the invariant |
| + |
| +**End state.** Every `ActivatorManaged` instance on every box is *nested*: |
| +its database and env files live under `/data/instances/<iid>/`, its socket |
| +under `/run/carlos/instances/<iid>/app.sock`, and it wakes as |
| +`carlos-instance@<iid>.service` inside a private mount + pid namespace |
| +(`TemporaryFileSystem=/data:ro`, a single `BindPaths=/data/instances/%I`, |
| +`InaccessiblePaths=/etc/carlos`, IMDS denied, read-only cgroupfs). A sibling's |
| +database, socket, token and the control-plane credentials are not |
| +permission-denied in that namespace; they *do not exist*. |
| + |
| +**The invariant, stated three ways because it must hold at three layers:** |
| + |
| +1. **The mint must not produce flat.** Once the fleet default is nested, the |
| + provisioning reconciler writes nested paths at first mint, so a freshly |
| + minted instance is never flat even for one wake. |
| +2. **The converge must not leave flat.** A box whose recorded intent is |
| + `nested` runs the layout pass on every serial move; a managed row still |
| + flat after a converge attempt is surfaced as a loud, logged drift line |
| + naming the host and its layout, never silently tolerated. |
| +3. **The status surface must fail on flat.** `carlos status --layout` renders |
| + a flat *managed* row as a failing check in red and exits non-zero, so an |
| + operator or a CI gate reads "box X has N flat instances" as the defect it |
| + is. |
| + |
| +Flat is a security defect, not a valid state. The whole build exists to make |
| +that true mechanically rather than by runbook discipline. |
| + |
| +**What is already landed (isolated-exec plans 1-3, on `land-ica`), so the |
| +build does not touch it:** |
| + |
| +- `internal/activator/layout.go` `Classify` already returns |
| + `flat`/`nested`/`fenced`/`conflict`/`foreign` from a row's two columns, a |
| + journal probe (`LayoutInputs.Journal`) and a peer scan (`LayoutInputs.Peers`). |
| +- `internal/activator/backend_exec.go` `Start` already dispatches a |
| + `LayoutNested` row to `startUnit`/`seedUnit` (the `carlos-instance@<iid>` |
| + unit); `FenceAndStop` already quiesces a host (`maintenance.go`). |
| +- `internal/provision/provision.go` already defines every nested path helper |
| + (`InstanceID`, `InstanceDir`, `NestedDBPath`, `NestedSocket`, |
| + `InstanceRunDir`, `LayoutJournalPath`) beside the flat ones |
| + (`DefaultDBPath`, `DefaultSocket`, `SocketDir`). |
| +- `internal/host/units/` already carries `carlos-instance@.service`, its seed |
| + and sidecar templates, the slices, the three `nosymfollow` mount units, the |
| + probe, and the `carlos-instance-control` socket/service. |
| + |
| +**What is missing (plans 4 and 5), which this plan builds:** the registry CAS, |
| +a generation-matched fence release, the fleet intent object, the whole |
| +`internal/layout` package (plan/journal/preflight/pass), the agent wiring, the |
| +status surface, the downgrade guards, and the operator trigger (console door + |
| +CLI verb). The reconciler still mints flat (its path write at ~388-391 is |
| +gated on `Record.DerivesPaths()` - an instance kind with no explicit upstreams |
| +- and always writes `DefaultSocket`/`DefaultDBPath`), and its drift block at |
| +~1108-1121 *logs* a nested-shaped managed row as a "different route shape" that |
| +`carlos remove` should fix (it does not rewrite the row, but it is noise that |
| +tells an operator a correct nested row is wrong); that is the first thing to |
| +fix. |
| + |
| +**Two code-state notes the implementer must carry (verified on `land-ica`):** |
| + |
| +- The classification struct is `activator.LayoutInputs` (fields `DataDir`, |
| + `SocketDir`, `Journal func(host) bool`, `Peers func() ([]registry.Route, |
| + error)`, `SameID`). Plans 4/5 and the runbook sometimes call it |
| + `LayoutFacts`; that is only the name of the `ExecBackend` *field* of type |
| + `LayoutInputs`, not a separate type. |
| +- Plans 4/5 place the console door, status door and view under |
| + `internal/console/*`. **That directory does not exist on `land-ica`;** the |
| + console lives under `cmd/console/` (and the root CLI in `cli.go`/`main.go`). |
| + Map every `internal/console/...` path in plans 4/5 to the actual console |
| + package before implementing increments B and E; the logic is unchanged, only |
| + the file home. |
| + |
| +--- |
| + |
| +## 2. Decomposition into landable increments |
| + |
| +Six increments, each its own amadan branch, ordered so the fleet is never |
| +worse off at any step. The spine is: make a nested row *tolerated* before one |
| +can exist; build the converter *dormant* (no intent -> no behaviour change); |
| +make the defect *visible*; add the *guards*; add the *trigger*; and only then, |
| +once the fleet is proven nested, flip the *mint* default so flat can never |
| +come back. Increments A-E change no running behaviour until an operator writes |
| +the intent object in E; F is the point of no easy return and goes last. |
| + |
| +The mapping to the existing plans: A and D and F are plan 4 task 1 and task 6 |
| +(split along the fleet-safety seam); B is plan 4 task 5; C is plan 4 tasks |
| +2-4; E is plan 5 tasks 1-4. This plan re-groups those tasks; it does not |
| +re-specify their code. Implement each increment with the matching plan-4/plan-5 |
| +tasks open beside it. |
| + |
| +### Increment A - Accept and derive nested; the primitives |
| + |
| +**One line:** add the compare-and-swap, the fence release and the intent type, |
| +and stop the reconciler treating a nested-shaped managed row as drift, so a |
| +nested row is a legal state the fleet tolerates. Mints nothing, converges |
| +nothing. |
| + |
| +**Depends on:** nothing (Classify and FenceAndStop are landed). |
| + |
| +**Files / functions (plan 4 task 1, plus the reconcile half of task 6):** |
| + |
| +- `internal/registry/registry.go` (modify): `SetInstancePaths(host, fromDB, |
| + fromSock, toDB, toSock) error` + `var ErrPathsChanged`. One `UPDATE ... SET |
| + db_path=?, socket=? WHERE host=? AND db_path=? AND socket=?`, guarded by |
| + **both** current column values; `n==0` re-reads the row and returns |
| + `ErrPathsChanged` naming what it holds now. It is `SetDBPath`'s sibling; the |
| + two columns move together because a row with one flat and one nested path is |
| + neither layout and every consumer refuses it. |
| +- `internal/activator/maintenance.go` (modify): `Unfence(host, generation) |
| + bool` (releases only the matching generation; a late earlier pass never |
| + removes a newer fence) and `Fenced(host) bool` (= `!wakeAllowed(host)`). This |
| + is the generation-matched release of the in-process fence `FenceAndStop` |
| + installs; it sits beside the existing `LiftFence`/`wakeAllowed`, which are |
| + not generation-matched and are not what the pass needs. *(Owned by the |
| + activator package; coordinate the edit there.)* |
| +- `internal/release/layoutintent.go` (create): `LayoutIntent{Schema, ID, |
| + RequestedAt, Mode, Scope}`, `LayoutScope{Box}`, `LayoutModeNone/Flat/Nested`, |
| + `LayoutIntentKey() = "control/layout.json"`, `LoadLayoutIntent`, |
| + `PutLayoutIntent`, `Applies(boxLabel)`, `ValidLayoutMode`. The reader |
| + tolerates an absent object (every deployment's state until an operator acts) |
| + and errors on a schema it does not understand. |
| +- `internal/provision/reconcile.go` (modify): the drift block at ~1108-1121 |
| + compares `routeFor(rec, ...)` (which derives flat `Socket`/`DBPath`) against |
| + the stored row and, on any `Socket`/`DBPath`/`Kind`/`Unit` mismatch, logs |
| + "different route shape ... not converged in place; carlos remove ... as a |
| + last resort". It only *logs* - it does not rewrite the row - but a nested row |
| + trips it on every pass, telling an operator a correct nested row is wrong. |
| + Add `provision.NestedPaths(dataDir, rt) bool` (shape-only: `DBPath` under |
| + `<dataDir>/instances` or `Socket` under `<SocketDir>/instances`) and, when it |
| + holds and the kind matches, log `row is on the nested layout ... the record's |
| + flat derivation is not drift` instead of the shape-mismatch message. This is |
| + the "reconcile drift fight" the brief names; it must land before anything |
| + creates a nested row, so the first nested row is not mistaken for corruption. |
| + |
| +**Data shapes:** `control/layout.json` = `{schema:1, id, requested_at, mode, |
| +scope:{box}}`. The CAS compares the two path columns only. |
| + |
| +**Ordering / CAS:** none at runtime yet - this increment adds the primitive |
| +and the toleration. The CAS is exercised only in C. |
| + |
| +**Fleet effect:** none. No row changes; the reconciler simply stops fighting a |
| +nested row that does not yet exist. |
| + |
| +### Increment B - The loud flag: `carlos status --layout` |
| + |
| +**One line:** surface every managed row's layout, render a flat managed row as |
| +a FAILING check, and exit non-zero when any managed host reads flat - the |
| +single smallest change that makes the regression visible everywhere. |
| + |
| +**Depends on:** nothing load-bearing (uses the landed `activator.Classify`). |
| +Independent of A; can land first. |
| + |
| +**Files / functions (plan 4 task 5):** |
| + |
| +- `internal/status/snapshot.go` (modify): `RowLayout{Layout, Reason}`, |
| + `LayoutReport{Mode, Rows map[string]RowLayout}`, `Box.Layout *LayoutReport` |
| + (private, like `Metrics`; never merged into the public `fleet.json`). |
| + `Probes` gains `Layout *LayoutReport`; `CollectBox` copies it. |
| +- `internal/host/agent.go` (modify): the snapshot writer fills `Probes.Layout` |
| + - `Mode` from the last intent the box saw (`"none"` before any), `Rows` from |
| + `Classify` over every `ActivatorManaged` row (`external` for an unflagged |
| + socket route). |
| +- `internal/console/apistatusinstances.go` + `authcli.go` (modify): a |
| + `GET /api/cli/status/layout` door behind the same fleet-operator gate as |
| + `/status/instances`, one entry per box with a `stale` flag. |
| +- root `status.go`/`status_instances.go` (modify): `carlos status --layout` |
| + prints `BOX HOST LAYOUT REASON`, sorted, stale boxes suffixed `(stale)`, and |
| + **exits non-zero when any managed host reads `flat`**. `internal/console/view.go` |
| + + `templates/app_instance.html` gain a `Layout` fact on the instance page. |
| +- `docs/site/cli.md` (modify): the `--layout` paragraph and a stub |
| + `#### Instance layout` heading under `carlos system` (the verb itself lands |
| + in E; the heading is stubbed now so the docsite gate is satisfied once it |
| + lands). |
| + |
| +**Fleet effect:** read-only. On day one every box reports every managed row |
| +flat, so `carlos status --layout` is red everywhere - which is correct: it is |
| +the inventory and the CI gate for the whole migration. |
| + |
| +**Deviation from plan 4:** plan 4 reads the box `Mode` from "the last intent |
| +`layoutLeg` saw", which only exists after C wires the pass. Landing B first, |
| +report `Mode: "none"` unconditionally and fill it in C. The per-row |
| +classification does not need the intent and is the load-bearing part. |
| + |
| +### Increment C - The converter: journal, plan, preflight, pass, agent wiring |
| + |
| +**One line:** build `internal/layout` (the recoverable one-shot converter) and |
| +wire it into the agent at start (`Recover`) and on every serial move |
| +(`Converge`) - dormant, because with no intent object no box converges. |
| + |
| +**Depends on:** A (`SetInstancePaths`, `Unfence`/`Fenced`, `LayoutIntent`). |
| + |
| +**Files / functions (plan 4 tasks 2, 3, 4):** |
| + |
| +- `internal/layout/plan.go` (create): `PlanFor(dataDir, socketDir, host, mode)` |
| + derives `From`/`To` endpoints and the seven `Artifact`s (db, db-wal, db-shm, |
| + env, sidecar-env, config-env, socket) from the host name alone. The nested |
| + destinations are expressed through the **raw** directory |
| + (`rawInstanceDir = <dataDir>/.instances/<iid>`, `rawRunDir = |
| + <socketDir>/.instances/<iid>`); flat mode is the exact reverse. `internal/layout` |
| + is the *only* non-test caller of the literal `.instances`, pinned by a grep |
| + test (§3.2 of the design). |
| +- `internal/layout/journal.go` (create): `Journal{Host, Mode, From, To, |
| + Artifacts, Phase, WrittenAt}`, phases `files`/`row`/`done`, `WriteJournal` |
| + (`O_EXCL`), `ReadJournal` (`O_NOFOLLOW|O_NONBLOCK`, regular-file only, 64 KB |
| + cap), `UpdatePhase` (temp + rename), `RemoveJournal`, and `Validate` - the |
| + recovery check that re-derives both endpoints from the journal's own |
| + direction and refuses a journal naming anything else (a journal is data, not |
| + an instruction). |
| +- `internal/layout/preflight.go` (create): `Preflight` (the `<host>` vs |
| + `<host>.config` ownership rule, both directions), `DestinationCheck` (two |
| + databases at the destination = refusal; an env already there = moved aside |
| + `.aside-<unix>`; config-env replaced), `EnsureDirs` (raw data dir 0700, raw |
| + run dir 2750 with the `carlos-sockets` gid). |
| +- `internal/layout/pass.go` (create): `Pass{Registry, Fence, DataDir, |
| + SocketDir, SocketsGID, Reload, Now, Logf, Generation}`, `Converge(ctx, mode)`, |
| + `Recover(ctx)`, `migrateHost`, `Classify`. The `Fence` interface is |
| + `FenceAndStop`+`Unfence`, satisfied by `*activator.Service`. |
| +- `internal/host/agent.go` (modify): `newLayoutPass(...)`; call `pass.Recover` |
| + at agent start (before the activator serves, so a half-moved host is never |
| + woken); add a `layoutLeg` that reads `control/layout.json` and, when it |
| + `Applies` to this box, runs `pass.Converge(mode)` - called in `fastTick` |
| + under the same `Serial != lastSerial` branch as the restart and schedule |
| + legs. Record the intent `Mode` for B's box report. |
| +- `internal/appconfig/syncexec.go` and `internal/ops/litestream.go` (modify): |
| + both grow a `fenced func(host) bool` seam; `SyncExec` skips a fenced host |
| + with a printed line, the litestream generator drops a fenced host's db into |
| + `skipped`, so neither writes a bundle or replicates a database mid-move. |
| + |
| +**Data shapes:** the journal above, at `/data/layout/<host>.json`. |
| + |
| +**CAS / ordering:** `migrateHost` runs the nine-step pass (quiesce -> |
| +re-read-under-fence -> ensure dirs -> destination check -> journal (O_EXCL) -> |
| +rename files -> phase row -> `SetInstancePaths` CAS -> phase done -> remove |
| +journal -> reload -> unfence). `Recover` completes any journal to its own |
| +endpoint, idempotently, at every crash boundary. Detail in §3. |
| + |
| +**Fleet effect:** none. `Converge` only runs when `LoadLayoutIntent` finds an |
| +object that `Applies` to the box; nothing writes that object until E. Every |
| +existing test passes unchanged. |
| + |
| +### Increment D - The downgrade guards |
| + |
| +**One line:** stop an older binary or a careless flag from ever meeting a |
| +nested row. |
| + |
| +**Depends on:** C (`NestedPaths`, the pass, the journal all exist). |
| + |
| +**Files / functions (plan 4 task 6, minus the reconcile-drift half landed in A):** |
| + |
| +- `internal/host/systemapply.go` (modify): refuse a candidate binary that |
| + predates the layout (`candidatePredatesLayout`: its staged tree lacks |
| + `data-instances.mount`) while reversal is incomplete |
| + (`layoutReversalIncomplete`: any row with nested-shaped paths, any |
| + fenced/conflict row, any journal). A pre-layout binary unmounts the layout |
| + with itself and would serve a nested row from an empty mountpoint directory, |
| + restoring a second database; this refusal, by path shape and not by flags, is |
| + what the runbook's "hard floor" (§5) rests on. |
| +- `cli.go` (modify): `carlos route --hibernate=false` refuses a nested row |
| + (`layout revert first`) - clearing hibernation would hand the db to the box |
| + replicator at the wrong path and take the row out of the converter's reach. |
| +- `internal/provision/reconcile.go` (modify): the flag-application path refuses |
| + the same clear on a nested row. |
| +- `docs/site/cli.md` (modify): one sentence under `carlos system` on the |
| + rollback-refused-while-nested rule. |
| + |
| +**Fleet effect:** none on a flat fleet; these are refusals that only bite once |
| +rows are nested. |
| + |
| +### Increment E - The operator trigger |
| + |
| +**One line:** the verb and console door that record the fleet intent and bump |
| +the serial, so boxes in scope converge; plus the console's own downgrade |
| +refusal and the control service's mountpoint inspection. |
| + |
| +**Depends on:** A (the intent writer), C (the pass boxes converge with), B (the |
| +status report the console guard reads). |
| + |
| +**Files / functions (plan 5 tasks 1-4):** |
| + |
| +- `internal/release/layoutintent.go` (modify): `RecordLayoutIntent` (fresh id, |
| + read-then-CAS, 3 retries), `LayoutFloor` (the first release carrying the |
| + mount units) + `PreLayout(version)`. |
| +- `internal/console/apisystemlayout.go` (create) + `authcli.go`: `GET`/`POST |
| + /api/cli/system/layout`, owner gate, strict JSON, audit, serial bump. This is |
| + the only writer of a nested path in the whole platform, indirectly. |
| +- `system.go` (modify): `carlos system layout migrate|revert|status [--box] |
| + [--yes]`. |
| +- `internal/console/layoutguard.go` (create): `layoutDowngradeBlocked` - the |
| + rollback and update doors refuse a pre-layout target while any box reports a |
| + nested/fenced/conflict row, is stale, or has no layout field (unknown |
| + blocks). By observed state, never by intent. |
| +- `internal/instancectl/server.go` (modify): `inspectMountpoints` at start - in |
| + a private mount namespace, unmount each bind and list the raw directory; a |
| + held file blocks nested readiness, so a layout unmounted by hand can never |
| + leave two databases. |
| +- `docs/site/cli.md` (modify): fill the `#### Instance layout` subsection with |
| + the three verb forms and the rollback order. |
| + |
| +**Fleet effect:** still none until an operator actually runs `carlos system |
| +layout migrate --box <box>`. That command is the migration. Scope it to one box |
| +for the flagship run (`--box`); never write a fleet-wide `nested` intent first. |
| + |
| +### Increment F - Mint nested by default |
| + |
| +**One line:** flip the reconciler mint so a freshly provisioned instance is |
| +nested from its first wake, closing the last door through which flat returns. |
| + |
| +**Depends on:** the fleet migrated and observed nested, and E landed. This is |
| +the point of no easy return and goes last. |
| + |
| +**Files / functions (runbook §8 guardrail 1; design §8.6 step 5):** |
| + |
| +- `internal/provision/reconcile.go` (modify): the mint (the runbook's |
| + ~388-391, `rt.Socket = DefaultSocket(...)`, `rt.DBPath = DefaultDBPath(...)`) |
| + writes `NestedSocket(host)`/`NestedDBPath(dataDir, host)` and calls |
| + `layout.EnsureDirs` first, when the box/fleet intent is `nested`. Equivalently |
| + mint flat and let the first `Converge` nest it; the mint default is cleaner |
| + because the instance is never flat even for one wake. |
| +- A follow-up release (out of scope here, named for sequencing) drops the flat |
| + spawn path for managed routes entirely and refuses to start on a box whose |
| + `layout_mode` is not `nested`. |
| + |
| +**Fleet effect:** new instances are born nested. Existing ones were already |
| +migrated by E. |
| + |
| +### Recovery / rollback (cross-cutting, built in C and E) |
| + |
| +The converter is recoverable *by construction*; recovery is not a separate |
| +increment but a property of C's `Recover` and E's mountpoint inspection: |
| + |
| +- **Interrupted mid-move:** the journal is still present, so the host stays |
| + fenced (serves 503, no corruption). On the next agent start or pass, |
| + `Recover` reads the journal, validates its endpoints against the host's |
| + derived paths, sees whether the row is at `from` (CAS had not run) or `to` |
| + (it had), completes the move to the journal's own endpoint, removes the |
| + journal and unfences. No operator hand-editing. |
| +- **Nested wake fails health:** run `carlos system layout revert` for that box |
| + (mode `flat`) - the same pass with endpoints swapped moves the six files back |
| + and CASes the row back; the instance wakes as a plain edge child again. No |
| + byte lost. |
| +- **Layout unmounted by hand:** E's `inspectMountpoints` finds files under the |
| + raw mountpoint and blocks nested wakes until an operator clears them, so the |
| + destination check never has to choose between two databases. |
| + |
| +--- |
| + |
| +## 3. The converter algorithm in detail |
| + |
| +For one host, `migrateHost` runs this order. Steps before the journal leave |
| +nothing changed on failure; steps after it leave a journal that `Recover` |
| +finishes. |
| + |
| +1. **Eligibility.** The activator's own predicate: `Kind=="instance"`, |
| + `Hibernate`, `Backing==""`, `Tunnel==""`, `Socket!=""`. An unflagged socket |
| + route is externally supervised (titogo's router) - reported `external`, |
| + never touched. `foreign` rows (a hand-set `--db` path that is neither the |
| + flat nor the nested default) are reported and skipped: during the transition |
| + the registry is writable by every flat tenant at the same uid, so a row is |
| + not a trustworthy instruction to move an arbitrary file; the pass only ever |
| + moves files named `<host>.*` at the platform's own paths. |
| +2. **Quiesce.** `FenceAndStop(ctx, host, gen)` with a fresh generation: SIGTERM |
| + drain, prove the process is gone, install the in-process fence. An unproven |
| + stop (`ErrStopUnproven`) **aborts the move and leaves the fence installed** - |
| + never move a database whose writer might still be alive, and never release a |
| + fence over an unproven stop. This is the SQLite-WAL safety point: once the |
| + writer is proven gone, the `-wal`/`-shm` sidecars are either checkpointed |
| + away or safe to move alongside the `.db`. The pass does not run a manual |
| + `PRAGMA wal_checkpoint`; proving the writer dead is what makes the move safe, |
| + and the three db files are all recorded present and moved together. |
| +3. **Re-read under the fence.** Re-`Get` the row; if its columns are no longer |
| + the plan's `From` endpoint, abort (`row moved under the fence`). The copy |
| + classified a moment ago could be stale. |
| +4. **Ensure dirs** (nested mode): `EnsureDirs` creates the raw |
| + `/data/.instances/<iid>` (0700) and `/run/carlos/.instances/<iid>` (2750, |
| + `carlos-sockets` gid) so the bind mounts show them. |
| +5. **Destination check.** A `.db` already at the destination is a hard refusal |
| + (two databases, refusing to choose). An env/sidecar-env already there is |
| + moved aside `<name>.aside-<unix>`; a stale config-env is removed (sync |
| + re-renders it). Record which artifacts are present at the source; a missing |
| + *required* (the db) is a refusal, a missing optional is recorded absent. |
| +6. **Journal, `O_EXCL`.** Write `/data/layout/<host>.json` with `from`, `to`, |
| + the artifact presence vector and `phase:"files"`. From this instant the host |
| + is `fenced`: `Start`, `Seed`, `Restore`, the moves pass, `SyncExec`'s writer |
| + and the litestream generator all refuse it. The fence survives an agent |
| + restart because it is a file. |
| +7. **Move files.** `os.Rename` each present artifact from its flat source to |
| + its nested destination. **Through the raw directory |
| + `/data/.instances/<iid>/`, never the bind mount `/data/instances/<iid>/`.** |
| + The bind mount is a different mount; `rename(2)` across it answers `EXDEV`. |
| + `PlanFor` already encodes raw destinations; this is the EXDEV gotcha the |
| + brief names, and the grep test pins `internal/layout` as the only caller of |
| + the raw path. Same filesystem, same owner, atomic per file, no root. Then |
| + `UpdatePhase(row)`. |
| +8. **Flip the row.** `SetInstancePaths(host, from.db, from.sock, to.db, |
| + to.sock)`: one CAS on both columns. `ErrPathsChanged` means the row moved |
| + underneath the pass; the fence stays and the operator sees it. Then |
| + `UpdatePhase(done)`. |
| +9. **Release.** Remove the journal; reload routing synchronously (the agent |
| + calls the edge's `Reload` and the activator's route refresh rather than |
| + waiting for the 2s poll); unfence. A request that copied the old route |
| + before the reload is caught by the wake's re-read of the row before |
| + `Acquire` (design §6.8). |
| + |
| +**In-flight requests.** While the journal exists `Classify` returns `fenced`; |
| +`Start`/`Seed` refuse with `refusing to wake: layout migration in flight`, and |
| +the edge returns a clean **503** for the bounded window (the drain, sub-second |
| +renames, and the health-gated wake on the first request after release). Not a |
| +hang, not a crash: the instance is deliberately held down while its files move. |
| +Zero effect on any other host on the box - the pass is per-host and serial. |
| + |
| +**Litestream re-point.** A hibernating exec instance's db is replicated by the |
| +activator's own per-instance lease, keyed on the row's `DBPath`; after the CAS |
| +the activator replicates the nested path on the next wake. The box-wide |
| +`carlos-litestream` generator discovers by glob and filters by registry claim; |
| +its glob must include `instances/*/*.db` (design §9) so a nested |
| +non-hibernating db is replicated, and a fenced host is excluded. After a host |
| +flips, regenerate the box replication config from the registry (`carlos ops |
| +litestream`) and restart replication if it reports `changed`, so the box |
| +replicator and the registry agree on what each database is rather than leaving |
| +a stale generation pointed at the vanished flat path. The nested instance stays |
| +`User=carlos` (not `DynamicUser`), so the agent can still open the file to |
| +replicate it - the key reason nested, not unit-backed, is the target (§5). |
| + |
| +**Idempotency / resume.** `Recover` runs at agent start and at the top of every |
| +`Converge`. For each journal: `phase:files` -> for each present artifact, at |
| +destination = done, at source = rename, at neither = error (fence stays); |
| +`phase:row` -> row at `to` means the CAS committed, row at `from` means CAS now; |
| +`phase:done` -> remove the journal. A journal that passes `Validate` is |
| +completed to its own endpoint *even if the fleet intent has since changed*, then |
| +the host is converged to the current intent in the same pass before release - |
| +so a forged or stale journal can at most add one round trip, never leave a |
| +host half-nested. A journal whose endpoints are not the host's derived paths, |
| +or whose row sits at neither endpoint, is refused and the host stays fenced for |
| +an operator. |
| + |
| +--- |
| + |
| +## 4. Test strategy |
| + |
| +### Unit / integration (CI, no systemd needed) |
| + |
| +Per the design §10, added package by package as each increment lands: |
| + |
| +- `internal/registry`: `SetInstancePaths` is a CAS on both columns - the happy |
| + flip, a stale re-CAS returning `ErrPathsChanged` and naming the current |
| + values, a half-matching CAS (only one column) that also misses, an absent |
| + host returning a no-route error. (Increment A.) |
| +- `internal/activator`: `Unfence` releases only the matching generation; an |
| + older generation does not release a newer fence; a second release reports |
| + nothing. (A.) |
| +- `internal/release`: `LayoutIntent` round-trip, box scope vs fleet, a newer |
| + schema is an error; `RecordLayoutIntent` replaces under CAS with a fresh id; |
| + `PreLayout` / `LayoutFloor`. (A, E.) |
| +- `internal/layout`: `PlanFor` derives every path from the host and the flat |
| + plan is the exact reverse of nested; the db destination is the raw directory |
| + (a destination through the bind mount would EXDEV); the journal is exclusive, |
| + phases advance, a symlink/FIFO journal is refused; `Validate` checks both |
| + endpoints from the journal's own direction and refuses a forged source or |
| + destination; `Preflight` refuses an ambiguous `x`/`x.config`; |
| + `DestinationCheck` refuses two databases and moves an env aside; `EnsureDirs` |
| + modes. The pass: `Converge` migrates / is idempotent / reverts byte-for-byte; |
| + an unproven stop refuses and keeps the fence and writes no journal; foreign |
| + and unmanaged rows skipped; **`Recover` completes every crash boundary** |
| + (after-journal, mid-files, after-files, after-cas, before-remove) to a |
| + consistent nested state with no journal; a forged journal is refused and the |
| + host stays fenced; a journal completes to its own endpoint before converging |
| + to a changed intent. The grep test pins the raw `.instances` caller; the |
| + bind-mount EXDEV test runs where CI can mount (skipped otherwise). (C.) |
| +- `internal/host`: `layoutLeg` converges only when the intent applies to this |
| + box; `SyncExec` and the litestream generator skip a fenced host; |
| + `systemapply` refuses a pre-layout candidate while reversal is incomplete and |
| + installs when the box is flat. (C, D.) |
| +- `internal/status`: the `layout` field for each state, the box mode, a |
| + pre-field box renders `unknown`. (B.) |
| +- `internal/console` / `system.go`: the layout door's owner gate and byte- |
| + identical 404, the serial bump, the downgrade refusal order; the CLI verb |
| + parse. (E.) |
| +- `internal/instancectl`: `inspectMountpoints` - empty listings pass, a held |
| + file fails and blocks readiness, a missing `unshare` is `unverified`. (E.) |
| + |
| +### On-box dress rehearsal (the order that matters) |
| + |
| +**Never the flagship first.** The runbook §4 fixes the host order; the build |
| +plan's obligation is to prove the *mechanism* on a throwaway before a real |
| +tenant, with the exact pre/post checks: |
| + |
| +1. Roll the built release to a dev box (boxes before console). Every row stays |
| + flat, `layout_mode` is `none`, the probe verdict is `ready`, nothing else |
| + changes. Cold-boot one dev box and confirm the three mounts, the control |
| + socket and the probe come up (the `DefaultDependencies=no` ordering must not |
| + deadlock). |
| +2. **Migrate a throwaway first** - `alpha-bravo-123`, then `alpha-bravo-456`, |
| + then `james-test-1` (or a fresh `alpha-bravo-*` test host). For the first one, |
| + `carlos system layout migrate --box <dev box>` or the per-host path, then |
| + verify ALL of: |
| + - **Socket moved:** `/run/carlos/instances/<iid>/app.sock` exists AND |
| + `/run/carlos/<host>.sock` does not. |
| + - **Backed by a unit:** `systemctl list-units --all 'carlos-instance@*'` |
| + lists `carlos-instance@<iid>.service` loaded/active; |
| + `systemctl show -p MainPID --value carlos-instance@<iid>.service` is a real |
| + pid. |
| + - **No longer an edge child:** `pstree -ap "$(systemctl show -p MainPID |
| + --value carlos-edge.service)"` no longer shows this tenant as a direct |
| + child; it now lives under `carlos-instances-<iid>.slice`. |
| + - **Isolation holds (spot-check once):** from inside, `carlos instance-probe` |
| + asserts only its own directories, read-only cgroupfs, no `/etc/carlos`, |
| + IMDS refused. (The control service refuses to start isolated instances |
| + until the probe passes, so a nested instance that is up has already |
| + proven this.) |
| + - **Data integrity:** the nested db at `/data/instances/<iid>/<host>.db` is |
| + the live one; a row count / app health check matches the pre-migration |
| + value. |
| + - **Litestream still replicating:** `carlos ops litestream` reports the |
| + nested path; the replica advances. |
| +3. **Both directions before any real tenant.** On one dev box run `layout |
| + revert` and confirm the host serves as a plain edge child again (byte-for- |
| + byte restore, aside files noted), then `migrate` again. |
| +4. Only then follow the runbook order through the low-stakes and staging hosts |
| + (`apple-testers-2`, `dev-jelly-manager`), and `jam` / `jelly-manager` |
| + LAST, each in a quiet window. |
| + |
| +### Abort / rollback criteria |
| + |
| +- **Abort the migration of a host** if: the quiesce reports `ErrStopUnproven` |
| + (the writer may be alive); the destination check finds a second database; the |
| + CAS returns `ErrPathsChanged`; or the post-migration verification fails any of |
| + the five checks above. In every case the journal or the fence holds the host |
| + safe; let `Recover` settle it or run `layout revert` for that box. |
| +- **Abort the whole roll** if: the cold-boot mount ordering deadlocks; the probe |
| + verdict is `not enforced` on any box (nested wakes are correctly blocked); |
| + or a dev box cannot complete a migrate+revert+migrate cycle cleanly. Do not |
| + touch the flagship until a dev box has round-tripped. |
| +- **Hard floor:** never roll a box binary below `LayoutFloor` while any journal |
| + is unresolved or any row is nested - D's `systemapply` guard enforces this, |
| + and E's console guard enforces it fleet-wide. |
| + |
| +--- |
| + |
| +## 5. Risks and decisions |
| + |
| +- **Nested (`User=carlos`) vs unit-backed `DynamicUser` (`carlos-app@`). |
| + Decision: nested.** The single most important data-ownership reason: nested |
| + keeps the instance uid `carlos`, so the files stay `carlos`-owned and the |
| + agent keeps full read/replicate access - the migration re-owns not one byte, |
| + needs no chown, and leaves hibernation/restore/lease/sidecars untouched. |
| + Unit-backed runs each tenant under a per-start ephemeral DynamicUser with a |
| + 0700 `StateDirectory` the agent (uid `carlos`) **cannot read**, so it could |
| + neither restore nor replicate that database; replication would have to move |
| + to the root `carlos-litestream` unit and every route would need an explicit |
| + `-db` or be replicated by nobody - a data-loss risk the moment it sleeps. |
| + `UnitBackend` also has inert `Acquire`/`Restore`/`Seed` and refuses sidecars: |
| + choosing it here is a rewrite of hibernation, not an isolation fix. Nested is |
| + the right target for the seven live hosts. |
| +- **`InstanceID` collisions for long FQDNs.** `InstanceID(host)` is the host |
| + itself for names <= 40 bytes and otherwise `_` + 22 bytes + 16 hex of a |
| + sha256 (also 40 bytes). The seven live tenants have short labels, but their |
| + registry rows key on full FQDNs. Record as **checked, not assumed**: before |
| + migrating, list the box's managed rows and confirm no two resolve to the same |
| + `provision.InstanceID`. Two colliding rows classify `conflict` and *neither* |
| + wakes; the console create-door must refuse a new host that collides. |
| +- **Litestream / lease rebinding.** Covered in §3: the activator's per-instance |
| + lease re-keys on the nested `DBPath` on the next wake; the box-wide generator |
| + needs the `instances/*/*.db` glob; regenerate and restart after each flip. |
| + The failure mode to watch is a stale generation pointed at the vanished flat |
| + path - which is why the step is in the pass's ordered sequence, not left to a |
| + later cron. |
| +- **The reconcile drift fight.** The live reconciler treats a row whose paths |
| + differ from its derived flat values as un-convergeable drift and would fight |
| + the converter. Increment A fixes this *before* any nested row exists; A is |
| + therefore a true prerequisite, not a nicety. |
| +- **Fencing so a half-migrated instance never wakes mixed.** The journal is the |
| + fence and it is a file, so it survives an agent restart; `Validate` refuses a |
| + forged or stale journal rather than acting on it; every waking consumer |
| + refuses a fenced host. A half-migrated instance serves 503 (fail-safe), never |
| + a torn database or a cross-tenant read. |
| +- **Bucket / registry CAS races.** The row flip is a two-column CAS |
| + (`ErrPathsChanged` on a miss); the intent object is written read-then-CAS |
| + with retries; the fence is generation-matched. A concurrent operator or a |
| + racing pass loses the CAS and learns the current state rather than |
| + overwriting it. |
| +- **Blast radius / why staging matters.** The intent object converges *every |
| + box in scope on every serial move*. Scope it to one box (`--box`) for the |
| + flagship; a fleet-wide `nested` intent is a fleet-wide change nobody approves |
| + box by box. The converter runs in the agent but moves files under `/data` and |
| + flips registry rows and spawns root-managed units through the control channel: |
| + root-adjacent and fleet-converging. Prove it on a throwaway, then staging, |
| + then the flagship last. |
| + |
| +--- |
| + |
| +## 6. Effort and sequencing |
| + |
| +Rough t-shirt sizes (one engineer, with plans 4/5 as the detailed spec): |
| + |
| +| Inc | Scope | Size | Depends on | |
| +|---|---|---|---| |
| +| A | CAS + Unfence + intent type + reconcile drift toleration | **M** | - | |
| +| B | `carlos status --layout`, the loud flag | **M** | - | |
| +| C | `internal/layout` package + agent wiring + fenced consumers | **L** | A | |
| +| D | downgrade guards (systemapply, --hibernate=false, reconcile) | **S-M** | C | |
| +| E | operator trigger (console door, CLI verb, downgrade guard, mountpoint inspect) | **L** | A, C, B | |
| +| F | mint nested by default | **S** | fleet proven nested, E | |
| + |
| +Critical path: A -> C -> E, with B parallel to A and D after C. The whole build |
| +is roughly two to three focused weeks; the converter (C) and the operator |
| +surface (E) are the bulk. |
| + |
| +**The minimal first landable branch:** **Increment B, `carlos status |
| +--layout`.** It touches no spawn path and no running instance, it depends only |
| +on the already-landed `Classify`, and it delivers immediate safe value: it |
| +turns "every box is flat" into a visible, red, non-zero-exit check - the |
| +inventory for the migration and the regression gate for "never again" - before |
| +any file ever moves. Land B, then A (the primitives + drift toleration), then C |
| +(the dormant converter), then D and E, and only after the fleet is observed |
| +nested, F. |
| + |
| +--- |
| + |
| +## Appendix: deviations from plans 4 and 5 |
| + |
| +- **Re-sequencing.** Plans 4/5 are ordered as a single implementer's task list |
| + (plan 4 tasks 1-6 then plan 5 tasks 1-4). This plan re-groups them along the |
| + fleet-safety seam into six branches (A-F) so the fleet is demonstrably no |
| + worse off after each merge, and pulls the status surface (plan 4 task 5) |
| + forward as the first branch because it is pure visibility. |
| +- **Reconcile drift toleration lands first (A), not with the other guards |
| + (plan 4 task 6).** A nested row must be a tolerated state before the converter |
| + can create one, or the reconciler fights it on the next pass. The rest of task |
| + 6 (the active refusals) stays in D. |
| +- **Box `layout_mode` in the status report starts as `none` (B) and is filled |
| + by the pass (C).** Plan 4 assumes the intent-tracking exists when task 5 |
| + lands; landing B first means reporting `none` until C wires `layoutLeg`. |
| +- **Mint-nested (F) is deliberately last and its own branch**, separated from |
| + the converter, because it is the point of no easy return and must not land |
| + until the fleet is observed nested. |
| + |
| +Everything else follows plans 4 and 5 as written; their task steps, code and |
| +tests are the implementation authority for each increment above. |