amadan
Isolated exec spec: revision 3 after codex round 2
Round 2 findings committed unedited. All 14 accepted: the migration becomes an operator verb (carlos system layout migrate|revert) on a two-layout release, so the updater's own rollback lands on a layout the old binary serves; exit records from ExecStopPost=+ with cgroup emptiness; journal outside tenant mounts; mounts with explicit ordering; carlos-sockets traversal kept; MemorySwapMax; probe errno checks.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YZ1ZvWj7naf73GjTX8MBv5
2 files changed,
+882
−407
docs/superpowers/specs/2026-09-11-isolated-exec-design.md+727 −407docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R2.md+155 −0
CI passed — run details
diff --git a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md| index 6efd6d780..62ee58737 100644 |
| --- a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| +++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| @@ -1,7 +1,7 @@ |
| # Isolated exec — every exec-backed instance in its own namespace |
| -**Date:** 2026-09-11 · **Status:** revision 2, after codex round 1 |
| -(19 findings, all accepted; §13) · **Requested by:** discussion #126 |
| +**Date:** 2026-09-11 · **Status:** revision 3, after codex rounds 1 and 2 |
| +(19 + 14 findings, all accepted; §13) · **Requested by:** discussion #126 |
| (Tito, with box evidence), approved in direction by Paul on 2026-09-11 |
| **Builds on:** discussion #126 and its companion #127; titogo's |
| @@ -10,8 +10,11 @@ systemd 257 and on carlos-tito-ie); `internal/edge/purge.go:53-107` and |
| `internal/activator/admintoken.go:6-14`, which already describe the gap; |
| the exec-child accounting design (2026-08-07, pinfra |
| `platform-design/superpowers/specs/`), the memory-pressure containment |
| -design (2026-08-28, same place), and the root-relay pattern |
| -(`carlos-stage.path` → `carlos adopt --stage-only`). |
| +design (2026-08-28, same place), the cross-user socket consumers group |
| +(issue #103, `internal/host/bootstrap.go:41-62`), the system-update |
| +design (2026-09-06) whose apply/rollback path the transition must |
| +survive, and the root-relay pattern (`carlos-stage.path` → |
| +`carlos adopt --stage-only`). |
| --- |
| @@ -29,14 +32,18 @@ Paul approved the third on the same day and asked that the one open |
| mechanism choice, how the unprivileged agent starts a root-managed unit, |
| be settled between Fable and codex during review. Codex's round 1 |
| concurred with the control service and showed why it must authenticate |
| -its caller (§5.4). |
| +its caller (§5.4). Round 2 turned the migration from an agent-start side |
| +effect into an operator-triggered fleet verb (§8), because the system |
| +updater's own rollback runs the *old* binary and cannot reverse a layout |
| +it never knew. |
| The shape in one sentence: **the activator keeps owning every exec-backed |
| instance exactly as it does today, but the process it owns is launched as |
| a systemd template unit, still uid `carlos`, inside a mount namespace in |
| which nothing but that instance's own directory exists, on a mount where |
| no symlink is ever followed.** No uid change, no console change, no new |
| -backing, and the migration is a rename the agent performs itself. |
| +backing. One release carries both layouts; an operator verb moves each |
| +fleet across; the same verb moves it back. |
| --- |
| @@ -66,7 +73,7 @@ Everything the instance owns sits flat in one directory, `DataDir` |
| | File | Written by | Read by | |
| |---|---|---| |
| | `<host>.db` (+ `-wal`, `-shm`, `.activator-<host>.db-litestream/`) | the instance; restore, seed and replication in the agent | the instance, the agent's in-process replicator (`internal/lease/lease.go:257-299`) | |
| -| `<host>.env` (admin token) | `EnsureAdminToken` (`admintoken.go:111-195`) | the agent (`backend_exec.go:507, 607, 790`; `internal/schedule/runner.go:461`) | |
| +| `<host>.env` (admin token) | `EnsureAdminToken` (`admintoken.go:111-195`) | the agent (`backend_exec.go:350, 507, 579, 607, 790`; `internal/schedule/runner.go:461`) | |
| | `<host>-sidecar.env` | hand-staged | `backend_exec.go:865-877` | |
| | `<host>.config.env` | `internal/appconfig/syncexec.go:872-926` | `backend_exec.go:786` | |
| @@ -74,6 +81,13 @@ All of it is `carlos:carlos`, and every instance is uid `carlos`. Mode |
| 0600 restricts nothing between siblings. `purge.go:53-95` says so in |
| prose, and names per-instance uids as "the real fix". |
| +The sockets are shared on purpose with one more party: `/run/carlos` is |
| +`2750 carlos:carlos-sockets` (`bootstrap.go:61`), and after the health |
| +wait the activator normalises each instance socket to `0660` |
| +(`backend_exec.go:744-750`), so a deployment's enrolled cross-user |
| +consumer (titogo's router, uid `titogo`, issue #103) can dial every |
| +instance socket on the box. That contract survives this change (§3.1). |
| + |
| ### 1.2 What a sibling can reach today |
| Everything below is a read (or write) by any exec-backed instance of any |
| @@ -120,9 +134,12 @@ the instance as its own environment (`backend_exec.go:785-794, 966-971`; |
| sibling's database and the agent's replicator uploads the sibling's data |
| under the attacker's replica. Replace the listening socket with a symlink |
| to any other socket on the box and the edge dials it as this host |
| -(codex R1-1, R1-3). A design that bounds the instance but not the paths |
| -the agent follows on its behalf has not bounded anything. §3.2 closes |
| -this at the kernel, for every reader, without a race. |
| +(codex R1-1, R1-3). And any *file* the agent reads as instructions from |
| +the instance's directory is the same primitive without a symlink |
| +(codex R2-1). A design that bounds the instance but not the paths the |
| +agent follows on its behalf, or the records it trusts from the |
| +instance's own directory, has not bounded anything. §3.2 closes the |
| +first at the kernel; §3.3 states the rule for the second. |
| ### 1.4 What the unit backing already gets right, and why it is not the answer |
| @@ -159,15 +176,17 @@ scoped to the three instance templates and does not extend to |
| `<DataDir>/<host>.*` to `<DataDir>/instances/<host>/` (§3). The agent |
| creates the directory and everything in it, as `carlos`, exactly as it |
| creates the flat files today. The registry's `DBPath` and `Socket` |
| - columns point into the new layout. |
| + columns point into the new layout once a host is migrated. |
| 2. **No symlink is ever followed under the instance trees.** |
| `/data/instances` and `/run/carlos/instances` are bind mounts of |
| themselves with `nosymfollow` (§3.2). The kernel refuses to traverse a |
| symlink there for every process on the box, agent and edge included, |
| with no check-then-use race. Files the instance must never touch (the |
| - log sockets, the control channel, its token) live outside those two |
| - trees, where no tenant mount reaches. |
| + log sockets, the exit records, the control channel, its token, the |
| + migration journal) live outside those two trees, where no tenant mount |
| + reaches. Nothing the agent does is ever directed by a file it read |
| + from an instance directory (§3.3). |
| 3. **A template unit per role.** The instance runs as |
| `carlos-instance@<host>.service`; its seed as |
| @@ -178,15 +197,16 @@ scoped to the three instance templates and does not extend to |
| with exactly the instance's own two directories bound back in, |
| `PrivatePIDs=yes`, `ProtectProc=invisible`, `RestrictNamespaces=yes`, |
| `IPAddressDeny=` for IMDS), and all three sit in one per-host slice |
| - that carries the host's memory tier. A sibling's database does not |
| - exist in this namespace; a sibling's socket does not exist; a sibling's |
| - pid does not exist. |
| + that carries the host's memory and swap tier and its whole-family OOM |
| + rule. A sibling's database does not exist in this namespace; a |
| + sibling's socket does not exist; a sibling's pid does not exist. |
| 4. **The agent stays the owner.** Lease, restore, seed ordering, |
| replication, hibernation, the control socket, `carlos logs`, the |
| next-due tick, sidecar co-lifecycle and residence moves are unchanged |
| - in `internal/activator`. What changes is the `Proc` the exec backend |
| - hands back: it is a unit, not an `*exec.Cmd` (§6). |
| + in `internal/activator`, with one bounded change in the sweeper (§6.7). |
| + What changes is the `Proc` the exec backend hands back: it is a unit, |
| + not an `*exec.Cmd` (§6). |
| 5. **One root channel, authenticated.** The agent cannot start a system |
| unit. A small root service, `carlos-instance-control`, socket-activated |
| @@ -196,31 +216,35 @@ scoped to the three instance templates and does not extend to |
| mount contains (§5). This is the synchronous form of the |
| `carlos-stage.path` pattern. |
| -6. **Launch through a trampoline.** Each unit's `ExecStart` is |
| - `/opt/carlos/carlos instance-exec <role> <host>`, which runs inside the |
| - namespace, reads the launch description the agent wrote into the |
| - instance directory, and `execve`s the app binary with the exact argv and |
| - environment the agent composed (§6.2). |
| +6. **Launch through a trampoline; exit through a record.** Each unit's |
| + `ExecStart` is `/opt/carlos/carlos instance-exec <role> <host>`, which |
| + runs inside the namespace, reads the launch description the agent |
| + wrote, and `execve`s the app binary with the exact argv and environment |
| + the agent composed (§6.2). Each unit's `ExecStopPost=+` runs outside |
| + the namespace as root and writes the invocation's exit record where |
| + the agent, and only the agent, reads it (§6.5). |
| 7. **Resource walls become slices.** `carlos.slice` carries the |
| 2026-08-28 outer wall; the edge unit and every instance sit under it; |
| each host has its own slice inside that, carrying the tier (§4.3, |
| §6.6). |
| -8. **The migration is a fenced rename.** On the first agent start on the |
| - new build every activator-managed instance is asleep (they died with |
| - the agent). Before the activator starts, the agent moves each such |
| - route's flat files into its directory under a per-host marker that |
| - every other consumer honours, and updates the two registry columns in |
| - one compare-and-swap. No root step, no chown, no operator action (§8). |
| - Routes the activator does not manage are not touched. |
| +8. **One release, two layouts, one operator verb.** This release's |
| + agent runs a *flat* host exactly as today (unisolated, `childProc`) |
| + and a *nested* host isolated. Nothing moves on upgrade, so the system |
| + updater's own rollback (which runs the previous binary) always lands |
| + on a layout that binary understands. `carlos system layout migrate` |
| + moves a fleet's hosts across under a journal the tenant cannot touch; |
| + `carlos system layout revert` moves them back; both are console |
| + verbs, reported per host in status (§8). The next release drops the |
| + flat path. |
| 9. **Addr routes are untouched, and now invisible.** A TCP-addr route's |
| `<host>.config.env` stays flat in `/data`, because the app's own |
| supervisor reads it there (titogo's `EnvironmentFile=-/data/%i.config.env`). |
| - Exec children can no longer see it. That closes the read #127 leads |
| - with; #127's ask for root-owned delivery stays open as its own change |
| - (§11). |
| + Isolated instances can no longer see it. That closes the read #127 |
| + leads with; #127's ask for root-owned delivery stays open as its own |
| + change (§11). |
| --- |
| @@ -239,20 +263,33 @@ scoped to the three instance templates and does not extend to |
| launch.run.json agent-written before each instance start (§6.2) |
| launch.seed.json agent-written before a seed |
| launch.sidecar.json agent-written before a sidecar start |
| - .migrating the migration fence (§8.1); absent when settled |
| <host>.db.corrupt a quarantined restore, as today |
| -/run/carlos/instances/ bind mount of itself, nosymfollow; 0755 carlos:carlos |
| -/run/carlos/instances/<host>/ 0700 carlos:carlos, created by the agent |
| - app.sock the instance's listener (rt.Socket) |
| - app.sock.control the agent's control listener (rt.Socket + ".control") |
| +/data/layout/ 0700 carlos:carlos; NOT under any tenant mount |
| + <host>.json the migration journal for a host in flight (§8.2) |
| +/run/carlos/instances/ bind mount of itself, nosymfollow; 2750 carlos:carlos-sockets |
| +/run/carlos/instances/<host>/ 2750 carlos:carlos-sockets, created by the agent |
| + app.sock the instance's listener (rt.Socket); 0660 after health, as today |
| + app.sock.control the agent's control listener (rt.Socket + ".control"); 0600 |
| /run/carlos/logs/ 0700 carlos:carlos; NOT bound into any namespace |
| <host>.run.sock the agent's log listener for the instance (§6.4) |
| <host>.seed.sock ... for the seed |
| <host>.sidecar.sock ... for the sidecar |
| +/run/carlos/exits/ 0750 root:carlos; NOT bound into any namespace |
| + <host>.<role>.<invocation>.json root-written exit record (§6.5) |
| /run/carlos/instance-control.sock the root channel (§5); not in any namespace |
| /run/carlos/instance-control.token 0600 carlos:carlos, root-written; not in any namespace |
| ``` |
| +The socket tree keeps the #103 contract: `/run/carlos/instances` and |
| +each host's run directory are `2750 carlos:carlos-sockets` (setgid, so |
| +the socket an instance binds inherits the group, exactly as |
| +`tmpfilesConf` arranges for `/run/carlos` today), the activator's |
| +post-health `0660` normalisation is unchanged, and the control socket |
| +stays `0600`. An enrolled consumer can traverse to and dial `app.sock` |
| +and nothing else (codex R2-10). Inside the namespace the instance owns |
| +its run directory and can loosen it; that changes nothing outside, |
| +because outside is where the mode is set and checked. |
| + |
| **One helper, one owner.** `internal/provision` gains |
| `InstanceDir(dataDir, host)` beside `DefaultDBPath`, and |
| `DefaultDBPath(dataDir, host)` and `DefaultSocket(host)` change to return |
| @@ -274,19 +311,19 @@ func DefaultSocket(host string) string { |
| func LogSocket(host, role string) string { |
| return filepath.Join(SocketDir, "logs", host+"."+role+".sock") |
| } |
| +func FlatDBPath(dataDir, host string) string { return filepath.Join(dataDir, host+".db") } |
| +func FlatSocket(host string) string { return filepath.Join(SocketDir, host+".sock") } |
| ``` |
| -Every DataDir-keyed builder in the tree moves onto these (the list is |
| -§9). `ExecBackend.defaultDBPath` (`backend_exec.go:160-162`) is deleted in |
| -favour of the provision helper, so there is one definition, not two. |
| +`FlatDBPath`/`FlatSocket` exist for this release only: they are how a row |
| +is classified (§8.1) and how the flat spawn path finds its files. Every |
| +DataDir-keyed builder in the tree moves onto these (the list is §9). |
| +`ExecBackend.defaultDBPath` (`backend_exec.go:160-162`) is deleted in |
| +favour of the provision helpers, so there is one definition, not two. |
| **File names inside the directory keep the host prefix.** It costs |
| nothing, it keeps every existing test fixture recognisable, and it means |
| -a file found outside its directory (a stray rename, a half-migration) is |
| -still attributable. The one thing the directory changes about names is |
| -that the `x.config.env`-versus-host-`x.config` ambiguity `reapConfigEnv` |
| -guards against (`syncexec.go:1009-1015`) can no longer arise for exec |
| -routes: there is exactly one host per directory. |
| +a file found outside its directory is still attributable. |
| **Why nested and not per-instance uids.** titogo's unit comment says it |
| best: "A flat `/data/titogo/%i.db` cannot be isolated — the bind would |
| @@ -302,52 +339,75 @@ that, and leaves option B open as a later hardening on the same layout |
| **Custom `DBPath`.** `carlos add --db` may stamp any absolute path on an |
| exec route (`cli.go:157-159` refuses only a relative one). Under this |
| design an exec route's `DBPath` must be inside `InstanceDir`, because |
| -that is the only data path the namespace reveals. `carlos add` and |
| -`carlos route` refuse an exec route whose `--db` is elsewhere, naming the |
| -directory. Existing rows are handled by the migration (§8.2). |
| +that is the only data path the namespace reveals. `carlos add` refuses a |
| +nested-layout exec route whose `--db` is elsewhere, naming the directory. |
| +Existing rows are handled by the migration (§8.3). |
| ### 3.2 Symlinks are never followed under the instance trees (codex R1-1, R1-2, R1-3) |
| Two mount units, embedded and installed like every other unit |
| -(`internal/host/bootstrap.go:33-34`), and two `tmpfiles.d` lines beside |
| -the existing `/run/carlos` one (`bootstrap.go:61`): |
| +(`internal/host/bootstrap.go:33-34`), and tmpfiles lines beside the |
| +existing `/run/carlos` one (`bootstrap.go:61`). Both mounts carry |
| +`DefaultDependencies=no`: a mount unit's default ordering places it |
| +before `local-fs.target`, and `systemd-tmpfiles-setup.service`, which |
| +creates the directories, runs after that target, so the defaults would |
| +form a cycle systemd breaks by dropping the mount (codex R2-2). The |
| +ordering is therefore written out in full, including shutdown: |
| ```ini |
| # data-instances.mount |
| [Unit] |
| Description=carlos instance data (bind, nosymfollow) |
| +DefaultDependencies=no |
| RequiresMountsFor=/data |
| -Before=carlos-edge.service carlos-litestream.service |
| +After=local-fs.target systemd-tmpfiles-setup.service |
| +Before=carlos-edge.service carlos-instance-control.service carlos-litestream.service |
| +Conflicts=umount.target |
| +Before=umount.target |
| [Mount] |
| What=/data/instances |
| Where=/data/instances |
| Type=none |
| Options=bind,nosymfollow,nosuid,nodev,noexec |
| [Install] |
| -WantedBy=local-fs.target |
| +WantedBy=multi-user.target |
| ``` |
| ```ini |
| # run-carlos-instances.mount |
| [Unit] |
| Description=carlos instance sockets (bind, nosymfollow) |
| +DefaultDependencies=no |
| After=systemd-tmpfiles-setup.service |
| -Before=carlos-edge.service |
| +Before=carlos-edge.service carlos-instance-control.service |
| +Conflicts=umount.target |
| +Before=umount.target |
| [Mount] |
| What=/run/carlos/instances |
| Where=/run/carlos/instances |
| Type=none |
| Options=bind,nosymfollow,nosuid,nodev,noexec |
| [Install] |
| -WantedBy=local-fs.target |
| +WantedBy=multi-user.target |
| ``` |
| ``` |
| -d /data/instances 0755 carlos carlos - |
| -d /run/carlos/instances 0755 carlos carlos - |
| -d /run/carlos/logs 0700 carlos carlos - |
| +d /data/instances 0755 carlos carlos - |
| +d /data/layout 0700 carlos carlos - |
| +d /run/carlos/instances 2750 carlos carlos-sockets - |
| +d /run/carlos/logs 0700 carlos carlos - |
| +d /run/carlos/exits 0750 root carlos - |
| ``` |
| +`carlos-edge.service`, `carlos-instance-control.service` and the three |
| +templates gain `Requires=` and `After=` on both mounts, so an edge that |
| +starts without them does not start at all rather than starting |
| +unprotected. The plan's first box task is a **cold boot** with the full |
| +unit set installed, then `systemctl list-dependencies --before |
| +data-instances.mount` and a `systemd-analyze verify` over the embedded |
| +set, pinned as a CI test that renders the units and runs |
| +`systemd-analyze verify` where the tool exists. |
| + |
| `nosymfollow` (Linux 5.10+, util-linux 2.36+; Debian 13 ships 6.12 and |
| 2.41) makes path resolution on that mount return `ELOOP` at any symlink, |
| for every process, in every namespace that sees the mount. It is a |
| @@ -371,10 +431,6 @@ What this closes, race-free and for every reader at once: |
| - the instance planting a symlink at a predictable temp name so the |
| agent's write truncates the target (R1-3). |
| -The log sockets and the control channel are additionally placed where no |
| -tenant mount reaches at all (`/run/carlos/logs`, `/run/carlos/instance-control.*`), |
| -so the question of a tenant touching them does not arise (R1-2). |
| - |
| Belt and braces in the agent, cheap and testable: |
| - every agent open of a file under an instance directory goes through |
| @@ -383,8 +439,8 @@ Belt and braces in the agent, cheap and testable: |
| (Linux 5.6+), so a symlink is refused with a named error even on a |
| box where the mount option is missing; the same helper is what the |
| agent uses to verify, at start, that both mounts carry `nosymfollow` |
| - (`/proc/self/mountinfo`), and it **refuses to wake any instance** until |
| - they do (§8.4); |
| + (`/proc/self/mountinfo`), and it **refuses to wake any nested host** |
| + until they do (§8.5); |
| - every agent write into an instance directory creates its temp file |
| with `O_CREAT|O_EXCL` and a random suffix, then `renameat`s over the |
| target (rename replaces the directory entry and never follows the |
| @@ -392,6 +448,27 @@ Belt and braces in the agent, cheap and testable: |
| - launch files (§6.2) are written the same way, and `instance-exec` |
| reads them through the same helper. |
| +### 3.3 Nothing in an instance directory is an instruction (codex R2-1) |
| + |
| +The agent reads three kinds of thing from an instance directory: the |
| +database (to replicate, restore and checkpoint), the two env files (to |
| +compose the instance's own environment), and nothing else. It never |
| +reads a path, a host name, a file list or a phase from there. In |
| +particular: |
| + |
| +- the migration journal lives in `/data/layout/<host>.json`, outside |
| + every tenant mount, and every source and destination path in it is |
| + *derived from the host name and the registry row* by the §3.1 helpers |
| + at plan time, never read back from a file (§8.2); |
| +- the launch file is agent-written and then read by the trampoline |
| + inside the namespace as the instance itself; it directs nothing the |
| + agent does (§6.2); |
| +- the exit record is root-written outside every tenant mount (§6.5). |
| + |
| +A test plants a forged journal-shaped file in an instance directory and |
| +a forged launch file naming a sibling, and asserts the sibling is |
| +untouched by a full migrate, revert, wake and stop cycle. |
| + |
| --- |
| ## 4. The units |
| @@ -400,9 +477,9 @@ Unit files under `internal/host/units/`, embedded and installed exactly as |
| the existing nineteen are (`bootstrap.go:33-34`, `systemapply.go:339-398`): |
| `carlos-instance@.service`, `carlos-instance-seed@.service`, |
| `carlos-instance-sidecar@.service`, `carlos.slice`, |
| -`carlos-instances.slice`, the two mount units of §3.2, and the two |
| -control units of §5.2. Nothing enables the templates; an instance unit |
| -exists only while the agent has started it. |
| +`carlos-instances.slice`, the two mount units of §3.2, the probe of |
| +§8.5, and the two control units of §5.2. Nothing enables the templates; |
| +an instance unit exists only while the agent has started it. |
| **Instance names are escaped hosts.** The unit instance is |
| `systemd-escape(host)`: for a valid host (`[a-z0-9.-]`) that only turns |
| @@ -423,12 +500,15 @@ Description=carlos exec-backed instance %I |
| # PartOf= would propagate the RESTART and race the agent's own startup |
| # (codex R1-7). |
| BindsTo=carlos-edge.service |
| -After=carlos-edge.service data-instances.mount run-carlos-instances.mount |
| Requires=data-instances.mount run-carlos-instances.mount |
| +After=carlos-edge.service data-instances.mount run-carlos-instances.mount |
| [Service] |
| Type=simple |
| ExecStart=/opt/carlos/carlos instance-exec run %I |
| +# Runs as ROOT, outside the namespace ("+"), after the last process of |
| +# this invocation is gone: writes the exit record the agent reads (§6.5). |
| +ExecStopPost=+/opt/carlos/carlos instance-exit run %I |
| User=carlos |
| Group=carlos |
| # One slice per host holds the instance, its seed and its sidecar, and |
| @@ -477,14 +557,11 @@ RestrictSUIDSGID=yes |
| MemoryDenyWriteExecute=yes |
| SystemCallArchitectures=native |
| -# --- per-service resources; the tier lives on the host slice (§4.3) --- |
| +# --- per-service resources; the tier and the family OOM rule live on the |
| +# host slice (§4.3) --- |
| TasksMax=256 |
| # The kernel picks a tenant before it picks anything else in carlos.slice. |
| OOMScoreAdjust=500 |
| -# One OOM kill takes every process of this service (memory.oom.group). |
| -# The sidecar is a sibling service; the activator co-stops it when the |
| -# instance exits, as it does today (activator.go:970-984). |
| -OOMPolicy=kill |
| ``` |
| Lines that are load-bearing and why: |
| @@ -509,17 +586,26 @@ Lines that are load-bearing and why: |
| for a same-uid ptrace to attach to. `PrivatePIDs=` needs systemd 257 |
| (Debian 13, which `infra/modules/region-host/variables.tf:134` pins) |
| and is **ignored, not refused**, on a kernel that cannot provide it, so |
| - §8.4 gates on observed behaviour, not on the directive (R1-15). |
| + §8.5 gates on observed behaviour, not on the directive (R1-15). |
| - **`RestrictNamespaces=yes`**: without it an unprivileged user namespace |
| could bind-mount over a fresh mountpoint and re-reveal what the tmpfs |
| hid. |
| - **`IPAddressDeny=`** for IMDS. The nft rule in pinfra allows uid |
| `carlos` because the agent needs IMDS; this line is what stops an |
| instance, also uid `carlos`, from fetching the box's role credentials. |
| + It is a BPF filter, so a denied connect fails with `EPERM`, which is |
| + how the probe tells enforcement from an unreachable network (§8.5). |
| - **`BindsTo=carlos-edge.service`** keeps today's "instances die with the |
| agent": when the edge stops, so do they, and nothing restarts them but |
| the agent's next wake. §6.7 adds the startup sweep for the case where |
| the coupling did not fire. |
| +- **`ExecStopPost=+`**: the `+` prefix runs the command with full |
| + privileges and *without* the unit's `User=` and file-system namespacing, |
| + so it can write under `/run/carlos/exits`, which no tenant mount |
| + contains. systemd runs it after the service's own processes have been |
| + signalled and reaped, with `$SERVICE_RESULT`, `$EXIT_CODE`, |
| + `$EXIT_STATUS` and `$INVOCATION_ID` in its environment. §6.5 says what |
| + the agent does with that and why it still checks the cgroup. |
| - **`StandardOutput=file:` to an `AF_UNIX` socket** is documented |
| systemd behaviour: a stream connection is made and used as stdout. It |
| is opened by PID 1 outside the namespace, so the path is the real one, |
| @@ -537,6 +623,7 @@ Differences: |
| |---|---|---| |
| | `Type=` | `oneshot` | `simple` | |
| | `ExecStart=` | `/opt/carlos/carlos instance-exec seed %I` | `/opt/carlos/carlos instance-exec sidecar %I` | |
| +| `ExecStopPost=` | `+/opt/carlos/carlos instance-exit seed %I` | `+/opt/carlos/carlos instance-exit sidecar %I` | |
| | `StandardOutput=` | `file:/run/carlos/logs/%I.seed.sock` | `file:/run/carlos/logs/%I.sidecar.sock` | |
| | timeouts | `TimeoutStartSec=120` (today's seed budget, `backend_exec.go:368-408`; a oneshot has no start timeout by default, R1-5), `TimeoutStopSec=20` | `TimeoutStopSec=20` | |
| @@ -545,7 +632,7 @@ hardening and coupling lines are byte-identical (§10). Three files rather |
| than one because seed and instance never run together but sidecar and |
| instance do, and a template's instance is the host, not a role. |
| -### 4.3 Slices and the memory walls (codex R1-8, R1-10) |
| +### 4.3 Slices and the memory walls (codex R1-8, R1-10, R2-11) |
| ```ini |
| # carlos.slice — the outer wall of the 2026-08-28 design, as a slice |
| @@ -576,13 +663,28 @@ reclaimable page cache, carries over without new numbers. |
| `OOMPolicy=continue` stays on the edge unit. |
| The per-host slice `carlos-instances-<esc>.slice` is created on demand |
| -by the first unit that names it. The control service sets |
| -`MemoryMax`/`MemoryHigh` on it, from the route's tier, before every |
| -start (§5.3, §6.6): `512M`/`460M` by default, `1G`/`920M` for an app whose |
| -manifest declares `capabilities.children`, exactly the values |
| -`internal/cgroup` writes today. Seed, instance and sidecar all charge |
| -that one cgroup, which is what today's shared `carlos-exec@<host>` gives |
| -them (`backend_exec.go:149-155, 373-375, 890-895`). |
| +by the first unit that names it. Before every start the control service |
| +sets three properties on it from the route's tier (§5.3, §6.6): |
| +`MemoryMax`, `MemoryHigh` (90% of max) and **`MemorySwapMax` equal to |
| +`MemoryMax`**, the last because with zram on the box a cgroup at |
| +`memory.max` otherwise spills into compressed swap and carries on, which |
| +`internal/cgroup/cgroup.go:313-318` learned live on the flagship |
| +(R2-11). The values are `512M`/`460M`/`512M` by default and |
| +`1G`/`920M`/`1G` for an app whose manifest declares |
| +`capabilities.children`, exactly what `internal/cgroup` writes today. |
| +Seed, instance and sidecar all charge that one cgroup, which is what |
| +today's shared `carlos-exec@<host>` gives them |
| +(`backend_exec.go:149-155, 373-375, 890-895`). |
| + |
| +**The family OOM rule** is `memory.oom.group=1` on the host slice's |
| +cgroup, written by the control service (root) directly to |
| +`/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/memory.oom.group` |
| +after `set-property`, because systemd has no directive for it on a |
| +slice. With it, an OOM kill anywhere in the family takes the instance and |
| +its sidecar together, which is what `internal/cgroup` gives them today |
| +and what a per-service `OOMPolicy=kill` would not (R1-8, R2 disposition). |
| +systemd does not manage that attribute on slices, so the write does not |
| +fight it; the plan's box task confirms it survives a `daemon-reload`. |
| The cgroup path is |
| `/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/` |
| @@ -629,7 +731,7 @@ ExecStart=/opt/carlos/carlos instance-control |
| # Root deliberately: the whole job is systemctl start/stop, and the |
| # caller is the unprivileged agent. Same rationale as carlos-stage.service. |
| ProtectSystem=strict |
| -ReadWritePaths=/run/carlos |
| +ReadWritePaths=/run/carlos /sys/fs/cgroup/carlos.slice |
| ProtectHome=yes |
| PrivateTmp=yes |
| PrivateDevices=yes |
| @@ -645,7 +747,9 @@ The socket is owned by `carlos` with mode 0600 and the unit is socket- |
| activated, so only the agent's uid can open it and it runs only while |
| there is work. It is not in any instance's namespace: `/run/carlos` |
| inside an instance is an empty tmpfs with the instance's own directory |
| -bound in. |
| +bound in. At start the service writes a fresh random token to |
| +`/run/carlos/instance-control.token` (`0600 carlos:carlos`, `O_EXCL` |
| +temp and rename) and runs the probe of §8.5. |
| ### 5.3 Protocol |
| @@ -658,33 +762,44 @@ service built itself: |
| | Verb | What it runs | Returns | |
| |---|---|---| |
| -| `POST /start?host=H&role=run\|sidecar&memory_max=N&memory_high=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=N`; `systemctl reset-failed <unit>`; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID`, or 502 with systemctl's stderr | |
| -| `POST /seed?host=H&memory_max=N&memory_high=N` | the same three, on `carlos-instance-seed@<esc>.service` (oneshot: `start` blocks until exit or `TimeoutStartSec`), then `systemctl show -p Result,ExecMainStatus,ExecMainCode,InvocationID` | 200 with `{result, status, code, invocation_id}`; a failed seed is a 200 whose result says so, not a 502 | |
| +| `POST /start?host=H&role=run\|sidecar&memory_max=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N`; write `memory.oom.group=1` on the slice cgroup; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID <unit>`, read immediately after the start returns; or 502 with systemctl's stderr | |
| +| `POST /seed?host=H&memory_max=N` | the same, on `carlos-instance-seed@<esc>.service` (oneshot: `start` blocks until exit or `TimeoutStartSec`) | 200 with `{invocation_id}`; the seed's result reaches the agent as an exit record (§6.5), not in this response | |
| | `POST /stop?host=H&role=run\|seed\|sidecar` | `systemctl stop <unit>` (blocking; also cancels a pending start job) | 204 | |
| -| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>` | 204 | |
| -| `GET /ready` | nothing | 204 once the service has verified both mounts carry `nosymfollow` and `PrivatePIDs=` is effective (§8.4); 503 with the reason otherwise | |
| +| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>`, tolerating "not loaded" | 204 | |
| +| `GET /ready` | nothing | 204 once the probe verdict (§8.5) is `ready`; 503 with the verdict otherwise | |
| + |
| +There is deliberately **no `reset-failed` inside `/start`**: `ResetFailedUnit` |
| +refuses a unit that is not loaded, and a template instance that has never |
| +started is not loaded, so the first wake of every host would fail (codex |
| +R2-3). `systemctl start` on a `failed` unit already clears the failed |
| +state; `/reset` exists for the sweeper's housekeeping only. |
| Validation, in order: token; `host` must satisfy `provision.ValidHostShape` |
| (`provision.go:262`); `role` must be one of the three literals; the |
| -memory values must parse as `Limit` values the agent already knows how to |
| +memory value must parse as a `Limit` the agent already knows how to |
| produce (`internal/cgroup`, `TierDefault`/`TierChildren`). Anything else |
| is a 400 with no systemctl call. The unit name is |
| `"carlos-instance" + suffix + "@" + escape(host) + ".service"` and the |
| slice name `"carlos-instances-" + escape(host) + ".slice"`, built from |
| the validated pieces, never from the request string. |
| -What this grants the agent: start, stop, reset and memory-tier control of |
| -units named `carlos-instance*@<host>` and their host slice. That is |
| -exactly what it has today over its own children. It grants nothing else, |
| -and a host that does not exist is a unit whose bind source does not |
| -exist, which fails at namespace setup and starts nothing. |
| +What this grants the agent: start, stop, reset and tier control of units |
| +named `carlos-instance*@<host>` and their host slice. That is exactly |
| +what it has today over its own children. It grants nothing else, and a |
| +host that does not exist is a unit whose bind source does not exist, |
| +which fails at namespace setup and starts nothing. |
| + |
| +`systemctl set-property --runtime` on a slice or on an instance of an |
| +installed template that has never been started loads the unit first and |
| +writes a runtime drop-in the following `start` honours |
| +(`SetUnitProperties` in `src/core/dbus-manager.c`; confirmed by codex |
| +R1-1). The plan's first task exercises it on a dev box for a slice that |
| +does not yet exist. |
| -`systemctl set-property --runtime` on an instance of an installed |
| -template that has never been started loads the unit first and writes a |
| -runtime drop-in the following `start` honours (`SetUnitProperties` in |
| -`src/core/dbus-manager.c`; confirmed by codex R1-1). The same holds for |
| -a slice that does not yet exist. The plan's first task exercises both on |
| -a dev box. |
| +Every call the agent makes to this service carries a deadline |
| +(`controlTimeout`, 30s; a blocking `start` that exceeds it is answered |
| +by a `stop` and reported as a failed wake), because §6.7 puts one such |
| +call on a path that must never stall the activator. |
| ### 5.4 The choice put to codex, and the ruling |
| @@ -720,24 +835,30 @@ installed and not relied on. |
| ## 6. The activator on units |
| -`internal/activator/activator.go` does not change. `doWake` |
| -(`activator.go:772`), `stop` (`:962-1047`), `Bounce` (`:946-956`), |
| -`sweepOnce` (`:1082`) and the `Proc`/`Rep` seams (`:194-207`) keep their |
| -contracts. In particular `sweepOnce` keeps calling `Exited()` under the |
| -activator mutex and `Exited()` stays a non-blocking read of a flag |
| -(R1-4); everything that talks to systemd runs in the proc's own |
| -goroutine. The changes are inside `ExecBackend`. |
| +`internal/activator/activator.go` changes in exactly one place, the |
| +sidecar restart in `sweepOnce` (§6.7, codex R2-6). `doWake` |
| +(`activator.go:772`), `stop` (`:962-1047`), `Bounce` (`:946-956`) and the |
| +`Proc`/`Rep` seams (`:194-207`) keep their contracts. `Exited()` stays a |
| +non-blocking read of a flag (R1-4); everything that talks to systemd |
| +runs in the proc's own goroutine or in the sidecar-start goroutine. The |
| +rest of the change is inside `ExecBackend`, which keeps its current |
| +`childProc` path for flat hosts for this one release (§8.1) and adds the |
| +unit path for nested hosts. |
| ### 6.1 Preparing a role, and `Start` |
| One helper, `prepare(rt, role)`, runs before **any** role starts, seed |
| -included (R1-5): |
| - |
| -1. Ensure `InstanceDir` and `InstanceRunDir` exist, 0700, and `/run/carlos/logs`. |
| -2. For `run`: `EnsureAdminToken(InstanceDir/<host>.env)` (unchanged), |
| - `instanceEnv` (unchanged, reading the two files from `InstanceDir` |
| - through `instancefs`), bind the control listener at |
| - `rt.Socket + ".control"` (unchanged, `edge.ListenControl`), and clear |
| +included (R1-5, R2-12): |
| + |
| +1. Ensure `InstanceDir` and `InstanceRunDir` exist (0700 and 2750 as in |
| + §3.1), and `/run/carlos/logs`. |
| +2. `EnsureAdminToken(InstanceDir/<host>.env)` for `run` **and** `seed` |
| + (a seed is the first thing a fresh instance runs and must see the |
| + token the instance will, `backend_exec.go:346-352`); `instanceEnv` |
| + for both, reading the two files from `InstanceDir` through |
| + `instancefs`, with the seed's removal of the control-channel variables |
| + as today. For `run` only: bind the control listener at |
| + `rt.Socket + ".control"` (unchanged, `edge.ListenControl`) and clear |
| any stale `app.sock` (nothing is running: the unit is not active, or |
| we would not be here). |
| 3. Bind the role's log listener at `LogSocket(host, role)` (§6.4). |
| @@ -745,9 +866,10 @@ included (R1-5): |
| `Start` is then: `prepare(rt, "run")`; `POST /start` with the route's |
| tier; `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` (unchanged); |
| -on failure `POST /stop`, wait for the proc's `done`, and clean up, |
| -exactly as the current failure path stops the child; return a `unitProc` |
| -carrying the invocation id. |
| +the `0660` normalisation of `app.sock` (unchanged); on failure |
| +`POST /stop`, wait for the proc's `done`, and clean up, exactly as the |
| +current failure path stops the child; return a `unitProc` carrying the |
| +invocation id. |
| ### 6.2 The trampoline and `launch.<role>.json` |
| @@ -798,16 +920,17 @@ visible to that instance and the agent, nobody else. An instance can |
| rewrite its own launch file while it runs; the next start reads it again |
| through `instancefs`, refuses any `argv[0]` but its own live link, and |
| executes only what a tenant could execute anyway: its own binary with an |
| -argv and env it chose. That is not a privilege. |
| +argv and env it chose. That is not a privilege, and the agent never |
| +reads the file (§3.3). |
| Seed: `launch.seed.json` carries the argv from `rt.Seed` |
| (`backend_exec.go:332-338`) with the DB path substituted as today; `POST /seed` |
| -blocks up to `TimeoutStartSec`; the result's `status` is the seed's exit |
| -code; a non-zero exit or a timeout runs the same cleanup as today |
| -(`backend_exec.go:438`) **after** `POST /stop?role=seed` has returned and |
| -the seed proc's `done` has closed, so the lease is never released |
| -(`activator.go:824-828`) while a seed can still write (R1-5). A |
| -cancelled `ctx` during a seed does the same stop-and-wait. |
| +blocks up to `TimeoutStartSec`; the seed's exit status arrives as its |
| +exit record (§6.5); a non-zero status, a timeout or a cancelled `ctx` |
| +runs `POST /stop?role=seed`, waits for the seed proc's `done`, and only |
| +then runs the same cleanup as today (`backend_exec.go:438`), so the |
| +lease is never released (`activator.go:824-828`) while a seed can still |
| +write (R1-5). |
| Sidecar: `launch.sidecar.json` carries `[live/<host>, "sidecar", "run"]` |
| and the sidecar env from `<host>-sidecar.env` only, as today |
| @@ -859,68 +982,107 @@ socket open, as it keeps the pipe open today. `KillMode=mixed` plus |
| when systemd reaps it. EOF on the log connection ends **log collection |
| for that invocation**; it does not mean the process is gone (R1-4). |
| -### 6.5 Exit observation (codex R1-4) |
| +### 6.5 Exit observation (codex R1-4, R2-4, R2-5) |
| `childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`), |
| -which is process death. A unit proc needs the same certainty, from |
| -systemd, attributed to the right invocation. Each `unitProc` runs one |
| -goroutine: |
| - |
| -1. **Fast signal**: inotify (`IN_MODIFY`, or `poll` for `POLLPRI`) on the |
| - unit's `cgroup.events` file under the host slice; `populated 0` means |
| - every process of the unit is gone. cgroup v2 files are world-readable, |
| - so the agent reads them as `carlos`, as the sampler does today. |
| -2. **Confirmation**: `systemctl show -p ActiveState,SubState,Result,ExecMainStatus,ExecMainCode,InvocationID <unit>` |
| - (read-only, unprivileged). The proc is exited when `ActiveState` is |
| - `inactive` or `failed` **and** the `InvocationID` matches the one the |
| - start returned (a newer invocation is a different proc). A fallback |
| - poll of the same `show` every 2s covers a missed inotify event. |
| -3. On confirmation: emit the member-facing event line with today's |
| - wording (`instance exited: ...` with the status or signal from |
| - `ExecMainStatus`/`ExecMainCode`, or `instance exited (exit 0)`), |
| - set `exited`, close `done`. |
| - |
| -The activator's "an exited line with no stopping line before it is a |
| -crash" reading is unchanged, and `sweepOnce`'s existing dead-child |
| -check (`activator.go:1105-1122`) sees a flag, never a blocking call. |
| +which is process death. A unit proc needs the same certainty, attributed |
| +to the right invocation, and it must not depend on systemd keeping an |
| +inactive unit loaded: systemd garbage-collects an unreferenced inactive |
| +unit and its `ExecMainStatus` with it, so a `show` after a quick clean |
| +exit can find nothing to show (R2-4). So the record is written by the |
| +unit itself, on the way out: |
| + |
| +**`carlos instance-exit <role> <host>`** runs as `ExecStopPost=+` (root, |
| +outside the namespace, §4.1). It writes |
| +`/run/carlos/exits/<host>.<role>.<INVOCATION_ID>.json` with `O_EXCL` |
| +(`0640 root:carlos`), containing `{invocation, result, code, status, at}` |
| +from `$INVOCATION_ID`, `$SERVICE_RESULT`, `$EXIT_CODE`, `$EXIT_STATUS`. |
| +It runs whether the service exited, was stopped, timed out or was |
| +OOM-killed, and it runs after the invocation's processes have been |
| +signalled and reaped by systemd's stop sequence. The directory is |
| +root-owned and outside every tenant mount, so nothing but this command |
| +writes there. |
| + |
| +Each `unitProc` runs one goroutine that watches that directory with |
| +inotify (`IN_CLOSE_WRITE`/`IN_MOVED_TO` on the file for its invocation) |
| +and, as fallback, polls every 2s. A record alone is not termination: a |
| +service enters `failed` after its final `SIGKILL` timeout **with processes |
| +still present** if one is unkillable (`src/core/service.c:4020-4022`), |
| +and the lease must not be released over a writer (R2-5). So the |
| +goroutine, on seeing the record (or on the fallback finding the unit |
| +`inactive`/`failed` with an empty or absent `InvocationID`, which is a |
| +unit already unloaded), then requires **cgroup emptiness**: the unit's |
| +cgroup directory under the host slice is absent, or its `cgroup.events` |
| +says `populated 0`. Only then does it emit the member-facing event line |
| +with today's wording (`instance exited: ...` with the status or signal |
| +from the record, or `instance exited (exit 0)`; `instance exited (status |
| +unknown: no exit record)` in the unloaded-without-record case), set |
| +`exited`, and close `done`. A populated cgroup after a record is logged |
| +every 2s as `instance <host>: processes remain after SIGKILL; not |
| +releasing`, and `done` stays open; that is a box-level incident, not a |
| +lease release. |
| + |
| +A record whose invocation is not this proc's (someone restarted the |
| +unit outside the activator, which `Restart=no` does not forbid) is |
| +logged as `unit restarted outside the activator`, the proc's own |
| +invocation is treated as gone (terminal, status unknown), and the |
| +sweeper's stand-down then `POST /stop`s whatever is running; the agent |
| +owns these units and any other invocation is an anomaly to end, not to |
| +adopt. |
| + |
| +The agent removes a record once it has consumed it; on start it removes |
| +every record older than its own start, since no proc can be waiting on |
| +them. `sweepOnce`'s existing dead-child check (`activator.go:1105-1122`) |
| +sees a flag, never a blocking call. |
| `Restart=no` in the template is what makes this sound: systemd never |
| restarts the unit behind the activator's back; a failed unit stays |
| -`failed` until the agent's next `start`, which `reset-failed`s first. |
| +`failed` until the agent's next `start`, which clears it. |
| -### 6.6 cgroups, tiers and the sampler (codex R1-8, R1-9) |
| +### 6.6 cgroups, tiers and the sampler (codex R1-8, R1-9, R2-13) |
| `internal/cgroup` today: `Manager.Start` (`cgroup.go:242-285`) clones the |
| child into `carlos-exec@<host>` under the edge unit's delegated subtree, |
| -writes `memory.max`/`memory.high` from the tier and `memory.oom.group=1`, |
| -and the sampler reads the subtree as a scan root (exec-child accounting |
| -design §4). `CleanupHost` (`:398-403`) removes the directory. |
| +writes `memory.max`/`memory.high`/`memory.swap.max` from the tier and |
| +`memory.oom.group=1`, and the sampler reads the subtree as a scan root |
| +(exec-child accounting design §4). `CleanupHost` (`:398-403`) removes the |
| +directory. |
| Under units, systemd owns the cgroup. So: |
| -- **Tier** travels in `POST /start` / `POST /seed` and lands as |
| - `MemoryMax`/`MemoryHigh` on the host slice via `set-property --runtime`. |
| - The `Tier func(registry.Route) cgroup.Limit` seam and its box wiring |
| - (`stagedManifestOfLiveVersion`) are unchanged; only the consumer |
| +- **Tier** travels in `POST /start` / `POST /seed` and lands on the host |
| + slice as the three properties of §4.3 plus the `memory.oom.group` |
| + write. The `Tier func(registry.Route) cgroup.Limit` seam and its box |
| + wiring (`stagedManifestOfLiveVersion`) are unchanged; only the consumer |
| changes. The agent logs `instance: <host>: memory.max=…` on every start |
| as it does today. Seed, instance and sidecar share the one cap. |
| -- **Sampler** gains |
| +- **Per-host sampling** gains |
| `/sys/fs/cgroup/carlos.slice/carlos-instances.slice/` as a scan root |
| and reads each `carlos-instances-<esc>.slice/memory.current` (the |
| slice's charge is the family's total, so no per-role summation is |
| needed). It **publishes under the existing key** |
| `activator.ExecName(host)` (`carlos-exec@<host>`), because that is the |
| key `internal/console/metricsview.go:728-762` recognises and everything |
| - else is discarded at the app boundary there. The discovery code is |
| - `internal/host/metrics_probes.go`; both files are consumers (§9). The |
| - console therefore shows the same per-host numbers with no change. |
| + else is discarded at the app boundary there. Flat hosts, for this one |
| + release, are still sampled from the delegated subtree as today; the |
| + two roots never both hold a given host. |
| +- **The edge's own reading.** Platform-unit discovery scans |
| + `/sys/fs/cgroup/system.slice` (`internal/host/metrics_probes.go:49`), |
| + where `carlos-edge.service` no longer lives once it moves into |
| + `carlos.slice`, and the accounting design defines the edge's reading |
| + as the hierarchical total including its children. So discovery adds |
| + `/sys/fs/cgroup/carlos.slice` as a root for the edge unit, and the |
| + value published under `carlos-edge.service` becomes **`carlos.slice`'s |
| + `memory.current`**: the agent plus every instance, which is what the |
| + key has always meant (R2-13). The edge unit's own leaf is published |
| + additionally under `carlos-edge.service/self` for the box view; the |
| + platform graph is continuous across the roll. |
| - **`Manager.Start`, `CleanupHost`, the clone3 path and the delegation |
| - dance** are no longer used by the exec backend. They stay in the tree |
| - until a release has run without them; deleting the package's spawn |
| - half is a follow-up. **`Delegate=` on the edge unit** stays for the |
| - same release, then goes. |
| + dance** stay in use for flat hosts during this release and are deleted |
| + with the flat path in the next (§8.6). **`Delegate=` on the edge |
| + unit** goes with them. |
| -### 6.7 Agent start and restart (codex R1-7) |
| +### 6.7 Agent start and restart; the sidecar restart (codex R1-7, R2-6) |
| Today an agent restart kills every child (`Pdeathsig`, the cgroup tree |
| dies with the unit). `BindsTo=carlos-edge.service` reproduces the stop |
| @@ -933,13 +1095,30 @@ start, before anything else: |
| 2. `POST /stop` for every unit that is not `inactive`, whatever its |
| role, and for every unit with a pending or running job (`stop` cancels |
| a queued start); |
| -3. wait for each to reach `inactive`/`failed`. |
| +3. wait for each to reach `inactive`/`failed` **and** for its cgroup to |
| + be empty (§6.5's rule), then remove stale exit records. |
| Every instance is therefore asleep, and no job is queued, when the |
| -migration (§8) and then the activator's state machine begin, which is |
| -the invariant the activator holds today. Adopting a running unit instead |
| -(re-acquiring its lease, re-attaching replication) is a later |
| -improvement with its own review; it is not needed for correctness. |
| +activator's state machine begins, which is the invariant the activator |
| +holds today. Adopting a running unit instead (re-acquiring its lease, |
| +re-attaching replication) is a later improvement with its own review; it |
| +is not needed for correctness. |
| + |
| +**The one change in `activator.go`.** `sweepOnce` restarts a dead |
| +sidecar by calling `StartSidecar` under `a.mu` |
| +(`activator.go:1165-1200`), justified there by "an exec is quick". A |
| +unit start is an HTTP call and a `systemctl start`, and a stall there |
| +would hold every other host's wake and stop. So the sidecar restart |
| +becomes: under `a.mu`, mark `e.sidecarStarting = true`, capture the |
| +entry's wake generation, and schedule the backoff exactly as now; then |
| +in a goroutine, `StartSidecar` with `controlTimeout`; then re-take |
| +`a.mu`, and if the entry is still awake with the same generation, |
| +install the proc as today, otherwise `Stop` it. The lock still makes a |
| +restart impossible to race with a stop, because the install step checks |
| +the state under the same lock the stop flips it under, and a stop that |
| +arrives during the start finds `sidecarStarting` and waits for the |
| +install step to see it. One test drives a stalled sidecar start while |
| +another host wakes and stops. |
| ### 6.8 What does not change |
| @@ -949,7 +1128,7 @@ meta dir is derived from `dbPath`, `lease.go:286`, so it lands inside the |
| instance directory), `NextDue`, the schedule runner's token read, the |
| control socket's purge and schedule mounts, `Bounce`, the sweeper's |
| hibernation policy, residence moves (`movespass.go`, once it reads the |
| -DB path from the row and honours the migration fence, §8.1 and §9), |
| +DB path from the row and honours the layout journal, §8.2 and §9), |
| `carlos restart` in all three scopes, `X-Carlos-Version`, the caps pass, |
| config-ack. All of these see a path change and nothing else. |
| @@ -957,176 +1136,257 @@ config-ack. All of these see a path change and nothing else. |
| ## 7. Config delivery |
| -`SyncExec` (`internal/appconfig/syncexec.go`) writes an exec route's |
| -bundle to `InstanceDir/<host>.config.env` instead of |
| +`SyncExec` (`internal/appconfig/syncexec.go`) writes a nested exec |
| +route's bundle to `InstanceDir/<host>.config.env` instead of |
| `<DataDir>/<host>.config.env`, through the `O_EXCL`-temp-and-rename |
| helper of §3.2 rather than `os.WriteFile` on a predictable `.tmp` name |
| -(`syncexec.go:921-926`; R1-3). Its ownership rule ("sync owns |
| -`*.config.env` outright, never touches `<host>.env` or |
| -`<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged and now |
| -trivially true per directory. `reapConfigEnv` (`:977-1024`) walks |
| -`<DataDir>/instances/*/` for exec routes as well as the flat `DataDir` |
| -for addr routes, and the bounce-on-change path is unchanged (the agent |
| -bounces the entry; the next wake reads the new file). |
| +(`syncexec.go:921-926`; R1-3), and a flat route's exactly as today. Its |
| +ownership rule ("sync owns `*.config.env` outright, never touches |
| +`<host>.env` or `<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged |
| +and, for a nested host, trivially true per directory. `reapConfigEnv` |
| +(`:977-1024`) walks `<DataDir>/instances/*/` for nested routes as well as |
| +the flat `DataDir` for flat and addr routes, and the bounce-on-change |
| +path is unchanged (the agent bounces the entry; the next wake reads the |
| +new file). |
| **Opted-in TCP-addr routes stay flat.** Their file is read by the app's |
| own supervisor at `/data/<host>.config.env` (titogo's |
| `EnvironmentFile=-/data/%i.config.env`), and moving it would break every |
| -such drop-in. Exec instances cannot see it any more, which is the read |
| -#127 opens with. #127's request, root-rendered delivery into |
| +such drop-in. Isolated instances cannot see it any more, which is the |
| +read #127 opens with. #127's request, root-rendered delivery into |
| `/etc/carlos/apps` for addr routes so that the file is not readable by |
| the agent either, is a separate, smaller change and stays open (§11). |
| -`reapConfigEnv`'s host-`x.config` ambiguity guard stays for the flat |
| -directory, where it still applies. |
| +`reapConfigEnv`'s host-`x.config` ambiguity guard and `SyncExec`'s |
| +collision refusal (`syncexec.go:389-395`) stay for the flat directory, |
| +where they still apply, and the migration's ownership preflight (§8.2) |
| +is derived from the same rule. |
| --- |
| -## 8. Migration and rollout |
| - |
| -### 8.1 What moves, and when (codex R1-11, R1-12) |
| - |
| -**Eligibility is the activator's own predicate.** A route is migrated if |
| -and only if the activator would manage it: `Kind == "instance"`, |
| -`Hibernate`, `Backing == ""`, `Tunnel == ""`, `Socket != ""` |
| -(`activator.go:1379-1384`, `edge.go:630-638`). An unflagged instance |
| -route with a socket is an externally supervised process (titogo's router |
| -family is the live example); its files and rows are **not touched**, it |
| -is not isolated by this change, and `carlos status` names it as such |
| -(§8.5). Making that shape isolatable is a separate discussion. |
| - |
| -On agent start, after the unit sweep (§6.7) and before the activator |
| -starts, for every eligible route, in one pass: |
| - |
| -1. If `rt.DBPath` and `rt.Socket` are the flat defaults |
| - (`<DataDir>/<host>.db`, `<SocketDir>/<host>.sock`), proceed; |
| - otherwise §8.2. |
| -2. `mkdir -p InstanceDir(host)` 0700. If the directory already holds a |
| - `<host>.db` **and** a flat `<host>.db` also exists, stop: log |
| - `migrate: <host>: two databases (flat and nested); refusing to choose`, |
| - leave everything, refuse the wake (§8.2). Never pick one silently. |
| -3. Write `.migrating` in the directory (contents: the flat and nested |
| - paths and a timestamp). **This is the fence**: `Start`, `Seed`, |
| - `Restore`, the residence-move pass (`movespass.go`), `SyncExec`'s |
| - writer, and the litestream generator all refuse to act on a host whose |
| - directory carries it, with the marker's contents as the reason. |
| -4. `os.Rename` each of `<host>.db`, `<host>.db-wal`, `<host>.db-shm`, |
| - `.activator-<host>.db-litestream`, `<host>.env`, `<host>-sidecar.env`, |
| - `<host>.config.env`, `<host>.db.corrupt` that exists, into the |
| - directory. Same filesystem, same owner, atomic per file, no root. |
| -5. `registry.SetInstancePaths(host, fromDB, fromSocket, toDB, toSocket)`: |
| - one transaction, compare-and-swap on both columns, a new sibling of |
| - `SetDBPath` (`registry.go:1057-1075`). |
| -6. Remove `.migrating`. Log `migrate: <host>: flat → instances/`. |
| - |
| -Recovery is the marker: a later start that finds `.migrating` re-runs |
| -steps 4 to 6 from the paths it names (a rename whose source is gone and |
| -whose destination exists is complete; one whose source is gone and whose |
| -destination is missing is an error that keeps the fence up). A CAS |
| -conflict at step 5 (the row changed under us) keeps the fence up, logs |
| -both values, and is retried on the next start; a host under a fence |
| -never wakes, never moves, never syncs, so a partial state cannot be |
| -acted on by a destructive consumer. The `MoveCutting` path in |
| -`movespass.go:285-305`, which today would treat an absent DB as |
| -"nothing to checkpoint" and complete the flip, is the case the fence |
| -exists for. |
| - |
| -The edge picks the new `Socket` up on its next registry poll |
| -(`edge.go:449-451`, ~2s); nothing is awake to be affected. |
| - |
| -### 8.2 Rows that are not eligible for the automatic move |
| - |
| -A row whose `DBPath` or `Socket` is not the flat default was set by hand |
| -(`carlos add --db`). The migration does not guess. It logs |
| -`migrate: <host>: db_path/socket not at the flat default; run carlos route --instance-dir <host>`, |
| -`Start` refuses the wake with the same sentence as its event line (so a |
| -member sees it in `carlos logs` and in the instance's status, §8.5), and |
| -`carlos route --instance-dir <host>` (a new flag on the existing verb) |
| -performs steps 2 to 6 for that one host from the paths the row actually |
| -names, as long as they are under `DataDir` and `SocketDir`. The pre-roll |
| -check (§8.4) lists these rows before any box runs the new build, so on a |
| -fleet whose rows are all at the default (which is every console- |
| -provisioned row, `reconcile.go:300-301`) this branch never fires. |
| - |
| -### 8.3 Rollback (codex R1-13) |
| - |
| -The previous binary honours `rt.DBPath` and `rt.Socket` from the row |
| -(`backend_exec.go:220-225, 557-562`), so on a migrated box it would open |
| -the nested database, try to bind a socket in a run directory it never |
| -creates, and read admin and config files at the flat paths. That is not |
| -a working rollback, and it is not the "fresh restore" the draft claimed. |
| - |
| -Rollback is therefore **an explicit reverse migration, run by the new |
| -binary before the old one starts**, as a leg of `carlos system rollback` |
| -(the system-update design's rollback path) and as a standalone verb for |
| -the SSM road: `carlos instances-layout revert`. It stops every instance |
| -unit, and for every migrated row reverses steps 3 to 6 under the same |
| -fence: files back to flat, columns CAS-updated back, marker removed. The |
| -nested copy is the only history (nothing ran flat in between), so |
| -nothing is discarded. If someone rolls back without it, the failure is |
| -loud and lossless: instances fail to bind and the journal names the |
| -cause; the fix is to re-install the new binary or run the revert from |
| -it. The layout is never "two histories": a flat database appearing |
| -beside a nested one is a refusal (§8.1 step 2), not a choice. |
| - |
| -### 8.4 Gate: no isolation, no tenants (codex R1-15) |
| - |
| -A box that cannot establish the namespace **does not start tenants**. |
| -There is no fallback unit, no "proceed without `PrivatePIDs=`". The |
| -draft's contingency is withdrawn: running the fleet with the same-uid |
| -`/proc/<pid>/root` traversal open is the defect, not a degraded mode. |
| - |
| -The gate is observed behaviour, not a version string. At start the |
| -control service (root) runs a probe unit, |
| -`carlos-instance-probe.service`, a oneshot with the exact namespace |
| -block of §4.1 whose `ExecStart` is `carlos instance-probe`: it asserts |
| -from inside that `/data` and `/run/carlos` list nothing but the probe's |
| -own bound directories, that `/proc` lists only itself (pid 1 is the |
| -probe), that a connect to `169.254.169.254:80` fails, and that a |
| -symlink it creates in its directory is refused with `ELOOP`. The service |
| -records the verdict; `GET /ready` returns it; the agent refuses every |
| -wake with the verdict's reason until it is 204. The pre-roll check reads |
| -the same verdict through `carlos status` on the first box before the |
| -fleet follows. |
| - |
| -### 8.5 Rollout order |
| - |
| -1. **Pre-roll check**, from a laptop, per fleet, through the console: |
| - `carlos instances --layout` lists every exec route's layout state as |
| - the box reports it in its status snapshot (§9: `internal/status`), |
| - one of `flat` (not yet migrated), `nested`, `ineligible` (hand-set |
| - paths, §8.2), `external` (unflagged, §8.1) or `fenced` (marker |
| - present, with the reason). Fix `ineligible` rows with |
| - `carlos route --db` before the roll, or accept that they refuse to |
| - wake until `carlos route --instance-dir` runs. The same field is what |
| - a member sees on their instance page and in `carlos status`, which is |
| - the member-visible proof that an instance is isolated (R1-18). |
| -2. **Dev fleet first** (`kass`, two boxes), boxes before console. After |
| - the roll: the probe verdict is `ready`; `systemctl list-units |
| - 'carlos-instance*'` shows units coming and going with requests; |
| - `carlos instances --layout` says `nested` for every row. The test that |
| - matters is from inside: the plan's verification task provisions a |
| - throwaway instance whose seed lists `/data` and `/run/carlos`, reads |
| - `/proc/1/status`, curls IMDS, and plants and follows a symlink, and |
| - reads the results back through `carlos logs`. |
| -3. **Flagship**, region by region, console box last. The AU box has failed |
| - every gate since 2026-09-07 for an unrelated reason; do not read its |
| - failure as this change's. |
| +## 8. Two layouts, one release, and the verb that moves between them |
| + |
| +### 8.1 Why nothing moves on upgrade (codex R2-9) |
| + |
| +`carlos system apply` is run by the **installed** binary: it stages the |
| +candidate, restarts the edge, gates it, and on a failed gate restores the |
| +previous files and restarts the edge again (`systemapply.go:360-372, |
| +738-748`); `carlos-system-recover.service` runs `carlos.prev` when a |
| +journal exists (`units/carlos-system-recover.service:17`); and a channel |
| +rollback from the console (`system.go:466`) is an ordinary downgrade |
| +install. In every one of those paths the code that runs after the |
| +downgrade is the *old* binary, which knows one layout. A migration that |
| +ran on the new agent's first start would therefore leave every failed |
| +first roll on a layout the recovering binary cannot serve. So: |
| + |
| +**This release changes nothing about a host on upgrade.** Its exec |
| +backend classifies each activator-managed route on every `Start`: |
| + |
| +| Row | Classification | Spawn path | |
| +|---|---|---| |
| +| `DBPath == FlatDBPath`, `Socket == FlatSocket` | `flat` | today's `childProc`, unchanged, unisolated | |
| +| `DBPath == DefaultDBPath` (nested), `Socket == DefaultSocket` (nested) | `nested` | `unitProc` (§6) | |
| +| a journal exists in `/data/layout/<host>.json` | `fenced` | refuses to wake; reason from the journal | |
| +| anything else | `custom` | today's `childProc`; reported for the operator (§8.3) | |
| + |
| +A fleet rolled to this release behaves byte-for-byte as before until an |
| +operator runs the verb. A rollback before the verb is a rollback of |
| +nothing. The classification is published per route in the status |
| +snapshot (§8.4) and is what every other consumer keys on. |
| + |
| +### 8.2 `carlos system layout migrate` and `revert` (codex R1-11, R1-12, R2-1, R2-7, R2-8) |
| + |
| +The verb has the shape of `carlos restart` (#113): the console writes |
| +`control/layout.json` (`{id, requested_at, direction: nested|flat, scope}`, |
| +an append log like `restart-scope.json`), bumps the serial, and every box |
| +in scope acts on the serial move; boxes remember the last id they acted |
| +on. An old box never reads the object; an old console answers 404. Scope |
| +is the fleet, or `--box <label>` for one box, which is how a fleet |
| +migrates one box at a time with the console showing each box's rows |
| +turning `nested`. |
| + |
| +On a box, the layout pass runs in the agent (uid `carlos`, no root |
| +needed) and, per host in scope, in this order: |
| + |
| +1. **Eligibility is the activator's own predicate.** `Kind == "instance"`, |
| + `Hibernate`, `Backing == ""`, `Tunnel == ""`, `Socket != ""` |
| + (`activator.go:1379-1384`, `edge.go:630-638`). An unflagged instance |
| + route with a socket is an externally supervised process (titogo's |
| + router family is the live example); its files and rows are **not |
| + touched**, it is not isolated by this change, and status reports it |
| + `external`. Only a `flat` row migrates; only a `nested` row reverts; |
| + `custom` and `fenced` rows are reported and skipped. |
| +2. **Quiesce.** The pass asks the activator to stop the host |
| + (`Bounce`-class stop with replication closed and the lease released, |
| + the same sequence `stop()` runs) and to refuse wakes for it until the |
| + pass is done. Nothing is writing. |
| +3. **Plan, in memory, from the row and the host name only.** Sources and |
| + destinations for `<host>.db`, `-wal`, `-shm`, the litestream shadow |
| + dir, `<host>.env`, `<host>-sidecar.env`, `<host>.config.env`, |
| + `<host>.db.corrupt`, computed by the §3.1 helpers. Nothing is read |
| + from the instance directory to build the plan (§3.3). |
| +4. **Ownership preflight** (R2-8). For every flat source name, compute |
| + every route on the box that could own that name under the flat |
| + rules: `<host>.config.env` is `host`'s sync-owned bundle **unless** a |
| + route `<host>.config` exists, in which case it is that route's |
| + instance env and is not moved with `host` (this is `SyncExec`'s own |
| + collision rule, `syncexec.go:389-395`). Reverting into a flat |
| + directory applies the same table in reverse: a name that would land |
| + on another route's file is a refusal, never an overwrite. A plan with |
| + any ambiguous name is refused for that host with the two owners |
| + named. |
| +5. **Destination check.** A destination that already exists is a |
| + refusal (`two databases (flat and nested); refusing to choose`), |
| + never a choice. Recorded in status; an operator resolves it on the |
| + box, which is the one box-local step this design keeps, for a state |
| + it never creates itself. |
| +6. **Journal.** Write `/data/layout/<host>.json` (`O_EXCL`), outside |
| + every tenant mount: `{host, direction, from: {db, socket}, to: {db, |
| + socket}, artifacts: [{name, present}], phase: "files"}`. Presence is |
| + recorded so recovery knows an absent optional artifact (a sidecar |
| + env, a quarantine) was absent, not lost (R2-7). From this moment the |
| + host is `fenced`: `Start`, `Seed`, `Restore`, the residence-move pass, |
| + `SyncExec`'s writer and the litestream generator all refuse it, with |
| + the journal's contents as the reason. The `MoveCutting` path in |
| + `movespass.go:285-305`, which would otherwise treat an absent DB as |
| + "nothing to checkpoint" and complete a flip, is the case the fence |
| + exists for (R1-12). |
| +7. **Files.** `os.Rename` each artifact recorded present. Same |
| + filesystem, same owner, atomic per file, no root. Then journal |
| + `phase: "row"`. |
| +8. **Row.** `registry.SetInstancePaths(host, from.db, from.socket, |
| + to.db, to.socket)`: one transaction, compare-and-swap on both columns, |
| + a new sibling of `SetDBPath` (`registry.go:1057-1075`). Then journal |
| + `phase: "done"`. |
| +9. Remove the journal; log `layout: <host>: flat → nested` (or the |
| + reverse); release the host to the activator. The edge picks the new |
| + `Socket` up on its next registry poll (`edge.go:449-451`, ~2s). |
| + |
| +**Recovery is the journal, and it is exact** (R2-7). On agent start and |
| +at the top of every layout pass, for each journal: |
| + |
| +- `phase: files`: for each artifact recorded present: at destination → |
| + done; at source → rename; at neither → error, fence stays, status |
| + says which file. Artifacts recorded absent are skipped. Then continue |
| + from step 7's end. |
| +- `phase: row`: if the row already equals `to` → the CAS committed |
| + before the crash; continue. If it equals `from` → CAS now. Anything |
| + else → error, fence stays, both values in status. |
| +- `phase: done`: remove the journal. |
| + |
| +A second clean agent start finds `nested` rows and no journals and does |
| +nothing (the classification table runs before any legacy-default test). |
| +A forged journal-shaped file inside an instance directory is not a |
| +journal: the pass reads `/data/layout` only. |
| + |
| +**Revert** is the same pass with `direction: flat`, the same journal, |
| +the same fence, the same preflight in reverse. It exists so that a fleet |
| +can be taken back to a layout the previous release serves **before** |
| +that release is installed: the operator's rollback order is `layout |
| +revert` (this release, verified `flat` in status) then `system rollback`. |
| +A `system rollback` or a failed-gate rollback that happens while hosts |
| +are `nested` lands the old binary on rows it honours (`rt.DBPath`, |
| +`rt.Socket`; `backend_exec.go:220-225, 557-562`) but on a run directory |
| +it never creates, so those hosts fail to bind their socket, loudly and |
| +losslessly, until the new binary is back or the revert has run from it. |
| +No data is ever in two places: the fence and the destination check see |
| +to that. |
| + |
| +### 8.3 `custom` rows |
| + |
| +A row whose `DBPath` or `Socket` is neither the flat nor the nested |
| +default was set by hand (`carlos add --db`; note `carlos route` has no |
| +`--db` flag, `cli.go:469-475`, so it was set at add time). The pass does |
| +not guess: the row is reported `custom` in status with both values, and |
| +the host keeps running on today's flat path. `carlos add` on the box, |
| +re-adding the route with the flat defaults, is the repair, and it is |
| +box-local because the row was made box-locally; a console-provisioned |
| +row (`reconcile.go:300-301`) is never `custom`. |
| + |
| +### 8.4 Status, through the API (codex R1-18) |
| + |
| +`internal/status/snapshot.go` gains a per-route `layout` field with one |
| +of `flat`, `nested`, `fenced` (with the journal's phase and reason), |
| +`custom` (with both values), `external`. `carlos status`, `carlos |
| +instances --layout` and the console's instance page render it; a box |
| +that predates the field shows `unknown (box predates layout reporting)`. |
| +It is the pre-roll inventory (every row `flat` or `custom` before the |
| +verb), the progress view during a migration, the member-visible proof |
| +that an instance is isolated (`nested`), and the reason a `fenced` host |
| +does not wake. `docs/site/cli.md` names the new verbs and flags (the |
| +docsite gate fails otherwise). |
| + |
| +### 8.5 Gate: no isolation, no nested tenants (codex R1-15, R2-14) |
| + |
| +A box that cannot establish the namespace **does not start nested |
| +tenants**. There is no fallback unit, no "proceed without |
| +`PrivatePIDs=`". Running the fleet with the same-uid `/proc/<pid>/root` |
| +traversal open is the defect, not a degraded mode. |
| + |
| +The gate is observed behaviour, not a version string. At start, and on |
| +`daemon-reload`, the control service (root) runs |
| +`carlos-instance-probe.service`, a oneshot with the exact namespace and |
| +`IPAddressDeny=` block of §4.1 whose `ExecStart` is `carlos |
| +instance-probe`. From inside it asserts, and reports each as a separate |
| +verdict line: |
| + |
| +- `/data` and `/run/carlos` list nothing but the probe's own bound |
| + directories; |
| +- `/proc` lists only itself, and its own pid is 1; |
| +- `connect()` to `169.254.169.254:80` **and** to `[fd00:ec2::254]:80` |
| + fails with **`EPERM`** (the BPF filter's answer), while `connect()` to |
| + `127.0.0.1:1` fails with `ECONNREFUSED` (the network is live and the |
| + filter, not the network, said no). Any other errno on the metadata |
| + addresses, including a timeout, is `not enforced` (R2-14); |
| +- a symlink it creates in its directory is refused with `ELOOP`; |
| +- `/proc/self/mountinfo` shows `nosymfollow` on both bound directories. |
| + |
| +The service records the verdict; `GET /ready` returns it; the agent |
| +refuses every nested wake with the verdict's failing line until it is |
| +204. The probe cannot pass while an instance would not be isolated, |
| +because it runs under the same unit text (a test asserts the probe's |
| +namespace block equals the templates'), and it fails only on a line |
| +that names the property that is missing. |
| + |
| +### 8.6 Rollout order |
| + |
| +1. **Roll this release** to the dev fleet (`kass`, two boxes), boxes |
| + before console, with `update-platform-binary.sh` enabling the new |
| + units by name before it restarts the edge (§9). Every row stays |
| + `flat`; the probe verdict is `ready`; nothing else changes. Cold-boot |
| + one dev box and confirm both mounts, the control socket and the probe |
| + come up (R2-2). |
| +2. **`carlos system layout migrate --box <dev box>`**, watch the rows |
| + turn `nested`, then the second box. The test that matters is from |
| + inside: the plan's verification task provisions a throwaway instance |
| + whose seed lists `/data` and `/run/carlos`, reads `/proc/1/status`, |
| + connects to both metadata addresses and to loopback, and plants and |
| + follows a symlink, and reads the results back through `carlos logs`. |
| + Run `layout revert` on one box and `migrate` again: both directions |
| + are exercised before the flagship sees either. |
| +3. **Flagship**: roll the release region by region, console box last; |
| + then `layout migrate` one box at a time. The AU box has failed every |
| + gate since 2026-09-07 for an unrelated reason; do not read its failure |
| + as this change's. |
| 4. **carlos-tito-ie**: Tito's operator rolls it from Tito's console once |
| - the flagship has run a day. The seven flagged instances migrate on the |
| - first agent start; the router family (unflagged) is reported |
| - `external` and left alone. Reply on #126 with the build and the check |
| - list above. |
| - |
| -**The roll script.** `update-platform-binary.sh` (pinfra) enables units |
| -by name; it gains `carlos-instance-control.socket`, both mount units and |
| -the probe, before it restarts the edge, and `bootstrap.go`'s |
| -`systemctlPlan` (`:165-179`) starts `.socket` and `.mount` units it |
| -enables, which it does not today (R1-14). `ensureSystemUnits` |
| -(`systemunits.go:34-83`) checks the control socket is *active*, not just |
| -enabled. An agent that finds the channel absent refuses wakes with |
| -`instance control socket not listening`, which is what a forgotten |
| -enable looks like from `carlos logs`. |
| + the flagship has run a day, then migrates. The seven flagged instances |
| + move; the router family (unflagged) is reported `external` and left |
| + alone. Reply on #126 with the build and the check list above. |
| +5. **The next release** removes the flat spawn path, `FlatDBPath`, |
| + `FlatSocket`, the `custom`/`flat` classifications, the spawn half of |
| + `internal/cgroup` and `Delegate=` on the edge, and refuses to start on |
| + a box with a `flat` row. |
| + |
| +**The roll script and the planner.** `update-platform-binary.sh:52-70` |
| +(pinfra) enables units by name; it gains `carlos-instance-control.socket`, |
| +both mount units and the probe, before it restarts the edge, and |
| +`bootstrap.go`'s `systemctlPlan` (`:165-179`) starts `.socket` and |
| +`.mount` units it enables, which it does not today (R1-14). |
| +`ensureSystemUnits` (`systemunits.go:34-83`) checks the control socket |
| +is *active*, not just enabled. An agent that finds the channel absent |
| +refuses nested wakes with `instance control socket not listening`, which |
| +is what a forgotten enable looks like from `carlos logs`; flat hosts are |
| +unaffected, which is the point of the two-layout release. |
| --- |
| @@ -1134,43 +1394,52 @@ enable looks like from `carlos logs`. |
| The rule (AGENTS.md): the consumer list is everything that parses the |
| shape, not everything that calls the API. From the sweep of 2026-09-11, |
| -extended by codex R1-9, R1-14, R1-17, R1-18: |
| +extended by codex R1-9, R1-14, R1-17, R1-18, R2-8, R2-9, R2-10, R2-13: |
| -**Path builders that move onto `InstanceDir` / the new defaults** |
| +**Path builders that move onto the helpers** |
| -- `internal/provision/provision.go:1011-1012` — `DefaultSocket`, `DefaultDBPath` (the definitions) |
| -- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); every read through `instancefs` |
| -- `internal/appconfig/syncexec.go:465, 872, 921-926` (exec routes only; addr routes stay flat; `O_EXCL` temp); `:977-1024` `reapConfigEnv` walks both |
| -- `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path) |
| -- `internal/host/movespass.go:285-305` — rebuilds `<DataDir>/<host>.db` from the record; reads `rt.DBPath` from the row instead, and honours the fence |
| -- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir"), the new `carlos route --instance-dir`, `carlos instances --layout`, `carlos instances-layout revert`; `docs/site/cli.md` names all three (the docsite gate fails otherwise) |
| -- `internal/provision/reconcile.go:300-301` (`routeFor` mint) — follows the helpers unchanged; `:461-464` and `:873-881` compare derived paths and follow too |
| +- `internal/provision/provision.go:1011-1012` — `DefaultSocket`, `DefaultDBPath` (the definitions), plus `InstanceDir`, `InstanceRunDir`, `LogSocket`, `FlatDBPath`, `FlatSocket` |
| +- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); `:744-750` (socket normalisation, unchanged); every nested read through `instancefs`; the classification of §8.1 |
| +- `internal/appconfig/syncexec.go:389-395, 465, 872, 921-926` (nested routes into the directory; flat and addr routes flat; `O_EXCL` temp); `:977-1024` `reapConfigEnv` walks both |
| +- `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path, by classification) |
| +- `internal/host/movespass.go:285-305` — rebuilds `<DataDir>/<host>.db` from the record; reads `rt.DBPath` from the row instead, and refuses a `fenced` host |
| +- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir" for nested rows), the new `carlos instances --layout` and `carlos system layout migrate|revert`; `docs/site/cli.md` names them |
| +- `internal/provision/reconcile.go:300-301` (`routeFor` mint) — new rows mint nested; `:461-464` and `:873-881` compare derived paths and log a `flat` row as drift for this one release (the message names the verb) |
| -**Readers of `rt.Socket` (no change; the value changes)** |
| +**Readers of `rt.Socket` (no change; the value changes per host)** |
| - `internal/edge/edge.go:536-538, 2000-2014` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| +**The socket consumers group** (R2-10): `internal/host/bootstrap.go:41-62` |
| +(`socketsGroup`, `tmpfilesConf` gains the lines of §3.2), the per-host |
| +run directory mode in `prepare`, and the #103 test that dials a migrated |
| +instance as an enrolled non-`carlos` uid |
| + |
| **Litestream and replication** |
| - `internal/lease/lease.go:286` — meta dir derived from `dbPath`; follows |
| -- `internal/ops/litestream.go:354-388` — the box-wide generator **discovers by glob and filters by registry claim**; it does not append claimed paths (R1-17). The glob gains `instances/*/*.db`, or nested non-hibernating databases are never replicated. Hibernating rows stay excluded at `:420-421` as today; the fence excludes a migrating host. Pinned in `ops_test.go` for both coverage and the `skipped` report |
| -- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine; ordered `After=data-instances.mount` |
| +- `internal/ops/litestream.go:354-388` — the box-wide generator **discovers by glob and filters by registry claim**; it does not append claimed paths (R1-17). The glob gains `instances/*/*.db`, or nested non-hibernating databases are never replicated. Hibernating rows stay excluded at `:420-421` as today; a `fenced` host is excluded. Pinned in `ops_test.go` for both coverage and the `skipped` report |
| +- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine; ordered after `data-instances.mount` |
| **Metrics and status** |
| -- `internal/host/metrics_probes.go` (discovery: the new scan root), `internal/console/metricsview.go:728-762` (the key allowlist: unchanged because the sampler publishes under `carlos-exec@<host>`) |
| -- `internal/status/snapshot.go` gains the per-route `layout` field of §8.5; `carlos status` and the console instance page render it |
| +- `internal/host/metrics_probes.go:49` (discovery: the two new roots), `internal/host/metrics_loop.go:659` (the edge's published value becomes the slice total), `internal/console/metricsview.go:728-762` (the key allowlist: unchanged because the sampler publishes under `carlos-exec@<host>`) |
| +- `internal/status/snapshot.go` gains the per-route `layout` field of §8.4; `carlos status` and the console instance page render it |
| + |
| +**The updater and the control plane** |
| + |
| +- `internal/host/systemapply.go:360-372, 738-748`, `carlos-system-recover.service`, `system.go:466`: unchanged, by design (§8.1); the layout verbs are a separate console door and bucket object (`control/layout.json`), a separate box pass, and a separate mark in `restart-marks.json`'s sibling `layout-marks.json` |
| +- `internal/console`: the `POST /api/cli/system/layout` door (fleet-operator gate, fresh auth), `carlos system layout` in `system.go` |
| **Units and installers** |
| -- `internal/host/units/` gains: three templates, two slices, two mounts, the probe, the control socket and service (§4, §5, §8.4); `carlos-edge.service` gains `Slice=` and drops its memory lines; `carlos-litestream.service` gains the `After=` |
| +- `internal/host/units/` gains: three templates, two slices, two mounts, the probe, the control socket and service (§4, §5, §8.5); `carlos-edge.service` gains `Slice=`, `Requires=`/`After=` on the mounts, and drops its memory lines; `carlos-litestream.service` gains the `After=` |
| - `internal/host/bootstrap.go:61` (tmpfiles lines), `:116-122` `enabledUnits` (control socket, mounts, probe), `:165-179` `systemctlPlan` (starts `.socket`/`.mount`) |
| - `internal/host/systemunits.go:34-83` (active check on the socket) |
| - `update-platform-binary.sh:52-70` (pinfra): enables the new units by name before restarting the edge |
| - `harden-host.sh` (pinfra) writes hardening drop-ins for `$TENANT_UNITS`; the instance templates carry their own and must not be listed there |
| -- the `carlos system rollback` leg (system-update design) runs the reverse migration first |
| -**Tests that pin the flat layout** (§10): `internal/activator/activator_test.go` (20 fixture sites), `backend_exec_test.go` (4), `admintoken_test.go` (8, path-agnostic), `internal/appconfig/syncexec_test.go` (50), `internal/ops/ops_test.go` (52), `internal/provision/reconcile_test.go` (20), `internal/host/movespass_test.go` (1), `schedule_wiring_test.go` (1), `schedule_token_fence_test.go` (2), root `e2e_test.go` (24), `instances_e2e_test.go`, `config_e2e_test.go`, `deploy_e2e_test.go`. |
| +**Tests that pin the flat layout** (§10): `internal/activator/activator_test.go` (20 fixture sites), `backend_exec_test.go` (4), `admintoken_test.go` (8, path-agnostic), `internal/appconfig/syncexec_test.go` (50), `internal/ops/ops_test.go` (52), `internal/provision/reconcile_test.go` (20), `internal/host/movespass_test.go` (1), `schedule_wiring_test.go` (1), `schedule_token_fence_test.go` (2), root `e2e_test.go` (24), `instances_e2e_test.go`, `config_e2e_test.go`, `deploy_e2e_test.go`. For this release they keep passing on the flat path; each gains a nested twin. |
| **Not consumers**: the console never reads box paths (`internal/console` |
| has no `DBPath`/`DataDir` reader); `readDiskFrac` is whole-volume; the |
| @@ -1182,66 +1451,98 @@ sentinel `.path` units name fixed files under `/data`, never a host. |
| Unit level, all runnable in CI (no systemd needed): |
| -- `internal/provision`: `InstanceDir`, new defaults, `LogSocket`, the |
| - escape function against `systemd-escape` fixtures, `carlos add` |
| - refusal of an exec `--db` outside the directory. |
| +- `internal/provision`: `InstanceDir`, the nested defaults, the flat |
| + helpers, `LogSocket`, the escape function against `systemd-escape` |
| + fixtures, `carlos add` refusal of a nested `--db` outside the directory. |
| - `instancefs`: refuses a symlink at every component, refuses `..`, |
| `O_EXCL` temp plus rename, the mountinfo check. |
| -- `internal/activator`: `prepare` writes `launch.<role>.json` with the |
| +- `internal/activator`: classification of the four row shapes; `prepare` |
| + mints the token for seed and run, writes `launch.<role>.json` with the |
| deduplicated env and the exact argv (the existing env-composition tests |
| move their assertion to the effective environment of an executed stub); |
| - the control client sends the tier and the token, retries once on 401 |
| - after re-reading the token; `unitProc.Stop` orders stop → done → close |
| - listeners; `Exited()` only after a confirmed terminal state with the |
| - matching invocation id, never on log EOF; the failed-health path posts |
| - `stop` and waits; seed timeout and cancellation stop-and-wait before |
| - cleanup; the startup sweep stops activating units and cancels jobs. |
| - The control channel is a seam (`Control interface { Start; Seed; Stop; |
| - Reset; Ready }`) with a fake, as `Systemctl` is today; the exit |
| - observer takes a fake `show` and a fake events file. |
| + the control client sends the tier and the token with a deadline, and |
| + retries once on 401 after re-reading the token; `unitProc.Stop` orders |
| + stop → done → close listeners; `Exited()` only after an exit record (or |
| + an unloaded unit) **and** an empty cgroup, never on log EOF, with a |
| + populated cgroup holding `done` open; a foreign invocation's record; |
| + the failed-health path posts `stop` and waits; seed timeout and |
| + cancellation stop-and-wait before cleanup; the startup sweep stops |
| + activating units, cancels jobs and waits for empty cgroups; the sidecar |
| + restart reserves under the lock and installs or stops after, with a |
| + stalled start while another host wakes and stops. The control channel |
| + is a seam (`Control interface { Start; Seed; Stop; Reset; Ready }`) |
| + with a fake, as `Systemctl` is today; the exit observer takes a fake |
| + records directory and a fake cgroup tree. |
| - `instance-exec`: reads the file through `instancefs`, refuses schema |
| ≠ 1, refuses an `argv[0]` that is not the live link, execs. |
| +- `instance-exit`: writes the record with `O_EXCL` from the four |
| + variables; refuses to overwrite. |
| - `instance-control`: 401 without the token; every 400 (bad host, bad |
| role, bad limit); the unit and slice names it builds for a host with a |
| - dash; the `set-property` → `reset-failed` → `start` ordering; the seed |
| - result mapping; `/ready` from the probe verdict; with a fake |
| - `systemctl` runner. |
| + dash; `set-property` (three properties) → `oom.group` write → `start` |
| + ordering with no `reset-failed`; `/reset` tolerating "not loaded"; |
| + `/ready` from the probe verdict; with a fake `systemctl` runner and a |
| + fake cgroup tree. |
| +- `instance-probe`: each verdict line, including `EPERM` versus |
| + `ECONNREFUSED` versus timeout on the metadata addresses, with a fake |
| + dialer. |
| - `logListener`: one connection per invocation, a second refused and |
| logged, line cap, EOF ends collection without marking exit. |
| -- Migration: eligible row moves eight files under the marker and |
| - CAS-updates two columns; ineligible and external rows are logged and |
| - left; idempotent re-run after a simulated crash between files and row; |
| - CAS conflict keeps the fence; two databases refuse; every fenced |
| - consumer (`Start`, `Seed`, `Restore`, moves pass, `SyncExec`, litestream |
| - generator) refuses a fenced host; the reverse migration restores the |
| - flat layout byte for byte; a partial rename followed by a |
| - `MoveCutting` pass does not complete the flip. |
| -- Units: a test over `unitsFS` asserts the three templates' namespace, |
| - hardening and coupling blocks are identical, that `PrivatePIDs=yes`, |
| - `TemporaryFileSystem=/data:ro`, `IPAddressDeny=169.254.169.254/32`, |
| - `Restart=no`, `BindsTo=carlos-edge.service`, `Slice=carlos-instances-%i.slice` |
| - and the `%I` paths are present, that the seed carries |
| +- Layout pass: eligibility by the activator's predicate (flagged only; |
| + external, tunnel, unit-backed untouched); quiesce before plan; the |
| + ownership preflight for `x`/`x.config` in both directions; destination |
| + collision refusal; the journal's phases and every crash boundary |
| + (before files, mid-files, after files, after CAS, before journal |
| + removal) recovered exactly, with absent optional artifacts; a CAS |
| + conflict keeps the fence; a second clean start does nothing; every |
| + fenced consumer (`Start`, `Seed`, `Restore`, moves pass, `SyncExec`, |
| + litestream generator) refuses a fenced host; a partial rename followed |
| + by a `MoveCutting` pass does not complete the flip; revert restores the |
| + flat layout byte for byte; a forged journal-shaped file and a forged |
| + launch file in an instance directory move nothing (§3.3). |
| +- Units: a test over `unitsFS` asserts the three templates' and the |
| + probe's namespace, hardening and coupling blocks are identical, that |
| + `PrivatePIDs=yes`, `TemporaryFileSystem=/data:ro`, |
| + `IPAddressDeny=169.254.169.254/32`, `Restart=no`, |
| + `BindsTo=carlos-edge.service`, `Slice=carlos-instances-%i.slice`, |
| + `ExecStopPost=+` and the `%I` paths are present, that the seed carries |
| `TimeoutStartSec=120`, that the control socket is `SocketMode=0600 |
| - SocketUser=carlos`, that both mounts carry `nosymfollow`, and that |
| - `carlos-edge.service` carries `Slice=carlos.slice` and no `MemoryMax=`. |
| + SocketUser=carlos`, that both mounts carry `nosymfollow` and |
| + `DefaultDependencies=no` with the ordering of §3.2, and that |
| + `carlos-edge.service` carries `Slice=carlos.slice` and no `MemoryMax=`; |
| + and, where `systemd-analyze` exists on the CI host, `systemd-analyze |
| + verify` over the rendered set. |
| - `internal/ops`: the nested glob, coverage of a nested non-hibernating |
| row, the `skipped` report, the fence. |
| -- `internal/host`: `systemctlPlan` starts sockets and mounts; the |
| - sampler's slice scan root publishes under `carlos-exec@<host>`. |
| -- `internal/status`: the `layout` field for each of the five states. |
| - |
| -Box level, in the plan's verification task and in the pre-roll check: |
| - |
| -- On a dev box: the probe verdict; wake an instance, |
| - `systemctl status carlos-instance@<esc>` active in |
| - `carlos-instances-<esc>.slice`; from inside via the throwaway |
| +- `internal/host`: `systemctlPlan` starts sockets and mounts; the two |
| + sampler roots publish per-host under `carlos-exec@<host>` and the edge |
| + as the slice total, flat hosts still from the delegated subtree; the |
| + #103 consumer dial through the nested run directory. |
| +- `internal/status`: the `layout` field for each of the five states and |
| + the pre-field box. |
| +- `internal/console` and `system.go`: the layout door's gate and |
| + refusal order, the bucket object, the scope forms. |
| + |
| +Box level, in the plan's verification task and in the rollout: |
| + |
| +- Cold boot with the full unit set; `systemd-analyze verify`; |
| + `list-dependencies` shows the mounts before the edge. |
| +- Probe verdict `ready`; a deliberately broken probe (a drop-in removing |
| + `IPAddressDeny=`) turns it `not enforced` and blocks nested wakes. |
| +- Migrate one host; wake it; `systemctl status carlos-instance@<esc>` |
| + active in `carlos-instances-<esc>.slice` with the three memory |
| + properties and `memory.oom.group=1`; from inside via the throwaway |
| instance's seed: `/data` and `/run/carlos` list one directory each, |
| - `/proc/1` is the app, IMDS refuses, a planted symlink gives `ELOOP`; |
| - `carlos logs` shows all of it for both instance and sidecar at once; |
| - hibernate, unit gone; `carlos restart` cycles it; |
| - `systemctl restart carlos-edge` stops it and does not start it; |
| - `carlos instances --layout` says `nested`; the console's memory graph |
| - for the app is continuous across the roll. |
| + `/proc/1` is the app, both metadata addresses `EPERM`, loopback |
| + `ECONNREFUSED`, a planted symlink `ELOOP`; `carlos logs` shows all of |
| + it for both instance and sidecar at once; a quick-exiting seed's record |
| + arrives after the unit is unloaded; hibernate, unit gone, record |
| + consumed; `carlos restart` cycles it; `systemctl restart carlos-edge` |
| + stops it and does not start it; an enrolled `carlos-sockets` uid dials |
| + `app.sock` and cannot open the control socket; `carlos instances |
| + --layout` says `nested`; the console's memory graphs for the app and |
| + for the platform are continuous across the migration; revert, and the |
| + previous release serves the host again. |
| --- |
| @@ -1252,8 +1553,8 @@ Box level, in the plan's verification task and in the pre-roll check: |
| without it. If a later finding needs a uid boundary, it lands on this |
| layout with a root chown in the control service. Not in this change. |
| - **Externally supervised exec routes** (unflagged instance routes with |
| - a socket, §8.1): not isolated by this change; their supervisor owns |
| - their process. Whether the platform should offer them the same |
| + a socket, §8.2 step 1): not isolated by this change; their supervisor |
| + owns their process. Whether the platform should offer them the same |
| namespace is a discussion to open once this lands. |
| - **Unit-backed console instances.** Not needed for isolation, and a |
| hibernation rewrite. Whether `backing=unit` hibernation works on the |
| @@ -1262,11 +1563,11 @@ Box level, in the plan's verification task and in the pre-roll check: |
| to `carlos-app@` (R1-19). |
| - **#127**: root-rendered config for opted-in addr routes into |
| `/etc/carlos/apps`. Smaller and independent; this change makes the file |
| - invisible to instances but not to the agent. |
| + invisible to isolated instances but not to the agent. |
| - **Adopting running units on agent start** instead of stopping them |
| (§6.7). |
| -- **Deleting the spawn half of `internal/cgroup`** and `Delegate=` from |
| - the edge unit, one release after this lands. |
| +- **The next release** (§8.6 step 5): drop the flat path and the spawn |
| + half of `internal/cgroup`. |
| - **Discussion #1 (memory pressure)**: no new numbers here (§4.3); the |
| slice shape makes a later change a unit-file edit, never a drop-in. |
| @@ -1287,11 +1588,14 @@ Box level, in the plan's verification task and in the pre-roll check: |
| spec). Keeps every activator invariant, needs one root channel, and |
| migrates with a rename. |
| -Within 3: trampoline over `EnvironmentFile=` (§6.2, one env composer), |
| -three templates over one (§4.2, the instance is the host), the control |
| -service with a token over polkit (§5.4, ruled with codex), and |
| +Within 3: trampoline over `EnvironmentFile=` (§6.2, one env composer); |
| +three templates over one (§4.2, the instance is the host); the control |
| +service with a token over polkit (§5.4, ruled with codex); |
| `nosymfollow` mounts over per-read symlink checks (§3.2, kernel-enforced |
| -and race-free where a check-then-open is neither). |
| +and race-free where a check-then-open is neither); an exit record from |
| +`ExecStopPost=+` over polling unit state (§6.5, survives unit unload); |
| +and an operator verb over an agent-start migration (§8.1, because the |
| +updater's rollback runs the old binary). |
| --- |
| @@ -1302,19 +1606,35 @@ and race-free where a check-then-open is neither). |
| R1-1 control service with token, `nosymfollow` (§3.2, §5.4); |
| R1-2 log sockets outside tenant mounts, per role (§3.1, §6.4); |
| R1-3 `nosymfollow` plus `instancefs` and `O_EXCL` temps (§3.2, §7); |
| - R1-4 exit from systemd state with invocation id, never from EOF (§6.5); |
| + R1-4 exit from a record plus cgroup emptiness, never from EOF (§6.5); |
| R1-5 seed timeout, stop verb, wait before cleanup (§4.2, §5.3, §6.2); |
| R1-6 per-role log listeners (§6.4); |
| R1-7 `BindsTo=` not `PartOf=`, sweep covers jobs (§4.1, §6.7); |
| - R1-8 per-host slice carries the tier for all three roles (§4.3, §6.6); |
| + R1-8 per-host slice carries the tier and `memory.oom.group` for all three roles (§4.3, §6.6); |
| R1-9 cgroup path under `carlos.slice`, key compatibility, `metrics_probes.go` (§6.6, §9); |
| R1-10 one common wall in `carlos.slice`, no new numbers (§4.3); |
| - R1-11 eligibility is the activator's predicate (§8.1); |
| - R1-12 the `.migrating` fence and its consumers (§8.1); |
| - R1-13 explicit reverse migration; no "restore" claim (§8.3); |
| - R1-14 planner starts sockets and mounts; roll script; active check (§8.5); |
| - R1-15 no `PrivatePIDs=` fallback; behavioural gate and probe (§8.4); |
| + R1-11 eligibility is the activator's predicate (§8.2); |
| + R1-12 the journal fence and its consumers (§8.2); |
| + R1-13 explicit reverse migration; no "restore" claim (§8.2); |
| + R1-14 planner starts sockets and mounts; roll script; active check (§8.6); |
| + R1-15 no `PrivatePIDs=` fallback; behavioural gate and probe (§8.5); |
| R1-16 env deduplication, effective-value test (§6.2); |
| R1-17 nested glob is discovery, not reporting (§9); |
| - R1-18 `layout` status through the API and CLI, `docs/site/cli.md` (§8.5, §9); |
| + R1-18 `layout` status through the API and CLI, `docs/site/cli.md`; repair is `carlos add`, not a `--db` flag `carlos route` does not have (§8.3, §8.4); |
| R1-19 `UnitBackend` claim removed (§1.4, §11). |
| +- **Round 2** (codex, `CODEX-FINDINGS-ISOLATED-EXEC-R2.md`, 14 new |
| + findings, verdict request changes). All accepted: |
| + R2-1 journal outside tenant mounts, plan from row and host only, nothing in an instance directory is an instruction (§3.3, §8.2); |
| + R2-2 mounts with `DefaultDependencies=no` and explicit ordering, cold-boot verification, `systemd-analyze verify` in CI (§3.2); |
| + R2-3 no `reset-failed` in `/start` (§5.3); |
| + R2-4 exit record from `ExecStopPost=+`, survives unit unload; foreign invocation handling (§6.5); |
| + R2-5 cgroup emptiness required on every path before `done` (§6.5); |
| + R2-6 sidecar restart reserved under the lock, started outside it with a deadline (§5.3, §6.7); |
| + R2-7 journal records artifact presence and phase; committed-row recognition; classification before legacy test (§8.1, §8.2); |
| + R2-8 ownership preflight in both directions from `SyncExec`'s rule (§8.2); |
| + R2-9 two-layout release and operator verb; the updater's rollback lands on a layout the old binary serves (§8.1, §8.2, §8.6); |
| + R2-10 `carlos-sockets` traversal preserved on the run directories (§1.1, §3.1); |
| + R2-11 `MemorySwapMax` on the host slice (§4.3); |
| + R2-12 seed prepares the admin token and merged env (§6.1); |
| + R2-13 edge discovered in `carlos.slice`, published as the slice total (§6.6); |
| + R2-14 probe distinguishes `EPERM` from unreachable, tests both metadata addresses (§8.5). |
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R2.md b/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R2.md| new file mode 100644 |
| index 000000000..3d430fa37 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-FINDINGS-ISOLATED-EXEC-R2.md |
| @@ -0,0 +1,155 @@ |
| +| Prior finding | Disposition at `d70f1109` | |
| +|---|---| |
| +| R1-1 | **Resolved for the reported attack.** Protected socket resolution and bearer authentication replace uid-only authorization. Retain the §5.4 ruling. | |
| +| R1-2 | **Resolved.** Log sockets move outside tenant mounts. | |
| +| R1-3 | **Resolved for the reported symlink attacks.** `nosymfollow`, confined opens and exclusive temporary files address them. The new recovery marker introduces separate authority exposure, R2-1. | |
| +| R1-4 | **Partially resolved.** EOF no longer means exit, and `Exited()` is nonblocking. Completion evidence and another blocking call under the activator mutex remain defective, R2-4–R2-6. | |
| +| R1-5 | **Partially resolved.** Timeout, cancellation and stop-before-cleanup are specified. Safe completion depends on R2-4/R2-5; seed preparation also regresses, R2-12. | |
| +| R1-6 | **Resolved.** Each role has its own log listener. | |
| +| R1-7 | **Resolved.** `BindsTo=` removes dependency-propagated restarts; the sweep includes activating units and pending jobs. | |
| +| R1-8 | **Partially resolved.** All roles share the tier. Whole-family OOM behavior remains incomplete: a sidecar OOM can leave the main process running, whereas the previous shared `memory.oom.group=1` killed both. | |
| +| R1-9 | **Resolved for per-host metrics.** Hierarchy, aggregation and published key are corrected. Moving the edge creates another discovery omission, R2-13. | |
| +| R1-10 | **Resolved.** The arbitrary split is withdrawn and the shared outer wall retained. | |
| +| R1-11 | **Resolved.** Migration excludes externally supervised and tunnel/unit-backed routes. | |
| +| R1-12 | **Partially resolved.** The fence covers the named consumers, but recovery mishandles optional artifacts, committed CAS operations and already-nested rows, R2-7. | |
| +| R1-13 | **Partially resolved.** Reverse migration replaces the incorrect restore story. The actual downgrade and recovery execution paths remain uncovered, R2-9. | |
| +| R1-14 | **Resolved.** Socket activation, installer changes and active-state checks are explicitly required. The new mount introduces a separate boot failure, R2-2. | |
| +| R1-15 | **Partially resolved.** The unsafe fallback is withdrawn. The replacement probe can report readiness without proving IMDS enforcement, R2-14. | |
| +| R1-16 | **Resolved.** Last-value-wins deduplication and an executed-child test are specified. | |
| +| R1-17 | **Resolved.** Nested discovery, registry filtering and hibernation exclusions are accurately described. | |
| +| R1-18 | **Partially resolved.** Remote status and CLI documentation are specified. Repair still uses the local-only `carlos route`; its prescribed `--db` flag does not exist ([cli.go:469](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/cli.go:469)). Pre-roll inspection also depends on a snapshot field that old boxes cannot publish. Define the remote repair operation and compatibility rollout. | |
| +| R1-19 | **Resolved.** The unsupported `UnitBackend` claim is removed. | |
| + |
| +**R2-1 — P1 — §§5.4, 3.1, 8.1: Retain the authenticated control service, but move authoritative recovery metadata outside tenant mounts.** |
| + |
| +The new `.migrating` file lives in the tenant’s writable directory. Recovery explicitly repeats filesystem operations using the paths recorded in that file. A tenant can forge a marker directing the agent to move a sibling’s database or env file. These are ordinary paths: `nosymfollow` does not prevent the operation. A later CAS refusal is too late because renames happen first. |
| + |
| +Evidence: [spec:242](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:242), [spec:1008](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1008), [spec:1022](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1022). |
| + |
| +**Correction:** keep the authoritative transaction record outside every tenant mount. Treat a tenant-local marker only as an untrusted refusal signal. Validate every source and destination against trusted host-specific metadata before moving anything. Test a forged sibling path and assert that the victim remains untouched. |
| + |
| +**R2-2 — P1 — §3.2: The socket-tree mount creates a boot ordering cycle.** |
| + |
| +`run-carlos-instances.mount` declares `After=systemd-tmpfiles-setup.service`. A local mount receives a default ordering before `local-fs.target`, while tmpfiles starts after that target. The resulting cycle is: |
| + |
| +`mount → local-fs.target → tmpfiles → mount` |
| + |
| +systemd must break the boot transaction. Dropping the required mount blocks the control service and tenant starts. Installing the mount on an already-booted box does not exercise this failure. |
| + |
| +Evidence: [spec:331](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:331), [systemd mount dependencies](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/mount.c), [tmpfiles unit](https://raw.githubusercontent.com/systemd/systemd/v257/units/systemd-tmpfiles-setup.service). |
| + |
| +**Correction:** specify an acyclic directory-preparation and mount sequence, including shutdown ordering. Verify a cold boot with both mounts and the readiness probe. |
| + |
| +**R2-3 — P1 — §5.3: Moving property-setting to the slice makes the first service start fail at `reset-failed`.** |
| + |
| +The revised sequence sets properties on the host **slice**, then resets the **service**, then starts it. The first operation loads only the slice. systemd’s `ResetFailedUnit` deliberately does not load an unknown service, so a never-loaded template instance can fail before reaching `start`. The specified error handling returns 502. |
| + |
| +Evidence: [spec:661](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:661); [`ResetFailedUnit` versus `SetUnitProperties`](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/dbus-manager.c), lines 781–788. |
| + |
| +**Correction:** load and retain the service before resetting it, or tolerate precisely the missing-unit reset outcome. Test a first wake with neither the service nor slice previously loaded. |
| + |
| +**R2-4 — P1 — §§5.3, 6.5: Invocation polling can lose the completion record permanently.** |
| + |
| +systemd unloads unreferenced inactive units and discards their execution state. Nothing specified pins these services or retains their results. A clean exit can disappear before the next `show`, leaving the original invocation permanently unmatched and `done` open. Successful seeds are particularly exposed: `/seed` reads their invocation only after the oneshot finishes. |
| + |
| +A replacement invocation has the same unresolved outcome. The observer ignores it forever; `Restart=no` does not prohibit an explicit restart by another authorized actor. |
| + |
| +Evidence: [spec:862](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:862), [systemd unit garbage collection](https://raw.githubusercontent.com/systemd/systemd/v257/man/systemd.unit.xml), [unit reference operations](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/dbus-manager.c). |
| + |
| +**Correction:** retain invocation identity and completion evidence until the agent acknowledges them, using a persistent unit reference or broker-owned records. Define recovery for a replaced invocation. Test immediate successful exit, delayed observation and an external restart. |
| + |
| +**R2-5 — P1 — §6.5: `failed` does not prove that every writer has stopped.** |
| + |
| +The fallback poll accepts a matching invocation in `inactive` or `failed` without requiring an empty cgroup. systemd explicitly enters failed state when processes remain after its final SIGKILL timeout. The proposed observer can therefore close `done`, permitting seed cleanup or lease release while a process remains. |
| + |
| +Evidence: [spec:869](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:869); [systemd service implementation](https://raw.githubusercontent.com/systemd/systemd/v257/src/core/service.c), lines 4020–4022. |
| + |
| +**Correction:** require proven cgroup emptiness as a completion condition, including on the fallback path. Define what a vanished cgroup proves and keep uncertainty from authorizing cleanup or lease release. |
| + |
| +**R2-6 — P1 — §§6, 6.2: Sidecar restart introduces blocking systemd work under the global activator mutex.** |
| + |
| +`sweepOnce` calls `StartSidecar(context.Background(), ...)` while holding `a.mu`. Its existing justification is that spawning is quick. The replacement performs synchronous HTTP and systemctl operations, with no specified deadline. A stalled control operation now prevents unrelated hosts from waking or stopping. |
| + |
| +Making `Exited()` nonblocking does not address this second call site, and the statement that `activator.go` remains unchanged is untenable. |
| + |
| +Evidence: [activator.go:1170](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/activator.go:1170), [activator.go:1196](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/activator.go:1196), [spec:812](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:812). |
| + |
| +**Correction:** reserve the sidecar start under the mutex, perform bounded control work outside it, then reconcile completion with any intervening stop. Test a stalled sidecar start while another host wakes and stops. |
| + |
| +**R2-7 — P1 — §8.1: The new recovery algorithm cannot complete common restart states.** |
| + |
| +Three specified branches fail: |
| + |
| +- Migration moves only artifacts that exist, but recovery treats an artifact absent at both source and destination as an error. Ordinary hosts lack several optional artifacts, including sidecar env and quarantined DB files. |
| +- A crash after the row CAS but before marker removal retries the original CAS against an already-updated row, producing a permanent conflict. |
| +- A clean subsequent startup sends an already-nested row through the “not flat defaults” branch into the hand-set-path refusal. |
| + |
| +Evidence: [spec:1001](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1001), [spec:1013](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1013), [spec:1022](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1022). |
| + |
| +**Correction:** record artifact presence and migration direction, recognize a committed destination row, and classify settled nested rows before testing legacy defaults. Test every crash boundary and a second clean agent restart. |
| + |
| +**R2-8 — P1 — §8.1: Migration ignores existing flat-file ownership collisions.** |
| + |
| +For routes `x` and `x.config`, `/data/x.config.env` belongs to the latter’s instance env. Config sync already refuses to write that pathname for `x`. Migration nevertheless moves every existing `<host>.config.env` into that host’s directory. Processing `x` first places the sibling’s credentials in `x`’s config layer and removes them from their owner. |
| + |
| +Preserving the reaper’s ambiguity guard does not protect migration. |
| + |
| +Evidence: [syncexec.go:389](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/appconfig/syncexec.go:389), [spec:1013](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1013). |
| + |
| +**Correction:** preflight ownership of every source pathname against the complete registry before renaming. Refuse ambiguous transactions or preserve the established owner. Apply equivalent checks when reversing into a flat directory containing excluded routes. |
| + |
| +**R2-9 — P1 — §8.3: The first upgrade’s rollback executes the old binary, which cannot invoke the new reverse migration.** |
| + |
| +The installed binary starts `system apply`. That running process installs the candidate, restarts the edge and gates it. If the new edge migrates data and then fails its gate, the **old process** executes its existing `rollback` → `restoreFiles`, restoring the old executable without reversing the layout. Boot recovery deliberately executes `carlos.prev` when an update journal exists. |
| + |
| +Furthermore, `carlos system rollback` repoints a channel through the console API. Its resulting downgrade follows ordinary box-side installation; adding a leg only to a local failure handler does not cover it. |
| + |
| +Evidence: [systemapply.go:366](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/systemapply.go:366), [systemapply.go:743](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/systemapply.go:743), [recovery unit:17](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/units/carlos-system-recover.service:17), [system.go:466](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/system.go:466). |
| + |
| +**Correction:** specify a compatibility release or an explicit transition protocol understood by the updater and recovery executable before enabling migration. Cover channel downgrade, failed-gate rollback and interrupted recovery. Reverse migration must also quiesce activation and replication, rather than merely stop tenant units. |
| + |
| +**R2-10 — P2 — §§3.1, 6.1, 9: The new socket directories exclude enrolled cross-user consumers.** |
| + |
| +Existing instances deliberately expose their listening sockets to members of `carlos-sockets`: `/run/carlos` supplies the inherited group, and the backend normalizes sockets to 0660. The new layout assigns `carlos:carlos` to the parent and 0700 to each host directory. An enrolled router cannot traverse the host directory; preserving the socket chmod does not help. |
| + |
| +Evidence: [bootstrap.go:41](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/bootstrap.go:41), [backend_exec.go:744](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/backend_exec.go:744), [spec:244](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:244). |
| + |
| +**Correction:** preserve the explicit socket-consumer contract through directory traversal and socket-group permissions, while retaining tenant namespace isolation. Test a migrated instance through an enrolled non-`carlos` router. |
| + |
| +**R2-11 — P2 — §§4.3, 6.6: The slice configuration drops the existing swap ceiling.** |
| + |
| +The current cgroup code sets `memory.swap.max` equal to the memory tier. Its comment records a real allocator escaping the intended limit through zram before that setting was added. The proposed slice receives only `MemoryMax` and `MemoryHigh`. |
| + |
| +Evidence: [cgroup.go:313](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/cgroup/cgroup.go:313), [spec:578](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:578), [systemd `MemorySwapMax` documentation](https://raw.githubusercontent.com/systemd/systemd/v257/man/systemd.resource-control.xml). |
| + |
| +**Correction:** set `MemorySwapMax` to the tier limit on the shared host slice. Verify the delivered limit under swap pressure. |
| + |
| +**R2-12 — P2 — §§6.1–6.2: First-boot seeds lose admin-token preparation.** |
| + |
| +The new preparation sequence calls `EnsureAdminToken` only for `run`. Today `Seed` explicitly creates the token before reading its environment so that initialization receives the same credential as the serving process. Under the proposed sequence, the first seed runs before that credential exists. |
| + |
| +Evidence: [backend_exec.go:346](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/activator/backend_exec.go:346), [spec:733](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:733). |
| + |
| +**Correction:** prepare the admin token and merged environment for both seed and run, retaining the seed’s removal of control-channel variables. Verify that seed and subsequent run receive the same token. |
| + |
| +**R2-13 — P2 — §§4.3, 6.6, 9: Moving the edge into `carlos.slice` removes it from metrics discovery.** |
| + |
| +Current platform-unit discovery scans `system.slice` and its immediate template slices. The new scan root covers `carlos-instances.slice`, but the relocated `carlos-edge.service` is its sibling. Neither scan finds the edge. |
| + |
| +The prior accounting design also defines the edge reading as the hierarchical total including children. Relocating discovery alone would change that meaning. |
| + |
| +Evidence: [metrics_probes.go:49](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/metrics_probes.go:49), [metrics_loop.go:659](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/internal/host/metrics_loop.go:659), [accounting design §4](/home/paulca/github.com/carlosframework/platform-infrastructure/platform-design/superpowers/specs/2026-08-07-exec-child-accounting-design.md), [spec:569](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:569). |
| + |
| +**Correction:** explicitly discover the relocated edge and define how the previous aggregate reading is preserved or replaced. Test platform metrics as well as member per-host metrics. |
| + |
| +**R2-14 — P2 — §8.4: The readiness probe can pass without proving IMDS isolation.** |
| + |
| +The probe accepts any failed IPv4 connection to IMDS. A routing failure or timeout also satisfies that assertion when the systemd IP filter is ineffective. It never tests the separately denied IPv6 metadata address. systemd documents that IP filtering depends on kernel/BPF support. |
| + |
| +Thus the recorded `ready` verdict is stronger than the test establishes. |
| + |
| +Evidence: [spec:1079](/home/paulca/amadan.net/carlos/platform/.claude/worktrees/isolated-exec-spec/docs/superpowers/specs/2026-09-11-isolated-exec-design.md:1079), [systemd IP filtering documentation](https://raw.githubusercontent.com/systemd/systemd/v257/man/systemd.resource-control.xml). |
| + |
| +**Correction:** distinguish enforced denial from unrelated network failure, test both address families, and keep unavailable evidence from producing `ready`. Include a negative test where filtering is disabled while the destination is unreachable. |
| + |
| +**Verdict: request changes.** |
| \ No newline at end of file |