| index 7edd2597e..6efd6d780 100644 |
| --- a/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| +++ b/docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| @@ -1,8 +1,8 @@ |
| # Isolated exec — every exec-backed instance in its own namespace |
| |
| -**Date:** 2026-09-11 · **Status:** draft, awaiting codex review round 1 · |
| -**Requested by:** discussion #126 (Tito, with box evidence), approved in |
| -direction by Paul on 2026-09-11 |
| +**Date:** 2026-09-11 · **Status:** revision 2, after codex round 1 |
| +(19 findings, all accepted; §13) · **Requested by:** discussion #126 |
| +(Tito, with box evidence), approved in direction by Paul on 2026-09-11 |
| |
| **Builds on:** discussion #126 and its companion #127; titogo's |
| `deploy/systemd/titogo@.service` (the namespace recipe, verified on |
| @@ -27,14 +27,16 @@ children in place with a per-host directory plus a uid or namespace. |
| Three approaches were compared against the code on 2026-09-11 (§12). |
| Paul approved the third on the same day and asked that the one open |
| mechanism choice, how the unprivileged agent starts a root-managed unit, |
| -be settled between Fable and codex during review (§5.4). |
| +be settled between Fable and codex during review. Codex's round 1 |
| +concurred with the control service and showed why it must authenticate |
| +its caller (§5.4). |
| |
| The shape in one sentence: **the activator keeps owning every exec-backed |
| instance exactly as it does today, but the process it owns is launched as |
| a systemd template unit, still uid `carlos`, inside a mount namespace in |
| -which nothing but that instance's own directory exists.** No uid change, |
| -no console change, no new backing, and the migration is a rename the |
| -agent performs itself. |
| +which nothing but that instance's own directory exists, on a mount where |
| +no symlink is ever followed.** No uid change, no console change, no new |
| +backing, and the migration is a rename the agent performs itself. |
| |
| --- |
| |
| @@ -104,7 +106,25 @@ above implies: |
| carries `IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128` for exactly |
| this reason; exec children carry nothing. |
| |
| -### 1.3 What the unit backing already gets right, and why it is not the answer |
| +### 1.3 The primitive under all of it: a symlink is a read |
| + |
| +A namespace hides a sibling's files from the instance. It does not hide |
| +them from the **agent**, which reads and writes the instance's directory |
| +from outside, as the same uid, by path. Any path the instance can write |
| +and the agent later opens is therefore a request the instance makes on |
| +the agent's behalf: replace `<host>.env` with a symlink to |
| +`/data/litestream.env` and the next wake hands the bucket credentials to |
| +the instance as its own environment (`backend_exec.go:785-794, 966-971`; |
| +`EnsureAdminToken` only refuses the *chmod* through a symlink, |
| +`admintoken.go:141-150`). Replace `<host>.db` with a symlink to a |
| +sibling's database and the agent's replicator uploads the sibling's data |
| +under the attacker's replica. Replace the listening socket with a symlink |
| +to any other socket on the box and the edge dials it as this host |
| +(codex R1-1, R1-3). A design that bounds the instance but not the paths |
| +the agent follows on its behalf has not bounded anything. §3.2 closes |
| +this at the kernel, for every reader, without a race. |
| + |
| +### 1.4 What the unit backing already gets right, and why it is not the answer |
| |
| `carlos-app@.service` (`DynamicUser=yes`, `StateDirectory=carlos-app/%i`, |
| root-0600 config in `/etc/carlos/apps`) has the property #126 asks for: a |
| @@ -126,8 +146,10 @@ One more thing the sweep found: `UnitBackend.systemctl` |
| (`backend_unit.go:94-103`) runs `systemctl start` as the unprivileged |
| agent. Nothing in this tree or in pinfra grants uid `carlos` the |
| `org.freedesktop.systemd1.manage-units` action, and polkit is not |
| -referenced anywhere. §5 gives the agent a root channel for unit |
| -start/stop; wiring `UnitBackend` to it is a follow-up (§11). |
| +referenced anywhere. Whether `backing=unit` hibernation works on the |
| +fleet today is a follow-up discussion (§11); this spec's channel is |
| +scoped to the three instance templates and does not extend to |
| +`carlos-app@` (codex R1-19). |
| |
| --- |
| |
| @@ -139,52 +161,61 @@ start/stop; wiring `UnitBackend` to it is a follow-up (§11). |
| creates the flat files today. The registry's `DBPath` and `Socket` |
| columns point into the new layout. |
| |
| -2. **A template unit per role.** The instance runs as |
| +2. **No symlink is ever followed under the instance trees.** |
| + `/data/instances` and `/run/carlos/instances` are bind mounts of |
| + themselves with `nosymfollow` (§3.2). The kernel refuses to traverse a |
| + symlink there for every process on the box, agent and edge included, |
| + with no check-then-use race. Files the instance must never touch (the |
| + log sockets, the control channel, its token) live outside those two |
| + trees, where no tenant mount reaches. |
| + |
| +3. **A template unit per role.** The instance runs as |
| `carlos-instance@<host>.service`; its seed as |
| `carlos-instance-seed@<host>.service`; its sidecar as |
| `carlos-instance-sidecar@<host>.service` (§4). All three run as |
| - `User=carlos`, and all three carry titogo's namespace: |
| - `TemporaryFileSystem=/data:ro` and `TemporaryFileSystem=/run/carlos:ro` |
| + `User=carlos`, all three carry titogo's namespace |
| + (`TemporaryFileSystem=/data:ro` and `TemporaryFileSystem=/run/carlos:ro` |
| with exactly the instance's own two directories bound back in, |
| `PrivatePIDs=yes`, `ProtectProc=invisible`, `RestrictNamespaces=yes`, |
| - `IPAddressDeny=` for IMDS. A sibling's database does not exist in this |
| - namespace; a sibling's socket does not exist; a sibling's pid does not |
| - exist. |
| + `IPAddressDeny=` for IMDS), and all three sit in one per-host slice |
| + that carries the host's memory tier. A sibling's database does not |
| + exist in this namespace; a sibling's socket does not exist; a sibling's |
| + pid does not exist. |
| |
| -3. **The agent stays the owner.** Lease, restore, seed ordering, |
| +4. **The agent stays the owner.** Lease, restore, seed ordering, |
| replication, hibernation, the control socket, `carlos logs`, the |
| next-due tick, sidecar co-lifecycle and residence moves are unchanged |
| in `internal/activator`. What changes is the `Proc` the exec backend |
| hands back: it is a unit, not an `*exec.Cmd` (§6). |
| |
| -4. **One root channel.** The agent cannot start a system unit. A small |
| - root service, `carlos-instance-control`, socket-activated on a unix |
| - socket only uid `carlos` can open, starts and stops units named |
| - `carlos-instance*@<host>` and nothing else (§5). This is the |
| - synchronous form of the `carlos-stage.path` pattern. §5.4 records the |
| - alternative (polkit) and the question put to codex. |
| +5. **One root channel, authenticated.** The agent cannot start a system |
| + unit. A small root service, `carlos-instance-control`, socket-activated |
| + on a unix socket only uid `carlos` can open, starts and stops units |
| + named `carlos-instance*@<host>` and nothing else, and accepts a request |
| + only when it carries a token the agent reads from a path no tenant |
| + mount contains (§5). This is the synchronous form of the |
| + `carlos-stage.path` pattern. |
| |
| -5. **Launch through a trampoline.** Each unit's `ExecStart` is |
| +6. **Launch through a trampoline.** Each unit's `ExecStart` is |
| `/opt/carlos/carlos instance-exec <role> <host>`, which runs inside the |
| namespace, reads the launch description the agent wrote into the |
| instance directory, and `execve`s the app binary with the exact argv and |
| - environment the agent composed (§6.2). Environment composition stays in |
| - Go, in one place, testable, with no dependence on systemd's |
| - `EnvironmentFile=` parser agreeing with the agent's. |
| - |
| -6. **Resource walls move to a slice.** Instances leave the edge unit's |
| - cgroup tree. `carlos-instances.slice` becomes the outer wall; the tier |
| - (`512M` / `1G`) becomes a runtime property the control service sets on |
| - the unit before starting it; the sampler gains the slice as a scan |
| - root (§6.6). |
| - |
| -7. **The migration is a rename.** On the first agent start on the new |
| - build every exec instance is asleep (they died with the agent). Before |
| - the activator starts, the agent moves each exec route's flat files into |
| - its directory and updates the two registry columns in one compare-and- |
| - swap. No root step, no chown, no operator action (§8). |
| - |
| -8. **Addr routes are untouched, and now invisible.** A TCP-addr route's |
| + environment the agent composed (§6.2). |
| + |
| +7. **Resource walls become slices.** `carlos.slice` carries the |
| + 2026-08-28 outer wall; the edge unit and every instance sit under it; |
| + each host has its own slice inside that, carrying the tier (§4.3, |
| + §6.6). |
| + |
| +8. **The migration is a fenced rename.** On the first agent start on the |
| + new build every activator-managed instance is asleep (they died with |
| + the agent). Before the activator starts, the agent moves each such |
| + route's flat files into its directory under a per-host marker that |
| + every other consumer honours, and updates the two registry columns in |
| + one compare-and-swap. No root step, no chown, no operator action (§8). |
| + Routes the activator does not manage are not touched. |
| + |
| +9. **Addr routes are untouched, and now invisible.** A TCP-addr route's |
| `<host>.config.env` stays flat in `/data`, because the app's own |
| supervisor reads it there (titogo's `EnvironmentFile=-/data/%i.config.env`). |
| Exec children can no longer see it. That closes the read #127 leads |
| @@ -195,21 +226,31 @@ start/stop; wiring `UnitBackend` to it is a follow-up (§11). |
| |
| ## 3. The instance directory |
| |
| +### 3.1 Layout |
| + |
| ``` |
| -/data/instances/<host>/ 0700 carlos:carlos, created by the agent |
| - <host>.db the database (+ -wal, -shm) |
| - .activator-<host>.db-litestream/ the in-process replicator's shadow |
| - <host>.env admin token (EnsureAdminToken, unchanged) |
| - <host>-sidecar.env hand-staged sidecar env (unchanged) |
| - <host>.config.env syncexec's rendered bundle (unchanged) |
| - launch.run.json agent-written before each instance start (§6.2) |
| - launch.seed.json agent-written before a seed |
| - launch.sidecar.json agent-written before a sidecar start |
| - <host>.db.corrupt a quarantined restore, as today |
| -/run/carlos/instances/<host>/ 0700 carlos:carlos, created by the agent |
| - app.sock the instance's listener (rt.Socket) |
| - app.sock.control the agent's control listener (rt.Socket + ".control") |
| - log.sock the agent's log listener (§6.4) |
| +/data/instances/ bind mount of itself, nosymfollow (§3.2); 0755 carlos:carlos |
| +/data/instances/<host>/ 0700 carlos:carlos, created by the agent |
| + <host>.db the database (+ -wal, -shm) |
| + .activator-<host>.db-litestream/ the in-process replicator's shadow |
| + <host>.env admin token (EnsureAdminToken, unchanged) |
| + <host>-sidecar.env hand-staged sidecar env (unchanged) |
| + <host>.config.env syncexec's rendered bundle (unchanged) |
| + launch.run.json agent-written before each instance start (§6.2) |
| + launch.seed.json agent-written before a seed |
| + launch.sidecar.json agent-written before a sidecar start |
| + .migrating the migration fence (§8.1); absent when settled |
| + <host>.db.corrupt a quarantined restore, as today |
| +/run/carlos/instances/ bind mount of itself, nosymfollow; 0755 carlos:carlos |
| +/run/carlos/instances/<host>/ 0700 carlos:carlos, created by the agent |
| + app.sock the instance's listener (rt.Socket) |
| + app.sock.control the agent's control listener (rt.Socket + ".control") |
| +/run/carlos/logs/ 0700 carlos:carlos; NOT bound into any namespace |
| + <host>.run.sock the agent's log listener for the instance (§6.4) |
| + <host>.seed.sock ... for the seed |
| + <host>.sidecar.sock ... for the sidecar |
| +/run/carlos/instance-control.sock the root channel (§5); not in any namespace |
| +/run/carlos/instance-control.token 0600 carlos:carlos, root-written; not in any namespace |
| ``` |
| |
| **One helper, one owner.** `internal/provision` gains |
| @@ -230,6 +271,9 @@ func InstanceRunDir(host string) string { |
| func DefaultSocket(host string) string { |
| return filepath.Join(InstanceRunDir(host), "app.sock") |
| } |
| +func LogSocket(host, role string) string { |
| + return filepath.Join(SocketDir, "logs", host+"."+role+".sock") |
| +} |
| ``` |
| |
| Every DataDir-keyed builder in the tree moves onto these (the list is |
| @@ -262,33 +306,135 @@ that is the only data path the namespace reveals. `carlos add` and |
| `carlos route` refuse an exec route whose `--db` is elsewhere, naming the |
| directory. Existing rows are handled by the migration (§8.2). |
| |
| +### 3.2 Symlinks are never followed under the instance trees (codex R1-1, R1-2, R1-3) |
| + |
| +Two mount units, embedded and installed like every other unit |
| +(`internal/host/bootstrap.go:33-34`), and two `tmpfiles.d` lines beside |
| +the existing `/run/carlos` one (`bootstrap.go:61`): |
| + |
| +```ini |
| +# data-instances.mount |
| +[Unit] |
| +Description=carlos instance data (bind, nosymfollow) |
| +RequiresMountsFor=/data |
| +Before=carlos-edge.service carlos-litestream.service |
| +[Mount] |
| +What=/data/instances |
| +Where=/data/instances |
| +Type=none |
| +Options=bind,nosymfollow,nosuid,nodev,noexec |
| +[Install] |
| +WantedBy=local-fs.target |
| +``` |
| + |
| +```ini |
| +# run-carlos-instances.mount |
| +[Unit] |
| +Description=carlos instance sockets (bind, nosymfollow) |
| +After=systemd-tmpfiles-setup.service |
| +Before=carlos-edge.service |
| +[Mount] |
| +What=/run/carlos/instances |
| +Where=/run/carlos/instances |
| +Type=none |
| +Options=bind,nosymfollow,nosuid,nodev,noexec |
| +[Install] |
| +WantedBy=local-fs.target |
| +``` |
| + |
| +``` |
| +d /data/instances 0755 carlos carlos - |
| +d /run/carlos/instances 0755 carlos carlos - |
| +d /run/carlos/logs 0700 carlos carlos - |
| +``` |
| + |
| +`nosymfollow` (Linux 5.10+, util-linux 2.36+; Debian 13 ships 6.12 and |
| +2.41) makes path resolution on that mount return `ELOOP` at any symlink, |
| +for every process, in every namespace that sees the mount. It is a |
| +per-mount flag, so it holds through the `BindPaths=` bind into the |
| +instance's namespace and through the agent's and the edge's ordinary |
| +lookups from outside. `readlink` still works; nothing the platform runs |
| +needs a symlink under these trees (SQLite, litestream, the app's listener |
| +and the agent's env files are all plain files). Hard links cannot escape |
| +either: a `link()` needs a source path the caller can resolve, and inside |
| +the namespace no sibling path resolves. |
| + |
| +What this closes, race-free and for every reader at once: |
| + |
| +- the instance replacing its own `app.sock` with a symlink to the root |
| + control socket, a sibling's socket, or a sibling's control socket, and |
| + having the edge dial it (R1-1); |
| +- the instance replacing `<host>.env`, `<host>.config.env` or `<host>.db` |
| + with a symlink to a sibling's file, `/data/litestream.env` or any |
| + agent-readable path, and having the agent read it into the instance's |
| + environment or replicate it into the instance's replica (R1-3); |
| +- the instance planting a symlink at a predictable temp name so the |
| + agent's write truncates the target (R1-3). |
| + |
| +The log sockets and the control channel are additionally placed where no |
| +tenant mount reaches at all (`/run/carlos/logs`, `/run/carlos/instance-control.*`), |
| +so the question of a tenant touching them does not arise (R1-2). |
| + |
| +Belt and braces in the agent, cheap and testable: |
| + |
| +- every agent open of a file under an instance directory goes through |
| + one helper, `instancefs.Open(dir, name, flags)`, built on |
| + `openat2(2)` with `RESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS` |
| + (Linux 5.6+), so a symlink is refused with a named error even on a |
| + box where the mount option is missing; the same helper is what the |
| + agent uses to verify, at start, that both mounts carry `nosymfollow` |
| + (`/proc/self/mountinfo`), and it **refuses to wake any instance** until |
| + they do (§8.4); |
| +- every agent write into an instance directory creates its temp file |
| + with `O_CREAT|O_EXCL` and a random suffix, then `renameat`s over the |
| + target (rename replaces the directory entry and never follows the |
| + destination); |
| +- launch files (§6.2) are written the same way, and `instance-exec` |
| + reads them through the same helper. |
| + |
| --- |
| |
| ## 4. The units |
| |
| -Three template files under `internal/host/units/`, embedded and installed |
| -exactly as the existing nineteen are (`internal/host/bootstrap.go:33-34`, |
| -`systemapply.go:339-398`): `carlos-instance@.service`, |
| -`carlos-instance-seed@.service`, `carlos-instance-sidecar@.service`, plus |
| -`carlos-instances.slice`. Nothing enables the templates; an instance unit |
| +Unit files under `internal/host/units/`, embedded and installed exactly as |
| +the existing nineteen are (`bootstrap.go:33-34`, `systemapply.go:339-398`): |
| +`carlos-instance@.service`, `carlos-instance-seed@.service`, |
| +`carlos-instance-sidecar@.service`, `carlos.slice`, |
| +`carlos-instances.slice`, the two mount units of §3.2, and the two |
| +control units of §5.2. Nothing enables the templates; an instance unit |
| exists only while the agent has started it. |
| |
| +**Instance names are escaped hosts.** The unit instance is |
| +`systemd-escape(host)`: for a valid host (`[a-z0-9.-]`) that only turns |
| +`-` into `\x2d`. `%I` (unescaped) is used wherever the host is a path |
| +component; `%i` (escaped) wherever it is part of a unit name, which is |
| +what lets a per-host slice be spelled in the template (R1-8). The agent |
| +and the control service share one Go escape function (a copy of |
| +systemd's rules for this character set, pinned by a test against |
| +`systemd-escape` output for the hosts in the fixture set). |
| + |
| ### 4.1 `carlos-instance@.service` |
| |
| ```ini |
| [Unit] |
| -Description=carlos exec-backed instance %i |
| -# Stop and restart of the edge propagate here: an agent restart cycles |
| -# every instance, exactly as it did when instances were its children. |
| -PartOf=carlos-edge.service |
| -After=carlos-edge.service |
| +Description=carlos exec-backed instance %I |
| +# Stop-only coupling: when the edge stops (including the stop half of a |
| +# restart) this unit stops, and nothing starts it again but the agent. |
| +# PartOf= would propagate the RESTART and race the agent's own startup |
| +# (codex R1-7). |
| +BindsTo=carlos-edge.service |
| +After=carlos-edge.service data-instances.mount run-carlos-instances.mount |
| +Requires=data-instances.mount run-carlos-instances.mount |
| |
| [Service] |
| Type=simple |
| -ExecStart=/opt/carlos/carlos instance-exec run %i |
| +ExecStart=/opt/carlos/carlos instance-exec run %I |
| User=carlos |
| Group=carlos |
| -Slice=carlos-instances.slice |
| +# One slice per host holds the instance, its seed and its sidecar, and |
| +# carries the tier (§4.3, §6.6). Slices referenced here are created on |
| +# demand; no per-host unit file exists. |
| +Slice=carlos-instances-%i.slice |
| # The activator owns recycling: a crash stands the entry down and the |
| # next request wakes it. systemd must not race that. |
| Restart=no |
| @@ -296,17 +442,18 @@ Restart=no |
| # TimeoutStopSec; today's actStopTimeout is 20s. |
| KillMode=mixed |
| TimeoutStopSec=20 |
| -# The log stream: stdout and stderr connect to the agent's per-instance |
| -# listener (§6.4). systemd opens this as root, outside the namespace. |
| -StandardOutput=file:/run/carlos/instances/%i/log.sock |
| +# The log stream: stdout and stderr connect to the agent's per-role |
| +# listener (§6.4). PID 1 opens this as root, outside the namespace, at a |
| +# path no tenant mount contains. |
| +StandardOutput=file:/run/carlos/logs/%I.run.sock |
| StandardError=inherit |
| |
| # --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) --- |
| ProtectSystem=strict |
| TemporaryFileSystem=/data:ro |
| TemporaryFileSystem=/run/carlos:ro |
| -BindPaths=/data/instances/%i /run/carlos/instances/%i |
| -ReadWritePaths=/data/instances/%i /run/carlos/instances/%i |
| +BindPaths=/data/instances/%I /run/carlos/instances/%I |
| +ReadWritePaths=/data/instances/%I /run/carlos/instances/%I |
| ProtectHome=yes |
| PrivateTmp=yes |
| PrivateDevices=yes |
| @@ -330,26 +477,25 @@ RestrictSUIDSGID=yes |
| MemoryDenyWriteExecute=yes |
| SystemCallArchitectures=native |
| |
| -# --- resources (memory-pressure design 2026-08-28 §2.2-2.3, moved here) --- |
| -# Defaults; the control service overrides MemoryMax/MemoryHigh per start |
| -# with the route's tier (§6.6). |
| -MemoryMax=512M |
| -MemoryHigh=460M |
| +# --- per-service resources; the tier lives on the host slice (§4.3) --- |
| TasksMax=256 |
| -# The kernel picks a tenant before it picks anything else on the box. |
| +# The kernel picks a tenant before it picks anything else in carlos.slice. |
| OOMScoreAdjust=500 |
| -# One OOM kill takes the whole family (memory.oom.group, as the exec |
| -# cgroup sets today). |
| +# One OOM kill takes every process of this service (memory.oom.group). |
| +# The sidecar is a sibling service; the activator co-stops it when the |
| +# instance exits, as it does today (activator.go:970-984). |
| OOMPolicy=kill |
| ``` |
| |
| Lines that are load-bearing and why: |
| |
| -- **`TemporaryFileSystem=/data:ro` + `BindPaths=/data/instances/%i`** is |
| +- **`TemporaryFileSystem=/data:ro` + `BindPaths=/data/instances/%I`** is |
| the isolation. The tmpfs is empty; the bind reveals one directory. A |
| sibling's database is not permission-denied, it does not exist. The |
| same for `/run/carlos`: a sibling's listening socket and control socket |
| - do not exist, which closes the socket-file impersonation in §1.2. |
| + do not exist, which closes the socket-file impersonation in §1.2. The |
| + bound directory is inside the `nosymfollow` mount (§3.2), so a symlink |
| + the instance plants there is dead for every reader. |
| - **The bind sources must exist before the unit starts**, or systemd |
| fails at namespace setup before any `ExecStartPre=` runs (titogo |
| verified this; it is why `titogo-instance-dir@.service` exists). Here |
| @@ -361,65 +507,86 @@ Lines that are load-bearing and why: |
| own pid namespace an instance's `/proc` lists only itself: no sibling |
| pid to traverse into `/proc/<pid>/root/`, no sibling to signal, nothing |
| for a same-uid ptrace to attach to. `PrivatePIDs=` needs systemd 257 |
| - (Debian 13, which `infra/modules/region-host/variables.tf:134` pins); |
| - §8.4 verifies it on every box before the roll. |
| + (Debian 13, which `infra/modules/region-host/variables.tf:134` pins) |
| + and is **ignored, not refused**, on a kernel that cannot provide it, so |
| + §8.4 gates on observed behaviour, not on the directive (R1-15). |
| - **`RestrictNamespaces=yes`**: without it an unprivileged user namespace |
| could bind-mount over a fresh mountpoint and re-reveal what the tmpfs |
| hid. |
| - **`IPAddressDeny=`** for IMDS. The nft rule in pinfra allows uid |
| `carlos` because the agent needs IMDS; this line is what stops an |
| instance, also uid `carlos`, from fetching the box's role credentials. |
| -- **`PartOf=carlos-edge.service`** keeps today's semantics: restarting |
| - the edge cycles every instance. Without it, an agent restart would leave |
| - units running that no `actEntry` knows about. §6.7 adds the startup |
| - sweep that covers the case where `PartOf=` did not fire (a crashed |
| - agent, `OOMPolicy=continue` keeping the edge up). |
| +- **`BindsTo=carlos-edge.service`** keeps today's "instances die with the |
| + agent": when the edge stops, so do they, and nothing restarts them but |
| + the agent's next wake. §6.7 adds the startup sweep for the case where |
| + the coupling did not fire. |
| - **`StandardOutput=file:` to an `AF_UNIX` socket** is documented |
| systemd behaviour: a stream connection is made and used as stdout. It |
| - is opened by PID 1 outside the namespace, so the path is the real one. |
| - §6.4. |
| + is opened by PID 1 outside the namespace, so the path is the real one, |
| + and the path is in `/run/carlos/logs`, which no tenant mount contains. |
| - **`/opt/carlos`** stays readable through `ProtectSystem=strict`; the |
| trampoline and the live symlink are all the instance needs from it. |
| |
| ### 4.2 `carlos-instance-seed@.service` and `carlos-instance-sidecar@.service` |
| |
| -Identical `[Service]` hardening and namespace blocks. Differences: |
| +Identical `[Unit]` coupling, `[Service]` namespace, hardening and |
| +resource blocks, and the same `Slice=carlos-instances-%i.slice`. |
| +Differences: |
| |
| | | seed | sidecar | |
| |---|---|---| |
| | `Type=` | `oneshot` | `simple` | |
| -| `ExecStart=` | `/opt/carlos/carlos instance-exec seed %i` | `/opt/carlos/carlos instance-exec sidecar %i` | |
| -| `StandardOutput=` | `file:/run/carlos/instances/%i/log.sock` | same | |
| -| `PartOf=` | `carlos-edge.service` | `carlos-edge.service` | |
| -| memory | shares the instance's tier via the slice; no per-unit override | same | |
| +| `ExecStart=` | `/opt/carlos/carlos instance-exec seed %I` | `/opt/carlos/carlos instance-exec sidecar %I` | |
| +| `StandardOutput=` | `file:/run/carlos/logs/%I.seed.sock` | `file:/run/carlos/logs/%I.sidecar.sock` | |
| +| timeouts | `TimeoutStartSec=120` (today's seed budget, `backend_exec.go:368-408`; a oneshot has no start timeout by default, R1-5), `TimeoutStopSec=20` | `TimeoutStopSec=20` | |
| |
| -A Go test over the embedded FS asserts that the three files' namespace |
| -and hardening lines are byte-identical (§10). Three files rather than one |
| -because seed and instance never run together but sidecar and instance |
| -do, and a template's `%i` is the host, not a role. |
| +A Go test over the embedded FS asserts that the three files' namespace, |
| +hardening and coupling lines are byte-identical (§10). Three files rather |
| +than one because seed and instance never run together but sidecar and |
| +instance do, and a template's instance is the host, not a role. |
| |
| -### 4.3 `carlos-instances.slice` |
| +### 4.3 Slices and the memory walls (codex R1-8, R1-10) |
| |
| ```ini |
| +# carlos.slice — the outer wall of the 2026-08-28 design, as a slice |
| [Unit] |
| -Description=carlos exec-backed instances (outer memory wall) |
| +Description=carlos (edge, activator and every exec-backed instance) |
| +[Slice] |
| +MemoryHigh=70% |
| +MemoryMax=80% |
| +``` |
| |
| +```ini |
| +# carlos-instances.slice — the parent of every per-host slice |
| +[Unit] |
| +Description=carlos exec-backed instances |
| [Slice] |
| -MemoryHigh=50% |
| -MemoryMax=60% |
| +# No cap of its own: the outer wall is carlos.slice, the inner walls are |
| +# the per-host slices. This level exists so the sampler has one root to |
| +# scan and so an operator can `systemctl status carlos-instances.slice`. |
| ``` |
| |
| -And `carlos-edge.service` drops from `MemoryHigh=70% MemoryMax=80%` to |
| -`MemoryHigh=15% MemoryMax=20%`. The 2026-08-28 design set 80% for |
| -"agent plus every awake instance" so that 20% remained for the box's |
| -other units. Instances now sit in a sibling cgroup, so the wall splits: |
| -60% for the instances, 20% for the agent (about 110 MB resident; 20% of |
| -a t4g.micro is 200 MB), the same 20% outside both. **This split is a |
| -review target**, not a measurement: the agent's ceiling should be |
| -checked against its RSS on the flagship before the roll. `OOMPolicy=continue` |
| -stays on the edge unit (it no longer matters for tenants, and it costs |
| -nothing). The `oom_score_adj` write in `internal/cgroup` |
| -(`ExecOOMScoreAdj`) becomes `OOMScoreAdjust=500` in the template. |
| +`carlos-edge.service` gains `Slice=carlos.slice` and **drops** its own |
| +`MemoryHigh=70% MemoryMax=80%`: the wall moves up one level and now |
| +bounds exactly what it bounded before, the agent plus every awake |
| +instance, so borrowing between them and the OOM candidate set (a tenant |
| +at `+500` is always picked before the agent) are unchanged. The |
| +2026-08-28 reasoning, including the measured 1.2 GB leaf that was mostly |
| +reclaimable page cache, carries over without new numbers. |
| +`OOMPolicy=continue` stays on the edge unit. |
| + |
| +The per-host slice `carlos-instances-<esc>.slice` is created on demand |
| +by the first unit that names it. The control service sets |
| +`MemoryMax`/`MemoryHigh` on it, from the route's tier, before every |
| +start (§5.3, §6.6): `512M`/`460M` by default, `1G`/`920M` for an app whose |
| +manifest declares `capabilities.children`, exactly the values |
| +`internal/cgroup` writes today. Seed, instance and sidecar all charge |
| +that one cgroup, which is what today's shared `carlos-exec@<host>` gives |
| +them (`backend_exec.go:149-155, 373-375, 890-895`). |
| + |
| +The cgroup path is |
| +`/sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/` |
| +(R1-9). |
| |
| --- |
| |
| @@ -438,13 +605,14 @@ waiting HTTP request. |
| |
| ```ini |
| # carlos-instance-control.socket |
| +[Unit] |
| +Description=carlos instance control socket (agent → root) |
| [Socket] |
| ListenStream=/run/carlos/instance-control.sock |
| SocketUser=carlos |
| SocketGroup=carlos |
| SocketMode=0600 |
| Accept=no |
| - |
| [Install] |
| WantedBy=sockets.target |
| ``` |
| @@ -453,13 +621,15 @@ WantedBy=sockets.target |
| # carlos-instance-control.service |
| [Unit] |
| Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent) |
| -Requires=carlos-instance-control.socket |
| +Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount |
| +After=data-instances.mount run-carlos-instances.mount |
| |
| [Service] |
| ExecStart=/opt/carlos/carlos instance-control |
| # Root deliberately: the whole job is systemctl start/stop, and the |
| # caller is the unprivileged agent. Same rationale as carlos-stage.service. |
| ProtectSystem=strict |
| +ReadWritePaths=/run/carlos |
| ProtectHome=yes |
| PrivateTmp=yes |
| PrivateDevices=yes |
| @@ -472,80 +642,79 @@ MemoryMax=64M |
| ``` |
| |
| The socket is owned by `carlos` with mode 0600 and the unit is socket- |
| -activated, so only the agent can reach it and it runs only while there is |
| -work. Instances cannot reach it: `/run/carlos` inside their namespace is |
| -an empty tmpfs with their own directory bound in; the control socket is |
| -not in it. |
| +activated, so only the agent's uid can open it and it runs only while |
| +there is work. It is not in any instance's namespace: `/run/carlos` |
| +inside an instance is an empty tmpfs with the instance's own directory |
| +bound in. |
| |
| ### 5.3 Protocol |
| |
| HTTP/1.1 over the unix socket, the shape `edge.ListenControl` already |
| -uses for the per-instance control socket. Four verbs, no shell, no |
| -argument passes through to `systemctl` except a unit name the service |
| -built itself: |
| +uses for the per-instance control socket. Every request carries |
| +`Authorization: Bearer <token>`; every response to a request without the |
| +right token is 401 and nothing runs (§5.4 says why). Five verbs, no |
| +shell, no argument passes through to `systemctl` except a unit name the |
| +service built itself: |
| |
| | Verb | What it runs | Returns | |
| |---|---|---| |
| -| `POST /start?host=H&role=run\|sidecar&memory_max=N&memory_high=N` | `systemctl set-property --runtime carlos-instance[-sidecar]@H.service MemoryMax=N MemoryHigh=N` then `systemctl start` (blocking: the start job completes) | 204, or 502 with systemctl's stderr | |
| -| `POST /seed?host=H` | `systemctl start carlos-instance-seed@H.service` (oneshot: blocks until exit), then `systemctl show -p Result,ExecMainStatus` | 200 with `{result, status}`; a failed seed is a 200 whose result says so, not a 502 | |
| -| `POST /stop?host=H&role=run\|sidecar` | `systemctl stop` (blocking) | 204 | |
| -| `POST /reset?host=H&role=…` | `systemctl reset-failed` | 204 | |
| +| `POST /start?host=H&role=run\|sidecar&memory_max=N&memory_high=N` | `systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=N`; `systemctl reset-failed <unit>`; `systemctl start <unit>` (blocking: the start job completes) | 200 with `{invocation_id}` from `systemctl show -p InvocationID`, or 502 with systemctl's stderr | |
| +| `POST /seed?host=H&memory_max=N&memory_high=N` | the same three, on `carlos-instance-seed@<esc>.service` (oneshot: `start` blocks until exit or `TimeoutStartSec`), then `systemctl show -p Result,ExecMainStatus,ExecMainCode,InvocationID` | 200 with `{result, status, code, invocation_id}`; a failed seed is a 200 whose result says so, not a 502 | |
| +| `POST /stop?host=H&role=run\|seed\|sidecar` | `systemctl stop <unit>` (blocking; also cancels a pending start job) | 204 | |
| +| `POST /reset?host=H&role=…` | `systemctl reset-failed <unit>` | 204 | |
| +| `GET /ready` | nothing | 204 once the service has verified both mounts carry `nosymfollow` and `PrivatePIDs=` is effective (§8.4); 503 with the reason otherwise | |
| |
| -Validation, in order: `host` must satisfy `provision.ValidHostShape` |
| +Validation, in order: token; `host` must satisfy `provision.ValidHostShape` |
| (`provision.go:262`); `role` must be one of the three literals; the |
| memory values must parse as `Limit` values the agent already knows how to |
| produce (`internal/cgroup`, `TierDefault`/`TierChildren`). Anything else |
| is a 400 with no systemctl call. The unit name is |
| -`"carlos-instance" + suffix + "@" + host + ".service"`, built from the |
| -validated pieces, never from the request string. |
| +`"carlos-instance" + suffix + "@" + escape(host) + ".service"` and the |
| +slice name `"carlos-instances-" + escape(host) + ".slice"`, built from |
| +the validated pieces, never from the request string. |
| |
| -What this grants the agent: start, stop and memory-tier control of units |
| -named `carlos-instance*@<host>`. That is exactly what it has today over |
| -its own children. It grants nothing else, and a host that does not exist |
| -is a unit whose bind source does not exist, which fails at namespace |
| -setup and starts nothing. |
| +What this grants the agent: start, stop, reset and memory-tier control of |
| +units named `carlos-instance*@<host>` and their host slice. That is |
| +exactly what it has today over its own children. It grants nothing else, |
| +and a host that does not exist is a unit whose bind source does not |
| +exist, which fails at namespace setup and starts nothing. |
| |
| -`systemctl set-property --runtime` on a template instance that has not |
| -started yet writes a runtime drop-in under `/run/systemd/system.control/` |
| -that the following `start` honours; the control service runs the two in |
| -that order and the plan's first task verifies it on a box. |
| +`systemctl set-property --runtime` on an instance of an installed |
| +template that has never been started loads the unit first and writes a |
| +runtime drop-in the following `start` honours (`SetUnitProperties` in |
| +`src/core/dbus-manager.c`; confirmed by codex R1-1). The same holds for |
| +a slice that does not yet exist. The plan's first task exercises both on |
| +a dev box. |
| |
| -### 5.4 The choice put to codex: this service, or polkit |
| +### 5.4 The choice put to codex, and the ruling |
| |
| The other way to let uid `carlos` start these units is polkit: a rules |
| file granting `org.freedesktop.systemd1.manage-units` to `carlos` when |
| -the unit name matches `carlos-instance*@*.service`, after which the |
| -agent's existing bare `systemctl` calls would work. |
| - |
| -Fable's advisement is the control service, for four reasons: |
| - |
| -1. **Polkit is not part of the platform.** Nothing in this tree or in |
| - pinfra references it, and no install step puts it on a box (whether |
| - the Debian 13 cloud image happens to carry `polkitd` is to be checked, |
| - not assumed). Relying on it means a daemon with a JavaScript rules |
| - engine on an image whose every other privileged path is a `carlos` |
| - verb in one binary. |
| -2. **A rules file is a second install path.** Units reach the box through |
| - `embed.FS` and `installUnits` (`systemapply.go:517-589`) and through |
| - `update-platform-binary.sh`'s `cp` of the staged units. A file under |
| - `/etc/polkit-1/rules.d` is neither; it would need its own install, |
| - its own drift check and its own rollback. |
| -3. **Start needs a tier and seed needs a wait.** `set-property` is a |
| - separate polkit action (`manage-unit-files` / `set-environment` |
| - family); the seed's oneshot result needs a `show` after the start. |
| - The control service carries all of that in one Go handler with one |
| - test file; polkit would carry it in a rules file plus the same Go. |
| -4. **It is the existing pattern, made synchronous.** `carlos-stage.service` |
| - and `carlos-restart.service` are root oneshots that run a `carlos` verb |
| - on the agent's request. Operators already know what those are for. |
| - |
| -The honest cost: a root-owned HTTP server whose input is a host name |
| -from the agent. The mitigations are the grammar check, the fixed verb |
| -set, the absence of any shell, the 0600 socket, and the fact that the |
| -agent is already the box's control plane. **Codex is asked, in the review |
| -brief, to advise for or against this choice with reasons, and to look for |
| -a way an instance could reach the control socket or make the agent send |
| -a request it did not intend.** |
| +the unit name matches `carlos-instance*@*.service`. Codex R1-1 confirmed |
| +that polkit can express the unit and verb restrictions (systemd passes |
| +`unit` and `verb`, including `set-property` and `reset-failed`, under |
| +that one action; the draft's claim that properties need a separate |
| +action was wrong), and also that it cannot constrain property values, |
| +that a uid-only grant is reachable by every tenant through the system |
| +bus, and that a wildcard on names also authorises transient units. |
| + |
| +Codex's advisement, concurring with the draft's: **the control service, |
| +provided it authenticates the request and not the socket's uid.** The |
| +reason is the proxy attack in §1.3: the edge dials `rt.Socket` as uid |
| +`carlos` in its own namespace, so before §3.2 an instance could point its |
| +own socket path at the control socket and have the edge deliver an |
| +attacker-written `POST /stop?host=<sibling>` with a perfectly authorised |
| +peer uid. A peer-uid check cannot tell that request from the agent's |
| +own. `nosymfollow` removes the redirection; the token removes the class: |
| +a request is the agent's only if it carries the token, and the token is |
| +readable only from `/run/carlos/instance-control.token`, root-written at |
| +service start, `0600 carlos:carlos`, at a path no instance namespace |
| +contains and the edge never serves. The agent reads it once at start and |
| +on 401. A tenant that could learn it would already have to be reading |
| +arbitrary agent-readable files, which is the property §3.2 denies. |
| + |
| +Ruled, 2026-09-11: control service with bearer token; polkit is not |
| +installed and not relied on. |
| |
| --- |
| |
| @@ -554,25 +723,31 @@ a request it did not intend.** |
| `internal/activator/activator.go` does not change. `doWake` |
| (`activator.go:772`), `stop` (`:962-1047`), `Bounce` (`:946-956`), |
| `sweepOnce` (`:1082`) and the `Proc`/`Rep` seams (`:194-207`) keep their |
| -contracts. The changes are inside `ExecBackend`. |
| - |
| -### 6.1 `Start` |
| - |
| -1. Ensure `InstanceDir` and `InstanceRunDir` exist, 0700. Clear any stale |
| - `app.sock`, `app.sock.control` and `log.sock` in the run dir (an |
| - earlier process is gone: the unit is not active, or we would not be |
| - here). |
| -2. `EnsureAdminToken(InstanceDir/<host>.env)` — unchanged. |
| -3. `instanceEnv` — unchanged, reading the two files from `InstanceDir`. |
| -4. Bind the control listener at `rt.Socket + ".control"` — unchanged |
| - (`edge.ListenControl`), now inside the run dir. |
| -5. Bind the log listener at `InstanceRunDir/log.sock` (§6.4). |
| -6. Write `launch.run.json` (§6.2). |
| -7. `POST /start` on the control socket with the route's tier. |
| -8. `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` — unchanged. On |
| - failure, `POST /stop` and clean up, exactly as the current failure |
| - path stops the child. |
| -9. Return a `unitProc`. |
| +contracts. In particular `sweepOnce` keeps calling `Exited()` under the |
| +activator mutex and `Exited()` stays a non-blocking read of a flag |
| +(R1-4); everything that talks to systemd runs in the proc's own |
| +goroutine. The changes are inside `ExecBackend`. |
| + |
| +### 6.1 Preparing a role, and `Start` |
| + |
| +One helper, `prepare(rt, role)`, runs before **any** role starts, seed |
| +included (R1-5): |
| + |
| +1. Ensure `InstanceDir` and `InstanceRunDir` exist, 0700, and `/run/carlos/logs`. |
| +2. For `run`: `EnsureAdminToken(InstanceDir/<host>.env)` (unchanged), |
| + `instanceEnv` (unchanged, reading the two files from `InstanceDir` |
| + through `instancefs`), bind the control listener at |
| + `rt.Socket + ".control"` (unchanged, `edge.ListenControl`), and clear |
| + any stale `app.sock` (nothing is running: the unit is not active, or |
| + we would not be here). |
| +3. Bind the role's log listener at `LogSocket(host, role)` (§6.4). |
| +4. Write `launch.<role>.json` (§6.2). |
| + |
| +`Start` is then: `prepare(rt, "run")`; `POST /start` with the route's |
| +tier; `waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout)` (unchanged); |
| +on failure `POST /stop`, wait for the proc's `done`, and clean up, |
| +exactly as the current failure path stops the child; return a `unitProc` |
| +carrying the invocation id. |
| |
| ### 6.2 The trampoline and `launch.<role>.json` |
| |
| @@ -585,14 +760,30 @@ contracts. The changes are inside `ExecBackend`. |
| ``` |
| |
| `carlos instance-exec <role> <host>` runs as `ExecStart`, inside the |
| -namespace, as `carlos`. It reads `/data/instances/<host>/launch.<role>.json`, |
| -refuses a schema it does not know, refuses an `argv[0]` that is not |
| -`/opt/carlos/live/<host>` (defence in depth: the file is agent-written and |
| -0600, but the check is one line), and calls `syscall.Exec(argv[0], argv, env)`. |
| -It never parses env syntax: the agent's `childEnv` (`backend_exec.go:802-814`) |
| -composes the list exactly as it does today, including the agent-owned |
| -removal of tenant-supplied `CARLOS_CONTROL_SOCKET` / `CARLOS_PURGE` |
| -(`backend_exec.go:636-660`), and the trampoline passes it through. |
| +namespace, as `carlos`. It reads `/data/instances/<host>/launch.<role>.json` |
| +through `instancefs`, refuses a schema it does not know, refuses an |
| +`argv[0]` that is not `/opt/carlos/live/<host>` (defence in depth: the |
| +file is agent-written; the check is one line), and calls |
| +`syscall.Exec(argv[0], argv, env)`. It never parses env syntax: the |
| +agent's `childEnv` (`backend_exec.go:802-814`) composes the list exactly |
| +as it does today, including the agent-owned removal of tenant-supplied |
| +`CARLOS_CONTROL_SOCKET` / `CARLOS_PURGE` (`backend_exec.go:636-660`), and |
| +the trampoline passes it through. |
| + |
| +**One entry per key, last value wins** (R1-16). `os/exec` deduplicates |
| +`cmd.Env` in favour of the last occurrence; `syscall.Exec` passes |
| +duplicates through and a Go child keeps the first. `childEnv` therefore |
| +gains the deduplication itself (a map keyed on the name, insertion order |
| +preserved), and the test asserts the effective value *inside* an |
| +executed stub, not the JSON array. |
| + |
| +What the current `exec.Command` path does that the trampoline must |
| +reproduce, and how: working directory (none is set today, and none is |
| +set; the app starts in `/`), inherited fds (none beyond stdio today; |
| +`syscall.Exec` inherits whatever the trampoline holds, which is stdio |
| +only), `WaitDelay` (replaced by `TimeoutStopSec`, §6.4), `Pdeathsig` |
| +(replaced by `BindsTo=` plus the startup sweep, §6.7), |
| +`oom_score_adj` (`OOMScoreAdjust=`), the cgroup placement (`Slice=`). |
| |
| Why not `EnvironmentFile=`: systemd's parser and the agent's |
| `parseEnvBytes` differ on quoting and continuation, and the agent-owned |
| @@ -600,15 +791,23 @@ variables would then have to be written into a file the tenant's |
| `<host>.env` overlay must not win over. One composer, one format, one |
| test. |
| |
| -The launch file is written fresh before every start and holds the same |
| -material the child could already read from `/proc/self/environ`. It is |
| -inside the instance's own directory, visible to that instance and the |
| -agent, nobody else. |
| +The launch file is written fresh before every start, `O_EXCL` temp plus |
| +rename (§3.2), and holds the same material the child could already read |
| +from `/proc/self/environ`. It is inside the instance's own directory, |
| +visible to that instance and the agent, nobody else. An instance can |
| +rewrite its own launch file while it runs; the next start reads it again |
| +through `instancefs`, refuses any `argv[0]` but its own live link, and |
| +executes only what a tenant could execute anyway: its own binary with an |
| +argv and env it chose. That is not a privilege. |
| |
| Seed: `launch.seed.json` carries the argv from `rt.Seed` |
| (`backend_exec.go:332-338`) with the DB path substituted as today; `POST /seed` |
| -blocks; the result's `status` is the seed's exit code; a non-zero exit |
| -runs the same cleanup as today (`backend_exec.go:438`). |
| +blocks up to `TimeoutStartSec`; the result's `status` is the seed's exit |
| +code; a non-zero exit or a timeout runs the same cleanup as today |
| +(`backend_exec.go:438`) **after** `POST /stop?role=seed` has returned and |
| +the seed proc's `done` has closed, so the lease is never released |
| +(`activator.go:824-828`) while a seed can still write (R1-5). A |
| +cancelled `ctx` during a seed does the same stop-and-wait. |
| |
| Sidecar: `launch.sidecar.json` carries `[live/<host>, "sidecar", "run"]` |
| and the sidecar env from `<host>-sidecar.env` only, as today |
| @@ -618,20 +817,21 @@ and the sidecar env from `<host>-sidecar.env` only, as today |
| |
| ```go |
| type unitProc struct { |
| - host string |
| - role string |
| - ctl *edge.ControlListener // as childProc.ctl today |
| - logs *logListener // §6.4 |
| - exited atomic.Bool |
| - done chan struct{} |
| + host, role string |
| + invocation string // from the start response |
| + ctl *edge.ControlListener // run only; as childProc.ctl today |
| + logs *logListener // §6.4 |
| + exited atomic.Bool |
| + done chan struct{} |
| } |
| func (p *unitProc) Stop(ctx context.Context) error // POST /stop, then wait for done or ctx |
| -func (p *unitProc) Exited() bool // §6.5 |
| +func (p *unitProc) Exited() bool // p.exited.Load() |
| ``` |
| |
| The control listener's close ordering (`backend_exec.go:992-1009`: signal |
| first, close the listener only after the process is gone) is preserved: |
| `Stop` posts the stop, waits for `done`, then closes `ctl` and `logs`. |
| +`done` closes only on proven termination (§6.5), never on log EOF. |
| |
| ### 6.4 Logs |
| |
| @@ -641,50 +841,54 @@ and, with `Logs` set, line-wise into `Logs.App` for `carlos logs`, capped |
| at `execLogLineMax`, with `WaitDelay` bounding a grandchild that holds |
| the pipe. |
| |
| -The unit connects its stdout and stderr to `log.sock` |
| -(`StandardOutput=file:` on an `AF_UNIX` path). The agent's `logListener` |
| -accepts exactly one connection per start (a second is closed: the unit |
| -is the only thing that should connect, and it connects once), and feeds |
| -it to the same `applog.NewLineWriter` and the same `io.MultiWriter` to |
| -the agent's stdout. Same bytes, same cap, same sink, same `carlos logs`. |
| -The listener is bound before `POST /start` so PID 1's connect cannot |
| -race it; it lives in the run dir, so only this instance can reach it, |
| -and a connect from inside the namespace is harmless (it would only be |
| -writing its own log). |
| +Each role's unit connects its stdout and stderr to that role's socket in |
| +`/run/carlos/logs` (`StandardOutput=file:` on an `AF_UNIX` path; R1-2, |
| +R1-6). The agent's `logListener` for a role accepts one connection per |
| +start (the unit connects exactly once, at exec; a second connection |
| +during the same invocation is closed and logged, and can only come from |
| +PID 1 since no tenant mount contains the path), and feeds it to the same |
| +`applog.NewLineWriter` and the same `io.MultiWriter` to the agent's |
| +stdout. Same bytes, same cap, same sink, same `carlos logs`, and the |
| +instance and its sidecar no longer contend for one connection. The |
| +listener is bound in `prepare` before the start request, so PID 1's |
| +connect cannot race it. |
| |
| Grandchildren: a process the app forked that outlives it keeps the |
| socket open, as it keeps the pipe open today. `KillMode=mixed` plus |
| `TimeoutStopSec=20` kills it on stop; on a crash the unit's cgroup empties |
| -when systemd reaps it, and EOF follows. The `WaitDelay` grace becomes the |
| -unit's stop timeout. |
| - |
| -### 6.5 Exit observation |
| - |
| -`childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`). |
| -`unitProc` has two signals and uses both: |
| - |
| -- **EOF on the log connection**, which fires when the last process |
| - holding the unit's stdout closes it: the fast path, and the one that |
| - matches today's "pipe closed" semantics. |
| -- **`systemctl show -p ActiveState,SubState,Result,ExecMainStatus,ExecMainCode <unit>`**, |
| - read-only and unprivileged, polled by `sweepOnce` for every awake entry |
| - (the sweeper already visits each entry every `Idle/4`; one `show` per |
| - visit is cheap and can be batched with `systemctl show` on several |
| - units in one call). |
| - |
| -Either signal marks the proc exited, emits the member-facing event line |
| -with the same wording as today (`instance exited: ...` with the exit |
| -status or signal from `ExecMainStatus`/`ExecMainCode`, or |
| -`instance exited (exit 0)`), and closes `done`. The activator's existing |
| -"an exited line with no stopping line before it is a crash" reading is |
| -unchanged. |
| +when systemd reaps it. EOF on the log connection ends **log collection |
| +for that invocation**; it does not mean the process is gone (R1-4). |
| + |
| +### 6.5 Exit observation (codex R1-4) |
| + |
| +`childProc` learns of exit from `cmd.Wait()` (`backend_exec.go:722-736`), |
| +which is process death. A unit proc needs the same certainty, from |
| +systemd, attributed to the right invocation. Each `unitProc` runs one |
| +goroutine: |
| + |
| +1. **Fast signal**: inotify (`IN_MODIFY`, or `poll` for `POLLPRI`) on the |
| + unit's `cgroup.events` file under the host slice; `populated 0` means |
| + every process of the unit is gone. cgroup v2 files are world-readable, |
| + so the agent reads them as `carlos`, as the sampler does today. |
| +2. **Confirmation**: `systemctl show -p ActiveState,SubState,Result,ExecMainStatus,ExecMainCode,InvocationID <unit>` |
| + (read-only, unprivileged). The proc is exited when `ActiveState` is |
| + `inactive` or `failed` **and** the `InvocationID` matches the one the |
| + start returned (a newer invocation is a different proc). A fallback |
| + poll of the same `show` every 2s covers a missed inotify event. |
| +3. On confirmation: emit the member-facing event line with today's |
| + wording (`instance exited: ...` with the status or signal from |
| + `ExecMainStatus`/`ExecMainCode`, or `instance exited (exit 0)`), |
| + set `exited`, close `done`. |
| + |
| +The activator's "an exited line with no stopping line before it is a |
| +crash" reading is unchanged, and `sweepOnce`'s existing dead-child |
| +check (`activator.go:1105-1122`) sees a flag, never a blocking call. |
| |
| `Restart=no` in the template is what makes this sound: systemd never |
| restarts the unit behind the activator's back; a failed unit stays |
| -`failed` until the next `start`, and `POST /start` runs `reset-failed` |
| -first so a failed state is never a reason a wake cannot proceed. |
| +`failed` until the agent's next `start`, which `reset-failed`s first. |
| |
| -### 6.6 cgroups, tiers and the sampler |
| +### 6.6 cgroups, tiers and the sampler (codex R1-8, R1-9) |
| |
| `internal/cgroup` today: `Manager.Start` (`cgroup.go:242-285`) clones the |
| child into `carlos-exec@<host>` under the edge unit's delegated subtree, |
| @@ -692,43 +896,50 @@ writes `memory.max`/`memory.high` from the tier and `memory.oom.group=1`, |
| and the sampler reads the subtree as a scan root (exec-child accounting |
| design §4). `CleanupHost` (`:398-403`) removes the directory. |
| |
| -Under units, systemd owns the cgroup: |
| -`/sys/fs/cgroup/carlos-instances.slice/carlos-instance@<host>.service/` |
| -and `.../carlos-instance-sidecar@<host>.service/`. So: |
| - |
| -- **Tier** travels in `POST /start` and lands as `MemoryMax`/`MemoryHigh` |
| - via `set-property --runtime`. The `Tier func(registry.Route) cgroup.Limit` |
| - seam and its box wiring (`stagedManifestOfLiveVersion`) are unchanged; |
| - only the consumer changes. The agent logs `instance: <host>: memory.max=…` |
| - on every start as it does today. |
| -- **Sampler** gains the slice as a scan root: a host's charge is the sum |
| - of its `carlos-instance@` and `carlos-instance-sidecar@` units, which |
| - keeps "a sidecar's load is that instance's work". cgroup v2 files are |
| - world-readable, so the agent reads them as `carlos` exactly as it reads |
| - the delegated subtree now. |
| +Under units, systemd owns the cgroup. So: |
| + |
| +- **Tier** travels in `POST /start` / `POST /seed` and lands as |
| + `MemoryMax`/`MemoryHigh` on the host slice via `set-property --runtime`. |
| + The `Tier func(registry.Route) cgroup.Limit` seam and its box wiring |
| + (`stagedManifestOfLiveVersion`) are unchanged; only the consumer |
| + changes. The agent logs `instance: <host>: memory.max=…` on every start |
| + as it does today. Seed, instance and sidecar share the one cap. |
| +- **Sampler** gains |
| + `/sys/fs/cgroup/carlos.slice/carlos-instances.slice/` as a scan root |
| + and reads each `carlos-instances-<esc>.slice/memory.current` (the |
| + slice's charge is the family's total, so no per-role summation is |
| + needed). It **publishes under the existing key** |
| + `activator.ExecName(host)` (`carlos-exec@<host>`), because that is the |
| + key `internal/console/metricsview.go:728-762` recognises and everything |
| + else is discarded at the app boundary there. The discovery code is |
| + `internal/host/metrics_probes.go`; both files are consumers (§9). The |
| + console therefore shows the same per-host numbers with no change. |
| - **`Manager.Start`, `CleanupHost`, the clone3 path and the delegation |
| dance** are no longer used by the exec backend. They stay in the tree |
| - until the last exec spawn is gone (`Seed` and sidecars move in the same |
| - change, so nothing uses them after it); deleting the package's spawn |
| - half is a follow-up once a release has run without it. |
| -- **`Delegate=` on the edge unit** stays for one release for the same |
| - reason, then goes. |
| + until a release has run without them; deleting the package's spawn |
| + half is a follow-up. **`Delegate=` on the edge unit** stays for the |
| + same release, then goes. |
| |
| -### 6.7 Agent start and restart |
| +### 6.7 Agent start and restart (codex R1-7) |
| |
| Today an agent restart kills every child (`Pdeathsig`, the cgroup tree |
| -dies with the unit). `PartOf=carlos-edge.service` reproduces that for a |
| -`systemctl restart carlos-edge`. For a crash, or a stop that did not |
| -propagate, the agent on start runs |
| -`systemctl list-units --plain --no-legend 'carlos-instance*@*'` |
| -(unprivileged) and `POST /stop`s every active one before the activator |
| -starts. Every instance is therefore asleep when the activator's state |
| -machine begins, which is the invariant it holds today. Adopting a running |
| -unit instead (re-acquiring its lease, re-attaching replication) is a |
| -later improvement with its own review; it is not needed for correctness. |
| - |
| -The migration (§8) runs in this same window, after the sweep and before |
| -the activator starts, for the same reason: nothing is running. |
| +dies with the unit). `BindsTo=carlos-edge.service` reproduces the stop |
| +half of that for a `systemctl restart carlos-edge` and does not propagate |
| +the start. For a crash, or a stop that did not propagate, the agent on |
| +start, before anything else: |
| + |
| +1. `systemctl list-units --all --plain --no-legend 'carlos-instance*@*' 'carlos-instances-*.slice'` |
| + and `systemctl list-jobs` (both unprivileged); |
| +2. `POST /stop` for every unit that is not `inactive`, whatever its |
| + role, and for every unit with a pending or running job (`stop` cancels |
| + a queued start); |
| +3. wait for each to reach `inactive`/`failed`. |
| + |
| +Every instance is therefore asleep, and no job is queued, when the |
| +migration (§8) and then the activator's state machine begin, which is |
| +the invariant the activator holds today. Adopting a running unit instead |
| +(re-acquiring its lease, re-attaching replication) is a later |
| +improvement with its own review; it is not needed for correctness. |
| |
| ### 6.8 What does not change |
| |
| @@ -738,9 +949,9 @@ meta dir is derived from `dbPath`, `lease.go:286`, so it lands inside the |
| instance directory), `NextDue`, the schedule runner's token read, the |
| control socket's purge and schedule mounts, `Bounce`, the sweeper's |
| hibernation policy, residence moves (`movespass.go`, once it reads the |
| -DB path from the row instead of rebuilding it, §9), `carlos restart` in |
| -all three scopes, `X-Carlos-Version`, the caps pass, config-ack. All of |
| -these see a path change and nothing else. |
| +DB path from the row and honours the migration fence, §8.1 and §9), |
| +`carlos restart` in all three scopes, `X-Carlos-Version`, the caps pass, |
| +config-ack. All of these see a path change and nothing else. |
| |
| --- |
| |
| @@ -748,10 +959,12 @@ these see a path change and nothing else. |
| |
| `SyncExec` (`internal/appconfig/syncexec.go`) writes an exec route's |
| bundle to `InstanceDir/<host>.config.env` instead of |
| -`<DataDir>/<host>.config.env`. Its ownership rule ("sync owns |
| +`<DataDir>/<host>.config.env`, through the `O_EXCL`-temp-and-rename |
| +helper of §3.2 rather than `os.WriteFile` on a predictable `.tmp` name |
| +(`syncexec.go:921-926`; R1-3). Its ownership rule ("sync owns |
| `*.config.env` outright, never touches `<host>.env` or |
| -`<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged and now trivially |
| -true per directory. `reapConfigEnv` (`:977-1024`) walks |
| +`<host>-sidecar.env`", `syncexec.go:1-31`) is unchanged and now |
| +trivially true per directory. `reapConfigEnv` (`:977-1024`) walks |
| `<DataDir>/instances/*/` for exec routes as well as the flat `DataDir` |
| for addr routes, and the bounce-on-change path is unchanged (the agent |
| bounces the entry; the next wake reads the new file). |
| @@ -771,130 +984,191 @@ directory, where it still applies. |
| |
| ## 8. Migration and rollout |
| |
| -### 8.1 What moves, and when |
| +### 8.1 What moves, and when (codex R1-11, R1-12) |
| + |
| +**Eligibility is the activator's own predicate.** A route is migrated if |
| +and only if the activator would manage it: `Kind == "instance"`, |
| +`Hibernate`, `Backing == ""`, `Tunnel == ""`, `Socket != ""` |
| +(`activator.go:1379-1384`, `edge.go:630-638`). An unflagged instance |
| +route with a socket is an externally supervised process (titogo's router |
| +family is the live example); its files and rows are **not touched**, it |
| +is not isolated by this change, and `carlos status` names it as such |
| +(§8.5). Making that shape isolatable is a separate discussion. |
| |
| On agent start, after the unit sweep (§6.7) and before the activator |
| -starts, for every route with `Kind` instance, `Backing == ""`, and a |
| -`Socket` (an exec-backed route), in one pass: |
| - |
| -1. If `rt.DBPath == filepath.Join(DataDir, host+".db")` and |
| - `rt.Socket == filepath.Join(SocketDir, host+".sock")` (the flat |
| - defaults), the row is **eligible**. |
| -2. `mkdir -p InstanceDir(host)` 0700. |
| -3. `os.Rename` each of `<host>.db`, `<host>.db-wal`, `<host>.db-shm`, |
| +starts, for every eligible route, in one pass: |
| + |
| +1. If `rt.DBPath` and `rt.Socket` are the flat defaults |
| + (`<DataDir>/<host>.db`, `<SocketDir>/<host>.sock`), proceed; |
| + otherwise §8.2. |
| +2. `mkdir -p InstanceDir(host)` 0700. If the directory already holds a |
| + `<host>.db` **and** a flat `<host>.db` also exists, stop: log |
| + `migrate: <host>: two databases (flat and nested); refusing to choose`, |
| + leave everything, refuse the wake (§8.2). Never pick one silently. |
| +3. Write `.migrating` in the directory (contents: the flat and nested |
| + paths and a timestamp). **This is the fence**: `Start`, `Seed`, |
| + `Restore`, the residence-move pass (`movespass.go`), `SyncExec`'s |
| + writer, and the litestream generator all refuse to act on a host whose |
| + directory carries it, with the marker's contents as the reason. |
| +4. `os.Rename` each of `<host>.db`, `<host>.db-wal`, `<host>.db-shm`, |
| `.activator-<host>.db-litestream`, `<host>.env`, `<host>-sidecar.env`, |
| `<host>.config.env`, `<host>.db.corrupt` that exists, into the |
| directory. Same filesystem, same owner, atomic per file, no root. |
| -4. `registry.SetInstancePaths(host, fromDB, fromSocket, toDB, toSocket)`: |
| +5. `registry.SetInstancePaths(host, fromDB, fromSocket, toDB, toSocket)`: |
| one transaction, compare-and-swap on both columns, a new sibling of |
| `SetDBPath` (`registry.go:1057-1075`). |
| -5. Log one line per host: `migrate: <host>: flat → instances/`. |
| - |
| -Order matters: files first, then the row. If the agent dies between 3 |
| -and 4, the next start finds a row still naming the flat paths and files |
| -already in the directory; step 3 is idempotent (a rename whose source is |
| -gone is skipped when the destination exists), so it completes and the |
| -row follows. If it dies between 4 and 5, nothing is missing. If a rename |
| -fails midway (a full disk cannot fail a same-filesystem rename, but a |
| -permission or I/O error can), the pass logs it, leaves that host's row |
| -untouched, and moves on; that host **does not wake** until it is |
| -migrated (§8.2), because a flat spawn is the bug. |
| +6. Remove `.migrating`. Log `migrate: <host>: flat → instances/`. |
| + |
| +Recovery is the marker: a later start that finds `.migrating` re-runs |
| +steps 4 to 6 from the paths it names (a rename whose source is gone and |
| +whose destination exists is complete; one whose source is gone and whose |
| +destination is missing is an error that keeps the fence up). A CAS |
| +conflict at step 5 (the row changed under us) keeps the fence up, logs |
| +both values, and is retried on the next start; a host under a fence |
| +never wakes, never moves, never syncs, so a partial state cannot be |
| +acted on by a destructive consumer. The `MoveCutting` path in |
| +`movespass.go:285-305`, which today would treat an absent DB as |
| +"nothing to checkpoint" and complete the flip, is the case the fence |
| +exists for. |
| |
| The edge picks the new `Socket` up on its next registry poll |
| (`edge.go:449-451`, ~2s); nothing is awake to be affected. |
| |
| -### 8.2 Rows that are not eligible |
| +### 8.2 Rows that are not eligible for the automatic move |
| |
| A row whose `DBPath` or `Socket` is not the flat default was set by hand |
| (`carlos add --db`). The migration does not guess. It logs |
| -`migrate: <host>: db_path/socket not at the flat default; run carlos route --instance-dir <host>` |
| -and the route's wake refuses with the same sentence as its reason, so a |
| -member sees it in `carlos logs` and an operator sees it in the journal. |
| +`migrate: <host>: db_path/socket not at the flat default; run carlos route --instance-dir <host>`, |
| +`Start` refuses the wake with the same sentence as its event line (so a |
| +member sees it in `carlos logs` and in the instance's status, §8.5), and |
| `carlos route --instance-dir <host>` (a new flag on the existing verb) |
| -performs steps 2 to 5 for that one host from the paths the row actually |
| +performs steps 2 to 6 for that one host from the paths the row actually |
| names, as long as they are under `DataDir` and `SocketDir`. The pre-roll |
| check (§8.4) lists these rows before any box runs the new build, so on a |
| fleet whose rows are all at the default (which is every console- |
| provisioned row, `reconcile.go:300-301`) this branch never fires. |
| |
| -### 8.3 Rollback |
| - |
| -An older binary on a migrated box computes the flat `DefaultDBPath`, |
| -finds no file, and restores from the replica into the flat path: correct |
| -data as of the last checkpoint at hibernation, which is what a fresh box |
| -gets today. The nested directory is left in place and is re-adopted, not |
| -re-restored, if the new build returns (step 3's idempotence handles a |
| -flat file that reappeared: the nested copy wins and the flat one is |
| -renamed to `<host>.db.rollback-<ts>` inside the directory, never |
| -deleted). The registry rows name the nested paths, which the old |
| -reconciler logs as shape drift (`reconcile.go:873-881`) and does not |
| -revert. So a rollback costs one restore per instance and no data. |
| - |
| -### 8.4 Rollout order |
| - |
| -1. **Pre-roll check**, from a laptop, per fleet: a new read-only |
| - `carlos ops instances --check` against the box's registry copy (or, |
| - until that exists, an SSM `carlos routes` read) listing every exec |
| - route whose paths are not at the flat default. Fix them with |
| - `carlos route --db` before the roll, or accept that they will refuse |
| - to wake until `carlos route --instance-dir` runs. |
| -2. **systemd version.** `systemctl --version` on every box, over the |
| - existing update path; `PrivatePIDs=` needs 257. Debian 13 ships 257; |
| - a box that answers lower drops the directive from all three templates |
| - **by decision, not by improvisation**: the fleet then runs with |
| - `ProtectProc=invisible` alone and the same-uid `/proc/<pid>/root` |
| - traversal is back until per-instance uids close it (§11). titogo's |
| - unit comment made the same contingency decision in advance. |
| -3. **Dev fleet first** (`kass`, two boxes), the usual boxes-before-console |
| - order. After the roll: `systemctl list-units 'carlos-instance*'` shows |
| - units coming and going with requests; `ls /data/instances` matches |
| - `carlos routes`. The test that matters is from inside an instance, |
| - not from a root shell: the plan's verification task provisions a |
| - throwaway instance whose seed argv lists `/data` and `/run/carlos`, |
| - reads `/proc/1/status`, and curls IMDS, and reads the output back |
| - through `carlos logs`. A planted `/data/probe.txt` must be absent |
| - from the listing. |
| -4. **Flagship**, region by region, console box last. The AU box has failed |
| +### 8.3 Rollback (codex R1-13) |
| + |
| +The previous binary honours `rt.DBPath` and `rt.Socket` from the row |
| +(`backend_exec.go:220-225, 557-562`), so on a migrated box it would open |
| +the nested database, try to bind a socket in a run directory it never |
| +creates, and read admin and config files at the flat paths. That is not |
| +a working rollback, and it is not the "fresh restore" the draft claimed. |
| + |
| +Rollback is therefore **an explicit reverse migration, run by the new |
| +binary before the old one starts**, as a leg of `carlos system rollback` |
| +(the system-update design's rollback path) and as a standalone verb for |
| +the SSM road: `carlos instances-layout revert`. It stops every instance |
| +unit, and for every migrated row reverses steps 3 to 6 under the same |
| +fence: files back to flat, columns CAS-updated back, marker removed. The |
| +nested copy is the only history (nothing ran flat in between), so |
| +nothing is discarded. If someone rolls back without it, the failure is |
| +loud and lossless: instances fail to bind and the journal names the |
| +cause; the fix is to re-install the new binary or run the revert from |
| +it. The layout is never "two histories": a flat database appearing |
| +beside a nested one is a refusal (§8.1 step 2), not a choice. |
| + |
| +### 8.4 Gate: no isolation, no tenants (codex R1-15) |
| + |
| +A box that cannot establish the namespace **does not start tenants**. |
| +There is no fallback unit, no "proceed without `PrivatePIDs=`". The |
| +draft's contingency is withdrawn: running the fleet with the same-uid |
| +`/proc/<pid>/root` traversal open is the defect, not a degraded mode. |
| + |
| +The gate is observed behaviour, not a version string. At start the |
| +control service (root) runs a probe unit, |
| +`carlos-instance-probe.service`, a oneshot with the exact namespace |
| +block of §4.1 whose `ExecStart` is `carlos instance-probe`: it asserts |
| +from inside that `/data` and `/run/carlos` list nothing but the probe's |
| +own bound directories, that `/proc` lists only itself (pid 1 is the |
| +probe), that a connect to `169.254.169.254:80` fails, and that a |
| +symlink it creates in its directory is refused with `ELOOP`. The service |
| +records the verdict; `GET /ready` returns it; the agent refuses every |
| +wake with the verdict's reason until it is 204. The pre-roll check reads |
| +the same verdict through `carlos status` on the first box before the |
| +fleet follows. |
| + |
| +### 8.5 Rollout order |
| + |
| +1. **Pre-roll check**, from a laptop, per fleet, through the console: |
| + `carlos instances --layout` lists every exec route's layout state as |
| + the box reports it in its status snapshot (§9: `internal/status`), |
| + one of `flat` (not yet migrated), `nested`, `ineligible` (hand-set |
| + paths, §8.2), `external` (unflagged, §8.1) or `fenced` (marker |
| + present, with the reason). Fix `ineligible` rows with |
| + `carlos route --db` before the roll, or accept that they refuse to |
| + wake until `carlos route --instance-dir` runs. The same field is what |
| + a member sees on their instance page and in `carlos status`, which is |
| + the member-visible proof that an instance is isolated (R1-18). |
| +2. **Dev fleet first** (`kass`, two boxes), boxes before console. After |
| + the roll: the probe verdict is `ready`; `systemctl list-units |
| + 'carlos-instance*'` shows units coming and going with requests; |
| + `carlos instances --layout` says `nested` for every row. The test that |
| + matters is from inside: the plan's verification task provisions a |
| + throwaway instance whose seed lists `/data` and `/run/carlos`, reads |
| + `/proc/1/status`, curls IMDS, and plants and follows a symlink, and |
| + reads the results back through `carlos logs`. |
| +3. **Flagship**, region by region, console box last. The AU box has failed |
| every gate since 2026-09-07 for an unrelated reason; do not read its |
| failure as this change's. |
| -5. **carlos-tito-ie**: Tito's operator rolls it from Tito's console once |
| - the flagship has run a day. The seven flat instances migrate on the |
| - first agent start. Reply on #126 with the build and the check list |
| - above. |
| +4. **carlos-tito-ie**: Tito's operator rolls it from Tito's console once |
| + the flagship has run a day. The seven flagged instances migrate on the |
| + first agent start; the router family (unflagged) is reported |
| + `external` and left alone. Reply on #126 with the build and the check |
| + list above. |
| + |
| +**The roll script.** `update-platform-binary.sh` (pinfra) enables units |
| +by name; it gains `carlos-instance-control.socket`, both mount units and |
| +the probe, before it restarts the edge, and `bootstrap.go`'s |
| +`systemctlPlan` (`:165-179`) starts `.socket` and `.mount` units it |
| +enables, which it does not today (R1-14). `ensureSystemUnits` |
| +(`systemunits.go:34-83`) checks the control socket is *active*, not just |
| +enabled. An agent that finds the channel absent refuses wakes with |
| +`instance control socket not listening`, which is what a forgotten |
| +enable looks like from `carlos logs`. |
| |
| --- |
| |
| ## 9. Every consumer of the shape |
| |
| The rule (AGENTS.md): the consumer list is everything that parses the |
| -shape, not everything that calls the API. From the sweep of 2026-09-11: |
| +shape, not everything that calls the API. From the sweep of 2026-09-11, |
| +extended by codex R1-9, R1-14, R1-17, R1-18: |
| |
| **Path builders that move onto `InstanceDir` / the new defaults** |
| |
| - `internal/provision/provision.go:1011-1012` — `DefaultSocket`, `DefaultDBPath` (the definitions) |
| -- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`) |
| -- `internal/appconfig/syncexec.go:465, 872` (exec routes only; addr routes stay flat); `:977-1024` `reapConfigEnv` walks both |
| +- `internal/activator/backend_exec.go:160-162` `defaultDBPath` (deleted); `:350, 507, 579, 607, 786, 790` (`<host>.env`, `<host>.config.env`); `:865-868` (`<host>-sidecar.env`); every read through `instancefs` |
| +- `internal/appconfig/syncexec.go:465, 872, 921-926` (exec routes only; addr routes stay flat; `O_EXCL` temp); `:977-1024` `reapConfigEnv` walks both |
| - `internal/schedule/runner.go:461` and `internal/host/schedule_wiring.go:163` (admin token path) |
| -- `internal/host/movespass.go:285` — rebuilds `<DataDir>/<host>.db` from the record; must read `rt.DBPath` from the row instead |
| -- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir") |
| +- `internal/host/movespass.go:285-305` — rebuilds `<DataDir>/<host>.db` from the record; reads `rt.DBPath` from the row instead, and honours the fence |
| +- `cli.go:135-136` (`carlos add` default), `:157-159` (relative-path refusal becomes "must be under InstanceDir"), the new `carlos route --instance-dir`, `carlos instances --layout`, `carlos instances-layout revert`; `docs/site/cli.md` names all three (the docsite gate fails otherwise) |
| - `internal/provision/reconcile.go:300-301` (`routeFor` mint) — follows the helpers unchanged; `:461-464` and `:873-881` compare derived paths and follow too |
| |
| **Readers of `rt.Socket` (no change; the value changes)** |
| |
| -- `internal/edge/edge.go:536-538` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| +- `internal/edge/edge.go:536-538, 2000-2014` (dial, rebuilt on every `Reload`), `internal/activator/backend_exec.go:594, 738`, `internal/status/snapshot.go:487`, `internal/host/probes.go:273`, `internal/host/schedule_wiring.go:343-369`, `internal/schedule/runner.go:237-238`, `internal/appconfig/configack.go:168`, `internal/host/socket_advice.go:32-79` (its `/data/%s.config.env` string is for addr routes and stays), console display sites in `apiroutes.go`, `relayops.go`, `apirelay.go` |
| |
| **Litestream and replication** |
| |
| - `internal/lease/lease.go:286` — meta dir derived from `dbPath`; follows |
| -- `internal/ops/litestream.go:354` — the box-wide generator's flat `*.db` glob does not see nested files, **but** `FleetDBsWithSkipped` (`:323-428`) claims every route's `DBPath` from the registry, which is nested after migration. The glob is the proposer, the registry disposes. Add `instances/*/*.db` to the glob anyway so an unrouted nested file is reported under `skipped` rather than silently ignored, and pin it in `ops_test.go` |
| -- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine |
| +- `internal/ops/litestream.go:354-388` — the box-wide generator **discovers by glob and filters by registry claim**; it does not append claimed paths (R1-17). The glob gains `instances/*/*.db`, or nested non-hibernating databases are never replicated. Hibernating rows stay excluded at `:420-421` as today; the fence excludes a migrating host. Pinned in `ops_test.go` for both coverage and the `skipped` report |
| +- `carlos-litestream.service` runs as `carlos` with `ReadWritePaths=/data`: reads nested 0700 directories fine; ordered `After=data-instances.mount` |
| + |
| +**Metrics and status** |
| + |
| +- `internal/host/metrics_probes.go` (discovery: the new scan root), `internal/console/metricsview.go:728-762` (the key allowlist: unchanged because the sampler publishes under `carlos-exec@<host>`) |
| +- `internal/status/snapshot.go` gains the per-route `layout` field of §8.5; `carlos status` and the console instance page render it |
| |
| **Units and installers** |
| |
| -- `internal/host/units/` gains four files (§4) and two (§5.2); `carlos-edge.service` changes its memory lines |
| -- `internal/host/bootstrap.go:116-122` `enabledUnits` gains `carlos-instance-control.socket`; `systemctlPlan` (`:165-180`) already starts sockets it enables |
| -- `update-platform-binary.sh:52-70` (pinfra) enables units **by name**: add `carlos-instance-control.socket` there, or the roll leaves the channel unarmed and every wake fails with "connection refused" on the control socket. `ensureSystemUnits` (`internal/host/systemunits.go:34-83`) is the belt for that: it gains the socket too |
| -- `harden-host.sh` (pinfra) writes hardening drop-ins for `$TENANT_UNITS`; the instance templates carry their own and must not be listed there (a drop-in that adds `ProtectProc=invisible` is harmless; one that adds `PrivateNetwork` or narrows `RestrictAddressFamilies` is not) |
| +- `internal/host/units/` gains: three templates, two slices, two mounts, the probe, the control socket and service (§4, §5, §8.4); `carlos-edge.service` gains `Slice=` and drops its memory lines; `carlos-litestream.service` gains the `After=` |
| +- `internal/host/bootstrap.go:61` (tmpfiles lines), `:116-122` `enabledUnits` (control socket, mounts, probe), `:165-179` `systemctlPlan` (starts `.socket`/`.mount`) |
| +- `internal/host/systemunits.go:34-83` (active check on the socket) |
| +- `update-platform-binary.sh:52-70` (pinfra): enables the new units by name before restarting the edge |
| +- `harden-host.sh` (pinfra) writes hardening drop-ins for `$TENANT_UNITS`; the instance templates carry their own and must not be listed there |
| +- the `carlos system rollback` leg (system-update design) runs the reverse migration first |
| |
| **Tests that pin the flat layout** (§10): `internal/activator/activator_test.go` (20 fixture sites), `backend_exec_test.go` (4), `admintoken_test.go` (8, path-agnostic), `internal/appconfig/syncexec_test.go` (50), `internal/ops/ops_test.go` (52), `internal/provision/reconcile_test.go` (20), `internal/host/movespass_test.go` (1), `schedule_wiring_test.go` (1), `schedule_token_fence_test.go` (2), root `e2e_test.go` (24), `instances_e2e_test.go`, `config_e2e_test.go`, `deploy_e2e_test.go`. |
| |
| @@ -908,43 +1182,66 @@ sentinel `.path` units name fixed files under `/data`, never a host. |
| |
| Unit level, all runnable in CI (no systemd needed): |
| |
| -- `internal/provision`: `InstanceDir`, new defaults, `carlos add` refusal |
| - of an exec `--db` outside the directory. |
| -- `internal/activator`: `Start` writes `launch.run.json` with the exact |
| - argv and env `childEnv` produces today (the existing env-composition |
| - tests move their assertion from `cmd.Env` to the file); the control |
| - client sends `start` with the tier; `unitProc.Stop` orders stop → done |
| - → close listeners; `Exited()` from log EOF and from a `show` answer; |
| - the failed-health path posts `stop`. The control channel is a seam |
| - (`Control interface { Start; Seed; Stop; Reset }`) with a fake, as |
| - `Systemctl` is today. |
| -- `instance-exec`: reads the file, refuses schema ≠ 1, refuses an |
| - `argv[0]` that is not the live link, execs (tested with a stub argv). |
| -- `instance-control`: every 400 (bad host, bad role, bad limit), the unit |
| - name it builds, the `set-property` before `start` ordering, the seed |
| - result mapping, with a fake `systemctl` runner. |
| -- `logListener`: one connection accepted, second refused, line cap, |
| - EOF → exited. |
| -- Migration: eligible row moves eight files and CAS-updates two columns; |
| - ineligible row is logged and left; idempotent re-run after a simulated |
| - crash between files and row; rollback re-adoption renames the flat |
| - reappearance. |
| -- Units: a test over `unitsFS` asserts the three templates' namespace and |
| - hardening blocks are identical, that `PrivatePIDs=yes`, |
| - `TemporaryFileSystem=/data:ro`, `IPAddressDeny=169.254.169.254/32` and |
| - `Restart=no` are present in each, and that the control socket is |
| - `SocketMode=0600` `SocketUser=carlos`. |
| -- `internal/ops`: the glob change and the skipped report. |
| -- `internal/cgroup`: the sampler's slice scan root. |
| +- `internal/provision`: `InstanceDir`, new defaults, `LogSocket`, the |
| + escape function against `systemd-escape` fixtures, `carlos add` |
| + refusal of an exec `--db` outside the directory. |
| +- `instancefs`: refuses a symlink at every component, refuses `..`, |
| + `O_EXCL` temp plus rename, the mountinfo check. |
| +- `internal/activator`: `prepare` writes `launch.<role>.json` with the |
| + deduplicated env and the exact argv (the existing env-composition tests |
| + move their assertion to the effective environment of an executed stub); |
| + the control client sends the tier and the token, retries once on 401 |
| + after re-reading the token; `unitProc.Stop` orders stop → done → close |
| + listeners; `Exited()` only after a confirmed terminal state with the |
| + matching invocation id, never on log EOF; the failed-health path posts |
| + `stop` and waits; seed timeout and cancellation stop-and-wait before |
| + cleanup; the startup sweep stops activating units and cancels jobs. |
| + The control channel is a seam (`Control interface { Start; Seed; Stop; |
| + Reset; Ready }`) with a fake, as `Systemctl` is today; the exit |
| + observer takes a fake `show` and a fake events file. |
| +- `instance-exec`: reads the file through `instancefs`, refuses schema |
| + ≠ 1, refuses an `argv[0]` that is not the live link, execs. |
| +- `instance-control`: 401 without the token; every 400 (bad host, bad |
| + role, bad limit); the unit and slice names it builds for a host with a |
| + dash; the `set-property` → `reset-failed` → `start` ordering; the seed |
| + result mapping; `/ready` from the probe verdict; with a fake |
| + `systemctl` runner. |
| +- `logListener`: one connection per invocation, a second refused and |
| + logged, line cap, EOF ends collection without marking exit. |
| +- Migration: eligible row moves eight files under the marker and |
| + CAS-updates two columns; ineligible and external rows are logged and |
| + left; idempotent re-run after a simulated crash between files and row; |
| + CAS conflict keeps the fence; two databases refuse; every fenced |
| + consumer (`Start`, `Seed`, `Restore`, moves pass, `SyncExec`, litestream |
| + generator) refuses a fenced host; the reverse migration restores the |
| + flat layout byte for byte; a partial rename followed by a |
| + `MoveCutting` pass does not complete the flip. |
| +- Units: a test over `unitsFS` asserts the three templates' namespace, |
| + hardening and coupling blocks are identical, that `PrivatePIDs=yes`, |
| + `TemporaryFileSystem=/data:ro`, `IPAddressDeny=169.254.169.254/32`, |
| + `Restart=no`, `BindsTo=carlos-edge.service`, `Slice=carlos-instances-%i.slice` |
| + and the `%I` paths are present, that the seed carries |
| + `TimeoutStartSec=120`, that the control socket is `SocketMode=0600 |
| + SocketUser=carlos`, that both mounts carry `nosymfollow`, and that |
| + `carlos-edge.service` carries `Slice=carlos.slice` and no `MemoryMax=`. |
| +- `internal/ops`: the nested glob, coverage of a nested non-hibernating |
| + row, the `skipped` report, the fence. |
| +- `internal/host`: `systemctlPlan` starts sockets and mounts; the |
| + sampler's slice scan root publishes under `carlos-exec@<host>`. |
| +- `internal/status`: the `layout` field for each of the five states. |
| |
| Box level, in the plan's verification task and in the pre-roll check: |
| |
| -- On a dev box: wake an instance, `systemctl status carlos-instance@<host>` |
| - active; `nsenter`-free check from inside via a throwaway instance whose |
| - seed lists `/data` and `/run/carlos`; `cat /proc/1/status` from inside |
| - shows the app as pid 1 (PrivatePIDs); IMDS curl from inside fails; |
| - `carlos logs` shows all of it; hibernate, unit gone; `carlos restart` |
| - cycles it; `systemctl restart carlos-edge` stops it (`PartOf=`). |
| +- On a dev box: the probe verdict; wake an instance, |
| + `systemctl status carlos-instance@<esc>` active in |
| + `carlos-instances-<esc>.slice`; from inside via the throwaway |
| + instance's seed: `/data` and `/run/carlos` list one directory each, |
| + `/proc/1` is the app, IMDS refuses, a planted symlink gives `ELOOP`; |
| + `carlos logs` shows all of it for both instance and sidecar at once; |
| + hibernate, unit gone; `carlos restart` cycles it; |
| + `systemctl restart carlos-edge` stops it and does not start it; |
| + `carlos instances --layout` says `nested`; the console's memory graph |
| + for the app is continuous across the roll. |
| |
| --- |
| |
| @@ -952,14 +1249,17 @@ Box level, in the plan's verification task and in the pre-roll check: |
| |
| - **Per-instance uids (option B).** The nested layout is its |
| prerequisite; the namespace closes every path the discussion names |
| - without it. If a box cannot run `PrivatePIDs=`, or a later finding |
| - needs a uid boundary, it lands on this layout with a root chown in the |
| - control service. Not in this change. |
| + without it. If a later finding needs a uid boundary, it lands on this |
| + layout with a root chown in the control service. Not in this change. |
| +- **Externally supervised exec routes** (unflagged instance routes with |
| + a socket, §8.1): not isolated by this change; their supervisor owns |
| + their process. Whether the platform should offer them the same |
| + namespace is a discussion to open once this lands. |
| - **Unit-backed console instances.** Not needed for isolation, and a |
| - hibernation rewrite. `UnitBackend` gets a working `start` for free from |
| - §5 (it can call the control service instead of bare `systemctl`); that |
| - wiring, and whether `backing=unit` hibernation works on the fleet today, |
| - is a follow-up discussion. |
| + hibernation rewrite. Whether `backing=unit` hibernation works on the |
| + fleet today (`UnitBackend` runs `systemctl` unprivileged, §1.4) is its |
| + own discussion; the control channel of §5 is deliberately not widened |
| + to `carlos-app@` (R1-19). |
| - **#127**: root-rendered config for opted-in addr routes into |
| `/etc/carlos/apps`. Smaller and independent; this change makes the file |
| invisible to instances but not to the agent. |
| @@ -967,9 +1267,8 @@ Box level, in the plan's verification task and in the pre-roll check: |
| (§6.7). |
| - **Deleting the spawn half of `internal/cgroup`** and `Delegate=` from |
| the edge unit, one release after this lands. |
| -- **Discussion #1 (memory pressure)**: the slice split in §4.3 should be |
| - measured on the flagship after the roll, and the numbers adjusted in the |
| - unit files, never by drop-in. |
| +- **Discussion #1 (memory pressure)**: no new numbers here (§4.3); the |
| + slice shape makes a later change a unit-file edit, never a drop-in. |
| |
| --- |
| |
| @@ -988,13 +1287,34 @@ Box level, in the plan's verification task and in the pre-roll check: |
| spec). Keeps every activator invariant, needs one root channel, and |
| migrates with a rename. |
| |
| -Within 3, two sub-choices were made and one is put to codex: trampoline |
| -over `EnvironmentFile=` (§6.2, one env composer), three templates over |
| -one (§4.2, `%i` is the host), and the control service over polkit |
| -(§5.4). |
| +Within 3: trampoline over `EnvironmentFile=` (§6.2, one env composer), |
| +three templates over one (§4.2, the instance is the host), the control |
| +service with a token over polkit (§5.4, ruled with codex), and |
| +`nosymfollow` mounts over per-read symlink checks (§3.2, kernel-enforced |
| +and race-free where a check-then-open is neither). |
| |
| --- |
| |
| ## 13. Review log |
| |
| -- Round 1: pending (`CODEX-REVIEW-BRIEF-ISOLATED-EXEC.md`). |
| +- **Round 1** (codex, GPT-6, `CODEX-FINDINGS-ISOLATED-EXEC.md`, 19 |
| + findings, verdict request changes). All accepted: |
| + R1-1 control service with token, `nosymfollow` (§3.2, §5.4); |
| + R1-2 log sockets outside tenant mounts, per role (§3.1, §6.4); |
| + R1-3 `nosymfollow` plus `instancefs` and `O_EXCL` temps (§3.2, §7); |
| + R1-4 exit from systemd state with invocation id, never from EOF (§6.5); |
| + R1-5 seed timeout, stop verb, wait before cleanup (§4.2, §5.3, §6.2); |
| + R1-6 per-role log listeners (§6.4); |
| + R1-7 `BindsTo=` not `PartOf=`, sweep covers jobs (§4.1, §6.7); |
| + R1-8 per-host slice carries the tier for all three roles (§4.3, §6.6); |
| + R1-9 cgroup path under `carlos.slice`, key compatibility, `metrics_probes.go` (§6.6, §9); |
| + R1-10 one common wall in `carlos.slice`, no new numbers (§4.3); |
| + R1-11 eligibility is the activator's predicate (§8.1); |
| + R1-12 the `.migrating` fence and its consumers (§8.1); |
| + R1-13 explicit reverse migration; no "restore" claim (§8.3); |
| + R1-14 planner starts sockets and mounts; roll script; active check (§8.5); |
| + R1-15 no `PrivatePIDs=` fallback; behavioural gate and probe (§8.4); |
| + R1-16 env deduplication, effective-value test (§6.2); |
| + R1-17 nested glob is discovery, not reporting (§9); |
| + R1-18 `layout` status through the API and CLI, `docs/site/cli.md` (§8.5, §9); |
| + R1-19 `UnitBackend` claim removed (§1.4, §11). |
amadan