amadan
carlos / platform Public
Isolated exec — every exec-backed instance in its own namespace
Date: 2026-09-11 · Status: revision 17; codex verdict after round 11 is accept with corrections, and the one correction is applied here (54 findings over 11 rounds, all accepted; §13); a whole-branch final review (plan 2) found six further items, fixed in revision 13 (§13); plan 3's task 4 review settled the requested-stop close, recorded in revision 14 (§6.5, §13 R13); the first roll's probe found two defects, fixed in revision 15 (§4.1, §8.5, §13 R15); awaiting Paul's review · Requested by: discussion #126 (Tito, with box evidence), approved in direction by Paul on 2026-09-11
Builds on: discussion #126 and its companion #127; titogo's deploy/systemd/titogo@.service (the namespace recipe, verified on systemd 257 and on carlos-tito-ie); internal/edge/purge.go:53-107 and internal/activator/admintoken.go:6-14, which already describe the gap; the exec-child accounting design (2026-08-07, pinfra platform-design/superpowers/specs/), the memory-pressure containment design (2026-08-28, same place), the cross-user socket consumers group (issue #103, internal/host/bootstrap.go:41-62), the system-update design (2026-09-06) whose apply/rollback path the transition must survive, and the root-relay pattern (carlos-stage.path → carlos adopt --stage-only).
---
0. How this spec was arrived at
Tito filed #126 after verifying on carlos-tito-ie that an exec-backed instance can read every other instance's database, admin token and delivered config, including the relay route's platform credentials, and including instances of other accounts' apps. The discussion offered two shapes: make console-provisioned instances unit-backed, or isolate exec children in place with a per-host directory plus a uid or namespace.
Three approaches were compared against the code on 2026-09-11 (§12). Paul approved the third on the same day and asked that the one open mechanism choice, how the unprivileged agent starts a root-managed unit, be settled between Fable and codex during review. Codex's round 1 concurred with the control service and showed why it must authenticate its caller (§5.4). Round 2 turned the migration from an agent-start side effect into an operator-triggered fleet verb (§8), because the system updater's own rollback runs the old binary and cannot reverse a layout it never knew. Round 3 drew the transition boundary: while a box still runs unisolated tenants, nothing that runs as carlos is a trustworthy caller, so the root channel authenticates the agent process, not a secret the agent holds (§5.4).
The shape in one sentence: the activator keeps owning every exec-backed instance exactly as it does today, but the process it owns is launched as a systemd template unit, still uid carlos, inside a mount namespace in which nothing but that instance's own directory exists, on a mount where no symlink is ever followed. No uid change, no console change, no new backing. One release carries both layouts; an operator verb moves each fleet across; the same verb moves it back.
---
1. What exists, and what is wrong
1.1 The agent and its children
carlos agent runs as User=carlos under carlos-edge.service (internal/host/units/carlos-edge.service), with CapabilityBoundingSet=CAP_NET_BIND_SERVICE, RestrictNamespaces=yes, ReadWritePaths=/data /run/carlos and Delegate=cpu memory io pids. It cannot change a child's uid and cannot build a mount namespace for one.
Every exec-backed instance is a plain child of that process: internal/activator/backend_exec.go:690-692 builds exec.Command(bin, "--socket", rt.Socket, "--db", dbPath) with cmd.Env = childEnv(childVars); no cmd.Dir, no SysProcAttr.Credential (the only SysProcAttr fields set anywhere in the tree are the cgroup manager's Setpgid, Pdeathsig, UseCgroupFD and CgroupFD, internal/cgroup/strict_linux.go:256, cgroup_linux.go:20-24). Seed (backend_exec.go:377-410) and sidecars (:885-911) are spawned the same way.
Everything the instance owns sits flat in one directory, DataDir (/data; backend_exec.go:52-53, internal/host/agent.go:186-188):
| File | Written by | Read by | |---|---|---| | <host>.db (+ -wal, -shm, .activator-<host>.db-litestream/) | the instance; restore, seed and replication in the agent | the instance, the agent's in-process replicator (internal/lease/lease.go:257-299) | | <host>.env (admin token) | EnsureAdminToken (admintoken.go:111-195) | the agent (backend_exec.go:350, 507, 579, 607, 790; internal/schedule/runner.go:461) | | <host>-sidecar.env | hand-staged | backend_exec.go:865-877 | | <host>.config.env | internal/appconfig/syncexec.go:872-926 | backend_exec.go:786 |
All of it is carlos:carlos, and every instance is uid carlos. Mode 0600 restricts nothing between siblings. purge.go:53-95 says so in prose, and names per-instance uids as "the real fix".
The sockets are shared on purpose with one more party: /run/carlos is 2750 carlos:carlos-sockets (bootstrap.go:61), and after the health wait the activator normalises each instance socket to 0660 (backend_exec.go:744-750), so a deployment's enrolled cross-user consumer (titogo's router, uid titogo, issue #103) can dial every instance socket on the box. That contract survives this change (§3.1).
1.2 What a sibling can reach today
Everything below is a read (or write) by any exec-backed instance of any app on the box, as verified by Tito on 2026-09-10 and as the file table above implies:
- every other exec instance's database, read and write;
- every other exec instance's admin token, which drives that host's schedule verbs (
admintoken.go:6-14); - every
<host>.config.envin/data, including the ones for TCP-addr routes CARLOS never spawns (carlos env exec-delivery file), which for Tito carry every platform credential the relay holds (#127); - every other exec instance's control socket,
rt.Socket + ".control"(backend_exec.go:594;purge.go:56-68): a purge for another host, or a schedule tick with a token read from that host's env file; - every other exec instance's listening socket file under
/run/carlos: same uid, so a sibling can unlink<other>.sockand bind its own listener there, and the edge (internal/edge/edge.go:536-538) will dial it as that host. This is not in #126; it follows from the same uid and the same directory; /data/registry.db,/data/litestream.env(the box-wide replicator's bucket credentials,carlos-litestream.service) and, on a console box whereconsole.serviceruns ascarloswithReadWritePaths=/data,/data/console/console.db. Verify the console unit's user on the flagship before treating the last item as fact; the unit is hand-written and not in git.- the instance metadata service.
harden-host.sh:36-44(pinfra) allows IMDS to uid 0 and uidcarlosand drops everyone else, because the agent reads IMDS. Exec children are uidcarlos.carlos-app@.servicecarriesIPAddressDeny=169.254.169.254/32 fd00:ec2::254/128for exactly this reason; exec children carry nothing.
1.3 The primitive under all of it: same uid, same paths
A namespace hides a sibling's files from the instance. It does not hide them from the agent, which reads and writes the instance's directory from outside, as the same uid, by path. Any path the instance can write and the agent later opens is therefore a request the instance makes on the agent's behalf: replace <host>.env with a symlink to /data/litestream.env and the next wake hands the bucket credentials to the instance as its own environment (backend_exec.go:785-794, 966-971; EnsureAdminToken only refuses the chmod through a symlink, admintoken.go:141-150). Replace <host>.db with a symlink to a sibling's database and the agent's replicator uploads the sibling's data under the attacker's replica. Replace the listening socket with a symlink to any other socket on the box and the edge dials it as this host (codex R1-1, R1-3). Any file the agent reads as instructions from the instance's directory is the same primitive without a symlink (R2-1). And for as long as a box still runs one unisolated tenant, every carlos-readable secret and every carlos-writable directory on the box is that tenant's too, so nothing that runs as carlos can be told apart from it by what it knows or where it writes (R3-1). A design that bounds the instance but not the paths the agent follows on its behalf, the records it trusts, or the callers it believes, has not bounded anything. §3.2 closes the symlinks at the kernel; §3.3 states the rule for records; §5.4 states the rule for callers.
1.4 What the unit backing already gets right, and why it is not the answer
carlos-app@.service (DynamicUser=yes, StateDirectory=carlos-app/%i, root-0600 config in /etc/carlos/apps) has the property #126 asks for: a peer's state is absent, not permission-denied. But UnitBackend (internal/activator/backend_unit.go) has none of the machinery the exec backend gives a self-serve instance. Its Acquire, Restore and Seed are inert (:113-132), replication belongs to the root carlos-litestream unit because the agent cannot read a DynamicUser's state directory (:12-19), NextDue returns nothing (:174-179), and sidecars are refused (:181-188). provision.Record has no backing field (internal/provision/provision.go:36-135) and the reconciler refuses to adopt a row that carries one (reconcile.go:426-436). The always-awake design already concluded that moving exec instances to units "needs a separately reviewed root/unit/socket maintenance protocol and data migration". Making console-provisioned instances unit-backed is a rewrite of hibernation, not an isolation fix.
One more thing the sweep found: UnitBackend.systemctl (backend_unit.go:94-103) runs systemctl start as the unprivileged agent. Nothing in this tree or in pinfra grants uid carlos the org.freedesktop.systemd1.manage-units action, and polkit is not referenced anywhere. Whether backing=unit hibernation works on the fleet today is a follow-up discussion (§11); this spec's channel is scoped to the three instance templates and does not extend to carlos-app@ (codex R1-19).
---
2. The design in one page
- Per-instance directory. An exec-backed instance's files move from
<DataDir>/<host>.*to<DataDir>/instances/<host>/(§3). The agent creates the directory and everything in it, ascarlos, exactly as it creates the flat files today. The registry'sDBPathandSocketcolumns point into the new layout once a host is migrated.
- No symlink is ever followed under the trees the agent binds or opens for a tenant.
/data/instances,/run/carlos/instancesand/run/carlos/logsare bind mounts of themselves withnosymfollow(§3.2). The kernel refuses to traverse a symlink there for every process on the box, agent, edge and PID 1 included, with no check-then-use race. Nothing the agent does is ever directed by a file it read from an instance directory, and a migration journal is honoured only when it says what the fleet's recorded intent already says (§3.3).
- A template unit per role. The instance runs as
carlos-instance@<host>.service; its seed ascarlos-instance-seed@<host>.service; its sidecar ascarlos-instance-sidecar@<host>.service(§4). All three run asUser=carlos, all three carry titogo's namespace (TemporaryFileSystem=/data:roandTemporaryFileSystem=/run/carlos:rowith exactly the instance's own two directories bound back in,PrivatePIDs=yes,ProtectProc=invisible,RestrictNamespaces=yes,IPAddressDeny=for IMDS), and all three sit in one per-host slice that carries the host's memory and swap tier. A sibling's database does not exist in this namespace; a sibling's socket does not exist; a sibling's pid does not exist.
- The agent stays the owner. Lease, restore, seed ordering, replication, hibernation, the control socket,
carlos logs, the next-due tick, sidecar co-lifecycle and residence moves are unchanged ininternal/activator, with two bounded changes inactivator.go(§6.3, §6.7). What changes is theProcthe exec backend hands back: it is a unit, not an*exec.Cmd(§6).
- One root channel, authenticating the process. The agent cannot start a system unit. A small root service,
carlos-instance-control, socket-activated on a unix socket in a root-owned directory, starts and stops units namedcarlos-instance*@<host>and nothing else, and accepts a request only from the edge unit's main process, as reported by systemd, never from a child of it (§5). This is the synchronous form of thecarlos-stage.pathpattern.
- Launch through a trampoline; exit through the unit's own record plus the kernel's word. Each unit's
ExecStartis/opt/carlos/carlos instance-exec <role> <host>, which runs inside the namespace, reads the launch description the agent wrote, andexecves the app binary with the exact argv and environment the agent composed (§6.2). Each unit'sExecStopPost=leaves an advisory exit record in the instance's own run directory; termination itself is proven by systemd's unit state and an empty cgroup, never by a file (§6.5).
- Resource walls become slices.
carlos.slicecarries the 2026-08-28 outer wall; the edge unit and every instance sit under it; each host has its own slice inside that, carrying the tier (§4.3, §6.6).
- One release, two layouts, one operator verb. This release's agent runs a flat host exactly as today (unisolated,
childProc) and a nested host isolated. Nothing moves on upgrade and no new row is minted nested, so the system updater's own rollback (which runs the previous binary) always lands on a layout that binary understands.carlos system layout migraterecords the fleet's intent; every box converges its eligible hosts to it on every serial move;revertrecords the opposite; status reports each host (§8). The next release drops the flat spawn path for activator-managed hosts.
- Addr routes and externally supervised routes are untouched, and now invisible. A TCP-addr route's
<host>.config.envstays flat in/data, because the app's own supervisor reads it there (titogo'sEnvironmentFile=-/data/%i.config.env); an unflagged socket route keeps its flat paths because its supervisor owns them. Isolated instances can no longer see either. That closes the read #127 leads with; #127's ask for root-owned delivery stays open as its own change (§11).
---
3. The instance directory
3.1 Layout
/data/.instances/ the RAW directory: 0700 carlos:carlos; opened by the layout pass only (§3.2, §8.2)
/data/instances/ nosymfollow bind mount OF /data/.instances (§3.2); the view everything else uses
/data/instances/<iid>/ 0700 carlos:carlos, created by the agent
<host>.db the database (+ -wal, -shm)
.activator-<host>.db-litestream/ the in-process replicator's shadow
<host>.env admin token (EnsureAdminToken, unchanged)
<host>-sidecar.env hand-staged sidecar env (unchanged)
<host>.config.env syncexec's rendered bundle (unchanged)
launch.run.json agent-written before each instance start (§6.2)
launch.seed.json agent-written before a seed
launch.sidecar.json agent-written before a sidecar start
<host>.db.corrupt a quarantined restore, as today
/data/layout/ 0700 carlos:carlos
<host>.json the migration journal for a host in flight (§8.2)
/run/carlos/instances/ nosymfollow bind mount OF /run/carlos/.instances; 2750 carlos:carlos-sockets
/run/carlos/instances/<iid>/ 2750 carlos:carlos-sockets, created by the agent
app.sock the instance's listener (rt.Socket); 0660 after health, as today
app.sock.control the agent's control listener (rt.Socket + ".control"); 0600
exit.<role>.<invocation>.json the unit's advisory exit record (§6.5); tenant-writable, and treated so
/run/carlos/logs/ nosymfollow bind mount OF /run/carlos/.logs; 0700 carlos:carlos; NOT bound into any namespace
<iid>.run.sock the agent's log listener for the instance (§6.4)
<iid>.seed.sock ... for the seed
<iid>.sidecar.sock ... for the sidecar
/run/carlos-control/ 0750 root:carlos; NOT under /run/carlos, NOT in any namespace
control.sock the root channel (§5); SocketUser=carlos, 0600<iid> is the instance identifier, bounded to 40 bytes (codex R5-3). A host may be 253 bytes (controlname.ValidHostShape), and a unix socket address holds 107 bytes of path, so a directory named after the host cannot carry app.sock.control for every valid host. provision.InstanceID(host) is the host itself when len(host) <= 40, and otherwise _ + the first 22 bytes of the host + - + the first 16 hex digits of sha256(host), which is also 40 bytes. The two domains cannot meet (codex R6-1): no valid host contains _ (controlname labels are [a-z0-9-]), so a hashed identifier is never equal to any short host, and two hashed identifiers are equal only on a 64-bit hash collision between two long hosts sharing a 22-byte prefix. It is the directory name under /data/instances and /run/carlos/instances, the log socket prefix, and, escaped, the unit instance and slice suffix (§4; _ needs no escaping). File names inside the directory keep the full host (a file name may be 255 bytes). The longest address this produces, /run/carlos/instances/<40>/app.sock.control, is 79 bytes.
Two rows on one box whose hosts nonetheless collide in InstanceID classify as conflict, which refuses to wake with both hosts named and is never the unisolated fallback: a host that is already nested must never be moved to the childProc path by the arrival of a second row (R6-1). The layout pass skips a conflict host, status reports it, and the console's create-instance door refuses a new host whose identifier collides with an existing row on the target box. The pass also preflights every socket address it is about to commit against 107 bytes before it moves a file.
The socket tree keeps the #103 contract: /run/carlos/instances and each host's run directory are 2750 carlos:carlos-sockets (setgid, so the socket an instance binds inherits the group, exactly as tmpfilesConf arranges for /run/carlos today), the activator's post-health 0660 normalisation is unchanged, and the control socket stays 0600. An enrolled consumer can traverse to and dial app.sock and nothing else (codex R2-10). Inside the namespace the instance owns its run directory and can loosen it; that changes nothing outside, because outside is where the mode is set and checked.
One helper, one owner. internal/provision gains InstanceDir(dataDir, host) beside DefaultDBPath, and the nested defaults exist alongside the flat ones. In this release no minting path uses the nested defaults (§8.1, codex R3-6): routeFor (reconcile.go:299-301) and carlos add (cli.go:118-136) keep minting flat paths, and the layout pass is the only thing that ever writes a nested path into a row.
func InstanceID(host string) string { // ≤ 40 bytes; see §3.1
if len(host) <= 40 { return host }
sum := sha256.Sum256([]byte(host))
return "_" + host[:22] + "-" + hex.EncodeToString(sum[:8]) // '_' never occurs in a host
}
func InstanceDir(dataDir, host string) string {
return filepath.Join(dataDir, "instances", InstanceID(host))
}
func NestedDBPath(dataDir, host string) string {
return filepath.Join(InstanceDir(dataDir, host), host+".db")
}
func InstanceRunDir(host string) string {
return filepath.Join(SocketDir, "instances", InstanceID(host))
}
func NestedSocket(host string) string {
return filepath.Join(InstanceRunDir(host), "app.sock")
}
func LogSocket(host, role string) string {
return filepath.Join(SocketDir, "logs", InstanceID(host)+"."+role+".sock")
}
// DefaultDBPath and DefaultSocket keep their flat meaning for this
// release; the next release, which drops the flat spawn path, points
// them at the nested helpers for activator-managed routes and keeps
// them flat for externally supervised ones (§8.6).ExecBackend.defaultDBPath (backend_exec.go:160-162) is deleted in favour of the provision helpers, so there is one definition, not two.
File names inside the directory keep the host prefix. It costs nothing, it keeps every existing test fixture recognisable, and it means a file found outside its directory is still attributable.
Why nested and not per-instance uids. titogo's unit comment says it best: "A flat /data/titogo/%i.db cannot be isolated — the bind would have to reveal all of /data/titogo, which is the regression itself." Per-instance uids (the discussion's option 2, titogo's "Option B") would additionally need a uid allocator, a chown as root on every migration and every restore, and a group arrangement that keeps the agent able to open the database for replication. The namespace gives the property the discussion asks for ("absent, not permission-denied") without any of that, and leaves option B open as a later hardening on the same layout (§11).
3.2 Symlinks are never followed under the trees the platform binds or opens for a tenant (codex R1-1, R1-2, R1-3, R3-1)
Three mount units, embedded and installed like every other unit (internal/host/bootstrap.go:33-34), and tmpfiles lines beside the existing /run/carlos one (bootstrap.go:61). All three carry DefaultDependencies=no: a mount unit's default ordering places it before local-fs.target, and systemd-tmpfiles-setup.service, which creates the directories, runs after that target, so the defaults would form a cycle systemd breaks by dropping the mount (codex R2-2). The ordering is therefore written out in full, including shutdown:
# data-instances.mount
[Unit]
Description=carlos instance data (bind, nosymfollow)
DefaultDependencies=no
RequiresMountsFor=/data
After=local-fs.target systemd-tmpfiles-setup.service
Before=carlos-edge.service carlos-instance-control.service carlos-litestream.service
Conflicts=umount.target
Before=umount.target
[Mount]
What=/data/.instances
Where=/data/instances
Type=none
Options=bind,nosymfollow,nosuid,nodev,noexec
[Install]
WantedBy=multi-user.targetThe source is a different directory from the mountpoint (codex R4-2). rename(2) refuses to cross a mount boundary with EXDEV even when both sides are one filesystem, so a bind of /data/instances onto itself would make the migration's rename from /data/<host>.db into it impossible. Binding /data/.instances onto /data/instances keeps the raw directory reachable on the /data mount: the layout pass renames /data/<host>.db → /data/.instances/<host>/<host>.db on one mount, atomically, and the file appears at /data/instances/<host>/<host>.db through the protected view. The raw path is opened by exactly one caller, the layout pass, and only as openat2(RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH) directory fds for renameat, so no lookup through it ever follows a symlink either; a CI test greps the tree for the literal .instances and fails on any other occurrence.
run-carlos-instances.mount and run-carlos-logs.mount are the same shape for /run/carlos/.instances → /run/carlos/instances and /run/carlos/.logs → /run/carlos/logs, with After=systemd-tmpfiles-setup.service and no RequiresMountsFor=; nothing is ever renamed into those, so their raw directories have no caller at all.
d /data/.instances 0700 carlos carlos -
d /data/instances 0700 carlos carlos -
d /data/layout 0700 carlos carlos -
d /run/carlos/.instances 2750 carlos carlos-sockets -
d /run/carlos/.logs 0700 carlos carlos -
f /etc/carlos/probe-secret 0640 root carlos - probe
d /run/carlos/instances 2750 carlos carlos-sockets -
d /run/carlos/logs 0700 carlos carlos -
d /run/carlos-control 0750 root carlos -carlos-edge.service, carlos-instance-control.service and the three templates gain Requires= and After= on the mounts they touch, so an edge that starts without them does not start at all rather than starting unprotected. The plan's first box task is a cold boot with the full unit set installed, then systemctl list-dependencies --before data-instances.mount and a systemd-analyze verify over the embedded set, pinned as a CI test that renders the units and runs systemd-analyze verify where the tool exists.
nosymfollow (Linux 5.10+, util-linux 2.36+; Debian 13 ships 6.12 and 2.41) makes path resolution on that mount return ELOOP at any symlink, for every process, in every namespace that sees the mount. It is a per-mount flag, so it holds through the BindPaths= bind into the instance's namespace, through the agent's and the edge's ordinary lookups from outside, and through PID 1's open of a StandardOutput=file: path. readlink still works; nothing the platform runs needs a symlink under these trees (SQLite, litestream, the app's listener and the agent's env files are all plain files). Hard links cannot escape either: a link() needs a source path the caller can resolve, and inside the namespace no sibling path resolves.
What this closes, race-free and for every reader at once:
- the instance replacing its own
app.sockwith a symlink to the root control socket, a sibling's socket, or a sibling's control socket, and having the edge dial it (R1-1); - the instance replacing
<host>.env,<host>.config.envor<host>.dbwith a symlink to a sibling's file,/data/litestream.envor any agent-readable path, and having the agent read it into the instance's environment or replicate it into the instance's replica (R1-3); - the instance planting a symlink at a predictable temp name so the agent's write truncates the target (R1-3);
- a still-unisolated flat tenant, during the transition, planting a symlink at a log socket path so that PID 1 opens a root-owned file as some unit's stdout (R3-1).
/run/carlos/logsiscarlos-owned because the agent must bind there, so a same-uid process can write it; the mount flag is what makes that write inert.
The flat tenants' own sockets under /run/carlos itself are not on such a mount (the directory holds flat sockets, the apps subtree and more, and this release does not change it). A flat tenant can therefore still point its socket at any socket on the box, by symlink or by editing the registry row it can reach as carlos, and have the edge dial it. §5.4 says why that no longer reaches the control service, and §6.1 names the two checks the edge and the agent make on every unix dial.
Belt and braces in the agent, cheap and testable:
- every agent open of a file under an instance directory goes through one helper,
instancefs.Open(dir, name, flags), built onopenat2(2)withRESOLVE_BENEATH | RESOLVE_NO_SYMLINKS | RESOLVE_NO_MAGICLINKS(Linux 5.6+), so a symlink is refused with a named error even on a box where the mount option is missing; the same helper is what the agent uses to verify, at start, that all three mounts carrynosymfollow(/proc/self/mountinfo), and it refuses to wake any nested host until they do (§8.5); - every agent write into an instance directory creates its temp file with
O_CREAT|O_EXCLand a random suffix, thenrenameats over the target (rename replaces the directory entry and never follows the destination); - launch files (§6.2) are written the same way, and
instance-execreads them through the same helper.
3.3 Nothing in an instance directory is an instruction, and a journal is not one either (codex R2-1, R3-1)
The agent reads three kinds of thing from an instance directory: the database (to replicate, restore and checkpoint), the two env files (to compose the instance's own environment), and the advisory exit record (to decorate an event line, §6.5). It never reads a path, a host name, a file list or a phase from there. In particular:
- the migration journal lives in
/data/layout/<host>.json, outside every tenant mount, and every source and destination path in it is derived from the host name and the registry row by the §3.1 helpers at plan time. On recovery the pass recomputes both endpoints for the journal's own direction and honours the journal only if its paths equal the recomputation and the row stands at one of them (§8.2); a journal that passes is completed to its own end and the host is then converged to the fleet's recorded intent (control/layout.json, a bucket object no tenant can write) before it is released. During the transition a flat tenant can write/data/layout(same uid); a journal it forges can therefore cause the agent to do exactly what the next layout pass would do for that host anyway, and nothing else (R3-1); - the launch file is agent-written and then read by the trampoline inside the namespace as the instance itself; it directs nothing the agent does (§6.2);
- the exit record is written inside the instance's run directory by the unit's own
ExecStopPost=, is therefore tenant-writable, and is treated as the tenant's claim about its own exit status: it never decides whether the process is gone (§6.5).
A test plants a forged journal (a sibling's paths under a host's name; a direction opposite to the fleet's intent), a forged launch file naming a sibling, and a forged exit record for a running unit, and asserts the sibling is untouched and the running unit is not treated as exited.
---
4. The units
Unit files under internal/host/units/, embedded and installed exactly as the existing nineteen are (bootstrap.go:33-34, systemapply.go:339-398): carlos-instance@.service, carlos-instance-seed@.service, carlos-instance-sidecar@.service, carlos.slice, carlos-instances.slice, the three mount units of §3.2, the probe of §8.5, and the two control units of §5.2. Nothing enables the templates; an instance unit exists only while the agent has started it.
Instance names are escaped instance identifiers. The unit instance is systemd-escape(InstanceID(host)) (§3.1): for a valid identifier ([a-z0-9.-]) that only turns - into \x2d. %I (unescaped) is used wherever the identifier is a path component; %i (escaped) wherever it is part of a unit name, which is what lets a per-host slice be spelled in the template (R1-8). The agent and the control service share one Go escape function (a copy of systemd's rules for this character set, pinned by a test against systemd-escape output for the hosts in the fixture set). The templates never need the full host: every path they name is keyed by <iid>, and the trampoline learns the host from the launch file (§6.2).
4.1 carlos-instance@.service
[Unit]
Description=carlos exec-backed instance %I
# Stop-only coupling: when the edge stops (including the stop half of a
# restart) this unit stops, and nothing starts it again but the agent.
# PartOf= would propagate the RESTART and race the agent's own startup
# (codex R1-7).
BindsTo=carlos-edge.service
Requires=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
After=carlos-edge.service data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
[Service]
Type=simple
ExecStart=/opt/carlos/carlos instance-exec run %I
# Runs inside the same namespace, as carlos, after this invocation's
# processes have been signalled: leaves an ADVISORY exit record in the
# instance's own run directory (§6.5). Not "+": the "+" prefix drops
# User= and the Private*/Protect* settings but keeps the explicit
# TemporaryFileSystem=/BindPaths= mounts (codex R3-2), so nothing run
# from here can see the host's /run/carlos anyway.
ExecStopPost=/opt/carlos/carlos instance-exit run %I
User=carlos
Group=carlos
# One slice per host holds the instance, its seed and its sidecar, and
# carries the tier (§4.3, §6.6). Slices referenced here are created on
# demand; no per-host unit file exists.
Slice=carlos-instances-%i.slice
# The activator owns recycling: a crash stands the entry down and the
# next request wakes it. systemd must not race that.
Restart=no
# SIGTERM to the app, SIGKILL to anything it left behind after
# TimeoutStopSec; today's actStopTimeout is 20s.
KillMode=mixed
TimeoutStopSec=20
# The log stream: stdout and stderr connect to the agent's per-role
# listener (§6.4). PID 1 opens this as root, outside the namespace, on
# the nosymfollow logs mount, at a path no tenant mount contains.
StandardOutput=file:/run/carlos/logs/%I.run.sock
StandardError=inherit
# --- the namespace (titogo@.service, 2026-07-31 design, on systemd 257) ---
ProtectSystem=strict
TemporaryFileSystem=/data:ro
TemporaryFileSystem=/run/carlos:ro
# Every other tree a carlos-uid unit writes on the box is hidden the same
# way (codex R9-2): the restore drills' cache under /var/cache, and
# /var/lib as a class. An instance needs nothing from either.
TemporaryFileSystem=/var/cache:ro
TemporaryFileSystem=/var/lib:ro
# Root-written credentials a carlos-GROUP process may read live here:
# /etc/carlos/relay-masters is root:carlos 0640 for carlos-relay.service,
# and an instance's Group=carlos would read it too (codex R10-1). An
# instance gets its configuration through its launch file and needs
# nothing from /etc/carlos.
InaccessiblePaths=/etc/carlos
BindPaths=/data/instances/%I /run/carlos/instances/%I
ReadWritePaths=/data/instances/%I /run/carlos/instances/%I
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
# Its own /dev/shm; PrivateDevices= would bind the host's (R15).
TemporaryFileSystem=/dev/shm:mode=1777,nosuid,nodev
# /sys/fs/cgroup read-only: the edge's delegated subtree is carlos-owned,
# and its cgroup.kill / cgroup.freeze would otherwise be one write away
# for a same-uid tenant (codex R9-1).
ProtectControlGroups=yes
ProtectProc=invisible
ProcSubset=pid
PrivatePIDs=yes
RestrictNamespaces=yes
NoNewPrivileges=yes
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
IPAddressDeny=169.254.169.254/32 fd00:ec2::254/128
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectClock=yes
ProtectHostname=yes
RemoveIPC=yes
LockPersonality=yes
RestrictRealtime=yes
# No RestrictSUIDSGID=: it refuses openat2 outright (R15).
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
# --- per-service resources; the tier lives on the host slice (§4.3) ---
TasksMax=256
# The kernel picks a tenant before it picks anything else in carlos.slice.
OOMScoreAdjust=500
# One OOM kill takes every process of THIS service (systemd sets
# memory.oom.group on the service cgroup for OOMPolicy=kill). The sidecar
# is a sibling service with its own family rule; §4.3 says why the two are
# not one family any more.
OOMPolicy=killLines that are load-bearing and why:
TemporaryFileSystem=/data:ro+BindPaths=/data/instances/%Iis the isolation. The tmpfs is empty; the bind reveals one directory. A sibling's database is not permission-denied, it does not exist. The same for/run/carlos: a sibling's listening socket and control socket do not exist, which closes the socket-file impersonation in §1.2. The bound directory is inside thenosymfollowmount (§3.2), so a symlink the instance plants there is dead for every reader.- The bind sources must exist before the unit starts, or systemd fails at namespace setup before any
ExecStartPre=runs (titogo verified this; it is whytitogo-instance-dir@.serviceexists). Here the agent is the supervisor and creates both directories before it asks for a start (§6.1), so no helper unit is needed. PrivatePIDs=yes+ProtectProc=invisible. Every instance is uidcarlos, andProtectProc=invisibledoes not hide same-uid peers (hidepid's test isptrace_may_access, which same-uid passes). With its own pid namespace an instance's/proclists only itself: no sibling pid to traverse into/proc/<pid>/root/, no sibling to signal, nothing for a same-uid ptrace to attach to.PrivatePIDs=needs systemd 257 (Debian 13, whichinfra/modules/region-host/variables.tf:134pins) and is ignored, not refused, on a kernel that cannot provide it, so §8.5 gates on observed behaviour, not on the directive (R1-15).RestrictNamespaces=yes: without it an unprivileged user namespace could bind-mount over a fresh mountpoint and re-reveal what the tmpfs hid.IPAddressDeny=for IMDS. The nft rule in pinfra allows uidcarlosbecause the agent needs IMDS; this line is what stops an instance, also uidcarlos, from fetching the box's role credentials. It is a cgroup BPF packet filter: a denied datagram send fails withEPERM, a denied TCPconnect()loses its SYN and times out. §8.5's probe is written for that (R3-4).BindsTo=carlos-edge.servicekeeps today's "instances die with the agent": when the edge stops, so do they, and nothing restarts them but the agent's next wake. §6.7 adds the startup sweep for the case where the coupling did not fire.StandardOutput=file:to anAF_UNIXsocket is documented systemd behaviour: a stream connection is made and used as stdout. It is opened by PID 1 outside the namespace, so the path is the real one, the path is in/run/carlos/logs, which no tenant mount contains, and that directory is on anosymfollowmount, so a same-uid process outside any namespace cannot turn it into a symlink either.ProtectControlGroups=yes(codex R9-1).ProtectSystem=strictleaves kernel API filesystems alone andProtectKernelTunables=leaves/sys/fs/cgroupwritable; the edge unit'sDelegate=subtree, including thesupervise/cgroup the agent moves itself into (internal/cgroup/cgroup.go:172), is created bycarlosand itscgroup.killandcgroup.freezeare owner-writable. A same-uid tenant with a writable cgroupfs could kill or freeze the agent with no pid and no socket. Read-only cgroupfs closes that; an instance has no business in cgroupfs at all.TemporaryFileSystem=/var/cache:roand/var/lib:ro(codex R9-2). The isolation property is "no same-uid path on the box but its own", and two trees outside/dataand/run/carlosare written bycarlos-uid units:carlos-hibernate-verify.serviceandcarlos-restore-verify.servicesetCacheDirectory=carlos-verifyand restore every host's database into/var/cache/carlos-verifyduring a drill (internal/ops/ops.go:274,internal/lease/check.go:79), and/var/libholds unit tenants' state. Both are hidden. §9 carries the audit rule that found them, so the next such unit is caught at review.InaccessiblePaths=/etc/carlos(codex R10-1).ProtectSystem=strictmakes/etcread-only, not unreadable.carlos-relay.serviceruns ascarlos:carlosand reads/etc/carlos/relay-masters(internal/relay/masters.go:101), which the relay design stages asroot:carlos 0640(2026-09-07-relay-stun-turn-design.md:325): a file everyGroup=carlosprocess can read, and the secret every app's relay key is derived from (internal/relay/server.go:665). The instance's group iscarlosfor the socket contract (§3.1), so the directory is hidden outright.host.envandapps/*.envthere are root-0600 and were already unreadable; hiding the directory is what makes the next0640file safe by default./opt/carlosstays readable throughProtectSystem=strict; the trampoline and the live symlink are all the instance needs from it.
4.2 carlos-instance-seed@.service and carlos-instance-sidecar@.service
Identical [Unit] coupling, [Service] namespace, hardening and resource blocks, and the same Slice=carlos-instances-%i.slice. Differences:
| | seed | sidecar | |---|---|---| | Type= | oneshot with RemainAfterExit=yes: after ExecStart exits the unit stays active (exited) or failed, loaded, with its InvocationID, Result and ExecMainStatus retained until the control service stops it, so the seed's result can never be garbage-collected before it is read (codex R2-4, R5 disposition) | simple | | ExecStart= | /opt/carlos/carlos instance-exec seed %I | /opt/carlos/carlos instance-exec sidecar %I | | ExecStopPost= | /opt/carlos/carlos instance-exit seed %I | /opt/carlos/carlos instance-exit sidecar %I | | StandardOutput= | file:/run/carlos/logs/%I.seed.sock | file:/run/carlos/logs/%I.sidecar.sock | | timeouts | TimeoutStartSec=120 (today's seed budget, backend_exec.go:368-408, 466; a oneshot has no start timeout by default, R1-5), TimeoutStopSec=20 | TimeoutStopSec=20 |
A Go test over the embedded FS asserts that the three files' namespace, hardening and coupling lines are byte-identical (§10). Three files rather than one because seed and instance never run together but sidecar and instance do, and a template's instance is the host, not a role.
4.3 Slices and the memory walls (codex R1-8, R1-10, R2-11, R3-3)
# carlos.slice — the outer wall of the 2026-08-28 design, as a slice
[Unit]
Description=carlos (edge, activator and every exec-backed instance)
[Slice]
MemoryHigh=70%
MemoryMax=80%# carlos-instances.slice — the parent of every per-host slice
[Unit]
Description=carlos exec-backed instances
[Slice]
# No cap of its own: the outer wall is carlos.slice, the inner walls are
# the per-host slices. This level exists so the sampler has one root to
# scan and so an operator can `systemctl status carlos-instances.slice`.carlos-edge.service gains Slice=carlos.slice and drops its own MemoryHigh=70% MemoryMax=80%: the wall moves up one level and now bounds exactly what it bounded before, the agent plus every awake instance, so borrowing between them and the OOM candidate set (a tenant at +500 is always picked before the agent) are unchanged. The 2026-08-28 reasoning, including the measured 1.2 GB leaf that was mostly reclaimable page cache, carries over without new numbers. OOMPolicy=continue stays on the edge unit.
The per-host slice carlos-instances-<esc>.slice is created on demand by the first unit that names it. Before every start the control service sets three properties on it from the route's tier (§5.3, §6.6): MemoryMax, MemoryHigh (90% of max) and MemorySwapMax equal to MemoryMax, the last because with zram on the box a cgroup at memory.max otherwise spills into compressed swap and carries on, which internal/cgroup/cgroup.go:313-318 learned live on the flagship (R2-11). The values are 512M/460M/512M by default and 1G/920M/1G for an app whose manifest declares capabilities.children, exactly what internal/cgroup writes today. Seed, instance and sidecar all charge that one cgroup, which is what today's shared carlos-exec@<host> gives them (backend_exec.go:149-155, 373-375, 890-895).
The family OOM rule changes, deliberately. Today memory.oom.group=1 on the shared carlos-exec@<host> cgroup means an OOM kill in the sidecar takes the instance with it. systemd writes memory.oom.group itself, for slices too, from the unit's cgroup context (default 0, cgroup_context_apply in src/core/cgroup.c), so a value written behind its back is overwritten on the next property application, which /start for the sidecar performs on a slice whose instance is already running (codex R3-3). There is no supported way to keep it set on the slice. So the rule becomes: each service is its own OOM family (OOMPolicy=kill, which sets the attribute on the service's cgroup), and a sidecar killed by the OOM killer is a sidecar that died, which the sweeper already restarts with backoff (activator.go:1165-1200) while the instance keeps serving. An instance killed by the OOM killer still stands its entry down and co-stops its sidecar as today. The instance surviving its sidecar's OOM is the better of the two behaviours; the spec records it as a difference rather than claiming parity.
The cgroup path is /sys/fs/cgroup/carlos.slice/carlos-instances.slice/carlos-instances-<esc>.slice/ (R1-9).
---
5. The root channel: carlos-instance-control
5.1 Why a channel at all
The agent is uid carlos. systemctl start on a system unit needs either root or a polkit grant; read-only systemctl show / is-active / list-units do not. The existing root relays (carlos-stage.path, carlos-restart.path) are asynchronous sentinel files, fine for a minute-scale adoption pass and wrong for a wake that must answer a waiting HTTP request.
5.2 Units
# carlos-instance-control.socket
[Unit]
Description=carlos instance control socket (agent → root)
[Socket]
ListenStream=/run/carlos-control/control.sock
SocketUser=carlos
SocketGroup=carlos
SocketMode=0600
Accept=no
[Install]
WantedBy=sockets.target# carlos-instance-control.service
[Unit]
Description=carlos instance control (root: start/stop carlos-instance*@ units for the agent)
Requires=carlos-instance-control.socket data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
After=data-instances.mount run-carlos-instances.mount run-carlos-logs.mount
[Service]
ExecStart=/opt/carlos/carlos instance-control
# Root deliberately: the whole job is systemctl start/stop, and the
# caller is the unprivileged agent. Same rationale as carlos-stage.service.
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
RestrictAddressFamilies=AF_UNIX
MemoryDenyWriteExecute=yes
MemoryMax=64MThe socket lives in /run/carlos-control, 0750 root:carlos: any carlos process can connect (search on the directory), none can create, rename or replace anything in it. The unit is socket-activated and runs only while there is work. It is hidden from every instance's namespace by InaccessiblePaths= in the shared block (§4.2): a connect from inside one is impossible, and the peer check refuses everything else. /run/carlos inside an instance is an empty tmpfs with the instance's own directory bound in.
5.3 Protocol
HTTP/1.1 over the unix socket, the shape edge.ListenControl already uses for the per-instance control socket. Before reading a request line the service authenticates the connection (§5.4); a connection that fails is closed without a response. Five verbs, no shell, no argument passes through to systemctl except a unit name the service built itself:
| Verb | What it runs | Returns | Agent deadline | |---|---|---|---| | POST /start?host=H&role=run\|sidecar&memory_max=N | systemctl set-property --runtime carlos-instances-<esc>.slice MemoryMax=N MemoryHigh=0.9N MemorySwapMax=N; systemctl start <unit> (blocking: the start job completes) | 200 with {invocation_id} from systemctl show -p InvocationID <unit>, read immediately after the start returns; or 502 with systemctl's stderr | 30s | | POST /seed?host=H&memory_max=N | the same set-property; systemctl start on carlos-instance-seed@<esc>.service (blocking: a oneshot's start job completes when ExecStart exits, or fails at TimeoutStartSec); if the InvocationID it left is the SAME one that was running before this start, an earlier request's stop never ran, so the stale unit is stopped and started again; the unit is now active (exited) or failed and stays loaded (RemainAfterExit=yes, §4.2); systemctl show -p InvocationID,Result,ExecMainStatus,ExecMainCode reads the retained result; systemctl stop (instant for an exited oneshot; clears it), on a context detached from the request so a disconnected client cannot leave it running; then wait for the cgroup to be empty (§6.5's rule, applied by the service as root) | 200 with {invocation_id, result, status, code}, always complete | 150s: 120s seed budget plus TimeoutStopSec and slack (R3-5) | | POST /stop?host=H&role=run\|seed\|sidecar | systemctl stop <unit> (blocking; also cancels a pending start job) | 204 | 30s | | POST /reset?host=H&role=… | systemctl reset-failed <unit>, tolerating "not loaded" | 204 | 30s | | POST /sweep | systemctl list-units --all and list-jobs over the three templates' instances; systemctl stop each that is not inactive or has a job; wait for terminal state and empty cgroup on each | 200 with the list of unit names swept, or 502 naming the unit that would not terminate | 60s | | GET /ready | nothing | 204 once the probe verdict (§8.5) is ready; 503 with the verdict otherwise | 5s |
There is deliberately no reset-failed inside /start: ResetFailedUnit refuses a unit that is not loaded, and a template instance that has never started is not loaded, so the first wake of every host would fail (codex R2-3). systemctl start on a failed unit already clears the failed state; /reset exists for the sweeper's housekeeping only.
Validation, in order: connection (§5.4); host must satisfy provision.ValidHostShape (provision.go:262); role must be one of the three literals; the memory value must parse as a Limit the agent already knows how to produce (internal/cgroup, TierDefault/TierChildren). Anything else is a 400 with no systemctl call. The unit name is "carlos-instance" + suffix + "@" + escape(InstanceID(host)) + ".service" and the slice name "carlos-instances-" + escape(InstanceID(host)) + ".slice", built from the validated host by the same InstanceID and escape functions the agent used to create the directories (§3.1, §4), never from the request string (codex R5-3 disposition).
What this grants the agent: start, stop, reset and tier control of units named carlos-instance*@<host> and their host slice. That is exactly what it has today over its own children. It grants nothing else, and a host that does not exist is a unit whose bind source does not exist, which fails at namespace setup and starts nothing.
systemctl set-property --runtime on a slice or on an instance of an installed template that has never been started loads the unit first and writes a runtime drop-in the following start honours (SetUnitProperties in src/core/dbus-manager.c; confirmed by codex R1-1). The plan's first task exercises it on a dev box for a slice that does not yet exist.
A /start or /stop that exceeds its deadline is answered by the agent with a /stop and reported as a failed wake or an unproven stop (§6.3); the deadlines exist because §6.7 puts one such call on a path that must never stall the activator, and the seed's is long because a seed is allowed to be slow (R3-5).
5.4 Who may call it, and the ruling (codex R1-1, R3-1)
The other way to let uid carlos start these units is polkit: a rules file granting org.freedesktop.systemd1.manage-units to carlos when the unit name matches carlos-instance*@*.service. Codex R1-1 confirmed that polkit can express the unit and verb restrictions, and also that it cannot constrain property values, that a uid-only grant is reachable by every tenant through the system bus, and that a wildcard on names also authorises transient units. Codex's advisement, concurring with the draft's: the control service, provided it authenticates the caller and not the socket's uid. Ruled, 2026-09-11: control service; polkit is not installed and not relied on.
Revision 2 authenticated the caller with a bearer token in a carlos-readable file. Round 3 showed why that is not a boundary during the transition: a flat tenant is uid carlos in the host namespace and reads that file as easily as the agent does (R3-1). A secret the agent holds cannot distinguish the agent from an unisolated child of it. What can is systemd's own record of which process is the agent:
- On every accepted connection the service reads
SO_PEERCREDand requiresuid == carlosandpid == MainPIDofcarlos-edge.serviceassystemctl show -p MainPIDreports it at that moment (root, one call, cached for one second). A flat tenant is a child of that pid, never the pid itself; a nested tenant cannot reach the socket at all. - The edge is the agent, so a request the edge proxies for a tenant (a flat tenant pointing its own socket at
control.sock, §3.2) would pass that check. It does not get that far, because of two checks on every unix dial the edge and the agent make (edge.go:2000-2014'sunixProxy, the activator's health and control dials, §6.1): 1. the reserved endpoint is refused before the dial: a path under/run/carlos-control/is never dialled as an upstream, so a registry row pointing atcontrol.sockis refused beforeconnect(). Nothing else is refused by path: a route whose operator stamped--socket /run/titogo/example.sockkeeps working exactly as today (codex R5-1); 2. the listener answers to the path that was dialled: afterconnect()and before a byte is sent,getpeername()on the connected socket returns the address the listener bound, which is its real path however it was reached; if that is not byte-equal to the path the edge dialled, the connection is closed. A symlink at/run/carlos/<host>.sockpointing anywhere, including a sibling's socket or the control socket, therefore yields a mismatch. Revision 4's rule, "refuse a peer whose uid is 0", is withdrawn: a listener created by PID 1 forcarlos-app@.socketreports root credentials inSO_PEERCREDhowever the fd is later passed (codex R4-1), so that rule would have refused every unit-backed app. The two checks above do not look at credentials at all.
A proxied request therefore never reaches the service, and the service never sees a connection from anything but the agent's own goroutines. No token: the file it lived in is gone, and with it the question of who can read it. Each check is one syscall or one cached systemctl show, each is testable with a real socket pair, and all of them hold on a box that still runs flat tenants, which is the whole transition. They also end, as a side effect, the flat-tenant socket impersonation §1.2 names, which a symlink could otherwise still perform during this release.
---
6. The activator on units
internal/activator/activator.go changes in exactly three places: the sidecar restart in sweepOnce (§6.7, codex R2-6), the lease release in stop (§6.3, codex R2-5), and the route the wake runs on (§6.8, codex R8-1). doWake (activator.go:772), Bounce (:946-956) and the Proc/Rep seams (:194-207) keep their contracts. Exited() stays a non-blocking read of a flag (R1-4); everything that talks to systemd runs in the proc's own goroutine or in the sidecar-start goroutine. The rest of the change is inside ExecBackend, which keeps its current childProc path for flat hosts for this one release (§8.1) and adds the unit path for nested hosts.
6.1 Preparing a role, and Start
One helper, prepare(rt, role), runs before any role starts, seed included (R1-5, R2-12):
- Ensure
InstanceDirandInstanceRunDirexist (0700 and 2750 as in §3.1), and/run/carlos/logs. EnsureAdminToken(InstanceDir/<host>.env)forrunandseed(a seed is the first thing a fresh instance runs and must see the token the instance will,backend_exec.go:346-352);instanceEnvfor both, reading the two files fromInstanceDirthroughinstancefs, with the seed's removal of the control-channel variables as today. Forrunonly: bind the control listener atrt.Socket + ".control"(unchanged,edge.ListenControl) and clear any staleapp.sock(nothing is running: the unit is not active, or we would not be here). Remove any staleexit.<role>.*.json.- Bind the role's log listener at
LogSocket(host, role)(§6.4). - Write
launch.<role>.json(§6.2).
Start is then: prepare(rt, "run"); POST /start with the route's tier; waitHealthyCtx(ctx, rt.Socket, spawnHealthTimeout) (unchanged, with the path and getpeername checks of §5.4 on its dial); the 0660 normalisation of app.sock (unchanged); on failure POST /stop, wait for the proc's done, and clean up, exactly as the current failure path stops the child; return a unitProc carrying the invocation id.
6.2 The trampoline and launch.<role>.json
{
"schema": 1,
"host": "<host>",
"argv": ["/opt/carlos/live/<host>", "--socket", "/run/carlos/instances/<iid>/app.sock", "--db", "/data/instances/<iid>/<host>.db"],
"env": ["PATH=...", "CARLOS_ADMIN_TOKEN=...", "CARLOS_CONTROL_SOCKET=...", "..."]
}carlos instance-exec <role> <iid> runs as ExecStart, inside the namespace, as carlos. It reads /data/instances/<iid>/launch.<role>.json through instancefs, refuses a schema it does not know, refuses an argv[0] that is not directly under /opt/carlos/live/ or that does not equal /opt/carlos/live/ plus the file's own host (defence in depth: the file is agent-written; the prefix check is the one that holds even against a rewritten file, since only the platform's live links live there), and calls syscall.Exec(argv[0], argv, env). It never parses env syntax: the agent's childEnv (backend_exec.go:802-814) composes the list exactly as it does today, including the agent-owned removal of tenant-supplied CARLOS_CONTROL_SOCKET / CARLOS_PURGE (backend_exec.go:636-660), and the trampoline passes it through.
One entry per key, last value wins (R1-16). os/exec deduplicates cmd.Env in favour of the last occurrence; syscall.Exec passes duplicates through and a Go child keeps the first. childEnv therefore gains the deduplication itself (a map keyed on the name, insertion order preserved), and the test asserts the effective value inside an executed stub, not the JSON array.
What the current exec.Command path does that the trampoline must reproduce, and how: working directory (none is set today, and none is set; the app starts in /), inherited fds (none beyond stdio today; syscall.Exec inherits whatever the trampoline holds, which is stdio only), WaitDelay (replaced by TimeoutStopSec, §6.4), Pdeathsig (replaced by BindsTo= plus the startup sweep, §6.7), oom_score_adj (OOMScoreAdjust=), the cgroup placement (Slice=).
Why not EnvironmentFile=: systemd's parser and the agent's parseEnvBytes differ on quoting and continuation, and the agent-owned variables would then have to be written into a file the tenant's <host>.env overlay must not win over. One composer, one format, one test.
The launch file is written fresh before every start, O_EXCL temp plus rename (§3.2), and holds the same material the child could already read from /proc/self/environ. It is inside the instance's own directory, visible to that instance and the agent, nobody else. An instance can rewrite its own launch file while it runs; the next start reads it again through instancefs, refuses any argv[0] but its own live link, and executes only what a tenant could execute anyway: its own binary with an argv and env it chose. That is not a privilege, and the agent never reads the file (§3.3).
Seed: launch.seed.json carries the argv from rt.Seed (backend_exec.go:332-338) with the DB path substituted as today; POST /seed blocks up to its 150s deadline; the seed's exit status arrives in the response or in its exit record (§6.5); a non-zero status, a timeout or a cancelled ctx runs POST /stop?role=seed, waits for the seed proc's done, and only then runs the same cleanup as today (backend_exec.go:438), so the lease is never released (activator.go:824-828) while a seed can still write (R1-5).
Sidecar: launch.sidecar.json carries [live/<host>, "sidecar", "run"] and the sidecar env from <host>-sidecar.env only, as today (backend_exec.go:885-911); POST /start?role=sidecar.
6.3 unitProc, and a stop that could not be proven (codex R2-5)
type unitProc struct {
host, role string
invocation string // from the start response
ctl *edge.ControlListener // run only; as childProc.ctl today
logs *logListener // §6.4
exited atomic.Bool
done chan struct{}
}
func (p *unitProc) Stop(ctx context.Context) error // POST /stop, then wait for done or ctx
func (p *unitProc) Exited() bool // p.exited.Load()The control listener's close ordering (backend_exec.go:992-1009: signal first, close the listener only after the process is gone) is preserved: Stop posts the stop, waits for done, then closes ctl and logs. done closes only on proven termination (§6.5), never on log EOF.
If ctx expires before done closes, Stop returns ErrStillRunning without closing the listeners. Today stop() then calls rep.Close regardless (activator.go:1014-1033), which releases the lease over a process that may still be writing. The second activator.go change: on ErrStillRunning, stop() does not call rep.Close; it marks the entry's outcome uncertain (the field exists, outcome.uncertain), leaves the entry in stopping, and sweepOnce retries proc.Stop on its cadence until done closes, after which the normal rep.Close runs. The lease's TTL bounds the worst case, as it does today when an agent dies mid-stop; what changes is that the agent never voluntarily releases over an unproven termination.
6.4 Logs
Today teeStd (backend_exec.go:123-137) pipes the child's stdout and stderr through the agent: straight to the agent's own stdout (journald) and, with Logs set, line-wise into Logs.App for carlos logs, capped at execLogLineMax, with WaitDelay bounding a grandchild that holds the pipe.
Each role's unit connects its stdout and stderr to that role's socket in /run/carlos/logs (StandardOutput=file: on an AF_UNIX path; R1-2, R1-6). The agent's logListener for a role accepts one connection per start (the unit connects exactly once, at exec; a second connection during the same invocation is closed and logged), and feeds it to the same applog.NewLineWriter and the same io.MultiWriter to the agent's stdout. Same bytes, same cap, same sink, same carlos logs, and the instance and its sidecar no longer contend for one connection. The listener is bound in prepare before the start request, so PID 1's connect cannot race it. A flat tenant, during the transition, can connect to a log socket (same uid, host namespace) and write lines into another host's log stream; that is a nuisance the transition accepts and the next release ends, and it cannot make anything read wrong, because nothing reads the log stream as an instruction.
Grandchildren: a process the app forked that outlives it keeps the socket open, as it keeps the pipe open today. KillMode=mixed plus TimeoutStopSec=20 kills it on stop; on a crash the unit's cgroup empties when systemd reaps it. EOF on the log connection ends log collection for that invocation; it does not mean the process is gone (R1-4).
6.5 Exit observation (codex R1-4, R2-4, R2-5, R3-2, R3-7)
childProc learns of exit from cmd.Wait() (backend_exec.go:722-736), which is process death. A unit proc needs the same certainty, attributed to the right invocation, and it must not depend on systemd keeping an inactive unit loaded: systemd garbage-collects an unreferenced inactive unit and its ExecMainStatus with it, so a show after a quick clean exit can find nothing to show (R2-4). Revision 3 tried to write the record from ExecStopPost=+ into a root-owned directory; the + prefix drops User= and the Private*/Protect* settings but keeps the explicit TemporaryFileSystem=/BindPaths= mounts (apply_mount_namespace in src/core/exec-invoke.c), so that command could not see the directory (R3-2). So the record is advisory, and truth is elsewhere:
carlos instance-exit <role> <host> runs as ExecStopPost= inside the namespace as carlos, after the invocation's processes have been signalled and reaped by systemd's stop sequence, with $INVOCATION_ID, $SERVICE_RESULT, $EXIT_CODE and $EXIT_STATUS in its environment. It writes /run/carlos/instances/<host>/exit.<role>.<INVOCATION_ID>.json (O_EXCL) with those four values. The directory is the instance's own, so the instance could have written that file itself; the record is therefore the tenant's claim about its own exit and is used for one thing only: the wording of the member-facing event line. It never decides whether the process is gone, and a record for a running unit is ignored (§3.3). The agent, as carlos, removes it once consumed, and prepare removes stale ones (R3-7).
Each unitProc runs one goroutine that establishes termination from two facts the tenant cannot forge:
- systemd's unit state:
systemctl show -p ActiveState,SubState,Result,InvocationID <unit>(read-only, unprivileged), polled every 2s and triggered early by inotify on the run directory (the record's arrival) and on the unit'scgroup.eventsfile. Terminal isinactiveorfailedwith the proc's ownInvocationID, orinactivewith an emptyInvocationID(the unit was already unloaded; the proc's invocation is gone). - cgroup emptiness: the unit's cgroup directory under the host slice is absent, or its
cgroup.eventssayspopulated 0. A service entersfailedafter its finalSIGKILLtimeout with processes still present if one is unkillable (src/core/service.c:4020-4022), and the lease must not be released over a writer (R2-5).
Only when both hold does the goroutine emit the event line with today's wording (instance exited: ... with the status or signal from the record if one exists for this invocation, or instance exited (exit 0), or instance exited (status unknown: no exit record)), set exited, and close done. A populated cgroup after a terminal state is logged every 2s as instance <host>: processes remain after SIGKILL; not releasing, and done stays open (§6.3); that is a box-level incident, not a lease release.
An InvocationID that is neither the proc's own nor empty means someone restarted the unit outside the activator, which Restart=no does not forbid. The goroutine logs unit restarted outside the activator and POST /stops the unit itself, then keeps waiting: done closes only when the unit is terminal and its cgroup is empty, whichever invocation the processes belonged to (codex R2-5 disposition). The agent owns these units and any other invocation is an anomaly to end, not to adopt; the invocation id decides the event line's wording and nothing about whether a writer is gone.
One exception, after the activator itself has asked for the stop (Stop was called, or a failed health check stopped a fresh unit): a unit that then reports an invocation that is not the proc's own, or none at all while not terminal, has been replaced. systemd starts a new invocation only once the previous stop job completed, which is its own termination proof for the old one, so the goroutine emits instance exited (status unknown: replaced after stop) and closes done without the cgroup gate (the cgroup now belongs to the new invocation and would never read empty). The goroutine does not stop the replacement: a stop it did not request is not its anomaly to end, and a discarded proc must never be able to stop a later, legitimate wake (plan 3, task 4 review).
The seed's outcome comes from systemd, always. The seed unit carries RemainAfterExit=yes (§4.2), so it stays loaded, with its Result and ExecMainStatus, until the control service reads them and stops it (§5.3). /seed therefore always returns a complete result and the seed's exit record is, like every other record, wording only.
sweepOnce's existing dead-child check (activator.go:1105-1122) sees a flag, never a blocking call. Restart=no in the template is what makes this sound: systemd never restarts the unit behind the activator's back; a failed unit stays failed until the agent's next start, which clears it.
6.6 cgroups, tiers and the sampler (codex R1-8, R1-9, R2-13)
internal/cgroup today: Manager.Start (cgroup.go:242-285) clones the child into carlos-exec@<host> under the edge unit's delegated subtree, writes memory.max/memory.high/memory.swap.max from the tier and memory.oom.group=1, and the sampler reads the subtree as a scan root (exec-child accounting design §4). CleanupHost (:398-403) removes the directory.
Under units, systemd owns the cgroup. So:
- Tier travels in
POST /start/POST /seedand lands on the host slice as the three properties of §4.3. TheTier func(registry.Route) cgroup.Limitseam and its box wiring (stagedManifestOfLiveVersion) are unchanged; only the consumer changes. The agent logsinstance: <host>: memory.max=…on every start as it does today. Seed, instance and sidecar share the one cap. - Per-host sampling gains
/sys/fs/cgroup/carlos.slice/carlos-instances.slice/as a scan root and reads eachcarlos-instances-<esc>.slice/memory.current(the slice's charge is the family's total, so no per-role summation is needed). It publishes under the existing keyactivator.ExecName(host)(carlos-exec@<host>), because that is the keyinternal/console/metricsview.go:728-762recognises and everything else is discarded at the app boundary there. Flat hosts, for this one release, are still sampled from the delegated subtree as today; the two roots never both hold a given host. - The edge's own reading. Platform-unit discovery scans
/sys/fs/cgroup/system.slice(internal/host/metrics_probes.go:49), wherecarlos-edge.serviceno longer lives once it moves intocarlos.slice, and the accounting design defines the edge's reading as the hierarchical total including its children. So discovery adds/sys/fs/cgroup/carlos.sliceas a root for the edge unit, and the value published undercarlos-edge.servicebecomescarlos.slice'smemory.current: the agent plus every instance, which is what the key has always meant (R2-13). The edge unit's own leaf is published additionally undercarlos-edge.service/selffor the box view; the platform graph is continuous across the roll. Manager.Start,CleanupHost, the clone3 path and the delegation dance stay in use for flat hosts during this release and are deleted with the flat spawn path in the next (§8.6).Delegate=on the edge unit goes with them.
6.7 Agent start and restart; the sidecar restart (codex R1-7, R2-6)
Today an agent restart kills every child (Pdeathsig, the cgroup tree dies with the unit). BindsTo=carlos-edge.service reproduces the stop half of that for a systemctl restart carlos-edge and does not propagate the start. For a crash, or a stop that did not propagate, the agent on start, before anything else:
POST /sweep(§5.3). The control service, as root, enumerates everycarlos-instance@*,carlos-instance-seed@*andcarlos-instance-sidecar@*unit and every pending or running job on them,systemctl stops each (which also cancels a queued start), waits for each to reach a terminal state and for its cgroup to be empty (§6.5's rule), and returns the list. The sweep is keyed on the unit names systemd reports, not on hosts, so it covers a unit whose identifier is hashed and whose registry row is gone: the agent could not name such a unit through/stop, which takes a host (codex R7-1).- The agent removes stale exit records under every run directory.
Every instance is therefore asleep, and no job is queued, when the activator's state machine begins, which is the invariant the activator holds today. Adopting a running unit instead (re-acquiring its lease, re-attaching replication) is a later improvement with its own review; it is not needed for correctness.
The first activator.go change. sweepOnce restarts a dead sidecar by calling StartSidecar under a.mu (activator.go:1165-1200), justified there by "an exec is quick". A unit start is an HTTP call and a systemctl start, and a stall there would hold every other host's wake and stop. So the sidecar restart becomes: under a.mu, mark e.sidecarStarting = true, capture the entry's wake generation, and schedule the backoff exactly as now; then in a goroutine, StartSidecar with its 30s deadline; then re-take a.mu, and if the entry is still awake with the same generation, install the proc as today, otherwise Stop it. The lock still makes a restart impossible to race with a stop, because the install step checks the state under the same lock the stop flips it under, and a stop that arrives during the start finds sidecarStarting and waits for the install step to see it. One test drives a stalled sidecar start while another host wakes and stops.
6.8 The wake runs on the committed row, not on a copy (codex R8-1)
The edge hands the activator the route it proxied for, and the activator copies it into the entry (activator.go:481); doWake then restores, seeds, spawns and replicates on that copy (:806, :870). The copy can be older than the registry: for the interval between a layout pass's row CAS and the edge's reload, the edge still holds the flat paths, and a request in that interval would carry a flat route to a host the registry already calls nested. On that copy the exec backend would classify the host flat, restore a second database at the vacated flat path, and run childProc, and RetireExcept (activator.go:1260) would keep the entry because its host name did not change.
The third activator.go change: at the top of doWake, before Acquire, the activator re-reads the row (reg.Get(host), one indexed read) and uses that row for the whole wake, restore through replication, storing it back into the entry. If the row's DBPath or Socket differ from the copy the edge supplied, the wake logs route refreshed: <old> → <new> and proceeds on the fresh one; if the host has no row, or the row is fenced or conflict, the wake refuses as §8.1 says. The exec backend's classification (§8.1) therefore always sees the committed row. One test delivers a saved flat route for a migrated host and asserts no flat artifact is created and no childProc starts.
6.9 What does not change
Acquire, Restore (including the .corrupt quarantine and PurgeLocalDBArtifacts, all keyed off dbPath), lease.Replicate (the meta dir is derived from dbPath, lease.go:286, so it lands inside the instance directory), NextDue, the schedule runner's token read, the control socket's purge and schedule mounts, Bounce, the sweeper's hibernation policy, residence moves (movespass.go, once it reads the DB path from the row and honours the layout journal, §8.2 and §9), carlos restart in all three scopes, X-Carlos-Version, the caps pass, config-ack. All of these see a path change and nothing else.
---
7. Config delivery
SyncExec (internal/appconfig/syncexec.go) writes a nested exec route's bundle to InstanceDir/<host>.config.env instead of <DataDir>/<host>.config.env, through the O_EXCL-temp-and-rename helper of §3.2 rather than os.WriteFile on a predictable .tmp name (syncexec.go:921-926; R1-3), and a flat route's exactly as today. Its ownership rule ("sync owns *.config.env outright, never touches <host>.env or <host>-sidecar.env", syncexec.go:1-31) is unchanged and, for a nested host, trivially true per directory. reapConfigEnv (:977-1024) walks <DataDir>/instances/*/ for nested routes as well as the flat DataDir for flat and addr routes, and the bounce-on-change path is unchanged (the agent bounces the entry; the next wake reads the new file).
Opted-in TCP-addr routes stay flat. Their file is read by the app's own supervisor at /data/<host>.config.env (titogo's EnvironmentFile=-/data/%i.config.env), and moving it would break every such drop-in. Isolated instances cannot see it any more, which is the read #127 opens with. #127's request, root-rendered delivery into /etc/carlos/apps for addr routes so that the file is not readable by the agent either, is a separate, smaller change and stays open (§11).
reapConfigEnv's host-x.config ambiguity guard and SyncExec's collision refusal (syncexec.go:389-395) stay for the flat directory, where they still apply, and the migration's ownership preflight (§8.2) is derived from the same rule.
---
8. Two layouts, one release, and the verb that moves between them
8.1 Why nothing moves on upgrade, and nothing is minted nested (codex R2-9, R3-6)
carlos system apply is run by the installed binary: it stages the candidate, restarts the edge, gates it, and on a failed gate restores the previous files and restarts the edge again (systemapply.go:360-372, 738-748); carlos-system-recover.service runs carlos.prev when a journal exists (units/carlos-system-recover.service:17); and a channel rollback from the console (system.go:466) is an ordinary downgrade install. In every one of those paths the code that runs after the downgrade is the old binary, which knows one layout. So:
This release changes nothing about a host on upgrade, and mints nothing nested. routeFor (reconcile.go:299-301, whose DerivesPaths predicate covers unflagged socket instances too, provision.go:280) and carlos add (cli.go:118-136) keep the flat defaults; a provisioning pass that runs during the candidate's gate creates a flat row the old binary serves (R3-6). Unflagged socket routes, whose supervisor owns their paths, keep flat paths in this release and every later one. The only writer of a nested path into a row is the layout pass, and it runs only on a recorded fleet intent.
The exec backend classifies each activator-managed route on every Start:
| Row | Classification | Spawn path | |---|---|---| | DBPath == DefaultDBPath (flat), Socket == DefaultSocket (flat) | flat | today's childProc, unchanged, unisolated | | DBPath == NestedDBPath, Socket == NestedSocket | nested | unitProc (§6) | | a journal exists in /data/layout/<host>.json | fenced | refuses to wake; reason from the journal | | another row on the box has the same InstanceID | conflict | refuses to wake; both hosts named (§3.1) | | anything else | foreign | today's childProc; reported for the operator (§8.3) |
A fleet rolled to this release behaves byte-for-byte as before until an operator records an intent. A rollback before that is a rollback of nothing. The classification is published per route in the status snapshot (§8.4) and is what every other consumer keys on.
8.2 carlos system layout migrate and revert (codex R1-11, R1-12, R2-1, R2-7, R2-8, R2-9, R3-1)
The verb has the shape of carlos restart (#113): the console writes control/layout.json ({id, requested_at, mode: nested|flat, scope}, the newest entry of an append log like restart-scope.json), bumps the serial, and every box in scope converges to it. Scope is the fleet, or --box <label> for one box, which is how a fleet migrates one box at a time with the console showing each box's rows turning nested. The recorded mode is the fleet's intent, and boxes converge to it on every serial move, not once: a row that becomes eligible later (a route flagged for hibernation after the fleet went nested, a freshly provisioned flat row) is migrated on the next pass when it is asleep. An old box never reads the object; an old console answers 404.
On a box, the layout pass runs in the agent (uid carlos, no root needed) and, per host in scope, in this order:
- Eligibility is the activator's own predicate.
Kind == "instance",Hibernate,Backing == "",Tunnel == "",Socket != ""(activator.go:1379-1384,edge.go:630-638). An unflagged instance route with a socket is an externally supervised process (titogo's router family is the live example); its files and rows are not touched, it is not isolated by this change, and status reports itexternal. Under modenested,flatrows migrate; under modeflat,nestedrows revert;fencedrows recover first;foreignrows are reported and skipped. - Quiesce. The pass asks the activator to stop the host (
Bounce-class stop with replication closed and the lease released, the same sequencestop()runs, including §6.3's refusal to release over an unproven stop) and to refuse wakes for it until the pass is done. Nothing is writing. - Plan, in memory, from the row and the host name only, and only for files named after the host. Each source follows its existing contract (codex R4-3): the database is the row's
DBPath, and-wal,-shm, the litestream shadow dir and.corruptare derived from it (backend_exec.go:273, 1041-1057,lease.go:286); the three env files areDataDir-relative regardless ofDBPath(backend_exec.go:785-794, 865-868). The socket is the row'sSocket. And every source must be a file named<host>plus a platform suffix, directly underDataDirorSocketDir: the row'sDBPathmust be<DataDir>/<host>.dband itsSocket<SocketDir>/<host>.sock, which is whatflatmeans (§8.1). A row naming any other path isforeign(§8.3) and the pass does not touch it, because during the transition the registry is writable by every flat tenant (same uid, §1.3) and a row is therefore not a trustworthy instruction to move an arbitrary file: a tenant that could point itsDBPathat/data/console/console.dband have the pass move it into the tenant's directory would have the platform's own database. Destinations are the §3.1 helpers. Nothing is read from the instance directory to build the plan (§3.3). - Ownership preflight (R2-8). For every flat source name, compute every route on the box that could own that name under the flat rules:
<host>.config.envishost's sync-owned bundle unless a route<host>.configexists, in which case it is that route's instance env and is not moved withhost(this isSyncExec's own collision rule,syncexec.go:389-395). Reverting into a flat directory applies the same table in reverse: a name that would land on another route's file is a refusal, never an overwrite. A plan with any ambiguous name is refused for that host with the two owners named. - Destination check. A destination
<host>.dbthat already exists is a refusal (two databases; refusing to choose), never a choice. For the two env files, which an older binary running between a migration and a revert can have minted fresh at the flat path (backend_exec.go:579; R2-9 disposition), an existing destination is moved aside as<name>.aside-<timestamp>in the destination directory and the source takes its place: the source is the file the instance ran with for longer, the aside copy is kept, and the event line names both.<host>.config.envat the destination is simply replaced: sync re-renders it on its next pass regardless. - Journal. Write
/data/layout/<host>.json(O_EXCL):{host, mode, from: {db, socket}, to: {db, socket}, artifacts: [{name, present}], phase: "files"}. Presence is recorded so recovery knows an absent optional artifact (a sidecar env, a quarantine) was absent, not lost (R2-7). From this moment the host isfenced:Start,Seed,Restore, the residence-move pass,SyncExec's writer and the litestream generator all refuse it, with the journal's contents as the reason. TheMoveCuttingpath inmovespass.go:285-305, which would otherwise treat an absent DB as "nothing to checkpoint" and complete a flip, is the case the fence exists for (R1-12). - Files.
os.Renameeach artifact recorded present. Same filesystem, same owner, atomic per file, no root. Then journalphase: "row". - Row.
registry.SetInstancePaths(host, from.db, from.socket, to.db, to.socket): one transaction, compare-and-swap on both columns, a new sibling ofSetDBPath(registry.go:1057-1075). Then journalphase: "done". - Remove the journal; log
layout: <host>: flat → nested(or the reverse); reload routing synchronously (the agent owns the edge and calls itsReload(),edge.go:449-451, and the activator'sRetireExcept/route refresh, rather than waiting for the 2s poll); only then release the host to the activator. A request that copied the old route before that reload is caught by §6.8 (codex R8-1).
Recovery is the journal, and it is exact, and it is checked (R2-7, R3-1, R4 disposition). On agent start and at the top of every layout pass, for each journal, the check is against both endpoints, because the row is at from before step 8 and at to after it, and it is derived from the journal's own direction (codex R5 disposition): for mode: nested, from must be the host-named flat paths of step 3 and to the nested paths the §3.1 helpers derive; for mode: flat the two are swapped, artifact sources included. The row must currently equal either from or to. A journal that fails any of those is refused (layout: <host>: journal does not match row; not acting), the host stays fenced, and status says so. A journal that passes is completed to its own endpoint even if the fleet's intent has since changed: a transaction runs to its recorded end, and the host is then converged to the current intent in the same pass, before it is released to the activator, so a forged or stale journal can at most add one round trip a host never runs unisolated through. Otherwise:
phase: files: for each artifact recorded present: at destination → done; at source → rename; at neither → error, fence stays, status says which file. Artifacts recorded absent are skipped. Then continue from step 7's end.phase: row: if the row already equalsto→ the CAS committed before the crash; continue. If it equalsfrom→ CAS now. Anything else → error, fence stays, both values in status.phase: done: remove the journal.
A second clean agent start finds nested rows and no journals and does nothing (the classification table runs before any legacy-default test).
Revert is the same pass under mode flat, the same journal, the same fence, the same preflight in reverse. It exists so that a fleet can be taken back to a layout the previous release serves before that release is installed: the operator's rollback order is layout revert (this release, verified flat in status) then system rollback. That order is enforced, not advised (codex R5-2). A downgrade removes this release's unit files: installUnits runs systemctl disable --now on every unit the candidate lacks (systemapply.go:522), so the bind mounts go with the binary, /data/instances/<iid>/ becomes an empty mountpoint directory, and the old backend, honouring the nested rt.DBPath (backend_exec.go:220-225), would find no database and restore a second copy into the uncovered directory, which the next mount would hide. So:
- This release's
carlos system applyrefuses to install a candidate that predates the layout (its version is below the release that carries the mount units) unless reversal is complete: no row on the box, whatever its flags, has aDBPathorSocketunder/data/instancesor/run/carlos/instances, no row isfencedorconflict, and no journal exists. The check is by path shape and not by activator eligibility, because a row can leave management while keeping nested paths:SetHibernate(false)(registry.go:1565), including when the reconciler applies a record's flag (reconcile.go:828), does not touch them (codex R7 disposition of R5-2).layout_modeis not consulted either, because it records intent, and a revert refused at quiescence or at the destination check leaves a nested row under a flat intent. The reason names the rows; the installer is this binary for every downgrade from here, because a downgrade is an install performed by the installed binary (§8.1). - Unflagging a nested row is refused.
carlos route --hibernate=falseand the reconciler's flag application both refuse to clearHibernateon a row whose paths are nested, withlayout revert firstas the reason (the reconciler logs it as it logs shape drift,reconcile.go:873-881), so a nested row is always activator-managed and the revert pass can always quiesce it. The classification table (§8.1) therefore never meets a nested row it cannot own. - The console's rollback door (
system.go:466) refuses a channel rollback to a pre-layout release while any box in the fleet reports anynested,fencedorconflictrow, or no layout field at all, and names the boxes and rows. - A failed-gate rollback of the upgrade to this release runs the old binary, but at that moment nothing is nested (§8.1). A failed-gate rollback of a later upgrade runs this binary, which knows the layout.
- Recovery for a layout that was unmounted anyway (an operator removing units by hand): at start the control service (root) inspects the underlying mountpoint directories in a private mount namespace (
unshare -m,umountthe bind there, list) and, if/data/instancesor/run/carlos/instancesholds anything under the mount, records the verdictmountpoint held files while unmountedandGET /readyrefuses nested wakes until an operator has moved or removed them; the layout pass's destination check (step 5) then sees whatever was moved into the raw directory. Nothing is chosen silently; the databases are never two.
The database therefore has one history in every path that is allowed to happen, and the two paths that are not allowed are refused with the revert named as the fix.
8.3 foreign rows
A row whose DBPath or Socket is neither the flat nor the nested default was set by hand (carlos add --db; carlos route has no --db flag, cli.go:469-475, so it was set at add time). The pass does not move it, for the reason step 3 gives: while flat tenants share the agent's uid, a row is not a trustworthy instruction to move an arbitrary file, and the only files the pass will move are the ones named after the host at the platform's own paths. A foreign row is reported with both values, keeps running on today's flat path, and carlos add on the box, re-adding the route with the defaults, is the repair: box-local because the row was made box-locally, and because a remote verb that moved operator-named files would be the primitive this section refuses to build (R1-18, accepted as the residual). A console-provisioned row (reconcile.go:300-301) is never foreign.
8.4 Status, through the API (codex R1-18)
internal/status/snapshot.go gains a per-route layout field with one of flat, nested, fenced (with the journal's phase and reason), conflict (with the other host), foreign (with both values), external, and a box-level layout_mode (flat, nested, or none before any intent). carlos status --layout (beside --instances, under the same operator gate; R14) and the console's instance page render them; a box that predates the field shows unknown (box predates layout reporting). It is the pre-roll inventory, the progress view during a migration, the member-visible proof that an instance is isolated (nested), and the reason a fenced host does not wake. docs/site/cli.md names the new verbs and flags (the docsite gate fails otherwise).
8.5 Gate: no isolation, no nested tenants (codex R1-15, R2-14, R3-4)
A box that cannot establish the namespace does not start nested tenants. There is no fallback unit, no "proceed without PrivatePIDs=". Running the fleet with the same-uid /proc/<pid>/root traversal open is the defect, not a degraded mode.
The gate is observed behaviour, not a version string. At start, and on daemon-reload, the control service (root) runs carlos-instance-probe.service, a oneshot with the exact namespace and IPAddressDeny= block of §4.1 whose ExecStart is carlos instance-probe. From inside it asserts, and reports each as a separate verdict line:
/dataand/run/carloslist nothing but the probe's own bound directories, and/var/cacheand/var/liblist nothing at all;access(W_OK)on the edge's own delegatedcgroup.procs(/sys/fs/cgroup/carlos.slice/carlos-edge.service/cgroup.procs, chowned to carlos byDelegate=oncarlos-edge.service) answersEROFS, neverEACCES(the kernel checks DAC before the mount flag, soEACCESthere means the file is not delegated, not a pass) orENOENT(the edge unit is not loaded); and/proc/self/mountinfoshows/sys/fs/cgroupitself carryingro, so theEROFSanswer is the mount flag's doing and not some other cause;/etc/carloscannot be opened (EACCESorENOENT, never a listing), and the fixture/etc/carlos/probe-secretcannot be read. The fixture is not written by the control service, whoseProtectSystem=strictmakes/etcread-only to it (codex R11-1); it is a tmpfiles line (§3.2,f /etc/carlos/probe-secret 0640 root carlos - probe) applied by bootstrap and byupdate-platform-binary.sh'ssystemd-tmpfiles --create, before any probe runs. The control service, before it accepts a probe verdict,stats the fixture and requiresroot:carlos 0640; a missing or differently-owned fixture is itself anot readyverdict line (probe fixture absent), never a vacuous pass;/proclists only itself, and its own pid is 1;- a symlink it creates in its directory is refused with
ELOOP, and/proc/self/mountinfoshowsnosymfollowon both bound directories; - IMDS, for a packet filter (R3-4).
IPAddressDeny=is a cgroup BPF egress filter; it drops packets, and a TCPconnect()whose SYN is dropped retries and times out rather than returningEPERM(tcp_connect,net/ipv4/tcp_output.c). So the probe sends a UDP datagram, where the filter's verdict comes straight back fromsendmsg:sendto(169.254.169.254:80)must fail withEPERM;sendto([fd00:ec2::254]:80)must fail withEPERM(v6: enforced) or withENETUNREACH(v6: unverified, no route to the address): the kernel's own answer for that exact destination, so no route query is needed and none is made (codex R4-4; the service'sRestrictAddressFamilies=AF_UNIXstands). As a control,sendto(127.0.0.1:9)must succeed (the network stack is live and the filter, not the stack, said no). A 2s TCPconnect()to the IPv4 address must not succeed. Any success on a metadata address, or any errno other than those named, isnot enforced.
The service records the verdict; GET /ready returns it; the agent refuses every nested wake with the verdict's failing line until it is
v6: unverifieddoes not block readiness on its own, because the boxes have no IPv6 route to that address today and a box with no route cannot reach it; it is bounded and watched: the probe re-runs every five minutes and ondaemon-reload, the verdict is part of the status snapshot (§8.4), and the moment a v6 route appears and the answer is notEPERMthe verdict flips tonot enforcedand nested wakes stop (codex R2-14 disposition). The probe cannot pass while an instance would not be isolated, because it runs under the same unit text (a test asserts the probe's namespace block equals the templates'), and it fails only on a line that names the property that is missing.
8.6 Rollout order
- Roll this release to the dev fleet (
kass, two boxes), boxes before console, withupdate-platform-binary.shenabling the new units by name before it restarts the edge (§9). Every row staysflat;layout_modeisnone; the probe verdict isready; nothing else changes. Cold-boot one dev box and confirm the three mounts, the control socket and the probe come up (R2-2). carlos system layout migrate --box <dev box>, watch the rows turnnested, then the second box. The test that matters is from inside: the plan's verification task provisions a throwaway instance whose seed lists/dataand/run/carlos, reads/proc/1/status, sends the three datagrams, and plants and follows a symlink, and reads the results back throughcarlos logs. Runlayout reverton one box andmigrateagain: both directions are exercised before the flagship sees either.- Flagship: roll the release region by region, console box last; then
layout migrateone box at a time. The AU box has failed every gate since 2026-09-07 for an unrelated reason; do not read its failure as this change's. - carlos-tito-ie: Tito's operator rolls it from Tito's console once the flagship has run a day, then migrates. The seven flagged instances move; the router family (unflagged) is reported
externaland left alone. Reply on #126 with the build and the check list above. - The next release removes the flat spawn path for activator-managed hosts, points
DefaultDBPath/DefaultSocketat the nested helpers for flagged routes (and keeps them flat for unflagged ones, whose supervisors own their paths), removes the spawn half ofinternal/cgroupandDelegate=on the edge, and refuses to start on a box whoselayout_modeis notnested. Brought forward in plan 5 (revision 16): on a box whose intent isnested, the reconciler andcarlos addmint a fresh activator-managed row at the nested paths (routeFor'snestedargument), so a new tenant never runs flat there, not even for one pass.DefaultDBPath/DefaultSocketstay flat for every other caller until the next release.
Nested by default (revision 17, Paul's decision 2026-10-06). From the release that carries plan 5 (v0.25.0), nested is the default, not an opt-in. When control/layout.json is absent, the effective intent on a box is nested if and only if the box is ready: the deployment's default is nested (CARLOS_LAYOUT_DEFAULT in host.env is unset or nested; flat is the per-deployment opt-out, for a self-hoster whose boxes are not ready), the running build is at or above LayoutFloor when it can be placed (a build that evaluates the rule carries the pass by construction, so an unplaceable dev build passes and a mis-tagged release below the floor does not), and the nested gate of §8.5 answers: instancefs.SelfCheck, the three nosymfollow mounts, and the control service's probe verdict. A box that is not ready behaves exactly as before the default (flat, mints flat) and says why in its status report (layout.default_blocked); a box that is nested by default says so (layout.default), so carlos status --layout can tell a box that was migrated on purpose from one that moved on its own.
A recorded intent always wins: an explicit flat is the opt-out, an explicit nested behaves as today, and --box scoping is how an operator keeps box-by-box control. The default never forces: the awake grace, the holds and the quarantine of plan 4 apply as they do under a recorded intent. The console's guards need no new rule: the downgrade guard and the migrate guard judge what each box REPORTS (its rows and its running release), so a box that is nested by default is treated exactly like one under an explicit intent wherever it reports nested rows. carlos add cannot reach the gate from an operator's shell, so with no intent recorded it mints nested only when the box's registry already holds an activator-managed row at the nested paths; a fresh box's first hand-added row is minted flat and moved by the pass on the next serial move.
Rollout, restated: the flagship cuts v0.25.0 with this in, rolls it, and each box migrates on its own once its probe reports ready. An operator who wants box-by-box control records carlos system layout migrate --box (or revert) first; the recorded intent then governs.
Why a default and not a verb: the §1.2 list is what every flat tenant can reach, and a fleet that stays flat because nobody ran a verb is a fleet that stays exposed by omission. The gate is what makes the default safe to have: a box that cannot isolate does nothing, loudly.
The roll script and the planner. update-platform-binary.sh:52-70 (pinfra) enables units by name; it gains carlos-instance-control.socket, the three mount units and the probe, before it restarts the edge, and bootstrap.go's systemctlPlan (:165-179) starts .socket and .mount units it enables, which it does not today (R1-14). ensureSystemUnits (systemunits.go:34-83) checks the control socket is active, not just enabled. An agent that finds the channel absent refuses nested wakes with instance control socket not listening, which is what a forgotten enable looks like from carlos logs; flat hosts are unaffected, which is the point of the two-layout release.
---
9. Every consumer of the shape
The rule (AGENTS.md): the consumer list is everything that parses the shape, not everything that calls the API. From the sweep of 2026-09-11, extended by codex R1-9, R1-14, R1-17, R1-18, R2-8, R2-9, R2-10, R2-13, R3-1, R3-6:
Path builders that move onto the helpers
internal/provision/provision.go:280(DerivesPaths, unchanged, but named: it is why unflagged routes mint flat),:1011-1012(DefaultSocket,DefaultDBPath, flat in this release), plusInstanceDir,InstanceRunDir,NestedDBPath,NestedSocket,LogSocketinternal/activator/backend_exec.go:160-162defaultDBPath(deleted);:350, 507, 579, 607, 786, 790(<host>.env,<host>.config.env);:865-868(<host>-sidecar.env);:744-750(socket normalisation, unchanged);:273, 1041-1057(quarantine and artifact paths, the migration's source contract); every nested read throughinstancefs; the classification of §8.1; the path andgetpeernamechecks on the health and control dials (§5.4)internal/edge/edge.go:2000-2014(unixProxy: the path andgetpeernamechecks on every dial, §5.4)- the layout pass's raw-directory renames (
/data/.instances, §3.2): the one caller, pinned by a grep test internal/appconfig/syncexec.go:389-395, 465, 872, 921-926(nested routes into the directory; flat and addr routes flat;O_EXCLtemp);:977-1024reapConfigEnvwalks bothinternal/schedule/runner.go:461andinternal/host/schedule_wiring.go:163(admin token path, by classification)internal/host/movespass.go:285-305— rebuilds<DataDir>/<host>.dbfrom the record; readsrt.DBPathfrom the row instead, and refuses afencedhostcli.go:118-136(carlos adddefaults, unchanged and named),:157-159(relative-path refusal), the newcarlos instances --layoutandcarlos system layout migrate|revert;docs/site/cli.mdnames theminternal/provision/reconcile.go:299-301(routeFormint, unchanged and named);:461-464and:873-881compare derived paths and log anestedrow as drift for this one release (the message names the layout mode)
Readers of rt.Socket (no change; the value changes per host)
internal/edge/edge.go:536-538(dial, rebuilt on everyReload),internal/activator/backend_exec.go:594, 738,internal/status/snapshot.go:487,internal/host/probes.go:273,internal/host/schedule_wiring.go:343-369,internal/schedule/runner.go:237-238,internal/appconfig/configack.go:168,internal/host/socket_advice.go:32-79(its/data/%s.config.envstring is for addr routes and stays), console display sites inapiroutes.go,relayops.go,apirelay.go
The socket consumers group (R2-10): internal/host/bootstrap.go:41-62 (socketsGroup, tmpfilesConf gains the lines of §3.2), the per-host run directory mode in prepare, and the #103 test that dials a migrated instance as an enrolled non-carlos uid
Litestream and replication
internal/lease/lease.go:286— meta dir derived fromdbPath; followsinternal/ops/litestream.go:354-388— the box-wide generator discovers by glob and filters by registry claim; it does not append claimed paths (R1-17). The glob gainsinstances/*/*.db, or nested non-hibernating databases are never replicated. Hibernating rows stay excluded at:420-421as today; afencedhost is excluded. Pinned inops_test.gofor both coverage and theskippedreportcarlos-litestream.serviceruns ascarloswithReadWritePaths=/data: reads nested 0700 directories fine; ordered afterdata-instances.mount
Metrics and status
internal/host/metrics_probes.go:49(discovery: the two new roots),internal/host/metrics_loop.go:659(the edge's published value becomes the slice total),internal/console/metricsview.go:728-762(the key allowlist: unchanged because the sampler publishes undercarlos-exec@<host>)internal/status/snapshot.gogains the per-routelayoutfield and the boxlayout_modeof §8.4;carlos statusand the console instance page render them
The updater and the control plane
internal/host/systemapply.go:360-372, 738-748,carlos-system-recover.service,system.go:466: unchanged, by design (§8.1); the layout verbs are a separate console door and bucket object (control/layout.json), a separate box pass that runs on every serial move, and a separate mark inrestart-marks.json's siblinglayout-marks.jsoninternal/console: thePOST /api/cli/system/layoutdoor (fleet-operator gate, fresh auth),carlos system layoutinsystem.go
Units and installers
internal/host/units/gains: three templates, two slices, three mounts, the probe, the control socket and service (§4, §5, §8.5);carlos-edge.servicegainsSlice=,Requires=/After=on the mounts, and drops its memory lines;carlos-litestream.servicegains theAfter=internal/host/bootstrap.go:61(tmpfiles lines),:116-122enabledUnits(control socket, mounts, probe),:165-179systemctlPlan(starts.socket/.mount)internal/host/systemunits.go:34-83(active check on the socket)update-platform-binary.sh:52-70(pinfra): enables the new units by name before restarting the edgeharden-host.sh(pinfra) writes hardening drop-ins for$TENANT_UNITS; the instance templates carry their own and must not be listed there- The same-uid audit (codex R9-1, R9-2): every unit in
internal/host/units/that runs asUser=carlosor delegates acarlos-owned tree, and every path it writes, is a path an isolated instance must not see. Today:carlos-edge.service(/data,/run/carlos, its delegated cgroup subtree),carlos-litestream.service(/data),carlos-hibernate-verify.serviceandcarlos-restore-verify.service(/var/cache/carlos-verify). A test overunitsFSlists everyUser=carlosunit'sReadWritePaths=,CacheDirectory=,StateDirectory=,RuntimeDirectory=andDelegate=and asserts each resulting tree is covered by aTemporaryFileSystem=,InaccessiblePaths=orProtectControlGroups=line in the templates, so a future unit that writes somewhere new fails the build until the templates hide it. Reads count too (codex R10-1): the same audit covers every file aUser=carlosorGroup=carlosunit reads that is not world-readable by design, which today is/etc/carlos/relay-masters(root:carlos 0640,internal/relay/masters.go:101) and the alternate path the relay design allows for it; the audit's second table lists each such file, its owner and mode, and the template line that hides it (InaccessiblePaths=/etc/carlos). A new credential input for acarlos-group reader is added to that table, or the test fails
Tests that pin the flat layout (§10): internal/activator/activator_test.go (20 fixture sites), backend_exec_test.go (4), admintoken_test.go (8, path-agnostic), internal/appconfig/syncexec_test.go (50), internal/ops/ops_test.go (52), internal/provision/reconcile_test.go (20), internal/host/movespass_test.go (1), schedule_wiring_test.go (1), schedule_token_fence_test.go (2), root e2e_test.go (24), instances_e2e_test.go, config_e2e_test.go, deploy_e2e_test.go. For this release they keep passing on the flat path; each gains a nested twin.
Not consumers: the console never reads box paths (internal/console has no DBPath/DataDir reader); readDiskFrac is whole-volume; the sentinel .path units name fixed files under /data, never a host.
---
10. Tests
Unit level, all runnable in CI (no systemd needed):
internal/provision:InstanceID(identity below 41 bytes, truncation plus hash above, never longer than 40, stable), the nested helpers keyed by it, the flat defaults unchanged,LogSocket, the escape function againstsystemd-escapefixtures, and the socket address preflight: a host that fits today's flat paths but whose identifier-keyed paths would exceed 107 bytes cannot exist, pinned by computing the longest address for a 253-byte host; a short host and a hashed identifier can never be equal (the_prefix); two long hosts colliding inInstanceIDclassify asconflictand refuse to wake, including a host that was alreadynestedwhen the second row arrived.instancefs: refuses a symlink at every component, refuses..,O_EXCLtemp plus rename, the mountinfo check for all three mounts.internal/edgeandinternal/activatordials, with real socket pairs: a path under/run/carlos-control/is refused beforeconnect(); a custom path elsewhere (/run/titogo/example.sock) is dialled as today; a symlink to another listener yields agetpeernamemismatch and is closed before a byte is sent; a listener created by another process and passed over (thecarlos-app@.socketshape) is accepted when its bound path is the one dialled.internal/host/systemapply: a pre-layout candidate is refused whilelayout_modeisnestedor a journal exists, and installed when the box isflat; the control service's start-time inspection of an underlying mountpoint that holds files (with a fake namespace runner).internal/console: the rollback door refuses a pre-layout release while any box reportsnested,fencedor no layout field, naming them.instance-control: a connection whose peer pid is not the edge'sMainPID, or whose uid is notcarlos, is closed without a response; every 400 (bad host, bad role, bad limit); the unit and slice names it builds for a host with a dash;set-property(three properties) →startordering with noreset-failed;/resettolerating "not loaded";/seedreading a retained result from anactive (exited)and from afailedoneshot, waiting for the empty cgroup, then stopping it;/readyfrom the probe verdict; with a fakesystemctlrunner and a fake peer.internal/activator: classification of the four row shapes;preparemints the token for seed and run, writeslaunch.<role>.jsonwith the deduplicated env and the exact argv (the existing env-composition tests move their assertion to the effective environment of an executed stub); the control client's per-verb deadlines (30s, 150s for seed);unitProc.Stoporders stop → done → close listeners and returnsErrStillRunningon expiry without closing;stop()skipsrep.CloseonErrStillRunningand the sweeper retries;Exited()only after a terminal unit state and an empty cgroup, never on log EOF or on a record alone, with a populated cgroup holdingdoneopen; a forged exit record for a running unit ignored; a foreign invocation; the failed-health path postsstopand waits; seed timeout and cancellation stop-and-wait before cleanup; the startup sweep stops activating units, cancels jobs and waits for empty cgroups; the sidecar restart reserves under the lock and installs or stops after, with a stalled start while another host wakes and stops. The control channel is a seam (Control interface { Start; Seed; Stop; Reset; Ready }) with a fake; the exit observer takes a fakeshow, a fake run directory and a fake cgroup tree.instance-exec: reads the file throughinstancefs, refuses schema ≠ 1, refuses anargv[0]that is not the live link, execs.instance-exit: writes the record withO_EXCLfrom the four variables into the run directory; refuses to overwrite.instance-probe: each verdict line, includingEPERMversusENETUNREACH(with and without a v6 route) versus other errnos on the datagram sends, the loopback control, and the TCP connect bound, with a fake socket layer.logListener: one connection per invocation, a second refused and logged, line cap, EOF ends collection without marking exit.- Layout pass: eligibility by the activator's predicate (flagged only; external, tunnel, unit-backed untouched); convergence on every serial move; a later-flagged row migrating on the next pass;
foreignrows (a custom basename, a subdirectory, a path outside the trees, a row pointing at the console database) reported and never moved; quiesce before plan; the ownership preflight forx/x.configin both directions; DB destination collision refused, env destination moved aside; the rename performed through the raw.instancesdirectory fds and never through the bind mount (a test with a real bind mount whereEXDEVwould otherwise fire, skipped where the CI host cannot mount); the journal's phases and every crash boundary (before files, mid-files, after files, after CAS, before journal removal) recovered exactly, with absent optional artifacts, with the row at either endpoint; a journal whosetois not the host's derived destination or whosefromis not the host's flat paths refused; a journal completed to its own endpoint and the host then converged to a changed intent before release; a CAS conflict keeps the fence; a second clean start does nothing; every fenced consumer (Start,Seed,Restore, moves pass,SyncExec, litestream generator) refuses a fenced host; a partial rename followed by aMoveCuttingpass does not complete the flip; revert restores the flat layout byte for byte, aside files included; a forged journal, a forged launch file and a forged exit record move nothing (§3.3). - Units: a test over
unitsFSasserts the three templates' and the probe's namespace, hardening and coupling blocks are identical, thatPrivatePIDs=yes,TemporaryFileSystem=/data:ro,IPAddressDeny=169.254.169.254/32,Restart=no,BindsTo=carlos-edge.service,Slice=carlos-instances-%i.slice,OOMPolicy=kill, anExecStopPost=without+, and the%Ipaths are present, that the seed carriesTimeoutStartSec=120, that the control socket isSocketMode=0600 SocketUser=carlosunder/run/carlos-control, that all three mounts carrynosymfollowandDefaultDependencies=nowith the ordering of §3.2, and thatcarlos-edge.servicecarriesSlice=carlos.sliceand noMemoryMax=; and, wheresystemd-analyzeexists on the CI host,systemd-analyze verifyover the rendered set. internal/ops: the nested glob, coverage of a nested non-hibernating row, theskippedreport, the fence.internal/host:systemctlPlanstarts sockets and mounts; the two sampler roots publish per-host undercarlos-exec@<host>and the edge as the slice total, flat hosts still from the delegated subtree; the #103 consumer dial through the nested run directory.internal/status: thelayoutfield for each of the six states, the box mode, and the pre-field box.internal/consoleandsystem.go: the layout door's gate and refusal order, the bucket object, the scope forms, convergence marks.
Box level, in the plan's verification task and in the rollout:
- Cold boot with the full unit set;
systemd-analyze verify;list-dependenciesshows the mounts before the edge. - Probe verdict
ready; a deliberately broken probe (a drop-in removingIPAddressDeny=) turns itnot enforcedand blocks nested wakes. - Migrate one host; wake it;
systemctl status carlos-instance@<esc>active incarlos-instances-<esc>.slicewith the three memory properties; from inside via the throwaway instance's seed:/dataand/run/carloslist one directory each,/proc/1is the app, the three datagrams answer as §8.5 says, a planted symlinkELOOP;carlos logsshows all of it for both instance and sidecar at once; a quick-exiting seed's status arrives; hibernate, unit gone, record consumed;carlos restartcycles it;systemctl restart carlos-edgestops it and does not start it; an enrolledcarlos-socketsuid dialsapp.sockand cannot open the control socket; a flat tenant on the same box cannot get a response fromcontrol.sockdirectly or through its own redirected socket;carlos instances --layoutsaysnested; the console's memory graphs for the app and for the platform are continuous across the migration; revert, and the previous release serves the host again.
---
11. Out of scope, and follow-ups
- Per-instance uids (option B). The nested layout is its prerequisite; the namespace closes every path the discussion names without it. If a later finding needs a uid boundary, it lands on this layout with a root chown in the control service. Not in this change.
- Externally supervised exec routes (unflagged instance routes with a socket, §8.2 step 1): not isolated by this change; their supervisor owns their process and their flat paths. Whether the platform should offer them the same namespace is a discussion to open once this lands.
- Unit-backed console instances. Not needed for isolation, and a hibernation rewrite. Whether
backing=unithibernation works on the fleet today (UnitBackendrunssystemctlunprivileged, §1.4) is its own discussion; the control channel of §5 is deliberately not widened tocarlos-app@(R1-19). - #127: root-rendered config for opted-in addr routes into
/etc/carlos/apps. Smaller and independent; this change makes the file invisible to isolated instances but not to the agent. - Adopting running units on agent start instead of stopping them (§6.7).
- The next release (§8.6 step 5): drop the flat spawn path and the spawn half of
internal/cgroup; with it, the transition-only exposures §3.2 and §6.4 name (a flat tenant redirecting its own socket, a flat tenant writing to a log socket) end. - Whole-family OOM across instance and sidecar (§4.3): not reproduced; recorded as a deliberate difference.
- Discussion #1 (memory pressure): no new numbers here (§4.3); the slice shape makes a later change a unit-file edit, never a drop-in.
11.1 Hardening after the layout (#214 items 1-3, #178), and what it leaves
Added to the templates, the control service and the probe (each property has a probe line, so ready reflects it):
- CPU and IO. The control service writes the tier's
CPUWeight=,CPUQuota=andIOWeight=beside its memory caps on the host slice (cgroup.SharesOf: 100/100%/100, children tier 100/200%/100). Nothing new on the wire: the shares follow from the memory tier. Residual:IOWeight=is enforced only by io.cost or BFQ; the probe saysunverifiedon a box with neither, which is the default on an EBSnonescheduler. Enabling io.cost is a box change, not made here. - Disk. No per-tenant quota:
/datais a directory on the root ext4 filesystem, and project quotas needtune2fs -O project,quotaon an unmounted filesystem plus aprjquotamount; every writer is uid carlos, so reserved blocks cannot tell a tenant from the agent. The guard refuses rather than stops: below max(1 GiB, 10%) free, starts and seeds answer 507, and the control service logs a warning every 30s while it lasts. Stopping the "biggest" tenant was tried and dropped in review: space held by unlinked-but-open files appears in no directory, so attribution can be gamed into stopping honest tenants, and a stop from the control service bypasses the activator's fence and holds. Residual: tenants already awake can keep writing past the floor until the disk is full. A dedicated/datavolume with project quotas (or systemd's quota directives on top of them) is the real fix. - The version store. Each instance sees an empty
/opt/carloswith/opt/carlos/carlosand its own build put back: the control service writes/run/carlos-control/exe/<iid>/{app -> /opt/carlos/live/<host>, live/<host> -> ../app}before every start, and the unit binds them at/opt/carlos/appand/opt/carlos/live.carlos-app@gets the same view by binding/opt/carlos/live/%iover an empty/opt/carlos. - Egress.
IPAddressDeny=covers loopback, RFC 1918, CGNAT, link-local, ULA, NAT64, site-local, multicast and reserved ranges. An allow is per address, both ways and on every port, so no loopback resolver is ever opened. That includes resolved's 127.0.0.53/54 stubs, because a tenant could bind the stub address and reach every wildcard listener through it. The namespace binds/run/systemd/resolve/resolv.conf(resolved's upstreams, when present) over/etc/resolv.conf. Before every probe, the control service re-opens the nameservers in that same file withIPAddressAllow=oncarlos-instances.slice. Link-local resolvers are refused except 169.254.169.253: an allow carries no zone, sofe80::1would open that address on every link. A nameserver that is one of the box's own addresses is never opened either: the control service reads them from procfs (/proc/net/fib_trieLOCAL routes,/proc/net/if_inet6) as it writes the allow, closes every resolver if it cannot, and reports a refusal as a failingresolver-localline. The probe'sdnsandresolver-portslines check from inside that each allowed resolver is reachable and does not answer on port 22. (The probe made the local-address check itself until v0.25.0, withnet.InterfaceAddrs; that is netlink, which the unit'sRestrictAddressFamilies=refuses, so no box was ever ready; #126.) Residuals: an allowed upstream is open on every port (it is a remote resolver, not the box), and a resolver change reaches running instances only at their next start. Loopback inside one instance is closed too, so a family that talks to itself over TCP loopback must use a unix socket in its own directory. Services the box exposes on a public interface address remain reachable, as from the internet. - IPC (#178).
PrivateIPC=yesand a private/dev/mqueue. Abstract unix sockets are scoped with Landlock (LANDLOCK_SCOPE_ABSTRACT_UNIX_SOCKET, ABI 6, Linux 6.12) by the trampoline before it execs; a kernel without it fails the probe and the trampoline refuses to exec. Residual: a sibling can still bind an abstract name first, which is a refusal of service to the owner, never a connection to it. A per-unit network namespace would close that too, at the cost of a veth or proxy for egress; not done.
---
12. Approaches considered
- Console-provisioned instances become unit-backed (#126 option 1). Rejected:
UnitBackendhas no restore, seed, replication, hibernate- from-replica, sidecar or move;provision.Recordcannot express a backing and the reconciler refuses one. A rewrite of self-serve hibernation, and the always-awake design already said so. - Per-instance uid from the agent (#126 option 2 as written). Impossible without
CAP_SETUIDon the agent, which is root-equivalent for a compromised agent; and a uid alone does not hide a sibling's socket file or pid. - Isolated exec: the agent's instance, systemd's namespace (this spec). Keeps every activator invariant, needs one root channel, and migrates with a rename.
Within 3: trampoline over EnvironmentFile= (§6.2, one env composer); three templates over one (§4.2, the instance is the host); the control service authenticating the agent's pid over polkit and over a token (§5.4, ruled with codex across two rounds); nosymfollow mounts over per-read symlink checks (§3.2, kernel-enforced and race-free where a check-then-open is neither); termination from unit state plus cgroup emptiness, with the unit's own record as decoration, over any record as proof (§6.5); and an operator verb recording a fleet intent that boxes converge to over an agent-start migration (§8.1, because the updater's rollback runs the old binary).
---
13. Review log
- Round 1 (codex, GPT-6,
CODEX-FINDINGS-ISOLATED-EXEC.md, 19 findings, verdict request changes). All accepted: R1-1 control service,nosymfollow(§3.2, §5.4); R1-2 log sockets outside tenant mounts, per role (§3.1, §6.4); R1-3nosymfollowplusinstancefsandO_EXCLtemps (§3.2, §7); R1-4 exit from unit state plus cgroup emptiness, never from EOF (§6.5); R1-5 seed timeout, stop verb, wait before cleanup (§4.2, §5.3, §6.2); R1-6 per-role log listeners (§6.4); R1-7BindsTo=notPartOf=, sweep covers jobs (§4.1, §6.7); R1-8 per-host slice carries the tier for all three roles; whole-family OOM recorded as a difference after R3-3 (§4.3); R1-9 cgroup path undercarlos.slice, key compatibility,metrics_probes.go(§6.6, §9); R1-10 one common wall incarlos.slice, no new numbers (§4.3); R1-11 eligibility is the activator's predicate (§8.2); R1-12 the journal fence and its consumers (§8.2); R1-13 explicit reverse migration; no "restore" claim (§8.2); R1-14 planner starts sockets and mounts; roll script; active check (§8.6); R1-15 noPrivatePIDs=fallback; behavioural gate and probe (§8.5); R1-16 env deduplication, effective-value test (§6.2); R1-17 nested glob is discovery, not reporting (§9); R1-18layoutstatus through the API and CLI;docs/site/cli.md; the box-local repair of aforeignrow accepted as the residual, with the reason (§8.3, §8.4); R1-19UnitBackendclaim removed (§1.4, §11). - Round 2 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R2.md, 14 new findings, verdict request changes). All accepted: R2-1 journal outside tenant mounts, plan from row and host only (§3.3, §8.2); R2-2 mounts withDefaultDependencies=noand explicit ordering, cold-boot verification (§3.2); R2-3 noreset-failedin/start(§5.3); R2-4 termination not dependent on unit retention;/seedcorrelation (§5.3, §6.5); R2-5 cgroup emptiness beforedone;stop()does not release overErrStillRunning(§6.3, §6.5); R2-6 sidecar restart reserved under the lock, started outside it with a deadline (§6.7); R2-7 journal records artifact presence and phase; committed-row recognition; classification before legacy test (§8.1, §8.2); R2-8 ownership preflight in both directions fromSyncExec's rule (§8.2); R2-9 two-layout release and operator verb; same-boot rollback behaviour stated; env aside rule (§8.1, §8.2); R2-10carlos-socketstraversal preserved on the run directories (§1.1, §3.1); R2-11MemorySwapMaxon the host slice (§4.3); R2-12 seed prepares the admin token and merged env (§6.1); R2-13 edge discovered incarlos.slice, published as the slice total (§6.6); R2-14 probe distinguishes enforcement from unreachability (§8.5, reworked by R3-4). - Round 3 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R3.md, 7 new findings, verdict request changes). All accepted: R3-1 no token; the control service authenticates the edge'sMainPID; every dial checks path andgetpeername(the root-peer rule of revision 4 withdrawn by R4-1);/run/carlos/logson anosymfollowmount; journals checked against both endpoints (§3.2, §3.3, §5.4); R3-2 exit record fromExecStopPost=without+, advisory only, in the run directory (§4.1, §6.5); R3-3 no manualmemory.oom.group; per-service families, difference recorded (§4.3); R3-4 UDPEPERMprobe with a loopback control and a no-route rule for v6 (§8.5); R3-5 150s seed deadline (§5.3); R3-6 nothing minted nested in this release; unflagged routes flat by contract (§3.1, §8.1, §8.6); R3-7 the record lives where the agent can delete it (§6.5). - Round 4 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R4.md, 4 new findings plus partials, verdict request changes). All accepted: R4-1 root-peer rule withdrawn; path allowlist before the dial andgetpeernameequality after it (§5.4, §6.1); R4-2/data/.instancesbound onto/data/instances; the pass renames on one mount through raw directory fds (§3.1, §3.2); R4-3 sources follow each file's existing contract; only host-named files at the platform's paths move; everything else isforeign(§8.2, §8.3); R4-4 no route query; the v6 answer comes fromsendtoitself; the probe re-runs and the verdict is watched (§8.5); R2-4/seedcorrelates by--no-blockthenInvocationID; a seed record may decide a seed's outcome because it can deceive only its author (§5.3, §6.5); R2-5 a foreign invocation is stopped anddonewaits for the cgroup (§6.5); R2-7 journals validated against both endpoints and completed to their own end before convergence (§8.2); R2-14v6: unverifiedbounded by the five-minute re-run and reported in status (§8.5); R1-18 theforeignresidual accepted with its reason (§8.3). - Round 5 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R5.md, 3 new findings plus 2 partials, verdict request changes). All accepted: R5-1 only the control endpoint is refused by path; custom sockets elsewhere dial as today (§5.4); R5-2system applyand the console's rollback door refuse a pre-layout release while any box is nested; underlying mountpoints inspected at start (§8.2); R5-3InstanceID, bounded to 40 bytes, keys every directory, socket and unit name; addresses preflighted against 107 bytes (§3.1, §4, §6.2); R2-4 the seed unit carriesRemainAfterExit=yes, so its result is retained until read; the record is wording again (§4.2, §5.3, §6.5); R2-7 recovery endpoints derived from the journal's own direction; §3.3 reconciled with §8.2 (§3.3, §8.2). - Round 6 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R6.md, 1 new finding plus 2 partials, verdict request changes). All accepted: R6-1 hashed identifiers carry a_prefix no host can have, 64-bit hash; a collision isconflict, which refuses to wake and is never the unisolated fallback; the create door refuses a colliding host (§3.1, §8.1); R5-2 the downgrade guards require completed reversal (nonested,fencedorconflictrow, no journal), not intent (§8.2); R5-3 the control service builds unit and slice names fromInstanceID(host)(§5.3). - Round 7 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R7.md, 1 new finding plus 1 partial, verdict request changes). All accepted: R7-1 the startup sweep isPOST /sweep, performed by the control service on the unit names systemd reports, so a hashed identifier without a row is still stopped (§5.3, §6.7); R5-2 the downgrade guard checks every row's path shape regardless of flags; unflagging a nested row is refused (§8.2); §10 aligned with theconflictrule. - Round 8 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R8.md, 1 new finding, every prior item resolved, verdict request changes). Accepted: R8-1 the wake re-reads the row beforeAcquireand runs on it; the layout pass reloads routing before releasing the fence (§6.8, §8.2). - Round 9 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R9.md, 2 new findings, every prior item resolved, verdict request changes). Both accepted: R9-1ProtectControlGroups=yesin the templates and the probe; the probe writescgroup.killand expectsEROFS(§4.1, §8.5); R9-2/var/cacheand/var/libhidden; the same-uid audit rule and its test added to §9 (§4.1, §8.5, §9). - Round 10 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R10.md, 1 new finding, every prior item resolved, verdict request changes). Accepted: R10-1InaccessiblePaths=/etc/carlosin the templates and the probe; the audit extends to credential inputs acarlos-group process reads (§4.1, §8.5, §9). - Round 11 (codex,
CODEX-FINDINGS-ISOLATED-EXEC-R11.md, 1 new finding, every prior item resolved, verdict accept with corrections). The correction is applied in this revision: R11-1 the probe fixture comes from tmpfiles at bootstrap, and the control service verifies its ownership and mode before accepting a verdict (§3.2, §8.5). - R12 (final review, plan 2): cgroup-ro targets the edge's delegated cgroup.procs; DAC precedes the mount flag so a root-owned file answers EACCES.
- R13 (task 4 review, plan 3): after a stop the activator requested, a unit reporting a foreign or absent invocation proves the old invocation over without the cgroup gate (§6.5); the observer never stops an invocation it did not start once its own stop was requested.
- R14 (plan 4, 2026-10-04; recorded in revision 16): the per-host layout view is
carlos status --layout, beside--instancesand under the same operator gate, notcarlos instances --layout: the instances verb lists the local registry, and the layout report lives in each box's private status snapshot, which only the console reads (§8.4). - R17 (plan 5 follow-up, Paul's decision 2026-10-06; revision 17): nested is the default when no intent is recorded, gated on the box's readiness;
CARLOS_LAYOUT_DEFAULT=flatis the deployment opt-out and a recordedflatintent the fleet opt-out (§8.6, "Nested by default"). - R15 (first roll, 2026-09-12; fixed 2026-09-25): the probe on dev's boxes answered
ready: falsetwice, and the review of the fix found a third gap. (a)RestrictSUIDSGID=yesmakesopenat2fail with ENOSYS on systemd 257, because systemd cannot read the mode insideopen_howand so blocks the whole call; everyinstancefsopen in the namespace is anopenat2. The line is dropped from all four templates. What it guarded is covered another way: with no capabilities an instance can only make a set-id file owned bycarlos, which gains nothing for anothercarlosprocess;NoNewPrivileges=yesvoids it inside the namespace; and no other uid can reach one (the tenant trees arenosuid, andPrivateTmp='s/tmpand/var/tmpsit under a root 0700 directory,/var/tmpbeing on the root filesystem and notnosuid). (b)control-unreachablecheckedLstat, butInaccessiblePaths=leaves a mode 000 root-owned node at the path. The check is now what a tenant could do, a listing of the directory and a dial of the socket, and passes only when both are refused for permission or absence; a refused connection or a timeout proves the path reachable and fails. (c)PrivateDevices=bind-mounts the host's/dev/shm, so every instance, all uidcarlos, shared one; each unit now gets its own tmpfs there, and the probe'sshm-privatecheck plants a file in the host's/dev/shm(the control service's tmpfiles fixture) and fails if it is visible. POSIX message queues and abstract unix sockets are still box-wide; they are a follow-up (§11). (d) 2026-10-04, plan 4:carlos-edge.servicecarried the same line, and the agent runs there: its nestedinstancefsopens and every rename of the layout pass areopenat2calls, so the firstlayout migratewould have met ENOSYS on its first host. The line is dropped from the edge unit too. The argument in (a) carries: the edge's one capability,CAP_NET_BIND_SERVICE, does not bear on set-id files (that takesCAP_CHOWN,CAP_FSETIDorCAP_SETUID), so anything the agent could mark set-id is owned bycarlos, and the agent and its flat children run underNoNewPrivileges=. Anopenatfallback ininstancefswas considered and rejected: it would be a second resolution path, exercised in production only when the hardening had regressed, and it would route around the regression rather than report it. Instead the agent runsinstancefs.SelfCheckin its own unit at start, and the nested-wake gate (§8.5) asks it first, so a blockedopenat2refuses nested wakes and the layout pass with the directive named. - R16 (plan 4 review, 2026-10-04): three gaps in the pass while flat tenants still share the agent's uid. (a) A tenant could make its own future directory under
/data/.instancesahead of the pass and hard-link a sibling's database orlitestream.envinto it (its own uid's files, soprotected_hardlinksallows it);EnsureDirsaccepted the directory and the destination check looked only at the planned names. Now a migration refuses an instance directory holding any entry the plan does not name, refuses any file about to move that has more than one name (the shadow directory walked), and checks both again after the renames and before the row flips; a failure there keeps the journal atfilesand the host fenced. (b) Deferring an awake host (plan 4 amendment, "when it is asleep") was unbounded, so a tenant that never idled kept raw-path access to its nested siblings for good. The deferral is bounded: an hour of grace (layout.DefaultAwakeGrace), after which the pass stops the host as §8.2 step 2 always allowed and moves it; the serial leg re-runs on its own ten minutes after any deferral so the bound holds on a quiet fleet; the intent gainsforcefor an operator who wants no grace; andcarlos status --layoutwarns on every box with nested rows beside flat ones, because that box is not isolated yet. (c) Files moved aside at a destination (<name>.aside-<unix>) were kept inside the instance directory, readable by the tenant that gets the directory; they are kept under/data/layout/asidenow, which no instance namespace contains. Still open: a flat tenant that is running while another host moves can plant between the check and the rename; the check is against what was planted before the pass arrived, and the closure is the bound in (b). (d) Re-check: (a)'s refusals were themselves escapable: a tenant could keep its own move refused with one syscall (ln /data/B.db /data/.x, or a file in its own future directory), and because the pass lifted its fence on a refusal the host woke flat again on its next request, for as long as it kept the refusal alive. The rule is now fail-closed on the way to nested: a flat host that has been stopped for its move and is then refused stays fenced and down (the pass keeps the fence, as it already did after a failed handover), status reports itfencedwith the refusal (held down after a refused move to nested), every later pass retries from the kept fence and releases the host once it has moved, and a flat intent lifts the fence. A nested host refused its revert is not held: it is already isolated. The mixed-layout warning countsforeignandexternalrows besideflatones, since all three run as uid carlos with no namespace; and the second-name refusal reports the inode and thefind <data> -xdev -samefile <path>that lists the other names. (e) Landing follow-ups: a planted ENTRY in a host's future directory is no longer a refusal (onetouchby a sibling would have held the host down until an operator acted, and the hold outlived the sibling's own migration); it is renamed into/data/layout/aside/<host>/with a random, no-replace name and the move goes on, logged and shown in status as a note. The hold stays for a second name (nlink > 1, no safe automatic fix) and for a symlink at the instance directory. The hold is persisted as/data/layout/<host>.holdand restored at agent start before anything wakes, by re-deriving it from the files: the marker is a hint to re-check, not the authority, because every file under/datais uid carlos's and a flat tenant can unlink it; a deleted marker therefore costs one grace after one restart, after which the next pass meets the same refusal and holds again, and a marker whose reason is gone is dropped rather than honoured. The serial leg re-runs on its own after a held result, as it does after a deferral, so a cleared reason is acted on within ten minutes on a quiet fleet.