carlos / platform Public

Sidecar isolation spec, revision 4: answers Astra round 3

Round 3 (committed unedited as -R3.md) found every hole in revision 3's
automatic transitions. Revision 4 removes them: CARLOS_NESTED_WORKERS and
CARLOS_FLAT_EXEC are explicit per box in root-owned host.env; worker
eligibility is fixed at agent start; install refuses a pre-floor binary
whenever host.env opts into workers (R3-1); no ratchet (R3-2, R3-4, R3-5);
D9's replicator check needs the config to exclude the file and an exact
running or stopped state (R3-3).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude agent, operated by paul@keymail.dev 9f2cd284316b2c819369c982b3399d256683f15a parent ce2bfb2
2 files changed, +277 −172
  • docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md +179 −172
  • docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md +98 −0
diff --git a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
index c48fdc073..c4c8a44aa 100644
--- a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
+++ b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
@@ -1,6 +1,6 @@
# Sidecar isolation: native sidecars on the nested tier, and no flat exec by default
-**Date:** 2026-10-24 · **Status:** revision 3, approved to build by Paul
+**Date:** 2026-10-24 · **Status:** revision 4, approved to build by Paul
2026-10-24 ("let's just do this and close the vulnerability and roll a
release when done"), for an independent
review (brief: `CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md`) and then Paul ·
@@ -22,6 +22,15 @@ not spawn history (D15); the create check uses the reconciler's own
placement set (D17); and fence retirement needs a complete process census
(D18).
+**Revision 4** answers Astra's round 3 (`CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md`,
+verdict reject: three P1s, two P2s), marked **(R3-n)**. Every finding was in
+an automatic transition that read flat-writable state or raced the agent,
+so revision 4 removes them: `CARLOS_NESTED_WORKERS` and `CARLOS_FLAT_EXEC`
+are explicit, per box, in root-owned `host.env`; the ratchet is gone; the
+agent's worker eligibility is fixed at its start; install refuses a
+pre-floor binary whenever `host.env` opts into workers (D11, D15, §8.2).
+Where revision 3's bullets below say otherwise, revision 4's sections win.
+
**Revision 3** answers Astra's round 2 (`CODEX-FINDINGS-SIDECAR-ISOLATION-R2.md`,
at 730ba093, verdict reject: three P1s, two P2s), marked **(R2-n)**, and
records Paul's rulings of 2026-10-24 (§11). It also cuts what ships to what
@@ -102,11 +111,11 @@ The decisions, for scanning (each is argued where it is made):
| D8 | The layout pass moves worker rows with their full extras inventory; a row whose database and its companions are absent on both sides moves without it (rev 2) | §3.8 |
| D9 | Every worker replicates through its lease only; the box-wide replicator and the restore drills exclude workers (rev 2) | §3.9 |
| D10 | A failed nested gate never reopens flat for a worker | §3.10 |
-| D11 | Workers move only once `carlos.prev` is itself worker-capable; install keeps both binaries at or above the floor (rev 3) | §3.11 |
+| D11 | Workers opt-in in `host.env`; eligibility fixed at agent start with both binaries at or above the floor; install refuses a pre-floor binary while opted in (rev 4) | §3.11 |
| D12 | `CARLOS_FLAT_EXEC=refuse\|report` plus an exact-host allowlist, read from `host.env` | §4.1 |
| D13 | The refusal sits beside the claim refusal in both spawn seams, with its own error | §4.2 |
| D14 | Refusals and a registry inventory of what `refuse` would stop are reported per host (rev 2) | §4.3 |
-| D15 | Unset `CARLOS_NESTED_WORKERS` is on; unset `CARLOS_FLAT_EXEC` is a per-box ratchet that turns to `refuse` once nothing would be refused (rev 3) | §4.4 |
+| D15 | Both settings explicit per box in root-owned `host.env`; no automatic default change; closure verified per box (rev 4) | §4.4 |
| D16 | A refused managed row nudges the layout pass for that host | §4.5 |
| D17 | (deferred) Creation is refused unless the reconciler's own placement set has a fresh box and every one reports nested workers runnable | §5 |
| D18 | (deferred) The fence retires per box on four observable conditions, the third a complete process census | §6 |
@@ -733,21 +742,24 @@ replication, hibernating or always-on, flat or nested:
first drill after the change is the evidence that those lineages are
sound after a period of two writers. One that fails is a finding for
that host, reported as any drill failure is.
-- **The running replicator, not the file, decides (R2-3).**
+- **The running replicator, not the file, decides (R2-3, R3-3).**
`/etc/litestream.yml` is regenerated by an operator (`carlos ops
- litestream`, `internal/ops/ops.go:149-175`, which prints `changed` for
- the caller to restart the replicator) and litestream reads it only at
- start (`carlos-litestream.service:16`). A config that no longer names a
- worker's database is therefore no evidence that the running daemon has
- let go of it. So the layout pass moves an always-on worker only when the
- box-wide replicator cannot be holding the file: `carlos-litestream` is
- not active, or it is active and its current invocation started after
- the config was last written (`ActiveEnterTimestamp` against the file's
- mtime, read through `systemctl show`) **and** that config does not name
- the file. Otherwise the move is deferred with the reason "the box-wide
- replicator still names this database: regenerate its config and restart
- it" (copy to review), and retried on every pass, so a failed restart is
- never taken for a done one.
+ litestream`, `internal/ops/ops.go:149-175`) and litestream reads it only
+ at start (`carlos-litestream.service:16`, `Restart=always` at `:32`).
+ The layout pass moves an always-on worker only when **all** of these
+ hold, read under the fence immediately before the first rename:
+ - `/etc/litestream.yml` exists and does not name the worker's database
+ (or does not exist at all), so any start from now on cannot open it;
+ - `carlos-litestream`'s `ActiveState` is exactly `active` with
+ `SubState` `running` and its `ExecMainStartTimestamp` later than the
+ config's mtime, **or** exactly `inactive` (or `failed`) with no main
+ PID and its cgroup empty;
+ - no observation failed.
+ Anything else (`activating`, `deactivating`, `reloading`, an unreadable
+ property, a config naming the file) defers the move with the reason
+ "the box-wide replicator may still hold this database" (copy to review),
+ retried on every pass. Because the config already excludes the file,
+ a restart or a pending start after the check opens a config without it.
- **The move is then path-blind** for every worker, as it already is for a
hibernating one: the lease lineage is keyed by host, so moving the file
moves nothing in the bucket.
@@ -774,64 +786,67 @@ recorded (`layoutleg.go:127-146`). For workers:
loudly"). An operator who needs a specific host up through a gate
failure adds it to the allowlist, knowingly (§4.1).
-### 3.11 Decision D11: downgrade (rev 3: R1-2, R2-2)
+### 3.11 Decision D11: downgrade (rev 4: R1-2, R2-2, R3-1)
A binary from before this change runs a nested worker row through the old
`StartWorkerStaged`: flat, with the env read from the flat directory and
no database path, and the row's own nested `DBPath` replicated by the old
lease code. Two copies, and unisolated.
-Revision 1 guarded `systemApplier.install` only. Round 1 named the other
-ways an old binary reaches a row: the rollback after a failed gate
-(`restoreFiles`, `systemapply.go:1784-1800`), the recovery unit, which can
-exec `carlos.prev` itself (`carlos-system-recover.service:41`), and a move
-interrupted before its row CAS. Revision 2 answered with a third protocol
-floor; round 2 showed that the root guard's separately installed copy
-(`guardarm.go:148-166`, overriding the recovery unit's `ExecStart`) would
-not know it (R2-2), and on flagship no protocol floor is published at all.
-
-Revision 3 closes every path from one fact about them. **Each path that
-puts a binary in front of the rows picks one of two files: the current
-`carlos` or `carlos.prev`.** The recovery unit's shell runs one or the
-other (`:41`); the guard copy chooses `carlos.prev` when an apply's journal
-is present and the current binary otherwise (`guardexec.go:66-80`), and its
-floor route chooses between the same two (`:95-110`); `restoreFiles` puts
-`carlos.prev` back. And root records `carlos.prev`'s release beside it
-before every apply (`<optDir>/carlos.prev.version`, written in `notePrev`,
-`systemapply.go:818-830`) and keeps an older `carlos.prev` when the
-running build is not recorded known-good (`decidePrev`, `:792-815`).
-
-**Decision.**
-
-- **Workers move only when both files can run them.** The layout pass
- treats worker rows as eligible (D8), and the reconciler mints nested
- worker rows (D7), only on a box whose running build and whose
- `carlos.prev.version` both parse as releases at or above
- `release.WorkerLayoutFloor` (the first release carrying this spec). An
- unreadable or unparseable `carlos.prev.version`, or none, reads as below.
- Status says `workers: waiting for carlos.prev` (a closed word) while it
- holds. In practice a box qualifies after its second roll at or above the
- floor (§13).
-- **Once true, it stays true.** Neither file goes below the floor without
- an install, and install refuses that:
- - a candidate whose version parses below `WorkerLayoutFloor`, or whose
- unit set lacks `carlos-instance-worker@.service` (for a candidate whose
- version does not parse), is refused while any worker host is named,
- beside R9's guard and on its row test (below);
- - `decidePrev` never replaces `carlos.prev` with a binary below the
- floor, because the running build is at or above it, and keeps the
- existing `carlos.prev` otherwise.
- So the guard copy, whichever of the two it picks, picks a worker-capable
- binary. It needs no change, and neither does the recovery unit.
-- **The row test is R9's, not a row-only one:** a worker host is named when
- its row has `NestedPaths`, when `Classify` calls it fenced or in
- conflict, or when it has a journal, whether or not the row exists
- (`layoutReversalIncomplete`, `systemapply.go:1273-1310`). D8's status
- change lets the console see worker hosts too, and `layoutDowngradeBlocked`
- (`layoutguard.go:24-69`) gains `release.WorkerLayoutFloor`, judged on
- those hosts, and blocks a channel move below it while any is named.
-- `carlos system layout revert` reverts worker rows with the rest (they are
- in the pass); after it, install accepts an older candidate again.
+Every path that puts a binary in front of the rows picks one of two files,
+the current `carlos` or `carlos.prev`: the recovery unit's shell
+(`carlos-system-recover.service:41`), the root guard's copy
+(`guardexec.go:66-110`), and `restoreFiles`. Root records `carlos.prev`'s
+release beside it before every apply (`carlos.prev.version`, `notePrev`)
+and keeps an older `carlos.prev` when the running build is not recorded
+known-good (`decidePrev`, `systemapply.go:792-815`).
+
+Revision 3 let the agent's eligibility follow `carlos.prev.version` live.
+Round 3 showed the race (R3-1): a downgrade's own `decidePrev` copies N
+into `carlos.prev`, which made the running N agent eligible, and it could
+mint or move a worker after install's row check and before the candidate
+replaced it.
+
+**Decision (rev 4).** Nested worker state exists only on an operator's
+explicit, root-owned say-so, and install keys its refusal on that say-so,
+not on state the agent creates:
+
+- **`CARLOS_NESTED_WORKERS` is opt-in:** unset or `off` means workers mint
+ and stay flat (today's behaviour); `on` is set in `host.env`, which is
+ root-owned (`/etc/carlos`, inaccessible to every tenant tier).
+- **The agent's eligibility is fixed at its own start.** At start it reads
+ `CARLOS_NESTED_WORKERS`, its own build, and `carlos.prev.version`. It
+ mints nested workers (D7) and moves worker rows (D8) only if all three
+ held then: the setting is `on`, and both builds parse at or above
+ `release.WorkerLayoutFloor`. Nothing that changes while it runs (an
+ install's `decidePrev` included) makes it eligible; only a restart does.
+ Status says `waiting_for_prev` or `off` while it does not hold.
+- **Install refuses a candidate below the floor** (version below
+ `WorkerLayoutFloor`, or, for an unparseable version, a unit set lacking
+ `carlos-instance-worker@.service`) **when `host.env` says
+ `CARLOS_NESTED_WORKERS=on`, or** when R9's row test names a worker host
+ (`layoutReversalIncomplete`, `systemapply.go:1273-1310`: nested paths,
+ fenced or conflict, or a journal). The first arm reads a root-owned
+ file, so it does not depend on what the running agent has done, and it
+ closes R3-1: an agent that could create nested state is one whose box
+ says `on`, and on such a box install never activates a pre-floor binary.
+ The refusal comes before the journal and names the setting and the
+ hosts, with the way back: revert the layout, set `off`, restart the
+ agent, then downgrade (copy to review).
+- **The two files stay worker-capable.** Once an agent has started
+ eligible, both `carlos` and `carlos.prev` were at or above the floor at
+ that moment; `decidePrev` only ever copies the running binary into
+ `carlos.prev`, and install will not put a pre-floor binary in `carlos`
+ while the setting is `on`. So the guard copy and the recovery unit, which
+ pick one of the two, need no change.
+- **Residual, stated:** an operator who sets `off` in `host.env` and
+ downgrades without restarting the agent first leaves a running agent
+ that may still create nested state during the install. The copy for
+ `off` says to restart; the row test catches any state created before the
+ install's check.
+- The console's `layoutDowngradeBlocked` (`layoutguard.go:24-69`) gains
+ `release.WorkerLayoutFloor` over observed worker hosts.
+- `carlos system layout revert` reverts worker rows with the rest.
### 3.12 Every consumer of the shape
@@ -887,17 +902,7 @@ logged and taken as the safe reading):
refused (§4.2).
- `report`: every spawn proceeds as today, and each flat or foreign spawn
of a host not on the allowlist is recorded as a would-be refusal (§4.3).
-- unset: the ratchet of D15 (rev 3): `report` until the box's inventory is
- first empty under a nested intent, `refuse` from then on, with one
- exception. On a box whose **effective intent is
- flat by an operator's choice**, unset stays `report` in every release,
- because that box's every row is flat on purpose. Flat by choice is
- either `CARLOS_LAYOUT_DEFAULT=flat` with no recorded intent, or a
- recorded `flat` intent, whatever the default says (R1-6). Flat by
- fallback (the default is nested but the box is not ready,
- `defaultLayoutIntent`, `layoutleg.go:115-146`) is not a choice, and
- unset means `refuse` there (D10). An intent the agent cannot read counts
- as not chosen.
+- unset: `report` (rev 4, D15). An operator sets `refuse` per box.
- an unknown value: `refuse`, logged. An operator who typed the variable
meant to choose, and the reading that cannot expose a tenant is the one
to take. (This is the opposite of `CARLOS_LAYOUT_DEFAULT`'s unknown-value
@@ -980,45 +985,37 @@ these seams; nothing here changes them.
- **`carlos status --layout`** shows the policy and each refused or
allowlisted host beside its layout word.
-### 4.4 Decision D15: sequencing, so no box loses a workload on upgrade (rev 3: R1-6, R2-4)
-
-Revisions 1 and 2 had a later release change both defaults behind a
-console guard. Round 2 showed the guard is judged when the channel moves,
-while each box applies the release later, so a workload created in between
-gets both changes unassessed (R2-4). Paul's ruling (§11 Q6) is that a
-default change must not stop what was running. Revision 3 makes the
-change happen on the box, at the one moment it is known to stop nothing.
-
-**Decision.**
-
-- **`CARLOS_NESTED_WORKERS` unset means on**, from release N, subject to
- D11's condition and the box's effective intent being nested. A worker
- that opens `CARLOS_SIDECAR_DB` and uses `CARLOS_AGENT_SOCKET` moves
- without noticing. Every native sidecar on our deployments today is the
- operator's, and §8.2's check covers them before release N rolls. From
- release N the documented contract is those two variables (D4).
-- **`CARLOS_FLAT_EXEC` unset is a ratchet.** It reads `report` until the
- first time the agent finds the box's inventory (D14) empty and the
- effective intent nested, and from then on `refuse`. The agent records
- that moment in `/data/.carlos/flat-exec-refuse` (written once, atomically,
- never removed by any code path) and reads it at start. So:
- - nothing that runs when the ratchet turns is refused by it, because the
- inventory was empty;
- - a row that arrives later meets `refuse`, which is the property: a new
- worker is minted nested, a new instance is nested by default, and a
- foreign row an operator adds by hand is refused and named in status
- (D14), with `carlos add` or the allowlist as the repair;
- - a box whose intent is flat by choice (§4.1) never turns.
- The file lives in the agent's data directory, which only `carlos`
- processes in the host namespace can write. Once the ratchet has turned
- there are none but the agent.
-- **An explicit setting always wins**: `refuse` or `report`, read at agent
- start as before, and the ratchet file is then ignored but not removed.
-- The inventory, the ratchet's state and its time are in status, so an
- operator can see which boxes have turned and why one has not.
-
-No console guard, no later release flipping a default, and no `host.env`
-edit on any box.
+### 4.4 Decision D15: sequencing, so no box loses a workload on upgrade (rev 4: R1-6, R2-4, R3-2, R3-4, R3-5)
+
+Revisions 1 to 3 changed a default automatically: behind a console guard
+(R2-4: the guard is judged before boxes activate), then with an on-box
+ratchet (R3-4: the inventory and the commit are not one transaction; R3-5:
+the ratchet file is under `/data`, which a flat tenant can write; R3-2: a
+flat tenant can make its own row foreign and hold the ratchet open
+forever). Every automatic transition so far has read state the flat tier
+can write. Revision 4 has none.
+
+**Decision.** Both settings are explicit, per box, in root-owned
+`host.env`, and nothing changes them but an operator:
+
+- **`CARLOS_FLAT_EXEC`:** unset means `report` in this release (no
+ automatic change, no ratchet, no file under `/data`). An operator sets
+ `refuse` on a box once its status shows what `refuse` would stop: the
+ inventory of D14, read from the registry, dormant rows included. Anything
+ still listed (a legitimate foreign row, or a row a flat tenant has made
+ foreign, R3-2) is the operator's to resolve before or by setting
+ `refuse`: repaired with `carlos add`, allowlisted knowingly, or simply
+ refused. Under `refuse` a row a tenant made foreign is refused like any
+ other, so a tenant cannot keep its flat process by editing its row.
+- **`CARLOS_NESTED_WORKERS`:** opt-in, as D11 says.
+- **Both take effect at agent start**, which an operator's `carlos system
+ host-env write` plus the agent restart provides.
+- **Closure is verified, not assumed (R3-2):** B1 counts as closed on a
+ box when its status shows `CARLOS_FLAT_EXEC=refuse` in force, an empty
+ allowlist, and no worker row `flat` or `foreign`. §8.2's last step checks
+ this on every box and lists any that do not.
+- Making `refuse` the unset default for new boxes is a later decision, not
+ this release's.
### 4.5 Decision D16: the stragglers
@@ -1172,7 +1169,7 @@ boxes and private services on the network.
| **The fence** (today, interim) | all of the above except metadata | reach metadata, for as long as it cannot leave its cgroup for an exempt one (§1.6; the fence's own hardening, recorded on a dev box by §9) |
| **Release N rolled** (§3 dark, `report`) | the same: nothing that runs has changed | (status now names every flat spawn) |
| **`CARLOS_NESTED_WORKERS=on`** on a box whose gate is ready | on that box, nothing from a new or moved sidecar: it sees its own directory only, no loopback, no private range, no metadata, no sibling pid. It can still get a flat process **if** the box mints flat (gate failing with no recorded intent, D10) or the row is foreign | read or write another tenant's files from its sidecar; call the agent socket for another host (D3) |
-| **`refuse` in force** (explicit, or the ratchet turned), empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace |
+| **`refuse` set** in `host.env`, empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace |
| **Console refusal on** (follow-up, D17) | (no change in power) | create a record the platform would place unisolated or never run; this is honesty, the boundary is the line above |
| **Fence retired** (follow-up, §6) | nothing new: there is no flat child left for the fence to filter | |
@@ -1228,56 +1225,51 @@ source is unreachable"; another provider's adapter supplies its own line.
---
-## 8. Rollout (rev 3)
+## 8. Rollout (rev 4)
### 8.1 Flags and defaults
-| Setting | Where | Unset means | Explicit values |
+| Setting | Where | Unset means | Set |
|---|---|---|---|
-| `CARLOS_NESTED_WORKERS` | box `host.env` | on (D15), effective only under D11's condition and a nested intent | `on`, `off` (keeps workers minted and kept flat) |
-| `CARLOS_FLAT_EXEC` | box `host.env` | the ratchet: `report` until the box's inventory is first empty under a nested intent, `refuse` from then on (D15); `report` forever under a flat-by-choice intent | `refuse`, `report` |
+| `CARLOS_NESTED_WORKERS` | box `host.env` (root-owned) | off: workers mint and stay flat | `on`: eligible from the next agent start under D11 |
+| `CARLOS_FLAT_EXEC` | box `host.env` | `report` | `refuse`, `report` |
| `CARLOS_FLAT_EXEC_ALLOW` | box `host.env` | empty | exact hosts (D12) |
-None of these needs setting on any of our boxes. Nothing is dark in
-release N except what D11 holds back until `carlos.prev` qualifies.
+Release N changes nothing that runs until an operator sets these.
### 8.2 Order
The house order: dev boxes, then the dev console, then a check, then
-flagship boxes, then the flagship console.
+flagship boxes, then the flagship console. Setting `host.env` values on a
+box is a production change under AGENTS.md's scoped approval: the plan for
+it (accounts, regions, the SSM parameters `carlos system host-env write`
+reads, the boxes, the restart, verification and the way back) goes to Paul
+once, before step 4.
1. **Before the cut: the sidecar check.** Every native sidecar on dev and
flagship (all the operator's, #370) is checked for a computed database
path, a hard-coded agent socket, and loopback or private-address use
- (§3.4, §7.3). Each one that needs it ships a release reading
- `CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` first. Where an always-on
- worker's database is named in a box's `/etc/litestream.yml`, the
- operator regenerates it (`carlos ops litestream`, which D9 now excludes
- workers from) and restarts the replicator; until then D9's check holds
- that worker's move (§3.9).
-2. **Release N to dev boxes, then the dev console.** On each box: status
- shows the inventory; workers stay flat (`waiting for carlos.prev`); D9's
- exclusion and D19's fence are live.
-3. **Release N's patch (N.1) to dev boxes and the dev console.** Now
- `carlos.prev` is N, so D11's condition holds and dev boxes move their
- workers: hibernating ones at once, always-on ones within the awake
- grace. Verify from inside with a throwaway sidecar in a dev account
- whose release lists `/data` and `/run/carlos`, reads `/proc/1/status`,
- sends the probe's datagrams, opens `CARLOS_SIDECAR_DB`, calls
- `/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, and prints it
- to `carlos sidecars logs`; then sleeps, wakes on its timer and on
- `carlos sidecars wake`, crashes once (restarted), is killed by SIGTERM
- from inside once (a crash, restarted), and finishes once (parked). Then
- watch each dev box's inventory empty and its ratchet turn to `refuse`.
- A foreign row made with `carlos add --db` is then refused and named.
-4. **Flagship:** N to boxes region by region, console box last, then the
- flagship console; then N.1 the same way. Watch each box move its workers
- and turn its ratchet. A box whose ratchet has not turned within the
- awake grace after N.1 is named in status with what holds it.
-5. **The fence** stays until its retirement follow-up (§13).
-
-N.1 can be whatever patch is next on `main`; its only job here is to make
-N the box's `carlos.prev`.
+ (§3.4, §7.3). Each that needs it ships a release reading
+ `CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` first. Any box whose
+ `/etc/litestream.yml` names an always-on worker's database has it
+ regenerated (`carlos ops litestream`, which now excludes workers) and the
+ replicator restarted (D9).
+2. **Release N to dev, then flagship** (boxes, then console). Status shows
+ each box's inventory; nothing moves.
+3. **Release N.1 the same way.** Now `carlos.prev` is N on every box.
+4. **Per box, dev first: `CARLOS_NESTED_WORKERS=on`** and an agent restart.
+ Workers move: hibernating ones at once, always-on ones within the awake
+ grace. On dev, verify from inside with a throwaway sidecar (namespace
+ listing, `/proc/1/status`, the probe's datagrams, `CARLOS_SIDECAR_DB`,
+ `/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, logs; sleep,
+ timer wake, `carlos sidecars wake`, a crash, a self-SIGTERM crash, a
+ finish).
+5. **Per box, once its inventory is empty or resolved:
+ `CARLOS_FLAT_EXEC=refuse`** and an agent restart.
+6. **Verify closure on every box (R3-2):** `refuse` in force, empty
+ allowlist, no worker row `flat` or `foreign`. Any box that fails is
+ named on #370 with what holds it.
+7. **The fence** stays until its retirement follow-up (§13).
### 8.3 Census
@@ -1374,16 +1366,15 @@ Unit and package level:
excludes an always-on worker and the restore census includes it; the
shrink guard accepts a path leaving for that reason and names it, and
still refuses any other shrink.
-- D11 (rev 3): the pass and the reconciler treat workers as flat while
- `carlos.prev.version` is missing, unparseable or below the floor, and
- status says `waiting for carlos.prev`; install refuses a candidate below
+- D11 (rev 4): the pass and the reconciler treat workers as flat unless,
+ at agent start, `CARLOS_NESTED_WORKERS=on` and both builds were at or
+ above the floor (missing or unparseable `carlos.prev.version` reads as
+ below); a change to `carlos.prev.version` while the agent runs changes
+ nothing; status says `off` or `waiting_for_prev`; install refuses a
+ pre-floor candidate whenever `host.env` says `on`; install refuses a candidate below
the floor, or one lacking the worker template, while a worker host is
named, including a host with only a journal.
-- D15's ratchet: unset reads `report` with a non-empty inventory; turns
- once on the first empty inventory under a nested intent, writes its file
- once, and stays `refuse` across an agent restart and a later non-empty
- inventory; never turns under a flat-by-choice intent; an explicit value
- wins and leaves the file.
+- D15 (rev 4): unset `CARLOS_FLAT_EXEC` is `report`; unset `CARLOS_NESTED_WORKERS` is off; both read once at agent start; no file under `/data` changes either.
- D19 replacement (R2-1): a hold `ReplaceWorker` whose worker start
answers uncertain restores nothing and stops the entry; an overlap one
falls back only after `RoleGone`.
@@ -1466,10 +1457,9 @@ packages' tests.
5. **A third protocol floor?** Superseded: revision 3's D11 needs none
(R2-2).
6. **How defaults reach existing boxes:** nothing that runs is stopped by a
- default change (recommendation taken). Revision 3 does it with the
- ratchet of D15 instead of keeping old defaults on established boxes,
- because the ratchet needs no per-box `host.env` change and turns every
- box once it is safe.
+ default change (recommendation taken). Revision 4 changes no default at
+ all: both settings are set per box by an operator in `host.env` (§8.2),
+ under the scoped approval production changes need.
## 12. Review disposition
@@ -1519,14 +1509,31 @@ judged reasonable. Citations spot-checked and held.
| R2-4 | P2: defaults change during the rollout window | **Accepted, by Paul's ruling (Q6).** No release flips a default behind a console guard; the flat-exec default is a per-box ratchet that turns only when nothing would be refused | D15, §4.4 |
| R2-5 | P2: D17 drops stale boxes that still run workers flat | **Deferred with D17.** The console check is not the boundary and is out of this release; the follow-up takes R2-5's fix (stale eligible boxes block) | §5, §13 |
+
+### 12.3 Round 3 (Astra, `CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md`, at 99017de5)
+
+Verdict: reject. R2-1 fixed; R2-5's deferral accepted. Every round-3
+finding was in an automatic transition that read state the flat tier can
+write, or that raced the agent. Revision 4 removes the transitions
+(`refuse-up-front-over-tolerating`).
+
+| Finding | | Disposition | Where |
+|---|---|---|---|
+| R3-1 | P1: install's `decidePrev` makes the agent eligible after the row check | **Accepted.** Eligibility is fixed at agent start; workers are opt-in in root-owned `host.env`, and install refuses a pre-floor candidate whenever `host.env` says `on` | D11, §3.11 |
+| R3-2 | P1: a flat tenant makes its row foreign and holds the ratchet open | **Accepted.** No ratchet; `refuse` is an operator's explicit setting, under which a foreign row is refused; closure is verified per box | D15, §4.4; §8.2 step 6 |
+| R3-3 | P1: "not active" is not proof the replicator let go | **Accepted.** Config must exclude the file in every case; only exactly `active/running` started after the config, or `inactive`/`failed` with no PID and an empty cgroup, passes; transitional or unreadable defers | D9, §3.9 |
+| R3-4 | P2: the ratchet overtakes a flat worker | **Accepted, by removal.** No ratchet | D15 |
+| R3-5 | P2: a flat tenant forges the ratchet file | **Accepted, by removal.** No file under `/data` decides policy; both settings are in root-owned `host.env` | D15 |
+
---
## 13. What release N carries, and what follows
-**Release N (and its patch N.1, §8.2):** D1 to D14, D16 and D19, with D11
-and D15 as in revision 3. That closes B1: a native sidecar runs nested on
-every box whose `carlos.prev` qualifies, and once a box's ratchet turns,
-nothing on it runs in the host namespace unless an operator allowlists it.
+**Release N (and its patch N.1, §8.2):** D1 to D14, D16 and D19, with D9,
+D11 and D15 as in revision 4. That closes B1: a native sidecar runs nested on
+every box whose `carlos.prev` qualifies, and once an operator sets
+`refuse` on a box (§8.2 steps 5 and 6), nothing on it runs in the host
+namespace unless an operator allowlists it.
**Follow-ups, each its own branch, filed as amadan discussions:**
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md b/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md
new file mode 100644
index 000000000..9ca90d53d
--- /dev/null
+++ b/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R3.md
@@ -0,0 +1,98 @@
+| Round 2 finding | Revision 3 assessment | Section |
+|---|---|---|
+| R2-1 — Uncertain hold replacement restores the previous build | **Fixed at design level.** Uncertainty reaches `startMayHaveRun`; hold attempts cannot restore the previous build, and the stop obligation survives until `RoleGone`. | D19, §3.5.1 |
+| R2-2 — Independently installed guard ignores the workers floor | **Original bypass addressed, but the replacement invariant is incomplete.** Both selectable binaries being capable would protect recovery; installation can race the creation of the state it checks. See R3-1. | D11, §3.11 |
+| R2-3 — Regenerated configuration does not prove daemon release | **Partially fixed.** The active-daemon check addresses the original stale-configuration case. The new “not active” alternative admits an unsafe transition. See R3-3. | D9, §3.9 |
+| R2-4 — Defaults change after the console’s assessment | **Partially fixed.** The console-to-activation gap is removed. The local inventory-to-ratchet transition still lacks admission ordering, and can precede worker readiness. See R3-4. | D15, §4.4 |
+| R2-5 — Stale eligible boxes are excluded from creation checks | **Acceptably deferred** with D17 under the owner’s ruling. | §11 Q1, §12.2, §13 |
+
+Reviewed revision 3 at `99017de51f3e3a3af604adf596c6c41185ed2ea8` against main `fef28dc5`. The shared checkout advanced during the review; the code cited below was checked against the requested baseline and is unchanged. Static review only: no files modified and no tests run. D17 and D18’s deferral is accepted.
+
+1. **R3-1 — P1: Installation can enable worker migration, pass its downgrade check, then install a worker-incapable binary.**
+
+ **Section:** D11, §3.11; §8.3’s instruction to leave `decidePrev` unchanged.
+
+ **Failure sequence:**
+
+ 1. A box runs known-good N, with a pre-N `carlos.prev` and a flat worker. D11 currently prevents its migration.
+ 2. An installation of a pre-N candidate begins. `decidePrev` copies N into `carlos.prev` and records N’s version. This immediately satisfies D11’s migration condition for the still-running N agent.
+ 3. The candidate’s worker-layout check sees only the flat row: no nested worker, conflict or journal yet. It passes.
+ 4. The agent migrates that worker, or mints a new nested worker, before the candidate is activated.
+ 5. Installation replaces `carlos` with the pre-N candidate and restarts the edge. That binary starts the nested worker through the old flat implementation.
+
+ The two-file observation is correct for the named recovery selectors. What fails is the claimed invariant that installation cannot put either file below the floor while worker state exists. The check and the state creation are not ordered.
+
+ **Main evidence:** Installation calls `decidePrev` before staging and the layout check at [systemapply.go:498](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:498). `notePrev` copies the binary and writes its version at [systemapply.go:841](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:841). The existing layout check is at [systemapply.go:525](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:525), before activation at [systemapply.go:569](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:569). The quiesced unit list excludes the edge at [systemapply.go:83](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:83), and the layout pass runs independently at [layoutleg.go:552](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/layoutleg.go:552). With no protocol floor armed, the activation guard proceeds without the registry check at [systemapply.go:1055](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/systemapply.go:1055).
+
+ **Suggested fix:** Serialize worker eligibility, nested minting and migration against binary activation. Recheck worker state at activation while holding that exclusion, and prevent the old agent from creating nested worker state between the check and its shutdown. Keep R9’s early refusal, but do not use it as the activation proof. Test a downgrade whose `decidePrev` first makes workers eligible, with a mint or migration inserted after the early check.
+
+2. **R3-2 — P1: A flat worker can keep the ratchet in `report` indefinitely by making its own row foreign.**
+
+ **Section:** D15, §4.4; D7, §3.7; D8, §3.8; closure claim in §13.
+
+ **Failure sequence:**
+
+ 1. During release N, a tenant starts a flat worker while `carlos.prev` remains below the worker floor.
+ 2. As uid `carlos`, it changes its registry row’s `DBPath` to another writable, nonstandard path. It retains its sidecar kind, identity and runnable arguments.
+ 3. N.1 installs successfully. Both binaries now qualify, but the row classifies as `foreign`. The layout pass skips it; reconciliation reports the unexpected path without repairing it.
+ 4. D14 continues listing the foreign row, so D15 never leaves `report`.
+ 5. Subsequent wakes still take D5’s foreign-to-flat arm. The platform continues starting tenant code in the host namespace after both releases have rolled.
+
+ This is continued authorization of fresh flat launches, beyond the residual risk from files planted before isolation. An ordinary legitimate foreign workload can also prevent the box-wide ratchet from turning; the hostile case makes the obstacle attacker-controlled.
+
+ **Main evidence:** `/data` is assigned to the service user at [bootstrap.go:661](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/bootstrap.go:661), including the registry ownership described at [bootstrap.go:475](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/bootstrap.go:475). Unexpected sidecar paths are explicitly reported but not rewritten at [sidecar_reconcile.go:616](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/provision/sidecar_reconcile.go:616). The pass skips foreign rows before quiescing them at [pass.go:294](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/layout/pass.go:294). Foreign paths remain usable by the exec backend at [paths.go:33](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/activator/paths.go:33).
+
+ **Suggested fix:** Do not let tenant-writable inventory grant indefinite permission for flat execution. Quarantine invalid managed-worker shapes, and distinguish operator-authorized legacy exceptions from rows a flat tenant can manufacture. Give unresolved legitimate exceptions an explicit resolution path. Rollout completion must require verified refusal on every intended box; installing N and N.1 alone does not close B1.
+
+3. **R3-3 — P1: “Not active” does not establish that the box-wide replicator cannot touch the database during migration.**
+
+ **Section:** D9, §3.9, running-replicator check.
+
+ **Failure sequence:**
+
+ 1. The box-wide daemon has opened an always-on worker’s database using the old configuration.
+ 2. A restart begins. The service becomes `deactivating` while its process still holds database handles.
+ 3. D9’s “not active” alternative admits migration without requiring configuration exclusion or completed termination.
+ 4. The worker and lease replicator stop, and migration renames database artifacts while the box-wide daemon can still checkpoint them.
+
+ There is a second window even if “not active” is implemented as strictly `inactive`: a pending start can launch the daemon on the unchanged configuration after the observation. The predicate does not prevent that start during the move.
+
+ **Main evidence:** The daemon reads `/etc/litestream.yml` at [carlos-litestream.service:16](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-litestream.service:16), needs writable database access for checkpointing at [line 23](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-litestream.service:23), and has `Restart=always` at [line 32](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-litestream.service:32). Migration renames artifacts individually at [pass.go:988](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/layout/pass.go:988). Competing checkpoint owners are the corruption case documented at [litestream.go:230](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/ops/litestream.go:230). Systemd distinguishes `deactivating` from stopped `inactive`: [https://www.freedesktop.org/software/systemd/man/latest/systemctl.html](https://www.freedesktop.org/software/systemd/man/latest/systemctl.html).
+
+ **Suggested fix:** Require configuration exclusion even when the service is stopped, prove the old invocation and its descendants have terminated, and prevent a conflicting start/configuration change through the move. Transitional states and unreadable observations must defer. Test migration during a slow restart and immediately before a pending start using the old configuration.
+
+4. **R3-4 — P2: The local ratchet can overtake a flat worker that has already started.**
+
+ **Section:** D15, §4.4; D14, §4.3; D7/D11, §§3.7 and 3.11.
+
+ **Failure sequence:**
+
+ 1. A release-N box has nested intent, no flat inventory, and a pre-N `carlos.prev`. The agent obtains an empty inventory for the ratchet.
+ 2. Before it commits the policy change, reconciliation creates a worker. D11 requires this worker to be minted flat. Its wake succeeds under `report`.
+ 3. The agent commits `refuse` from the earlier empty inventory.
+ 4. The worker reaches `max_run` or needs a restart. Its old process stops, but the replacement is refused. D11 still prevents migration until another qualifying installation.
+
+ Moving the decision onto the box removes the console rollout delay, but does not make inventory assessment atomic with admission. Even without the race, an empty release-N box can turn before D11 qualifies, contradicting D15’s assurance that subsequently arriving workers are minted nested.
+
+ **Main evidence:** Reconciliation publishes new sidecar rows through `Upsert` at [sidecar_reconcile.go:461](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/provision/sidecar_reconcile.go:461). Worker startup reaches the flat spawn at [backend_exec.go:3125](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/activator/backend_exec.go:3125). An always-on worker’s run ceiling triggers recycling at [activator.go:4709](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/activator/activator.go:4709); replacement retires the previous child before spawning at [activator.go:3560](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/activator/activator.go:3560).
+
+ **Suggested fix:** Make the ratchet transition a local admission transaction: order registry publication and in-flight flat starts against the final inventory assessment and policy commit. Require worker mint readiness, including D11, before automatic transition. This needs no console creation pause. Test a flat worker admitted between inventory collection and the proposed commit.
+
+5. **R3-5 — P2: A flat tenant can forge the persistent ratchet before the agent earns it.**
+
+ **Section:** D15, §4.4, `/data/.carlos/flat-exec-refuse`.
+
+ **Failure sequence:**
+
+ 1. Release N runs flat workers under `report`; inventory is nonempty and `carlos.prev` does not qualify.
+ 2. A worker creates the marker that the agent would write after a successful transition.
+ 3. On the next agent restart, the marker selects `refuse` despite the nonempty inventory.
+ 4. Previously working flat workloads are refused, and D11 prevents their migration.
+
+ The assertion that only the agent can write the marker *after* isolation does not authenticate a marker written *before* isolation. Atomic replacement and “never removed by code” do not establish its origin.
+
+ **Main evidence:** The edge runs as `carlos` with writable `/data` at [carlos-edge.service:28](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-edge.service:28) and [line 37](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-edge.service:37). Bootstrap assigns `/data` to that user at [bootstrap.go:661](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/bootstrap.go:661). Native workers are ordinary exec children at [backend_exec.go:3125](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/activator/backend_exec.go:3125).
+
+ **Suggested fix:** Persist the committed transition behind an authority boundary flat tenants cannot write or replace, including its parent directories, and validate that provenance at startup. An agent-owned file under `/data` cannot serve as that proof. Test a preplanted marker while flat inventory remains.
+
+**Overall verdict: reject.** D19 resolves the replacement issue, but D11 does not yet preserve its invariant across activation, and D15 does not guarantee a safe or inevitable transition. D9 also permits migration without proving exclusive database ownership. Release N plus N.1 therefore does not establish §13’s closure claim.
\ No newline at end of file