carlos / platform Public

Sidecar isolation spec rev 4.1 and plan: Astra round 4 (accept with fixes) folded in

R4-1 D9's check covers every worker move; R4-2 rollout sets host.env lines
per box over SSM, hostenv allowlist gains the names; R4-3 inventory computed
from each snapshot's registry read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 14 hours ago d86c9e96fe15c0ea69b68d2e92dae16617e48c7e parent 209b832

3 files changed, +110 −12

docs/superpowers/plans/2026-10-24-sidecar-isolation.md +12 −4
docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md +35 −8
docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R4.md +63 −0

diff --git a/docs/superpowers/plans/2026-10-24-sidecar-isolation.md b/docs/superpowers/plans/2026-10-24-sidecar-isolation.md

index 4034f455d..e5f81ce81 100644

--- a/docs/superpowers/plans/2026-10-24-sidecar-isolation.md

+++ b/docs/superpowers/plans/2026-10-24-sidecar-isolation.md

@@ -17,7 +17,7 @@ workers. Nothing in `host.env` changes on any box.

units embedded in `internal/host/units`, cgroup v2, litestream.

**Spec:** `docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md`,

-revision 4 (9f2cd284). Read §0, §3, §4, §8, §13 before any task. Decisions

+revision 4.1. Read §0, §3, §4, §8, §13 before any task. Decisions

are cited as D-numbers; this plan implements D1 to D14, D16, D19 and D9/D11/D15

as revised in rev 4. D17 and D18 are follow-ups and are NOT built here.

@@ -339,8 +339,8 @@ The largest task; one implementer, Fable-tier.

`SubState=running` with `ExecMainStartTimestamp` after the config's

mtime, or exactly `inactive`/`failed` with `MainPID=0` and an empty

cgroup. Any other state or any failed observation holds (spec rev 4 D9,

- R3-3). Applied to always-on worker rows, read under the fence right

- before the first rename.

+ R3-3). Applied to every worker row's move (R4-1: not only always-on), read

+ under the fence right before the first rename.

- Missing-database rule: `db` not required when `db`, `db-wal`, `db-shm`,

`litestream-shadow`, `corrupt` and `<db>-litestream` are all absent at

both endpoints; then the journal records it absent and the rest moves.

@@ -349,6 +349,7 @@ The largest task; one implementer, Fable-tier.

suffix is refused like any journal mismatch.

- [ ] Tests: `TestPassMovesHibernatingWorkerAsleep`;

+ `TestFormerlyAlwaysOnWorkerMoveHeldWhileReplicatorNamesFile` (R4-1);

`TestPassDefersAwakeAlwaysOnWorkerThenStopsOnce`;

`TestFencedWorkerRefusesEveryWakeSource` (timer, `carlos sidecars wake`,

boot pass, sweep's always-on start — Review Focus 3, including an agent

@@ -383,6 +384,8 @@ The largest task; one implementer, Fable-tier.

`Metrics.Layout.Workers`) and its contract test

- Modify: `internal/host/layoutleg.go` (D16: a refused managed or worker

row on a nested-intent box marks the layout leg dirty)

+- Modify: `internal/hostenv/hostenv.go:70` (allow `CARLOS_NESTED_WORKERS`,

+ `CARLOS_FLAT_EXEC`, `CARLOS_FLAT_EXEC_ALLOW`; R4-2)

- Test: `internal/activator/flatexec_test.go`, `internal/host/flatexec_test.go`,

`internal/activator/spawn_enumeration_test.go`

@@ -396,7 +399,9 @@ The largest task; one implementer, Fable-tier.

(unset → `report`, unknown → `refuse`); `func flatExecInventory(routes []registry.Route, classify func(registry.Route) activator.Layout, allow map[string]bool) []InventoryRow`

(rows that would be refused: managed, worker and foreign exec rows

classifying `flat`/`foreign`, not allowlisted, awake or asleep),

- refreshed on every layout pass into status. Status also carries spec rev

+ computed from each status snapshot's own registry read

+ (`internal/host/agent.go:4002`), never from the layout pass (R4-3); a

+ failed computation is `unknown`, which never counts as empty or closed. Status also carries spec rev

4 D15's closure check: `refuse` in force, empty allowlist, no worker row

`flat` or `foreign`.

@@ -412,6 +417,9 @@ The largest task; one implementer, Fable-tier.

`TestClosureCheck` (true only with refuse, empty allowlist, no flat or

foreign worker row);

`TestStatusListsForeignRowInInventory` (Review Focus 2);

+ `TestInventoryComputedUnderFlatDefault`, `...UnderUnreadableIntent`,

+ `...UnderFailedGate`, `TestInventoryUnknownIsNotClosed` (R4-3);

+ `TestHostEnvAllowsWorkerAndFlatExecNames` (R4-2);

`TestRefusedManagedRowNudgesPass` and not for foreign; status contract

test passes with the new fields and they never reach `fleet.json`.

- [ ] Commit.

diff --git a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md

index c4c8a44aa..47d5615ff 100644

--- a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md

+++ b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md

@@ -1,6 +1,6 @@

# Sidecar isolation: native sidecars on the nested tier, and no flat exec by default

-**Date:** 2026-10-24 · **Status:** revision 4, approved to build by Paul

+**Date:** 2026-10-24 · **Status:** revision 4.1, approved to build by Paul

2026-10-24 ("let's just do this and close the vulnerability and roll a

release when done"), for an independent

review (brief: `CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md`) and then Paul ·

@@ -746,8 +746,12 @@ replication, hibernating or always-on, flat or nested:

`/etc/litestream.yml` is regenerated by an operator (`carlos ops

litestream`, `internal/ops/ops.go:149-175`) and litestream reads it only

at start (`carlos-litestream.service:16`, `Restart=always` at `:32`).

- The layout pass moves an always-on worker only when **all** of these

- hold, read under the fence immediately before the first rename:

+ The layout pass moves **any** worker row only when **all** of these

+ hold, read under the fence immediately before the first rename. Not

+ only always-on ones (R4-1): a worker that was always-on when the running

+ replicator started and has since been switched to hibernating

+ (`sidecar_reconcile.go:583` changes `Hibernate` in place) may still be

+ open in it, so the row's current flag proves nothing about the daemon:

- `/etc/litestream.yml` exists and does not name the worker's database

(or does not exist at all), so any start from now on cannot open it;

- `carlos-litestream`'s `ActiveState` is exactly `active` with

@@ -966,7 +970,12 @@ these seams; nothing here changes them.

(`flat_refused`, `foreign_refused`, `flat_reported`, `foreign_reported`),

first and last time and a count, the shape `ConfigRefusal` already has

(`snapshot.go:256-274`). Error text never goes in it.

-- **Inventory (R1-6).** Beside the spawn record, every snapshot carries

+- **Inventory (R1-6, R4-3).** Computed from the same registry read each

+ status snapshot already makes (`agent.go:4002`), never from the layout

+ pass, which returns early on a flat default, an unreadable intent or a

+ failed gate. An inventory that could not be computed is reported as

+ `unknown` and never counts as empty or as closed. Beside the spawn

+ record, every snapshot carries

what `refuse` would stop **from the registry, not from history**: each

row the agent could spawn through a seam (activator-managed, worker and

foreign exec rows, awake or asleep) that classifies `flat` or `foreign`

@@ -1242,16 +1251,22 @@ Release N changes nothing that runs until an operator sets these.

The house order: dev boxes, then the dev console, then a check, then

flagship boxes, then the flagship console. Setting `host.env` values on a

box is a production change under AGENTS.md's scoped approval: the plan for

-it (accounts, regions, the SSM parameters `carlos system host-env write`

-reads, the boxes, the restart, verification and the way back) goes to Paul

-once, before step 4.

+it (accounts, regions, the boxes, the restart, verification and the way

+back) goes to Paul once, before step 4. **How the lines get there (R4-2):**

+flagship and dev boxes do not build `host.env` with `carlos system

+host-env write`; their deployment repos set single lines over SSM with an

+idempotent, atomic script (the pattern of carloku-infrastructure's

+`scripts/set-alert-sns-topic.sh`), and that is the path used here, one box

+at a time, so two boxes in a region can differ. For deployments that do use

+`host-env write`, its allowlist (`internal/hostenv/hostenv.go:70`) gains

+the three names, at region scope; per-box choice there is out of scope.

1. **Before the cut: the sidecar check.** Every native sidecar on dev and

flagship (all the operator's, #370) is checked for a computed database

path, a hard-coded agent socket, and loopback or private-address use

(§3.4, §7.3). Each that needs it ships a release reading

`CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` first. Any box whose

- `/etc/litestream.yml` names an always-on worker's database has it

+ `/etc/litestream.yml` names any worker's database has it

regenerated (`carlos ops litestream`, which now excludes workers) and the

replicator restarted (D9).

2. **Release N to dev, then flagship** (boxes, then console). Status shows

@@ -1525,6 +1540,18 @@ write, or that raced the agent. Revision 4 removes the transitions

| R3-4 | P2: the ratchet overtakes a flat worker | **Accepted, by removal.** No ratchet | D15 |

| R3-5 | P2: a flat tenant forges the ratchet file | **Accepted, by removal.** No file under `/data` decides policy; both settings are in root-owned `host.env` | D15 |

+### 12.4 Round 4 (Astra, `CODEX-FINDINGS-SIDECAR-ISOLATION-R4.md`, at 9f2cd284 and plan fea6a05e)

+Verdict: **accept with fixes**. R3-1 to R3-5 fixed. Revision 4.1 makes the

+three fixes:

+|---|---|---|---|

+| R4-2 | P2: `carlos system host-env write` rejects the new names | **Accepted.** Rollout uses the deployments' per-box SSM line script; `hostenv`'s allowlist gains the three names for region-scoped use | §8.2; plan Task 9 |

---

## 13. What release N carries, and what follows

diff --git a/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R4.md b/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R4.md

new file mode 100644

index 000000000..e29a4bc02

--- /dev/null

+++ b/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R4.md

@@ -0,0 +1,63 @@

+| R3 finding | Revision 4 assessment | Section |

+|---|---|---|

+| R3-1 — Installation changes worker eligibility mid-flight | **Fixed** under the prescribed operator sequence. Eligibility is fixed at agent start; installation refuses a pre-floor candidate while `host.env` enables workers. | D11, §3.11 |

+| R3-2 — Foreign row holds the automatic ratchet open | **Fixed.** Explicit refusal replaces the ratchet and applies to foreign rows. Closure requires verification. | D15, §4.4; §8.2 |

+| R3-3 — “Not active” permits migration over a live replicator | **Fixed for rows receiving the check.** Transitional states defer; stopped states require an empty cgroup and configuration exclusion. A new coverage gap is R4-1 below. | D9, §3.9 |

+| R3-4 — Ratchet overtakes an admitted flat worker | **Fixed by removal.** Policy changes require an operator setting and restart. | D15, §4.4 |

+| R3-5 — Tenant forges the ratchet file | **Fixed by removal.** No tenant-writable marker determines policy. | D15, §4.4 |

+Reviewed spec `9f2cd284316b2c819369c982b3399d256683f15a` and plan `fea6a05e`, against the brief’s main baseline `fef28dc5`. Static review only; no files modified or tests run. The owner’s decisions on creation, explicit settings, and D17/D18 remain accepted.

+1. **R4-1 — P1: A formerly always-on worker can bypass D9’s replicator check.**

+ **Section:** D9, §3.9; §8.2 step 1; implementation plan Task 8.

+ **Failure steps:**

+ 1. Before release N, an always-on worker’s database enters the box-wide configuration. The daemon opens it.

+ 2. The app changes its sidecar manifest to enable hibernation. Reconciliation changes the existing row’s `Hibernate` flag without restarting the box-wide daemon.

+ 3. The box follows the N/N.1 rollout. The worker is now hibernating, so §8.2’s check for configuration naming an **always-on** worker need not select it.

+ 4. The operator enables nested workers. Task 8 explicitly applies `boxReplicatorHolds` only to always-on worker rows, so this row bypasses the check.

+ 5. Migration stops the worker and its lease replicator, then renames database artifacts while the box-wide daemon can still checkpoint them.

+ The current hibernation flag does not establish which databases an existing daemon opened. This is a bypass of the new check, independent of whether that check correctly handles `deactivating`.

+ **Main evidence:** Reconciliation changes hibernation in place at [sidecar_reconcile.go:583](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/provision/sidecar_reconcile.go:583). The box-wide daemon reads its configuration at startup and has writable database access at [carlos-litestream.service:16](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/units/carlos-litestream.service:16). Migration renames artifacts individually at [pass.go:988](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/layout/pass.go:988).

+ **Fix:** Apply D9’s release check to every worker migration, regardless of current `Hibernate`. Widen the rollout inspection similarly. Add a test starting with an always-on worker in the loaded configuration, switching it to hibernating, and proving migration remains blocked until the daemon releases it.

+2. **R4-2 — P2: The specified configuration command cannot write the new settings.**

+ **Section:** D11, §3.11; D15, §4.4; §8.2; implementation plan Tasks 7, 9 and 13.

+ **Failure steps:**

+ 1. N and N.1 roll successfully.

+ 2. The operator adds `CARLOS_NESTED_WORKERS=on` to the parameters consumed by the specified `carlos system host-env write` command.

+ 3. Assembly rejects the variable as unapproved and writes nothing. The same happens with `CARLOS_FLAT_EXEC` and `CARLOS_FLAT_EXEC_ALLOW`.

+ 4. The planned rollout cannot activate migration or refusal through its named command. No implementation task changes this command’s allowlist.

+ There is also a scope mismatch to resolve: this command assembles shared and regional parameters, not independently selected per-box settings.

+ **Main evidence:** The permitted names are enumerated at [hostenv.go:70](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/hostenv/hostenv.go:70); unknown names are rejected at [hostenv.go:207](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/hostenv/hostenv.go:207). The command fetches parameters and replaces the file at [cli.go:68](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/hostenv/cli.go:68). Its shared/regional source selection is at [hostenv.go:307](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/hostenv/hostenv.go:307).

+ **Fix:** Add an explicit configuration-writing task. Support the three names and specify how independent per-box choices survive subsequent regeneration. Test the actual write path, including two boxes in one region retaining different operator choices. Alternatively, name a concrete per-box configuration path in the rollout and implement that path.

+3. **R4-3 — P2: Task 9’s inventory refresh contradicts D14’s snapshot contract.**

+ **Section:** D14, §4.3; D15, §4.4; implementation plan Task 9.

+ **Failure steps:**

+ 1. A box has `CARLOS_LAYOUT_DEFAULT=flat`, no recorded layout intent, and dormant flat workloads.

+ 2. Task 9 refreshes inventory “on every layout pass.” The layout leg takes its flat-default early return without running convergence.

+ 3. Status snapshots continue publishing without obtaining the inventory D14 promises from the current registry. Depending on initialization, it remains absent or stale.

+ 4. An operator cannot reliably assess what setting `refuse` will stop. If an uninitialized empty inventory is displayed as clear, following the rollout can strand those workloads.

+ The same dependency affects unreadable intent and failed-gate paths. Even where passes run, their cadence does not satisfy “every snapshot.”

+ **Main evidence:** An unreadable intent returns before convergence at [layoutleg.go:637](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/layoutleg.go:637); the flat-default path does likewise at [layoutleg.go:656](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/layoutleg.go:656). The status loop already obtains current registry rows independently at [agent.go:4002](/home/paulca/.cache/tmp/claude-1001/sec/design-sidecar-isolation/internal/host/agent.go:4002).

+ **Fix:** Change Task 9 to calculate inventory from each snapshot’s registry read, independently of migration eligibility or execution. An unavailable inventory must not count as empty or satisfy closure. Test dormant rows with flat defaults, unreadable intent, and a failed nested gate.

+**Verdict: accept with fixes.** The revised operator-controlled mechanisms address R3. R4-1 must be fixed before migration is enabled; R4-2 and R4-3 are required to make the specified rollout executable and its verification reliable.

\ No newline at end of file