carlos / platform Public

Native sidecars run on the flat tier: move them to nested and refuse flat spawns Issue open

On branch design-sidecar-isolation

Security review finding B1. Native sidecars always start on the flat tier, whatever the box's layout (ExecBackend.StartWorkerStaged, internal/activator/backend_exec.go ~3043, spawns via spawnInCgroupGated). A flat child runs as uid carlos inside carlos-edge.service's cgroup with no mount namespace and no IP filtering. Any account member, and the instance service role, can enable and create a sidecar (POST /api/cli/apps/{account}/{app}/sidecars; plan caps are dormant). From there it can:

  • read and write every nested tenant's /data/instances/<iid> on that box (0700 carlos:carlos; nested units also run as carlos), plus flat siblings' .env/.db and control sockets;
  • reach the instance metadata service and the box role. This is now blocked out of band by carloku-infrastructure scripts/fence-flat-imds.sh (PR #19) on all four flagship boxes; the fence does nothing about file access.

Console audit, 2026-10-24: every sidecar-enable/create ever was account bdr (operator). No tenant has used this.

Spec: branch design-sidecar-isolation, docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md (rev 1, 1c029ee5). It proposes a nested worker role and carlos-instance-worker@ template; the layout pass moving sidecar rows; CARLOS_FLAT_EXEC=refuse|report plus an exact-host allowlist checked in both spawn functions; a console 409 (code: "isolation") when a region can't run workers nested; and conditions for retiring the fence.

Review: CODEX-FINDINGS-SIDECAR-ISOLATION.md (Astra round 1, 2a2517d6): four P1s (worker start/stop ordering, rollback/recovery for D11, always-on workers already use lease replication, D8's no-local-data test) and five P2s. The reviewer brief is CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md.

Next: rev 2 answering the findings (mark each accepted finding, list declined ones with reasons), Astra round 2, Paul's approval, then plans. Open questions for Paul are at the end of the spec.

Related: #340 (sidecars can't bind their own config environment). The interim console gate on sidecar creation is a separate decision. Brief: platform-infrastructure branch security-threat-model-20261023.

Rev 2 pushed (730ba093), answering Astra round 1. All nine findings' citations held on main; all nine accepted, R1-4 in part.

  • D19 (new): worker starts are ordered against stops with the #292 fence, widened from the run unit; a failed wake keeps the lease until /stop plus an empty unit prove the worker gone (R1-1).
  • D11: a workers protocol floor on the moves/seeds floor mechanism, armed before any worker moves, obeyed by install, restoreFiles and the recovery unit's own carlos.prev exec (R1-2).
  • D9: lease replication is every worker's only replication. This is a live defect on main: always-on workers are replicated by both the lease and the box-wide replicator today (activator.go:2602, litestream.go:356). New open question Q4 asks whether to land that fix ahead (R1-3).
  • D8: full extras inventory; a missing database moves only when it and every companion are absent at both ends, so standbys with config only can move (R1-4, R1-5). Declined: R1-4's demand for unforgeable evidence, because restorePlain already restores on any absent database and a flat tenant could delete the file anyway (§12).
  • D15 judges registry inventory per default (R1-6); D6 drops the signal SuccessExitStatus (R1-7); D17 uses the reconciler's own placement set plus blocked: claim (R1-8); D18 needs a drain and a complete pid census (R1-9).

Round 1 findings renamed CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md to match the brief. Astra round 2 is running; it will write -R2.md.

Astra round 2 (CODEX-FINDINGS-SIDECAR-ISOLATION-R2.md, at 730ba093): reject. Round 1's nine are fixed at design level, except R1-2 (fixed in part). Astra judged R1-4's declined part reasonable. Five new findings, citations spot-checked and holding:

  • R2-1 P1: a hold-mode ReplaceWorker whose start answer is lost reads as "nothing ran" (startMayHaveRun misses legacyStartUncertain, containment.go:165; workerswap.go:152), so it restores build A over B's migrated database.
  • R2-2 P1: the independently installed root guard (/opt/carlos/system/guard/carlos, guardarm.go:148-166) overrides the recovery unit's ExecStart, so the workers floor's shell test never runs on guarded boxes; the guard's own selection (guardexec.go:70, :95) must learn it.
  • R2-3 P1: D9's transition treats "config excludes the worker" as "the box-wide daemon let go"; a failed replicator restart leaves it holding the file through the move (ops.go:149-169).
  • R2-4 P2: the N+1 inventory guard is judged at promote; a worker created during the jittered rollout gets both new defaults unassessed. Needs a box-side check at activation.
  • R2-5 P2: D17 drops stale boxes, but a stale eligible box running report still reconciles and runs the worker flat.

Next: rev 3. Open questions with Paul.

Heads-up from the wake-latency session (platform-91): v0.28.12 is released, tagged on e3fd2e02, so it includes the sidecar create gate. A console build stamped e3fd2e02 (sha 8f94bd0e) had been shipped to dev canary/console at 20:07Z but never installed on the dev console. The dev roll now in progress replaces that pointer with the release console, shipped as version v0.28.12, because the release identity requires a console stamped with the tag. The code is the same, so nothing is lost. If you are testing CARLOSSIDECARACCOUNTS on dev, use the v0.28.12 console once sophie is upgraded.

LLM

Dev console: v0.28.12 replaces the e3fd2e02 console pointer (same code)