amadan
Sidecar isolation: implementation plan for release N (spec rev 3)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+466
−0
docs/superpowers/plans/2026-10-24-sidecar-isolation.md+466 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5 go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0 go: downloading github.com/aws/smithy-go v1.28.1 go: downloading github.com/superfly/ltx v0.5.1 go: downloading golang.org/x/net v0.56.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6 go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading golang.org/x/crypto v0.53.0 go: downloading golang.org/x/sys v0.46.0 go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0 go: downloading github.com/oschwald/maxminddb-golang v1.13.1 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6 go: downloading github.com/pion/logging v0.2.4 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading github.com/superfly/ltx v0.5.1 go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0 go: downloading github.com/klauspost/compress v1.18.0 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading golang.org/x/text v0.38.0 go: downloading golang.org/x/text v0.38.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/prometheus/common v0.45.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.45.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/google/uuid v1.6.0 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/google/uuid v1.6.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/plans/2026-10-24-sidecar-isolation.md b/docs/superpowers/plans/2026-10-24-sidecar-isolation.md| new file mode 100644 |
| index 000000000..af998ccb9 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-10-24-sidecar-isolation.md |
| @@ -0,0 +1,466 @@ |
| +# Sidecar isolation (B1) Implementation Plan |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Native sidecars run on the nested tier, and a box refuses every |
| +flat tenant spawn once nothing on it would be refused, closing security |
| +finding B1 in release N and its patch N.1. |
| + |
| +**Architecture:** A fifth nested role, `worker`, with its own unit template, |
| +started through the existing root control channel; the layout pass moves |
| +worker rows the way it moves instance rows; one policy check in the two |
| +spawn seams refuses flat and foreign launches, with an unset policy that |
| +ratchets to `refuse` per box. D9 fixes double replication of always-on |
| +workers. Nothing in `host.env` changes on any box. |
| + |
| +**Tech Stack:** Go (module `github.com/carlosframework/platform`), systemd |
| +units embedded in `internal/host/units`, cgroup v2, litestream. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md`, |
| +revision 3 (99017de5). Read §0, §3, §4, §8, §13 before any task. Decisions |
| +are cited as D-numbers; this plan implements D1 to D14, D16, D19 and D11/D15 |
| +as revised in rev 3. D17 and D18 are follow-ups and are NOT built here. |
| + |
| +## Global Constraints |
| + |
| +- Base: merge `origin/main` (e3fd2e02 or later) into the branch before Task 1. |
| +- `release.WorkerLayoutFloor` = the tag of release N. Use |
| + `SemVer{Major: 0, Minor: 29, Patch: 0}` (N = v0.29.0; the latest published |
| + tag is v0.28.12). If the release is cut under another tag, change this one |
| + constant and its test. |
| +- Root-run files (`internal/instancectl/*`, `internal/host/units/*`, |
| + `internal/host/systemapply.go`) change root-read census digests: every |
| + task touching them runs `make census-card` and notes its claims in |
| + `implementation/census-claims-sidecar-isolation.txt` |
| + (`<file>:<unit><TAB><what changed>`); Task 12 runs |
| + `make census-propose NAME=sidecar-isolation CLAIMS=<that file>`, which |
| + writes the overlay, its chain line and the card at |
| + `docs/superpowers/notes/census-proposal-sidecar-isolation.md` (AGENTS.md, |
| + "The census: an approval, not a decision"). Paul approves the card before |
| + merge. A `reads`/`new` class needs hand rows (#369). |
| +- Every user-facing string marked "copy to review" in the spec is written |
| + as a placeholder constant in one file per package |
| + (`*_copy.go`), collected in Task 12, reviewed, then filled. Error |
| + fragments, log lines and JSON keys are not copy. |
| +- Closed status words, exactly: flat-exec reasons `flat_refused`, |
| + `foreign_refused`, `flat_reported`, `foreign_reported`; intent sources |
| + `nested`, `flat_choice`, `flat_fallback`, `unknown`; ratchet states |
| + `report`, `refuse`; workers readiness `nested`, `off`, |
| + `waiting_for_prev`, `blocked`. |
| +- Env names, exactly: `CARLOS_NESTED_WORKERS` (`on`|`off`, unset = on), |
| + `CARLOS_FLAT_EXEC` (`refuse`|`report`, unset = ratchet, unknown = |
| + refuse), `CARLOS_FLAT_EXEC_ALLOW` (comma-separated exact hosts, |
| + `provision.ValidHostShape`), `CARLOS_SIDECAR_DB` (agent-owned, always set |
| + for a worker). |
| +- Ratchet file: `<DataDir>/.carlos/flat-exec-refuse`, mode 0600, content |
| + the RFC 3339 time it turned, written with write-temp-then-rename, never |
| + removed by code. |
| +- Use `registry.Route.IsSidecar()`, never the `"sidecar"` literal. |
| +- Wire changes are additive (status snapshot fields `omitempty`). |
| +- Tests go through production wiring where the package already does |
| + (AGENTS.md; `production-wired-witnesses`): real `instancectl.Server` on a |
| + unix socket with faked `Systemctl`, real embedded units, real |
| + `layout.Pass` over a real registry. |
| +- Locally run only the touched packages' tests with `-run`; CI runs the |
| + full gate. Go: default `GOCACHE`; `GOFLAGS=-mod=mod GOPROXY=off`. |
| +- Cross-compile gate: anything Linux-only needs `//go:build linux` and a |
| + `!linux` stub. |
| + |
| +## Review Focus |
| + |
| +1. **A worker that computes `/data/<host>.db` itself starts nested and |
| + crashes.** It must read as a crash (D6), restart with backoff, and say |
| + in `carlos sidecars logs` that the database open failed. Pinned in |
| + Task 6's crash test with a stub that opens the flat path. |
| +2. **A box with a foreign row (`carlos add --db`) never turns its |
| + ratchet.** Status must name the row and the reason. Pinned in Task 9's |
| + status test. |
| +3. **Agent restart mid-move of an always-on worker.** The journal recovery |
| + must finish or revert the move, and no wake source may start the |
| + worker while fenced. Pinned in Task 8 (fenced wake-source tests, boot |
| + pass included). |
| +4. **A worker token presented on another worker's per-host socket, or on |
| + the shared socket after the move.** Uniform 401. Pinned in Task 5. |
| +5. **`carlos.prev.version` missing on an old box** (written only since the |
| + moves floor work): workers stay flat with `waiting_for_prev`, never |
| + move. Pinned in Task 8. |
| + |
| +--- |
| + |
| +### Task 0: Branch setup |
| + |
| +- [ ] Merge `origin/main` into `design-sidecar-isolation` (no conflicts |
| + expected: main's only sidecar change is `internal/console/sidecarsgate.go`). |
| +- [ ] `go build ./... && go vet ./internal/activator ./internal/instancectl ./internal/layout` |
| + clean. Commit the merge. Push. |
| + |
| +### Task 1: D9, one replication owner (no flag) |
| + |
| +**Files:** |
| +- Modify: `internal/ops/litestream.go` (the `Hibernate` exclusion at |
| + `:356-357`; `CoverageShortfall`) |
| +- Modify: `internal/ops/hibernate_verify.go:45-60` (`HibernatingHosts`) |
| +- Test: `internal/ops/litestream_test.go`, `internal/ops/hibernate_verify_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func ActivatorReplicated(rt registry.Route) bool` in |
| + `internal/ops`: `rt.Backing != registry.BackingUnit && rt.DBPath != "" && |
| + (rt.Hibernate || rt.IsSidecar())`. `HibernatingHosts` keeps its name and |
| + selects by this predicate (doc comment updated). |
| + |
| +- [ ] Test `TestFleetDBsExcludesAlwaysOnWorker`: registry with an always-on |
| + sidecar row (`Kind "sidecar"`, `Hibernate false`, `DBPath /data/w.db`) |
| + and a file there; generated DB set does not contain `/data/w.db`. |
| +- [ ] Test `TestHibernatingHostsIncludesAlwaysOnWorker`. |
| +- [ ] Test `TestCoverageShortfallAcceptsActivatorReplicatedDrop`: prior |
| + config names `/data/w.db`; new set lacks it because the row is |
| + `ActivatorReplicated`; no shortfall, and the drop is named on stderr. |
| + A drop for any other reason is still a shortfall. |
| +- [ ] Implement; run `go test ./internal/ops -run 'ActivatorReplicated|AlwaysOnWorker|CoverageShortfall'`. |
| +- [ ] Commit. |
| + |
| +### Task 2: the `worker` role at root (D1, D19 root half) |
| + |
| +**Files:** |
| +- Modify: `internal/instancectl/names.go` (`RoleWorker = "worker"`, |
| + `UnitPrefixWorker = "carlos-instance-worker@"`, `prefixFor`, `ParseRole`, |
| + `ParseUnitName`) |
| +- Modify: `internal/instancectl/server.go` (`handleStart` `:461-530`, the |
| + stop, reset and sweep handlers: admit `RoleWorker`; worker tier rule = run |
| + role's, always `setTier`; `gen != 0` with `RoleWorker` is 400) |
| +- Modify: `internal/instancectl/legacystart.go` (fence keyed by host and |
| + role: `legacyStopping`, `legacyBarrier`, attempt maps become keyed by |
| + `roleHost{role Role; host string}`; applies to `RoleRun` and `RoleWorker`) |
| +- Modify: `internal/instanceexec/exec.go:44-53`, `exit.go` (accept `worker`) |
| +- Test: `internal/instancectl/server_test.go`, `legacystart_test.go`, |
| + `names_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `instancectl.RoleWorker`, `instancectl.UnitPrefixWorker`; |
| + `UnitName(RoleWorker, host)` = `carlos-instance-worker@<InstanceID(host)>.service`; |
| + `SliceName(host)` unchanged. |
| + |
| +- [ ] Tests: `TestParseUnitNameWorker`; `TestStartWorkerSetsTier`; |
| + `TestStartWorkerRefusesGen`; `TestWorkerStartValidatesStagedExe` (the |
| + worker host's own app); and the #292 shape for the worker role: |
| + `TestWorkerStopCancelsAndJoinsHeldStart` (held handler never submits |
| + after `/stop` answers), `TestWorkerStartAcceptedBeforeStopIsRefused`, |
| + `TestWorkerJoinPastBudgetAnswersError`, |
| + `TestWorkerStopDoesNotFenceRunUnit` and its reverse. Copy the existing |
| + run-unit tests in `legacystart_test.go`, parameterised by role. |
| +- [ ] Implement. `clearLegacyClaimLaunch` stays run-only. |
| +- [ ] `go test ./internal/instancectl ./internal/instanceexec -count=1`. |
| +- [ ] `make census-card`; add this task's claims. Commit. |
| + |
| +### Task 3: the worker unit template (D1, D6) |
| + |
| +**Files:** |
| +- Create: `internal/host/units/carlos-instance-worker@.service` |
| +- Modify: `internal/host/units_isolation_test.go:231` |
| + (`TestInstanceTemplatesShareOneNamespace` over five templates) |
| +- Modify: wherever embedded units are listed for install (grep |
| + `carlos-instance-sidecar@.service` in `internal/host/*.go`) |
| + |
| +The file is `carlos-instance-sidecar@.service` with exactly five lines |
| +different: `Description=` ends `(worker)`; `ExecStart=/opt/carlos/carlos |
| +instance-exec worker %I`; `ExecStopPost=/opt/carlos/carlos instance-exit |
| +worker %I`; `StandardOutput=file:/run/carlos/logs/%I.worker.sock`; and the |
| +`SuccessExitStatus=` line deleted. |
| + |
| +- [ ] Test `TestWorkerTemplateDiffersInFiveLines`: line diff against the |
| + sidecar template is exactly those five. |
| +- [ ] Widen the namespace identity test. Run |
| + `go test ./internal/host -run 'Templates|WorkerTemplate'`. |
| +- [ ] `make census-card`; add this task's claims. Commit. |
| + |
| +### Task 4: `Classify` and `WorkerManaged` (D2) |
| + |
| +**Files:** |
| +- Modify: `internal/activator/layout.go:63-111` |
| +- Modify: `internal/registry` or `internal/activator` (wherever |
| + `ActivatorManaged` lives): add `WorkerManaged` |
| +- Test: `internal/activator/layout_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func WorkerManaged(rt registry.Route) bool` = |
| + `rt.IsSidecar() && rt.Backing == "" && rt.Tunnel == ""`, beside |
| + `ActivatorManaged`. `Classify` on a sidecar row follows spec D2's table. |
| + The conflict scan's peer predicate is `ActivatorManaged(p) || WorkerManaged(p)`. |
| + |
| +- [ ] Table test `TestClassifyWorkerRows`: journal → fenced; identifier |
| + collision with an instance row → conflict, both named; `Socket ""` + |
| + `DefaultDBPath` → flat; `Socket ""` + `NestedDBPath` → nested; a socket |
| + on a sidecar row → foreign; any other path → foreign. |
| +- [ ] Implement. `paths.go` needs no change; assert so with |
| + `TestPathsForNestedWorker` (nested DB, empty socket, nested run dir). |
| +- [ ] Commit. |
| + |
| +### Task 5: per-host agent socket and the worker's `prepare` (D3) |
| + |
| +**Files:** |
| +- Modify: `internal/agentsock/agentsock.go` (add a host-pinned wrapper) |
| +- Modify: `internal/activator/prepare.go` (`prepareKeyedAt` for `RoleWorker`) |
| +- Modify: `internal/activator/activator.go` (`workerEnv` `:3321-3328` |
| + takes the socket path from the backend) |
| +- Test: `internal/agentsock/agentsock_test.go`, `internal/activator/prepare_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `func (s *Server) ServePinned(ctx context.Context, path, host string) error` |
| + in `agentsock`: binds `path` mode 0600, answers only requests whose |
| + `Carlos-Sidecar-Host` equals `host` (exact, single header value; absent, |
| + repeated or other → the existing uniform `refuse`), then the existing |
| + token check. Returns when ctx is done; removes the socket file. |
| +- Produces: `func (b *ExecBackend) agentSocketFor(rt registry.Route) string`: |
| + `filepath.Join(provision.InstanceRunDir(rt.Host), "agent.sock")` for a |
| + nested worker row, the shared socket otherwise. |
| +- `prepare` for `RoleWorker`: directories, env layers through `instancefs`, |
| + no admin token, the log listener, and the pinned agent listener started |
| + before it returns; the listener is closed by `unitProc.closeListeners`. |
| + |
| +- [ ] Tests: `TestPinnedSocketRefusesOtherHost`, |
| + `TestPinnedSocketRefusesRepeatedHostHeader`, |
| + `TestPinnedSocketRefusesAbsentHostHeader`, |
| + `TestPinnedSocketMode0600`, `TestWorkerTokenRefusedOnSharedSocketAfterMove` |
| + (Review Focus 4), `TestPrepareWorkerBindsAgentSocketBeforeReturn`, |
| + `TestPrepareWorkerHasNoAdminToken`. |
| +- [ ] Implement; run `go test ./internal/agentsock ./internal/activator -run 'Pinned|PrepareWorker|SharedSocket'`. |
| +- [ ] Commit. |
| + |
| +### Task 6: `StartWorkerStaged` nested arm, `CARLOS_SIDECAR_DB`, crash reading, start ordering (D4, D5, D6, D19 agent half) |
| + |
| +The largest task; one implementer, Fable-tier. |
| + |
| +**Files:** |
| +- Modify: `internal/activator/backend_exec.go` (`StartWorkerStaged` |
| + `:3049-3155`; `LegacyRunGone` `:2595-2640` generalised) |
| +- Modify: `internal/activator/activator.go` (`workerEnv`; the failed-wake |
| + branch `:2543-2566`; respawn error arm `:3739`) |
| +- Modify: `internal/activator/containment.go` (`startMayHaveRun` |
| + `:160-171`, `legacyStartMayRun`) |
| +- Modify: `internal/activator/workerswap.go` (unchanged-incarnation |
| + branch `:152-160`) |
| +- Modify: `internal/activator/unitproc.go` (add `Crashed`) |
| +- Modify: `internal/activator/worker_gen_test.go:50-63` (rewrite |
| + `TestKillHostProvesANestedLayoutWorkersHostCgroup`) |
| +- Test: `internal/activator/worker_nested_test.go` (new) |
| + |
| +**Interfaces:** |
| +- Consumes: Task 2's `RoleWorker`; Task 4's `Classify`; Task 5's |
| + `agentSocketFor` and `prepare`. |
| +- Produces: `func (b *ExecBackend) RoleGone(ctx context.Context, rt registry.Route, role instancectl.Role) error` |
| + (the body of `LegacyRunGone`, with `LegacyRunGone` now calling it with |
| + `RoleRun`); `func (p *unitProc) Crashed() bool` per spec D6. |
| +- `workerEnv` adds `CARLOS_SIDECAR_DB=<rt.DBPath>` (agent-owned, applied |
| + over tenant layers) for flat and nested workers. |
| +- Nested arm order (spec D5): claim refusal → `nestedGate` → `prepare(rt, |
| + RoleWorker)` → `launch.worker.json` with argv `[liveLink(host), args...]` |
| + and composed env → `launchGate` → control `/start` for `RoleWorker` |
| + always naming the staged build (`StagedExe` from the resolved id) → |
| + `newUnitProc(rt, RoleWorker, ...)`. Fenced and conflict refuse with the |
| + reason. Flat and foreign run today's body. |
| +- An error from the nested `/start` other than an answered refusal that ran |
| + nothing is wrapped so that both `legacyStartMayRun` and |
| + `startMayHaveRun` return true. |
| + |
| +- [ ] Tests (fake systemd running the trampoline stub, real control server): |
| + `TestStartWorkerStagedNestedNamesStagedBuild` (never the live link); |
| + `TestStartWorkerStagedNestedAsksExecGateBeforeStart`; |
| + `TestLaunchWorkerJSONCarriesArgvAndEnv` (`CARLOS_SIDECAR_DB`, per-host |
| + `CARLOS_AGENT_SOCKET`); `TestFlatWorkerGetsSidecarDB`; |
| + `TestStartWorkerStagedFencedAndConflictRefuse`; |
| + `TestUnitProcCrashedOnExit143` and `OnExit141` (true); |
| + `TestUnitProcNotCrashedOnExit0`; `TestUnitProcIgnoresForgedExitRecord`; |
| + `TestUnitProcCollectedInvocationNotCrash`; |
| + `TestRequestedStopNeverReachesProcCrashed`; |
| + `TestNestedWorkerComputedFlatPathCrashLoops` (Review Focus 1: stub opens |
| + `/data/<host>.db`, reads as crash, restarts with backoff, log line names |
| + the open failure); |
| + `TestFailedNestedWorkerWakeKeepsLeaseUntilRoleGone` (start response lost, |
| + handler held; lease kept; released only after `RoleGone`); |
| + `TestHoldReplaceWorkerUncertainStartRestoresNothing` (R2-1); |
| + `TestOverlapReplaceWorkerFallsBackOnlyAfterRoleGone`; |
| + the rewritten `KillHost` test; end to end |
| + `TestNestedWorkerLifecycle` (wake, ready, next-due, idle, sleep, timer |
| + wake, crash restart, finish park). |
| +- [ ] Implement. Run `go test ./internal/activator -run 'Worker|UnitProc|RoleGone|KillHost' -count=1`. |
| +- [ ] Commit. |
| + |
| +### Task 7: minting (D7) and the D11 readiness condition |
| + |
| +**Files:** |
| +- Modify: `internal/provision/sidecar_reconcile.go` (`sidecarRouteFor` |
| + `:131-148`; `convergeSidecar` shape check `:621-626`) |
| +- Create: `internal/host/workerready.go` (the box's workers readiness) |
| +- Modify: `internal/release/layoutintent.go` (add `WorkerLayoutFloor`) |
| +- Test: `internal/provision/sidecar_reconcile_test.go`, `internal/host/workerready_test.go` |
| + |
| +**Interfaces:** |
| +- Produces: `release.WorkerLayoutFloor = SemVer{0, 29, 0}`. |
| +- Produces: `func WorkersReady(getenv func(string) string, optDir string, running string) (ready bool, word string, reason string)` |
| + in `internal/host`: `CARLOS_NESTED_WORKERS` unset or `on` (`off` → |
| + `off`); `running` and the content of `<optDir>/carlos.prev.version` both |
| + parse at or above `WorkerLayoutFloor` (otherwise `waiting_for_prev`, |
| + including missing/unreadable/unparseable — Review Focus 5). Effective |
| + intent and gate are checked by callers. Exposed to the reconciler and |
| + the pass through their existing layout-reader seams (`mintLayoutReader`, |
| + `reconcile.go:471-490`, gains a `workers bool`). |
| +- `sidecarRouteFor` mints `NestedDBPath` exactly when the instance mint |
| + would mint nested **and** `WorkersReady`; `convergeSidecar` accepts the |
| + flat or nested path for the row's host. |
| + |
| +- [ ] Tests: `TestWorkersReadyTable` (unset, on, off, prev missing, prev |
| + below, prev unparseable, running below, both at floor); |
| + `TestSidecarMintNestedOnlyWhenWorkersReady`; |
| + `TestConvergeSidecarAcceptsBothPaths`; co-spawn check unchanged. |
| +- [ ] Commit. |
| + |
| +### Task 8: the layout pass moves worker rows (D8, D9 running check, D11) |
| + |
| +**Files:** |
| +- Modify: `internal/layout/pass.go` (eligibility `:264`, `:529`, `:747`; |
| + the missing-database refusal `:347-355`; `presence` `:805-830`) |
| +- Modify: `internal/layout/plan.go` (`PlanFor`: worker hosts by |
| + `provision.SidecarSuffix` get `Socket ""` endpoints; `Required` for `db` |
| + per D8 rev 2) |
| +- Modify: `internal/appconfig/syncexec.go` (`nestedRoute`/`ConfigEnvPath` |
| + `:66-79` learn the worker shape; fenced skip covers worker hosts) |
| +- Modify: `internal/host/layoutleg.go:193-199` (report worker rows) |
| +- Create: `internal/layout/replicator.go` (`boxReplicatorHolds`) |
| +- Test: `internal/layout/pass_worker_test.go` (new), `internal/appconfig/syncexec_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 4's `WorkerManaged`, Task 7's readiness (passed into |
| + `layout.Pass` as `WorkersReady func() bool`; nil = false). |
| +- Produces: `func boxReplicatorHolds(dbPath string, show func(unit, prop string) (string, error), configPath string) (bool, string)`: |
| + true (hold, with reason) when `carlos-litestream` is active and |
| + (its `ActiveEnterTimestamp` is before the config's mtime, or the config |
| + names `dbPath`). Applied to always-on worker rows only. |
| +- Missing-database rule: `db` not required when `db`, `db-wal`, `db-shm`, |
| + `litestream-shadow`, `corrupt` and `<db>-litestream` are all absent at |
| + both endpoints; then the journal records it absent and the rest moves. |
| + A companion without its database refuses with the names. |
| +- Worker test is the host suffix; a row whose `Kind` disagrees with the |
| + suffix is refused like any journal mismatch. |
| + |
| +- [ ] Tests: `TestPassMovesHibernatingWorkerAsleep`; |
| + `TestPassDefersAwakeAlwaysOnWorkerThenStopsOnce`; |
| + `TestFencedWorkerRefusesEveryWakeSource` (timer, `carlos sidecars wake`, |
| + boot pass, sweep's always-on start — Review Focus 3, including an agent |
| + restart mid-move); |
| + `TestWorkerJournalKindSuffixMismatchRefused`; `TestWorkerExtrasMove`; |
| + `TestStandbyRowMovesConfigOnly`; |
| + `TestCompanionWithoutDatabaseRefused`; |
| + `TestTwoBoxFailoverAfterConfigOnlyMove`; |
| + `TestRevertWorkerByteForByte`; |
| + `TestAlwaysOnMoveHeldWhileReplicatorNamesFile` and |
| + `...HeldWhileReplicatorStartedBeforeConfig`, `...ProceedsWhenInactive`; |
| + `TestWorkersNotEligibleWithoutReadiness` (Review Focus 5); |
| + `TestConfigEnvPathNestedWorker`; `TestLayoutReportListsWorkerRows`. |
| +- [ ] Commit. |
| + |
| +### Task 9: the flat-exec policy, inventory and ratchet (D12 to D16) |
| + |
| +**Files:** |
| +- Create: `internal/activator/flatexec.go` (`FlatExecPolicy`, `Admit`, |
| + `ErrFlatExecRefused`) |
| +- Modify: `internal/activator/backend_exec.go:455-509` |
| + (`spawnInCgroupGated`), `internal/activator/containment.go:234-276` |
| + (`spawnInCgroupAtTier`): the check after the claim refusal |
| +- Modify: `internal/activator/swap.go:225-245`, `activator.go:3474-3479` |
| + (classify the new error) |
| +- Create: `internal/host/flatexec.go` (env reading, the ratchet, the |
| + inventory, wiring into the agent and the status snapshot) |
| +- Modify: `internal/status/snapshot.go` (additive `Metrics.FlatExec`, |
| + `Metrics.Layout.Workers`) and its contract test |
| +- Modify: `internal/host/layoutleg.go` (D16: a refused managed or worker |
| + row on a nested-intent box marks the layout leg dirty) |
| +- Test: `internal/activator/flatexec_test.go`, `internal/host/flatexec_test.go`, |
| + `internal/activator/spawn_enumeration_test.go` |
| + |
| +**Interfaces:** |
| +- Produces in `activator`: `type FlatExecPolicy struct{ Mode atomic.Value /* "refuse"|"report" */; Allow map[string]bool; Record func(host, reason string) }`; |
| + `func (p *FlatExecPolicy) Admit(rt registry.Route, layout Layout) error` |
| + (nil policy = today's behaviour); `var ErrFlatExecRefused` wrapping |
| + `hostbox.ErrUnsupported`. `ExecBackend.FlatExec *FlatExecPolicy`. |
| +- Produces in `host`: `func flatExecFromEnv(getenv func(string) string) (mode string, explicit bool, allow []string, dropped []string)`; |
| + `type flatExecRatchet struct{ path string }` with `Turned() (bool, time.Time)` |
| + and `Turn(now time.Time) error`; `func flatExecInventory(routes []registry.Route, classify func(registry.Route) activator.Layout, allow map[string]bool) []InventoryRow`. |
| + The agent evaluates on every layout pass: if not explicit, intent nested, |
| + inventory empty and not turned → `Turn`, then `Mode` = `refuse`. |
| + Flat-by-choice intent never turns. Explicit always wins. |
| + |
| +- [ ] Tests: `TestFlatExecEnvTable` (each value, unset, unknown → refuse, |
| + invalid allowlist entry dropped and reported, wildcards rejected); |
| + `TestAdmitRefusesFlatAndForeignNotAllowlisted`; both seams refuse |
| + before `Cgroups` is touched (including nil manager); |
| + `TestEveryExecCommandReachesASeam` (enumerates `exec.Command` in |
| + `internal/activator`, four today); swap gate and respawn veto classify |
| + `ErrFlatExecRefused`; `TestRatchetTurnsOnFirstEmptyInventory`, |
| + `...StaysAcrossRestartAndLaterRow`, `...NeverUnderFlatChoice`, |
| + `...ExplicitWinsAndKeepsFile`, `...RecordedFlatIntentIsChoice`, |
| + `...GateFallbackIsNotChoice`; `TestInventoryIncludesDormantRow`; |
| + `TestStatusNamesForeignRowHoldingRatchet` (Review Focus 2); |
| + `TestRefusedManagedRowNudgesPass` and not for foreign; status contract |
| + test passes with the new fields and they never reach `fleet.json`. |
| +- [ ] Commit. |
| + |
| +### Task 10: downgrade guards (D11) |
| + |
| +**Files:** |
| +- Modify: `internal/host/systemapply.go` (`:510-528` beside R9's guard; |
| + the row test `layoutReversalIncomplete` `:1273-1310` covers worker hosts) |
| +- Modify: `internal/console/layoutguard.go:24-69` |
| +- Test: `internal/host/systemapply_test.go`, `internal/console/layoutguard_test.go` |
| + |
| +**Interfaces:** |
| +- Consumes: Task 7's `release.WorkerLayoutFloor`. |
| +- Install refuses (before the journal) a candidate whose version parses |
| + below `WorkerLayoutFloor`, or whose staged unit set lacks |
| + `carlos-instance-worker@.service`, while `layoutReversalIncomplete` |
| + names a worker host (host has `provision.SidecarSuffix`). Reason text is |
| + copy (Task 12). `layoutDowngradeBlocked` gains the same floor over |
| + observed worker hosts. |
| + |
| +- [ ] Tests: `TestInstallRefusesPreWorkerCandidateWithNestedWorker`, |
| + `...WithWorkerJournalOnly`, `...AcceptsWhenNoWorkerHost`, |
| + `...UnparseableVersionJudgedByUnitSet`; console |
| + `TestDowngradeBlockedBelowWorkerFloor`. |
| +- [ ] `make census-card`; add this task's claims. Commit. |
| + |
| +### Task 11: every other consumer of the shape (spec §3.12) |
| + |
| +**Files:** the sampler's slice-to-host mapping (`internal/host/metrics_probes.go`); |
| +`carlos status --instances`' unit matcher (grep `UnitPrefixSidecar` and |
| +`carlos-instance-sidecar@` outside `instancectl`); engines `arm`'s "no |
| +route spawns workers" check (engines design §6.2) becomes "no worker row |
| +classifies flat or foreign"; `internal/hostbox/inventoryscope.go:48` |
| +(assert only). |
| + |
| +- [ ] First step: re-run the consumer sweep (`grep -rn 'IsSidecar()\|UnitPrefix\|carlos-instance-' --include=*.go`) |
| + and list any consumer not in spec §3.12 in the task's commit message. |
| +- [ ] Tests per consumer: a nested worker's slice maps to its sidecar |
| + host; `status --instances` lists a worker unit; engines arm refuses with |
| + a flat worker row and passes with a nested one. |
| +- [ ] Commit. |
| + |
| +### Task 12: copy, docs, census proposal |
| + |
| +- [ ] Collect every `*_copy.go` placeholder and the docs paragraphs |
| + (`docs/site/cli.md`: sidecar section with `CARLOS_SIDECAR_DB` and |
| + `CARLOS_AGENT_SOCKET` as the contract, `carlos status --layout`'s new |
| + lines, the three box settings and the ratchet; README box-settings list) |
| + into one deck; run it through a Luna pass on the house rules |
| + (`large-copy-decks-delegated`), apply verbatim. |
| +- [ ] `make census-propose NAME=sidecar-isolation CLAIMS=implementation/census-claims-sidecar-isolation.txt`; the card goes to Paul for approval. |
| +- [ ] `go test ./internal/docsite` passes. |
| +- [ ] Commit. |
| + |
| +### Task 13: box-level verification (dev, after the build) |
| + |
| +Not code. Run on the root runner or a dev box, recorded in the branch's |
| +discussion: a real nested worker unit (namespace listing, probe datagrams, |
| +`PrivatePIDs`, per-host socket reachable and shared one absent, logs, crash |
| +including self-SIGTERM read as crash, requested stop leaving `failed` and |
| +the next start succeeding, edge restart and boot-pass restart of an |
| +always-on worker); §1.6's cgroup move attempt under the fence. Then the |
| +release steps of spec §8.2. |