amadan
Sidecar isolation design (security finding B1), revision 1, and its review brief
Native sidecars start as plain edge children on every box, whatever its layout. The spec runs them as units of a new carlos-instance-worker@ template, moves existing sidecar rows through the layout pass, refuses flat and foreign spawns unless host.env allows them (reporting first, refusing by default one release later), refuses sidecar creates the target region cannot isolate, and states when the interim metadata fence can go. Draft for an independent review, then Paul.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 files changed,
+1254
−0
docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md+1092 −0docs/superpowers/specs/CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md+162 −0
CI passed — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (0 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (18 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (170 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.587s
ok github.com/carlosframework/platform/cmd/moveprobe 11.858s
ok github.com/carlosframework/platform/cmd/zddprobe 0.775s
ok github.com/carlosframework/platform/internal/activator 60.387s
ok github.com/carlosframework/platform/internal/agentsock 0.029s
ok github.com/carlosframework/platform/internal/alerting 0.009s
ok github.com/carlosframework/platform/internal/analytics 0.060s
ok github.com/carlosframework/platform/internal/appconfig 2.924s
ok github.com/carlosframework/platform/internal/apperr 0.192s
ok github.com/carlosframework/platform/internal/applog 0.270s
ok github.com/carlosframework/platform/internal/awsinv 0.007s
ok github.com/carlosframework/platform/internal/boxchoice 0.021s
ok github.com/carlosframework/platform/internal/boxid 0.625s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.010s
ok github.com/carlosframework/platform/internal/caps 0.023s
ok github.com/carlosframework/platform/internal/certwarm 0.005s
ok github.com/carlosframework/platform/internal/cgroup 0.782s
ok github.com/carlosframework/platform/internal/changelog 0.511s
ok github.com/carlosframework/platform/internal/claimactivation 1.222s
ok github.com/carlosframework/platform/internal/clamd 0.066s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 7.906s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.033s
ok github.com/carlosframework/platform/internal/dbquery 0.019s
ok github.com/carlosframework/platform/internal/dbreads 2.077s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.010s
ok github.com/carlosframework/platform/internal/dnsrec 0.012s
ok github.com/carlosframework/platform/internal/docsite 0.031s
ok github.com/carlosframework/platform/internal/domains 2.771s
ok github.com/carlosframework/platform/internal/doorkey 0.014s
ok github.com/carlosframework/platform/internal/economics 0.015s
ok github.com/carlosframework/platform/internal/edge 45.493s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.135s
ok github.com/carlosframework/platform/internal/gate 1.656s
ok github.com/carlosframework/platform/internal/genselect 0.018s
ok github.com/carlosframework/platform/internal/geo 0.018s
ok github.com/carlosframework/platform/internal/geoip 0.029s
ok github.com/carlosframework/platform/internal/historyreadback 0.157s
ok github.com/carlosframework/platform/internal/host 141.885s
ok github.com/carlosframework/platform/internal/hostactors 6.789s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.005s
ok github.com/carlosframework/platform/internal/hostbox 3.004s
ok github.com/carlosframework/platform/internal/hostclaim 0.743s
ok github.com/carlosframework/platform/internal/hostenv 0.007s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 5.371s
ok github.com/carlosframework/platform/internal/hostfence 0.036s
ok github.com/carlosframework/platform/internal/hostfiles 10.365s
ok github.com/carlosframework/platform/internal/hostfloor 0.029s
ok github.com/carlosframework/platform/internal/hostguard 1.067s
ok github.com/carlosframework/platform/internal/hostops 2.064s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.002s
ok github.com/carlosframework/platform/internal/hostquarantine 0.005s
ok github.com/carlosframework/platform/internal/hostrelocate 0.615s
ok github.com/carlosframework/platform/internal/hostresolve 0.258s
ok github.com/carlosframework/platform/internal/hostwrite 0.002s
ok github.com/carlosframework/platform/internal/infraview 0.005s
ok github.com/carlosframework/platform/internal/instancectl 5.235s
ok github.com/carlosframework/platform/internal/instanceexec 2.819s
ok github.com/carlosframework/platform/internal/instancefs 0.019s
ok github.com/carlosframework/platform/internal/instancekey 0.022s
ok github.com/carlosframework/platform/internal/instanceupdates 0.024s
ok github.com/carlosframework/platform/internal/layout 1.843s
ok github.com/carlosframework/platform/internal/lbhealth 0.017s
ok github.com/carlosframework/platform/internal/lease 10.335s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.583s
ok github.com/carlosframework/platform/internal/ledger 0.017s
ok github.com/carlosframework/platform/internal/mail 0.255s
ok github.com/carlosframework/platform/internal/maintenance 0.104s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.562s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.083s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 22.682s
ok github.com/carlosframework/platform/internal/moves/archive 1.021s
ok github.com/carlosframework/platform/internal/objstore 0.291s
ok github.com/carlosframework/platform/internal/ops 0.218s
ok github.com/carlosframework/platform/internal/park 0.279s
ok github.com/carlosframework/platform/internal/plans 0.013s
ok github.com/carlosframework/platform/internal/privatefile 0.005s
ok github.com/carlosframework/platform/internal/provision 24.036s
ok github.com/carlosframework/platform/internal/publicrelease 0.008s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.661s
ok github.com/carlosframework/platform/internal/relay 7.231s
ok github.com/carlosframework/platform/internal/release 26.317s
ok github.com/carlosframework/platform/internal/rollout 3.852s
ok github.com/carlosframework/platform/internal/rolloutwords 0.005s
ok github.com/carlosframework/platform/internal/rootaws 0.008s
ok github.com/carlosframework/platform/internal/rootparticipation 0.062s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.561s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.005s
ok github.com/carlosframework/platform/internal/schedule 2.686s
ok github.com/carlosframework/platform/internal/seeds 0.012s
ok github.com/carlosframework/platform/internal/sim/engines 2.612s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.542s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 4.222s
ok github.com/carlosframework/platform/internal/status 0.074s
ok github.com/carlosframework/platform/internal/steering 0.006s
ok github.com/carlosframework/platform/internal/stopevidence 0.319s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.004s
ok github.com/carlosframework/platform/internal/tunnelclient 0.005s
ok github.com/carlosframework/platform/internal/unixdial 0.005s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.673s
=== internal_console@0 (80 s)
ok github.com/carlosframework/platform/internal/console 78.687s
=== internal_console@1 (43 s)
ok github.com/carlosframework/platform/internal/console 41.855s
=== internal_console@2 (42 s)
ok github.com/carlosframework/platform/internal/console 41.006s
=== internal_console@3 (41 s)
ok github.com/carlosframework/platform/internal/console 40.118s
=== internal_console@4 (42 s)
ok github.com/carlosframework/platform/internal/console 40.398s
=== internal_console@5 (41 s)
ok github.com/carlosframework/platform/internal/console 39.383s
=== internal_console@6 (37 s)
ok github.com/carlosframework/platform/internal/console 35.845s
=== internal_console@7 (46 s)
ok github.com/carlosframework/platform/internal/console 44.608s
=== root@0 (109 s)
ok github.com/carlosframework/platform 107.800s
=== root@1 (83 s)
ok github.com/carlosframework/platform 82.391s
=== root@2 (92 s)
ok github.com/carlosframework/platform 90.786s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50528418 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/24 18:02:16 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E189290F334462, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
=== RUN TestS3StoreMinioContract
=== RUN TestS3StoreMinioContract/conditional_get_and_etag_listing
=== RUN TestS3StoreMinioContract/bump_through_every_mutation_path
--- PASS: TestS3StoreMinioContract (0.02s)
--- PASS: TestS3StoreMinioContract/conditional_get_and_etag_listing (0.01s)
--- PASS: TestS3StoreMinioContract/bump_through_every_mutation_path (0.01s)
PASS
ok github.com/carlosframework/platform/internal/release 0.031s
Bucket created successfully `local/dbf-deploy`.
Bucket created successfully `local/dbf-data-ie`.
Bucket created successfully `local/dbf-data-uk`.
local/dbf-deploy versioning is enabled
Created policy `dbf-console` successfully.
Created policy `dbf-ukbox` successfully.
Added user `dbfconsole` successfully.
Added user `dbfukbox` successfully.
Attached Policies: [dbf-console]
To User: dbfconsole
Attached Policies: [dbf-ukbox]
To User: dbfukbox
=== RUN TestDBForwardMinio
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 18:02:18 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000002
2026/10/24 18:02:18 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000002 maxTXID=0000000000000002 size=370
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000002
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000002 txid.db=0000000000000003
2026/10/24 18:02:18 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000003 maxTXID=0000000000000003 size=454
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000003 txid.db=0000000000000003
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000000 txid.db=0000000000000001
2026/10/24 18:02:18 INFO ltx file uploaded system=store db=source.db replica=file level=0 minTXID=0000000000000001 maxTXID=0000000000000001 size=720
2026/10/24 18:02:18 INFO replica sync system=store db=source.db replica=file txid.replica=0000000000000001 txid.db=0000000000000001
2026/10/24 18:02:18 console: CARLOS_SECRETS_PUBKEY is unset, so no region outside the home can be chosen automatically
2026/10/24 18:02:18 console: could not notify member@example.com of a CLI token issue: keymail: lookup member@example.com: keymail: GET /api/federation/lookup?addr=member%40example.com&from=console%40127.0.0.1%3A38213: could not determine target mailbox
agent.go:135: dbreads: dbr_qhb7rufcniuzwf4luac2pdtjji: planned 3 files (1544 bytes) and restored to 3 in 10ms
agent.go:135: dbreads: dbr_qhb7rufcniuzwf4luac2pdtjji: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/24 18:02:19 console: db read dbr_qhb7rufcniuzwf4luac2pdtjji: relayed 1 chunks (102 bytes) in 1.004s, ended done
agent.go:135: dbreads: dbr_yghy5deik4gbd5ft54r2wxqfse: planned 1 files (720 bytes) and restored to 1 in 8ms
agent.go:135: dbreads: dbr_yghy5deik4gbd5ft54r2wxqfse: output 5 bytes in 1 confirmed chunks in 1ms
2026/10/24 18:02:20 console: db read dbr_yghy5deik4gbd5ft54r2wxqfse: relayed 1 chunks (102 bytes) in 1.003s, ended done
agent.go:135: dbreads: dbr_hpe5kclmexwbadij74utmfg5ti: planned 3 files (1544 bytes) and held through 3 in 9ms
agent.go:135: dbreads: dbr_hpe5kclmexwbadij74utmfg5ti: output 1544 bytes in 1 confirmed chunks in 1ms
2026/10/24 18:02:21 console: db read dbr_hpe5kclmexwbadij74utmfg5ti: relayed 1 chunks (1641 bytes) in 1.003s, ended done
=== RUN TestDBForwardMinio/by_disconnect
=== NAME TestDBForwardMinio
agent.go:135: dbreads: dbr_swsubrfbgm5c7lngymuhrdkenu: planned 3 files (1544 bytes) and restored to 3 in 9ms
2026/10/24 18:02:21 console: db read dbr_swsubrfbgm5c7lngymuhrdkenu: relayed 0 chunks (0 bytes) in 13ms, ended client gone
agent.go:135: dbreads: dbr_swsubrfbgm5c7lngymuhrdkenu: output 5 bytes in 1 confirmed chunks in 45ms
=== RUN TestDBForwardMinio/by_timeout
=== NAME TestDBForwardMinio
agent.go:135: dbreads: dbr_uuuovbeiasxjnoklk5py42gsyq: planned 3 files (1544 bytes) and restored to 3 in 10ms
2026/10/24 18:02:41 console: db read dbr_uuuovbeiasxjnoklk5py42gsyq: relayed 0 chunks (0 bytes) in 19.265s, ended no-result
agent.go:135: dbreads: dbr_uuuovbeiasxjnoklk5py42gsyq: output 5 bytes in 1 confirmed chunks in 19.245s
--- PASS: TestDBForwardMinio (22.64s)
--- PASS: TestDBForwardMinio/by_disconnect (0.16s)
--- PASS: TestDBForwardMinio/by_timeout (19.37s)
PASS
ok github.com/carlosframework/platform/internal/console 22.658s
minio contract: PASS
diff --git a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md| new file mode 100644 |
| index 000000000..83da7c395 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md |
| @@ -0,0 +1,1092 @@ |
| +# Sidecar isolation: native sidecars on the nested tier, and no flat exec by default |
| + |
| +**Date:** 2026-10-24 · **Status:** draft, revision 1, for an independent |
| +review (brief: `CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md`) and then Paul · |
| +**Requested by:** security review finding B1, confirmed live 2026-10-24 · |
| +**Main at:** fef28dc5 (every `file:line` below is on that commit unless it |
| +says otherwise) |
| + |
| +**Builds on:** the isolated-exec design |
| +(`2026-09-11-isolated-exec-design.md`, revision 17; "the nested design" |
| +below), its plan 4 amendments (`../plans/2026-09-25-isolated-exec-4-layout-amendments.md`, |
| +rulings R1 to R25 and findings S1 to S15), the sidecar-members design |
| +(`2026-08-17-sidecar-members-design.md`; "the sidecar design"), and the |
| +fleet-internal engines design (`2026-10-21-fleet-internal-engines-design.md` |
| +§6.2), which already names the same gap from the other side. |
| + |
| +--- |
| + |
| +## 0. The design in one page |
| + |
| +A native sidecar (a `Kind "sidecar"` registry row, the sidecar design's |
| +standalone worker) always starts as a plain child of the edge, on every |
| +box, whatever the box's layout. A plain child runs as uid `carlos` in the |
| +host namespace, so it can read every flat tenant's files, write into every |
| +nested tenant's directory, and reach the instance metadata service. Any |
| +member of an account, and the app's own instance credential, can create |
| +one. This spec closes that in four moves: |
| + |
| +1. **Workers run nested.** A native sidecar runs as a unit of a new |
| + template, `carlos-instance-worker@.service`, with the instance |
| + template's namespace byte for byte, through the root control channel |
| + that already starts every nested role. Its row carries nested paths; |
| + the layout pass moves existing sidecar rows the way it moves instance |
| + rows; `StartWorkerStaged` dispatches on the row's layout like `Start` |
| + and `StartSidecar` already do (§3). |
| +2. **Flat exec needs an operator's say-so.** The two spawn seams every |
| + flat child passes through refuse a flat or foreign launch unless the |
| + box allows it (`CARLOS_FLAT_EXEC` and `CARLOS_FLAT_EXEC_ALLOW` in |
| + `host.env`). The refusal reuses the claim-mode refusal's place, error |
| + class and handling. It ships reporting only, is switched on box by |
| + box, and becomes the default one release later, behind a console guard |
| + that will not move the channel while any box would lose a workload |
| + (§4). |
| +3. **The console refuses a sidecar it cannot place isolated**, at create |
| + time, from what the target region's boxes report, instead of writing a |
| + record that lands flat or never runs (§5). |
| +4. **The interim fence retires** on a box once nothing flat can run there: |
| + the policy refuses, the allowlist is empty, no flat child cgroup is |
| + populated, and the nested probe's metadata line is enforced (§6). |
| + |
| +§7 is the threat model, step by step. §8 is the rollout, dev first. §11 |
| +lists three questions that are Paul's. |
| + |
| +The decisions, for scanning (each is argued where it is made): |
| + |
| +| | Decision | Section | |
| +|---|---|---| |
| +| D1 | A new `worker` role and template; the existing `sidecar` role stays the legacy co-spawn's | §3.1 | |
| +| D2 | A worker row's paths: nested DB, empty socket; `Classify` learns the socketless shape | §3.2 | |
| +| D3 | A nested worker's agent socket is a per-host listener in its own run directory | §3.3 | |
| +| D4 | Every worker is told its database path (`CARLOS_SIDECAR_DB`), flat and nested | §3.4 | |
| +| D5 | `StartWorkerStaged` dispatches on layout; the nested arm passes the staged build to root | §3.5 | |
| +| D6 | `unitProc` answers `Crashed()` from systemd's result, never from the exit record | §3.6 | |
| +| D7 | The sidecar reconciler mints nested exactly when the instance reconciler does | §3.7 | |
| +| D8 | The layout pass moves worker rows; a row with nothing on this box moves by row alone | §3.8 | |
| +| D9 | An always-on worker's box-wide replica starts a new lineage at its nested path | §3.9 | |
| +| D10 | A failed nested gate never reopens flat for a worker | §3.10 | |
| +| D11 | Downgrade below the worker layout is refused by unit set on the box and by floor at the console | §3.11 | |
| +| D12 | `CARLOS_FLAT_EXEC=refuse\|report` plus an exact-host allowlist, read from `host.env` | §4.1 | |
| +| D13 | The refusal sits beside the claim refusal in both spawn seams, with its own error | §4.2 | |
| +| D14 | Refusals are reported per host with closed reason words, like config refusals | §4.3 | |
| +| D15 | Ship `report`; operators switch boxes to `refuse`; the next release defaults to `refuse` behind a console guard | §4.4 | |
| +| D16 | A refused managed row nudges the layout pass for that host | §4.5 | |
| +| D17 | Creation is refused unless every fresh box in scope reports nested workers ready | §5 | |
| +| D18 | The fence retires per box on four observable conditions; an allowlisted host keeps it | §6 | |
| + |
| +--- |
| + |
| +## 1. What exists on main |
| + |
| +### 1.1 Three tiers |
| + |
| +`Classify` (`internal/activator/layout.go:78-111`) sorts a row into `flat`, |
| +`nested`, `fenced`, `conflict` or `foreign`. `Start` dispatches on it |
| +(`internal/activator/backend_exec.go:1834-1843`): nested runs a unit |
| +(`startUnitAt`), fenced and conflict refuse, and everything else, flat and |
| +foreign alike, runs `startChildAt` (`:1995`), a plain exec child. |
| + |
| +- **Unit tier** (`carlos-app@`, `Backing=unit`): DynamicUser, outside this |
| + spec. |
| +- **Nested tier** (`carlos-instance@`, `-seed@`, `-sidecar@`, `-cmd@`, |
| + `-gen@`): uid `carlos` inside a mount namespace where `/data` and |
| + `/run/carlos` are empty tmpfs with the host's own two directories bound |
| + back (`internal/host/units/carlos-instance-sidecar@.service:53`, `:73-74`), |
| + `/etc/carlos` and the control socket directory inaccessible (`:96`), |
| + `PrivatePIDs`, `ProtectControlGroups`, and an egress filter that drops |
| + loopback, private ranges, link-local and both metadata addresses (`:142`). |
| +- **Flat and foreign tier**: a child of the agent, uid `carlos`, in |
| + `carlos-exec@<host>` under `carlos-edge.service`'s delegated subtree. |
| + The edge unit is `User=carlos` with `ReadWritePaths=/data /run/carlos` |
| + and `Delegate=cpu memory io pids` |
| + (`internal/host/units/carlos-edge.service:28`, `:37`, `:101`) and has no |
| + `IPAddressDeny=` line at all. |
| + |
| +### 1.2 Native sidecars always start flat |
| + |
| +The sidecar reconciler mints `DBPath: DefaultDBPath(dataDir, host)` and no |
| +socket (`internal/provision/sidecar_reconcile.go:131-148`, the path at |
| +`:141`). `StartWorkerStaged` (`backend_exec.go:3049-3155`) never asks the |
| +layout: it resolves the build, composes the env from the flat-or-nested |
| +`pathsFor` (`internal/activator/paths.go:33-54`), and spawns with |
| +`spawnInCgroupGated` (`backend_exec.go:3125-3130`). Its two siblings both |
| +dispatch: `Start` (above) and `StartSidecar`, the legacy co-spawn |
| +(`:2920-2928`), whose nested arm is `startSidecarUnitFrom` (`:3170-3227`). |
| +`carlos run` goes further and refuses anything not nested |
| +(`internal/activator/cmdunit.go:172-175`). |
| + |
| +A test pins the current behaviour as intended: |
| +`TestKillHostProvesANestedLayoutWorkersHostCgroup` |
| +(`internal/activator/worker_gen_test.go:50-63`) says "StartWorker spawns |
| +every worker into the flat host cgroup, whatever the row's layout". The |
| +sidecar design recorded this as an open gap in its own §15.4. |
| + |
| +### 1.3 Who can create one |
| + |
| +`POST /api/cli/apps/{account}/{app}/sidecars` admits the instance service |
| +role (`internal/console/authcli.go:791`). The enable door |
| +(`PUT .../sidecars-manifest`, `:789`) is person-only, gated by |
| +`apiAccountApp` (`internal/console/apisidecars.go:525-527`), so any member |
| +of the account can enable and then create. The plan cap check |
| +(`apisidecars.go:314-324`) is dormant. Nothing in the create path asks |
| +whether the target boxes can isolate the workload (`:216-391`). |
| + |
| +### 1.4 The other flat spawns |
| + |
| +Every tenant process the agent starts outside a unit goes through one of |
| +two seams, and both already carry one refusal, claim mode: |
| + |
| +- `spawnInCgroupGated` (`backend_exec.go:455-509`; refusal at `:456-458`) |
| +- `spawnInCgroupAtTier` (`internal/activator/containment.go:234-276`; |
| + refusal at `:235-237`), which delegates to the first when it places no |
| + child. |
| + |
| +Four call sites reach them with tenant code: the flat seed |
| +(`backend_exec.go:1055`), the flat instance (`:2241`), the legacy co-spawn |
| +sidecar (`:2988`) and the native worker (`:3125`). So on a box that is |
| +otherwise nested, these are flat: every native sidecar; every `foreign` |
| +instance row (an operator's `carlos add --socket/--db`); every |
| +activator-managed row the layout pass deferred or refused (awake within |
| +grace, a two-databases refusal, a missing database, `pass.go:286-353`); the |
| +seed of any of those; and every row on a box whose deployment chose |
| +`CARLOS_LAYOUT_DEFAULT=flat` or whose nested gate failed while no intent was |
| +recorded (`internal/host/layoutleg.go:127-146`). |
| + |
| +### 1.5 What a flat child reaches |
| + |
| +As uid `carlos` in the host namespace, with the edge's own view of the box: |
| + |
| +- every flat tenant's database, `<host>.env` (admin token), |
| + `<host>.config.env` and control socket in `/data` and `/run/carlos`; |
| + `registry.db`, `litestream.env`; on the console box, the console's |
| + database (the nested design's §1.2 list, unchanged for this tier); |
| +- **every nested tenant's directory, for writing**, through the raw tree |
| + `/data/.instances` (0700 `carlos`). The layout pass's own preflight says |
| + so (`internal/layout/preflight.go:181-190`), and the engines design draws |
| + the consequence (§6.2): a flat process can plant a library and an |
| + `LD_PRELOAD` in a nested sibling's env file and run inside that |
| + sibling's slice. **A flat tenant therefore voids the nested tier's |
| + guarantee for every nested tenant on the same box**, not only for flat |
| + ones; |
| +- the delegated cgroup subtree, which `carlos` owns: `cgroup.kill` and |
| + `cgroup.freeze` on the agent's own `supervise/` leaf |
| + (`internal/cgroup/cgroup.go:269-276`; the nested design's R9-1), and |
| + writes to its own `carlos-exec@<host>` limits; |
| +- the per-wake `CARLOS_SIDECAR_TOKEN` of every other flat worker, through |
| + `/proc/<pid>/environ` (same uid); |
| +- loopback listeners, the VPC and the instance metadata service, and |
| + through it the box role. |
| + |
| +### 1.6 The interim fence, and why it is weaker than it looks |
| + |
| +`carloku-infrastructure scripts/fence-flat-imds.sh` (not in this |
| +repository; described from the finding) drops metadata traffic from the |
| +edge's cgroup except from the agent's `supervise` leaf and one exempt |
| +operator app. Two limits, the second not in the finding: |
| + |
| +- It does nothing for the `/data` reads and writes or the cgroup writes |
| + above. |
| +- **If it keys on cgroup path, a flat child can step out of it.** Cgroup v2 |
| + lets a process move a pid between two cgroups when it can write |
| + `cgroup.procs` in both and in their common ancestor. `Delegate=` chowns |
| + the edge unit's cgroup and its `cgroup.procs` to `carlos`, and every |
| + directory below it is made by the agent as `carlos`. A flat child can |
| + therefore write its own pid into `supervise/cgroup.procs` (or the exempt |
| + app's cgroup) and open a socket there. The plan's first box task |
| + verifies this on a dev box (§9, box level) before anyone relies on the |
| + fence for anything. |
| + |
| +The repository already reads the box's metadata gate once: |
| +`nftGateUIDs` in the root adopt pass (`internal/host/adopt_advice.go:179-199`) |
| +parses `nft -n list ruleset` for the uid allow rule harden-host writes. |
| + |
| +### 1.7 What the existing `sidecar` role is for |
| + |
| +`instancectl.ParseRole` accepts `run`, `seed`, `sidecar`, `cmd` |
| +(`internal/instancectl/names.go:28-44`). The `sidecar` role is **the legacy |
| +co-spawn's unit**: a companion of a nested instance, keyed by the |
| +*instance's* host, launched as `[live/<host>, "sidecar", "run"]` with the |
| +single-layer `<host>-sidecar.env` (`backend_exec.go:3170-3227`; |
| +`internal/activator/prepare.go:94-101`), in the instance's slice (§4.3 of |
| +the nested design). The control service couples it to the instance's |
| +generations: a sidecar start naming a staged build leaves a populated slice's |
| +tier to the generation owner (`internal/instancectl/server.go:511-516`). No |
| +native sidecar has ever started through it. |
| + |
| +### 1.8 Where the problem statement and main disagree |
| + |
| +- Native sidecar rows do not classify `flat`: a sidecar row has `Socket ""`, |
| + which is neither the flat nor the nested socket, so `Classify` returns |
| + `foreign` for every one (`layout.go:103-110`). Nothing acts on that today, |
| + because `StartWorkerStaged` never asks, and status skips sidecar rows |
| + (`layoutleg.go:193-199`, ruling R15). |
| +- Claim mode does not "refuse flat spawns" specifically: with |
| + `HostWrites.Enabled` it refuses **every** current launch shape, nested |
| + ones included (`ErrClaimIsolationUnavailable`, |
| + `internal/activator/rolelaunch.go:15-17`; the nested arms at |
| + `containment.go:328-330`, `backend_exec.go:3171-3173`, `:1486-1488`; the |
| + worker at `internal/activator/workerbuild.go:111-113`). What this spec |
| + reuses is its refusal *point* and its error handling, not its scope. |
| +- A worker is never told where its database is. Its argv is the manifest's |
| + `args` alone and its environment carries no path |
| + (`backend_exec.go:3049-3130`; `workerEnv`, `internal/activator/activator.go:3321-3328`). |
| + A worker that opens a database must be computing `/data/<host>.db` |
| + itself, which stops working the moment its row is nested (§3.4). |
| +- The fence may be bypassable by cgroup migration (§1.6). |
| +- The flat tier is not only a cross-tenant read of flat siblings: it is a |
| + write into nested siblings (§1.5). |
| + |
| +--- |
| + |
| +## 2. Prior rulings: kept, and the ones this reopens |
| + |
| +Kept unchanged: the nested design's namespace block and its identity test |
| +across templates (§4.1, §4.2), the control service and its caller rule |
| +(§5.4), termination from unit state and an empty cgroup (§6.5), the gate |
| +(§8.5: no isolation, no nested tenants), the journal and its recovery |
| +rules (§3.3, §8.2), nested by default (§8.6, revision 17), and plan 4's |
| +R1 to R25 except as listed. From the sidecar design: standalone members, |
| +the synthetic host, the edge's refusal on Kind, each pass's removal |
| +authority scoped to its own Kind, and the legacy co-spawn left in place |
| +(§2, §6, §9.4, §10). |
| + |
| +Reopened, each because this problem forces it: |
| + |
| +- **Sidecar design §7.1, "one agent socket at `/run/carlos/agent.sock`".** |
| + A nested namespace has an empty `/run/carlos`, so a nested worker cannot |
| + see that path. Reopened for nested workers only (D3). |
| +- **Plan 4 R15 and S6, "skip `Kind sidecar` in the layout report; sidecars |
| + are outside the pass".** Both were right while sidecars could not be |
| + nested. Once they can, a nested sidecar row that status omits is a row |
| + the console's downgrade guard cannot see (`internal/console/layoutguard.go:59-64` |
| + judges `Layout.Rows` only), and a channel rollback would strand it. |
| + Reopened: worker rows are reported and moved (D8). |
| +- **Sidecar design §9.3, "metrics correct for free via `carlos-exec@<host>`".** |
| + Correct for flat workers only. A nested worker is sampled from its |
| + slice, and the sampler must map that slice back to a sidecar host (§3.12). |
| +- **The pass's `db.Required` rule** (`internal/layout/plan.go:271`, enforced |
| + at `internal/layout/pass.go:347-355`). It is code, not a spec ruling, but |
| + the refusal policy turns it from a cosmetic skip into an outage, so it |
| + is named here (D8). |
| + |
| +Not reopened, though it looks as if it might need to be: plan 4 R14's |
| +"defer an awake host". Plan 5 already bounded it with an awake grace |
| +(`DefaultAwakeGrace`, an hour, then one brief stop; `pass.go:126-157`, |
| +`:320-326`), and that is exactly what an always-on worker needs. |
| + |
| +--- |
| + |
| +## 3. Native sidecars on the nested tier |
| + |
| +### 3.1 Decision D1: a `worker` role and `carlos-instance-worker@.service` |
| + |
| +**Decision.** Add a fifth original role, `worker`, with its own template |
| +`carlos-instance-worker@.service`. The `sidecar` role stays the legacy |
| +co-spawn's, untouched. |
| + |
| +The template is the sidecar template with four lines changed and nothing |
| +else: `Description=` (`(worker)`), `ExecStart=/opt/carlos/carlos |
| +instance-exec worker %I`, `ExecStopPost=/opt/carlos/carlos instance-exit |
| +worker %I` and `StandardOutput=file:/run/carlos/logs/%I.worker.sock`. It |
| +binds the same executable sources as the other original templates |
| +(`/run/carlos-control/exe/%I/app` and `live`): a worker's host has no run |
| +or sidecar unit of its own, and its seed (which `Seed` already dispatches |
| +by layout, `backend_exec.go:905-914`) runs before it, never beside it, so |
| +the cmd role's reason for separate sources (merge ruling M4) does not |
| +arise. Same `Slice=carlos-instances-%i.slice`, same |
| +`Restart=no`, `KillMode=mixed`, `TimeoutStopSec=20`, `TasksMax`, |
| +`OOMScoreAdjust`, `OOMPolicy=kill`, same `WorkingDirectory`, and the |
| +namespace block byte for byte, held by |
| +`TestInstanceTemplatesShareOneNamespace` (`internal/host/units_isolation_test.go:231`) |
| +widened to the new file. |
| + |
| +Why not reuse `sidecar`, which would need no root change at all: |
| + |
| +- **The host is different.** A co-spawn sidecar's unit is keyed by its |
| + instance's host and shares that instance's slice; a worker *is* its |
| + host. The control service's sidecar branch is written for the first |
| + case (`server.go:511-516`), and retiring the co-spawn (sidecar design |
| + §10) must not take native workers with it. |
| +- **Downgrade detection.** Plan 4 R9 detects a pre-layout candidate by its |
| + unit set (`data-instances.mount` absent; `internal/host/systemapply.go:510`, |
| + `:1233`), because version strings are unreliable for dev builds. A new |
| + template gives D11 the same robust test. Reusing `sidecar` would leave |
| + only a version floor. |
| +- **Launch shape.** The trampoline can check, per role, what it is handed. |
| + A worker's argv is the manifest's `args`, not the co-spawn's fixed |
| + `sidecar run`. |
| + |
| +Root-side changes, all small: `names.go` gains `RoleWorker`, |
| +`UnitPrefixWorker = "carlos-instance-worker@"`, its `prefixFor`, |
| +`ParseRole` and `ParseUnitName` cases; the control service admits `worker` |
| +on `/start`, `/stop`, `/reset` and `/sweep` with the run role's tier rule |
| +(always set the slice's tier: a worker's slice is its own) and validates a |
| +named staged build with `validateStagedExe` as it does today |
| +(`internal/instancectl/stagedexe.go:45-78`, whose registry view is |
| +Kind-agnostic, `internal/host/claimstop_run.go:121-141`); |
| +`instance-exec` and `instance-exit` accept `worker` |
| +(`internal/instanceexec/exec.go:44-53`). No new verb, no new socket, no new |
| +root relay. The census implications are in §8.3. |
| + |
| +### 3.2 Decision D2: a worker row's paths, and `Classify` |
| + |
| +**Decision.** A nested worker row has `DBPath = NestedDBPath(dataDir, host)` |
| +and `Socket = ""`. A flat worker row is today's: `DefaultDBPath` and |
| +`Socket = ""`. `Classify` gains a branch, taken when `rt.IsSidecar()`: |
| + |
| +| Worker row | Layout | |
| +|---|---| |
| +| journal present | `fenced` (unchanged rule, first) | |
| +| identifier collides with another managed row | `conflict` | |
| +| `Socket == ""`, `DBPath == DefaultDBPath` | `flat` | |
| +| `Socket == ""`, `DBPath == NestedDBPath` | `nested` | |
| +| anything else (a socket on a sidecar row, any other path) | `foreign` | |
| + |
| +The conflict scan's peer predicate widens from `ActivatorManaged(p)` |
| +(`layout.go:97`) to `ActivatorManaged(p) || WorkerManaged(p)`, where |
| +`WorkerManaged(rt) = rt.IsSidecar() && rt.Backing == "" && rt.Tunnel == ""`. |
| +Sidecar hosts are long (`<name>.<app>.<sqid>.sidecar.internal`), so most |
| +hash to a bounded identifier (`internal/provision/provision.go:2973-2979`); |
| +a collision with an instance host is a 64-bit hash collision, and the |
| +existing rule (refuse both, name both) handles it. |
| + |
| +The `InstanceID` socket bound still holds: the longest path a worker |
| +needs is `/run/carlos/instances/<40>/agent.sock`, 72 bytes. |
| + |
| +### 3.3 Decision D3: the agent socket, per host |
| + |
| +A worker's only channel is the agent socket (`/v1/ready`, `/v1/next-due`, |
| +`/v1/idle`, `/v1/heartbeat`; `internal/agentsock/agentsock.go:98-106`), at |
| +`/run/carlos/agent.sock` mode 0660 (`internal/activator/activator.go:110`, |
| +`agentsock.go:75`), handed to the child as `CARLOS_AGENT_SOCKET` |
| +(`workerEnv`, `activator.go:3321-3328`; set at `:2469` and `:3623`). Inside |
| +the namespace `/run/carlos` is an empty tmpfs, so that path does not exist. |
| + |
| +**Decision.** For a nested worker, the agent binds a second listener for |
| +that host alone at `<InstanceRunDir(host)>/agent.sock`, mode 0600, in |
| +`prepare` for the worker role, before the start request, and closes it |
| +after the proc's `done`, in the order `prepare` already uses for the |
| +control listener (nested design §6.3). The run directory is bound into the |
| +namespace at the same path, so `CARLOS_AGENT_SOCKET` names the same string |
| +inside and out. The listener is served by the same `agentsock.Server`, |
| +wrapped so that the host it answers for is fixed: a request whose |
| +`Carlos-Sidecar-Host` is not the listener's host gets the same uniform 401 |
| +as every other refusal, before the token comparison. The token check stays |
| +the boundary; the pinning means a token leaked from one worker is useless |
| +on another's listener. |
| + |
| +Why not bind `/run/carlos/agent.sock` into the namespace: its source is in |
| +`/run/carlos` itself, which is not on a `nosymfollow` mount, so during the |
| +transition a flat tenant could replace it with a symlink and PID 1 would |
| +bind whatever it names (the nested design's §3.2 argument, applied to a |
| +bind source). The run directory is on the `nosymfollow` mount, and the |
| +agent creates the socket there itself. |
| + |
| +The run directory is `2750 carlos:carlos-sockets` (§3.1 of the nested |
| +design), so an enrolled cross-user consumer can traverse it; 0600 on the |
| +socket keeps it out. The tenant owns its run directory inside the namespace |
| +and can loosen the socket's mode or replace the file; either only breaks |
| +its own channel, because the agent listens and never dials. |
| + |
| +Flat workers keep `/run/carlos/agent.sock` for as long as any run (§4). |
| + |
| +### 3.4 Decision D4: tell every worker where its database is |
| + |
| +**Decision.** `workerEnv` gains `CARLOS_SIDECAR_DB=<rt.DBPath>`, agent-owned |
| +(applied over the tenant's env files, like the token), for flat and nested |
| +workers alike, in the first release that carries this spec, before any |
| +worker row is ever nested. The sidecar page of `docs/site/cli.md` |
| +(`:970-1020`) says a worker opens its database at that path and nowhere |
| +else (copy to review). |
| + |
| +A worker that computes `/data/<host>.db` itself today will fail when its |
| +row is nested: `/data` is a read-only tmpfs in the namespace, so the open |
| +fails and the worker crashes. Which deployed apps do that is a fact the |
| +rollout's check step establishes before any flagship box turns nested |
| +workers on (§8.2, §11 Q3). |
| + |
| +### 3.5 Decision D5: `StartWorkerStaged` dispatches on layout |
| + |
| +**Decision.** `StartWorkerStaged` keeps its first half (argv from `rt.Args`, |
| +build resolution or `VerifyStaged`, `LiveExecGate`, env layers, the replica |
| +key gate, `runBeforeExec`; `backend_exec.go:3050-3118`) and then |
| +dispatches on `b.pathsFor(rt).layout`: |
| + |
| +- `nested`: the claim refusal (as `startSidecarUnitFrom` has it, |
| + `:3171-3173`), `nestedGate` (`:2678-2688`), `prepare(rt, RoleWorker)` |
| + (the directories, the env layers read through `instancefs`, no admin |
| + token, the log listener, the per-host agent listener of D3), a |
| + `launch.worker.json` with argv `[liveLink(host), args...]` and the |
| + composed env, then the control service's start for the worker role |
| + **always naming the staged build** the first half resolved (`StagedExe` |
| + from `id`). The worker contract is that it runs a build by path, never |
| + the live link (the zero-downtime spec §7.9, cited at `:3027-3033`); root |
| + binds exactly the file the agent hashed. The ctx's exec gate |
| + (`launchGate`) is asked immediately before the start request, as the |
| + nested instance start asks it (the `launchHook` contract, `backend_exec.go:299-306`). |
| + Returns a `unitProc` for the worker role. |
| +- `fenced`, `conflict`: refuse, naming the reason, as `StartSidecar` does. |
| +- `flat`, `foreign`: today's body, which now meets the flat-exec policy |
| + of §4 in `spawnInCgroupGated`. |
| + |
| +The activator side changes in one place: `workerEnv`'s socket argument |
| +comes from the backend for the row (the per-host path for a nested row, |
| +`a.agentSocket()` otherwise), and `w.launchLayout` (`activator.go:2436-2438`) |
| +already records the layout the launch used, so a failed wake's cleanup |
| +looks for descendants in the right place. |
| + |
| +`TestKillHostProvesANestedLayoutWorkersHostCgroup` |
| +(`worker_gen_test.go:50-63`) is rewritten: a nested worker's cgroup is its |
| +unit's, and `KillHost` proves the slice, as for every other nested role. |
| + |
| +### 3.6 Decision D6: crash versus finish under a unit |
| + |
| +The activator tells a hibernating worker's crash (restart with backoff) |
| +from its finish (park) by `procCrashed`, which asks the proc's optional |
| +`Crashed()` and defaults to "not a crash" (`activator.go:404-418`). |
| +`childProc` answers from `ProcessState` (`backend_exec.go:3329-3335`). |
| +`unitProc` has no `Crashed` (`internal/activator/unitproc.go:87-203`), so |
| +every nested worker crash would read as a finish and park. |
| + |
| +**Decision.** `unitProc.Crashed()` is true only after the observer's two |
| +termination facts hold (nested design §6.5) **and** systemd reported the |
| +proc's own invocation as `failed` with `Result` other than `success`. |
| +The advisory exit record never decides it (§3.3 of the nested design: the |
| +record is the tenant's claim). The template's `SuccessExitStatus=129 130 141 |
| +143` (the trampoline's signal convention, |
| +`carlos-instance-sidecar@.service:41-46`) means a stop by SIGTERM is a |
| +finish, as for a flat child stopped the same way. An `inactive` unit whose |
| +invocation was garbage-collected reads as "not a crash", the safe default |
| +`procCrashed` already documents; a failed unit is not collected until it |
| +is reset or started again, so a real crash keeps its `Result`. |
| + |
| +### 3.7 Decision D7: minting |
| + |
| +**Decision.** `sidecarRouteFor` takes the same `nested` answer the instance |
| +mint takes (`mintLayoutReader`, `internal/provision/reconcile.go:471-490`; |
| +`RouteFor`'s `nested` arm, `:456-462`) and, when it is true and |
| +`CARLOS_NESTED_WORKERS` is on (§8.1), mints `DBPath = NestedDBPath`. Every |
| +other answer mints flat, as today. `convergeSidecar`'s shape check |
| +(`sidecar_reconcile.go:621-626`) accepts either the flat or the nested path |
| +for the row's host, and reports anything else as today. |
| + |
| +The mint-time "one host, one mechanism" check (`coSpawnStaged`, |
| +`sidecar_reconcile.go:156-159`) looks in the flat data directory; it keeps |
| +looking there, because a co-spawn file is attached to an *instance* host, |
| +and an instance host can never end in `.sidecar.internal`. |
| + |
| +### 3.8 Decision D8: moving existing sidecar rows |
| + |
| +**Decision.** The layout pass's eligibility (`pass.go:264`, `:529`, `:747`) |
| +widens from `ActivatorManaged(rt)` to `ActivatorManaged(rt) || |
| +WorkerManaged(rt)`, behind `CARLOS_NESTED_WORKERS` (§8.1). Everything else |
| +about the move is plan 4's: |
| + |
| +- **Quiesce.** `FenceAndStop` on the worker's entry. A hibernating worker |
| + that is asleep moves at once. An awake one, including every always-on |
| + worker, is deferred for the awake grace and then stopped once |
| + (`pass.go:320-326`). While fenced, every wake source refuses: the timer |
| + wake, `carlos sidecars wake`, and the boot pass and sweep that start an |
| + always-on worker with no live entry (sidecar design §7.2). The plan pins |
| + each of the three with a test, because the last two do not come in |
| + through a request. |
| +- **Plan.** `PlanFor` derives both endpoints with `Socket ""` for a worker |
| + host, and the worker test is the host's suffix |
| + (`provision.SidecarSuffix`), never the row's `Kind` alone: recovery |
| + recomputes endpoints from the host name (§3.3 of the nested design), and |
| + a same-uid flat tenant can edit the row. The row's `Kind` must agree with |
| + the suffix, or the journal is refused like any mismatch. The nine |
| + artifacts keep their order (`plan.go:98-101`); `socket` is recorded |
| + absent on both sides. The journal format does not change. |
| +- **A row with nothing on this box moves by row alone.** Today the pass |
| + refuses a host whose database is missing (`pass.go:347-355`), so a row |
| + minted flat on a box that never woke it stays flat forever. Under the |
| + refusal policy that becomes a box that refuses the row on its first wake |
| + here, which is exactly a failover. **Decision:** when every one of the |
| + nine artifacts is absent at both endpoints, the pass journals the |
| + absence and does the row CAS only. The next wake restores from the |
| + bucket into the nested path. This changes `Required` for the database |
| + from "must exist" to "must exist unless nothing exists", for instance |
| + rows too. |
| +- **Config.** `ConfigEnvPath` decides nested by both columns |
| + (`internal/appconfig/syncexec.go:66-79`), so a nested worker's |
| + `.config.env` would keep landing flat. It learns the worker shape (D2's |
| + table), and SyncExec's fenced skip covers worker hosts. |
| +- **Status.** `layoutStatusReport` reports worker rows (`layoutleg.go:193-199` |
| + widens), so `carlos status --layout`, the console's downgrade guard and |
| + the migrate guard all see them. A row's word gains nothing new: a worker |
| + is `flat`, `nested`, `fenced`, `conflict` or `foreign` like any other |
| + row, and the report's per-host key already admits any host |
| + (`internal/status/snapshot.go:230-254`). |
| + |
| +### 3.9 Decision D9: an always-on worker's replica lineage |
| + |
| +Hibernating workers replicate through the lease (`.activator-<host>.db-litestream`, |
| +one of the nine artifacts), keyed by host in the bucket, so their move is |
| +path-blind. An always-on worker is not hibernating, so the box-wide |
| +replicator claims its database (`internal/ops/litestream.go:356-357`, |
| +`:393-398`), and that replicator keys a replica by the database's full path |
| +(`replicaKey`, `:139-162`). Moving the file therefore moves the replica. |
| +This is new: no always-on row has ever been moved by the pass, because |
| +`ActivatorManaged` requires `Hibernate`. |
| + |
| +**Decision.** Accept a new lineage at the nested path. The pass, for an |
| +always-on worker: fences (the generator then drops the host, |
| +`FleetDBsWithSkipped`'s `fenced`), waits until the regenerated config no |
| +longer names the flat path and the box-wide replicator has reloaded, |
| +moves, and releases; the generator then names the nested path. The old |
| +lineage stays in the bucket under its old key (retention as configured). |
| +`CoverageShortfall`'s shrink guard is taught that a journaled host's flat |
| +path leaving the config is expected. The plan must confirm on a dev box |
| +that litestream releases the file on reload before the rename; if it does |
| +not, the pass stops the replicator for the rename, which is a box-wide |
| +pause of seconds, and says so in the event line. |
| + |
| +### 3.10 Decision D10: while a box's nested gate fails |
| + |
| +The gate (§8.5 of the nested design) refuses every nested wake while the |
| +mounts, the probe or the control service are not ready, and |
| +`defaultLayoutIntent` falls back to flat on such a box when no intent is |
| +recorded (`layoutleg.go:127-146`). For workers: |
| + |
| +- a nested worker row refuses to start, with the gate's failing line as |
| + the reason, exactly as a nested instance does; |
| +- a flat worker row is not migrated (the pass is gated, plan 4 R3); |
| +- a new worker row on a box with no recorded intent is minted flat |
| + (D7 follows the instance mint); |
| +- **Decision:** none of this reopens the flat path. Under `report` (§4) a |
| + flat row runs flat, as it does today. Under `refuse`, it is refused. A |
| + box that cannot isolate runs nothing it cannot isolate, which is the |
| + nested design's ruling ("a box that cannot isolate does nothing, |
| + loudly"). An operator who needs a specific host up through a gate |
| + failure adds it to the allowlist, knowingly (§4.1). |
| + |
| +### 3.11 Decision D11: downgrade |
| + |
| +A binary from before this change runs a nested worker row through the old |
| +`StartWorkerStaged`: flat, with the env read from the flat directory and |
| +no database path, and the row's own nested `DBPath` replicated by the old |
| +lease code. Two copies, and unisolated. |
| + |
| +**Decision.** |
| + |
| +- Box: `systemApplier.install` refuses a candidate whose unit set lacks |
| + `carlos-instance-worker@.service` while any row whose host carries the |
| + sidecar suffix has `NestedPaths` (`provision.go:3020-3023`), beside |
| + R9's guard (`systemapply.go:1273-1290`). The reason names the rows and |
| + says to revert the layout first (copy to review). |
| +- Console: `layoutDowngradeBlocked` (`layoutguard.go:24-69`) gains a |
| + second floor, `release.WorkerLayoutFloor`, judged on the same observed |
| + rows. With D8 reporting worker rows, a nested worker row blocks a |
| + rollback below that floor the way a nested instance row blocks one |
| + below `LayoutFloor` (`internal/release/layoutintent.go:211`). |
| +- `carlos system layout revert` reverts worker rows with the rest (they |
| + are in the pass). |
| + |
| +### 3.12 Every consumer of the shape |
| + |
| +The rule this repo has paid for three times: the consumer list is |
| +everything that parses the shape, and for sidecars, everything whose |
| +predicate happens to require a socket (sidecar design §15.2). Known on |
| +main; the plan's first task re-runs the sweep on the day it starts: |
| + |
| +- `activator/layout.go:63-65`, `:97`, `:103-110` (D2) |
| +- `activator/paths.go:33-54` (follows D2 unchanged) |
| +- `activator/backend_exec.go:3049-3155` (D5), `:2914-2928` (unchanged) |
| +- `activator/prepare.go:36-110` (the worker role: env layers, no token, |
| + agent listener) |
| +- `activator/unitproc.go` (D6) |
| +- `activator/activator.go:2469`, `:3623` (the socket per row, D5) |
| +- `agentsock` (the host-pinned listener, D3) |
| +- `appconfig/syncexec.go:66-79` (D8) |
| +- `layout/plan.go:244-276`, `layout/pass.go:264`, `:529`, `:747`, |
| + `:347-355` (D8) |
| +- `provision/sidecar_reconcile.go:131-148`, `:621-626` (D7) |
| +- `host/layoutleg.go:193-199` (D8) |
| +- `host/systemapply.go:1273-1290` (D11) |
| +- `console/layoutguard.go` (D11) |
| +- `ops/litestream.go` (D9) |
| +- the sampler's slice-to-host mapping (`internal/host/metrics_probes.go`, |
| + the nested design §6.6) and `carlos status --instances`' unit matcher |
| + (sidecar design §14.4), which must recognise `carlos-instance-worker@` |
| +- `internal/hostbox/inventoryscope.go:48` accepts either path already |
| +- the claim launch shapes (`internal/host/claim*`, `internal/hostbox`): |
| + out of scope while claims are dormant; claims-on keeps refusing workers |
| + (§10) |
| +- engines `arm` (engines design §6.2): "no route spawns workers" becomes |
| + "no worker row classifies flat or foreign" |
| + |
| +--- |
| + |
| +## 4. The flat-exec policy |
| + |
| +### 4.1 Decision D12: the setting |
| + |
| +Two lines in `host.env`, read the way `CARLOS_LAYOUT_DEFAULT` is |
| +(`layoutleg.go:68-84`: `os.Getenv`, trimmed, lower-cased, an unknown value |
| +logged and taken as the safe reading): |
| + |
| + CARLOS_FLAT_EXEC=refuse # or report |
| + CARLOS_FLAT_EXEC_ALLOW=shop.example.com,tools.example.com |
| + |
| +- `refuse`: a flat or foreign spawn of a host not on the allowlist is |
| + refused (§4.2). |
| +- `report`: every spawn proceeds as today, and each flat or foreign spawn |
| + of a host not on the allowlist is recorded as a would-be refusal (§4.3). |
| +- unset: `report` in the release that carries this spec; `refuse` from the |
| + next (D15). With `CARLOS_LAYOUT_DEFAULT=flat` (a deployment that has |
| + chosen not to isolate), unset stays `report` in every release, because |
| + that deployment's every row is flat by its operator's choice. |
| +- an unknown value: `refuse`, logged. An operator who typed the variable |
| + meant to choose, and the reading that cannot expose a tenant is the one |
| + to take. (This is the opposite of `CARLOS_LAYOUT_DEFAULT`'s unknown-value |
| + rule, which falls back to flat, because there the safe reading is the |
| + one that cannot strand a database.) |
| +- The allowlist names **exact hosts**, comma-separated, each checked with |
| + `provision.ValidHostShape` (`provision.go:386`). No wildcards, no |
| + suffixes: one entry reopening `*.sidecar.internal` would undo the whole |
| + change. An invalid entry is dropped and reported. |
| + |
| +**An allowlisted host is trusted as the agent.** It runs as `carlos` in the |
| +host namespace and can read and write everything §1.5 lists, nested |
| +tenants' directories included. The allowlist exists for an operator's own |
| +workload that genuinely cannot run nested, and `carlos status` says, for |
| +every allowlisted host, that it is unisolated (copy to review). |
| + |
| +The values are read once at agent start, like the layout default. Changing |
| +them takes an agent restart, which is the operator act that also makes the |
| +change visible in the journal. |
| + |
| +### 4.2 Decision D13: the refusal point |
| + |
| +**Decision.** Both seams gain one check immediately after the claim |
| +refusal: |
| + |
| +```go |
| +// spawnInCgroupGated, backend_exec.go:456-458, and spawnInCgroupAtTier, |
| +// containment.go:235-237, each: |
| +if b.Cfg.HostWrites.Enabled { |
| + return nil, nil, ErrClaimIsolationUnavailable |
| +} |
| +if err := b.FlatExec.Admit(rt); err != nil { // nil Admit: today's behaviour |
| + return nil, nil, err |
| +} |
| +``` |
| + |
| +`ErrFlatExecRefused` wraps `hostbox.ErrUnsupported`, as the claim error |
| +does (`rolelaunch.go:17`), and joins it in the two places that already |
| +classify the claim error: the swap gate's no-ownership list |
| +(`internal/activator/swap.go:225-245`), so a refused successor keeps the |
| +old generation serving rather than half-switching, and the worker |
| +respawn's veto (`activator.go:3474-3479`), so a refused respawn is a |
| +recorded veto, not a crash loop. Because every tenant spawn outside a unit |
| +goes through one of the two seams (§1.4), one check covers the instance, |
| +the seed, the co-spawn sidecar and the worker. A test enumerates every |
| +`exec.Command` in `internal/activator` and fails on one that does not reach |
| +a seam (four today: `backend_exec.go:1056`, `:2242`, `:2989`, `:3126`). |
| + |
| +The check runs before `Cgroups` is touched, so it also covers a box with |
| +no delegated subtree, where `StartGated` falls back to a plain start in the |
| +agent's own cgroup (`cgroup.go:392-395`). |
| + |
| +The platform's own children are not tenant code and do not pass through |
| +these seams; nothing here changes them. |
| + |
| +### 4.3 Decision D14: what a refusal looks like |
| + |
| +- **Box status.** `Box.Metrics.FlatExec` (additive, private like the rest |
| + of `Metrics`): the policy in force, the allowlist, and per host the |
| + last refusal or would-be refusal with a closed reason word |
| + (`flat_refused`, `foreign_refused`, `flat_reported`, `foreign_reported`), |
| + first and last time and a count, the shape `ConfigRefusal` already has |
| + (`snapshot.go:256-274`). Error text never goes in it. |
| +- **Wake failure.** An instance wake refused this way answers the visitor |
| + as any refused wake does (the member-safe reason path); a worker's state |
| + word in `carlos sidecars ls` is a new one (copy to review: "Held: this |
| + box will not run it unisolated"). |
| +- **Event line.** One per host per hour, as the replica key gate logs |
| + (`internal/activator/replicakey.go:78-90`), naming the host, the layout, |
| + and what to do (copy to review). |
| +- **`carlos status --layout`** shows the policy and each refused or |
| + allowlisted host beside its layout word. |
| + |
| +### 4.4 Decision D15: sequencing, so no box loses a workload on upgrade |
| + |
| +1. **Release N** carries §3, §4 and §5, with `CARLOS_FLAT_EXEC` defaulting |
| + to `report` and `CARLOS_NESTED_WORKERS` off. Rolling it changes nothing |
| + that runs; status starts showing every flat spawn a `refuse` box would |
| + stop. |
| +2. **Per box**, after `CARLOS_NESTED_WORKERS=on` has moved its sidecars |
| + and its status shows no would-be refusal for a stretch that covers the |
| + box's slowest hibernating workload's wake interval, an operator sets |
| + `CARLOS_FLAT_EXEC=refuse` (with an allowlist if one is genuinely |
| + needed). Dev boxes first (§8.2). |
| +3. **Release N+1** makes `refuse` the unset default (and turns |
| + `CARLOS_NESTED_WORKERS` on by default, §8.1). The console's system |
| + channel door refuses to move a channel to N+1 while any box that will |
| + follow it has the variable unset and reports a would-be refusal in its |
| + last fresh snapshot, or is stale, naming the boxes and hosts (copy to |
| + review). That is `layoutDowngradeBlocked`'s pattern (judge observed |
| + state; unknown blocks), applied to an upgrade. |
| +4. A box that needs the old behaviour after N+1 says so explicitly: |
| + `CARLOS_FLAT_EXEC=report`. |
| + |
| +Claim-mode boxes are unaffected throughout: they already refuse every |
| +legacy launch shape before this check is reached. |
| + |
| +### 4.5 Decision D16: the stragglers |
| + |
| +A managed row can still be flat on a nested box for a moment: a row the |
| +pass deferred while awake that has since gone to sleep, before the next |
| +serial move runs the pass (plan 4 R14 runs it on serial moves). Under |
| +`refuse` its next wake is refused. |
| + |
| +**Decision.** A refusal of an `ActivatorManaged` or `WorkerManaged` row on |
| +a box whose effective intent is nested marks the layout leg dirty for that |
| +host (R14's single-flight and dirty flag), so the pass runs within seconds, |
| +finds the host asleep, and moves it. The refused request answers as a |
| +retryable wake failure; the next one wakes nested. A foreign row is not |
| +nudged: the pass does not move foreign rows (nested design §8.3), and its |
| +repair stays the box-local `carlos add`. |
| + |
| +--- |
| + |
| +## 5. Decision D17: creation-time refusal |
| + |
| +**Decision.** `handleAPISidecarCreate` refuses before any write when the |
| +app's sidecars could not run isolated where they would be placed. The |
| +check runs after the region is resolved (`apisidecars.go:268-278`) and |
| +before the plan cap and the claim branch (`:315`, `:353`), so it covers |
| +the claim-mode create too: |
| + |
| +- The console reads every box's snapshot the way `carlos system status` |
| + does (`systemBoxes`, as `layoutDowngradeBlocked` uses it), keeps the |
| + boxes in the resolved region (or, for a fleet-placed app, the fleet's |
| + boxes), and drops stale ones. |
| +- Each box reports a new additive field, `Metrics.Layout.Workers`: |
| + `nested` when its binary carries this spec, `CARLOS_NESTED_WORKERS` is |
| + on, its effective intent is nested and its nested gate answers; `off` or |
| + `blocked: <reason>` otherwise. The reason is private (it may name a |
| + path), like the layout report's reasons. |
| +- The create is refused when no fresh box is in scope, or when any fresh |
| + box in scope reports anything but `nested`. Stale boxes neither pass |
| + nor block: the box-side refusal (§4) is the boundary, and this door is |
| + the early, honest answer in front of it. |
| +- The refusal is a 409 with `code: "isolation"`, so the CLI and an app's |
| + own code can tell it from a cap or a duplicate. The text names the |
| + region and says that sidecars there cannot run isolated yet (copy to |
| + review). Members never see box labels or reasons. |
| +- Off by default in release N behind `CONSOLE_REQUIRE_ISOLATED_WORKERS`; |
| + on by default from N+1. |
| + |
| +Records that already exist are not touched: their rows move with D8 or are |
| +refused by §4. `carlos sidecars enable` is not gated: it changes policy, |
| +not what runs. |
| + |
| +Why per region and not per box: a sidecar record is minted on every box |
| +in its region and pool (sidecar design §8, `ReconcileSidecars`), and |
| +whichever box takes its lease runs it. A record is only as isolated as the |
| +weakest box that may run it. |
| + |
| +--- |
| + |
| +## 6. Decision D18: retiring the interim fence |
| + |
| +**Decision.** `fence-flat-imds.sh` may be removed from a box when all four |
| +hold on that box, and each is observable through the API (`carlos status`), |
| +with no box access: |
| + |
| +1. `CARLOS_FLAT_EXEC` in force is `refuse` (explicitly or by default); |
| +2. the allowlist is empty; |
| +3. no flat child is running: the agent reports the number of populated |
| + `carlos-exec@*` cgroups under its delegated subtree (it already walks |
| + them, `listExecCgroups`, `internal/host/metrics_probes.go:253`), and it |
| + is zero; |
| +4. the nested probe's metadata line is enforced (`v4: enforced`; nested |
| + design §8.5). |
| + |
| +With an allowlisted host on the box, the fence stays, and is no longer |
| +"interim": it is the deployment's documented control for a host the |
| +operator has chosen to trust, and it belongs in the AWS hardening script |
| +beside the uid rule (§7.4), not in an out-of-band one. Given §1.6, it is a |
| +control against accident, not against that host: an allowlisted host is |
| +already trusted as the agent (§4.1). |
| + |
| +Removal is a carloku-infrastructure change, applied with the scoped |
| +approval production infrastructure needs (AGENTS.md, 2026-09-15), dev |
| +boxes first. |
| + |
| +--- |
| + |
| +## 7. Threat model |
| + |
| +### 7.1 Attacker and assets |
| + |
| +**Attacker:** a member of any account, or code holding an app's instance |
| +credential, who can enable and create sidecars and ship the release they |
| +run. They control the worker's code entirely. They do not hold box access, |
| +an operator role, or another account's credentials. |
| + |
| +**Assets:** other tenants' data, tokens and configuration on the box; the |
| +agent's integrity (its cgroup, its socket, the files it parses); the box's |
| +cloud credentials and everything they reach (the deployment bucket, |
| +regional secrets, on the console box the ability to mint IAM users); other |
| +boxes and private services on the network. |
| + |
| +### 7.2 Step by step |
| + |
| +| After | The attacker can | The attacker cannot | |
| +|---|---|---| |
| +| **Today** | everything in §1.5: read flat siblings; write nested siblings' raw directories and so run code inside them; kill or freeze the agent through cgroupfs; steal other flat workers' tokens; reach loopback, the VPC and metadata, and through it the box role | nothing the edge itself cannot do, short of root | |
| +| **The fence** (today, interim) | all of the above except metadata from its own cgroup; and very likely metadata too, by moving its pid into the exempt leaf (§1.6) | reach metadata from `carlos-exec@<host>` while it stays there | |
| +| **Release N rolled** (§3 dark, `report`) | the same: nothing that runs has changed | (status now names every flat spawn) | |
| +| **`CARLOS_NESTED_WORKERS=on`** on a box whose gate is ready | on that box, nothing from a new or moved sidecar: it sees its own directory only, no loopback, no private range, no metadata, no sibling pid. It can still get a flat process **if** the box mints flat (gate failing with no recorded intent, D10) or the row is foreign | read or write another tenant's files from its sidecar; call the agent socket for another host (D3) | |
| +| **`CARLOS_FLAT_EXEC=refuse`**, empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace | |
| +| **Console refusal on** | (no change in power) | create a record the platform would place unisolated or never run; this is honesty, the boundary is the line above | |
| +| **Fence retired** (§6) | nothing new: there is no flat child left for the fence to filter | | |
| + |
| +### 7.3 What remains after the last step |
| + |
| +- **The nested tier's own residuals,** unchanged and already stated in the |
| + nested design: every tenant is still uid `carlos` and the namespace is |
| + the boundary (§11, option B); the agent reads tenant-writable files and |
| + is protected by `nosymfollow` and `instancefs` (§3.2, §3.3); an allowed |
| + upstream resolver is open on every port; a public interface address of |
| + the box is reachable as from the internet; a sibling can squat an |
| + abstract socket name (a refusal of service, not a connection); disk |
| + exhaustion past the floor (§11.1). |
| +- **Loopback.** Closed for every nested role, workers included, by the |
| + template's `IPAddressDeny=localhost` (`carlos-instance-sidecar@.service:142`). |
| + A worker that relied on reaching something on 127.0.0.1 stops working |
| + when it moves; the rollout check looks for it (§8.2). |
| +- **Unit-backed tenants and the doc engine.** `carlos-app@` units are |
| + outside this spec. The doc-engine grant deliberately leaves loopback open |
| + for the engine's own browser and office processes (README, the |
| + doc-engine preconditions), which is why its box must be dedicated (engines |
| + design §5.9). Nothing here changes that, and nothing here relies on it. |
| +- **Allowlisted hosts.** Trusted as the agent (§4.1). The policy makes |
| + that trust explicit and visible; it does not reduce it. |
| +- **The past.** A box that ran flat tenants before step 5 may already hold |
| + things a flat tenant planted in nested siblings' directories. The pass |
| + quarantines unexpected entries in a directory it is about to fill |
| + (`preflight.go:181-190`), but a directory filled earlier is only as |
| + trustworthy as that period (the engines design's §6.2 residual), and |
| + the credentials the flat tier could reach should be treated as exposed |
| + (§11 Q2). |
| +- **Externally supervised routes** (unflagged instance rows with a socket): |
| + the agent never spawns them; untouched, as the nested design §8.2 step 1 |
| + rules. |
| +- **Claim mode,** when it is activated, refuses workers until the root |
| + role launcher learns the worker role (§10). |
| + |
| +### 7.4 Provider boundary |
| + |
| +The core property is provider-neutral: **no tenant process runs in the |
| +host namespace, and no tenant process can reach the box's credential source |
| +or another tenant's state.** Everything in §3, §4 and §5 is core: roles, |
| +templates, the policy, the console check. The namespace's egress filter |
| +already denies every link-local and private range, which covers the |
| +metadata endpoints of the major clouds without naming them. |
| + |
| +The AWS adapter holds the rest: the two metadata addresses named in the |
| +template for greppability (`169.254.169.254/32`, `fd00:ec2::254/128`), the |
| +probe's metadata lines, the box role and what it can reach, harden-host's |
| +uid rule (read back by `adopt_advice.go:179-199`), and the interim fence. |
| +D18's fourth condition is that adapter's statement of "the credential |
| +source is unreachable"; another provider's adapter supplies its own line. |
| + |
| +--- |
| + |
| +## 8. Rollout |
| + |
| +### 8.1 Flags and defaults |
| + |
| +| Setting | Where | Release N | Release N+1 | Gone in | |
| +|---|---|---|---|---| |
| +| `CARLOS_NESTED_WORKERS` (`on`\|`off`) | box `host.env` | off: workers mint flat, the pass skips them, dispatch still honours any nested row | on unless `off` | N+2 | |
| +| `CARLOS_FLAT_EXEC` (`refuse`\|`report`) | box `host.env` | `report` | `refuse` unless set, or unless `CARLOS_LAYOUT_DEFAULT=flat` | stays (it is the self-hoster's opt-out) | |
| +| `CARLOS_FLAT_EXEC_ALLOW` | box `host.env` | empty | empty | stays | |
| +| `CONSOLE_REQUIRE_ISOLATED_WORKERS` | console env | off | on unless `false` | N+2 | |
| + |
| +Dark in N: every behaviour change. The dispatch in `StartWorkerStaged` is |
| +live in N, but no nested worker row can exist until a box turns |
| +`CARLOS_NESTED_WORKERS` on, so it is inert. |
| + |
| +### 8.2 Order |
| + |
| +The house order: dev boxes, then the dev console, then a check, then |
| +flagship boxes, then the flagship console. |
| + |
| +1. **Release N to dev boxes, then the dev console.** Nothing changes. |
| + `carlos status --layout` lists sidecar rows as `flat` and the flat-exec |
| + section lists each as `flat_reported`. |
| +2. **Dev boxes: `CARLOS_NESTED_WORKERS=on`.** Watch sidecar rows turn |
| + `nested`, hibernating ones at once, always-on ones within the awake |
| + grace. The check is from inside: a throwaway sidecar in a dev account |
| + whose release lists `/data` and `/run/carlos`, reads `/proc/1/status`, |
| + sends the probe's datagrams, opens `CARLOS_SIDECAR_DB`, calls |
| + `/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, and prints |
| + all of it to `carlos sidecars logs`; then sleeps, wakes on its timer, |
| + wakes on `carlos sidecars wake`, crashes once (restarted with backoff), |
| + and finishes once (parked). An always-on twin is moved by the grace |
| + stop, and its replica appears under the nested key. |
| +3. **Dev boxes: `CARLOS_FLAT_EXEC=refuse`.** A foreign row made on a dev |
| + box with `carlos add --db` is refused, status says so, and `carlos add` |
| + with the defaults repairs it. Verify §1.6's cgroup migration on a dev |
| + box both before and after: before, a flat child can move itself into |
| + `supervise`; after, there is no flat child to try. |
| +4. **Dev console: `CONSOLE_REQUIRE_ISOLATED_WORKERS=true`.** A create in a |
| + dev region whose boxes report `nested` succeeds; with one box's |
| + `CARLOS_NESTED_WORKERS` off, it is refused with `code: "isolation"`. |
| +5. **The check before flagship:** establish, for every app with native |
| + sidecars on the flagship, whether its worker opens its database by |
| + computed path, uses the agent socket by a hard-coded path, or reaches |
| + anything on loopback or a private address (§3.4, §7.3). Each one that |
| + does ships a release that reads `CARLOS_SIDECAR_DB` and |
| + `CARLOS_AGENT_SOCKET` before its boxes turn nested workers on (§11 Q3). |
| +6. **Flagship boxes,** region by region, console box last: steps 2 and 3. |
| + Then the **flagship console**: step 4. |
| +7. **Fence retirement** per box when §6's four conditions hold, dev first, |
| + under its own scoped approval. |
| +8. **Release N+1** cut and rolled in the same order, behind the console |
| + guard of D15. |
| + |
| +### 8.3 Census |
| + |
| +Root-run code this touches, each of which changes a digest the root-read |
| +census pins (`internal/provision/plan14_root_reads_test.go`) and so needs |
| +a census proposal through the #280 process in the same branch: |
| + |
| +- `internal/instancectl/names.go`, `server.go` (the worker role on the |
| + existing verbs; no new verb); |
| +- `internal/host/units/carlos-instance-worker@.service` (a new embedded |
| + unit; templates are never enabled, so `update-platform-binary.sh` needs |
| + no new name); |
| +- `internal/host/systemapply.go` (D11's guard); |
| +- the probe, only if the plan gives it a worker-template line (it should |
| + not need one: the namespace block is identical, and the identity test |
| + holds that). |
| + |
| +No new root pass, path unit or relay. `instance-exec` and `instance-exit` |
| +run as `carlos` inside the namespace. |
| + |
| +### 8.4 Docs and copy |
| + |
| +`docs/site/cli.md` (the sidecar section, `carlos status --layout`, the |
| +three box settings) and the README's box-settings list. Every new string |
| +in this spec marked "copy to review" goes through copy review before it is |
| +written into a file: the refusal texts of §3.11, §4.3, §4.4 and §5, the |
| +worker state word, the allowlist warning, and the docs paragraphs. Error |
| +fragments and JSON keys are not copy. |
| + |
| +--- |
| + |
| +## 9. Tests |
| + |
| +Through production wiring wherever the repo already does it: a real |
| +`activator.Service` over a real `ExecBackend`, a real `instancectl.Server` |
| +on a real unix socket with its `Systemctl` and peer seams faked (the shape |
| +the `internal/host` launch tests use), the real embedded unit files, and a |
| +real `layout.Pass` against a real registry. |
| + |
| +Unit and package level: |
| + |
| +- `Classify`: the worker table of D2, including a sidecar row carrying a |
| + socket (`foreign`) and a worker colliding with an instance identifier |
| + (`conflict`, both named). |
| +- `StartWorkerStaged`: nested, fenced, conflict, flat and foreign arms; |
| + the nested arm names the staged build to root, never the live link, and |
| + asks the exec gate before the start request; `launch.worker.json` |
| + carries the manifest argv and the composed env with `CARLOS_SIDECAR_DB` |
| + and the per-host `CARLOS_AGENT_SOCKET`; the old |
| + `TestKillHostProvesANestedLayoutWorkersHostCgroup` replaced. |
| +- The per-host agent listener: bound before the start, closed after |
| + `done`; a token for host A refused on B's listener and on the shared |
| + socket alike; the uniform 401; mode 0600. |
| +- `unitProc.Crashed`: a failed own invocation with `exit-code` is a crash; |
| + `success` after SIGTERM is not; a forged exit record claiming either is |
| + ignored; an unloaded invocation is not a crash. |
| +- The activator: a nested worker's wake, ready, next-due, idle, sleep, |
| + timer wake, crash restart and finish park, end to end over the real |
| + control server and a fake systemd that runs the trampoline's stub. |
| +- The control service: `worker` admitted on the four verbs, refused where |
| + it does not belong; the worker's unit and slice names; the tier always |
| + set; a staged build validated with the worker host's own app. |
| +- Templates: the identity test across five templates; the worker |
| + template's four differing lines and nothing else. |
| +- Reconciler: mint nested exactly when the instance mint does and the flag |
| + is on; converge accepts both paths; the co-spawn check unchanged. |
| +- Layout pass: a hibernating worker moves asleep; an always-on worker is |
| + deferred for the grace and then stopped once; every wake source of a |
| + fenced worker refuses (timer, wake verb, boot pass, sweep); a journal |
| + whose host suffix and row `Kind` disagree is refused; a row with no |
| + artifacts anywhere moves by CAS alone, and a row with some artifacts but |
| + no database is still refused; revert brings a worker back byte for byte. |
| +- `ConfigEnvPath` and SyncExec for a nested worker; the litestream |
| + generator and shrink guard across an always-on worker's move. |
| +- Flat-exec policy: each value, unset under both layout defaults, an |
| + unknown value, an invalid allowlist entry; the refusal from both seams; |
| + the enumeration test over `exec.Command` in `internal/activator`; the |
| + swap gate and the respawn veto classify the new error; the nudge of |
| + D16 runs the pass for a refused managed row and not for a foreign one. |
| +- Status: the `FlatExec` section and `Layout.Workers` pass the contract |
| + allowlist (`internal/status` `contract_test.go`) and never reach |
| + `fleet.json`; worker rows in the layout report. |
| +- Console: the create door's refusal order (after the region, before the |
| + cap and the claim branch), every case of D17, `code: "isolation"`; the |
| + downgrade guard's worker floor; the N+1 channel guard of D15. |
| +- `systemapply`: a candidate without the worker template is refused while |
| + a nested worker row exists, and installs when none does. |
| + |
| +Kernel and box level (root runner or a dev box; these need real cgroup v2 |
| +and systemd): |
| + |
| +- §1.6's cgroup migration from a flat child into `supervise`, and its |
| + metadata reach under the fence, recorded as evidence before and after |
| + `refuse`. |
| +- A real nested worker unit: namespace listing, the probe's datagrams, |
| + `PrivatePIDs`, the per-host agent socket reachable from inside and the |
| + shared one absent, the log stream in `carlos sidecars logs`, a crash |
| + read as a crash, `systemctl restart carlos-edge` stopping it and the |
| + boot pass starting an always-on one again. |
| +- The always-on move's litestream release (D9). |
| + |
| +The full gate runs once before handoff; locally, only the changed |
| +packages' tests. |
| + |
| +--- |
| + |
| +## 10. Out of scope |
| + |
| +- **Claim-mode workers.** Claims-on refuses every legacy launch shape |
| + today, workers included (`workerbuild.go:111-113`). When claims are |
| + activated, the root role launcher needs a worker role of its own; that is |
| + the claim work's, and until then a claims-on box runs no workers. |
| +- **Retiring the legacy co-spawn** (sidecar design §10). Unchanged, and D1 |
| + keeps it separable. |
| +- **Per-instance uids** (nested design §11, option B). |
| +- **Rotating what the flat tier could reach.** An incident decision, not a |
| + design one (§11 Q2). |
| +- **Unit-backed tenants and the doc engine** (§7.3). |
| + |
| +--- |
| + |
| +## 11. Open questions for Paul |
| + |
| +1. **Pause sidecar creation now?** Turning `CONSOLE_REQUIRE_ISOLATED_WORKERS` |
| + on before any box reports nested workers refuses every sidecar create |
| + until nested workers roll. It stops new flat sidecars today at the cost |
| + of the feature for that time. Existing sidecars keep running. |
| +2. **Treat what the flat tier could reach as exposed?** B1 was confirmed |
| + live, and §1.6 suggests the fence may not have held against a |
| + deliberate tenant. That would put the box roles' reachable secrets |
| + (regional SSM, the deployment bucket, and on the console box the IAM |
| + user minting) in scope for rotation, and possibly a look at the |
| + bucket's write history. That is an incident call. |
| +3. **Who confirms the apps that use native sidecars are ready to move?** |
| + §8.2 step 5 needs each such app's owner to confirm, or ship, a worker |
| + that reads `CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` and does not |
| + rely on loopback, before its flagship boxes turn nested workers on. And |
| + does the operator app the fence exempts need the flat allowlist at all, |
| + or can it move to a unit or to the nested tier first, so that every |
| + box's allowlist can be empty? |
diff --git a/docs/superpowers/specs/CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md| new file mode 100644 |
| index 000000000..49b30818e |
| --- /dev/null |
| +++ b/docs/superpowers/specs/CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md |
| @@ -0,0 +1,162 @@ |
| +# Review brief: sidecar isolation (security finding B1) |
| + |
| +You are an independent second reader of a design spec for the CARLOS |
| +platform (this repository). Read it cold and adversarially. You are looking |
| +for the way a tenant still gets a process in the host namespace, or the way |
| +an upgrade, a downgrade or a failure leaves a workload down or its data in |
| +two places. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md`, |
| +at the commit named in the prompt. Main is `fef28dc5`. |
| + |
| +## What it claims |
| + |
| +Native sidecars (`Kind "sidecar"` rows) start today as plain children of the |
| +edge on every box, whatever its layout, so any account member can run code |
| +as uid `carlos` in the host namespace. The spec claims that after its six |
| +steps (§7.2) no tenant process can run in the host namespace on a box with |
| +`CARLOS_FLAT_EXEC=refuse` and an empty allowlist, by any spawn path, in any |
| +layout state including a failed nested gate; that existing sidecars move to |
| +the nested tier without losing data or a wake; that no box loses a workload |
| +on upgrade; and that the interim metadata fence can then be removed. |
| + |
| +Read §0's decision table, §1.8 (where the spec says the problem statement |
| +and main disagree) and §2 (what it reopens) first. |
| + |
| +## Ground rules |
| + |
| +- Check against the code on `main`, not the spec's description of it. Cite |
| + `file:line` for every claim about existing code. |
| +- The nested design's rulings (`2026-09-11-isolated-exec-design.md`) and |
| + plan 4's amendments R1 to R25 are fixed unless §2 of this spec reopens |
| + them. Do not argue for per-instance uids, unit-backed console instances |
| + or polkit. Do report a ruling this spec silently breaks. |
| +- The sidecar design's rulings (`2026-08-17-sidecar-members-design.md` §2) |
| + are fixed: standalone members, the synthetic host, hibernation, the edge's |
| + refusal on Kind. |
| +- A mechanism the spec names, bounds and assigns is complete at spec level. |
| + Do not ask for plan-level detail. |
| + |
| +## Code to read |
| + |
| +- **Spawn seams:** `internal/activator/backend_exec.go` (`spawnInCgroupGated`, |
| + `StartWorkerStaged`, `StartSidecar`, `startSidecarUnitFrom`, `Start`, |
| + `Seed`, `seedChild`, `spawnChildFrom`, `nestedGate`, `childProc.Crashed`), |
| + `containment.go` (`spawnInCgroupAtTier`, `startRole`), `rolelaunch.go`, |
| + `workerbuild.go`, `workerswap.go`, `swap.go` (the gate classification), |
| + `unitproc.go`, `prepare.go`, `paths.go`, `layout.go`, `cmdunit.go`. |
| +- **The activator:** `activator.go` (worker wake, `workerEnv`, |
| + `respawnWorkerLocked`, `restartWorkerLocked`, `procCrashed`, the boot pass |
| + and the sweep's always-on start, `wakeAllowed`), `maintenance.go` |
| + (`FenceAndStop`). |
| +- **Root side:** `internal/instancectl/names.go`, `server.go`, |
| + `stagedexe.go`; `internal/instanceexec/exec.go`, `exit.go`; |
| + `internal/host/units/carlos-instance*@.service`, `carlos-edge.service`. |
| +- **Layout pass:** `internal/layout/pass.go`, `plan.go`, `journal.go`, |
| + `preflight.go`; `internal/host/layoutleg.go`, `systemapply.go`. |
| +- **Provisioning and config:** `internal/provision/sidecar_reconcile.go`, |
| + `reconcile.go` (`RouteFor`, `mintLayoutReader`), `provision.go` |
| + (`InstanceID`, `NestedPaths`); `internal/appconfig/syncexec.go`; |
| + `internal/ops/litestream.go`. |
| +- **Agent socket:** `internal/agentsock/agentsock.go`, `internal/host/agent.go` |
| + (`serveAgentSocket`, the boot pass). |
| +- **Console:** `internal/console/apisidecars.go`, `authcli.go`, |
| + `layoutguard.go`; `internal/status/snapshot.go` and its contract test. |
| +- **Cgroups:** `internal/cgroup/cgroup.go` (the delegation dance, the |
| + `supervise` leaf, `StartGated`'s nil-manager path). |
| + |
| +## Attack these |
| + |
| +1. **A fifth spawn path.** §4.2 says every tenant process the agent starts |
| + outside a unit goes through `spawnInCgroupGated` or `spawnInCgroupAtTier`. |
| + Find one that does not: a worker swap or recycle, a seed fallback, the |
| + probe-wake path, a maintenance or schedule runner, a replica path, a |
| + `carlos run` fallback, the moves pass, anything that `exec`s a staged |
| + build or the live link. Then find a policy bypass: an allowlist value |
| + that matches more than intended, a host the check is asked about that is |
| + not the host being spawned, the policy read at the wrong time. |
| +2. **The fence bypass of §1.6.** Is it right that a flat child (uid |
| + `carlos`, in `carlos-exec@<host>` under the edge's `Delegate=` subtree) |
| + can write its pid into the `supervise` leaf's `cgroup.procs`? Cite the |
| + kernel's delegation rules and systemd's `Delegate=` chown set. If it is |
| + wrong, say what stops it. If it is right, does anything else in the repo |
| + rely on the leaf as a boundary? |
| +3. **D1, a new role versus reusing `sidecar`.** Is the new template |
| + justified, and is it really "four lines and nothing else"? What does the |
| + control service do with `role=sidecar` that a worker would have broken, |
| + and what does it do with `role=run` that a worker now needs (the legacy |
| + start fence, `clearLegacyClaimLaunch`, the tier rule)? Do the sweep, |
| + `ParseUnitName`, the sampler and `carlos status --instances` all learn |
| + the new prefix in the spec? |
| +4. **D3, the per-host agent socket.** Can a nested worker reach the shared |
| + `/run/carlos/agent.sock`? Can a flat tenant during the transition reach |
| + or redirect a nested worker's listener (the run directory is on a |
| + `nosymfollow` mount; the agent binds the socket)? Does host pinning hold |
| + if the header is absent, repeated or differently cased? |
| +5. **D4 and §8.2 step 5.** Is `CARLOS_SIDECAR_DB` the right fix, and is there |
| + any other way a worker today learns or assumes its paths (cwd, a relative |
| + path, the config bundle, `CARLOS_SIDECAR_HOST`)? What happens to a worker |
| + that ignores it on its first nested start: does it crash-loop, and does |
| + the crash read as a crash (D6)? |
| +6. **D6, crash versus finish.** Does systemd keep `Result` and the invocation |
| + for a failed simple service long enough for the observer to read it, |
| + across `reset-failed` from the sweep or the next start? Can a worker make |
| + a crash read as a finish, or the reverse, given the trampoline's exit |
| + convention and `SuccessExitStatus=`? |
| +7. **D8, moving rows.** Can any wake source start a fenced worker: the |
| + timer, `carlos sidecars wake`, the boot pass, the sweep's always-on |
| + start, a converge onto a new build, a `max_run` recycle? Does the |
| + host-suffix rule for a worker journal hold against a flat tenant editing |
| + the row's `Kind` or `Socket`? Is the "moves by row alone" rule safe: can a |
| + flat tenant hide a database (rename it away) to get a row moved and then |
| + restored from an older bucket copy, or to leave two databases? |
| +8. **D9, always-on replication.** Is a new replica lineage at the nested |
| + path actually what happens, and is anything lost between the fence and |
| + the regenerated config? Does `CoverageShortfall` refuse the config write |
| + mid-move as written today? What do the restore drills |
| + (`carlos-restore-verify`, `carlos-hibernate-verify`) do with the old |
| + lineage? |
| +9. **D10 and D15, availability.** Walk a box through upgrade to N, the two |
| + box settings, a gate failure, an agent restart, and N+1, and find the |
| + moment a workload that ran before is refused without its operator having |
| + chosen that. Include a box with `CARLOS_LAYOUT_DEFAULT=flat`, a box with |
| + a recorded `flat` intent, a box whose layout intent is unreadable, a |
| + multi-box region where only one box holds a row's database, and a |
| + customer-fleet box that publishes under `fleets/<name>/`. |
| +10. **D11, downgrade.** Can a rollback, a failed-gate restore, the recover |
| + unit or a console channel move put a binary from before this change in |
| + front of a nested worker row? Is "unit set lacks the worker template" |
| + the right test on every path R9 covers? |
| +11. **D17, the console check.** Can a record land unisolated anyway: a |
| + region with no fresh box, a box that joins the region after the create, |
| + a pool or placement (`no-inbound`) narrower than the region, a direct |
| + bucket write, the claim-mode create path, an app moving region? Does the |
| + refusal leak which boxes exist or what they report to a member? |
| +12. **D18, retiring the fence.** Are the four conditions sufficient and |
| + observable without box access? Could they all hold while a flat child |
| + still runs (a child outside the delegated subtree, a box with no |
| + delegation where children sit in the edge's own cgroup)? |
| +13. **§7, the threat model.** Is anything a malicious member can still do |
| + after the last step missing from §7.3? Is the provider boundary of §7.4 |
| + drawn so that no core code names an AWS address or service? |
| +14. **Census and rollout.** Does §8.3 list every root-run file the change |
| + touches? Is anything that the spec says is dark actually live in |
| + release N? |
| + |
| +## Output |
| + |
| +Findings ranked P1 (a tenant still gets a host-namespace process, data ends |
| +up in two places or is lost, or the spec cannot be built against `main`), |
| +P2 (a real gap a builder would hit, or an availability failure on upgrade), |
| +P3 (worth fixing). For each: the section, the failure as concrete steps, |
| +the code citation, and a suggested fix. End with an overall verdict: accept, |
| +accept with fixes, or reject. Do not report wording or style. |
| + |
| +Write the findings to `docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md`. |
| + |
| +## Round 2 and later |
| + |
| +Read your previous findings (`CODEX-FINDINGS-SIDECAR-ISOLATION-R<n-1>.md`) |
| +and the spec's disposition of them first. Start with a table: each previous |
| +finding, and whether the new revision fixes it (cite the section). Then |
| +report only new findings. |