amadan
Sidecar isolation spec, revision 3: answers Astra round 2, Paul's rulings
R2-1 (uncertain hold replacement) and R2-3 (running replicator, not the config file) accepted; R2-2 answered with no new floor: workers move only once carlos.prev is worker-capable, and install keeps both binaries above the floor. R2-4 answered by Paul's Q6 ruling: unset CARLOS_FLAT_EXEC is a per-box ratchet. D17 (and R2-5) and D18 deferred to follow-ups (§13). Approved to build by Paul 2026-10-24.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+284
−249
docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md+284 −249
CI cancelled — superseded by a newer push — run details
CI log
=== gate ===
.amadan/test-gate.sh
go: downloading golang.org/x/net v0.56.0
go: downloading golang.org/x/crypto v0.53.0
go: downloading golang.org/x/sys v0.46.0
go: downloading github.com/pion/stun/v3 v3.0.1
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading golang.org/x/sync v0.21.0
go: downloading golang.org/x/text v0.38.0
go: downloading golang.org/x/crypto v0.53.0
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading golang.org/x/text v0.38.0
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading github.com/pion/transport/v3 v3.0.8
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading github.com/pion/transport/v3 v3.0.8
=== launchctx (1 s)
ok amadan.net/carlos/platform.git/launchctx 0.012s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (26 s)
go: downloading golang.org/x/crypto v0.53.0
go: downloading golang.org/x/text v0.38.0
go: downloading golang.org/x/tools v0.47.0
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading github.com/pion/transport/v3 v3.0.8
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading golang.org/x/mod v0.37.0
=== gofmt (1 s)
gofmt -l .: clean
=== rest (178 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.627s
ok github.com/carlosframework/platform/cmd/moveprobe 11.954s
ok github.com/carlosframework/platform/cmd/zddprobe 0.761s
ok github.com/carlosframework/platform/internal/activator 59.981s
ok github.com/carlosframework/platform/internal/agentsock 0.020s
ok github.com/carlosframework/platform/internal/alerting 0.007s
ok github.com/carlosframework/platform/internal/analytics 0.074s
ok github.com/carlosframework/platform/internal/appconfig 2.885s
ok github.com/carlosframework/platform/internal/apperr 0.190s
ok github.com/carlosframework/platform/internal/applog 0.312s
ok github.com/carlosframework/platform/internal/awsinv 0.015s
ok github.com/carlosframework/platform/internal/boxchoice 0.008s
ok github.com/carlosframework/platform/internal/boxid 0.892s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.042s
ok github.com/carlosframework/platform/internal/caps 0.036s
ok github.com/carlosframework/platform/internal/certwarm 0.006s
ok github.com/carlosframework/platform/internal/cgroup 0.768s
ok github.com/carlosframework/platform/internal/changelog 0.596s
ok github.com/carlosframework/platform/internal/claimactivation 1.261s
ok github.com/carlosframework/platform/internal/clamd 0.068s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 7.994s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.049s
ok github.com/carlosframework/platform/internal/dbquery 0.036s
ok github.com/carlosframework/platform/internal/dbreads 2.114s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.014s
ok github.com/carlosframework/platform/internal/dnsrec 0.024s
ok github.com/carlosframework/platform/internal/docsite 0.042s
ok github.com/carlosframework/platform/internal/domains 2.967s
ok github.com/carlosframework/platform/internal/doorkey 0.023s
ok github.com/carlosframework/platform/internal/economics 0.014s
ok github.com/carlosframework/platform/internal/edge 46.187s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.008s
ok github.com/carlosframework/platform/internal/fleetapi 0.141s
ok github.com/carlosframework/platform/internal/gate 1.676s
ok github.com/carlosframework/platform/internal/genselect 0.032s
ok github.com/carlosframework/platform/internal/geo 0.013s
ok github.com/carlosframework/platform/internal/geoip 0.037s
ok github.com/carlosframework/platform/internal/historyreadback 0.216s
ok github.com/carlosframework/platform/internal/host 142.370s
ok github.com/carlosframework/platform/internal/hostactors 5.592s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.016s
ok github.com/carlosframework/platform/internal/hostbox 3.030s
ok github.com/carlosframework/platform/internal/hostclaim 0.762s
ok github.com/carlosframework/platform/internal/hostenv 0.016s
ok github.com/carlosframework/platform/internal/hostevidence 0.006s
ok github.com/carlosframework/platform/internal/hostexec 5.350s
ok github.com/carlosframework/platform/internal/hostfence 0.050s
ok github.com/carlosframework/platform/internal/hostfiles 10.378s
ok github.com/carlosframework/platform/internal/hostfloor 0.022s
ok github.com/carlosframework/platform/internal/hostguard 1.026s
ok github.com/carlosframework/platform/internal/hostops 1.927s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.016s
ok github.com/carlosframework/platform/internal/hostquarantine 0.018s
ok github.com/carlosframework/platform/internal/hostrelocate 0.620s
ok github.com/carlosframework/platform/internal/hostresolve 0.175s
ok github.com/carlosframework/platform/internal/hostwrite 0.017s
ok github.com/carlosframework/platform/internal/infraview 0.020s
ok github.com/carlosframework/platform/internal/instancectl 5.241s
ok github.com/carlosframework/platform/internal/instanceexec 2.804s
ok github.com/carlosframework/platform/internal/instancefs 0.015s
ok github.com/carlosframework/platform/internal/instancekey 0.019s
ok github.com/carlosframework/platform/internal/instanceupdates 0.023s
ok github.com/carlosframework/platform/internal/layout 1.744s
ok github.com/carlosframework/platform/internal/lbhealth 0.022s
ok github.com/carlosframework/platform/internal/lease 10.134s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.556s
ok github.com/carlosframework/platform/internal/ledger 0.012s
ok github.com/carlosframework/platform/internal/mail 0.294s
ok github.com/carlosframework/platform/internal/maintenance 0.105s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.596s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.082s
ok github.com/carlosframework/platform/internal/metrics 0.006s
ok github.com/carlosframework/platform/internal/moves 22.357s
ok github.com/carlosframework/platform/internal/moves/archive 1.055s
ok github.com/carlosframework/platform/internal/objstore 0.289s
ok github.com/carlosframework/platform/internal/ops 0.229s
ok github.com/carlosframework/platform/internal/park 0.307s
ok github.com/carlosframework/platform/internal/plans 0.009s
ok github.com/carlosframework/platform/internal/privatefile 0.004s
ok github.com/carlosframework/platform/internal/provision 31.365s
ok github.com/carlosframework/platform/internal/publicrelease 0.009s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.713s
ok github.com/carlosframework/platform/internal/relay 7.242s
ok github.com/carlosframework/platform/internal/release 26.079s
ok github.com/carlosframework/platform/internal/rollout 3.855s
ok github.com/carlosframework/platform/internal/rolloutwords 0.003s
ok github.com/carlosframework/platform/internal/rootaws 0.012s
ok github.com/carlosframework/platform/internal/rootparticipation 0.060s
ok github.com/carlosframework/platform/internal/rootstate 0.028s
ok github.com/carlosframework/platform/internal/runs 0.558s
ok github.com/carlosframework/platform/internal/s3compat 0.007s
ok github.com/carlosframework/platform/internal/safefile 0.009s
ok github.com/carlosframework/platform/internal/schedule 2.715s
ok github.com/carlosframework/platform/internal/seeds 0.011s
ok github.com/carlosframework/platform/internal/sim/engines 2.670s
ok github.com/carlosframework/platform/internal/sim/hostlife 11.282s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.955s
ok github.com/carlosframework/platform/internal/status 0.067s
ok github.com/carlosframework/platform/internal/steering 0.007s
ok github.com/carlosframework/platform/internal/stopevidence 0.299s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.004s
ok github.com/carlosframework/platform/internal/unixdial 0.004s
ok github.com/carlosframework/platform/internal/updatewords 0.004s
ok github.com/carlosframework/platform/internal/usage 0.005s
ok github.com/carlosframework/platform/internal/version 0.003s
ok github.com/carlosframework/platform/internal/vet 1.703s
=== internal_console@0 (91 s)
ok github.com/carlosframework/platform/internal/console 89.395s
=== internal_console@1 (45 s)
ok github.com/carlosframework/platform/internal/console 43.130s
=== internal_console@2 (41 s)
ok github.com/carlosframework/platform/internal/console 39.617s
=== internal_console@3 (41 s)
ok github.com/carlosframework/platform/internal/console 39.225s
=== internal_console@4 (44 s)
ok github.com/carlosframework/platform/internal/console 42.116s
=== internal_console@5 (41 s)
ok github.com/carlosframework/platform/internal/console 39.996s
=== internal_console@6 (38 s)
ok github.com/carlosframework/platform/internal/console 36.399s
=== internal_console@7 (47 s)
ok github.com/carlosframework/platform/internal/console 45.305s
=== root@0 (110 s)
ok github.com/carlosframework/platform 108.369s
=== root@1 (84 s)
ok github.com/carlosframework/platform 82.674s
=== root@2 (91 s)
ok github.com/carlosframework/platform 89.359s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50528418 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: cached, digest verified
mc: cached, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.024s
=== RUN TestT4StorageProbeMinio
2026/10/24 20:33:17 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E19166DC6D4E75, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.025s
=== RUN TestS3StoreMinioContract
=== RUN TestS3StoreMinioContract/conditional_get_and_etag_listing
=== RUN TestS3StoreMinioContract/bump_through_every_mutation_path
--- PASS: TestS3StoreMinioContract (0.03s)
--- PASS: TestS3StoreMinioContract/conditional_get_and_etag_listing (0.01s)
--- PASS: TestS3StoreMinioContract/bump_through_every_mutation_path (0.02s)
PASS
ok github.com/carlosframework/platform/internal/release 0.032s
Bucket created successfully `local/dbf-deploy`.
Bucket created successfully `local/dbf-data-ie`.
Bucket created successfully `local/dbf-data-uk`.
local/dbf-deploy versioning is enabled
Created policy `dbf-console` successfully.
Created policy `dbf-ukbox` successfully.
Added user `dbfconsole` successfully.
Added user `dbfukbox` successfully.
Attached Policies: [dbf-console]
To User: dbfconsole
Attached Policies: [dbf-ukbox]
To User: dbfukbox
amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md| index aa4f7c01c..c48fdc073 100644 |
| --- a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md |
| +++ b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md |
| @@ -1,6 +1,8 @@ |
| # Sidecar isolation: native sidecars on the nested tier, and no flat exec by default |
| -**Date:** 2026-10-24 · **Status:** draft, revision 2, for an independent |
| +**Date:** 2026-10-24 · **Status:** revision 3, approved to build by Paul |
| +2026-10-24 ("let's just do this and close the vulnerability and roll a |
| +release when done"), for an independent |
| review (brief: `CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md`) and then Paul · |
| **Requested by:** security review finding B1, confirmed live 2026-10-24 · |
| **Main at:** fef28dc5 (every `file:line` below is on that commit unless it |
| @@ -20,6 +22,25 @@ not spawn history (D15); the create check uses the reconciler's own |
| placement set (D17); and fence retirement needs a complete process census |
| (D18). |
| +**Revision 3** answers Astra's round 2 (`CODEX-FINDINGS-SIDECAR-ISOLATION-R2.md`, |
| +at 730ba093, verdict reject: three P1s, two P2s), marked **(R2-n)**, and |
| +records Paul's rulings of 2026-10-24 (§11). It also cuts what ships to what |
| +closes the hole, in one release and its patch (§13): |
| + |
| +- **D11 needs no new floor.** Workers become nested on a box only once its |
| + `carlos.prev` is itself a release that understands nested workers. Every |
| + path that can put an older binary in front of a row (rollback, the |
| + recovery unit, the root guard's copy) picks `carlos.prev` or the current |
| + binary, so both are then worker-capable (R2-2). |
| +- **D15 needs no console guard and no `host.env` edit.** An unset |
| + `CARLOS_FLAT_EXEC` turns to `refuse` on its own, on each box, the first |
| + time that box's inventory is empty, and never turns back. An unset |
| + `CARLOS_NESTED_WORKERS` is on. So no default change stops a workload that |
| + was running when it changed (R2-4). |
| +- **Deferred to follow-ups:** the console's create-time check (D17, and so |
| + R2-5) and the fence's retirement (D18). Neither is the boundary; the box's |
| + refusal is. |
| + |
| **Builds on:** the isolated-exec design |
| (`2026-09-11-isolated-exec-design.md`, revision 17; "the nested design" |
| below), its plan 4 amendments (`../plans/2026-09-25-isolated-exec-4-layout-amendments.md`, |
| @@ -81,14 +102,14 @@ The decisions, for scanning (each is argued where it is made): |
| | D8 | The layout pass moves worker rows with their full extras inventory; a row whose database and its companions are absent on both sides moves without it (rev 2) | §3.8 | |
| | D9 | Every worker replicates through its lease only; the box-wide replicator and the restore drills exclude workers (rev 2) | §3.9 | |
| | D10 | A failed nested gate never reopens flat for a worker | §3.10 | |
| -| D11 | A workers protocol floor, armed before any worker moves, holds every install, rollback and recovery selection (rev 2) | §3.11 | |
| +| D11 | Workers move only once `carlos.prev` is itself worker-capable; install keeps both binaries at or above the floor (rev 3) | §3.11 | |
| | D12 | `CARLOS_FLAT_EXEC=refuse\|report` plus an exact-host allowlist, read from `host.env` | §4.1 | |
| | D13 | The refusal sits beside the claim refusal in both spawn seams, with its own error | §4.2 | |
| | D14 | Refusals and a registry inventory of what `refuse` would stop are reported per host (rev 2) | §4.3 | |
| -| D15 | Ship `report`; operators switch boxes; N+1 changes a default only on boxes where the change stops nothing, judged from inventory (rev 2) | §4.4 | |
| +| D15 | Unset `CARLOS_NESTED_WORKERS` is on; unset `CARLOS_FLAT_EXEC` is a per-box ratchet that turns to `refuse` once nothing would be refused (rev 3) | §4.4 | |
| | D16 | A refused managed row nudges the layout pass for that host | §4.5 | |
| -| D17 | Creation is refused unless the reconciler's own placement set has a fresh box and every one reports nested workers runnable (rev 2) | §5 | |
| -| D18 | The fence retires per box on four observable conditions, the third a complete process census; an allowlisted host keeps it (rev 2) | §6 | |
| +| D17 | (deferred) Creation is refused unless the reconciler's own placement set has a fresh box and every one reports nested workers runnable | §5 | |
| +| D18 | (deferred) The fence retires per box on four observable conditions, the third a complete process census | §6 | |
| --- |
| @@ -528,6 +549,19 @@ only for an unproven generation or a legacy start marked uncertain |
| start on an entry that keeps its lease throughout, so a respawn's |
| uncertain start needs no lease decision; the next stop of that entry, by |
| any path, goes through the fenced `/stop`, which orders it. |
| +- **Replacements (R2-1).** An uncertain start is not "nothing ran". The |
| + nested worker arm's uncertain outcome also answers true to |
| + `startMayHaveRun` (`containment.go:160-171`, which today knows only |
| + `startUncertain` and an unproven generation), so it reaches |
| + `ReplaceWorker` as "may have executed". `ReplaceWorker`'s |
| + unchanged-incarnation branch (`workerswap.go:152-160`) then treats it as |
| + the attempt's build having run: a hold attempt restores nothing and stops |
| + the entry (the F2 rule for a hold build exec'd before a refusal), and an |
| + overlap attempt falls back only after the attempted unit is proven gone |
| + through the fenced `/stop` and `RoleGone`. The respawn's error arm |
| + (`activator.go:3739`) keeps the obligation on the entry until that proof, |
| + so no path restores the old build over a database the new one may have |
| + migrated. |
| - `KillHost` inspecting the slice alone is not the proof, and the rewrite |
| of `TestKillHostProvesANestedLayoutWorkersHostCgroup` says so. |
| @@ -579,7 +613,7 @@ forever (R1-7). That is not the flat contract. |
| **Decision.** `sidecarRouteFor` takes the same `nested` answer the instance |
| mint takes (`mintLayoutReader`, `internal/provision/reconcile.go:471-490`; |
| `RouteFor`'s `nested` arm, `:456-462`) and, when it is true and |
| -`CARLOS_NESTED_WORKERS` is on (§8.1) and the workers floor holds (D11), mints `DBPath = NestedDBPath`. Every |
| +`CARLOS_NESTED_WORKERS` is on (§8.1) and D11's `carlos.prev` condition holds, mints `DBPath = NestedDBPath`. Every |
| other answer mints flat, as today. `convergeSidecar`'s shape check |
| (`sidecar_reconcile.go:621-626`) accepts either the flat or the nested path |
| for the row's host, and reports anything else as today. |
| @@ -593,7 +627,7 @@ and an instance host can never end in `.sidecar.internal`. |
| **Decision.** The layout pass's eligibility (`pass.go:264`, `:529`, `:747`) |
| widens from `ActivatorManaged(rt)` to `ActivatorManaged(rt) || |
| -WorkerManaged(rt)`, behind `CARLOS_NESTED_WORKERS` (§8.1) and the workers floor (D11). Everything else |
| +WorkerManaged(rt)`, behind `CARLOS_NESTED_WORKERS` (§8.1) and D11's `carlos.prev` condition. Everything else |
| about the move is plan 4's: |
| - **Quiesce.** `FenceAndStop` on the worker's entry. A hibernating worker |
| @@ -699,11 +733,24 @@ replication, hibernating or always-on, flat or nested: |
| first drill after the change is the evidence that those lineages are |
| sound after a period of two writers. One that fails is a finding for |
| that host, reported as any drill failure is. |
| +- **The running replicator, not the file, decides (R2-3).** |
| + `/etc/litestream.yml` is regenerated by an operator (`carlos ops |
| + litestream`, `internal/ops/ops.go:149-175`, which prints `changed` for |
| + the caller to restart the replicator) and litestream reads it only at |
| + start (`carlos-litestream.service:16`). A config that no longer names a |
| + worker's database is therefore no evidence that the running daemon has |
| + let go of it. So the layout pass moves an always-on worker only when the |
| + box-wide replicator cannot be holding the file: `carlos-litestream` is |
| + not active, or it is active and its current invocation started after |
| + the config was last written (`ActiveEnterTimestamp` against the file's |
| + mtime, read through `systemctl show`) **and** that config does not name |
| + the file. Otherwise the move is deferred with the reason "the box-wide |
| + replicator still names this database: regenerate its config and restart |
| + it" (copy to review), and retried on every pass, so a failed restart is |
| + never taken for a done one. |
| - **The move is then path-blind** for every worker, as it already is for a |
| hibernating one: the lease lineage is keyed by host, so moving the file |
| - moves nothing in the bucket, and no box-wide reload sits inside the |
| - move. Revision 1's wait for a regenerated config, and the dev-box check |
| - on litestream releasing the file, are gone. |
| + moves nothing in the bucket. |
| This part has no dependency on the rest of the spec and is not behind |
| `CARLOS_NESTED_WORKERS`. It can land first (§11 Q4). |
| @@ -727,78 +774,64 @@ recorded (`layoutleg.go:127-146`). For workers: |
| loudly"). An operator who needs a specific host up through a gate |
| failure adds it to the allowlist, knowingly (§4.1). |
| -### 3.11 Decision D11: downgrade |
| +### 3.11 Decision D11: downgrade (rev 3: R1-2, R2-2) |
| A binary from before this change runs a nested worker row through the old |
| `StartWorkerStaged`: flat, with the env read from the flat directory and |
| no database path, and the row's own nested `DBPath` replicated by the old |
| lease code. Two copies, and unisolated. |
| -Revision 1 guarded `systemApplier.install` only. Round 1 showed three |
| -other ways an old binary gets in front of a nested worker row (R1-2): |
| - |
| -- the update starts the new edge before its health gate |
| - (`systemapply.go:600-612`), so the new agent may already have moved |
| - workers when the gate fails and `restoreFiles` (`:1784-1800`) puts |
| - `carlos.prev` back, with no worker check; |
| -- the recovery unit can **exec `carlos.prev` itself** |
| - (`internal/host/units/carlos-system-recover.service:41`), so a check |
| - that only the new binary makes cannot hold that path; |
| -- a row-only predicate misses a move interrupted before its row CAS: the |
| - row still reads flat while its files are half-way. R9's guard counts |
| - journals, fenced rows and conflicts for exactly this reason |
| - (`systemapply.go:1273-1310`). |
| - |
| -The repo already has the mechanism for "an old binary must not run once |
| -this box has state it cannot read": the moves protocol floor, and the seeds |
| -floor built on it (seeded-instances design §3.3; `internal/seeds/floor.go:41-42`, |
| -`internal/host/rootseedsfloor.go`). A floor, once armed on a box, makes |
| -root give a verdict on `carlos.prev`, and the recovery unit runs |
| -`carlos.prev` only when that verdict names the armed floor |
| -(`carlos-system-recover.service:9-30`, `:41`); `restoreFiles` checks the |
| -same floor before it changes anything (`systemapply.go:1790-1797`). |
| - |
| -**Decision.** A third floor, the **workers floor**, on that mechanism: |
| - |
| -- **Published and armed before anything moves.** An operator publishes |
| - `control/workers-protocol.json` naming the first release that carries |
| - this spec. A box arms it locally as it arms the seeds floor |
| - (`armFloor`, `systemapply.go:908-925`), and gives its `carlos.prev` |
| - verdict. The layout pass treats worker rows as eligible (D8) only on a |
| - box whose workers floor is armed **and** whose `carlos.prev` is either |
| - excluded or at or above the floor; until then `CARLOS_NESTED_WORKERS=on` |
| - is reported as `blocked: workers floor` and moves nothing. The reconciler |
| - mints nested worker rows (D7) under the same condition. So every nested |
| - worker row, journal or fence on a box exists only after every binary that |
| - box can select would refuse to run below it. |
| -- **Every activation path obeys it.** Install refuses a candidate below an |
| - armed workers floor, beside the moves floor's refusal. `restoreFiles` |
| - refuses a `carlos.prev` below it, so a failed gate leaves the box on the |
| - new build, journal kept, as the moves floor does. The recovery unit's |
| - `ExecStart` gains the third test, the same shape as the seeds floor's |
| - (`/data/.moves/workers-floor.min-release`, verdict |
| - `/opt/carlos/carlos.prev.workers-allowed`), so the shell never execs a |
| - `carlos.prev` below it. |
| -- **The row test, for the console and for status,** is R9's, not a |
| - row-only one: a worker host is named when its row has `NestedPaths`, |
| - when `Classify` calls it fenced or in conflict, or when it has a |
| - journal, whether or not the row exists (`layoutReversalIncomplete`, |
| - `systemapply.go:1273-1310`, which already covers every host this way; |
| - D8's status change makes the console see worker hosts too). The console's |
| - `layoutDowngradeBlocked` (`layoutguard.go:24-69`) gains |
| - `release.WorkerLayoutFloor`, judged on those observed hosts, and blocks a |
| - channel move below it while any is named. |
| -- **No disarm.** Neither existing floor has one, and this one adds none: a |
| - floor only ever rises. A deployment that must go below it reverts its |
| - worker rows first and then has an operator remove the floor files by |
| - hand, knowingly. The plan documents that step beside the revert. |
| -- `carlos system layout revert` reverts worker rows with the rest (they |
| - are in the pass). |
| - |
| -The unit-set test of revision 1 (candidate lacks |
| -`carlos-instance-worker@.service`) stays as a second net for a candidate |
| -whose version cannot be parsed, the way R9 pairs the mount unit with |
| -`LayoutFloor` (`systemapply.go:510-528`). |
| +Revision 1 guarded `systemApplier.install` only. Round 1 named the other |
| +ways an old binary reaches a row: the rollback after a failed gate |
| +(`restoreFiles`, `systemapply.go:1784-1800`), the recovery unit, which can |
| +exec `carlos.prev` itself (`carlos-system-recover.service:41`), and a move |
| +interrupted before its row CAS. Revision 2 answered with a third protocol |
| +floor; round 2 showed that the root guard's separately installed copy |
| +(`guardarm.go:148-166`, overriding the recovery unit's `ExecStart`) would |
| +not know it (R2-2), and on flagship no protocol floor is published at all. |
| + |
| +Revision 3 closes every path from one fact about them. **Each path that |
| +puts a binary in front of the rows picks one of two files: the current |
| +`carlos` or `carlos.prev`.** The recovery unit's shell runs one or the |
| +other (`:41`); the guard copy chooses `carlos.prev` when an apply's journal |
| +is present and the current binary otherwise (`guardexec.go:66-80`), and its |
| +floor route chooses between the same two (`:95-110`); `restoreFiles` puts |
| +`carlos.prev` back. And root records `carlos.prev`'s release beside it |
| +before every apply (`<optDir>/carlos.prev.version`, written in `notePrev`, |
| +`systemapply.go:818-830`) and keeps an older `carlos.prev` when the |
| +running build is not recorded known-good (`decidePrev`, `:792-815`). |
| + |
| +**Decision.** |
| + |
| +- **Workers move only when both files can run them.** The layout pass |
| + treats worker rows as eligible (D8), and the reconciler mints nested |
| + worker rows (D7), only on a box whose running build and whose |
| + `carlos.prev.version` both parse as releases at or above |
| + `release.WorkerLayoutFloor` (the first release carrying this spec). An |
| + unreadable or unparseable `carlos.prev.version`, or none, reads as below. |
| + Status says `workers: waiting for carlos.prev` (a closed word) while it |
| + holds. In practice a box qualifies after its second roll at or above the |
| + floor (§13). |
| +- **Once true, it stays true.** Neither file goes below the floor without |
| + an install, and install refuses that: |
| + - a candidate whose version parses below `WorkerLayoutFloor`, or whose |
| + unit set lacks `carlos-instance-worker@.service` (for a candidate whose |
| + version does not parse), is refused while any worker host is named, |
| + beside R9's guard and on its row test (below); |
| + - `decidePrev` never replaces `carlos.prev` with a binary below the |
| + floor, because the running build is at or above it, and keeps the |
| + existing `carlos.prev` otherwise. |
| + So the guard copy, whichever of the two it picks, picks a worker-capable |
| + binary. It needs no change, and neither does the recovery unit. |
| +- **The row test is R9's, not a row-only one:** a worker host is named when |
| + its row has `NestedPaths`, when `Classify` calls it fenced or in |
| + conflict, or when it has a journal, whether or not the row exists |
| + (`layoutReversalIncomplete`, `systemapply.go:1273-1310`). D8's status |
| + change lets the console see worker hosts too, and `layoutDowngradeBlocked` |
| + (`layoutguard.go:24-69`) gains `release.WorkerLayoutFloor`, judged on |
| + those hosts, and blocks a channel move below it while any is named. |
| +- `carlos system layout revert` reverts worker rows with the rest (they are |
| + in the pass); after it, install accepts an older candidate again. |
| ### 3.12 Every consumer of the shape |
| @@ -854,8 +887,9 @@ logged and taken as the safe reading): |
| refused (§4.2). |
| - `report`: every spawn proceeds as today, and each flat or foreign spawn |
| of a host not on the allowlist is recorded as a would-be refusal (§4.3). |
| -- unset: `report` in the release that carries this spec; `refuse` from the |
| - next (D15), with one exception. On a box whose **effective intent is |
| +- unset: the ratchet of D15 (rev 3): `report` until the box's inventory is |
| + first empty under a nested intent, `refuse` from then on, with one |
| + exception. On a box whose **effective intent is |
| flat by an operator's choice**, unset stays `report` in every release, |
| because that box's every row is flat on purpose. Flat by choice is |
| either `CARLOS_LAYOUT_DEFAULT=flat` with no recorded intent, or a |
| @@ -946,46 +980,45 @@ these seams; nothing here changes them. |
| - **`carlos status --layout`** shows the policy and each refused or |
| allowlisted host beside its layout word. |
| -### 4.4 Decision D15: sequencing, so no box loses a workload on upgrade |
| - |
| -1. **Release N** carries §3, §4 and §5, with `CARLOS_FLAT_EXEC` defaulting |
| - to `report` and `CARLOS_NESTED_WORKERS` off. Rolling it changes nothing |
| - that runs; status starts showing every flat spawn a `refuse` box would |
| - stop. |
| -2. **Per box**, after `CARLOS_NESTED_WORKERS=on` has moved its sidecars |
| - and its inventory (D14) lists nothing, an operator sets |
| - `CARLOS_FLAT_EXEC=refuse` (with an allowlist if one is genuinely |
| - needed). Dev boxes first (§8.2). |
| -3. **Release N+1** changes two defaults, and the console guards each one |
| - separately, because they are different questions (R1-6): whether a box |
| - may still run flat, and whether its workers are ready to move. |
| - |
| - | Default that changes in N+1 | It changes nothing on a box when | Otherwise | |
| - |---|---|---| |
| - | `CARLOS_FLAT_EXEC` unset → `refuse` | the variable is set explicitly, or the effective intent is flat by choice (§4.1), or the inventory lists no row and the intent is known | blocked | |
| - | `CARLOS_NESTED_WORKERS` unset → `on` | the variable is set explicitly, or the box has no flat worker row | blocked | |
| - |
| - The system channel door refuses to move a channel to N+1 while any box |
| - that will follow it is blocked on either line, or is stale, or reports |
| - an inventory it could not complete, naming the boxes, the line and the |
| - hosts (copy to review). That is `layoutDowngradeBlocked`'s pattern |
| - (judge observed state; unknown blocks), applied to an upgrade. A box's |
| - history of spawns plays no part: a dormant row that never woke is in |
| - the inventory all the same. |
| -4. A box that needs the old behaviour after N+1 says so explicitly, and |
| - each opt-out covers only its own line: `CARLOS_FLAT_EXEC=report` keeps |
| - flat spawns running; `CARLOS_NESTED_WORKERS=off` keeps workers minted |
| - and kept flat. An operator who sets only the first still has workers |
| - move, and status says so beside the setting. |
| - |
| -Whether a worker is ready to move is not something a box can observe (a |
| -worker that computes `/data/<host>.db` looks the same as one that reads |
| -`CARLOS_SIDECAR_DB` until it starts nested). So the second line never |
| -infers readiness: on a box with worker rows, only an operator's explicit |
| -setting clears it, after §8.2 step 6's check. |
| - |
| -Claim-mode boxes are unaffected throughout: they already refuse every |
| -legacy launch shape before this check is reached. |
| +### 4.4 Decision D15: sequencing, so no box loses a workload on upgrade (rev 3: R1-6, R2-4) |
| + |
| +Revisions 1 and 2 had a later release change both defaults behind a |
| +console guard. Round 2 showed the guard is judged when the channel moves, |
| +while each box applies the release later, so a workload created in between |
| +gets both changes unassessed (R2-4). Paul's ruling (§11 Q6) is that a |
| +default change must not stop what was running. Revision 3 makes the |
| +change happen on the box, at the one moment it is known to stop nothing. |
| + |
| +**Decision.** |
| + |
| +- **`CARLOS_NESTED_WORKERS` unset means on**, from release N, subject to |
| + D11's condition and the box's effective intent being nested. A worker |
| + that opens `CARLOS_SIDECAR_DB` and uses `CARLOS_AGENT_SOCKET` moves |
| + without noticing. Every native sidecar on our deployments today is the |
| + operator's, and §8.2's check covers them before release N rolls. From |
| + release N the documented contract is those two variables (D4). |
| +- **`CARLOS_FLAT_EXEC` unset is a ratchet.** It reads `report` until the |
| + first time the agent finds the box's inventory (D14) empty and the |
| + effective intent nested, and from then on `refuse`. The agent records |
| + that moment in `/data/.carlos/flat-exec-refuse` (written once, atomically, |
| + never removed by any code path) and reads it at start. So: |
| + - nothing that runs when the ratchet turns is refused by it, because the |
| + inventory was empty; |
| + - a row that arrives later meets `refuse`, which is the property: a new |
| + worker is minted nested, a new instance is nested by default, and a |
| + foreign row an operator adds by hand is refused and named in status |
| + (D14), with `carlos add` or the allowlist as the repair; |
| + - a box whose intent is flat by choice (§4.1) never turns. |
| + The file lives in the agent's data directory, which only `carlos` |
| + processes in the host namespace can write. Once the ratchet has turned |
| + there are none but the agent. |
| +- **An explicit setting always wins**: `refuse` or `report`, read at agent |
| + start as before, and the ratchet file is then ignored but not removed. |
| +- The inventory, the ratchet's state and its time are in status, so an |
| + operator can see which boxes have turned and why one has not. |
| + |
| +No console guard, no later release flipping a default, and no `host.env` |
| +edit on any box. |
| ### 4.5 Decision D16: the stragglers |
| @@ -1004,7 +1037,11 @@ repair stays the box-local `carlos add`. |
| --- |
| -## 5. Decision D17: creation-time refusal |
| +## 5. Decision D17: creation-time refusal (deferred, §13) |
| + |
| +**Revision 3:** not in release N (Paul, §11 Q1). The follow-up takes this |
| +section as written plus R2-5's fix: a stale or incomplete box that the |
| +placement set cannot exclude blocks creation. |
| **Decision.** `handleAPISidecarCreate` refuses before any write when the |
| app's sidecars could not run isolated where they would be placed. The |
| @@ -1031,8 +1068,8 @@ the claim-mode create too: |
| serves the wrong pool or accepts inbound work, and block on an |
| irrelevant box. |
| - Each box reports a new additive field, `Metrics.Layout.Workers`: |
| - `nested` when its binary carries this spec, its workers floor is armed |
| - with a `carlos.prev` verdict (D11), `CARLOS_NESTED_WORKERS` is on, its |
| + `nested` when its binary carries this spec, D11's `carlos.prev` condition holds, |
| + `CARLOS_NESTED_WORKERS` is on, its |
| effective intent is nested, its nested gate answers, **and it can launch |
| a worker at all**: a box with claim control armed refuses every worker |
| launch (`workerbuild.go:111-113`) and reports `blocked: claim` (R1-8). |
| @@ -1061,7 +1098,11 @@ weakest box that may run it. |
| --- |
| -## 6. Decision D18: retiring the interim fence |
| +## 6. Decision D18: retiring the interim fence (deferred, §13) |
| + |
| +**Revision 3:** not in release N. The fence stays on every box until the |
| +follow-up builds the census below and an operator removes the fence under |
| +its own scoped approval. |
| **Decision.** `fence-flat-imds.sh` may be removed from a box when all four |
| hold on that box, and each is observable through the API (`carlos status`), |
| @@ -1131,9 +1172,9 @@ boxes and private services on the network. |
| | **The fence** (today, interim) | all of the above except metadata | reach metadata, for as long as it cannot leave its cgroup for an exempt one (§1.6; the fence's own hardening, recorded on a dev box by §9) | |
| | **Release N rolled** (§3 dark, `report`) | the same: nothing that runs has changed | (status now names every flat spawn) | |
| | **`CARLOS_NESTED_WORKERS=on`** on a box whose gate is ready | on that box, nothing from a new or moved sidecar: it sees its own directory only, no loopback, no private range, no metadata, no sibling pid. It can still get a flat process **if** the box mints flat (gate failing with no recorded intent, D10) or the row is foreign | read or write another tenant's files from its sidecar; call the agent socket for another host (D3) | |
| -| **`CARLOS_FLAT_EXEC=refuse`**, empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace | |
| -| **Console refusal on** | (no change in power) | create a record the platform would place unisolated or never run; this is honesty, the boundary is the line above | |
| -| **Fence retired** (§6) | nothing new: there is no flat child left for the fence to filter | | |
| +| **`refuse` in force** (explicit, or the ratchet turned), empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace | |
| +| **Console refusal on** (follow-up, D17) | (no change in power) | create a record the platform would place unisolated or never run; this is honesty, the boundary is the line above | |
| +| **Fence retired** (follow-up, §6) | nothing new: there is no flat child left for the fence to filter | | |
| ### 7.3 What remains after the last step |
| @@ -1187,75 +1228,56 @@ source is unreachable"; another provider's adapter supplies its own line. |
| --- |
| -## 8. Rollout |
| +## 8. Rollout (rev 3) |
| ### 8.1 Flags and defaults |
| -| Setting | Where | Release N | Release N+1 | Gone in | |
| -|---|---|---|---|---| |
| -| `CARLOS_NESTED_WORKERS` (`on`\|`off`) | box `host.env` | off: workers mint flat, the pass skips them, dispatch still honours any nested row. On does nothing until the workers floor is armed (D11) | on unless `off`; the N+1 channel guard (D15) | N+2 | |
| -| `CARLOS_FLAT_EXEC` (`refuse`\|`report`) | box `host.env` | `report` | `refuse` unless set, or unless the intent is flat by choice (§4.1); the N+1 channel guard (D15) | stays (it is the self-hoster's opt-out) | |
| -| `control/workers-protocol.json` | deployment bucket | published before any box turns workers on | stays | stays (a floor only rises) | |
| -| `CARLOS_FLAT_EXEC_ALLOW` | box `host.env` | empty | empty | stays | |
| -| `CONSOLE_REQUIRE_ISOLATED_WORKERS` | console env | off | on unless `false` | N+2 | |
| - |
| -Dark in N: every behaviour change except two. The dispatch in |
| -`StartWorkerStaged` is live in N, but no nested worker row can exist until |
| -a box arms the workers floor and turns `CARLOS_NESTED_WORKERS` on, so it is |
| -inert. **D9 is live in N on purpose**: the box-wide replicator stops |
| -replicating workers on the first generation after the roll (§3.9), and |
| -the shrink guard says so per host. D19's root fence is live too, and |
| -inert until a worker unit exists. |
| +| Setting | Where | Unset means | Explicit values | |
| +|---|---|---|---| |
| +| `CARLOS_NESTED_WORKERS` | box `host.env` | on (D15), effective only under D11's condition and a nested intent | `on`, `off` (keeps workers minted and kept flat) | |
| +| `CARLOS_FLAT_EXEC` | box `host.env` | the ratchet: `report` until the box's inventory is first empty under a nested intent, `refuse` from then on (D15); `report` forever under a flat-by-choice intent | `refuse`, `report` | |
| +| `CARLOS_FLAT_EXEC_ALLOW` | box `host.env` | empty | exact hosts (D12) | |
| + |
| +None of these needs setting on any of our boxes. Nothing is dark in |
| +release N except what D11 holds back until `carlos.prev` qualifies. |
| ### 8.2 Order |
| The house order: dev boxes, then the dev console, then a check, then |
| flagship boxes, then the flagship console. |
| -1. **Release N to dev boxes, then the dev console.** Nothing that runs |
| - changes. `carlos status --layout` lists sidecar rows as `flat` and the |
| - flat-exec inventory lists each. Each always-on worker's path leaves the |
| - box-wide config, named in the shrink guard's event line, and the next |
| - restore drill covers its lease lineage (D9). |
| -2. **Publish the workers floor** for the deployment, and wait until every |
| - dev box reports it armed with a `carlos.prev` verdict (D11). |
| -3. **Dev boxes: `CARLOS_NESTED_WORKERS=on`.** Watch sidecar rows turn |
| - `nested`, hibernating ones at once, always-on ones within the awake |
| - grace. The check is from inside: a throwaway sidecar in a dev account |
| +1. **Before the cut: the sidecar check.** Every native sidecar on dev and |
| + flagship (all the operator's, #370) is checked for a computed database |
| + path, a hard-coded agent socket, and loopback or private-address use |
| + (§3.4, §7.3). Each one that needs it ships a release reading |
| + `CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` first. Where an always-on |
| + worker's database is named in a box's `/etc/litestream.yml`, the |
| + operator regenerates it (`carlos ops litestream`, which D9 now excludes |
| + workers from) and restarts the replicator; until then D9's check holds |
| + that worker's move (§3.9). |
| +2. **Release N to dev boxes, then the dev console.** On each box: status |
| + shows the inventory; workers stay flat (`waiting for carlos.prev`); D9's |
| + exclusion and D19's fence are live. |
| +3. **Release N's patch (N.1) to dev boxes and the dev console.** Now |
| + `carlos.prev` is N, so D11's condition holds and dev boxes move their |
| + workers: hibernating ones at once, always-on ones within the awake |
| + grace. Verify from inside with a throwaway sidecar in a dev account |
| whose release lists `/data` and `/run/carlos`, reads `/proc/1/status`, |
| sends the probe's datagrams, opens `CARLOS_SIDECAR_DB`, calls |
| - `/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, and prints |
| - all of it to `carlos sidecars logs`; then sleeps, wakes on its timer, |
| - wakes on `carlos sidecars wake`, crashes once (restarted with backoff), |
| - is killed by SIGTERM from inside its own process once (a crash, |
| - restarted), and finishes once (parked). An always-on twin is moved by |
| - the grace stop, and its lease lineage continues under its host key. |
| - A standby box that has the twin's config and has never held its lease |
| - moves its row with config only, and the twin then fails over to it. |
| -4. **Dev boxes: `CARLOS_FLAT_EXEC=refuse`,** then an edge restart so the |
| - census can read `clean` (D18). A foreign row made on a dev box with |
| - `carlos add --db` is refused, status says so, and `carlos add` with the |
| - defaults repairs it. Record the §1.6 cgroup migration attempt on a dev |
| - box before and after: before, whether a flat child can move itself into |
| - `supervise` under the fence; after, there is no flat child to try. |
| -5. **Dev console: `CONSOLE_REQUIRE_ISOLATED_WORKERS=true`.** A create in a |
| - dev region whose eligible boxes report `nested` succeeds; with one |
| - eligible box's `CARLOS_NESTED_WORKERS` off, it is refused with |
| - `code: "isolation"`; with only an ineligible box (another pool) off, it |
| - succeeds. |
| -6. **The check before flagship:** establish, for every app with native |
| - sidecars on the flagship, whether its worker opens its database by |
| - computed path, uses the agent socket by a hard-coded path, or reaches |
| - anything on loopback or a private address (§3.4, §7.3). Each one that |
| - does ships a release that reads `CARLOS_SIDECAR_DB` and |
| - `CARLOS_AGENT_SOCKET` before its boxes turn nested workers on (§11 Q3). |
| -7. **Flagship:** release N to flagship boxes and console as in step 1; |
| - the workers floor as in step 2; then boxes region by region, console |
| - box last, steps 3 and 4. Then the **flagship console**: step 5. |
| -8. **Fence retirement** per box when §6's four conditions hold, dev first, |
| - under its own scoped approval. |
| -9. **Release N+1** cut and rolled in the same order, behind the console |
| - guard of D15. |
| + `/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, and prints it |
| + to `carlos sidecars logs`; then sleeps, wakes on its timer and on |
| + `carlos sidecars wake`, crashes once (restarted), is killed by SIGTERM |
| + from inside once (a crash, restarted), and finishes once (parked). Then |
| + watch each dev box's inventory empty and its ratchet turn to `refuse`. |
| + A foreign row made with `carlos add --db` is then refused and named. |
| +4. **Flagship:** N to boxes region by region, console box last, then the |
| + flagship console; then N.1 the same way. Watch each box move its workers |
| + and turn its ratchet. A box whose ratchet has not turned within the |
| + awake grace after N.1 is named in status with what holds it. |
| +5. **The fence** stays until its retirement follow-up (§13). |
| + |
| +N.1 can be whatever patch is next on `main`; its only job here is to make |
| +N the box's `carlos.prev`. |
| ### 8.3 Census |
| @@ -1269,14 +1291,10 @@ a census proposal through the #280 process in the same branch: |
| - `internal/host/units/carlos-instance-worker@.service` (a new embedded |
| unit; templates are never enabled, so `update-platform-binary.sh` needs |
| no new name); |
| -- `internal/host/systemapply.go` (D11: install, `restoreFiles`, `armFloor`) |
| - and the workers floor's root files beside `rootseedsfloor.go` |
| - (`rootworkersfloor.go`, its verdict and its boot read in |
| - `rootmovesfloor_boot.go`); |
| -- `internal/host/units/carlos-system-recover.service` (D11: the third |
| - floor test in `ExecStart`) (R1-2); |
| -- the `floor-allows-prev` verb, if the workers floor's verdict file needs |
| - a case of its own there; |
| +- `internal/host/systemapply.go` (D11, rev 3: install's refusal of a |
| + candidate below `WorkerLayoutFloor` while a worker host is named; |
| + `decidePrev` unchanged). The recovery unit, the guard and the floor files |
| + are not touched (R2-2); |
| - the probe, only if the plan gives it a worker-template line (it should |
| not need one: the namespace block is identical, and the identity test |
| holds that). |
| @@ -1356,14 +1374,23 @@ Unit and package level: |
| excludes an always-on worker and the restore census includes it; the |
| shrink guard accepts a path leaving for that reason and names it, and |
| still refuses any other shrink. |
| -- D11: the workers floor arms like the seeds floor; the pass and the |
| - reconciler treat workers as flat until it is armed with a verdict; |
| - install refuses a candidate below it; `restoreFiles` refuses a |
| - `carlos.prev` below it and keeps the journal; the recovery unit's |
| - `ExecStart`, run by the real shell against fixture files, execs the |
| - current binary when the verdict does not name the floor (the existing |
| - recover-unit tests' shape); the row test names a worker host with only a |
| - journal. |
| +- D11 (rev 3): the pass and the reconciler treat workers as flat while |
| + `carlos.prev.version` is missing, unparseable or below the floor, and |
| + status says `waiting for carlos.prev`; install refuses a candidate below |
| + the floor, or one lacking the worker template, while a worker host is |
| + named, including a host with only a journal. |
| +- D15's ratchet: unset reads `report` with a non-empty inventory; turns |
| + once on the first empty inventory under a nested intent, writes its file |
| + once, and stays `refuse` across an agent restart and a later non-empty |
| + inventory; never turns under a flat-by-choice intent; an explicit value |
| + wins and leaves the file. |
| +- D19 replacement (R2-1): a hold `ReplaceWorker` whose worker start |
| + answers uncertain restores nothing and stops the entry; an overlap one |
| + falls back only after `RoleGone`. |
| +- D9's running-replicator check (R2-3): a move is deferred while |
| + `carlos-litestream` is active and started before the config's mtime, or |
| + its config names the file; it proceeds when the replicator is inactive or |
| + restarted on a config without it. |
| - Flat-exec policy: each value, unset under both layout defaults, an |
| unknown value, an invalid allowlist entry; the refusal from both seams; |
| the enumeration test over `exec.Command` in `internal/activator`; the |
| @@ -1376,20 +1403,15 @@ Unit and package level: |
| `contract_test.go`) and never reach `fleet.json`; worker rows in the |
| layout report; a dormant flat row that has never woken is in the |
| inventory. |
| -- D18's census: a fixture subtree with only the agent's pid is `clean`; a |
| +- (follow-up) D18's census: a fixture subtree with only the agent's pid is `clean`; a |
| pid in the agent's own leaf that the agent did not start is |
| `populated`; an unreadable `cgroup.procs`, a missing root, and a box with |
| no delegation are `unknown`. |
| -- Console: the create door's refusal order (after the region, before the |
| - cap and the claim branch), every case of D17 including a ready box in |
| - the wrong pool, an inbound box for a `no-inbound` record, an unready box |
| - outside the set, and a claim-mode box (`blocked: claim`); |
| - `code: "isolation"`; the downgrade guard's worker floor; the N+1 channel |
| - guard of D15, each line separately, with a dormant row and with |
| - `CARLOS_FLAT_EXEC=report` set but `CARLOS_NESTED_WORKERS` unset. |
| -- `systemapply`: a candidate without the worker template, or below the |
| - workers floor, is refused while a worker host is named, and installs |
| - when none is. |
| +- Console: the downgrade guard's worker floor. (D17's door tests go with |
| + the follow-up.) |
| +- `systemapply`: a candidate without the worker template, or below |
| + `WorkerLayoutFloor`, is refused while a worker host is named, and |
| + installs when none is. |
| Kernel and box level (root runner or a dev box; these need real cgroup v2 |
| and systemd): |
| @@ -1404,7 +1426,7 @@ and systemd): |
| leaving the unit `failed` and the next start succeeding, |
| `systemctl restart carlos-edge` stopping it and the boot pass starting |
| an always-on one again. |
| -- D18's census on a real box: `clean` after an edge restart under |
| +- (follow-up) D18's census on a real box: `clean` after an edge restart under |
| `refuse`; `populated` with a stray process left in the agent's leaf. |
| The full gate runs once before handoff; locally, only the changed |
| @@ -1427,44 +1449,27 @@ packages' tests. |
| --- |
| -## 11. Open questions for Paul |
| - |
| -1. **Pause sidecar creation now?** Turning `CONSOLE_REQUIRE_ISOLATED_WORKERS` |
| - on before any box reports nested workers refuses every sidecar create |
| - until nested workers roll. It stops new flat sidecars today at the cost |
| - of the feature for that time. Existing sidecars keep running. |
| -2. **Treat what the flat tier could reach as exposed?** B1 was confirmed |
| - live. The console audit (#370) found every sidecar ever created was the |
| - operator account's, so no tenant has used the sidecar path. Round 1 |
| - reports the hardened fence closes the cgroup escape (§1.6). Flat |
| - instance rows (foreign rows, rows the pass refused or deferred) had the |
| - same reach, though, and before the fence nothing stopped metadata. That |
| - would put the box roles' reachable secrets (regional SSM, the deployment |
| - bucket, and on the console box the IAM user minting) in scope for |
| - rotation, and possibly a look at the bucket's write history. That is an |
| - incident call; the inventory of D14, once release N is out, says which |
| - tenant hosts ran flat. |
| -3. **Who confirms the apps that use native sidecars are ready to move?** |
| - §8.2 step 6 needs a worker that reads `CARLOS_SIDECAR_DB` and |
| - `CARLOS_AGENT_SOCKET` and does not rely on loopback, before its flagship |
| - boxes turn nested workers on. Every sidecar today is the operator's, so |
| - this is our own check rather than a customer's. And does the operator |
| - app the fence exempts need the flat allowlist at all, or can it move to |
| - a unit or to the nested tier first, so that every box's allowlist can |
| - be empty? |
| -4. **Land D9 ahead of the rest?** The double replication of always-on |
| - workers (§1.8, R1-3) is live on main now and risks a corrupt replica |
| - for every always-on sidecar. D9 is small (one predicate in `internal/ops`, |
| - the shrink guard's exception, the restore census) and independent of |
| - isolation. Recommendation: its own branch and release now, ahead of |
| - release N, with a restore drill of each always-on worker's lease lineage |
| - after it rolls. |
| -5. **The workers floor is a third protocol floor.** It adds a third test to |
| - the recovery unit's `ExecStart` and a third published control object. |
| - The alternative is weaker: R9's unit-set test cannot hold the recovery |
| - unit's own `carlos.prev` exec (R1-2). Recommendation: accept the floor. |
| - |
| ---- |
| +## 11. Questions for Paul, and his rulings (2026-10-24) |
| + |
| +1. **Pause sidecar creation now?** Ruled: no ("don't worry about pausing"). |
| + The console check (D17) is deferred to a follow-up (§13). |
| +2. **Treat what the flat tier could reach as exposed?** Still Paul's |
| + incident call, outside this build. Every sidecar ever created was the |
| + operator's (#370); flat instance rows had the same reach, and before the |
| + fence nothing stopped metadata. Release N's inventory shows which tenant |
| + hosts ran flat on each box. |
| +3. **The operator app the fence exempts:** moves with everything else; no |
| + box carries an allowlist entry (recommendation taken under "just do |
| + this"). |
| +4. **Land D9 ahead of the rest?** Ruled by "just do this": D9 ships in |
| + release N with the rest, as one release, rather than separately. |
| +5. **A third protocol floor?** Superseded: revision 3's D11 needs none |
| + (R2-2). |
| +6. **How defaults reach existing boxes:** nothing that runs is stopped by a |
| + default change (recommendation taken). Revision 3 does it with the |
| + ratchet of D15 instead of keeping old defaults on established boxes, |
| + because the ratchet needs no per-box `host.env` change and turns every |
| + box once it is safe. |
| ## 12. Review disposition |
| @@ -1499,3 +1504,33 @@ this spec removes. Root-held evidence of every past materialisation on |
| every box would be a new mechanism for a loss the attacker can cause |
| anyway. What the finding rightly caught, the incomplete inventory and the |
| companion files, is fixed. |
| + |
| + |
| +### 12.2 Round 2 (Astra, `CODEX-FINDINGS-SIDECAR-ISOLATION-R2.md`, at 730ba093) |
| + |
| +Verdict: reject. Round 1's findings fixed or fixed in part; R1-4's decline |
| +judged reasonable. Citations spot-checked and held. |
| + |
| +| Finding | | Disposition | Where | |
| +|---|---|---|---| |
| +| R2-1 | P1: an uncertain hold replacement falls back over a migrated database | **Accepted.** The worker arm's uncertain start answers "may have run" to `startMayHaveRun` too; `ReplaceWorker` treats it as executed: a hold attempt restores nothing and stops the entry through the fenced stop and `RoleGone` | D19, §3.5.1 | |
| +| R2-2 | P1: the root guard's copy ignores a new floor | **Accepted, differently.** No new floor: workers move only once `carlos.prev` is itself worker-capable, and install keeps both files at or above the floor; the guard picks one of those two, so it needs no change | D11, §3.11 | |
| +| R2-3 | P1: a regenerated config is not proof the replicator let go | **Accepted.** An always-on worker's move waits until the running replicator started after a config that does not name the file | D9, §3.9 | |
| +| R2-4 | P2: defaults change during the rollout window | **Accepted, by Paul's ruling (Q6).** No release flips a default behind a console guard; the flat-exec default is a per-box ratchet that turns only when nothing would be refused | D15, §4.4 | |
| +| R2-5 | P2: D17 drops stale boxes that still run workers flat | **Deferred with D17.** The console check is not the boundary and is out of this release; the follow-up takes R2-5's fix (stale eligible boxes block) | §5, §13 | |
| + |
| +--- |
| + |
| +## 13. What release N carries, and what follows |
| + |
| +**Release N (and its patch N.1, §8.2):** D1 to D14, D16 and D19, with D11 |
| +and D15 as in revision 3. That closes B1: a native sidecar runs nested on |
| +every box whose `carlos.prev` qualifies, and once a box's ratchet turns, |
| +nothing on it runs in the host namespace unless an operator allowlists it. |
| + |
| +**Follow-ups, each its own branch, filed as amadan discussions:** |
| + |
| +- **D17**, the console's create-time check, with R2-5's fix. |
| +- **D18**, retiring the interim fence: the process census in status, then |
| + the carloku-infrastructure change under its own scoped approval. |
| +- Copy and docs for anything those add. |