amadan

carlos / platform Public

Sidecar isolation spec, revision 2: answers Astra round 1

All nine round 1 findings accepted (R1-4 in part). New D19 orders a
worker's starts against its stops (the #292 fence, widened); D11 becomes a
workers protocol floor that install, rollback and the recovery unit obey;
D9 makes lease replication every worker's only replication, fixing a
double-replication defect live on main; D8 narrows the no-database move;
D15 judges registry inventory; D17 uses the reconciler's placement set;
D18 needs a complete process census. §12 is the disposition. Round 1
findings renamed to the brief's -R1 convention.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
claude 730ba093dffd58338aac556737d5b0fbbe994fd9 parent 2a2517d
2 files changed, +593 −184
  • docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md +593 −184
  • docs/superpowers/specs/{CODEX-FINDINGS-SIDECAR-ISOLATION.md => CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md} +0 −0

CI failed — run details

CI log 
=== gate ===
.amadan/test-gate.sh
go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0
go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5
go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0
go: downloading github.com/aws/smithy-go v1.28.1
go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0
go: downloading github.com/superfly/ltx v0.5.1
go: downloading golang.org/x/net v0.56.0
go: downloading modernc.org/sqlite v1.55.0
go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6
go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0
go: downloading golang.org/x/crypto v0.53.0
go: downloading github.com/oschwald/maxminddb-golang v1.13.1
go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0
go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1
go: downloading golang.org/x/sys v0.46.0
go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6
go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6
go: downloading github.com/benbjohnson/litestream v0.5.14
go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0
go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2
go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0
go: downloading github.com/pion/logging v0.2.4
go: downloading github.com/pion/stun/v3 v3.0.1
go: downloading github.com/pion/turn/v4 v4.1.4
go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6
go: downloading github.com/pierrec/lz4/v4 v4.1.22
go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40
go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16
go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4
go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4
go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8
go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12
go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40
go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40
go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23
go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31
go: downloading github.com/pion/dtls/v3 v3.0.7
go: downloading github.com/pion/transport/v3 v3.0.8
go: downloading golang.org/x/sync v0.21.0
go: downloading github.com/prometheus/client_golang v1.17.0
go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18
go: downloading github.com/lmittmann/tint v1.1.3
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading github.com/pion/transport/v4 v4.0.1
go: downloading github.com/pion/randutil v0.1.0
go: downloading github.com/wlynxg/anet v0.0.5
go: downloading github.com/cespare/xxhash/v2 v2.3.0
go: downloading github.com/beorn7/perks v1.0.1
go: downloading google.golang.org/protobuf v1.36.10
go: downloading github.com/prometheus/common v0.45.0
go: downloading github.com/prometheus/client_model v0.5.0
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading golang.org/x/text v0.38.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0
go: downloading modernc.org/sqlite v1.55.0
go: downloading github.com/pierrec/lz4/v4 v4.1.22
go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0
go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0
go: downloading github.com/klauspost/compress v1.18.0
go: downloading golang.org/x/text v0.38.0
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading google.golang.org/protobuf v1.36.10
go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0
go: downloading modernc.org/libc v1.74.1
go: downloading modernc.org/libc v1.74.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/google/uuid v1.6.0
go: downloading modernc.org/mathutil v1.7.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/google/uuid v1.6.0
go: downloading modernc.org/mathutil v1.7.1
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
=== launchctx (2 s)
ok  	amadan.net/carlos/platform.git/launchctx	0.011s
?   	amadan.net/carlos/platform.git/launchctx/controlname	[no test files]
?   	amadan.net/carlos/platform.git/launchctx/wireid	[no test files]
=== vet (43 s)
go: downloading modernc.org/sqlite v1.55.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0
go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0
go: downloading github.com/pierrec/lz4/v4 v4.1.22
go: downloading golang.org/x/tools v0.47.0
go: downloading github.com/klauspost/compress v1.18.0
go: downloading golang.org/x/text v0.38.0
go: downloading google.golang.org/protobuf v1.36.10
go: downloading github.com/prometheus/procfs v0.12.0
go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0
go: downloading golang.org/x/mod v0.37.0
go: downloading modernc.org/libc v1.74.1
go: downloading modernc.org/memory v1.11.0
go: downloading github.com/dustin/go-humanize v1.0.1
go: downloading github.com/google/uuid v1.6.0
go: downloading modernc.org/mathutil v1.7.1
go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec
=== gofmt (0 s)
gofmt -l .: clean
=== rest (190 s)
?   	github.com/carlosframework/platform/cmd/changelog	[no test files]
ok  	github.com/carlosframework/platform/cmd/console	3.936s
ok  	github.com/carlosframework/platform/cmd/moveprobe	12.209s
ok  	github.com/carlosframework/platform/cmd/zddprobe	0.837s
ok  	github.com/carlosframework/platform/internal/activator	59.464s
ok  	github.com/carlosframework/platform/internal/agentsock	0.035s
ok  	github.com/carlosframework/platform/internal/alerting	0.007s
ok  	github.com/carlosframework/platform/internal/analytics	0.022s
ok  	github.com/carlosframework/platform/internal/appconfig	3.287s
ok  	github.com/carlosframework/platform/internal/apperr	0.178s
ok  	github.com/carlosframework/platform/internal/applog	0.355s
ok  	github.com/carlosframework/platform/internal/awsinv	0.012s
ok  	github.com/carlosframework/platform/internal/boxchoice	0.010s
ok  	github.com/carlosframework/platform/internal/boxid	1.804s
?   	github.com/carlosframework/platform/internal/boxid/boxidtest	[no test files]
?   	github.com/carlosframework/platform/internal/boxid/internal/signtest	[no test files]
ok  	github.com/carlosframework/platform/internal/boxvolumes	0.052s
ok  	github.com/carlosframework/platform/internal/caps	0.036s
ok  	github.com/carlosframework/platform/internal/certwarm	0.008s
ok  	github.com/carlosframework/platform/internal/cgroup	0.928s
ok  	github.com/carlosframework/platform/internal/changelog	0.839s
ok  	github.com/carlosframework/platform/internal/claimactivation	1.711s
ok  	github.com/carlosframework/platform/internal/clamd	0.094s
?   	github.com/carlosframework/platform/internal/cliflag	[no test files]
?   	github.com/carlosframework/platform/internal/controlname	[no test files]
?   	github.com/carlosframework/platform/internal/controlsock	[no test files]
ok  	github.com/carlosframework/platform/internal/dbdoor	8.135s
ok  	github.com/carlosframework/platform/internal/dbdoor/dbdoortest	0.042s
ok  	github.com/carlosframework/platform/internal/dbquery	0.011s
ok  	github.com/carlosframework/platform/internal/dbreads	2.143s
?   	github.com/carlosframework/platform/internal/dbreads/dbreadstest	[no test files]
ok  	github.com/carlosframework/platform/internal/dbverify	0.006s
ok  	github.com/carlosframework/platform/internal/dnsrec	0.006s
ok  	github.com/carlosframework/platform/internal/docsite	0.036s
ok  	github.com/carlosframework/platform/internal/domains	2.997s
ok  	github.com/carlosframework/platform/internal/doorkey	0.016s
ok  	github.com/carlosframework/platform/internal/economics	0.006s
ok  	github.com/carlosframework/platform/internal/edge	49.026s
?   	github.com/carlosframework/platform/internal/edgeproof	[no test files]
ok  	github.com/carlosframework/platform/internal/edgesfile	0.010s
ok  	github.com/carlosframework/platform/internal/fleetapi	0.171s
ok  	github.com/carlosframework/platform/internal/gate	1.704s
ok  	github.com/carlosframework/platform/internal/genselect	0.017s
ok  	github.com/carlosframework/platform/internal/geo	0.016s
ok  	github.com/carlosframework/platform/internal/geoip	0.038s
ok  	github.com/carlosframework/platform/internal/historyreadback	0.221s
ok  	github.com/carlosframework/platform/internal/host	142.417s
ok  	github.com/carlosframework/platform/internal/hostactors	6.571s
ok  	github.com/carlosframework/platform/internal/hostactors/awsmembership	0.018s
ok  	github.com/carlosframework/platform/internal/hostbox	3.284s
ok  	github.com/carlosframework/platform/internal/hostclaim	0.787s
ok  	github.com/carlosframework/platform/internal/hostenv	0.018s
ok  	github.com/carlosframework/platform/internal/hostevidence	0.018s
ok  	github.com/carlosframework/platform/internal/hostexec	5.576s
ok  	github.com/carlosframework/platform/internal/hostfence	0.050s
ok  	github.com/carlosframework/platform/internal/hostfiles	10.422s
ok  	github.com/carlosframework/platform/internal/hostfloor	0.026s
ok  	github.com/carlosframework/platform/internal/hostguard	1.051s
ok  	github.com/carlosframework/platform/internal/hostops	2.014s
?   	github.com/carlosframework/platform/internal/hostops/fencerelease	[no test files]
ok  	github.com/carlosframework/platform/internal/hostops/internal/fencerelease	0.017s
ok  	github.com/carlosframework/platform/internal/hostquarantine	0.016s
ok  	github.com/carlosframework/platform/internal/hostrelocate	0.609s
ok  	github.com/carlosframework/platform/internal/hostresolve	0.151s
ok  	github.com/carlosframework/platform/internal/hostwrite	0.017s
ok  	github.com/carlosframework/platform/internal/infraview	0.018s
ok  	github.com/carlosframework/platform/internal/instancectl	5.274s
ok  	github.com/carlosframework/platform/internal/instanceexec	2.857s
ok  	github.com/carlosframework/platform/internal/instancefs	0.021s
ok  	github.com/carlosframework/platform/internal/instancekey	0.017s
ok  	github.com/carlosframework/platform/internal/instanceupdates	0.026s
ok  	github.com/carlosframework/platform/internal/layout	1.794s
ok  	github.com/carlosframework/platform/internal/lbhealth	0.020s
ok  	github.com/carlosframework/platform/internal/lease	10.105s
ok  	github.com/carlosframework/platform/internal/lease/leasetest	0.539s
ok  	github.com/carlosframework/platform/internal/ledger	0.017s
ok  	github.com/carlosframework/platform/internal/mail	0.241s
ok  	github.com/carlosframework/platform/internal/maintenance	0.098s
ok  	github.com/carlosframework/platform/internal/maintenanceexec	5.553s
ok  	github.com/carlosframework/platform/internal/maintenanceguard	0.108s
ok  	github.com/carlosframework/platform/internal/metrics	0.005s
ok  	github.com/carlosframework/platform/internal/moves	29.051s
ok  	github.com/carlosframework/platform/internal/moves/archive	0.992s
ok  	github.com/carlosframework/platform/internal/objstore	0.263s
ok  	github.com/carlosframework/platform/internal/ops	0.188s
ok  	github.com/carlosframework/platform/internal/park	0.269s
ok  	github.com/carlosframework/platform/internal/plans	0.010s
ok  	github.com/carlosframework/platform/internal/privatefile	0.004s
ok  	github.com/carlosframework/platform/internal/provision	50.141s
ok  	github.com/carlosframework/platform/internal/publicrelease	0.010s
?   	github.com/carlosframework/platform/internal/regionname	[no test files]
ok  	github.com/carlosframework/platform/internal/registry	5.718s
ok  	github.com/carlosframework/platform/internal/relay	7.206s
ok  	github.com/carlosframework/platform/internal/release	29.435s
--- FAIL: TestAPointerClearRequeuesBeforeTheNextChannelIsRead (0.00s)
    manager_test.go:1185: the cleared build was requeued only after every channel was read
FAIL
FAIL	github.com/carlosframework/platform/internal/rollout	4.122s
ok  	github.com/carlosframework/platform/internal/rolloutwords	0.003s
ok  	github.com/carlosframework/platform/internal/rootaws	0.007s
ok  	github.com/carlosframework/platform/internal/rootparticipation	0.067s
ok  	github.com/carlosframework/platform/internal/rootstate	0.027s
ok  	github.com/carlosframework/platform/internal/runs	0.563s
ok  	github.com/carlosframework/platform/internal/s3compat	0.005s
ok  	github.com/carlosframework/platform/internal/safefile	0.004s
ok  	github.com/carlosframework/platform/internal/schedule	2.834s
ok  	github.com/carlosframework/platform/internal/seeds	0.013s
ok  	github.com/carlosframework/platform/internal/sim/engines	3.128s
ok  	github.com/carlosframework/platform/internal/sim/hostlife	10.241s
ok  	github.com/carlosframework/platform/internal/sim/litestreamfacts	3.877s
ok  	github.com/carlosframework/platform/internal/status	0.094s
ok  	github.com/carlosframework/platform/internal/steering	0.006s
ok  	github.com/carlosframework/platform/internal/stopevidence	0.311s
ok  	github.com/carlosframework/platform/internal/svcrole	0.002s
ok  	github.com/carlosframework/platform/internal/telemetry	0.005s
ok  	github.com/carlosframework/platform/internal/termfence	0.009s
ok  	github.com/carlosframework/platform/internal/tunnelclient	0.004s
ok  	github.com/carlosframework/platform/internal/unixdial	0.002s
ok  	github.com/carlosframework/platform/internal/updatewords	0.007s
ok  	github.com/carlosframework/platform/internal/usage	0.005s
ok  	github.com/carlosframework/platform/internal/version	0.006s
ok  	github.com/carlosframework/platform/internal/vet	1.758s
FAIL
=== internal_console@0 (83 s)
ok  	github.com/carlosframework/platform/internal/console	81.554s
=== internal_console@1 (48 s)
ok  	github.com/carlosframework/platform/internal/console	46.657s
=== internal_console@2 (43 s)
ok  	github.com/carlosframework/platform/internal/console	41.152s
=== internal_console@3 (47 s)
ok  	github.com/carlosframework/platform/internal/console	44.869s
=== internal_console@4 (47 s)
ok  	github.com/carlosframework/platform/internal/console	45.604s
=== internal_console@5 (48 s)
ok  	github.com/carlosframework/platform/internal/console	45.771s
=== internal_console@6 (44 s)
ok  	github.com/carlosframework/platform/internal/console	41.813s
=== internal_console@7 (50 s)
ok  	github.com/carlosframework/platform/internal/console	48.232s
=== root@0 (113 s)
ok  	github.com/carlosframework/platform	111.163s
=== root@1 (84 s)
ok  	github.com/carlosframework/platform	82.459s
=== root@2 (94 s)
ok  	github.com/carlosframework/platform	92.127s

FAILED: rest
make: *** [Makefile:19: test] Error 1

amadan: step gate failed: exit status 2

amadan: job failed at step gate — its output is above, not at the tail of this log
diff --git a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
index 83da7c395..aa4f7c01c 100644
--- a/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
+++ b/docs/superpowers/specs/2026-10-24-sidecar-isolation-design.md
@@ -1,11 +1,25 @@
# Sidecar isolation: native sidecars on the nested tier, and no flat exec by default
-**Date:** 2026-10-24 · **Status:** draft, revision 1, for an independent
+**Date:** 2026-10-24 · **Status:** draft, revision 2, for an independent
review (brief: `CODEX-REVIEW-BRIEF-SIDECAR-ISOLATION.md`) and then Paul ·
**Requested by:** security review finding B1, confirmed live 2026-10-24 ·
**Main at:** fef28dc5 (every `file:line` below is on that commit unless it
says otherwise)
+**Revision 2** answers Astra's round 1 (`CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md`,
+reviewed at 1c029ee5, verdict reject: four P1s, five P2s). Every finding's
+citations were checked against main and held. Each accepted finding is
+marked **(R1-n)** where the text answers it; §12 is the disposition table,
+with the one part declined and why. The largest changes: worker starts are
+ordered against stops like the run unit's (D19, new); downgrade is held by a
+workers protocol floor that the recovery unit also obeys (D11, rewritten);
+every worker replicates through its lease only, which also fixes a
+double-replication defect live on main today (D9, rewritten); the move of a
+row with no database is narrowed (D8); the upgrade guard judges inventory,
+not spawn history (D15); the create check uses the reconciler's own
+placement set (D17); and fence retirement needs a complete process census
+(D18).
+
**Builds on:** the isolated-exec design
(`2026-09-11-isolated-exec-design.md`, revision 17; "the nested design"
below), its plan 4 amendments (`../plans/2026-09-25-isolated-exec-4-layout-amendments.md`,
@@ -45,11 +59,12 @@ one. This spec closes that in four moves:
time, from what the target region's boxes report, instead of writing a
record that lands flat or never runs (§5).
4. **The interim fence retires** on a box once nothing flat can run there:
- the policy refuses, the allowlist is empty, no flat child cgroup is
- populated, and the nested probe's metadata line is enforced (§6).
+ the policy refuses, the allowlist is empty, a complete census of the
+ edge's process tree finds nothing but the agent and its own children,
+ and the nested probe's metadata line is enforced (§6).
§7 is the threat model, step by step. §8 is the rollout, dev first. §11
-lists three questions that are Paul's.
+lists the questions that are Paul's. §12 is the review disposition.
The decisions, for scanning (each is argued where it is made):
@@ -60,19 +75,20 @@ The decisions, for scanning (each is argued where it is made):
| D3 | A nested worker's agent socket is a per-host listener in its own run directory | §3.3 |
| D4 | Every worker is told its database path (`CARLOS_SIDECAR_DB`), flat and nested | §3.4 |
| D5 | `StartWorkerStaged` dispatches on layout; the nested arm passes the staged build to root | §3.5 |
-| D6 | `unitProc` answers `Crashed()` from systemd's result, never from the exit record | §3.6 |
+| D19 | A worker's starts are ordered against its stops, and a failed wake keeps the lease until the worker unit is proven gone (rev 2) | §3.5.1 |
+| D6 | A nested worker's crash is any unsuccessful result its own invocation reports; the template marks no signal exit successful (rev 2) | §3.6 |
| D7 | The sidecar reconciler mints nested exactly when the instance reconciler does | §3.7 |
-| D8 | The layout pass moves worker rows; a row with nothing on this box moves by row alone | §3.8 |
-| D9 | An always-on worker's box-wide replica starts a new lineage at its nested path | §3.9 |
+| D8 | The layout pass moves worker rows with their full extras inventory; a row whose database and its companions are absent on both sides moves without it (rev 2) | §3.8 |
+| D9 | Every worker replicates through its lease only; the box-wide replicator and the restore drills exclude workers (rev 2) | §3.9 |
| D10 | A failed nested gate never reopens flat for a worker | §3.10 |
-| D11 | Downgrade below the worker layout is refused by unit set on the box and by floor at the console | §3.11 |
+| D11 | A workers protocol floor, armed before any worker moves, holds every install, rollback and recovery selection (rev 2) | §3.11 |
| D12 | `CARLOS_FLAT_EXEC=refuse\|report` plus an exact-host allowlist, read from `host.env` | §4.1 |
| D13 | The refusal sits beside the claim refusal in both spawn seams, with its own error | §4.2 |
-| D14 | Refusals are reported per host with closed reason words, like config refusals | §4.3 |
-| D15 | Ship `report`; operators switch boxes to `refuse`; the next release defaults to `refuse` behind a console guard | §4.4 |
+| D14 | Refusals and a registry inventory of what `refuse` would stop are reported per host (rev 2) | §4.3 |
+| D15 | Ship `report`; operators switch boxes; N+1 changes a default only on boxes where the change stops nothing, judged from inventory (rev 2) | §4.4 |
| D16 | A refused managed row nudges the layout pass for that host | §4.5 |
-| D17 | Creation is refused unless every fresh box in scope reports nested workers ready | §5 |
-| D18 | The fence retires per box on four observable conditions; an allowlisted host keeps it | §6 |
+| D17 | Creation is refused unless the reconciler's own placement set has a fresh box and every one reports nested workers runnable (rev 2) | §5 |
+| D18 | The fence retires per box on four observable conditions, the third a complete process census; an allowlisted host keeps it (rev 2) | §6 |
---
@@ -177,24 +193,26 @@ As uid `carlos` in the host namespace, with the edge's own view of the box:
- loopback listeners, the VPC and the instance metadata service, and
through it the box role.
-### 1.6 The interim fence, and why it is weaker than it looks
+### 1.6 The interim fence, and what it does not cover
`carloku-infrastructure scripts/fence-flat-imds.sh` (not in this
-repository; described from the finding) drops metadata traffic from the
-edge's cgroup except from the agent's `supervise` leaf and one exempt
-operator app. Two limits, the second not in the finding:
+repository; described from the finding and the round 1 review) drops
+metadata traffic from the edge's cgroup except from the agent's
+`supervise` leaf. Two limits:
- It does nothing for the `/data` reads and writes or the cgroup writes
above.
-- **If it keys on cgroup path, a flat child can step out of it.** Cgroup v2
- lets a process move a pid between two cgroups when it can write
- `cgroup.procs` in both and in their common ancestor. `Delegate=` chowns
- the edge unit's cgroup and its `cgroup.procs` to `carlos`, and every
- directory below it is made by the agent as `carlos`. A flat child can
- therefore write its own pid into `supervise/cgroup.procs` (or the exempt
- app's cgroup) and open a socket there. The plan's first box task
- verifies this on a dev box (§9, box level) before anyone relies on the
- fence for anything.
+- **It keys on cgroup path**, so it holds only while a flat child cannot
+ move its pid into an exempt cgroup. Revision 1 argued that it could
+ (cgroup v2 allows the move to a writer of `cgroup.procs` at both ends and
+ their common ancestor, and `Delegate=` hands the edge's subtree to
+ `carlos`). The round 1 review reports that the fence as hardened since
+ makes `supervise/cgroup.procs` unwritable by `carlos` and exempts nothing
+ else, which closes that move (the kernel's destination and
+ common-ancestor checks; systemd's delegation ownership). This spec does
+ not rely on the fence either way. The plan's first box task still records
+ the attempt on a dev box (§9, box level), because the fence is the only
+ control between now and step 7 of §8.2.
The repository already reads the box's metadata gate once:
`nftGateUIDs` in the root adopt pass (`internal/host/adopt_advice.go:179-199`)
@@ -232,9 +250,21 @@ native sidecar has ever started through it.
(`backend_exec.go:3049-3130`; `workerEnv`, `internal/activator/activator.go:3321-3328`).
A worker that opens a database must be computing `/data/<host>.db`
itself, which stops working the moment its row is nested (§3.4).
-- The fence may be bypassable by cgroup migration (§1.6).
+- The fence keys on cgroup path; whether a flat child can leave its cgroup
+ is the fence's own hardening, not this spec's (§1.6).
- The flat tier is not only a cross-tenant read of flat siblings: it is a
write into nested siblings (§1.5).
+- **Every always-on worker is replicated twice today (R1-3).** A worker's
+ wake trades its lease for running replication unconditionally
+ (`activator.go:2602`), and always-on supervision keeps that lease and
+ replicator for as long as the row exists (`activator.go:4599-4627`). The
+ box-wide generator excludes only hibernating rows (`litestream.go:356-357`),
+ so it also replicates the same file, under its path key. The same file
+ warns that two replicators on one WAL fight over checkpointing and can
+ corrupt the replica (`litestream.go:230-234`). The restore drill's
+ census has the same `Hibernate` test (`internal/ops/hibernate_verify.go:45-60`).
+ This is a live defect, independent of isolation; D9 fixes it and §11 Q4
+ asks whether to land that part first.
---
@@ -268,6 +298,18 @@ Reopened, each because this problem forces it:
at `internal/layout/pass.go:347-355`). It is code, not a spec ruling, but
the refusal policy turns it from a cosmetic skip into an outage, so it
is named here (D8).
+- **#292's scope, "for the original run unit only"**
+ (`internal/instancectl/legacystart.go:11-34`). The ordered start and stop
+ was the owner's ruling for the run unit, the one role whose start the
+ agent abandoned with the lease at stake. A nested worker is the second
+ (R1-1). Widened to the worker role, unchanged in mechanism (D19).
+- **The box-wide replicator's `Hibernate` test** (`litestream.go:356-357`)
+ and the restore census that mirrors it (`hibernate_verify.go:45-60`).
+ Both meant "the activator owns this file's replication", and for workers
+ they have been wrong since always-on workers existed (R1-3, D9).
+- **Revision 1's own D1 "four lines and nothing else"**: the worker
+ template also drops the signal statuses from `SuccessExitStatus=` (R1-7,
+ D6).
Not reopened, though it looks as if it might need to be: plan 4 R14's
"defer an awake host". Plan 5 already bounded it with an awake grace
@@ -284,10 +326,11 @@ Not reopened, though it looks as if it might need to be: plan 4 R14's
`carlos-instance-worker@.service`. The `sidecar` role stays the legacy
co-spawn's, untouched.
-The template is the sidecar template with four lines changed and nothing
+The template is the sidecar template with five lines changed and nothing
else: `Description=` (`(worker)`), `ExecStart=/opt/carlos/carlos
instance-exec worker %I`, `ExecStopPost=/opt/carlos/carlos instance-exit
-worker %I` and `StandardOutput=file:/run/carlos/logs/%I.worker.sock`. It
+worker %I`, `StandardOutput=file:/run/carlos/logs/%I.worker.sock`, and no
+`SuccessExitStatus=` line (D6, R1-7). It
binds the same executable sources as the other original templates
(`/run/carlos-control/exe/%I/app` and `live`): a worker's host has no run
or sidecar unit of its own, and its seed (which `Seed` already dispatches
@@ -320,7 +363,8 @@ Root-side changes, all small: `names.go` gains `RoleWorker`,
`UnitPrefixWorker = "carlos-instance-worker@"`, its `prefixFor`,
`ParseRole` and `ParseUnitName` cases; the control service admits `worker`
on `/start`, `/stop`, `/reset` and `/sweep` with the run role's tier rule
-(always set the slice's tier: a worker's slice is its own) and validates a
+(always set the slice's tier: a worker's slice is its own) and the run
+role's start fence (D19), and validates a
named staged build with `validateStagedExe` as it does today
(`internal/instancectl/stagedexe.go:45-78`, whose registry view is
Kind-agnostic, `internal/host/claimstop_run.go:121-141`);
@@ -328,6 +372,11 @@ Kind-agnostic, `internal/host/claimstop_run.go:121-141`);
(`internal/instanceexec/exec.go:44-53`). No new verb, no new socket, no new
root relay. The census implications are in §8.3.
+A worker has one unit per host and no generation units: its replacement
+(`ReplaceWorker`, `internal/activator/workerswap.go:13-40`) and its respawn
+are stop-then-start on the same entry, so `/start` for the worker role
+always has `gen == 0`.
+
### 3.2 Decision D2: a worker row's paths, and `Classify`
**Decision.** A nested worker row has `DBPath = NestedDBPath(dataDir, host)`
@@ -437,35 +486,100 @@ looks for descendants in the right place.
`TestKillHostProvesANestedLayoutWorkersHostCgroup`
(`worker_gen_test.go:50-63`) is rewritten: a nested worker's cgroup is its
-unit's, and `KillHost` proves the slice, as for every other nested role.
+unit's, and `KillHost` alone is not the proof for a nested worker (D19).
+
+#### 3.5.1 Decision D19: a worker's start is ordered against its stop (R1-1)
+
+The failure: a nested worker's `/start` times out on the agent's side while
+the control service's handler is still running. The failed wake's cleanup
+sees an empty slice and gives the lease back; another box takes it; the
+late handler then submits the first box's start. Two workers own one
+database. This is exactly the race #292 closed for the run unit
+(`legacystart.go:11-34`), and the root fence today covers `RoleRun` only
+(`server.go:470-480`). On the agent side, a failed wake keeps its lease
+only for an unproven generation or a legacy start marked uncertain
+(`activator.go:2543-2566`), and the proof is hard-coded to the run unit
+(`LegacyRunGone`, `backend_exec.go:2595-2640`).
+
+**Decision.** The worker role gets the run unit's ordering, end to end:
+
+- **Root.** `handleStart` registers a worker start as an attempt
+ (`beginLegacyStart`) exactly as it does a run start; `/stop` of the
+ worker role fences the host for that role, cancels and joins the attempts
+ in flight within `legacyJoinBudget`, applies the accepted-connection
+ watermark, and only then stops the unit. The fence is keyed by host and
+ role, so a worker's stop never fences an instance's start or the
+ reverse. One that cannot be joined answers an error, and the agent keeps
+ the lease, as for the run unit.
+- **Agent.** The nested worker arm marks a start error uncertain the way
+ the run arm does (`legacyStartUncertain`, read by `legacyStartMayRun`,
+ `containment.go:74-79`) for
+ every outcome short of an answered refusal that ran nothing
+ (`launchRefused`, `containment.go:55-71`). The failed
+ wake's cleanup then takes the same branch as an uncertain legacy start
+ (`activator.go:2552-2554`): the prover, generalised from `LegacyRunGone`
+ to `RoleGone(ctx, rt, role)`, stops the worker unit through `/stop`
+ (which, once answered, means no earlier request can still submit a
+ start) and then requires the unit `inactive` or `failed` with its cgroup
+ empty or gone. Only that proof releases the lease. Anything short of it
+ keeps the lease, and the host stays this box's until a later stop
+ proves it.
+- **Respawns.** `respawnWorkerLocked` and `restartWorkerLocked` stop then
+ start on an entry that keeps its lease throughout, so a respawn's
+ uncertain start needs no lease decision; the next stop of that entry, by
+ any path, goes through the fenced `/stop`, which orders it.
+- `KillHost` inspecting the slice alone is not the proof, and the rewrite
+ of `TestKillHostProvesANestedLayoutWorkersHostCgroup` says so.
### 3.6 Decision D6: crash versus finish under a unit
The activator tells a hibernating worker's crash (restart with backoff)
from its finish (park) by `procCrashed`, which asks the proc's optional
`Crashed()` and defaults to "not a crash" (`activator.go:404-418`).
-`childProc` answers from `ProcessState` (`backend_exec.go:3329-3335`).
-`unitProc` has no `Crashed` (`internal/activator/unitproc.go:87-203`), so
-every nested worker crash would read as a finish and park.
-
-**Decision.** `unitProc.Crashed()` is true only after the observer's two
-termination facts hold (nested design §6.5) **and** systemd reported the
-proc's own invocation as `failed` with `Result` other than `success`.
-The advisory exit record never decides it (§3.3 of the nested design: the
-record is the tenant's claim). The template's `SuccessExitStatus=129 130 141
-143` (the trampoline's signal convention,
-`carlos-instance-sidecar@.service:41-46`) means a stop by SIGTERM is a
-finish, as for a flat child stopped the same way. An `inactive` unit whose
-invocation was garbage-collected reads as "not a crash", the safe default
-`procCrashed` already documents; a failed unit is not collected until it
-is reset or started again, so a real crash keeps its `Result`.
+`childProc` answers from `ProcessState`: any unsuccessful exit, a signal
+death included, is a crash (`backend_exec.go:3329-3335`). `unitProc` has
+no `Crashed` (`internal/activator/unitproc.go:87-203`), so every nested
+worker crash would read as a finish and park.
+
+Revision 1 kept the sidecar template's `SuccessExitStatus=129 130 141 143`
+(`carlos-instance-sidecar@.service:41-46`). The trampoline reports a
+tenant's signal death as `128 + signal` (`initExitCode`,
+`internal/instanceexec/init_linux.go:135-141`), so a worker that died of
+SIGTERM or SIGPIPE on its own would have read as success and parked
+forever (R1-7). That is not the flat contract.
+
+**Decision.**
+
+- The worker template has **no** `SuccessExitStatus=` line. Exit 0 is the
+ only success, as for a flat child.
+- `unitProc.Crashed()` is true only after the observer's two termination
+ facts hold (nested design §6.5) **and** systemd reports the proc's own
+ invocation with `Result` other than `success`. The advisory exit record
+ never decides it (§3.3 of the nested design: the record is the tenant's
+ claim).
+- A stop the agent asked for is told apart by the agent, not by the exit
+ status. Every reader of `procCrashed` reads a child that exited on its
+ own: the sweep's unexpected-exit arm (`activator.go:4592-4627`), the
+ clean-park mark (`:4504`), and the two settle windows
+ (`wakeplan.go:579`, `workerswap.go:186`). None of them is reached for an
+ entry the agent is stopping (`beginStopLocked`). A requested stop of a worker may leave
+ its unit `failed` (the trampoline answers systemd's SIGTERM with 143);
+ the termination facts and D19's proof already accept `failed`, and the
+ next start of a failed unit needs no reset.
+- An `inactive` unit whose invocation was garbage-collected reads as "not
+ a crash", the safe default `procCrashed` documents. A `failed` unit is
+ not collected until it is reset or started again, so a real crash keeps
+ its `Result` until the observer has read it.
+- A worker that sends SIGTERM to its own PID 1 makes a crash read as
+ whatever the trampoline then reports. It can only mislabel its own
+ exit, which is its own availability.
### 3.7 Decision D7: minting
**Decision.** `sidecarRouteFor` takes the same `nested` answer the instance
mint takes (`mintLayoutReader`, `internal/provision/reconcile.go:471-490`;
`RouteFor`'s `nested` arm, `:456-462`) and, when it is true and
-`CARLOS_NESTED_WORKERS` is on (§8.1), mints `DBPath = NestedDBPath`. Every
+`CARLOS_NESTED_WORKERS` is on (§8.1) and the workers floor holds (D11), mints `DBPath = NestedDBPath`. Every
other answer mints flat, as today. `convergeSidecar`'s shape check
(`sidecar_reconcile.go:621-626`) accepts either the flat or the nested path
for the row's host, and reports anything else as today.
@@ -479,7 +593,7 @@ and an instance host can never end in `.sidecar.internal`.
**Decision.** The layout pass's eligibility (`pass.go:264`, `:529`, `:747`)
widens from `ActivatorManaged(rt)` to `ActivatorManaged(rt) ||
-WorkerManaged(rt)`, behind `CARLOS_NESTED_WORKERS` (§8.1). Everything else
+WorkerManaged(rt)`, behind `CARLOS_NESTED_WORKERS` (§8.1) and the workers floor (D11). Everything else
about the move is plan 4's:
- **Quiesce.** `FenceAndStop` on the worker's entry. A hibernating worker
@@ -498,16 +612,50 @@ about the move is plan 4's:
the suffix, or the journal is refused like any mismatch. The nine
artifacts keep their order (`plan.go:98-101`); `socket` is recorded
absent on both sides. The journal format does not change.
-- **A row with nothing on this box moves by row alone.** Today the pass
- refuses a host whose database is missing (`pass.go:347-355`), so a row
- minted flat on a box that never woke it stays flat forever. Under the
- refusal policy that becomes a box that refuses the row on its first wake
- here, which is exactly a failover. **Decision:** when every one of the
- nine artifacts is absent at both endpoints, the pass journals the
- absence and does the row CAS only. The next wake restores from the
- bucket into the nested path. This changes `Required` for the database
- from "must exist" to "must exist unless nothing exists", for instance
- rows too.
+- **The app's own entries move too.** A worker's move takes the full extras
+ inventory (`ScanExtras`, `internal/layout/extras.go:14-50`, called from
+ `presence`, `pass.go:805-830`) exactly as an instance's does. Nothing in
+ this spec narrows it (R1-4).
+- **A row whose database was never here moves without one** (R1-4, R1-5).
+ Today the pass refuses a host whose database is missing (`pass.go:347-355`),
+ so a row minted flat on a box that never woke it stays flat forever.
+ Under the refusal policy that becomes a box that refuses the row on its
+ first wake here, which is exactly a failover. Revision 1 allowed a
+ row-only move when all nine artifacts were absent. That missed the
+ ordinary standby: config delivery writes every sidecar's `.config.env`
+ whether or not it has run (`internal/appconfig/syncexec.go:879-890`,
+ `:1523`), so a box that has never held the lease has config and no
+ database.
+ **Decision:** the database stops being `Required` when, under the fence,
+ **the database and every one of its companions are absent at both
+ endpoints**: `db`, `db-wal`, `db-shm`, `litestream-shadow`, `corrupt`,
+ and the box-wide replicator's `<db>-litestream` directory. The pass then
+ journals the database absent and moves everything that is present (the
+ env files, the config bundle, every extra) under the journal as it moves
+ any artifact. The next wake restores from the bucket into the nested
+ path, which is what `restorePlain` does on any box whose database path
+ is empty (`backend_exec.go:795-807`). A companion without its database
+ (a WAL with no database, say) is evidence of local history that went
+ missing, and the move is refused, as today, with the names in the
+ reason; an operator decides. This applies to instance rows too.
+
+ Why filesystem absence is enough here, and the part of R1-4 this
+ declines (§12): the platform's own rule is already that an absent
+ database means "restore from the bucket". `restorePlain` takes an
+ absent path as a cold box on every wake, flat or nested, today. A flat
+ tenant that renames a stopped sibling's database (the attack R1-4
+ describes) gets the same restore on that sibling's next flat wake with
+ no layout pass involved, and it could delete the file outright. So the
+ move adds no loss the flat tier did not already allow, and the flat
+ tier is what this spec removes. A renamed copy under a name the host
+ owns (`<host>.db.old`) is an extra and moves with the host; one under a
+ name no host owns is reported as unattributed, as today
+ (`pass.go:280-284`). Asking for evidence a flat tenant cannot forge
+ would mean root-held evidence of every past materialisation on every
+ box, for a loss the flat tenant can cause anyway.
+- **Failover.** The plan's test runs two boxes where only one has ever
+ held the database: the other's row moves with config only, and the
+ host then fails over to it nested, restoring from the bucket (R1-5).
- **Config.** `ConfigEnvPath` decides nested by both columns
(`internal/appconfig/syncexec.go:66-79`), so a nested worker's
`.config.env` would keep landing flat. It learns the worker shape (D2's
@@ -519,28 +667,46 @@ about the move is plan 4's:
row, and the report's per-host key already admits any host
(`internal/status/snapshot.go:230-254`).
-### 3.9 Decision D9: an always-on worker's replica lineage
-
-Hibernating workers replicate through the lease (`.activator-<host>.db-litestream`,
-one of the nine artifacts), keyed by host in the bucket, so their move is
-path-blind. An always-on worker is not hibernating, so the box-wide
-replicator claims its database (`internal/ops/litestream.go:356-357`,
-`:393-398`), and that replicator keys a replica by the database's full path
-(`replicaKey`, `:139-162`). Moving the file therefore moves the replica.
-This is new: no always-on row has ever been moved by the pass, because
-`ActivatorManaged` requires `Hibernate`.
-
-**Decision.** Accept a new lineage at the nested path. The pass, for an
-always-on worker: fences (the generator then drops the host,
-`FleetDBsWithSkipped`'s `fenced`), waits until the regenerated config no
-longer names the flat path and the box-wide replicator has reloaded,
-moves, and releases; the generator then names the nested path. The old
-lineage stays in the bucket under its old key (retention as configured).
-`CoverageShortfall`'s shrink guard is taught that a journaled host's flat
-path leaving the config is expected. The plan must confirm on a dev box
-that litestream releases the file on reload before the rename; if it does
-not, the pass stops the replicator for the rename, which is a box-wide
-pause of seconds, and says so in the event line.
+### 3.9 Decision D9: one replication owner for every worker (R1-3)
+
+Revision 1 assumed the box-wide replicator owned an always-on worker's
+database and planned a new path-keyed lineage for the move. Main says
+otherwise (§1.8): every worker's wake starts lease replication
+(`activator.go:2602`), always-on supervision keeps it
+(`activator.go:4599-4627`), and the box-wide generator replicates the same
+file too, because its exclusion tests `Hibernate` alone
+(`litestream.go:356-357`). Two replicators on one WAL is the corruption
+case the file itself warns about (`litestream.go:230-234`).
+
+**Decision.** The activator's lease replication is a worker's only
+replication, hibernating or always-on, flat or nested:
+
+- **One predicate, three readers.** `ops` gains `ActivatorReplicated(rt)`:
+ `rt.Backing != BackingUnit && rt.DBPath != "" && (rt.Hibernate ||
+ rt.IsSidecar())`. The box-wide generator's exclusion
+ (`litestream.go:356-357`) and the restore census (`HibernatingHosts`,
+ `hibernate_verify.go:45-60`, renamed for what it now selects) both use
+ it, so the two cannot drift into a gap that belongs to neither, which is
+ what that census's own comment promises.
+- **The transition.** On the first generation after this change, each
+ always-on worker's path leaves the box-wide config. `CoverageShortfall`'s
+ shrink guard is taught that a path leaving because its row became
+ `ActivatorReplicated` is expected, and names each one in the event line.
+ The lease lineage has been running all along, keyed by host, so nothing
+ stops being replicated. The old path-keyed lineage stays in the bucket
+ under its old key, under the retention it has.
+- **The restore drill** now covers always-on workers' lease lineages. The
+ first drill after the change is the evidence that those lineages are
+ sound after a period of two writers. One that fails is a finding for
+ that host, reported as any drill failure is.
+- **The move is then path-blind** for every worker, as it already is for a
+ hibernating one: the lease lineage is keyed by host, so moving the file
+ moves nothing in the bucket, and no box-wide reload sits inside the
+ move. Revision 1's wait for a regenerated config, and the dev-box check
+ on litestream releasing the file, are gone.
+
+This part has no dependency on the rest of the spec and is not behind
+`CARLOS_NESTED_WORKERS`. It can land first (§11 Q4).
### 3.10 Decision D10: while a box's nested gate fails
@@ -568,21 +734,72 @@ A binary from before this change runs a nested worker row through the old
no database path, and the row's own nested `DBPath` replicated by the old
lease code. Two copies, and unisolated.
-**Decision.**
-
-- Box: `systemApplier.install` refuses a candidate whose unit set lacks
- `carlos-instance-worker@.service` while any row whose host carries the
- sidecar suffix has `NestedPaths` (`provision.go:3020-3023`), beside
- R9's guard (`systemapply.go:1273-1290`). The reason names the rows and
- says to revert the layout first (copy to review).
-- Console: `layoutDowngradeBlocked` (`layoutguard.go:24-69`) gains a
- second floor, `release.WorkerLayoutFloor`, judged on the same observed
- rows. With D8 reporting worker rows, a nested worker row blocks a
- rollback below that floor the way a nested instance row blocks one
- below `LayoutFloor` (`internal/release/layoutintent.go:211`).
+Revision 1 guarded `systemApplier.install` only. Round 1 showed three
+other ways an old binary gets in front of a nested worker row (R1-2):
+
+- the update starts the new edge before its health gate
+ (`systemapply.go:600-612`), so the new agent may already have moved
+ workers when the gate fails and `restoreFiles` (`:1784-1800`) puts
+ `carlos.prev` back, with no worker check;
+- the recovery unit can **exec `carlos.prev` itself**
+ (`internal/host/units/carlos-system-recover.service:41`), so a check
+ that only the new binary makes cannot hold that path;
+- a row-only predicate misses a move interrupted before its row CAS: the
+ row still reads flat while its files are half-way. R9's guard counts
+ journals, fenced rows and conflicts for exactly this reason
+ (`systemapply.go:1273-1310`).
+
+The repo already has the mechanism for "an old binary must not run once
+this box has state it cannot read": the moves protocol floor, and the seeds
+floor built on it (seeded-instances design §3.3; `internal/seeds/floor.go:41-42`,
+`internal/host/rootseedsfloor.go`). A floor, once armed on a box, makes
+root give a verdict on `carlos.prev`, and the recovery unit runs
+`carlos.prev` only when that verdict names the armed floor
+(`carlos-system-recover.service:9-30`, `:41`); `restoreFiles` checks the
+same floor before it changes anything (`systemapply.go:1790-1797`).
+
+**Decision.** A third floor, the **workers floor**, on that mechanism:
+
+- **Published and armed before anything moves.** An operator publishes
+ `control/workers-protocol.json` naming the first release that carries
+ this spec. A box arms it locally as it arms the seeds floor
+ (`armFloor`, `systemapply.go:908-925`), and gives its `carlos.prev`
+ verdict. The layout pass treats worker rows as eligible (D8) only on a
+ box whose workers floor is armed **and** whose `carlos.prev` is either
+ excluded or at or above the floor; until then `CARLOS_NESTED_WORKERS=on`
+ is reported as `blocked: workers floor` and moves nothing. The reconciler
+ mints nested worker rows (D7) under the same condition. So every nested
+ worker row, journal or fence on a box exists only after every binary that
+ box can select would refuse to run below it.
+- **Every activation path obeys it.** Install refuses a candidate below an
+ armed workers floor, beside the moves floor's refusal. `restoreFiles`
+ refuses a `carlos.prev` below it, so a failed gate leaves the box on the
+ new build, journal kept, as the moves floor does. The recovery unit's
+ `ExecStart` gains the third test, the same shape as the seeds floor's
+ (`/data/.moves/workers-floor.min-release`, verdict
+ `/opt/carlos/carlos.prev.workers-allowed`), so the shell never execs a
+ `carlos.prev` below it.
+- **The row test, for the console and for status,** is R9's, not a
+ row-only one: a worker host is named when its row has `NestedPaths`,
+ when `Classify` calls it fenced or in conflict, or when it has a
+ journal, whether or not the row exists (`layoutReversalIncomplete`,
+ `systemapply.go:1273-1310`, which already covers every host this way;
+ D8's status change makes the console see worker hosts too). The console's
+ `layoutDowngradeBlocked` (`layoutguard.go:24-69`) gains
+ `release.WorkerLayoutFloor`, judged on those observed hosts, and blocks a
+ channel move below it while any is named.
+- **No disarm.** Neither existing floor has one, and this one adds none: a
+ floor only ever rises. A deployment that must go below it reverts its
+ worker rows first and then has an operator remove the floor files by
+ hand, knowingly. The plan documents that step beside the revert.
- `carlos system layout revert` reverts worker rows with the rest (they
are in the pass).
+The unit-set test of revision 1 (candidate lacks
+`carlos-instance-worker@.service`) stays as a second net for a candidate
+whose version cannot be parsed, the way R9 pairs the mount unit with
+`LayoutFloor` (`systemapply.go:510-528`).
+
### 3.12 Every consumer of the shape
The rule this repo has paid for three times: the consumer list is
@@ -603,9 +820,13 @@ main; the plan's first task re-runs the sweep on the day it starts:
`:347-355` (D8)
- `provision/sidecar_reconcile.go:131-148`, `:621-626` (D7)
- `host/layoutleg.go:193-199` (D8)
-- `host/systemapply.go:1273-1290` (D11)
+- `host/systemapply.go:1273-1310`, `:1784-1800`, `armFloor` (D11)
+- `host/units/carlos-system-recover.service` (D11)
- `console/layoutguard.go` (D11)
-- `ops/litestream.go` (D9)
+- `ops/litestream.go`, `ops/hibernate_verify.go` (D9)
+- `instancectl/legacystart.go`, `server.go` (D19)
+- `provision/sidecar_reconcile.go` `poolServed`, `placeable` (D17, shared
+ with the console)
- the sampler's slice-to-host mapping (`internal/host/metrics_probes.go`,
the nested design §6.6) and `carlos status --instances`' unit matcher
(sidecar design §14.4), which must recognise `carlos-instance-worker@`
@@ -634,9 +855,15 @@ logged and taken as the safe reading):
- `report`: every spawn proceeds as today, and each flat or foreign spawn
of a host not on the allowlist is recorded as a would-be refusal (§4.3).
- unset: `report` in the release that carries this spec; `refuse` from the
- next (D15). With `CARLOS_LAYOUT_DEFAULT=flat` (a deployment that has
- chosen not to isolate), unset stays `report` in every release, because
- that deployment's every row is flat by its operator's choice.
+ next (D15), with one exception. On a box whose **effective intent is
+ flat by an operator's choice**, unset stays `report` in every release,
+ because that box's every row is flat on purpose. Flat by choice is
+ either `CARLOS_LAYOUT_DEFAULT=flat` with no recorded intent, or a
+ recorded `flat` intent, whatever the default says (R1-6). Flat by
+ fallback (the default is nested but the box is not ready,
+ `defaultLayoutIntent`, `layoutleg.go:115-146`) is not a choice, and
+ unset means `refuse` there (D10). An intent the agent cannot read counts
+ as not chosen.
- an unknown value: `refuse`, logged. An operator who typed the variable
meant to choose, and the reading that cannot expose a tenant is the one
to take. (This is the opposite of `CARLOS_LAYOUT_DEFAULT`'s unknown-value
@@ -700,6 +927,15 @@ these seams; nothing here changes them.
(`flat_refused`, `foreign_refused`, `flat_reported`, `foreign_reported`),
first and last time and a count, the shape `ConfigRefusal` already has
(`snapshot.go:256-274`). Error text never goes in it.
+- **Inventory (R1-6).** Beside the spawn record, every snapshot carries
+ what `refuse` would stop **from the registry, not from history**: each
+ row the agent could spawn through a seam (activator-managed, worker and
+ foreign exec rows, awake or asleep) that classifies `flat` or `foreign`
+ and is not allowlisted, by host and layout word; the effective intent
+ with its source (`nested`, `flat_choice`, `flat_fallback`, `unknown`);
+ and the count of flat worker rows. A dormant row is in it on the first
+ snapshot after this release, whether it has woken or not. The inventory
+ is what D15 judges; the spawn record is what an operator reads.
- **Wake failure.** An instance wake refused this way answers the visitor
as any refused wake does (the member-safe reason path); a worker's state
word in `carlos sidecars ls` is a new one (copy to review: "Held: this
@@ -717,19 +953,36 @@ these seams; nothing here changes them.
that runs; status starts showing every flat spawn a `refuse` box would
stop.
2. **Per box**, after `CARLOS_NESTED_WORKERS=on` has moved its sidecars
- and its status shows no would-be refusal for a stretch that covers the
- box's slowest hibernating workload's wake interval, an operator sets
+ and its inventory (D14) lists nothing, an operator sets
`CARLOS_FLAT_EXEC=refuse` (with an allowlist if one is genuinely
needed). Dev boxes first (§8.2).
-3. **Release N+1** makes `refuse` the unset default (and turns
- `CARLOS_NESTED_WORKERS` on by default, §8.1). The console's system
- channel door refuses to move a channel to N+1 while any box that will
- follow it has the variable unset and reports a would-be refusal in its
- last fresh snapshot, or is stale, naming the boxes and hosts (copy to
- review). That is `layoutDowngradeBlocked`'s pattern (judge observed
- state; unknown blocks), applied to an upgrade.
-4. A box that needs the old behaviour after N+1 says so explicitly:
- `CARLOS_FLAT_EXEC=report`.
+3. **Release N+1** changes two defaults, and the console guards each one
+ separately, because they are different questions (R1-6): whether a box
+ may still run flat, and whether its workers are ready to move.
+
+ | Default that changes in N+1 | It changes nothing on a box when | Otherwise |
+ |---|---|---|
+ | `CARLOS_FLAT_EXEC` unset → `refuse` | the variable is set explicitly, or the effective intent is flat by choice (§4.1), or the inventory lists no row and the intent is known | blocked |
+ | `CARLOS_NESTED_WORKERS` unset → `on` | the variable is set explicitly, or the box has no flat worker row | blocked |
+
+ The system channel door refuses to move a channel to N+1 while any box
+ that will follow it is blocked on either line, or is stale, or reports
+ an inventory it could not complete, naming the boxes, the line and the
+ hosts (copy to review). That is `layoutDowngradeBlocked`'s pattern
+ (judge observed state; unknown blocks), applied to an upgrade. A box's
+ history of spawns plays no part: a dormant row that never woke is in
+ the inventory all the same.
+4. A box that needs the old behaviour after N+1 says so explicitly, and
+ each opt-out covers only its own line: `CARLOS_FLAT_EXEC=report` keeps
+ flat spawns running; `CARLOS_NESTED_WORKERS=off` keeps workers minted
+ and kept flat. An operator who sets only the first still has workers
+ move, and status says so beside the setting.
+
+Whether a worker is ready to move is not something a box can observe (a
+worker that computes `/data/<host>.db` looks the same as one that reads
+`CARLOS_SIDECAR_DB` until it starts nested). So the second line never
+infers readiness: on a box with worker rows, only an operator's explicit
+setting clears it, after §8.2 step 6's check.
Claim-mode boxes are unaffected throughout: they already refuse every
legacy launch shape before this check is reached.
@@ -760,18 +1013,36 @@ before the plan cap and the claim branch (`:315`, `:353`), so it covers
the claim-mode create too:
- The console reads every box's snapshot the way `carlos system status`
- does (`systemBoxes`, as `layoutDowngradeBlocked` uses it), keeps the
- boxes in the resolved region (or, for a fleet-placed app, the fleet's
- boxes), and drops stale ones.
+ does (`systemBoxes`, as `layoutDowngradeBlocked` uses it) and drops
+ stale ones.
+- **The placement set is the reconciler's, computed by the reconciler's
+ own code (R1-8).** A box can run the record exactly when
+ `ReconcileSidecars` would mint it there: the record's region is the
+ box's, the box serves the record's pool (`poolServed`), and the record's
+ placement is one the box accepts (`placeable`, with `no-inbound`
+ against the box's `CARLOS_INBOUND`)
+ (`internal/provision/sidecar_reconcile.go:343-356`). The console builds
+ the record it is about to write (region, pool, placement, as
+ `apisidecars.go:268-300` resolves them) and asks those same functions,
+ exported from `provision`, about each fresh box. So the box reports its
+ pools and its inbound setting in its snapshot, beside the field below;
+ a box that does not report them is not in the set and does not pass.
+ Revision 1's region-wide set could pass a region whose only ready box
+ serves the wrong pool or accepts inbound work, and block on an
+ irrelevant box.
- Each box reports a new additive field, `Metrics.Layout.Workers`:
- `nested` when its binary carries this spec, `CARLOS_NESTED_WORKERS` is
- on, its effective intent is nested and its nested gate answers; `off` or
- `blocked: <reason>` otherwise. The reason is private (it may name a
- path), like the layout report's reasons.
-- The create is refused when no fresh box is in scope, or when any fresh
- box in scope reports anything but `nested`. Stale boxes neither pass
- nor block: the box-side refusal (§4) is the boundary, and this door is
- the early, honest answer in front of it.
+ `nested` when its binary carries this spec, its workers floor is armed
+ with a `carlos.prev` verdict (D11), `CARLOS_NESTED_WORKERS` is on, its
+ effective intent is nested, its nested gate answers, **and it can launch
+ a worker at all**: a box with claim control armed refuses every worker
+ launch (`workerbuild.go:111-113`) and reports `blocked: claim` (R1-8).
+ `off` or `blocked: <reason>` otherwise. The reason is private (it may
+ name a path), like the layout report's reasons.
+- The create is refused when the placement set has no fresh box, or when
+ any fresh box in it reports anything but `nested`. A box outside the
+ set never blocks. Stale boxes neither pass nor block: the box-side
+ refusal (§4) is the boundary, and this door is the early, honest answer
+ in front of it.
- The refusal is a 409 with `code: "isolation"`, so the CLI and an app's
own code can tell it from a cap or a duplicate. The text names the
region and says that sidecars there cannot run isolated yet (copy to
@@ -796,12 +1067,31 @@ weakest box that may run it.
hold on that box, and each is observable through the API (`carlos status`),
with no box access:
-1. `CARLOS_FLAT_EXEC` in force is `refuse` (explicitly or by default);
+1. `CARLOS_FLAT_EXEC` in force is `refuse` (explicitly or by default),
+ and has been since this agent process started;
2. the allowlist is empty;
-3. no flat child is running: the agent reports the number of populated
- `carlos-exec@*` cgroups under its delegated subtree (it already walks
- them, `listExecCgroups`, `internal/host/metrics_probes.go:253`), and it
- is zero;
+3. **no flat child is running, by a complete census (R1-9)**. Revision 1
+ counted populated `carlos-exec@*` cgroups. That misses what lives
+ outside them: a spawn with no cgroup manager starts in the agent's own
+ cgroup (`cgroup.go:392-395`), the PID-move fallback leaves any
+ grandchild already forked in the agent's cgroup (`cgroup.go:360-375`),
+ and `listExecCgroups` answers an empty list for a missing or unreadable
+ root and skips unreadable groups (`metrics_probes.go:253-280`). So:
+ - **Drain.** Condition 1 holds only for an agent that read `refuse` at
+ its own start. The edge unit has the default `KillMode=control-group`
+ (`carlos-edge.service`), so the stop before that start killed every
+ process in its subtree, flat children included. A box switched to
+ `refuse` reports the condition only after its next edge restart.
+ - **Census.** The agent reads `cgroup.procs` of **every** cgroup in the
+ edge unit's subtree, its own leaf included, and reports `clean` only
+ when every pid it finds is the agent itself or a platform child it
+ started outside the tenant seams and still holds (it keeps that pid
+ set). Any other pid, any cgroup it could not read, a subtree root it
+ could not open, or a box with no delegation where it cannot tell its
+ own cgroup's members apart, reports `unknown` with the reason, and
+ `unknown` blocks retirement exactly as a populated count would.
+ - The census is a snapshot field (`Metrics.FlatExec.Census`: `clean`,
+ `populated`, `unknown`, with the time), read through `carlos status`.
4. the nested probe's metadata line is enforced (`v4: enforced`; nested
design §8.5).
@@ -838,7 +1128,7 @@ boxes and private services on the network.
| After | The attacker can | The attacker cannot |
|---|---|---|
| **Today** | everything in §1.5: read flat siblings; write nested siblings' raw directories and so run code inside them; kill or freeze the agent through cgroupfs; steal other flat workers' tokens; reach loopback, the VPC and metadata, and through it the box role | nothing the edge itself cannot do, short of root |
-| **The fence** (today, interim) | all of the above except metadata from its own cgroup; and very likely metadata too, by moving its pid into the exempt leaf (§1.6) | reach metadata from `carlos-exec@<host>` while it stays there |
+| **The fence** (today, interim) | all of the above except metadata | reach metadata, for as long as it cannot leave its cgroup for an exempt one (§1.6; the fence's own hardening, recorded on a dev box by §9) |
| **Release N rolled** (§3 dark, `report`) | the same: nothing that runs has changed | (status now names every flat spawn) |
| **`CARLOS_NESTED_WORKERS=on`** on a box whose gate is ready | on that box, nothing from a new or moved sidecar: it sees its own directory only, no loopback, no private range, no metadata, no sibling pid. It can still get a flat process **if** the box mints flat (gate failing with no recorded intent, D10) or the row is foreign | read or write another tenant's files from its sidecar; call the agent socket for another host (D3) |
| **`CARLOS_FLAT_EXEC=refuse`**, empty allowlist | nothing flat on that box, by any of the four spawn paths, in any layout state: a flat or foreign row is refused, the gate failing included | start any process in the host namespace |
@@ -903,24 +1193,33 @@ source is unreachable"; another provider's adapter supplies its own line.
| Setting | Where | Release N | Release N+1 | Gone in |
|---|---|---|---|---|
-| `CARLOS_NESTED_WORKERS` (`on`\|`off`) | box `host.env` | off: workers mint flat, the pass skips them, dispatch still honours any nested row | on unless `off` | N+2 |
-| `CARLOS_FLAT_EXEC` (`refuse`\|`report`) | box `host.env` | `report` | `refuse` unless set, or unless `CARLOS_LAYOUT_DEFAULT=flat` | stays (it is the self-hoster's opt-out) |
+| `CARLOS_NESTED_WORKERS` (`on`\|`off`) | box `host.env` | off: workers mint flat, the pass skips them, dispatch still honours any nested row. On does nothing until the workers floor is armed (D11) | on unless `off`; the N+1 channel guard (D15) | N+2 |
+| `CARLOS_FLAT_EXEC` (`refuse`\|`report`) | box `host.env` | `report` | `refuse` unless set, or unless the intent is flat by choice (§4.1); the N+1 channel guard (D15) | stays (it is the self-hoster's opt-out) |
+| `control/workers-protocol.json` | deployment bucket | published before any box turns workers on | stays | stays (a floor only rises) |
| `CARLOS_FLAT_EXEC_ALLOW` | box `host.env` | empty | empty | stays |
| `CONSOLE_REQUIRE_ISOLATED_WORKERS` | console env | off | on unless `false` | N+2 |
-Dark in N: every behaviour change. The dispatch in `StartWorkerStaged` is
-live in N, but no nested worker row can exist until a box turns
-`CARLOS_NESTED_WORKERS` on, so it is inert.
+Dark in N: every behaviour change except two. The dispatch in
+`StartWorkerStaged` is live in N, but no nested worker row can exist until
+a box arms the workers floor and turns `CARLOS_NESTED_WORKERS` on, so it is
+inert. **D9 is live in N on purpose**: the box-wide replicator stops
+replicating workers on the first generation after the roll (§3.9), and
+the shrink guard says so per host. D19's root fence is live too, and
+inert until a worker unit exists.
### 8.2 Order
The house order: dev boxes, then the dev console, then a check, then
flagship boxes, then the flagship console.
-1. **Release N to dev boxes, then the dev console.** Nothing changes.
- `carlos status --layout` lists sidecar rows as `flat` and the flat-exec
- section lists each as `flat_reported`.
-2. **Dev boxes: `CARLOS_NESTED_WORKERS=on`.** Watch sidecar rows turn
+1. **Release N to dev boxes, then the dev console.** Nothing that runs
+ changes. `carlos status --layout` lists sidecar rows as `flat` and the
+ flat-exec inventory lists each. Each always-on worker's path leaves the
+ box-wide config, named in the shrink guard's event line, and the next
+ restore drill covers its lease lineage (D9).
+2. **Publish the workers floor** for the deployment, and wait until every
+ dev box reports it armed with a `carlos.prev` verdict (D11).
+3. **Dev boxes: `CARLOS_NESTED_WORKERS=on`.** Watch sidecar rows turn
`nested`, hibernating ones at once, always-on ones within the awake
grace. The check is from inside: a throwaway sidecar in a dev account
whose release lists `/data` and `/run/carlos`, reads `/proc/1/status`,
@@ -928,27 +1227,34 @@ flagship boxes, then the flagship console.
`/v1/ready` and `/v1/next-due` over `CARLOS_AGENT_SOCKET`, and prints
all of it to `carlos sidecars logs`; then sleeps, wakes on its timer,
wakes on `carlos sidecars wake`, crashes once (restarted with backoff),
- and finishes once (parked). An always-on twin is moved by the grace
- stop, and its replica appears under the nested key.
-3. **Dev boxes: `CARLOS_FLAT_EXEC=refuse`.** A foreign row made on a dev
- box with `carlos add --db` is refused, status says so, and `carlos add`
- with the defaults repairs it. Verify §1.6's cgroup migration on a dev
- box both before and after: before, a flat child can move itself into
- `supervise`; after, there is no flat child to try.
-4. **Dev console: `CONSOLE_REQUIRE_ISOLATED_WORKERS=true`.** A create in a
- dev region whose boxes report `nested` succeeds; with one box's
- `CARLOS_NESTED_WORKERS` off, it is refused with `code: "isolation"`.
-5. **The check before flagship:** establish, for every app with native
+ is killed by SIGTERM from inside its own process once (a crash,
+ restarted), and finishes once (parked). An always-on twin is moved by
+ the grace stop, and its lease lineage continues under its host key.
+ A standby box that has the twin's config and has never held its lease
+ moves its row with config only, and the twin then fails over to it.
+4. **Dev boxes: `CARLOS_FLAT_EXEC=refuse`,** then an edge restart so the
+ census can read `clean` (D18). A foreign row made on a dev box with
+ `carlos add --db` is refused, status says so, and `carlos add` with the
+ defaults repairs it. Record the §1.6 cgroup migration attempt on a dev
+ box before and after: before, whether a flat child can move itself into
+ `supervise` under the fence; after, there is no flat child to try.
+5. **Dev console: `CONSOLE_REQUIRE_ISOLATED_WORKERS=true`.** A create in a
+ dev region whose eligible boxes report `nested` succeeds; with one
+ eligible box's `CARLOS_NESTED_WORKERS` off, it is refused with
+ `code: "isolation"`; with only an ineligible box (another pool) off, it
+ succeeds.
+6. **The check before flagship:** establish, for every app with native
sidecars on the flagship, whether its worker opens its database by
computed path, uses the agent socket by a hard-coded path, or reaches
anything on loopback or a private address (§3.4, §7.3). Each one that
does ships a release that reads `CARLOS_SIDECAR_DB` and
`CARLOS_AGENT_SOCKET` before its boxes turn nested workers on (§11 Q3).
-6. **Flagship boxes,** region by region, console box last: steps 2 and 3.
- Then the **flagship console**: step 4.
-7. **Fence retirement** per box when §6's four conditions hold, dev first,
+7. **Flagship:** release N to flagship boxes and console as in step 1;
+ the workers floor as in step 2; then boxes region by region, console
+ box last, steps 3 and 4. Then the **flagship console**: step 5.
+8. **Fence retirement** per box when §6's four conditions hold, dev first,
under its own scoped approval.
-8. **Release N+1** cut and rolled in the same order, behind the console
+9. **Release N+1** cut and rolled in the same order, behind the console
guard of D15.
### 8.3 Census
@@ -957,18 +1263,28 @@ Root-run code this touches, each of which changes a digest the root-read
census pins (`internal/provision/plan14_root_reads_test.go`) and so needs
a census proposal through the #280 process in the same branch:
-- `internal/instancectl/names.go`, `server.go` (the worker role on the
- existing verbs; no new verb);
+- `internal/instancectl/names.go`, `server.go`, `legacystart.go` (the
+ worker role on the existing verbs and under the start fence, D19; no new
+ verb);
- `internal/host/units/carlos-instance-worker@.service` (a new embedded
unit; templates are never enabled, so `update-platform-binary.sh` needs
no new name);
-- `internal/host/systemapply.go` (D11's guard);
+- `internal/host/systemapply.go` (D11: install, `restoreFiles`, `armFloor`)
+ and the workers floor's root files beside `rootseedsfloor.go`
+ (`rootworkersfloor.go`, its verdict and its boot read in
+ `rootmovesfloor_boot.go`);
+- `internal/host/units/carlos-system-recover.service` (D11: the third
+ floor test in `ExecStart`) (R1-2);
+- the `floor-allows-prev` verb, if the workers floor's verdict file needs
+ a case of its own there;
- the probe, only if the plan gives it a worker-template line (it should
not need one: the namespace block is identical, and the identity test
holds that).
No new root pass, path unit or relay. `instance-exec` and `instance-exit`
-run as `carlos` inside the namespace.
+run as `carlos` inside the namespace. D9's files (`internal/ops`) are not
+pinned by the root-read census today; the plan's census task confirms that
+on the day it runs.
### 8.4 Docs and copy
@@ -1003,40 +1319,77 @@ Unit and package level:
- The per-host agent listener: bound before the start, closed after
`done`; a token for host A refused on B's listener and on the shared
socket alike; the uniform 401; mode 0600.
-- `unitProc.Crashed`: a failed own invocation with `exit-code` is a crash;
- `success` after SIGTERM is not; a forged exit record claiming either is
- ignored; an unloaded invocation is not a crash.
+- `unitProc.Crashed`: a failed own invocation with `exit-code` is a crash,
+ including status 143 and 141 from a worker that died of SIGTERM or
+ SIGPIPE on its own; exit 0 is not; a forged exit record claiming either
+ is ignored; an unloaded invocation is not a crash; an agent-requested
+ stop that leaves the unit `failed` never reaches a `procCrashed` reader.
- The activator: a nested worker's wake, ready, next-due, idle, sleep,
timer wake, crash restart and finish park, end to end over the real
control server and a fake systemd that runs the trampoline's stub.
+- D19, over the real control server: a worker `/start` whose client gives
+ up while the handler is held keeps the lease on the agent side; the
+ `/stop` that follows cancels and joins it, and the held handler never
+ submits; a start accepted before a stop is refused however late it
+ registers; a join past the budget answers an error and the lease stays;
+ a worker stop does not fence the same host's run unit, or the reverse.
- The control service: `worker` admitted on the four verbs, refused where
it does not belong; the worker's unit and slice names; the tier always
- set; a staged build validated with the worker host's own app.
+ set; a staged build validated with the worker host's own app; `gen` other
+ than 0 refused for the worker role.
- Templates: the identity test across five templates; the worker
- template's four differing lines and nothing else.
+ template's five differing lines and nothing else.
- Reconciler: mint nested exactly when the instance mint does and the flag
is on; converge accepts both paths; the co-spawn check unchanged.
- Layout pass: a hibernating worker moves asleep; an always-on worker is
deferred for the grace and then stopped once; every wake source of a
fenced worker refuses (timer, wake verb, boot pass, sweep); a journal
- whose host suffix and row `Kind` disagree is refused; a row with no
- artifacts anywhere moves by CAS alone, and a row with some artifacts but
- no database is still refused; revert brings a worker back byte for byte.
-- `ConfigEnvPath` and SyncExec for a nested worker; the litestream
- generator and shrink guard across an always-on worker's move.
+ whose host suffix and row `Kind` disagree is refused; a worker's
+ app-made extras move with it; a row whose database and companions are
+ absent on both sides moves its config and extras with the database
+ journaled absent; a row with a WAL, shadow or meta directory but no
+ database is refused; two boxes where only one has held the database,
+ and the other then wakes the host nested from the bucket; revert brings
+ a worker back byte for byte.
+- `ConfigEnvPath` and SyncExec for a nested worker.
+- D9: `ActivatorReplicated` is the one predicate; the box-wide generator
+ excludes an always-on worker and the restore census includes it; the
+ shrink guard accepts a path leaving for that reason and names it, and
+ still refuses any other shrink.
+- D11: the workers floor arms like the seeds floor; the pass and the
+ reconciler treat workers as flat until it is armed with a verdict;
+ install refuses a candidate below it; `restoreFiles` refuses a
+ `carlos.prev` below it and keeps the journal; the recovery unit's
+ `ExecStart`, run by the real shell against fixture files, execs the
+ current binary when the verdict does not name the floor (the existing
+ recover-unit tests' shape); the row test names a worker host with only a
+ journal.
- Flat-exec policy: each value, unset under both layout defaults, an
unknown value, an invalid allowlist entry; the refusal from both seams;
the enumeration test over `exec.Command` in `internal/activator`; the
swap gate and the respawn veto classify the new error; the nudge of
- D16 runs the pass for a refused managed row and not for a foreign one.
-- Status: the `FlatExec` section and `Layout.Workers` pass the contract
- allowlist (`internal/status` `contract_test.go`) and never reach
- `fleet.json`; worker rows in the layout report.
+ D16 runs the pass for a refused managed row and not for a foreign one;
+ unset under a recorded `flat` intent with a nested default is `report`,
+ and under a gate fallback is `refuse`.
+- Status: the `FlatExec` section (spawn record, inventory, census) and
+ `Layout.Workers` pass the contract allowlist (`internal/status`
+ `contract_test.go`) and never reach `fleet.json`; worker rows in the
+ layout report; a dormant flat row that has never woken is in the
+ inventory.
+- D18's census: a fixture subtree with only the agent's pid is `clean`; a
+ pid in the agent's own leaf that the agent did not start is
+ `populated`; an unreadable `cgroup.procs`, a missing root, and a box with
+ no delegation are `unknown`.
- Console: the create door's refusal order (after the region, before the
- cap and the claim branch), every case of D17, `code: "isolation"`; the
- downgrade guard's worker floor; the N+1 channel guard of D15.
-- `systemapply`: a candidate without the worker template is refused while
- a nested worker row exists, and installs when none does.
+ cap and the claim branch), every case of D17 including a ready box in
+ the wrong pool, an inbound box for a `no-inbound` record, an unready box
+ outside the set, and a claim-mode box (`blocked: claim`);
+ `code: "isolation"`; the downgrade guard's worker floor; the N+1 channel
+ guard of D15, each line separately, with a dormant row and with
+ `CARLOS_FLAT_EXEC=report` set but `CARLOS_NESTED_WORKERS` unset.
+- `systemapply`: a candidate without the worker template, or below the
+ workers floor, is refused while a worker host is named, and installs
+ when none is.
Kernel and box level (root runner or a dev box; these need real cgroup v2
and systemd):
@@ -1047,9 +1400,12 @@ and systemd):
- A real nested worker unit: namespace listing, the probe's datagrams,
`PrivatePIDs`, the per-host agent socket reachable from inside and the
shared one absent, the log stream in `carlos sidecars logs`, a crash
- read as a crash, `systemctl restart carlos-edge` stopping it and the
- boot pass starting an always-on one again.
-- The always-on move's litestream release (D9).
+ (including a self-inflicted SIGTERM) read as a crash, a requested stop
+ leaving the unit `failed` and the next start succeeding,
+ `systemctl restart carlos-edge` stopping it and the boot pass starting
+ an always-on one again.
+- D18's census on a real box: `clean` after an edge restart under
+ `refuse`; `populated` with a stray process left in the agent's leaf.
The full gate runs once before handoff; locally, only the changed
packages' tests.
@@ -1078,15 +1434,68 @@ packages' tests.
until nested workers roll. It stops new flat sidecars today at the cost
of the feature for that time. Existing sidecars keep running.
2. **Treat what the flat tier could reach as exposed?** B1 was confirmed
- live, and §1.6 suggests the fence may not have held against a
- deliberate tenant. That would put the box roles' reachable secrets
- (regional SSM, the deployment bucket, and on the console box the IAM
- user minting) in scope for rotation, and possibly a look at the
- bucket's write history. That is an incident call.
+ live. The console audit (#370) found every sidecar ever created was the
+ operator account's, so no tenant has used the sidecar path. Round 1
+ reports the hardened fence closes the cgroup escape (§1.6). Flat
+ instance rows (foreign rows, rows the pass refused or deferred) had the
+ same reach, though, and before the fence nothing stopped metadata. That
+ would put the box roles' reachable secrets (regional SSM, the deployment
+ bucket, and on the console box the IAM user minting) in scope for
+ rotation, and possibly a look at the bucket's write history. That is an
+ incident call; the inventory of D14, once release N is out, says which
+ tenant hosts ran flat.
3. **Who confirms the apps that use native sidecars are ready to move?**
- §8.2 step 5 needs each such app's owner to confirm, or ship, a worker
- that reads `CARLOS_SIDECAR_DB` and `CARLOS_AGENT_SOCKET` and does not
- rely on loopback, before its flagship boxes turn nested workers on. And
- does the operator app the fence exempts need the flat allowlist at all,
- or can it move to a unit or to the nested tier first, so that every
- box's allowlist can be empty?
+ §8.2 step 6 needs a worker that reads `CARLOS_SIDECAR_DB` and
+ `CARLOS_AGENT_SOCKET` and does not rely on loopback, before its flagship
+ boxes turn nested workers on. Every sidecar today is the operator's, so
+ this is our own check rather than a customer's. And does the operator
+ app the fence exempts need the flat allowlist at all, or can it move to
+ a unit or to the nested tier first, so that every box's allowlist can
+ be empty?
+4. **Land D9 ahead of the rest?** The double replication of always-on
+ workers (§1.8, R1-3) is live on main now and risks a corrupt replica
+ for every always-on sidecar. D9 is small (one predicate in `internal/ops`,
+ the shrink guard's exception, the restore census) and independent of
+ isolation. Recommendation: its own branch and release now, ahead of
+ release N, with a restore drill of each always-on worker's lease lineage
+ after it rolls.
+5. **The workers floor is a third protocol floor.** It adds a third test to
+ the recovery unit's `ExecStart` and a third published control object.
+ The alternative is weaker: R9's unit-set test cannot hold the recovery
+ unit's own `carlos.prev` exec (R1-2). Recommendation: accept the floor.
+
+---
+
+## 12. Review disposition
+
+### 12.1 Round 1 (Astra, `CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md`, at 1c029ee5)
+
+Verdict: reject. Every citation was checked against main and held.
+
+| Finding | | Disposition | Where |
+|---|---|---|---|
+| R1-1 | P1: no start/stop ordering, two workers on one lease | **Accepted.** The run unit's #292 fence, widened to the worker role; the failed wake keeps the lease until `/stop` and an empty unit prove it gone | D19, §3.5.1; §2 |
+| R1-2 | P1: D11 holds install but not rollback or recovery | **Accepted.** A workers protocol floor on the moves/seeds floor mechanism, armed before anything moves, held by install, `restoreFiles` and the recovery unit; R9's journal-aware row test; recover unit added to the census | D11, §3.11; §8.3; §11 Q5 |
+| R1-3 | P1: always-on workers already use lease replication | **Accepted.** One predicate makes lease replication every worker's only replication; the box-wide generator and the restore census use it. Recorded as a live defect | D9, §3.9; §1.8; §11 Q4 |
+| R1-4 | P1: "nine artifacts absent" is no evidence of no local data | **Accepted in part.** The full extras inventory moves with every worker; the row-only exception is replaced by "database and every companion absent at both ends", and a companion without its database refuses as today. **Declined:** evidence a flat tenant cannot manufacture (below) | D8, §3.8 |
+| R1-5 | P2: standby boxes with config and no database are stranded | **Accepted.** The narrowed rule moves config and extras with the database journaled absent; a two-box failover test | D8, §3.8; §9; §8.2 step 3 |
+| R1-6 | P2: D15 judges spawn history; `report` does not cover the worker default | **Accepted.** A registry inventory, dormant rows included, judged per default; flat by choice covers a recorded `flat` intent; the two defaults are guarded separately and opt out separately | D14, §4.3; D15, §4.4; §4.1 |
+| R1-7 | P2: `SuccessExitStatus` turns real crashes into finishes | **Accepted.** No signal statuses in the worker template; requested stops are told apart by the agent; D1's "four lines" reopened to five | D6, §3.6; D1, §3.1 |
+| R1-8 | P2: D17 checks a different placement set; claim mode passes | **Accepted.** The reconciler's own `poolServed` and `placeable`, over fresh boxes, nonempty; readiness includes the workers floor and launch capability (`blocked: claim`) | D17, §5 |
+| R1-9 | P2: D18's cgroup count is not proof | **Accepted.** `refuse` read at this agent's start (the edge restart drains the subtree), and a pid census of the whole edge subtree where anything unreadable or unaccounted is `unknown` and blocks | D18, §6 |
+
+The review's closing note, that the hardened fence already closes §1.6's
+cgroup escape, is taken: §1.6 and §7.2 no longer claim the fence is
+bypassable, and the dev-box record of the attempt stays.
+
+**Declined: R1-4's demand for evidence a flat tenant cannot manufacture.**
+The platform's own rule already treats an absent database as "restore from
+the bucket": `restorePlain` does that on every wake, flat or nested
+(`backend_exec.go:795-807`). A flat tenant that renames a stopped sibling's
+database gets that restore on the sibling's next flat wake today, with no
+layout pass involved, and could delete the file instead. The row move adds
+no loss the flat tier does not already allow, and the flat tier is what
+this spec removes. Root-held evidence of every past materialisation on
+every box would be a new mechanism for a loss the attacker can cause
+anyway. What the finding rightly caught, the incomplete inventory and the
+companion files, is fixed.
diff --git a/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION.md b/docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md
similarity index 100%
rename from docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION.md
rename to docs/superpowers/specs/CODEX-FINDINGS-SIDECAR-ISOLATION-R1.md