carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Fleet-internal engines, Plan 1: the simulation — Implementation Plan
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** A seeded model of spec rev 14's safety rules (invariant I1, the
6fence, the launch gate, the operator clear, the dedicated engine box, the
7call path's identity and grants), with a mutation harness proving that
8every rule is load-bearing, before any production code is written.
9
10**Architecture:** A test-only package, `internal/sim/engines`, in the shape
11of `internal/sim/hostlife`: a deterministic world driven by a seeded RNG,
12named protocol rules that a mutation can switch off (`ENGINESIM_MUTATE`),
13invariants checked after every step, deterministic witness scenarios (one
14per Astra finding), and `mutate.sh` running one prebuilt test binary per
15rule. It imports nothing from production: it models the spec, so a
16finding here is a spec defect, fixed in the spec before Plans 2 to 5.
17
18**Tech Stack:** Go standard library only (`math/rand`, `testing`), bash for
19the harness.
20
21**Spec:** `docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md`
22(rev 14, approved 2026-10-21). Rulings R1 to R4 are in its §2.1.
23
24### The series
25
26| Plan | Builds | Depends on |
27| --- | --- | --- |
28| **1 (this)** | The model and mutation sweep | spec |
29| 2 | The internal mark: `routes.internal`, classification before serving, the edge's fence and box-wide epoch, the launch gate on every activator path, the internal refusal (:443, home :80), minted groups, `--public`, the dedicated-box rule, `doc_engine` excludes `ports`, deploy watch tickets | 1 |
30| 3 | Engine names and grants: console `carlos engines`, signed `control/engines.json`, the edge's grant mirror with the durable floor | 2 |
31| 4 | The call path: arming, the engine socket owner with the peer-cgroup check, the door transition and uid rule, the signed hop caller, engine-box admission (exact route, recheck, per-app cap), the engine transport and limits | 3 |
32| 5 | Docs, copy review, the rollout survey and runbook | 4 |
33
34Plans 2 to 5 are written after this plan's sweep is green, because a rule
35the sweep cannot catch, or a violation it finds, changes the spec they
36implement.
37
38## Global Constraints
39
40- Test-only package: every file ends in `_test.go` (plus `mutate.sh` and
41 the expected-outcomes file). No production package imports it; it
42 imports no production package.
43- Deterministic: one seed gives one trace. No wall clock, no goroutines,
44 no map iteration order in a decision (sort keys first).
45- Environment: `ENGINESIM_SEEDS` (sweep size; gate default 200),
46 `ENGINESIM_SEED` (one seed, verbose trace), `ENGINESIM_MUTATE` (one rule
47 or a comma-separated set), `ENGINESIM_LIST=1` (print the rules a run
48 would check), `ENGINESIM_SCRATCH` (default `$TMPDIR/enginesim`).
49- CPU budget on DOCS-T1: one 3000-seed mutation sweep per revision, serial,
50 `nice -n 19`; 30k or more only on Paul's say-so. Scratch under
51 `$TMPDIR/enginesim`, removed once after the sweep, never by pattern.
52- No `t.Parallel()` in the sweep or witness tests (package-level `mutated`
53 is written by the harness's environment only, but witnesses read it).
54- Spec values the model uses: mirror poll 30 s, stale bound 10 min, record
55 lifetime 72 h (as ticks: 1 tick = 10 s); fence wait bound 30 s; one
56 internal group per engine box; engine names have no dot.
57- Rule names and invariant ids below are the vocabulary of Plans 2 to 5's
58 test names; keep them exactly.
59
60## Review Focus
61
621. **A witness that passes for the wrong reason.** Each witness must also
63 run with no mutation and hold; a witness that violates unmutated is a
64 model bug. (Task 6, `TestWitnessesHoldUnmutated`.)
652. **Actions the random walk never reaches.** A rule guarding an action
66 that no seed fires is untested however green the sweep looks. (Task 6,
67 `TestEveryActionFires`: each action count > 0 over 200 seeds.)
683. **Residuals generated as if they were violations.** The spec's three
69 stated residuals (fresh-box replay, a declaring build with an unwritten
70 manifest running across the upgrade, flat-era tampering) must not be
71 generated by the default world, and a flag must generate each so a test
72 shows it is counted as `residual`, not `violation`. (Task 5,
73 `TestResidualsAreCountedNotFailed`.)
744. **Nondeterminism.** A trace that differs between two runs of one seed
75 makes every reported seed unreplayable. (Task 1, `TestSeedDeterministic`.)
765. **Availability rules invisible to safety invariants.** Rules such as
77 `static-minted-public` and `owner-nonempty-unique` only protect
78 availability; without a liveness invariant their mutation is never
79 caught. (Task 3, INV-AV.)
80
81---
82
83### Task 1: The world, the step loop and the harness
84
85**Files:**
86- Create: `internal/sim/engines/world_test.go`, `internal/sim/engines/sim_test.go`,
87 `internal/sim/engines/rules_test.go`, `internal/sim/engines/invariants_test.go`
88
89**Interfaces:**
90- Produces:
91 - `type build struct{ Ver int; Declares, ManifestOK bool }`
92 - `type proc struct{ ID int; Group string; Ver int; Slice string; Unisolated, OwnPort bool }`
93 (`Slice` is the cgroup the peer check reads: `inst:<group>`,
94 `exec:<host>`, `unit:<host>`)
95 - `type row struct{ Host, Kind, Owner, Backend string; Platform, Grant bool; PathTo map[string]string }`
96 (`Kind` is `instance|service|static`; `Backend` a socket path or address,
97 `""` for static)
98 - `type mark int` with `markNull, markPublic, markInternal`
99 - `type req struct{ ID, Epoch int; Row, Backend string; Public bool; Caller *caller; Held bool }`
100 - `type caller struct{ Account, App, Engine string }`
101 - `type box struct{ ID int; Rows map[string]*row; Mark map[string]mark; Wrote, Fenced, Epoch int; ReloadFails, Armed bool; Live, Good map[string]int; Staged map[string][]int; Procs []*proc; Admitted []*req }`
102 - `type world struct{ seed int64; rng *rand.Rand; boxes []*box; builds map[int]build; step int; trace, violations, residuals []string; stats map[string]int }`
103 - `func newWorld(seed int64, verbose bool) *world`
104 - `func runSeed(seed int64, verbose bool) *world`
105 - `func (w *world) fail(inv, format string, args ...any)` (records a violation once per invariant id)
106 - `func on(rule string) bool` and `var allRules []string` (rules_test.go,
107 the hostlife idiom: `ENGINESIM_MUTATE` read in `init`)
108 - `func (w *world) group(b *box, host string) string` (a row's group key:
109 its owner for a minted instance row, else its own host)
110
111- [ ] **Step 1: Write the failing tests** in `sim_test.go`:
112 `TestSeedDeterministic` (two `runSeed(7, false)` traces are equal),
113 `TestEngines` (the sweep: `ENGINESIM_SEEDS`, default 200; one seed with
114 `ENGINESIM_SEED`; reports violations per invariant with the first seed,
115 hostlife's `sweep` shape), `TestRulesListed` (`ENGINESIM_LIST=1` prints
116 `allRules` one per line and runs nothing).
117- [ ] **Step 2: Run** `go test ./internal/sim/engines/ -run 'TestSeedDeterministic|TestEngines|TestRulesListed' -count=1`.
118 Expected: FAIL, package has no world.
119- [ ] **Step 3: Implement** the types above, `newWorld` (two boxes: box 0 the
120 engine's home, box 1 a calling box; one engine group `eng` with a
121 minted platform row `eng.acct.apps.example`; one public app group `shop`
122 on box 1; builds 1 to 4 with `Declares` and `ManifestOK` drawn per
123 world), a step loop of 400 chaos steps then 200 calm steps choosing among
124 registered actions (Tasks 2 to 5 register theirs in an `actions` slice),
125 and `final()`. `allRules` starts empty; each later task appends.
126- [ ] **Step 4: Run** the same command. Expected: PASS (no actions yet, no
127 violations).
128- [ ] **Step 5: Commit** `enginesim: the world, step loop and sweep (#335 plan 1)`.
129
130### Task 2: The fence and the launch gate (I1)
131
132**Files:**
133- Create: `internal/sim/engines/launch_test.go`, `internal/sim/engines/edge_test.go`
134- Modify: `rules_test.go`, `invariants_test.go`
135
136**Interfaces:**
137- Consumes: Task 1's types, `on`, `fail`, `group`.
138- Produces:
139 - `func (w *world) launch(b *box, g string, ver int, path string) bool`
140 (`path` is one of `adopt`, `swap`, `fallback`, `recovery`, `seed`,
141 `cmd`, `systemd`, `socket`; returns false when the gate refuses)
142 - `func (w *world) writeMark(b *box, g string, m mark)` (bumps `Wrote`)
143 - `func (w *world) reload(b *box)` (fails when `ReloadFails`; on success
144 swaps the map, bumps `Epoch` on any `0`→`1`, cancels every `Admitted`
145 request with an older epoch, then sets `Fenced = Wrote`)
146 - `func (w *world) fenced(b *box, g string) bool`
147 - `func (w *world) admitPublic(b *box, host, path string) *req`
148 (resolves alias, path map and minted row to the final row; refuses when
149 that row's group is internal or `NULL`; else appends to `Admitted`)
150 - `func (w *world) deliver(r *req)` (a held request resumes; records the
151 process it reaches)
152 - Invariants: **INV-P** (a public request is delivered to a process whose
153 build declares `doc_engine`), **INV-I1** (a declaring process exists
154 while `!fenced`).
155 - Rules: `set-before-repoint`, `fence-needs-reload`,
156 `fence-drains-admitted`, `fence-box-wide`, `gate-adopt`, `gate-swap`,
157 `gate-fallback`, `gate-recovery`, `gate-seed`, `gate-cmd`,
158 `gate-waits-when-written`, `unreadable-declares`,
159 `live-link-only-adoption`, `mark-on-final-upstream`.
160 - Actions: adopt, reload tick, reload-fails toggle, public request
161 (host, alias or path), hold and resume, wake with recovery or fallback
162 to `Good`, swap, seed, cmd, systemd restart, socket activation.
163 - Out of the model: deploy watch tickets (spec §5.8). A ticket request is
164 authenticated, not public; Plan 2 tests its scope (GET or HEAD, `/` or
165 the signed probe path, bound host and version) directly.
166
167- [ ] **Step 1: Write the failing witnesses** in `witness_test.go` (create it
168 with hostlife's `witness` struct and `TestWitnesses`), one per rule, each
169 a fixed sequence that violates the named invariant when its rule is off:
170 - `wSetBeforeRepoint` (R2-1): adopt declaring build over a public route;
171 a public request admitted between repoint and mark → INV-P.
172 - `wFenceDrainsAdmitted` (R3-1): request admitted at `0`, held across the
173 adoption, resumed → INV-P.
174 - `wFenceNeedsReload` and `wGateWaitsWhenWritten` (R10-1): reload failing,
175 adoption times out with `1` written, recovery wake picks a declaring
176 `Good` → INV-I1.
177 - `wGateFallback` (R9-1): `--public` (stubbed here as `writeMark(0)` after
178 stop) then a failed first wake falls back to declaring `Good` → INV-P.
179 - `wGateSwap` (R7-1), `wGateSeed`, `wGateCmd`, `wGateRecovery`: a
180 declaring build started by that path under `0` → INV-I1.
181 - `wUnreadableDeclares`: a build with `ManifestOK=false` and
182 `Declares=true` launched under `0` → INV-I1.
183 - `wFenceBoxWide` (R13-2): a request admitted through a `service` row that
184 is then deleted, held, resumed after a declaring launch → INV-P.
185 - `wMarkOnFinalUpstream` (R1-2): a public request via an alias and via a
186 path map to the engine → INV-P.
187 - `wLiveLinkOnly` (R11): systemd restarts a unit whose live link was moved
188 by something other than adoption → INV-I1.
189- [ ] **Step 2: Run** `ENGINESIM_MUTATE=<rule> go test ./internal/sim/engines/ -run TestWitnesses -count=1`
190 for one rule. Expected: FAIL to compile (no `launch`, `reload`).
191- [ ] **Step 3: Implement** the interfaces above. Each rule guards exactly
192 one step of the mechanism in spec §5.1 ("The invariant", "Setting the
193 mark"), and `on(rule)` false removes only that step.
194- [ ] **Step 4: Run** `go test ./internal/sim/engines/ -count=1` (unmutated:
195 every witness holds, the sweep is clean) and, for each rule,
196 `ENGINESIM_MUTATE=<rule> go test ./internal/sim/engines/ -run TestWitnesses -count=1`
197 (its witness violates). Expected: as stated.
198- [ ] **Step 5: Commit** `enginesim: the fence and the launch gate, I1 (#335 plan 1)`.
199
200### Task 3: Classification, minted groups and the operator clear
201
202**Files:**
203- Create: `internal/sim/engines/classify_test.go`
204- Modify: `rules_test.go`, `invariants_test.go`, `witness_test.go`
205
206**Interfaces:**
207- Consumes: Task 2's `launch`, `writeMark`, `reload`, `admitPublic`.
208- Produces:
209 - `func (w *world) upgrade(b *box)` (every instance group to `NULL`, then
210 classification before listeners: `1` if a grant in the group, the live
211 build declares, or any staged build declares; `0` if the live manifest
212 reads and none holds; else stays `NULL`)
213 - `func (w *world) operatorPublic(b *box, g string) bool` (spec §5.1
214 steps 1 to 4: under the adoption lock, refuse if live declares or a
215 grant remains; stop the group; refuse if any process of the group
216 remains; write `0`)
217 - Invariant **INV-AV** (at the end of the calm phase, every row of a
218 group whose mark is `0` answers a public request; liveness, checked in
219 `final()` only).
220 - Rules: `classify-null-refused`, `classify-staged`,
221 `minted-follow-owner`, `grant-marks-group`, `static-minted-public`,
222 `owner-nonempty-unique`, `public-checks-live`, `public-stops-group`,
223 `public-holds-lock`.
224 - Actions: upgrade, grant and revoke on any row of a group (minted
225 included), operator clear, add a TCP-backed instance and its minted row,
226 add a static app and its minted row and canary.
227- [ ] **Step 1: Write the failing witnesses**: `wClassifyStaged` (R8-1: an
228 awake route repointed without a restart, upgraded → INV-P),
229 `wClassifyNullRefused` (R2-2), `wMintedFollowOwner` (R5-1),
230 `wGrantMarksGroup` (R6-1: grant on the minted row of a non-declaring
231 release → INV-P via the minted host), `wStaticMintedPublic` (R6-2 →
232 INV-AV), `wOwnerNonEmptyUnique` (R7-2: two TCP instances of one app →
233 INV-AV with the rule off, ambiguous owner refused with it on),
234 `wPublicChecksLive`, `wPublicStopsGroup` (R7-1/R8-2: generation still
235 serving), `wPublicHoldsLock` (adoption repoints during the clear).
236- [ ] **Step 2: Run** the witnesses. Expected: FAIL to compile.
237- [ ] **Step 3: Implement** the interfaces above per spec §5.1
238 ("Platform-minted rows", "Becoming public is never automatic",
239 "Classification before serving").
240- [ ] **Step 4: Run** unmutated (all hold, sweep clean) and each rule
241 mutated against its witness (violates).
242- [ ] **Step 5: Commit** `enginesim: classification, minted groups, the operator clear (#335 plan 1)`.
243
244### Task 4: The dedicated engine box and ingress
245
246**Files:**
247- Create: `internal/sim/engines/dedicated_test.go`
248- Modify: `rules_test.go`, `witness_test.go`
249
250**Interfaces:**
251- Consumes: Tasks 2 and 3.
252- Produces:
253 - `func (w *world) addServiceRow(b *box, host, backend string)` (local or,
254 on box 1, aimed at box 0's address)
255 - `func (w *world) directConnect(b *box, port string, public bool) *proc`
256 (models a connection that bypasses the edge; reaches a process only if
257 the world's ingress precondition flag is violated or a `ports` listener
258 exists)
259 - Rules: `ports-excluded`, `dedicated-launch-check`,
260 `dedicated-edge-refusal`, `ingress-precondition` (the world flag; its
261 mutation lets direct connections through, showing the precondition is
262 load-bearing).
263 - Actions: add or delete a service row (local, remote), an engine binding
264 its own port, a release declaring `ports` with `doc_engine`.
265- [ ] **Step 1: Write the failing witnesses**: `wPortsExcluded` (R11-1),
266 `wDedicatedLaunchCheck` and `wDedicatedEdgeRefusal` (R12-2),
267 `wIngressPrivate` (R13-1: a remote service row aimed at the engine's
268 address with the precondition off → INV-P), `wOwnPort` (R12-1, same).
269- [ ] **Step 2: Run** the witnesses. Expected: FAIL to compile.
270- [ ] **Step 3: Implement** per spec §5.9 and §5.1 "The edge is the only
271 ingress".
272- [ ] **Step 4: Run** unmutated and mutated as in Task 2.
273- [ ] **Step 5: Commit** `enginesim: the dedicated engine box and ingress (#335 plan 1)`.
274
275### Task 5: Calls, identity, grants and arming
276
277**Files:**
278- Create: `internal/sim/engines/calls_test.go`
279- Modify: `rules_test.go`, `invariants_test.go`, `witness_test.go`
280
281**Interfaces:**
282- Consumes: Tasks 1 to 4.
283- Produces:
284 - `type record struct{ Rev, Expires int; Grants map[string]bool }` (key
285 `account/app/engine`) and `type mirror struct{ Rec *record; Floor, ReadAt int; Persisted *record; PersistedFloor int }`,
286 added to `box` as `Mirror mirror`
287 - `func (w *world) publish(rev int)`, `func (w *world) replay(rev int)`
288 (a bucket writer restores an older signed record),
289 `func (w *world) mirrorTick(b *box)`, `func (w *world) restartEdge(b *box)`
290 - `func (w *world) engineCall(from *box, p *proc, viaDoor bool, uid string, host, engine, path string)`
291 - `func (w *world) launchUnisolated(b *box, kind string) bool` (`flat`,
292 `foreign`, `worker`; refused while armed), `func (w *world) arm(b *box) bool`,
293 `func (w *world) disarm(b *box)`
294 - Invariants: **INV-G** (an engine call reaches an engine route although
295 the newest record the console published does not grant it, beyond the
296 freshness bound, or reaches a route other than the engine's exact
297 route), **INV-ID** (the caller the engine box records differs from the
298 process's real group), **INV-ARM** (an unisolated tenant process
299 exists on an armed box while an engine socket is open).
300 - Rules: `cgroup-identity`, `door-uid`, `hop-caller-signed`,
301 `engine-box-recheck`, `exact-route-engine`, `floor-persisted`,
302 `stale-clears-grants`, `armed-refuses-unisolated`, `arm-checks-box`.
303 - Residual flags (off by default, each with a test): `freshBoxReplay`,
304 `unwrittenManifestAcrossUpgrade`, `flatEraTamper`; when on, the world
305 records `residual:<name>` instead of a violation for the matching
306 sequence.
307- [ ] **Step 1: Write the failing witnesses**: `wCgroupIdentity` (R1-1),
308 `wDoorUID` (R2-3), `wHopCallerSigned` (a client-set caller header),
309 `wEngineBoxRecheck` (G4: calling box table stale), `wExactRouteEngine`
310 (R4-2: path map from engine A's host to engine B), `wFloorPersisted`
311 (R1-4: revoke, replay older record, restart), `wStaleClearsGrants`,
312 `wArmedRefusesUnisolated` (R3-2/R4-1: foreign and worker launches),
313 `wArmChecksBox`; and `TestResidualsAreCountedNotFailed` (each flag on:
314 `residuals` non-empty, `violations` empty).
315- [ ] **Step 2: Run** the witnesses. Expected: FAIL to compile.
316- [ ] **Step 3: Implement** per spec §5.2 to §5.4 and §6.2.
317- [ ] **Step 4: Run** unmutated and mutated as in Task 2, and the residual
318 test.
319- [ ] **Step 5: Commit** `enginesim: calls, identity, grants and arming (#335 plan 1)`.
320
321### Task 6: The mutation harness and the first sweep
322
323**Files:**
324- Create: `internal/sim/engines/mutate.sh`, `internal/sim/engines/expected-outcomes.txt`
325- Modify: `internal/sim/engines/sim_test.go`, `witness_test.go`
326- Modify: `docs/superpowers/specs/2026-10-21-fleet-internal-engines-design.md` (§8: the sweep result)
327
328**Interfaces:**
329- Consumes: every rule and witness from Tasks 2 to 5.
330- Produces: `mutate.sh <seeds> [rule|rule,rule|@all]`, hostlife's
331 verdicts (`CAUGHT`, `MISSED`, `ERROR`, `EXPECTED-MISS`, `STALE`) with the
332 `ENGINESIM_` variables; `expected-outcomes.txt` in hostlife's format.
333
334- [ ] **Step 1: Write the failing tests**: `TestWitnessesHoldUnmutated`
335 (every witness passes with `mutated == ""`), `TestEveryActionFires`
336 (over 200 seeds every registered action's `stats` count is above zero),
337 `TestEveryRuleHasWitness` (each name in `allRules` has a witness or an
338 `expected-outcomes.txt` line).
339- [ ] **Step 2: Run** `go test ./internal/sim/engines/ -count=1`. Expected:
340 FAIL until the harness file and coverage hold.
341- [ ] **Step 3: Implement** `mutate.sh` by adapting
342 `internal/sim/hostlife/mutate.sh` (same build-once, run-per-rule shape,
343 `ENGINESIM_` names, scratch `$TMPDIR/enginesim`), and fill
344 `expected-outcomes.txt` only for rules a reason makes uncatchable alone.
345- [ ] **Step 4: Run the sweep** once, serial and niced:
346 `go test -c -o "$TMPDIR/enginesim/engines.test" ./internal/sim/engines/ && nice -n 19 internal/sim/engines/mutate.sh 3000`.
347 Expected: every rule `CAUGHT` or `EXPECTED-MISS`, no `MISSED`, `ERROR`
348 or `STALE`; the unmutated sweep at 3000 seeds has no violation. Remove
349 `$TMPDIR/enginesim` afterwards.
350- [ ] **Step 5: Record** the verdict table in spec §8 under "Simulation"
351 (rules, verdicts, first seeds, the date), and if any rule is `MISSED` or
352 the unmutated sweep fails, stop: write the defect up on #335 and bring it
353 to Paul before Plan 2.
354- [ ] **Step 6: Commit** `enginesim: mutation harness and the first 3000-seed sweep (#335 plan 1)`.
355