Isolated Exec, Plan 4: amendments before execution (2026-09-25)

Plan 4 (2026-09-12-isolated-exec-4-layout.md) was written against main at c1d40ee and spec revision 14. Before building it on branch isolated-exec-layout (cut from 52f71143, spec revision 15), an audit checked it against main. These rulings amend the plan and take precedence over its text where they differ. Each names the task it changes. LB means load-bearing: it must be applied when that task runs.

The audit's evidence (the drift table, semantic findings S1 to S15, and the conflict table C1 to C13) is summarised in each ruling. Line references are to main at 52f71143.

R1 (LB, T3/T4; spec §8.2 step 2, §6.3, §6.7). How the pass proves a host is stopped. > Add func (a *Service) Fenced(host string) bool and func (a *Service) StopUncertain(host string) bool (true when stopHistory[host] exists and is not complete, or when an entry exists) to internal/activator/maintenance.go. In the pass, treat FenceAndStop returning ErrStopUnproven as quiesced only when svc.Quiesced(host) && !svc.StopUncertain(host) and the agent's startup unit sweep succeeded (pass a SweptAtStart bool into Pass). Otherwise, lift the fence at this pass's generation, return Result{Outcome: "deferred", Reason: "stop unproven"}, and write no journal. Never leave a layout fence installed after a refused quiesce. Add a test that runs Converge against a real testActivator service for an entry-less host and asserts it migrates and that the host can wake afterwards.

R2 (LB, T1/T3; #175 instance-moves §6.1 G12, spec §8.2 step 2). The layout fence and the move fence. > Do not add Unfence. Use the existing activator.(*Service).LiftFence everywhere (the Fence interface is FenceAndStop + LiftFence). Drop TestUnfenceReleasesOnlyTheMatchingGeneration. In convergeHost, skip with Outcome: "deferred" any row whose Status is neither "" nor "ready" (this covers the cut's "moving"), and any host where svc.Fenced(host) is already true before this pass fences. After FenceAndStop, re-read the row and also refuse if its Status is no longer ready or it is no longer ActivatorManaged. The pass lifts only its own generation. Add a test with a pre-installed fence at a lower generation: the pass defers the host and the original fence is still in place afterwards.

R3 (LB, T3/T4; spec §8.5, §13 R15). The probe gate. > Pass gains NestedOK func() error. The agent passes the same nestedGate(instancefs.MountNoSymfollow, eb.Control.Ready) value it gives eb.NestedOK (agent.go:330). Converge(ModeNested) calls it once per pass before touching any host. On error it returns a single Result{Host: "", Outcome: "refused", Reason: "isolation not ready: <err>"} and moves nothing. Converge(ModeFlat) and Recover are never gated.

R4 (LB, T3/T4; spec §3.3, §8.2 "Recovery is the journal"). Recovery converges before it releases. > The signature becomes Recover(ctx, intent Mode), where "" means no intent. recoverHost completes the journal under its fence. If intent != "" && intent != j.Mode, it then runs the migrate steps for that host under the same fence generation, and releases only once the host is at the intent. Converge(ctx, mode) calls Recover(ctx, mode). At agent start, run Recover after eb.Control.Sweep succeeds and before svc.Start() (agent.go:367-372), with the intent loaded by one LoadLayoutIntent from the deploy store. If the intent cannot be read, complete journals but keep any host whose completed endpoint is flat and whose pre-journal layout was nested fenced (journal kept at done) until an intent is read. X5's test must record fence lifts and assert none happens between recovery and convergence.

R5 (LB, T2/T3; spec §8.2 steps 3 and 6, §3.1). The two missing artifacts. > Append two optional artifacts to artifactNames, after socket: litestream-shadow (<dir>/.activator-<host>.db-litestream, a directory, derived as lease.go:287 does) and corrupt (<db>.corrupt). The destination check treats an existing destination shadow dir as a refusal (two replication states), like db, and moves an existing destination .corrupt aside, like env. Validate's length check becomes 9.

R6 (LB, T3; spec §3.2, §3.3). Renames through directory fds. > Add internal/instancefs helpers, MkdirBeneath(dir, name string, perm os.FileMode) error and RenameBetween(srcDir, srcName, dstDir, dstName string) error. Build them on the existing openDir (openat2 RESOLVE_NO_SYMLINKS|RESOLVE_BENEATH) plus unix.Mkdirat / unix.Renameat, with !linux stubs returning errors.ErrUnsupported. EnsureDirs creates <data>/.instances/<iid> and <run>/.instances/<iid> through MkdirBeneath from fds on <data>/.instances and <run>/.instances, then refuses unless an Lstat says it is a directory. Every artifact move uses RenameBetween. Add a test that plants <data>/.instances/<iid> as a symlink and asserts the pass refuses and moves nothing.

R7 (LB, T1; spec §8.2 first paragraph). The intent's wire shape. > The shape is release.LayoutIntents{Schema int; Requests []LayoutIntent}, oldest first, at control/layout.json. LoadLayoutIntent(ctx, s, boxLabel) returns the newest request whose scope applies to boxLabel. AppendLayoutIntent(ctx, s, in) is the CAS-append writer, mirroring AppendRestartScope (restartscope.go:259) including the retry. A schema above 1 is an error, as planned.

R8 (LB, T2/T3; plan's own tests). One socket directory. > PlanFor derives both socket endpoints from its socketDir argument: flat filepath.Join(socketDir, host+".sock"), nested filepath.Join(socketDir, "instances", InstanceID(host), "app.sock"), which is exactly LayoutInputs.sockets (layout.go:50-56). Pass.layoutInputs passes p.socketDir(), not the raw field. In box() and every T3 fixture the row's Socket is filepath.Join(rd, host+".sock"). Production is unchanged, because both default to provision.SocketDir.

R9 (LB, T6; spec §8.2 "That order is enforced"). Where the downgrade guard goes. > Put the guard in systemApplier.install right after newPlan, newUnits, err := a.stagePlan(ctx, staged, newDir) (systemapply.go:339). The candidate predates the layout when !slices.Contains(newUnits, "data-instances.mount"). Delete candidatePredatesLayout(stagedRoot). Refuse with a.record(req, fmt.Errorf("%w: …", errSystemRefused, …)) before the journal is written. The tests cover a pre-layout unit set with a nested row (refused), the same set with every row flat (installs), and a post-layout set with nested rows (installs).

R10 (LB, T5; spec §8.4). The instance page's layout fact. > Do not add RouteView.Layout or touch Assemble. Add Layout string to instanceDetail (instancepage.go:13-18). Fill it in handleAppInstance from one status.LoadBoxes read: the first box whose report names the host, otherwise unknown (box predates layout reporting). The template line is <dt>Layout</dt><dd>{{.Layout}}</dd> inside {{with .InstanceDetail}}. Members see the word only, never the reason (reasons carry paths). The strings go through copy review before merge.

R11 (LB, T3; spec §3.2 "a CI test greps the tree"). The grep test's pattern. > The grep test matches the path literal, not the substring: pattern ["/]\.instances\b over --include=*.go, across internal/ and the repository root's *.go. The allowlist is internal/layout/ and internal/host/bootstrap.go. Before committing, confirm the test passes on current main plus this task (the substring pattern fails on console/server.go:2932 and instances_cli.go:109 today).

R12 (minor, T6; spec §8.2 downgrade rules, gap). Guard carlos add too. > In runAdd, before Upsert, if a row for the host exists and provision.NestedPaths(dataDir(), existing) is true, log.Fatalf("add %s: its paths are nested; layout revert first", host). Record this in the task report as a spec gap, so the spec owner can add it to §8.2's list.

R13 (minor, T4; spec §8.2 step 6). Serialize with config sync. > migrateHost and recoverHost hold the agent's configSyncMu (agent.go:913) from journal write to journal removal. The pass takes a Serialize func(func()) seam; the agent wires it to configSyncMu.Lock/Unlock, and it is nil in tests. Keep the fenced skip in SyncExec as a second net. Add it as a variadic option (...SyncOption) so the 62 existing call sites stay as they are.

R14 (minor, T4; spec §8.2 "when it is asleep"). Keep the serial loop moving. > layoutLeg runs Converge in its own goroutine behind a TryLock single-flight, with a dirty flag that re-runs once if a serial move arrives mid-pass. On the serial-move leg, convergeHost defers a host that is awake (!svc.Quiesced(host)) with Outcome: "deferred", Reason: "awake". Insert the leg after scheduleLeg and before systemLeg.observe (agent.go:1657-1667).

R15 (minor, T5; snapshot.go:295-305, contract_test.go:26-80). Status placement. > Put the report under the existing private exception as BoxMetrics.Layout *LayoutReport, not at the Box root. Add "layout", "metrics.layout": {"mode","rows"} and "metrics.layout.rows": {wildcardAnyKey} to allowedBoxKeys, and assert allowedFleetKeys has no layout. Report only instance-kind rows: external for unflagged socket instances, and skip Kind "sidecar". Mode is "none" when the last-seen intent is "". The console door also reads fleets/<name>/ boxes the way apisystemupdate.go:233 does.

R16 (minor, T1/T4 tests; store.go:37-41). > The newer-schema fixture is written by s.Delete(ctx, key) then s.Put(ctx, key, body, ""). The T4 test switches intents by Get for the etag, then Put with that etag (or through AppendLayoutIntent once R7 lands).

R17 (minor, T3; go vet lostcancel). > deadline(ctx) returns (context.Context, context.CancelFunc) and each caller defer cancel()s. No //nolint.

R18 (minor, T2/T3; review hygiene). > Delete errNoSuchArtifact and var _ = filepath.Join. List journal_linux.go/journal_other.go under Files. Rename the test to TestReadJournalRefusesASymlink, or add a FIFO case (unix.Mkfifo, linux-only file). Define openLayoutRegistry(t, dir) in pass_linux_test.go as registry.Open(filepath.Join(dir, "registry.db")) plus t.Cleanup(Close).

R19 (minor, T3; spec §3.1). > Preflight refuses when len(plan.To.Socket)+len(".control") > provision.MaxUnixPath (for mode nested), naming the length.

R20 (minor, T4; agent.go:510-520). > Publish the edge's reload the way setUpstreamSource publishes probes: var layoutReload atomic.Pointer[func()], set in the RunWithActivatorReady hook to func() { _ = e.Reload() }. Pass.Reload loads it and is a no-op before the edge is ready, when the 2 s poll covers.

R21 (minor, all; current file locations). > - carlos status --layout goes in RunStatus (cli.go:3342-3375) and statusUsage. The fetch and render go in status_instances.go beside statusInstances. The CLI table test is required. > - The unflag refusal goes in runRouteHibernate (cli.go:1386, before :1443). > - Reconcile edits go at reconcile.go:967-978 (flag) and :1011-1021 (drift). > - The door is registered as s.cliRoute("GET /api/cli/status/layout", svcrole.Set{}, s.handleAPIStatusLayout), gated by cliUser + s.store.IsOperator → 404, as in apistatusinstances.go:91-104. > - One predicate, layout.JournalPresent(dd, host) (Lstat not-ErrNotExist), is used by the pass, SyncExec's option, the litestream fenced hook in ops.go:117 (via provision.LayoutJournalPath, no layout import needed) and layoutReversalIncomplete.

R22 (minor, T5/T6 docs; spec §8.4, house copy rules). > T5 documents carlos status --layout only. It does not add a #### Instance layout section for a verb that does not exist yet; plan 5 writes that section. T6's sentence under ### carlos system states the refusal and names "revert the layout first" without a command line. All new user-facing strings (the CLI help, the refusal messages, the page fact) go through copy review.

R23 (minor, plan header; D49, D50). > Cite spec revision 15. The branch is isolated-exec-layout. Gate env is GOCACHE=$TMPDIR/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp/g. Use the commit trailer from the controller's current attribution.

R24 (minor, T3 tests; S5). > Add a Replica: true flat managed row to box() and assert it migrates and reverts like any other. There is no replica_of column; the replica is its own host.

Load-bearing rulings: 11 (R1-R11). Minor: 13 (R12-R24).

R25 (minor, T5/T6; copy-review rule). > The user-facing strings T5 and T6 introduce (CLI help and table headings for carlos status --layout, the instance page's Layout fact and its unknown text, the refusal messages from system apply, carlos route --hibernate=false, carlos add and the reconciler, and the docs/site/cli.md sentences) go through copy review with Paul before they are written into files. No em dashes.

Findings the rulings answer

S1. Quiesce is unprovable for every asleep host after an agent restart, and the plan then never releases the fence. For an entry-less host, FenceAndStop returns ErrStopUnproven unless stopHistory[host] is complete (maintenance.go:90-99). stopHistory is in-process memory (activator.go:626). After any restart or roll, no asleep host has history. The plan's global constraint and migrateHost keep the fence installed on ErrStopUnproven and never release it. The first Converge under a nested intent would therefore fence every asleep managed flat host on the box. doWake then refuses them all with ErrFenced (activator.go:1060) until the next agent restart: a box-wide outage. Recover at start (before svc.Start, so there are no entries) hits the same answer. The journaled host stays fenced, never wakes, never gains history, and never recovers. The plan's fakeFence returns success, which hides all of this. The #175 worker meets the same fact and handles it: it ignores ErrStopUnproven and asks svc.Quiesced(host) instead (movespass.go:104-114, 395).

S2. The #175 move fence and this plan's layout fence share one slot per host, and they collide. a.fences is map[host]uint64 (activator.go:624). FenceAndStop overwrites any prior fence at a lower-or-equal generation (maintenance.go:80-85). LiftFence deletes only on an exact match (:133-145). The move's generation is Move.RequestedAt.UnixNano() (movespass.go:240). The layout pass's default is time.Now().UnixNano(), which is later. Consequences:

• A layout pass on a host whose cut is in progress (Status "moving", fence installed) overwrites the move's fence, then lifts its own generation. That removes the move's G12 fence until the next moves round re-installs it. The cut itself triggers this: nudgeSerial (movespass.go:363-366, 432) bumps the serial, and the serial move runs fullPass and then the layout leg in the same tick.
• A layout fence left installed after an unproven stop (S1) is always newer than any move requested earlier. Every later cfg.Fence then fails with ErrFenceGeneration, which is not ErrStopUnproven, so the cut errors every pass and is stuck until restart (movespass.go:110-113, 364-367).
• runMovesPass also runs from instancesLoop (agent.go:2334), a different goroutine from fastTick. The layout pass and a moves round can therefore interleave. The move's journal check (movespass.go:381) covers files mid-move, but only if the layout pass never lifts a fence it did not install.
• A moving row is about to leave this box. Migrating it is wasted work that races the purge (PurgeLocalDBArtifacts(dbPath), movespass.go:425).

S3. SyncExec and litestream race the pass. SyncExec runs from configSyncLoop and configGenLoop through syncExecSerialized (agent.go:918-922), on goroutines separate from fastTick. The flat writer uses a fixed temp name <host>.config.env.tmp and then a rename (syncexec.go:1176-1180). A per-route fenced(host) check before rendering is a check-then-act. SyncExec can pass the check, the pass can then move config.env, and SyncExec then lands a flat config.env for a row that is now nested. That leaves an orphan, or two bundles. The litestream generator does not matter: hibernating rows are excluded already (litestream.go:344-346, 436-438), and only hibernating rows migrate.

S4. The probe gate (R15, §8.5) is not consulted before migrating to nested. eb.NestedOK = nestedGate(instancefs.MountNoSymfollow, eb.Control.Ready) (agent.go:330, 700-715) refuses every nested wake unless the three mounts carry nosymfollow and /ready answers ready. On dev, the probe answered not-ready for 13 days (R15 a/b). The plan's Converge(nested) would still move rows. Each moved flat host that woke fine would become a nested host that refuses to wake. §8.5: "A box that cannot establish the namespace does not start nested tenants."

S5. Replica rows (#162) need no special case, and there is no replica_of. A replica is its own host with its own record (provision/replica.go:44-66). It is minted flat by routeFor with Replica: true (reconcile.go:364-380), and only on the box its placement names (reconcile.go:202-211). #175 notes that a replica never gets a route on a platform box (provision.go MoveRequest, "G14") and refuses to move one. Its DB, env files and socket are named after its own host, so PlanFor derives them correctly and it migrates like any activator-managed row. Nothing reads a replica's paths across hosts. The home↔replica lease protocol is titogo's own, over the network. No change is needed beyond a test row.

S6. Sidecar rows (Kind sidecar, merged from sidecar-members) are outside the pass, and status must say so. sidecarRouteFor mints DBPath: DefaultDBPath(dataDir, sidecarHost) with Socket "" (provision/sidecarreconcile.go, sidecarRouteFor). ActivatorManaged excludes them (Kind != instance) and so does Classify's conflict scan. The plan's status rows say "external for an unflagged socket instance route", and a sidecar is not one. Without an explicit skip, a sidecar would render as nothing or as foreign. coSpawnStaged (sidecarreconcile.go:156) is a new hand-built flat-path reader of <DataDir>/<host>-sidecar.env, but its host is a sidecar host (*.sidecar.internal, sidecar.go:31), never a migratable row. No change needed.

S7. Per-environment and private object storage (#412) add no on-box path. 96d4dacb says "Nothing on a box changes: config delivery already resolves per (account, app, environment)". 91d16d0a is a console and manifest policy. Neither introduces a host-named file under DataDir. No impact.

S8. The §9 sweep for new hand-built consumers since c1d40eeb found no new reader of a migratable host's flat shape. The diff grep of non-test Go for DefaultDBPath|DefaultSocket|.DBPath|.env"|config.env|-sidecar.env|.sock" finds four:

• movespass.go:393-397, which now follows the row;
• status DBBytes(r.DBPath), which follows the row;
• sidecarRouteFor, sidecar rows only;
• coSpawnStaged (S6).

Pre-existing readers listed in spec §9 are unchanged. No ruling needed.

S9. carlos add on an existing nested host re-flattens it: a third unflag and flatten path the downgrade guards miss. runAdd defaults --db to DefaultDBPath (cli.go:139). Upsert's ON CONFLICT overwrites socket, db_path and hibernate (registry.go:987-1006). SetHibernate's own doc names it as the third writer (:1705-1708). A re-add of a nested host leaves the row flat and unflagged over nested files. The next wake restores a second database at the flat path. Spec §8.2 lists only carlos route --hibernate=false and the reconciler. §8.3 names carlos add as the repair for foreign rows, which is still fine, because foreign ≠ nested.

S10. Recovery releases the host before converging it, and at start it does not converge at all. Spec §3.3 and §8.2 say a journal that passes is completed to its own end and then "converged to the current intent in the same pass, before it is released to the activator". Plan recoverHost does defer p.Fence.LiftFence(...) and returns. Converge fences again later. At start, the plan runs Recover only. fastTick's first successful serial read only seeds (agent.go:1624-1638), so no converge happens until someone bumps the serial. §3.3's forged-journal argument depends on that ordering. A flat tenant (same uid) can forge a valid mode: flat journal for a nested host. Recovery reverts it, releases it, and the host then runs unisolated until the next deploy.

S11. Two spec-named artifacts are missing from the on-disk artifact list. §8.2 step 3: "-wal, -shm, the litestream shadow dir and .corrupt are derived from it". The shadow dir is <dir>/.activator-<host>.db-litestream (lease/lease.go:287). The quarantine is <db>.corrupt (backend_exec.go:50, 314). Step 6 names "a quarantine" as an optional artifact. The plan's artifactNames has seven entries and calls itself "part of the on-disk contract". Adding entries later changes the journal format.

S12. Renames and directory creation use path strings that a same-uid tenant can redirect. /data/.instances is 0700 carlos (bootstrap.go:69), and during the transition flat tenants run as carlos (spec §1.3). A tenant can plant /data/.instances/<iid> as a symlink before EnsureDirs. os.MkdirAll accepts an existing symlink to a directory, and os.Rename(src, "/data/.instances/<iid>/<host>.db") then lands the DB wherever the link points. Spec §3.2: the raw path is opened "only as openat2(RESOLVE_NO_SYMLINKS | RESOLVE_BENEATH) directory fds for renameat".

S13. The intent object's wire shape differs from the spec and will be frozen by the first rolled box. The spec says control/layout.json is "{id, requested_at, mode, scope}, the newest entry of an append log like restart-scope.json" (§8.2), i.e. {schema, requests:[…]} (restartscope.go:113-120). The plan reads a single flat object. Plan 5 writes the spec's shape. A plan-4 box reads mode:"", ValidLayoutMode fails, it logs "not converging", and it never converges.

S14. Status placement, privacy and fleet boxes. Box fields other than Metrics and Sidecars are "meant for the public page" (snapshot.go:300-305). The contract allowlist (contract_test.go:26-80) fails on a new key, and it handles hostname-keyed maps with wildcardAnyKey under metrics (:78-79). The plan puts a hostname-keyed Layout at the Box root. Separately, customer-fleet boxes publish under fleets/<name>/status/boxes/ (fleets.go:997; apisystemupdate.go:233). A door reading only status.LoadBoxes(ctx, s.cfg.Releases) misses them.

S15. Converge runs inline in the 2 s serial loop and stops awake hosts. fastTick runs its legs synchronously (agent.go:1642-1669). FenceAndStop on an awake host waits for the stop, and the plan gives it up to 60 s per host. That stalls the restart, wake, schedule and system legs and the fast sweep. Under a standing nested intent, every freshly minted flat row that is awake gets bounced on the next serial move (every deploy or config write). Spec §8.2: a later-eligible row "is migrated on the next pass when it is asleep".

Count: 15 findings. 12 need action. S5, S7 and S8 are evidence of no impact.

---