amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Isolated Exec, Plan 5 of 5: The Console Door and the Last Two Guards
2
3> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
5**Goal:** Give an operator the verb that records the fleet's layout intent (`carlos system layout migrate|revert`, optionally `--box`), through a console door that writes `control/layout.json` and bumps the serial so every box in scope converges (plan 4's pass); refuse, at the console, a platform rollback or update to a release before the layout while any box still reports a nested, fenced or conflict row or no layout field; and make the root control service inspect the underlying mountpoint directories at start so a layout that was unmounted by hand cannot leave two databases.
6
7**Architecture:** The CLI verb follows `carlos system rollback` (a `requireDeployAPISession`, a confirmation, one POST). The door follows `handleAPISystemRollback` (owner gate, bucket check, strict JSON, audit, serial bump) and writes the intent with `release.PutLayoutIntent` under a read-then-CAS. The downgrade refusal is one helper both system doors call before moving a pointer, reading `status.LoadBoxes` and the `Layout` report plan 4 added. The mountpoint inspection is a `Run` seam on `instancectl.Server` that executes `unshare -m` as root and folds a verdict line into every probe report.
8
9**Tech Stack:** Go 1.26, `internal/release` (`LayoutIntent`, `SystemPointer`, `SemVer`), `internal/console`, `internal/status`, `internal/instancectl`.
10
11**Spec:** `docs/superpowers/specs/2026-09-11-isolated-exec-design.md` (revision 14): §8.2 (the verb's shape, scope, convergence; the downgrade rules; the unmounted-mountpoint recovery), §8.4, §8.6, §10 (`internal/console` and `system.go` bullets), §13 R5-2. Plan 4 defines `LayoutIntent` and the status `Layout` report this plan consumes.
12
13## Global Constraints
14
15- Gate and env as plan 4. Console security gates are ordered session, CSRF, owner (byte-identical 404), fresh auth last (AGENTS.md); the layout door uses `apiSystemOwner` exactly as the update and rollback doors do.
16- The intent object is the only writer of a nested path anywhere in the platform, indirectly: recording `nested` is an operator's act, audited, and needs a confirmation in the CLI unless `--yes`.
17- The downgrade refusal is by observed state (each box's `Layout` report), never by the intent, and a box that reports no layout field or is stale counts as unknown and blocks.
18- `LayoutFloor` is the first release that carries the mount units; it is a constant with a test, and the plan's report names it so it can be corrected when the release is cut.
19- Copy: no em dashes; comments are full sentences saying why; log lines use `->`.
20- Commit after every task with the trailer `Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>`.
21
22---
23
24## File Structure
25
26| File | Responsibility |
27|---|---|
28| `internal/release/layoutintent.go` (modify) | `LayoutFloor`, `PreLayout(version string) bool`; `RecordLayoutIntent(ctx, s, mode, box, actor)` (read etag, CAS put, 3 retries, fresh id) |
29| `internal/console/apisystemlayout.go` (create) | `handleAPISystemLayoutGet`, `handleAPISystemLayoutSet` |
30| `internal/console/authcli.go` (modify) | two routes beside `/api/cli/system/rollback` |
31| `internal/console/layoutguard.go` (create) | `layoutDowngradeBlocked(ctx, s, target string) (reason string)` |
32| `internal/console/apisystemupdate.go` (modify) | rollback and update doors call the guard before moving a pointer |
33| `system.go` (modify) | `runSystemLayout`; `systemVerb` learns `layout`; `systemUsage` |
34| `docs/site/cli.md` (modify) | the `#### Instance layout` subsection plan 4 stubbed gets the verb |
35| `internal/instancectl/server.go` (modify) | `Run` seam; `inspectMountpoints` at start; `startLines` folded into every report |
36| `internal/instancectl/server_test.go` (modify) | inspection tests with a fake `Run` |
37| `internal/host/units/carlos-instance-control.service` (modify) | nothing new needed: root, `ProtectSystem=strict` leaves `/data` visible; add a comment on the inspection |
38| spec §8.4 (modify) | `carlos instances --layout` becomes `carlos status --layout` (revision 15, §13 R14) |
39
40---
41
42### Task 1: The intent writer and the console door
43
44**Files:**
45- Modify: `internal/release/layoutintent.go`, `internal/release/layoutintent_test.go`
46- Create: `internal/console/apisystemlayout.go`, `internal/console/apisystemlayout_test.go`
47- Modify: `internal/console/authcli.go`
48
49**Interfaces:**
50- Produces:
51 ```go
52 // release
53 func RecordLayoutIntent(ctx context.Context, s Store, mode, box, actor string) (*LayoutIntent, error) // fresh 16-byte hex id, RequestedAt now, CAS over the existing etag, 3 retries on ErrCASConflict
54 // console
55 GET /api/cli/system/layout -> 200 {"mode":"none|flat|nested","box":"","id":"","requested_at":""} (owner gate; 200 with mode none when absent)
56 POST /api/cli/system/layout {"mode":"nested|flat","box":"<label or empty>"} -> 200 {"mode","box","id"}; 400 on a bad mode or a label that is not a box label shape; 404 for a non-owner
57 ```
58
59- [ ] **Step 1: Write the failing tests**
60
61`internal/release/layoutintent_test.go` (append):
62
63```go
64func TestRecordLayoutIntentReplacesUnderCAS(t *testing.T) {
65 s := NewDirStore(t.TempDir())
66 ctx := context.Background()
67 first, err := RecordLayoutIntent(ctx, s, LayoutModeNested, "kass-1", "op@example.com")
68 if err != nil || first.ID == "" || first.Mode != LayoutModeNested || first.Scope.Box != "kass-1" {
69 t.Fatalf("first = %+v %v", first, err)
70 }
71 second, err := RecordLayoutIntent(ctx, s, LayoutModeFlat, "", "op@example.com")
72 if err != nil || second.ID == first.ID || second.Mode != LayoutModeFlat || second.Scope.Box != "" {
73 t.Fatalf("second = %+v %v", second, err)
74 }
75 got, _, _ := LoadLayoutIntent(ctx, s)
76 if got.ID != second.ID {
77 t.Fatalf("object holds %s, want the latest %s", got.ID, second.ID)
78 }
79 if _, err := RecordLayoutIntent(ctx, s, "sideways", "", "op"); err == nil {
80 t.Fatal("bad mode accepted")
81 }
82}
83```
84
85`internal/console/apisystemlayout_test.go` (copy the fixture of the nearest `apisystemupdate_test.go` test that exercises `apiSystemOwner`: a server with an ops-owner principal, a member principal, and a `DirStore` as `cfg.Releases`):
86
87```go
88func TestSystemLayoutDoorIsOwnerOnlyAndRecordsTheIntent(t *testing.T) {
89 s, owner, member := newSystemDoorFixture(t) // as the rollback door test builds it
90 // A member gets the byte-identical 404.
91 rec := postAs(t, s, member, "/api/cli/system/layout", `{"mode":"nested"}`)
92 if rec.Code != 404 {
93 t.Fatalf("member: %d", rec.Code)
94 }
95 rec = postAs(t, s, owner, "/api/cli/system/layout", `{"mode":"sideways"}`)
96 if rec.Code != 400 {
97 t.Fatalf("bad mode: %d %s", rec.Code, rec.Body)
98 }
99 rec = postAs(t, s, owner, "/api/cli/system/layout", `{"mode":"nested","box":"kass-1"}`)
100 if rec.Code != 200 {
101 t.Fatalf("record: %d %s", rec.Code, rec.Body)
102 }
103 in, found, err := release.LoadLayoutIntent(context.Background(), s.cfg.Releases)
104 if err != nil || !found || in.Mode != "nested" || in.Scope.Box != "kass-1" {
105 t.Fatalf("object = %+v %v %v", in, found, err)
106 }
107 if !s.serialBumped() { // whatever the fixture exposes for bumpSerial; else read the serial object
108 t.Error("serial not bumped")
109 }
110 rec = getAs(t, s, owner, "/api/cli/system/layout")
111 if rec.Code != 200 || !strings.Contains(rec.Body.String(), `"mode":"nested"`) {
112 t.Fatalf("get: %d %s", rec.Code, rec.Body)
113 }
114}
115```
116
117- [ ] **Step 2: Run the tests to verify they fail**
118
119Expected: build failure.
120
121- [ ] **Step 3: Write the code**
122
123`layoutintent.go` (append):
124
125```go
126// LayoutFloor is the first platform release that carries the instance
127// mount units. A pointer moved below it while any box is nested would
128// unmount the layout from under a nested row (design §8.2), so both system
129// doors refuse such a move until every box reports flat. Correct this
130// constant when the release carrying the layout is cut.
131var LayoutFloor = SemVer{Major: 0, Minor: 20, Patch: 0}
132
133// PreLayout reports whether a release tag predates the layout. A tag that
134// does not parse is treated as pre-layout: refusing is the safe answer for
135// a version nobody can place.
136func PreLayout(version string) bool {
137 v, ok := ParseSemVer(strings.TrimPrefix(version, "v"))
138 if !ok {
139 return true
140 }
141 return v.Compare(LayoutFloor) < 0
142}
143
144// RecordLayoutIntent replaces the intent with a fresh one. A CAS conflict is
145// retried: the other writer recorded a different intent and ours must
146// still land, with its own id, as the newer one.
147func RecordLayoutIntent(ctx context.Context, s Store, mode, box, actor string) (*LayoutIntent, error) {
148 if !ValidLayoutMode(mode) {
149 return nil, fmt.Errorf("layout intent: mode %q is not flat or nested", mode)
150 }
151 id := make([]byte, 16)
152 if _, err := io.ReadFull(rand.Reader, id); err != nil {
153 return nil, err
154 }
155 in := &LayoutIntent{Schema: 1, ID: hex.EncodeToString(id), RequestedAt: time.Now().UTC().Format(time.RFC3339), Mode: mode, Scope: LayoutScope{Box: box}}
156 var lastErr error
157 for attempt := 0; attempt < 3; attempt++ {
158 _, etag, err := s.Get(ctx, LayoutIntentKey())
159 if err != nil && !errors.Is(err, os.ErrNotExist) {
160 return nil, err
161 }
162 if err != nil {
163 etag = ""
164 }
165 if err := PutLayoutIntent(ctx, s, in, etag); err != nil {
166 if errors.Is(err, ErrCASConflict) {
167 lastErr = err
168 continue
169 }
170 return nil, err
171 }
172 return in, nil
173 }
174 return nil, fmt.Errorf("layout intent: %s is being written by someone else; nothing was recorded: %w", LayoutIntentKey(), lastErr)
175}
176```
177
178(Check `ParseSemVer`'s treatment of a leading `v` and drop the `TrimPrefix` if it already handles it.)
179
180`internal/console/apisystemlayout.go`:
181
182```go
183package console
184
185import (
186 "context"
187 "errors"
188 "log"
189 "net/http"
190 "os"
191
192 "github.com/carlosframework/platform/internal/release"
193)
194
195// The layout door (isolated-exec design §8.2). It records the fleet's
196// intent and bumps the serial; every box in scope converges on its next
197// serial move. It moves no file itself: the pass on each box does, with
198// the box's own registry as the authority on what is eligible.
199func (s *Server) handleAPISystemLayoutGet(w http.ResponseWriter, r *http.Request) {
200 if _, ok := s.apiSystemOwner(w, r); !ok {
201 return
202 }
203 if !s.systemBucket(w) {
204 return
205 }
206 in, found, err := release.LoadLayoutIntent(r.Context(), s.cfg.Releases)
207 if err != nil {
208 jsonError(w, http.StatusBadGateway, err.Error())
209 return
210 }
211 out := map[string]string{"mode": "none", "box": "", "id": "", "requested_at": ""}
212 if found {
213 out["mode"], out["box"], out["id"], out["requested_at"] = in.Mode, in.Scope.Box, in.ID, in.RequestedAt
214 }
215 writeJSON(w, http.StatusOK, out)
216}
217
218func (s *Server) handleAPISystemLayoutSet(w http.ResponseWriter, r *http.Request) {
219 addr, ok := s.apiSystemOwner(w, r)
220 if !ok {
221 return
222 }
223 if !s.systemBucket(w) {
224 return
225 }
226 var in struct {
227 Mode string `json:"mode"`
228 Box string `json:"box"`
229 }
230 if !readJSONStrictLimit(w, r, &in, 1<<10) {
231 return
232 }
233 if !release.ValidLayoutMode(in.Mode) {
234 jsonError(w, http.StatusBadRequest, "mode must be nested or flat")
235 return
236 }
237 if in.Box != "" && !validBoxLabel(in.Box) {
238 jsonError(w, http.StatusBadRequest, "box must be a box label")
239 return
240 }
241 rec, err := release.RecordLayoutIntent(r.Context(), s.cfg.Releases, in.Mode, in.Box, addr)
242 if err != nil {
243 jsonError(w, http.StatusBadGateway, err.Error())
244 return
245 }
246 scope := "the fleet"
247 if in.Box != "" {
248 scope = "box " + in.Box
249 }
250 s.auditSystem(addr, "system-layout", "recorded layout "+in.Mode+" for "+scope)
251 s.bumpSerial(r.Context())
252 writeJSON(w, http.StatusOK, map[string]string{"mode": rec.Mode, "box": rec.Scope.Box, "id": rec.ID})
253}
254```
255
256`validBoxLabel` is whatever the console already uses to validate a box label (the fleets code has one; else `controlname.ValidHostShape`-style: 1..63 of `[a-z0-9-]`). Routes in `authcli.go` beside the rollback door: `s.cliRoute("GET /api/cli/system/layout", svcrole.Set{}, s.handleAPISystemLayoutGet)` and `POST` likewise.
257
258- [ ] **Step 4: Run the tests to verify they pass**
259
260Expected: PASS.
261
262- [ ] **Step 5: gofmt and commit**
263
264```bash
265git add internal/release internal/console
266git commit -m "console: the layout intent door and its writer (plan 5, task 1)"
267```
268
269---
270
271### Task 2: `carlos system layout` and the docs
272
273**Files:**
274- Modify: `system.go`, `system_test.go` (if the file has verb tests), `docs/site/cli.md`, the spec (§8.4 one line, §13 R14, header revision 15)
275
276**Interfaces:**
277- Produces: `carlos system layout migrate [--box <label>] [--yes]`, `carlos system layout revert [--box <label>] [--yes]`, `carlos system layout status` (prints the recorded intent). `systemVerb` accepts `layout`.
278
279- [ ] **Step 1: Write the failing test**
280
281A `systemVerb` test (append to the file that tests it, or a new `system_layout_test.go`): `systemVerb([]string{"layout", "migrate", "--box", "x"})` returns verb `layout` and rest `[migrate --box x]`.
282
283- [ ] **Step 2: Run it to verify it fails**
284
285- [ ] **Step 3: Write the code**
286
287In `systemVerb`'s case list add `"layout"`. In `runSystem` add `case "layout": runSystemLayout(rest)`.
288
289```go
290func runSystemLayout(args []string) {
291 if len(args) == 0 {
292 fmt.Fprint(os.Stderr, systemLayoutUsage)
293 os.Exit(2)
294 }
295 sub, rest := args[0], args[1:]
296 fs := flag.NewFlagSet("system layout "+sub, flag.ExitOnError)
297 consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)")
298 box := fs.String("box", "", "one box's label; the default is the whole fleet")
299 yes := fs.Bool("yes", false, "do not ask")
300 fs.Usage = flagUsage(fs, systemLayoutUsage)
301 fs.Parse(rest)
302 sess := requireDeployAPISession(*consoleFlag)
303 ctx := context.Background()
304 switch sub {
305 case "status":
306 st, body, err := sess.getJSON(ctx, "/api/cli/system/layout")
307 if err != nil {
308 log.Fatalf("carlos system layout status: %v", err)
309 }
310 if st != http.StatusOK {
311 log.Fatalf("carlos system layout status: %s", apiErrorFromBody(st, body))
312 }
313 var out struct{ Mode, Box, ID, RequestedAt string }
314 _ = json.Unmarshal(body, &out)
315 scope := "the fleet"
316 if out.Box != "" {
317 scope = "box " + out.Box
318 }
319 if out.Mode == "none" {
320 fmt.Fprintln(stdout, "no layout intent recorded; every box is on the flat layout")
321 return
322 }
323 fmt.Fprintf(stdout, "intent: %s for %s (recorded %s)\nrun `carlos status --layout` to see what each box reports\n", out.Mode, scope, out.RequestedAt)
324 return
325 case "migrate", "revert":
326 default:
327 fmt.Fprintf(os.Stderr, "carlos system layout: unknown verb %q\n\n%s", sub, systemLayoutUsage)
328 os.Exit(2)
329 }
330 mode := "nested"
331 verb := "migrate every activator-managed instance on %s into its own directory and namespace"
332 if sub == "revert" {
333 mode = "flat"
334 verb = "move every isolated instance on %s back to the flat layout the previous release serves"
335 }
336 scope := "every box"
337 if *box != "" {
338 scope = "box " + *box
339 }
340 if !*yes {
341 if !interactiveTerminal() {
342 log.Fatal("no terminal to confirm on; pass --yes")
343 }
344 if !promptYes(stdout, fmt.Sprintf(verb+"? each host is stopped, moved and released one at a time [y/N] ", scope)) {
345 return
346 }
347 }
348 st, body, err := sess.postJSON(ctx, "/api/cli/system/layout", map[string]string{"mode": mode, "box": *box})
349 if err != nil {
350 log.Fatalf("carlos system layout %s: %v", sub, err)
351 }
352 if st != http.StatusOK {
353 log.Fatalf("carlos system layout %s: %s", sub, apiErrorFromBody(st, body))
354 }
355 fmt.Fprintf(stdout, "%s layout %s for %s; boxes converge on their next serial move. Watch with `carlos status --layout`.\n", colorFor(stdout).ok("recorded"), mode, scope)
356}
357
358const systemLayoutUsage = `usage: carlos system layout migrate [--box <label>] [--yes]
359 carlos system layout revert [--box <label>] [--yes]
360 carlos system layout status
361
362migrate records the intent "nested": every activator-managed instance on the
363boxes in scope is stopped, moved into /data/instances/<id>/ and, from its next
364wake, runs in its own mount and pid namespace. revert records "flat" and moves
365them back; run it before rolling the platform back to a release that predates
366the layout. status prints the recorded intent. Deployment owner only.
367`
368```
369
370`systemUsage` gains a `layout` line under "The platform's own binary". `docs/site/cli.md`: fill the `#### Instance layout` subsection plan 4 stubbed with the three forms, what `nested` means for a member (an instance reads its own files and nothing else), the rollback order (`layout revert`, verify with `carlos status --layout`, then `system rollback`), and that the console refuses a rollback to a pre-layout release while any box reports otherwise. Spec: §8.4 `carlos instances --layout` -> `carlos status --layout`; §13 add `R14 (plan 4): the per-host layout view is carlos status --layout, beside --instances, under the same operator gate`; header revision 15.
371
372- [ ] **Step 4: Gate with the docsite test**
373
374- [ ] **Step 5: Commit**
375
376```bash
377git add system.go system_test.go docs/site/cli.md docs/superpowers/specs/2026-09-11-isolated-exec-design.md
378git commit -m "carlos system layout migrate|revert|status; docs and spec revision 15 (plan 5, task 2)"
379```
380
381---
382
383### Task 3: The downgrade refusal at the console
384
385**Files:**
386- Create: `internal/console/layoutguard.go`, `internal/console/layoutguard_test.go`
387- Modify: `internal/console/apisystemupdate.go` (rollback door; the update door when the requested version is pre-layout)
388
389**Interfaces:**
390- Produces: `func (s *Server) layoutDowngradeBlocked(ctx context.Context, target string) string` (empty = allowed; else a reason naming boxes and rows). Blocks when `release.PreLayout(target)` and any box snapshot is stale, has no `Layout` report, or reports a row whose layout is `nested`, `fenced` or `conflict`.
391
392- [ ] **Step 1: Write the failing test**
393
394`layoutguard_test.go`: with two box snapshots written through `status` into the fixture store (one flat, one with a nested row), `layoutDowngradeBlocked(ctx, "v0.19.0")` names the second box and the host; with both flat it returns ""; with target `"v0.21.0"` it returns "" regardless; a box without a `Layout` report blocks with "reports no layout field"; a stale box blocks with "stale".
395
396- [ ] **Step 2: Run it to verify it fails**
397
398- [ ] **Step 3: Write the code**
399
400```go
401// layoutDowngradeBlocked is design §8.2's rollback rule at the console: a
402// pointer must not move below the layout floor while any box could still
403// hold a nested row. It judges observed state, never the intent, because
404// a revert refused on a box leaves a nested row under a flat intent; and a
405// box that has not reported, or reports no layout at all, is unknown and
406// blocks, because the cost of guessing is a database hidden under a mount.
407func (s *Server) layoutDowngradeBlocked(ctx context.Context, target string) string {
408 if !release.PreLayout(target) {
409 return ""
410 }
411 boxes, err := status.LoadBoxes(ctx, s.cfg.Releases)
412 if err != nil {
413 return "could not read the boxes' snapshots: " + err.Error()
414 }
415 now := time.Now()
416 var reasons []string
417 for _, b := range boxes {
418 switch {
419 case status.BoxStale(b, now, status.SnapshotPeriod):
420 reasons = append(reasons, b.Label+" is stale")
421 case b.Layout == nil:
422 reasons = append(reasons, b.Label+" reports no layout field")
423 default:
424 for host, row := range b.Layout.Rows {
425 if row.Layout == "nested" || row.Layout == "fenced" || row.Layout == "conflict" {
426 reasons = append(reasons, b.Label+": "+host+" is "+row.Layout)
427 }
428 }
429 }
430 }
431 if len(reasons) == 0 {
432 return ""
433 }
434 sort.Strings(reasons)
435 return target + " predates the instance layout and the fleet is not flat: " + strings.Join(reasons, "; ") + ". Run `carlos system layout revert`, wait for `carlos status --layout` to show every row flat, then retry"
436}
437```
438
439In `handleAPISystemRollback`, after the channel check and before `RollbackSystem`: read the pointer (`release.LoadSystemPointer` or whatever `RollbackSystem` reads) to learn the rollback target (`History[0].Version`, or however `RollbackSystem` chooses; mirror it exactly), then `if reason := s.layoutDowngradeBlocked(r.Context(), target); reason != "" { jsonError(w, http.StatusConflict, reason); return }`. In `handleAPISystemUpdate`, when `in.Version` is set and `release.PreLayout(in.Version)`, the same check before `startSystemJob`.
440
441- [ ] **Step 4: Gate**
442
443- [ ] **Step 5: Commit**
444
445```bash
446git add internal/console
447git commit -m "console: a pre-layout rollback or update is refused while any box reports a nested, fenced or unknown row (plan 5, task 3)"
448```
449
450---
451
452### Task 4: The control service inspects the underlying mountpoints at start
453
454**Files:**
455- Modify: `internal/instancectl/server.go`, `internal/instancectl/server_test.go`, `internal/host/units/carlos-instance-control.service` (comment only)
456
457**Interfaces:**
458- Produces: `Server.Run func(ctx context.Context, args ...string) (string, error)` (production: `exec.CommandContext(...).CombinedOutput()`); `func (s *Server) inspectMountpoints(ctx) []VerdictLine` run once from `Run()` before `probeLoop`, its lines kept in `s.startLines` and prepended to every report `RunProbe` stores; a failing line is `{Check: "mountpoint-<name>", Verdict: "fail", Detail: "mountpoint held files while unmounted: <n> entries"}`; an inspection that cannot run (no `unshare`) is `{Check: "mountpoint", Verdict: "unverified", Detail: ...}` and does not block readiness on its own.
459
460- [ ] **Step 1: Write the failing tests**
461
462`server_test.go` (append): a `fakeRun` keyed like `fakeCtl` answering `unshare` invocations. (a) Both listings empty: no fail line; `/ready` after a passing probe is 204. (b) The `/data/instances` listing returns two names: the report carries `mountpoint-data-instances fail` and `/ready` is 503 even after a passing probe report. (c) `unshare` fails to exec: an `unverified` line and readiness follows the probe alone.
463
464- [ ] **Step 2: Run them to verify they fail**
465
466- [ ] **Step 3: Write the code**
467
468```go
469// inspectMountpoints looks UNDER the two bind mounts (design §8.2, "Recovery
470// for a layout that was unmounted anyway"): in a private mount namespace it
471// unmounts each bind and lists the raw mountpoint directory. Anything there
472// was written while the layout was unmounted, and the layout pass's
473// destination check would otherwise never see it. The verdict is recorded
474// once at start; it clears only when an operator moves or removes the
475// files and restarts the service.
476func (s *Server) inspectMountpoints(ctx context.Context) []VerdictLine {
477 var lines []VerdictLine
478 for _, m := range []struct{ name, where string }{{"data-instances", "/data/instances"}, {"run-carlos-instances", "/run/carlos/instances"}} {
479 out, err := s.Run(ctx, "unshare", "-m", "--propagation", "private", "sh", "-c", "umount "+m.where+" 2>/dev/null; ls -A "+m.where)
480 if err != nil {
481 lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "unverified", Detail: strings.TrimSpace(out + " " + err.Error())})
482 continue
483 }
484 if n := len(strings.Fields(out)); n > 0 {
485 lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "fail", Detail: fmt.Sprintf("mountpoint held files while unmounted: %d entries under %s", n, m.where)})
486 continue
487 }
488 lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "ok"})
489 }
490 return lines
491}
492```
493
494In `RunProbe`'s `set`, prepend `s.startLines` and recompute `Ready` as "no line with verdict fail" (the probe's own `Ready` plus the start lines). In `Run()`, set `s.Run` to the exec runner and call `s.startLines = s.inspectMountpoints(ctx)` before `go s.probeLoop(ctx)`. Unit file comment: one paragraph naming the inspection and why it runs as root.
495
496- [ ] **Step 4: Gate**
497
498`go test ./internal/instancectl/ -race -count=3` outside the sandbox, plus the full gate.
499
500- [ ] **Step 5: Commit**
501
502```bash
503git add internal/instancectl internal/host/units
504git commit -m "instance-control: the underlying mountpoints are inspected at start and a held file blocks readiness (plan 5, task 4)"
505```
506
507---
508
509## Self-review
510
511**Spec coverage.** §8.2 verb shape (console writes the object, bumps the serial, boxes converge; `--box` scope): Tasks 1 and 2. Downgrade rules: `system apply` refusal (plan 4), unflag refusals (plan 4), the console's rollback door (Task 3, extended to an update naming a pre-layout version), the unmounted-mountpoint recovery (Task 4). §8.4 CLI naming (revision 15): Task 2. §8.6 rollout order: docs in Task 2. **Not built, by ruling:** `layout-marks.json` (§9 names it): convergence to an intent is idempotent and stateless, so a mark adds nothing the status report does not already say; recorded as a ruling in the ledger rather than built.
512
513**Placeholder scan.** Task 1's console test names the fixture to copy (`newSystemDoorFixture`, `postAs`, `getAs`): the implementer uses the real helper names from `apisystemupdate_test.go`. Task 3's rollback-target read says "mirror `RollbackSystem`" rather than quoting it; the implementer reads that function. Everything else carries code.
514
515**Type consistency.** `LayoutFloor`, `PreLayout`, `RecordLayoutIntent`, `handleAPISystemLayoutGet/Set`, `layoutDowngradeBlocked`, `runSystemLayout`, `systemLayoutUsage`, `inspectMountpoints`, `Server.Run`, `startLines` are spelled identically across tasks; plan 4's `LayoutIntent`, `LayoutScope`, `PutLayoutIntent`, `LoadLayoutIntent`, `Box.Layout`, `LayoutReport.Rows`, `RowLayout.Layout` are consumed by those names.
516