amadan
isolated exec: plan 5, the console door and the last two guards
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
1 file changed,
+515
−0
docs/superpowers/plans/2026-09-12-isolated-exec-5-console.md+515 −0
CI passed — run details
diff --git a/docs/superpowers/plans/2026-09-12-isolated-exec-5-console.md b/docs/superpowers/plans/2026-09-12-isolated-exec-5-console.md| new file mode 100644 |
| index 000000000..cacd8a425 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-09-12-isolated-exec-5-console.md |
| @@ -0,0 +1,515 @@ |
| +# Isolated Exec, Plan 5 of 5: The Console Door and the Last Two Guards |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +**Goal:** Give an operator the verb that records the fleet's layout intent (`carlos system layout migrate|revert`, optionally `--box`), through a console door that writes `control/layout.json` and bumps the serial so every box in scope converges (plan 4's pass); refuse, at the console, a platform rollback or update to a release before the layout while any box still reports a nested, fenced or conflict row or no layout field; and make the root control service inspect the underlying mountpoint directories at start so a layout that was unmounted by hand cannot leave two databases. |
| + |
| +**Architecture:** The CLI verb follows `carlos system rollback` (a `requireDeployAPISession`, a confirmation, one POST). The door follows `handleAPISystemRollback` (owner gate, bucket check, strict JSON, audit, serial bump) and writes the intent with `release.PutLayoutIntent` under a read-then-CAS. The downgrade refusal is one helper both system doors call before moving a pointer, reading `status.LoadBoxes` and the `Layout` report plan 4 added. The mountpoint inspection is a `Run` seam on `instancectl.Server` that executes `unshare -m` as root and folds a verdict line into every probe report. |
| + |
| +**Tech Stack:** Go 1.26, `internal/release` (`LayoutIntent`, `SystemPointer`, `SemVer`), `internal/console`, `internal/status`, `internal/instancectl`. |
| + |
| +**Spec:** `docs/superpowers/specs/2026-09-11-isolated-exec-design.md` (revision 14): §8.2 (the verb's shape, scope, convergence; the downgrade rules; the unmounted-mountpoint recovery), §8.4, §8.6, §10 (`internal/console` and `system.go` bullets), §13 R5-2. Plan 4 defines `LayoutIntent` and the status `Layout` report this plan consumes. |
| + |
| +## Global Constraints |
| + |
| +- Gate and env as plan 4. Console security gates are ordered session, CSRF, owner (byte-identical 404), fresh auth last (AGENTS.md); the layout door uses `apiSystemOwner` exactly as the update and rollback doors do. |
| +- The intent object is the only writer of a nested path anywhere in the platform, indirectly: recording `nested` is an operator's act, audited, and needs a confirmation in the CLI unless `--yes`. |
| +- The downgrade refusal is by observed state (each box's `Layout` report), never by the intent, and a box that reports no layout field or is stale counts as unknown and blocks. |
| +- `LayoutFloor` is the first release that carries the mount units; it is a constant with a test, and the plan's report names it so it can be corrected when the release is cut. |
| +- Copy: no em dashes; comments are full sentences saying why; log lines use `->`. |
| +- Commit after every task with the trailer `Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>`. |
| + |
| +--- |
| + |
| +## File Structure |
| + |
| +| File | Responsibility | |
| +|---|---| |
| +| `internal/release/layoutintent.go` (modify) | `LayoutFloor`, `PreLayout(version string) bool`; `RecordLayoutIntent(ctx, s, mode, box, actor)` (read etag, CAS put, 3 retries, fresh id) | |
| +| `internal/console/apisystemlayout.go` (create) | `handleAPISystemLayoutGet`, `handleAPISystemLayoutSet` | |
| +| `internal/console/authcli.go` (modify) | two routes beside `/api/cli/system/rollback` | |
| +| `internal/console/layoutguard.go` (create) | `layoutDowngradeBlocked(ctx, s, target string) (reason string)` | |
| +| `internal/console/apisystemupdate.go` (modify) | rollback and update doors call the guard before moving a pointer | |
| +| `system.go` (modify) | `runSystemLayout`; `systemVerb` learns `layout`; `systemUsage` | |
| +| `docs/site/cli.md` (modify) | the `#### Instance layout` subsection plan 4 stubbed gets the verb | |
| +| `internal/instancectl/server.go` (modify) | `Run` seam; `inspectMountpoints` at start; `startLines` folded into every report | |
| +| `internal/instancectl/server_test.go` (modify) | inspection tests with a fake `Run` | |
| +| `internal/host/units/carlos-instance-control.service` (modify) | nothing new needed: root, `ProtectSystem=strict` leaves `/data` visible; add a comment on the inspection | |
| +| spec §8.4 (modify) | `carlos instances --layout` becomes `carlos status --layout` (revision 15, §13 R14) | |
| + |
| +--- |
| + |
| +### Task 1: The intent writer and the console door |
| + |
| +**Files:** |
| +- Modify: `internal/release/layoutintent.go`, `internal/release/layoutintent_test.go` |
| +- Create: `internal/console/apisystemlayout.go`, `internal/console/apisystemlayout_test.go` |
| +- Modify: `internal/console/authcli.go` |
| + |
| +**Interfaces:** |
| +- Produces: |
| + ```go |
| + // release |
| + func RecordLayoutIntent(ctx context.Context, s Store, mode, box, actor string) (*LayoutIntent, error) // fresh 16-byte hex id, RequestedAt now, CAS over the existing etag, 3 retries on ErrCASConflict |
| + // console |
| + GET /api/cli/system/layout -> 200 {"mode":"none|flat|nested","box":"","id":"","requested_at":""} (owner gate; 200 with mode none when absent) |
| + POST /api/cli/system/layout {"mode":"nested|flat","box":"<label or empty>"} -> 200 {"mode","box","id"}; 400 on a bad mode or a label that is not a box label shape; 404 for a non-owner |
| + ``` |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`internal/release/layoutintent_test.go` (append): |
| + |
| +```go |
| +func TestRecordLayoutIntentReplacesUnderCAS(t *testing.T) { |
| + s := NewDirStore(t.TempDir()) |
| + ctx := context.Background() |
| + first, err := RecordLayoutIntent(ctx, s, LayoutModeNested, "kass-1", "op@example.com") |
| + if err != nil || first.ID == "" || first.Mode != LayoutModeNested || first.Scope.Box != "kass-1" { |
| + t.Fatalf("first = %+v %v", first, err) |
| + } |
| + second, err := RecordLayoutIntent(ctx, s, LayoutModeFlat, "", "op@example.com") |
| + if err != nil || second.ID == first.ID || second.Mode != LayoutModeFlat || second.Scope.Box != "" { |
| + t.Fatalf("second = %+v %v", second, err) |
| + } |
| + got, _, _ := LoadLayoutIntent(ctx, s) |
| + if got.ID != second.ID { |
| + t.Fatalf("object holds %s, want the latest %s", got.ID, second.ID) |
| + } |
| + if _, err := RecordLayoutIntent(ctx, s, "sideways", "", "op"); err == nil { |
| + t.Fatal("bad mode accepted") |
| + } |
| +} |
| +``` |
| + |
| +`internal/console/apisystemlayout_test.go` (copy the fixture of the nearest `apisystemupdate_test.go` test that exercises `apiSystemOwner`: a server with an ops-owner principal, a member principal, and a `DirStore` as `cfg.Releases`): |
| + |
| +```go |
| +func TestSystemLayoutDoorIsOwnerOnlyAndRecordsTheIntent(t *testing.T) { |
| + s, owner, member := newSystemDoorFixture(t) // as the rollback door test builds it |
| + // A member gets the byte-identical 404. |
| + rec := postAs(t, s, member, "/api/cli/system/layout", `{"mode":"nested"}`) |
| + if rec.Code != 404 { |
| + t.Fatalf("member: %d", rec.Code) |
| + } |
| + rec = postAs(t, s, owner, "/api/cli/system/layout", `{"mode":"sideways"}`) |
| + if rec.Code != 400 { |
| + t.Fatalf("bad mode: %d %s", rec.Code, rec.Body) |
| + } |
| + rec = postAs(t, s, owner, "/api/cli/system/layout", `{"mode":"nested","box":"kass-1"}`) |
| + if rec.Code != 200 { |
| + t.Fatalf("record: %d %s", rec.Code, rec.Body) |
| + } |
| + in, found, err := release.LoadLayoutIntent(context.Background(), s.cfg.Releases) |
| + if err != nil || !found || in.Mode != "nested" || in.Scope.Box != "kass-1" { |
| + t.Fatalf("object = %+v %v %v", in, found, err) |
| + } |
| + if !s.serialBumped() { // whatever the fixture exposes for bumpSerial; else read the serial object |
| + t.Error("serial not bumped") |
| + } |
| + rec = getAs(t, s, owner, "/api/cli/system/layout") |
| + if rec.Code != 200 || !strings.Contains(rec.Body.String(), `"mode":"nested"`) { |
| + t.Fatalf("get: %d %s", rec.Code, rec.Body) |
| + } |
| +} |
| +``` |
| + |
| +- [ ] **Step 2: Run the tests to verify they fail** |
| + |
| +Expected: build failure. |
| + |
| +- [ ] **Step 3: Write the code** |
| + |
| +`layoutintent.go` (append): |
| + |
| +```go |
| +// LayoutFloor is the first platform release that carries the instance |
| +// mount units. A pointer moved below it while any box is nested would |
| +// unmount the layout from under a nested row (design §8.2), so both system |
| +// doors refuse such a move until every box reports flat. Correct this |
| +// constant when the release carrying the layout is cut. |
| +var LayoutFloor = SemVer{Major: 0, Minor: 20, Patch: 0} |
| + |
| +// PreLayout reports whether a release tag predates the layout. A tag that |
| +// does not parse is treated as pre-layout: refusing is the safe answer for |
| +// a version nobody can place. |
| +func PreLayout(version string) bool { |
| + v, ok := ParseSemVer(strings.TrimPrefix(version, "v")) |
| + if !ok { |
| + return true |
| + } |
| + return v.Compare(LayoutFloor) < 0 |
| +} |
| + |
| +// RecordLayoutIntent replaces the intent with a fresh one. A CAS conflict is |
| +// retried: the other writer recorded a different intent and ours must |
| +// still land, with its own id, as the newer one. |
| +func RecordLayoutIntent(ctx context.Context, s Store, mode, box, actor string) (*LayoutIntent, error) { |
| + if !ValidLayoutMode(mode) { |
| + return nil, fmt.Errorf("layout intent: mode %q is not flat or nested", mode) |
| + } |
| + id := make([]byte, 16) |
| + if _, err := io.ReadFull(rand.Reader, id); err != nil { |
| + return nil, err |
| + } |
| + in := &LayoutIntent{Schema: 1, ID: hex.EncodeToString(id), RequestedAt: time.Now().UTC().Format(time.RFC3339), Mode: mode, Scope: LayoutScope{Box: box}} |
| + var lastErr error |
| + for attempt := 0; attempt < 3; attempt++ { |
| + _, etag, err := s.Get(ctx, LayoutIntentKey()) |
| + if err != nil && !errors.Is(err, os.ErrNotExist) { |
| + return nil, err |
| + } |
| + if err != nil { |
| + etag = "" |
| + } |
| + if err := PutLayoutIntent(ctx, s, in, etag); err != nil { |
| + if errors.Is(err, ErrCASConflict) { |
| + lastErr = err |
| + continue |
| + } |
| + return nil, err |
| + } |
| + return in, nil |
| + } |
| + return nil, fmt.Errorf("layout intent: %s is being written by someone else; nothing was recorded: %w", LayoutIntentKey(), lastErr) |
| +} |
| +``` |
| + |
| +(Check `ParseSemVer`'s treatment of a leading `v` and drop the `TrimPrefix` if it already handles it.) |
| + |
| +`internal/console/apisystemlayout.go`: |
| + |
| +```go |
| +package console |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "log" |
| + "net/http" |
| + "os" |
| + |
| + "github.com/carlosframework/platform/internal/release" |
| +) |
| + |
| +// The layout door (isolated-exec design §8.2). It records the fleet's |
| +// intent and bumps the serial; every box in scope converges on its next |
| +// serial move. It moves no file itself: the pass on each box does, with |
| +// the box's own registry as the authority on what is eligible. |
| +func (s *Server) handleAPISystemLayoutGet(w http.ResponseWriter, r *http.Request) { |
| + if _, ok := s.apiSystemOwner(w, r); !ok { |
| + return |
| + } |
| + if !s.systemBucket(w) { |
| + return |
| + } |
| + in, found, err := release.LoadLayoutIntent(r.Context(), s.cfg.Releases) |
| + if err != nil { |
| + jsonError(w, http.StatusBadGateway, err.Error()) |
| + return |
| + } |
| + out := map[string]string{"mode": "none", "box": "", "id": "", "requested_at": ""} |
| + if found { |
| + out["mode"], out["box"], out["id"], out["requested_at"] = in.Mode, in.Scope.Box, in.ID, in.RequestedAt |
| + } |
| + writeJSON(w, http.StatusOK, out) |
| +} |
| + |
| +func (s *Server) handleAPISystemLayoutSet(w http.ResponseWriter, r *http.Request) { |
| + addr, ok := s.apiSystemOwner(w, r) |
| + if !ok { |
| + return |
| + } |
| + if !s.systemBucket(w) { |
| + return |
| + } |
| + var in struct { |
| + Mode string `json:"mode"` |
| + Box string `json:"box"` |
| + } |
| + if !readJSONStrictLimit(w, r, &in, 1<<10) { |
| + return |
| + } |
| + if !release.ValidLayoutMode(in.Mode) { |
| + jsonError(w, http.StatusBadRequest, "mode must be nested or flat") |
| + return |
| + } |
| + if in.Box != "" && !validBoxLabel(in.Box) { |
| + jsonError(w, http.StatusBadRequest, "box must be a box label") |
| + return |
| + } |
| + rec, err := release.RecordLayoutIntent(r.Context(), s.cfg.Releases, in.Mode, in.Box, addr) |
| + if err != nil { |
| + jsonError(w, http.StatusBadGateway, err.Error()) |
| + return |
| + } |
| + scope := "the fleet" |
| + if in.Box != "" { |
| + scope = "box " + in.Box |
| + } |
| + s.auditSystem(addr, "system-layout", "recorded layout "+in.Mode+" for "+scope) |
| + s.bumpSerial(r.Context()) |
| + writeJSON(w, http.StatusOK, map[string]string{"mode": rec.Mode, "box": rec.Scope.Box, "id": rec.ID}) |
| +} |
| +``` |
| + |
| +`validBoxLabel` is whatever the console already uses to validate a box label (the fleets code has one; else `controlname.ValidHostShape`-style: 1..63 of `[a-z0-9-]`). Routes in `authcli.go` beside the rollback door: `s.cliRoute("GET /api/cli/system/layout", svcrole.Set{}, s.handleAPISystemLayoutGet)` and `POST` likewise. |
| + |
| +- [ ] **Step 4: Run the tests to verify they pass** |
| + |
| +Expected: PASS. |
| + |
| +- [ ] **Step 5: gofmt and commit** |
| + |
| +```bash |
| +git add internal/release internal/console |
| +git commit -m "console: the layout intent door and its writer (plan 5, task 1)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 2: `carlos system layout` and the docs |
| + |
| +**Files:** |
| +- Modify: `system.go`, `system_test.go` (if the file has verb tests), `docs/site/cli.md`, the spec (§8.4 one line, §13 R14, header revision 15) |
| + |
| +**Interfaces:** |
| +- Produces: `carlos system layout migrate [--box <label>] [--yes]`, `carlos system layout revert [--box <label>] [--yes]`, `carlos system layout status` (prints the recorded intent). `systemVerb` accepts `layout`. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +A `systemVerb` test (append to the file that tests it, or a new `system_layout_test.go`): `systemVerb([]string{"layout", "migrate", "--box", "x"})` returns verb `layout` and rest `[migrate --box x]`. |
| + |
| +- [ ] **Step 2: Run it to verify it fails** |
| + |
| +- [ ] **Step 3: Write the code** |
| + |
| +In `systemVerb`'s case list add `"layout"`. In `runSystem` add `case "layout": runSystemLayout(rest)`. |
| + |
| +```go |
| +func runSystemLayout(args []string) { |
| + if len(args) == 0 { |
| + fmt.Fprint(os.Stderr, systemLayoutUsage) |
| + os.Exit(2) |
| + } |
| + sub, rest := args[0], args[1:] |
| + fs := flag.NewFlagSet("system layout "+sub, flag.ExitOnError) |
| + consoleFlag := fs.String("console", "", "console origin (default: the only logged-in one)") |
| + box := fs.String("box", "", "one box's label; the default is the whole fleet") |
| + yes := fs.Bool("yes", false, "do not ask") |
| + fs.Usage = flagUsage(fs, systemLayoutUsage) |
| + fs.Parse(rest) |
| + sess := requireDeployAPISession(*consoleFlag) |
| + ctx := context.Background() |
| + switch sub { |
| + case "status": |
| + st, body, err := sess.getJSON(ctx, "/api/cli/system/layout") |
| + if err != nil { |
| + log.Fatalf("carlos system layout status: %v", err) |
| + } |
| + if st != http.StatusOK { |
| + log.Fatalf("carlos system layout status: %s", apiErrorFromBody(st, body)) |
| + } |
| + var out struct{ Mode, Box, ID, RequestedAt string } |
| + _ = json.Unmarshal(body, &out) |
| + scope := "the fleet" |
| + if out.Box != "" { |
| + scope = "box " + out.Box |
| + } |
| + if out.Mode == "none" { |
| + fmt.Fprintln(stdout, "no layout intent recorded; every box is on the flat layout") |
| + return |
| + } |
| + fmt.Fprintf(stdout, "intent: %s for %s (recorded %s)\nrun `carlos status --layout` to see what each box reports\n", out.Mode, scope, out.RequestedAt) |
| + return |
| + case "migrate", "revert": |
| + default: |
| + fmt.Fprintf(os.Stderr, "carlos system layout: unknown verb %q\n\n%s", sub, systemLayoutUsage) |
| + os.Exit(2) |
| + } |
| + mode := "nested" |
| + verb := "migrate every activator-managed instance on %s into its own directory and namespace" |
| + if sub == "revert" { |
| + mode = "flat" |
| + verb = "move every isolated instance on %s back to the flat layout the previous release serves" |
| + } |
| + scope := "every box" |
| + if *box != "" { |
| + scope = "box " + *box |
| + } |
| + if !*yes { |
| + if !interactiveTerminal() { |
| + log.Fatal("no terminal to confirm on; pass --yes") |
| + } |
| + if !promptYes(stdout, fmt.Sprintf(verb+"? each host is stopped, moved and released one at a time [y/N] ", scope)) { |
| + return |
| + } |
| + } |
| + st, body, err := sess.postJSON(ctx, "/api/cli/system/layout", map[string]string{"mode": mode, "box": *box}) |
| + if err != nil { |
| + log.Fatalf("carlos system layout %s: %v", sub, err) |
| + } |
| + if st != http.StatusOK { |
| + log.Fatalf("carlos system layout %s: %s", sub, apiErrorFromBody(st, body)) |
| + } |
| + fmt.Fprintf(stdout, "%s layout %s for %s; boxes converge on their next serial move. Watch with `carlos status --layout`.\n", colorFor(stdout).ok("recorded"), mode, scope) |
| +} |
| + |
| +const systemLayoutUsage = `usage: carlos system layout migrate [--box <label>] [--yes] |
| + carlos system layout revert [--box <label>] [--yes] |
| + carlos system layout status |
| + |
| +migrate records the intent "nested": every activator-managed instance on the |
| +boxes in scope is stopped, moved into /data/instances/<id>/ and, from its next |
| +wake, runs in its own mount and pid namespace. revert records "flat" and moves |
| +them back; run it before rolling the platform back to a release that predates |
| +the layout. status prints the recorded intent. Deployment owner only. |
| +` |
| +``` |
| + |
| +`systemUsage` gains a `layout` line under "The platform's own binary". `docs/site/cli.md`: fill the `#### Instance layout` subsection plan 4 stubbed with the three forms, what `nested` means for a member (an instance reads its own files and nothing else), the rollback order (`layout revert`, verify with `carlos status --layout`, then `system rollback`), and that the console refuses a rollback to a pre-layout release while any box reports otherwise. Spec: §8.4 `carlos instances --layout` -> `carlos status --layout`; §13 add `R14 (plan 4): the per-host layout view is carlos status --layout, beside --instances, under the same operator gate`; header revision 15. |
| + |
| +- [ ] **Step 4: Gate with the docsite test** |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add system.go system_test.go docs/site/cli.md docs/superpowers/specs/2026-09-11-isolated-exec-design.md |
| +git commit -m "carlos system layout migrate|revert|status; docs and spec revision 15 (plan 5, task 2)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 3: The downgrade refusal at the console |
| + |
| +**Files:** |
| +- Create: `internal/console/layoutguard.go`, `internal/console/layoutguard_test.go` |
| +- Modify: `internal/console/apisystemupdate.go` (rollback door; the update door when the requested version is pre-layout) |
| + |
| +**Interfaces:** |
| +- Produces: `func (s *Server) layoutDowngradeBlocked(ctx context.Context, target string) string` (empty = allowed; else a reason naming boxes and rows). Blocks when `release.PreLayout(target)` and any box snapshot is stale, has no `Layout` report, or reports a row whose layout is `nested`, `fenced` or `conflict`. |
| + |
| +- [ ] **Step 1: Write the failing test** |
| + |
| +`layoutguard_test.go`: with two box snapshots written through `status` into the fixture store (one flat, one with a nested row), `layoutDowngradeBlocked(ctx, "v0.19.0")` names the second box and the host; with both flat it returns ""; with target `"v0.21.0"` it returns "" regardless; a box without a `Layout` report blocks with "reports no layout field"; a stale box blocks with "stale". |
| + |
| +- [ ] **Step 2: Run it to verify it fails** |
| + |
| +- [ ] **Step 3: Write the code** |
| + |
| +```go |
| +// layoutDowngradeBlocked is design §8.2's rollback rule at the console: a |
| +// pointer must not move below the layout floor while any box could still |
| +// hold a nested row. It judges observed state, never the intent, because |
| +// a revert refused on a box leaves a nested row under a flat intent; and a |
| +// box that has not reported, or reports no layout at all, is unknown and |
| +// blocks, because the cost of guessing is a database hidden under a mount. |
| +func (s *Server) layoutDowngradeBlocked(ctx context.Context, target string) string { |
| + if !release.PreLayout(target) { |
| + return "" |
| + } |
| + boxes, err := status.LoadBoxes(ctx, s.cfg.Releases) |
| + if err != nil { |
| + return "could not read the boxes' snapshots: " + err.Error() |
| + } |
| + now := time.Now() |
| + var reasons []string |
| + for _, b := range boxes { |
| + switch { |
| + case status.BoxStale(b, now, status.SnapshotPeriod): |
| + reasons = append(reasons, b.Label+" is stale") |
| + case b.Layout == nil: |
| + reasons = append(reasons, b.Label+" reports no layout field") |
| + default: |
| + for host, row := range b.Layout.Rows { |
| + if row.Layout == "nested" || row.Layout == "fenced" || row.Layout == "conflict" { |
| + reasons = append(reasons, b.Label+": "+host+" is "+row.Layout) |
| + } |
| + } |
| + } |
| + } |
| + if len(reasons) == 0 { |
| + return "" |
| + } |
| + sort.Strings(reasons) |
| + return target + " predates the instance layout and the fleet is not flat: " + strings.Join(reasons, "; ") + ". Run `carlos system layout revert`, wait for `carlos status --layout` to show every row flat, then retry" |
| +} |
| +``` |
| + |
| +In `handleAPISystemRollback`, after the channel check and before `RollbackSystem`: read the pointer (`release.LoadSystemPointer` or whatever `RollbackSystem` reads) to learn the rollback target (`History[0].Version`, or however `RollbackSystem` chooses; mirror it exactly), then `if reason := s.layoutDowngradeBlocked(r.Context(), target); reason != "" { jsonError(w, http.StatusConflict, reason); return }`. In `handleAPISystemUpdate`, when `in.Version` is set and `release.PreLayout(in.Version)`, the same check before `startSystemJob`. |
| + |
| +- [ ] **Step 4: Gate** |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/console |
| +git commit -m "console: a pre-layout rollback or update is refused while any box reports a nested, fenced or unknown row (plan 5, task 3)" |
| +``` |
| + |
| +--- |
| + |
| +### Task 4: The control service inspects the underlying mountpoints at start |
| + |
| +**Files:** |
| +- Modify: `internal/instancectl/server.go`, `internal/instancectl/server_test.go`, `internal/host/units/carlos-instance-control.service` (comment only) |
| + |
| +**Interfaces:** |
| +- Produces: `Server.Run func(ctx context.Context, args ...string) (string, error)` (production: `exec.CommandContext(...).CombinedOutput()`); `func (s *Server) inspectMountpoints(ctx) []VerdictLine` run once from `Run()` before `probeLoop`, its lines kept in `s.startLines` and prepended to every report `RunProbe` stores; a failing line is `{Check: "mountpoint-<name>", Verdict: "fail", Detail: "mountpoint held files while unmounted: <n> entries"}`; an inspection that cannot run (no `unshare`) is `{Check: "mountpoint", Verdict: "unverified", Detail: ...}` and does not block readiness on its own. |
| + |
| +- [ ] **Step 1: Write the failing tests** |
| + |
| +`server_test.go` (append): a `fakeRun` keyed like `fakeCtl` answering `unshare` invocations. (a) Both listings empty: no fail line; `/ready` after a passing probe is 204. (b) The `/data/instances` listing returns two names: the report carries `mountpoint-data-instances fail` and `/ready` is 503 even after a passing probe report. (c) `unshare` fails to exec: an `unverified` line and readiness follows the probe alone. |
| + |
| +- [ ] **Step 2: Run them to verify they fail** |
| + |
| +- [ ] **Step 3: Write the code** |
| + |
| +```go |
| +// inspectMountpoints looks UNDER the two bind mounts (design §8.2, "Recovery |
| +// for a layout that was unmounted anyway"): in a private mount namespace it |
| +// unmounts each bind and lists the raw mountpoint directory. Anything there |
| +// was written while the layout was unmounted, and the layout pass's |
| +// destination check would otherwise never see it. The verdict is recorded |
| +// once at start; it clears only when an operator moves or removes the |
| +// files and restarts the service. |
| +func (s *Server) inspectMountpoints(ctx context.Context) []VerdictLine { |
| + var lines []VerdictLine |
| + for _, m := range []struct{ name, where string }{{"data-instances", "/data/instances"}, {"run-carlos-instances", "/run/carlos/instances"}} { |
| + out, err := s.Run(ctx, "unshare", "-m", "--propagation", "private", "sh", "-c", "umount "+m.where+" 2>/dev/null; ls -A "+m.where) |
| + if err != nil { |
| + lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "unverified", Detail: strings.TrimSpace(out + " " + err.Error())}) |
| + continue |
| + } |
| + if n := len(strings.Fields(out)); n > 0 { |
| + lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "fail", Detail: fmt.Sprintf("mountpoint held files while unmounted: %d entries under %s", n, m.where)}) |
| + continue |
| + } |
| + lines = append(lines, VerdictLine{Check: "mountpoint-" + m.name, Verdict: "ok"}) |
| + } |
| + return lines |
| +} |
| +``` |
| + |
| +In `RunProbe`'s `set`, prepend `s.startLines` and recompute `Ready` as "no line with verdict fail" (the probe's own `Ready` plus the start lines). In `Run()`, set `s.Run` to the exec runner and call `s.startLines = s.inspectMountpoints(ctx)` before `go s.probeLoop(ctx)`. Unit file comment: one paragraph naming the inspection and why it runs as root. |
| + |
| +- [ ] **Step 4: Gate** |
| + |
| +`go test ./internal/instancectl/ -race -count=3` outside the sandbox, plus the full gate. |
| + |
| +- [ ] **Step 5: Commit** |
| + |
| +```bash |
| +git add internal/instancectl internal/host/units |
| +git commit -m "instance-control: the underlying mountpoints are inspected at start and a held file blocks readiness (plan 5, task 4)" |
| +``` |
| + |
| +--- |
| + |
| +## Self-review |
| + |
| +**Spec coverage.** §8.2 verb shape (console writes the object, bumps the serial, boxes converge; `--box` scope): Tasks 1 and 2. Downgrade rules: `system apply` refusal (plan 4), unflag refusals (plan 4), the console's rollback door (Task 3, extended to an update naming a pre-layout version), the unmounted-mountpoint recovery (Task 4). §8.4 CLI naming (revision 15): Task 2. §8.6 rollout order: docs in Task 2. **Not built, by ruling:** `layout-marks.json` (§9 names it): convergence to an intent is idempotent and stateless, so a mark adds nothing the status report does not already say; recorded as a ruling in the ledger rather than built. |
| + |
| +**Placeholder scan.** Task 1's console test names the fixture to copy (`newSystemDoorFixture`, `postAs`, `getAs`): the implementer uses the real helper names from `apisystemupdate_test.go`. Task 3's rollback-target read says "mirror `RollbackSystem`" rather than quoting it; the implementer reads that function. Everything else carries code. |
| + |
| +**Type consistency.** `LayoutFloor`, `PreLayout`, `RecordLayoutIntent`, `handleAPISystemLayoutGet/Set`, `layoutDowngradeBlocked`, `runSystemLayout`, `systemLayoutUsage`, `inspectMountpoints`, `Server.Run`, `startLines` are spelled identically across tasks; plan 4's `LayoutIntent`, `LayoutScope`, `PutLayoutIntent`, `LoadLayoutIntent`, `Box.Layout`, `LayoutReport.Rows`, `RowLayout.Layout` are consumed by those names. |