amadan
carlos / platform Public
Isolated Exec, Plan 4 of 5: The Layout Pass on the Box
> For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.
Goal: Give a box the pass that moves an activator-managed host between the flat layout and the nested one, exactly and recoverably: a journal in /data/layout, a two-column compare-and-swap on the row, renames through the raw .instances directory, a fence that holds in-process and across an agent restart, convergence to a recorded fleet intent on every serial move, the layout field in the box snapshot, and the guards that stop an older binary from ever meeting a nested row.
Architecture: A new package internal/layout owns the plan (from the row and host name only), the journal (phases files, row, done), recovery (checked against both endpoints derived from the journal's own direction), and the pass (Converge and Recover). The registry gains SetInstancePaths. The activator gains Unfence, the release of the in-process fence FenceAndStop installs. The agent runs Recover at start and Converge on every serial move against control/layout.json (the intent object plan 5's console door writes; this plan defines it and reads it). The box snapshot carries layout_mode and each managed row's classification. carlos system apply, carlos route --hibernate=false and the provisioning reconciler refuse the three things a downgrade must never meet.
Tech Stack: Go 1.26, internal/activator (Classify, FenceAndStop), internal/instancefs, internal/provision, internal/registry, internal/release (Store, DirStore for tests), internal/status.
Spec: docs/superpowers/specs/2026-09-11-isolated-exec-design.md (revision 14): §3.2 (raw directory, EXDEV), §3.3 (a journal is not an instruction), §8.1, §8.2 (all nine steps, recovery, revert, the downgrade rules), §8.3, §8.4, §9, §10 ("Layout pass" bullet). Rulings in §13 are not to be reopened.
Global Constraints
- Gate before every commit:
go build ./... && go vet ./... && gofmt -l .empty&& go test ./... -count=1; envGOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off TMPDIR=/tmp/g; socket-binding packages need the sandbox off. Cross-compile gate: darwin, windows, linux-arm64; Linux-only syscalls in_linux.gowith!linuxstubs. - Nothing in this plan mints a nested row on its own. A row becomes nested only inside
Convergeunder a recorded intent ofnested. Without an intent object every box behaves as before, and every existing test keeps passing without assertion edits. - The plan for a host is built from the row and the host name only (§8.2 step 3): sources are
<DataDir>/<host>.db(plus-wal,-shm),<DataDir>/<host>.env,<DataDir>/<host>-sidecar.env,<DataDir>/<host>.config.env,<SocketDir>/<host>.sock; destinations are the §3.1 helpers. A row whoseDBPathorSocketis neither the flat nor the nested default isforeignand is never moved. Nothing is read from an instance directory to build the plan. - Renames of the database and its sidecar files cross into the tree through the RAW directory
/data/.instances/<iid>/(never through the/data/instancesbind mount, which is a different mount and answersEXDEV). The one caller of the literal.instancesisinternal/layout; a grep test pins it. - The journal is written
O_EXCL; recovery refuses a journal whosefrom/todo not equal the host's derived endpoints for the journal's ownmode, or whose row equals neither endpoint. A refused journal keeps the host fenced. - Quiesce is
activator.FenceAndStopwith a fresh generation; an unproven stop (ErrStopUnproven) refuses the host for this pass and leaves the in-process fence installed (never released over an unproven stop). - Copy: no em dashes anywhere; comments are full sentences saying why, never process history; log lines use
->between values. - Commit after every task with the trailer
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>. Branchworktree-isolated-exec-spec.
---
File Structure
| File | Responsibility | |---|---| | internal/registry/registry.go (modify) | SetInstancePaths(host, fromDB, fromSock, toDB, toSock): one statement, CAS on both columns | | internal/activator/maintenance.go (modify) | Unfence(host, generation); Fenced(host) bool | | internal/release/layoutintent.go (create) | LayoutIntent{Schema, Mode, Scope, RequestedAt, ID}, LayoutIntentKey, LoadLayoutIntent, LayoutModeNone/Flat/Nested | | internal/layout/plan.go (create) | Mode, Endpoint, Artifact, Plan, PlanFor(dataDir, socketDir, host, mode), rawInstanceDir | | internal/layout/journal.go (create) | Journal, phases, WriteJournal (OEXCL), ReadJournal, UpdatePhase, Validate | | internal/layout/preflight.go (create) | ownership preflight (x vs x.config), destination check, EnsureDirs | | internal/layout/pass.go (create) | Pass{Registry, Fence, DataDir, SocketDir, Reload, Now, Logf}, Converge(ctx, mode, scope), Recover(ctx), migrateHost | | `internal/layout/*test.go (create) | plan, journal, preflight, pass with a real registry and a temp tree; the raw-dir rename test with a bind mount (skipped when it cannot mount) | | internal/host/agent.go (modify) | wire: Recover at start, Converge in fastTick on a serial move; the layoutReload seam | | internal/appconfig/syncexec.go, internal/ops/litestream.go (modify) | skip a fenced host (journal exists) | | internal/status/snapshot.go (modify) | Box.Layout *LayoutReport{Mode, Rows map[string]RowLayout}; CollectBox takes a classifier | | internal/console/apistatusinstances.go, root statusinstances.go (modify) | GET /api/cli/status/layout; carlos status --layout | | internal/console/view.go, templates/appinstance.html (modify) | RouteView.Layout; a Layout fact on the instance page | | internal/host/systemapply.go (modify) | refuse a candidate that predates the layout while reversal is incomplete | | cli.go, internal/provision/reconcile.go (modify) | refuse --hibernate=false on a nested row; the reconciler's flag application refuses likewise; drift log names the layout | | docs/site/cli.md (modify) | carlos status --layout; the layout section under carlos system` (the verb itself is plan 5) |
---
Task 1: SetInstancePaths, Unfence, the intent object
Files:
- Modify:
internal/registry/registry.go(afterSetDBPath),internal/registry/registry_test.go - Modify:
internal/activator/maintenance.go,internal/activator/maintenance_test.go - Create:
internal/release/layoutintent.go,internal/release/layoutintent_test.go
Interfaces:
- Produces: ``
go // registry func (r *Registry) SetInstancePaths(host, fromDB, fromSock, toDB, toSock string) error // ErrPathsChanged wrapped when the CAS misses var ErrPathsChanged = errors.New("registry: instance paths changed underneath this pass") // activator func (a *Service) Unfence(host string, generation uint64) bool // true when the fence with this generation was removed func (a *Service) Fenced(host string) bool // release const ( LayoutModeNone = ""; LayoutModeFlat = "flat"; LayoutModeNested = "nested" ) type LayoutIntent struct { Schema int; ID string; RequestedAt string; Mode string; Scope LayoutScope } type LayoutScope struct { Box string } // "" = the fleet func LayoutIntentKey() string // "control/layout.json" func LoadLayoutIntent(ctx, s Store) (*LayoutIntent, bool, error) // found=false when absent; schema>1 is an error func (in *LayoutIntent) Applies(boxLabel string) bool func ValidLayoutMode(m string) bool``
- [ ] Step 1: Write the failing tests
internal/registry/registry_test.go (append; use the file's own newTestRegistry/Open helper and Upsert of a minimal Route{Host, Kind: "instance", Socket, DBPath}):
func TestSetInstancePathsIsACompareAndSwapOnBothColumns(t *testing.T) {
r := openTestRegistry(t)
if err := r.Upsert(Route{Host: "h.example.com", Kind: "instance", Socket: "/run/carlos/h.example.com.sock", DBPath: "/data/h.example.com.db"}); err != nil {
t.Fatal(err)
}
if err := r.SetInstancePaths("h.example.com", "/data/h.example.com.db", "/run/carlos/h.example.com.sock", "/data/instances/h.example.com/h.example.com.db", "/run/carlos/instances/h.example.com/app.sock"); err != nil {
t.Fatalf("CAS from the current values: %v", err)
}
rt, _ := r.Get("h.example.com")
if rt.DBPath != "/data/instances/h.example.com/h.example.com.db" || rt.Socket != "/run/carlos/instances/h.example.com/app.sock" {
t.Fatalf("row after CAS = %q %q", rt.DBPath, rt.Socket)
}
// A second CAS from the OLD values misses on both columns and names the current ones.
err := r.SetInstancePaths("h.example.com", "/data/h.example.com.db", "/run/carlos/h.example.com.sock", "/x", "/y")
if !errors.Is(err, ErrPathsChanged) || !strings.Contains(err.Error(), "/data/instances/h.example.com/h.example.com.db") {
t.Fatalf("stale CAS = %v", err)
}
// A CAS where only ONE column matches also misses: both must agree.
err = r.SetInstancePaths("h.example.com", "/data/instances/h.example.com/h.example.com.db", "/run/carlos/h.example.com.sock", "/x", "/y")
if !errors.Is(err, ErrPathsChanged) {
t.Fatalf("half-matching CAS = %v", err)
}
if err := r.SetInstancePaths("absent.example.com", "a", "b", "c", "d"); err == nil || strings.Contains(err.Error(), "changed") {
t.Fatalf("absent host = %v, want a no-route error", err)
}
}internal/activator/maintenance_test.go (append; testActivator is in activator_test.go):
func TestUnfenceReleasesOnlyTheMatchingGeneration(t *testing.T) {
a, _, _, _ := testActivator(t, time.Hour)
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
defer cancel()
if _, err := a.FenceAndStop(ctx, "h.example.com", 7); err != nil && !errors.Is(err, ErrStopUnproven) {
t.Fatal(err)
}
if !a.Fenced("h.example.com") {
t.Fatal("fence not installed")
}
if a.Unfence("h.example.com", 6) {
t.Fatal("an older generation must not release a newer fence")
}
if !a.Fenced("h.example.com") {
t.Fatal("fence lost to an older generation")
}
if !a.Unfence("h.example.com", 7) || a.Fenced("h.example.com") {
t.Fatal("the matching generation must release the fence")
}
if a.Unfence("h.example.com", 7) {
t.Fatal("a second release must report nothing to release")
}
}internal/release/layoutintent_test.go:
package release
import (
"context"
"testing"
)
func TestLayoutIntentRoundTripAndScope(t *testing.T) {
s := NewDirStore(t.TempDir())
ctx := context.Background()
if _, found, err := LoadLayoutIntent(ctx, s); err != nil || found {
t.Fatalf("absent object: found=%v err=%v", found, err)
}
in := &LayoutIntent{Schema: 1, ID: "abc", RequestedAt: "2026-09-12T00:00:00Z", Mode: LayoutModeNested, Scope: LayoutScope{Box: "kass-1"}}
if err := PutLayoutIntent(ctx, s, in, ""); err != nil {
t.Fatal(err)
}
got, found, err := LoadLayoutIntent(ctx, s)
if err != nil || !found || got.Mode != LayoutModeNested || got.ID != "abc" {
t.Fatalf("round trip: %+v %v %v", got, found, err)
}
if !got.Applies("kass-1") || got.Applies("kass-2") {
t.Error("box scope must match only the named box")
}
got.Scope.Box = ""
if !got.Applies("anything") {
t.Error("an empty scope is the fleet")
}
if err := s.PutRaw(ctx, LayoutIntentKey(), []byte(`{"schema":2,"mode":"nested"}`)); err != nil {
t.Fatal(err)
}
if _, _, err := LoadLayoutIntent(ctx, s); err == nil {
t.Fatal("a newer schema must be an error, not a guess")
}
}(PutLayoutIntent(ctx, s, in, etag) and the DirStore's raw put: use whatever Store method the file's siblings use for a CAS put (s.Put(ctx, key, body, etag)); if there is no PutRaw, write the newer-schema object through s.Put with etag "" after deleting the key, or through the DirStore's directory directly.)
- [ ] Step 2: Run the tests to verify they fail
Run: TMPDIR=/tmp/g GOCACHE=/tmp/claude-1001/gocache GOFLAGS=-mod=mod GOPROXY=off go test ./internal/registry ./internal/activator ./internal/release -run 'TestSetInstancePaths|TestUnfence|TestLayoutIntent' -count=1 Expected: build failures for the undefined symbols.
- [ ] Step 3: Write the code
internal/registry/registry.go, after SetDBPath:
// ErrPathsChanged is SetInstancePaths's answer when the row's two path
// columns no longer hold the values the caller compared against.
var ErrPathsChanged = errors.New("registry: instance paths changed underneath this pass")
// SetInstancePaths moves a row between the flat and the nested layout
// (isolated-exec design §8.2 step 8): one UPDATE guarded by BOTH columns'
// current values, so a row that was migrated, reverted or re-added between
// the pass's read and its write is left exactly as it is and the caller
// learns what it holds now. It is SetDBPath's sibling; the two columns are
// swapped together because a row with one flat path and one nested path is
// neither layout and every consumer would refuse it.
func (r *Registry) SetInstancePaths(host, fromDB, fromSock, toDB, toSock string) error {
res, err := r.sql.Exec(`UPDATE routes SET db_path = ?, socket = ? WHERE host = ? AND db_path = ? AND socket = ?`, toDB, toSock, host, fromDB, fromSock)
if err != nil {
return err
}
n, err := res.RowsAffected()
if err != nil {
return err
}
if n == 0 {
var curDB, curSock string
serr := r.sql.QueryRow(`SELECT db_path, socket FROM routes WHERE host = ?`, host).Scan(&curDB, &curSock)
if serr == sql.ErrNoRows {
return fmt.Errorf("no route for host %q", host)
}
if serr != nil {
return serr
}
return fmt.Errorf("%w: %s now holds db %q socket %q", ErrPathsChanged, host, curDB, curSock)
}
return nil
}internal/activator/maintenance.go (append):
// Unfence releases the fence FenceAndStop installed for host, and only
// when the generation matches: a pass that fenced a host is the one that
// may release it, and a later pass's fence is never removed by an earlier
// pass finishing late. It reports whether a fence was removed.
func (a *Service) Unfence(host string, generation uint64) bool {
a.mu.Lock()
defer a.mu.Unlock()
if a.fences == nil || generation == 0 || a.fences[host] != generation {
return false
}
delete(a.fences, host)
return true
}
// Fenced reports whether a maintenance fence is installed for host.
func (a *Service) Fenced(host string) bool { return !a.wakeAllowed(host) }internal/release/layoutintent.go:
// The fleet's layout intent (isolated-exec design §8.2): one small object
// the console writes and every box converges to on every serial move. It
// records a MODE, not a request, so a row that becomes eligible after the
// intent was recorded is migrated on the next pass, and a box that was away
// converges when it returns. An old box never reads the key; an old console
// never writes it.
package release
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
)
const (
LayoutModeNone = ""
LayoutModeFlat = "flat"
LayoutModeNested = "nested"
)
// LayoutScope narrows an intent to one box by label; empty is the fleet.
type LayoutScope struct {
Box string `json:"box,omitempty"`
}
// LayoutIntent is control/layout.json.
type LayoutIntent struct {
Schema int `json:"schema"` // 1
ID string `json:"id"`
RequestedAt string `json:"requested_at"`
Mode string `json:"mode"`
Scope LayoutScope `json:"scope"`
}
func LayoutIntentKey() string { return "control/layout.json" }
func ValidLayoutMode(m string) bool { return m == LayoutModeFlat || m == LayoutModeNested }
// Applies reports whether the intent addresses the box with this label.
func (in *LayoutIntent) Applies(boxLabel string) bool {
return in != nil && (in.Scope.Box == "" || in.Scope.Box == boxLabel)
}
// LoadLayoutIntent reads the object. found=false means it is absent, which
// is every deployment's state until an operator records an intent. A newer
// schema is an error: a box that cannot be sure what the intent means must
// do nothing.
func LoadLayoutIntent(ctx context.Context, s Store) (*LayoutIntent, bool, error) {
raw, _, err := s.Get(ctx, LayoutIntentKey())
if errors.Is(err, os.ErrNotExist) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
var in LayoutIntent
if err := json.Unmarshal(raw, &in); err != nil {
return nil, false, fmt.Errorf("layout intent: %w", err)
}
if in.Schema > 1 {
return nil, false, fmt.Errorf("layout intent: schema %d is newer than this binary understands; upgrade carlos", in.Schema)
}
if !ValidLayoutMode(in.Mode) {
return nil, false, fmt.Errorf("layout intent: mode %q is not flat or nested", in.Mode)
}
return &in, true, nil
}
// PutLayoutIntent writes the object with the caller's etag (create-only when
// empty). Plan 5's console door is its caller; it lives here so the reader
// and the writer share one shape.
func PutLayoutIntent(ctx context.Context, s Store, in *LayoutIntent, etag string) error {
if !ValidLayoutMode(in.Mode) {
return fmt.Errorf("layout intent: mode %q is not flat or nested", in.Mode)
}
in.Schema = 1
body, err := json.Marshal(in)
if err != nil {
return err
}
_, err = s.Put(ctx, LayoutIntentKey(), body, etag)
return err
}(Match Store.Put's real signature from the sibling AppendRestartScope; it returns (etag, error).)
- [ ] Step 4: Run the tests to verify they pass
Run the same command as step 2. Expected: PASS.
- [ ] Step 5: gofmt and commit
gofmt -l internal/registry internal/activator internal/release
git add internal/registry internal/activator internal/release
git commit -m "layout: SetInstancePaths, Unfence, the fleet layout intent object (plan 4, task 1)"---
Task 2: The plan and the journal
Files:
- Create:
internal/layout/plan.go,internal/layout/journal.go,internal/layout/plan_test.go,internal/layout/journal_test.go
Interfaces:
- Produces: ``
go package layout type Mode string // "nested" | "flat" (release.LayoutModeNested/Flat) type Endpoint struct { DB, Socket string } type Artifact struct { Name string; Src, Dst string; Present bool; Required bool } // Name: db, db-wal, db-shm, env, sidecar-env, config-env, socket type Plan struct { Host, IID string; Mode Mode; From, To Endpoint; Artifacts []Artifact } func PlanFor(dataDir, socketDir, host string, mode Mode) (*Plan, error) // error for an unknown mode or an invalid host func rawInstanceDir(dataDir, host string) string // <dataDir>/.instances/<iid> func rawRunDir(socketDir, host string) string // <socketDir>/.instances/<iid> type Phase string; const ( PhaseFiles Phase = "files"; PhaseRow = "row"; PhaseDone = "done" ) type Journal struct { Host string; Mode Mode; From, To Endpoint; Artifacts []JournalArtifact; Phase Phase; WrittenAt string } type JournalArtifact struct { Name string; Present bool } func WriteJournal(dataDir string, j *Journal) error // O_EXCL; ErrJournalExists func ReadJournal(dataDir, host string) (*Journal, bool, error) // found=false when absent; 64KB cap; a FIFO or symlink is an error func UpdatePhase(dataDir string, j *Journal, p Phase) error // rewrite via temp + rename in /data/layout func RemoveJournal(dataDir, host string) error func Validate(j *Journal, dataDir, socketDir string, row registry.Route) (atFrom bool, err error) // §8.2 recovery check``
The socket artifact records presence for reporting only; nothing is renamed for it (a socket file is a leftover and both prepare and SecureUnixListen unlink before binding). It is Required: false and Src/Dst are the two socket paths.
- [ ] Step 1: Write the failing tests
internal/layout/plan_test.go:
package layout
import (
"path/filepath"
"strings"
"testing"
"github.com/carlosframework/platform/internal/provision"
)
func TestPlanForNestedDerivesEveryPathFromTheHost(t *testing.T) {
p, err := PlanFor("/data", "/run/carlos", "h.example.com", Mode("nested"))
if err != nil {
t.Fatal(err)
}
iid := provision.InstanceID("h.example.com")
if p.From != (Endpoint{DB: "/data/h.example.com.db", Socket: "/run/carlos/h.example.com.sock"}) {
t.Errorf("from = %+v", p.From)
}
if p.To != (Endpoint{DB: provision.NestedDBPath("/data", "h.example.com"), Socket: provision.NestedSocket("h.example.com")}) {
t.Errorf("to = %+v", p.To)
}
byName := map[string]Artifact{}
for _, a := range p.Artifacts {
byName[a.Name] = a
}
if byName["db"].Src != "/data/h.example.com.db" || byName["db"].Dst != filepath.Join("/data/.instances", iid, "h.example.com.db") || !byName["db"].Required {
t.Errorf("db artifact = %+v (the destination must be the RAW directory)", byName["db"])
}
if byName["env"].Dst != filepath.Join("/data/.instances", iid, "h.example.com.env") || byName["env"].Required {
t.Errorf("env artifact = %+v", byName["env"])
}
if byName["db-wal"].Src != "/data/h.example.com.db-wal" || byName["db-shm"].Src != "/data/h.example.com.db-shm" {
t.Errorf("sidecar files = %+v %+v", byName["db-wal"], byName["db-shm"])
}
if byName["config-env"].Src != "/data/h.example.com.config.env" || byName["sidecar-env"].Src != "/data/h.example.com-sidecar.env" {
t.Errorf("env files = %+v %+v", byName["config-env"], byName["sidecar-env"])
}
if strings.Contains(byName["db"].Dst, "/data/instances/") {
t.Error("a destination through the bind mount would EXDEV")
}
}
func TestPlanForFlatIsTheExactReverse(t *testing.T) {
n, _ := PlanFor("/data", "/run/carlos", "h.example.com", Mode("nested"))
f, err := PlanFor("/data", "/run/carlos", "h.example.com", Mode("flat"))
if err != nil {
t.Fatal(err)
}
if f.From != n.To || f.To != n.From {
t.Errorf("flat endpoints %+v %+v are not nested's swapped %+v %+v", f.From, f.To, n.From, n.To)
}
for i := range n.Artifacts {
if f.Artifacts[i].Src != n.Artifacts[i].Dst || f.Artifacts[i].Dst != n.Artifacts[i].Src {
t.Errorf("artifact %s not reversed: %+v vs %+v", n.Artifacts[i].Name, f.Artifacts[i], n.Artifacts[i])
}
}
}
func TestPlanForRefusesBadInput(t *testing.T) {
if _, err := PlanFor("/data", "/run/carlos", "h.example.com", Mode("sideways")); err == nil {
t.Error("unknown mode accepted")
}
if _, err := PlanFor("/data", "/run/carlos", "../h", Mode("nested")); err == nil {
t.Error("invalid host accepted")
}
}internal/layout/journal_test.go:
package layout
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/carlosframework/platform/internal/registry"
)
func journalFor(t *testing.T, dd string, mode Mode) *Journal {
t.Helper()
p, err := PlanFor(dd, "/run/carlos", "h.example.com", mode)
if err != nil {
t.Fatal(err)
}
return p.Journal([]bool{true, false, false, true, false, true, false})
}
func TestJournalWriteIsExclusiveAndPhasesAdvance(t *testing.T) {
dd := t.TempDir()
j := journalFor(t, dd, Mode("nested"))
if err := WriteJournal(dd, j); err != nil {
t.Fatal(err)
}
if err := WriteJournal(dd, j); !errors.Is(err, ErrJournalExists) {
t.Fatalf("second write = %v, want ErrJournalExists", err)
}
got, found, err := ReadJournal(dd, "h.example.com")
if err != nil || !found || got.Phase != PhaseFiles || got.Mode != Mode("nested") || len(got.Artifacts) != 7 {
t.Fatalf("read back = %+v %v %v", got, found, err)
}
if err := UpdatePhase(dd, got, PhaseRow); err != nil {
t.Fatal(err)
}
got, _, _ = ReadJournal(dd, "h.example.com")
if got.Phase != PhaseRow {
t.Errorf("phase = %s", got.Phase)
}
if err := RemoveJournal(dd, "h.example.com"); err != nil {
t.Fatal(err)
}
if _, found, _ := ReadJournal(dd, "h.example.com"); found {
t.Error("journal survives removal")
}
}
func TestReadJournalRefusesASymlinkAndAFIFO(t *testing.T) {
dd := t.TempDir()
dir := filepath.Join(dd, "layout")
os.MkdirAll(dir, 0o700)
target := filepath.Join(dd, "elsewhere.json")
os.WriteFile(target, []byte(`{"host":"h.example.com"}`), 0o600)
if err := os.Symlink(target, filepath.Join(dir, "h.example.com.json")); err != nil {
t.Skip(err)
}
if _, _, err := ReadJournal(dd, "h.example.com"); err == nil {
t.Error("a symlinked journal was read")
}
}
func TestValidateChecksBothEndpointsFromTheJournalsOwnDirection(t *testing.T) {
dd := t.TempDir()
j := journalFor(t, dd, Mode("nested"))
flat := registry.Route{Host: "h.example.com", DBPath: j.From.DB, Socket: j.From.Socket}
nested := registry.Route{Host: "h.example.com", DBPath: j.To.DB, Socket: j.To.Socket}
if atFrom, err := Validate(j, dd, "/run/carlos", flat); err != nil || !atFrom {
t.Errorf("row at from: %v %v", atFrom, err)
}
if atFrom, err := Validate(j, dd, "/run/carlos", nested); err != nil || atFrom {
t.Errorf("row at to: %v %v", atFrom, err)
}
other := registry.Route{Host: "h.example.com", DBPath: "/data/other.db", Socket: j.From.Socket}
if _, err := Validate(j, dd, "/run/carlos", other); err == nil {
t.Error("a row at neither endpoint was accepted")
}
// A forged journal whose `to` points somewhere else is refused whatever the row says.
forged := *j
forged.To.DB = "/data/console/console.db"
if _, err := Validate(&forged, dd, "/run/carlos", flat); err == nil {
t.Error("a journal with a foreign destination was accepted")
}
forged = *j
forged.From.DB = "/data/console/console.db"
if _, err := Validate(&forged, dd, "/run/carlos", flat); err == nil {
t.Error("a journal with a foreign source was accepted")
}
// The flat direction swaps the two, and Validate derives from the mode.
r := journalFor(t, dd, Mode("flat"))
if atFrom, err := Validate(r, dd, "/run/carlos", nested); err != nil || !atFrom {
t.Errorf("revert journal, row at from (nested): %v %v", atFrom, err)
}
}- [ ] Step 2: Run the tests to verify they fail
Run: go test ./internal/layout -count=1 (env as above). Expected: build failure.
- [ ] Step 3: Write
plan.go
// Package layout is the pass that moves an activator-managed host between
// the flat layout (<data>/<host>.db, /run/carlos/<host>.sock) and the nested
// one (<data>/instances/<iid>/, /run/carlos/instances/<iid>/), and back
// (isolated-exec design §8.2). It builds its plan from the row and the host
// name only, journals every step in /data/layout, and recovers from the
// journal exactly. Nothing in an instance directory is read to decide
// anything (§3.3).
package layout
import (
"errors"
"fmt"
"path/filepath"
"github.com/carlosframework/platform/internal/controlname"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/release"
)
// Mode is the direction of a move: the layout the host ends in.
type Mode string
const (
ModeNested Mode = release.LayoutModeNested
ModeFlat Mode = release.LayoutModeFlat
)
// Endpoint is the pair of row columns a layout is defined by.
type Endpoint struct {
DB string `json:"db"`
Socket string `json:"socket"`
}
// Artifact is one file the pass moves. Src and Dst are RAW paths for files
// inside the instance tree (design §3.2): the bind mount at /data/instances
// is a different mount from /data/.instances and rename(2) between them
// answers EXDEV, so the pass renames through the raw directory and the
// mount shows the result.
type Artifact struct {
Name string
Src, Dst string
// Required marks the database: a missing required source is an error,
// a missing optional one is recorded absent and skipped (R2-7).
Required bool
Present bool
}
// Plan is everything the pass will do for one host, computed before it
// touches anything.
type Plan struct {
Host string
IID string
Mode Mode
From, To Endpoint
Artifacts []Artifact
}
// artifactNames is the fixed order of the seven artifacts. The journal
// records presence by this order, so it is part of the on-disk contract.
var artifactNames = []string{"db", "db-wal", "db-shm", "env", "sidecar-env", "config-env", "socket"}
// rawInstanceDir and rawRunDir are the ONLY places the platform names the
// raw directories behind the two bind mounts; a grep test pins that.
func rawInstanceDir(dataDir, host string) string {
return filepath.Join(dataDir, ".instances", provision.InstanceID(host))
}
func rawRunDir(socketDir, host string) string {
return filepath.Join(socketDir, ".instances", provision.InstanceID(host))
}
// flatFiles and nestedFiles are the seven artifact paths under each layout,
// in artifactNames order. The nested set is expressed through the raw
// directories.
func flatFiles(dataDir, socketDir, host string) []string {
return []string{
provision.DefaultDBPath(dataDir, host),
provision.DefaultDBPath(dataDir, host) + "-wal",
provision.DefaultDBPath(dataDir, host) + "-shm",
filepath.Join(dataDir, host+".env"),
filepath.Join(dataDir, host+"-sidecar.env"),
filepath.Join(dataDir, host+".config.env"),
filepath.Join(socketDir, host+".sock"),
}
}
func nestedFiles(dataDir, socketDir, host string) []string {
d := rawInstanceDir(dataDir, host)
return []string{
filepath.Join(d, host+".db"),
filepath.Join(d, host+".db-wal"),
filepath.Join(d, host+".db-shm"),
filepath.Join(d, host+".env"),
filepath.Join(d, host+"-sidecar.env"),
filepath.Join(d, host+".config.env"),
filepath.Join(rawRunDir(socketDir, host), "app.sock"),
}
}
// PlanFor derives the plan for host in the given direction from the host
// name alone (design §8.2 step 3). The endpoints are the row values the
// pass will compare against and swap; the artifacts are the files it will
// rename.
func PlanFor(dataDir, socketDir, host string, mode Mode) (*Plan, error) {
if mode != ModeNested && mode != ModeFlat {
return nil, fmt.Errorf("layout: mode %q is not nested or flat", mode)
}
if !controlname.ValidHostShape(host) {
return nil, fmt.Errorf("layout: %q is not a hostname", host)
}
flat := Endpoint{DB: provision.DefaultDBPath(dataDir, host), Socket: provision.DefaultSocket(host)}
nested := Endpoint{DB: provision.NestedDBPath(dataDir, host), Socket: provision.NestedSocket(host)}
src, dst := flatFiles(dataDir, socketDir, host), nestedFiles(dataDir, socketDir, host)
p := &Plan{Host: host, IID: provision.InstanceID(host), Mode: mode, From: flat, To: nested}
if mode == ModeFlat {
p.From, p.To = nested, flat
src, dst = dst, src
}
for i, name := range artifactNames {
p.Artifacts = append(p.Artifacts, Artifact{Name: name, Src: src[i], Dst: dst[i], Required: name == "db"})
}
return p, nil
}
// Journal builds the journal for this plan with the given presence
// vector (one bool per artifact, artifactNames order).
func (p *Plan) Journal(present []bool) *Journal {
j := &Journal{Host: p.Host, Mode: p.Mode, From: p.From, To: p.To, Phase: PhaseFiles}
for i, a := range p.Artifacts {
j.Artifacts = append(j.Artifacts, JournalArtifact{Name: a.Name, Present: i < len(present) && present[i]})
}
return j
}
var errNoSuchArtifact = errors.New("layout: no such artifact")provision.DefaultSocket(host) is <SocketDir>/<host>.sock with the compiled-in provision.SocketDir; socketDir is only used for the artifact paths so tests can point it at a temp dir. controlname.ValidHostShape is what FenceAndStop uses.
- [ ] Step 4: Write
journal.go
package layout
import (
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"time"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
)
// Phase is how far a move got. files: sources are being renamed; row: the
// files are at their destinations and the row is being swapped; done: the
// row is swapped and only the journal remains.
type Phase string
const (
PhaseFiles Phase = "files"
PhaseRow Phase = "row"
PhaseDone Phase = "done"
)
// Journal is /data/layout/<host>.json (design §8.2 step 6). It records
// presence so recovery can tell an optional artifact that was absent from
// one that was lost (R2-7). It is NOT an instruction (§3.3): Validate
// re-derives both endpoints from the host and the mode and refuses a
// journal that names anything else.
type Journal struct {
Host string `json:"host"`
Mode Mode `json:"mode"`
From Endpoint `json:"from"`
To Endpoint `json:"to"`
Artifacts []JournalArtifact `json:"artifacts"`
Phase Phase `json:"phase"`
WrittenAt string `json:"written_at"`
}
type JournalArtifact struct {
Name string `json:"name"`
Present bool `json:"present"`
}
var ErrJournalExists = errors.New("layout: a journal already exists for this host")
const journalMax = 64 << 10
func journalDir(dataDir string) string { return filepath.Join(dataDir, "layout") }
// WriteJournal creates the journal O_EXCL. An existing file, whoever wrote
// it, is a refusal: the pass never overwrites a record of a move it did not
// make.
func WriteJournal(dataDir string, j *Journal) error {
if err := os.MkdirAll(journalDir(dataDir), 0o700); err != nil {
return err
}
if j.WrittenAt == "" {
j.WrittenAt = time.Now().UTC().Format(time.RFC3339)
}
raw, err := json.MarshalIndent(j, "", " ")
if err != nil {
return err
}
f, err := os.OpenFile(provision.LayoutJournalPath(dataDir, j.Host), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o600)
if errors.Is(err, os.ErrExist) {
return ErrJournalExists
}
if err != nil {
return err
}
if _, err := f.Write(append(raw, '\n')); err != nil {
f.Close()
return err
}
return f.Close()
}
// ReadJournal reads the host's journal without following a symlink and
// without blocking on a special file: /data/layout is agent-owned, but a
// flat tenant shares the agent's uid during the transition and can plant
// things there.
func ReadJournal(dataDir, host string) (*Journal, bool, error) {
path := provision.LayoutJournalPath(dataDir, host)
f, err := os.OpenFile(path, os.O_RDONLY|syscallNofollow|syscallNonblock, 0)
if errors.Is(err, os.ErrNotExist) {
return nil, false, nil
}
if err != nil {
return nil, false, err
}
defer f.Close()
fi, err := f.Stat()
if err != nil {
return nil, false, err
}
if !fi.Mode().IsRegular() {
return nil, false, fmt.Errorf("layout: %s is not a regular file", path)
}
raw, err := io.ReadAll(io.LimitReader(f, journalMax+1))
if err != nil {
return nil, false, err
}
if len(raw) > journalMax {
return nil, false, fmt.Errorf("layout: %s is larger than %d bytes", path, journalMax)
}
var j Journal
if err := json.Unmarshal(raw, &j); err != nil {
return nil, false, fmt.Errorf("layout: %s: %w", path, err)
}
return &j, true, nil
}
// UpdatePhase rewrites the journal with a new phase through a temp file and
// rename in the same directory, so a crash leaves either the old phase or
// the new one, never a torn file.
func UpdatePhase(dataDir string, j *Journal, p Phase) error {
j.Phase = p
raw, err := json.MarshalIndent(j, "", " ")
if err != nil {
return err
}
dir := journalDir(dataDir)
tmp, err := os.CreateTemp(dir, "."+j.Host+".tmp-*")
if err != nil {
return err
}
name := tmp.Name()
if _, err := tmp.Write(append(raw, '\n')); err != nil {
tmp.Close()
os.Remove(name)
return err
}
if err := tmp.Close(); err != nil {
os.Remove(name)
return err
}
if err := os.Rename(name, provision.LayoutJournalPath(dataDir, j.Host)); err != nil {
os.Remove(name)
return err
}
return nil
}
func RemoveJournal(dataDir, host string) error {
err := os.Remove(provision.LayoutJournalPath(dataDir, host))
if errors.Is(err, os.ErrNotExist) {
return nil
}
return err
}
// Validate is the recovery check of design §8.2: the journal's endpoints
// must be exactly what PlanFor derives for its host and mode, and the row
// must sit at one of them. It returns whether the row is at `from` (the
// CAS has not run) or at `to` (it has).
func Validate(j *Journal, dataDir, socketDir string, row registry.Route) (atFrom bool, err error) {
if j.Host != row.Host {
return false, fmt.Errorf("layout: journal is for %s, row is %s", j.Host, row.Host)
}
p, err := PlanFor(dataDir, socketDir, j.Host, j.Mode)
if err != nil {
return false, err
}
if j.From != p.From || j.To != p.To {
return false, fmt.Errorf("layout: %s: journal endpoints %+v -> %+v are not the derived %+v -> %+v; not acting", j.Host, j.From, j.To, p.From, p.To)
}
if len(j.Artifacts) != len(artifactNames) {
return false, fmt.Errorf("layout: %s: journal lists %d artifacts, expected %d", j.Host, len(j.Artifacts), len(artifactNames))
}
for i, a := range j.Artifacts {
if a.Name != artifactNames[i] {
return false, fmt.Errorf("layout: %s: journal artifact %d is %q, expected %q", j.Host, i, a.Name, artifactNames[i])
}
}
switch (Endpoint{DB: row.DBPath, Socket: row.Socket}) {
case j.From:
return true, nil
case j.To:
return false, nil
}
return false, fmt.Errorf("layout: %s: row holds db %q socket %q, at neither journal endpoint; not acting", j.Host, row.DBPath, row.Socket)
}syscallNofollow/syscallNonblock are unix.O_NOFOLLOW|unix.O_NONBLOCK in journal_linux.go and 0 in journal_other.go (//go:build !linux), so the package builds everywhere and the checks bite on Linux.
- [ ] Step 5: Run the tests to verify they pass
Run: go test ./internal/layout -count=1. Expected: PASS.
- [ ] Step 6: gofmt and commit
gofmt -l internal/layout
git add internal/layout
git commit -m "layout: the plan from the host name and the exclusive, validated journal (plan 4, task 2)"---
Task 3: Preflight, directories, and the pass
Files:
- Create:
internal/layout/preflight.go,internal/layout/pass.go,internal/layout/pass_linux_test.go,internal/layout/preflight_test.go,internal/layout/rawdir_test.go
Interfaces:
- Produces: ``
go // preflight.go func Preflight(p *Plan, routes []registry.Route) error // ownership: a name that another route owns under the flat rules refuses the plan naming both owners func DestinationCheck(p *Plan, now func() time.Time) (aside []string, err error) // db present at dst = refusal; env/sidecar-env present at dst = moved aside <name>.aside-<unix>; config-env replaced func EnsureDirs(dataDir, socketDir, host string, socketsGID int) error // raw <data>/.instances/<iid> 0700; raw <run>/.instances/<iid> 2750 gid socketsGID (skip chown when gid<0) // pass.go type Fence interface { FenceAndStop(ctx, host string, generation uint64) (*activator.StopEvidence, error); Unfence(host string, generation uint64) bool } type Pass struct { Registry *registry.Registry Fence Fence // nil: no activator (tests); the pass still journals and moves DataDir, SocketDir string SocketsGID int // carlos-sockets gid; -1 in tests Reload func() // routing reload after a row flip; nil = the 2s poll Now func() time.Time Logf func(string, ...any) Generation func() uint64 // monotonic; default time.Now().UnixNano() } type Result struct { Host string; Outcome string; Reason string } // Outcome: migrated | reverted | recovered | skipped | refused | unchanged func (p *Pass) Recover(ctx context.Context) []Result // every journal in /data/layout func (p *Pass) Converge(ctx context.Context, mode Mode) []Result // every managed row not already in mode func (p *Pass) Classify(rt registry.Route, peers []registry.Route) (activator.Layout, string)``
- [ ] Step 1: Write the failing tests
internal/layout/preflight_test.go:
package layout
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/registry"
)
func TestPreflightRefusesAConfigEnvOwnedByAnotherRoute(t *testing.T) {
p, _ := PlanFor("/data", "/run/carlos", "x.example.com", ModeNested)
routes := []registry.Route{{Host: "x.example.com"}, {Host: "x.example.com.config"}}
err := Preflight(p, routes)
if err == nil || !strings.Contains(err.Error(), "x.example.com.config") {
t.Fatalf("ambiguous config-env accepted: %v", err)
}
if err := Preflight(p, []registry.Route{{Host: "x.example.com"}}); err != nil {
t.Fatalf("unambiguous plan refused: %v", err)
}
// Reverting applies the same table: a flat destination another route owns is a refusal.
r, _ := PlanFor("/data", "/run/carlos", "x.example.com", ModeFlat)
if err := Preflight(r, routes); err == nil {
t.Fatal("revert onto another route's file accepted")
}
}
func TestDestinationCheckRefusesTwoDatabasesAndMovesEnvAside(t *testing.T) {
dd := t.TempDir()
p, _ := PlanFor(dd, filepath.Join(dd, "run"), "h.example.com", ModeNested)
os.MkdirAll(rawInstanceDir(dd, "h.example.com"), 0o700)
os.WriteFile(p.Artifacts[0].Dst, []byte("x"), 0o600)
if _, err := DestinationCheck(p, time.Now); err == nil || !strings.Contains(err.Error(), "two databases") {
t.Fatalf("two databases not refused: %v", err)
}
os.Remove(p.Artifacts[0].Dst)
os.WriteFile(p.Artifacts[3].Dst, []byte("fresh env minted by an older binary"), 0o600)
aside, err := DestinationCheck(p, func() time.Time { return time.Unix(1700000000, 0) })
if err != nil {
t.Fatal(err)
}
if len(aside) != 1 || !strings.HasSuffix(aside[0], ".env.aside-1700000000") {
t.Fatalf("aside = %v", aside)
}
if _, err := os.Stat(p.Artifacts[3].Dst); !os.IsNotExist(err) {
t.Error("env destination still occupied")
}
}
func TestEnsureDirsCreatesTheRawDirectoriesWithTheRightModes(t *testing.T) {
dd, rd := t.TempDir(), t.TempDir()
if err := EnsureDirs(dd, rd, "h.example.com", -1); err != nil {
t.Fatal(err)
}
fi, err := os.Stat(rawInstanceDir(dd, "h.example.com"))
if err != nil || fi.Mode().Perm() != 0o700 {
t.Errorf("data dir: %v %v", fi, err)
}
fi, err = os.Stat(rawRunDir(rd, "h.example.com"))
if err != nil || fi.Mode().Perm() != 0o750 || fi.Mode()&os.ModeSetgid == 0 {
t.Errorf("run dir: %v %v", fi, err)
}
}internal/layout/pass_linux_test.go (//go:build linux; the registry opens a real SQLite file, so use registry.Open(filepath.Join(dir, "registry.db")) or the constructor the registry tests use):
//go:build linux
package layout
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
)
type fakeFence struct{ fences map[string]uint64; unproven map[string]bool }
func (f *fakeFence) FenceAndStop(_ context.Context, host string, gen uint64) (*activator.StopEvidence, error) {
if f.fences == nil {
f.fences = map[string]uint64{}
}
f.fences[host] = gen
if f.unproven[host] {
return nil, activator.ErrStopUnproven
}
return &activator.StopEvidence{}, nil
}
func (f *fakeFence) Unfence(host string, gen uint64) bool {
if f.fences[host] != gen {
return false
}
delete(f.fences, host)
return true
}
// box is a temp data dir, a temp socket dir and a registry with one flat
// managed row whose files exist.
func box(t *testing.T) (*Pass, *fakeFence, string) {
t.Helper()
dd := t.TempDir()
rd := filepath.Join(dd, "run")
os.MkdirAll(rd, 0o755)
reg := openTestRegistry(t, dd)
host := "h.example.com"
rt := registry.Route{Host: host, Kind: "instance", Hibernate: true, Socket: provision.DefaultSocket(host), DBPath: provision.DefaultDBPath(dd, host)}
if err := reg.Upsert(rt); err != nil {
t.Fatal(err)
}
os.WriteFile(rt.DBPath, []byte("db"), 0o600)
os.WriteFile(rt.DBPath+"-wal", []byte("wal"), 0o600)
os.WriteFile(filepath.Join(dd, host+".env"), []byte("CARLOS_ADMIN_TOKEN=t\n"), 0o600)
f := &fakeFence{}
var logs []string
p := &Pass{Registry: reg, Fence: f, DataDir: dd, SocketDir: rd, SocketsGID: -1, Now: time.Now,
Logf: func(format string, a ...any) { logs = append(logs, format) }}
return p, f, host
}
func TestConvergeNestedMovesFilesFlipsTheRowAndRemovesTheJournal(t *testing.T) {
p, f, host := box(t)
res := p.Converge(context.Background(), ModeNested)
if len(res) != 1 || res[0].Outcome != "migrated" {
t.Fatalf("results = %+v", res)
}
rt, _ := p.Registry.Get(host)
if rt.DBPath != provision.NestedDBPath(p.DataDir, host) || rt.Socket != provision.NestedSocket(host) {
t.Fatalf("row = %q %q", rt.DBPath, rt.Socket)
}
raw := rawInstanceDir(p.DataDir, host)
for _, name := range []string{host + ".db", host + ".db-wal", host + ".env"} {
if _, err := os.Stat(filepath.Join(raw, name)); err != nil {
t.Errorf("%s not at the raw destination: %v", name, err)
}
}
if _, err := os.Stat(provision.DefaultDBPath(p.DataDir, host)); !os.IsNotExist(err) {
t.Error("flat db still present")
}
if _, found, _ := ReadJournal(p.DataDir, host); found {
t.Error("journal left behind after a clean move")
}
if len(f.fences) != 0 {
t.Error("fence left installed after a clean move")
}
// Idempotent: a second converge does nothing.
if res := p.Converge(context.Background(), ModeNested); len(res) != 1 || res[0].Outcome != "unchanged" {
t.Fatalf("second converge = %+v", res)
}
// Revert restores the flat layout byte for byte.
if res := p.Converge(context.Background(), ModeFlat); len(res) != 1 || res[0].Outcome != "reverted" {
t.Fatalf("revert = %+v", res)
}
if b, err := os.ReadFile(provision.DefaultDBPath(p.DataDir, host)); err != nil || string(b) != "db" {
t.Errorf("flat db after revert: %q %v", b, err)
}
}
func TestConvergeRefusesAnUnprovenStopAndKeepsTheFence(t *testing.T) {
p, f, host := box(t)
f.unproven = map[string]bool{host: true}
res := p.Converge(context.Background(), ModeNested)
if res[0].Outcome != "refused" || !strings.Contains(res[0].Reason, "unproven") {
t.Fatalf("results = %+v", res)
}
if f.fences[host] == 0 {
t.Error("fence released over an unproven stop")
}
if _, found, _ := ReadJournal(p.DataDir, host); found {
t.Error("a journal was written before quiescence")
}
}
func TestConvergeSkipsForeignAndUnmanagedRows(t *testing.T) {
p, _, _ := box(t)
p.Registry.Upsert(registry.Route{Host: "custom.example.com", Kind: "instance", Hibernate: true, Socket: provision.DefaultSocket("custom.example.com"), DBPath: "/data/sub/store.db"})
p.Registry.Upsert(registry.Route{Host: "ext.example.com", Kind: "instance", Hibernate: false, Socket: provision.DefaultSocket("ext.example.com"), DBPath: provision.DefaultDBPath(p.DataDir, "ext.example.com")})
out := map[string]Result{}
for _, r := range p.Converge(context.Background(), ModeNested) {
out[r.Host] = r
}
if out["custom.example.com"].Outcome != "skipped" || !strings.Contains(out["custom.example.com"].Reason, "foreign") {
t.Errorf("foreign row: %+v", out["custom.example.com"])
}
if _, listed := out["ext.example.com"]; listed {
t.Errorf("an unmanaged row was considered: %+v", out["ext.example.com"])
}
}
func TestRecoverCompletesEveryCrashBoundary(t *testing.T) {
// Drive the pass through migrateHost with a hook that aborts after each
// step, then Recover, and assert the end state is the nested layout with
// no journal, every time.
steps := []string{"after-journal", "mid-files", "after-files", "after-cas", "before-remove"}
for _, step := range steps {
t.Run(step, func(t *testing.T) {
p, _, host := box(t)
p.abortAt = step
p.Converge(context.Background(), ModeNested)
p.abortAt = ""
if _, found, _ := ReadJournal(p.DataDir, host); !found {
t.Fatalf("no journal after aborting at %s", step)
}
res := p.Recover(context.Background())
if len(res) != 1 || res[0].Outcome != "recovered" {
t.Fatalf("recover = %+v", res)
}
rt, _ := p.Registry.Get(host)
if rt.DBPath != provision.NestedDBPath(p.DataDir, host) {
t.Errorf("row not nested after recovery from %s: %q", step, rt.DBPath)
}
if _, err := os.Stat(filepath.Join(rawInstanceDir(p.DataDir, host), host+".db")); err != nil {
t.Errorf("db not at destination after recovery from %s", step)
}
if _, found, _ := ReadJournal(p.DataDir, host); found {
t.Error("journal remains after recovery")
}
})
}
}
func TestRecoverRefusesAForgedJournalAndKeepsTheHostFenced(t *testing.T) {
p, _, host := box(t)
j := &Journal{Host: host, Mode: ModeNested, From: Endpoint{DB: "/data/console/console.db", Socket: provision.DefaultSocket(host)},
To: Endpoint{DB: provision.NestedDBPath(p.DataDir, host), Socket: provision.NestedSocket(host)}, Phase: PhaseFiles}
for _, n := range artifactNames {
j.Artifacts = append(j.Artifacts, JournalArtifact{Name: n, Present: true})
}
if err := WriteJournal(p.DataDir, j); err != nil {
t.Fatal(err)
}
res := p.Recover(context.Background())
if res[0].Outcome != "refused" || !strings.Contains(res[0].Reason, "not acting") {
t.Fatalf("forged journal: %+v", res)
}
if _, found, _ := ReadJournal(p.DataDir, host); !found {
t.Error("forged journal removed; the host must stay fenced for an operator")
}
if _, err := os.Stat(filepath.Join(p.DataDir, "console")); err == nil {
t.Error("something touched the forged source")
}
}
func TestConvergeRecoversAJournalBeforeConvergingToANewIntent(t *testing.T) {
p, _, host := box(t)
p.abortAt = "after-files"
p.Converge(context.Background(), ModeNested)
p.abortAt = ""
// The intent changed to flat while the move was in flight: the journal
// completes to ITS endpoint (nested), then the host converges to flat.
res := p.Converge(context.Background(), ModeFlat)
outcomes := []string{}
for _, r := range res {
outcomes = append(outcomes, r.Outcome)
}
rt, _ := p.Registry.Get(host)
if rt.DBPath != provision.DefaultDBPath(p.DataDir, host) {
t.Fatalf("row after recovery and convergence: %q (outcomes %v)", rt.DBPath, outcomes)
}
if _, err := os.Stat(provision.DefaultDBPath(p.DataDir, host)); err != nil {
t.Errorf("db not back at the flat path: %v", err)
}
}internal/layout/rawdir_test.go (the grep test and, where it can, a real bind mount):
package layout
import (
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// The raw directories are named in exactly one package: a second caller
// would be a second way to bypass the bind mount's nosymfollow.
func TestRawInstancesDirectoryIsNamedOnlyHere(t *testing.T) {
root := filepath.Join("..", "..")
out, err := exec.Command("grep", "-rl", "--include=*.go", `\.instances`, filepath.Join(root, "internal"), filepath.Join(root, "main.go")).Output()
if err != nil && len(out) == 0 {
t.Skip("grep unavailable")
}
for _, line := range strings.Split(strings.TrimSpace(string(out)), "\n") {
if line == "" || strings.HasSuffix(line, "_test.go") {
continue
}
if !strings.Contains(line, "internal/layout/") && !strings.Contains(line, "internal/host/bootstrap.go") {
t.Errorf("%s names the raw .instances directory; only internal/layout (and bootstrap's tmpfiles lines) may", line)
}
}
}
// With a real bind mount, a rename THROUGH the mount answers EXDEV and the
// pass's rename through the raw directory succeeds. Needs CAP_SYS_ADMIN;
// skipped where it cannot mount.
func TestRenameThroughTheRawDirectoryAvoidsEXDEV(t *testing.T) {
if os.Geteuid() != 0 {
t.Skip("needs root to bind mount")
}
base := t.TempDir()
raw, view := filepath.Join(base, ".instances"), filepath.Join(base, "instances")
os.MkdirAll(raw, 0o700)
os.MkdirAll(view, 0o700)
if out, err := exec.Command("mount", "--bind", "-o", "nosymfollow", raw, view).CombinedOutput(); err != nil {
t.Skipf("cannot bind mount here: %v %s", err, out)
}
defer exec.Command("umount", view).Run()
src := filepath.Join(base, "h.db")
os.WriteFile(src, []byte("x"), 0o600)
if err := os.Rename(src, filepath.Join(view, "h.db")); err == nil {
t.Fatal("rename through the bind mount succeeded; the EXDEV premise of §3.2 does not hold here")
}
if err := os.Rename(src, filepath.Join(raw, "h.db")); err != nil {
t.Fatalf("rename through the raw directory: %v", err)
}
if _, err := os.Stat(filepath.Join(view, "h.db")); err != nil {
t.Fatalf("the view does not show the renamed file: %v", err)
}
}- [ ] Step 2: Run the tests to verify they fail
Run: go test ./internal/layout -count=1. Expected: build failure.
- [ ] Step 3: Write
preflight.go
package layout
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
"github.com/carlosframework/platform/internal/registry"
)
// Preflight is the ownership check of design §8.2 step 4. Under the flat
// rules <host>.config.env is host's rendered bundle UNLESS a route named
// <host>.config exists, in which case it is that route's instance env
// (SyncExec's own collision rule). Moving it with host would take another
// route's file; reverting onto it would overwrite one. Either is a refusal
// naming both owners.
func Preflight(p *Plan, routes []registry.Route) error {
hosts := map[string]bool{}
for _, rt := range routes {
hosts[rt.Host] = true
}
if other := p.Host + ".config"; hosts[other] {
return fmt.Errorf("layout: %s: %s.config.env is owned by both %s (its bundle) and %s (its instance env); refusing to move it", p.Host, p.Host, p.Host, other)
}
if strings.HasSuffix(p.Host, ".config") {
if owner := strings.TrimSuffix(p.Host, ".config"); hosts[owner] {
return fmt.Errorf("layout: %s: its env file %s.env is also %s's rendered bundle; refusing to move it", p.Host, p.Host, owner)
}
}
return nil
}
// DestinationCheck is step 5: a database already at the destination is a
// refusal (two databases; refusing to choose). An env file already at the
// destination, which an older binary running between a migration and a
// revert can have minted fresh, is moved aside as <name>.aside-<unix> so
// the source, the file the instance ran with for longer, takes its place.
// The rendered bundle is simply replaced; sync re-renders it anyway.
func DestinationCheck(p *Plan, now func() time.Time) (aside []string, err error) {
for _, a := range p.Artifacts {
if a.Name == "socket" {
continue
}
_, statErr := os.Lstat(a.Dst)
if errors.Is(statErr, os.ErrNotExist) {
continue
}
if statErr != nil {
return aside, statErr
}
switch a.Name {
case "db", "db-wal", "db-shm":
return aside, fmt.Errorf("layout: %s: two databases (%s exists at the destination); refusing to choose", p.Host, a.Dst)
case "env", "sidecar-env":
to := fmt.Sprintf("%s.aside-%d", a.Dst, now().Unix())
if err := os.Rename(a.Dst, to); err != nil {
return aside, err
}
aside = append(aside, to)
case "config-env":
if err := os.Remove(a.Dst); err != nil {
return aside, err
}
}
}
return aside, nil
}
// EnsureDirs creates the raw directories the bind mounts show: the data
// directory 0700 (the tenant's own ground) and the run directory 2750 with
// the carlos-sockets group, so an enrolled consumer can reach app.sock. The
// directories must exist before the row flips, because Restore and prepare
// both assume them.
func EnsureDirs(dataDir, socketDir, host string, socketsGID int) error {
if err := os.MkdirAll(rawInstanceDir(dataDir, host), 0o700); err != nil {
return err
}
run := rawRunDir(socketDir, host)
if err := os.MkdirAll(run, 0o750); err != nil {
return err
}
if err := os.Chmod(run, 0o750|os.ModeSetgid); err != nil {
return err
}
if socketsGID >= 0 {
if err := os.Chown(run, -1, socketsGID); err != nil {
return err
}
}
return nil
}
func lstatPresent(path string) (bool, error) {
_, err := os.Lstat(path)
if errors.Is(err, os.ErrNotExist) {
return false, nil
}
return err == nil, err
}
var _ = filepath.Join- [ ] Step 4: Write
pass.go
package layout
import (
"context"
"errors"
"fmt"
"os"
"sort"
"time"
"github.com/carlosframework/platform/internal/activator"
"github.com/carlosframework/platform/internal/provision"
"github.com/carlosframework/platform/internal/registry"
)
// Fence is what the pass needs from the activator: stop the host and keep
// it stopped (FenceAndStop), and let it wake again (Unfence).
type Fence interface {
FenceAndStop(ctx context.Context, host string, generation uint64) (*activator.StopEvidence, error)
Unfence(host string, generation uint64) bool
}
// Pass is the layout pass for one box.
type Pass struct {
Registry *registry.Registry
Fence Fence
DataDir string
SocketDir string
SocketsGID int
Reload func()
Now func() time.Time
Logf func(string, ...any)
Generation func() uint64
// abortAt is a test seam: the step name after which migrateHost returns
// as if the agent had died there.
abortAt string
}
// Result is what the pass says about one host.
type Result struct {
Host string
Outcome string // migrated | reverted | recovered | skipped | refused | unchanged
Reason string
}
const quiesceTimeout = 60 * time.Second
func (p *Pass) now() time.Time {
if p.Now != nil {
return p.Now()
}
return time.Now()
}
func (p *Pass) logf(format string, a ...any) {
if p.Logf != nil {
p.Logf(format, a...)
}
}
func (p *Pass) generation() uint64 {
if p.Generation != nil {
return p.Generation()
}
return uint64(time.Now().UnixNano())
}
func (p *Pass) socketDir() string {
if p.SocketDir != "" {
return p.SocketDir
}
return provision.SocketDir
}
func (p *Pass) layoutInputs(routes []registry.Route) activator.LayoutInputs {
return activator.LayoutInputs{
DataDir: p.DataDir,
SocketDir: p.SocketDir,
Journal: func(host string) bool {
_, found, err := ReadJournal(p.DataDir, host)
return found || (err != nil && !errors.Is(err, os.ErrNotExist))
},
Peers: func() ([]registry.Route, error) { return routes, nil },
}
}
// Classify is the box's one classification, shared with the status
// snapshot so both say the same thing about a row.
func (p *Pass) Classify(rt registry.Route, peers []registry.Route) (activator.Layout, string) {
return activator.Classify(rt, p.layoutInputs(peers))
}
// Recover completes or refuses every journal in /data/layout (design §8.2,
// "Recovery is the journal"). It runs at agent start and at the top of
// every Converge.
func (p *Pass) Recover(ctx context.Context) []Result {
entries, err := os.ReadDir(journalDir(p.DataDir))
if err != nil {
return nil
}
routes, err := p.Registry.List()
if err != nil {
p.logf("layout: recover: list routes: %v", err)
return nil
}
byHost := map[string]registry.Route{}
for _, rt := range routes {
byHost[rt.Host] = rt
}
var out []Result
for _, e := range entries {
name := e.Name()
if len(name) < 6 || name[len(name)-5:] != ".json" || name[0] == '.' {
continue
}
host := name[:len(name)-5]
out = append(out, p.recoverHost(ctx, host, byHost))
}
return out
}
func (p *Pass) recoverHost(ctx context.Context, host string, byHost map[string]registry.Route) Result {
j, found, err := ReadJournal(p.DataDir, host)
if err != nil {
return Result{host, "refused", "journal unreadable: " + err.Error() + "; not acting"}
}
if !found {
return Result{host, "unchanged", ""}
}
rt, ok := byHost[host]
if !ok {
return Result{host, "refused", "journal names a host with no row; not acting"}
}
atFrom, err := Validate(j, p.DataDir, p.socketDir(), rt)
if err != nil {
return Result{host, "refused", err.Error()}
}
plan, _ := PlanFor(p.DataDir, p.socketDir(), host, j.Mode)
gen := p.generation()
if p.Fence != nil {
if _, err := p.Fence.FenceAndStop(deadline(ctx), host, gen); err != nil {
return Result{host, "refused", "recovery: stop unproven: " + err.Error()}
}
defer p.Fence.Unfence(host, gen)
}
switch j.Phase {
case PhaseFiles:
if err := p.moveFiles(plan, j); err != nil {
return Result{host, "refused", err.Error()}
}
if err := UpdatePhase(p.DataDir, j, PhaseRow); err != nil {
return Result{host, "refused", err.Error()}
}
fallthrough
case PhaseRow:
if atFrom {
if err := p.Registry.SetInstancePaths(host, j.From.DB, j.From.Socket, j.To.DB, j.To.Socket); err != nil {
return Result{host, "refused", "row: " + err.Error()}
}
}
if err := UpdatePhase(p.DataDir, j, PhaseDone); err != nil {
return Result{host, "refused", err.Error()}
}
fallthrough
case PhaseDone:
if err := RemoveJournal(p.DataDir, host); err != nil {
return Result{host, "refused", err.Error()}
}
default:
return Result{host, "refused", fmt.Sprintf("journal phase %q unknown; not acting", j.Phase)}
}
p.reload()
p.logf("layout: %s: recovered to %s (journal phase was %s)", host, j.Mode, j.Phase)
return Result{host, "recovered", string(j.Mode)}
}
// moveFiles renames every artifact recorded present that is still at its
// source; one already at its destination is done; one at neither is an
// error that keeps the fence.
func (p *Pass) moveFiles(plan *Plan, j *Journal) error {
for i, a := range plan.Artifacts {
if a.Name == "socket" || !j.Artifacts[i].Present {
continue
}
atDst, err := lstatPresent(a.Dst)
if err != nil {
return err
}
if atDst {
continue
}
atSrc, err := lstatPresent(a.Src)
if err != nil {
return err
}
if !atSrc {
return fmt.Errorf("layout: %s: %s is at neither %s nor %s; not acting", plan.Host, a.Name, a.Src, a.Dst)
}
if err := os.Rename(a.Src, a.Dst); err != nil {
return fmt.Errorf("layout: %s: rename %s: %w", plan.Host, a.Name, err)
}
if p.abortAt == "mid-files" {
return errAborted
}
}
return nil
}
var errAborted = errors.New("layout: aborted by test seam")
// Converge runs Recover, then moves every managed row that is not already
// in mode (design §8.2 steps 1 to 9). Rows are visited in host order so a
// log reads the same way twice.
func (p *Pass) Converge(ctx context.Context, mode Mode) []Result {
out := p.Recover(ctx)
routes, err := p.Registry.List()
if err != nil {
p.logf("layout: converge: list routes: %v", err)
return out
}
sort.Slice(routes, func(i, j int) bool { return routes[i].Host < routes[j].Host })
for _, rt := range routes {
if !activator.ActivatorManaged(rt) {
continue
}
if ctx.Err() != nil {
break
}
out = append(out, p.convergeHost(ctx, rt, routes, mode))
}
return out
}
func (p *Pass) convergeHost(ctx context.Context, rt registry.Route, routes []registry.Route, mode Mode) Result {
layout, reason := p.Classify(rt, routes)
switch layout {
case activator.LayoutFenced:
return Result{rt.Host, "refused", reason}
case activator.LayoutConflict:
return Result{rt.Host, "refused", reason}
case activator.LayoutForeign:
return Result{rt.Host, "skipped", "foreign: " + reason}
case activator.LayoutNested:
if mode == ModeNested {
return Result{rt.Host, "unchanged", ""}
}
case activator.LayoutFlat:
if mode == ModeFlat {
return Result{rt.Host, "unchanged", ""}
}
}
return p.migrateHost(ctx, rt, routes, mode)
}
// migrateHost is steps 2 to 9 for one host. Every early return before the
// journal is written leaves nothing changed; every return after it leaves
// a journal Recover completes.
func (p *Pass) migrateHost(ctx context.Context, rt registry.Route, routes []registry.Route, mode Mode) Result {
host := rt.Host
plan, err := PlanFor(p.DataDir, p.socketDir(), host, mode)
if err != nil {
return Result{host, "refused", err.Error()}
}
if err := Preflight(plan, routes); err != nil {
return Result{host, "refused", err.Error()}
}
gen := p.generation()
if p.Fence != nil {
if _, err := p.Fence.FenceAndStop(deadline(ctx), host, gen); err != nil {
// The fence stays: an unproven stop is never released over.
return Result{host, "refused", "stop unproven: " + err.Error()}
}
defer p.Fence.Unfence(host, gen)
}
// Re-read the row under the fence: the copy we classified could be stale.
fresh, err := p.Registry.Get(host)
if err != nil || fresh == nil {
return Result{host, "refused", "row vanished under the fence"}
}
if (Endpoint{DB: fresh.DBPath, Socket: fresh.Socket}) != plan.From {
return Result{host, "refused", fmt.Sprintf("row moved under the fence: db %q socket %q", fresh.DBPath, fresh.Socket)}
}
if mode == ModeNested {
if err := EnsureDirs(p.DataDir, p.socketDir(), host, p.SocketsGID); err != nil {
return Result{host, "refused", err.Error()}
}
}
aside, err := DestinationCheck(plan, p.now)
if err != nil {
return Result{host, "refused", err.Error()}
}
present := make([]bool, len(plan.Artifacts))
for i, a := range plan.Artifacts {
ok, err := lstatPresent(a.Src)
if err != nil {
return Result{host, "refused", err.Error()}
}
if a.Required && !ok {
return Result{host, "refused", fmt.Sprintf("%s is missing at %s", a.Name, a.Src)}
}
present[i] = ok
}
j := plan.Journal(present)
if err := WriteJournal(p.DataDir, j); err != nil {
return Result{host, "refused", err.Error()}
}
if p.abortAt == "after-journal" {
return Result{host, "refused", "aborted"}
}
if err := p.moveFiles(plan, j); err != nil {
return Result{host, "refused", err.Error()}
}
if p.abortAt == "after-files" {
return Result{host, "refused", "aborted"}
}
if err := UpdatePhase(p.DataDir, j, PhaseRow); err != nil {
return Result{host, "refused", err.Error()}
}
if err := p.Registry.SetInstancePaths(host, plan.From.DB, plan.From.Socket, plan.To.DB, plan.To.Socket); err != nil {
return Result{host, "refused", "row: " + err.Error()}
}
if p.abortAt == "after-cas" {
return Result{host, "refused", "aborted"}
}
if err := UpdatePhase(p.DataDir, j, PhaseDone); err != nil {
return Result{host, "refused", err.Error()}
}
if p.abortAt == "before-remove" {
return Result{host, "refused", "aborted"}
}
if err := RemoveJournal(p.DataDir, host); err != nil {
return Result{host, "refused", err.Error()}
}
p.reload()
outcome := "migrated"
if mode == ModeFlat {
outcome = "reverted"
}
p.logf("layout: %s: %s -> %s (%d file(s) moved aside: %v)", host, plan.From.DB, plan.To.DB, len(aside), aside)
return Result{host, outcome, ""}
}
// reload asks the edge and the activator to pick the new row up now rather
// than on the next poll, so the host's next request runs on the committed
// row (§6.8 covers a request that copied the old one).
func (p *Pass) reload() {
if p.Reload != nil {
p.Reload()
}
}
func deadline(ctx context.Context) context.Context {
c, _ := context.WithTimeout(ctx, quiesceTimeout) //nolint:govet
return c
}(Give deadline a proper cancel: return (ctx, cancel) and defer cancel() at both call sites; the snippet compresses it.) A Result{host, "refused", "aborted"} from a test-seam abort leaves the journal, which is what the recovery tests need.
- [ ] Step 5: Run the tests to verify they pass
Run: go test ./internal/layout -count=1 -race. Expected: PASS; the bind-mount test skips without root.
- [ ] Step 6: gofmt and commit
gofmt -l internal/layout
git add internal/layout
git commit -m "layout: preflight, raw directories, and the recoverable pass (plan 4, task 3)"---
Task 4: Agent wiring and the fenced consumers
Files:
- Modify:
internal/host/agent.go(the pass at start and on a serial move),internal/host/agent_layout_test.go(create) - Modify:
internal/appconfig/syncexec.go,internal/ops/litestream.goand their tests
Interfaces:
- Produces:
func newLayoutPass(reg *registry.Registry, svc *activator.Service, dd string, reload func()) *layout.Pass;func layoutLeg(ctx, store release.Store, boxLabel string, pass *layout.Pass)(reads the intent, converges when it applies, logs each non-unchanged result);fastDepsgainslayoutLeg func(ctx)called afterscheduleLeginfastTick. appconfig.SyncExecandops.FleetDBsWithSkippedgain afenced func(host string) boolseam (nil = never);SyncExecskips a fenced host with a printed line;FleetDBsWithSkippedputs a fenced host's db inskipped.
- [ ] Step 1: Write the failing tests
internal/host/agent_layout_test.go:
package host
import (
"context"
"strings"
"testing"
"github.com/carlosframework/platform/internal/layout"
"github.com/carlosframework/platform/internal/release"
)
func TestLayoutLegConvergesOnlyWhenTheIntentAppliesToThisBox(t *testing.T) {
s := release.NewDirStore(t.TempDir())
ctx := context.Background()
calls := []layout.Mode{}
fake := func(mode layout.Mode) { calls = append(calls, mode) }
// No intent: nothing runs.
layoutLegWith(ctx, s, "kass-1", fake)
if len(calls) != 0 {
t.Fatal("converged with no intent")
}
release.PutLayoutIntent(ctx, s, &release.LayoutIntent{Mode: release.LayoutModeNested, Scope: release.LayoutScope{Box: "kass-2"}}, "")
layoutLegWith(ctx, s, "kass-1", fake)
if len(calls) != 0 {
t.Fatal("converged for another box's intent")
}
release.PutLayoutIntent(ctx, s, &release.LayoutIntent{Mode: release.LayoutModeFlat}, "")
// The second put replaces: read the etag first in the real code; here the DirStore accepts "" as overwrite or the test deletes the key. Adapt to the store's CAS rule.
layoutLegWith(ctx, s, "kass-1", fake)
if len(calls) != 1 || calls[0] != layout.ModeFlat {
t.Fatalf("calls = %v", calls)
}
}(layoutLegWith(ctx, store, label, converge func(layout.Mode)) is the testable core; layoutLeg binds it to the pass.)
internal/appconfig/syncexec_test.go (append): copy TestSyncExecWritesANestedRouteIntoItsInstanceDir and pass fenced returning true for the host; assert no file is written and the output names the fence. internal/ops/ops_test.go (append): a flat route whose fenced returns true lands in skipped.
- [ ] Step 2: Run the tests to verify they fail
Expected: build failures.
- [ ] Step 3: Write the code
agent.go, where the exec backend and activator are built (after svc.Routes = reg):
// The layout pass (isolated-exec design §8.2): recovers any journal
// left by a previous agent before the activator starts, so a host mid-
// move is never woken on a half-moved layout, and converges to the
// fleet's recorded intent on every serial move after that.
pass := newLayoutPass(reg, svc, dd, func() { ed.Reload() })
for _, r := range pass.Recover(ctx) {
if r.Outcome != "unchanged" {
log.Printf("agent: layout recovery: %s: %s %s", r.Host, r.Outcome, r.Reason)
}
}(ed is the edge handle the agent holds; use the real variable name. If the agent does not own an Edge value in this function, pass nil and say so in a comment: the 2s registry poll reloads routing, and §6.8's refresh covers a request that raced it.)
func newLayoutPass(reg *registry.Registry, svc *activator.Service, dd string, reload func()) *layout.Pass {
gid := -1
if g, err := user.LookupGroup(socketsGroup); err == nil {
if n, err := strconv.Atoi(g.Gid); err == nil {
gid = n
}
}
return &layout.Pass{Registry: reg, Fence: svc, DataDir: dd, SocketsGID: gid, Reload: reload, Logf: log.Printf}
}
// layoutLeg rides the serial move like the restart and schedule legs: it
// reads the intent, and when the intent addresses this box it converges.
// Reading the object on every move is one Get; converging is idempotent
// and the results say what changed.
func layoutLeg(ctx context.Context, store release.Store, boxLabel string, pass *layout.Pass) {
layoutLegWith(ctx, store, boxLabel, func(mode layout.Mode) {
for _, r := range pass.Converge(ctx, mode) {
if r.Outcome != "unchanged" {
log.Printf("agent: layout: %s: %s %s", r.Host, r.Outcome, r.Reason)
}
}
})
}
func layoutLegWith(ctx context.Context, store release.Store, boxLabel string, converge func(layout.Mode)) {
in, found, err := release.LoadLayoutIntent(ctx, store)
if err != nil {
log.Printf("agent: layout intent: %v (not converging this move)", err)
return
}
if !found || !in.Applies(boxLabel) {
return
}
converge(layout.Mode(in.Mode))
}socketsGroup is bootstrap's constant for carlos-sockets (read bootstrap.go for its name). The box label is whatever the agent already uses for its snapshot (status.CollectBox's label source). fastDeps gains layoutLeg func(context.Context); fastTick calls it right after scheduleLeg under the same doc.Serial != d.lastSerial branch; the agent sets it to func(ctx) { layoutLeg(ctx, store, label, pass) }. When *activator.Service is nil (no activator armed) the pass gets Fence: nil (an untyped nil, not a nil *Service, so the interface is nil).
syncexec.go: SyncExec(..., ack func(ConfigAck), fenced func(host string) bool) (append the parameter; update the two callers in agent.go with func(h string) bool { _, found, err := layout.ReadJournal(dd, h); return found || err != nil }); in the per-route loop, before rendering: if fenced != nil && fenced(rt.Host) { fmt.Printf(" %s: layout migration in flight; bundle not written this pass\n", rt.Host); continue }. litestream.go: FleetDBsWithSkipped(reg, dataDir, fenced func(host string) bool); a route whose fenced(r.Host) is true is not added to claimed, so its db lands in skipped with the rest. Update callers and the LitestreamConfig generator's call site.
- [ ] Step 4: Gate
Run the full gate (all packages, sandbox off). Expected: green; every existing syncexec and ops test passes with nil for the new parameter.
- [ ] Step 5: gofmt and commit
gofmt -l internal/host internal/appconfig internal/ops
git add internal/host internal/appconfig internal/ops
git commit -m "agent: the layout pass at start and on every serial move; sync and litestream honour the fence (plan 4, task 4)"---
Task 5: Status, the console page, carlos status --layout
Files:
- Modify:
internal/status/snapshot.go,internal/status/snapshot_test.go - Modify:
internal/host/agent.go(the snapshot writer passes the classifier) - Modify:
internal/console/apistatusinstances.go(new door),internal/console/authcli.go(route),internal/console/view.go,internal/console/templates/app_instance.html - Modify: root
status.go/status_instances.go(--layout),docs/site/cli.md
Interfaces:
- Produces: ``
go // status type RowLayout struct { Layout stringjson:"layout"; Reason stringjson:"reason,omitempty"} type LayoutReport struct { Mode stringjson:"mode"; Rows map[string]RowLayoutjson:"rows"} // Mode: none|flat|nested (the intent this box last converged to, "" before any) // Box gains: Layout *LayoutReportjson:"layout,omitempty"// CollectBox(routes, p Probes, now) unchanged; Probes gains Layout *LayoutReport (the agent fills it from pass.Classify over the managed rows and the last intent it saw) // console GET /api/cli/status/layout -> {"boxes":[{"box":label,"mode":...,"stale":bool,"rows":[{"host","layout","reason"}]}]} // fleet-operator gate, same as /status/instances // RouteView gains Layout string (from the box snapshot that reports the host; "unknown (box predates layout reporting)" when none does) // CLI: carlos status --layout``
- [ ] Step 1: Write the failing tests
internal/status/snapshot_test.go: TestCollectBoxCarriesTheLayoutReport (a Probes{Layout: &LayoutReport{Mode: "nested", Rows: {"h.example.com": {Layout: "nested"}}}} round-trips through CollectBox and JSON; a Probes without one yields no layout key in the JSON). Console: a handler test in the style of apistatusinstances_test.go asserting the door is closed to non-operators (404) and lists two boxes' rows, marking a stale box. CLI: a table-rendering test if status_instances.go has one for --instances; otherwise the console test suffices.
- [ ] Step 2: Run the tests to verify they fail
Expected: build failure.
- [ ] Step 3: Write the code
In snapshot.go add the two types and the Box.Layout field with a doc comment: hostnames here are private, like Metrics; the public fleet.json merge never copies it (check merge/the public writer and assert in the test that the fleet document has no layout key). In Probes add Layout *LayoutReport; CollectBox copies it. In the agent's snapshot writer, build it: mode from the last intent layoutLeg saw (a small atomic.Value on the agent, "none" before any), rows from pass.Classify(rt, routes) for every activator.ActivatorManaged row, plus external for an unflagged socket instance route.
Console: handleAPIStatusLayout beside handleAPIStatusInstances, same gate (apiFleetOperator or whatever the instances door uses), reading status.LoadBoxes and emitting one entry per box with Stale from status.BoxStale. Route it in authcli.go next to /api/cli/status/instances. RouteView.Layout is filled in Assemble from a map the caller builds once from LoadBoxes (host -> layout word, first box that reports the host wins; missing -> the "unknown (box predates layout reporting)" string); the template gains <dt>Layout</dt><dd>{{.Row.Layout}}</dd> after Channel.
CLI: carlos status --layout fetches /api/cli/status/layout and prints BOX HOST LAYOUT REASON sorted by box then host, stale boxes suffixed (stale). docs/site/cli.md: under ### carlos status (find it) add the --layout paragraph: what the six words mean and that a fenced or conflict host does not wake; under ### carlos system add a #### Instance layout subsection describing layout migrate|revert as the verb plan 5 adds (write it now so the docsite gate sees the heading once the verb lands; mark the verb "arrives with the console door").
- [ ] Step 4: Gate, including the docsite gate
Run: the full gate plus go test ./internal/docsite/.... Expected: green.
- [ ] Step 5: gofmt and commit
git add internal/status internal/host internal/console status.go status_instances.go docs/site/cli.md
git commit -m "status: the layout report per box, the console's layout fact, carlos status --layout (plan 4, task 5)"---
Task 6: The downgrade guards
Files:
- Modify:
internal/host/systemapply.go,internal/host/systemapply_test.go - Modify:
cli.go(route--hibernate=false),internal/provision/reconcile.go,internal/provision/reconcile_test.go - Modify:
docs/site/cli.md(one sentence undercarlos system: a rollback to a release before the layout is refused while any row is nested)
Interfaces:
- Produces: ``
go // host func layoutReversalIncomplete(reg *registry.Registry, dataDir string) (reasons []string) // rows with DBPath or Socket under /data/instances or /run/carlos/instances (by path shape, any flags), fenced or conflict rows, any journal func candidatePredatesLayout(stagedRoot string) bool // the staged etc/systemd/system tree lacks data-instances.mount // provision func NestedPaths(dataDir string, rt registry.Route) bool // DBPath under <dataDir>/instances or Socket under <SocketDir>/instances (shape only)``
- [ ] Step 1: Write the failing tests
internal/host/systemapply_test.go: using the file's applier fixture, a staged root without data-instances.mount and a registry with one nested row makes Run record a refusal naming the row (errSystemRefused, message contains layout revert); the same staged root with every row flat installs; a staged root WITH the mount unit installs regardless of nested rows. internal/provision/reconcile_test.go: a record with Hibernate: false against an existing provisioned nested row logs layout revert first and leaves Hibernate true; the drift log for a nested row says nested layout instead of the shape-mismatch text. cli.go: a test in the CLI's route test file (or a unit test of the extracted check refuseUnflagNested(rt, dataDir) error) that a nested row refuses --hibernate=false with layout revert first.
- [ ] Step 2: Run the tests to verify they fail
Expected: FAIL.
- [ ] Step 3: Write the code
provision.NestedPaths (provision.go, beside the nested helpers): strings.HasPrefix(rt.DBPath, filepath.Join(dataDir, "instances")+"/") || strings.HasPrefix(rt.Socket, filepath.Join(SocketDir, "instances")+"/").
systemapply.go, in Run after stageCandidate and before install:
// A release before the layout removes the mount units with the binary
// and then serves a nested row from an empty mountpoint directory
// (design §8.2, "That order is enforced"): so a candidate that lacks the
// mount unit is refused while anything on the box is nested, fenced,
// in conflict or mid-move. The check is by path shape and not by flags,
// because a row can leave management while keeping nested paths.
if candidatePredatesLayout(staged) {
if reasons := layoutReversalIncomplete(a.reg, a.dataDir); len(reasons) > 0 {
return a.record(req, fmt.Errorf("%w: %s predates the instance layout and this box still has: %s; run `carlos system layout revert` and wait for every row to report flat first", errSystemRefused, req.Version, strings.Join(reasons, "; ")))
}
}(a.reg/a.dataDir: give the applier a registry-opening seam openRegistry func() (*registry.Registry, error) and the data dir it already knows; a registry that cannot be opened is itself a refusal reason.) candidatePredatesLayout stats <stagedRoot>/etc/systemd/system/data-instances.mount. layoutReversalIncomplete lists rows with provision.NestedPaths, rows a layout.Pass{Registry: reg, DataDir: dd}.Classify calls fenced or conflict, and every *.json in <dd>/layout.
cli.go runRoute, before SetHibernate(rt.Host, false): if !on && provision.NestedPaths(dataDir, rt) { log.Fatalf("route: %s: its paths are nested; layout revert first (a nested row must stay activator-managed so the revert pass can quiesce it)", rt.Host) } (the CLI knows the data dir the same way carlos add does). reconcile.go, in the owned branch where rec.Hibernate != nil && existing.Hibernate != *rec.Hibernate: when !*rec.Hibernate && provision.NestedPaths(cfg.DataDir, existing) log provision: %s: record asks to clear hibernate but the row's paths are nested; layout revert first and skip the write. In the drift block, when provision.NestedPaths(cfg.DataDir, existing) and want.Kind == existing.Kind, log provision: %s: row is on the nested layout (db %q socket %q); the record's flat derivation is not drift instead of the shape-mismatch message.
- [ ] Step 4: Gate
Full gate plus the docsite gate. Expected: green.
- [ ] Step 5: gofmt and commit
git add internal/host internal/provision cli.go docs/site/cli.md
git commit -m "isolated exec: a pre-layout candidate, an unflag, and a reconcile flag-clear all refuse a nested row (plan 4, task 6)"---
Self-review
Spec coverage. §8.2 steps 1 to 9: Task 3 (convergeHost/migrateHost: eligibility via ActivatorManaged and Classify; quiesce via FenceAndStop; plan from host name; preflight; destination check; O_EXCL journal; renames through the raw directory; two-column CAS; journal removal, reload, unfence). Recovery against both endpoints from the journal's direction, every phase, refusal keeps the fence: Tasks 2 and 3. Convergence on every serial move and a later-eligible row: Task 4. Revert: Task 3 (ModeFlat). Downgrade rules (system apply refusal by path shape, unflag refusal in the CLI and the reconciler): Task 6. §8.3 foreign rows skipped and reported: Task 3. §8.4 status field, box mode, console page, CLI: Task 5 (carlos status --layout instead of carlos instances --layout: the status family already carries the operator's per-box view and its gate; spec wording to follow in the next revision). §9 fenced consumers (SyncExec writer, litestream generator): Task 4; movespass and the activator already refuse. §3.2 raw-directory rename pinned by a grep test and a bind-mount test: Task 3. Left to plan 5: the console door and carlos system layout migrate|revert (writers of the intent object this plan reads), layout-marks, the console rollback door's refusal, and the control service's unmounted-mountpoint inspection.
Placeholder scan. Task 5's console and CLI steps describe the code by reference to their siblings (handleAPIStatusInstances, --instances) rather than quoting it: the implementer copies the sibling. Everything else carries code. No "similar to Task N".
Type consistency. Mode, Endpoint, Artifact, Plan, Journal, JournalArtifact, Phase*, PlanFor, WriteJournal, ReadJournal, UpdatePhase, RemoveJournal, Validate, Preflight, DestinationCheck, EnsureDirs, Pass, Fence, Result, Recover, Converge, Classify, rawInstanceDir, rawRunDir, SetInstancePaths, ErrPathsChanged, Unfence, Fenced, LayoutIntent, LayoutScope, LoadLayoutIntent, PutLayoutIntent, LayoutIntentKey, RowLayout, LayoutReport, NestedPaths, layoutReversalIncomplete, candidatePredatesLayout are spelled identically across tasks.