carlos / platform Public

host: paging for moves through the alert sinks, rate-limited per op (§13)

moves-phase1-6 Task 5 (preflight L8). The box pages through moveCfg.Page
only what it alone sees, each rule once, its state the op journal's
paged:<rule> step, so a restarted agent does not page it again:
- stuck, an incident, once per entry into stuck (stuck-<retry epoch>);
  pageStuck moves to movespaging.go and is the one stuck page;
- rolled_back, a warning, where this box closes a rollback or a revert;
- a stop evidence refused for a stray process, an incident from the
  cut's stopping, and M-C20 on the app's platform log (approved
  verbatim);
- a refused start check, an incident from the probe owners (a Page
  field, wired when claim Plan 10 registers the owner set).
The console's move pass keeps release-deadline (hourly), preparing
(once, from moves.PreparingSince), orphan, close-overdue and
abort-overdue; M-C22 stays Plan 5 Task 7a's.

Copy: MV-N385 to MV-N390 pending; M-C20 approved. Census:
recoverRun.finalize's B8 digest (one page call after the close).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 1 week ago 583cf86ebf916218f01638673f876186d6351069 parent c34b88d

11 files changed, +445 −33

approved_moves_copy_test.go +5 −0
docs/superpowers/plans/2026-10-06-moves-phase1-copy.md +8 −2
internal/host/movescut.go +3 −0
internal/host/movespaging.go +193 −0
internal/host/movespaging_test.go +215 −0
internal/host/movespass.go +5 −3
internal/host/movesprobeowners.go +11 −1
internal/host/movesrecover.go +1 −0
internal/host/movesrevert.go +1 −0
internal/host/movesrollback.go +1 −25
internal/provision/plan9_b8_census_test.go +2 −2

diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go

index d28d31ca9..771a07634 100644

--- a/approved_moves_copy_test.go

+++ b/approved_moves_copy_test.go

@@ -83,6 +83,7 @@ var movesCopyIDs = []string{

"MV-N365", "MV-N366", "MV-N367", "MV-N368", "MV-N369", "MV-N370", "MV-N371",

"MV-N372", "MV-N373", "MV-N374", "MV-N375", "MV-N376", "MV-N377", "MV-N378", "MV-N379",

"MV-N380", "MV-N381",

+ "MV-N385", "MV-N386", "MV-N387", "MV-N388", "MV-N389", "MV-N390",

}

// copyRow is one row of the record. The File cell is the file the sentence is

@@ -616,6 +617,7 @@ var movesScanFiles = []string{

"internal/host/movesmeasure_linux.go",

"internal/host/movesmeasure_other.go",

"internal/host/movesparks.go",

+ "internal/host/movespaging.go",

"internal/host/movespass.go",

"internal/host/movesphase1.go",

"internal/host/movespolicy.go",

@@ -786,6 +788,9 @@ var movesReturnedSentences = map[string]map[string]bool{

"internal/console/apiinstancemove.go": {"moveDeleteRefusal": true},

// The moves evidence's anomaly page (Plan 5 Task 9a, MV-N336 and MV-N337).

"internal/host/movesrefusals.go": {"movesAnomalyPage": true},

+ // The box's move pages (Plan 6 Task 5, MV-N385 to MV-N390) and M-C20,

+ // the app's platform log line for a stray process.

+ "internal/host/movespaging.go": {"rolledBackPage": true, "strayPage": true, "strayLine": true, "startRefusedPage": true},

// The console's move pass pages (Plan 5 Task 7, MV-N325 to MV-N334).

"internal/console/movesrecovery.go": {"moveWaitingPage": true, "movePreparingPageText": true, "moveOrphanPage": true, "moveCloseOverduePage": true, "moveAbortOverduePage": true},

}

diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

index 9c0cfd279..4a7e57d81 100644

--- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

+++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

@@ -32,7 +32,7 @@ Columns: id, file, text, status.

| M-C22 | move.go | <host> is waiting for a box in <region>. Its last box was terminated before the move finished. Its data up to the last completed upload is kept in <region>. It will start when a box joins the region, and any writes after that upload will be reported then. | approved |

@@ -312,7 +312,7 @@ The anomaly page (Task 9a's quiet refusals, `internal/host/movesrefusals.go`): a

## Plan 6

-(completed by Plan 6 Task 7. Task 2 takes MV-N319 to MV-N324, and made the abort door speak M-C13 on a console without the host claim, so M-C13 is now approved verbatim. Task 2b made the delete door speak M-C19 through moveDeleteRefusal in internal/console/apiinstancemove.go, so its File cell moves there (preflight L6) and it is approved verbatim. Task 1a takes MV-N350 to MV-N371 and Task 1b MV-N372 to MV-N379, Task 3 MV-N380 (the member line for stuck, preflight L9) and MV-N381 (the name a member line gives the home region, which a box knows only as the record's empty region; Task 3 fix round 1, ruling 4). Task 3 fix round 1 (ruling 6, preflight L12) moved M-C1 and M-C5 to M-C9 into shared templates in internal/moves/operatorcopy.go, which both move.go and the member stream (internal/host/movesevents.go) call, so their File cells, and MV-N380's, name that file; their text is unchanged (Plan 5 owns MV-N325 to MV-N349) for `carlos move` (move.go, and its usage line in main.go), and speaks M-C1 to M-C12 and M-C22 verbatim from move.go, so those rows are approved. Task 1b speaks M-C18 verbatim from move.go and MV-N11 from the resolve door of a console without the host claim, so both are approved)

+(completed by Plan 6 Task 7. Task 2 takes MV-N319 to MV-N324, and made the abort door speak M-C13 on a console without the host claim, so M-C13 is now approved verbatim. Task 2b made the delete door speak M-C19 through moveDeleteRefusal in internal/console/apiinstancemove.go, so its File cell moves there (preflight L6) and it is approved verbatim. Task 1a takes MV-N350 to MV-N371 and Task 1b MV-N372 to MV-N379, Task 3 MV-N380 (the member line for stuck, preflight L9) and MV-N381 (the name a member line gives the home region, which a box knows only as the record's empty region; Task 3 fix round 1, ruling 4). Task 3 fix round 1 (ruling 6, preflight L12) moved M-C1 and M-C5 to M-C9 into shared templates in internal/moves/operatorcopy.go, which both move.go and the member stream (internal/host/movesevents.go) call, so their File cells, and MV-N380's, name that file; their text is unchanged (Plan 5 owns MV-N325 to MV-N349) for `carlos move` (move.go, and its usage line in main.go), and speaks M-C1 to M-C12 and M-C22 verbatim from move.go, so those rows are approved. Task 1b speaks M-C18 verbatim from move.go and MV-N11 from the resolve door of a console without the host claim, so both are approved. Task 4 adds no sentence: the detail answer's timings reuse MV-N319 when their read fails. Task 5 takes MV-N385 to MV-N390, the box's pages (internal/host/movespaging.go, preflight L8), and speaks M-C20 verbatim from movespaging.go, on the app's platform log when the cut's stop evidence finds a stray process, so M-C20 is approved)

|---|---|---|---|

@@ -349,6 +349,12 @@ The anomaly page (Task 9a's quiet refusals, `internal/host/movesrefusals.go`): a

## Review of 2026-10-09

diff --git a/internal/host/movescut.go b/internal/host/movescut.go

index daeceb755..db322735c 100644

--- a/internal/host/movescut.go

+++ b/internal/host/movescut.go

@@ -422,6 +422,9 @@ func (c *cutRun) stopping(ctx context.Context) (bool, error) {

}

if !rep.TreeStopped() {

c.w.notice(host+"#evidence", "moves: %s: the instance is not proven stopped yet (%s); retrying until the cut's deadline", host, failedSteps(rep))

+ if strayRefused(rep) {

+ c.w.pageStray(ctx, c.rec)

+ }

return false, nil

}

j, err := c.journal()

diff --git a/internal/host/movespaging.go b/internal/host/movespaging.go

new file mode 100644

index 000000000..2eef5f67a

--- /dev/null

+++ b/internal/host/movespaging.go

@@ -0,0 +1,193 @@

+package host

+import (

+ "context"

+ "fmt"

+ "log"

+ "strconv"

+ "time"

+ "github.com/carlosframework/platform/internal/alerting"

+ "github.com/carlosframework/platform/internal/moves"

+ "github.com/carlosframework/platform/internal/provision"

+// Paging for moves from the box (spec §13; Plan 6 Task 5; preflight L8). The

+// box pages, through moveCfg.Page (PageOperator, the agent's alert sinks;

+// Plan 5 Task 0), only what it alone sees:

+//

+// - stuck, an incident, once per entry into stuck (pageStuck): the rule is

+// stuck-<epoch>, the record's recovery retry epoch, which only an

+// operator's retry advances, so a retried move that sticks again pages

+// again and a replay of the same mark does not;

+// - rolled_back, a warning, once per op, where this box closes a rollback

+// (recoverRun.finalize) or a revert (revertRun.close);

+// - a stop evidence refused for a process the move cannot account for

+// (StepStray), an incident, once per op, from the cut's stopping; the

+// app's platform log then says M-C20;

+// - a refused start check (moves.StartCheck), an incident, once per op,

+// from the probe owners (movesProbeOwners.startCheck).

+//

+// The console's move pass (internal/console/movesrecovery.go, Plan 5 Task 7)

+// keeps the rules whose facts it holds, and the box does not repeat them:

+// rolling_back past release_deadline (an incident, once an hour), preparing

+// with no promotable round for 60 minutes (a warning, once, measured from

+// moves.PreparingSince, the six-hour abort's own start), an orphaned claim,

+// an overdue close and an overdue abort. A move converted to relocating with

+// no live box in its region (M-C22) is Plan 5 Task 7a's page.

+//

+// A rule's rate state is the op journal on this box (moves.Journal, step

+// paged:<rule>, the time it was marked), so a restarted agent does not page

+// it again. The step is written before the page is sent: a crash between the

+// two, or a send that fails (logged), pages nothing more, as the console's

+// move_pages records an attempt whether or not delivery worked (pages are

+// best effort). A journal that cannot be read or written sends nothing, so a

+// rule that fires every pass can never flood. Unwired (nil Page), nothing

+// pages and nothing is marked.

+// pagedStep is the journal step that records a rule's page.

+const pagedStep = "paged:"

+// The rules' names in the journal.

+const (

+ pageRuleRolledBack = "rolled_back"

+ pageRuleStray = "stray"

+ pageRuleStartCheck = "start_check"

+// pageRuleStuck is the stuck rule of one entry into stuck: its retry epoch.

+func pageRuleStuck(m *provision.MoveState) string {

+ return "stuck-" + strconv.FormatUint(m.RecoveryRetryEpoch, 10)

+// pageOnce sends one page for rule unless the journal of (host, op) under

+// dataDir says it was sent: it marks the journal, then pages. sent is true

+// only when the page was handed to page without error.

+func pageOnce(ctx context.Context, page func(ctx context.Context, level, subject, body string) error, dataDir, host, op, rule string, now time.Time, level, subject, body string) (sent bool) {

+ if page == nil {

+ return false

+ }

+ j, err := moves.ReadJournal(dataDir, host, op)

+ if err != nil {

+ log.Printf("moves: %s: the %s page of %s is not sent: its journal: %v", host, rule, op, err)

+ return false

+ }

+ if j.Done(pagedStep + rule) {

+ return false

+ }

+ j.Mark(pagedStep+rule, now.UTC().Format(time.RFC3339))

+ if err := j.Write(dataDir); err != nil {

+ log.Printf("moves: %s: the %s page of %s is not sent: its journal: %v", host, rule, op, err)

+ return false

+ }

+ if err := page(ctx, level, subject, body); err != nil {

+ log.Printf("moves: %s: the %s page of %s: %v", host, rule, op, err)

+ return false

+ }

+ return true

+// pageOnce is the worker's pageOnce: its pager, journals and clock.

+func (w *movesWorker) pageOnce(ctx context.Context, host, op, rule, level, subject, body string) bool {

+ return pageOnce(ctx, w.cfg.Page, w.cfg.DataDir, host, op, rule, w.now(), level, subject, body)

+// pageStuck pages an operator once a MoveMarkStuck has landed (moves spec

+// §13: stuck is an incident; K11), once per entry into stuck. role is this

+// box's part in the move, "source" or "target", and reason the recorded

+// stuck sentence. The page names the host, the move, the box and the

+// reason, and nothing else: no path, no error text, no secret. Its journal

+// is the one the stopped half keeps: a rollback's source keeps its

+// recovery's (the rollback id), every other half the move id's.

+func (w *movesWorker) pageStuck(ctx context.Context, rec provision.Record, role, reason string) {

+ m := rec.Move

+ if m == nil || m.ID == "" {

+ return

+ }

+ op := m.ID

+ if role == "source" && m.State == provision.MoveRollingBack && m.Rollback != nil && m.Rollback.ID != "" {

+ op = m.Rollback.ID

+ }

+ subject, body := stuckPage(rec.Host, m.ID, w.cfg.Box, role, reason)

+ w.pageOnce(ctx, rec.Host, op, pageRuleStuck(m), alerting.LevelIncident, subject, body)

+// pageRolledBack pages a warning once this box has closed a move rolled

+// back (§13): rec is the record as the close found it, op the journal the

+// closing half keeps (the recovery's rollback id, or the revert's move id).

+func (w *movesWorker) pageRolledBack(ctx context.Context, rec provision.Record, op string) {

+ if rec.Move == nil || rec.Move.ID == "" {

+ return

+ }

+ subject, body := rolledBackPage(rec.Host, rec.Move.ID, w.cfg.Box)

+ w.pageOnce(ctx, rec.Host, op, pageRuleRolledBack, alerting.LevelWarning, subject, body)

+// strayRefused reports whether a stop evidence report was refused for a

+// process the move cannot account for: its StepStray ran and failed. A step

+// that never ran is not a refusal.

+func strayRefused(rep moves.StoppedReport) bool {

+ c, ok := rep.Steps[moves.StepStray]

+ return ok && !c.OK

+// pageStray pages an incident once per op for a stop evidence refused by a

+// stray process (§13), and the first time it is sent the app's platform log

+// says M-C20. rec is the cut's record.

+func (w *movesWorker) pageStray(ctx context.Context, rec provision.Record) {

+ if rec.Move == nil || rec.Move.ID == "" {

+ return

+ }

+ subject, body := strayPage(rec.Host, rec.Move.ID, w.cfg.Box)

+ if !w.pageOnce(ctx, rec.Host, rec.Move.ID, pageRuleStray, alerting.LevelIncident, subject, body) {

+ return

+ }

+ if sink := w.cfg.Events; sink != nil {

+ sink.Event(rec.Account, rec.App, rec.Host, "warn", strayLine(w.cfg.Box, rec.Host))

+ }

+// pageStartRefused pages an incident once per op for a refused §5.1 start

+// check on this box (§13). The probe owners' lock is held across the

+// journal's read and write, as their other journal writes hold it.

+func (m *movesProbeOwners) pageStartRefused(ctx context.Context, host, op string) {

+ now := time.Now()

+ if m.Now != nil {

+ now = m.Now()

+ }

+ subject, body := startRefusedPage(host, op, m.Box)

+ m.mu.Lock()

+ defer m.mu.Unlock()

+ pageOnce(ctx, m.Page, m.DataDir, host, op, pageRuleStartCheck, now, alerting.LevelIncident, subject, body)

+// rolledBackPage is the rolled_back page: the subject (MV-N385) and the

+// body (MV-N386).

+func rolledBackPage(host, id, box string) (subject, body string) {

+ subject = fmt.Sprintf("carlos move rolled back: %s", host)

+ body = fmt.Sprintf("The move %s of %s rolled back, and %s is serving on %s again.", id, host, host, box)

+ return subject, body

+// strayPage is the stray process's page: the subject (MV-N387) and the

+// body (MV-N388).

+func strayPage(host, id, box string) (subject, body string) {

+ subject = fmt.Sprintf("carlos move stray process: %s", host)

+ body = fmt.Sprintf("%s is running a process for %s that the move %s cannot account for, so it cannot prove %s stopped. The cut keeps checking until its deadline.",

+ box, host, id, host)

+ return subject, body

+// strayLine is M-C20, the app's platform log line for the stray process.

+func strayLine(box, host string) string {

+ return fmt.Sprintf("%s is running a process for %s that this move cannot account for. An operator has been paged.", box, host)

+// startRefusedPage is the refused start check's page: the subject

+// (MV-N389) and the body (MV-N390).

+func startRefusedPage(host, id, box string) (subject, body string) {

+ subject = fmt.Sprintf("carlos move start refused: %s", host)

+ body = fmt.Sprintf("%s refused to start %s for the move %s: what it holds of %s is not the generation the host claim names. Nothing was started.",

+ box, host, id, host)

+ return subject, body

diff --git a/internal/host/movespaging_test.go b/internal/host/movespaging_test.go

new file mode 100644

index 000000000..628d19d80

--- /dev/null

+++ b/internal/host/movespaging_test.go

@@ -0,0 +1,215 @@

+package host

+import (

+ "context"

+ "errors"

+ "strings"

+ "testing"

+ "time"

+ "github.com/carlosframework/platform/internal/moves"

+ "github.com/carlosframework/platform/internal/provision"

+// pagesWith counts the pages whose subject starts with prefix.

+func pagesWith(pages pagesSeen, prefix string) (n int, level string) {

+ for _, p := range pages {

+ if strings.HasPrefix(p.subject, prefix) {

+ n++

+ level = p.level

+ }

+ return n, level

+// pagedInJournal reports whether op's journal on dataDir records rule's page.

+func pagedInJournal(t *testing.T, dataDir, host, op, rule string) bool {

+ t.Helper()

+ j, err := moves.ReadJournal(dataDir, host, op)

+ if err != nil {

+ t.Fatal(err)

+ }

+ return j.Done(pagedStep + rule)

+// TestPagingIsRateLimited (Review Focus 4; spec §13; preflight L8): a rule

+// that fires every pass must not flood. On the box, stuck pages once per

+// entry into stuck (a retry that sticks again pages again), rolled_back once

+// per op, a stray process at the stop once per op, and a refused start check

+// once per op; each rule's state is the op journal, so a restarted agent

+// does not page it again. The console's own rules are tested where they

+// live: rolling_back past release_deadline once an hour

+// (TestConsolePagesThroughAlerting), preparing once

+// (TestRecoveryPassAbortsAfterSixHours) from moves.PreparingSince

+// (TestPreparingPageRunsFromPreparingSince).

+func TestPagingIsRateLimited(t *testing.T) {

+ ctx := context.Background()

+ t.Run("stuck, once per entry", func(t *testing.T) {

+ dd := t.TempDir()

+ var pages pagesSeen

+ worker := func() *movesWorker {

+ return &movesWorker{cfg: moveCfg{DataDir: dd, Box: "uk-box", Page: pages.hook(), Now: func() time.Time { return time.Date(2026, 10, 11, 9, 0, 0, 0, time.UTC) }}}

+ }

+ rec := provision.Record{Host: p1Host, Move: &provision.MoveState{ID: p1ID, State: provision.MoveRollingBack,

+ Rollback: &provision.MoveRollback{ID: "rb_abcdefghijklmnopqrstuvwxyz", TargetPhase: provision.RbTargetReturning}}}

+ w := worker()

+ for i := 0; i < 3; i++ {

+ w.pageStuck(ctx, rec, "target", provision.StuckReturnTargetDatabase)

+ }

+ worker().pageStuck(ctx, rec, "target", provision.StuckReturnTargetDatabase) // a restarted agent

+ if n, level := pagesWith(pages, "carlos move stuck: "); n != 1 || level != "incident" {

+ t.Fatalf("stuck pages %d (%s), want one incident: %+v", n, level, pages)

+ }

+ if !pagedInJournal(t, dd, p1Host, p1ID, "stuck-0") {

+ t.Fatal("the target's journal of the move does not record the stuck page")

+ }

+ // An operator's retry advances the epoch; stuck again is a new entry.

+ rec.Move.RecoveryRetryEpoch = 1

+ w.pageStuck(ctx, rec, "target", provision.StuckReturnTargetDatabase)

+ if n, _ := pagesWith(pages, "carlos move stuck: "); n != 2 {

+ t.Fatalf("stuck pages after a retry %d, want two: %+v", n, pages)

+ }

+ // The source of a rollback keeps its recovery's journal.

+ w.pageStuck(ctx, rec, "source", provision.StuckBaselineMismatch)

+ if n, _ := pagesWith(pages, "carlos move stuck: "); n != 3 || !pagedInJournal(t, dd, p1Host, "rb_abcdefghijklmnopqrstuvwxyz", "stuck-1") {

+ t.Fatalf("the source's stuck page %d, want one more, in the recovery's journal: %+v", n, pages)

+ }

+ })

+ t.Run("rolled_back, once per op, at the recovery's close", func(t *testing.T) {

+ h := newRecoverHarness(t, false)

+ h.released(t)

+ h.srcPass(t)

+ h.opened(t)

+ h.closeOut(t)

+ if rec := h.read(t, p1Host); rec.Move != nil || rec.LastMove == nil || rec.LastMove.Outcome != "rolled_back" {

+ t.Fatalf("not closed: %+v %+v", rec.Move, rec.LastMove)

+ }

+ h.srcPass(t)

+ h.sw = nil // a restarted agent

+ h.srcPass(t)

+ h.rollbackPass(t)

+ n, level := pagesWith(h.pages, "carlos move rolled back: ")

+ if n != 1 || level != "warning" {

+ t.Fatalf("rolled_back pages %d (%s), want one warning: %+v", n, level, h.pages)

+ }

+ want := "The move " + p1ID + " of " + p1Host + " rolled back, and " + p1Host + " is serving on ie-box again."

+ for _, p := range h.pages {

+ if strings.HasPrefix(p.subject, "carlos move rolled back: ") && p.body != want {

+ t.Fatalf("the page's body %q, want %q", p.body, want)

+ }

+ })

+ t.Run("rolled_back, once per op, at the revert's close", func(t *testing.T) {

+ h := newRevertHarness(t)

+ h.fromStopped(t)

+ h.arm(t)

+ var pages pagesSeen

+ h.cfg.Page = pages.hook()

+ h.drive(t)

+ h.closed(t)

+ h.pass(t)

+ if n, level := pagesWith(pages, "carlos move rolled back: "); n != 1 || level != "warning" {

+ t.Fatalf("rolled_back pages %d (%s), want one warning: %+v", n, level, pages)

+ }

+ })

+ t.Run("a stray process at the stop, once per op, with M-C20", func(t *testing.T) {

+ h := newCutHarness(t)

+ h.ready(t)

+ var pages pagesSeen

+ sink := &memberSink{}

+ h.cfg.Page, h.cfg.Events = pages.hook(), sink

+ evidence := h.cfg.Evidence

+ h.cfg.Evidence = func(ctx context.Context, host, location string, dbs []string) (moves.StoppedReport, error) {

+ r, err := evidence(ctx, host, location, dbs)

+ r.Steps[moves.StepStray] = moves.Check{Reason: "1 process(es) besides the agent in its leaf"}

+ r.Stray = []int{4242}

+ return r, err

+ }

+ h.cut(t)

+ for i := 0; i < 3; i++ {

+ h.pass(t)

+ }

+ h.freshWorker() // a restarted agent

+ h.pass(t)

+ if m := h.move(t); m.CutPhase != provision.CutStopping || h.ev.calls < 4 {

+ t.Fatalf("the stray process did not hold the cut at stopping: %+v after %d evidence calls", m, h.ev.calls)

+ }

+ if n, level := pagesWith(pages, "carlos move stray process: "); n != 1 || level != "incident" {

+ t.Fatalf("stray pages %d (%s), want one incident: %+v", n, level, pages)

+ }

+ mc20 := "bab/app/" + p1Host + " warn ie-box is running a process for " + p1Host + " that this move cannot account for. An operator has been paged."

+ said := 0

+ for _, e := range sink.events {

+ if e == mc20 {

+ said++

+ }

+ if said != 1 {

+ t.Fatalf("M-C20 said %d times, want once: %q", said, sink.events)

+ }

+ if !pagedInJournal(t, h.cfg.DataDir, p1Host, p1ID, pageRuleStray) {

+ t.Fatal("the cut's journal does not record the stray page")

+ }

+ // Evidence that fails for any other reason pages nothing.

+ h2 := newCutHarness(t)

+ h2.ready(t)

+ var pages2 pagesSeen

+ h2.cfg.Page = pages2.hook()

+ h2.ev.stopped = false // the database is still held open: StepFDs

+ h2.cut(t)

+ h2.pass(t)

+ if len(pages2) != 0 {

+ t.Fatalf("an unproven stop with no stray process paged: %+v", pages2)

+ }

+ })

+ t.Run("a refused start check, once per op", func(t *testing.T) {

+ f := newPOFixture(t)

+ var pages pagesSeen

+ f.owners.Page = pages.hook()

+ o := f.read()

+ in, err := f.owners.Select(ctx, f.route, o, poProbe)

+ if err != nil {

+ t.Fatal(err)

+ }

+ admitted := f.admit(o, in)

+ spawn := poWork{action: "spawn", f: f}

+ f.writeLocal("mv_other", poOp, f.inc)

+ for i := 0; i < 3; i++ {

+ if err := f.owners.Materialize(ctx, admitted, in, spawn); !errors.Is(err, moves.ErrStartFiles) {

+ t.Fatalf("files of another op: %v", err)

+ }

+ // A restarted agent: the owners' memory is gone, the journal is not.

+ f.owners.mu.Lock()

+ f.owners.published, f.owners.fences = nil, nil

+ f.owners.mu.Unlock()

+ f.writeLocal(poOp, "src", f.inc)

+ if err := f.owners.Materialize(ctx, admitted, in, spawn); err == nil {

+ t.Fatal("a database of another generation started")

+ }

+ if n, level := pagesWith(pages, "carlos move start refused: "); n != 1 || level != "incident" {

+ t.Fatalf("start check pages %d (%s), want one incident: %+v", n, level, pages)

+ }

+ if !pagedInJournal(t, f.dataDir, poHost, poOp, pageRuleStartCheck) {

+ t.Fatal("the landing's journal does not record the start check page")

+ }

+ })

+ t.Run("a journal that cannot be written sends nothing", func(t *testing.T) {

+ var pages pagesSeen

+ if pageOnce(ctx, pages.hook(), t.TempDir(), "bad/host", p1ID, pageRuleStray, time.Now(), "incident", "s", "b") || len(pages) != 0 {

+ t.Fatalf("paged without a journal: %+v", pages)

+ }

+ if pageOnce(ctx, nil, t.TempDir(), p1Host, p1ID, pageRuleStray, time.Now(), "incident", "s", "b") {

+ t.Fatal("an unwired pager reported a page")

+ }

+ })

diff --git a/internal/host/movespass.go b/internal/host/movespass.go

index 5d7712bc1..f516507bb 100644

--- a/internal/host/movespass.go

+++ b/internal/host/movespass.go

@@ -652,9 +652,11 @@ type moveCfg struct {

// Page pages an operator (moves spec §13, K11): PageOperator in

// production (armMovesPass), nil in tests, where nothing pages. The box

- // arms call it where a MoveMarkStuck lands (pageStuck): the rollback's

- // return on the target, and the recovery, its dirty return's install

- // and the revert on the source.

+ // arms call it once per rule through the op journal (movespaging.go):

+ // where a MoveMarkStuck lands (pageStuck: the rollback's return on the

+ // target, and the recovery, its dirty return's install and the revert

+ // on the source), where this box closes a rollback or a revert, and

+ // where the cut's stop evidence finds a stray process.

Page func(ctx context.Context, level, subject, body string) error

// The source's recovery after a rollback (§7.8, movesrecover.go).

diff --git a/internal/host/movesprobeowners.go b/internal/host/movesprobeowners.go

index 3ea0cf25e..b92ce3ca1 100644

--- a/internal/host/movesprobeowners.go

+++ b/internal/host/movesprobeowners.go

@@ -127,6 +127,10 @@ type movesProbeOwners struct {

Release func(ctx context.Context, rt registry.Route, version string, work hostwrite.Work) error

NewID func() string

Now func() time.Time

+ // Page pages an operator for a refused start check, once per op

+ // (movespaging.go, §13): PageOperator once claim Plan 10 registers this

+ // owner set; nil, nothing pages.

+ Page func(ctx context.Context, level, subject, body string) error

mu sync.Mutex

published map[string]ClaimPublicationReceipt

@@ -857,7 +861,13 @@ func (m *movesProbeOwners) startCheck(ctx context.Context, o hostclaim.Observed,

if err != nil {

return err

}

- return moves.StartCheck(fresh.Claim, marker, files, db)

+ if err := moves.StartCheck(fresh.Claim, marker, files, db); err != nil {

+ // A refused start check pages an operator once per op (§13,

+ // movespaging.go).

+ m.pageStartRefused(ctx, o.Host, fresh.Claim.Op.ID)

+ return err

+ }

+ return nil

}

// readMarker is the location's current generation.json, nil when absent.

diff --git a/internal/host/movesrecover.go b/internal/host/movesrecover.go

index 6c81656fd..5b518ad59 100644

--- a/internal/host/movesrecover.go

+++ b/internal/host/movesrecover.go

@@ -820,6 +820,7 @@ func (r *recoverRun) finalize(ctx context.Context) (bool, error) {

}

r.w.logTransition(ctx, r.s, pre, *closed)

log.Printf("moves: %s: move %s is rolled back and closed", host, id)

+ r.w.pageRolledBack(ctx, r.rec, r.rb)

nudgeSerial(ctx, r.s)

if !released {

if err := r.fault("after-close-record"); err != nil {

diff --git a/internal/host/movesrevert.go b/internal/host/movesrevert.go

index 23c381cf0..3f21b694c 100644

--- a/internal/host/movesrevert.go

+++ b/internal/host/movesrevert.go

@@ -726,6 +726,7 @@ func (r *revertRun) close(ctx context.Context, releaseClaim bool) (bool, error)

return false, err

}

log.Printf("moves: %s: move %s is reverted and closed", host, id)

+ r.w.pageRolledBack(ctx, r.rec, id)

nudgeSerial(ctx, r.s)

if releaseClaim {

if err := r.fault("after-close-record"); err != nil {

diff --git a/internal/host/movesrollback.go b/internal/host/movesrollback.go

index b9a9eea6d..8d4e0775f 100644

--- a/internal/host/movesrollback.go

+++ b/internal/host/movesrollback.go

@@ -14,7 +14,6 @@ import (

"strings"

"time"

- "github.com/carlosframework/platform/internal/alerting"

"github.com/carlosframework/platform/internal/appconfig"

"github.com/carlosframework/platform/internal/hostbox"

"github.com/carlosframework/platform/internal/hostclaim"

@@ -125,30 +124,7 @@ func (w *movesWorker) now() time.Time {

return time.Now()

}

-// pageStuck pages an operator once a MoveMarkStuck has landed (moves spec

-// §13: stuck is an incident; K11). role is this box's part in the move,

-// "source" or "target", and reason the recorded stuck sentence. The page

-// names the host, the move, the box and the reason, and nothing else: no

-// path, no error text, no secret. Delivery is best effort: a replay that

-// lands the same mark again pages again, and a crash between the mark and

-// the send, or a send that fails (logged, never retried), pages nothing.

-// Unwired (nil Page), nothing pages; a failed page never stops the arm.

-func (w *movesWorker) pageStuck(ctx context.Context, rec provision.Record, role, reason string) {

- f := w.cfg.Page

- if f == nil {

- return

- }

- id := ""

- if rec.Move != nil {

- id = rec.Move.ID

- }

- subject, body := stuckPage(rec.Host, id, w.cfg.Box, role, reason)

- if err := f(ctx, alerting.LevelIncident, subject, body); err != nil {

- log.Printf("moves: %s: the page for the stuck mark of %s: %v", rec.Host, id, err)

- }

-// stuckPage is the stuck mark's page: the subject (MV-N313) and the body

+// stuckPage is the stuck mark's page (pageStuck, movespaging.go): the subject (MV-N313) and the body

// (MV-N314).

func stuckPage(host, id, box, role, reason string) (subject, body string) {

subject = fmt.Sprintf("carlos move stuck: %s", host)

diff --git a/internal/provision/plan9_b8_census_test.go b/internal/provision/plan9_b8_census_test.go

index 26c405d3a..96cfdbf27 100644

--- a/internal/provision/plan9_b8_census_test.go

+++ b/internal/provision/plan9_b8_census_test.go

@@ -543,8 +543,8 @@ var plan9B8CensusInventory = []censusEntry{

{"internal/host/moveslandcommit.go", "landingRun.open", "record-writer", "key:SetRecordMove", "febac996d3bb9caf4021cbc4bcffe1c900bddba4287ff95f083bd5e0751e9bc2"},

// Closes a committed move through SetRecordMove (MoveClose, LastMove done) once nothing is owed under the op, then releases the claim (ReleaseMove via releaseAfterClose): record first, ruling M4. A claim already released by this move gets only the record's close.

{"internal/host/movesparks.go", "movesWorker.closeMove", "record-writer", "key:SetRecordMove", "380a2a0f8cc450573256f359a77da5e451ad506fef84aaeceb9696e3d2904211"},

- // Closes the rollback record through SetRecordMove once nothing is owed under the op, then releases the claim (record first, ruling M4); local journal parking uses separate storage.

- {"internal/host/movesrecover.go", "recoverRun.finalize", "record-writer", "key:SetRecordMove", "509591bb643d58d9314314d4d7c3fd245468502317de750d9e7b28e7a3cd2673"},

+ // Closes the rollback record through SetRecordMove once nothing is owed under the op, then releases the claim (record first, ruling M4); local journal parking uses separate storage. Plan 6 Task 5: after the close it pages rolled_back once (pageRolledBack, the recovery's journal), which writes no record.

+ {"internal/host/movesrecover.go", "recoverRun.finalize", "record-writer", "key:SetRecordMove", "34fd48b2d014c5b1cf9480ae93f9e2c0fa6ff1f1ed7600cc34ee495e29c49315"},

// Advances the source recovery phase through SetRecordMove and its sequenced writer.

{"internal/host/movesrecover.go", "recoverRun.phase", "record-writer", "key:SetRecordMove", "be641eaba77b940ce471696720b637b9dfb9454de0893aca1fcc40750135719d"},

// Records probe inputs through SetRecordMove; phase and stuck helpers use the same writer.