carlos / platform Public

offline move: keep each door's evidence current, and say a stale report is stale (defect 7)

Harness Run 9, finding 7, most of the 366 s outage. The doors believe a box's
report for two minutes, but boxes rewrote theirs only on the moves pass a
door's serial bump started, or every 5 to 6 minutes. An operator who ran flip
more than two minutes after seal was refused until the timed pass, and flip
called the stale report "still runs a process" (MV-O31), so the operator went
looking for a process that was not there.

Boxes: while an offline move is in a state whose reports a door judges by age
(stopping to landed), each source box and the sealer re-run their own step
every 15 s plus jitter (movesWorker.offlineReassert, offlineReassertLoop),
which rewrites the report: offlineSourceStop as a source, offlineKeepStop as
the sealer, under the pass lock, in offlineRound's order. The seal itself and
the target's landing are not repeated (they run on the bump's pass); with no
such move in flight the tick reads nothing. The serial bump still drives
every step: each door bumps, every box's fast loop runs the pass set, and the
sealer nudges at sealed. The box status fields the doors read (claims on, the
edges file's hosts) already follow within one 2 s edges-file poll.

Console: seal, flip and open refuse a report that is there but older than two
minutes with MV-O178 ("the report from <box> is not current. it reports again
within a minute. try again then"), the sealer's included; a missing report
keeps M-C10, a real process keeps MV-O31. MV-O178 is pending Paul's copy
review (moves-phase1-copy.md, movesCopyIDs).

TestOfflineMoveE2E drives the real doors with only production's triggers in
between: each box's fastTick on the serial (nudges included) and the
re-assert tick. Back to back, and three minutes behind every step (the view
reads MV-O178 until the re-assert runs), start to open is accepted first try
at every door.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 4 days ago 78d4ea6015c77906df087e4e4a819a484085ab19 parent ab78c51

10 files changed, +432 −17

approved_moves_copy_test.go +1 −0
docs/superpowers/notes/census-proposal-moves-offline-fixes.md +10 −0
docs/superpowers/plans/2026-10-06-moves-phase1-copy.md +6 −0
internal/console/apiinstancemove_offline.go +31 −6
internal/console/apiinstancemove_offline_test.go +9 −4
internal/host/agent.go +17 −0
internal/host/movesoffline.go +93 −1
internal/host/movesoffline_e2e_test.go +185 −5
internal/host/movesoffline_test.go +77 −0
internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv +3 −1

diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go

index 47f478a5b..99219804e 100644

--- a/approved_moves_copy_test.go

+++ b/approved_moves_copy_test.go

@@ -105,6 +105,7 @@ var movesCopyIDs = []string{

"MV-O142", "MV-O143", "MV-O144", "MV-O145", "MV-O146", "MV-O147", "MV-O148", "MV-O149", "MV-O150", "MV-O151", "MV-O152", "MV-O153", "MV-O154", "MV-O155", "MV-O156", "MV-O157",

"MV-O158", "MV-O159", "MV-O160", "MV-O161", "MV-O162", "MV-O163", "MV-O164", "MV-O165", "MV-O166", "MV-O167", "MV-O168", "MV-O169", "MV-O170", "MV-O171",

"MV-O172", "MV-O173", "MV-O174", "MV-O175", "MV-O176", "MV-O177",

+ "MV-O178",

}

// copyRow is one row of the record. The File cell is the file the sentence is

diff --git a/docs/superpowers/notes/census-proposal-moves-offline-fixes.md b/docs/superpowers/notes/census-proposal-moves-offline-fixes.md

index 9f2981b41..b9c9f20fb 100644

--- a/docs/superpowers/notes/census-proposal-moves-offline-fixes.md

+++ b/docs/superpowers/notes/census-proposal-moves-offline-fixes.md

@@ -35,6 +35,16 @@ move's own units.

Envelopes re-pinned: `internal/appconfig`, `internal/host`.

+### Defect 7: fresh reports at every door step

+No unit row. The re-assert (`movesWorker.offlineReassert`, run by

+`offlineReassertLoop` in the agent every 15 s or so) reads the record from the

+deployment store and calls the source and sealer steps

+census-ruling-moves-offline already classified (`offlineSourceStop`,

+`offlineKeepStop`); it has no root read of its own. The console's stale-report

+refusal (MV-O178) reads nothing new. Envelopes re-pinned: `internal/console`,

+`internal/host`.

## The seal

The accepted-legacy rows add frozen declarations, so the legacy seal moves.

diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

index 504dcc7a4..b278bbec2 100644

--- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

+++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

@@ -682,3 +682,9 @@ The landing's item reasons (`internal/host/movesoffline_land.go`, Task 5 fix rou

+A door's answer to a box report it cannot believe for its age (harness Run 9, finding 7). `flip` answered MV-O31 for a source report older than two minutes, and an operator who reads "still runs a process" goes looking for a process that is not there. `seal`, `flip` and `open` now answer MV-O178 for any report that is there but stale, the sealer's included; a report that is missing keeps M-C10. A box taking part in a move rewrites its report about every 15 seconds, so the step is retried rather than looked into. `<box>` is the box whose report it is. Pending until Paul reviews it.

+|---|---|---|---|

diff --git a/internal/console/apiinstancemove_offline.go b/internal/console/apiinstancemove_offline.go

index 4d0813f9a..74d178a72 100644

--- a/internal/console/apiinstancemove_offline.go

+++ b/internal/console/apiinstancemove_offline.go

@@ -53,7 +53,10 @@ import (

const offlineMoveCapability = "offline-move-1"

// offlineReportFresh is how recently a box's report must be written for a

-// door to believe it. Boxes rewrite their reports on every moves pass.

+// door to believe it. Boxes rewrite their reports on every moves pass, and a

+// source or sealer box every 15 s or so while its report is one a door reads

+// (internal/host offlineReassertEvery), so a report older than this is from a

+// box that has stopped reporting, and the door says so (MV-O178).

const offlineReportFresh = 2 * time.Minute

// Every sentence a door speaks has its row in the copy record (MV-O20 to

@@ -113,6 +116,16 @@ func (s *Server) offlineWaitRefusal(box, from, host string) *offlineRefusal {

return &offlineRefusal{status: http.StatusConflict, msg: fmt.Sprintf("Waiting for %s in %s to confirm it has stopped %s. %s is offline.", box, from, host, host)}

}

+// offlineStaleRefusal is MV-O178: box's report is there but older than

+// offlineReportFresh. That says nothing about the box's process or its stop,

+// only that the door cannot believe the report yet; a box taking part in a

+// move rewrites it every few seconds, so the step is retried, not looked into

+// (harness Run 9, finding 7: flip answered MV-O31 for a stale report, and an

+// operator went looking for a process that was not there).

+func offlineStaleRefusal(box string) *offlineRefusal {

+ return &offlineRefusal{status: http.StatusConflict, msg: fmt.Sprintf("the report from %s is not current. it reports again within a minute. try again then", box)}

// offlineRefusal is what a door refuses with: the status and the sentence.

type offlineRefusal struct {

status int

@@ -798,7 +811,10 @@ func (s *Server) offlineSealCheck(f offlineFleet, ev offlineEvidence, mv *provis

var data []string

for _, b := range src {

rep, ok := ev.get(moves.OfflineRoleSource, b.Box)

- if !ok || !offlineFresh(rep.OfflineReport, now) || !offlineStopped(rep.OfflineReport) || offlineServes(b, host) {

+ if ok && !offlineFresh(rep.OfflineReport, now) {

+ return "", nil, offlineStaleRefusal(b.Box)

+ }

+ if !ok || !offlineStopped(rep.OfflineReport) || offlineServes(b, host) {

return "", nil, s.offlineWaitRefusal(b.Box, from, host)

}

boxes = append(boxes, b.Box)

@@ -841,8 +857,11 @@ func (s *Server) offlineFlipCheck(ev offlineEvidence, mv *provision.MoveState, h

}

for _, box := range mv.Boxes {

rep, ok := ev.get(moves.OfflineRoleSource, box)

- if !ok || !offlineFresh(rep.OfflineReport, now) {

- return processRefusal(box, host)

+ if !ok {

+ return s.offlineWaitRefusal(box, from, host)

+ }

+ if !offlineFresh(rep.OfflineReport, now) {

+ return offlineStaleRefusal(box)

}

if !offlineStopped(rep.OfflineReport) {

return s.offlineWaitRefusal(box, from, host)

@@ -874,12 +893,18 @@ func (s *Server) offlineOpenCheck(ev offlineEvidence, mv *provision.MoveState, h

return offlineStateRefusal(mv.State, host)

}

seal, ok := ev.get(moves.OfflineRoleSealer, mv.Sealer)

- if !ok || !offlineFresh(seal.OfflineReport, now) || seal.TombstoneOwner != moves.TombstoneOwner(mv.ID) || !seal.TombstoneUntil.After(now) {

+ if ok && !offlineFresh(seal.OfflineReport, now) {

+ return offlineStaleRefusal(mv.Sealer)

+ }

+ if !ok || seal.TombstoneOwner != moves.TombstoneOwner(mv.ID) || !seal.TombstoneUntil.After(now) {

return offlineStateRefusal(mv.State, host)

}

for _, box := range mv.Boxes {

rep, ok := ev.get(moves.OfflineRoleSource, box)

- if !ok || !offlineFresh(rep.OfflineReport, now) || !offlineStopped(rep.OfflineReport) {

+ if ok && !offlineFresh(rep.OfflineReport, now) {

+ return offlineStaleRefusal(box)

+ }

+ if !ok || !offlineStopped(rep.OfflineReport) {

return s.offlineWaitRefusal(box, from, host)

}

diff --git a/internal/console/apiinstancemove_offline_test.go b/internal/console/apiinstancemove_offline_test.go

index acf194a07..e588d9da8 100644

--- a/internal/console/apiinstancemove_offline_test.go

+++ b/internal/console/apiinstancemove_offline_test.go

@@ -428,6 +428,9 @@ func msgO25(state string) string {

func msgMC10(box string) string {

return "Waiting for " + box + " in ie to confirm it has stopped " + offlineHost + ". " + offlineHost + " is offline."

}

+func msgO178(box string) string {

+ return "the report from " + box + " is not current. it reports again within a minute. try again then"

func msgO31(box string) string {

return box + " still runs a process for " + offlineHost + ". wait for it to stop, then flip"

}

@@ -732,7 +735,7 @@ func TestOfflineSealDoor(t *testing.T) {

r.At = time.Now().Add(-5 * time.Minute)

}

}, "ie-a", "ie-b")

- w.refused(http.MethodPost, "/seal", "", http.StatusConflict, msgMC10("ie-b"))

+ w.refused(http.MethodPost, "/seal", "", http.StatusConflict, msgO178("ie-b"))

// A box still listing the host has not stopped serving it.

w.stopped(nil, "ie-a", "ie-b")

still := ieOff("ie-b")

@@ -830,8 +833,10 @@ func TestOfflineFlipDoor(t *testing.T) {

w.refused(http.MethodPost, "/flip", "", http.StatusConflict, msgO31("ie-box"))

w.stopped(func(_ string, r *moves.OfflineReport) { r.LayoutJournal = true }, "ie-box")

w.refused(http.MethodPost, "/flip", "", http.StatusConflict, msgO31("ie-box"))

+ // A stale report is not a process (harness Run 9, finding 7): MV-O178,

+ // never MV-O31.

w.stopped(func(_ string, r *moves.OfflineReport) { r.At = time.Now().Add(-3 * time.Minute) }, "ie-box")

- w.refused(http.MethodPost, "/flip", "", http.StatusConflict, msgO31("ie-box"))

+ w.refused(http.MethodPost, "/flip", "", http.StatusConflict, msgO178("ie-box"))

w.stopped(func(_ string, r *moves.OfflineReport) { r.Fenced = false }, "ie-box")

w.refused(http.MethodPost, "/flip", "", http.StatusConflict, msgMC10("ie-box"))

// All clear.

@@ -876,7 +881,7 @@ func TestOfflineOpenDoorRechecksTheStop(t *testing.T) {

r.TombstoneOwner = moves.TombstoneOwner(w.id)

r.TombstoneUntil = time.Now().Add(time.Hour)

})

- w.refused(http.MethodPost, "/open", "", http.StatusConflict, msgO25(provision.OfflineLanded))

+ w.refused(http.MethodPost, "/open", "", http.StatusConflict, msgO178("ie-box"))

reset()

// A tombstone that is another move's, or none, or run out.

for name, edit := range map[string]func(*moves.OfflineReport){

@@ -901,7 +906,7 @@ func TestOfflineOpenDoorRechecksTheStop(t *testing.T) {

w.stopped(func(_ string, r *moves.OfflineReport) { r.Fenced = false }, "ie-box")

w.refused(http.MethodPost, "/open", "", http.StatusConflict, msgMC10("ie-box"))

w.stopped(func(_ string, r *moves.OfflineReport) { r.At = time.Now().Add(-10 * time.Minute) }, "ie-box")

- w.refused(http.MethodPost, "/open", "", http.StatusConflict, msgMC10("ie-box"))

+ w.refused(http.MethodPost, "/open", "", http.StatusConflict, msgO178("ie-box"))

reset()

// The seal changed under the move.

sealKey := moves.OfflineSealKey(offlineHost, w.id)

diff --git a/internal/host/agent.go b/internal/host/agent.go

index 1bd1078f8..451941db1 100644

--- a/internal/host/agent.go

+++ b/internal/host/agent.go

@@ -3322,6 +3322,9 @@ func instancesLoop(reg *registry.Registry, writes hostwrite.Config, bs *buildSta

// predicate, already covered by TestShouldSkipPublisherElection, answers

// both.

isFleetBox := shouldSkipPublisherElection(os.Getenv)

+ // An offline move's participants keep their reports current between the

+ // passes below (finding 7); idle, it reads nothing.

+ go offlineReassertLoop(ctx, store)

delay := time.Minute

for {

time.Sleep(delay)

@@ -3353,6 +3356,20 @@ func instancesLoop(reg *registry.Registry, writes hostwrite.Config, bs *buildSta

}

+// offlineReassertLoop runs the armed moves worker's offlineReassert every

+// offlineReassertEvery plus up to 5 s of jitter, for the life of the agent:

+// while this box takes part in an offline move, its source and sealer reports

+// stay current for the doors whatever the serial does. Thin and untested by

+// convention, like instancesLoop; the tick is offlineReassert.

+func offlineReassertLoop(ctx context.Context, s release.Store) {

+ for {

+ time.Sleep(offlineReassertEvery + rand.N(5*time.Second))

+ if w := armedMovesWorker.Load(); w != nil {

+ w.offlineReassert(ctx, s)

+ }

// retentionLoop runs the release-retention pass forever on the slow

// reconcile cadence (5min + jitter, first tick after 1 minute) — never the

// 2s fast pass: pruning is not urgent, and a policy written now is enforced

diff --git a/internal/host/movesoffline.go b/internal/host/movesoffline.go

index cb1019574..baa6202d8 100644

--- a/internal/host/movesoffline.go

+++ b/internal/host/movesoffline.go

@@ -139,6 +139,7 @@ type offlineView struct {

type offlineSnapshot struct {

moves map[string]provision.MoveState // host -> its offline move

+ owners map[string][2]string // host -> the account and app its record lives under

failed bool // the listing failed: keep everything

// unreadable are the hosts whose record read but did not decode, or

// disagreed with its path (provision.ListAllRecordsStrict): their move

@@ -157,10 +158,11 @@ func newOfflineView(cfg moveCfg, written func(host string)) *offlineView {

// (fail closed for keep only; the hold never reads the view) and logged at

// most once a minute each.

func (v *offlineView) refresh(recs []provision.Record, unreadable map[string]bool) {

- next := &offlineSnapshot{moves: map[string]provision.MoveState{}, unreadable: map[string]bool{}}

+ next := &offlineSnapshot{moves: map[string]provision.MoveState{}, owners: map[string][2]string{}, unreadable: map[string]bool{}}

for _, rec := range recs {

if rec.Move.Offline() {

next.moves[rec.Host] = *rec.Move

+ next.owners[rec.Host] = [2]string{rec.Account, rec.App}

}

now := time.Now()

@@ -319,6 +321,96 @@ func (w *movesWorker) offlineRound(ctx context.Context, s release.Store, rec pro

return errors.Join(errs...)

}

+// offlineReassertEvery is how often a box that takes part in an offline move

+// rewrites the reports a door judges by age (harness Run 9, finding 7). The

+// doors believe a report for two minutes (offlineReportFresh, internal/

+// console), and the moves pass runs on a serial move or every five to six

+// minutes. So a report written on the pass one door's bump started was stale

+// by the time an operator who waited over two minutes ran the next door, and

+// flip and open refused until the timed pass: most of Run 9's 366 s offline.

+// At this cadence a participant's report is never much more than 20 s old.

+const offlineReassertEvery = 15 * time.Second

+// offlineReassertRoles says which of this box's steps keep its report current

+// while a move is in m's state: a source box's stop from stopping to landed,

+// and the sealer's keep from sealed to landed. Those are the reports seal,

+// flip and open judge by age. The seal itself (sealing) and the target's

+// landing are not repeated here: they are long, and they run on the pass a

+// door's serial bump starts. From open on no door reads a report by age.

+func offlineReassertRoles(m *provision.MoveState, source, sealer bool) (asSource, asSealer bool) {

+ switch m.State {

+ case provision.OfflineStopping, provision.OfflineSealing:

+ return source, false

+ case provision.OfflineSealed, provision.OfflineLanding, provision.OfflineLanded:

+ return source, sealer

+ }

+ return false, false

+// offlineReassert is one tick of offlineReassertLoop: for each offline move

+// the view holds in which this box's report is judged by age, it reads the

+// record again and runs this box's source or sealer step for it, in

+// offlineRound's order, under the pass lock. Each step is an ensure that ends

+// by rewriting the box's report (offlineSourceStop, offlineKeepStop). With no

+// such move in flight it reads nothing at all. It returns how many records it

+// acted on.

+func (w *movesWorker) offlineReassert(ctx context.Context, s release.Store) int {

+ cfg := w.cfg

+ v := cfg.Offline

+ if !cfg.OfflineCapable || v == nil || s == nil || cfg.Box == "" {

+ return 0

+ }

+ snap := v.snap.Load()

+ if snap == nil || snap.failed || len(snap.moves) == 0 {

+ return 0

+ }

+ here := cfg.parkRegion()

+ roles := func(m *provision.MoveState) (bool, bool) {

+ source := here != "" && park.PlacementMatches(cfg.ParkHomeRegion, m.From, here)

+ return offlineReassertRoles(m, source, m.Sealer == cfg.Box)

+ }

+ var hosts []string

+ for host, m := range snap.moves {

+ if asSource, asSealer := roles(&m); asSource || asSealer {

+ hosts = append(hosts, host)

+ }

+ if len(hosts) == 0 {

+ return 0

+ }

+ sort.Strings(hosts)

+ w.pass.Lock()

+ defer w.pass.Unlock()

+ acted := 0

+ for _, host := range hosts {

+ owner := snap.owners[host]

+ rec, err := provision.GetRecord(ctx, s, owner[0], owner[1], host)

+ if err != nil {

+ w.logTrouble(host, fmt.Errorf("offline move: read the record to report again: %w", err))

+ continue

+ }

+ if !rec.Move.Offline() {

+ continue

+ }

+ asSource, asSealer := roles(rec.Move)

+ if !asSource && !asSealer {

+ continue

+ }

+ acted++

+ var errs []error

+ if asSource {

+ errs = append(errs, w.offlineSourceStep(ctx, s, *rec))

+ }

+ if asSealer {

+ errs = append(errs, w.offlineSealerStep(ctx, s, *rec))

+ }

+ if err := errors.Join(errs...); err != nil {

+ w.logTrouble(host, err)

+ }

+ return acted

// now is cfg.Now, or the wall clock when a test left it unset.

func (c moveCfg) now() time.Time {

if c.Now != nil {

diff --git a/internal/host/movesoffline_e2e_test.go b/internal/host/movesoffline_e2e_test.go

index 32f356db6..398b87e3a 100644

--- a/internal/host/movesoffline_e2e_test.go

+++ b/internal/host/movesoffline_e2e_test.go

@@ -8,6 +8,7 @@ import (

"encoding/base64"

"encoding/json"

"errors"

+ "fmt"

"io"

"net/http"

"net/http/httptest"

@@ -407,14 +408,24 @@ func (w *e2eWorld) publish(b *offBox) {

// report freshness by.

func (w *e2eWorld) tick(b *offBox) {

w.t.Helper()

- w.wd.mu.Lock()

- w.wd.now = time.Now()

- w.wd.mu.Unlock()

+ w.tickAt(b, time.Now())

+// tickAt is tick with the boxes' clock reading at.

+func (w *e2eWorld) tickAt(b *offBox, at time.Time) {

+ w.t.Helper()

+ w.setBoxClock(at)

b.reconcile()

w.publish(b)

b.pass()

}

+func (w *e2eWorld) setBoxClock(at time.Time) {

+ w.wd.mu.Lock()

+ w.wd.now = at

+ w.wd.mu.Unlock()

func (w *e2eWorld) tickAll() {

w.t.Helper()

w.tick(w.ie)

@@ -663,8 +674,15 @@ func (w *e2eWorld) begin() {

t.Fatal("ie-box still lists the host after its stop")

}

- // Config sync's render at rest, standing in: the source's config.env

- // from the bundle (store binding and secret).

+ w.renderSourceConfig()

+// renderSourceConfig stands in for config sync's render at rest: the

+// source's config.env from the bundle (store binding and secret), and the

+// source's bytes the move is judged against.

+func (w *e2eWorld) renderSourceConfig() {

+ t := w.t

+ t.Helper()

hr, err := w.ie.w.cfg.RenderHeld(context.Background(), w.wd.s, *w.ie.row(offHost), w.id, nil, func(string) error { return nil })

if err != nil {

t.Fatal(err)

@@ -1048,6 +1066,168 @@ func TestOfflineMoveE2E(t *testing.T) {

t.Run("nested source with launch files", func(t *testing.T) { offlineMoveDone(t, landOpts{nested: true}) })

t.Run("no files directory", func(t *testing.T) { offlineMoveDone(t, landOpts{noFiles: true}) })

t.Run("rollback after a tampered archive", offlineMoveRollback)

+ // Harness Run 9, finding 7: the doors driven with nothing between them

+ // but what production runs (each door's serial bump, the boxes' own

+ // nudges and the offline re-assert loop), first back to back, then with

+ // the operator three minutes behind at every step, past the doors' two

+ // minutes of belief in a report.

+ t.Run("doors back to back", func(t *testing.T) { offlineMoveDoorsApart(t, 0) })

+ t.Run("doors three minutes apart", func(t *testing.T) { offlineMoveDoorsApart(t, 3*time.Minute) })

+// e2eLoops are what acts on the two boxes between door calls in production,

+// and nothing else: each box's fast-converge tick (fastTick, the real serial

+// rule: a moved serial runs the pass set once per clamp, and a nudge made

+// during the pass runs it again), whose pass set is the agent's order

+// (reconcile, the edges file, the moves pass); and each box's offline

+// re-assert tick (offlineReassert). Nothing else ticks a box.

+type e2eLoops struct {

+ w *e2eWorld

+ fast map[*offBox]*fastDeps

+ tickAt time.Time // fastTick's own clock, stepped past the clamp each round

+ boxAt time.Time // what the boxes' clock reads during a serial pass

+ passes int // serial passes run

+ again int // re-assert ticks run

+func (w *e2eWorld) loops() *e2eLoops {

+ l := &e2eLoops{w: w, fast: map[*offBox]*fastDeps{}, tickAt: time.Now()}

+ for _, b := range []*offBox{w.ie, w.uk} {

+ // The sweep's registry is empty: the sweep stages releases and is

+ // not what this test is about. The serial leg is.

+ l.fast[b] = &fastDeps{reg: restartTestRegistry(w.t), store: w.rs, fullPass: func(context.Context) {

+ l.passes++

+ w.tickAt(b, l.boxAt)

+ }}

+ }

+ l.round() // the boot read seeds each box's serial

+ return l

+// round is one fastTick on each box; it reports whether either ran its pass.

+func (l *e2eLoops) round() bool {

+ before := l.passes

+ l.tickAt = l.tickAt.Add(fastFullClamp + time.Second)

+ for _, b := range []*offBox{l.w.ie, l.w.uk} {

+ fastTick(context.Background(), l.fast[b], l.tickAt)

+ }

+ return l.passes > before

+// after is what the boxes do once a door has bumped the serial, when the

+// operator runs the next door gap later: the serial passes at the time of

+// the bump, then the re-assert loop's ticks through the gap, up to now.

+func (l *e2eLoops) after(gap time.Duration) {

+ t := l.w.t

+ t.Helper()

+ l.boxAt = time.Now().Add(-gap)

+ rounds := 0

+ for l.round() {

+ if rounds++; rounds > 4 {

+ t.Fatal("the serial kept moving after four rounds of passes")

+ }

+ if gap > 0 {

+ // Without the re-assert, a door would read the reports as the bump's

+ // passes left them: older than it believes. The status view says

+ // so, and never as a process (MV-O31).

+ if v := l.w.status(); v.Next != "" || !strings.Contains(v.Waiting, "is not current") {

+ t.Fatalf("%s behind the bump, the view read %+v; want a report that is not current", gap, v)

+ }

+ for at := l.boxAt.Add(offlineReassertEvery); !at.After(time.Now()); at = at.Add(offlineReassertEvery) {

+ l.reassert(at)

+ }

+func (l *e2eLoops) reassert(at time.Time) {

+ l.w.setBoxClock(at)

+ for _, b := range []*offBox{l.w.ie, l.w.uk} {

+ b.w.offlineReassert(context.Background(), l.w.rs)

+ }

+ l.again++

+// offlineMoveDoorsApart drives start to open with the operator gap behind

+// each door's bump, and wants every door accepted on its first or second try,

+// the second after one more re-assert tick. It logs the time each door took

+// to its answer and from start to open.

+func offlineMoveDoorsApart(t *testing.T, gap time.Duration) {

+ w := newE2EWorld(t, landOpts{})

+ l := w.loops()

+ tries := map[string]int{}

+ door := func(verb, body, want string) e2eView {

+ t.Helper()

+ for try := 1; ; try++ {

+ code, out := w.door(verb, body)

+ if code == http.StatusAccepted {

+ tries[verbName(verb)] = try

+ v := w.decode(out)

+ if v.State != want {

+ t.Fatalf("%s: state %q, want %q", verbName(verb), v.State, want)

+ }

+ return v

+ }

+ if try == 2 {

+ t.Fatalf("%s refused twice, %s behind the previous step: %d %s", verbName(verb), gap, code, out)

+ }

+ l.reassert(time.Now())

+ }

+ next := func(step, verb string) {

+ t.Helper()

+ if v := w.status(); v.Next != verb {

+ t.Fatalf("after %s, %s behind: next %q, want %q (waiting %q)", step, gap, v.Next, verb, v.Waiting)

+ }

+ timers, err := w.ie.reg.TimersFor(offHost)

+ if err != nil {

+ t.Fatal(err)

+ }

+ w.srcTimers = map[string]registry.Timer{}

+ for _, tm := range timers {

+ w.srcTimers[tm.Name] = tm

+ }

+ began := time.Now()

+ var took []string

+ timed := func(name string, f func()) {

+ at := time.Now()

+ f()

+ took = append(took, fmt.Sprintf("%s %s", name, time.Since(at).Round(time.Millisecond)))

+ }

+ timed("start", func() {

+ w.id = door("", `{"region":"uk"}`, provision.OfflineStopping).ID

+ l.after(gap)

+ })

+ next("start", "seal")

+ w.renderSourceConfig()

+ timed("seal", func() {

+ door("seal", "", provision.OfflineSealing)

+ l.after(gap)

+ })

+ if st := w.wd.record().Move.State; st != provision.OfflineSealed {

+ t.Fatalf("state %s after the seal's passes", st)

+ }

+ next("seal", "flip")

+ timed("flip", func() {

+ door("flip", "", provision.OfflineLanding)

+ l.after(gap)

+ })

+ if st := w.wd.record().Move.State; st != provision.OfflineLanded {

+ t.Fatalf("state %s after the flip's passes", st)

+ }

+ next("flip", "open")

+ timed("open", func() {

+ door("open", "", provision.OfflineOpen)

+ l.after(0)

+ })

+ next("open", "finish")

+ total := time.Since(began).Round(time.Millisecond)

+ t.Logf("operator gap %s: start to open in %s of test time (%s); %d serial passes, %d re-assert ticks; tries %v",

+ gap, total, strings.Join(took, ", "), l.passes, l.again, tries)

+ // The move is the real one: the target holds what the source had.

+ w.requireTarget()

}

func offlineMoveDone(t *testing.T, opts landOpts) {

diff --git a/internal/host/movesoffline_test.go b/internal/host/movesoffline_test.go

index b1b8da306..368d0e5f4 100644

--- a/internal/host/movesoffline_test.go

+++ b/internal/host/movesoffline_test.go

@@ -1281,3 +1281,80 @@ func TestOfflineKeep(t *testing.T) {

}

+// readCountingStore counts the reads made through it.

+type readCountingStore struct {

+ release.Store

+ mu sync.Mutex

+ reads int

+func (c *readCountingStore) Get(ctx context.Context, key string) ([]byte, string, error) {

+ c.mu.Lock()

+ c.reads++

+ c.mu.Unlock()

+ return c.Store.Get(ctx, key)

+func (c *readCountingStore) List(ctx context.Context, prefix string) ([]string, error) {

+ c.mu.Lock()

+ c.reads++

+ c.mu.Unlock()

+ return c.Store.List(ctx, prefix)

+// Harness Run 9, finding 7: between moves passes, a source box rewrites its

+// report on the re-assert tick while a door judges that report by age

+// (stopping to landed), and only then. It never runs the seal itself, it

+// reads nothing while no offline move is in flight, and a box that is

+// neither a source nor the sealer does nothing.

+func TestOfflineReassertRewritesOnlyTheReportsADoorReads(t *testing.T) {

+ wd := newOffWorld(t)

+ a := wd.box("ie-a", "", nil)

+ uk := wd.box("uk-box", "uk", nil)

+ a.tick()

+ uk.tick()

+ cs := &readCountingStore{Store: wd.s}

+ ctx := context.Background()

+ if n := a.w.offlineReassert(ctx, cs); n != 0 || cs.reads != 0 {

+ t.Fatalf("no move in flight: acted on %d, %d store reads", n, cs.reads)

+ }

+ wd.start()

+ a.tick() // the start door's bump

+ uk.tick()

+ wd.advance(3 * time.Minute)

+ for _, state := range []string{provision.OfflineStopping, provision.OfflineSealing} {

+ if state == provision.OfflineSealing {

+ wd.seal()

+ wd.advance(time.Minute)

+ }

+ if n := a.w.offlineReassert(ctx, cs); n != 1 {

+ t.Fatalf("%s: the source acted on %d records, want 1", state, n)

+ }

+ if r := wd.report(moves.OfflineRoleSource, "ie-a"); !r.At.Equal(wd.clock()) || !r.Held || !r.Fenced {

+ t.Fatalf("%s: the source's report %+v, want rewritten at %v", state, r, wd.clock())

+ }

+ if n := uk.w.offlineReassert(ctx, cs); n != 0 {

+ t.Fatalf("%s: the target acted on %d records", state, n)

+ }

+ if _, _, err := wd.s.Get(ctx, moves.OfflineSealKey(offHost, offID)); !errors.Is(err, os.ErrNotExist) {

+ t.Fatalf("the re-assert ran the seal (err %v)", err)

+ }

+ // From open on, no door reads a report by age: nothing is rewritten.

+ wd.sealed()

+ wd.flip()

+ wd.landed()

+ wd.open()

+ a.pass() // the view learns the state the way it always does

+ at := wd.report(moves.OfflineRoleSource, "ie-a").At

+ wd.advance(time.Minute)

+ if n := a.w.offlineReassert(ctx, cs); n != 0 {

+ t.Fatalf("open: the source acted on %d records", n)

+ }

+ if r := wd.report(moves.OfflineRoleSource, "ie-a"); !r.At.Equal(at) {

+ t.Fatalf("open: the source's report was rewritten at %v", r.At)

+ }

diff --git a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

index 842946e9b..4ff5acd9f 100644

--- a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

+++ b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

@@ -1,8 +1,10 @@

# Census overlay moves-offline-fixes (harness Run 9's defects 6 to 8 in the offline operator move, on main after 54ae7fb9), loaded LAST, after armed-box-updates. PROPOSED-moves-offline-fixes: every row is rev3=PROPOSED-moves-offline-fixes, and the ruling is asked of the census owner (platform-1c). The proposal, with each unit's executing identity: docs/superpowers/notes/census-proposal-moves-offline-fixes.md.

# Every unit here runs in the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos) and makes no root decision, so each is proposed accepted legacy at main's disposition (J1 root-route for registry and route reads, J4 root-input for file reads), as census-ruling-moves-offline proposed its units.

# Defect 6 (config.env for a nested row whose instance directory is not made yet): SyncExecWithCallbacks, renderExecConfigBefore and configGenLoop re-frozen at main's disposition; configGenTick and pendingDirs.appeared new. The seal moves.

+# Defect 7 (fresh reports at every door step): envelope only. The source and sealer re-assert (movesWorker.offlineReassert, offlineReassertLoop) reads no root input of its own: it reads the record from the deployment store and calls the steps census-ruling-moves-offline already classified. The console's stale-report refusal reads nothing new. internal/console and internal/host re-pinned; the seal stands.

@coverage internal/appconfig 2c32778cc777f026f1dd7b6404b11e3fabf1b6bd4912799d6ef26bfb59d42a07

-@coverage internal/host e0ca6a7c7d5d4f1d167e77c34c61795b1f35dcf60b8a2c8209340758b59a106a

+@coverage internal/console 7b015747c5ef8a3ee7473ad63f7f99200ce14c63c1b7a47945c212ad9239756b

+@coverage internal/host 78443272fea18130457dd94f7c970ee8b1e82f110d9d1bba46e48500aaf6d283

@legacy 783751049991bdd17a140bba6ce9a3049a4075b1170e4882c7207b20bed24b41 internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f

internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f

@legacy 356c576628a285021657e989a015ef938bb1da3498d8146a734903a23fea1e4a internal/appconfig/syncexec.go renderExecConfigBefore legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from config sync and the moves worker's held render: main's body with one Lstat of a nested row's instance directory after its write failed, to tell a directory not made yet from any other failure; it follows no link and changes no outcome but the error's kind file:Lstat@3893,file:ReadFile@951,registry:Environment@3190,registry:Host@1070,registry:Host@2586,registry:Host@4088,registry:Host@4320,registry:Host@4576,registry:Host@4669,registry:Host@876,whole:Route@2522,whole:Route@257,whole:Route@3715,whole:Route@680 a60a9a2e2158f3a8dadf6fcf46810930dad89b67d1acc23fb57c324a7d9ac1f1