moves offline: Astra round 1, pin the stager, clean by staged_at, no I/O under the round's lock
Three P2s from Astra's review of moves-offline-precopy. 1. An abort skipped cleanup when the round that copied never reported (a crash, or a failed report write). Cleanup now follows the record: any move carrying staged_at is cleaned by its stager on abort, whatever the reports say. The prefix deleted is the host's in the target's bucket, never the box's own. 2. A box that joined mid-move could become the stager, clean and report reverted while the previous stager's round was still copying. The stage door now pins the stager on the record (move.stager, additive); only that box ever stages or cleans for the move, its cleanup follows its own last round under its pass lock, and the console's finish of a staged abort waits for that box's stager report saying reverted. 3. A receipt write could make the receipts GET under the round's accounting mutex, so a stalled read kept another copy's failure from stopping the round. The book is now read before any copy starts whenever the round copies anything; set and drop touch memory only. Each has a test that failed before its fix. Copy: Luna's rewrites of MV-O182 and MV-O183 applied verbatim; MV-O179 to MV-O189 approved by Luna 2026-10-21; MV-O56 withdrawn. No new sentences.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 files changed,
+295
−114
approved_moves_copy_test.go+1 −1docs/superpowers/plans/2026-10-06-moves-phase1-copy.md+14 −15internal/console/apiinstancemove_offline.go+20 −2internal/console/apiinstancemove_offline_stage_test.go+19 −1internal/host/movesoffline_stage.go+28 −58internal/host/movesoffline_stage_test.go+78 −2internal/lease/mirror.go+14 −2internal/lease/mirror_parallel_test.go+44 −0internal/lease/mirror_test.go+9 −1internal/moves/offline.go+9 −7internal/provision/offline_move.go+15 −3internal/provision/offline_stage_test.go+27 −13internal/provision/provision.go+6 −0move.go+4 −4move_test.go+3 −3offline_move_e2e_test.go+4 −2
CI cancelled — superseded by a newer push — run details
diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go| index 7c1d62bb7..3eac4dc67 100644 |
| --- a/approved_moves_copy_test.go |
| +++ b/approved_moves_copy_test.go |
| @@ -96,7 +96,7 @@ var movesCopyIDs = []string{ |
| "MV-O20", "MV-O21", "MV-O22", "MV-O23", "MV-O24", "MV-O25", "MV-O26", "MV-O27", "MV-O28", "MV-O29", "MV-O30", "MV-O31", |
| "MV-O32", "MV-O33", |
| "MV-O34", "MV-O35", "MV-O36", "MV-O37", "MV-O38", "MV-O39", "MV-O40", "MV-O41", "MV-O42", "MV-O43", |
| - "MV-O50", "MV-O51", "MV-O52", "MV-O53", "MV-O54", "MV-O55", "MV-O56", "MV-O57", "MV-O58", "MV-O59", "MV-O60", "MV-O61", "MV-O62", "MV-O63", "MV-O64", "MV-O65", |
| + "MV-O50", "MV-O51", "MV-O52", "MV-O53", "MV-O54", "MV-O55", "MV-O57", "MV-O58", "MV-O59", "MV-O60", "MV-O61", "MV-O62", "MV-O63", "MV-O64", "MV-O65", |
| "MV-O66", "MV-O67", "MV-O68", "MV-O69", "MV-O70", "MV-O71", "MV-O72", "MV-O73", "MV-O74", "MV-O75", "MV-O76", "MV-O77", "MV-O78", "MV-O79", "MV-O80", "MV-O81", |
| "MV-O82", "MV-O83", "MV-O84", "MV-O85", "MV-O86", "MV-O87", "MV-O88", "MV-O89", "MV-O90", "MV-O91", "MV-O92", "MV-O93", "MV-O94", "MV-O95", "MV-O96", "MV-O97", |
| "MV-O98", |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md| index fe73bd1c7..82873e003 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| @@ -543,7 +543,7 @@ Two more refusals of the console doors (Task 6), for cases MV-O20 to MV-O31 do n |
| | MV-O50 | internal/console/apiinstancemove_offline.go | <host> is steered or runs under a unit. offline moves cannot carry either yet | approved | |
| | MV-O51 | internal/console/apiinstancemove_offline.go | <box> already serves <host>. an offline move needs the region it moves to to be empty of it | approved | |
| -Seven more lines of `carlos move offline` (`move.go`, Task 7), for places MV-O1 to MV-O10 do not reach: what `abort` says once every box has put the instance back (the plan's runbook has the operator run `finish` next, and no row said so), the lines `status` prints for each box and for each manifest item (`<status>` is ok, waiting, failed or unchecked; `<detail>` and `<phase>` are box diagnostics, not copy; an item with no detail uses the shorter line), the usage line, and the line `status` prints for a path a source box could not park and left where it was (MV-O58; `<reason>` is the box's own reason, not copy). A box whose report is not current says so: `flip` and `open` refuse on a stale report, so `status` has to show it. MV-O56 follows MV-O57 in the table only by number. |
| +Seven more lines of `carlos move offline` (`move.go`, Task 7), for places MV-O1 to MV-O10 do not reach: what `abort` says once every box has put the instance back (the plan's runbook has the operator run `finish` next, and no row said so), the lines `status` prints for each box and for each manifest item (`<status>` is ok, waiting, failed or unchecked; `<detail>` and `<phase>` are box diagnostics, not copy; an item with no detail uses the shorter line), the usage line, and the line `status` prints for a path a source box could not park and left where it was (MV-O58; `<reason>` is the box's own reason, not copy). A box whose report is not current says so: `flip` and `open` refuse on a stale report, so `status` has to show it. MV-O56 was the usage line before `stage`; MV-O181 replaced it, and Luna approved its withdrawal on 2026-10-21. |
| | id | file | text | status | |
| |---|---|---|---| |
| @@ -552,7 +552,6 @@ Seven more lines of `carlos move offline` (`move.go`, Task 7), for places MV-O1 |
| | MV-O54 | move.go | <box> is <phase> as the <role>. Its report is not current. | approved | |
| | MV-O55 | move.go | Item <id> is <status>: <detail>. | approved | |
| | MV-O57 | move.go | Item <id> is <status>. | approved | |
| -| MV-O56 | move.go | carlos move offline needs start, seal, flip, open, finish, abort or status. | pending | |
| | MV-O58 | move.go | <box> left <path> in place: <reason>. | approved | |
| | MV-O65 | move.go | <box> could not clean up <host>: <reason>. Run carlos move offline status --host <host> for each item. | approved | |
| @@ -691,22 +690,22 @@ A door's answer to a box report it cannot believe for its age (harness Run 9, fi |
| ## Offline move: stage (pre-copy) |
| -`carlos move offline stage` copies the instance's replica to the region it moves to while it keeps serving, so the seal has less to copy with the instance offline (branch `moves-offline-precopy`). Pending Luna's review: Paul has handed the review of these rows to Luna, so each is `pending` until Luna approves it. |
| +`carlos move offline stage` copies the instance's replica to the region it moves to while it keeps serving, so the seal has less to copy with the instance offline (branch `moves-offline-precopy`). Paul handed the review of these rows to Luna. Luna approved MV-O179 to MV-O189 on 2026-10-21, with two rewrites applied as given (MV-O182 and MV-O183). |
| MV-O179 and MV-O180 are console door refusals (`internal/console/apiinstancemove_offline.go`), lower case like the other door sentences. MV-O179 answers a `stage` or `start` that names another region than the one the instance is staged for (`<to>` is that region's name). MV-O180 is the stage door's own box rule: every box in both regions must publish `offline-stage-1`. |
| -MV-O181 is the usage line with `stage` added. It replaces MV-O56, which is set back to `pending` here because the code no longer says it: MV-O56 is withdrawn once Luna approves MV-O181. MV-O182 to MV-O188 are `carlos move offline` output (`move.go`): what `stage` says when it posts and when its round has run, what `status` says while the move is staging, and what `abort` says for a move that was staged and never started (nothing is moved back, only the staged copy is removed). In MV-O183, `<staged>` and `<total>` are object counts, `<staged_mb>` and `<total_mb>` are megabytes with one decimal, and `<time>` is the round's end as `2006-01-02 15:04 UTC`. In MV-O185, `<reason>` is the stager box's own diagnostic, not copy. MV-O189 is the CLI page's paragraph for `stage`. |
| +MV-O181 is the usage line with `stage` added. It replaces MV-O56, whose withdrawal Luna approved with it. MV-O182 to MV-O188 are `carlos move offline` output (`move.go`): what `stage` says when it posts and when its round has run, what `status` says while the move is staging, and what `abort` says for a move that was staged and never started (nothing is moved back, only the staged copy is removed). In MV-O183, `<staged>` and `<total>` are object counts, `<staged_mb>` and `<total_mb>` are megabytes with one decimal, and `<time>` is the round's end as `2006-01-02 15:04 UTC`. In MV-O185, `<reason>` is the stager box's own diagnostic, not copy. MV-O189 is the CLI page's paragraph for `stage`. |
| | id | file | text | status | |
| |---|---|---|---| |
| -| MV-O179 | internal/console/apiinstancemove_offline.go | <host> is staged for <to>. abort the move to move it to another region | pending | |
| -| MV-O180 | internal/console/apiinstancemove_offline.go | <box> cannot stage an offline move. it runs an older release | pending | |
| -| MV-O181 | move.go | carlos move offline needs stage, start, seal, flip, open, finish, abort or status. | pending | |
| -| MV-O182 | move.go | Staging <host> in <to>. <host> keeps serving while it copies. | pending | |
| -| MV-O183 | move.go | Staged <staged> of <total> objects (<staged_mb> of <total_mb> MB) of <host> in <to> at <time>. | pending | |
| -| MV-O184 | move.go | <host> keeps serving. Run carlos move offline start --host <host> --to <to> when you are ready to take it offline. | pending | |
| -| MV-O185 | move.go | The last staging round failed: <reason>. Run carlos move offline stage --host <host> --to <to> to try again. | pending | |
| -| MV-O186 | move.go | Waiting for the first staging round of <host>. | pending | |
| -| MV-O187 | move.go | Removing the staged copy of <host> from <to>. | pending | |
| -| MV-O188 | move.go | The staged copy of <host> is removed. Run carlos move offline finish --host <host> to end the move. | pending | |
| -| MV-O189 | docs/site/cli.md | `stage` is optional and comes before `start`. It copies the instance's data to the region it moves to while the instance keeps serving, and each time it runs it copies only what is new. The more it has copied, the less is left to copy while the instance is offline. Until `start`, `abort` removes the staged copy and the instance is never offline. | pending | |
| +| MV-O179 | internal/console/apiinstancemove_offline.go | <host> is staged for <to>. abort the move to move it to another region | approved | |
| +| MV-O180 | internal/console/apiinstancemove_offline.go | <box> cannot stage an offline move. it runs an older release | approved | |
| +| MV-O181 | move.go | carlos move offline needs stage, start, seal, flip, open, finish, abort or status. | approved | |
| +| MV-O182 | move.go | Staging <host> in <to>. <host> keeps serving while the data is copied. | approved | |
| +| MV-O183 | move.go | Staged <staged> of <total> objects for <host> in <to> (<staged_mb> of <total_mb> MB) at <time>. | approved | |
| +| MV-O184 | move.go | <host> keeps serving. Run carlos move offline start --host <host> --to <to> when you are ready to take it offline. | approved | |
| +| MV-O185 | move.go | The last staging round failed: <reason>. Run carlos move offline stage --host <host> --to <to> to try again. | approved | |
| +| MV-O186 | move.go | Waiting for the first staging round of <host>. | approved | |
| +| MV-O187 | move.go | Removing the staged copy of <host> from <to>. | approved | |
| +| MV-O188 | move.go | The staged copy of <host> is removed. Run carlos move offline finish --host <host> to end the move. | approved | |
| +| MV-O189 | docs/site/cli.md | `stage` is optional and comes before `start`. It copies the instance's data to the region it moves to while the instance keeps serving, and each time it runs it copies only what is new. The more it has copied, the less is left to copy while the instance is offline. Until `start`, `abort` removes the staged copy and the instance is never offline. | approved | |
diff --git a/internal/console/apiinstancemove_offline.go b/internal/console/apiinstancemove_offline.go| index e74a26c8d..c8fe845a5 100644 |
| --- a/internal/console/apiinstancemove_offline.go |
| +++ b/internal/console/apiinstancemove_offline.go |
| @@ -620,8 +620,17 @@ func (s *Server) handleAPIInstanceOfflineStage(w http.ResponseWriter, r *http.Re |
| return |
| } |
| } |
| + // The stager is pinned here, for the whole move: the one box that ever |
| + // writes this move's staging prefix, and the one whose cleanup an abort |
| + // waits for (Astra round 1, finding 2). offlineSelect has already |
| + // required at least one source box. |
| + var names []string |
| + for _, b := range fleet.boxesAt(d.rec.Region) { |
| + names = append(names, b.Box) |
| + } |
| + stager := moves.OfflineStager(names) |
| s.offlineBegin(w, r, d, func(id string) func(*provision.Record) error { |
| - return provision.OfflineStage(id, d.rec.Region, to, d.addr, time.Now()) |
| + return provision.OfflineStage(id, d.rec.Region, to, d.addr, stager, time.Now()) |
| }) |
| } |
| @@ -899,6 +908,14 @@ func (s *Server) offlineStepFor(ctx context.Context, d offlineDoor, mv *provisio |
| if ref := offlineFinishCheck(ev, mv, host, "", sealerPhase, moves.OfflinePhaseReverted, boxes); ref != nil { |
| return offlineStep{}, ref |
| } |
| + // A staged move also waits for its pinned stager to say the staged |
| + // copy is gone, whatever its last round reported (Astra round 1, |
| + // findings 1 and 2). |
| + if mv.StagedAt != nil { |
| + if rep, ok := ev.get(moves.OfflineRoleStager, mv.Stager); !ok || rep.Phase != moves.OfflinePhaseReverted { |
| + return offlineStep{}, offlineStateRefusal(mv.State, host) |
| + } |
| + } |
| return offlineStep{mut: provision.OfflineFinish(mv.ID, now)}, nil |
| } |
| return offlineStep{}, offlineStateRefusal(mv.State, host) |
| @@ -1166,6 +1183,7 @@ type offlineView struct { |
| From string `json:"from,omitempty"` |
| To string `json:"to,omitempty"` |
| Sealer string `json:"sealer,omitempty"` |
| + Stager string `json:"stager,omitempty"` |
| Boxes []string `json:"boxes,omitempty"` |
| RequestedBy string `json:"requested_by,omitempty"` |
| RequestedAt int64 `json:"requested_at,omitempty"` |
| @@ -1243,7 +1261,7 @@ func (s *Server) offlineViewOf(ctx context.Context, rec *provision.Record) offli |
| now := time.Now() |
| v.State, v.ID = mv.State, mv.ID |
| v.From, v.To = s.offlineName(mv.From), s.offlineName(mv.To) |
| - v.Sealer, v.Boxes = mv.Sealer, mv.Boxes |
| + v.Sealer, v.Boxes, v.Stager = mv.Sealer, mv.Boxes, mv.Stager |
| v.NeverOffline = !mv.StartedOffline() |
| if mv.StagedAt != nil { |
| v.StagedAt = mv.StagedAt.UnixNano() |
diff --git a/internal/console/apiinstancemove_offline_stage_test.go b/internal/console/apiinstancemove_offline_stage_test.go| index debcaba96..d21d5c6e4 100644 |
| --- a/internal/console/apiinstancemove_offline_stage_test.go |
| +++ b/internal/console/apiinstancemove_offline_stage_test.go |
| @@ -53,7 +53,7 @@ func TestOfflineStageThenStartThenSeal(t *testing.T) { |
| w := newOfflineWorld(t, ieOff("ie-box"), ukOff("uk-box"), us) |
| v := w.stage() |
| rec := w.record() |
| - if rec.Region != "ie" || rec.Move.State != provision.OfflineStaging || rec.Move.To != "uk" || rec.Move.StagedAt == nil { |
| + if rec.Region != "ie" || rec.Move.State != provision.OfflineStaging || rec.Move.To != "uk" || rec.Move.StagedAt == nil || rec.Move.Stager != "ie-box" { |
| t.Fatalf("staged record: %q %+v", rec.Region, rec.Move) |
| } |
| if !v.CanAbort || v.Next != offlineVerbStart || v.Stage != nil { |
| @@ -166,6 +166,13 @@ func TestOfflineAbortFromStagingDoor(t *testing.T) { |
| } |
| w.refused(http.MethodPost, "/finish", "", http.StatusConflict, msgO25(provision.OfflineReverting)) |
| w.report(moves.OfflineRoleSource, "ie-box", moves.OfflinePhaseReverted, nil) |
| + // Every source box has reverted, but the pinned stager has not said |
| + // the staged copy is gone (Astra round 1, findings 1 and 2): no finish, |
| + // whatever its last round reported, or another box's stager report. |
| + w.refused(http.MethodPost, "/finish", "", http.StatusConflict, msgO25(provision.OfflineReverting)) |
| + w.report(moves.OfflineRoleStager, "ie-other", moves.OfflinePhaseReverted, nil) |
| + w.refused(http.MethodPost, "/finish", "", http.StatusConflict, msgO25(provision.OfflineReverting)) |
| + w.report(moves.OfflineRoleStager, "ie-box", moves.OfflinePhaseReverted, nil) |
| v := w.ok("/finish", "", "none") |
| if v.Last == nil || v.Last.Outcome != provision.OfflineOutcomeAborted || v.Last.OfflineS != 0 { |
| t.Fatalf("last = %+v", v.Last) |
| @@ -189,3 +196,14 @@ func TestOfflineStageViewShowsAFailedRound(t *testing.T) { |
| t.Fatalf("stage view %+v", st.Stage) |
| } |
| } |
| + |
| +// The stage door pins the stager: the lexically first eligible source box, |
| +// whatever order the fleet lists them in. |
| +func TestOfflineStagePinsTheStager(t *testing.T) { |
| + w := newOfflineWorld(t, ieOff("ie-c"), ieOff("ie-a"), ukOff("uk-box")) |
| + v := w.stage() |
| + if rec := w.record(); rec.Move.Stager != "ie-a" || v.Stager != "ie-a" { |
| + t.Fatalf("stager %q (view %q), want ie-a", rec.Move.Stager, v.Stager) |
| + } |
| +} |
| + |
diff --git a/internal/host/movesoffline_stage.go b/internal/host/movesoffline_stage.go| index cd2089e14..ef934d921 100644 |
| --- a/internal/host/movesoffline_stage.go |
| +++ b/internal/host/movesoffline_stage.go |
| @@ -13,7 +13,6 @@ import ( |
| "github.com/carlosframework/platform/internal/activator" |
| "github.com/carlosframework/platform/internal/lease" |
| "github.com/carlosframework/platform/internal/moves" |
| - "github.com/carlosframework/platform/internal/park" |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/release" |
| ) |
| @@ -56,40 +55,19 @@ import ( |
| // box that does not publish it. |
| const capOfflineStage1 = "offline-stage-1" |
| -// offlineStagerOf is moves.OfflineStager over the boxes of the record's |
| -// source placement that the console's doors count (offlineCanHold and the |
| -// capability), read from the edges files now: the console's status view |
| -// names the same box. |
| -func offlineStagerOf(ctx context.Context, s release.Store, home, from string) (string, error) { |
| - bodies, err := loadEdgesFileBodies(ctx, s) |
| - if err != nil { |
| - return "", err |
| - } |
| - var boxes []string |
| - for _, body := range bodies { |
| - var t edgesFile |
| - if err := json.Unmarshal(body, &t); err != nil || t.V < 1 || t.Box == "" { |
| - continue |
| - } |
| - if t.Placement != from && !park.PlacementMatches(home, from, park.PlacementRegion(t.Placement)) { |
| - continue |
| - } |
| - if !t.HibernationConfigured || !t.ServesSharedPool || !containsString(t.Capabilities, capOfflineStage1) { |
| - continue |
| - } |
| - boxes = append(boxes, t.Box) |
| - } |
| - return moves.OfflineStager(boxes), nil |
| -} |
| - |
| // offlineStageStep is the stager's step in offline-staging: one round, then |
| -// the stager report. Any other source box does nothing. |
| +// the stager report. The stager is the box the stage door pinned on the |
| +// record (MoveState.Stager); no other box ever writes this move's staging |
| +// prefix, whatever the fleet looks like now (Astra round 1, finding 2). |
| func (w *movesWorker) offlineStageStep(ctx context.Context, s release.Store, rec provision.Record) error { |
| cfg := w.cfg |
| m, host := rec.Move, rec.Host |
| if cfg.Box == "" { |
| return errors.New("offline move: this box has no label to report under") |
| } |
| + if m.Stager == "" || m.Stager != cfg.Box { |
| + return nil |
| + } |
| var request time.Time |
| if m.StagedAt != nil { |
| request = m.StagedAt.UTC() |
| @@ -101,13 +79,6 @@ func (w *movesWorker) offlineStageStep(ctx context.Context, s release.Store, rec |
| if prev != nil && prev.StageRequest.Equal(request) && (prev.Phase == moves.OfflinePhaseStaged || prev.Phase == moves.OfflinePhaseFailed) { |
| return nil // this request has had its round |
| } |
| - stager, err := offlineStagerOf(ctx, s, cfg.ParkHomeRegion, m.From) |
| - if err != nil { |
| - return fmt.Errorf("offline move %s: read the fleet's edges files: %w", m.ID, err) |
| - } |
| - if stager != cfg.Box { |
| - return nil |
| - } |
| r := moves.OfflineReport{Op: m.ID, Host: host, Box: cfg.Box, Role: moves.OfflineRoleStager, Phase: moves.OfflinePhaseStaged, StageRequest: request} |
| p, err := w.offlineStageRound(ctx, s, rec) |
| r.At = cfg.now().UTC() |
| @@ -215,33 +186,30 @@ func (w *movesWorker) ownStagerReport(ctx context.Context, s release.Store, rec |
| return &r, nil |
| } |
| -// offlineUnstage is the stager's part of an abort: once any box has staged |
| -// for this move, the stager the fleet names now deletes the target's |
| -// staging prefix and the copy receipts, and says so in its stager report |
| -// (phase reverted). The source step calls it before it reports reverted, |
| -// so the finish that ends the abort waits for it. Every part is an ensure, |
| -// and a move that was never staged does nothing at all, so an abort from |
| -// stopping is exactly what it was. The staging prefix is this host's; no |
| -// other move of the host can be in flight while this record carries this |
| -// one. |
| +// offlineUnstage is the stager's part of an abort of a staged move. A move |
| +// that carries staged_at may have staged copies whatever its reports say: a |
| +// round can land copies and die before its report (Astra round 1, finding |
| +// 1). So the pinned stager, and only it, deletes the target's staging |
| +// prefix and the copy receipts and says so in its stager report (phase |
| +// reverted), once, before its source step reports reverted. Its own last |
| +// round has ended by then: rounds run only in staging, and both run under |
| +// this box's pass lock. The console's finish of a staged move waits for |
| +// that report (finding 2). A move that was never staged does nothing here, |
| +// so an abort from stopping is what it was. The prefix deleted is this |
| +// host's in the target's bucket, never this box's own bucket, where a |
| +// moves/<host>/ prefix is not this move's (the stage round refuses to write |
| +// there for the same reason). |
| func (w *movesWorker) offlineUnstage(ctx context.Context, s release.Store, rec provision.Record) error { |
| cfg := w.cfg |
| m, host := rec.Move, rec.Host |
| - stagers, err := readOfflineReports(ctx, s, host, m.ID, moves.OfflineRoleStager) |
| - if err != nil { |
| - return fmt.Errorf("offline move %s: read the stager reports: %w", m.ID, err) |
| - } |
| - if len(stagers) == 0 { |
| + if m.StagedAt == nil || m.Stager == "" || m.Stager != cfg.Box { |
| return nil |
| } |
| - stager, err := offlineStagerOf(ctx, s, cfg.ParkHomeRegion, m.From) |
| + own, err := w.ownStagerReport(ctx, s, rec) |
| if err != nil { |
| - return fmt.Errorf("offline move %s: read the fleet's edges files: %w", m.ID, err) |
| - } |
| - if stager != cfg.Box { |
| - return nil |
| + return fmt.Errorf("offline move %s: read the stager report: %w", m.ID, err) |
| } |
| - if own, ok := stagers[cfg.Box]; ok && own.Phase == moves.OfflinePhaseReverted { |
| + if own != nil && own.Phase == moves.OfflinePhaseReverted { |
| return nil |
| } |
| tgt, err := w.offlineTarget(ctx, s, m) |
| @@ -256,11 +224,13 @@ func (w *movesWorker) offlineUnstage(ctx context.Context, s release.Store, rec p |
| if prefix == "moves/" || !strings.HasPrefix(prefix, "moves/") { |
| return fmt.Errorf("offline move %s: no staging prefix for %s", m.ID, host) |
| } |
| - if err := lease.DeletePrefix(ctx, mirror, tgt.DataBucket, prefix); err != nil { |
| - return fmt.Errorf("offline move %s: delete %s/%s: %w", m.ID, tgt.DataBucket, prefix, err) |
| + if tgt.DataBucket != cfg.OwnBucket { |
| + if err := lease.DeletePrefix(ctx, mirror, tgt.DataBucket, prefix); err != nil { |
| + return fmt.Errorf("offline move %s: delete %s/%s: %w", m.ID, tgt.DataBucket, prefix, err) |
| + } |
| + log.Printf("moves: %s: offline move %s: the staged copy in %s/%s is deleted", host, m.ID, tgt.DataBucket, prefix) |
| } |
| clearOfflineReceipts(ctx, mirror, cfg.OwnBucket, host) |
| - log.Printf("moves: %s: offline move %s: the staged copy in %s/%s is deleted", host, m.ID, tgt.DataBucket, prefix) |
| r := moves.OfflineReport{Op: m.ID, Host: host, Box: cfg.Box, Role: moves.OfflineRoleStager, Phase: moves.OfflinePhaseReverted, At: cfg.now().UTC()} |
| return writeOfflineReport(ctx, s, r) |
| } |
diff --git a/internal/host/movesoffline_stage_test.go b/internal/host/movesoffline_stage_test.go| index 1225c2db9..559cf448b 100644 |
| --- a/internal/host/movesoffline_stage_test.go |
| +++ b/internal/host/movesoffline_stage_test.go |
| @@ -50,8 +50,11 @@ func (sw *sealWorld) mustStagePass(b *offBox) { |
| } |
| } |
| -func (sw *sealWorld) stage() { |
| - sw.move(provision.OfflineStage(offID, "", "uk", "op@example.com", sw.clock())) |
| +func (sw *sealWorld) stage() { sw.stageBy("ie-a") } |
| + |
| +// stageBy stages with the stager the door pinned. |
| +func (sw *sealWorld) stageBy(stager string) { |
| + sw.move(provision.OfflineStage(offID, "", "uk", "op@example.com", stager, sw.clock())) |
| } |
| // restage is a second carlos move offline stage: one more round. |
| @@ -339,3 +342,76 @@ func TestOfflineStageRefusesTheSourcesOwnBucket(t *testing.T) { |
| t.Fatalf("stager report %+v", r) |
| } |
| } |
| + |
| +// Astra round 1, finding 1: a round that copied and then never wrote its |
| +// report (the box crashed, or the report write failed) still leaves its |
| +// copy to the abort. Cleanup follows the record's staged_at, never the |
| +// reports. |
| +func TestOfflineAbortCleansARoundThatNeverReported(t *testing.T) { |
| + sw := newSealWorld(t) |
| + ie := sw.ie |
| + sw.instance(ie) |
| + sw.stage() |
| + sw.stagerEdges(ie) |
| + // The round's copies land; the box dies before its report. |
| + if _, err := ie.w.offlineStageRound(context.Background(), sw.s, sw.record()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(sw.stagedRel()) == 0 { |
| + t.Fatal("fixture: nothing staged") |
| + } |
| + if _, ok := sw.reportIf(moves.OfflineRoleStager, "ie-a"); ok { |
| + t.Fatal("fixture: the round reported") |
| + } |
| + sw.abort() |
| + sw.mustStagePass(ie) |
| + if n := len(sw.stagedRel()); n != 0 { |
| + t.Fatalf("%d staged objects left by a round that never reported", n) |
| + } |
| + if r := sw.report(moves.OfflineRoleStager, "ie-a"); r.Phase != moves.OfflinePhaseReverted { |
| + t.Fatalf("stager report %+v", r) |
| + } |
| +} |
| + |
| +// Astra round 1, finding 2: a box that joins mid-move, lexically before the |
| +// stager, never stages and never cleans for it. Only the pinned stager |
| +// writes the staging prefix, so its cleanup, after its own last round, is |
| +// the one that counts. |
| +func TestOfflineAbortWaitsForThePinnedStager(t *testing.T) { |
| + sw := newSealWorld(t) |
| + sw.instance(sw.ie) |
| + ieB := sw.sourceBox("ie-b") |
| + sw.stageBy("ie-b") |
| + sw.mustStagePass(ieB) |
| + if r := sw.report(moves.OfflineRoleStager, "ie-b"); r.Phase != moves.OfflinePhaseStaged { |
| + t.Fatalf("stager report %+v", r) |
| + } |
| + // ie-a joins, publishing the capability, and the operator stages again. |
| + sw.stagerEdges(sw.ie) |
| + sw.restage() |
| + spy := sw.spyCopies() |
| + sw.mustStagePass(sw.ie) |
| + if _, ok := sw.reportIf(moves.OfflineRoleStager, "ie-a"); ok || len(spy.take()) != 0 { |
| + t.Fatal("a box that is not the pinned stager staged") |
| + } |
| + sw.abort() |
| + sw.mustStagePass(sw.ie) // ie-a's revert: not its copy to remove |
| + if _, ok := sw.reportIf(moves.OfflineRoleStager, "ie-a"); ok { |
| + t.Fatal("a box that is not the pinned stager acknowledged the cleanup") |
| + } |
| + // ie-b's round, which began before the abort, lands its copies now. |
| + sw.fake.Put(sealIE, sealPrefix+"/"+offHost+"/scratch/late.bin", []byte("late")) |
| + if _, err := ieB.w.offlineStageRound(context.Background(), sw.s, sw.record()); err != nil { |
| + t.Fatal(err) |
| + } |
| + if len(sw.stagedRel()) == 0 { |
| + t.Fatal("fixture: the late round staged nothing") |
| + } |
| + sw.mustStagePass(ieB) // the pinned stager's revert, after its last round |
| + if n := len(sw.stagedRel()); n != 0 { |
| + t.Fatalf("%d staged objects left after the pinned stager's cleanup", n) |
| + } |
| + if r := sw.report(moves.OfflineRoleStager, "ie-b"); r.Phase != moves.OfflinePhaseReverted { |
| + t.Fatalf("stager report %+v", r) |
| + } |
| +} |
diff --git a/internal/lease/mirror.go b/internal/lease/mirror.go| index 451bea05f..a1aa65870 100644 |
| --- a/internal/lease/mirror.go |
| +++ b/internal/lease/mirror.go |
| @@ -269,6 +269,18 @@ func MirrorPrefixKeep(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, |
| // neither depends on that order. The first failure stops the round: no |
| // further copy starts, the ones in flight are cancelled, and every copy |
| // that did land keeps its receipt (the best-effort save below). |
| + // |
| + // No network request is ever made under mu (Astra round 1, finding 3): |
| + // a receipts read that stalled there would hold every other worker, |
| + // the first failure's included, so the round could not stop. The book |
| + // is read here, before any copy, whenever this round copies anything |
| + // and keeps receipts; set and drop then touch memory only. (A round |
| + // that copies nothing still reads no receipts.) |
| + if len(jobs) > 0 { |
| + if err := book.load(ctx); err != nil { |
| + return round, err |
| + } |
| + } |
| var mu sync.Mutex |
| err = forEachParallel(ctx, mirrorWorkers(m), len(jobs), func(cctx context.Context, i int) error { |
| r := jobs[i] |
| @@ -306,8 +318,8 @@ func MirrorPrefixKeep(ctx context.Context, m MirrorStore, srcBucket, srcPrefix, |
| if dstETag == "" { |
| dstETag = s.ETag |
| } |
| - // A failed read here loses this receipt (one recopy next |
| - // round), never a wrong skip. |
| + // The book was read before the copies began, so this is a |
| + // write to memory, never a request under mu. |
| return book.set(ctx, r, copyReceipt{SrcETag: s.ETag, SrcSize: s.Size, DstETag: dstETag, DstVersionID: ver}) |
| } |
| book.drop(r) |
diff --git a/internal/lease/mirror_parallel_test.go b/internal/lease/mirror_parallel_test.go| index 5cd834ec7..27bec5312 100644 |
| --- a/internal/lease/mirror_parallel_test.go |
| +++ b/internal/lease/mirror_parallel_test.go |
| @@ -381,3 +381,47 @@ func TestForEachParallel(t *testing.T) { |
| t.Fatal(err) |
| } |
| } |
| + |
| +// Astra round 1, finding 3: no network request is made while the round's |
| +// accounting is locked. A receipts read that stalls once a copy has landed |
| +// (the lazy load a receipt write used to make under the lock) must not keep |
| +// another copy's failure from stopping the round: the book is read before |
| +// any copy starts. |
| +func TestMirrorPrefixAReceiptsReadNeverBlocksTheFirstFailure(t *testing.T) { |
| + f := newFakeMirror() |
| + f.workers = 2 |
| + f.putMultipart("src", "instances/h/ltx/1/a.ltx", "a", 100) // lands, needs a receipt |
| + f.putSized("src", "instances/h/ltx/0/b.ltx", "b", 10) // fails |
| + var landed atomic.Bool |
| + release := make(chan struct{}) |
| + t.Cleanup(func() { close(release) }) |
| + f.onCopy = func(_ context.Context, key string) error { |
| + if strings.HasSuffix(key, "b.ltx") { |
| + // Fail only once a's copy has landed and its receipt is due. |
| + for deadline := time.Now().Add(5 * time.Second); !landed.Load() && time.Now().Before(deadline); time.Sleep(time.Millisecond) { |
| + } |
| + time.Sleep(20 * time.Millisecond) |
| + return errors.New("injected copy failure") |
| + } |
| + landed.Store(true) |
| + return nil |
| + } |
| + f.onGet = func(string) { |
| + if landed.Load() { |
| + <-release // a receipts read that stalls after a copy |
| + } |
| + } |
| + done := make(chan error, 1) |
| + go func() { |
| + _, err := MirrorPrefix(context.Background(), f, "src", "instances/h/", "dst", "moves/h/", testReceipts) |
| + done <- err |
| + }() |
| + select { |
| + case err := <-done: |
| + if err == nil || !strings.Contains(err.Error(), "injected copy failure") { |
| + t.Fatalf("round error = %v, want the injected failure", err) |
| + } |
| + case <-time.After(3 * time.Second): |
| + t.Fatal("a receipts read made under the round's lock kept the first failure from stopping the round") |
| + } |
| +} |
diff --git a/internal/lease/mirror_test.go b/internal/lease/mirror_test.go| index 169449135..d1afab49e 100644 |
| --- a/internal/lease/mirror_test.go |
| +++ b/internal/lease/mirror_test.go |
| @@ -31,7 +31,9 @@ type fakeMirror struct { |
| beforeCopy func(srcKey string) |
| // onCopy runs after the precondition check and before the object |
| // lands; ctx is the copy's own. An error it returns is the copy's. |
| - onCopy func(ctx context.Context, srcKey string) error |
| + onCopy func(ctx context.Context, srcKey string) error |
| + // onGet runs at the top of Get, without the lock. |
| + onGet func(key string) |
| failPut bool |
| failGet bool |
| puts int |
| @@ -157,6 +159,12 @@ func (f *fakeMirror) Delete(_ context.Context, bucket, key string) error { |
| } |
| func (f *fakeMirror) Get(_ context.Context, bucket, key string) ([]byte, error) { |
| + f.mu.Lock() |
| + on := f.onGet |
| + f.mu.Unlock() |
| + if on != nil { |
| + on(key) |
| + } |
| f.mu.Lock() |
| defer f.mu.Unlock() |
| f.gets++ |
diff --git a/internal/moves/offline.go b/internal/moves/offline.go| index 7fa926649..670a9d1c6 100644 |
| --- a/internal/moves/offline.go |
| +++ b/internal/moves/offline.go |
| @@ -120,7 +120,8 @@ func OfflineSourced() []OfflineItem { |
| // The roles a report names: a source-region box that stopped the host, the |
| // one source box that seals, the target box that lands, and the one source |
| // box that pre-copies the replica while the move is offline-staging (the |
| -// stager: the lexically first box of the source region, OfflineStager). |
| +// stager, which the stage door picks with OfflineStager and pins on the |
| +// record as move.stager). |
| const ( |
| OfflineRoleSource = "source" |
| OfflineRoleSealer = "sealer" |
| @@ -399,12 +400,13 @@ func ValidateOfflineSeal(r OfflineReport, host, op string) error { |
| return nil |
| } |
| -// OfflineStager is the box that pre-copies the replica while a move is |
| -// offline-staging: the lexically first of boxes, the source region's boxes |
| -// as the fleet's edges files list them, "" for none. Every source box and |
| -// the console compute it from the same listing, so one box copies; two |
| -// boxes that briefly disagree (a box joined between their listings) only |
| -// copy twice, which the receipts make harmless (lease.Receipts). |
| +// OfflineStager is the stage door's choice of the box that pre-copies the |
| +// replica while a move is offline-staging: the lexically first of boxes, |
| +// the source region's eligible boxes as the fleet's edges files list them, |
| +// "" for none. The door pins it on the record (MoveState.Stager) and it |
| +// never changes for the move: a box that joins later never stages or |
| +// cleans for it, so exactly one box ever writes the staging prefix (Astra |
| +// round 1, finding 2). |
| func OfflineStager(boxes []string) string { |
| stager := "" |
| for _, b := range boxes { |
diff --git a/internal/provision/offline_move.go b/internal/provision/offline_move.go| index 5ad990a5d..1db2bcf66 100644 |
| --- a/internal/provision/offline_move.go |
| +++ b/internal/provision/offline_move.go |
| @@ -130,9 +130,21 @@ func OfflineStart(id, from, to, by string, now time.Time) func(*Record) error { |
| } |
| // OfflineStage is OfflineStart into offline-staging: the same refusals and |
| -// the same stamps, plus staged_at. The instance keeps serving. |
| -func OfflineStage(id, from, to, by string, now time.Time) func(*Record) error { |
| - return offlineBegin(id, from, to, by, now, OfflineStaging) |
| +// the same stamps, plus staged_at and the stager, the one box that copies |
| +// for this move (a stage that names none is malformed). The instance keeps |
| +// serving. |
| +func OfflineStage(id, from, to, by, stager string, now time.Time) func(*Record) error { |
| + begin := offlineBegin(id, from, to, by, now, OfflineStaging) |
| + return func(rec *Record) error { |
| + if stager == "" { |
| + return fmt.Errorf("%w: %s", ErrMoveConflict, refusedMalformedStart) |
| + } |
| + if err := begin(rec); err != nil { |
| + return err |
| + } |
| + rec.Move.Stager = stager |
| + return nil |
| + } |
| } |
| func offlineBegin(id, from, to, by string, now time.Time, state string) func(*Record) error { |
diff --git a/internal/provision/offline_stage_test.go b/internal/provision/offline_stage_test.go| index eddd28036..c01ae23ed 100644 |
| --- a/internal/provision/offline_stage_test.go |
| +++ b/internal/provision/offline_stage_test.go |
| @@ -21,20 +21,20 @@ func TestOfflineStageThenStartThenTheWholeMove(t *testing.T) { |
| s := release.NewDirStore(t.TempDir()) |
| rec := offlineRecord(t, s) |
| - got, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)) |
| + got, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)) |
| if err != nil { |
| t.Fatal(err) |
| } |
| m := got.Move |
| if !m.Offline() || m.State != OfflineStaging || m.ID != offID || m.To != "uk" || got.Region != "" || |
| - m.StagedAt == nil || !m.StagedAt.Equal(now) || !m.RequestedAt.Equal(now) { |
| + m.StagedAt == nil || !m.StagedAt.Equal(now) || !m.RequestedAt.Equal(now) || m.Stager != "box-a" { |
| t.Fatalf("staged record: %+v region %q", m, got.Region) |
| } |
| // Staging is a move in flight: no second move starts beside it. |
| if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStart(otherOff, "", "uk", "ops@example.com", now)); !errors.Is(err, ErrMoveConflict) { |
| t.Fatalf("a start beside a staged move: %v", err) |
| } |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(otherOff, "", "us", "ops@example.com", now)); !errors.Is(err, ErrMoveConflict) { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(otherOff, "", "us", "ops@example.com", "box-a", now)); !errors.Is(err, ErrMoveConflict) { |
| t.Fatalf("a stage beside a staged move: %v", err) |
| } |
| @@ -43,7 +43,7 @@ func TestOfflineStageThenStartThenTheWholeMove(t *testing.T) { |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if got.Move.State != OfflineStopping || got.Move.RequestedBy != "other@example.com" || !got.Move.RequestedAt.Equal(start) || !got.Move.StagedAt.Equal(now) { |
| + if got.Move.State != OfflineStopping || got.Move.RequestedBy != "other@example.com" || !got.Move.RequestedAt.Equal(start) || !got.Move.StagedAt.Equal(now) || got.Move.Stager != "box-a" { |
| t.Fatalf("started: %+v", got.Move) |
| } |
| // The rest of the move is the ordinary path; offline time counts from |
| @@ -74,7 +74,7 @@ func TestOfflineStartStagedRefusals(t *testing.T) { |
| } |
| s := release.NewDirStore(t.TempDir()) |
| rec := offlineRecord(t, s) |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)); err != nil { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)); err != nil { |
| t.Fatal(err) |
| } |
| if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStartStaged(otherOff, "ops@example.com", now)); !errors.Is(err, ErrMoveConflict) { |
| @@ -91,7 +91,7 @@ func TestOfflineStartStagedRefusals(t *testing.T) { |
| // No other step leaves staging: seal through finish all refuse. |
| s = release.NewDirStore(t.TempDir()) |
| rec = offlineRecord(t, s) |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)); err != nil { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)); err != nil { |
| t.Fatal(err) |
| } |
| before := offlineGet(t, s, rec) |
| @@ -112,7 +112,7 @@ func TestOfflineStartStagedIsAlwaysAfterTheStage(t *testing.T) { |
| now := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC) |
| s := release.NewDirStore(t.TempDir()) |
| rec := offlineRecord(t, s) |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)); err != nil { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)); err != nil { |
| t.Fatal(err) |
| } |
| got, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStartStaged(offID, "ops@example.com", now.Add(-time.Second))) |
| @@ -132,7 +132,7 @@ func TestOfflineAbortFromStaging(t *testing.T) { |
| now := time.Date(2026, 10, 21, 9, 0, 0, 0, time.UTC) |
| s := release.NewDirStore(t.TempDir()) |
| rec := offlineRecord(t, s) |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)); err != nil { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)); err != nil { |
| t.Fatal(err) |
| } |
| got, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineAbort(offID)) |
| @@ -147,7 +147,7 @@ func TestOfflineAbortFromStaging(t *testing.T) { |
| s = release.NewDirStore(t.TempDir()) |
| rec = offlineRecord(t, s) |
| for _, mut := range []func(*Record) error{ |
| - OfflineStage(offID, "", "uk", "ops@example.com", now), |
| + OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now), |
| OfflineStartStaged(offID, "ops@example.com", now.Add(20*time.Minute)), |
| OfflineAbort(offID), |
| } { |
| @@ -164,12 +164,12 @@ func TestOfflineAbortFromStaging(t *testing.T) { |
| // staged_at is additive like the other offline fields: a staged journal |
| // round-trips, and every earlier shape still does (TestOfflineFieldsAreAdditiveOnTheWire). |
| func TestOfflineStagedAtIsAdditiveOnTheWire(t *testing.T) { |
| - const staged = `{"schema":1,"host":"a.go.example.com","slug":"s","app":"app","account":"bab","region":"ie","channel":"stable","created_at":"2026-09-01T08:00:00Z","move":{"id":"mv_abcdefghijklmnopqrstuvwxyz","from":"ie","to":"uk","state":"offline-staging","requested_by":"ops@example.com","requested_at":"2026-10-21T09:00:00Z","updated_at":"2026-10-21T09:00:00Z","kind":"offline","staged_at":"2026-10-21T09:00:00Z"}}` |
| + const staged = `{"schema":1,"host":"a.go.example.com","slug":"s","app":"app","account":"bab","region":"ie","channel":"stable","created_at":"2026-09-01T08:00:00Z","move":{"id":"mv_abcdefghijklmnopqrstuvwxyz","from":"ie","to":"uk","state":"offline-staging","requested_by":"ops@example.com","requested_at":"2026-10-21T09:00:00Z","updated_at":"2026-10-21T09:00:00Z","kind":"offline","staged_at":"2026-10-21T09:00:00Z","stager":"box-a"}}` |
| rec, err := DecodeRecord([]byte(staged)) |
| if err != nil { |
| t.Fatal(err) |
| } |
| - if rec.Move.State != OfflineStaging || rec.Move.StagedAt == nil || rec.Move.StartedOffline() { |
| + if rec.Move.State != OfflineStaging || rec.Move.StagedAt == nil || rec.Move.StartedOffline() || rec.Move.Stager != "box-a" { |
| t.Fatalf("staged journal: %+v", rec.Move) |
| } |
| if out, _ := json.Marshal(rec); string(out) != staged { |
| @@ -188,11 +188,11 @@ func TestOfflineRestage(t *testing.T) { |
| if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineRestage(offID, now)); !errors.Is(err, ErrMoveConflict) { |
| t.Fatalf("restage at rest: %v", err) |
| } |
| - if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", now)); err != nil { |
| + if _, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "box-a", now)); err != nil { |
| t.Fatal(err) |
| } |
| got, err := SetRecordMove(ctx, s, rec.Account, rec.App, rec.Host, OfflineRestage(offID, now.Add(5*time.Minute))) |
| - if err != nil || !got.Move.StagedAt.Equal(now.Add(5*time.Minute)) || !got.Move.RequestedAt.Equal(now) || got.Move.StartedOffline() || got.Move.State != OfflineStaging { |
| + if err != nil || !got.Move.StagedAt.Equal(now.Add(5*time.Minute)) || !got.Move.RequestedAt.Equal(now) || got.Move.StartedOffline() || got.Move.State != OfflineStaging || got.Move.Stager != "box-a" { |
| t.Fatalf("restage: %v %+v", err, got.Move) |
| } |
| // A clock behind the last request still moves it forward. |
| @@ -212,3 +212,17 @@ func TestOfflineRestage(t *testing.T) { |
| t.Fatalf("restage after the start: %v", err) |
| } |
| } |
| + |
| +// The stager is pinned at the stage: a stage that names none is malformed. |
| +func TestOfflineStageNamesItsStager(t *testing.T) { |
| + s := release.NewDirStore(t.TempDir()) |
| + rec := offlineRecord(t, s) |
| + before := offlineGet(t, s, rec) |
| + if _, err := SetRecordMove(context.Background(), s, rec.Account, rec.App, rec.Host, OfflineStage(offID, "", "uk", "ops@example.com", "", time.Now())); !errors.Is(err, ErrMoveConflict) { |
| + t.Fatalf("a stage with no stager: %v", err) |
| + } |
| + if after := offlineGet(t, s, rec); !reflect.DeepEqual(after, before) { |
| + t.Fatal("a refused stage changed the record") |
| + } |
| +} |
| + |
diff --git a/internal/provision/provision.go b/internal/provision/provision.go| index 420ec20cb..844cbed4c 100644 |
| --- a/internal/provision/provision.go |
| +++ b/internal/provision/provision.go |
| @@ -1072,6 +1072,12 @@ type MoveState struct { |
| // move aborted before its start was never offline (OfflineFinish). |
| // Additive and omitempty like the three above. |
| StagedAt *time.Time `json:"staged_at,omitempty"` |
| + // Stager is the one source box that pre-copies a staged move, named |
| + // by the stage door and never changed (Astra round 1, finding 2): no |
| + // other box ever writes the staging prefix for this move, and an abort |
| + // of a staged move finishes only once this box has removed the staged |
| + // copy. Additive and omitempty. |
| + Stager string `json:"stager,omitempty"` |
| } |
| // MoveInputs (what the target was checked against, §7.3 T3) is NOT |
diff --git a/move.go b/move.go| index 3480e8ee4..579078a79 100644 |
| --- a/move.go |
| +++ b/move.go |
| @@ -84,7 +84,7 @@ const ( |
| moveOtherLoss = "The console asks to confirm a loss carlos move does not know. Use carlos ops host resolve." |
| moveLossChanged = "The move changed since it was read. Nothing was declared. Run the command again." |
| // The offline subcommand named no verb, or one it does not have (MV-O181, |
| - // which replaces MV-O56 now that stage is a verb). |
| + // which replaced MV-O56 when stage became a verb). |
| moveOfflineUsage = "carlos move offline needs stage, start, seal, flip, open, finish, abort or status." |
| ) |
| @@ -1057,7 +1057,7 @@ func (w *offlineRun) sayStaged(v offlineView) { |
| return |
| } |
| at := time.Unix(c.RoundAt, 0).UTC().Format("2006-01-02 15:04 UTC") |
| - moveSay(w.m.out, fmt.Sprintf("Staged %d of %d objects (%s of %s MB) of %s in %s at %s.", c.Done, c.Total, megabytes(c.DoneBytes), megabytes(c.TotalBytes), host, v.To, at)) |
| + moveSay(w.m.out, fmt.Sprintf("Staged %d of %d objects for %s in %s (%s of %s MB) at %s.", c.Done, c.Total, host, v.To, megabytes(c.DoneBytes), megabytes(c.TotalBytes), at)) |
| } |
| // sayStartWhenReady is MV-O184: what comes after a stage. |
| @@ -1418,7 +1418,7 @@ func (w *offlineRun) sayServed(v offlineView) { |
| func (w *offlineRun) headline(r offlineReply) { |
| v, host := r.v, w.m.host |
| if v.State == provision.OfflineStaging { |
| - moveSay(w.m.out, fmt.Sprintf("Staging %s in %s. %s keeps serving while it copies.", host, v.To, host)) |
| + moveSay(w.m.out, fmt.Sprintf("Staging %s in %s. %s keeps serving while the data is copied.", host, v.To, host)) |
| w.sayStaged(v) |
| if v.Next == "start" { |
| w.sayStartWhenReady(v) |
| @@ -1564,7 +1564,7 @@ func (w *offlineRun) runStage(ctx context.Context, to string) int { |
| if r.v.State != provision.OfflineStaging { |
| return w.stray(r) |
| } |
| - moveSay(w.m.out, fmt.Sprintf("Staging %s in %s. %s keeps serving while it copies.", host, r.v.To, host)) |
| + moveSay(w.m.out, fmt.Sprintf("Staging %s in %s. %s keeps serving while the data is copied.", host, r.v.To, host)) |
| asked := r.v.StagedAt |
| return w.follow(ctx, r, func(r offlineReply) (int, bool) { |
| if r.v.State != provision.OfflineStaging { |
diff --git a/move_test.go b/move_test.go| index d09ddffd9..bff16ba4c 100644 |
| --- a/move_test.go |
| +++ b/move_test.go |
| @@ -1816,7 +1816,7 @@ func TestMoveOfflineFailedReportAfterTheOpen(t *testing.T) { |
| // ---- carlos move offline stage ---- |
| const ( |
| - oStaging = "Staging acme.shop.example.com in uk. acme.shop.example.com keeps serving while it copies." |
| + oStaging = "Staging acme.shop.example.com in uk. acme.shop.example.com keeps serving while the data is copied." |
| oReady = "acme.shop.example.com keeps serving. Run carlos move offline start --host acme.shop.example.com --to uk when you are ready to take it offline." |
| oUnstage = "Removing the staged copy of acme.shop.example.com from uk." |
| oUnstaged = "The staged copy of acme.shop.example.com is removed. Run carlos move offline finish --host acme.shop.example.com to end the move." |
| @@ -1848,7 +1848,7 @@ func TestMoveOfflineStage(t *testing.T) { |
| }, |
| } |
| code, out := runOfflineAgainst(t, c, "stage", "--to", "uk") |
| - want := oStaging + "\nStaged 305 of 305 objects (1580.0 of 1580.0 MB) of acme.shop.example.com in uk at 2026-10-21 09:00 UTC.\n" + oReady + "\n" |
| + want := oStaging + "\nStaged 305 of 305 objects for acme.shop.example.com in uk (1580.0 of 1580.0 MB) at 2026-10-21 09:00 UTC.\n" + oReady + "\n" |
| if code != 0 || out != want { |
| t.Fatalf("exit %d, output:\n%s\nwant 0:\n%s", code, out, want) |
| } |
| @@ -1885,7 +1885,7 @@ func TestMoveOfflineStatusWhileStaging(t *testing.T) { |
| } |
| c = &fakeOfflineConsole{gets: []frame{stageView(200, 200, "staged", frame{"next": "start"})}} |
| code, out = runOfflineAgainst(t, c, "status") |
| - want := oStaging + "\nStaged 305 of 305 objects (1580.0 of 1580.0 MB) of acme.shop.example.com in uk at 2026-10-21 09:00 UTC.\n" + oReady + "\n" |
| + want := oStaging + "\nStaged 305 of 305 objects for acme.shop.example.com in uk (1580.0 of 1580.0 MB) at 2026-10-21 09:00 UTC.\n" + oReady + "\n" |
| if code != 0 || out != want { |
| t.Fatalf("exit %d, output:\n%s\nwant 0:\n%s", code, out, want) |
| } |
diff --git a/offline_move_e2e_test.go b/offline_move_e2e_test.go| index fff3f5766..5299341ab 100644 |
| --- a/offline_move_e2e_test.go |
| +++ b/offline_move_e2e_test.go |
| @@ -294,14 +294,14 @@ func TestMoveOfflineStageAgainstTheRealDoors(t *testing.T) { |
| } |
| return "", result{} |
| } |
| - staging := "Staging " + host + " in uk. " + host + " keeps serving while it copies." |
| + staging := "Staging " + host + " in uk. " + host + " keeps serving while the data is copied." |
| ready := host + " keeps serving. Run carlos move offline start --host " + host + " --to uk when you are ready to take it offline." |
| id, r := stage(time.Time{}) |
| if r.code != 0 { |
| t.Fatalf("stage exit %d:\n%s", r.code, r.out) |
| } |
| wantLines(t, r.out, staging, ready) |
| - if !strings.Contains(r.out, "Staged 288 of 288 objects (72.0 of 72.0 MB) of "+host+" in uk at ") { |
| + if !strings.Contains(r.out, "Staged 288 of 288 objects for "+host+" in uk (72.0 of 72.0 MB) at ") { |
| t.Fatalf("stage output lacks the progress line:\n%s", r.out) |
| } |
| // Again: a new request, answered by a new round. |
| @@ -326,6 +326,8 @@ func TestMoveOfflineStageAgainstTheRealDoors(t *testing.T) { |
| for deadline := time.Now().Add(20 * time.Second); time.Now().Before(deadline) && record().Move.State != provision.OfflineReverting; time.Sleep(10 * time.Millisecond) { |
| } |
| e.put(id, "ie-box", moves.OfflineRoleSource, moves.OfflinePhaseReverted, func(r *moves.OfflineReport) { r.Held = false }) |
| + // The pinned stager says the staged copy is gone: the finish waits for it. |
| + e.put(id, "ie-box", moves.OfflineRoleStager, moves.OfflinePhaseReverted, func(r *moves.OfflineReport) { r.Held, r.Fenced, r.Quiesced = false, false, false }) |
| select { |
| case r := <-aborted: |
| if r.code != 3 { |