carlos / platform Public

offline-move sealer: fix round 1 (lost stop, files refusal, no files dir, copy)

- I1: a tombstone that goes between the sealer's LockState and Reassert
  comes back as lease.ErrTombstoneLost; the sealer treats it as a lost
  stop (the "reassert" fault seam deletes the lock there in a test: one
  page, nothing taken again).
- I2: the lost-stop page splits by state. Before open it says to abort;
  from open or finishing on it says abort is refused and points to
  carlos move offline status and the offline move runbook.
- M2: every pass after the seal reads the source reports first; one
  written since the seal at a position past the sealed one is a lost
  stop in the db-tail item, with the tombstone left as it is.
- I3: the files refusal for an entry beside the files directory says the
  move can carry only the files directory and points to the runbook. It
  never tells the operator to move or remove the app's data.
- A host with no files directory (seen on real nested hosts) seals with
  an empty files item: the archive carries env alone and extracts to an
  empty files directory. Tested nested and flat.
- Copy: rows MV-O66 to MV-O98, all pending: the page subject, both
  bodies, the two causes, the failed item's reason, the four lease
  states, the files refusal, the other refusals and the passed items'
  reasons. Each refusal is now a literal where it is decided, so the
  gate reads it; the two that are an error's own text are listed in
  movesDiagnosticWrites with exact counts. movesoffline_seal.go is in
  movesScanFiles. The env item's reason no longer carries the digest
  (it is the seal's EnvSHA256): tracing it through the field reads ran
  the copy scan out of memory.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 4 days ago 4927cdfb32c6c6a953814721bce4cedc09ebddd0 parent 120af0f

4 files changed, +477 −101

approved_moves_copy_test.go +17 −0
docs/superpowers/plans/2026-10-06-moves-phase1-copy.md +43 −0
internal/host/movesoffline_seal.go +143 −51
internal/host/movesoffline_seal_test.go +274 −50

diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go

index 490e1bb38..ab3f90371 100644

--- a/approved_moves_copy_test.go

+++ b/approved_moves_copy_test.go

@@ -97,6 +97,9 @@ var movesCopyIDs = []string{

"MV-O32", "MV-O33",

"MV-O34", "MV-O35", "MV-O36", "MV-O37", "MV-O38", "MV-O39", "MV-O40", "MV-O41", "MV-O42", "MV-O43",

"MV-O50", "MV-O51", "MV-O52", "MV-O53", "MV-O54", "MV-O55", "MV-O56", "MV-O57", "MV-O58", "MV-O59", "MV-O60", "MV-O61", "MV-O62", "MV-O63", "MV-O64", "MV-O65",

+ "MV-O66", "MV-O67", "MV-O68", "MV-O69", "MV-O70", "MV-O71", "MV-O72", "MV-O73", "MV-O74", "MV-O75", "MV-O76", "MV-O77", "MV-O78", "MV-O79", "MV-O80", "MV-O81",

+ "MV-O82", "MV-O83", "MV-O84", "MV-O85", "MV-O86", "MV-O87", "MV-O88", "MV-O89", "MV-O90", "MV-O91", "MV-O92", "MV-O93", "MV-O94", "MV-O95", "MV-O96", "MV-O97",

+ "MV-O98",

}

// copyRow is one row of the record. The File cell is the file the sentence is

@@ -1262,6 +1265,7 @@ var movesScanFiles = []string{

"internal/host/movespaging.go",

"internal/host/movespass.go",

"internal/host/movesoffline.go",

+ "internal/host/movesoffline_seal.go",

"internal/host/movesphase1.go",

"internal/host/movespolicy.go",

"internal/host/movesprobe.go",

@@ -1447,6 +1451,9 @@ var movesReturnedSentences = map[string]map[string]bool{

// The box's move pages (Plan 6 Task 5, MV-N385 to MV-N390) and M-C20,

// the operators' sentence for a stray process (the stray page's body).

"internal/host/movespaging.go": {"rolledBackPage": true, "strayPage": true, "strayLine": true, "startRefusedPage": true},

+ // The offline sealer's lost-stop page, its two causes and the lease

+ // states the first names (offline move Task 4, MV-O66 to MV-O75).

+ "internal/host/movesoffline_seal.go": {"offlineLostStopPage": true, "offlineLeaseCause": true, "offlinePastSealCause": true, "offlineLeaseText": true},

// The console's move pass pages (Plan 5 Task 7, MV-N325 to MV-N334), and

// the dirty fenced target's page (moves after claim Plan 12, Task 5,

// MV-N450 and MV-N451).

@@ -1546,6 +1553,16 @@ var movesDiagnosticWrites = map[string]map[string]movesDiagnosticWrite{

Count: 2,

+ "internal/host/movesoffline_seal.go": {

+ "offlineTakeSeal|sealRefusal.Reason": {

+ Why: "the env-pin refusal for a CARLOS_SECRETS_KEY that does not decode is moves.ReportKeyFromSecrets' own error text (err.Error()), a box diagnostic that `carlos move offline status` shows as MV-O55's <detail>; it is not copy",

+ Count: 1,

+ },

+ "offlineArchive|sealRefusal.Reason": {

+ Why: "the files refusal for an archive Build refuses is archive.Build's own error text (err.Error()), whose sentinels are already rows MV-N77 to MV-N81 and MV-N86; a box diagnostic that `carlos move offline status` shows as MV-O55's <detail>",

+ Count: 1,

+ },

}

// movesServerText names, per file, the type that carries text the console

diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

index 2cdb820ff..f6db016aa 100644

--- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

+++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md

@@ -566,3 +566,46 @@ The source box's reasons for a path it left in place (`internal/host/movesofflin

+The sealer's lost-stop page (`internal/host/movesoffline_seal.go`, Task 4 fix round 1, review I2 and M2). After the seal the stop is kept, never taken again; when it no longer holds, the sealer pages an operator once per move. The body splits by state: before `open`, `abort` puts the instance back with nothing lost; from `open` on, `abort` is refused (MV-O40) and the move is fixed forward by hand. `<state>` is the record's offline state (`offline-sealed`, `offline-open` and so on). `<cause>` is one of the two causes below, and the failed item's reason (MV-O71) carries the same cause. `<lease>` is one of the four lease states MV-O72 to MV-O75; `<owner>` is a lease owner (`moved:<id>` or a box's lease id) and `<time>` an RFC 3339 time.

+|---|---|---|---|

+| MV-O67 | internal/host/movesoffline_seal.go | The offline move <id> of <host> is <state>, and its stop no longer holds: <cause>. The move cannot go on. Read carlos move offline status --host <host> and abort the move. | pending |

+| MV-O68 | internal/host/movesoffline_seal.go | The offline move <id> of <host> is <state>, and its stop no longer holds: <cause>. It cannot be aborted once it is open. Read carlos move offline status --host <host> and follow the offline move runbook. | pending |

+The sealer's item reasons, which `carlos move offline status` shows as MV-O55's `<detail>`. The copy gate reads every Check reason, so each has a row. MV-O76 is the `files` refusal for an entry the app keeps beside its files directory (review I3): the move cannot carry it, and moving it into the files directory or removing it would lose the app's data, so the row offers neither and points to the runbook; `<paths>` are the entries' paths on the box. MV-O77 to MV-O84 are the other refusals. Two refusals carry an error's own text and are box diagnostics, not rows (`movesDiagnosticWrites`): the archive's own refusal (its sentinels are MV-N77 to MV-N81 and MV-N86) and a secrets key that does not decode. MV-O85 to MV-O98 are the reasons of the items that passed; `<rung>` is the replication rung, quoted.

+|---|---|---|---|

diff --git a/internal/host/movesoffline_seal.go b/internal/host/movesoffline_seal.go

index f0e7b62b6..cbfb60cde 100644

--- a/internal/host/movesoffline_seal.go

+++ b/internal/host/movesoffline_seal.go

@@ -45,7 +45,7 @@ import (

// 1. this box's own report for the pass says held, fenced, quiesced, no

// process and no layout journal, and no source box keeps a sidecar;

-// 2. the local database is checkpointed when it is ahead of the replica

+// 2. the local database is checkpointed unless it is behind the replica

// (cfg.Checkpoint takes the lease, syncs and releases it);

// 3. lease.Tombstone, which never replaces a live holder;

// 4. only then the final position is read from the replica, and every

@@ -63,18 +63,20 @@ import (

// objects overwritten), so a crash anywhere before it converges. Once

// seal.json exists the sealed position is fixed, and from then on the

// tombstone is never taken again: each pass reads lease.LockState first,

-// and a tombstone that is absent, expired or anyone else's is a lost stop.

-// The pass reports failed and pages; the operator aborts. Only a tombstone

-// held by this move is re-asserted (lease.Reassert, If-Match on the ETag it

-// reads). After it is lifted (finish, abort) it is never re-asserted.

+// and a tombstone that is absent, expired or anyone else's is a lost stop,

+// as is a source report written since the seal at a position past the

+// sealed one. The pass reports failed and pages: before open the operator

+// aborts, from open on the runbook fixes it forward. Only a tombstone held

+// by this move is re-asserted (lease.Reassert, If-Match on the ETag it

+// reads, and lease.ErrTombstoneLost, never a new tombstone, when it went

+// in between). After it is lifted (finish, abort) it is never re-asserted.

// The objects the sealer writes beside the sealed archive, under

// moves.OfflineArchivePrefix(host, id) in the target's data bucket. The

// landing (Task 5) reads them there.

const (

// offlineArchiveSealed is the sealed files archive (archive.Seal of

- // archive.BuildFiltered's output). Its sha256 is the seal's

- // FilesSealed.

+ // archive.Build's output). Its sha256 is the seal's FilesSealed.

offlineArchiveSealed = "files.sealed"

// offlineArchiveManifest is its archive.Manifest (JSON), with the

// sealed data key and every entry.

@@ -165,7 +167,11 @@ func (t tombState) free() bool {

return t.owner == "" || !time.Now().Before(t.expires)

}

-func (t tombState) String() string {

+func (t tombState) String() string { return offlineLeaseText(t) }

+// offlineLeaseText is the lease as the lost stop's page and the sealer's

+// report name it (copy rows MV-O72 to MV-O75).

+func offlineLeaseText(t tombState) string {

switch {

case t.owner == "":

return "absent"

@@ -239,15 +245,16 @@ func replicaPosition(objs map[string]lease.MirrorObject, prefix string) uint64 {

// sealRefusal is a seal item that failed for a reason a later pass will

// meet again: the pass reports it (phase failed) and the operator aborts.

+// Reason is the item's reason as `carlos move offline status` shows it.

+// Each refusal is written as a literal where it is decided, so the copy

+// gate reads every sentence a Reason can carry (copy rows MV-O76 to

+// MV-O84); the two that carry an error's own text (the archive's refusal

+// and the secrets key's) are box diagnostics (movesDiagnosticWrites).

type sealRefusal struct {

- item, reason string

+ item, Reason string

}

-func (r *sealRefusal) Error() string { return r.item + ": " + r.reason }

-func sealRefuse(item, format string, args ...any) error {

- return &sealRefusal{item: item, reason: fmt.Sprintf(format, args...)}

+func (r *sealRefusal) Error() string { return r.item + ": " + r.Reason }

// offlineSeal is the sealer's step in offline-sealing.

func (w *movesWorker) offlineSeal(ctx context.Context, s release.Store, rec provision.Record) error {

@@ -269,7 +276,7 @@ func (w *movesWorker) offlineSeal(ctx context.Context, s release.Store, rec prov

seal, err := w.offlineTakeSeal(ctx, s, rec, items)

var ref *sealRefusal

if errors.As(err, &ref) {

- items[ref.item] = moves.Check{OK: false, Reason: ref.reason}

+ items[ref.item] = moves.Check{OK: false, Reason: ref.Reason}

r := moves.OfflineReport{Op: m.ID, Host: host, Box: w.cfg.Box, Role: moves.OfflineRoleSealer, Phase: moves.OfflinePhaseFailed,

At: w.cfg.now().UTC(), Items: items}

return errors.Join(fmt.Errorf("offline move %s: %s cannot be sealed: %w", m.ID, host, err), writeOfflineReport(ctx, s, r))

@@ -328,7 +335,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

}

for _, b := range boxes {

if reports[b].Sidecar {

- return seal, sealRefuse("sidecar", "%s keeps a sidecar file for %s", b, host)

+ return seal, &sealRefusal{item: "sidecar", Reason: fmt.Sprintf("%s keeps a sidecar file for %s", b, host)}

}

items["sidecar"] = moves.Check{OK: true, Reason: "no source box keeps a sidecar file"}

@@ -396,7 +403,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

}

log.Printf("moves: %s: offline move %s: checkpointed the local database (position %d, replica %d)", host, m.ID, local.LocalTXID, replica)

case local.LocalTXID > replica:

- return seal, sealRefuse("db-tail", "this box's local position %d is past the replica's %d and there is no database at %s to checkpoint", local.LocalTXID, replica, dbPath)

+ return seal, &sealRefusal{item: "db-tail", Reason: fmt.Sprintf("this box's local position %d is past the replica's %d and there is no database at %s to checkpoint", local.LocalTXID, replica, dbPath)}

}

if err := w.sealFault("checkpoint"); err != nil {

return seal, err

@@ -418,7 +425,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

}

final := replicaPosition(objs, srcPrefix)

if final == 0 {

- return seal, sealRefuse("db", "the replica at %s/%s holds no database", bucket, srcPrefix)

+ return seal, &sealRefusal{item: "db", Reason: fmt.Sprintf("the replica at %s/%s holds no database", bucket, srcPrefix)}

}

mine, err := w.localState(host, rt) // after the checkpoint, not as reported before it

if err != nil {

@@ -430,7 +437,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

pos = mine.LocalTXID

}

if pos > final {

- return seal, sealRefuse("db-tail", "%s's local position %d is past the replica's final position %d", b, pos, final)

+ return seal, &sealRefusal{item: "db-tail", Reason: fmt.Sprintf("%s's local position %d is past the replica's final position %d", b, pos, final)}

}

items["db-tail"] = moves.Check{OK: true, Reason: fmt.Sprintf("every source box is at or below TXID %d", final)}

@@ -488,7 +495,10 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

}

if len(stray) > 0 {

sort.Strings(stray)

- return seal, sealRefuse("files", "%s keeps %s beside its files directory %s, which the move carries alone; move it into %s or remove it", host, strings.Join(stray, ", "), files, files)

+ // What the app keeps there is the app's data, in a place it reads

+ // from: moving it into the files directory changes that place, and

+ // removing it loses it. So the refusal offers neither (MV-O76).

+ return seal, &sealRefusal{item: "files", Reason: fmt.Sprintf("%s keeps %s beside its files directory, and this move can carry only the files directory. see the offline move runbook", host, strings.Join(stray, ", "))}

}

a, err := w.offlineArchive(ctx, rec, mirror, tgt.DataBucket, prefix, files, envPath)

if err != nil {

@@ -526,16 +536,16 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

if err != nil {

return seal, err

}

- envWhat := "absent on the source, so none is carried"

+ // The digest itself is the seal's EnvSHA256, not part of the reason.

+ items["env"] = moves.Check{OK: true, Reason: "absent on the source, so none is carried"}

if envSHA != "" {

- envWhat = "carried in the archive, sha256 " + envSHA

+ items["env"] = moves.Check{OK: true, Reason: "carried in the archive"}

}

- items["env"] = moves.Check{OK: true, Reason: envWhat}

if cfg.LaunchLayers == nil {

- return seal, sealRefuse("env-pin", "this box has no exec backend to compose the launch environment")

+ return seal, &sealRefusal{item: "env-pin", Reason: "this box has no exec backend to compose the launch environment"}

}

if cfg.SecretsKey == "" {

- return seal, sealRefuse("env-pin", "this box holds no CARLOS_SECRETS_KEY to key the env report")

+ return seal, &sealRefusal{item: "env-pin", Reason: "this box holds no CARLOS_SECRETS_KEY to key the env report"}

}

env, instance, _, err := cfg.LaunchLayers(*rt)

if err != nil {

@@ -543,7 +553,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

}

mac, err := moves.ReportKeyFromSecrets(cfg.SecretsKey)

if err != nil {

- return seal, sealRefuse("env-pin", "the env report key: %v", err)

+ return seal, &sealRefusal{item: "env-pin", Reason: err.Error()}

}

envReport := moves.ReportLaunch(mac, env, instance)

envReportBody, err := json.Marshal(envReport)

@@ -567,7 +577,7 @@ func (w *movesWorker) offlineTakeSeal(ctx context.Context, s release.Store, rec

return seal, fmt.Errorf("read the live build: %w", err)

}

if version == "" {

- return seal, sealRefuse("release", "no build is live for %s on this box", host)

+ return seal, &sealRefusal{item: "release", Reason: fmt.Sprintf("no build is live for %s on this box", host)}

}

items["release"] = moves.Check{OK: true, Reason: "live build " + version}

items["replication"] = moves.Check{OK: true, Reason: fmt.Sprintf("replication %q", rt.Replication)}

@@ -640,15 +650,22 @@ func sameListing(source map[string]lease.MirrorObject, srcPrefix string, staged

// stray is every entry the app keeps beside its files directory, which

// the flat files directory cannot carry: on a nested row every entry of

// the instance directory that is neither the platform's

-// (layout.NestedPlatformNames) nor <host>; on a flat row every entry of

+// (layout.IsNestedPlatformName: the seed marker and the launch descriptors

+// included) nor <host>; on a flat row every entry of

// the data directory the layout's naming rules give this host

// (layout.FlatLeftovers; <host>-blobs is the live example) that is

// neither. The seal refuses on any, rather than leave it behind.

+//

+// A host that never kept a files directory (an app that writes nothing of

+// its own) has none at the files path: the archive then carries the env

+// file alone, FilesPlain is the digest of that, and the landing extracts

+// an empty files directory.

+//

// configPath is the rendered configuration beside the env file, which the

// effective instance key may come from.

func (w *movesWorker) offlineFilesOf(host string, rt *registry.Route) (files, envPath, configPath string, stray []string, err error) {

dd := w.cfg.DataDir

- platform := layout.NestedPlatformNames(host)

+ platform := func(name string) bool { return layout.IsNestedPlatformName(host, name) }

if rt != nil && provision.NestedPaths(dd, *rt) {

raw := layout.RawInstanceDir(dd, host)

ents, err := os.ReadDir(raw)

@@ -656,7 +673,7 @@ func (w *movesWorker) offlineFilesOf(host string, rt *registry.Route) (files, en

return "", "", "", nil, err

}

for _, e := range ents {

- if !platform[e.Name()] && e.Name() != host {

+ if !platform(e.Name()) && e.Name() != host {

stray = append(stray, filepath.Join(raw, e.Name()))

}

@@ -673,7 +690,7 @@ func (w *movesWorker) offlineFilesOf(host string, rt *registry.Route) (files, en

}

for _, p := range layout.FlatLeftovers(dd, host, hosts) {

- if name := filepath.Base(p); !platform[name] && name != host {

+ if name := filepath.Base(p); !platform(name) && name != host {

stray = append(stray, p)

}

@@ -702,7 +719,7 @@ func (w *movesWorker) offlineArchive(ctx context.Context, rec provision.Record,

cfg := w.cfg

var out sealedArchive

if cfg.ConfigPub == nil {

- return out, sealRefuse("files", "this box holds no CARLOS_SECRETS_KEY to seal the archive under")

+ return out, &sealRefusal{item: "files", Reason: "this box holds no CARLOS_SECRETS_KEY to seal the archive under"}

}

dir := moves.OfflineParkDir(cfg.DataDir, rec.Host, rec.Move.ID, moves.OfflineRoleSealer)

if dir == "" {

@@ -726,7 +743,7 @@ func (w *movesWorker) offlineArchive(ctx context.Context, rec provision.Record,

case errors.Is(err, archive.ErrChanged):

return out, fmt.Errorf("archive the files: %w; built again next pass", err)

case err != nil:

- return out, sealRefuse("files", "%v", err)

+ return out, &sealRefusal{item: "files", Reason: err.Error()}

}

if _, err := built.Seek(0, io.SeekStart); err != nil {

return out, err

@@ -859,7 +876,10 @@ func decodeSeal(raw []byte, host, id string) (*moves.OfflineReport, error) {

// offlineKeepStop is every pass after the seal, until the tombstone is

// lifted: read the lease first; a tombstone that is not this move's any

-// more is a lost stop, never taken again; this move's is re-asserted. In

+// more is a lost stop, never taken again. Then the source reports: one

+// written since the seal at a position past the sealed one is a lost stop

+// too (a write the stop did not hold off, which the staged copy lacks).

+// Only then is this move's tombstone re-asserted. In

// offline-sealing (a seal written by a pass that crashed before the CAS)

// it also makes the CAS. Then it rewrites the living sealer report: the

// seal's values and items, the tombstone's owner and expiry, and At.

@@ -878,16 +898,34 @@ func (w *movesWorker) offlineKeepStop(ctx context.Context, s release.Store, rec

return err

}

if !st.mine(m.ID) {

- return w.offlineLostStop(ctx, s, rec, seal, st)

+ return w.offlineLeaseLost(ctx, s, rec, seal, st)

+ }

+ if seal != nil {

+ if err := w.offlineCheckPositions(ctx, s, rec, seal); err != nil {

+ return err

+ }

}

bucket, loc, region, err := cfg.tombstoneAt(host)

if err != nil {

return err

}

+ if err := w.sealFault("reassert"); err != nil {

+ return err

+ }

if err := lease.Reassert(ctx, bucket, loc, region, moves.TombstoneOwner(m.ID), moves.TombstoneTTL); err != nil {

var held *lease.HeldError

if errors.As(err, &held) {

- return w.offlineLostStop(ctx, s, rec, seal, tombState{owner: held.Owner, expires: held.ExpiresAt})

+ return w.offlineLeaseLost(ctx, s, rec, seal, tombState{owner: held.Owner, expires: held.ExpiresAt})

+ }

+ if errors.Is(err, lease.ErrTombstoneLost) {

+ // Gone between the read above and Reassert's own: the same

+ // lost stop, and Reassert wrote nothing. Read it again for

+ // what it is now (absent, when even that read fails).

+ now, rerr := w.offlineLock(ctx, host)

+ if rerr != nil {

+ now = tombState{}

+ }

+ return w.offlineLeaseLost(ctx, s, rec, seal, now)

}

return fmt.Errorf("offline move %s: re-assert the tombstone of %s: %w", m.ID, host, err)

}

@@ -895,7 +933,7 @@ func (w *movesWorker) offlineKeepStop(ctx context.Context, s release.Store, rec

return err

}

if !st.mine(m.ID) {

- return w.offlineLostStop(ctx, s, rec, seal, st)

+ return w.offlineLeaseLost(ctx, s, rec, seal, st)

}

if m.State == provision.OfflineSealing {

src := provision.MoveSource{

@@ -929,16 +967,56 @@ func (w *movesWorker) offlineKeepStop(ctx context.Context, s release.Store, rec

return writeOfflineReport(ctx, s, living)

}

-// offlineLostStop reports a stop that no longer holds: the tombstone is

-// absent, expired, lifted or someone else's after the seal. It is never

-// taken again (a wake may have written the replica in between, so the

-// sealed position is no longer the source's), the living report says

-// failed with the lease item not OK, and an operator is paged once. The

-// operator aborts the move.

-func (w *movesWorker) offlineLostStop(ctx context.Context, s release.Store, rec provision.Record, seal *moves.OfflineReport, st tombState) error {

+// offlineCheckPositions is the backstop behind the holds and the fence:

+// every source report written since the seal (at or after its At) must be

+// at or below the sealed final position. One past it is a box that wrote

+// what the staged copy does not hold, so the sealed position is no longer

+// the instance's: a lost stop, in the db-tail item.

+func (w *movesWorker) offlineCheckPositions(ctx context.Context, s release.Store, rec provision.Record, seal *moves.OfflineReport) error {

+ reports, err := readOfflineReports(ctx, s, rec.Host, rec.Move.ID, moves.OfflineRoleSource)

+ if err != nil {

+ return fmt.Errorf("read the source reports: %w", err)

+ }

+ boxes := make([]string, 0, len(reports))

+ for b := range reports {

+ boxes = append(boxes, b)

+ }

+ sort.Strings(boxes)

+ for _, b := range boxes {

+ r := reports[b]

+ if r.At.Before(seal.At) || r.LocalTXID <= seal.FinalTXID {

+ continue

+ }

+ return w.offlineLostStop(ctx, s, rec, seal, "db-tail", offlinePastSealCause(b, r.LocalTXID, seal.FinalTXID))

+ }

+ return nil

+// offlineLeaseLost is the lost stop of a tombstone that is absent,

+// expired, lifted or someone else's, in the lease item.

+func (w *movesWorker) offlineLeaseLost(ctx context.Context, s release.Store, rec provision.Record, seal *moves.OfflineReport, st tombState) error {

+ return w.offlineLostStop(ctx, s, rec, seal, "lease", offlineLeaseCause(st))

+// offlineLeaseCause and offlinePastSealCause are the two causes of a lost

+// stop as its page and the failed item say them (MV-O69, MV-O70).

+func offlineLeaseCause(st tombState) string {

+ return fmt.Sprintf("the source's lease is %s, not this move's tombstone", offlineLeaseText(st))

+func offlinePastSealCause(box string, pos, final uint64) string {

+ return fmt.Sprintf("%s reports TXID %d, past the sealed position %d", box, pos, final)

+// offlineLostStop reports a stop that no longer holds after the seal:

+// cause says why, and item is the seal item it fails. The tombstone is

+// never taken again (a wake may have written the replica in between, so

+// the sealed position is no longer the source's), the living report says

+// failed with that item not OK and no tombstone owner (so open refuses),

+// and an operator is paged once per move.

+func (w *movesWorker) offlineLostStop(ctx context.Context, s release.Store, rec provision.Record, seal *moves.OfflineReport, item, cause string) error {

cfg := w.cfg

m, host := rec.Move, rec.Host

- reason := fmt.Sprintf("lost stop: the tombstone at the source's lease is %s, not held by %s", st, moves.TombstoneOwner(m.ID))

r := moves.OfflineReport{Op: m.ID, Host: host, Box: cfg.Box, Role: moves.OfflineRoleSealer}

if seal != nil {

r = *seal

@@ -950,14 +1028,28 @@ func (w *movesWorker) offlineLostStop(ctx context.Context, s release.Store, rec

for k, v := range r.Items {

items[k] = v

}

- items["lease"] = moves.Check{OK: false, Reason: reason}

+ items[item] = moves.Check{OK: false, Reason: "lost stop: " + cause}

r.Items = items

werr := writeOfflineReport(ctx, s, r)

- subject := fmt.Sprintf("offline move of %s lost its stop", host)

- body := fmt.Sprintf("The offline move %s of %s is %s, and the tombstone that stops every wake in the source region is no longer held: %s. The move cannot go on. Read carlos move offline status --host %s and abort the move.", m.ID, host, m.State, st, host)

+ subject, body := offlineLostStopPage(host, m.ID, m.State, cause)

w.firePage(ctx, host, m.ID, offlinePageLostStop, alerting.LevelIncident, subject, body)

- log.Printf("moves: %s: offline move %s: LOST STOP: %s", host, m.ID, reason)

- return errors.Join(fmt.Errorf("offline move %s: %s: %s", m.ID, host, reason), werr)

+ log.Printf("moves: %s: offline move %s: LOST STOP: %s", host, m.ID, cause)

+ return errors.Join(fmt.Errorf("offline move %s: %s: lost stop: %s", m.ID, host, cause), werr)

+// offlineLostStopPage is the lost stop's page (copy rows MV-O66 to

+// MV-O68; the item's reason is MV-O71): its subject, and a body for the move's state. Before open,

+// abort puts the instance back with nothing lost, so the body says to

+// abort. From open on, abort is refused (MV-O40) and the move is fixed

+// forward by hand, so the body points to the status and the runbook.

+func offlineLostStopPage(host, id, state, cause string) (subject, body string) {

+ subject = fmt.Sprintf("offline move of %s lost its stop", host)

+ if state == provision.OfflineOpen || state == provision.OfflineFinishing {

+ body = fmt.Sprintf("The offline move %s of %s is %s, and its stop no longer holds: %s. It cannot be aborted once it is open. Read carlos move offline status --host %s and follow the offline move runbook.", id, host, state, cause, host)

+ return subject, body

+ }

+ body = fmt.Sprintf("The offline move %s of %s is %s, and its stop no longer holds: %s. The move cannot go on. Read carlos move offline status --host %s and abort the move.", id, host, state, cause, host)

+ return subject, body

}

// offlineFinishLift is the sealer's step in offline-finishing. Until every

@@ -1004,7 +1096,7 @@ func (w *movesWorker) offlineFinishLift(ctx context.Context, s release.Store, re

case st.free():

default:

seal, _ := decodeSeal(raw, host, m.ID)

- return w.offlineLostStop(ctx, s, rec, seal, st)

+ return w.offlineLeaseLost(ctx, s, rec, seal, st)

}

if err := w.sealFault("lift"); err != nil {

return err

diff --git a/internal/host/movesoffline_seal_test.go b/internal/host/movesoffline_seal_test.go

index 1d6004da0..02158838d 100644

--- a/internal/host/movesoffline_seal_test.go

+++ b/internal/host/movesoffline_seal_test.go

@@ -60,6 +60,7 @@ type sealWorld struct {

priv *ecdh.PrivateKey

ie, uk *offBox

pages []string

+ bodies []string // each page's body, beside pages

}

func newSealWorld(t *testing.T) *sealWorld {

@@ -127,6 +128,7 @@ func (sw *sealWorld) arm(b *offBox) {

c.LaunchLayers = (&activator.ExecBackend{DataDir: b.dd}).LaunchLayers

c.Page = func(_ context.Context, level, subject, body string) error {

sw.pages = append(sw.pages, level+": "+subject)

+ sw.bodies = append(sw.bodies, body)

return nil

}

@@ -622,7 +624,7 @@ func TestOfflineSealRefusesASourceAheadOfTheFinalPosition(t *testing.T) {

}

err := sw.pass(ie)

var ref *sealRefusal

- if !errors.As(err, &ref) || ref.item != "db-tail" || !strings.Contains(ref.reason, "ie-b") {

+ if !errors.As(err, &ref) || ref.item != "db-tail" || !strings.Contains(ref.Reason, "ie-b") {

t.Fatalf("pass: %v", err)

}

r := sw.report(moves.OfflineRoleSealer, "ie-a")

@@ -976,7 +978,7 @@ func TestOfflineSealFailsAFileOverTheCapAndAFIFO(t *testing.T) {

c.setup()

err := sw.pass(ie)

var ref *sealRefusal

- if !errors.As(err, &ref) || ref.item != "files" || !strings.Contains(ref.reason, c.want) {

+ if !errors.As(err, &ref) || ref.item != "files" || !strings.Contains(ref.Reason, c.want) {

t.Fatalf("%s: %v", c.name, err)

}

r := sw.report(moves.OfflineRoleSealer, "ie-a")

@@ -1015,63 +1017,35 @@ func TestOfflineSealArchivesANestedInstanceFlat(t *testing.T) {

sw := newSealWorld(t)

ie := sw.ie

sw.instance(ie)

- ctx := context.Background()

- // The bind mount at <dd>/instances, as a link in the fixture.

- if err := os.MkdirAll(filepath.Join(ie.dd, ".instances"), 0o755); err != nil {

- t.Fatal(err)

- }

- if err := os.Symlink(".instances", filepath.Join(ie.dd, "instances")); err != nil {

- t.Fatal(err)

- }

- // The layout pass works on the row's own paths; the sockets live in a

- // short directory of the test's, as the pass's tests keep them.

- sd, err := os.MkdirTemp("", "ly")

- if err != nil {

- t.Fatal(err)

- }

- t.Cleanup(func() { os.RemoveAll(sd) })

- rt := ie.row(offHost)

- if err := ie.reg.SetInstancePaths(offHost, rt.DBPath, rt.Socket, rt.DBPath, filepath.Join(sd, offHost+".sock")); err != nil {

- t.Fatal(err)

- }

- pass := &layout.Pass{Registry: ie.reg, DataDir: ie.dd, SocketDir: sd, SocketsGID: -1, Now: time.Now, SweptAtStart: true, Logf: t.Logf}

- res := pass.Converge(ctx, layout.ModeNested)

- migrated := false

- for _, r := range res {

- if r.Host == offHost && r.Outcome == "migrated" {

- migrated = true

- }

- if !migrated {

- t.Fatalf("the layout pass: %+v", res)

+ raw := sw.nest(ie)

+ // More platform entries, which stay behind: the generation and files

+ // markers, the adoption record, the seed marker of a seeded database,

+ // and the launch descriptors (fixed and per generation) as a real

+ // nested host keeps them.

+ for _, name := range []string{offHost + ".db.gen", ".files-gen", offHost + instancekey.AdoptionSuffix, offHost + ".db" + activator.SeedMarkerSuffix,

+ "launch.run.json", "launch.seed.json", "launch.gen.7.json"} {

+ writeFile(t, filepath.Join(raw, name), "platform")

}

- ie.nested = true

- raw := layout.RawInstanceDir(ie.dd, offHost)

- if rt := ie.row(offHost); !provision.NestedPaths(ie.dd, *rt) {

- t.Fatalf("fixture: row %+v is not nested", rt)

- }

- if lexists(filepath.Join(ie.dd, offHost)) {

- t.Fatal("fixture: the layout pass left the flat files directory")

- }

- for _, name := range []string{offHost + ".db", offHost + ".env", offHost + ".config.env", ".activator-" + offHost + ".db-litestream", filepath.Join(offHost, "instance.key")} {

- if !lexists(filepath.Join(raw, name)) {

- t.Fatalf("fixture: the layout pass did not move %s into the instance directory", name)

- }

- // More platform entries, which stay behind.

- writeFile(t, filepath.Join(raw, offHost+".db.gen"), "generation")

- writeFile(t, filepath.Join(raw, ".files-gen"), "marker")

- writeFile(t, filepath.Join(raw, offHost+instancekey.AdoptionSuffix), "{}")

// An entry of the app's beside its files directory, which no flat

// files directory can hold.

stray := filepath.Join(raw, offHost+"-blobs", "b")

writeFile(t, stray, "blob")

sw.startAndSeal("ie-a", ie)

- err = sw.pass(ie)

+ err := sw.pass(ie)

var ref *sealRefusal

- if !errors.As(err, &ref) || ref.item != "files" || !strings.Contains(ref.reason, offHost+"-blobs") {

+ if !errors.As(err, &ref) || ref.item != "files" || !strings.Contains(ref.Reason, offHost+"-blobs") {

t.Fatalf("a stray entry: %v", err)

}

+ // The refusal says the move cannot carry it, and offers no fix that

+ // moves or removes the app's data (review I3).

+ if !strings.Contains(ref.Reason, "this move can carry only the files directory. see the offline move runbook") || strings.Contains(ref.Reason, "move it") || strings.Contains(ref.Reason, "remove") {

+ t.Fatalf("the refusal: %q", ref.Reason)

+ }

+ for _, name := range []string{"launch.run.json", "launch.gen.7.json", ".db.seed"} {

+ if strings.Contains(ref.Reason, name) {

+ t.Fatalf("the refusal names the platform's %s: %q", name, ref.Reason)

+ }

if r := sw.report(moves.OfflineRoleSealer, "ie-a"); r.Phase != moves.OfflinePhaseFailed || r.Items["files"].OK {

t.Fatalf("report %+v", r)

}

@@ -1108,6 +1082,59 @@ func TestOfflineSealArchivesANestedInstanceFlat(t *testing.T) {

}

+// nest makes b's instance of offHost nested with the real layout pass,

+// which takes the flat files directory <data>/<host> into the instance

+// directory as <instance dir>/<host>, and returns the raw instance

+// directory.

+func (sw *sealWorld) nest(ie *offBox) string {

+ t := sw.t

+ t.Helper()

+ ctx := context.Background()

+ // The bind mount at <dd>/instances, as a link in the fixture.

+ if err := os.MkdirAll(filepath.Join(ie.dd, ".instances"), 0o755); err != nil {

+ t.Fatal(err)

+ }

+ if err := os.Symlink(".instances", filepath.Join(ie.dd, "instances")); err != nil {

+ t.Fatal(err)

+ }

+ // The layout pass works on the row's own paths; the sockets live in a

+ // short directory of the test's, as the pass's tests keep them.

+ sd, err := os.MkdirTemp("", "ly")

+ if err != nil {

+ t.Fatal(err)

+ }

+ t.Cleanup(func() { os.RemoveAll(sd) })

+ rt := ie.row(offHost)

+ if err := ie.reg.SetInstancePaths(offHost, rt.DBPath, rt.Socket, rt.DBPath, filepath.Join(sd, offHost+".sock")); err != nil {

+ t.Fatal(err)

+ }

+ pass := &layout.Pass{Registry: ie.reg, DataDir: ie.dd, SocketDir: sd, SocketsGID: -1, Now: time.Now, SweptAtStart: true, Logf: t.Logf}

+ res := pass.Converge(ctx, layout.ModeNested)

+ migrated := false

+ for _, r := range res {

+ if r.Host == offHost && r.Outcome == "migrated" {

+ migrated = true

+ }

+ if !migrated {

+ t.Fatalf("the layout pass: %+v", res)

+ }

+ ie.nested = true

+ raw := layout.RawInstanceDir(ie.dd, offHost)

+ if rt := ie.row(offHost); !provision.NestedPaths(ie.dd, *rt) {

+ t.Fatalf("fixture: row %+v is not nested", rt)

+ }

+ if lexists(filepath.Join(ie.dd, offHost)) {

+ t.Fatal("fixture: the layout pass left the flat files directory")

+ }

+ for _, name := range []string{offHost + ".db", offHost + ".env", offHost + ".config.env", ".activator-" + offHost + ".db-litestream", filepath.Join(offHost, "instance.key")} {

+ if !lexists(filepath.Join(raw, name)) {

+ t.Fatalf("fixture: the layout pass did not move %s into the instance directory", name)

+ }

+ return raw

// Finishing lifts the tombstone only once every box the seal named reports

// gone: until then each pass keeps the stop. After the lift a wake takes

// the lease, the report says lifted, and a later pass changes nothing.

@@ -1179,3 +1206,200 @@ func TestOfflineSealFinishingLiftsOnlyAfterEverySourceIsGone(t *testing.T) {

t.Fatalf("paged: %v", sw.pages)

}

+// A host that never kept a files directory (seen on real nested hosts: the

+// instance directory holds the database, the env files and the launch

+// descriptors, and no <host>) seals with an empty files item: the archive

+// carries the env file alone, FilesPlain is archive.PlainDigest of no files

+// directory and that env file (FilesEntries 1), and it extracts to an

+// empty files directory beside the env file. Nested and flat alike.

+func TestOfflineSealAHostWithNoFilesDirectory(t *testing.T) {

+ for _, nested := range []bool{true, false} {

+ t.Run(map[bool]string{true: "nested", false: "flat"}[nested], func(t *testing.T) {

+ sw := newSealWorld(t)

+ ie := sw.ie

+ sw.instance(ie)

+ files := filepath.Join(ie.dd, offHost)

+ envPath := filepath.Join(ie.dd, offHost+".env")

+ if nested {

+ raw := sw.nest(ie)

+ files, envPath = filepath.Join(raw, offHost), filepath.Join(raw, offHost+".env")

+ writeFile(t, filepath.Join(raw, "launch.run.json"), "{}")

+ }

+ if err := os.RemoveAll(files); err != nil {

+ t.Fatal(err)

+ }

+ sw.startAndSeal("ie-a", ie)

+ sw.mustPass(ie)

+ if sw.record().Move.State != provision.OfflineSealed {

+ t.Fatal("not sealed")

+ }

+ seal, _, _ := sw.sealRecord()

+ want, wantBytes, wantEntries, err := archive.PlainDigest(files, envPath)

+ if err != nil {

+ t.Fatal(err)

+ }

+ if seal.FilesPlain != want || seal.FilesBytes != wantBytes || seal.FilesEntries != wantEntries || wantEntries != 1 {

+ t.Fatalf("seal files %s %d bytes %d entries, want %s %d bytes 1 entry (env only)", seal.FilesPlain, seal.FilesBytes, seal.FilesEntries, want, wantBytes)

+ }

+ if r := sw.report(moves.OfflineRoleSealer, "ie-a"); r.Phase != moves.OfflinePhaseSealed || !r.Items["files"].OK {

+ t.Fatalf("report %+v", r)

+ }

+ got, env := sw.openArchive(seal)

+ ents, err := os.ReadDir(got)

+ if err != nil || len(ents) != 0 {

+ t.Fatalf("the landing's files directory: %v entries, %v", len(ents), err)

+ }

+ if readString(t, env) != readString(t, envPath) {

+ t.Fatal("the env file did not travel byte for byte")

+ }

+ if lexists(files) {

+ t.Fatal("the seal made a files directory on the source")

+ }

+ })

+ }

+// The tombstone goes between the sealer's LockState and Reassert's own

+// read: Reassert reports it lost (lease.ErrTombstoneLost) and writes

+// nothing, so the sealer pages a lost stop once and never takes it again.

+func TestOfflineSealLostBetweenTheReadAndTheReassert(t *testing.T) {

+ sw := newSealWorld(t)

+ ie := sw.ie

+ sw.instance(ie)

+ sw.startAndSeal("ie-a", ie)

+ sw.mustPass(ie)

+ ctx := context.Background()

+ key := sealPrefix + "/" + offHost + "/lock.json"

+ var after string

+ t.Cleanup(func() { offlineSealFaults.Delete(ie.w) })

+ offlineSealFaults.Store(ie.w, func(point string) error {

+ if point == "reassert" {

+ sw.fake.BareDelete(sealIE, key)

+ after = sw.fake.Snapshot()

+ }

+ return nil

+ })

+ err := ie.w.offlineSealerStep(ctx, sw.s, sw.record())

+ if err == nil || !strings.Contains(err.Error(), "lost stop") {

+ t.Fatalf("pass: %v", err)

+ }

+ if after == "" || sw.fake.Snapshot() != after {

+ t.Fatal("the bucket was written after the tombstone went")

+ }

+ if _, ok := sw.fake.Current(sealIE, key); ok {

+ t.Fatal("the tombstone was taken again")

+ }

+ r := sw.report(moves.OfflineRoleSealer, "ie-a")

+ if r.Phase != moves.OfflinePhaseFailed || r.Items["lease"].OK || r.Items["lease"].Reason != "lost stop: "+offlineLeaseCause(tombState{}) {

+ t.Fatalf("report %+v", r)

+ }

+ // The next pass finds it gone at LockState: still nothing written,

+ // still one page.

+ offlineSealFaults.Delete(ie.w)

+ if err := ie.w.offlineSealerStep(ctx, sw.s, sw.record()); err == nil || !strings.Contains(err.Error(), "lost stop") {

+ t.Fatalf("second pass: %v", err)

+ }

+ if sw.fake.Snapshot() != after {

+ t.Fatal("the second pass wrote to the bucket")

+ }

+ if len(sw.pages) != 1 {

+ t.Fatalf("pages %v, want one", sw.pages)

+ }

+// The lost stop's page says abort only while abort can still put the

+// instance back. From open on, abort is refused, so it points to the

+// status and the runbook instead. Each state in its own world: the page

+// goes out once per move.

+func TestOfflineSealLostStopPageByState(t *testing.T) {

+ for _, state := range []string{provision.OfflineSealed, provision.OfflineOpen, provision.OfflineFinishing} {

+ t.Run(state, func(t *testing.T) {

+ sw := newSealWorld(t)

+ ie := sw.ie

+ sw.instance(ie)

+ sw.startAndSeal("ie-a", ie)

+ sw.mustPass(ie)

+ ctx := context.Background()

+ if state != provision.OfflineSealed {

+ sw.flip()

+ sw.landed()

+ sw.open()

+ }

+ if state == provision.OfflineFinishing {

+ sw.finishing()

+ }

+ if err := ie.w.offlineSealerStep(ctx, sw.s, sw.record()); err != nil {

+ t.Fatalf("the stop held: %v", err)

+ }

+ if got := sw.record().Move.State; got != state {

+ t.Fatalf("fixture: state %s", got)

+ }

+ sw.fake.BareDelete(sealIE, sealPrefix+"/"+offHost+"/lock.json")

+ if err := ie.w.offlineSealerStep(ctx, sw.s, sw.record()); err == nil || !strings.Contains(err.Error(), "lost stop") {

+ t.Fatalf("lost: %v", err)

+ }

+ if len(sw.pages) != 1 || sw.pages[0] != "incident: offline move of "+offHost+" lost its stop" {

+ t.Fatalf("pages %v", sw.pages)

+ }

+ body := sw.bodies[0]

+ if !strings.HasPrefix(body, "The offline move "+offID+" of "+offHost+" is "+state+", and its stop no longer holds: the source's lease is absent, not this move's tombstone. ") {

+ t.Fatalf("body: %s", body)

+ }

+ abort := strings.HasSuffix(body, "The move cannot go on. Read carlos move offline status --host "+offHost+" and abort the move.")

+ runbook := strings.HasSuffix(body, "It cannot be aborted once it is open. Read carlos move offline status --host "+offHost+" and follow the offline move runbook.")

+ if wantAbort := state == provision.OfflineSealed; abort != wantAbort || runbook == wantAbort {

+ t.Fatalf("body in %s: %s", state, body)

+ }

+ })

+ }

+// The backstop behind the holds and the fence (review M2): a source report

+// written since the seal at a position past the sealed one is a lost stop

+// in the db-tail item, even with the tombstone held. Nothing is written to

+// the lease, and the page names the box.

+func TestOfflineSealLostStopOnASourcePastTheSealedPosition(t *testing.T) {

+ sw := newSealWorld(t)

+ ie := sw.ie

+ sw.instance(ie)

+ ieB := sw.sourceBox("ie-b")

+ sw.startAndSeal("ie-a", ie, ieB)

+ sw.mustPass(ie)

+ if sw.record().Move.State != provision.OfflineSealed {

+ t.Fatal("not sealed")

+ }

+ seal, _, _ := sw.sealRecord()

+ ctx := context.Background()

+ // A pass with every report at or below the seal keeps the stop.

+ ieB.tick()

+ if err := ie.w.offlineSealerStep(ctx, sw.s, sw.record()); err != nil {

+ t.Fatalf("at the seal: %v", err)

+ }

+ // ie-b writes past the sealed position, and reports it.

+ meta := filepath.Join(activatorMetaDir(provision.DefaultDBPath(ieB.dd, offHost)), "ltx", "0")

+ writeFile(t, filepath.Join(meta, ltxName(seal.FinalTXID+1, seal.FinalTXID+3)), "ltx")

+ ieB.tick()

+ if r := sw.report(moves.OfflineRoleSource, "ie-b"); r.LocalTXID != seal.FinalTXID+3 || r.At.Before(seal.At) {

+ t.Fatalf("fixture: ie-b reports %d at %s", r.LocalTXID, r.At)

+ }

+ before := sw.fake.Snapshot()

+ err := ie.w.offlineSealerStep(ctx, sw.s, sw.record())

+ if err == nil || !strings.Contains(err.Error(), "lost stop") {

+ t.Fatalf("past the seal: %v", err)

+ }

+ if sw.fake.Snapshot() != before {

+ t.Fatal("the lease was written after the stop was lost")

+ }

+ cause := offlinePastSealCause("ie-b", seal.FinalTXID+3, seal.FinalTXID)

+ r := sw.report(moves.OfflineRoleSealer, "ie-a")

+ if r.Phase != moves.OfflinePhaseFailed || r.Items["db-tail"].OK || r.Items["db-tail"].Reason != "lost stop: "+cause || r.TombstoneOwner != "" || !r.Items["lease"].OK {

+ t.Fatalf("report %+v", r)

+ }

+ if len(sw.bodies) != 1 || !strings.Contains(sw.bodies[0], "its stop no longer holds: "+cause+".") {

+ t.Fatalf("pages %v %v", sw.pages, sw.bodies)

+ }

+ if owner, _, _ := sw.lock(); owner != moves.TombstoneOwner(offID) {

+ t.Fatalf("the tombstone is %s", owner)

+ }