amadan
offline-move: the release item refuses a target live link to another build (final review HIGH)
A return move after an upgrade (A to B on v1, v2 on B, B back to A) found A's root-owned live link still naming v1, which retirement leaves. The channel names v2, so the landing passed, and a wake before ordinary adoption could exec v1 against the sealed v2 database. The release item now also reads <opt>/live/<host> on the target: absent is fine (nothing runs until adoption stages the sealed build); a link that does not resolve to the sealed version's VersionPath fails the item with MV-O177 (pending copy). open re-verifies every landing item, so it keeps the hold. The link is only read; root gains no code path. Runbook P3 says to abort such a return move and to read the target's link before starting one. Tests: the return-move regression (landing fails at release, a scheduled tick on the held row is refused, open keeps the hold, the link untouched, a link naming the sealed build opens), a link appearing before open, and the fresh-target premise. The Plan 14 root read census changes (landRun.release, new landRun.liveNotSealed, the internal/host envelope) and is NOT re-pinned: it waits for the census owner's ruling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 files changed,
+168
−5
approved_moves_copy_test.go+1 −1docs/superpowers/plans/2026-10-06-moves-phase1-copy.md+2 −1docs/superpowers/plans/2026-10-20-offline-operator-move.md+2 −0internal/host/movesoffline_land.go+42 −3internal/host/movesoffline_land_test.go+121 −0
diff --git a/approved_moves_copy_test.go b/approved_moves_copy_test.go| index 36666a1a1..47f478a5b 100644 |
| --- a/approved_moves_copy_test.go |
| +++ b/approved_moves_copy_test.go |
| @@ -104,7 +104,7 @@ var movesCopyIDs = []string{ |
| "MV-O126", "MV-O127", "MV-O128", "MV-O129", "MV-O130", "MV-O131", "MV-O132", "MV-O133", "MV-O134", "MV-O135", "MV-O136", "MV-O137", "MV-O138", "MV-O139", "MV-O140", "MV-O141", |
| "MV-O142", "MV-O143", "MV-O144", "MV-O145", "MV-O146", "MV-O147", "MV-O148", "MV-O149", "MV-O150", "MV-O151", "MV-O152", "MV-O153", "MV-O154", "MV-O155", "MV-O156", "MV-O157", |
| "MV-O158", "MV-O159", "MV-O160", "MV-O161", "MV-O162", "MV-O163", "MV-O164", "MV-O165", "MV-O166", "MV-O167", "MV-O168", "MV-O169", "MV-O170", "MV-O171", |
| - "MV-O172", "MV-O173", "MV-O174", "MV-O175", "MV-O176", |
| + "MV-O172", "MV-O173", "MV-O174", "MV-O175", "MV-O176", "MV-O177", |
| } |
| // copyRow is one row of the record. The File cell is the file the sentence is |
diff --git a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md| index 633163f8b..0d2e8c798 100644 |
| --- a/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| +++ b/docs/superpowers/plans/2026-10-06-moves-phase1-copy.md |
| @@ -610,7 +610,7 @@ The sealer's item reasons, which `carlos move offline status` shows as MV-O55's |
| | MV-O97 | internal/host/movesoffline_seal.go | replication "<rung>" | pending | |
| | MV-O98 | internal/host/movesoffline_seal.go | next due <unix time> | pending | |
| -The landing's item reasons (`internal/host/movesoffline_land.go`, Task 5 fix round 1), which `carlos move offline status` shows as MV-O55's `<detail>` for the target's items. The copy gate reads every Check reason, so each has a row. A reason never carries an error's text: the error behind a reason that waits or fails is the box's log line. `<names>` is the list of variables a failed env pin finds different, each as `<name> (<kind>)` from `moves.CompareEnv`, never a value. MV-O110 to MV-O176. |
| +The landing's item reasons (`internal/host/movesoffline_land.go`, Task 5 fix round 1), which `carlos move offline status` shows as MV-O55's `<detail>` for the target's items. The copy gate reads every Check reason, so each has a row. A reason never carries an error's text: the error behind a reason that waits or fails is the box's log line. `<names>` is the list of variables a failed env pin finds different, each as `<name> (<kind>)` from `moves.CompareEnv`, never a value. MV-O110 to MV-O177. MV-O177 is the `release` item's failure when the target box keeps a live link for the host that names another build than the sealed one (final review, return moves). |
| | id | file | text | status | |
| |---|---|---|---| |
| @@ -681,3 +681,4 @@ The landing's item reasons (`internal/host/movesoffline_land.go`, Task 5 fix rou |
| | MV-O174 | internal/host/movesoffline_land.go | not checked: the row is not held for this move | pending | |
| | MV-O175 | internal/host/movesoffline_land.go | not checked: the seal could not be used | pending | |
| | MV-O176 | internal/host/movesoffline_land.go | the row could not be read back | pending | |
| +| MV-O177 | internal/host/movesoffline_land.go | the target box still holds an older build of this instance. see the offline move runbook | pending | |
diff --git a/docs/superpowers/plans/2026-10-20-offline-operator-move.md b/docs/superpowers/plans/2026-10-20-offline-operator-move.md| index bd14adf70..f45dab1db 100644 |
| --- a/docs/superpowers/plans/2026-10-20-offline-operator-move.md |
| +++ b/docs/superpowers/plans/2026-10-20-offline-operator-move.md |
| @@ -577,6 +577,8 @@ Every step waits for Paul's go. Nothing touches production before P2. |
| **P3. After `open`, if something is wrong.** The target is the only live copy and may have taken writes, so going back is a new offline move from `uk` to the home, which needs UK's staging grant (P0.4). Its landing on the IE box lifts any stale tombstone and finds no warm cache (the source half parked at `open`). If the grant is not in place: fix forward on the UK box, and file the gap. |
| +- A return move's landing fails the `release` item with MV-O177 ("the target box still holds an older build of this instance") when the box the instance left still keeps its live link `/opt/carlos/live/<host>` naming the build it ran then, and the app has been promoted since. Retirement leaves that link because it is root-owned, and nothing in this tool or in ordinary adoption repoints it while the row is held, so there is no safe road to clear it from here: abort the move (safe before `open`), and the instance serves again from where it is. Before starting a return move, read the link on the target box (`readlink /opt/carlos/live/<host>`): if it names a different version from the one the instance runs now, do not start, and take it to Paul. |
| + |
| **Cleanup recipe for an abandoned move** (documented, rarely needed): read `instances/<host>/lock.json` in the source data bucket with `lease.LockState`; if the owner is `moved:<id>` for a move that is no longer in flight, delete it. The 7-day expiry frees it anyway. |
| ## Interaction with the host claim |
diff --git a/internal/host/movesoffline_land.go b/internal/host/movesoffline_land.go| index 487bad3fb..8ac434d5a 100644 |
| --- a/internal/host/movesoffline_land.go |
| +++ b/internal/host/movesoffline_land.go |
| @@ -1171,9 +1171,10 @@ func (r *landRun) replication() { |
| r.ok("replication") |
| } |
| -// release: the app's channel resolves to the build the seal recorded. A |
| -// promote during the move fails it; the build itself arrives after open |
| -// through ordinary adoption. |
| +// release: the app's channel resolves to the build the seal recorded, and |
| +// the target holds no live link to another build of the host |
| +// (liveNotSealed). A promote during the move fails it; the build itself |
| +// arrives after open through ordinary adoption. |
| func (r *landRun) release(ctx context.Context) { |
| cfg := r.cfg() |
| if r.m.Source.Release != "" && r.m.Source.Release != r.seal.Release { |
| @@ -1194,5 +1195,43 @@ func (r *landRun) release(ctx context.Context) { |
| r.landFail("release", "the channel points at another build than the sealed one") |
| return |
| } |
| + if err := r.liveNotSealed(); err != nil { |
| + r.landFailErr("release", "the target box still holds an older build of this instance. see the offline move runbook", err) |
| + return |
| + } |
| r.ok("release") |
| } |
| + |
| +// liveNotSealed is the release item's second half: nil when the target has |
| +// no live link for the host, or when the link names the sealed build. The |
| +// link (<opt>/live/<host>) is the one file ExecBackend.ensureLiveLink and |
| +// the activator exec. A box the instance once left keeps it, root-owned and |
| +// still naming the build it ran then (retirePaths leaves it); a wake before |
| +// ordinary adoption execs whatever it names, since ensureLiveLink accepts |
| +// any executable and default update settings gate no version. So on a |
| +// return move after an upgrade the old build could open the sealed |
| +// database. Any other link fails the item, and open, which verifies every |
| +// item again, never releases the hold over it. No link is fine: nothing can |
| +// run until adoption stages the sealed build. The link is only read here, |
| +// never changed. The error is the box's log line. |
| +func (r *landRun) liveNotSealed() error { |
| + optDir := r.cfg().OptDir |
| + if optDir == "" { |
| + optDir = release.OptDir() |
| + } |
| + link := release.LiveLink(optDir, r.host) |
| + if _, err := os.Lstat(link); errors.Is(err, os.ErrNotExist) { |
| + return nil |
| + } else if err != nil { |
| + return err |
| + } |
| + id, err := release.ResolveLiveLink(optDir, r.host) |
| + if err != nil { |
| + return err |
| + } |
| + want := release.VersionPath(optDir, release.RouteAccount(*r.rt), r.seal.Release, r.rt.App) |
| + if id.Version != r.seal.Release || id.StagedPath != want { |
| + return fmt.Errorf("%s names build %q at %s, the seal recorded %q at %s", link, id.Version, id.StagedPath, r.seal.Release, want) |
| + } |
| + return nil |
| +} |
diff --git a/internal/host/movesoffline_land_test.go b/internal/host/movesoffline_land_test.go| index f97d76596..011d743fa 100644 |
| --- a/internal/host/movesoffline_land_test.go |
| +++ b/internal/host/movesoffline_land_test.go |
| @@ -9,6 +9,8 @@ import ( |
| "encoding/json" |
| "errors" |
| "io/fs" |
| + "net/http" |
| + "net/http/httptest" |
| "os" |
| "path/filepath" |
| "reflect" |
| @@ -779,6 +781,11 @@ func TestOfflineLandFailsEachMismatch(t *testing.T) { |
| func TestOfflineLandOpenReleasesOnlyWhenEveryItemVerifies(t *testing.T) { |
| lw := newLandWorld(t) |
| ctx := context.Background() |
| + // A fresh target: no live link for the host, which the release item |
| + // accepts (nothing can run until adoption stages the sealed build). |
| + if lexists(release.LiveLink(lw.uk.opt, offHost)) { |
| + t.Fatal("fixture: the target already has a live link") |
| + } |
| lw.flipAndLand(1) |
| lw.requireLanded() |
| lw.wd.open() |
| @@ -1135,3 +1142,117 @@ func TestOfflineLandANestedSourceWithLaunchFiles(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// --- the target's own live build (final review, return moves) -------------------- |
| + |
| +const landOlderBuild = "the target box still holds an older build of this instance. see the offline move runbook" |
| + |
| +// stageLive gives the target box a live link for the host naming version v |
| +// of the app, as a box the instance once left keeps it after retirement: a |
| +// staged executable in the version store and the link to it. |
| +func (lw *landWorld) stageLive(v string) string { |
| + t := lw.t |
| + t.Helper() |
| + writeFile(t, release.VersionPath(lw.uk.opt, lw.wd.acct, v, "app"), "#!/bin/sh\nexit 0\n") |
| + if err := release.PointLive(lw.uk.opt, offHost, lw.wd.acct, v, "app"); err != nil { |
| + t.Fatal(err) |
| + } |
| + target, err := os.Readlink(release.LiveLink(lw.uk.opt, offHost)) |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + return target |
| +} |
| + |
| +// requireNoWake: the row is still held landing by the move, the edges file |
| +// does not list it (so the edge never hands it to the activator), and a |
| +// scheduled tick for it, the one wake source that reaches a held row, is |
| +// answered with the maintenance page without reaching the exec backend. |
| +func (lw *landWorld) requireNoWake(when string) { |
| + t := lw.t |
| + t.Helper() |
| + rt := lw.uk.row(offHost) |
| + if rt == nil || rt.Status != "landing" || rt.HoldOp != offID { |
| + t.Fatalf("%s: the target row %+v is not held by the move", when, rt) |
| + } |
| + lw.uk.edges() |
| + if lw.uk.lists(offHost) { |
| + t.Fatalf("%s: the edges file lists the host", when) |
| + } |
| + proxied := false |
| + rec := httptest.NewRecorder() |
| + lw.uk.svc.ServeScheduled(rec, httptest.NewRequest("POST", "http://"+offHost+"/jobs/remind", nil), *rt, |
| + http.HandlerFunc(func(http.ResponseWriter, *http.Request) { proxied = true })) |
| + if proxied || rec.Code != http.StatusServiceUnavailable { |
| + t.Fatalf("%s: a scheduled tick on the target was answered %d (proxied %v)", when, rec.Code, proxied) |
| + } |
| + if rt := lw.uk.row(offHost); rt == nil || rt.Status != "landing" || rt.HoldOp != offID { |
| + t.Fatalf("%s: after the tick the target row is %+v", when, rt) |
| + } |
| +} |
| + |
| +// A return move after an upgrade: the target box is one the instance left |
| +// on an older build, and it still keeps that build's live link (retirement |
| +// leaves the root-owned link). The channel names the sealed build, so only |
| +// the link can tell. The landing fails the release item with MV-O177 and |
| +// every other item lands; the record stays offline-landing; nothing can |
| +// wake the row; the link is left exactly as it was. Open, reached even so, |
| +// verifies the item again and keeps the hold. A link that names the sealed |
| +// build passes, and open then releases the row. |
| +func TestOfflineLandRefusesAnOlderLiveBuildOnTheTarget(t *testing.T) { |
| + lw := newLandWorld(t) |
| + old := lw.stageLive("v0") |
| + lw.flipAndLand(2) |
| + if st := lw.state(); st != provision.OfflineLanding { |
| + t.Fatalf("state %s with an older live build on the target", st) |
| + } |
| + r := lw.report() |
| + if r.Phase != moves.OfflinePhaseFailed || r.Items["release"].OK || r.Items["release"].Reason != landOlderBuild { |
| + t.Fatalf("phase %s, release %+v", r.Phase, r.Items["release"]) |
| + } |
| + for id, c := range r.Items { |
| + if id != "release" && !c.OK { |
| + t.Fatalf("%s not OK: %+v", id, c) |
| + } |
| + } |
| + lw.requireNoWake("after the landing") |
| + if got, err := os.Readlink(release.LiveLink(lw.uk.opt, offHost)); err != nil || got != old { |
| + t.Fatalf("the live link is %q (%v), want it left at %q", got, err, old) |
| + } |
| + |
| + // Open, forced past the door's landed gate, verifies again and refuses. |
| + lw.wd.landed() |
| + lw.wd.open() |
| + lw.uk.tick() |
| + lw.uk.tick() |
| + if r := lw.report(); r.Phase == moves.OfflinePhaseOpened || r.Items["release"].Reason != landOlderBuild { |
| + t.Fatalf("open with an older live build: %s %+v", r.Phase, r.Items["release"]) |
| + } |
| + lw.requireNoWake("at open") |
| + lw.requireNoTargetLease() |
| + |
| + // The same link naming the sealed build: open releases the row. |
| + lw.stageLive(sealVersion) |
| + lw.uk.tick() |
| + if rt := lw.uk.row(offHost); rt == nil || rt.Status != "ready" || rt.HoldOp != "" { |
| + t.Fatalf("not opened with the sealed live build: %+v (%+v)", rt, lw.report().Items) |
| + } |
| + if r := lw.report(); r.Phase != moves.OfflinePhaseOpened { |
| + t.Fatalf("phase %s", r.Phase) |
| + } |
| +} |
| + |
| +// A live link that names another build fails at open too: the landing was |
| +// clean (no link), and the link appears before open. |
| +func TestOfflineLandOpenRefusesALiveBuildThatAppeared(t *testing.T) { |
| + lw := newLandWorld(t) |
| + lw.flipAndLand(1) |
| + lw.requireLanded() |
| + lw.stageLive("v0") |
| + lw.wd.open() |
| + lw.uk.tick() |
| + if r := lw.report(); r.Phase == moves.OfflinePhaseOpened || r.Items["release"].Reason != landOlderBuild { |
| + t.Fatalf("open with another live build: %s %+v", r.Phase, r.Items["release"]) |
| + } |
| + lw.requireNoWake("at open") |
| +} |