amadan
schedule: a source box's tick leaves a held or moved host alone (defect 8)
Harness Run 9, defect 8. After open the source keeps its row, held, until finishing, and open has lifted the fence that used to refuse the scheduled token. So the first every-minute tick after open minted a fresh <host>.env (an admin token) inside the instance directory the move had just parked: a stray secret for a host served from another region, and a non-empty directory a later return move's landing would meet. The schedule runner gains Skip, a host this box must not touch: no token, no delivery, no bookkeeping, no status report. It is asked at the tick's route read and again, after a fresh row read, just before the token, so a hold written in between is seen. The agent sets it to scheduleSkip: any held row (delivery to one is refused by the activator anyway, and a hold whose move has just ended stays a hold until reconcile removes the row), and a host whose offline move's record names another region (offlineView.elsewhere). Every per-instance file writer the scheduler path reaches: the token (EnsureAdminToken, the env file and its directory) is the only one. Delivery to a held row wakes nothing (ServeScheduled answers maintenance), and the converger writes registry rows and the bucket status only. Config sync, instance-key backfill and the layout pass already skip rows that are not ready. TestOfflineMoveE2E now runs the source's real runner after open with a due schedule: no env file at either layout's path, no timer row moves, no store write. Without Skip it writes the env file, as on sophie. Census: overlay rows for newScheduleDeps, scheduleSkip, Runner.Tick and Runner.fire (PROPOSED-moves-offline-fixes); the seal moves.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
9 files changed,
+255
−5
docs/superpowers/notes/census-proposal-moves-offline-fixes.md+28 −3internal/host/movesoffline.go+22 −0internal/host/movesoffline_e2e_test.go+63 −0internal/host/movesoffline_test.go+32 −0internal/host/schedule_wiring.go+24 −0internal/provision/plan14_root_reads_test.go+1 −1internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv+10 −1internal/schedule/runner.go+25 −0internal/schedule/runner_test.go+50 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate === go: downloading github.com/aws/aws-sdk-go-v2 v1.44.0 go: downloading github.com/aws/aws-sdk-go-v2/config v1.32.6 go: downloading github.com/aws/aws-sdk-go-v2/service/sns v1.43.0 go: downloading github.com/superfly/ltx v0.5.1 go: downloading github.com/oschwald/maxminddb-golang v1.13.1 go: downloading github.com/aws/aws-sdk-go-v2/service/autoscaling v1.74.0 go: downloading github.com/aws/aws-sdk-go-v2/service/ec2 v1.324.0 go: downloading github.com/aws/smithy-go v1.28.1 go: downloading golang.org/x/crypto v0.53.0 go: downloading github.com/aws/aws-sdk-go-v2/service/s3 v1.105.0 go: downloading modernc.org/sqlite v1.55.0 go: downloading github.com/benbjohnson/litestream v0.5.14 go: downloading github.com/aws/aws-sdk-go-v2/service/acm v1.45.0 go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.68.0 go: downloading github.com/aws/aws-sdk-go-v2/service/iam v1.58.2 go: downloading github.com/aws/aws-sdk-go-v2/service/resourcegroupstaggingapi v1.37.0 go: downloading github.com/aws/aws-sdk-go-v2/service/route53 v1.64.1 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.45.6 go: downloading golang.org/x/sys v0.46.0 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.19.6 go: downloading github.com/aws/aws-sdk-go-v2/service/ssm v1.74.0 go: downloading github.com/klauspost/compress v1.18.0 go: downloading golang.org/x/net v0.56.0 go: downloading github.com/pion/logging v0.2.4 go: downloading github.com/pion/stun/v3 v3.0.1 go: downloading github.com/pion/turn/v4 v4.1.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sesv2 v1.66.6 go: downloading github.com/aws/aws-sdk-go-v2/service/costexplorer v1.67.5 go: downloading github.com/pierrec/lz4/v4 v4.1.22 go: downloading github.com/prometheus/client_golang v1.17.0 go: downloading golang.org/x/sync v0.21.0 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.18.16 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.8.4 go: downloading github.com/aws/aws-sdk-go-v2/service/signin v1.0.4 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.30.8 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.12 go: downloading github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.18 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.4.40 go: downloading github.com/aws/aws-sdk-go-v2/internal/v4a v1.4.38 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.19 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.13.40 go: downloading github.com/lmittmann/tint v1.1.3 go: downloading github.com/mattn/go-isatty v0.0.20 go: downloading github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.14 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.9.23 go: downloading github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.31 go: downloading github.com/pion/dtls/v3 v3.0.7 go: downloading github.com/pion/transport/v3 v3.0.8 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.7.40 go: downloading github.com/wlynxg/anet v0.0.5 go: downloading github.com/pion/randutil v0.1.0 go: downloading github.com/pion/transport/v4 v4.0.1 go: downloading github.com/prometheus/client_model v0.5.0 go: downloading github.com/prometheus/procfs v0.12.0 go: downloading google.golang.org/protobuf v1.36.10 go: downloading github.com/cespare/xxhash/v2 v2.3.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/prometheus/common v0.45.0 go: downloading github.com/matttproud/golang_protobuf_extensions/v2 v2.0.0 go: downloading golang.org/x/text v0.38.0 go: downloading modernc.org/libc v1.74.1 go: downloading modernc.org/memory v1.11.0 go: downloading github.com/dustin/go-humanize v1.0.1 go: downloading github.com/google/uuid v1.6.0 go: downloading modernc.org/mathutil v1.7.1 go: downloading github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec cd launchctx && go vet ./... && go test ./... -count=1 ok amadan.net/carlos/platform.git/launchctx 0.012s ? amadan.net/carlos/platform.git/launchctx/controlname [no test files] ? amadan.net/carlos/platform.git/launchctx/wireid [no test files] go vet ./... go: downloading golang.org/x/tools v0.47.0 go: downloading golang.org/x/mod v0.37.0 gofmt -l .: clean go test ./... -count=1 -timeout 30m amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/notes/census-proposal-moves-offline-fixes.md b/docs/superpowers/notes/census-proposal-moves-offline-fixes.md| index b9c9f20fb..033cc16d1 100644 |
| --- a/docs/superpowers/notes/census-proposal-moves-offline-fixes.md |
| +++ b/docs/superpowers/notes/census-proposal-moves-offline-fixes.md |
| @@ -45,8 +45,33 @@ census-ruling-moves-offline already classified (`offlineSourceStop`, |
| refusal (MV-O178) reads nothing new. Envelopes re-pinned: `internal/console`, |
| `internal/host`. |
| +### Defect 8: the source's scheduler wrote the moved host's env file after open |
| + |
| +The schedule runner is built only by the agent (`armSchedule`, from |
| +`runAgent`); no root process makes one. |
| + |
| +| unit | change | executing identity | disposition | |
| +|---|---|---|---| |
| +| `internal/host/schedule_wiring.go` `newScheduleDeps` | changed legacy: the runner's `Skip` is set to `scheduleSkip`; no new read site | the agent | re-frozen, J1 | |
| +| `internal/host/schedule_wiring.go` `scheduleSkip` | new: reads a row's status (any hold) and host (the armed offline view says the record names another region) to tell the runner to leave the host alone; it only withholds a tick | the agent's schedule runner | accepted legacy, J1 (as the runner's own hold checks) | |
| +| `internal/schedule/runner.go` `Runner.Tick` | changed legacy: a host `Skip` names gets no tick and no bookkeeping at the pre-filter | the agent's schedule runner | re-frozen at census-ruling-seeded-instances' disposition | |
| +| `internal/schedule/runner.go` `Runner.fire` | re-pinned: with `Skip` set, the row is read again before the token, and a skipped host gets no token and no record | the agent's schedule runner (overlay e names the service controller and root callbacks for the claim design; neither sets `Skip`) | hint, E2, unchanged | |
| + |
| +Envelopes re-pinned: `internal/host`, `internal/schedule`. |
| + |
| ## The seal |
| -The accepted-legacy rows add frozen declarations, so the legacy seal moves. |
| -The test pins the new value after the overlay loads; main's seal is still |
| -checked first, so a change on main's side shows on its own. |
| +The accepted-legacy rows add frozen declarations, so the legacy seal moves, |
| +from `393e7f45…bec29` (main, after moves-offline and armed-box-updates) to |
| +`0d4e20c578532f1ca600de8569e1a46036bd959c14efbe879fb8bca47dcf4568`. The test |
| +checks main's seal before it loads this overlay, so a change on main's side |
| +still shows on its own. |
| + |
| +## What the ruling is asked to decide |
| + |
| +1. That every unit above runs only in the agent and makes no root decision, |
| + so accepted legacy at main's disposition is the right footing. |
| +2. That `Runner.fire` keeps its class and task (hint, E2): the change only |
| + withholds a token and a record for a host `Skip` names. |
| +3. That the re-assert of defect 7 needs no row: it calls only steps already |
| + classified under census-ruling-moves-offline. |
diff --git a/internal/host/movesoffline.go b/internal/host/movesoffline.go| index baa6202d8..9138cfdc5 100644 |
| --- a/internal/host/movesoffline.go |
| +++ b/internal/host/movesoffline.go |
| @@ -213,6 +213,28 @@ func (v *offlineView) keep(host string) bool { |
| return false |
| } |
| +// elsewhere reports whether host's offline move has its record name a region |
| +// other than this box's: the target's from landing to finishing, the source's |
| +// while reverting. Such a host is not this box's to touch whatever its row |
| +// says (scheduleSkip). A view with no listing yet, and a host with no |
| +// offline move, answer false: the row's own hold answers for those. |
| +func (v *offlineView) elsewhere(host string) bool { |
| + snap := v.snap.Load() |
| + if snap == nil || snap.failed || v.region == "" { |
| + return false |
| + } |
| + m, ok := snap.moves[host] |
| + if !ok { |
| + return false |
| + } |
| + region := m.From |
| + switch m.State { |
| + case provision.OfflineLanding, provision.OfflineLanded, provision.OfflineOpen, provision.OfflineFinishing: |
| + region = m.To |
| + } |
| + return !park.PlacementMatches(v.home, region, v.region) |
| +} |
| + |
| // refreshOfflineView reads the record list once, outside a moves pass: at |
| // agent start, before the reconcile loops, next to refreshMovesHolds. A |
| // failed listing leaves keep failing closed until a listing is read. |
diff --git a/internal/host/movesoffline_e2e_test.go b/internal/host/movesoffline_e2e_test.go| index 398b87e3a..ee87816cb 100644 |
| --- a/internal/host/movesoffline_e2e_test.go |
| +++ b/internal/host/movesoffline_e2e_test.go |
| @@ -35,6 +35,7 @@ import ( |
| "github.com/carlosframework/platform/internal/provision" |
| "github.com/carlosframework/platform/internal/registry" |
| "github.com/carlosframework/platform/internal/release" |
| + "github.com/carlosframework/platform/internal/schedule" |
| "github.com/carlosframework/platform/internal/status" |
| ) |
| @@ -1270,6 +1271,7 @@ func offlineMoveDone(t *testing.T, opts landOpts) { |
| t.Fatal("ie-box's row went before finishing") |
| } |
| w.requireTombstoned("after open") |
| + w.requireSourceSchedulerQuiet() |
| // The writes so far, before the first wake (which is not the tool's). |
| w.requireClosedWriteSet("through open") |
| w.firstWake() |
| @@ -1310,6 +1312,67 @@ func offlineMoveDone(t *testing.T, opts landOpts) { |
| held.Release(ctx) |
| } |
| +// requireSourceSchedulerQuiet is harness Run 9's defect 8. After open the |
| +// source keeps its row, held, until finishing, its timers with it, and open |
| +// has lifted the fence. The source's own schedule runner, built as the agent |
| +// builds it (newScheduleDeps over ie-box's registry, the deployment store, |
| +// its activator and data directory), ticks with every timer due: it writes |
| +// nothing for the host. No env file at either layout's path (the tick used |
| +// to mint one, a fresh admin token, in the parked instance directory), no |
| +// timer row moves, and no store write. |
| +func (w *e2eWorld) requireSourceSchedulerQuiet() { |
| + t := w.t |
| + t.Helper() |
| + ie := w.ie |
| + if ie.svc.Fenced(offHost) { |
| + t.Fatal("fixture: ie-box is still fenced after open") |
| + } |
| + // Run 9's app declared a one-minute schedule; the row is the source's |
| + // as the converger writes it, due now. (The fixture's own timers are |
| + // the seal's, kept for the digest, and are not all ones the runner can |
| + // fire.) |
| + if err := ie.reg.UpsertTimer(registry.Timer{Host: offHost, Name: "every-minute", Kind: schedule.KindSchedule, Spec: "every 1m0s", Path: "/tick", NextAt: 1}); err != nil { |
| + t.Fatal(err) |
| + } |
| + before, err := ie.reg.TimersFor(offHost) |
| + if err != nil || len(before) == 0 { |
| + t.Fatalf("fixture: ie-box keeps no timers for the host after open (%v)", err) |
| + } |
| + mark := w.rs.mark() |
| + sd := newScheduleDeps(ie.reg, w.rs, ie.svc, nil, ie.dd) |
| + sd.runner.Tick(context.Background()) |
| + // A delivery the tick started holds the host's claim until it ends. |
| + for deadline := time.Now().Add(10 * time.Second); sd.runner.Busy(offHost); { |
| + if time.Now().After(deadline) { |
| + t.Fatal("the source's tick did not finish") |
| + } |
| + time.Sleep(5 * time.Millisecond) |
| + } |
| + for _, p := range []string{ |
| + filepath.Join(ie.dd, offHost+".env"), |
| + filepath.Join(provision.InstanceDir(ie.dd, offHost), offHost+".env"), |
| + filepath.Join(layout.RawInstanceDir(ie.dd, offHost), offHost+".env"), |
| + } { |
| + if lexists(p) { |
| + t.Fatalf("the source's scheduler wrote %s after open", p) |
| + } |
| + } |
| + after, err := ie.reg.TimersFor(offHost) |
| + // The runner's daily sweep of finished once-rows is box-wide and is not |
| + // a tick of the host (the timers digest leaves that history out, joint |
| + // N14): only the live rows are compared. |
| + live := func(ts []registry.Timer) []registry.Timer { |
| + return slices.DeleteFunc(slices.Clone(ts), func(tm registry.Timer) bool { return tm.Kind == "once" && tm.NextAt == 0 }) |
| + } |
| + before, after = live(before), live(after) |
| + if err != nil || !reflect.DeepEqual(after, before) { |
| + t.Fatalf("the source's tick moved the host's timers (%v):\nbefore %+v\nafter %+v", err, before, after) |
| + } |
| + if ops := w.rs.since(mark); len(ops) != 0 { |
| + t.Fatalf("the source's tick wrote to the store: %v", ops) |
| + } |
| +} |
| + |
| // requireRetiredAfterFinish: finishing removed the row and nothing parked |
| // came back or went. |
| func (w *e2eWorld) requireRetiredAfterFinish(at moves.OfflineReport) { |
diff --git a/internal/host/movesoffline_test.go b/internal/host/movesoffline_test.go| index 368d0e5f4..b44a184d9 100644 |
| --- a/internal/host/movesoffline_test.go |
| +++ b/internal/host/movesoffline_test.go |
| @@ -1358,3 +1358,35 @@ func TestOfflineReassertRewritesOnlyTheReportsADoorReads(t *testing.T) { |
| t.Fatalf("open: the source's report was rewritten at %v", r.At) |
| } |
| } |
| + |
| +// elsewhere, scheduleSkip's second half (harness Run 9, defect 8): the box |
| +// whose region the record does not name, through every state the record |
| +// moves through. |
| +func TestOfflineElsewhereFollowsTheRecordsRegion(t *testing.T) { |
| + wd := newOffWorld(t) |
| + a := wd.box("ie-a", "", nil) |
| + uk := wd.box("uk-box", "uk", nil) |
| + a.tick() |
| + uk.tick() |
| + check := func(when string, ieElsewhere, ukElsewhere bool) { |
| + t.Helper() |
| + a.pass() |
| + uk.pass() |
| + if got := a.w.cfg.Offline.elsewhere(offHost); got != ieElsewhere { |
| + t.Fatalf("%s: ie-a elsewhere = %v", when, got) |
| + } |
| + if got := uk.w.cfg.Offline.elsewhere(offHost); got != ukElsewhere { |
| + t.Fatalf("%s: uk-box elsewhere = %v", when, got) |
| + } |
| + } |
| + check("no move", false, false) |
| + wd.start() |
| + check("stopping", false, true) |
| + wd.seal() |
| + wd.sealed() |
| + check("sealed", false, true) |
| + wd.flip() |
| + check("landing", true, false) |
| + wd.abort() |
| + check("reverting", false, true) |
| +} |
diff --git a/internal/host/schedule_wiring.go b/internal/host/schedule_wiring.go| index b89e0dcf7..8bcd4dcd3 100644 |
| --- a/internal/host/schedule_wiring.go |
| +++ b/internal/host/schedule_wiring.go |
| @@ -87,6 +87,7 @@ func newScheduleDeps(reg *registry.Registry, store release.Store, svc *activator |
| Reg: reg, |
| Deliver: &activatorDeliverer{svc: svc}, |
| Token: scheduleToken(dd, svc, reg), |
| + Skip: scheduleSkip, |
| Event: event, |
| Box: box, |
| Build: version.Build, |
| @@ -145,6 +146,29 @@ func armSchedule(reg *registry.Registry, store release.Store, storeErr error, sv |
| } |
| } |
| +// scheduleSkip is the runner's Skip (harness Run 9, defect 8): a host whose |
| +// row this box holds, and a host whose offline move's record names another |
| +// region. After open the source keeps its row, held, until finishing, and |
| +// open has lifted the fence that used to refuse the token; so the source's |
| +// tick minted a fresh <host>.env (an admin token) inside the instance |
| +// directory the move had just parked, a stray secret for a host served |
| +// elsewhere, and recorded a delivery the activator refuses for a held row |
| +// anyway (ServeScheduled). The token is the only instance file the runner |
| +// writes; delivery wakes nothing for a held row, and the converger writes |
| +// registry rows and the bucket status only. Any hold counts, not only an |
| +// offline move's: no held row is ever served a tick, and a hold whose move |
| +// has just ended is still a hold until reconcile removes or releases the |
| +// row. |
| +func scheduleSkip(rt registry.Route) bool { |
| + if registry.IsHold(rt.Status) { |
| + return true |
| + } |
| + if v := armedOfflineView(); v != nil { |
| + return v.elsewhere(rt.Host) |
| + } |
| + return false |
| +} |
| + |
| // scheduleToken is the runner's Token: ensure-then-read on the activator's |
| // own <dataDir>/<host>.env (spec §6.4). |
| // |
diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go| index 40809ef76..9ad0f4987 100644 |
| --- a/internal/provision/plan14_root_reads_test.go |
| +++ b/internal/provision/plan14_root_reads_test.go |
| @@ -996,7 +996,7 @@ func TestPlan14RootReadCensus(t *testing.T) { |
| if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv"), &inv, true); err != nil { |
| t.Fatal(err) |
| } |
| - seal = "247e7d2d5e9213e08e2d631d3416c2fe2ca3044f9ea8c70d3e515139a822d007" |
| + seal = "0d4e20c578532f1ca600de8569e1a46036bd959c14efbe879fb8bca47dcf4568" |
| errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...) |
| errs = append(errs, checkRootReadDeclarations(got, inv)...) |
| if len(errs) > 0 { |
diff --git a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv| index 4ff5acd9f..2cb302bf0 100644 |
| --- a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv |
| +++ b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv |
| @@ -2,9 +2,11 @@ |
| # Every unit here runs in the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos) and makes no root decision, so each is proposed accepted legacy at main's disposition (J1 root-route for registry and route reads, J4 root-input for file reads), as census-ruling-moves-offline proposed its units. |
| # Defect 6 (config.env for a nested row whose instance directory is not made yet): SyncExecWithCallbacks, renderExecConfigBefore and configGenLoop re-frozen at main's disposition; configGenTick and pendingDirs.appeared new. The seal moves. |
| # Defect 7 (fresh reports at every door step): envelope only. The source and sealer re-assert (movesWorker.offlineReassert, offlineReassertLoop) reads no root input of its own: it reads the record from the deployment store and calls the steps census-ruling-moves-offline already classified. The console's stale-report refusal reads nothing new. internal/console and internal/host re-pinned; the seal stands. |
| +# Defect 8 (the source's scheduler wrote the moved host's env file after open): Runner.Tick and newScheduleDeps re-frozen at their dispositions, scheduleSkip new (J1), Runner.fire re-pinned at its class and task (hint, E2: overlay e's reclassification); internal/host and internal/schedule re-pinned. The seal moves. |
| @coverage internal/appconfig 2c32778cc777f026f1dd7b6404b11e3fabf1b6bd4912799d6ef26bfb59d42a07 |
| @coverage internal/console 7b015747c5ef8a3ee7473ad63f7f99200ce14c63c1b7a47945c212ad9239756b |
| -@coverage internal/host 78443272fea18130457dd94f7c970ee8b1e82f110d9d1bba46e48500aaf6d283 |
| +@coverage internal/host d1a6f949230f617898b8f308ee1f458b7ab3f396f8095885fb997ead8728f455 |
| +@coverage internal/schedule fb5b5ecb590bcbb4cef33b2e884ad0f676aadbf2e23eebf82a8918342bffdbc3 |
| @legacy 783751049991bdd17a140bba6ce9a3049a4075b1170e4882c7207b20bed24b41 internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f |
| internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f |
| @legacy 356c576628a285021657e989a015ef938bb1da3498d8146a734903a23fea1e4a internal/appconfig/syncexec.go renderExecConfigBefore legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from config sync and the moves worker's held render: main's body with one Lstat of a nested row's instance directory after its write failed, to tell a directory not made yet from any other failure; it follows no link and changes no outcome but the error's kind file:Lstat@3893,file:ReadFile@951,registry:Environment@3190,registry:Host@1070,registry:Host@2586,registry:Host@4088,registry:Host@4320,registry:Host@4576,registry:Host@4669,registry:Host@876,whole:Route@2522,whole:Route@257,whole:Route@3715,whole:Route@680 a60a9a2e2158f3a8dadf6fcf46810930dad89b67d1acc23fb57c324a7d9ac1f1 |
| @@ -15,3 +17,10 @@ internal/host/agent.go configGenLoop legacy accepted-legacy accepted-legacy;rev3 |
| internal/host/agent.go configGenTick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as configGenLoop); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): new, configGenLoop's tick over the routes it listed; it also runs the agent's own config pass once an instance directory the last pass waited for exists whole:Route@268 751870d636c3ea7b3670a7e00c71d085e94ae9b013545e319047a4609f514943 |
| @legacy 237af935c475d587a83b4e7a33d21e1f3fd32f36333a8440b9f3b44b21b8509e internal/host/agent.go pendingDirs.appeared legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-input (J4, as the moves units' isDir and lexists); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): new, an Lstat that never follows a link, of the instance directories the last config pass could not write into; it decides only whether the agent runs its own config pass again file:Lstat@137 bc87759cf499661d1058712b03230924e1669a8a1710b88e4053c8b4f7bd3aa3 |
| internal/host/agent.go pendingDirs.appeared legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-input (J4, as the moves units' isDir and lexists); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos): new, an Lstat that never follows a link, of the instance directories the last config pass could not write into; it decides only whether the agent runs its own config pass again file:Lstat@137 bc87759cf499661d1058712b03230924e1669a8a1710b88e4053c8b4f7bd3aa3 |
| +@legacy fb3a335703bd3d6008ff82252d64aa9f41676650a874eb786291d1d201e57910 internal/host/schedule_wiring.go newScheduleDeps legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from armSchedule: main's body with the runner's Skip set to scheduleSkip; no new read site registry:App@1479,registry:App@388,registry:Host@1546,registry:Host@396,whole:Route@1474,whole:Route@383 aa93bed2781d1005cf864ebaaf6ac4597d6f547c45d6a1063f10c4a483ba46c3 |
| +internal/host/schedule_wiring.go newScheduleDeps legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from armSchedule: main's body with the runner's Skip set to scheduleSkip; no new read site registry:App@1479,registry:App@388,registry:Host@1546,registry:Host@396,whole:Route@1474,whole:Route@383 aa93bed2781d1005cf864ebaaf6ac4597d6f547c45d6a1063f10c4a483ba46c3 |
| +@legacy 48af22e2b896b940598175e9c4988e2837b14a72e88b7ee1ba8346098f209606 internal/host/schedule_wiring.go scheduleSkip legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as the runner's own hold checks); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its schedule runner: new, reads a row's status (any hold) and host (the armed offline view: the record names another region) to tell the runner to leave the host alone; it only withholds a tick, never grants one registry:Host@155,registry:Status@64 4f471727429b5b79bfa909693a09e588051b9f490eafd8480ee9893581362d64 |
| +internal/host/schedule_wiring.go scheduleSkip legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as the runner's own hold checks); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its schedule runner: new, reads a row's status (any hold) and host (the armed offline view: the record names another region) to tell the runner to leave the host alone; it only withholds a tick, never grants one registry:Host@155,registry:Status@64 4f471727429b5b79bfa909693a09e588051b9f490eafd8480ee9893581362d64 |
| +@legacy 826058c66c70c6601b64db3044629013a839aed8c9383a6c24c9a86a2772ebd0 internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter; no root decision registry:Host@1184,registry:Host@1226,registry:Host@1247,registry:Host@1271,registry:Host@5289,registry:ImportGate@2801,registry:ImportGate@3716,registry:Status@2594,registry:Status@3702,whole:Route@1653,whole:Route@1943,whole:Route@2072,whole:Route@3482,whole:Route@3673,whole:Route@4806,whole:Route@5155,whole:Route@5330,whole:Route@5366,whole:Timer@1051,whole:Timer@1155,whole:Timer@1177,whole:Timer@1240,whole:Timer@1264,whole:Timer@1280,whole:Timer@4481,whole:Timer@4810,whole:Timer@5370,whole:Timer@820,whole:Timer@902 9892ec7759802c53a4f372a615110f994f2c0697b3353b483aa9a37dbb9d3649 |
| +internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter; no root decision registry:Host@1184,registry:Host@1226,registry:Host@1247,registry:Host@1271,registry:Host@5289,registry:ImportGate@2801,registry:ImportGate@3716,registry:Status@2594,registry:Status@3702,whole:Route@1653,whole:Route@1943,whole:Route@2072,whole:Route@3482,whole:Route@3673,whole:Route@4806,whole:Route@5155,whole:Route@5330,whole:Route@5366,whole:Timer@1051,whole:Timer@1155,whole:Timer@1177,whole:Timer@1240,whole:Timer@1264,whole:Timer@1280,whole:Timer@4481,whole:Timer@4810,whole:Timer@5370,whole:Timer@820,whole:Timer@902 9892ec7759802c53a4f372a615110f994f2c0697b3353b483aa9a37dbb9d3649 |
| +internal/schedule/runner.go Runner.fire hint E2 accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (main: J1) PROPOSED E46: captured claim selection and root journal mutation; root RPC outside service host exclusion; SQL/route data never grants root authority. Executing identity: service controller; root for inventory/retirement callbacks. moves-offline-fixes: with Skip set (only the agent's runner sets it, scheduleSkip), the row is read again before the token and a skipped host gets no token and no record; class and task unchanged registry:Host@243,registry:Host@385,registry:Host@691,registry:Host@837,registry:Name@252,registry:Name@394,registry:Spec@148,registry:Spec@260,whole:Route@1005,whole:Route@1202,whole:Route@1504,whole:Route@1567,whole:Route@1735,whole:Route@662,whole:Route@742,whole:Route@768,whole:Route@879,whole:Timer@1009,whole:Timer@108,whole:Timer@1206,whole:Timer@1306,whole:Timer@1508,whole:Timer@1739,whole:Timer@317,whole:Timer@883 1aaf0e167a6852ab6958cb409aaac63523334b4595e1cb8a0e30ac96f4499e9d |
diff --git a/internal/schedule/runner.go b/internal/schedule/runner.go| index f340d5d24..e69324eff 100644 |
| --- a/internal/schedule/runner.go |
| +++ b/internal/schedule/runner.go |
| @@ -70,6 +70,15 @@ type Runner struct { |
| // inside fire for the rows that get that far, so a tick delivers only |
| // when the hold and the admission both allow it. |
| Gate *HostGate |
| + // Skip names a host this box must not touch at all, whatever the hold |
| + // wiring says: its due timers are neither delivered nor written, no |
| + // token is minted for it and nothing is reported. The agent sets it for |
| + // a held row and for a host whose offline move's record names another |
| + // region (harness Run 9, defect 8: after open, the source's tick minted |
| + // a fresh <host>.env in the instance directory the move had parked). |
| + // Asked at the tick's route read and again just before the token, so a |
| + // hold written in between is seen. Nil: none is skipped. |
| + Skip func(rt registry.Route) bool |
| // beforeClaim is the tests' pause between a tick's route read and its |
| // claim (TestDrainRaceCannotAdvanceBookkeeping). Nil in the agent. |
| @@ -187,6 +196,9 @@ func (r *Runner) Tick(ctx context.Context) { |
| // Converge prunes these; until it does, they are not ours to fire. |
| continue |
| } |
| + if r.Skip != nil && r.Skip(*rt) { |
| + continue // not this box's host to touch: no bookkeeping either |
| + } |
| // A held row (moves spec §5.2, §9.4 "frozen") is skipped with no |
| // bookkeeping at all: before this, its delivery was refused by the |
| // activator and recorded as a retry, so a held row's attempts |
| @@ -498,6 +510,19 @@ func (r *Runner) fire(ctx context.Context, rt registry.Route, t registry.Timer) |
| } |
| return |
| } |
| + if r.Skip != nil { |
| + // The token is the one write a tick makes on the instance's own |
| + // ground, so the row is read again right before it. |
| + var current *registry.Route |
| + err := withTimerWork(ctx, func() error { |
| + var readErr error |
| + current, readErr = r.Reg.Get(rt.Host) |
| + return readErr |
| + }) |
| + if err != nil || current == nil || r.Skip(*current) { |
| + return |
| + } |
| + } |
| start := r.now() |
| token, err := r.Token(rt.Host) |
| if err != nil { |
diff --git a/internal/schedule/runner_test.go b/internal/schedule/runner_test.go| index 155e2cb39..d333b2c6d 100644 |
| --- a/internal/schedule/runner_test.go |
| +++ b/internal/schedule/runner_test.go |
| @@ -1400,3 +1400,53 @@ func TestGatedTimersDoNotStarveAnotherHost(t *testing.T) { |
| }) |
| } |
| } |
| + |
| +// Skip (harness Run 9, defect 8): a host the agent says not to touch, here a |
| +// held row with the move-hold wiring off (an offline move's source after |
| +// open), gets no token, no delivery and no bookkeeping. And a hold written |
| +// between the tick's route read and the fire is seen before the token, the |
| +// one write a tick makes on the instance's own ground. |
| +func TestSkipMintsNoTokenAndRecordsNothing(t *testing.T) { |
| + d := &fakeDeliverer{status: 204} |
| + reg, r := runnerFixture(t, d) |
| + now := at("2026-10-20T17:27:04Z") |
| + r.Now = func() time.Time { return now } |
| + tokens := 0 |
| + r.Token = func(string) (string, error) { tokens++; return "tok", nil } |
| + r.Skip = func(rt registry.Route) bool { return registry.IsHold(rt.Status) } |
| + if err := reg.UpsertTimer(registry.Timer{Host: holdHost, Name: "every-minute", Kind: KindSchedule, Spec: "every 1m0s", Path: "/tick", NextAt: now.Unix() - 5}); err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := reg.SetHold(holdHost, "moving", "mv_a"); err != nil { |
| + t.Fatal(err) |
| + } |
| + before := timersOf(t, reg, holdHost) |
| + r.tick() |
| + if tokens != 0 || len(d.calls) != 0 || !sameTimers(timersOf(t, reg, holdHost), before) || r.Busy(holdHost) { |
| + t.Fatalf("a skipped host: %d tokens, %d deliveries, timers %+v (were %+v)", tokens, len(d.calls), timersOf(t, reg, holdHost), before) |
| + } |
| + |
| + // The row reads ready at the tick's read; the hold lands before the claim. |
| + if ok, err := reg.ReleaseHold(holdHost, "mv_a"); err != nil || !ok { |
| + t.Fatalf("release hold = %v %v", ok, err) |
| + } |
| + r.beforeClaim = func(host string) { |
| + if err := reg.SetHold(host, "moving", "mv_b"); err != nil { |
| + t.Error(err) |
| + } |
| + } |
| + r.tick() |
| + if tokens != 0 || len(d.calls) != 0 || !sameTimers(timersOf(t, reg, holdHost), before) { |
| + t.Fatalf("a hold written before the fire: %d tokens, %d deliveries", tokens, len(d.calls)) |
| + } |
| + |
| + // Ready again, nothing skipped: the tick fires as it always has. |
| + if ok, err := reg.ReleaseHold(holdHost, "mv_b"); err != nil || !ok { |
| + t.Fatalf("release hold = %v %v", ok, err) |
| + } |
| + r.beforeClaim = nil |
| + r.tick() |
| + if tokens != 1 || len(d.calls) != 1 { |
| + t.Fatalf("a ready host: %d tokens, %d deliveries", tokens, len(d.calls)) |
| + } |
| +} |