carlos / platform Public

schedule: a skipped host's due rows no longer fill the runner's batch (fix round 1, finding 1)

Astra's review, HIGH. With the hold wiring off, as on an offline move's
boxes, Tick read DueTimers (64 rows) and applied Skip after the limit, so a
moving host with 64 earlier due timers filled every batch and starved every
other instance's timers.

Tick's query is now Runner.dueRows: with Skip set, held hosts are left out
by the query (DueUnheldTimers, as with the hold wiring armed), and any host
Skip names on other grounds is added to the exclusion and the batch read
again, so a skipped host's rows never take a slot. Runner.fire is unchanged.

Regression: Gate nil, 64 due rows of a skipped host (held, and not held but
named by Skip) and one of an ordinary host; the ordinary one fires on the
first tick. It fails with the old query.

Census: Runner.Tick re-pinned, Runner.dueRows new (accepted legacy, J1),
internal/schedule envelope, seal d3b7eb06.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

claude 4 days ago 005d8037182f8eac8fd3349594a1e061b9734385 parent 574abf7

4 files changed, +93 −11

internal/provision/plan14_root_reads_test.go +1 −1
internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv +6 −3
internal/schedule/runner.go +44 −7
internal/schedule/runner_test.go +42 −0

diff --git a/internal/provision/plan14_root_reads_test.go b/internal/provision/plan14_root_reads_test.go

index 9ad0f4987..080727663 100644

--- a/internal/provision/plan14_root_reads_test.go

+++ b/internal/provision/plan14_root_reads_test.go

@@ -996,7 +996,7 @@ func TestPlan14RootReadCensus(t *testing.T) {

if err = loadRootReadInventory(filepath.Join(root, "internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv"), &inv, true); err != nil {

t.Fatal(err)

}

- seal = "0d4e20c578532f1ca600de8569e1a46036bd959c14efbe879fb8bca47dcf4568"

+ seal = "d3b7eb063b214441776cc5381cf44dba91aabce4466e990e685f8fc29c26e9a5"

errs := append(compareRootReads(got, inv), checkRootReadLegacy(inv, seal)...)

errs = append(errs, checkRootReadDeclarations(got, inv)...)

if len(errs) > 0 {

diff --git a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

index 2cb302bf0..37a9f1480 100644

--- a/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

+++ b/internal/provision/testdata/plan14-root-reads-moves-offline-fixes.tsv

@@ -3,10 +3,11 @@

# Defect 6 (config.env for a nested row whose instance directory is not made yet): SyncExecWithCallbacks, renderExecConfigBefore and configGenLoop re-frozen at main's disposition; configGenTick and pendingDirs.appeared new. The seal moves.

# Defect 7 (fresh reports at every door step): envelope only. The source and sealer re-assert (movesWorker.offlineReassert, offlineReassertLoop) reads no root input of its own: it reads the record from the deployment store and calls the steps census-ruling-moves-offline already classified. The console's stale-report refusal reads nothing new. internal/console and internal/host re-pinned; the seal stands.

# Defect 8 (the source's scheduler wrote the moved host's env file after open): Runner.Tick and newScheduleDeps re-frozen at their dispositions, scheduleSkip new (J1), Runner.fire re-pinned at its class and task (hint, E2: overlay e's reclassification); internal/host and internal/schedule re-pinned. The seal moves.

+# Fix round 1, finding 1 (Astra: skipped timers could starve other hosts): Runner.Tick re-pinned at its re-frozen disposition (its due query moved into dueRows); Runner.dueRows new, accepted legacy (J1, as Runner.Tick): held hosts excluded in the query and hosts Skip names paged past before the limit; internal/schedule re-pinned. The seal moves.

@coverage internal/appconfig 2c32778cc777f026f1dd7b6404b11e3fabf1b6bd4912799d6ef26bfb59d42a07

@coverage internal/console 7b015747c5ef8a3ee7473ad63f7f99200ce14c63c1b7a47945c212ad9239756b

@coverage internal/host d1a6f949230f617898b8f308ee1f458b7ab3f396f8095885fb997ead8728f455

-@coverage internal/schedule fb5b5ecb590bcbb4cef33b2e884ad0f676aadbf2e23eebf82a8918342bffdbc3

+@coverage internal/schedule a76efa679b9089eff4f6dae8e6f0adc5d92de628e66e16d4a00c81ba70e086ff

@legacy 783751049991bdd17a140bba6ce9a3049a4075b1170e4882c7207b20bed24b41 internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f

internal/appconfig/syncexec.go SyncExecWithCallbacks legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), through syncExecSerialized: main's body, with a nested row whose write found no instance directory named and skipped (errInstanceDirAbsent, WhenInstanceDirAbsent) instead of failing the pass; no new read site file:Stat@28119,registry:App@10156,registry:App@10371,registry:App@15749,registry:App@28004,registry:Environment@10164,registry:Environment@10379,registry:Environment@15757,registry:Environment@28012,registry:Host@1031,registry:Host@1460,registry:Host@15584,registry:Host@15682,registry:Host@16514,registry:Host@16865,registry:Host@17800,registry:Host@18341,registry:Host@21268,registry:Host@21485,registry:Host@22043,registry:Host@22233,registry:Host@22242,registry:Host@22261,registry:Host@25162,registry:Host@25779,registry:Host@25831,registry:Host@28513,registry:Host@831,registry:Host@902,registry:Host@996,whole:Route@10049,whole:Route@10110,whole:Route@10151,whole:Route@10268,whole:Route@10302,whole:Route@10366,whole:Route@1412,whole:Route@1441,whole:Route@15193,whole:Route@15243,whole:Route@15744,whole:Route@24758,whole:Route@25172,whole:Route@25876,whole:Route@27903,whole:Route@27936,whole:Route@27999,whole:Route@28100,whole:Route@28750,whole:Route@28823,whole:Route@693,whole:Route@721,whole:Route@750,whole:Route@776,whole:Route@9984 8d58864244e2a823103fab97653d48b7d586843eee5fcb5848194447c2f0210f

@legacy 356c576628a285021657e989a015ef938bb1da3498d8146a734903a23fea1e4a internal/appconfig/syncexec.go renderExecConfigBefore legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=route-and-input (J1,J4, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from config sync and the moves worker's held render: main's body with one Lstat of a nested row's instance directory after its write failed, to tell a directory not made yet from any other failure; it follows no link and changes no outcome but the error's kind file:Lstat@3893,file:ReadFile@951,registry:Environment@3190,registry:Host@1070,registry:Host@2586,registry:Host@4088,registry:Host@4320,registry:Host@4576,registry:Host@4669,registry:Host@876,whole:Route@2522,whole:Route@257,whole:Route@3715,whole:Route@680 a60a9a2e2158f3a8dadf6fcf46810930dad89b67d1acc23fb57c324a7d9ac1f1

@@ -21,6 +22,8 @@ internal/host/agent.go pendingDirs.appeared legacy accepted-legacy accepted-lega

internal/host/schedule_wiring.go newScheduleDeps legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, main's disposition); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from armSchedule: main's body with the runner's Skip set to scheduleSkip; no new read site registry:App@1479,registry:App@388,registry:Host@1546,registry:Host@396,whole:Route@1474,whole:Route@383 aa93bed2781d1005cf864ebaaf6ac4597d6f547c45d6a1063f10c4a483ba46c3

@legacy 48af22e2b896b940598175e9c4988e2837b14a72e88b7ee1ba8346098f209606 internal/host/schedule_wiring.go scheduleSkip legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as the runner's own hold checks); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its schedule runner: new, reads a row's status (any hold) and host (the armed offline view: the record names another region) to tell the runner to leave the host alone; it only withholds a tick, never grants one registry:Host@155,registry:Status@64 4f471727429b5b79bfa909693a09e588051b9f490eafd8480ee9893581362d64

internal/host/schedule_wiring.go scheduleSkip legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as the runner's own hold checks); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its schedule runner: new, reads a row's status (any hold) and host (the armed offline view: the record names another region) to tell the runner to leave the host alone; it only withholds a tick, never grants one registry:Host@155,registry:Status@64 4f471727429b5b79bfa909693a09e588051b9f490eafd8480ee9893581362d64

-@legacy 826058c66c70c6601b64db3044629013a839aed8c9383a6c24c9a86a2772ebd0 internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter; no root decision registry:Host@1184,registry:Host@1226,registry:Host@1247,registry:Host@1271,registry:Host@5289,registry:ImportGate@2801,registry:ImportGate@3716,registry:Status@2594,registry:Status@3702,whole:Route@1653,whole:Route@1943,whole:Route@2072,whole:Route@3482,whole:Route@3673,whole:Route@4806,whole:Route@5155,whole:Route@5330,whole:Route@5366,whole:Timer@1051,whole:Timer@1155,whole:Timer@1177,whole:Timer@1240,whole:Timer@1264,whole:Timer@1280,whole:Timer@4481,whole:Timer@4810,whole:Timer@5370,whole:Timer@820,whole:Timer@902 9892ec7759802c53a4f372a615110f994f2c0697b3353b483aa9a37dbb9d3649

-internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter; no root decision registry:Host@1184,registry:Host@1226,registry:Host@1247,registry:Host@1271,registry:Host@5289,registry:ImportGate@2801,registry:ImportGate@3716,registry:Status@2594,registry:Status@3702,whole:Route@1653,whole:Route@1943,whole:Route@2072,whole:Route@3482,whole:Route@3673,whole:Route@4806,whole:Route@5155,whole:Route@5330,whole:Route@5366,whole:Timer@1051,whole:Timer@1155,whole:Timer@1177,whole:Timer@1240,whole:Timer@1264,whole:Timer@1280,whole:Timer@4481,whole:Timer@4810,whole:Timer@5370,whole:Timer@820,whole:Timer@902 9892ec7759802c53a4f372a615110f994f2c0697b3353b483aa9a37dbb9d3649

+@legacy f938c1e7ab247e5531c6786342ea7c523a87ef60c42c94448b4c57b3fda01597 internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter, and its due query moved into Runner.dueRows; no root decision registry:Host@1352,registry:Host@1394,registry:Host@1415,registry:Host@1439,registry:Host@5457,registry:ImportGate@2969,registry:ImportGate@3884,registry:Status@2762,registry:Status@3870,whole:Route@1821,whole:Route@2111,whole:Route@2240,whole:Route@3650,whole:Route@3841,whole:Route@4974,whole:Route@5323,whole:Route@5498,whole:Route@5534,whole:Timer@1219,whole:Timer@1323,whole:Timer@1345,whole:Timer@1408,whole:Timer@1432,whole:Timer@1448,whole:Timer@4649,whole:Timer@4978,whole:Timer@5538 0c7ef6fbd18e1be821561b82bd059ffdb5e9ac3fe1a35d3b6e1cb8aada3a9edc

+internal/schedule/runner.go Runner.Tick legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: census-ruling-seeded-instances' body with a host Skip names given no tick and no bookkeeping at the pre-filter, and its due query moved into Runner.dueRows; no root decision registry:Host@1352,registry:Host@1394,registry:Host@1415,registry:Host@1439,registry:Host@5457,registry:ImportGate@2969,registry:ImportGate@3884,registry:Status@2762,registry:Status@3870,whole:Route@1821,whole:Route@2111,whole:Route@2240,whole:Route@3650,whole:Route@3841,whole:Route@4974,whole:Route@5323,whole:Route@5498,whole:Route@5534,whole:Timer@1219,whole:Timer@1323,whole:Timer@1345,whole:Timer@1408,whole:Timer@1432,whole:Timer@1448,whole:Timer@4649,whole:Timer@4978,whole:Timer@5538 0c7ef6fbd18e1be821561b82bd059ffdb5e9ac3fe1a35d3b6e1cb8aada3a9edc

internal/schedule/runner.go Runner.fire hint E2 accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (main: J1) PROPOSED E46: captured claim selection and root journal mutation; root RPC outside service host exclusion; SQL/route data never grants root authority. Executing identity: service controller; root for inventory/retirement callbacks. moves-offline-fixes: with Skip set (only the agent's runner sets it, scheduleSkip), the row is read again before the token and a skipped host gets no token and no record; class and task unchanged registry:Host@243,registry:Host@385,registry:Host@691,registry:Host@837,registry:Name@252,registry:Name@394,registry:Spec@148,registry:Spec@260,whole:Route@1005,whole:Route@1202,whole:Route@1504,whole:Route@1567,whole:Route@1735,whole:Route@662,whole:Route@742,whole:Route@768,whole:Route@879,whole:Timer@1009,whole:Timer@108,whole:Timer@1206,whole:Timer@1306,whole:Timer@1508,whole:Timer@1739,whole:Timer@317,whole:Timer@883 1aaf0e167a6852ab6958cb409aaac63523334b4595e1cb8a0e30ac96f4499e9d

+@legacy 32d1784c5ae37f6209554d5a3dc02c95d58c8a9e5593ea5c610ec6041891cc0f internal/schedule/runner.go Runner.dueRows legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as Runner.Tick); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: new, Tick's due query moved out of Tick: held hosts left out by the query when the hold wiring is armed or Skip is set, and each host Skip names (one registry row read per host in the batch) added to the exclusion and the batch read again, so a skipped host's rows never fill the limit; it only withholds ticks registry:Host@489,registry:Host@525,registry:Host@567,registry:Host@647,whole:Route@590,whole:Route@611,whole:Timer@186,whole:Timer@264,whole:Timer@366,whole:Timer@388,whole:Timer@472,whole:Timer@699 6e0d3045eebd802a7d3936a84d7fcbd8a9edd2741e5a13ed8c595c7dcfdaa2c1

+internal/schedule/runner.go Runner.dueRows legacy accepted-legacy accepted-legacy;rev3=PROPOSED-moves-offline-fixes;basis=root-route (J1, as Runner.Tick); the agent (/opt/carlos/carlos agent, carlos-edge.service, user carlos), from its scheduler: new, Tick's due query moved out of Tick: held hosts left out by the query when the hold wiring is armed or Skip is set, and each host Skip names (one registry row read per host in the batch) added to the exclusion and the batch read again, so a skipped host's rows never fill the limit; it only withholds ticks registry:Host@489,registry:Host@525,registry:Host@567,registry:Host@647,whole:Route@590,whole:Route@611,whole:Timer@186,whole:Timer@264,whole:Timer@366,whole:Timer@388,whole:Timer@472,whole:Timer@699 6e0d3045eebd802a7d3936a84d7fcbd8a9edd2741e5a13ed8c595c7dcfdaa2c1

diff --git a/internal/schedule/runner.go b/internal/schedule/runner.go

index e69324eff..24435450d 100644

--- a/internal/schedule/runner.go

+++ b/internal/schedule/runner.go

@@ -151,13 +151,13 @@ func (r *Runner) Tick(ctx context.Context) {

// the same reason: a held row is skipped below with no bookkeeping, so

// a held host's rows would sit at the front of every tick's batch and

// starve the rest. Off, the query is main's.

- var due []registry.Timer

- var err error

- if r.holdsArmed() {

- due, err = r.Reg.DueUnheldTimers(now.Unix(), dueBatch, r.busyHosts())

- } else {

- due, err = r.Reg.DueTimers(now.Unix(), dueBatch, r.busyHosts())

- }

+ //

+ // Skip set (the agent's scheduleSkip) is the same rule for the hosts it

+ // names, before the limit too (Astra, moves-offline-fixes review 1): a

+ // held host's rows are left out by the query, and a host Skip names on

+ // other grounds is added to the exclusion and the batch read again, so

+ // a skipped host's due rows can never fill the batch.

+ due, err := r.dueRows(now)

if err != nil {

log.Printf("schedule: due query: %v", err)

return

@@ -287,6 +287,43 @@ func (r *Runner) Tick(ctx context.Context) {

}

+// dueRows is Tick's due query: busy hosts excluded, held hosts too when the

+// hold wiring is armed or Skip is set, and, with Skip set, every host it

+// names, read again with each one found added to the exclusion. Each read

+// either has no skipped host or adds at least one host to the exclusion, so

+// the loop ends within one read per host with due rows. A host whose row

+// cannot be read is left in the batch: the tick's own read reports it.

+func (r *Runner) dueRows(now time.Time) ([]registry.Timer, error) {

+ exclude := r.busyHosts()

+ for {

+ var due []registry.Timer

+ var err error

+ if r.holdsArmed() || r.Skip != nil {

+ due, err = r.Reg.DueUnheldTimers(now.Unix(), dueBatch, exclude)

+ } else {

+ due, err = r.Reg.DueTimers(now.Unix(), dueBatch, exclude)

+ }

+ if err != nil || r.Skip == nil || len(due) == 0 {

+ return due, err

+ }

+ var skipped []string

+ seen := map[string]bool{}

+ for _, t := range due {

+ if seen[t.Host] {

+ continue

+ }

+ seen[t.Host] = true

+ if rt, err := r.Reg.Get(t.Host); err == nil && rt != nil && r.Skip(*rt) {

+ skipped = append(skipped, t.Host)

+ }

+ if len(skipped) == 0 {

+ return due, nil

+ }

+ exclude = append(exclude, skipped...)

+ }

// busyHosts is the in-flight set as DueTimers wants it. A snapshot: a host

// that goes busy after this read is caught by claim instead.

func (r *Runner) busyHosts() []string {

diff --git a/internal/schedule/runner_test.go b/internal/schedule/runner_test.go

index d333b2c6d..d174327a9 100644

--- a/internal/schedule/runner_test.go

+++ b/internal/schedule/runner_test.go

@@ -1450,3 +1450,45 @@ func TestSkipMintsNoTokenAndRecordsNothing(t *testing.T) {

t.Fatalf("a ready host: %d tokens, %d deliveries", tokens, len(d.calls))

}

+// Astra's review of moves-offline-fixes, finding 1: with the hold wiring off

+// (Gate nil, as on an offline move's boxes), a host Skip names must not fill

+// the due batch. 64 earlier due rows of a skipped host, held or not, and one

+// later row of an ordinary host: the ordinary one fires on the first tick.

+func TestSkippedHostsDoNotStarveAnotherHostsTimer(t *testing.T) {

+ for _, held := range []bool{true, false} {

+ d := &fakeDeliverer{status: 204}

+ reg, r := runnerFixture(t, d)

+ now := at("2026-10-20T17:27:04Z")

+ r.Now = func() time.Time { return now }

+ const other = "b.bab.oncarlos.com"

+ if err := reg.Upsert(registry.Route{Host: other, Kind: "instance", Account: "bab", App: "app", Status: "ready", Hibernate: true}); err != nil {

+ t.Fatal(err)

+ }

+ if held {

+ if err := reg.SetHold(holdHost, "moving", "mv_a"); err != nil {

+ t.Fatal(err)

+ }

+ r.Skip = func(rt registry.Route) bool { return registry.IsHold(rt.Status) }

+ } else {

+ // Ready, but named elsewhere (an offline move's record names

+ // another region): only Skip knows.

+ r.Skip = func(rt registry.Route) bool { return rt.Host == holdHost }

+ }

+ for i := range dueBatch {

+ if err := reg.UpsertTimer(registry.Timer{Host: holdHost, Name: fmt.Sprintf("t%02d", i), Kind: KindSchedule, Spec: "every 1m0s", Path: "/tick", NextAt: now.Unix() - 1000 + int64(i)}); err != nil {

+ t.Fatal(err)

+ }

+ if err := reg.UpsertTimer(registry.Timer{Host: other, Name: "later", Kind: KindSchedule, Spec: "every 1m0s", Path: "/tick", NextAt: now.Unix() - 5}); err != nil {

+ t.Fatal(err)

+ }

+ if r.Gate != nil {

+ t.Fatal("fixture: the hold wiring is armed")

+ }

+ r.tick()

+ if len(d.calls) != 1 || d.hosts[0] != other {

+ t.Fatalf("held %v: deliveries to %v, want exactly one to %s", held, d.hosts, other)

+ }