Scheduled ticks must not hold an instance awake

Date: 2026-08-30 Status: approved in principle (Paul, 2026-08-30, after the flagship measurement below). Two decisions were taken in the brainstorm and are marked (ruling); everything else is marked (decision) and can be overturned cheaply.

1. The problem, as found

Asked whether one member was hogging resources on the flagship, the answer was no — and the way the answer was reached turned up a bug underneath it.

james@keymail.dev (account bac, app jam) declared one schedule on 2026-08-29: sweep, every 5 minutes, hitting /jobs/sweep. The handler answers 204 in 1 millisecond. From the moment that schedule landed (2026-08-29 21:25Z) the instance has been resident continuously — present in every 60-second metrics sample since, 24 hours and counting, where the comparable tenant seapointish sleeps and wakes ten times a day.

It is not the app's fault and it is not the schedule's fault. It is one line of accounting.

Scheduled delivery is otherwise exactly right. activatorDeliverer sends the tick through the activator into the instance's unix socket, never through the public edge, and its doc comment states the property it was built for (internal/host/schedule_wiring.go:216):

> a hibernating instance WAKES for its tick, exactly as it would for a > visitor, and stays awake for the tick's duration because the activator's > in-flight count holds it open. No separate keep-awake to get wrong.

The tick reaches the instance through Service.ServeHTTP (internal/activator/activator.go:1141) — the same door the edge uses — which calls beginRequest, whose release stamps e.lastActive = a.now() (activator.go:594). The sweeper hibernates on e.inflight == 0 && now.Sub(e.lastActive) >= a.cfg.Idle (activator.go:933).

CARLOS_HIBERNATE_IDLE defaults to 15m. The spec is 5m. The clock is reset three times before it can ever expire, so the sweeper's condition is unreachable for the life of the schedule. Every tick buys fifteen minutes of residency for one millisecond of work.

Why it is worth fixing rather than tolerating

Three reasons, in ascending order of weight.

It bills. instance_hours is a metered, priced quantity, and in rate-sheet mode it is the one term with no free-tier subtraction applied — compare billable(...) on egress, storage, apps and memory against the raw meters[usage.MeterInstanceHours] in internal/economics/compute.go:120, at InstanceHourUSD = $0.01. A 5-minute schedule accrues ~730 instance-hours a month: $7.30 billed for roughly four seconds of work.

It defeats the capacity model. The 2026-08-30 sizing measurement put peak concurrent awake instances at 74 of the top 300 accounts on a 90s window. That number assumes instances sleep between requests. An app with any schedule tighter than the idle window never sleeps at all, so as schedules spread the model degrades toward "everything is always awake" — which is the assumption the r8g.2xlarge recommendation was chosen against.

It is invisible. Nothing tells the member. carlos schedule ls reports what will fire and what fired last; nothing says "and this interval means your instance never hibernates". James could not have known, and neither could anyone reading the console.

2. What is already right, and must stay right

Worth stating, because two of these were nearly re-engineered before the code was read properly.

There are already two wake paths for scheduled work, and they are enough. The runner polls DueTimers every 15 seconds and delivers, waking the instance through the activator; separately the sweeper reads DueInstances and wakes any sleeping host whose recorded due time has arrived (activator.go:909-914). Platform timers are covered by the first.

NextDue is therefore not the mechanism to change. GET /api/next-due over the instance socket (internal/activator/backend_exec.go:506) asks the app about work the platform cannot see — its own in-process schedule. It is not the platform's timer index and should not become one. This spec adds no column, no SetNextDue change and no second waker. (decision)

The sidecar precedent already solves the accounting half. BeginSidecarRequest (activator.go:526) passes traffic to an awake instance without registering activity — "no in-flight count, no lastActive stamp (the sidecar's own long-lived WebSocket must not hold its instance awake; that non-count is the whole reason the pair can sleep)". The shape needed here is that shape, differing in one respect: a tick must wake a sleeping instance, where a sidecar request is refused.

3. The change

Three parts. Together they are perhaps a hundred lines.

3.1 A non-stamping, waking entry point

Add Service.ServeScheduled(w, r, rt, proxy) bool, sibling to ServeHTTP, and a beginScheduledRequest behind it that:

• wakes a sleeping instance, exactly as beginRequest does — the tick is the wake source, and that must not change;
• takes the in-flight count for the duration, so a tick is never cut off mid-delivery by a concurrent sweep;
• on release, decrements without stamping lastActive.

activatorDeliverer.Deliver calls it instead of ServeHTTP (schedule_wiring.go:317).

No wire marker. (decision) The sidecar identifies itself with SidecarHeader because it has to: it dials its instance's public URL and its traffic arrives through the edge, so the only place to carry the signal is the request. A scheduled tick never leaves the process — "the tick never leaves the box: no DNS, no TLS, no edge auth, nothing an outsider could aim at the same path" (schedule_wiring.go:219) — so the call site itself is the signal. A dedicated method has no spoofing surface at all, where a header, even one that is safe by construction, would have to be argued about.

X-Carlos-Schedule-At is untouched. It is the app-facing half of the tick contract and continues to reach the handler.

3.2 The guard: don't sleep into an imminent tick

Without a guard, part 3.1 alone converts jam from one resident process into 288 wake/sleep cycles a day. Each cycle is a full SQLite checkpoint-and-restore traversal, and the wake path is where 2026-08-15-sqlite-corruption-defence-design.md (issue #200) records an unclosed integrity gap. Multiplying a scheduled app's traversals by ~28× to save 26 MB is a bad trade.

So: in sweepOnce, before hibernating an otherwise-idle host, skip the sleep when this box holds a timer for that host due sooner than cfg.Idle away.

hibernate when:  inflight == 0
             &&  now - lastActive >= cfg.Idle
             &&  (no timer for this host, or nextAt - now > cfg.Idle)

The threshold is cfg.Idle itself, not a new constant. (ruling — Paul, 2026-08-30) If the gap to the next tick is shorter than the window we would have stayed awake for anyway, staying awake is the cheaper answer. It adds no second tuning number to drift out of step with the first, and it composes with the idle window rather than fighting it:

| Idle window | Schedule | Behaviour | |---|---|---| | 15m | every 5m | stays resident (today's behaviour, unchanged) | | 90s | every 5m | sleeps ~3.5 of every 5 minutes | | 90s | every 60s | stays resident | | 15m | every 1h | sleeps ~45 minutes |

Residency is measured from each WAKE, not from each tick: a tick that wakes a sleeping instance stamps lastActive via the wake path's own stamp, so the instance gets one full idle window of residency starting at that wake before it becomes eligible to sleep again. That is why the 15m/1h row sleeps ~45 minutes (60 minus the 15-minute window it earns on waking), not 59 (60 minus the one-minute tick that would apply if the clock started at delivery). Rows 2 and 3 were already computed this way; anyone extending this table — or the §1 billing arithmetic that reads off it — should keep using the wake, not the tick, as the start of the window.

Implementation constraint, from the code as it stands: sweepOnce does its DueInstances read deliberately before taking a.mu — "it's a registry read, and nothing under a.mu should wait on SQLite" (activator.go:911). The earliest-due lookup joins that pre-lock batch as one query returning host → earliest next_at, never a per-entry query inside the loop.

A host with no timers is unaffected: no row, no guard, sleep as today.

A tick-held host still converges. The guard only skips the sleep; it must not skip the build-convergence check that runs after it. A host held awake by the guard never reaches a "next wake" to converge on — that is what the guard guarantees — so it is now in the same class as an always-awake sidecar host, and reaches the convergence arm on every sweep exactly as a sidecar does. Without this, a carlos ship behind a tight enough schedule would be served stale indefinitely. (ruling — Paul, review round 1, 2026-08-31, overruling an earlier "converges at its next genuine sleep" draft of this section that assumed a sleep the guard makes impossible.)

3.3 Tell the member

The box is the only place that knows both numbers — it holds the timer rows and it constructed the activator with cfg.Idle. So the box decides and reports.

Add HoldsAwake bool \json:"holds_awake"\` to schedule.TimerStatus (internal/schedule/status.go:24`) — true when this row's interval is shorter than the box's idle window, so the row's instance stays resident between ticks. Written wherever the status object is already written (after every fire and every converge that changed a host's rows). Additive, per the repo's wire-shape rule: anything written before this change must still parse, and a status object from an older box simply lacks the field.

carlos schedule ls renders it — one line, in the member's own vocabulary: that this schedule fires more often than the deployment's idle window, so the instance stays awake between ticks and accrues instance-hours accordingly. carlos schedule set shows the same line once the converger has written the status.

Warn, never refuse. (decision) A tight schedule is a legitimate thing to want, and the platform serves it correctly today. The defect is that it was silent, not that it was allowed.

4. The consequence to be explicit about

Not every host is inert under this change — it depends which side of the idle window the host's schedule sits on.

A host whose schedule fires more often than the idle window — jam, 5m against the flagship's 15m window — is unaffected. The guard holds it resident for the same reason the accounting bug used to: its next tick is always inside the window, so the sweeper's answer is "stay awake" today and after this lands. jam will not begin sleeping when this ships.

A host whose schedule fires less often than the idle window is not inert, if that host also takes visitor traffic. Before this branch, every tick — regardless of how far apart — reset lastActive, so a tick landing between two visits kept pushing hibernation out. After this branch a tick no longer resets the clock, so that host loses the extra runway its ticks used to buy it: it can now hibernate up to one full idle window earlier than before, measured from its last real visitor request rather than from whichever tick last happened to land. That is the fix working as designed — a tick was never meant to count as activity — but it is a real, observable behaviour change on this branch, not a case of "provably alters nothing". Whether the flagship carries any host in this shape today is exactly what §7's rollout check exists to establish before rolling.

The sequencing that remains deliberate is narrower than "inert, then behaviour change": this change runs under every tenant instance on every box, so shipping it and confirming its effect matches §7's expectations before separately tightening CARLOS_HIBERNATE_IDLE 15m → 90s keeps the two changes distinguishable if something needs reverting. The idle-window tightening is the change that turns jam from "stays resident" into "sleeps and wakes"; this branch is the change that stops loose-schedule hosts getting undeserved extra runway from their ticks.

(ruling — Paul, 2026-08-30) The two land as separate branches.

5. What is not built

• No NextDue rework, no new registry column, no second waker. §2.
• No per-route idle override. One knob per deployment, plus a guard that reads it, covers every case found. A per-route window is a product decision nobody has asked for.
• No new tuning constant. §3.2.
• No special case for carlos schedule run. A manual fire is FireTimerNow, which moves next_at and nothing else (internal/registry/timers.go:132); the runner cannot tell it from a recurring fire without a new column. A schema change to buy "the instance stays up 15 minutes after you press run" is not worth it. Manual runs are ticks. (decision — reversed during the brainstorm, after reading the code: the first instinct was to treat a human's press as visitor activity.)
• No use of routes.next_due in the guard. It looks like free extra coverage — the app's own declared due time, already in the registry, already read by DueInstances — and it is a trap. That column is written only at sleep and is deliberately not cleared on wake: "the registry's stale past value while the host is up is harmless" (activator.go:1007). A guard reading it on an awake host would compare against a timestamp in the past, conclude the next delivery is imminent forever, and wedge itself open. The value could only be made current by probing the app on every sweep pass, which is a socket round trip per idle host per tick. The guard covers platform timers; app-declared schedules stay uncovered, and are fine there because their intervals (hourly and up, in every case on the fleet today) are far wider than any plausible idle window.
• No change to sidecar accounting. It is already correct and is the precedent this follows.

6. Testing

All of it runs without a box.

internal/activator:

• a scheduled delivery leaves lastActive untouched; a visitor request to the same host moves it — the two asserted against each other in one test, since the bug is precisely that they were the same path;
• a tick against a sleeping host wakes it, and the in-flight count holds it through delivery (a sweep racing the tick must not stop it mid-flight);
• the guard: a host idle past the window with a timer due inside cfg.Idle stays resident; the same host with the timer beyond cfg.Idle hibernates; a host with no timer rows hibernates as before;
• the guard's query runs before a.mu — asserted structurally, the way the existing sweep tests pin their ordering.

internal/host:

• activatorDeliverer.Deliver routes through ServeScheduled, not ServeHTTP, and a delivery to a host that was asleep when the tick came due still returns the handler's status;
• X-Carlos-Schedule-At still reaches the handler unchanged.

internal/schedule:

• a status object written by a box that computed the verdict round-trips, and one written without the field still parses.

7. Rollout

Ordinary: merge, roll the fleet, confirm the change lands where it should and nowhere else — see §4, this is not a blanket "confirm inert". Two checks:

• jam-shaped hosts — schedule interval shorter than the box's idle window — must not move at all. jam itself must still be resident afterwards, and seapointish (no schedule) must still be doing its ten wake/sleep cycles a day. A change to either here is a bug, not a win.
• Before rolling, enumerate hosts whose schedule interval is longer than the idle window. Expect their wake/sleep cycle counts to rise slightly — this branch stops their ticks buying free runway between real visits, so a small rise there is the change working as designed. Only a large or otherwise unexplained jump is worth investigating.

The separate idle-window tightening (15m → 90s) is verified on the next branch, against the same apps.