| 1 | # Scheduled ticks must not hold an instance awake |
| 2 | |
| 3 | **Date:** 2026-08-30 |
| 4 | **Status:** approved in principle (Paul, 2026-08-30, after the flagship |
| 5 | measurement below). Two decisions were taken in the brainstorm and are marked |
| 6 | **(ruling)**; everything else is marked **(decision)** and can be overturned |
| 7 | cheaply. |
| 8 | |
| 9 | ## 1. The problem, as found |
| 10 | |
| 11 | Asked whether one member was hogging resources on the flagship, the answer was |
| 12 | no — and the way the answer was reached turned up a bug underneath it. |
| 13 | |
| 14 | `james@keymail.dev` (account `bac`, app `jam`) declared one schedule on |
| 15 | 2026-08-29: `sweep`, every 5 minutes, hitting `/jobs/sweep`. The handler |
| 16 | answers `204` in **1 millisecond**. From the moment that schedule landed |
| 17 | (2026-08-29 21:25Z) the instance has been resident **continuously** — present |
| 18 | in every 60-second metrics sample since, 24 hours and counting, where the |
| 19 | comparable tenant `seapointish` sleeps and wakes ten times a day. |
| 20 | |
| 21 | It is not the app's fault and it is not the schedule's fault. It is one line |
| 22 | of accounting. |
| 23 | |
| 24 | Scheduled delivery is otherwise exactly right. `activatorDeliverer` sends the |
| 25 | tick through the activator into the instance's unix socket, never through the |
| 26 | public edge, and its doc comment states the property it was built for |
| 27 | (`internal/host/schedule_wiring.go:216`): |
| 28 | |
| 29 | > a hibernating instance WAKES for its tick, exactly as it would for a |
| 30 | > visitor, and stays awake for the tick's duration because the activator's |
| 31 | > in-flight count holds it open. No separate keep-awake to get wrong. |
| 32 | |
| 33 | The tick reaches the instance through `Service.ServeHTTP` |
| 34 | (`internal/activator/activator.go:1141`) — the same door the edge uses — which |
| 35 | calls `beginRequest`, whose release stamps `e.lastActive = a.now()` |
| 36 | (`activator.go:594`). The sweeper hibernates on |
| 37 | `e.inflight == 0 && now.Sub(e.lastActive) >= a.cfg.Idle` (`activator.go:933`). |
| 38 | |
| 39 | `CARLOS_HIBERNATE_IDLE` defaults to 15m. The spec is 5m. The clock is reset |
| 40 | three times before it can ever expire, so the sweeper's condition is |
| 41 | unreachable for the life of the schedule. **Every tick buys fifteen minutes of |
| 42 | residency for one millisecond of work.** |
| 43 | |
| 44 | ### Why it is worth fixing rather than tolerating |
| 45 | |
| 46 | Three reasons, in ascending order of weight. |
| 47 | |
| 48 | **It bills.** `instance_hours` is a metered, priced quantity, and in |
| 49 | rate-sheet mode it is the one term with no free-tier subtraction applied — |
| 50 | compare `billable(...)` on egress, storage, apps and memory against the raw |
| 51 | `meters[usage.MeterInstanceHours]` in `internal/economics/compute.go:120`, at |
| 52 | `InstanceHourUSD` = $0.01. A 5-minute schedule accrues ~730 instance-hours a |
| 53 | month: **$7.30 billed for roughly four seconds of work**. |
| 54 | |
| 55 | **It defeats the capacity model.** The 2026-08-30 sizing measurement put peak |
| 56 | concurrent awake instances at 74 of the top 300 accounts on a 90s window. That |
| 57 | number assumes instances sleep between requests. An app with any schedule |
| 58 | tighter than the idle window never sleeps at all, so as schedules spread the |
| 59 | model degrades toward "everything is always awake" — which is the assumption |
| 60 | the r8g.2xlarge recommendation was chosen against. |
| 61 | |
| 62 | **It is invisible.** Nothing tells the member. `carlos schedule ls` reports |
| 63 | what will fire and what fired last; nothing says "and this interval means your |
| 64 | instance never hibernates". James could not have known, and neither could |
| 65 | anyone reading the console. |
| 66 | |
| 67 | ## 2. What is already right, and must stay right |
| 68 | |
| 69 | Worth stating, because two of these were nearly re-engineered before the code |
| 70 | was read properly. |
| 71 | |
| 72 | **There are already two wake paths for scheduled work, and they are enough.** |
| 73 | The runner polls `DueTimers` every 15 seconds and delivers, waking the |
| 74 | instance through the activator; separately the sweeper reads `DueInstances` |
| 75 | and wakes any sleeping host whose recorded due time has arrived |
| 76 | (`activator.go:909-914`). Platform timers are covered by the first. |
| 77 | |
| 78 | **`NextDue` is therefore not the mechanism to change.** `GET /api/next-due` |
| 79 | over the instance socket (`internal/activator/backend_exec.go:506`) asks the |
| 80 | *app* about work the platform cannot see — its own in-process schedule. It is |
| 81 | not the platform's timer index and should not become one. This spec adds no |
| 82 | column, no `SetNextDue` change and no second waker. **(decision)** |
| 83 | |
| 84 | **The sidecar precedent already solves the accounting half.** |
| 85 | `BeginSidecarRequest` (`activator.go:526`) passes traffic to an awake instance |
| 86 | without registering activity — "no in-flight count, no lastActive stamp (the |
| 87 | sidecar's own long-lived WebSocket must not hold its instance awake; that |
| 88 | non-count is the whole reason the pair can sleep)". The shape needed here is |
| 89 | that shape, differing in one respect: a tick must *wake* a sleeping instance, |
| 90 | where a sidecar request is refused. |
| 91 | |
| 92 | ## 3. The change |
| 93 | |
| 94 | Three parts. Together they are perhaps a hundred lines. |
| 95 | |
| 96 | ### 3.1 A non-stamping, waking entry point |
| 97 | |
| 98 | Add `Service.ServeScheduled(w, r, rt, proxy) bool`, sibling to `ServeHTTP`, |
| 99 | and a `beginScheduledRequest` behind it that: |
| 100 | |
| 101 | - wakes a sleeping instance, exactly as `beginRequest` does — the tick is the |
| 102 | wake source, and that must not change; |
| 103 | - takes the in-flight count for the duration, so a tick is never cut off |
| 104 | mid-delivery by a concurrent sweep; |
| 105 | - on release, decrements **without** stamping `lastActive`. |
| 106 | |
| 107 | `activatorDeliverer.Deliver` calls it instead of `ServeHTTP` |
| 108 | (`schedule_wiring.go:317`). |
| 109 | |
| 110 | **No wire marker. (decision)** The sidecar identifies itself with |
| 111 | `SidecarHeader` because it has to: it dials its instance's public URL and its |
| 112 | traffic arrives through the edge, so the only place to carry the signal is the |
| 113 | request. A scheduled tick never leaves the process — "the tick never leaves |
| 114 | the box: no DNS, no TLS, no edge auth, nothing an outsider could aim at the |
| 115 | same path" (`schedule_wiring.go:219`) — so the call site itself is the signal. |
| 116 | A dedicated method has no spoofing surface at all, where a header, even one |
| 117 | that is safe by construction, would have to be argued about. |
| 118 | |
| 119 | `X-Carlos-Schedule-At` is untouched. It is the app-facing half of the tick |
| 120 | contract and continues to reach the handler. |
| 121 | |
| 122 | ### 3.2 The guard: don't sleep into an imminent tick |
| 123 | |
| 124 | Without a guard, part 3.1 alone converts `jam` from one resident process into |
| 125 | **288 wake/sleep cycles a day**. Each cycle is a full SQLite |
| 126 | checkpoint-and-restore traversal, and the wake path is where |
| 127 | `2026-08-15-sqlite-corruption-defence-design.md` (issue #200) records an |
| 128 | unclosed integrity gap. Multiplying a scheduled app's traversals by ~28× to |
| 129 | save 26 MB is a bad trade. |
| 130 | |
| 131 | So: in `sweepOnce`, before hibernating an otherwise-idle host, skip the sleep |
| 132 | when this box holds a timer for that host due sooner than `cfg.Idle` away. |
| 133 | |
| 134 | ``` |
| 135 | hibernate when: inflight == 0 |
| 136 | && now - lastActive >= cfg.Idle |
| 137 | && (no timer for this host, or nextAt - now > cfg.Idle) |
| 138 | ``` |
| 139 | |
| 140 | **The threshold is `cfg.Idle` itself, not a new constant. (ruling — Paul, |
| 141 | 2026-08-30)** If the gap to the next tick is shorter than the window we would |
| 142 | have stayed awake for anyway, staying awake *is* the cheaper answer. It adds |
| 143 | no second tuning number to drift out of step with the first, and it composes |
| 144 | with the idle window rather than fighting it: |
| 145 | |
| 146 | | Idle window | Schedule | Behaviour | |
| 147 | |---|---|---| |
| 148 | | 15m | every 5m | stays resident (today's behaviour, unchanged) | |
| 149 | | 90s | every 5m | sleeps ~3.5 of every 5 minutes | |
| 150 | | 90s | every 60s | stays resident | |
| 151 | | 15m | every 1h | sleeps ~45 minutes | |
| 152 | |
| 153 | Residency is measured from each WAKE, not from each tick: a tick that wakes a |
| 154 | sleeping instance stamps `lastActive` via the wake path's own stamp, so the |
| 155 | instance gets one full idle window of residency starting at that wake before |
| 156 | it becomes eligible to sleep again. That is why the 15m/1h row sleeps ~45 |
| 157 | minutes (60 minus the 15-minute window it earns on waking), not 59 (60 minus |
| 158 | the one-minute tick that would apply if the clock started at delivery). Rows |
| 159 | 2 and 3 were already computed this way; anyone extending this table — or the |
| 160 | §1 billing arithmetic that reads off it — should keep using the wake, not the |
| 161 | tick, as the start of the window. |
| 162 | |
| 163 | **Implementation constraint, from the code as it stands:** `sweepOnce` does |
| 164 | its `DueInstances` read deliberately before taking `a.mu` — "it's a registry |
| 165 | read, and nothing under a.mu should wait on SQLite" (`activator.go:911`). The |
| 166 | earliest-due lookup joins that pre-lock batch as one query returning |
| 167 | `host → earliest next_at`, never a per-entry query inside the loop. |
| 168 | |
| 169 | A host with no timers is unaffected: no row, no guard, sleep as today. |
| 170 | |
| 171 | **A tick-held host still converges.** The guard only skips the *sleep*; it |
| 172 | must not skip the build-convergence check that runs after it. A host held |
| 173 | awake by the guard never reaches a "next wake" to converge on — that is what |
| 174 | the guard guarantees — so it is now in the same class as an always-awake |
| 175 | sidecar host, and reaches the convergence arm on every sweep exactly as a |
| 176 | sidecar does. Without this, a `carlos ship` behind a tight enough schedule |
| 177 | would be served stale indefinitely. **(ruling — Paul, review round 1, |
| 178 | 2026-08-31, overruling an earlier "converges at its next genuine sleep" |
| 179 | draft of this section that assumed a sleep the guard makes impossible.)** |
| 180 | |
| 181 | ### 3.3 Tell the member |
| 182 | |
| 183 | The box is the only place that knows both numbers — it holds the timer rows |
| 184 | and it constructed the activator with `cfg.Idle`. So the box decides and |
| 185 | reports. |
| 186 | |
| 187 | Add `HoldsAwake bool \`json:"holds_awake"\`` to `schedule.TimerStatus` |
| 188 | (`internal/schedule/status.go:24`) — true when this row's interval is shorter |
| 189 | than the box's idle window, so the row's instance stays resident between |
| 190 | ticks. Written |
| 191 | wherever the status object is already written (after every fire and every |
| 192 | converge that changed a host's rows). Additive, per the repo's wire-shape |
| 193 | rule: anything written before this change must still parse, and a status |
| 194 | object from an older box simply lacks the field. |
| 195 | |
| 196 | `carlos schedule ls` renders it — one line, in the member's own vocabulary: |
| 197 | that this schedule fires more often than the deployment's idle window, so the |
| 198 | instance stays awake between ticks and accrues instance-hours accordingly. |
| 199 | `carlos schedule set` shows the same line once the converger has written the |
| 200 | status. |
| 201 | |
| 202 | **Warn, never refuse. (decision)** A tight schedule is a legitimate thing to |
| 203 | want, and the platform serves it correctly today. The defect is that it was |
| 204 | silent, not that it was allowed. |
| 205 | |
| 206 | ## 4. The consequence to be explicit about |
| 207 | |
| 208 | **Not every host is inert under this change — it depends which side of the |
| 209 | idle window the host's schedule sits on.** |
| 210 | |
| 211 | A host whose schedule fires *more often* than the idle window — `jam`, 5m |
| 212 | against the flagship's 15m window — is unaffected. The guard holds it |
| 213 | resident for the same reason the accounting bug used to: its next tick is |
| 214 | always inside the window, so the sweeper's answer is "stay awake" today and |
| 215 | after this lands. `jam` will not begin sleeping when this ships. |
| 216 | |
| 217 | A host whose schedule fires *less often* than the idle window is not inert, |
| 218 | if that host also takes visitor traffic. Before this branch, every tick — |
| 219 | regardless of how far apart — reset `lastActive`, so a tick landing between |
| 220 | two visits kept pushing hibernation out. After this branch a tick no longer |
| 221 | resets the clock, so that host loses the extra runway its ticks used to buy |
| 222 | it: it can now hibernate up to one full idle window earlier than before, |
| 223 | measured from its last real visitor request rather than from whichever tick |
| 224 | last happened to land. That is the fix working as designed — a tick was |
| 225 | never meant to count as activity — but it is a real, observable behaviour |
| 226 | change on *this* branch, not a case of "provably alters nothing". Whether the |
| 227 | flagship carries any host in this shape today is exactly what §7's rollout |
| 228 | check exists to establish before rolling. |
| 229 | |
| 230 | The sequencing that remains deliberate is narrower than "inert, then |
| 231 | behaviour change": this change runs under every tenant instance on every |
| 232 | box, so shipping it and confirming its effect matches §7's expectations |
| 233 | before separately tightening `CARLOS_HIBERNATE_IDLE` 15m → 90s keeps the two |
| 234 | changes distinguishable if something needs reverting. The idle-window |
| 235 | tightening is the change that turns `jam` from "stays resident" into |
| 236 | "sleeps and wakes"; this branch is the change that stops loose-schedule |
| 237 | hosts getting undeserved extra runway from their ticks. |
| 238 | |
| 239 | **(ruling — Paul, 2026-08-30)** The two land as separate branches. |
| 240 | |
| 241 | ## 5. What is not built |
| 242 | |
| 243 | - **No `NextDue` rework, no new registry column, no second waker.** §2. |
| 244 | - **No per-route idle override.** One knob per deployment, plus a guard that |
| 245 | reads it, covers every case found. A per-route window is a product decision |
| 246 | nobody has asked for. |
| 247 | - **No new tuning constant.** §3.2. |
| 248 | - **No special case for `carlos schedule run`.** A manual fire is |
| 249 | `FireTimerNow`, which moves `next_at` and nothing else |
| 250 | (`internal/registry/timers.go:132`); the runner cannot tell it from a |
| 251 | recurring fire without a new column. A schema change to buy "the instance |
| 252 | stays up 15 minutes after you press run" is not worth it. Manual runs are |
| 253 | ticks. **(decision — reversed during the brainstorm, after reading the |
| 254 | code: the first instinct was to treat a human's press as visitor activity.)** |
| 255 | - **No use of `routes.next_due` in the guard.** It looks like free extra |
| 256 | coverage — the app's own declared due time, already in the registry, already |
| 257 | read by `DueInstances` — and it is a trap. That column is written only at |
| 258 | sleep and is deliberately not cleared on wake: "the registry's stale past |
| 259 | value while the host is up is harmless" (`activator.go:1007`). A guard |
| 260 | reading it on an awake host would compare against a timestamp in the past, |
| 261 | conclude the next delivery is imminent forever, and wedge itself open. The |
| 262 | value could only be made current by probing the app on every sweep pass, |
| 263 | which is a socket round trip per idle host per tick. The guard covers |
| 264 | platform timers; app-declared schedules stay uncovered, and are fine there |
| 265 | because their intervals (hourly and up, in every case on the fleet today) |
| 266 | are far wider than any plausible idle window. |
| 267 | - **No change to sidecar accounting.** It is already correct and is the |
| 268 | precedent this follows. |
| 269 | |
| 270 | ## 6. Testing |
| 271 | |
| 272 | All of it runs without a box. |
| 273 | |
| 274 | `internal/activator`: |
| 275 | |
| 276 | - a scheduled delivery leaves `lastActive` untouched; a visitor request to the |
| 277 | same host moves it — the two asserted against each other in one test, since |
| 278 | the bug is precisely that they were the same path; |
| 279 | - a tick against a *sleeping* host wakes it, and the in-flight count holds it |
| 280 | through delivery (a sweep racing the tick must not stop it mid-flight); |
| 281 | - the guard: a host idle past the window with a timer due inside `cfg.Idle` |
| 282 | stays resident; the same host with the timer beyond `cfg.Idle` hibernates; |
| 283 | a host with no timer rows hibernates as before; |
| 284 | - the guard's query runs before `a.mu` — asserted structurally, the way the |
| 285 | existing sweep tests pin their ordering. |
| 286 | |
| 287 | `internal/host`: |
| 288 | |
| 289 | - `activatorDeliverer.Deliver` routes through `ServeScheduled`, not |
| 290 | `ServeHTTP`, and a delivery to a host that was asleep when the tick came due |
| 291 | still returns the handler's status; |
| 292 | - `X-Carlos-Schedule-At` still reaches the handler unchanged. |
| 293 | |
| 294 | `internal/schedule`: |
| 295 | |
| 296 | - a status object written by a box that computed the verdict round-trips, and |
| 297 | one written without the field still parses. |
| 298 | |
| 299 | ## 7. Rollout |
| 300 | |
| 301 | Ordinary: merge, roll the fleet, confirm the change lands where it should |
| 302 | and nowhere else — see §4, this is not a blanket "confirm inert". Two checks: |
| 303 | |
| 304 | - **`jam`-shaped hosts — schedule interval shorter than the box's idle |
| 305 | window — must not move at all.** `jam` itself must still be resident |
| 306 | afterwards, and `seapointish` (no schedule) must still be doing its ten |
| 307 | wake/sleep cycles a day. A change to either here is a bug, not a win. |
| 308 | - **Before rolling, enumerate hosts whose schedule interval is longer than |
| 309 | the idle window.** Expect their wake/sleep cycle counts to rise slightly — |
| 310 | this branch stops their ticks buying free runway between real visits, so a |
| 311 | small rise there is the change working as designed. Only a large or |
| 312 | otherwise unexplained jump is worth investigating. |
| 313 | |
| 314 | The separate idle-window tightening (15m → 90s) is verified on the *next* |
| 315 | branch, against the same apps. |
| 316 | |
amadan