reminders: make the platform's wake carry work with it
A reminder for a 14:00 meeting arrived at 14:47, and the instance was
awake for almost all of the 57 minutes it was late. Hibernation was not
the cause; the wake was.
`carlos logs` for 2026-09-04 shows the shape of it:
13:57:08 instance stopping → 13:57:38 instance woke
14:13:08 instance stopping → 14:13:38 instance woke
14:29:08 instance stopping → 14:29:38 instance woke
14:45:08 instance stopping → 14:45:38 instance woke
Awake fifteen and a half minutes, asleep thirty seconds, over and over,
sending nothing. The activator was doing its job perfectly: it asked
/api/next-due before hibernating, recorded the answer, and woke the
process at it. But a wake IS only a process start. No request arrives
with it, and the drain had no boot entrance — so the woken instance did
nothing, idled out, hibernated, answered the same still-overdue instant,
and was woken again. The 6h declared schedule was hours away. The mail
finally went when somebody loaded a page.
The design said three entrances — boot, tick, request. Two were built.
It also specified the one-shot timer that makes the wake arrive as a
POST rather than as a bare exec, and that was never written at all:
`grep ScheduleAt` across this repo returned nothing.
Four fixes, each with a test that fails without it, and each mutation
tested — the guard was broken on purpose and the named test watched to
fail before it was believed.
1. App.DrainAtBoot, called from main before Serve. The entrance the
spec named and nobody built. Synchronous: the agent binds the socket
before spawning us, so a request arriving mid-drain waits, and the
pass is bounded by drainDeadline.
2. A one-shot platform timer, upserted by name, so the wake arrives as
a POST to /jobs/reminders. Named `reminder-next`, NOT `reminders` as
the design document proposed: `carlos schedule set` already declares
a recurring schedule of that name on this app, one-shot timers share
the namespace, and ScheduleAt answers ErrDeclaredSchedule for a
declared name. The obvious name would have been refused on every
call — silently, since a refusal is a log line we move on from. Held
as swappable package vars so a test can watch what the app asks for
without standing up a control socket.
3. NextDueAt now answers with work this instance CAN DO. It returned
MIN(due_at) over every unfired row, including rows a drain had
claimed and failed to send — which are deliberately unclaimable for
the five-minute lease. So an undeliverable address left next_due
permanently in the past and reproduced the whole wake/sleep loop on
its own: wake, claim nothing, idle fifteen minutes, hibernate, wake.
A claimed row now answers when its lease expires.
4. The rearm goes in the signed-in middleware, after the handler, not
one call per writer. Every write to the reminder queue happens inside
a request in that group — create, edit, add or remove a reminder,
exclude an occurrence, delete — so one call covers all of them and no
future writer has to remember. It costs an indexed MIN and a
comparison; nothing reaches the control socket unless the answer
moved. The tick route is outside that group and rearms for itself.
AGENTS.md and the design doc are corrected in the same commit, per this
file's own rule. The design's item 1 said `NextDue` "wakes the process
for it" and stopped there — the gap in the sentence is the gap that got
built, so it now says all three parts out loud.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0155Ae1Q1aR51HX97nmBovG6
10 files changed,
+620
−25
AGENTS.md+21 −1cmd/calendar/main.go+16 −0docs/superpowers/specs/2026-09-02-calendar-design.md+33 −5internal/calendar/app.go+13 −2internal/calendar/handlers.go+4 −0internal/calendar/handlers_jobs.go+25 −9internal/calendar/store_reminder_next.go+31 −8internal/calendar/testing.go+71 −0internal/calendar/wake.go+154 −0internal/calendartest/wake_test.go+252 −0
diff --git a/AGENTS.md b/AGENTS.md| index 3113245..bd8db02 100644 |
| --- a/AGENTS.md |
| +++ b/AGENTS.md |
| @@ -120,7 +120,27 @@ relying on remembering one, make it a test instead. |
| be hibernated mid-send. |
| - **Ticks are expensive.** `CARLOS_HIBERNATE_IDLE` defaults to 15 |
| minutes, so each tick buys a quarter hour of billed residency. The |
| - declared schedule is 6h and the precision comes from `NextDue`. |
| + declared schedule is 6h; the precision comes from `NextDue` AND the |
| + one-shot timer below, which are two halves of one mechanism. |
| +- **A wake is only a process start.** `NextDue` says WHEN to wake us and |
| + delivers no work. The precision path needs all three of `NextDue`, the |
| + one-shot timer (`carlos.ScheduleAt`, wake.go) and the boot drain. Two |
| + of the three shipped, and the instance woke and slept every sixteen |
| + minutes sending nothing while a reminder went out 57 minutes late. |
| + `TestAWakeWithNoRequestStillSendsTheReminder` is the guard. |
| +- **The one-shot timer is `reminder-next`, never `reminders`.** The |
| + declared schedule owns that name, one-shot timers share the namespace, |
| + and `ScheduleAt` answers `ErrDeclaredSchedule` — which the app logs |
| + and carries on from, so the failure is invisible. |
| +- **`NextDue` answers when there is work this instance CAN DO**, not |
| + when a row came due. A claimed row is unclaimable for its five-minute |
| + lease; answering with its past due time wakes the instance to do |
| + nothing, again every sixteen minutes, for as long as one address |
| + bounces. |
| +- **The rearm lives in the request middleware, after the handler** — |
| + never one call per writer. Every queue write happens inside a |
| + signed-in request, so one call covers all of them and nothing new has |
| + to remember. |
| **Frontend** |
| - Write the **`rst-*` attribute grammar** (`rst-btn="primary"`), never |
diff --git a/cmd/calendar/main.go b/cmd/calendar/main.go| index 78a4e27..fef3201 100644 |
| --- a/cmd/calendar/main.go |
| +++ b/cmd/calendar/main.go |
| @@ -13,6 +13,7 @@ |
| package main |
| import ( |
| + "context" |
| "log/slog" |
| "os" |
| "strings" |
| @@ -83,6 +84,21 @@ func main() { |
| // The same page a handler's own error path renders, so a panic and |
| // a handled failure look identical to whoever hits one. |
| opts.ErrorPage = calendar.ErrorPage |
| + |
| + // The wake's other half, and it must happen BEFORE Serve. |
| + // |
| + // The activator wakes this process at the instant NextDue named, |
| + // and that is all a wake is — no request arrives with it. Without |
| + // a drain here the woken instance sits idle for its fifteen-minute |
| + // window, hibernates, answers the same still-overdue instant, and |
| + // is woken again thirty seconds later: a checkpoint-and-restore |
| + // round trip every sixteen minutes, sending nothing, until somebody |
| + // happens to load a page. |
| + // |
| + // It also re-registers the one-shot timer, so a wake that arrives |
| + // with no timer behind it leaves with one. |
| + app.DrainAtBoot(context.Background()) |
| + |
| if err := rastrillo.Serve(opts); err != nil { |
| logger.Error("serve failed", "err", err) |
| os.Exit(1) |
diff --git a/docs/superpowers/specs/2026-09-02-calendar-design.md b/docs/superpowers/specs/2026-09-02-calendar-design.md| index c86f847..17f561b 100644 |
| --- a/docs/superpowers/specs/2026-09-02-calendar-design.md |
| +++ b/docs/superpowers/specs/2026-09-02-calendar-design.md |
| @@ -171,16 +171,44 @@ A forever-daily standup costs one row; firing it computes the next. |
| **Three ways a reminder fires**, and the redundancy is the design: |
| -1. `Options.NextDue` answers the activator's poll with |
| - `MIN(due_at)`. The platform hibernates knowing when to wake, and |
| - wakes the process for it. This is the precision path. |
| +1. **The precision path, and it has three parts, not one.** |
| + `Options.NextDue` answers the activator's poll and the platform |
| + wakes the process at the answer — but a wake IS only a process |
| + start, and nothing arrives with it. So the app also registers a |
| + one-shot timer (`carlos.ScheduleAt`, upserted by name after any |
| + request that could have moved the queue), which arrives as a POST to |
| + `/jobs/reminders`; and it **drains at boot**, for the wake that has |
| + no timer behind it. |
| + |
| + Writing only the first part is what shipped, and it failed in |
| + exactly the way the omission predicts: the platform woke a process |
| + that did nothing, idled its fifteen minutes, hibernated, answered |
| + the same still-overdue instant, and was woken again — a |
| + checkpoint-and-restore round trip every sixteen minutes, sending |
| + nothing, until somebody loaded a page. A reminder due at 13:50 on |
| + 2026-09-04 arrived at 14:47. Fixed 2026-09-05. |
| + |
| + Two details that are not optional: |
| + |
| + - **The timer is named `reminder-next`, never `reminders`.** The |
| + declared schedule owns that name, one-shot timers share the |
| + namespace, and `ScheduleAt` answers `ErrDeclaredSchedule` for a |
| + declared name — so the obvious name refuses every registration, |
| + silently. |
| + - **`NextDue` answers when there is work this instance CAN DO**, not |
| + when a row came due. A claimed row is unclaimable for its |
| + five-minute lease, so it answers when the lease expires. Reporting |
| + its past `due_at` reproduces the same wake loop for anybody with an |
| + undeliverable address. |
| 2. The declared schedule, **every 6 hours**, POSTs `/jobs/reminders`. |
| Six and not five minutes: `CARLOS_HIBERNATE_IDLE` defaults to 15 |
| minutes, so every tick buys a quarter hour of billed residency — a |
| 5-minute schedule accrues ~730 instance-hours a month for seconds of |
| - work. |
| + work. It is the backstop, not the precision. |
| 3. Any signed-in page view drains one opportunistically, bounded to one |
| - per request. |
| + per request. The same middleware rearms the timer after every |
| + request, which is what makes the rearm complete: every write to the |
| + queue happens inside one of these. |
| **The tick route is mounted OUTSIDE `csrf.Protect`.** `csrf.Protect` |
| refuses any POST carrying no `Sec-Fetch-Site`, `Origin` or `Referer`, |
diff --git a/internal/calendar/app.go b/internal/calendar/app.go| index 58ccd29..30f2c77 100644 |
| --- a/internal/calendar/app.go |
| +++ b/internal/calendar/app.go |
| @@ -36,6 +36,10 @@ type App struct { |
| // auth is held so the harness can mint a session the same way a |
| // real sign-in does, rather than by writing a session row by hand. |
| auth *rauth.Auth |
| + |
| + // h is the handler set, held so DrainAtBoot can run the same drain |
| + // the tick route runs rather than a second copy of it. |
| + h *handlers |
| } |
| // New resolves every fail-closed setting, applies the schema, and wires |
| @@ -124,7 +128,11 @@ func New(cfg Config) (*App, error) { |
| mailRecorder: recorder, |
| auth: authPlugin, |
| } |
| - h := &handlers{store: store, logger: logger, origin: origin, mailer: mailer} |
| + h := &handlers{ |
| + store: store, logger: logger, origin: origin, mailer: mailer, |
| + timer: &wakeTimer{logger: logger}, |
| + } |
| + a.h = h |
| a.mux = a.routes(h, authPlugin, list, origin) |
| return a, nil |
| } |
| @@ -135,7 +143,10 @@ func New(cfg Config) (*App, error) { |
| // |
| // Without it, a reminder due at 09:00 on an instance nobody visits |
| // would arrive whenever somebody next loaded a page. With it, the |
| -// platform wakes the process at 09:00 to send it. |
| +// platform wakes the process at 09:00. |
| +// |
| +// Waking is only half of it: the wake carries no request. DrainAtBoot |
| +// and the one-shot timer in wake.go are what turn it into a send. |
| func (a *App) NextDue() time.Time { |
| return a.store.NextDueAt(context.Background()) |
| } |
diff --git a/internal/calendar/handlers.go b/internal/calendar/handlers.go| index fa66a94..3f902f2 100644 |
| --- a/internal/calendar/handlers.go |
| +++ b/internal/calendar/handlers.go |
| @@ -20,6 +20,10 @@ type handlers struct { |
| // mailer sends reminders. The same sender sign-in uses. |
| mailer mail.Sender |
| + |
| + // timer is the platform's one-shot wake, re-pointed at the next due |
| + // reminder after anything that could have moved it. See wake.go. |
| + timer *wakeTimer |
| } |
| // secure reports whether cookies this app sets should carry Secure. It |
diff --git a/internal/calendar/handlers_jobs.go b/internal/calendar/handlers_jobs.go| index 77f7873..9f55f30 100644 |
| --- a/internal/calendar/handlers_jobs.go |
| +++ b/internal/calendar/handlers_jobs.go |
| @@ -26,6 +26,12 @@ func (h *handlers) drainReminders(w http.ResponseWriter, r *http.Request) { |
| http.Error(w, "forbidden", http.StatusForbidden) |
| return |
| } |
| + // Outside the signed-in group, so this route rearms for itself. |
| + // Deferred rather than tail-called: the 500 path below returns |
| + // early, and a drain that failed partway through has still moved |
| + // the queue. |
| + defer h.rearm(r.Context()) |
| + |
| sent, err := h.drain(r.Context(), time.Now()) |
| if err != nil { |
| // 5xx so the platform retries. The occurrence key makes a |
| @@ -128,12 +134,21 @@ func (h *handlers) sendReminder(ctx context.Context, due DueReminder, now time.T |
| return h.mailer.Send(ctx, due.Address, subject, body) |
| } |
| -// drainOpportunistically is the third entrance, as middleware on the |
| -// signed-in group. |
| +// drainOpportunistically is the request entrance, as middleware on the |
| +// signed-in group. It also rearms the platform's wake timer. |
| +// |
| +// The drain is GET and HEAD only. A drain inside a POST would put an |
| +// email send between somebody's form submission and its redirect, which |
| +// is the one moment they are already waiting. |
| // |
| -// GET and HEAD only. A drain inside a POST would put an email send |
| -// between somebody's form submission and its redirect, which is the one |
| -// moment they are already waiting. |
| +// The rearm runs on every method and AFTER the handler, which is what |
| +// makes it complete: every write to the reminder queue happens inside a |
| +// request in this group — creating an event, adding or removing a |
| +// reminder, editing a time, excluding an occurrence, deleting the lot — |
| +// so one call here covers all of them, and no future writer has to |
| +// remember to re-point the timer. It costs an indexed MIN and a |
| +// comparison; nothing reaches the control socket unless the answer |
| +// moved. |
| // |
| // The cost is honest: when a reminder IS due, one page load carries one |
| // SMTP send. The claim query is a single indexed read and returns |
| @@ -147,16 +162,17 @@ func (h *handlers) drainOpportunistically(next http.Handler) http.Handler { |
| h.DrainOnRequest(r.Context()) |
| } |
| next.ServeHTTP(w, r) |
| + h.rearm(r.Context()) |
| }) |
| } |
| // DrainOnRequest sends at most one overdue reminder. |
| // |
| // Three ways a reminder gets sent, and the redundancy is the point: |
| -// the platform's own wake from NextDue, the declared schedule's tick, |
| -// and this. If the first two are misconfigured the app still reminds |
| -// people, just less punctually, rather than silently never reminding |
| -// anybody. |
| +// DrainAtBoot on the platform's own wake, the tick — from the declared |
| +// 6h schedule or from the one-shot timer wake.go registers — and this. |
| +// If the first two are misconfigured the app still reminds people, just |
| +// less punctually, rather than silently never reminding anybody. |
| // |
| // Bounded to a single reminder per request so that loading a page never |
| // becomes slow because a queue got long. |
diff --git a/internal/calendar/store_reminder_next.go b/internal/calendar/store_reminder_next.go| index 8538dc3..2dee76b 100644 |
| --- a/internal/calendar/store_reminder_next.go |
| +++ b/internal/calendar/store_reminder_next.go |
| @@ -2,6 +2,7 @@ package calendar |
| import ( |
| "context" |
| + "database/sql" |
| "time" |
| ) |
| @@ -42,23 +43,45 @@ func (s *Store) nextOccurrenceAfter(e Event, now int64, offsetMin int) (int64, s |
| } |
| // NextDueAt is what Options.NextDue answers: the earliest moment this |
| -// instance has work. |
| +// instance has work IT CAN ACTUALLY DO. |
| // |
| // The platform asks a running instance this before hibernating it, and |
| // wakes it at the answer. That is what makes a reminder fire on a |
| // machine that is asleep almost all the time — without it, a reminder |
| // would arrive whenever the next person happened to load a page. |
| // |
| +// "Can actually do" is the whole subtlety, and getting it wrong is |
| +// expensive rather than merely late. A row a drain claimed and failed |
| +// to send is unclaimable for claimLease, deliberately — the lease IS |
| +// the backoff. Answering with its due_at, an instant already in the |
| +// past, told the platform to wake us at once for work no drain could |
| +// take: wake, claim nothing, idle fifteen minutes, hibernate, answer |
| +// the same past instant, wake again. One undeliverable address bought a |
| +// checkpoint-and-restore round trip every sixteen minutes, forever. |
| +// |
| +// So a claimed row answers when its lease expires and an unclaimed one |
| +// answers its due time, whichever comes first across the queue. |
| +// |
| // Zero means nothing is scheduled, and the platform hibernates |
| // indefinitely. |
| func (s *Store) NextDueAt(ctx context.Context) time.Time { |
| - var q ReminderQueue |
| - err := s.g.WithContext(ctx). |
| - Where("fired_at IS NULL"). |
| - Order("due_at ASC"). |
| - First(&q).Error |
| - if err != nil { |
| + row := s.g.WithContext(ctx).Raw(` |
| +SELECT MIN(CASE WHEN claimed_at IS NULL THEN due_at |
| + ELSE MAX(due_at, claimed_at + ?) END) |
| +FROM reminder_queues |
| +WHERE fired_at IS NULL`, int64(claimLease.Seconds())).Row() |
| + |
| + var next sql.NullInt64 |
| + if err := row.Scan(&next); err != nil { |
| + // Never a reason to refuse to hibernate: an instance that |
| + // cannot read its own queue will not send anything either, and |
| + // holding it awake only spends money to be equally broken. |
| + s.logger.Error("calendar: read next due reminder", |
| + "stage", "reminders.nextdue", "err", scrubErr(err)) |
| + return time.Time{} |
| + } |
| + if !next.Valid { |
| return time.Time{} |
| } |
| - return time.Unix(q.DueAt, 0) |
| + return time.Unix(next.Int64, 0) |
| } |
diff --git a/internal/calendar/testing.go b/internal/calendar/testing.go| index e6d8a65..3c27187 100644 |
| --- a/internal/calendar/testing.go |
| +++ b/internal/calendar/testing.go |
| @@ -2,6 +2,7 @@ package calendar |
| import ( |
| "bytes" |
| + "context" |
| "database/sql" |
| "fmt" |
| "io" |
| @@ -12,6 +13,8 @@ import ( |
| "os" |
| "path/filepath" |
| "strings" |
| + "sync" |
| + "time" |
| ) |
| // TB is the slice of *testing.T this file needs. |
| @@ -162,3 +165,71 @@ func (a *App) LastMailForTest() (string, error) { |
| } |
| return a.mailRecorder.last(), nil |
| } |
| + |
| +// WakeTimerCall is one thing the app asked the platform's timer |
| +// registry to do. |
| +type WakeTimerCall struct { |
| + Name string |
| + Path string |
| + At time.Time |
| + |
| + // Cancel is true for a removal, where At is the zero time. |
| + Cancel bool |
| +} |
| + |
| +// WakeTimerLog is what CaptureWakeTimerForTest hands back. |
| +type WakeTimerLog struct { |
| + mu sync.Mutex |
| + calls []WakeTimerCall |
| +} |
| + |
| +func (l *WakeTimerLog) add(c WakeTimerCall) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + l.calls = append(l.calls, c) |
| +} |
| + |
| +// Calls is every registration and cancellation, in order. |
| +func (l *WakeTimerLog) Calls() []WakeTimerCall { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + return append([]WakeTimerCall(nil), l.calls...) |
| +} |
| + |
| +// Last is the most recent call, and false when there were none. |
| +func (l *WakeTimerLog) Last() (WakeTimerCall, bool) { |
| + l.mu.Lock() |
| + defer l.mu.Unlock() |
| + if len(l.calls) == 0 { |
| + return WakeTimerCall{}, false |
| + } |
| + return l.calls[len(l.calls)-1], true |
| +} |
| + |
| +// CaptureWakeTimerForTest redirects the platform's schedule calls into |
| +// a log for the life of the test, and restores them afterwards. |
| +// |
| +// Swapping the two package vars rather than faking a control socket: |
| +// a socket test would be exercising rastrillo's transport instead of |
| +// this app's decisions, and a unix path under t.TempDir() sits close |
| +// enough to the 108-byte limit to fail on somebody else's machine. What |
| +// is worth asserting here is WHEN this app asks for a timer and what it |
| +// asks for — which is exactly what the vars carry. |
| +// |
| +// Off CARLOS the real calls answer ErrNotOnCarlos, so a test that does |
| +// not capture them exercises the same paths and observes nothing. |
| +func CaptureWakeTimerForTest(t TB) *WakeTimerLog { |
| + t.Helper() |
| + log := &WakeTimerLog{} |
| + atWas, cancelWas := scheduleAt, scheduleCancel |
| + scheduleAt = func(_ context.Context, name string, at time.Time, path string) error { |
| + log.add(WakeTimerCall{Name: name, Path: path, At: at}) |
| + return nil |
| + } |
| + scheduleCancel = func(_ context.Context, name string) error { |
| + log.add(WakeTimerCall{Name: name, Cancel: true}) |
| + return nil |
| + } |
| + t.Cleanup(func() { scheduleAt, scheduleCancel = atWas, cancelWas }) |
| + return log |
| +} |
diff --git a/internal/calendar/wake.go b/internal/calendar/wake.go| new file mode 100644 |
| index 0000000..de7f294 |
| --- /dev/null |
| +++ b/internal/calendar/wake.go |
| @@ -0,0 +1,154 @@ |
| +package calendar |
| + |
| +import ( |
| + "context" |
| + "errors" |
| + "log/slog" |
| + "sync" |
| + "time" |
| + |
| + "github.com/carlosframework/rastrillo/carlos" |
| +) |
| + |
| +// Making the platform's wake carry work with it. |
| +// |
| +// NextDue tells the activator WHEN to wake this instance. It says |
| +// nothing about what to do on arrival, and a wake is only a process |
| +// start: no request, no tick, nothing. So the app also registers a |
| +// one-shot platform timer, which fires as a POST to the drain route — |
| +// and drains at boot as well, for the wake that arrives with no timer |
| +// behind it. |
| + |
| +// wakeTimerName is this app's handle on its one-shot timer. |
| +// |
| +// Deliberately NOT "reminders", which the design document proposed and |
| +// which cannot work: `carlos schedule set` already declares a recurring |
| +// schedule of that name on this app, one-shot timers share that |
| +// namespace, and ScheduleAt answers ErrDeclaredSchedule for a declared |
| +// name. Every registration would have been refused — silently, since a |
| +// refusal is a log line the app moves on from. |
| +const wakeTimerName = "reminder-next" |
| + |
| +// wakePath is what the platform POSTs when the timer fires: the same |
| +// route the declared 6h schedule uses, behind the same carlos.Tick |
| +// guard. One handler, however the work arrives. |
| +const wakePath = "/jobs/reminders" |
| + |
| +// scheduleAt and scheduleCancel are the platform calls, held as vars so |
| +// a test can see what the app asks for without standing up a control |
| +// socket. CaptureWakeTimerForTest is the only thing that replaces them. |
| +var ( |
| + scheduleAt = carlos.ScheduleAt |
| + scheduleCancel = carlos.ScheduleCancel |
| +) |
| + |
| +// wakeTimer keeps one platform timer pointing at the next moment this |
| +// instance has work. |
| +// |
| +// ONE timer, upserted by name — never one per reminder. The host's |
| +// ceiling is 1,000 pending timers shared across every app on the box, |
| +// and a timer per occurrence would need exactly the materialised series |
| +// the queue refuses to hold. |
| +type wakeTimer struct { |
| + logger *slog.Logger |
| + |
| + mu sync.Mutex |
| + // at is the instant last successfully registered, so the common |
| + // case — nothing changed since the last page load — costs a |
| + // comparison instead of a socket round trip. Zero means cancelled. |
| + at int64 |
| + // known is false until the first assertion lands. Timers live in |
| + // the box's registry rather than in this process, so a fresh boot |
| + // inherits whatever the last one left and must assert once even if |
| + // the answer is "nothing". |
| + known bool |
| +} |
| + |
| +// assert points the timer at next, or cancels it when next is the zero |
| +// time. |
| +// |
| +// Failures are logged and NOT recorded as asserted, so the following |
| +// request retries. That is the right shape for the transient case (a |
| +// wedged agent) and harmless for the permanent one, because the drain |
| +// still has two other entrances. |
| +func (w *wakeTimer) assert(ctx context.Context, next time.Time) { |
| + var want int64 |
| + if !next.IsZero() { |
| + want = next.Unix() |
| + } |
| + |
| + // The lock is held across the socket call on purpose. Releasing it |
| + // first would let two requests register different instants and let |
| + // the slower one's answer win, which is a timer pointing at the |
| + // wrong moment with nothing to correct it. The call only happens |
| + // when the answer moved, so the common request pays a comparison. |
| + w.mu.Lock() |
| + defer w.mu.Unlock() |
| + if w.known && w.at == want { |
| + return |
| + } |
| + |
| + var err error |
| + if want == 0 { |
| + err = scheduleCancel(ctx, wakeTimerName) |
| + } else { |
| + err = scheduleAt(ctx, wakeTimerName, next, wakePath) |
| + } |
| + |
| + switch { |
| + case err == nil: |
| + w.at, w.known = want, true |
| + case errors.Is(err, carlos.ErrNotOnCarlos): |
| + // A laptop, or a test. Expected, and recorded as asserted so a |
| + // development log does not carry a line per page load. |
| + w.at, w.known = want, true |
| + case errors.Is(err, carlos.ErrDeclaredSchedule): |
| + // A configuration collision, not a transient failure: the name |
| + // belongs to a declared schedule and every call will be |
| + // refused until one of the two names changes. Said loudly |
| + // because the symptom otherwise is only that reminders are |
| + // late. |
| + w.logger.Error("calendar: the wake timer's name is a declared schedule; "+ |
| + "no one-shot timer can be registered and reminders will fire late", |
| + "stage", "reminders.timer", "name", wakeTimerName) |
| + default: |
| + w.logger.Error("calendar: assert wake timer", |
| + "stage", "reminders.timer", "err", scrubErr(err)) |
| + } |
| +} |
| + |
| +// rearm re-points the timer at whatever the queue now says is next. |
| +// |
| +// Cheap enough to call after every request: one indexed MIN over a |
| +// table with at most one row per reminder rule, then a comparison that |
| +// returns without touching the socket unless the answer moved. |
| +func (h *handlers) rearm(ctx context.Context) { |
| + h.timer.assert(ctx, h.store.NextDueAt(ctx)) |
| +} |
| + |
| +// DrainAtBoot is the entrance a platform wake needs, and the one the |
| +// design named that was never built. |
| +// |
| +// The activator asks a hibernating instance for its next due moment, |
| +// records it, and wakes the process at that instant. Nothing arrives |
| +// with the wake. Without this the woken instance served no request, |
| +// idled out its fifteen-minute window, hibernated, answered the same |
| +// still-overdue instant, and was woken again thirty seconds later. |
| +// Observed live on 2026-09-04: four wake/sleep pairs between 13:57 and |
| +// 14:45, no mail sent, and the reminder finally delivered 57 minutes |
| +// late when somebody loaded a page. |
| +// |
| +// Synchronous, before Serve. The agent binds this instance's socket |
| +// before it spawns the process, so a request arriving mid-drain waits |
| +// rather than failing, and the pass is bounded by drainDeadline. |
| +func (a *App) DrainAtBoot(ctx context.Context) { |
| + sent, err := a.h.drain(ctx, time.Now()) |
| + if err != nil { |
| + a.logger.Error("calendar: drain at boot", |
| + "stage", "reminders.drain", "err", scrubErr(err)) |
| + } |
| + if sent > 0 { |
| + a.logger.Info("calendar: reminders drained at boot", "sent", sent) |
| + } |
| + a.h.rearm(ctx) |
| +} |
diff --git a/internal/calendartest/wake_test.go b/internal/calendartest/wake_test.go| new file mode 100644 |
| index 0000000..e93c718 |
| --- /dev/null |
| +++ b/internal/calendartest/wake_test.go |
| @@ -0,0 +1,252 @@ |
| +package calendartest |
| + |
| +import ( |
| + "context" |
| + "net/url" |
| + "strings" |
| + "testing" |
| + "time" |
| + |
| + calendar "amadan.net/oficina/calendar/internal/calendar" |
| +) |
| + |
| +// A wake that carries no request must still send the reminder. |
| +// |
| +// This is the bug the whole file exists for, and it was live: the |
| +// activator asks a hibernating instance GET /api/next-due, records the |
| +// answer, and wakes the process at that moment. The process came up, |
| +// nobody visited it, the 6h tick was hours away — and the drain has no |
| +// boot entrance, so the wake did nothing at all. The instance idled its |
| +// fifteen minutes, hibernated, was asked for its next due time again, |
| +// answered with the SAME still-overdue instant, and was woken again |
| +// thirty seconds later. On 2026-09-04 that loop ran from 13:57 to |
| +// 14:45, four wake/sleep pairs, and the reminder went out 57 minutes |
| +// late when somebody finally loaded a page. |
| +// |
| +// The spec named three entrances — boot, tick, request. Only two were |
| +// built. |
| +func TestAWakeWithNoRequestStillSendsTheReminder(t *testing.T) { |
| + app, session := signedIn(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(2 * time.Minute) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"Woken for this"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(15 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"none"}, |
| + }) |
| + |
| + before, _ := app.LastMailForTest() |
| + |
| + // Exactly what main.go does after New and before Serve. No request, |
| + // no tick, no session — just a process that has come up. |
| + app.DrainAtBoot(context.Background()) |
| + |
| + after, err := app.LastMailForTest() |
| + if err != nil { |
| + t.Fatalf("LastMailForTest: %v", err) |
| + } |
| + if after == before { |
| + t.Fatal("a boot with an overdue reminder sent nothing — " + |
| + "the platform's wake has no work attached to it, so the instance " + |
| + "will hibernate and be woken for the same reminder forever") |
| + } |
| + if !strings.Contains(after, "Woken for this") { |
| + t.Errorf("the boot drain sent something else:\n%s", after) |
| + } |
| +} |
| + |
| +// Queueing a reminder must register the one-shot platform timer, so the |
| +// wake arrives as a POST to the drain rather than as a bare process |
| +// start. |
| +// |
| +// NextDue alone is only half the mechanism: it tells the activator WHEN |
| +// to wake us and nothing about what to do on arrival. ScheduleAt is the |
| +// half that delivers the work. |
| +func TestQueueingAReminderAssertsTheWakeTimer(t *testing.T) { |
| + timers := calendar.CaptureWakeTimerForTest(t) |
| + |
| + app, session := signedIn(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(3 * time.Hour) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"Later"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(30 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"none"}, |
| + }) |
| + |
| + call, ok := timers.Last() |
| + if !ok { |
| + t.Fatal("creating an event with a reminder registered no platform timer — " + |
| + "the platform will wake this instance with nothing to do") |
| + } |
| + if call.Cancel { |
| + t.Fatalf("the last timer call was a cancel, with a reminder queued") |
| + } |
| + if call.Path != "/jobs/reminders" { |
| + t.Errorf("the timer POSTs to %q, want /jobs/reminders", call.Path) |
| + } |
| + // Ten minutes before the event, which is the person's default. |
| + want := start.Add(-10 * time.Minute) |
| + if diff := call.At.Sub(want); diff > time.Minute || diff < -time.Minute { |
| + t.Errorf("the timer is set for %v, want about %v", call.At, want) |
| + } |
| +} |
| + |
| +// The timer's name may not be the name of the declared schedule. |
| +// |
| +// `carlos schedule set` already declares a recurring schedule called |
| +// "reminders" on this app, and the platform keeps one-shot timers in |
| +// that same namespace: ScheduleAt refuses a declared name with |
| +// ErrDeclaredSchedule. The design document says to call the timer |
| +// "reminders", which would have failed on every single call — silently, |
| +// because there is nothing to notice a refusal the app logs and moves |
| +// on from. |
| +func TestTheWakeTimerDoesNotCollideWithTheDeclaredSchedule(t *testing.T) { |
| + timers := calendar.CaptureWakeTimerForTest(t) |
| + |
| + app, session := signedIn(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(3 * time.Hour) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"Later"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(30 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"none"}, |
| + }) |
| + |
| + call, ok := timers.Last() |
| + if !ok { |
| + t.Fatal("no timer was registered at all") |
| + } |
| + if call.Name == "reminders" { |
| + t.Error(`the one-shot timer is named "reminders", which is the declared ` + |
| + `schedule's name — the platform answers 409 and every registration ` + |
| + `is refused`) |
| + } |
| +} |
| + |
| +// A send that failed must not leave the instance in a wake/sleep loop. |
| +// |
| +// FailReminder refreshes the claim rather than clearing it, so the row |
| +// is deliberately unclaimable for five minutes. NextDue reported its |
| +// due_at anyway — an instant in the PAST — so the platform woke the |
| +// instance immediately, the drain could claim nothing, the instance |
| +// idled fifteen minutes and hibernated, and was woken again at once. |
| +// One bad address therefore bought a checkpoint-and-restore round trip |
| +// every sixteen minutes, indefinitely. |
| +// |
| +// NextDue must answer when there is work this instance can actually |
| +// DO. |
| +func TestAFailedSendDoesNotWakeTheInstanceInALoop(t *testing.T) { |
| + t.Setenv("CARLOS_ADMIN_TOKEN", "test-token") |
| + |
| + mailer := &pickyMailer{refuse: "Doomed"} |
| + app := newAppWithMailer(t, mailer) |
| + session := app.SignInForTest(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(1 * time.Minute) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"Doomed standup"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(15 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"daily"}, |
| + }) |
| + |
| + if rec := tick(t, h); rec.Code != 204 { |
| + t.Fatalf("drain: status %d, want 204\n%s", rec.Code, rec.Body.String()) |
| + } |
| + |
| + due := app.NextDue() |
| + if due.IsZero() { |
| + t.Fatal("a failed send unqueued the reminder entirely") |
| + } |
| + if !due.After(time.Now()) { |
| + t.Errorf("NextDue = %v, which is already past. The row is claimed for "+ |
| + "another five minutes and cannot be drained, so the platform will "+ |
| + "wake this instance, find nothing to do, idle, hibernate and wake "+ |
| + "again — forever", due) |
| + } |
| +} |
| + |
| +// Draining must leave the timer pointing at whatever is next, not at |
| +// the occurrence just sent. |
| +func TestTheDrainRearmsTheTimerForTheNextOccurrence(t *testing.T) { |
| + t.Setenv("CARLOS_ADMIN_TOKEN", "test-token") |
| + timers := calendar.CaptureWakeTimerForTest(t) |
| + |
| + app, session := signedIn(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(2 * time.Minute) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"Daily thing"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(15 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"daily"}, |
| + }) |
| + |
| + if rec := tick(t, h); rec.Code != 204 { |
| + t.Fatalf("drain: status %d, want 204", rec.Code) |
| + } |
| + |
| + call, ok := timers.Last() |
| + if !ok { |
| + t.Fatal("the drain registered no timer for the next occurrence") |
| + } |
| + if call.Cancel { |
| + t.Fatal("the drain cancelled the timer with the series still running") |
| + } |
| + if until := time.Until(call.At); until < 20*time.Hour || until > 26*time.Hour { |
| + t.Errorf("the rearmed timer is %v away, want about a day", until) |
| + } |
| +} |
| + |
| +// With nothing left to warn about, the timer is cancelled rather than |
| +// left pointing at an instant that will wake the instance for no |
| +// reason. |
| +func TestTheLastReminderCancelsTheTimer(t *testing.T) { |
| + t.Setenv("CARLOS_ADMIN_TOKEN", "test-token") |
| + timers := calendar.CaptureWakeTimerForTest(t) |
| + |
| + app, session := signedIn(t, "someone@example.com") |
| + h := app.Handler() |
| + |
| + start := time.Now().UTC().Add(2 * time.Minute) |
| + createEvent(t, h, session, url.Values{ |
| + "title": {"One off"}, |
| + "start": {start.Format("2006-01-02T15:04")}, |
| + "end": {start.Add(15 * time.Minute).Format("2006-01-02T15:04")}, |
| + "kind": {"meeting"}, |
| + "status": {"confirmed"}, |
| + "repeat": {"none"}, |
| + }) |
| + |
| + if rec := tick(t, h); rec.Code != 204 { |
| + t.Fatalf("drain: status %d, want 204", rec.Code) |
| + } |
| + |
| + call, ok := timers.Last() |
| + if !ok { |
| + t.Fatal("no timer calls at all") |
| + } |
| + if !call.Cancel { |
| + t.Errorf("with the queue empty the app left a timer set for %v — "+ |
| + "the platform will wake this instance for nothing", call.At) |
| + } |
| +} |