amadan
console/cli/spec: run wakes on the tick, 502 on an unlistable status prefix, doc defects
I4: the CLI header and `carlos schedule` usage, and handleAPIScheduleRun's doc, said a hibernating instance is never woken by these verbs — the opposite of what happens. Now: the console write wakes nothing; the box's runner wakes the instance when the tick is due, within ~15s of a run. I5: handleAPIScheduleList answers 502 "could not read instance reports — try again" when ListStatus could not list at all (nil, err); a partial result with a joined error is still 200 with what is knowable. Both pinned. I8: schedule.Mount's doc no longer claims the bearer closes the same-uid sibling gap — purge.go's framing (other UNIX users + unauthenticated dialers; the same-uid residual is reach, not privilege; per-instance uids are the fix). Spec §8's sentence about svcroutes_test.go's door lists replaced with what that test actually spells out. Minors: purge_listen_windows.go names ListenControl's caller and the CARLOS_CONTROL_SOCKET strip; spec §4.3 example gains due_at, §6.3/§7.1 say precisely what X-Carlos-Schedule-At is and when it moves, §7 gains the app-author notes (token in the environment, Host and X-Forwarded-*, a curl for PUT /schedule, the deploy-straddle cut, once-timers at boot), §11 step 2 says what ls shows for an instance asleep since before the roll.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
6 files changed,
+161
−23
docs/superpowers/specs/2026-08-23-scheduled-work-design.md+77 −10internal/console/apischedule.go+16 −3internal/console/apischedule_test.go+46 −0internal/edge/purge_listen_windows.go+4 −2internal/schedule/control.go+11 −4schedule.go+7 −4
diff --git a/docs/superpowers/specs/2026-08-23-scheduled-work-design.md b/docs/superpowers/specs/2026-08-23-scheduled-work-design.md| index cf1622e1e..c35738c7c 100644 |
| --- a/docs/superpowers/specs/2026-08-23-scheduled-work-design.md |
| +++ b/docs/superpowers/specs/2026-08-23-scheduled-work-design.md |
| @@ -154,11 +154,17 @@ converge that changed the host's rows: |
| "unsupported": "", |
| "timers": [ |
| {"name": "sync", "kind": "schedule", "spec": "every 6h", "path": "/jobs/sync", |
| - "next_at": 1787004000, "last_at": 1786982400, "last_status": "200", "last_ms": 3210, "attempts": 0} |
| + "next_at": 1787004000, "due_at": 1787004000, |
| + "last_at": 1786982400, "last_status": "200", "last_ms": 3210, "attempts": 0} |
| ] |
| } |
| ``` |
| +`due_at` is the occurrence `next_at` will deliver — the value the tick's |
| +`X-Carlos-Schedule-At` carries (§6.3). They differ after a retry (`next_at` |
| +is the retry instant, `due_at` still the occurrence being retried) and are |
| +`0` on a row a pre-`due_at` box wrote. |
| + |
| This is what makes `carlos schedule ls` honest for a member with no box |
| access (CLAUDE.md, "Operate via API and CLI"): the declaration alone says |
| what was asked; the status objects say what each instance is actually going |
| @@ -294,7 +300,7 @@ DueTimers(now, 64) |
| | `Authorization` | `Bearer <CARLOS_ADMIN_TOKEN>` — the instance's own token (§6.4) | |
| | `X-Carlos-Schedule` | the timer name | |
| | `X-Carlos-Schedule-Kind` | `schedule` / `once` | |
| -| `X-Carlos-Schedule-At` | the unix time this occurrence was due | |
| +| `X-Carlos-Schedule-At` | the **occurrence**: the row's `due_at` — the natural instant a schedule was due, the `at` of a once-timer, or the moment of a `carlos schedule run`. Identical on every retry of that occurrence; moves only when the row advances. (Falls back to `next_at` on a row written before `due_at` existed.) | |
| | `User-Agent` | `carlos-schedule/<build>` | |
| | `Content-Length` | `0` | |
| @@ -325,10 +331,21 @@ billed wire bytes for its own tick, and a `POST` is never cache-served. |
| **At-least-once** is the guarantee: a box that dies between wake and tick |
| still has the row due, and fires it again on recovery. The handler must be |
| -idempotent for its occurrence (`X-Carlos-Schedule-At` is the key an app |
| -should dedupe on if it cares). Nothing is ever fired twice *concurrently* for |
| -one host: the runner serializes per host and marks the row before taking the |
| -next. |
| +idempotent for its occurrence, and `X-Carlos-Schedule-At` is that |
| +occurrence's key: the box stores it as `due_at`, set when a row is created |
| +or its spec/path changes (`= next_at`), when a fire advances the row (`=` |
| +the new `next_at`), and by `run` (`= now`); a **retry keeps it** — only |
| +`next_at` moves to the retry instant. So the five attempts of one failed |
| +occurrence all carry the same value, and a redelivery after a box crash |
| +carries the value the first attempt did. Nothing is ever fired twice |
| +*concurrently* for one host: the runner keeps a per-host in-flight mark |
| +across ticks and skips a host whose delivery is still running, and marks the |
| +row before taking the next. |
| + |
| +A tick launches per-host work and returns; it never waits for a delivery. |
| +A 20-minute handler on one host holds only its own host's next rows, never |
| +another host's — the bound is `MaxParallelHosts` (4) deliveries in flight |
| +across the box, not per tick. |
| A timer whose `next_at` is in the past by more than one occurrence when the |
| runner sees it (box was down) fires **once** — §6.1's coalescing. |
| @@ -401,11 +418,39 @@ Rules the doc states plainly: |
| compare (seapointish's `platform.go` is the reference). A tick without a |
| valid token is an internet request to that path; treat it as one. |
| - Work inside the request. 2xx = done; 5xx = please retry; 4xx = don't. |
| -- Be idempotent per occurrence: `X-Carlos-Schedule-At` identifies it. |
| +- Be idempotent per occurrence: `X-Carlos-Schedule-At` identifies it, and |
| + it is stable across retries — every retry of one failed occurrence |
| + carries the same value, and a redelivery after a box crash carries the |
| + value the first attempt did (§6.3). A schedule's value is the natural |
| + instant it was due; a once-timer's is its `at`; a `carlos schedule run` |
| + is its own occurrence, stamped with the moment of the run. If you record |
| + which occurrences you have completed, key on this header, never on the |
| + wall clock. |
| - The handler is an ordinary route; the app may also call it from a |
| "Sync now" button (with its own auth). Nothing about a tick is magic |
| beyond the headers. |
| +What else the app can rely on: |
| + |
| +- **The token is in the environment.** `CARLOS_ADMIN_TOKEN` is in the |
| + instance's process environment (minted by the platform, §6.4); compare |
| + the bearer against it. Never accept a tick on a header alone. |
| +- **The tick looks like a request from the edge.** `Host` is the route host |
| + (`acme.bdn.oncarlos.com`), and the unix-socket proxy sets |
| + `X-Forwarded-Host`/`X-Forwarded-Proto` as it does for a visitor. There is |
| + no client address behind a tick. Nothing the edge would have added — |
| + cache, compression, usage accounting — is in the path. |
| +- **A long job that straddles a deploy is cut, and re-run.** A deploy |
| + cycles the instance after its converge grace (~90s); a tick still open |
| + at that point is cut with the process, the runner sees a transport |
| + error and retries — at-least-once, so the occurrence runs again. Jobs |
| + longer than that should be resumable from the occurrence key. |
| +- **Once-timers survive a restart, and you may re-assert them at boot.** |
| + They live in the box's registry, not the process, so a restart loses |
| + none. An app that keeps its own record and re-registers on start can: |
| + the control socket is bound BEFORE the spawn, so `PUT /schedule/{name}` |
| + works from the first line of `main`. |
| + |
| ### 7.2 Registering a once-timer (control socket) |
| The agent already hands exec children `CARLOS_CONTROL_SOCKET` (purge |
| @@ -420,6 +465,14 @@ keeps its "only when true" rule). |
| | `DELETE /schedule/{name}` | `204`, idempotent | |
| | `GET /schedule` | this host's timers, both kinds (the §4.3 shape) | |
| +From inside the instance, with nothing but curl: |
| + |
| +```sh |
| +curl --unix-socket "$CARLOS_CONTROL_SOCKET" -X PUT http://carlos/schedule/remind \ |
| + -H "Authorization: Bearer $CARLOS_ADMIN_TOKEN" \ |
| + -d '{"at":"2026-09-01T08:00:00Z","path":"/jobs/remind"}' |
| +``` |
| + |
| **Auth (decision):** every request carries `Authorization: Bearer |
| <CARLOS_ADMIN_TOKEN>` — the same token the tick presents, so the app already |
| has it. Be precise about what that buys. The bearer closes the channel |
| @@ -474,8 +527,17 @@ seconds. |
| Audit lines: `schedule set <name>`, `schedule rm <name>`, `schedule run |
| <name>`. Store errors are logged, never returned (they carry bucket names). |
| -`svcroutes_test.go`'s door lists are extended — the new doors are Operate |
| -doors, and the test that pins the closed set must be told. |
| +The one exception in shape, not in rule: `GET …/schedules` answers 502 with |
| +a fixed sentence when the status prefix cannot be listed at all — a 200 |
| +with `instances: []` there would read as "no instance has reported yet" for |
| +a bucket the console could not reach. Some reports plus an unreadable one is |
| +still 200 with what is knowable, the rest logged. |
| + |
| +`svcroutes_test.go` spells out only two lists: the doors a zero-role |
| +service principal may reach (the equality on the empty role set) and the |
| +families a service principal may never open. Operate doors join neither, so |
| +the new doors need no entry there — they are covered by the role gate |
| +itself. |
| ## 9. The CLI: `carlos schedule` |
| @@ -547,7 +609,12 @@ Verified live, in this order, with zero box access: |
| 1. `carlos schedule set -app <a> -name ping -every 1m -path /jobs/ping` on an |
| app whose handler exists (a test app on the flagship account). |
| -2. `carlos schedule ls` shows `next` per instance within ~5s. |
| +2. `carlos schedule ls` shows `next` per instance within ~5s — including an |
| + instance that has been hibernating since before the roll: the converge |
| + writes its rows and report without waking it, and its first tick mints |
| + the admin token it never had (§6.4) and wakes it. Before that fix, such |
| + an instance showed `last … → no instance token` five times and then a |
| + `failed after 5 attempts` line, and was never woken. |
| 3. `carlos logs -app <a> -f` shows `schedule ping fired POST /jobs/ping → 204` |
| within the minute; the instance was asleep when it fired. |
| 4. `carlos schedule run -name ping` → a fire within ~15s. |
diff --git a/internal/console/apischedule.go b/internal/console/apischedule.go| index f6d22c723..4a7ee8cd0 100644 |
| --- a/internal/console/apischedule.go |
| +++ b/internal/console/apischedule.go |
| @@ -149,6 +149,13 @@ func (s *Server) declaredSchedule(ctx context.Context, w http.ResponseWriter, r |
| // the other way round: it is the answer, not a contribution to it, so a store |
| // error there is a 502. |
| // |
| +// The line between those two is whether ListStatus could LIST at all. A |
| +// failed listing comes back as (nil, err) and means nothing is knowable — |
| +// answering 200 with `instances: []` there would print "no instance has |
| +// reported yet" for a bucket the console could not reach, and a member would |
| +// read a broken console as a fleet that has not caught up. That is a 502. |
| +// Some reports plus a joined error is the partial case above. |
| +// |
| // No write budget and no audit line: it is a read. |
| func (s *Server) handleAPIScheduleList(w http.ResponseWriter, r *http.Request) { |
| _, acct, app, ok := s.apiScheduleApp(w, r, false, "this console cannot read schedules") |
| @@ -166,6 +173,10 @@ func (s *Server) handleAPIScheduleList(w http.ResponseWriter, r *http.Request) { |
| insts, err := schedule.ListStatus(ctx, s.cfg.Releases, acct.Sqid, app) |
| if err != nil { |
| log.Printf("console: api schedule status %s/%s: %v", acct.Sqid, app, err) |
| + if insts == nil { |
| + jsonError(w, http.StatusBadGateway, "could not read instance reports — try again") |
| + return |
| + } |
| } |
| writeJSON(w, http.StatusOK, scheduleAnswer(acct.Sqid, app, d, insts)) |
| } |
| @@ -256,9 +267,11 @@ func (s *Server) handleAPIScheduleRemove(w http.ResponseWriter, r *http.Request) |
| // then announce — the same mechanism and the same rules as restart-gen. |
| // |
| // It answers "requested", not "fired", for restart's reason: what lands here |
| -// is a request in a bucket. Whether a given instance fires it depends on that |
| -// box seeing the serial move, and on the instance being awake — hibernating |
| -// ones are never woken by this. |
| +// is a request in a bucket. This write wakes nothing itself; each box's |
| +// converger marks the row due when it sees the serial move, and the box's |
| +// runner delivers on its next tick — waking a hibernating instance for it |
| +// the way a visitor would (spec §6.3) — so a fire follows within ~15s, and |
| +// `carlos logs` is where it shows. |
| func (s *Server) handleAPIScheduleRun(w http.ResponseWriter, r *http.Request) { |
| addr, acct, app, ok := s.apiScheduleApp(w, r, true, "this console cannot request a run") |
| if !ok { |
diff --git a/internal/console/apischedule_test.go b/internal/console/apischedule_test.go| index 601d8f09b..0ef43eb3e 100644 |
| --- a/internal/console/apischedule_test.go |
| +++ b/internal/console/apischedule_test.go |
| @@ -417,3 +417,49 @@ func TestAPIScheduleWithoutABucketIs503(t *testing.T) { |
| } |
| } |
| } |
| + |
| +// scheduleListBrokenStore cannot LIST the status prefix at all: the shape a |
| +// bucket outage takes on the read door. |
| +type scheduleListBrokenStore struct{ release.Store } |
| + |
| +func (b scheduleListBrokenStore) List(context.Context, string) ([]string, error) { |
| + return nil, errors.New(brokenStoreErr) |
| +} |
| + |
| +// A total failure to read the instance reports is a 502 with a fixed |
| +// sentence, never a 200 that reads as "no instance has reported yet": a |
| +// member with no box access would take a broken console for a fleet that |
| +// has not caught up. The bucket's own error text stays in the log. |
| +func TestAPIScheduleListTotalStatusFailureIs502(t *testing.T) { |
| + fx, token := scheduleFixture(t, "code-schedule-ls-broken") |
| + fx.srv.cfg.Releases = scheduleListBrokenStore{fx.rel} |
| + status, raw := getRaw(t, fx.ts.URL+"/api/cli/apps/bab/titogo/schedules", token) |
| + if status != http.StatusBadGateway { |
| + t.Fatalf("list with an unlistable status prefix = %d %s, want 502", status, raw) |
| + } |
| + if !bytes.Contains([]byte(raw), []byte("could not read instance reports")) || bytes.Contains([]byte(raw), []byte("carlos-prod-secret-bucket")) { |
| + t.Fatalf("502 body = %s, want the fixed sentence and none of the store's text", raw) |
| + } |
| +} |
| + |
| +// One unreadable report among readable ones is the PARTIAL case: the door |
| +// answers 200 with what is knowable and logs the rest, so one box writing |
| +// rubbish never blanks the list for every other instance of the app. |
| +func TestAPIScheduleListPartialStatusFailureIs200(t *testing.T) { |
| + fx, token := scheduleFixture(t, "code-schedule-ls-partial") |
| + good := schedule.Status{Host: "good.bab.oncarlos.com", Box: "carlos-ie-1", ReportedAt: time.Now().UTC().Format(time.RFC3339)} |
| + if err := schedule.WriteStatus(context.Background(), fx.rel, release.OpsAccount, "titogo", good); err != nil { |
| + t.Fatal(err) |
| + } |
| + if _, err := fx.rel.Put(context.Background(), schedule.StatusKey(release.OpsAccount, "titogo", "bad.bab.oncarlos.com"), []byte("{not json"), ""); err != nil { |
| + t.Fatal(err) |
| + } |
| + status, raw := getRaw(t, fx.ts.URL+"/api/cli/apps/bab/titogo/schedules", token) |
| + if status != http.StatusOK { |
| + t.Fatalf("list with one bad report = %d %s, want 200", status, raw) |
| + } |
| + res := decodeScheduleList(t, []byte(raw)) |
| + if len(res.Instances) != 1 || res.Instances[0].Host != "good.bab.oncarlos.com" { |
| + t.Fatalf("instances = %+v, want the one readable report", res.Instances) |
| + } |
| +} |
diff --git a/internal/edge/purge_listen_windows.go b/internal/edge/purge_listen_windows.go| index 59b875fb8..cd213e987 100644 |
| --- a/internal/edge/purge_listen_windows.go |
| +++ b/internal/edge/purge_listen_windows.go |
| @@ -15,8 +15,10 @@ import ( |
| // a listener would bind and serve while the umask that makes it owner-only |
| // does not exist, quietly downgrading a security property into a no-op on |
| // the one platform nobody here can test. An error at the boundary keeps the |
| -// guarantee honest. ListenPurge's caller (internal/activator/backend_exec.go) |
| -// already treats a failure here as "no purge listener". |
| +// guarantee honest. The caller is ListenControl's, in |
| +// internal/activator/backend_exec.go, which treats a failure here as "no |
| +// control listener": the instance runs without its control verbs and |
| +// CARLOS_CONTROL_SOCKET is stripped from its environment. |
| func secureUnixListen(socketPath string) (net.Listener, error) { |
| return nil, errors.New("purge listeners need a unix socket with owner-only permissions; not supported on windows") |
| } |
diff --git a/internal/schedule/control.go b/internal/schedule/control.go| index 7f37a1c2a..2828e5fcf 100644 |
| --- a/internal/schedule/control.go |
| +++ b/internal/schedule/control.go |
| @@ -23,10 +23,17 @@ type TimerStore interface { |
| // Mount adds the once-timer verbs to an instance's control socket (spec |
| // §7.2). host is fixed here and never read from a request — the |
| -// confused-deputy guarantee edge.ListenPurge gives — and every verb needs |
| -// the host's own CARLOS_ADMIN_TOKEN as a bearer, which is what closes the |
| -// same-uid-sibling gap purge.go documents: a sibling can compute this |
| -// socket's path, but not this instance's token. |
| +// confused-deputy guarantee edge.ListenControl gives — and every verb needs |
| +// the host's own CARLOS_ADMIN_TOKEN as a bearer. Be precise about what that |
| +// bearer buys (internal/edge/purge.go has the long form): it closes the |
| +// channel against OTHER UNIX users on the box and against any |
| +// unauthenticated dialer — reaching the socket is no longer enough. It does |
| +// NOT close the same-uid sibling gap: every exec child shares the agent's |
| +// uid, the token file is 0600 owner-only, so a sibling can read another |
| +// host's token and present it here. That residual is bounded — a sibling |
| +// with host B's token could already dial B's own socket with it; these |
| +// verbs add reach (a platform wake plus a POST at a chosen time and path), |
| +// not privilege — and per-instance uids are the real fix. |
| func Mount(mux *http.ServeMux, host string, store TimerStore, token func() (string, error), now func() time.Time) { |
| authed := func(fn http.HandlerFunc) http.HandlerFunc { |
| return func(w http.ResponseWriter, r *http.Request) { |
diff --git a/schedule.go b/schedule.go| index f92b5d0a7..69e809b97 100644 |
| --- a/schedule.go |
| +++ b/schedule.go |
| @@ -2,8 +2,10 @@ |
| // front over internal/console/apischedule.go). Pull-based, restart's shape |
| // and reason: the console touches one small object (schedules.json, or the |
| // run-signal touch object beside it), and every box serving the app notices |
| -// within seconds and acts on its own. Nothing is pushed at a box, no |
| -// hibernating instance is woken by any of these four verbs. |
| +// within seconds and acts on its own. Nothing is pushed at a box: the |
| +// console write itself wakes nothing, and it is the box's own runner that |
| +// wakes a hibernating instance when its tick is due — within ~15s of a |
| +// `run`, or at the next natural occurrence of a schedule. |
| // |
| // The vocabulary follows restart's honesty rule on purpose: `set` says |
| // "recorded", `rm` says "removed", `run` says "requested" — never "fired" |
| @@ -42,8 +44,9 @@ const scheduleUsage = `usage: carlos schedule <ls|set|rm|run> [flags] |
| -console <origin> console to act through (default: the only logged-in one) |
| Signed-in only (carlos auth login): a schedule is a console-authorized write, |
| -the same shape as carlos restart — boxes act on it within seconds, and a |
| -hibernating instance is never woken by any of these four verbs. |
| +the same shape as carlos restart — boxes act on it within seconds. The write |
| +itself wakes nothing; a box's runner wakes a hibernating instance when its |
| +tick is due (within ~15s of a run, or at the schedule's next occurrence). |
| ` |
| const scheduleLsUsage = `usage: carlos schedule ls -app <a> [-account <a>] [-account-id <id>] [-console <origin>] |