amadan
spec: carlos deploy — seconds-to-live for every non-baking channel
Design for the deploy signal (control/serial), the generalized fast converge set (#101), Materialize on cold wake (#77), the carlos deploy verb, and the X-Carlos-Version edge header.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+257
−0
docs/superpowers/specs/2026-08-08-carlos-deploy-design.md+257 −0
diff --git a/docs/superpowers/specs/2026-08-08-carlos-deploy-design.md b/docs/superpowers/specs/2026-08-08-carlos-deploy-design.md| new file mode 100644 |
| index 000000000..6122f2550 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-08-carlos-deploy-design.md |
| @@ -0,0 +1,257 @@ |
| +# carlos deploy — seconds-to-live for every non-baking channel |
| + |
| +2026-08-08. Extends the pipelines spec (2026-08-02-pipelines-design.md), the |
| +instance-provisioning spec (2026-08-04, "Known gap (Task 8 review)"), and the |
| +bake-controls spec (2026-08-04). Implements the principle of issue #101 |
| +("Nothing should wait minutes to wire up once the information to do it |
| +exists") and the preferred fix of issue #77 (Materialize). Touches |
| +`internal/host` (agent.go), `internal/activator` (backend_exec.go), |
| +`internal/release` (adopt.go, a new serial object), the edge proxy |
| +(response header), and the CLI (`carlos deploy`). |
| + |
| +## 1. Problem |
| + |
| +`carlos ship` + `carlos promote` complete in seconds, and then the feature is |
| +live... eventually. The latency between "the bucket has everything needed" |
| +and "the URL serves it" is three loops that each chose a cadence in |
| +isolation (issue #101's framing): |
| + |
| +| Step | Owner | Cadence today | |
| +|---|---|---| |
| +| Route mint for a new instance | `instancesLoop` (`internal/host/agent.go`) | 5 min + jitter | |
| +| Adoption (stage + symlink + try-restart) | `adoptLoop` | 1 h + 0–10 min jitter | |
| +| Adoption, `canary/*` routes only | `canaryAdoptLoop` | 2 s + jitter | |
| +| Cold wake with nothing staged | `ExecBackend.Materialize` | **never** (nil in production → 503) | |
| + |
| +The per-app `production` flag (bake-controls spec) is NOT the missing piece: |
| +it gates only the supply-chain bake window, which non-production apps already |
| +skip. The wait is pure poll cadence. |
| + |
| +Routes on `canary/*` channels already have the target experience: promote → |
| +staged and restarted in ~3–5 s. The problem is that the fast path is scoped |
| +by *channel name* when the useful scope is *outstanding work* — plus one gap |
| +the name-scope hides: a box has no way to notice a fresh promote to an |
| +already-converged non-canary route without polling that channel's pointer, |
| +and polling every pointer at 2 s is the poll storm #101 warns against. |
| + |
| +## 2. Shape of the fix |
| + |
| +Four pieces, all pull-based. The console never gains an inbound path to a |
| +box (#101's hard constraint); boxes keep converging with the console down. |
| + |
| +1. **A deploy signal** — one bucket object the console bumps on writes that |
| + create immediate box work; boxes poll that single object every 2 s and run |
| + the normal (fully verified, hold-respecting) pass set on a change. |
| +2. **A fast set** — routes with outstanding work are swept every 2 s until |
| + converged, regardless of channel name; the set drains to empty. |
| + Subsumes `canaryAdoptLoop`. |
| +3. **Materialize on cold wake** — an unstaged exec route stages its channel |
| + head on demand at wake instead of 503ing until the hourly pass. |
| +4. **`carlos deploy`** — ship → promote → watch until the URL provably |
| + serves the shipped version, via a new `X-Carlos-Version` edge response |
| + header. |
| + |
| +## 3. The deploy signal: `control/serial` |
| + |
| +A new unsigned bucket object at `control/serial` (top level, NOT under |
| +`channels/` or any per-app prefix — nothing that lists or parses app shapes |
| +sees it): |
| + |
| +```json |
| +{"schema": 1, "serial": 41, "bumped_at": "2026-08-08T10:15:00Z"} |
| +``` |
| + |
| +**Console bumps it** (read-increment-write, last-writer-wins — see below) |
| +after any bucket write a box loop consumes: channel promote, instance record |
| +create/enable/update, app config (env) write, production-flag flip. One bump |
| +may coalesce many writes; that is the point. |
| + |
| +**Boxes poll it every 2 s** in the fast loop (§4). One GET per box per 2 s — |
| +O(1) in fleet size, ~43k requests/day/box, cents/month. The box remembers |
| +the last serial it acted on (in-memory is enough; a restart just triggers |
| +one extra pass) and reacts to **any change**, not only increments — a reset |
| +or races between two console instances must never wedge the box into |
| +ignoring bumps. On change it runs the standard pass set: instances |
| +reconcile (`provision.Reconcile`), adopt pass (`release.Pass`), exec config |
| +sync — each with every signature check, bake hold, and ratchet exactly as |
| +today. Full pass-set runs are clamped to at most one per 5 s so a bump storm |
| +degrades to steady-state polling, not a stampede. |
| + |
| +**Security posture.** The object is unsigned because it carries no |
| +authority: it can only trigger *verified* work sooner. An attacker with |
| +bucket write access could bump it forever — and could already write garbage |
| +pointers that fail verification; either way the damage is bounded extra |
| +GETs, capped by the clamp. Verification of what gets adopted is unchanged. |
| + |
| +**Wire shape (additive, fail-safe).** New object; nothing existing parses |
| +it. Consumer list (everything that PARSES the shape, per the repo rule): the |
| +box fast loop, and its tests. Old boxes ignore it and keep the hourly |
| +cadence — the safe direction. Operator-side scripts that write pointers |
| +directly (pinfra `ship-app.sh`/`promote-app.sh`) do not bump; those flows |
| +stay hourly until the scripts learn to (non-goal here). |
| + |
| +**Console concurrency.** Two console processes bumping concurrently may |
| +lose an increment (S3 has no CAS on this path). Acceptable: the value's |
| +magnitude is meaningless, only *change* is; a lost increment still changed |
| +the value once, and the writes it announced are found by the pass the other |
| +bump triggers. `bumped_at` uses the console clock and is informational. |
| + |
| +## 4. The fast set: cadence as a function of outstanding work |
| + |
| +`canaryAdoptLoop` generalizes into a **fast converge loop**: |
| + |
| +- **Membership** (a route has outstanding work) — any of: |
| + - no live symlink yet (newly minted, or staging previously failed); |
| + - the last adopt pass computed an unstaged, hold-eligible head for it |
| + (the pass already computes this "as its ordinary business" — #101); |
| + - its instance record exists but the route is not yet minted |
| + (`provision.Reconcile`'s outstanding case). |
| +- **Cadence**: 2 s + jitter while the set is non-empty; the loop goes |
| + entirely quiet (serial polling only) when it drains. Per-route failure |
| + backoff doubles 2 s → 5 min cap, so a permanently broken route (missing |
| + artifact, spurious 403s under load — #101's cost note) cannot hammer the |
| + store forever; the hourly pass remains the floor for everything. |
| +- **Bake interaction**: a route whose head is inside a bake window is NOT |
| + outstanding work — it is *held*, drops out of the fast set, and re-enters |
| + when the window expires (the hourly pass or a serial bump re-notices it). |
| + Fast convergence never polls a held route at 2 s for 72 h. |
| +- `canary/*` routes lose their special case: a canary promote bumps the |
| + serial like any other, the pass notices the unstaged head, the route joins |
| + the fast set, converges, drains. Behavior is equal or better; the |
| + channel-name check (`isCanaryRoute`) is deleted. Production channel names |
| + pinned in `internal/release/laddercheck_test.go` are untouched — this |
| + changes when pointers are *polled*, never which channels are *legal*. |
| +- The sandbox rules move with the code: the fast loop stays inside |
| + `carlos-edge.service` (ProtectSystem=strict), so the standing DO-NOTs on |
| + `canaryAdoptLoop` — no `NagExpiredCanaries`, no `WithCapabilities` — |
| + apply verbatim to its successor and must be carried over in comments. |
| + The capability-converge consequence recorded there (new binary runs |
| + against the previous unit shape until the hourly root pass) is unchanged. |
| + |
| +Restart semantics are already correct and untouched: the adopt pass |
| +try-restarts awake units onto the new symlink; hibernating instances get the |
| +pure repoint and pick the new binary up on wake. Once adoption is fast, |
| +"new version on next refresh" follows for both warm and cold instances. |
| + |
| +## 5. Materialize: cold wake stages on demand |
| + |
| +`ExecBackend.Materialize` (`internal/activator/backend_exec.go`, nil in |
| +production today) gets wired: on wake of a host whose live symlink is |
| +missing, resolve the route's channel pointer (full signature verification), |
| +check the bake hold (a held head is NOT materialized — cold wake must not |
| +become a bake bypass; the 503 is then correct and the message says "held"), |
| +stage the artifact, point the symlink, and let the wake proceed. First |
| +visitor pays a slow request instead of receiving a 503 for up to an hour. |
| + |
| +Implementation is a refactor-to-share of `adoptHost`'s staging half, not a |
| +second staging path. Per-host singleflight (the activator's existing wake |
| +serialization) bounds concurrent materializations; this design does not |
| +depend on fixing the #76 beginRequest race, and doesn't worsen it. |
| + |
| +This retires "no live binary" as a wake-failure class (issue #77's preferred |
| +fix, and the instance-provisioning design's own recorded preference). |
| + |
| +## 6. The verb: `carlos deploy` |
| + |
| +Member CLI, console-API surface — no AWS access, per the operate-via-API |
| +rule. One command: |
| + |
| +``` |
| +carlos deploy [-target linux-arm64] [-channel <ch>] [-host <host>] [-timeout 120s] |
| +``` |
| + |
| +1. **Resolve** app from `.carlos/config` / `-app` like every other verb. |
| +2. **Resolve channel**: the channel the app's instances follow (instances |
| + API). Exactly one distinct channel → use it. Several → error listing |
| + them, demanding `-channel` or `-host`. None (no instances) → error with |
| + the `instances create` hint; deploy does not create instances (YAGNI). |
| +3. **Ship** — the existing ship path, flags passed through. |
| +4. **Promote** — the existing promote path, to the resolved channel. The |
| + console bumps `control/serial` as part of the promote write. |
| +5. **Watch** — poll the instance URL(s) until `X-Carlos-Version` equals the |
| + shipped version, then print the URL(s) and exit 0. Default timeout 120 s. |
| + On timeout: exit 1 with diagnostics — the version the header currently |
| + reports, or "no header" (edge predates this feature → say so and suggest |
| + the hourly window), or connection state (route not minted yet / wake |
| + failing). |
| +6. **Bake-held channel** (production app promoting to beta/stable inside a |
| + window): ship and promote complete, then deploy prints the hold ("held |
| + until ~T, use the console fast-track if this is an emergency") and exits |
| + 0 **without watching** — the requested writes succeeded and the hold is |
| + policy, not failure. Deploy never bypasses or waives bake; fast-track |
| + stays a console/passkey action (bake-controls spec §5). |
| + |
| +Step-by-step verbs remain first-class; deploy is composition, not |
| +replacement. |
| + |
| +## 7. The version tell: `X-Carlos-Version` |
| + |
| +The edge sets `X-Carlos-Version: <version>` on every proxied response for |
| +versioned routes, read from the route's live symlink target — the edge |
| +already knows it; this is a `Server:`-header-grade disclosure, approved as |
| +public. It is the deploy watch's proof and doubles as the human |
| +"verify with the binary you built" tell (`curl -sI | grep -i carlos`). |
| + |
| +Consumer list for the header (everything that PARSES it): the deploy watch, |
| +and tests. Absent header = old edge or unversioned route; the watch treats |
| +it as "cannot confirm", never as a match. |
| + |
| +Additionally, the instances API response gains an additive |
| +`staged_version` field where the console can know it, chipping at issue #78 |
| +(no member-facing wake visibility). Freshness is best-effort (the console's |
| +view may lag box-local truth); the header is the authoritative watch and |
| +the API field is explicitly NOT used by deploy's exit criterion. |
| + |
| +## 8. What this deliberately does not change |
| + |
| +- **No push channel.** Boxes stay unreachable by design; everything here is |
| + the box pulling one extra tiny object. |
| +- **No bake/ladder/passkey changes.** Holds, sign-off gates, the stable |
| + step-up, fast-track, and the production flag's 72 h ratchet are all |
| + untouched; the fast set explicitly excludes held routes. |
| +- **No new dev-mode flag.** The unconverged-work predicate makes a cadence |
| + knob unnecessary; the existing production flag keeps meaning exactly |
| + "bake applies". |
| +- **No operator-script integration.** pinfra ship/promote scripts keep the |
| + hourly cadence until separately taught to bump the serial. |
| +- **Existing wire shapes.** Channel pointers, manifests, flag objects, |
| + bake-clock files: unchanged. The two new shapes (serial object, response |
| + header) are additive and ignored by everything that predates them. |
| + |
| +## 9. Testing |
| + |
| +- Serial: parse/shape test; change-not-increment reaction; clamp behavior. |
| +- Fast set: membership predicate unit tests (unstaged, unreconciled, held → |
| + excluded, drained → quiet); backoff schedule; `canaryAdoptLoop` parity |
| + test showing canary routes converge at least as fast as before. |
| +- Materialize: dir-store staging on wake; held head refuses with the held |
| + message; missing artifact fails the wake with the existing error class. |
| +- Header: set for versioned routes, absent for unversioned; value tracks a |
| + symlink swap. |
| +- Deploy verb: e2e against a test console + dir store — every test that |
| + shells the carlos binary builds env via `carlosTestEnv` and sets |
| + `cmd.Dir` to a scratch dir (repo rule; the 40-account incident). |
| +- `laddercheck` untouched and green: production channel names still legal. |
| +- Backward-parse: pre-change pointers and manifests still adopt (repo rule: |
| + wire-shape changes are additive). |
| + |
| +## 10. Rollout |
| + |
| +Deploy verb is useful immediately (canary/* routes are already fast); full |
| +seconds-to-live needs the box halves live. Order per the standing rule |
| +(promote → adopt → restart): ship the new binary, adopt on both boxes, then |
| +restart `carlos-edge.service` (long-running unit — serial loop and header |
| +don't exist until restart). Flagship and tito both; verify with the header |
| +itself from the edge (`--resolve` rule), not a cached DNS answer. |
| + |
| +## Relationship to open work |
| + |
| +- #101 — implements its two suggested shapes plus the serial signal for the |
| + notice-a-promote gap; comment left linking this spec. |
| +- #77 — closed by §5 (Materialize, its preferred fix). |
| +- #78 — advanced by §7's `staged_version` field (not fully closed). |
| +- #95 — env-set first-sighting absorption: the serial bump on config writes |
| + plus the pass-set run covers the reported symptom; verify during |
| + implementation and close if so. |
| +- #76 — activator wake singleflight race: untouched, explicitly not |
| + worsened (§5). |