amadan
spec: pipelines — channels are maturity, environments are glue
Two axes, independent. Channels are platform-defined and ordered (canary, edge, beta, stable) because promotion needs a shared vocabulary and a direction. Environments are app-defined and unordered because the platform cannot know what "Stripe sandbox" means, and the moment it tries, the next app breaks the enum. The evidence is in the fleet. All eighteen Tito routes follow one channel, go.tito.io included, so what distinguishes contacts-canary from production is data and config — the environment axis, expressed by hand eighteen times. Meanwhile both production apps are parked on migration canary channels with no stable pointer at all, because nothing knows what a channel is for. Ordered promotion runs from edge upward; canary sits outside the chain, since a canary is a branch's build and what reaches edge is the artifact built from main after the merge. Hotfixes bypass explicitly and are recorded as hotfixes, because an invisible bypass is indistinguishable from a broken gate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 file changed,
+257
−0
docs/superpowers/specs/2026-08-02-pipelines-design.md+257 −0
diff --git a/docs/superpowers/specs/2026-08-02-pipelines-design.md b/docs/superpowers/specs/2026-08-02-pipelines-design.md| new file mode 100644 |
| index 000000000..53ae4acab |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-02-pipelines-design.md |
| @@ -0,0 +1,257 @@ |
| +# Pipelines — channels are maturity, environments are glue |
| + |
| +**Status:** approved 2026-08-02. |
| +**Supersedes nothing.** Extends the multi-account design (accounts own apps) and |
| +the domains design (routes own hostnames) with the axis both were missing: how a |
| +build moves from someone's worktree to production, and what it is wired to when |
| +it gets there. |
| + |
| +## 1. Why |
| + |
| +Two things are true of every deployment on this platform today, and both are |
| +accidents rather than decisions. |
| + |
| +**Production is parked on migration channels.** `go.tito.io` follows |
| +`canary/platform-adoption`. `keymail.dev` follows `canary/keymail`. Neither app |
| +has a `stable` pointer at all. The channel each production host follows is named |
| +after the migration that put it there, and nothing says that is wrong, because |
| +nothing knows what a channel is *for*. |
| + |
| +**The environment axis is expressed eighteen times by hand.** Every one of |
| +Tito's eighteen routes follows that same single channel — `contacts-canary`, |
| +`modal-canary`, `timeslots-canary` included. They run byte-identical builds. |
| +What distinguishes them is their data and their configuration, so their |
| +"canary-ness" is a hostname convention, not a platform mechanism. The |
| +credentials that make each one what it is live in eighteen separate per-host env |
| +files, and a sandbox key reaching a production instance is a copy-paste away. |
| + |
| +Underneath both: **the deploy path lives in people's heads.** On 2026-08-02 that |
| +produced three failures in one afternoon — an app deployed through a tool that |
| +no longer deployed it, a release adopted in the wrong order (restart before |
| +adoption, so the old binary re-exec'd), and a verification run against a stale |
| +DNS answer that reported success from the wrong host. A pipeline is not |
| +primarily an automation feature. It is where the deploy path stops being |
| +folklore. |
| + |
| +## 2. The two axes |
| + |
| +A route binds exactly one of each, and they are independent. |
| + |
| +| | **channel** | **environment** | |
| +|---|---|---| |
| +| answers | how mature is this build | what glue is it wired to | |
| +| defined by | the platform | the app | |
| +| vocabulary | fixed: `canary` → `edge` → `beta` → `stable` | arbitrary, the app's own names | |
| +| ordered | yes — promotion needs a ladder | no — they are siblings | |
| +| selects | which release the route runs | which config bundle the route gets | |
| + |
| +The asymmetry is the design. Channels are ordered and platform-defined because |
| +promotion is meaningless without an order and a shared vocabulary. Environments |
| +are unordered and app-defined because the platform cannot know what "Stripe |
| +sandbox" means, and the moment it tries, the next app breaks the enum. Tito |
| +already needs `demo`, `connect`, `contacts`, `timeslots`, `products`, |
| +`importer`, `wfs`, `yste-demo`, `vicky`, `tunnel` and `local`; no fixed list was |
| +ever going to survive that. |
| + |
| +Keeping the vocabularies separate is also what stops `dev` — which names an |
| +environment — being used for a rung, which is a build maturity. A rung called |
| +`dev` would imply dev glue, and the whole point is that those are chosen |
| +independently. |
| + |
| +### The rule this replaces |
| + |
| +> The platform needs a dev *deployment*; apps need dev *channels*. |
| + |
| +You cannot develop infrastructure without a second copy of it, so CARLOS itself |
| +gets a separate deployment. But once CARLOS *is* an app's infrastructure, that |
| +app does not need separate infrastructure for its dev work — it needs an earlier |
| +rung on the same platform, and the platform supplies the isolation. This retires |
| +per-app dev boxes as a category, not just keymail's. |
| + |
| +## 3. The ladder |
| + |
| +``` |
| +canary/<name> a specific change. Shipped by `carlos canary` from a worktree |
| + or a branch. May be wired to ANY environment, including prod. |
| + Disposable by default; promotable if it earns it. |
| + ↓ |
| +edge merged to main. Subscribers pick it up as it lands — no human |
| + in the loop, by definition. |
| + ↓ |
| +beta promoted on sign-off by a human with rights on the account. |
| + ↓ |
| +stable promoted on sign-off, after the supply-chain bake window. |
| +``` |
| + |
| +`canary` is a rung, not a side-channel: a canary instance of Eleven added to the |
| +production home app, routing through production everything, is a first-class |
| +participant in production and not a sandbox exercise. Sandboxing it would make |
| +it a different thing that proves nothing. |
| + |
| +**Ordering is enforced on promotion, from `edge` upward.** A version may only |
| +enter `beta` from `edge`, and `stable` from `beta`; `carlos promote` refuses |
| +otherwise. This is the whole value of the ladder: "it was signed off on beta" |
| +means something only if everything on stable came through beta. |
| + |
| +`canary` is deliberately outside that chain. A canary is a *branch's* build, and |
| +the build that reaches `edge` is the one produced from `main` after the merge — |
| +a different artifact with a different version. Nothing is promoted from canary to |
| +edge; merging is what puts a change on edge. Canary is where a change is proven |
| +before it is a candidate at all, which is why it is the one rung with no gate |
| +into it and no gate out. |
| + |
| +**Hotfixes get an explicit bypass, never an implicit one.** `carlos promote |
| +-hotfix` may place a version on any rung, and it is recorded as a hotfix in the |
| +audit trail. A bypass that is invisible is indistinguishable from a broken gate. |
| + |
| +**Promotion moves a pointer, never an artifact.** The same content-addressed |
| +build walks the ladder; only the signed channel pointer changes. This is already |
| +how `promote` works and is why "we tested this on beta" is a meaningful claim — |
| +the bytes on stable are the bytes that were on beta. |
| + |
| +**Promotion never changes environment.** If it did, beta and stable could be |
| +talking to different databases and the ladder would prove nothing. Environment |
| +is a property of the route, not of the release. |
| + |
| +### Gates |
| + |
| +| rung | gate | |
| +|---|---| |
| +| `canary` | none — this is where work happens | |
| +| `edge` | merged to main. Automatic. | |
| +| `beta` | sign-off by an account member, recorded | |
| +| `stable` | sign-off by an account **owner**, recorded, plus the bake window | |
| + |
| +Sign-off on `stable` requires passkey step-up (multi-account spec §7a). Placing |
| +a build in front of every user of an app is precisely the class of action for |
| +which a live session should not be sufficient proof of presence. |
| + |
| +The bake window is a supply-chain hold, not a sign-off — it answers "has this |
| +artifact existed long enough for a compromised dependency to surface", which is |
| +a different question from "does this work". Both are required for `stable`; |
| +neither substitutes for the other. |
| + |
| +## 4. Environments |
| + |
| +An environment is a **named, reusable bundle of glue configuration**, defined by |
| +the app, bound per route. |
| + |
| +It is deliberately *not* a new isolation primitive. Routes already isolate: |
| +every instance has its own data directory, its own unit, its own socket. What is |
| +missing is a way for many routes to share one definition of "the Stripe sandbox |
| +credentials", so that changing it is one edit rather than eighteen, and so that |
| +which routes use it is a question with an answer. |
| + |
| +``` |
| +environments/<account-sqid>/<app>/<name>.json the bundle, in the bucket |
| +``` |
| + |
| +A bundle holds key/value configuration and references to secrets; the platform |
| +stores, scopes and audits it, and has no opinion on what any key means. A route |
| +names one environment; the activator materialises it into the instance's |
| +environment at spawn. |
| + |
| +**Secrets are referenced, never inlined.** A bundle names a secret by reference |
| +(SSM parameter, or the deployment's own secret store); the value is resolved at |
| +spawn time on the box. A bucket object that contains live credentials is a |
| +bucket object that leaks live credentials. |
| + |
| +**Defaults and overrides.** An app may define a `default` bundle that every |
| +route inherits, with a named environment layering on top. Eighteen routes |
| +sharing one Stripe key and differing in three values each should be expressible |
| +as one bundle plus three-line overrides. |
| + |
| +## 5. Guard rails |
| + |
| +**Prod-wired canaries are conspicuous.** A route on a `canary/*` channel bound |
| +to a production environment is an unreviewed build in production. That is |
| +allowed and useful, and it must be *visible* — flagged in the console's app view |
| +and on the fleet status page — rather than discoverable only by reading the |
| +registry. Visibility, not permission: nothing here restricts what a canary may |
| +do. |
| + |
| +**Canaries expire.** Every `canary/*` channel carries a TTL (default 7 days). |
| +Expiry does not kill the route; it flags it as expired in the console and on the |
| +status page, and nags. The hazard is not a prod-wired canary — it is one that |
| +outlives the branch it was testing and quietly becomes permanent infrastructure |
| +nobody remembers deploying. |
| + |
| +**Adoption must actually run.** `carlos-adopt.timer` was `enabled` but never |
| +`active` on the flagship: the unit was symlinked and never armed, so no release |
| +ever adopted automatically and every deploy silently required a manual pass. |
| +`bootstrap` and any unit-refreshing script must arm timers, and a host whose |
| +adoption timer is not running is a fleet-status defect, not a silent one. |
| + |
| +## 6. What `edge` requires that does not exist |
| + |
| +`edge` means merged to main, with subscribers picking it up as it lands. Three |
| +things stand between that and today: |
| + |
| +1. **Nothing builds on merge.** Every artifact deployed so far was built on a |
| + developer's machine. `edge` needs a build triggered by main advancing, which |
| + ships and promotes to `edge` with no human step. |
| +2. **Nothing promotes automatically.** `promote` is a manual verb with a signing |
| + key read from SSM. An automatic promotion needs an identity that can sign for |
| + `edge` and *only* `edge` — a key scoped to the bottom rungs, so that |
| + compromising CI cannot place a build on stable. |
| +3. **Adoption is hourly.** "As soon as it lands" and a 60-minute poll are not |
| + the same claim. Adoption latency has to drop for the rung to mean what it |
| + says. |
| + |
| +## 7. Console surface |
| + |
| +One ladder view per app: four rungs, what version sits on each, when it got |
| +there and who promoted it, with promote actions between adjacent rungs. Routes |
| +are listed under the rung they subscribe to, with their environment beside them, |
| +so "what is running where, wired to what" is one screen rather than an |
| +inference. |
| + |
| +Promotion to `stable` goes through passkey step-up. Every promotion writes an |
| +audit row: version, from rung, to rung, actor, whether it was a hotfix. |
| + |
| +## 8. Migration |
| + |
| +1. **Give both production apps a `stable` pointer.** `go.tito.io` and |
| + `keymail.dev` move off their migration canary channels onto `stable`, with |
| + their current versions. This is a pointer write and a route update; no |
| + artifact moves. |
| +2. **Tito's eighteen routes** keep one channel (`stable`) and gain environments |
| + named for what they already are. |
| +3. **`dev.keymail.dev`** becomes a route on the flagship subscribed to `edge`, |
| + bound to a `dev` environment — retiring the last reason keymail's legacy EC2 |
| + stack exists. |
| +4. **`canary.sh` and the keymail dev box** are retired once `carlos canary` can |
| + stand up a canary with a hostname and a chosen environment. The WebKit gate |
| + then runs against a platform canary. |
| + |
| +## 9. Scope |
| + |
| +**In, first slice:** the ladder as ordered platform vocabulary; ordered |
| +promotion with the hotfix bypass; per-route channel subscription surfaced; |
| +environments as named bundles with defaults and overrides; the console ladder |
| +view; prod-wired-canary flagging; canary TTL; the adopt-timer arming fix; |
| +production onto `stable`. |
| + |
| +**Deferred, named:** build-on-merge automation and the `edge`-scoped signing |
| +identity (§6.1, §6.2) — these need a CI story this repo does not have yet; |
| +sub-hourly adoption (§6.3); retiring `canary.sh` and the keymail dev box (§8.4); |
| +per-environment schema-migration handling. |
| + |
| +A pipeline whose bottom rung is not yet automatic is still worth having: the |
| +ordering, the visibility and the audit trail are the parts that were missing |
| +today, and none of them need CI. |
| + |
| +## 10. Testing |
| + |
| +- Ordered promotion: `edge → beta` and `beta → stable` succeed; every skip is |
| + refused, and the refusal names `-hotfix`. Shipping to `canary` needs no |
| + promotion and is never refused by ordering. |
| +- A hotfix promotion is recorded as one and is visible as one. |
| +- Promotion never mutates a route's environment — asserted directly, because |
| + this is the invariant that makes the ladder meaningful. |
| +- Environment resolution: default plus override produces exactly the expected |
| + key set; a secret reference is never written to the bucket in cleartext. |
| +- A prod-wired canary is flagged in the console and on the status page. |
| +- An expired canary is flagged and does not stop serving. |
| +- Step-up is required for `stable` promotion and refused without it. |
| +- Adoption timer state is reported by the fleet status page. |