amadan
docs: emergency runbook for critical security releases under the bake window (#830)
The clock stays; expediting is box-side. Writes down the invariant that keeps every override honest (it must ride a different credential than the signing key the delay defends against), the step-by-step procedure, the honest cost, and the two designed-but-unbuilt rungs (consent-based expedite, second-key co-signature) so the rationale lives in the repo, not in a chat transcript.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+75
−0
docs/server-releases.md+75 −0
diff --git a/docs/server-releases.md b/docs/server-releases.md| index f938f33..90c2caf 100644 |
| --- a/docs/server-releases.md |
| +++ b/docs/server-releases.md |
| @@ -156,6 +156,81 @@ The mechanics (`bake.go`): |
| (Watch → Custom → Releases), so a publish you didn't make lands in your |
| inbox through a channel the release key never touches. |
| +### Critical security releases — the emergency runbook |
| + |
| +A critical fix wants to reach the fleet in minutes; the bake window makes |
| +beta/stable wait days. Both are security properties, and the resolution is |
| +**not** a faster feed: the clock stays untouched, and expediting is done from |
| +the box side. The rule that keeps every current and future override honest: |
| + |
| +> **An override is safe exactly when it cannot be exercised with the secret |
| +> the delay defends against.** The bake window defends against a compromised |
| +> *release signing key*; every expedite path below rides *box root* or an |
| +> *instance admin's deliberate act* — independent credentials. A same-key |
| +> override (a `critical: true` field in the manifest) is the one thing never |
| +> to build: it is the first flag an attacker with the key would set, and it |
| +> converts the whole defence into decoration. |
| + |
| +The procedure, for a fix that can't wait out the hold: |
| + |
| +1. **Land the fix on `main`** (normal PR flow — an emergency is when review |
| + discipline pays, not when it lapses). |
| +2. **Publish and converge the edge fleet** — edge has no hold, so this is |
| + already the whole story for every edge-riding space: |
| + |
| + ```sh |
| + ./deploy/release-server.sh publish "security: <one line>" && \ |
| + ./deploy/release-server.sh ship |
| + ``` |
| + |
| +3. **Promote to beta and stable.** The channel heads move immediately; only |
| + *installation* is held. This also fires the release-alert push to every |
| + subscribed space's admins — for self-hosted boxes, that push (and the |
| + admin console's "available" line) *is* the expedite channel: their |
| + operator applies it with their own root, on their own judgement. |
| +4. **Expedite boxes you operate** (requires root on each box — that's the |
| + point): |
| + |
| + ```sh |
| + lchat update run --ignore-bake # whole box: install held heads now |
| + lchat deploy <host> --version <ver> # or surgically, one instance |
| + ``` |
| + |
| + `--ignore-bake` skips only the bake clock; pins, major pins, and |
| + updates-off are still honoured. |
| +5. **Say what happened.** The release notes and the changelog entry should |
| + name the fix plainly (members see the changelog banner after their space |
| + updates). If the vuln is sensitive, "security fix — update promptly" is |
| + enough; silent security releases train nobody to act on the next alert. |
| + |
| +What this costs, honestly: a self-hosted box whose operator ignores the push |
| +installs the fix only when the hold elapses (up to ~3.75 days on stable with |
| +maximum jitter). That is the deliberate trade — the window pays for itself |
| +the day a *malicious* release ships, and a malicious binary on an E2EE |
| +messenger (key exfiltration) is unrecoverable in a way a bounded patch lag |
| +is not. Expect social engineering in both directions: an attacker will label |
| +malware "critical security fix" precisely to make humans expedite it, which |
| +is why the expedite paths end at a human with an independent credential |
| +rather than at a feed flag. |
| + |
| +Two designed-but-unbuilt rungs, for when self-hoster numbers make |
| +"wait for each operator" unacceptable (build in this order): |
| + |
| +- **Consent-based expedite**: make the release alert actionable — an |
| + "Install now" button in the admin console writes a one-shot, per-version |
| + consent the box updater honours. The override power distributes to each |
| + space's human; an attacker with the signing key would have to socially |
| + engineer every admin individually. No new keys, no publisher power. |
| +- **Second-key co-signature**: a separate *emergency key*, stored apart from |
| + the release key (offline / second operator — the value is that no single |
| + machine compromise yields both), whose co-signature shortens the hold. |
| + Three rules keep it from becoming the attack: the co-sig **binds the exact |
| + sha and expires within hours** (no hoarding, no replay onto a later build); |
| + the reduced hold is a **floor, never zero** (2–6h + jitter — the earliest |
| + boxes stay canaries even in an emergency); and expedited releases alert |
| + **louder, not quieter**, so abuse of the mechanism is the most visible |
| + event in the system. |
| + |
| ### Major version pinning |
| Releases carry two names: the **sha** (`version`, the artifact id everything |