| 1 | # 🤖 Server releases: channels, publishing, and the signed feed |
| 2 | |
| 3 | How server builds get released, how the three channels work, and the trust |
| 4 | model behind instance auto-updating. The operator entry point is |
| 5 | `./deploy/ops publish|promote|ship`; the machinery those verbs drive is |
| 6 | `deploy/release-server.sh`, and the format/signing code is `release.go` |
| 7 | (shared by the publisher and the box updater so they can't drift). The box-side updater |
| 8 | that consumes this feed is described in `docs/instance-router.md` |
| 9 | (§ Auto-updating). |
| 10 | |
| 11 | ## The shape |
| 12 | |
| 13 | The source repo is private, so a box can't read its GitHub releases without a |
| 14 | credential we don't want to hand out. Everything therefore publishes to public |
| 15 | repos, split by audience (2026-07-22 — before the split, both lived on the |
| 16 | releases repo, and the per-build wall made its page unreadable): |
| 17 | |
| 18 | - the **releases repo** (`elevenmessenger/releases`) is the HUMAN surface: the |
| 19 | Mac app's releases + Sparkle appcast (see `ios/release-mac.sh`), the rolling |
| 20 | `server` feed release, enterprise. GitHub's "Latest" badge belongs to the |
| 21 | newest Mac release — Sparkle's feed URL rides `latest/download/appcast.xml`, |
| 22 | and `release-server.sh` warns loudly when that invariant breaks. |
| 23 | - the **builds repo** (`elevenmessenger/builds`) takes the per-commit |
| 24 | `server-<ver>` build artifacts — machine consumers only. Builds published |
| 25 | before the split stay on the releases repo; promote resolves per version, |
| 26 | so old pins and rollbacks keep working. |
| 27 | |
| 28 | - **One release per build**: tag `server-<ver>` (ver = the git short sha that |
| 29 | `deploy/build-linux.sh` stamps into the binary as `main.buildVersion`), |
| 30 | carrying **both linux architectures** — `lchat-linux-amd64` and |
| 31 | `lchat-linux-arm64` — plus `lchat-darwin-{arm64,amd64}` (the website's |
| 32 | `lchat try` download for Macs; never referenced by the manifest) and the |
| 33 | **sibling-service binaries** `lchat-{connect,push,home}-linux-{amd64,arm64}`, |
| 34 | each with its `.sha256`. |
| 35 | |
| 36 | The service binaries are how connect/push/home deploy at all now that the |
| 37 | box has no root SSH: their sha256s ride *inside* the signed manifest entry |
| 38 | (`services`), and the box's root update pass (`updaterservices.go`) |
| 39 | sha-compares each loose `/opt/lchat/<name>` against the channel head and |
| 40 | swaps + bounces any that drifted — same signature, same bake window, same |
| 41 | rollback-and-quarantine shape as the control binary's own self-update. A box |
| 42 | without a service's file is skipped, so nothing installs uninvited. (A |
| 43 | promote whose entry had to be rebuilt from release assets carries no |
| 44 | `services` block, and the converge simply skips until the next publish.) |
| 45 | - **One rolling release**: tag `server`, whose assets live at fixed URLs |
| 46 | (re-uploaded with `--clobber`, so the URLs never change): |
| 47 | - `manifest.json` — the **feed**, updated on every publish/promote: |
| 48 | |
| 49 | ``` |
| 50 | https://github.com/elevenmessenger/releases/releases/download/server/manifest.json |
| 51 | ``` |
| 52 | - `lchat-linux-{amd64,arm64}` and `lchat-darwin-{arm64,amd64}` |
| 53 | (+ `.sha256`) — copies of the current **stable** binaries, refreshed by |
| 54 | `promote … stable`. These JSON-free URLs are what `deploy/install.sh` |
| 55 | (linux) and the website's try snippet (any OS) download. |
| 56 | - `install.sh` — the self-hoster installer, refreshed on every publish. |
| 57 | - **The releases repo README** is the public self-hosting guide — synced from |
| 58 | `docs/self-hosting.md` on every publish (this source repo is private, so the |
| 59 | downloads page doubles as the documentation). |
| 60 | |
| 61 | > Operator gotcha (learned live): GitHub's asset CDN can serve a |
| 62 | > freshly-clobbered asset **stale for a few minutes**, and brand-new assets can |
| 63 | > briefly 404 — different clients may even hit different cache nodes. An update |
| 64 | > pass right after a publish may quietly no-op; the next hourly pass absorbs |
| 65 | > it. Don't chase ghosts during that window. |
| 66 | |
| 67 | ## Channels |
| 68 | |
| 69 | | channel | meaning | moves when | |
| 70 | |---------|---------|------------| |
| 71 | | `edge` | bleeding edge — tracks `main` | every `publish` | |
| 72 | | `beta` | near-stable, for people who want to test | `promote <ver> beta` | |
| 73 | | `stable` | well tested, should work in all cases — the default channel | `promote <ver> stable` | |
| 74 | |
| 75 | A channel is just a pointer in the manifest. Promotion re-signs the manifest; |
| 76 | no rebuild, no new artifacts. A channel may lag (or be unset early on) — that's |
| 77 | normal. Pointing a channel **back** at an older version is allowed and is how a |
| 78 | bad release gets pulled; updaters follow the channel wherever it points. |
| 79 | |
| 80 | ## The manifest |
| 81 | |
| 82 | A signed wrapper — the Ed25519 signature covers the exact base64 payload bytes |
| 83 | (verify first, decode after), so there is no JSON canonicalisation to get wrong: |
| 84 | |
| 85 | ```json |
| 86 | { "payload": "<base64 of the payload JSON>", "sig": "<base64 Ed25519 signature>" } |
| 87 | ``` |
| 88 | |
| 89 | Payload: |
| 90 | |
| 91 | ```json |
| 92 | { |
| 93 | "schema": 1, |
| 94 | "generated_at": "2026-07-08T12:00:00Z", |
| 95 | "channels": { |
| 96 | "edge": { "version": "a1b2c3d", |
| 97 | "url": "https://github.com/elevenmessenger/releases/releases/download/server-a1b2c3d/lchat-linux-amd64", |
| 98 | "sha256": "…64 hex…", "size": 123456, |
| 99 | "arm64_url": "…/server-a1b2c3d/lchat-linux-arm64", |
| 100 | "arm64_sha256": "…64 hex…", "arm64_size": 123456, |
| 101 | "notes": "one-liner", "published_at": "…" }, |
| 102 | "beta": { "…": "…" }, |
| 103 | "stable": { "…": "…" } |
| 104 | } |
| 105 | } |
| 106 | ``` |
| 107 | |
| 108 | The flat `url`/`sha256` fields are the **amd64** binary — kept flat so every |
| 109 | updater ever shipped can read them; `arm64_*` are additive and optional (an |
| 110 | arm64 box facing a release without them skips it rather than installing the |
| 111 | wrong architecture — `releaseEntry.forArch` in `release.go`). |
| 112 | |
| 113 | ## Trust model |
| 114 | |
| 115 | The updater downloads and *executes* native code, so the chain matters: |
| 116 | |
| 117 | - The **public key** is baked into the binary (`releasePubKeyB64` in |
| 118 | `release.go`). The **private key** (an Ed25519 seed) exists only in the |
| 119 | password manager — Bitwarden (legacy copy in 1Password), item *"Eleven |
| 120 | Messenger server release signing key"* — and in the signing process's |
| 121 | memory while signing (`LCHAT_RELEASE_KEY`, the seed value in the |
| 122 | environment; it is never written to disk). The older file mode |
| 123 | (`LCHAT_RELEASE_KEY_FILE`) remains as a fallback. |
| 124 | - The signed payload carries each binary's **sha256**, so the binaries |
| 125 | themselves need no separate signature: a downloaded binary is hashed and |
| 126 | checked against the signed manifest before it is ever staged. Compromising |
| 127 | GitHub (or the download path) without the key yields nothing installable. |
| 128 | - **`generated_at` is stamped at signing time** and consumers persist the last |
| 129 | value they accepted, refusing anything older — so replaying a stale (but |
| 130 | validly signed) manifest can't downgrade a fleet. An intentional pull-back is |
| 131 | a *re-signed* manifest with a newer timestamp, which still applies. |
| 132 | |
| 133 | ### The bake window (supply-chain hold) |
| 134 | |
| 135 | Everything above assumes the signing key is honest. The bake window is the |
| 136 | defence for the day it isn't — a stolen seed, or a compromised operator |
| 137 | machine signing a malicious build. beta and stable subscribers don't install |
| 138 | a release the moment it appears; the box waits until the release has **sat at |
| 139 | the channel head for a while**: |
| 140 | |
| 141 | | channel | hold | |
| 142 | |---------|------| |
| 143 | | `edge` | none — it *is* the operator's own deploy path | |
| 144 | | `beta` | 24 h | |
| 145 | | `stable` | 72 h | |
| 146 | |
| 147 | Real supply-chain incidents are bimodal: noisy ones (malicious npm/PyPI |
| 148 | releases, stolen-token publishes) are overwhelmingly detected and yanked |
| 149 | within hours to ~3 days; stealthy targeted ones take weeks and no tolerable |
| 150 | hold defends against them (that's what key hygiene is for). 24 h absorbs the |
| 151 | caught-within-a-day mass; 72 h additionally **spans a weekend** — a |
| 152 | Friday-night malicious publish is seen by a human on Monday with margin to |
| 153 | pull the channel back, and a pull-back to a previously-seen version applies |
| 154 | *immediately* (evacuation never bakes). This is also why stable-channel lag |
| 155 | saved every distro from xz-utils. |
| 156 | |
| 157 | The mechanics (`bake.go`): |
| 158 | |
| 159 | - **The clock is the box's own first sighting** — `lchat update run` records |
| 160 | "first saw `<ver>` at the head of `<chan>`" in |
| 161 | `/opt/lchat/channel-first-seen.json` and installs only once that record is |
| 162 | old enough. Never a manifest timestamp: an attacker holding the key |
| 163 | controls every timestamp in the feed and would simply backdate one, but |
| 164 | cannot backdate what the box already saw. A missing/corrupt clock file |
| 165 | fails **closed** (everything bakes from now). |
| 166 | - **Jitter**: each box waits `hold × (1 + j)`, `j ∈ [0, 0.25)` minted once |
| 167 | per box — a fleet of self-hosted boxes doesn't install in the same hour, |
| 168 | so the earliest are canaries for the rest. |
| 169 | - **There is deliberately NO publisher-side bypass** ("critical fix, skip the |
| 170 | hold") — that flag is the first thing an attacker would set. The bypass is |
| 171 | local and explicit, requiring root on the box: |
| 172 | `lchat update run --ignore-bake`, or a plain |
| 173 | `lchat deploy <host> --version <ver>`. A held release shows in |
| 174 | `lchat update status` as `(held until <time>)` and in the admin consoles as |
| 175 | "auto-installs after a N-hour safety hold". |
| 176 | - **Detection inside the window** is what makes the hold worth anything: each |
| 177 | instance's own 6-hourly feed check pushes the space's admins when its |
| 178 | channel head *changes* (`releasealert.go`) — the alert fires on observation |
| 179 | of the feed, out-of-band from the signing key, so a rogue publish can't be |
| 180 | quiet. edge sightings stay silent (its riders are the operators doing the |
| 181 | publishing; the post-install changelog banner covers arrival). |
| 182 | Belt-and-braces for operators: also GitHub-watch the releases repo |
| 183 | (Watch → Custom → Releases), so a publish you didn't make lands in your |
| 184 | inbox through a channel the release key never touches. |
| 185 | |
| 186 | ### Critical security releases — the emergency runbook |
| 187 | |
| 188 | A critical fix wants to reach the fleet in minutes; the bake window makes |
| 189 | beta/stable wait days. Both are security properties, and the resolution is |
| 190 | **not** a faster feed: the clock stays untouched, and expediting is done from |
| 191 | the box side. The rule that keeps every current and future override honest: |
| 192 | |
| 193 | > **An override is safe exactly when it cannot be exercised with the secret |
| 194 | > the delay defends against.** The bake window defends against a compromised |
| 195 | > *release signing key*; every expedite path below rides *box root* or an |
| 196 | > *instance admin's deliberate act* — independent credentials. A same-key |
| 197 | > override (a `critical: true` field in the manifest) is the one thing never |
| 198 | > to build: it is the first flag an attacker with the key would set, and it |
| 199 | > converts the whole defence into decoration. |
| 200 | |
| 201 | The procedure, for a fix that can't wait out the hold: |
| 202 | |
| 203 | 1. **Land the fix on `main`** (normal PR flow — an emergency is when review |
| 204 | discipline pays, not when it lapses). |
| 205 | 2. **Publish and converge the edge fleet** — edge has no hold, so this is |
| 206 | already the whole story for every edge-riding space: |
| 207 | |
| 208 | ```sh |
| 209 | ./deploy/release-server.sh publish "security: <one line>" && \ |
| 210 | ./deploy/release-server.sh ship |
| 211 | ``` |
| 212 | |
| 213 | 3. **Promote to beta and stable.** The channel heads move immediately; only |
| 214 | *installation* is held. This also fires the release-alert push to every |
| 215 | subscribed space's admins — for self-hosted boxes, that push (and the |
| 216 | admin console's "available" line) *is* the expedite channel: their |
| 217 | operator applies it with their own root, on their own judgement. |
| 218 | 4. **Expedite boxes you operate** (requires root on each box — that's the |
| 219 | point): |
| 220 | |
| 221 | ```sh |
| 222 | lchat update run --ignore-bake # whole box: install held heads now |
| 223 | lchat deploy <host> --version <ver> # or surgically, one instance |
| 224 | ``` |
| 225 | |
| 226 | `--ignore-bake` skips only the bake clock; pins, major pins, and |
| 227 | updates-off are still honoured. |
| 228 | 5. **Say what happened.** The release notes and the changelog entry should |
| 229 | name the fix plainly (members see the changelog banner after their space |
| 230 | updates). If the vuln is sensitive, "security fix — update promptly" is |
| 231 | enough; silent security releases train nobody to act on the next alert. |
| 232 | |
| 233 | What this costs, honestly: a self-hosted box whose operator ignores the push |
| 234 | installs the fix only when the hold elapses (up to ~3.75 days on stable with |
| 235 | maximum jitter). That is the deliberate trade — the window pays for itself |
| 236 | the day a *malicious* release ships, and a malicious binary on an E2EE |
| 237 | messenger (key exfiltration) is unrecoverable in a way a bounded patch lag |
| 238 | is not. Expect social engineering in both directions: an attacker will label |
| 239 | malware "critical security fix" precisely to make humans expedite it, which |
| 240 | is why the expedite paths end at a human with an independent credential |
| 241 | rather than at a feed flag. |
| 242 | |
| 243 | Two designed-but-unbuilt rungs, for when self-hoster numbers make |
| 244 | "wait for each operator" unacceptable (build in this order): |
| 245 | |
| 246 | - **Consent-based expedite**: make the release alert actionable — an |
| 247 | "Install now" button in the admin console writes a one-shot, per-version |
| 248 | consent the box updater honours. The override power distributes to each |
| 249 | space's human; an attacker with the signing key would have to socially |
| 250 | engineer every admin individually. No new keys, no publisher power. |
| 251 | - **Second-key co-signature**: a separate *emergency key*, stored apart from |
| 252 | the release key (offline / second operator — the value is that no single |
| 253 | machine compromise yields both), whose co-signature shortens the hold. |
| 254 | Three rules keep it from becoming the attack: the co-sig **binds the exact |
| 255 | sha and expires within hours** (no hoarding, no replay onto a later build); |
| 256 | the reduced hold is a **floor, never zero** (2–6h + jitter — the earliest |
| 257 | boxes stay canaries even in an emergency); and expedited releases alert |
| 258 | **louder, not quieter**, so abuse of the mechanism is the most visible |
| 259 | event in the system. |
| 260 | |
| 261 | ### Major version pinning |
| 262 | |
| 263 | Releases carry two names: the **sha** (`version`, the artifact id everything |
| 264 | deploys by) and a **human semver** (`semver`, stamped at publish time from |
| 265 | `deploy/SERVER_SEMVER` — bump that file as part of cutting a release; plain |
| 266 | `MAJOR.MINOR.PATCH`, no suffixes). The semver exists for one consumer: an |
| 267 | instance that wants minors and patches but not to be auto-updated across a |
| 268 | breaking major. |
| 269 | |
| 270 | - **Setting the pin**: `update_major` is an instance setting like the version |
| 271 | pin — `lchat update major <host> <n>` / `lchat update major <host> clear` |
| 272 | on the box, or `update_major` through the admin settings API. |
| 273 | - **Semantics are halt-and-notify, not LTS.** A channel head whose major is |
| 274 | ≤ the pin installs as normal; one that crosses it is *held* — the pass says |
| 275 | so, `update status` shows `(held by major pin N)`, and the admin console |
| 276 | explains it on the available line. Nothing auto-installs until the admin |
| 277 | raises or clears the pin. There are no per-major channel heads: we don't |
| 278 | maintain old-major branches, so a pinned instance's updates simply stop at |
| 279 | the boundary (that's the honest promise; per-major heads can be added if an |
| 280 | LTS branch ever actually exists). |
| 281 | - **Fail closed**: a release with no semver at all also holds a major-pinned |
| 282 | instance — the pin is a promise not to cross a boundary, so an unlabelled |
| 283 | release must never sneak past it. (`release-server.sh` warns when it |
| 284 | publishes without `deploy/SERVER_SEMVER`, and when a `promote` has to |
| 285 | rebuild an entry from release assets, which loses the semver.) |
| 286 | - **Pull-backs always pass**: an older major is never held. |
| 287 | |
| 288 | ## The S3 mirror |
| 289 | |
| 290 | GitHub's asset CDN is the feed's single point of failure — and it has served |
| 291 | freshly-clobbered manifests stale (and brand-new binaries 404) for minutes |
| 292 | after a publish (the gotcha above). The feed therefore also mirrors to S3 |
| 293 | (bucket: `infra/prod/releases.tf` — public-read GETs so boxes fetch |
| 294 | anonymously, IAM-only writes, versioned for manifest rollback): |
| 295 | |
| 296 | - **Publish side:** set `LCHAT_RELEASE_MIRROR_BUCKET` and |
| 297 | `deploy/release-server.sh` copies everything it uploads to GitHub into |
| 298 | `s3://<bucket>/releases/…` too — `manifest.json` at the top (refreshed on |
| 299 | publish *and* promote), binaries under `server-<ver>/`. Unset, the script |
| 300 | is unchanged. Mirror writes are best-effort: a failure warns rather than |
| 301 | aborting a publish whose GitHub half already landed — a lagging mirror is |
| 302 | harmless because updaters use whichever verified feed is fresher. |
| 303 | - **Fetch side:** set `LCHAT_RELEASE_MIRROR_URL` (a plain HTTPS base, e.g. |
| 304 | `https://<bucket>.s3.<region>.amazonaws.com/releases`) and the updater |
| 305 | falls back to the mirror when the primary fetch fails — or when it verifies |
| 306 | but is *stale*: generated_at older than the newest feed the box ever |
| 307 | accepted (the anti-replay stamp), which is exactly what the post-publish |
| 308 | CDN window looks like; a fresher mirror then wins. Binary downloads fall |
| 309 | back the same way. Every fallback logs loudly; a feed merely *equal* to the |
| 310 | stamp is indistinguishable from "no new release", so steady-state passes |
| 311 | never touch the mirror. |
| 312 | |
| 313 | The mirror adds availability, never trust: both paths run the identical |
| 314 | Ed25519 manifest verification and sha256 binary check, so a compromised |
| 315 | mirror can only cause a refusal, not an install. |
| 316 | |
| 317 | ## Publishing |
| 318 | |
| 319 | ```sh |
| 320 | # cut a release from clean, landed main and point edge at it — the signing |
| 321 | # key comes from Bitwarden automatically (deploy/vault-items.sh), unlocking |
| 322 | # interactively if the vault is locked; no key export, nothing on disk |
| 323 | ./deploy/ops publish "what changed, one line" |
| 324 | |
| 325 | # make every fleet box catch up NOW instead of on the hourly timer |
| 326 | ./deploy/ops ship |
| 327 | |
| 328 | # promote when it's proven |
| 329 | ./deploy/ops promote a1b2c3d beta |
| 330 | ./deploy/ops promote a1b2c3d stable |
| 331 | ``` |
| 332 | |
| 333 | `ship` loops the box's `lchat update run` until every un-pinned instance is |
| 334 | current on its channel — absorbing GitHub's stale-CDN window on a fresh |
| 335 | manifest — and each pass restarts updated instances' bot sidecars in |
| 336 | lock-step. Pinned instances (canaries) are reported and left alone. With the |
| 337 | whole fleet riding `edge`, `publish … && ship` is a complete manual deploy. |
| 338 | |
| 339 | **Hibernating hosts** (docs/planet-scale.md) have no unit to restart and are |
| 340 | never woken just to update. The pass instead reads their prefs from the |
| 341 | parked DB (read-only, fail-closed — a host whose prefs can't be read is |
| 342 | skipped, never moved) and **repoints the live symlink** — the next wake |
| 343 | execs the new binary. They're processed *after* every unit-managed host, so |
| 344 | a boot-broken build that fails a health check on an awake instance poisons |
| 345 | the version before any sleeper is moved. Two stated trades vs the unit |
| 346 | path: there is no per-host health check (a bad build surfaces at wake — |
| 347 | recover with `lchat rollback <host>`, which repoints back without waking; |
| 348 | `lchat deploy <host> --version <ver>` likewise repoints), and a |
| 349 | currently-awake child keeps its old binary until it next sleeps. A |
| 350 | sidecar-bearing space that never sleeps therefore still needs the flip-off |
| 351 | flow (`lchat hibernate <host> --off`, update, `--on`). `lchat update status` |
| 352 | reads a sleeping host's prefs the same parked-DB way, so its CHANNEL/AUTO |
| 353 | columns (including a pin) show real values instead of `?` — an invisible pin |
| 354 | is how a fleet operation stomps a canary. |
| 355 | |
| 356 | Same rule as deploys and TestFlight: **publish from landed `main`**, never a |
| 357 | feature branch (the script refuses a dirty tree). Publishing is manual for now; |
| 358 | a CI job cutting `edge` on every push to `main` is the natural next step. |
| 359 | |
| 360 | `lchat release` subcommands (`keygen` / `sign` / `verify` / `payload`) are the |
| 361 | plumbing the script drives — `verify` checks a manifest exactly the way a box |
| 362 | will, and `keygen` prints a new public key for baking in (only needed if the |
| 363 | key is ever rotated; rotation requires shipping the new public key via the |
| 364 | *old* key's channel first). |
| 365 | |
amadan