🤖 Server releases: channels, publishing, and the signed feed

How server builds get released, how the three channels work, and the trust model behind instance auto-updating. The operator entry point is ./deploy/ops publish|promote|ship; the machinery those verbs drive is deploy/release-server.sh, and the format/signing code is release.go (shared by the publisher and the box updater so they can't drift). The box-side updater that consumes this feed is described in docs/instance-router.md (§ Auto-updating).

The shape

The source repo is private, so a box can't read its GitHub releases without a credential we don't want to hand out. Everything therefore publishes to public repos, split by audience (2026-07-22 — before the split, both lived on the releases repo, and the per-build wall made its page unreadable):

• the releases repo (elevenmessenger/releases) is the HUMAN surface: the Mac app's releases + Sparkle appcast (see ios/release-mac.sh), the rolling server feed release, enterprise. GitHub's "Latest" badge belongs to the newest Mac release — Sparkle's feed URL rides latest/download/appcast.xml, and release-server.sh warns loudly when that invariant breaks.
• the builds repo (elevenmessenger/builds) takes the per-commit server-<ver> build artifacts — machine consumers only. Builds published before the split stay on the releases repo; promote resolves per version, so old pins and rollbacks keep working.

• One release per build: tag server-<ver> (ver = the git short sha that deploy/build-linux.sh stamps into the binary as main.buildVersion), carrying both linux architectures — lchat-linux-amd64 and lchat-linux-arm64 — plus lchat-darwin-{arm64,amd64} (the website's lchat try download for Macs; never referenced by the manifest) and the sibling-service binaries lchat-{connect,push,home}-linux-{amd64,arm64}, each with its .sha256.

The service binaries are how connect/push/home deploy at all now that the box has no root SSH: their sha256s ride inside the signed manifest entry (services), and the box's root update pass (updaterservices.go) sha-compares each loose /opt/lchat/<name> against the channel head and swaps + bounces any that drifted — same signature, same bake window, same rollback-and-quarantine shape as the control binary's own self-update. A box without a service's file is skipped, so nothing installs uninvited. (A promote whose entry had to be rebuilt from release assets carries no services block, and the converge simply skips until the next publish.)

• One rolling release: tag server, whose assets live at fixed URLs (re-uploaded with --clobber, so the URLs never change): - manifest.json — the feed, updated on every publish/promote:

`` https://github.com/elevenmessenger/releases/releases/download/server/manifest.json ` - lchat-linux-{amd64,arm64} and lchat-darwin-{arm64,amd64} (+ .sha256) — copies of the current **stable** binaries, refreshed by promote … stable. These JSON-free URLs are what deploy/install.sh (linux) and the website's try snippet (any OS) download. - install.sh` — the self-hoster installer, refreshed on every publish.

• The releases repo README is the public self-hosting guide — synced from docs/self-hosting.md on every publish (this source repo is private, so the downloads page doubles as the documentation).

> Operator gotcha (learned live): GitHub's asset CDN can serve a > freshly-clobbered asset stale for a few minutes, and brand-new assets can > briefly 404 — different clients may even hit different cache nodes. An update > pass right after a publish may quietly no-op; the next hourly pass absorbs > it. Don't chase ghosts during that window.

Channels

| channel | meaning | moves when | |---------|---------|------------| | edge | bleeding edge — tracks main | every publish | | beta | near-stable, for people who want to test | promote <ver> beta | | stable | well tested, should work in all cases — the default channel | promote <ver> stable |

A channel is just a pointer in the manifest. Promotion re-signs the manifest; no rebuild, no new artifacts. A channel may lag (or be unset early on) — that's normal. Pointing a channel back at an older version is allowed and is how a bad release gets pulled; updaters follow the channel wherever it points.

The manifest

A signed wrapper — the Ed25519 signature covers the exact base64 payload bytes (verify first, decode after), so there is no JSON canonicalisation to get wrong:

{ "payload": "<base64 of the payload JSON>", "sig": "<base64 Ed25519 signature>" }

Payload:

{
  "schema": 1,
  "generated_at": "2026-07-08T12:00:00Z",
  "channels": {
    "edge":   { "version": "a1b2c3d",
                "url": "https://github.com/elevenmessenger/releases/releases/download/server-a1b2c3d/lchat-linux-amd64",
                "sha256": "…64 hex…", "size": 123456,
                "arm64_url": "…/server-a1b2c3d/lchat-linux-arm64",
                "arm64_sha256": "…64 hex…", "arm64_size": 123456,
                "notes": "one-liner", "published_at": "…" },
    "beta":   { "…": "…" },
    "stable": { "…": "…" }
  }
}

The flat url/sha256 fields are the amd64 binary — kept flat so every updater ever shipped can read them; arm64_* are additive and optional (an arm64 box facing a release without them skips it rather than installing the wrong architecture — releaseEntry.forArch in release.go).

Trust model

The updater downloads and executes native code, so the chain matters:

• The public key is baked into the binary (releasePubKeyB64 in release.go). The private key (an Ed25519 seed) exists only in the password manager — Bitwarden (legacy copy in 1Password), item "Eleven Messenger server release signing key" — and in the signing process's memory while signing (LCHAT_RELEASE_KEY, the seed value in the environment; it is never written to disk). The older file mode (LCHAT_RELEASE_KEY_FILE) remains as a fallback.
• The signed payload carries each binary's sha256, so the binaries themselves need no separate signature: a downloaded binary is hashed and checked against the signed manifest before it is ever staged. Compromising GitHub (or the download path) without the key yields nothing installable.
• generated_at is stamped at signing time and consumers persist the last value they accepted, refusing anything older — so replaying a stale (but validly signed) manifest can't downgrade a fleet. An intentional pull-back is a re-signed manifest with a newer timestamp, which still applies.

The bake window (supply-chain hold)

Everything above assumes the signing key is honest. The bake window is the defence for the day it isn't — a stolen seed, or a compromised operator machine signing a malicious build. beta and stable subscribers don't install a release the moment it appears; the box waits until the release has sat at the channel head for a while:

| channel | hold | |---------|------| | edge | none — it is the operator's own deploy path | | beta | 24 h | | stable | 72 h |

Real supply-chain incidents are bimodal: noisy ones (malicious npm/PyPI releases, stolen-token publishes) are overwhelmingly detected and yanked within hours to ~3 days; stealthy targeted ones take weeks and no tolerable hold defends against them (that's what key hygiene is for). 24 h absorbs the caught-within-a-day mass; 72 h additionally spans a weekend — a Friday-night malicious publish is seen by a human on Monday with margin to pull the channel back, and a pull-back to a previously-seen version applies immediately (evacuation never bakes). This is also why stable-channel lag saved every distro from xz-utils.

The mechanics (bake.go):

• The clock is the box's own first sighting — lchat update run records "first saw <ver> at the head of <chan>" in /opt/lchat/channel-first-seen.json and installs only once that record is old enough. Never a manifest timestamp: an attacker holding the key controls every timestamp in the feed and would simply backdate one, but cannot backdate what the box already saw. A missing/corrupt clock file fails closed (everything bakes from now).
• Jitter: each box waits hold × (1 + j), j ∈ [0, 0.25) minted once per box — a fleet of self-hosted boxes doesn't install in the same hour, so the earliest are canaries for the rest.
• There is deliberately NO publisher-side bypass ("critical fix, skip the hold") — that flag is the first thing an attacker would set. The bypass is local and explicit, requiring root on the box: lchat update run --ignore-bake, or a plain lchat deploy <host> --version <ver>. A held release shows in lchat update status as (held until <time>) and in the admin consoles as "auto-installs after a N-hour safety hold".
• Detection inside the window is what makes the hold worth anything: each instance's own 6-hourly feed check pushes the space's admins when its channel head changes (releasealert.go) — the alert fires on observation of the feed, out-of-band from the signing key, so a rogue publish can't be quiet. edge sightings stay silent (its riders are the operators doing the publishing; the post-install changelog banner covers arrival). Belt-and-braces for operators: also GitHub-watch the releases repo (Watch → Custom → Releases), so a publish you didn't make lands in your inbox through a channel the release key never touches.

Critical security releases — the emergency runbook

A critical fix wants to reach the fleet in minutes; the bake window makes beta/stable wait days. Both are security properties, and the resolution is not a faster feed: the clock stays untouched, and expediting is done from the box side. The rule that keeps every current and future override honest:

> An override is safe exactly when it cannot be exercised with the secret > the delay defends against. The bake window defends against a compromised > release signing key; every expedite path below rides box root or an > instance admin's deliberate act — independent credentials. A same-key > override (a critical: true field in the manifest) is the one thing never > to build: it is the first flag an attacker with the key would set, and it > converts the whole defence into decoration.

The procedure, for a fix that can't wait out the hold:

1. Land the fix on main (normal PR flow — an emergency is when review discipline pays, not when it lapses).
2. Publish and converge the edge fleet — edge has no hold, so this is already the whole story for every edge-riding space:

``sh ./deploy/release-server.sh publish "security: <one line>" && \ ./deploy/release-server.sh ship ``

1. Promote to beta and stable. The channel heads move immediately; only installation is held. This also fires the release-alert push to every subscribed space's admins — for self-hosted boxes, that push (and the admin console's "available" line) is the expedite channel: their operator applies it with their own root, on their own judgement.
2. Expedite boxes you operate (requires root on each box — that's the point):

``sh lchat update run --ignore-bake # whole box: install held heads now lchat deploy <host> --version <ver> # or surgically, one instance ``

--ignore-bake skips only the bake clock; pins, major pins, and updates-off are still honoured.

1. Say what happened. The release notes and the changelog entry should name the fix plainly (members see the changelog banner after their space updates). If the vuln is sensitive, "security fix — update promptly" is enough; silent security releases train nobody to act on the next alert.

What this costs, honestly: a self-hosted box whose operator ignores the push installs the fix only when the hold elapses (up to ~3.75 days on stable with maximum jitter). That is the deliberate trade — the window pays for itself the day a malicious release ships, and a malicious binary on an E2EE messenger (key exfiltration) is unrecoverable in a way a bounded patch lag is not. Expect social engineering in both directions: an attacker will label malware "critical security fix" precisely to make humans expedite it, which is why the expedite paths end at a human with an independent credential rather than at a feed flag.

Two designed-but-unbuilt rungs, for when self-hoster numbers make "wait for each operator" unacceptable (build in this order):

• Consent-based expedite: make the release alert actionable — an "Install now" button in the admin console writes a one-shot, per-version consent the box updater honours. The override power distributes to each space's human; an attacker with the signing key would have to socially engineer every admin individually. No new keys, no publisher power.
• Second-key co-signature: a separate emergency key, stored apart from the release key (offline / second operator — the value is that no single machine compromise yields both), whose co-signature shortens the hold. Three rules keep it from becoming the attack: the co-sig binds the exact sha and expires within hours (no hoarding, no replay onto a later build); the reduced hold is a floor, never zero (2–6h + jitter — the earliest boxes stay canaries even in an emergency); and expedited releases alert louder, not quieter, so abuse of the mechanism is the most visible event in the system.

Major version pinning

Releases carry two names: the sha (version, the artifact id everything deploys by) and a human semver (semver, stamped at publish time from deploy/SERVER_SEMVER — bump that file as part of cutting a release; plain MAJOR.MINOR.PATCH, no suffixes). The semver exists for one consumer: an instance that wants minors and patches but not to be auto-updated across a breaking major.

• Setting the pin: update_major is an instance setting like the version pin — lchat update major <host> <n> / lchat update major <host> clear on the box, or update_major through the admin settings API.
• Semantics are halt-and-notify, not LTS. A channel head whose major is ≤ the pin installs as normal; one that crosses it is held — the pass says so, update status shows (held by major pin N), and the admin console explains it on the available line. Nothing auto-installs until the admin raises or clears the pin. There are no per-major channel heads: we don't maintain old-major branches, so a pinned instance's updates simply stop at the boundary (that's the honest promise; per-major heads can be added if an LTS branch ever actually exists).
• Fail closed: a release with no semver at all also holds a major-pinned instance — the pin is a promise not to cross a boundary, so an unlabelled release must never sneak past it. (release-server.sh warns when it publishes without deploy/SERVER_SEMVER, and when a promote has to rebuild an entry from release assets, which loses the semver.)
• Pull-backs always pass: an older major is never held.

The S3 mirror

GitHub's asset CDN is the feed's single point of failure — and it has served freshly-clobbered manifests stale (and brand-new binaries 404) for minutes after a publish (the gotcha above). The feed therefore also mirrors to S3 (bucket: infra/prod/releases.tf — public-read GETs so boxes fetch anonymously, IAM-only writes, versioned for manifest rollback):

• Publish side: set LCHAT_RELEASE_MIRROR_BUCKET and deploy/release-server.sh copies everything it uploads to GitHub into s3://<bucket>/releases/… too — manifest.json at the top (refreshed on publish and promote), binaries under server-<ver>/. Unset, the script is unchanged. Mirror writes are best-effort: a failure warns rather than aborting a publish whose GitHub half already landed — a lagging mirror is harmless because updaters use whichever verified feed is fresher.
• Fetch side: set LCHAT_RELEASE_MIRROR_URL (a plain HTTPS base, e.g. https://<bucket>.s3.<region>.amazonaws.com/releases) and the updater falls back to the mirror when the primary fetch fails — or when it verifies but is stale: generated_at older than the newest feed the box ever accepted (the anti-replay stamp), which is exactly what the post-publish CDN window looks like; a fresher mirror then wins. Binary downloads fall back the same way. Every fallback logs loudly; a feed merely equal to the stamp is indistinguishable from "no new release", so steady-state passes never touch the mirror.

The mirror adds availability, never trust: both paths run the identical Ed25519 manifest verification and sha256 binary check, so a compromised mirror can only cause a refusal, not an install.

Publishing

# cut a release from clean, landed main and point edge at it — the signing
# key comes from Bitwarden automatically (deploy/vault-items.sh), unlocking
# interactively if the vault is locked; no key export, nothing on disk
./deploy/ops publish "what changed, one line"

# make every fleet box catch up NOW instead of on the hourly timer
./deploy/ops ship

# promote when it's proven
./deploy/ops promote a1b2c3d beta
./deploy/ops promote a1b2c3d stable

ship loops the box's lchat update run until every un-pinned instance is current on its channel — absorbing GitHub's stale-CDN window on a fresh manifest — and each pass restarts updated instances' bot sidecars in lock-step. Pinned instances (canaries) are reported and left alone. With the whole fleet riding edge, publish … && ship is a complete manual deploy.

Hibernating hosts (docs/planet-scale.md) have no unit to restart and are never woken just to update. The pass instead reads their prefs from the parked DB (read-only, fail-closed — a host whose prefs can't be read is skipped, never moved) and repoints the live symlink — the next wake execs the new binary. They're processed after every unit-managed host, so a boot-broken build that fails a health check on an awake instance poisons the version before any sleeper is moved. Two stated trades vs the unit path: there is no per-host health check (a bad build surfaces at wake — recover with lchat rollback <host>, which repoints back without waking; lchat deploy <host> --version <ver> likewise repoints), and a currently-awake child keeps its old binary until it next sleeps. A sidecar-bearing space that never sleeps therefore still needs the flip-off flow (lchat hibernate <host> --off, update, --on). lchat update status reads a sleeping host's prefs the same parked-DB way, so its CHANNEL/AUTO columns (including a pin) show real values instead of ? — an invisible pin is how a fleet operation stomps a canary.

Same rule as deploys and TestFlight: publish from landed main, never a feature branch (the script refuses a dirty tree). Publishing is manual for now; a CI job cutting edge on every push to main is the natural next step.

lchat release subcommands (keygen / sign / verify / payload) are the plumbing the script drives — verify checks a manifest exactly the way a box will, and keygen prints a new public key for baking in (only needed if the key is ever rotated; rotation requires shipping the new public key via the old key's channel first).