| 1 | # Eleven → Carloku: customer-fleet migration |
| 2 | |
| 3 | **Date:** 2026-08-29 |
| 4 | **Status:** Approved design (Paul). Revised 2026-08-30 twice: first after an |
| 5 | adversarial review (Fable) found three Criticals in the original wave 1, then |
| 6 | again after Paul reframed the work into **two phases — prove it greenfield, |
| 7 | then move production hosts one at a time**. That reframe dissolves most of what |
| 8 | the review found, so this document supersedes the box-conversion plan entirely. |
| 9 | |
| 10 | **Refreshed 2026-09-05** against main, after the spec sat unmerged on branch |
| 11 | `eleven` for a week while the platform moved 269 commits. Three passages had |
| 12 | gone stale and are amended in place, each marked and dated where it sits: |
| 13 | phase 5's tick-cadence argument (void — #371 landed), gap 3 (demoted), and |
| 14 | gap 6 (closed — the file it names was already deleted). Nothing else changed; |
| 15 | the design and its decisions stand as approved. |
| 16 | |
| 17 | **Account:** `eleven` (sqid `bdr`) on `https://console.carloku.com` |
| 18 | **Supersedes:** `infra/eleven-fleet.md`, which was factually wrong about |
| 19 | Eleven's starting state (see "The reframe"). That runbook was deleted from main |
| 20 | in the open-source redaction pass (`2adc71f`, 2026-08-27), so this document is |
| 21 | now the only account of where Eleven starts. |
| 22 | |
| 23 | ## The reframe this spec exists for |
| 24 | |
| 25 | `infra/eleven-fleet.md` says Eleven "moves from Tito's own self-hosted CARLOS |
| 26 | install onto Carloku's console" — a control-plane re-point, "not a data |
| 27 | migration: there is no data to migrate." The first half is wrong. |
| 28 | |
| 29 | **Eleven is not on CARLOS at all.** Checked 2026-08-29: |
| 30 | |
| 31 | - `home.elevenmessenger.com` resolves to `89.167.122.227` and serves 200 with |
| 32 | **no `X-Carlos-Version`** header. It is not behind a CARLOS edge. |
| 33 | - The repo carries a complete operator surface of its own: `deploy/ops` with |
| 34 | build/sign/publish, `edge|beta|stable` channels, fleet sweeps, canaries and |
| 35 | a dev mesh; `lchat router` with its own cert cache and wildcard DNS-01; |
| 36 | `activator.go` with its own hibernation, litestream replication and |
| 37 | single-writer lease. |
| 38 | - CARLOS's equivalents were **ported from that code** — commit `845ffd4`, |
| 39 | "activator, releases/channels, fleet ops (ported from Eleven + titogo)". |
| 40 | |
| 41 | So this is a first-time onboarding onto a control plane that is Eleven's own |
| 42 | machinery, generalised. That is why the overlap is total rather than partial, |
| 43 | and why most of the work is *deleting* Eleven's operator surface rather than |
| 44 | translating it. |
| 45 | |
| 46 | The second half of the old runbook's claim survives, for a different reason |
| 47 | than it gave, and it is now verified rather than inherited — see D4. |
| 48 | |
| 49 | ## Verified starting state (2026-08-29) |
| 50 | |
| 51 | Every line here was checked, not assumed. |
| 52 | |
| 53 | | Fact | How it was checked | |
| 54 | |---|---| |
| 55 | | Account `eleven` (`bdr`) exists on Carloku | `carlos auth whoami` | |
| 56 | | It has **zero apps** | `carlos instances list -account eleven` → "account eleven has no apps" | |
| 57 | | It has **no fleets** | `carlos fleets ls -account eleven` → "no fleets" | |
| 58 | | Production is two boxes: one Hetzner, one Fasthosts, neither EC2 | Paul, 2026-08-29 | |
| 59 | | Boxes serve directly; no CARLOS edge in front | `curl -sI https://home.elevenmessenger.com` | |
| 60 | | Eleven's AWS account is `427880590669` | `sts get-caller-identity` on the vault's dev-agent key | |
| 61 | | Hibernate bucket `eleven-prod-instances-427880590669`, prefix `prod/` | vault item note + `s3api list-objects-v2` | |
| 62 | | Layout is `prod/<host>/` by FQDN, plus `prod/mesh/` | `s3api list-objects-v2 --delimiter /` | |
| 63 | | **31 instances**: 23 `*.11ch.at`, 5 `*.luan.chat`, 2 `*.elevenmessenger.com`, 1 `chat.paulca.com` | same listing | |
| 64 | | All four zones are on **DNSimple** | `dig NS` on each | |
| 65 | | Fleet API surface (#73) merged | PR #145 MERGED 2026-08-11 | |
| 66 | | `region-remote-host`'s `deployment_bucket_arn` is optional | `infra/modules/region-remote-host/variables.tf:11` | |
| 67 | | `carlos apps place` / `apps create --place` exist | `apps.go:111`, `apps.go:382` | |
| 68 | | Sidecar members **not merged** | PR #223 OPEN, last updated 2026-08-18; re-checked 2026-09-05 — still unmerged, no sidecar package on main | |
| 69 | | Eleven already runs a tofu-managed AWS node: EC2 t4g.small arm64, EIP, S3 bucket, IAM role | `infra/dev-mesh/{compute,storage,route53,dns}.tf` | |
| 70 | | `dev.11ch.at` is delegated from DNSimple into a Route53 zone, live at `54.154.95.205` | `aws_route53_zone.dev` + `dnsimple_zone_record.dev_delegation`; `dig NS` | |
| 71 | | `https://amadan.net/eleven/infrastructure` exists and is empty | `git ls-remote` → exit 0, no refs | |
| 72 | | `lchat relocate` is mesh-internal and cannot target a CARLOS box | `relocate.go:1-30` (`home_region`, `fanOutMeshRoute`, `candidatesFor`) | |
| 73 | | Status bucket `eleven-status-427880590669` is 521 objects / ~270 MB, of which `prompts/` is 422 / ~270 MB — the site itself is ~620 KB | `aws s3 ls --recursive --summarize` | |
| 74 | | `carlos ledger` provides append / blob / publish / verify, credential-free verification | `carlos ledger -h` | |
| 75 | |
| 76 | So every precondition `infra/eleven-fleet.md` lists as "pending merge" has |
| 77 | landed, and the two it assumes are already met — apps created, fleet |
| 78 | existing — are both false. |
| 79 | |
| 80 | ## Decisions |
| 81 | |
| 82 | | # | Decision | Rationale | |
| 83 | |---|---|---| |
| 84 | | **D1** | **Fresh boxes, never converted in place.** The two existing production boxes are retired when empty, not migrated. | Converting a live box in place carried the review's worst findings: a shared registry file, a fleet-wide lease drain, and a ports 80/443 handover under production traffic. A fresh box has none of them. | |
| 85 | | **D2** | **Sidecars are proven in phase 1 and hardened in phase 3.** | Sidecars do not spin up today (Paul) and the reason is unknown, so a greenfield box is where the answer is cheapest to find — and no production behaviour can regress while looking. | |
| 86 | | **D3** | Sidecars run on a **third fleet box with `CARLOS_INBOUND=none`**, on ordinary outbound-only networking — no NAT gateway, admin access retained. | Eleven's own `docs/bots.md`: the bot host transiently holds thread plaintext and keys, so "root on the chat server reads every bot thread." Moving it off the serving boxes restores the guarantee. The full private-subnet posture is not needed for that and costs the break-glass channel. | |
| 87 | | **D4** | **Boxes serve directly** in phases 1-2. Carloku is control plane only. | A fleet box with no deployment bucket has its cert issuance gate cleared and "issues and keeps its own certs, as before" (`internal/edge/certsync.go:117`). No Eleven traffic transits Carloku, and no metadata leaves Eleven's boxes. | |
| 88 | | **D5** | **Edge-fronted ingress is phase 4**, gated on two written entry conditions. | Fleet routes cannot be steered today, and the WebSocket hop is unmeasured. See "Phase 4". | |
| 89 | | **D6** | The **sidecar box goes in Eleven's AWS account** (`427880590669`). | It is an ordinary `hashicorp/aws` EC2 instance, it sits beside the buckets that plane already holds, and `region-remote-host` is close to the right module with `deployment_bucket_arn` omitted. | |
| 90 | | **D7** | **Two phases.** Phase 1 proves the whole product greenfield on a throwaway namespace; phase 2 moves the 31 production hosts. | Phase 1 turns the review's unknowns into rehearsals at zero risk: it proves release delivery, teaches the manifest shape at four hosts instead of 31, and answers why sidecars do not spin up today. | |
| 91 | | **D8** | Phase 2 moves hosts **one at a time by DNS**, keeping each address identical. | `docs/multi-edge.md`: the address is the identity, and changing it re-keys every member. A per-host A record overrides the wildcard, so hosts cross individually and reversibly. `lchat relocate` cannot do this — it is mesh-internal (`home_region`, `fanOutMeshRoute`, `candidatesFor`) and a CARLOS box is not a mesh peer. | |
| 92 | | **D9** | All new boxes are **AWS EC2 in Eleven's account**, tofu-managed from a root at `https://amadan.net/eleven/infrastructure`. | Tito covers the bill for now (Paul, 2026-08-30). The tofu root is Eleven-owned and lives outside the platform repos, per the customer-fleet rule. The repo exists and is empty. | |
| 93 | |
| 94 | ## Topology |
| 95 | |
| 96 | DNS stays pointed at Eleven's boxes. Each box runs `carlos agent` — edge plus |
| 97 | activator — issues its own wildcard certificates, and serves directly. |
| 98 | Carloku is reached outbound over the tunnel and carries control only: |
| 99 | releases, channels, config, placement, fleet visibility, wake. |
| 100 | |
| 101 | **Certificates.** A fleet box carrying no deployment bucket of its own has |
| 102 | `armCertIssuance` clear the issuance gate, because there is nothing to publish |
| 103 | to and nothing to mirror from; it issues for itself. Wildcards come from |
| 104 | `newWildcardFromEnv` under the same gate. |
| 105 | |
| 106 | **DNS provider — a constraint, not a coincidence.** `internal/edge/dns.go` |
| 107 | selects exactly one provider per box from `CARLOS_DNS_PROVIDER`. Eleven serves |
| 108 | four zones (`11ch.at`, `elevenmessenger.com`, `luan.chat`, `paulca.com`) and |
| 109 | **all four are on DNSimple**, so `CARLOS_DNS_PROVIDER=dnsimple` plus |
| 110 | `CARLOS_DNSIMPLE_TOKEN` covers the whole estate. Had any zone been elsewhere, |
| 111 | a directly-serving box could not have issued both wildcards and D4 would not |
| 112 | have been available. Record this: adding a zone at a second DNS provider |
| 113 | breaks the direct-serve path. |
| 114 | |
| 115 | The token is the vault's `lchat DNSimple DNS-01 token (11ch.at — Eleven |
| 116 | Messenger acct 176940)` — password slot, per `deploy/vault-items.sh`. |
| 117 | |
| 118 | **Three wildcards, not four — deliberately.** `CARLOS_WILDCARD_DOMAINS` lists |
| 119 | `11ch.at`, `elevenmessenger.com` and `luan.chat`. It omits `paulca.com`, |
| 120 | because Eleven serves exactly one host there (`chat.paulca.com`) and a |
| 121 | wildcard would grant the box authority to mint certificates for the whole of |
| 122 | a personal domain it has no other business in. An exactly-named route gets an |
| 123 | exact certificate through the ordinary registry host policy, which is the |
| 124 | correct outcome. Any future member custom domain follows the same rule: one |
| 125 | host means an exact name, a zone of instances means a wildcard. |
| 126 | |
| 127 | ## Phase 1 — the greenfield proving ground |
| 128 | |
| 129 | A CARLOS-native Eleven, standing beside production and touching none of it. |
| 130 | Everything here is throwaway: a namespace nobody's keys are bound to, on a box |
| 131 | that can be destroyed and rebuilt. |
| 132 | |
| 133 | ### Amended 2026-09-05 — phase 1 starts on the shared fleet, not a fresh box |
| 134 | |
| 135 | **Paul, 2026-09-05:** *"Since Eleven is decentralised anyway, I think the first |
| 136 | goal will be getting a running instance on `https://console.carloku.com/a/eleven/` |
| 137 | and then we go from here migrating the other apps."* |
| 138 | |
| 139 | This is a smaller and better first step than the box below, and it reorders |
| 140 | phase 1 rather than replacing it. The original phase 1 opens with |
| 141 | infrastructure — port `infra/dev-mesh` into a new tofu root, stand up an EC2 |
| 142 | box, point it at `carlos agent` — and only then proves the product. But **none |
| 143 | of phase 1's five proofs actually need Eleven to own the box.** Account |
| 144 | `eleven` (`bdr`) has no fleet, so an app created in it lands on the **shared |
| 145 | Carloku fleet**, which already exists and is already running. That defers the |
| 146 | tofu root, the AWS spend and the `eleven/infrastructure` repo out of the |
| 147 | critical path entirely. |
| 148 | |
| 149 | It also matches the product: Eleven is decentralised, one instance per space, |
| 150 | so a single instance *is* a working Eleven — not a stub standing in for one. |
| 151 | There is no all-at-once cutover to rehearse. |
| 152 | |
| 153 | **Two findings from 2026-09-05 that make this cheap:** |
| 154 | |
| 155 | 1. **The launch contract already matches, exactly.** CARLOS spawns an app as |
| 156 | `exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)` |
| 157 | (`internal/activator/backend_exec.go:691`). Eleven spawns its own instances |
| 158 | as `exec.Command(bin, "--socket", rt.Socket, "--db", dbPath)` |
| 159 | (`messenger/activator.go:1218`) — the same line, because CARLOS's activator |
| 160 | was ported from Eleven's. `lchat` needs **no code change** to be launched by |
| 161 | a CARLOS box. |
| 162 | |
| 163 | 2. **Verified by running it.** `lchat --socket <path> --db <path>` on an empty |
| 164 | directory boots, creates its SQLite database, prints an admin link, logs |
| 165 | `lchat listening on unix:<path>`, and answers **HTTP 200** with the app |
| 166 | shell over that socket. From nothing, with no config and no environment. |
| 167 | |
| 168 | **The one mismatch found: architecture.** `carlos ship --target` defaults to |
| 169 | `linux-arm64` and the shared fleet is Graviton, but the binary committed to |
| 170 | Eleven's repo is `lchat-linux-amd64`. Phase 1 needs an arm64 build. This is a |
| 171 | build-matrix line, not a design problem. |
| 172 | |
| 173 | **So phase 1 now runs in two steps.** Step A — one instance on the shared |
| 174 | fleet, which proves release delivery (proof 1), the manifest shape (proof 2) |
| 175 | and that the product works on CARLOS (proof 3), with no infrastructure at all. |
| 176 | Step B — the box below, which is still needed for the proofs step A cannot |
| 177 | reach: sidecars (proof 4) and the disposition of `lchat router`'s hooks and |
| 178 | links (proof 5), both of which want a box Eleven controls. Everything from |
| 179 | "The infrastructure already exists in the right shape" onward is step B. |
| 180 | |
| 181 | ### Step A is done — 2026-09-05 |
| 182 | |
| 183 | Eleven runs on Carloku. `proving.bdr.oncarlos.com` serves Eleven's app shell |
| 184 | over HTTPS from the shared fleet, on a binary shipped through `carlos`. |
| 185 | |
| 186 | The whole path, for the record: |
| 187 | |
| 188 | ARCH=arm64 ./deploy/build-linux.sh # in Eleven's repo → lchat-linux-arm64 |
| 189 | carlos apps create --account eleven --app proving |
| 190 | carlos deploy --account eleven --app proving --target linux-arm64 \ |
| 191 | --version ffd3de0 lchat-linux-arm64 |
| 192 | carlos instances enable --account eleven --app proving |
| 193 | carlos instances create --account eleven --app proving \ |
| 194 | --host proving.bdr.oncarlos.com --channel edge |
| 195 | |
| 196 | **What that proves.** |
| 197 | |
| 198 | - **Proof 1, release delivery, end to end.** `deploy` shipped `ffd3de0` and |
| 199 | promoted it to `edge`; the box's reconcile pass materialised the route and |
| 200 | the adopt installed the binary. The edge stamps `x-carlos-version: ffd3de0`. |
| 201 | - **Proof 2, the manifest shape.** `instances enable` pinned the app to |
| 202 | `bdr.oncarlos.com` and the one-label rule held: `proving` is one label under |
| 203 | the account's platform subdomain, and the console checked the host against |
| 204 | the manifest before minting the record. |
| 205 | - **Proof 3, the product runs on CARLOS.** The box logs |
| 206 | `lchat listening on unix:/run/carlos/proving.bdr.oncarlos.com.sock |
| 207 | (db: /data/proving.bdr.oncarlos.com.db)`, and the served page carries |
| 208 | Eleven's own CSP — `connect-src` naming `home.elevenmessenger.com`, the |
| 209 | giphy and youtube-nocookie sources — not a platform placeholder. Signing in |
| 210 | and creating a space is the remaining half of this proof and needs a person. |
| 211 | - **Hibernation works, unprompted.** The record read `Setting up`, then served, |
| 212 | then went to `Asleep` on its own. The instance woke for a request and slept |
| 213 | again with nothing configured for it. |
| 214 | |
| 215 | **Two findings worth carrying.** |
| 216 | |
| 217 | 1. **The admin link names the wrong host.** On first boot lchat prints |
| 218 | `http://localhost:1747/admin#token=…` because it has no idea what hostname |
| 219 | fronts it — it only ever knew a socket. An operator has to hand-edit the |
| 220 | host before the link works. Harmless once, tedious across 31 hosts, and a |
| 221 | trap for anyone who pastes it as printed. Worth a fix in Eleven: derive the |
| 222 | link's origin from a public-host env var when one is set. |
| 223 | |
| 224 | 2. **The architecture default is a real trip hazard.** `carlos ship` and |
| 225 | `carlos deploy` default `--target linux-arm64`; Eleven's committed binary is |
| 226 | `lchat-linux-amd64` and its `build-linux.sh` defaults `ARCH=amd64`. The two |
| 227 | defaults disagree, and nothing catches it until the box tries to exec the |
| 228 | binary. Eleven's build matrix should produce arm64 for Carloku. |
| 229 | |
| 230 | ### Step B — the proving box |
| 231 | |
| 232 | **The infrastructure already exists in the right shape.** `infra/dev-mesh/` in |
| 233 | Eleven's repo is complete tofu for exactly this box — EC2 `t4g.small` arm64 |
| 234 | (commented as "the sizing floor for a node that runs a sidecar"), security |
| 235 | group, IAM role and instance profile, EIP, a versioned S3 bucket with a |
| 236 | lifecycle rule, and `dev.11ch.at` delegated from DNSimple into a Route53 zone. |
| 237 | `ops dev rebuild` already destroys and recreates it from nothing. Phase 1 does |
| 238 | not design new infrastructure; it ports that shape into |
| 239 | `https://amadan.net/eleven/infrastructure` and points the box at `carlos agent` |
| 240 | instead of `lchat router`. |
| 241 | |
| 242 | **Use a fresh namespace, not `dev.11ch.at`** — the existing dev mesh keeps |
| 243 | working, and the two must not contend for one zone. |
| 244 | |
| 245 | **DNS provider.** The phase-1 zone is on **Route53**, so this box runs |
| 246 | `CARLOS_DNS_PROVIDER=route53`. Production is on DNSimple. Since the setting is |
| 247 | per box, the two coexist with no conflict — the single-provider constraint bites |
| 248 | only within one box. |
| 249 | |
| 250 | ### What phase 1 must prove |
| 251 | |
| 252 | Each of these is a review finding converted from an unknown into a rehearsal: |
| 253 | |
| 254 | 1. **Release delivery end to end** (the review's C1) — `carlos ship` Eleven's |
| 255 | binary, set channel pointers, and watch a box-side adopt install |
| 256 | `/opt/carlos/versions/…` and mint the live links. This is also where the |
| 257 | **signing-key migration** gets settled: Eleven signs its own releases today |
| 258 | (`deploy/ops publish`, key from Bitwarden), and from here on the authority is |
| 259 | Carloku's release key. |
| 260 | 2. **The manifest shape** (C3) — author an app and its manifest as a Carloku |
| 261 | operator (`carlos instances enable -domain <zone>`), then mint instances |
| 262 | through the ordinary door. Learn the one-label rule at small scale. |
| 263 | 3. **The product actually works on CARLOS** — log in to home, create a space, |
| 264 | invite a member, exchange messages. This is the part no amount of code |
| 265 | reading can establish. |
| 266 | 4. **Sidecars** (Open Q7) — provision a bot from nothing. Today's failure is |
| 267 | *unexplained*, and a greenfield box is where the explanation is cheapest to |
| 268 | find. If #223 has not landed, this is also where its `args` field and the |
| 269 | `<host>-sidecar.env` rename get exercised. |
| 270 | 5. **What of `lchat router` CARLOS cannot replace.** Hooks (`in.<host>`) and |
| 271 | links sit behind the router today, and `in.<host>` is two labels — which |
| 272 | `ValidHost` refuses. Phase 1 is where their disposition is decided, with |
| 273 | nothing depending on the answer. |
| 274 | |
| 275 | Phase 1 is finished when a person can sign in, make a space, talk in it, and |
| 276 | watch a bot answer — with `carlos` the only control plane involved. |
| 277 | |
| 278 | ## Phase 2 — move the production hosts, one at a time |
| 279 | |
| 280 | Fresh CARLOS boxes in Eleven's AWS account. The two existing boxes keep serving |
| 281 | until they are empty, then retire. **No box is ever converted in place**, which |
| 282 | is what removes the registry collision, the fleet-wide lease drain and the |
| 283 | ports 80/443 handover from this plan altogether. |
| 284 | |
| 285 | **The state does not move, again.** Old and new boxes point at the *same* |
| 286 | hibernate bucket and prefix, and `HibernatePrefix(prefix, host)` is `prod/<host>` |
| 287 | on both sides — verified against the live bucket. A moved host is restored by the |
| 288 | new box from the keys lchat already wrote. There is no database copy in this |
| 289 | plan, and `docs/instance-migration.md`'s copy step is not needed here. |
| 290 | |
| 291 | **The lease makes it safe by construction.** Both sides use the same litestream |
| 292 | `lss3.Leaser`, per host, at `<prefix>/<host>/lock.json`, TTL 15 minutes. Only one |
| 293 | box can hold a host's lease, so a half-moved host cannot be served twice. What |
| 294 | the TTL costs is a **bounded per-host wait**, scheduled deliberately, rather |
| 295 | than the fleet-wide outage that in-place conversion would have caused. |
| 296 | |
| 297 | ### Per-host move |
| 298 | |
| 299 | For each of the 31 hosts, in an order you choose (start with the least busy): |
| 300 | |
| 301 | 1. **Create the instance record** on the right app for that host's zone (per |
| 302 | phase 1's manifest work). One host at a time; there is no bulk import. |
| 303 | 2. **Let the host sleep**, or put it to sleep, so its lease is not held. |
| 304 | 3. **Drop its route on the old box**, so lchat will not wake it again. |
| 305 | 4. **Flip that host's A record** to the new box. A specific record beats the |
| 306 | `*.11ch.at` wildcard, so this moves exactly one host and nothing else. |
| 307 | 5. **Wake and verify** — the new box restores from `prod/<host>/` and serves. |
| 308 | Allow for the lease TTL if the old box held it recently. |
| 309 | 6. *Rollback:* flip the A record back and restore the old box's route. The |
| 310 | bucket state is shared, so nothing is stranded. |
| 311 | |
| 312 | **The wildcard moves last**, once every host has an explicit record, and the |
| 313 | old boxes come out of DNS entirely before they are destroyed. |
| 314 | |
| 315 | **Two box-side publishers retire with the old boxes** (phase 5): |
| 316 | `deploy/status-publish.sh` generates `fleet.json` every five minutes, and |
| 317 | `deploy/ssh-ledger-collect.sh` publishes the SSH access ledger on its own |
| 318 | timer. Both stop when the boxes are destroyed, and the public status page |
| 319 | decays silently rather than loudly. Resolve their rehoming **before** the last |
| 320 | box is destroyed, not after. |
| 321 | |
| 322 | **The sibling services move too.** Hooks and links are router-fronted on the |
| 323 | production boxes, so retiring `lchat router` retires them. They are part of |
| 324 | phase 2's surface — not, as an earlier draft claimed, out of scope — and |
| 325 | `in.<host>` needs the disposition phase 1 decided. |
| 326 | |
| 327 | ### What phase 2 still carries |
| 328 | |
| 329 | Fresh boxes dissolve the registry collision (C2), the ports 80/443 handover and |
| 330 | its reboot race, and the fleet-wide lease drain. What remains real: |
| 331 | |
| 332 | - **Release delivery** must work on the new boxes (proven in phase 1). |
| 333 | - **Manifests and 31 instance records** still have to be authored. |
| 334 | - **`CARLOS_POOLS` stays unset.** `inScope` gates pool first |
| 335 | (`reconcile.go:212`) and an empty `Pools` serves only the `""` pool |
| 336 | (`reconcile.go:1006-1011`) — correct only while every Eleven manifest carries |
| 337 | `Pool: ""`. Pin it. |
| 338 | - **Do not pass `--region` to `instances create`.** App placement overrides the |
| 339 | request body (`apiinstances.go:558-561`), so the flag does nothing here. |
| 340 | |
| 341 | ## Phase 3 — sidecars |
| 342 | |
| 343 | **Precondition: PR #223 merged and deployed to Carloku.** It is 55 commits and |
| 344 | +15.7k lines and has not moved since 2026-08-18. It does not go onto the |
| 345 | critical path of phases 1-2 (D2). |
| 346 | |
| 347 | **Why sidecars will not spawn AFTER the migration — and why that is a |
| 348 | different question from why they do not run today.** An earlier draft claimed |
| 349 | the `sidecar run` hardcode explained Paul's "sidecars don't spin up today". |
| 350 | That was wrong, and the review caught it: today's boxes run **Eleven's own** |
| 351 | activator, which spawns `bot run` from `/data/<host>-bot.env` and is a working |
| 352 | mechanism (messenger `activator.go:19-27,243`). Whatever stops sidecars today |
| 353 | is not this, and remains unexplained — establish it on a live box rather than |
| 354 | carrying a guess forward. |
| 355 | |
| 356 | What the hardcode does explain is a **post-migration** failure. CARLOS's exec |
| 357 | backend spawns `<release binary> sidecar run` (`backend_exec.go:826`) when |
| 358 | `/data/<host>-sidecar.env` exists, while Eleven's binary dispatches `bot` |
| 359 | (`main.go:190`) — so after the flip the child would exit on a usage error. |
| 360 | PR #223 closes it: a sidecar record carries a required `args` argv field, so |
| 361 | `carlos sidecars enable -app <app> -args 'bot run'` names the command instead |
| 362 | of hardcoding it. |
| 363 | |
| 364 | There is a second, quieter mismatch on the same path. CARLOS reads |
| 365 | `<DataDir>/<host>-sidecar.env` — its own comment calls this "Eleven's |
| 366 | `<host>-bot.env`, renamed" (`backend_exec.go:798`). Eleven's existing staged |
| 367 | env files are therefore **invisible** to CARLOS even after #223's `args` fix. |
| 368 | Phase 3 needs an explicit restage or rename step for every sidecar env file — |
| 369 | and phase 1 is where that is discovered rather than debugged in production. |
| 370 | |
| 371 | **Steps.** Provision the third box in account `427880590669` via tofu; attach |
| 372 | it as a fleet box with `CARLOS_INBOUND=none`; enrol the bot sidecars with |
| 373 | `-args 'bot run'` and `-placement no-inbound`. #223 refuses to mint routed |
| 374 | instances on a no-inbound box, so the isolation is platform-enforced rather |
| 375 | than conventional. |
| 376 | |
| 377 | **The hooks sidecar does not belong here.** `lchat hooks run` serves |
| 378 | `in.<host>` behind the router — it is **inbound**. #223's sidecar class is |
| 379 | defined as "a serving class that nothing can dial", so modelling it as a #223 |
| 380 | sidecar would make it permanently unreachable. But it cannot simply become an |
| 381 | ordinary instance with a route either: `in.<host>` is **two labels** under its |
| 382 | zone and `ValidHost` accepts only one. Its disposition is decided in phase 1, |
| 383 | where nothing depends on the answer. |
| 384 | |
| 385 | ## Phase 4 — edge-fronted ingress (deferred, gated) |
| 386 | |
| 387 | The eventual goal is members reaching a nearby edge. It is deferred behind two |
| 388 | entry conditions, both of which must be written down as met before any DNS |
| 389 | moves: |
| 390 | |
| 391 | 1. **Fleet-placed routes can be steered.** They cannot today. |
| 392 | `internal/steering/steering.go:313` excludes non-shared-pool routes |
| 393 | defensively — "the door refuses them" — and the CLI says `steering … |
| 394 | latency|off (shared pool only, v1)`. Steered names must also be in |
| 395 | `PLATFORM_APPS_DOMAIN`'s zone; a custom domain reaches steering by CNAME to |
| 396 | `edge.<apps-domain>`. |
| 397 | 2. **The WebSocket hop is measured against direct-serve** on a real |
| 398 | transatlantic path. Eleven is WebSocket fan-out (`hub.go`). Nearest-edge |
| 399 | wins the TLS and TCP handshake once; thereafter every message pays |
| 400 | edge → tunnel → box for the life of the connection. With boxes in |
| 401 | Germany/UK and edges in IE/US/UK/AU, that is plausibly a net loss in steady |
| 402 | state. Measure it; do not assume it either way. |
| 403 | |
| 404 | Two things this phase is **not**: |
| 405 | |
| 406 | - It is not Eleven's multi-edge. `docs/multi-edge.md`'s north star is "a space |
| 407 | live on several edges at once", built on the append-only-log-is-a-CRDT |
| 408 | property. CARLOS routes to the one box holding the lease; it does not |
| 409 | replicate instance state and should not learn to. Replicated spaces stay |
| 410 | Eleven's. |
| 411 | - It is not delivered by `instances set-upstreams`, which is health-probed |
| 412 | failover, not geo-routing. |
| 413 | |
| 414 | Direct-serve remains the rollback path in every case. |
| 415 | |
| 416 | ## Phase 5 — the public record: status site and prompt registry |
| 417 | |
| 418 | Both live in one S3 bucket today (`eleven-status-427880590669`, served at |
| 419 | `status.elevenmessenger.com` via CloudFront), but measurement shows they are two |
| 420 | unrelated problems: |
| 421 | |
| 422 | | | Objects | Bytes | |
| 423 | |---|---|---| |
| 424 | | Whole bucket | 521 | 283,396,762 (~270 MB) | |
| 425 | | `prompts/` alone | 422 | 282,761,414 (~270 MB) | |
| 426 | | **Everything else** | **99** | **~620 KB** | |
| 427 | |
| 428 | **99.8% of the bucket is the prompt corpus.** The status site proper is 620 KB. |
| 429 | |
| 430 | ### The status site → a CARLOS static app |
| 431 | |
| 432 | 620 KB across 99 objects is a straightforward static app, and `carlos deploy` |
| 433 | serves static natively. But hosting is the easy half. `docs/status.md` describes |
| 434 | two kinds of content and only one is static: |
| 435 | |
| 436 | - **Committed data** — `carbon.json`, `economics.json`, `contributors.json`, |
| 437 | `music-*.json`, `ssh/*.json`. Authored, reviewed, pushed with `ops pages |
| 438 | push`. Moves cleanly to a static app; `ops pages push` retires with it. |
| 439 | - **Generated data** — `fleet.json` and friends, produced **every five minutes |
| 440 | by `deploy/status-publish.sh` on the router box**: instance counts, awake vs |
| 441 | hibernating, version distribution, shared-service health, replication health. |
| 442 | |
| 443 | **The generator is the dependency, and it is a phase 2 problem.** That script |
| 444 | runs on a box phase 2 retires, so the status page starts decaying the moment the |
| 445 | last host crosses unless the generator has somewhere to go. Sequence this |
| 446 | accordingly: the static half can move early, but the generated half must be |
| 447 | resolved before the old boxes are destroyed. |
| 448 | |
| 449 | **The generator becomes a scheduled CARLOS app** (Paul, 2026-08-30). `carlos |
| 450 | schedule set --app <a> --name fleet --every 30m --path /generate` declares it, |
| 451 | and a box's runner "wakes a hibernating instance when its tick is due" — so a |
| 452 | sleeping generator is the supported path, not a trick. `--every` has a |
| 453 | **1-minute floor**, so any cadence under discussion is legal. |
| 454 | |
| 455 | **Amended 2026-09-05: the cadence argument this spec carried is void.** |
| 456 | The original text said "use 30 minutes, not 5 — because at 5 it never sleeps", |
| 457 | reasoning that a 5-minute tick resets `lastActive` three times per 15-minute |
| 458 | idle window and pins the instance awake. That was true when written and is not |
| 459 | true now. `Scheduled ticks must not hold an instance awake` (#371, on main |
| 460 | 2026-08-31) added `Service.ServeScheduled` and `beginScheduledRequest` |
| 461 | (`activator.go:1261`), which wakes a sleeping instance and holds the in-flight |
| 462 | count for the tick's duration but releases **without** stamping `lastActive`; |
| 463 | `activatorDeliverer.Deliver` calls it instead of `ServeHTTP` |
| 464 | (`schedule_wiring.go:349`). A tick now buys no residency at all. |
| 465 | |
| 466 | **So pick the cadence from what the page needs, not from the idle window.** |
| 467 | The status page's own contract is a five-minute `fleet.json`, and five minutes |
| 468 | is now affordable: the instance wakes, generates, and sleeps again one idle |
| 469 | window after the last *real* request. Match the existing cadence unless |
| 470 | something else argues against it. |
| 471 | |
| 472 | **Where it runs still matters, for a smaller reason than before.** |
| 473 | `CARLOS_HIBERNATE_IDLE` is still read once from the **box** environment |
| 474 | (`activator.go:92`) with no per-route or per-app override — gap 3 below stands. |
| 475 | But #371 removes the sharp edge that made placement urgent: the generator no |
| 476 | longer needs a short idle to sleep, so it no longer wants a box configured |
| 477 | against its neighbours. Prefer the **shared Carloku fleet** anyway — a |
| 478 | generator reading the console API needs no fleet-local knowledge, it keeps |
| 479 | Eleven's boxes for Eleven's members, and it is the cheaper rung — but this is |
| 480 | now a preference, not a constraint. |
| 481 | |
| 482 | Worth exploring rather than assuming: **CARLOS already knows much of what |
| 483 | `fleet.json` reports** — instances, awake state, versions, heartbeats are all |
| 484 | console-API facts. Part of the page may become an API consumer rather than a |
| 485 | box-side generator, which would remove the box dependency entirely. What CARLOS |
| 486 | does *not* know is Eleven's own aggregate vocabulary (allocation band words, |
| 487 | fleet-backup replication health), so some generator is likely to survive; the |
| 488 | question is what it runs on and what it reads. |
| 489 | |
| 490 | **The privacy rules are non-negotiable and must survive the move.** |
| 491 | `docs/status.md` states them as holding *by construction* in the generator — |
| 492 | aggregates only, never a hostname. A rehomed generator must carry the same |
| 493 | boundary, and the existing contract test (`test/status-contract.test.mjs`) is |
| 494 | the thing that proves it. Port the test with the generator, or the guarantee |
| 495 | becomes a claim. |
| 496 | |
| 497 | ### The prompt registry → `carlos ledger`, not git |
| 498 | |
| 499 | Amadan keeps its prompt registry in git because it is small. Eleven's is **270 |
| 500 | MB and grows per session, per contributor**, in a versioned bucket — Eleven's own |
| 501 | docs say why it is not committed: *"far too large to commit."* Git would store |
| 502 | every revision forever and every clone would pull all of it. Three orders of |
| 503 | magnitude separate the two cases; the same answer does not fit both. |
| 504 | |
| 505 | `carlos ledger` already provides the right primitives, and per the open-ledgers |
| 506 | work it was designed from Eleven's redaction rules in the first place: |
| 507 | |
| 508 | - `blob` — content-addressed, create-only uploads (re-uploading identical bytes |
| 509 | is a no-op), which is exactly the corpus's shape. |
| 510 | - `append` — hash-chained entries for the session index. |
| 511 | - `publish` — the public manifest, with a title. |
| 512 | - `verify` — walks a **published** chain over plain HTTPS and re-hashes every |
| 513 | entry, needing **no credentials**. Anyone can check it. |
| 514 | |
| 515 | **On rewriting for redaction.** Git's rewritability looks like an advantage here |
| 516 | and is not the property to want: a force-push makes tampering *silent*, while a |
| 517 | hash chain with public `verify` makes it *detectable*. Redaction belongs |
| 518 | **before** append — which is what `tools/prompt-registry.mjs` already does, |
| 519 | redacting credentials and annotating each redaction in place. If something slips |
| 520 | through, both models require a repair; only the ledger makes that repair |
| 521 | visible. Treat immutability as the feature and the pre-append redaction pass as |
| 522 | the control. |
| 523 | |
| 524 | **The hybrid, made concrete** — it is the split Eleven already uses, rehomed: |
| 525 | `prompts.html` is committed *code* and belongs in the static site's repo; the |
| 526 | corpus is *data* and belongs in ledger blobs. They compose, because `ledger |
| 527 | verify --url <status-host>` points at exactly the host the static app serves. |
| 528 | |
| 529 | ### Open for this phase |
| 530 | |
| 531 | - **How much of the generator collapses into console-API reads** once it is a |
| 532 | scheduled app. Where it runs is settled (shared fleet, `--every 30m`); what |
| 533 | it still needs a box-side view for is not. |
| 534 | - **What one ledger entry is** — a session, a batch, a contributor-day — and how |
| 535 | a contributor appends without holding broad credentials. |
| 536 | - **Whether the music and SSH ledgers follow the same path.** Both are accrued |
| 537 | public records with the same shape; the SSH ledger additionally has a box-side |
| 538 | collector with the same retirement problem as the fleet generator. |
| 539 | - **What happens to the existing corpus** at cutover: re-uploaded as blobs |
| 540 | (content-addressing makes it idempotent) or left in the old bucket behind a |
| 541 | redirect. |
| 542 | |
| 543 | ## What stays Eleven's, permanently |
| 544 | |
| 545 | - **Mesh replication** (`meshreplicate.go`, `prod/mesh/`). CARLOS deliberately |
| 546 | has no mesh prefix — `internal/activator/activator.go:117` records the seam. |
| 547 | - **The reaper** (`reaper.go`), reclaiming never-claimed zero-member spaces. |
| 548 | CARLOS has no equivalent and needs none. |
| 549 | - **Self-serve space creation** (`controld.go`, `cloud.go`). This becomes a |
| 550 | console API caller holding a **service credential** (#220/#227, merged) |
| 551 | rather than a local root daemon — but the quota, slug and capacity policy |
| 552 | stay Eleven's. |
| 553 | - **The entire E2EE layer**, which CARLOS never sees and must never need to. |
| 554 | |
| 555 | ## Capability map |
| 556 | |
| 557 | | Eleven today | CARLOS | Verdict | |
| 558 | |---|---|---| |
| 559 | | `ops publish` — build + sign | `carlos ship`, `release-keygen` | replace | |
| 560 | | `edge\|beta\|stable`, `ops promote` | `carlos promote` / `pipeline` / `channels` | replace | |
| 561 | | `ops ship` sweep, `AUTO=pinned` | `carlos adopt`, `instances set-channel` | replace | |
| 562 | | `lchat router` subdomain routing | edge + registry routes | replace | |
| 563 | | certcache + wildcard DNS-01 | edge wildcard manager, DNSimple backend | replace — same design | |
| 564 | | `activator.go` hibernate + lease | CARLOS activator | replace — *ported from Eleven* | |
| 565 | | Bitwarden vault contract | `carlos secrets` | replace with care — the vault contract is deliberate | |
| 566 | | `controld` / `cloud.go` | console API + service credential | rehost, keep policy | |
| 567 | | `reaper.go`, `meshreplicate.go` | — | **stays Eleven's** | |
| 568 | | bot sidecars | PR #223 | **phases 1 and 3** | |
| 569 | | hooks sidecar (`in.<host>`) | *neither* | **not a #223 sidecar, and not an instance record either**: two labels fails `ValidHost`. Disposition decided in phase 1. | |
| 570 | |
| 571 | ## Gaps to file against the platform |
| 572 | |
| 573 | 1. **The reconcile dry-run has no operator door.** `ReconcileConfig.DryRun` |
| 574 | exists and is documented as design §4.3's backfill mitigation, but nothing |
| 575 | outside `internal/provision` sets it; the only CLI-reachable `DryRun` is |
| 576 | `status.DryRun`, the unrelated leak check. An operator cannot ask a box |
| 577 | "what would you adopt, and what does adopting it let you later delete?" |
| 578 | |
| 579 | **The fresh-box strategy (D1) demotes this from blocking to wanted.** A box |
| 580 | that boots with an empty registry and mints its own records is the ordinary |
| 581 | path, not a backfill — the dangerous case was adopting a live box's |
| 582 | pre-existing lchat rows, and no box is converted in place any more. Still |
| 583 | worth building: the question is the right one to be able to ask, and the |
| 584 | next customer arriving with an existing estate will be a genuine backfill. |
| 585 | |
| 586 | 2. **No bulk instance-record import.** Phase 2 is 31 hand-authored |
| 587 | `instances create` calls. Tolerable once at this size; it is a real gap the |
| 588 | moment a second customer arrives with an existing estate, and it is worth |
| 589 | filing now while the pain is concrete. |
| 590 | |
| 591 | 3. **No per-app hibernation idle.** `CARLOS_HIBERNATE_IDLE` is box-wide |
| 592 | (`activator.go:92`) with no route- or app-level override. Scheduled-work |
| 593 | apps want a short idle (wake, work, sleep); interactive apps want a long one |
| 594 | (do not make a returning member pay a cold wake). One box cannot serve both |
| 595 | well today. **Amended 2026-09-05: this is no longer urgent.** #371 means a |
| 596 | tick does not stamp `lastActive`, so a scheduled app sleeps on the box's own |
| 597 | idle without needing one of its own — which was the case that made this |
| 598 | gap bite. It stays a real gap for apps whose *own* traffic wants a different |
| 599 | idle from their neighbours', but phase 5 no longer depends on it. Still |
| 600 | worth a `Route.Idle` override. |
| 601 | |
| 602 | 4. **Fleet steering.** Blocks phase 4. Latency steering is shared-pool only in |
| 603 | v1; fleet-placed routes are refused at the door and excluded by the |
| 604 | converger. |
| 605 | 5. **No "list an account's apps" verb.** `carlos instances list -account X` |
| 606 | answers only "account X has no apps"; there is no listing for an account |
| 607 | that *has* apps without naming one. This is a product gap under AGENTS.md's |
| 608 | API-and-CLI rule, found while establishing the starting state. |
| 609 | 6. ~~**`infra/eleven-fleet.md` is wrong.**~~ **Closed 2026-09-05: the file no |
| 610 | longer exists.** It was deleted from main in the open-source redaction pass |
| 611 | (`2adc71f`, 2026-08-27) — two days *before* this spec was written, so the |
| 612 | spec's "Supersedes" line described a file that was already gone. Nothing to |
| 613 | rewrite; this document is the only account of Eleven's starting state. |
| 614 | 7. **Multi-provider wildcards.** One `CARLOS_DNS_PROVIDER` per box. Not |
| 615 | blocking today (all four zones are DNSimple), but it is a latent constraint |
| 616 | on any directly-serving box whose zones are split. |
| 617 | |
| 618 | ## Open questions |
| 619 | |
| 620 | The two-phase reframe settled most of the previous list — the registry |
| 621 | disposition, the lease-drain window and the front-door handover are all gone |
| 622 | with in-place conversion. What genuinely remains: |
| 623 | |
| 624 | 1. **Which of the 31 hosts belongs to which app**, and therefore how many apps |
| 625 | there are. Needs an inventory before phase 2 starts; phase 1 establishes the |
| 626 | shape at four hosts first. |
| 627 | |
| 628 | 2. **The disposition of `in.<host>`** (hooks). Two labels under the zone, which |
| 629 | `ValidHost` can never accept — so it is neither a #223 sidecar nor an |
| 630 | instance record. Decided in phase 1. |
| 631 | |
| 632 | 3. **The signing-key migration.** Whether Eleven's release key is retired at |
| 633 | cutover or cross-published, and what happens to releases already signed with |
| 634 | it. Phase 1 settles it in practice. |
| 635 | |
| 636 | 4. **Why do sidecars not spin up today?** Eleven's own activator has a working |
| 637 | `bot run` path (`activator.go:19-27,243`), so the previously-assumed |
| 638 | explanation is void. Phase 1 answers this from nothing. |
| 639 | |
| 640 | 5. **What does a hibernating bot sidecar persist across wakes?** The Go sidecar |
| 641 | writes a `session.json` byte-compatible with the Node one. Whether that lives |
| 642 | on local disk, in a bucket, or is re-minted per wake decides whether the |
| 643 | sidecar box needs a data bucket of its own. |
| 644 | |
| 645 | 6. **What the old boxes cost to keep running during phase 2**, and therefore how |
| 646 | long the parallel window can be. Both estates run at once until the last host |
| 647 | crosses. |
| 648 | |
| 649 | 7. **Where the estate lives long-term.** AWS is the phase-2 answer with Tito |
| 650 | covering the bill (D9), but `docs/planet-scale.md` treats Hetzner as the |
| 651 | production plane deliberately, on cost. Worth revisiting once real running |
| 652 | costs are visible. |
| 653 | |
| 654 | ## Security finding (out of scope here; file separately) |
| 655 | |
| 656 | The vault item *"AWS dev account (Eleven free tier) — dev-agent access key"* |
| 657 | resolves to `arn:aws:iam::427880590669:user/dev-agent` carrying |
| 658 | `AdministratorAccess` — on the same account that holds every production |
| 659 | instance's parked state. The name describes something much smaller than the |
| 660 | grant. The contrast is in the same vault: `activator-prod` is properly scoped |
| 661 | and refuses `s3:GetBucketLocation`. Narrow the grant or rename the item; it is |
| 662 | unrelated to this migration but was found by it. |
| 663 | |
| 664 | ## What this spec does not do |
| 665 | |
| 666 | - It does not move Eleven's data. Old and new boxes share one hibernate bucket |
| 667 | and prefix, and `HibernatePrefix` produces the same keys on both sides — a |
| 668 | moved host is restored in place, not copied. |
| 669 | - It does not convert any existing box. The two production boxes serve until |
| 670 | empty, then retire. |
| 671 | - It does not change any space's address. Phase 1 runs on a throwaway |
| 672 | namespace precisely because an address is an identity, and phase 2 moves |
| 673 | hosts without renaming them. |
| 674 | - It does not build replicated spaces. That stays Eleven's mesh work. |
| 675 | - It does not change Eleven's E2EE model in any way. |
| 676 | |
amadan