amadan
docs: Eleven→Carloku fleet migration runbook
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+152
−0
infra/eleven-fleet.md+152 −0
diff --git a/infra/eleven-fleet.md b/infra/eleven-fleet.md| new file mode 100644 |
| index 000000000..4c10f803f |
| --- /dev/null |
| +++ b/infra/eleven-fleet.md |
| @@ -0,0 +1,152 @@ |
| +# Eleven → Carloku fleet migration runbook |
| + |
| +Eleven (elevenmessenger.com, Tito's flagship CARLOS app) moves from Tito's |
| +own self-hosted CARLOS install (`console.platform.tito.io`, on |
| +`carlos-tito-ie`) onto Carloku's console (`console.carloku.com`) as a |
| +**customer fleet**: Eleven's account on Carloku owns fleet `eleven`, the two |
| +production boxes (already in Tito's own AWS account, `titogo`) attach as |
| +fleet boxes, and every Eleven app is placed `fleet/eleven` — opted out of |
| +Carloku's shared fleet by construction. Eleven's data never leaves Eleven's |
| +own bucket. The boxes' network posture does not change: outbound-only dial. |
| + |
| +Executed by Paul, one box at a time, with verification and an abort path at |
| +every step. Scope is the **fleet attach + placement** flip only — Eleven's |
| +account and app records on Carloku are assumed to already exist (see |
| +Preconditions). Not a data migration: there is no data to migrate. |
| + |
| +## Preconditions (must be true before Step 1) |
| + |
| +- [ ] **#73 (fleet API surface) merged and deployed to Carloku's flagship |
| + edge.** *Not true at time of writing* — branch `worktree-fleet-api` has |
| + the design (`docs/superpowers/specs/2026-08-11-fleet-api-surface-design.md`) |
| + and the first edge piece (`VerifyDynamic`, `2c9f697`/`b2b0867`), but the |
| + store door (`/store/o/<key>` etc.), the `internal/fleetapi` client, and |
| + the env-driven store selection in `openDeployStore` aren't built yet. |
| + Without it a fleet box still needs RW on Carloku's platform deployment |
| + bucket to run — unacceptable for boxes in Tito's own AWS account. |
| +- [ ] **`region-remote-host`'s `deployment_bucket_arn` is optional.** Also |
| + #73 rollout item 4 — today the variable is required, no default |
| + (`infra/modules/region-remote-host/variables.tf`). Eleven's tofu root |
| + must be able to omit the grant. |
| +- [ ] **Eleven's account exists on Carloku**, passkey-reachable, apps already |
| + created under it (`carlos accounts create -name eleven -owner |
| + <address>`, then the normal `apps create` flow) — currently unplaced, |
| + since the fleet doesn't exist yet. |
| +- [ ] **Eleven-owned tofu root**, `region-remote-host`/`paulca/pi-fleet` |
| + pattern, applied in `titogo` against Eleven's existing data bucket — |
| + `deployment_bucket_arn` **omitted** (#73: no platform-bucket creds |
| + needed at all, only the box's own bucket). |
| +- [ ] `carlos` agent binary on both boxes built from a post-#73 `main` |
| + (stale-checkout mismatch is a recorded trap — build fresh). |
| + |
| +## Ceremony |
| + |
| +Signed in to Carloku (`carlos auth login -console https://console.carloku.com`) |
| +as a member of Eleven's account, or `-account eleven` as ops. |
| + |
| +**1. Create the fleet** |
| + |
| + carlos fleets create -bucket <eleven's data bucket> -account eleven eleven |
| + |
| +Mints the record only; the bucket is taken on trust that it exists (it |
| +does). The fleets namespace is flat and global — a 409 "that name is taken" |
| +doesn't necessarily mean *this* attempt took it. |
| + |
| +*Verify:* `carlos fleets ls -account eleven` shows fleet `eleven`, correct |
| +bucket, no boxes. |
| +*Abort:* no `fleets delete` verb exists (`create|add-box|rotate-token| |
| +tag-box|detach-box|ls` is the full set). A fleet with the wrong bucket has |
| +no clean undo short of an operator-side registry fix — get the bucket name |
| +right first. |
| + |
| +**2. Add box 1** |
| + |
| + carlos fleets add-box -account eleven eleven <box-1-label> |
| + |
| +Token prints **exactly once** — capture it now (`rotate-token` mints a new |
| +one; the plaintext is never re-shown). |
| + |
| +*Verify:* `carlos fleets ls -account eleven` shows the box, status `active`. |
| +*Abort:* if added in error and never dialed, `carlos fleets detach-box |
| +-account eleven eleven <box-1-label>` disables the row (no delete verb, but |
| +detach suffices). |
| + |
| +**3. Flip box 1's env** |
| + |
| +Stage a new `/etc/carlos/host.env` (keep the old file as |
| +`host.env.pre-carloku` — the entire rollback for this step). Post-#73 |
| +contract — no `CARLOS_DEPLOYMENT_BUCKET`, no platform-bucket half of |
| +`/etc/carlos/credentials`, hibernate-bucket creds only: |
| + |
| + AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials |
| + CARLOS_HIBERNATE_BUCKET=<eleven's data bucket> |
| + CARLOS_FLEET=eleven |
| + CARLOS_FLEET_BOX=<box-1-label> |
| + CARLOS_REGION=fleet/eleven |
| + CARLOS_STATUS_PREFIX=fleets/eleven/ |
| + CARLOS_TUNNEL_URL=https://<Carloku's reserved tunnel hostname> |
| + CARLOS_TUNNEL_NAME=eleven/<box-1-label> |
| + CARLOS_TUNNEL_TOKEN=<from Step 2 — shown once> |
| + |
| +`CARLOS_FLEET_BOX` defaults from `CARLOS_TUNNEL_NAME`'s label half, but |
| +write it explicitly. `/etc/carlos/credentials` now holds ONLY the |
| +hibernate-bucket key from the Eleven-owned tofu root — any platform-bucket |
| +key pair the file carried for tito's own console is removed here, not just |
| +unused. Restart the agent unit to pick up the new env. |
| + |
| +*Verify:* |
| +- `journalctl -u carlos-edge` shows "tunnel connected". |
| +- Carloku fleet page (`/a/<eleven-sqid>/fleets/eleven`) shows box 1 with a |
| + fresh heartbeat, tunnel connected. |
| +- `carlos fleets ls -account eleven` still shows the box `active` (env flip |
| + changes which console it dials, not registry status). |
| + |
| +**Do not proceed to box 2 until box 1 verifies clean** — the whole point of |
| +flipping one box at a time is that box 2 keeps serving Eleven's traffic |
| +under its prior config while box 1 is proven. |
| + |
| +*Abort:* restore `host.env.pre-carloku`, restart the unit — the box re-dials |
| +whatever it dialed before, nothing on Carloku needs touching. For a |
| +suspected-compromised token rather than a bad env, `carlos fleets |
| +detach-box -account eleven eleven <box-1-label>` is the hard cut — closes |
| +the channel immediately; `rotate-token` only refuses the *next* dial-in. |
| + |
| +**4. Repeat Steps 2-3 for box 2**, only once box 1 is verified clean. |
| + |
| +## App placement |
| + |
| +For each Eleven app (already created under the account, per Preconditions): |
| + |
| + carlos apps place -app <name> -target fleet/eleven -account eleven |
| + |
| +`-target` is required and tri-state (`""` clears placement — never omit it; |
| +an omitted `-target` is a usage error, not a no-op). Placement re-targets |
| +*serving* only, converging within a pass — it does not copy state (v1's |
| +"nothing migrates," restated as D6 in the fleets-console spec). That caveat |
| +is moot here: Eleven's instance state was never anywhere but Eleven's own |
| +bucket, so there's nothing to leave behind. Only the control plane moves. |
| + |
| +*Verify per app:* `curl -sI https://<app-host>` (`--resolve` at Carloku's |
| +edge IP ahead of any DNS cutover — never trust a cached answer) shows |
| +`X-Carlos-Version` matching what Eleven expects. |
| + |
| +*Abort:* `-target ""` un-places, but on Carloku that lands the app on the |
| +**shared** fleet, which has no access to Eleven's bucket — not a rollback to |
| +a working state. If a box misbehaves, fix or detach it and leave placement |
| +at `fleet/eleven`: the dark-fleet 503 (age shown, nothing silently wrong) is |
| +the intended failure mode here, not un-placing. |
| + |
| +## Post-flip |
| + |
| +- [ ] **Revoke the old platform-bucket IAM grant.** Whatever principal gave |
| + Eleven's boxes RW on tito's own platform deployment bucket (from |
| + tito's own `platform-infrastructure` tofu) is no longer needed once |
| + both boxes verify clean on Carloku — revoke it; don't delete the |
| + principal blind without confirming nothing else references it. |
| +- [ ] **DNS for elevenmessenger.com — a Paul decision.** Point the host |
| + directly at Carloku's edge (clean cutover, `X-Carlos-Version` is |
| + immediately authoritative, but TTL/propagation lag is live downtime |
| + risk for a flagship app) versus leaving DNS on tito's own edge proxying |
| + or CNAMEing through to Carloku (slower, an extra hop, but a fallback |
| + that doesn't depend on DNS propagation). Both are compatible with |
| + everything above; neither is assumed here. |