| 1 | # 🤖 Migrating an instance from one server to another |
| 2 | |
| 3 | > 🤖 **Not yet encoded — do not run from this page.** There is no |
| 4 | > `ops` verb for instance migration yet. When a migration is next needed, |
| 5 | > the move is to build one from this record (`ops migrate <host> <box>` or |
| 6 | > similar, per `deploy/README.md § The operator surface`'s no-cookbooks |
| 7 | > rule), then run the verb — and mark this page historical. |
| 8 | |
| 9 | Status: **design record** (the shape below was validated once in practice). Moving a running messenger |
| 10 | instance to a new box is **data movement + a DNS flip**, done so it's reversible: |
| 11 | copy the DB, keep the old instance running as a fallback, and only cut DNS over |
| 12 | once the new box is verified. The flip itself is reversible too. |
| 13 | |
| 14 | The key property: **keep the hostname the same**. Browser keys (and passkeys, for |
| 15 | the home server) are per-origin, so an unchanged origin means members' keys, |
| 16 | sessions, and history are all unaffected — only the DNS record moves. |
| 17 | |
| 18 | ## Before you start |
| 19 | |
| 20 | - SSH to both the source and destination boxes. |
| 21 | - The destination is already bootstrapped (`lchat bootstrap`; router running). |
| 22 | - Know the instance's **admin token** and any per-instance config (see step 1). |
| 23 | |
| 24 | ## Steps |
| 25 | |
| 26 | ### 1. Recover the instance's config from the source |
| 27 | |
| 28 | The instance's DB stores only the **hash** of the admin token — the plaintext |
| 29 | lives in the source's deploy env / secret store. Recover it, plus any per-instance |
| 30 | config the instance reads (`LCHAT_PUSH_RELAY_URL`/`_TOKEN`, `LCHAT_HOME_ORIGIN`, |
| 31 | `LCHAT_APP_STORE_URL`, APNs vars, …). You'll write these into the destination's |
| 32 | EnvironmentFile in step 4. **Preserve the admin token** — minting a new one |
| 33 | invalidates every saved admin link. |
| 34 | |
| 35 | ### 2. Snapshot the DB consistently |
| 36 | |
| 37 | Use SQLite's `VACUUM INTO` — it produces a single, compacted, transaction-consistent |
| 38 | file and is **safe on a live DB** (no need to stop the instance): |
| 39 | |
| 40 | ```sh |
| 41 | sqlite3 /path/to/lchat.db "VACUUM INTO '/tmp/snapshot.db'" |
| 42 | sqlite3 /tmp/snapshot.db "PRAGMA integrity_check" # expect: ok |
| 43 | ``` |
| 44 | |
| 45 | If the source hides the DB behind an opaque store (many identically-named DB files), |
| 46 | identify the right one by **content** — e.g. grep the files for the instance's |
| 47 | `admin_token_hash` (`hex(sha256(admin token))`), which is unique per instance. |
| 48 | |
| 49 | `VACUUM INTO` snapshots the **DB only**. Attachment/photo/video bytes live in a |
| 50 | **sibling directory** `<db>-blobs/` (e.g. `/data/<host>-blobs/`), not in the DB — |
| 51 | so it must be copied separately (step 3), or migrated messages lose their images |
| 52 | and videos. The files are immutable and content-addressed by a random id, so a |
| 53 | live copy is safe. (An older instance that predates disk-backed blobs has no such |
| 54 | directory — its blobs are still inside the DB and need nothing extra.) |
| 55 | |
| 56 | ### 3. Copy the snapshot to the destination |
| 57 | |
| 58 | Copy `/tmp/snapshot.db` to the destination as `/data/<host>.db` and verify it |
| 59 | landed intact (`integrity_check` + a couple of row counts against the source). |
| 60 | **Also copy the blob directory** if the source has one: |
| 61 | |
| 62 | ```sh |
| 63 | rsync -a /data/<host>-blobs/ <dest>:/data/<host>-blobs/ |
| 64 | ``` |
| 65 | |
| 66 | These are the disk-backed attachment bytes the DB references by id; copy them at |
| 67 | or after the DB snapshot (blobs uploaded to the source afterward are subject to |
| 68 | the same gap check as messages — see step 5). |
| 69 | |
| 70 | **Also copy the VAPID keyfile** (`/data/<host>.vapid`, another DB sibling — |
| 71 | kept out of the DB so S3 replicas never carry a private key). Without it the |
| 72 | destination re-mints the pair on first boot and every member's web-push |
| 73 | subscription silently breaks until their next visit re-subscribes them. |
| 74 | |
| 75 | ### 4. Import on the destination (preserve token + config) |
| 76 | |
| 77 | With the migrated DB already at `/data/<host>.db`, run `add-instance` with |
| 78 | `--admin-token` so it **imports** the existing token instead of minting a new one |
| 79 | (a fresh token would rewrite `admin_token_hash` on boot and orphan every saved |
| 80 | admin link). It writes the EnvironmentFile, registers the route, starts the unit |
| 81 | against the pre-placed DB, health-checks it, and reloads the router: |
| 82 | |
| 83 | ```sh |
| 84 | lchat add-instance <host> \ |
| 85 | --admin-token "<existing token>" \ |
| 86 | --push-relay-url https://push.example.com --push-relay-token "<existing relay token>" |
| 87 | ``` |
| 88 | |
| 89 | Pass whatever per-instance config the source had (`--push-relay-url`/`-token`, |
| 90 | `--home-origin`, `--app-store-url`). Verify **before any DNS change**, over the |
| 91 | instance's unix socket: `GET /` → 200, an admin-token request → 200, and the row |
| 92 | counts match the source. |
| 93 | |
| 94 | ### 5. Gap check |
| 95 | |
| 96 | Confirm nothing was written to the source between the snapshot and the cutover |
| 97 | (compare a row count, e.g. `messages`, source-now vs snapshot). For a busy |
| 98 | instance, freeze writes or take the snapshot at cutover time — writes that land on |
| 99 | the source after the snapshot are **not** on the destination. |
| 100 | |
| 101 | ### 6. Cut over DNS |
| 102 | |
| 103 | Flip the host's A record to the destination's IP. **Lower the record's TTL ahead of |
| 104 | time** (to ~60s, ideally a TTL-worth before the cutover) so the flip propagates |
| 105 | fast; otherwise resolvers that cached the old IP keep hitting the old box until the |
| 106 | old TTL expires (harmless while the old box is still up as a fallback). |
| 107 | |
| 108 | The TLS cert **can't be pre-issued**: autocert's HTTP-01 validates by resolving the |
| 109 | host over public DNS, so the cert only issues once DNS points at the destination. |
| 110 | Authoritative DNS updates in seconds (only resolver *caches* lag), so the cert |
| 111 | issues on the first HTTPS hit right after the flip. |
| 112 | |
| 113 | ### 7. Verify + keep the fallback |
| 114 | |
| 115 | `GET https://<host>/` → 200 with a valid (Let's Encrypt) cert, serving the migrated |
| 116 | data. **Leave the source instance running** until you're confident — the migration |
| 117 | is a copy, so flipping DNS back is an instant rollback. Retire the source only |
| 118 | afterward. |
| 119 | |
| 120 | ## Bots (sidecars) |
| 121 | |
| 122 | The instance's **bot agent-users migrate with the DB** (they're rows), but the |
| 123 | **sidecar** — the Node process and its session files (the bots' identities) — is |
| 124 | separate and does *not* come across in the DB copy. Do **not** re-run `add-bot` on |
| 125 | the destination: that mints a *new* set of agent identities (duplicate bots in the |
| 126 | Helpers catalog). Instead either copy the sidecar's session files to the |
| 127 | destination and start `lchat-bot@<host>` there, or point the existing sidecar at |
| 128 | the destination (it follows the host over DNS and reconnects with its persisted |
| 129 | token). Plan the bot move as its own step. |
| 130 | |
| 131 | ## What makes this safe |
| 132 | |
| 133 | - **Reversible**: the source is copied, never moved; DNS flip is reversible; the |
| 134 | cert issues automatically post-flip. |
| 135 | - **No key churn**: same origin ⇒ per-origin browser keys/passkeys untouched. |
| 136 | - **Consistent snapshot**: `VACUUM INTO` captures a clean point-in-time DB; the |
| 137 | sibling `<db>-blobs/` dir (immutable, content-addressed files) copies alongside. |
| 138 | |
amadan