🤖 Migrating an instance from one server to another

> 🤖 Not yet encoded — do not run from this page. There is no > ops verb for instance migration yet. When a migration is next needed, > the move is to build one from this record (ops migrate <host> <box> or > similar, per deploy/README.md § The operator surface's no-cookbooks > rule), then run the verb — and mark this page historical.

Status: design record (the shape below was validated once in practice). Moving a running messenger instance to a new box is data movement + a DNS flip, done so it's reversible: copy the DB, keep the old instance running as a fallback, and only cut DNS over once the new box is verified. The flip itself is reversible too.

The key property: keep the hostname the same. Browser keys (and passkeys, for the home server) are per-origin, so an unchanged origin means members' keys, sessions, and history are all unaffected — only the DNS record moves.

Before you start

• SSH to both the source and destination boxes.
• The destination is already bootstrapped (lchat bootstrap; router running).
• Know the instance's admin token and any per-instance config (see step 1).

Steps

1. Recover the instance's config from the source

The instance's DB stores only the hash of the admin token — the plaintext lives in the source's deploy env / secret store. Recover it, plus any per-instance config the instance reads (LCHAT_PUSH_RELAY_URL/_TOKEN, LCHAT_HOME_ORIGIN, LCHAT_APP_STORE_URL, APNs vars, …). You'll write these into the destination's EnvironmentFile in step 4. Preserve the admin token — minting a new one invalidates every saved admin link.

2. Snapshot the DB consistently

Use SQLite's VACUUM INTO — it produces a single, compacted, transaction-consistent file and is safe on a live DB (no need to stop the instance):

sqlite3 /path/to/lchat.db "VACUUM INTO '/tmp/snapshot.db'"
sqlite3 /tmp/snapshot.db "PRAGMA integrity_check"   # expect: ok

If the source hides the DB behind an opaque store (many identically-named DB files), identify the right one by content — e.g. grep the files for the instance's admin_token_hash (hex(sha256(admin token))), which is unique per instance.

VACUUM INTO snapshots the DB only. Attachment/photo/video bytes live in a sibling directory <db>-blobs/ (e.g. /data/<host>-blobs/), not in the DB — so it must be copied separately (step 3), or migrated messages lose their images and videos. The files are immutable and content-addressed by a random id, so a live copy is safe. (An older instance that predates disk-backed blobs has no such directory — its blobs are still inside the DB and need nothing extra.)

3. Copy the snapshot to the destination

Copy /tmp/snapshot.db to the destination as /data/<host>.db and verify it landed intact (integrity_check + a couple of row counts against the source). Also copy the blob directory if the source has one:

rsync -a /data/<host>-blobs/ <dest>:/data/<host>-blobs/

These are the disk-backed attachment bytes the DB references by id; copy them at or after the DB snapshot (blobs uploaded to the source afterward are subject to the same gap check as messages — see step 5).

Also copy the VAPID keyfile (/data/<host>.vapid, another DB sibling — kept out of the DB so S3 replicas never carry a private key). Without it the destination re-mints the pair on first boot and every member's web-push subscription silently breaks until their next visit re-subscribes them.

4. Import on the destination (preserve token + config)

With the migrated DB already at /data/<host>.db, run add-instance with --admin-token so it imports the existing token instead of minting a new one (a fresh token would rewrite admin_token_hash on boot and orphan every saved admin link). It writes the EnvironmentFile, registers the route, starts the unit against the pre-placed DB, health-checks it, and reloads the router:

lchat add-instance <host> \
  --admin-token "<existing token>" \
  --push-relay-url https://push.example.com --push-relay-token "<existing relay token>"

Pass whatever per-instance config the source had (--push-relay-url/-token, --home-origin, --app-store-url). Verify before any DNS change, over the instance's unix socket: GET / → 200, an admin-token request → 200, and the row counts match the source.

5. Gap check

Confirm nothing was written to the source between the snapshot and the cutover (compare a row count, e.g. messages, source-now vs snapshot). For a busy instance, freeze writes or take the snapshot at cutover time — writes that land on the source after the snapshot are not on the destination.

6. Cut over DNS

Flip the host's A record to the destination's IP. Lower the record's TTL ahead of time (to ~60s, ideally a TTL-worth before the cutover) so the flip propagates fast; otherwise resolvers that cached the old IP keep hitting the old box until the old TTL expires (harmless while the old box is still up as a fallback).

The TLS cert can't be pre-issued: autocert's HTTP-01 validates by resolving the host over public DNS, so the cert only issues once DNS points at the destination. Authoritative DNS updates in seconds (only resolver caches lag), so the cert issues on the first HTTPS hit right after the flip.

7. Verify + keep the fallback

GET https://<host>/ → 200 with a valid (Let's Encrypt) cert, serving the migrated data. Leave the source instance running until you're confident — the migration is a copy, so flipping DNS back is an instant rollback. Retire the source only afterward.

Bots (sidecars)

The instance's bot agent-users migrate with the DB (they're rows), but the sidecar — the Node process and its session files (the bots' identities) — is separate and does not come across in the DB copy. Do not re-run add-bot on the destination: that mints a new set of agent identities (duplicate bots in the Helpers catalog). Instead either copy the sidecar's session files to the destination and start lchat-bot@<host> there, or point the existing sidecar at the destination (it follows the host over DNS and reconnects with its persisted token). Plan the bot move as its own step.

What makes this safe

• Reversible: the source is copied, never moved; DNS flip is reversible; the cert issues automatically post-flip.
• No key churn: same origin ⇒ per-origin browser keys/passkeys untouched.
• Consistent snapshot: VACUUM INTO captures a clean point-in-time DB; the sibling <db>-blobs/ dir (immutable, content-addressed files) copies alongside.