amadan
plan: keymail → flagship migration — recon, CARLOS fit, rehearsal + cutover
Recon 2026-08-01: keymail runs in the flagship account already (prod i-088560dc704eb6dee / keymail.dev); embedded litestream restore-on-boot is the migration mechanism; multi-process relay+instances needs a static user; one required code change (public-https origin behind the edge). ba4337e shipped to the deployment bucket, not promoted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+303
−0
docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md+303 −0
diff --git a/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md b/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md| new file mode 100644 |
| index 000000000..f64092292 |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md |
| @@ -0,0 +1,303 @@ |
| +# 🤖 Plan: Keymail → CARLOS flagship migration |
| + |
| +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
| + |
| +> 🤖 Written by an LLM (Claude), per factor X. |
| + |
| +**Goal:** Keymail's production instance (`keymail.dev`) serving from the CARLOS |
| +flagship box (`i-092c0c1eea75723cb` / 99.81.104.219), riding carlos release |
| +channels, with zero change to its public identity — because Keymail is the |
| +**identity provider for the platform console** and its issuer origin |
| +(`https://keymail.dev`) and passkey RP ID (`keymail.dev`) must survive the |
| +move byte-identical. |
| + |
| +**Playbook:** mirror the Kass migration (plan B + session lessons: ship |
| +`-target linux-arm64`, route-before-adopt, per-host live link `live/<host>`, |
| +`/data` 0711, rehearse on a copy, DNS flip HELD for Paul's explicit go). |
| + |
| +## Where Keymail lives today (recon 2026-08-01) |
| + |
| +- Repo: `/home/paulca/github.com/keymaildev/keymail` (origin/main `ba4337e`). |
| + Pure Go, `CGO_ENABLED=0`, **one static binary** (`go build -trimpath |
| + -ldflags="-s -w"`, per `deploy/deploy.sh`). |
| +- Runs in the **same AWS account as the flagship** (271376211898, eu-west-1): |
| + - prod: `i-088560dc704eb6dee` (t4g.nano, EIP 34.250.110.189) = `keymail.dev` |
| + - dev: `i-026386726a440d140` (t4g.small, EIP 54.247.184.202) = `dev.keymail.dev` |
| + - Stack: `keymail/deploy/` OpenTofu, state |
| + `s3://keymail-tfstate-271376211898/deploy/terraform.tfstate`; artifact |
| + bucket `keymail-deploy-271376211898`. |
| +- Serving shape (from `deploy/user_data.sh.tftpl`): one systemd unit |
| + `keymail-relay.service`, `User=keymail` (static system user), |
| + `ExecStart=keymail relay -tls -host keymail.dev -rp-id keymail.dev -data |
| + /opt/keymail/data -sock-dir /run/keymail -pow 20`. The relay terminates its |
| + own TLS via autocert on :443/:80 today, serves the **status site on |
| + `status.keymail.dev`** (Host-routed in-process), runs the **inbound MX on |
| + :25** (`KEYMAIL_MX_ADDR=:25`), and **spawns one `keymail instance` child |
| + process per mailbox** (provision.go), proxying `/api/*` to them over unix |
| + sockets in `/run/keymail`. |
| +- Data: SQLite — relay owns `registry.db` + `identity.db` (passkey identity + |
| + wrapped key vault), each instance owns its inbox DB, under |
| + `/opt/keymail/data`. **Litestream is embedded in the binary** (backup.go): |
| + every process streams its own WAL to |
| + `s3://keymail-blobs-271376211898/db/<host>/<name>/…` and **restores on boot |
| + when the local file is absent** — a lost box is a reconnect. No sidecar, no |
| + litestream.yml. |
| +- Blobs: sealed attachment ciphertext in `s3://keymail-blobs-271376211898` |
| + (`KEYMAIL_BLOB_BUCKET`), hand-rolled SigV4 client + instance-role creds. |
| +- Mail: SES in this account (identities `keymail.dev` and `dev.keymail.dev` |
| + are **already verified here** — no SES move at all); per-mailbox SES tenants |
| + (`KEYMAIL_SES_IDENTITY_ARN`); outbound via instance role, not SMTP creds. |
| +- Secrets actually on the box (all via systemd env, none in files we track): |
| + - `KEYMAIL_GOOGLE_CLIENT_SECRET` (Gmail offload import; source of truth |
| + 1Password `op://Keymail/Google/oauth client secret`, stamped by deploy). |
| + - Everything else is per-boot in-memory (challengeKey, bridgeKey) or |
| + IAM-role-derived. **Keymail the OAuth provider holds no long-lived signing |
| + key**: the *client* (console) pins its ECDSA P-256 `sign_pub` into the |
| + grant; keymail stores only grant rows + token hashes in the user's inbox |
| + DB, so grants and tokens migrate with the data and console tokens stay |
| + valid across the move. |
| + |
| +## CARLOS fit assessment |
| + |
| +- **Multi-process app family** (relay + N `keymail instance` children) → |
| + **static `keymail` system user**, exactly like today. `carlos-app@` |
| + (DynamicUser) is per-unit and cannot cover relay-spawned children — the |
| + template's own documented caveat. Unit: a bespoke `keymail-relay.service` |
| + with the harden-host sandbox drop-ins, `RuntimeDirectory=keymail`, |
| + `StateDirectory`-equivalent under `/data/keymail`. |
| +- **Edge fit:** relay supports plain HTTP behind a proxy (`-addr` / |
| + `KEYMAIL_RELAY_ADDR`, default :8443; `-tls` off). Whole-app `routed` tenant, |
| + TCP upstream on loopback — same shape as Kass. Routes needed: |
| + `keymail.dev` **and** `status.keymail.dev` → same upstream (relay Host-routes |
| + the status site itself; the edge must preserve Host). |
| +- **Listener conflicts:** none on :80/:443 (edge owns them; relay goes |
| + loopback-only). **:25 is real**: the inbound MX must keep binding :25 on the |
| + flagship box directly (the edge doesn't speak SMTP) — |
| + `AmbientCapabilities=CAP_NET_BIND_SERVICE` + a new SG ingress rule. |
| +- **âš One required code change (K0):** relay derives its public origin scheme |
| + from `-tls` (relay.go: `scheme := "https"` only when `-tls`). Behind the |
| + edge (`-tls` off) WebAuthn `RPOrigins` becomes `http://keymail.dev` and |
| + **every passkey sign-in fails** (clientDataJSON origin is |
| + `https://keymail.dev`). Keymail needs a small flag/env (e.g. |
| + `KEYMAIL_PUBLIC_HTTPS=1` or `-public-origin`) so a TLS-terminating proxy |
| + upstream still yields an https origin. Done in a keymail worktree with |
| + tests, per keymail's CLAUDE.md. |
| +- **IAM/infra deltas on the flagship host role** (module currently scopes to |
| + the two carlos buckets + `/carlos/*` SSM only): add `keymail-blobs-…` bucket |
| + RW and the SES send + tenant actions (copy the shape of keymail deploy's |
| + `aws_iam_role_policy.ses_send`). |
| +- **Identity-provider constraints:** issuer origin `https://keymail.dev` and |
| + RP ID `keymail.dev` are load-bearing (console `CONSOLE_KEYMAIL` defaults to |
| + `https://keymail.dev`; client_id is the console's origin). Hostname must not |
| + change, scheme must stay https, and the console must be able to reach it |
| + from the same box (loopback-hairpin via the edge is fine post-cutover). |
| +- **Capacity note:** prod is a t4g.nano today; the flagship t4g.small already |
| + hosts kass + console + edge. The dev relay once OOMed a nano re-spawning its |
| + instance fleet — check prod mailbox count before cutover and watch memory. |
| + |
| +## Global constraints |
| + |
| +- NO DNS changes, no touching the running keymail service, no production data |
| + writes, until the explicitly-gated steps say so — and the cutover flip is |
| + **HELD for Paul's explicit go**. |
| +- Secret VALUES never appear in commands, files, or transcripts — SSM names |
| + only; copies are operator-run scripts (classifier blocks release-key reads |
| + and prod-data sync for agents). |
| +- Keymail's existing `deploy/` stack stays untouched until post-cutover |
| + retirement (it will drift once we stop its service — same as Kass, expected). |
| +- Rehearsals must never write to the **live** backup prefix |
| + `s3://keymail-blobs-271376211898/db/keymail.dev/` — a rehearsal relay |
| + pointed at the live bucket would stream its own WAL over production's |
| + lineage. Rehearsals always run against a **copied** prefix in |
| + `carlos-data-ie-271376211898`. |
| + |
| +--- |
| + |
| +### Task K0: keymail code change — public-https origin behind a proxy |
| + |
| +- [ ] In a keymail worktree: add `-public-https` (env `KEYMAIL_PUBLIC_HTTPS=1`) |
| + to `keymail relay`: with it set and `-tls` off, the identity origin is |
| + `https://<host>` (WebAuthn RPOrigins) and anything else scheme-derived |
| + follows. Tests: identity origin assertion with the flag on/off. |
| +- [ ] `go test ./...`, `go vet`, `gofmt -l` clean; PR into keymail main. |
| +- [ ] Re-ship the merged sha to the flagship deployment bucket |
| + (`carlos ship -app keymail -kind binary -version <sha> -target linux-arm64`). |
| + (ba4337e is already shipped for pipeline validation; the cutover binary |
| + is the K0 sha.) |
| + |
| +### Task K1: flagship stack prep (tofu — no DNS) |
| + |
| +- [ ] `platform-infrastructure/flagship`: new `keymail.tf` — |
| + SG ingress tcp/25 (world) on the host SG; IAM policy on |
| + `carlos-flagship-host`: RW on `arn:aws:s3:::keymail-blobs-271376211898` |
| + (+`/*`), and `ses:SendEmail`, `ses:SendRawEmail`, |
| + `ses:CreateTenant`/tenant-association actions matching keymail deploy's |
| + `ses_send` policy. Nothing DNS, nothing touching keymail's own stack. |
| +- [ ] `tofu plan` — verify only additive SG/IAM; apply (operator gate as |
| + established). |
| + |
| +### Task K2: stage secrets into SSM (names only; operator script) |
| + |
| +- [ ] `scripts/copy-keymail-secrets.sh` (mirrors `copy-kass-secrets.sh`): |
| + reads 1Password `op://Keymail/Google/oauth client secret` → SSM |
| + **`/carlos/keymail/google-oauth-client-secret`** (SecureString), |
| + byte-verify readback. That is keymail's only runtime secret; SES and |
| + blobs ride the instance role. Optional: **`/carlos/keymail/bridge-key`** |
| + if we want the MX bridge key pinned across restarts (today per-boot). |
| + |
| +### Task K3: box-side restore rehearsal (no prod impact, loopback only) |
| + |
| +- [ ] Copy the live backup prefix (read-only on source, operator script): |
| + `aws s3 sync s3://keymail-blobs-271376211898/db/keymail.dev/ |
| + s3://carlos-data-ie-271376211898/db/keymail.dev/` — the rehearsal |
| + relay's `KEYMAIL_BACKUP_BUCKET=carlos-data-ie-271376211898` then |
| + restores from and streams to the **copy** (prefix `db/<host>` is fixed |
| + in code; the bucket is the isolation boundary). |
| +- [ ] On the flagship box (SSM, staged repo script `rehearse-keymail.sh`): |
| + static user `keymail`; `/data/keymail` (0700, /data 0711); |
| + `/etc/carlos/apps/keymail.env` (0600 root): `KEYMAIL_RELAY_ADDR=127.0.0.1:8446`, |
| + `KEYMAIL_HOST=keymail.dev`, `KEYMAIL_RP_ID=keymail.dev`, |
| + `KEYMAIL_DATA=/data/keymail`, `KEYMAIL_SOCK_DIR=/run/keymail`, |
| + `KEYMAIL_BACKUP_BUCKET=carlos-data-ie-271376211898`, |
| + `KEYMAIL_BLOB_BUCKET=keymail-blobs-271376211898` (blob reads are safe; |
| + blob writes are content-addressed ciphertext — acceptable, or omit for a |
| + stricter rehearsal), `KEYMAIL_PUBLIC_HTTPS=1`. **No `KEYMAIL_SES_FROM`** |
| + (outbound off — a rehearsal must not email real people), **no |
| + `KEYMAIL_MX_ADDR`** (no :25), no Google secret needed. |
| +- [ ] `keymail-relay.service` (User=keymail, hardening drop-ins per |
| + harden-host.sh, RuntimeDirectory=keymail) → start → verify: unit active, |
| + `registry.db` + `identity.db` restored (log line `backup: restored …`), |
| + instance children spawned, `curl -s 127.0.0.1:8446/.well-known/keymail` |
| + and a federation lookup of `paul@keymail.dev` 200 — restored data, all |
| + on loopback, nothing public. |
| +- [ ] Record freeze-to-serving time (target < 10 min); then stop the |
| + rehearsal unit and clear `/data/keymail` (it must be empty at cutover so |
| + restore-on-boot pulls the fresh live state). |
| + |
| +### Task K4: public rehearsal + FULL console sign-in round-trip (pre-cutover gate) |
| + |
| +Real-passkey testing at `keymail.dev` is impossible before the flip (RP ID |
| +binds to the hostname; sslip.io can't claim it), so the round-trip runs on a |
| +rehearsal hostname with a fresh mailbox — it proves every moving part except |
| +the DNS binding itself: |
| + |
| +- [ ] Fresh empty keymail at `keymail.99.81.104.219.sslip.io`: second env/unit |
| + (`KEYMAIL_HOST=keymail.99.81.104.219.sslip.io`, own data dir + port, |
| + backup bucket = carlos-data-ie, `KEYMAIL_PUBLIC_HTTPS=1`); edge routes |
| + (route-before-adopt): `carlos add -app keymail -kind instance -addr |
| + 127.0.0.1:8447 -channel canary/rehearsal keymail.99.81.104.219.sslip.io` |
| + → `carlos adopt` (live link `live/<host>`); LE cert issues on first hit. |
| +- [ ] Sign up a test mailbox there with a NEW passkey (browser). |
| +- [ ] Point a canary console at it: console canary instance with |
| + `CONSOLE_KEYMAIL=https://keymail.99.81.104.219.sslip.io` — run the full |
| + ceremony: OAuth 2.1 PKCE authorize → approve → token exchange with |
| + `sign_pub` pinned → **sealed confirmation envelope lands in the Apps |
| + inbox** → returning sign-in via the confirmation link. All four legs |
| + green = the migrated provider stack works end-to-end. |
| +- [ ] Tear down the rehearsal route + unit; keep notes on timings. |
| + |
| +### Task K5: cutover (HELD — Paul's explicit go required) |
| + |
| +Pre-flight: K0 sha promoted (canary/rehearsal channel; stable follows its |
| +bake), K1 applied, K2 verified, K3+K4 green, `/data/keymail` empty, |
| +prod mailbox count + box memory headroom checked, TTL on `keymail.dev` A is |
| +already 300. |
| + |
| +Order (single operator session, minutes of downtime): |
| + |
| +1. Routes exist + adopted on flagship: `keymail.dev`, `status.keymail.dev` |
| + → 127.0.0.1:8446 (certs can't issue pre-flip; edge will get them on first |
| + post-flip hit). |
| +2. **Stop** `keymail-relay.service` on old prod `i-088560dc704eb6dee` (SSM). |
| + Wait ~30s for the final WAL frames to land (litestream sync ~1s, monitor |
| + poll 10s); verify the newest LTX timestamps under |
| + `db/keymail.dev/{registry,identity}/` are post-stop. |
| +3. **Start** `keymail-relay.service` on the flagship with |
| + `KEYMAIL_BACKUP_BUCKET=keymail-blobs-271376211898` (the LIVE bucket — the |
| + only moment a flagship relay may point at it), `KEYMAIL_MX_ADDR=:25`, |
| + `KEYMAIL_SES_FROM=keymail.dev` + `KEYMAIL_SES_IDENTITY_ARN`, Google secret |
| + from `/carlos/keymail/google-oauth-client-secret`, `KEYMAIL_PUBLIC_HTTPS=1`. |
| + Restore-on-boot pulls the just-settled state. Never run both relays at |
| + once against the live bucket. |
| +4. Smoke pre-flip: loopback `/.well-known/keymail` + federation lookup 200 on |
| + restored data; `curl --resolve keymail.dev:443:99.81.104.219` through the |
| + edge (cert pending is expected until DNS). |
| +5. **Flip DNSimple** `keymail.dev` A + `status.keymail.dev` A → 99.81.104.219 |
| + (TTL 300). MX follows automatically — the MX record targets `keymail.dev` |
| + itself. Watch the LE cert issue. |
| +6. Verify (Paul, in a browser): real passkey sign-in at `https://keymail.dev`; |
| + **console sign-in round-trip at platform.carlosframework.com against the |
| + real provider** (existing grant should work without re-auth — grant rows |
| + moved with the inbox DB); send a keymail→keymail message; send an inbound |
| + ordinary email from outside and watch the MX accept-and-seal; outbound |
| + bridge send via SES. |
| +7. Old prod box left stopped-but-intact (service down, disk untouched) as the |
| + rollback body; retire after a 30-day freeze like Kass. |
| + |
| +**Rollback** (any failure before/at step 6): stop the flagship relay, restart |
| +`keymail-relay.service` on the old box (its local data is present, so it |
| +resumes without restore), flip the A records back. Caveat: anything accepted |
| +by the flagship relay during the window exists only in the streamed lineage — |
| +the old box resumes from its own disk, so those minutes fork the backup |
| +lineage and their messages are lost to the rolled-back prod. Keep the window |
| +short; do the flip at a quiet hour. |
| + |
| +### Task K6: post-cutover |
| + |
| +- [ ] Promote the K0 sha to keymail's stable channel after the supply-chain |
| + bake; keymail updates now ride carlos channels. |
| +- [ ] Decide `dev.keymail.dev` (recommend: migrate it as a follow-on wave with |
| + the same recipe, or leave it on its box until the deploy stack retires). |
| +- [ ] Note keymail `deploy/` tofu drift (service stopped + DNS moved outside |
| + its state) in its README; plan stack retirement with the old boxes. |
| +- [ ] Consider unix-socket upstream for the relay (edge gap: loopback TCP |
| + cross-talk between tenants) once the edge supports it. |
| + |
| +## Special risks — Keymail is the console's identity provider |
| + |
| +- **Sign-in outage window:** during cutover, console sign-ins and sealed |
| + confirmation sends fail until the new relay serves. Existing console |
| + sessions (cookies) survive; the console itself does not depend on keymail to |
| + keep serving. No circular dependency at rest — but never schedule a console |
| + deploy in the same window. |
| +- **Origin is the contract:** `https://keymail.dev` is simultaneously the |
| + OAuth issuer base (`CONSOLE_KEYMAIL`), the client-visible authorize origin, |
| + and the WebAuthn RP origin. The K0 https-origin fix is the single most |
| + dangerous omission — without it the migrated provider comes up with |
| + `http://` RPOrigins and every passkey ceremony fails **after** DNS has |
| + flipped. |
| +- **Grants/tokens:** live in each user's inbox DB (`oauth` tables — client |
| + origin, scopes, pinned `sign_pub`, token *hashes*). They move with |
| + litestream; the console's stored bearer tokens remain valid. Verify by |
| + round-trip, not assumption (K5.6). |
| +- **Passkeys:** `identity.db` (credentials + wrapped vault) is bound to RP ID |
| + `keymail.dev`. Hostname stability rule: the name never changes, or every |
| + passkey is orphaned. |
| +- **MX gap:** between old-stop and new-serving, inbound SMTP senders retry |
| + (that's SMTP's job) — but the :25 SG rule and `KEYMAIL_MX_ADDR` must be live |
| + on the flagship *before* the flip or mail bounces after retries expire. |
| + |
| +## SSM names (values never in this repo) |
| + |
| +- `/carlos/keymail/google-oauth-client-secret` — Gmail offload import secret. |
| +- `/carlos/keymail/bridge-key` — optional pinned MX bridge key. |
| +- (existing) `/carlos/release-key` — promotion signing; operator-only. |
| + |
| +## Prep already completed (2026-08-01) |
| + |
| +- keymail `ba4337e` (origin/main) built `CGO_ENABLED=0 GOOS=linux GOARCH=arm64 |
| + -trimpath` and **shipped** to `carlos-flagship-271376211898` |
| + (`apps/keymail/releases/ba4337e/linux-arm64` + manifest). NOT promoted — no |
| + channel, no route, no box touched; the cutover binary will be the K0 sha. |
| + |
| +## Open questions for Paul |
| + |
| +1. K0 flag shape: `-public-https` env-only, or a general `-public-origin`? |
| +2. Does `dev.keymail.dev` migrate in the same wave (dev-first would be the |
| + ideal live rehearsal — but it's the fleet every keymail session tests |
| + against, so its downtime has its own cost)? |
| +3. Prod mailbox count / t4g.small headroom with kass + console + edge already |
| + resident — bump the flagship box first? |
| +4. Pin the bridge key across restarts (`/carlos/keymail/bridge-key`) or keep |
| + per-boot? |
| +5. Woodstar boxes live in this account too — sequencing vs. keymail? |