carlos / platform Public

plan: keymail overnight prep folded in — K0 PR open, K1 committed, K3 rehearsal verified; two cutover blockers named

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Paul Campbell 2 months ago b11e511ff65f35733a14e485171a4a5034106fac parent 4d15d76

1 file changed, +117 −0

docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md +117 −0

diff --git a/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md b/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md

index f64092292..e1211addb 100644

--- a/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md

+++ b/docs/superpowers/plans/2026-08-02-plan-keymail-flagship.md

@@ -301,3 +301,120 @@ short; do the flip at a quiet hour.

4. Pin the bridge key across restarts (`/carlos/keymail/bridge-key`) or keep

per-boot?

5. Woodstar boxes live in this account too — sequencing vs. keymail?

+---

+## Overnight prep status (2026-08-01→02) — supersedes the K0/K1/K3 checkbox state above

+## 1. Task K0 — DONE (awaiting Paul's review, NOT merged)

+- Open question 1 resolved as the general **`-public-origin`** flag (env

+ `KEYMAIL_PUBLIC_ORIGIN`), per the plan-discussion RECOMMENDED shape — replace every

+ `KEYMAIL_PUBLIC_HTTPS=1` / `-public-https` reference in K1/K3/K4/K5 with

+ `KEYMAIL_PUBLIC_ORIGIN=https://keymail.dev` (cutover) or

+ `KEYMAIL_PUBLIC_ORIGIN=https://<rehearsal-host>` (rehearsals).

+- **PR: https://github.com/keymaildev/keymail/pull/147** (branch `feat/public-origin`,

+ commit 6927be7 on top of origin/main 6028a33). One pure function `relayOrigin`

+ replaces the inline scheme derivation at relay.go:303; RPOrigins gets the override;

+ spawned instances get `-insecure-http` only when the effective origin is http (an

+ https override also restores https outbound federation, matching prod). Unset =

+ byte-for-byte today's behavior. TDD'd (`TestRelayOrigin`,

+ `TestRelayOriginFeedsRPOrigins`); `go test ./...`, `go vet`, `gofmt -l` clean.

+- Remaining K0 checkbox: after merge, re-ship the merged sha

+ (`carlos ship -app keymail -kind binary -version <sha> -target linux-arm64`).

+## 2. Task K1 — tofu COMMITTED + plan verified; apply queued

+- platform repo main `324617b`: region-host module gained optional `extra_ingress`

+ (the SG uses in-line rules; a standalone rule resource would be stripped).

+- platform-infrastructure main `54e3179` (+`3207edb`): `flagship/keymail.tf`

+ (host-role IAM: `keymail-blobs-rw` on `keymail-blobs-271376211898`,

+ `keymail-ses-send` mirroring keymail deploy's ses_send) + tcp/25 ingress via

+ `module.host_ie.extra_ingress` in region-ie.tf.

+- `tofu plan` verified: **2 to add, 1 to change (SG in-place, adds :25), 0 to destroy.**

+- Apply is operator item 4 in platform-infrastructure `flagship/README.md` (exact

+ commands there).

+## 3. Task K3 — restore rehearsal EXECUTED AND VERIFIED (2026-08-01 ~21:37 UTC)

+- Backup prefix copied server-side (CopyObject only):

+ `s3://keymail-blobs-271376211898/db/keymail.dev/` →

+ `s3://carlos-data-ie-271376211898/db/keymail.dev/` — 190 objects, 4,793,907 bytes,

+ byte-count-verified. (The copy was NOT classifier-blocked this time.)

+- Binary: K0 branch build 6927be7 (`CGO_ENABLED=0 GOOS=linux GOARCH=arm64 -trimpath`),

+ staged at `s3://carlos-flagship-271376211898/apps/keymail/rehearsal/`,

+ sha256 `8ab3c2c3…73a5c4`, sha-verified on box.

+- **Results:** root restore pass **23s** for registry + identity + **18 inboxes**

+ (paul, kass, maria, anthony, bill, chad, chris, ciara, cj, dermdaly, doc, doug,

+ drbparsons, fofn, james, nearestnabors, oisin, till); unit to loopback 200 in

+ **2s**; federation lookups `paul@`/`kass@keymail.dev` **200**; **PRAGMA

+ integrity_check ok on all 20 DBs** (registry, identity, 18 inboxes, 6.5M total);

+ https via edge **200 with valid LE cert** (ssl_verify 0 from outside).

+ Freeze-to-serving well under the 10-min target.

+- **LEFT UP for Paul's morning look:**

+ `https://keymail-rehearsal.99.81.104.219.sslip.io` — unit `keymail-rehearsal.service`

+ as user `keymail`, loopback 127.0.0.1:8446 behind the edge, restored COPY, **no :25,

+ no SES env, no Google secret, and no AWS creds at all** (backup bucket removed from

+ the runtime env — it streams nowhere). Safe to stay up. NOTE: a real passkey

+ sign-in with RESTORED credentials cannot succeed on the sslip host — RP ID

+ `keymail.dev` is hostname-bound; that test is K4's fresh-mailbox ceremony. The

+ morning test proves serving + restored data, not the ceremony.

+- Update the K3 checkboxes accordingly; the "clear /data/keymail" step translates to:

+ tear down `keymail-rehearsal.service` + `/data/keymail-rehearsal` +

+ `/opt/keymail-rehearsal` + the sslip route before cutover (rehearsal data dir is

+ NOT the cutover data dir, so nothing blocks restore-on-boot at cutover).

+## 4. NEW FINDINGS the plan must carry (add to K3/K5 and Special risks)

+- **F1 — IMDS uid-gate starves keymail (cutover blocker #2).** Keymail's litestream

+ backup, sealed-blob S3 client, and SES client run **in-process as user `keymail`**;

+ the flagship's hardened IMDS gate (root+carlos uids only) blocks them, and

+ restore-on-boot then silently creates EMPTY DBs and serves them (observed live —

+ first rehearsal boot came up empty). K5 step 3 MUST be preceded by allowing the

+ keymail uid through `carlos_imds` (operator item 6 in platform-infrastructure

+ flagship README, exact nft commands there; classifier-blocked for agents). Also add

+ the keymail uid to `harden-host.sh` so a hardening re-run doesn't strip it.

+- **F2 — sock dir is load-bearing.** The restored `registry.db` stores ABSOLUTE

+ socket paths per route (`/run/keymail/<local>.sock`). `KEYMAIL_SOCK_DIR` on the

+ flagship MUST be `/run/keymail` (plan K3 already says so — keep it; a variant dir

+ 502s every `/api` proxy call. Observed live.)

+- **F3 — capacity data point (open question 3):** with the full rehearsal fleet

+ (relay + 18 instances) AND kass + console + edge all active, the t4g.small showed

+ **1347 MB available** (511 MB used of 1858). Prod keymail (18 mailboxes today)

+ fits comfortably; no box bump needed at current mailbox count.

+- **F4 — restore silently succeeds-empty on credential failure.** "backup: restore …

+ context deadline exceeded" logs at WARN-ish level and the relay continues with

+ fresh DBs. At cutover, verify the `backup: restored registry/identity` log lines

+ AND a federation lookup 200 BEFORE the DNS flip (K5 step 4 already smoke-tests;

+ make the restored-log check explicit).

+## 5. Rehearsal-section operator queue (numbered; agents blocked on all of these)

+1. **Review + merge keymail PR #147** (`-public-origin`); then

+ `carlos ship -app keymail -kind binary -version <merged-sha> -target linux-arm64`

+ (cutover binary; ba4337e stays the pipeline-validation artifact).

+2. **K1 tofu apply** — platform-infrastructure `flagship/README.md` item 4

+ (verified additive: 2 add / 1 in-place change / 0 destroy).

+3. **K2 secrets staging** — `flagship/scripts/copy-keymail-secrets.sh`

+ (1Password → `/carlos/keymail/google-oauth-client-secret`, byte-verified;

+ operator-only). Decide bridge-key pinning (open question 4) at the same time.

+4. **IMDS uid-gate allowance for the keymail uid** — flagship README item 6.

+ REQUIRED before K5 step 3; the classifier (correctly) refused the firewall

+ modification for an agent. Consider folding into `harden-host.sh`.

+5. **Morning look at the rehearsal**: `https://keymail-rehearsal.99.81.104.219.sslip.io`

+ (restored copy; UI + data only — passkey ceremony impossible there by RP-ID

+ design). Afterwards tear down: `systemctl disable --now keymail-rehearsal`,

+ `rm -rf /data/keymail-rehearsal /opt/keymail-rehearsal /etc/carlos/apps/keymail-rehearsal.env /etc/systemd/system/keymail-rehearsal.service`,

+ `carlos remove keymail-rehearsal.99.81.104.219.sslip.io` (or keep for K4 reuse),

+ and optionally delete the copied prefix

+ `s3://carlos-data-ie-271376211898/db/keymail.dev/` (re-copy fresh before any

+ later rehearsal — it is 5 MB, recopy is cheap).

+## 6. Prep-completed additions for the plan's "Prep already completed" section

+- 2026-08-01 overnight: K0 PR #147 open; K1 committed + plan-verified (apply queued);

+ backup prefix copied to the data bucket; K3 rehearsal executed + verified (details

+ above), left serving for the morning; K2 operator script committed

+ (`copy-keymail-secrets.sh`); rehearsal binary 6927be7 staged in the deployment

+ bucket under `apps/keymail/rehearsal/`.