amadan
woodstar: Plan D cutover runbook — delegation live in the zone, D0-D5 phases, flip held
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+225
−0
docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md+225 −0
diff --git a/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md b/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md| new file mode 100644 |
| index 000000000..88478728a |
| --- /dev/null |
| +++ b/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md |
| @@ -0,0 +1,225 @@ |
| +# Woodstar → Carloku Cutover Runbook (Plan D) |
| + |
| +> **For agentic workers:** this is an OPERATIONAL runbook, not a code |
| +> plan. Execute phases in order in a session with SSM (profile |
| +> `keymail`) and SSH to the old box. Phases D0–D3 are safe any time; |
| +> **D4 (cutover) is HELD for Paul's explicit go** — never start it on |
| +> your own authority. Checkbox syntax tracks progress. |
| + |
| +**Goal:** `woodstar.app`, `status.woodstar.app`, and working |
| +`*.woodstar.app` TLS + routing serve from the Carloku flagship box, |
| +per spec `docs/superpowers/specs/2026-08-12-woodstar-carloku-migration-design.md` |
| +(§7 is the contract this runbook executes). |
| + |
| +## Fixed values |
| + |
| +| Thing | Value | |
| +|---|---| |
| +| Flagship box | `i-092c0c1eea75723cb` / `99.81.104.219` (aws profile `keymail`) | |
| +| Old prod box | `ec2-user@52.30.232.124` (SSH only, no SSM; Route53-confirmed live 2026-08-16 — the Caddyfile's `34.241.196.33` is stale) | |
| +| Devbox (NOT migrating) | `dev.woodstar.app` = `34.253.20.182` | |
| +| woodstar.app zone | Route53 `Z07702842ZC4MUKZR07JB`, **keymail account** | |
| +| Delegation zone | `acme.oncarlos.com` (oncarlos.com zone `Z04313755ASMHAWD536L`, Route53, keymail account — edge publishes TXT here via `CARLOS_DNS_PROVIDER=route53` suffix-walk) | |
| +| Delegation CNAME | `_acme-challenge.woodstar.app` → `_acme-challenge.woodstar.app.acme.oncarlos.com` — **DONE 2026-08-16** (stale Caddy TXT leftovers deleted in the same change batch) | |
| +| Old-box cert headroom | apex + wildcard both expire **2026-10-21**; old-box renewal is blocked by the delegation CNAME from its ~Sept 21 renewal window — cutover must land before then, rollback keeps existing certs | |
| +| W0 build sha | woodstar `origin/main` (`0bdcb07` at runbook time; re-read at D2 — always build ALL SIX binaries from ONE sha, `GOOS=linux GOARCH=arm64`, from a CLEAN checkout: `git status --porcelain` empty, never the stale local tree) | |
| +| Ports (loopback only) | relay 8451, media 8452, indexer 8453, notify 8454, unfurl 8455, apbridge 8456 | |
| +| Data | `/data/woodstar` 0700 owner `woodstar` (`/data` stays 0711) | |
| +| App env file | `/etc/carlos/apps/woodstar.env` 0600 root: `WOODSTAR_APPLE_APP_ID=J5NDF5PCD8.app.woodstar.ios` — in place BEFORE the flip or iOS passkeys break | |
| +| Backup prefixes | `s3://carlos-data-ie-271376211898/woodstar/{relay,indexer,apbridge,media}` (host role already RW); old bucket `woodstar-backups-271376211898` frozen as pre-migration history | |
| +| Replicated DBs | relay.db, indexer.db, apbridge.db — **unfurl.db deliberately excluded** (cache, repo rule); `apbridge.seed` = federation identity flat FILE, copied not litestreamed | |
| +| Apps (7) | `woodstar-relay/-media/-indexer/-notify/-unfurl/-apbridge` (kind binary, linux-arm64) + `woodstar-status` (kind static) | |
| + |
| +## Standing constraints (spec §7, restated) |
| + |
| +- No DNS A-record changes and no touching running woodstar services |
| + until D4; **the flip is Paul's explicit go**. |
| +- Rehearsal reads COPIES only — never point litestream at the live |
| + `prod/` prefixes (lineage fork); never run old-box and flagship |
| + stacks against the same replication prefixes at once. |
| +- No `carlos bootstrap` on the live flagship. |
| +- **Founder trap:** never start units against an empty `/data/woodstar` |
| + reachable on the real hostname — first registrant becomes founder. |
| + Restore is explicit; there is no restore-on-boot. |
| +- Public hostnames attach via the product surface (#82/#151 doors) — |
| + no `carlos add` over SSM for them. |
| + |
| +--- |
| + |
| +## D0 — Platform preconditions (no prod impact beyond two unit restarts) |
| + |
| +State at runbook time: track live (box binary `c42ec0c`, console |
| +`3f28e9f`, both contain Plans B+C); delegation CNAME done. One item |
| +open: |
| + |
| +- [ ] **D0.1 — set the delegation zone on the flagship** (this exact |
| + command was permission-blocked for the authoring session; run it or |
| + approve its retry): |
| + |
| +```bash |
| +cd ~/github.com/carlosframework/platform-infrastructure && \ |
| +aws --profile keymail ssm send-command --instance-ids i-092c0c1eea75723cb \ |
| + --document-name AWS-RunShellScript --parameters "$(jq -n '{commands: [ |
| + "grep -q CARLOS_ACME_DELEGATION_ZONE /etc/carlos/host.env || echo CARLOS_ACME_DELEGATION_ZONE=acme.oncarlos.com >> /etc/carlos/host.env", |
| + "systemctl restart carlos-edge.service console.service", |
| + "sleep 2; systemctl is-active carlos-edge.service console.service" |
| + ]}')" |
| +``` |
| + |
| + Both `console.service` and `carlos-edge.service` source |
| + `/etc/carlos/host.env` (verified 2026-08-16), so the one line covers |
| + door and issuance. Inert until a claim opts in. |
| + |
| +- [ ] **D0.2 — verify:** `curl -sI https://console.carloku.com/ | grep |
| + -i x-carlos-version` still answers (post-restart); box journal shows |
| + NO `CARLOS_ACME_DELEGATION_ZONE … invalid` line from either unit. |
| + |
| +## D1 — Old-box pre-flight (SSH, read-only, any time) |
| + |
| +All over `ssh ec2-user@52.30.232.124`. Output feeds D2's unit files — |
| +record it in the session and in this file's margin notes. |
| + |
| +- [ ] **D1.1 — units:** `systemctl cat woodstar-*.service` (expect |
| + six). Capture EVERY ExecStart flag and Environment line verbatim — |
| + especially apbridge's `-domain`/`-seed`/`-bridgy`/`-bsky-api` and |
| + any relay `-passkey-rp`/`-media`/`-imports` values. Six units are |
| + the target; a seventh unit or a flag the spec never mentioned is a |
| + FINDING — stop and record it before D2. |
| +- [ ] **D1.2 — sizes:** per-service RSS (`systemctl status`), total |
| + `du -sh /opt/woodstar/data` — re-confirm the t4g.nano-scale verdict. |
| +- [ ] **D1.3 — data dir:** `ls -la /opt/woodstar/data` — expect |
| + relay.db, indexer.db, apbridge.db, apbridge.seed, unfurl.db, media/. |
| + Anything else is a finding. |
| +- [ ] **D1.4 — Caddyfile actuals vs repo reference:** diff the box's |
| + live Caddyfile against `infra/prod-Caddyfile`; specifically whether |
| + `/unfurl*` and `/img*` handles exist live (repo copy lacks both — |
| + §0 said the repo copy is stale). |
| +- [ ] **D1.5 — litestream actuals:** `/etc/litestream.yml` — which DBs |
| + replicate, to which prefixes; **does `apbridge.seed` have ANY backup |
| + today?** If not: copy it to |
| + `s3://woodstar-backups-271376211898/apbridge/apbridge.seed` NOW as |
| + its own pre-flight fix (it is the federation identity), before |
| + anything else proceeds. |
| +- [ ] **D1.6 — backup freshness:** newest object timestamps under the |
| + old bucket's prefixes; note the media backup mechanism (script/timer) |
| + and cadence. |
| + |
| +## D2 — Flagship tenant build-out (pre-flip, real hostnames NOT routed yet) |
| + |
| +- [ ] **D2.1 — account decision (Paul, one line):** which console |
| + account owns the seven apps. Recommendation: a dedicated `woodstar` |
| + account (own statements, own tokens), same pattern as tito=bae on |
| + the tito console. Needed before any `carlos apps create`. |
| +- [ ] **D2.2 — apps:** `carlos apps create` × 7 under that account |
| + (six binary + `woodstar-status` static). |
| +- [ ] **D2.3 — build + ship:** from a clean woodstar checkout at |
| + origin/main: six `CGO_ENABLED=0 GOOS=linux GOARCH=arm64 go build` |
| + binaries, `carlos ship` + promote each on its app's canary channel. |
| + Status site: `carlos ship -kind static` of the status content. |
| +- [ ] **D2.4 — box-side (SSM, root):** system user `woodstar` |
| + (sysusers), `/data/woodstar` 0700, `/etc/carlos/apps/woodstar.env` |
| + (0600, the AASA line), six hardened units |
| + `woodstar-{relay,media,indexer,notify,unfurl,apbridge}.service` |
| + patterned on `keymail-relay.service` + `scripts/harden-host.sh` |
| + drop-ins. ExecStart flags = D1.1 verbatim, transposed: paths to |
| + `/data/woodstar`, ports to the 845x map, PLUS relay fan-out flags: |
| + |
| + ``` |
| + -addr 127.0.0.1:8451 -db /data/woodstar/relay.db -name woodstar |
| + -media http://127.0.0.1:8452 -media-upstream http://127.0.0.1:8452 |
| + -indexer-upstream http://127.0.0.1:8453 -notify-upstream http://127.0.0.1:8454 |
| + -unfurl-upstream http://127.0.0.1:8455 -apbridge-upstream http://127.0.0.1:8456 |
| + -domain woodstar.app |
| + ``` |
| + |
| + **`-media` AND `-media-upstream` are BOTH required** (importer vs |
| + fan-out — final-review catch). apbridge keeps |
| + `-domain woodstar.app -seed /data/woodstar/apbridge.seed` + D1's |
| + `-bridgy`/`-bsky-api` values. Units installed but **NOT started** |
| + (founder trap) — except as D3 rehearses on copies. |
| +- [ ] **D2.5 — routes via product surface:** relay's primary route as |
| + a declared route record (#137 vocabulary) on the flagship; |
| + `status.woodstar.app` attach for `woodstar-status` (first-route |
| + static attach or alias — whichever the live #82 door supports for |
| + the shape). **`status.woodstar.app`'s exact-host route MUST exist |
| + before the wildcard alias serves** — with the alias live and the |
| + status route missing, status 302s into the SPA and the status page |
| + silently becomes the app. Then attach `woodstar.app` as alias to the |
| + relay's route. |
| +- [ ] **D2.6 — wildcard, pre-flip cert proof:** `carlos domains attach |
| + woodstar.app -wildcard -catchall` (door refuses until D0.1 is done — |
| + that refusal text is itself a check). Watch the edge obtain |
| + `*.woodstar.app` via the delegation CNAME; then handshake-verify |
| + against the flagship IP without DNS: |
| + `curl -sv --resolve paul.woodstar.app:443:99.81.104.219 |
| + https://paul.woodstar.app/ -o /dev/null 2>&1 | grep -E |
| + "subject:|expire"` → subject `CN=*.woodstar.app`. Issuance does not |
| + depend on the A records: this is the whole point of §4. |
| + |
| +## D3 — Rehearsal (copies only, tear down after) |
| + |
| +- [ ] **D3.1** server-side CopyObject of the old bucket's backups into |
| + `s3://carlos-data-ie-271376211898/woodstar-rehearsal/`. |
| +- [ ] **D3.2** on the flagship: `litestream restore` the three DBs from |
| + the rehearsal prefix + media sync + seed-file copy into |
| + `/data/woodstar-rehearsal`; `PRAGMA integrity_check` on all three. |
| +- [ ] **D3.3** transient units (systemd-run, rehearsal ports 8461-8466, |
| + same flags re-pathed); apbridge with `-bridgy` and `-bsky-api` |
| + pointed at throwaway values so rehearsal never dials the real |
| + federation; an sslip.io rehearsal route through the edge. |
| +- [ ] **D3.4 — the fan-out path check through the edge** (degraded- |
| + not-broken is the named failure shape; never skip): feed renders |
| + restored posts; `/media/*` images load; `/img/*` unfurl card images |
| + load (the leg the spec almost lost); `/api/search` returns; |
| + `/stream/*` SSE streams; `/subscribe` WS connects (SSE/WS through |
| + TWO proxied hops edge→relay→notify — verified, not assumed); |
| + `/unfurl` answers; webfinger + `/users/*` answer from the bridge; |
| + woodstar's own WebKit `console-check.mjs` passes. |
| +- [ ] **D3.5** verify exact-host-beats-wildcard precedence with the |
| + rehearsal routes; record freeze-to-serving time (the cutover window |
| + estimate); tear down units, data, routes. |
| + |
| +## D4 — Cutover (**HELD: starts only on Paul's explicit go**; quiet hour; TTLs already 300) |
| + |
| +- [ ] **D4.1** old box (SSH): stop the six `woodstar-*` units → one |
| + final media-backup pass → wait ~30 s for litestream's final WAL |
| + frames → stop litestream → copy `apbridge.seed` off-box → verify |
| + newest backup timestamps are post-stop. |
| +- [ ] **D4.2** flagship: litestream restore the three DBs from the OLD |
| + bucket's live prefixes + media sync + seed file into |
| + `/data/woodstar`; `chown -R woodstar`; `PRAGMA integrity_check`; |
| + start the six units; enable flagship litestream config + media timer |
| + against the **fresh `carlos-data-ie` prefixes** (order: stop-old → |
| + restore-new → start-new; the two stacks never share prefixes). |
| +- [ ] **D4.3** smoke BEFORE DNS: loopback `/info` 200; AASA correct at |
| + `/.well-known/apple-app-site-association`; webfinger answers from |
| + the bridge; `curl --resolve woodstar.app:443:99.81.104.219` through |
| + the edge. |
| +- [ ] **D4.4** flip Route53 zone `Z07702842ZC4MUKZR07JB`: |
| + `woodstar.app` A and `*.woodstar.app` A → `99.81.104.219`. Watch |
| + per-host certs for the two named hosts mint on first post-flip hit. |
| +- [ ] **D4.5 — Paul's browser pass:** passkey sign-in, post, image |
| + upload + display, search, live notification, DM, link-card unfurl, |
| + iOS AASA validation, status page, a `paul.woodstar.app` profile |
| + redirect + its `/.well-known/atproto-did`, and federation (webfinger |
| + from a real Fediverse server / Bridgy Fed still resolving). |
| + |
| +## Rollback (any point before Paul blesses D4.5) |
| + |
| +Stop flagship units + its litestream; restart old-box units + |
| +litestream; flip the two A records back to `52.30.232.124`. Writes |
| +accepted by the flagship during the window live only in its lineage — |
| +keep the window short. Old box afterwards: units stopped, disk intact, |
| +frozen 30 days. (Old-box certs stay valid to 2026-10-21 regardless of |
| +the delegation CNAME.) |
| + |
| +## D5 — Post-cutover (spec §9) |
| + |
| +- [ ] Promote the W0 sha to stable after the 72 h supply-chain bake. |
| +- [ ] Repoint status publishing: `carlos ship -kind static` + promote |
| + replaces `scripts/status-publish.sh`. |
| +- [ ] Note woodstar `infra/` tofu drift; retire the old prod instance |
| + after the 30-day freeze; zone, bucket, devbox, IAM stay. |
| +- [ ] Update woodstar's AGENTS.md "The boxes" section and the |
| + `infra/prod-Caddyfile` reference-copy header (describes a box that |
| + no longer serves). |