amadan
woodstar runbook: D4 EXECUTED — woodstar.app live on the flagship; three catches recorded
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+53
−4
docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md+53 −4
diff --git a/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md b/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md| index 63681bbb4..cfb4fa573 100644 |
| --- a/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md |
| +++ b/docs/superpowers/plans/2026-08-16-woodstar-cutover-runbook.md |
| @@ -299,21 +299,70 @@ dominates: final media pass + 30 s WAL wait). |
| ## D4 — Cutover (**HELD: starts only on Paul's explicit go**; quiet hour; TTLs already 300) |
| -- [ ] **D4.1** old box (SSH): stop the six `woodstar-*` units → one |
| +**EXECUTED 2026-08-17 evening on Paul's "Go ahead with the full |
| +transition." Timeline and catches:** |
| + |
| +- **D4.1 freeze 18:26:15–18:26:52Z** (37 s): six units + media timer |
| + stopped, final media pass, 35 s WAL wait, `litestream.service` |
| + stopped. Losslessness proven by mtimes: every on-disk WAL write |
| + (relay/indexer 10:36:25, apbridge 18:13:05) matched its uploaded |
| + segment to the second; the 18:26:52 `.db` mtimes are litestream's |
| + shutdown checkpoint of those same frames. |
| +- **CATCH 1 — old-bucket read denied:** `carlos-flagship-host` has no |
| + policy on `woodstar-backups-271376211898` (the spec's "host role |
| + already has RW" held only for carlos-data-ie). First restore attempt |
| + aborted cleanly at step one. Fix: session-side |
| + `aws s3 sync` of the frozen `prod/` into |
| + `carlos-data-ie…/woodstar-cutover/` (box-readable), restore |
| + repointed. No IAM change. |
| +- **D4.2 restore run: 10 s, integrity `ok` × 3**, five units up — |
| + relay failed: its `EnvironmentFile=/etc/carlos/apps/woodstar.env` |
| + had been SWEPT (see catch 2). Unit stamps landed for all six routes |
| + (`try-restart` wiring confirmed by the verb's own output). |
| +- **CATCH 2 — env-file sweep:** the route-scoped config-delivery pass |
| + (PR #206) owns `/etc/carlos/apps` and removed the hand-written |
| + `woodstar.env` (dir mtime 18:04). `carlos env set -app |
| + woodstar-relay WOODSTAR_APPLE_APP_ID=…` was set through the door |
| + (kept as the durable record) but did NOT render a file for this |
| + route shape — **follow-up gap to file**: config delivery for |
| + kind:service unit-stamped routes. Interim fix = unit-level |
| + `Environment=` line, which is exactly what the old box ran. |
| +- **CATCH 3 — root-owned WAL sidecars:** with the relay down through |
| + catches 1–2, the litestream restart (root) opened `relay.db` first |
| + and minted root-owned `-wal`/`-shm`; the woodstar user then got |
| + SQLITE_CANTOPEN (14). Fix: re-chown `relay.db*` + restart. **Rule |
| + for any re-run: start ALL app units and see them active BEFORE |
| + restarting carlos-litestream.** |
| +- **D4.3 smoke:** loopback info 200 + AASA + DID; edge smoke via |
| + `woodstar-relay.bdm.oncarlos.com` (feed 200) and the wildcard-cert |
| + catch-all (`paul.woodstar.app --resolve` → 302 + DID) — never a TLS |
| + probe of the two per-host names pre-flip. |
| +- **D4.4 flip 21:05Z:** both A records UPSERTed → 99.81.104.219; |
| + INSYNC in ~10 s; per-host certs minted on first hit |
| + (`CN=woodstar.app` exp 2026-11-15). **Full public battery green:** |
| + feed/AASA/media/search 200, webfinger `activity+json`, WS 101, |
| + DID `did:plc:f53efijrjuwujt7ppnp5gzf3` identical, www 301 |
| + URI-preserved, status page serving. |
| +- Old box: units stopped, disk intact, frozen 30 days (spec §7 |
| + rollback posture). Rollback script staged at |
| + `…/ops/woodstar/d4-rollback.sh` (flagship half) + two A records |
| + back — unused. |
| + |
| +- [x] **D4.1** old box (SSH): stop the six `woodstar-*` units → one |
| final media-backup pass → wait ~30 s for litestream's final WAL |
| frames → stop litestream → copy `apbridge.seed` off-box → verify |
| newest backup timestamps are post-stop. |
| -- [ ] **D4.2** flagship: litestream restore the three DBs from the OLD |
| +- [x] **D4.2** flagship: litestream restore the three DBs from the OLD |
| bucket's live prefixes + media sync + seed file into |
| `/data/woodstar`; `chown -R woodstar`; `PRAGMA integrity_check`; |
| start the six units; enable flagship litestream config + media timer |
| against the **fresh `carlos-data-ie` prefixes** (order: stop-old → |
| restore-new → start-new; the two stacks never share prefixes). |
| -- [ ] **D4.3** smoke BEFORE DNS: loopback `/info` 200; AASA correct at |
| +- [x] **D4.3** smoke BEFORE DNS: loopback `/info` 200; AASA correct at |
| `/.well-known/apple-app-site-association`; webfinger answers from |
| the bridge; `curl --resolve woodstar.app:443:99.81.104.219` through |
| the edge. |
| -- [ ] **D4.4** flip Route53 zone `Z07702842ZC4MUKZR07JB`: |
| +- [x] **D4.4** flip Route53 zone `Z07702842ZC4MUKZR07JB`: |
| `woodstar.app` A and `*.woodstar.app` A → `99.81.104.219`. Watch |
| per-host certs for the two named hosts mint on first post-flip hit. |
| - [ ] **D4.5 — Paul's browser pass:** passkey sign-in, post, image |