amadan
durability design: builder-check calls (box prefixes, restore-box, escrow before launch, heartbeat, deletions); questions for Paul
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+72
−30
docs/superpowers/specs/2026-10-23-durability-design.md+72 −30
CI cancelled — superseded by a newer push — run details
CI log
=== gate === .amadan/test-gate.sh amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-23-durability-design.md b/docs/superpowers/specs/2026-10-23-durability-design.md| index bd507feb2..d85b1367c 100644 |
| --- a/docs/superpowers/specs/2026-10-23-durability-design.md |
| +++ b/docs/superpowers/specs/2026-10-23-durability-design.md |
| @@ -131,6 +131,11 @@ and covers every database on the box. |
| `welp.bae`. For hibernating instances the activator's embedded |
| replicator runs one Litestream DB per file. Box-wide replication still |
| never touches a hibernating instance's files. |
| +- **Each box replicates under its own prefix**, `boxes/<box-label>/` |
| + plus the full path, in its region's data bucket. Several boxes can share |
| + a data bucket under region HA, and a rebuilt box must never write into |
| + the dead box's lineage. The 0.5.15 cutover (below) starts new lineages |
| + anyway, so this is the one free moment to choose the scheme. My call. |
| - **The generator runs itself.** The agent re-runs the `ops litestream` |
| generation on route changes and every 10 minutes, and restarts the |
| replicator only on "changed". The shrink guard stays. A shrink still |
| @@ -162,8 +167,14 @@ and covers every database on the box. |
| keeps its change-counter freshness check and never treats schema-only |
| success as current. They publish their verdicts to `control/verify/<box>.json`, and the |
| console shows them. |
| -- New alert rules, through the existing alert delivery (SNS email today; |
| - the webhook stays optional): |
| +- **Delivery that doesn't need AWS, and that outlives the console box.** |
| + The webhook URL becomes a sealed box secret (§5.4), so a non-AWS |
| + deployment has a supported path. SNS stays the AWS adapter. Each box's |
| + agent also pings an optional heartbeat URL (`CARLOS_HEARTBEAT_URL`) |
| + every 5 minutes, and the off-account runner (§5.5) checks the |
| + deployment's heartbeats too. Losing a box, including the console's, then |
| + raises an alert from outside it. My call. |
| +- New alert rules: |
| - box-wide Litestream not active, or restarting (slice A of #284 already |
| adds `restarts@<unit>`) |
| - a drill failed, or hasn't run in 36 hours |
| @@ -187,6 +198,11 @@ platform has never said whether those are durable, and they aren't. |
| My recommendation, which Paul needs to confirm (§9, question 1): back them |
| up. |
| +- **Scope:** exec instance directories, unit tenants' state directories, |
| + and directories an addr route declares with a new `--files` flag (for |
| + example `/data/woodstar`). Purpose: disaster recovery. Files following |
| + an instance between boxes stays with moves' existing archive for now. |
| + My call. |
| - An hourly, incremental, content-addressed snapshot of each instance's |
| directory to the bucket, under `instances/<host>/files/`, skipping |
| databases (Litestream has those), `-wal`, `-shm`, `.env` (§5.4) and the |
| @@ -203,13 +219,16 @@ up. |
| ### 5.4 Keys and box settings |
| -- **Instance `.env` escrow.** When `<host>.env` is minted or changed, the |
| - platform seals it to the deployment's secrets public key |
| +- **Instance `.env` escrow.** When `<host>.env` is minted or changed, and |
| + before the app's first launch, the platform seals it to the deployment's secrets public key |
| (`appconfig.SealBound`, bound to account and app) and writes it to |
| `instances/<host>/env.sealed` in the bucket. Restore needs |
| `CARLOS_SECRETS_KEY`, which is in SSM in four regions and in 1Password, |
| - so a rebuilt box can unseal it. Nothing on a box can read another |
| - instance's escrow without that key. |
| + so a rebuilt box can unseal it. A failed escrow write holds the first |
| + launch, which already depends on the bucket for its artifact, so this |
| + adds no new dependency. Before minting, a box checks for an existing |
| + escrow and restores it instead, so a rebuilt box never mints a new key |
| + over sealed data. My call. |
| - **Platform-owned box secrets** (relay masters, `release.env`): the same |
| sealing, at `control/box-secrets/<box>/<name>.sealed`. Bootstrap restores |
| them. |
| @@ -243,11 +262,17 @@ destination in step with the deployment's buckets. |
| - **How:** incremental by listing, comparing ETag and size, then Get from |
| the source and Put to the destination. Server-side copy can't cross |
| providers. Litestream's LTX files are immutable, so most runs move only |
| - new segments. It runs every 15 minutes, on the box holding the |
| - housekeeper lease, so exactly one box does it. |
| + new segments. It runs every 15 minutes. Where it runs is Paul's |
| + call (§9 question 1). A box can't do it without widening roles that are |
| + narrow on purpose: no region box reads another region's data, and only |
| + the console with scanning on reads tenant buckets. |
| +- **Deletions** are copied as delete markers, so a restore matches the |
| + source. Noncurrent versions at the destination expire after the |
| + retention period, which bounds the cost of Litestream's constant small |
| + writes. My call. |
| - **The destination:** Hetzner Object Storage (Paul, 2026-10-23: "we use |
| - our Hetzner account"), versioned, with object lock in governance mode |
| - and a 30-day default retention (my call). The destination credentials |
| + our Hetzner account"), versioned, with object lock (mode and retention: |
| + §9 question 4). The destination credentials |
| can write but not delete locked versions, so a compromised box can't |
| destroy the copy. |
| - **Encryption:** objects are sealed client-side to the deployment's |
| @@ -266,14 +291,18 @@ destination in step with the deployment's buckets. |
| ### 5.6 Rebuilding a box |
| -- **`carlos system restore-registry --from <box-label>`** copies the dead |
| - box's latest registry snapshot into place on a fresh box before the edge |
| - starts, and refuses to start if the label is a live box (the clobber risk |
| - `snapshot_loop.go` notes). It replaces the hand-run `aws s3 cp` step. |
| -- **Bootstrap restores, in order:** `host.env` from the host-env escrow; |
| - sealed box secrets; the registry (when told to, with `--from`); then it |
| - starts the edge. Tenants restore themselves on first wake. Their database |
| - comes from Litestream, their files from §5.3, their `.env` from §5.4. |
| +- **`carlos system restore-box --from <box-label>`** brings a fresh box up |
| + as the dead one. It refuses if the label's heartbeat is current (the |
| + clobber risk `snapshot_loop.go` notes). In order it restores: |
| + 1. `host.env` from the host-env escrow |
| + 2. sealed box secrets |
| + 3. the registry from the dead box's latest snapshot |
| + 4. every box-wide replica under the dead box's prefix: the console, |
| + unit tenants, addr-route services and non-hibernating instances |
| + 5. their files (§5.3) and `.env` escrows (§5.4) |
| + Then it starts the edge, and the box replicates under its own new prefix. |
| + It replaces the hand-run `aws s3 cp`. Hibernating instances restore |
| + themselves on first wake, as today, with files and `.env` alongside. |
| - **Hand-made units on IE** move into code. The console unit is the |
| platform's to ship, since the console is part of the platform. Woodstar, |
| keymail-relay and amadan-backup are tenant services and move into |
| @@ -299,9 +328,9 @@ destination in step with the deployment's buckets. |
| marker first, or deleting a store breaks. That has to land before or |
| with the versioning. Version |
| `keymail-blobs` in tofu. |
| -- Drop `s3:DeleteObjectVersion` from the region-host data-bucket statement |
| - unless something needs it. The builder checks who added it in 3ca43d33 |
| - and why. |
| +- Narrow `s3:DeleteObjectVersion` on the region-host data-bucket statement |
| + to the prefixes host lifecycle claim Plan 2 uses (spec C3, §6.2 and |
| + §6.5). The claim needs it, so dropping it outright would break it. |
| - Add a noncurrent-version lifecycle to the data buckets (30 days), so |
| versioning's cost is bounded. |
| - Merge `amadan-backup` properly or replace it. Once §5.1 lets amadan |
| @@ -323,7 +352,7 @@ destination in step with the deployment's buckets. |
| | 5 | §5.4 `.env` and box-secret escrow; SSM read narrowing | platform | medium | |
| | 6 | §5.5 `carlos ops offsite` to Hetzner | platform | large | |
| | 7 | §5.3 app-file snapshots | platform | large | |
| -| 8 | §5.6 `restore-registry`, bootstrap restore order, units into code, runbook, rehearsal | platform + carloku | medium | |
| +| 8 | §5.6 `restore-box`, bootstrap restore order, units into code, runbook, rehearsal | platform + carloku | medium | |
| | 9 | §5.7 smaller fixes | both | small | |
| Each platform step goes through the usual spec review, plan, build, Astra |
| @@ -356,13 +385,26 @@ just carloku's choice. |
| ## 9. Questions for Paul |
| -1. **App files (§5.3):** back up files apps write on disk (my |
| +1. **Where the off-account copier runs (§5.5).** My recommendation: a small |
| + Hetzner server pulls from AWS with a read-only IAM user, writes to |
| + Hetzner storage, and also checks heartbeats. Nothing in AWS can then |
| + reach the copy at all, and no box's role widens. The alternative is a |
| + box in AWS with a role widened to read every region's data and every |
| + tenant bucket. |
| +2. **App files (§5.3):** back up files apps write on disk (my |
| recommendation), or declare them not durable and move apps like `welp` |
| - to object storage? |
| -2. **Off-account encryption (§5.5):** seal objects client-side before they |
| - reach Hetzner (my recommendation), or rely on Hetzner's server-side |
| - encryption? |
| -3. **Retention:** 30 days of object lock at Hetzner and 7 days of |
| - point-in-time history in-account. Longer? |
| -4. **#290:** this design needs a replica-freshness signal; the design there |
| + to object storage? Discussion #355. |
| +3. **Data location:** Hetzner's object storage is only in Germany and |
| + Finland. Is a copy of US and AU tenants' data in the EU acceptable, or |
| + does each region need its own destination? |
| +4. **Object lock mode and retention:** compliance mode makes the copy truly |
| + undeletable for the retention period, by anyone, Paul included. |
| + Governance mode can be overridden by any key allowed to bypass it, and |
| + Hetzner's keys may not be restrictable that way. My recommendation: |
| + compliance, 14 days at the destination, 7 days of point-in-time history |
| + in-account. |
| +5. **Seal before Hetzner (§5.5):** seal objects client-side (my |
| + recommendation), or rely on Hetzner's server-side encryption? Sealing |
| + means restores always need `CARLOS_SECRETS_KEY` from 1Password. |
| +6. **#290:** this design needs a replica-freshness signal; the design there |
| is still awaiting your ruling. |