amadan
durability design: every byte off the box, every box rebuildable (draft)
From the 2026-10-22/23 live audit of carloku.com: the console database had no copy anywhere, the Irish box's Litestream config is hand-written and stale, instance keys and several secrets live only on their box, and nothing in the account has a copy outside it. Six workstreams, cheapest risk reduction first; questions for Paul in §9.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
1 file changed,
+368
−0
docs/superpowers/specs/2026-10-23-durability-design.md+368 −0
CI cancelled — superseded by a newer push — run details
CI log
=== gate ===
.amadan/test-gate.sh
=== launchctx (2 s)
ok amadan.net/carlos/platform.git/launchctx 0.034s
? amadan.net/carlos/platform.git/launchctx/controlname [no test files]
? amadan.net/carlos/platform.git/launchctx/wireid [no test files]
=== vet (19 s)
=== gofmt (0 s)
gofmt -l .: clean
=== rest (174 s)
? github.com/carlosframework/platform/cmd/changelog [no test files]
ok github.com/carlosframework/platform/cmd/console 3.608s
ok github.com/carlosframework/platform/cmd/moveprobe 11.883s
ok github.com/carlosframework/platform/cmd/zddprobe 0.745s
ok github.com/carlosframework/platform/internal/activator 60.633s
ok github.com/carlosframework/platform/internal/agentsock 0.043s
ok github.com/carlosframework/platform/internal/alerting 0.010s
ok github.com/carlosframework/platform/internal/analytics 0.043s
ok github.com/carlosframework/platform/internal/appconfig 2.935s
ok github.com/carlosframework/platform/internal/apperr 0.199s
ok github.com/carlosframework/platform/internal/applog 0.284s
ok github.com/carlosframework/platform/internal/awsinv 0.014s
ok github.com/carlosframework/platform/internal/boxchoice 0.007s
ok github.com/carlosframework/platform/internal/boxid 0.831s
? github.com/carlosframework/platform/internal/boxid/boxidtest [no test files]
? github.com/carlosframework/platform/internal/boxid/internal/signtest [no test files]
ok github.com/carlosframework/platform/internal/boxvolumes 0.029s
ok github.com/carlosframework/platform/internal/caps 0.031s
ok github.com/carlosframework/platform/internal/certwarm 0.007s
ok github.com/carlosframework/platform/internal/cgroup 0.760s
ok github.com/carlosframework/platform/internal/changelog 0.535s
ok github.com/carlosframework/platform/internal/claimactivation 1.228s
ok github.com/carlosframework/platform/internal/clamd 0.066s
? github.com/carlosframework/platform/internal/cliflag [no test files]
? github.com/carlosframework/platform/internal/controlname [no test files]
? github.com/carlosframework/platform/internal/controlsock [no test files]
ok github.com/carlosframework/platform/internal/dbdoor 8.167s
ok github.com/carlosframework/platform/internal/dbdoor/dbdoortest 0.016s
ok github.com/carlosframework/platform/internal/dbquery 0.016s
ok github.com/carlosframework/platform/internal/dbreads 2.251s
? github.com/carlosframework/platform/internal/dbreads/dbreadstest [no test files]
ok github.com/carlosframework/platform/internal/dbverify 0.008s
ok github.com/carlosframework/platform/internal/dnsrec 0.010s
ok github.com/carlosframework/platform/internal/docsite 0.030s
ok github.com/carlosframework/platform/internal/domains 2.954s
ok github.com/carlosframework/platform/internal/doorkey 0.015s
ok github.com/carlosframework/platform/internal/economics 0.008s
ok github.com/carlosframework/platform/internal/edge 45.803s
? github.com/carlosframework/platform/internal/edgeproof [no test files]
ok github.com/carlosframework/platform/internal/edgesfile 0.012s
ok github.com/carlosframework/platform/internal/fleetapi 0.177s
ok github.com/carlosframework/platform/internal/gate 1.722s
ok github.com/carlosframework/platform/internal/genselect 0.049s
ok github.com/carlosframework/platform/internal/geo 0.017s
ok github.com/carlosframework/platform/internal/geoip 0.065s
ok github.com/carlosframework/platform/internal/historyreadback 0.249s
ok github.com/carlosframework/platform/internal/host 144.226s
ok github.com/carlosframework/platform/internal/hostactors 5.486s
ok github.com/carlosframework/platform/internal/hostactors/awsmembership 0.005s
ok github.com/carlosframework/platform/internal/hostbox 3.065s
ok github.com/carlosframework/platform/internal/hostclaim 0.787s
ok github.com/carlosframework/platform/internal/hostenv 0.007s
ok github.com/carlosframework/platform/internal/hostevidence 0.012s
ok github.com/carlosframework/platform/internal/hostexec 5.331s
ok github.com/carlosframework/platform/internal/hostfence 0.069s
ok github.com/carlosframework/platform/internal/hostfiles 10.322s
ok github.com/carlosframework/platform/internal/hostfloor 0.023s
ok github.com/carlosframework/platform/internal/hostguard 1.066s
ok github.com/carlosframework/platform/internal/hostops 2.014s
? github.com/carlosframework/platform/internal/hostops/fencerelease [no test files]
ok github.com/carlosframework/platform/internal/hostops/internal/fencerelease 0.006s
ok github.com/carlosframework/platform/internal/hostquarantine 0.007s
ok github.com/carlosframework/platform/internal/hostrelocate 0.690s
ok github.com/carlosframework/platform/internal/hostresolve 0.188s
ok github.com/carlosframework/platform/internal/hostwrite 0.004s
ok github.com/carlosframework/platform/internal/infraview 0.003s
ok github.com/carlosframework/platform/internal/instancectl 5.243s
ok github.com/carlosframework/platform/internal/instanceexec 2.833s
ok github.com/carlosframework/platform/internal/instancefs 0.010s
ok github.com/carlosframework/platform/internal/instancekey 0.005s
ok github.com/carlosframework/platform/internal/instanceupdates 0.009s
ok github.com/carlosframework/platform/internal/layout 1.694s
ok github.com/carlosframework/platform/internal/lbhealth 0.003s
ok github.com/carlosframework/platform/internal/lease 10.140s
ok github.com/carlosframework/platform/internal/lease/leasetest 0.496s
ok github.com/carlosframework/platform/internal/ledger 0.018s
ok github.com/carlosframework/platform/internal/mail 0.313s
ok github.com/carlosframework/platform/internal/maintenance 0.102s
ok github.com/carlosframework/platform/internal/maintenanceexec 5.533s
ok github.com/carlosframework/platform/internal/maintenanceguard 0.115s
ok github.com/carlosframework/platform/internal/metrics 0.014s
ok github.com/carlosframework/platform/internal/moves 22.516s
ok github.com/carlosframework/platform/internal/moves/archive 1.028s
ok github.com/carlosframework/platform/internal/objstore 0.306s
ok github.com/carlosframework/platform/internal/ops 0.228s
ok github.com/carlosframework/platform/internal/park 0.270s
ok github.com/carlosframework/platform/internal/plans 0.013s
ok github.com/carlosframework/platform/internal/privatefile 0.007s
ok github.com/carlosframework/platform/internal/provision 23.628s
ok github.com/carlosframework/platform/internal/publicrelease 0.010s
? github.com/carlosframework/platform/internal/regionname [no test files]
ok github.com/carlosframework/platform/internal/registry 5.666s
ok github.com/carlosframework/platform/internal/relay 7.234s
ok github.com/carlosframework/platform/internal/release 26.314s
ok github.com/carlosframework/platform/internal/rollout 3.837s
ok github.com/carlosframework/platform/internal/rolloutwords 0.004s
ok github.com/carlosframework/platform/internal/rootaws 0.007s
ok github.com/carlosframework/platform/internal/rootparticipation 0.069s
ok github.com/carlosframework/platform/internal/rootstate 0.027s
ok github.com/carlosframework/platform/internal/runs 0.602s
ok github.com/carlosframework/platform/internal/s3compat 0.006s
ok github.com/carlosframework/platform/internal/safefile 0.007s
ok github.com/carlosframework/platform/internal/schedule 2.651s
ok github.com/carlosframework/platform/internal/seeds 0.028s
ok github.com/carlosframework/platform/internal/sim/engines 2.725s
ok github.com/carlosframework/platform/internal/sim/hostlife 10.841s
ok github.com/carlosframework/platform/internal/sim/litestreamfacts 3.751s
ok github.com/carlosframework/platform/internal/status 0.078s
ok github.com/carlosframework/platform/internal/steering 0.007s
ok github.com/carlosframework/platform/internal/stopevidence 0.308s
ok github.com/carlosframework/platform/internal/svcrole 0.002s
ok github.com/carlosframework/platform/internal/telemetry 0.004s
ok github.com/carlosframework/platform/internal/termfence 0.005s
ok github.com/carlosframework/platform/internal/tunnelclient 0.008s
ok github.com/carlosframework/platform/internal/unixdial 0.003s
ok github.com/carlosframework/platform/internal/updatewords 0.008s
ok github.com/carlosframework/platform/internal/usage 0.007s
ok github.com/carlosframework/platform/internal/version 0.002s
ok github.com/carlosframework/platform/internal/vet 1.699s
=== internal_console@0 (85 s)
ok github.com/carlosframework/platform/internal/console 84.013s
=== internal_console@1 (41 s)
ok github.com/carlosframework/platform/internal/console 40.086s
=== internal_console@2 (44 s)
ok github.com/carlosframework/platform/internal/console 42.741s
=== internal_console@3 (43 s)
ok github.com/carlosframework/platform/internal/console 41.225s
=== internal_console@4 (45 s)
ok github.com/carlosframework/platform/internal/console 43.981s
=== internal_console@5 (42 s)
ok github.com/carlosframework/platform/internal/console 40.514s
=== internal_console@6 (40 s)
ok github.com/carlosframework/platform/internal/console 38.398s
=== internal_console@7 (36 s)
ok github.com/carlosframework/platform/internal/console 35.018s
=== root@0 (107 s)
ok github.com/carlosframework/platform 105.872s
=== root@1 (90 s)
ok github.com/carlosframework/platform 88.448s
=== root@2 (80 s)
ok github.com/carlosframework/platform 78.822s
=== cross ===
--- shard 0 of 5 ---
cross-compiling darwin/arm64
--- shard 1 of 5 ---
cross-compiling darwin/amd64
--- shard 2 of 5 ---
cross-compiling linux/arm64
linux/arm64 carlos binary: 50528418 bytes (ceiling 52000000)
--- shard 3 of 5 ---
cross-compiling windows/amd64
--- shard 4 of 5 ---
cross-compiling windows/arm64
=== minio-contract ===
minio contract: arm64, minio RELEASE.2025-09-07T16-13-09Z, mc RELEASE.2025-08-13T08-35-41Z
minio: downloading https://github.com/minio/minio/releases/download/RELEASE.2025-09-07T16-13-09Z/minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
minio: downloaded, digest verified
mc: downloading https://github.com/minio/mc/releases/download/RELEASE.2025-08-13T08-35-41Z/mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
mc: downloaded, digest verified
minio healthy on 127.0.0.1:20999
Added `local` successfully.
Bucket created successfully `local/contract-bucket`.
Bucket created successfully `local/denied-bucket`.
Created policy `contract-only` successfully.
Added user `scopeduser` successfully.
Attached Policies: [contract-only]
To User: scopeduser
buckets and scoped user ready
=== RUN TestDeliveredContractMinio
=== RUN TestDeliveredContractMinio/put
=== RUN TestDeliveredContractMinio/get_round-trips_the_body
=== RUN TestDeliveredContractMinio/list_shows_the_key
=== RUN TestDeliveredContractMinio/head_and_list_agree_on_the_version
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict
=== RUN TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed
=== RUN TestDeliveredContractMinio/delete_removes_it
=== RUN TestDeliveredContractMinio/cross-bucket_denial
--- PASS: TestDeliveredContractMinio (0.02s)
--- PASS: TestDeliveredContractMinio/put (0.00s)
--- PASS: TestDeliveredContractMinio/get_round-trips_the_body (0.00s)
--- PASS: TestDeliveredContractMinio/list_shows_the_key (0.00s)
--- PASS: TestDeliveredContractMinio/head_and_list_agree_on_the_version (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/not_by_PutObjectTagging (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_its_own_object/and_not_by_x-amz-tagging_on_upload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route (0.01s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CreateMultipartUpload (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/not_by_CopyObject_with_a_REPLACE_tagging_directive (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/but_multipart_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_copy_itself_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/a_POST-policy_forged_tag_with_a_matching_version_is_still_not_a_verdict (0.00s)
--- PASS: TestDeliveredContractMinio/the_app_cannot_tag_by_any_other_route/and_an_ordinary_tagged-with-something-else_upload_is_allowed (0.00s)
--- PASS: TestDeliveredContractMinio/delete_removes_it (0.00s)
--- PASS: TestDeliveredContractMinio/cross-bucket_denial (0.00s)
PASS
ok github.com/carlosframework/platform/internal/objstore 0.023s
=== RUN TestT4StorageProbeMinio
2026/10/23 11:09:08 moves: T4: STORE: list: operation error S3: ListObjectsV2, https response error StatusCode: 403, RequestID: 18E12409131FB052, HostID: 7c420053e0fbcd7f41653fdd92ffa2802c903d05303b1aa15adc2e8bb65d183f, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method.
--- PASS: TestT4StorageProbeMinio (0.01s)
PASS
ok github.com/carlosframework/platform/internal/host 0.022s
=== RUN TestS3StoreMinioContract
=== RUN TestS3StoreMinioContract/conditional_get_and_etag_listing
=== RUN TestS3StoreMinioContract/bump_through_every_mutation_path
--- PASS: TestS3StoreMinioContract (0.03s)
--- PASS: TestS3StoreMinioContract/conditional_get_and_etag_listing (0.01s)
--- PASS: TestS3StoreMinioContract/bump_through_every_mutation_path (0.02s)
PASS
ok github.com/carlosframework/platform/internal/release 0.030s
Bucket created successfully `local/dbf-deploy`.
Bucket created successfully `local/dbf-data-ie`.
Bucket created successfully `local/dbf-data-uk`.
local/dbf-deploy versioning is enabled
Created policy `dbf-console` successfully.
Created policy `dbf-ukbox` successfully.
Added user `dbfconsole` successfully.
Added user `dbfukbox` successfully.
Attached Policies: [dbf-console]
To User: dbfconsole
Attached Policies: [dbf-ukbox]
To User: dbfukbox
amadan: job cancelled — a newer push superseded it
diff --git a/docs/superpowers/specs/2026-10-23-durability-design.md b/docs/superpowers/specs/2026-10-23-durability-design.md| new file mode 100644 |
| index 000000000..bd507feb2 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-10-23-durability-design.md |
| @@ -0,0 +1,368 @@ |
| +# Durability: every byte off the box, every box rebuildable |
| + |
| +Status: DRAFT, 2026-10-23. Written from a live audit of carloku.com on |
| +2026-10-22 and 23. Platform-wide: carloku.com is the first deployment it |
| +lands on, and Tito's install has the same gaps (account 665106592142, being |
| +handled from the Tito side). |
| + |
| +## 1. Why |
| + |
| +Paul asked three questions on 2026-10-22. Which keys aren't backed up or in |
| +1Password? Which databases aren't on Litestream? If we lost every Carloku |
| +box, could we recover? The audit's answer was "tenants mostly yes, the |
| +console no". The console database, which holds every account, had no copy |
| +anywhere. Several other things lived only on a box. |
| + |
| +Paul's terms, 2026-10-23: |
| + |
| +- "If the Irish box disappeared, there's nothing hypercritical on it, but |
| + anything more than a few hours worth of data would be a massive pain." |
| +- "Low hanging fruit first, and by end of session, everything should have |
| + resilience and durability built in." |
| +- Standing rule: carloku.com consumes the platform like any self-hoster. |
| + If carloku needs something, the platform provides it, and nobody works |
| + around the platform by hand on a box. |
| +- Standing rule: AWS features are adapters. A deployment on one |
| + S3-compatible bucket plus hosts plus DNS must get the same durability. |
| + |
| +## 2. Targets |
| + |
| +| | Target | Notes | |
| +|---|---|---| |
| +| Data loss (RPO), any database | seconds to minutes | Litestream, as today for hibernating instances | |
| +| Data loss (RPO), app files on disk | at most 1 hour | new; see §5.3 | |
| +| Data loss, everything, if the AWS account is lost | at most 1 hour behind | the off-account copy, §5.5 | |
| +| Downtime (RTO), one box | under 4 hours, by runbook | Paul: "nothing hypercritical" | |
| +| Downtime (RTO), every box | one working day | same runbook, run per box | |
| +| Proof | every copy restore-tested on a schedule, failures alert | a backup nobody has restored doesn't count | |
| + |
| +The 1-hour file RPO and the 4-hour RTO are my call, sized to "a few hours". |
| + |
| +## 3. Goals, non-goals, anti-goals |
| + |
| +**Goals** |
| + |
| +1. Everything a box holds that isn't rebuildable from elsewhere has an |
| + up-to-date copy off the box: databases, app files, keys, box settings. |
| +2. Everything in the AWS account that matters has a copy outside it. |
| +3. A box can be rebuilt from those copies using platform verbs and a |
| + written runbook, and that has been rehearsed. |
| +4. When any copy stops being current, an alert fires and reaches a person. |
| +5. All of it is platform behaviour, on by default for new deployments. A |
| + self-hoster gets it by configuring a destination, not by copying |
| + carloku's scripts. |
| + |
| +**Non-goals** |
| + |
| +- Hot standby or automatic failover. The region-HA plans (2a, 2b merged |
| + dormant) cover that separately. This design is about losing nothing and |
| + being able to rebuild. |
| +- An AWS Organizations or multi-account setup for carloku. |
| +- Backing up what is rebuildable: release binaries in `/opt/carlos/versions`, |
| + static sites in `/data/static` (served from the bucket), GeoIP data, |
| + certificates (republished from `control/certs/`), ClamAV signatures, |
| + amadan's `amadan-materialized/repo.git` trees (3.5 GB; "pure scratch: |
| + rebuildable anytime from the store, never replicated", amadan |
| + `internal/repod/githttp.go`). |
| + |
| +**Anti-goals** (would make this a failure even if it works) |
| + |
| +- Any config on a box maintained by hand. Today's `/etc/litestream.yml` on |
| + the Irish box is exactly the thing that dropped the console database. |
| +- A copy that no drill ever restores. |
| +- An alert rule that can never fire (today's lag rule is wired to nil). |
| +- Two Litestream writers on one database's WAL; it corrupts the replica. |
| +- Requiring AWS for any of it. |
| +- Secrets printed to terminals, logs, or SSM command history. |
| + |
| +## 4. Where things stand (2026-10-23) |
| + |
| +Done this session: |
| + |
| +- `console.db`, the `proving-bots` sidecar, Kass's fourth instance database |
| + and Woodstar's `push.db` and `unfurl.db` now replicate. They were added by |
| + hand to the Irish box's existing hand-written `/etc/litestream.yml` |
| + (backup `/etc/litestream.yml.bak-20261023`), and restore-tested: integrity |
| + ok, 58 of 58 accounts. Interim only: §5.1 replaces the file. |
| +- `carloku-infrastructure/scripts/backup-to-1password.sh` (uncommitted) |
| + copies the 12 `/carlos/*` SSM secrets and an encrypted per-box archive |
| + (`/etc/carlos`, units and drop-ins, instance `.env` files, Woodstar seeds) |
| + into 1Password. Paul runs it; agents can't read the values. |
| + |
| +Still open, from the audit: |
| + |
| +| Gap | Where | |
| +|---|---| |
| +| The Irish Litestream config is hand-written, runs v0.3.13 (24h history, the 7-day setting is ignored), and nothing regenerates it | IE | |
| +| UK, US and AU have no box-wide Litestream, so a non-hibernating instance there has no backup | UK, US, AU | |
| +| `welp.bae`'s data (2.6 GB: `identity.db`, `registry.db`, a mailbox tree) is unreplicated; the route's DBPath names a file that doesn't exist | IE | |
| +| App files beside databases have no copy: `/data/blobs` (1.1 GB, shared by flat apps), `comic-cleaner` blobs (256 MB), others | IE | |
| +| Instance keys and admin tokens (`/data/<host>.env`, 295 on IE) exist only on their box | all | |
| +| Relay masters, Keymail push keys, `release.env`: box-only | IE | |
| +| `host.env` has no copy. The platform's `carlos system host-env` SSM escrow is built for region-HA group boxes: its required list includes the claim and HA switches carloku's boxes don't set, and its allowlist refuses credentials such as `CARLOS_DNSIMPLE_TOKEN` | all | |
| +| Release, CLI-update and apt signing keys, Kass peppers: SSM only, one account | account | |
| +| No copy of anything outside the AWS account; tofu state is in the same account | account | |
| +| `carlos-store-*` tenant buckets and `keymail-blobs` aren't versioned | account | |
| +| The Irish box role can `s3:DeleteObjectVersion` on its data bucket (region-host v0.23.0+) | IE | |
| +| No EBS snapshots | all | |
| +| No alert for Litestream down, a failed drill, a stale replica, or an unreplicated database; restore-verify publishes nothing; hibernate-verify isn't armed | all | |
| +| No restore-from-snapshot verb for the registry; no carloku box-loss runbook; never rehearsed | all | |
| +| Hand-made units on IE (console, Woodstar, keymail-relay, amadan-backup) aren't in any repo; Keymail's IMDS uid rule isn't in `harden-host.sh` | IE | |
| +| amadan-hub's instance replica stopped 2026-08-08; covered only by the hourly `amadan-backup.timer` from an unmerged branch | IE | |
| + |
| +## 5. The design |
| + |
| +Six workstreams, ordered so the cheapest risk reduction lands first (§6). |
| + |
| +### 5.1 The platform owns the whole replication set |
| + |
| +The box-wide replication config is generated, kept current automatically, |
| +and covers every database on the box. |
| + |
| +- **Routes can name more than one database.** DBPath becomes a list. Any |
| + entry may be a directory, meaning every non-empty `*.db` beneath it (the |
| + existing `treeDBs` rule), for any backing, not just `BackingUnit`. Set by |
| + `carlos add --db` (repeatable) and a new `carlos route --db` for addr |
| + routes. This lets Kass (a tree in its state dir) and Woodstar (services |
| + proxied by addr, databases in `/data/woodstar/`) be described. |
| +- **Exec instances replicate their whole instance directory's databases.** |
| + For a nested-layout instance, every `*.db` under |
| + `/data/instances/<id>/` is claimed, not only `<host>.db`. That fixes |
| + `welp.bae`. For hibernating instances the activator's embedded |
| + replicator runs one Litestream DB per file. Box-wide replication still |
| + never touches a hibernating instance's files. |
| +- **The generator runs itself.** The agent re-runs the `ops litestream` |
| + generation on route changes and every 10 minutes, and restarts the |
| + replicator only on "changed". The shrink guard stays. A shrink still |
| + needs an operator decision, and it now raises an alert instead of |
| + failing quietly in a log. |
| +- **Every box runs box-wide Litestream.** Bootstrap installs the pinned |
| + binary (0.5.15, checksum-verified, from the release bucket rather than |
| + GitHub), writes `/data/litestream.env`, and enables the unit. The unit |
| + runs as root with the capabilities unit tenants need: `CAP_DAC_OVERRIDE`, |
| + `CAP_DAC_READ_SEARCH`, `CAP_FOWNER` and `CAP_CHOWN`. That's today's |
| + hand-made `user.conf` drop-in, moved into the shipped unit. `CAP_CHOWN` |
| + was added on IE on 2026-10-23, after Tito's review found it missing: when |
| + root is the first to open a database whose `-wal` and `-shm` are missing, |
| + SQLite chowns them to the database's owner, and without the capability |
| + they stay root-owned and the app can't write them. |
| +- **Cutover on the Irish box** happens once the generator describes |
| + everything the hand file names. Moving 0.3.13 to 0.5.15 changes the |
| + replica format, so every box-wide database starts a new lineage. The old |
| + 0.3 replicas stay in the versioned bucket until the new ones have passed |
| + a restore drill, then expire through a lifecycle rule. The plan written |
| + on 2026-08-15 (`2026-08-15-flagship-litestream-0.5-upgrade.md` in |
| + platform-infrastructure) is the starting point. |
| + |
| +### 5.2 Proof: drills and alerts that fire |
| + |
| +- `restore-verify` and `hibernate-verify` are armed by bootstrap on every |
| + box. A drill proves currency, not just shape. A replica stuck at its |
| + first snapshot passes `integrity_check` and a table count, so the drill |
| + keeps its change-counter freshness check and never treats schema-only |
| + success as current. They publish their verdicts to `control/verify/<box>.json`, and the |
| + console shows them. |
| +- New alert rules, through the existing alert delivery (SNS email today; |
| + the webhook stays optional): |
| + - box-wide Litestream not active, or restarting (slice A of #284 already |
| + adds `restarts@<unit>`) |
| + - a drill failed, or hasn't run in 36 hours |
| + - a database on disk that no replication claims, older than 15 minutes |
| + (the generator's "claimed by no route" list, turned into an alert) |
| + - a shrink the guard refused |
| + - the off-account copy (§5.5) more than 2 hours behind |
| + - file snapshots (§5.3) more than 2 hours behind |
| +- Replica freshness, meaning "this replica is still tracking its source", |
| + is #290's open question. This design needs a freshness signal and takes |
| + whichever design Paul rules for on #290. Until then the daily drill's |
| + existing freshness check (`internal/ops/freshness.go`) is the signal. |
| +- The lag rule's nil wiring (`metrics_loop.go:492`) gets fixed or the rule |
| + removed. My call: removed, since the freshness alert replaces it. |
| + |
| +### 5.3 App files on disk |
| + |
| +Apps write files beside their databases: mailboxes, uploads, blobs. The |
| +platform has never said whether those are durable, and they aren't. |
| + |
| +My recommendation, which Paul needs to confirm (§9, question 1): back them |
| +up. |
| + |
| +- An hourly, incremental, content-addressed snapshot of each instance's |
| + directory to the bucket, under `instances/<host>/files/`, skipping |
| + databases (Litestream has those), `-wal`, `-shm`, `.env` (§5.4) and the |
| + replicator's own dot-directories. Unchanged files cost nothing to |
| + re-snapshot. Keep 7 days of snapshots. |
| +- The flat layout's shared `/data/blobs` can't be attributed to one host |
| + (`internal/layout/extras.go`). It's snapshotted as one box-level set |
| + until the layout pass moves its contents into instance directories. |
| +- `carlos instances files restore <host> [--at <time>]` restores them. |
| + The wake path restores them too when a hibernating instance wakes on a |
| + box that lacks them, as it already does for the database. |
| +- Moves reuse it: an offline move's archive could be built from the latest |
| + snapshot. That's a later simplification, not part of this work. |
| + |
| +### 5.4 Keys and box settings |
| + |
| +- **Instance `.env` escrow.** When `<host>.env` is minted or changed, the |
| + platform seals it to the deployment's secrets public key |
| + (`appconfig.SealBound`, bound to account and app) and writes it to |
| + `instances/<host>/env.sealed` in the bucket. Restore needs |
| + `CARLOS_SECRETS_KEY`, which is in SSM in four regions and in 1Password, |
| + so a rebuilt box can unseal it. Nothing on a box can read another |
| + instance's escrow without that key. |
| +- **Platform-owned box secrets** (relay masters, `release.env`): the same |
| + sealing, at `control/box-secrets/<box>/<name>.sealed`. Bootstrap restores |
| + them. |
| +- **`host.env`:** the existing `carlos system host-env` escrow becomes |
| + usable by any box, not only region-HA group boxes. The claim and HA |
| + switches move from required to optional for a box that isn't in a group. |
| + Credentials its allowlist refuses on purpose (`CARLOS_DNSIMPLE_TOKEN`) |
| + go to the sealed box secrets above instead. The bucket-sealed copy is |
| + the non-AWS fallback (§7). |
| +- **Signing keys** (release, CLI update, apt) **and Kass peppers:** |
| + 1Password through the script. They stay operator-held; the platform |
| + doesn't escrow its own signing keys. |
| +- **Tenant secrets the platform doesn't own** (Keymail's APNs key in |
| + `/etc/keymail-push/`, Woodstar's bridge seeds) belong in the tenant's |
| + sealed app config. Until each app moves them there, the 1Password script |
| + is their copy. That's carloku's housekeeping, not a platform feature. |
| +- **Narrow who can read `/carlos/*` SSM.** Every box can read every |
| + parameter today, including the release key. Fix it, with a separate path |
| + for the release key that only the console's role can read. |
| + |
| +### 5.5 The copy outside the account |
| + |
| +A platform verb, `carlos ops offsite`, keeps a second S3-compatible |
| +destination in step with the deployment's buckets. |
| + |
| +- **What it copies:** the deployment bucket, every region's data bucket, |
| + every `carlos-store-*` tenant bucket, and anything named in |
| + `CARLOS_OFFSITE_EXTRA` (on carloku: the tofu state bucket and |
| + `keymail-blobs`). It copies current versions only; history lives in |
| + versioning at both ends. |
| +- **How:** incremental by listing, comparing ETag and size, then Get from |
| + the source and Put to the destination. Server-side copy can't cross |
| + providers. Litestream's LTX files are immutable, so most runs move only |
| + new segments. It runs every 15 minutes, on the box holding the |
| + housekeeper lease, so exactly one box does it. |
| +- **The destination:** Hetzner Object Storage (Paul, 2026-10-23: "we use |
| + our Hetzner account"), versioned, with object lock in governance mode |
| + and a 30-day default retention (my call). The destination credentials |
| + can write but not delete locked versions, so a compromised box can't |
| + destroy the copy. |
| +- **Encryption:** objects are sealed client-side to the deployment's |
| + secrets key before upload (my recommendation, §9 question 2). Then |
| + Hetzner holds only ciphertext, and the copy still works with |
| + `CARLOS_SECRETS_KEY` from 1Password. |
| +- **Configuration:** `CARLOS_OFFSITE_ENDPOINT`, `_BUCKET` and `_REGION`, |
| + plus credentials sealed in the deployment config. Not `CARLOS_S3_ENDPOINT`: |
| + that's process-wide and would redirect the primary. |
| +- **It publishes** `control/offsite.json` (last complete pass, objects and |
| + bytes behind) for the alert in §5.2. |
| +- **Restore:** `carlos ops offsite restore --to <bucket>` reverses it, so a |
| + new account or provider can be seeded from it. |
| +- **Route53:** the console exports each zone's records to |
| + `control/dns/<zone>.json` daily, so the copy includes DNS. |
| + |
| +### 5.6 Rebuilding a box |
| + |
| +- **`carlos system restore-registry --from <box-label>`** copies the dead |
| + box's latest registry snapshot into place on a fresh box before the edge |
| + starts, and refuses to start if the label is a live box (the clobber risk |
| + `snapshot_loop.go` notes). It replaces the hand-run `aws s3 cp` step. |
| +- **Bootstrap restores, in order:** `host.env` from the host-env escrow; |
| + sealed box secrets; the registry (when told to, with `--from`); then it |
| + starts the edge. Tenants restore themselves on first wake. Their database |
| + comes from Litestream, their files from §5.3, their `.env` from §5.4. |
| +- **Hand-made units on IE** move into code. The console unit is the |
| + platform's to ship, since the console is part of the platform. Woodstar, |
| + keymail-relay and amadan-backup are tenant services and move into |
| + `carloku-infrastructure` as files plus an install script. The Keymail |
| + IMDS uid rule moves into `harden-host.sh`. |
| +- **EBS snapshots:** a daily DLM policy on every carloku box, 7 days kept, |
| + in `carloku-infrastructure`. Carloku's AWS-only safety net. It catches |
| + anything this design missed, and it's not a platform feature. |
| +- **Runbook** in `carloku-infrastructure/RUNBOOK-box-loss.md`, adapted |
| + from Tito's `RUNBOOK-edge-failover.md`, using only platform verbs. |
| +- **Rehearsal:** |
| + - First, rebuild a dev box (`carlos-dev-*`, account 640012953448) from |
| + nothing, timed. |
| + - Then restore one carloku database and one instance's files from the |
| + Hetzner copy alone. |
| + - No live flagship box is destroyed to rehearse. |
| + |
| +### 5.7 Smaller fixes |
| + |
| +- Version the `carlos-store-*` buckets at creation (`EnsureBucket`), with a |
| + 30-day noncurrent-version expiry, and backfill existing ones. The |
| + console's bucket-delete path must then delete every version and delete |
| + marker first, or deleting a store breaks. That has to land before or |
| + with the versioning. Version |
| + `keymail-blobs` in tofu. |
| +- Drop `s3:DeleteObjectVersion` from the region-host data-bucket statement |
| + unless something needs it. The builder checks who added it in 3ca43d33 |
| + and why. |
| +- Add a noncurrent-version lifecycle to the data buckets (30 days), so |
| + versioning's cost is bounded. |
| +- Merge `amadan-backup` properly or replace it. Once §5.1 lets amadan |
| + declare its databases (a tree under `/data/amadan/instances`), it's |
| + ordinary replication. |
| +- Remove leftovers the generator lists as unclaimed, after a check: |
| + `opgate-a5wpd4.bet` (renamed), `seapointish-purge-backup.db` (410 MB), |
| + `contra-fresh-*`. |
| + |
| +## 6. Order |
| + |
| +| Step | What | Who | Size | |
| +|---|---|---|---| |
| +| 0 | console.db and four others replicating (done); 1Password script (done) | done | | |
| +| 1 | Run the 1Password script | Paul | minutes | |
| +| 2 | Commit the script; IMDS rule into `harden-host.sh`; EBS snapshots (DLM); version `keymail-blobs` | carloku ops | small | |
| +| 3 | §5.1 multi-DB routes, whole-instance-dir replication, self-running generator, Litestream on every box; then the IE cutover to 0.5.15 | platform | large | |
| +| 4 | §5.2 drills armed and published; new alert rules | platform | medium | |
| +| 5 | §5.4 `.env` and box-secret escrow; SSM read narrowing | platform | medium | |
| +| 6 | §5.5 `carlos ops offsite` to Hetzner | platform | large | |
| +| 7 | §5.3 app-file snapshots | platform | large | |
| +| 8 | §5.6 `restore-registry`, bootstrap restore order, units into code, runbook, rehearsal | platform + carloku | medium | |
| +| 9 | §5.7 smaller fixes | both | small | |
| + |
| +Each platform step goes through the usual spec review, plan, build, Astra |
| +review, release and roll. Paul's "by end of session" is the goal. I won't |
| +skip review rounds to meet it. Steps 1 and 2 are today. Steps 3 to 9 are |
| +separate plans, written and built in this order. |
| + |
| +## 7. Provider boundary |
| + |
| +| Piece | Core (any S3-compatible store) | AWS adapter | Without AWS | |
| +|---|---|---|---| |
| +| Replication, drills, alerts, file snapshots, escrow, offsite, restore-registry | all core | none | same | |
| +| `host.env` escrow | bucket-sealed fallback | SSM (existing) | uses the sealed bucket copy | |
| +| SSM read narrowing | n/a | IAM | n/a | |
| +| EBS snapshots | n/a | carloku tofu only | operator's choice | |
| +| Route53 export | DNS provider interface | Route53 | the deployment's DNS provider, or none | |
| + |
| +Core packages don't import the AWS SDK, enforced by the existing import |
| +test. The offsite destination is any S3-compatible endpoint. Hetzner is |
| +just carloku's choice. |
| + |
| +## 8. Left to the builder |
| + |
| +- Package layout, names of internal types, unit file details. |
| +- The file-snapshot format, as long as it's incremental, content-addressed, |
| + restorable at a point in time and sealed when it leaves the account. |
| +- How the agent triggers regeneration (route-change hook plus timer, or |
| + timer only), as long as a new database is claimed within 10 minutes. |
| +- Exact alert thresholds within the targets in §2. |
| + |
| +## 9. Questions for Paul |
| + |
| +1. **App files (§5.3):** back up files apps write on disk (my |
| + recommendation), or declare them not durable and move apps like `welp` |
| + to object storage? |
| +2. **Off-account encryption (§5.5):** seal objects client-side before they |
| + reach Hetzner (my recommendation), or rely on Hetzner's server-side |
| + encryption? |
| +3. **Retention:** 30 days of object lock at Hetzner and 7 days of |
| + point-in-time history in-account. Longer? |
| +4. **#290:** this design needs a replica-freshness signal; the design there |
| + is still awaiting your ruling. |