amadan
spec: object-storage primitive design (#55)
Apps declare a store, operator grants, console provisions (S3+IAM), credentials delivered as sealed config vars; two-key rotation with explicit finish; purge sweeps the store via the console door.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 file changed,
+369
−0
docs/superpowers/specs/2026-08-10-object-storage-primitive-design.md+369 −0
diff --git a/docs/superpowers/specs/2026-08-10-object-storage-primitive-design.md b/docs/superpowers/specs/2026-08-10-object-storage-primitive-design.md| new file mode 100644 |
| index 000000000..e18424b1a |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-08-10-object-storage-primitive-design.md |
| @@ -0,0 +1,369 @@ |
| +# Object-storage primitive — design |
| + |
| +Issue #55. Apps running on CARLOS keep needing object storage, and today |
| +every one solves it by hand: create a bucket, write an IAM policy, mint |
| +keys out-of-band, paste endpoint + credentials into env. The primitive: |
| +an app declares "I need a store", the operator grants it for the |
| +deployment, and CARLOS provisions the backend, mints scoped credentials, |
| +and delivers the coordinates to the app's units as env — plus rotation |
| +and teardown as platform verbs. |
| + |
| +Decided with Paul, 2026-08-10. The section-by-section decisions and the |
| +alternatives they beat are inline; the shape follows the capability |
| +model's one-sentence security split (a need that only shapes the app's |
| +own world is the app's to declare; a need that consumes shared, |
| +operator-paid resources is the operator's to grant). |
| + |
| +## 1. Scope and v1 success criterion |
| + |
| +**v1 is done when titogo's file-upload feature works in production on |
| +the tito deployment via the primitive**: store declared, the dangling |
| +`titogo-uploads` bucket adopted, prefix-scoped credentials minted, and |
| +`TITO_S3_*` env delivered to titogo's instances — with zero titogo code |
| +changes. AWS is the only real backend in v1; Minio appears only as the |
| +CI contract test (§11) that keeps "S3-compatible" honest and marks the |
| +seam where Hetzner/R2/Minio provisioning arrives later. |
| + |
| +Control plane only, never the data plane. Apps keep their own S3 client |
| +and talk to the backend directly; a byte-proxy in the data path is a |
| +hop, a SPOF, and a pager, and is permanently out of scope. CARLOS owns |
| +provisioning, credentials, delivery, rotation, teardown. |
| + |
| +**The delivered contract** — everything an app may assume about its |
| +store: the S3 wire API, Put / Get / Delete / List on its own bucket, |
| +with the delivered credentials, against the delivered endpoint. Nothing |
| +more: no versioning, no SSE-KMS, no conditional-write or |
| +list-consistency promises. The contract is pinned by the Minio CI test, |
| +and it is deliberately worded per-*credential*, not per-*app*, so |
| +per-instance scoped credentials can arrive later without re-wording. |
| + |
| +## 2. The record: `apps/<acct>/<app>/store.json` |
| + |
| +One new bucket object per store, under the app prefix, CAS |
| +read-modify-write through the same If-Match discipline as |
| +`config.json`. Three blocks matching the three actors: |
| + |
| +```json |
| +{ |
| + "schema": 1, |
| + "app": "titogo", |
| + "declared": { |
| + "env_prefix": "TITO_S3", |
| + "at": "2026-08-10T12:00:00Z", "by": "member@example.com" |
| + }, |
| + "grant": { |
| + "backend": "aws", |
| + "bucket": "titogo-uploads", |
| + "at": "2026-08-10T12:05:00Z", "by": "paul@rslw.com" |
| + }, |
| + "provisioned": { |
| + "bucket": "titogo-uploads", |
| + "region": "eu-west-1", |
| + "endpoint": "", |
| + "iam_user": "carlos-store-tito-titogo", |
| + "user_arn": "arn:aws:iam::…:user/carlos-store/carlos-store-tito-titogo", |
| + "key_id": "AKIA…", |
| + "prev_key_id": "", |
| + "at": "2026-08-10T12:05:07Z" |
| + } |
| +} |
| +``` |
| + |
| +- `declared` — the member's half, written by `carlos store create`. |
| + `env_prefix` is the one delivery knob (§7); default `CARLOS_STORE`, |
| + charset `^[A-Z][A-Z0-9_]{0,31}$`. |
| +- `grant` — the operator's half, written by the grant door (§4). |
| + `bucket` set = adopt this existing bucket; empty = mint the name. |
| + `backend` is recorded so the record is self-describing even though v1 |
| + only ever writes `"aws"`. |
| +- `provisioned` — written only by the console after a successful |
| + provisioning walk. It records coordinates and the access-key **id**; |
| + the secret key exists nowhere except sealed inside `config.json`. |
| + |
| +**Why an app-level record and not a release-manifest capability.** The |
| +capability model's removal semantics — manifest stops declaring it, |
| +converge pass removes it — are exactly wrong for a bucket full of data: |
| +a rollback to a pre-store version would read as a teardown instruction. |
| +A store is a property of the app, not of a build (the same reasoning |
| +that put `exec_delivery` in the config plane). Store lifecycle is |
| +explicit verbs, never inferred from manifests; rollbacks are inert. |
| + |
| +## 3. Package |
| + |
| +`internal/objstore`, parallel to `internal/appconfig` and |
| +`internal/release`. It owns the record shape and CAS helpers, the |
| +`Provisioner` interface and its AWS implementation, the provisioning / |
| +rotation / teardown walks, and the env-set rendering. The console is |
| +its only production caller; the box side never imports the provisioner |
| +(boxes gain no backend authority — §9 is built around that). |
| + |
| +```go |
| +type Provisioner interface { |
| + EnsureBucket(ctx, name string) (region string, err error) |
| + EnsureUser(ctx, name, path, bucketARN string) (arn string, err error) |
| + MintKey(ctx, user string) (id, secret string, err error) |
| + DeleteKey(ctx, user, id string) error // NoSuchEntity == success |
| + EmptyBucket(ctx, name string) error // paginated, resumable |
| + DeleteBucket(ctx, name string) error // NoSuchBucket == success |
| + DeleteUser(ctx, user string) error // keys+policy first; NoSuchEntity == success |
| +} |
| +``` |
| + |
| +Every method is idempotent: "already exists" and "already gone" are |
| +successes, because every walk that calls them must be re-runnable (§5). |
| + |
| +## 4. Actors and doors |
| + |
| +- **Member declares**: `carlos store create -app titogo |
| + [-env-prefix TITO_S3]` → `POST /api/cli/apps/{acct}/{app}/store`. |
| + Ordinary member auth (the `apiAccountApp` token + byte-identical-404 |
| + gate). Declaring writes `declared` and provisions nothing. |
| +- **Operator grants**: `carlos store grant -app titogo |
| + [-bucket titogo-uploads]` → operator-gated console endpoint, same |
| + gate class as production-mode flag flips: operator identity + |
| + `requireFreshAuth`, ordered after the owner-404 gate per the console |
| + gate-order rule. The grant handler runs the provisioning walk inline |
| + (§5) — grant and provision are one verb, because a recorded desire |
| + with no acted-on effect is the provisioned-but-not-plumbed gap this |
| + issue exists to close. |
| +- **Console provisions**: the console is the single control-plane actor |
| + holding backend authority (§6). Boxes need zero new authority; |
| + delivery rides existing rails end to end. |
| + |
| +**Why console-provisions beat the alternatives.** A box-side converge |
| +pass would put IAM-admin credentials on every box and need single-actor |
| +election for minting. An operator-run CLI with local admin creds would |
| +make provisioning a manual runbook step again. The console is already |
| +the deployment's write door, already bumps `control/serial`, already |
| +audits, and already holds `CARLOS_SECRETS_PUBKEY` (the safe, |
| +encrypt-only half) for sealing. |
| + |
| +**Why sync-idempotent and not a reconciler loop.** The grant handler |
| +does the backend calls right there; every step is check-then-create, so |
| +a half-failed walk is repaired by running the verb again, and the error |
| +surfaces at the operator's terminal where a human is already standing. |
| +A console reconciler loop would self-heal without a retry, but adds an |
| +always-on loop holding admin authority for a verb an operator runs a |
| +handful of times a year, and moves failures from an exit code to |
| +"check status later". Idempotency gives us retry-to-repair without |
| +either cost. |
| + |
| +## 5. The provisioning walk |
| + |
| +Run by the grant handler, in order, each step check-then-create: |
| + |
| +1. **Resolve the record**: `store.json` must have `declared`; write |
| + `grant` (CAS). |
| +2. **Ensure bucket** — minted name |
| + `<CARLOS_STORE_BUCKET_PREFIX><acct-slug>-<app>` unless `grant.bucket` |
| + names an adopted one. Bucket-per-app, not prefix-per-app: a hard |
| + blast-radius wall per app, teardown is one bucket delete, and the |
| + IAM policy is a two-statement bucket scope instead of the fiddly |
| + per-backend prefix-condition dialect the issue warns about. |
| +3. **Ensure IAM user** under path `/carlos-store/`, inline policy |
| + scoped to exactly that bucket (`s3:ListBucket` on the bucket ARN; |
| + `s3:GetObject/PutObject/DeleteObject` on `arn:…:::<bucket>/*`). |
| +4. **Mint access key.** |
| +5. **Write the env set** into `apps/<acct>/<app>/config.json` through |
| + the existing `appconfig` setters (§7). Coordinates as `kind:"env"`, |
| + both key halves as `kind:"secret"` sealed with |
| + `CARLOS_SECRETS_PUBKEY`. |
| +6. **Write `provisioned`** into `store.json` (CAS). |
| +7. **Audit + `bumpSerial`.** Boxes converge within seconds: config-gen |
| + touch → 2s serial poll → `appconfig.Sync`/`SyncExec` → restart. |
| + |
| +A failure at any step exits non-zero with the step named; re-running |
| +the grant resumes (steps 2–4 find their resources existing, step 5–6 |
| +rewrites are CAS'd). Nothing is provisioned that isn't recorded within |
| +the same walk, and the walk is ordered so the *record of the secret* |
| +(step 5) lands before the record that claims success (step 6). |
| + |
| +## 6. Backend configuration and authority |
| + |
| +Console environment only — the inert-until-configured convention: |
| + |
| +- `CARLOS_STORE_BACKEND` — `aws` is the only accepted value in v1. |
| + Unset: every store verb answers 503 "object storage is not configured |
| + for this deployment". |
| +- `CARLOS_STORE_REGION` — bucket + credential region. |
| +- `CARLOS_STORE_ENDPOINT` — empty for AWS; recorded into |
| + `provisioned.endpoint` and delivered (§7) when set. |
| +- `CARLOS_STORE_BUCKET_PREFIX` — minted-name prefix, e.g. |
| + `carlos-store-tito-`. Required when the backend is set; refusing to |
| + default it keeps two deployments in one AWS account from colliding. |
| + |
| +The console role gains, in pinfra tofu: IAM verbs |
| +(`CreateUser/DeleteUser/PutUserPolicy/DeleteUserPolicy/CreateAccessKey/ |
| +DeleteAccessKey/ListAccessKeys`) restricted to the `/carlos-store/` |
| +user path, and S3 admin verbs |
| +(`CreateBucket/DeleteBucket/ListBucket/DeleteObject`) on |
| +`arn:aws:s3:::<CARLOS_STORE_BUCKET_PREFIX>*` **plus each adopted |
| +bucket's ARN, added explicitly when adopting** (the `titogo-uploads` |
| +grant lands with its ARN in the same tofu change). The role can never |
| +touch users or buckets outside those scopes. |
| + |
| +## 7. Credential delivery: the env surface |
| + |
| +Canonical names, one prefix knob. With `env_prefix` `P` (default |
| +`CARLOS_STORE`), the walk writes: |
| + |
| +| Var | Kind | Value | |
| +|---|---|---| |
| +| `P_BUCKET` | env | provisioned bucket name | |
| +| `P_REGION` | env | provisioned region | |
| +| `P_ENDPOINT` | env | only written when non-empty (non-AWS) | |
| +| `P_ACCESS_KEY` | secret | access-key id, sealed | |
| +| `P_SECRET_KEY` | secret | secret key, sealed | |
| + |
| +titogo sets `env_prefix: "TITO_S3"` and its existing |
| +`TITO_S3_BUCKET/REGION/ACCESS_KEY/SECRET_KEY` (+ optional |
| +`TITO_S3_ENDPOINT`) readers are fed with zero code changes; new apps |
| +get `CARLOS_STORE_*` by default. The access-key id is sealed alongside |
| +the secret — it isn't sensitive alone, but delivering the pair the same |
| +way keeps rotation a single-shaped rewrite. |
| + |
| +These are ordinary `config.json` vars: materialization, restart, |
| +per-route env files, exec spawn-time merge, and purge sweep are all |
| +existing, tested rails, and `carlos env list` shows the store vars |
| +where a member already looks. The trade: a member's |
| +`carlos env unset TITO_S3_BUCKET` breaks their own store wiring until |
| +the next grant/rotate rewrite. v1 accepts that (it is their own app, |
| +and `carlos store status` shows the drift by comparing config against |
| +`provisioned`); var-ownership marking is deliberately not built. |
| + |
| +## 8. Rotation: two keys, explicit finish |
| + |
| +`carlos store rotate -app X` — member-facing (it touches only the |
| +app's own store; the no-infra rule says the member must not need an |
| +operator to cycle a credential): |
| + |
| +1. If `prev_key_id` is pending from an earlier rotate, delete it now |
| + (this is also what makes the AWS two-keys-per-user limit the state |
| + machine: at most one current + one pending-delete). |
| +2. Mint the new key; reseal `P_ACCESS_KEY`/`P_SECRET_KEY` (CAS). |
| +3. `store.json`: `prev_key_id` ← old `key_id`, `key_id` ← new. Audit, |
| + bump serial. |
| + |
| +The old key **stays active** until `carlos store rotate -finish -app X` |
| +deletes it. No guessing about convergence, no window where a |
| +not-yet-restarted unit holds a dead credential; the platform never |
| +claims a rotation completed that it cannot observe. `carlos store |
| +status` and the console card nag while a finish is pending. An |
| +operator who needs a hard cut (leaked key) runs `rotate` then `-finish` |
| +back to back and eats the restart race knowingly. |
| + |
| +## 9. Teardown |
| + |
| +**App purge sweeps the store.** The 30-day trash cooloff is already |
| +the platform's "are you sure"; a purged app keeping a paid, |
| +credentialed bucket is the dangling-estate failure mode this issue |
| +exists to kill. Trash itself (the fuse's start) revokes nothing: routes |
| +stop, data survives until purge, restore keeps the store. |
| + |
| +The split that makes it honest: auto-purge runs **box-side** |
| +(`domains.Reconcile`), and boxes have no backend authority — so: |
| + |
| +- **Box-side `PurgeApp` skips** any app whose `store.json` carries a |
| + `provisioned` block, logging one line per pass naming the console |
| + door as the fix. The app stays trashed and unreachable; only the |
| + final sweep waits. |
| +- **Console "Delete now"** (`handlePurgeApp`) does the full job inline, |
| + before the prefix sweep: `EmptyBucket` → `DeleteBucket` → |
| + `DeleteUser` (keys + policy first), all idempotent, then the normal |
| + `release.PurgeApp`. Teardown failure aborts the purge — a retry |
| + re-enters cleanly — rather than sweeping `store.json` and stranding |
| + an unrecorded bucket. |
| +- **`carlos store destroy -app X`** — operator door, fresh auth, and |
| + the CLI requires typing the bucket name back — tears down the store |
| + while keeping the app: backend teardown, delete the store env vars |
| + from `config.json`, delete `store.json`, bump serial. This is the |
| + undo for adoption mistakes and the migration verb. |
| + |
| +**Ordering rule, stated once**: any path that destroys backend state |
| +reads `store.json` first and deletes it last. A missing or unreadable |
| +record must never be interpreted as "nothing to tear down" on a path |
| +that then deletes the record's prefix — hence the box-side skip. |
| + |
| +## 10. CLI and console surface |
| + |
| +Console API, following the #113 three-layer pattern (handler in |
| +`internal/console`, `apiSession` method in `apimode.go`, verb file |
| +dispatched from `main.go`; API-mode only, direct-bucket mode refused |
| +with a verb-specific message): |
| + |
| +| Endpoint | Gate | Verb | |
| +|---|---|---| |
| +| `POST /api/cli/apps/{acct}/{app}/store` | member | `carlos store create` | |
| +| `GET …/store` | member | `carlos store status` | |
| +| `POST …/store/rotate` | member | `carlos store rotate` | |
| +| `POST …/store/rotate/finish` | member | `carlos store rotate -finish` | |
| +| `POST …/store/grant` | operator + fresh auth | `carlos store grant` | |
| +| `DELETE …/store` | operator + fresh auth | `carlos store destroy` | |
| + |
| +`carlos store status` reports: declared/granted/provisioned state, the |
| +delivered env names, key id + age, pending-finish nag, and drift (a |
| +store var missing from `config.json`). Every write handler audits and |
| +bumps the serial. The console app page gains a read-only **Store** |
| +card (backend, bucket, key age, pending-finish nag); management stays |
| +CLI-first in v1. |
| + |
| +## 11. Testing |
| + |
| +- **Provisioner fake** with per-step failure injection: for each step |
| + of the walk, fail it once, re-run the grant, assert completion and |
| + exactly-one of every resource. Same for teardown and rotate. |
| +- **CAS conflict** on `store.json`: two concurrent writers, second |
| + gets `ErrCASConflict`. |
| +- **Console gate order**: non-member gets the byte-identical 404 on |
| + every store endpoint before any fresh-auth redirect; member cannot |
| + reach grant/destroy. |
| +- **Rotate state machine**: rotate → rotate (pending prev) → finish; |
| + the two-key ceiling is never exceeded. |
| +- **Purge split**: box-side purge skips a provisioned store and logs; |
| + console-side purge tears down before sweeping; teardown failure |
| + leaves `store.json` intact. |
| +- **Minio contract test** (CI, dockerized `minio` + `mc admin` minting |
| + a bucket-scoped user): Put/Get/Delete/List succeed through the |
| + delivered-env client path; a cross-bucket access is denied. This |
| + pins the delivered contract (§1) and doubles as the local dev-loop |
| + story. |
| + |
| +The standard gate applies: `go build ./...`, `go vet ./...`, |
| +`go test ./... -count=1`, `gofmt -l` clean, `GOOS=darwin` compile. |
| + |
| +## 12. Rollout order |
| + |
| +Wire-shape review: `store.json` is a new additive object; the |
| +delivered env vars are ordinary config vars every deployed box already |
| +materializes. One real ordering constraint: |
| + |
| +1. **Platform binaries to both boxes first** — boxes must know the |
| + purge-skip (§9) before any store exists, or an old box's auto-purge |
| + could sweep `store.json` and strand a bucket. (Consistent with the |
| + standing rule: platform binaries first, feature-bearing writes |
| + second.) |
| +2. Console binaries (the only actor that provisions). |
| +3. pinfra tofu: console-role IAM path grant, `titogo-uploads` ARN, |
| + console env (`CARLOS_STORE_*`) on tito. |
| +4. `carlos store create -app titogo -env-prefix TITO_S3` (member) and |
| + `carlos store grant -app titogo -bucket titogo-uploads` (operator) |
| + on the tito deployment; verify uploads end-to-end in titogo. |
| + |
| +## 13. Non-goals (v1) |
| + |
| +- **Per-instance scoped credentials** — the contract is worded |
| + per-credential so they can arrive later without app changes; the |
| + mechanics (per-prefix policy dialects, credential-estate growth) are |
| + explicitly deferred. |
| +- **Hetzner / R2 / Minio provisioning** — the `Provisioner` seam and |
| + the Minio contract test mark the door; no second implementation |
| + ships in v1. |
| +- **Per-environment stores** — one store per app per deployment; |
| + `environments/<name>.json` overlays can override the env vars by |
| + hand if someone truly needs a split before the primitive grows one. |
| +- **Data-plane anything** — no proxy, no signed-URL service, no |
| + metering/quotas. |
| +- **Var-ownership marking** in `config.json` (§7 trade accepted). |
| +- **Migrating the tito litestream replica estate** — existing |
| + hand-made replica buckets stay grandfathered; only `titogo-uploads` |
| + is adopted. |