| 1 | # A store per environment: object storage an instance can own |
| 2 | |
| 3 | **Date:** 2026-09-19 |
| 4 | **Status:** approved in direction (James, 2026-09-19). Two **(ruling)**s were |
| 5 | taken in that conversation; everything else is a **(decision)** and can be |
| 6 | overturned cheaply. |
| 7 | **Discussion:** amadan `carlos/platform` #159. |
| 8 | **Extends:** `2026-09-02-object-storage-self-serve-design.md` (the account |
| 9 | entitlement) and the object-storage primitive it builds on. |
| 10 | |
| 11 | ## 1. The problem, as found |
| 12 | |
| 13 | `store.json` is per app. `MintedBucket` is `prefix + acct + "-" + app` and |
| 14 | `UserName` is `"carlos-store-" + acct + "-" + app` |
| 15 | (`internal/objstore/walk.go:62,69`), and `writeEnv` delivers the coordinates |
| 16 | through `appconfig.Apply` (`walk.go:689`) — the app's **default** bundle. So |
| 17 | every instance of an app shares one bucket and one credential, and there is |
| 18 | no way to give one instance its own. |
| 19 | |
| 20 | That is correct for an app whose instances are one deployment. It is the |
| 21 | blocker for a managed, multi-tenant product, where one app has one instance |
| 22 | per customer: the only options are one bucket with per-tenant key prefixes, |
| 23 | where isolation becomes app logic and one path bug crosses tenants, or the |
| 24 | app bringing its own S3 and re-implementing what the platform already does. |
| 25 | |
| 26 | ### What is already right, and must stay right |
| 27 | |
| 28 | - **The entitlement is per ACCOUNT, not per app** (`store/entitlements/<sqid>.json`). |
| 29 | Approval is already orthogonal to whatever unit owns a store, so nothing |
| 30 | here adds an approval step. |
| 31 | - **The walks are idempotent and resumable**, and their lost-CAS retries are |
| 32 | guarded on the key the walk started from — the one shape the walk comments |
| 33 | call "NOT survivable" (`walk.go:258-289`). |
| 34 | - **The record's three null-or-present blocks** are what make "declared", |
| 35 | "granted" and "provisioned" distinguishable. |
| 36 | - **`release.PurgeApp` hand-parses `grant`/`provisioned`** out of the store |
| 37 | record and refuses an app's auto-purge while either is present |
| 38 | (`internal/release/trash.go:202-221`). The field names are frozen. |
| 39 | - **The per-app walk lock exists** (`internal/console/storelock.go`). |
| 40 | |
| 41 | ## 2. The shape, in one paragraph |
| 42 | |
| 43 | A store may belong to a **named config environment** instead of to the app as |
| 44 | a whole. `carlos store create --app a --environment e` declares one; the walk |
| 45 | mints a bucket and an IAM user named for `(account, app, environment)` and |
| 46 | delivers the coordinates into **that environment's bundle**, which is the |
| 47 | bundle the instances bound to it already read. An instance gets its own |
| 48 | bucket by being the only instance bound to its environment. Nothing on a box |
| 49 | changes. |
| 50 | |
| 51 | ## 3. Rulings |
| 52 | |
| 53 | - **(ruling, James 2026-09-19) A store is keyed on `(account, app, |
| 54 | environment)`, not on a route host.** Config delivery already resolves per |
| 55 | `(account, app, environment)` (`appconfig.LoadMerged`, `syncexec.go`'s |
| 56 | `groupKey`); there is no per-host config source, and adding one means a new |
| 57 | precedence layer inside the box-side sync that every route's config passes |
| 58 | through. Keying on the environment reuses the delivery axis exactly as it |
| 59 | is. |
| 60 | - **(ruling, James 2026-09-19) Teardown stays an explicit verb.** Deleting an |
| 61 | instance or abandoning an environment never destroys a bucket. |
| 62 | `provision.DeleteRecord` already "deliberately leaves the parked database |
| 63 | and blobs untouched", and `objstore.Teardown` is only ever called from a |
| 64 | door a person drove. Nothing in this change becomes the platform's first |
| 65 | automatic destroyer of tenant data. §8 covers what stops an abandoned |
| 66 | store from billing silently. |
| 67 | |
| 68 | ## 4. Wire shapes |
| 69 | |
| 70 | **(decision) The default store does not move.** With no `--environment`, every |
| 71 | key, name and delivery target is byte-for-byte what it is today: |
| 72 | `apps/<acct>/<app>/store.json`, bucket `<prefix><acct>-<app>`, user |
| 73 | `carlos-store-<acct>-<app>`, delivered to `config.json`. Existing records, |
| 74 | buckets and credentials are untouched, and every existing test stays green |
| 75 | unchanged. |
| 76 | |
| 77 | **(decision) An environment store is a sibling collection.** |
| 78 | |
| 79 | apps/<acct>/<app>/store.json the app's own store (unchanged) |
| 80 | apps/<acct>/<app>/stores/<env>.json one per named environment |
| 81 | |
| 82 | Nothing in the bucket nests keys under an instance today; the nearest |
| 83 | precedent is `apps/<sqid>/<app>/sidecars/<name>.json`, a flat sibling |
| 84 | collection, and this follows it. |
| 85 | |
| 86 | **(decision) The record shape is `objstore.Record`, unchanged**, with one |
| 87 | additive field: |
| 88 | |
| 89 | ```go |
| 90 | // Environment is the named config bundle this store belongs to. "" is the |
| 91 | // app's own store — the only shape that existed before 2026-09-19 — and is |
| 92 | // omitted from the JSON, so every record written before this field existed |
| 93 | // parses as the app store it is. |
| 94 | Environment string `json:"environment,omitempty"` |
| 95 | ``` |
| 96 | |
| 97 | Like `App`, it is stamped by `Put` from the key it is written to and never |
| 98 | trusted from a caller: a record that lies about its own identity is refused, |
| 99 | not guessed at. |
| 100 | |
| 101 | **(decision) Names gain the environment as a third component.** |
| 102 | |
| 103 | MintedBucket: <prefix><acct>-<app>-<env> |
| 104 | UserName: carlos-store-<acct>-<app>-<env> |
| 105 | |
| 106 | > **Implementation gate.** Both are bounded before this lands. S3 bucket |
| 107 | > names are at most 63 characters and IAM user names at most 64; account, |
| 108 | > app and environment names are each already bounded by their own |
| 109 | > validators. The walk must refuse a combination that cannot be named, |
| 110 | > **before** any backend call, with a sentence naming which part is too |
| 111 | > long — never a half-provisioned estate discovered at `CreateBucket`. |
| 112 | |
| 113 | **(decision) Delivery targets the environment's bundle.** `writeEnv` calls |
| 114 | `appconfig.ApplyEnvironment(..., env, ...)` when `Environment != ""`, and |
| 115 | `appconfig.Apply` otherwise. The variable names are unchanged |
| 116 | (`<PREFIX>_BUCKET`, `_REGION`, `_ENDPOINT`, `_ACCESS_KEY`, `_SECRET_KEY`), |
| 117 | and the two key halves are still sealed before they are written. |
| 118 | |
| 119 | Because a named bundle overlays the default **per key**, an app with both an |
| 120 | app store and an environment store delivers the environment's coordinates to |
| 121 | instances bound to it and the app's to everyone else. That is the overlay |
| 122 | doing exactly what it already does. |
| 123 | |
| 124 | ## 5. What a member sees |
| 125 | |
| 126 | ``` |
| 127 | carlos store create --app a --environment e [--env-prefix P] |
| 128 | carlos store status --app a [--environment e] |
| 129 | carlos store grant --app a --environment e (operator, unchanged gates) |
| 130 | carlos store rotate --app a --environment e [--finish] |
| 131 | ``` |
| 132 | |
| 133 | **(ruling, James 2026-09-21) There is no `carlos store destroy`.** An earlier |
| 134 | draft of this section listed one. Destroying a store stays what it already |
| 135 | was: a console action behind the heaviest ceremony in the platform — owner |
| 136 | role, CSRF, the bucket name typed exactly, and fresh passkey auth. A CLI verb |
| 137 | cannot ask for a passkey, so adding one would have quietly traded that |
| 138 | ceremony for a bearer token on the one action with no trash and no undo. |
| 139 | |
| 140 | The console's destroy door takes the environment instead (§8), and the |
| 141 | settings page grows one card per environment store, each with its own |
| 142 | confirm-and-destroy form. |
| 143 | |
| 144 | **(decision) `status` with no `--environment` lists every store the app has**, |
| 145 | the app's own first, then one line per environment: its bucket, its state, |
| 146 | and which instances read it. An app with only the default store prints what |
| 147 | it prints today. |
| 148 | |
| 149 | **(decision) The doors refuse what cannot be delivered.** A store for an |
| 150 | environment name that fails `appconfig.ValidEnvironmentName` is refused. An |
| 151 | environment bundle that does not exist yet is **not** refused: delivery |
| 152 | creates it, matching "an environment does not require a default to exist". |
| 153 | |
| 154 | **(decision) Sharing is visible, not refused.** Two instances bound to one |
| 155 | environment share its store, exactly as they share its config. `status` says |
| 156 | which instances read each store, so a shared bucket is something you can see |
| 157 | rather than something you discover. |
| 158 | |
| 159 | ## 6. Authority |
| 160 | |
| 161 | Unchanged in every respect. `apiAccountApp` first, then the operator |
| 162 | allowlist for `grant`, then the write budget, then the backend gates. The |
| 163 | account entitlement remains the single approval: **in an approved account, |
| 164 | creating an environment store provisions it with no new approval**, which is |
| 165 | what lets a signup path mint one without an operator in the loop. |
| 166 | |
| 167 | **(decision) The walk lock key gains the environment** — |
| 168 | `acct + "/" + app + "/" + env`. Two stores of one app share no estate: their |
| 169 | buckets, users and delivered bundles are all distinct, so serialising across |
| 170 | them would cost concurrency and buy nothing. That is the same argument |
| 171 | `storelock.go` already makes for two apps in one account. |
| 172 | |
| 173 | ## 7. The purge guard |
| 174 | |
| 175 | `release.PurgeApp` today does one `Get` of `store.json` and refuses the |
| 176 | purge while `grant` or `provisioned` is present. With environment stores it |
| 177 | must also **list `apps/<acct>/<app>/stores/`** and apply the same test to |
| 178 | every record there. |
| 179 | |
| 180 | **This is the one change in this spec that is a safety property rather than a |
| 181 | feature.** A miss means an app is auto-purged while a bucket, an IAM user and |
| 182 | a live access key survive with nothing naming them. It lands in the first |
| 183 | slice, with the test that proves it, and `release` keeps hand-parsing the two |
| 184 | frozen field names rather than importing `objstore`. |
| 185 | |
| 186 | ## 8. Metering, and the abandoned store |
| 187 | |
| 188 | **(decision) v1 sums every store of an app into the app's existing usage |
| 189 | key.** `storagePass` gets one `Get` per app today and writes |
| 190 | `day.Apps[acct+"/"+app]`. It will instead read the app store plus every |
| 191 | record under `stores/`, size each, and sum. The `StorageDay` wire shape does |
| 192 | not change, so nothing downstream of usage has to move, and an account's bill |
| 193 | is the same arithmetic over more buckets. |
| 194 | |
| 195 | Per-store attribution — what discussion #140 asks for — is a later, additive |
| 196 | change to the usage shape, and this spec deliberately does not make it. |
| 197 | |
| 198 | **Because teardown is explicit (§3), an abandoned environment store keeps |
| 199 | billing.** Three things make that visible rather than silent: |
| 200 | |
| 201 | - `store status` lists every store of the app, including ones no instance |
| 202 | reads. An orphan is on the screen. |
| 203 | - A store whose environment no bundle and no instance references is named in |
| 204 | `status` as **unused**, in those words. |
| 205 | - The purge guard (§7) refuses to auto-purge the app while any store is |
| 206 | provisioned, so the estate cannot be forgotten by being trashed. |
| 207 | |
| 208 | ## 9. Scanning |
| 209 | |
| 210 | **(decision) v1 scans the app's own store only.** `Sweep` is |
| 211 | `(acct, app)`-scoped and keeps one cursor per app |
| 212 | (`internal/objstore/sweep.go`), and the scan-result namespace is the same |
| 213 | pair. Extending both is real work with no bearing on whether a tenant can |
| 214 | have its own bucket. |
| 215 | |
| 216 | What this spec will not do is let the surface lie about it: `store status` on |
| 217 | an environment store says scanning is not available for it yet, and the scan |
| 218 | doors refuse `--environment` with that sentence rather than silently scanning |
| 219 | the wrong bucket or claiming a clean result. |
| 220 | |
| 221 | ## 10. Slices |
| 222 | |
| 223 | 1. **The primitive.** ✅ `Environment` on the record, the key layout, the name |
| 224 | builders and their length refusal, `writeEnv`'s delivery target, and the |
| 225 | walks threaded through. Plus §7's purge guard and its test. |
| 226 | 2. **The doors.** ✅ `?environment=` on create/status/grant/rotate/finish, the |
| 227 | lock key, `storeReply` carrying the environment, and the refusals of §5. |
| 228 | 3. **The CLI and the docs.** ✅ `--environment` on create/status/grant/rotate, |
| 229 | `status` listing every store, and `docs/site/cli.md`. |
| 230 | 4. **Teardown.** ✅ The account-teardown path covering every store, and the |
| 231 | console destroy door taking an environment — with one card and one |
| 232 | confirm-and-destroy form per environment store on the settings page. No |
| 233 | CLI verb, per §5's ruling. |
| 234 | 5. **Metering.** ✅ §8's sum, over the app's own store and every environment |
| 235 | store, into the one existing usage key. |
| 236 | |
| 237 | **Not built, and named here so it is not mistaken for shipped: §8's "unused" |
| 238 | line.** `store status` lists every environment that has a store, which is |
| 239 | where an abandoned one becomes visible — but it does not say whether any |
| 240 | INSTANCE is bound to that environment, which is what "unused" claims. Doing |
| 241 | it properly means reading the app's instance records and joining on their |
| 242 | environment binding, and a line that said "unused" from anything less would |
| 243 | be a guess about a live bucket. The listing is the honest half; the join is |
| 244 | follow-up work. |
| 245 | |
| 246 | ## 11. Non-goals |
| 247 | |
| 248 | - Per-host stores. §3's ruling. |
| 249 | - Automatic teardown of anything. §3's ruling. |
| 250 | - Per-store usage attribution (#140). |
| 251 | - Scanning environment stores (§9). |
| 252 | - Any change to the account entitlement, its approval queue or its |
| 253 | notifications. |
| 254 | |
amadan