| 1 | # Private storage per instance: policy on the app, provisioning by the platform |
| 2 | |
| 3 | **Date:** 2026-09-22 |
| 4 | **Status:** implemented in this branch. Scope confirmed with James on |
| 5 | 2026-09-22: §5 (the policy), §6 (the derived name), §7 (provisioning at |
| 6 | create) and §8 (the ceiling). §10's two items are NOT built and are named |
| 7 | there so they are not mistaken for shipped. |
| 8 | **Issue:** GitHub #412 (a better answer to #410, which this makes |
| 9 | unnecessary). |
| 10 | **Builds on:** `2026-09-19-per-environment-store-design.md` (#408) — the |
| 11 | per-environment store primitive this spends. Nothing here works without it. |
| 12 | |
| 13 | ## 1. The problem, as found |
| 14 | |
| 15 | #408 gave a store an environment to belong to. It did not give anyone a way |
| 16 | to reach one without a person. |
| 17 | |
| 18 | Creating an environment store is three acts: `carlos store create --app a |
| 19 | --environment e`, then a provisioning walk, then binding an instance record |
| 20 | to `e`. The first two run behind `apiAccountApp` plus the account |
| 21 | entitlement, and the CLI verb that drives them is a member's. An |
| 22 | `instance`-role service credential — the one a public signup path can hold — |
| 23 | can create an instance record and nothing else. |
| 24 | |
| 25 | So every signup on Managed Jelly (lazyatom/jam#187) stops for a person to |
| 26 | run one command. #410 asked for the fix that widens the role. This is the |
| 27 | fix that widens nobody: **the human declares the policy once, on the app, |
| 28 | and the create door does the rest under that standing declaration.** |
| 29 | |
| 30 | ### What is already right, and must stay right |
| 31 | |
| 32 | - **The entitlement is per ACCOUNT** (`store/entitlements/<sqid>.json`), and |
| 33 | #408 §6 already says that in an approved account, creating an environment |
| 34 | store provisions it with no new approval. That sentence is the whole |
| 35 | authority story here; this spec adds no new one. |
| 36 | - **A store is keyed on `(account, app, environment)`** (#408 §3, ruling). |
| 37 | Deriving the environment from the host at create time keeps that key |
| 38 | exactly and adds no per-host config source. |
| 39 | - **Teardown is explicit** (#408 §3, ruling). Nothing here destroys |
| 40 | anything, and §10 says what that costs. |
| 41 | - **The create door's refusal order** — gate, budget, manifest, host, |
| 42 | region, channel, write — and its byte-identical 404 before membership is |
| 43 | proven. Everything this spec adds sits AFTER the manifest read, so a |
| 44 | non-member still cannot learn that an app carries the policy. |
| 45 | |
| 46 | ## 2. The shape, in one paragraph |
| 47 | |
| 48 | An app may declare that **every instance it creates gets a config |
| 49 | environment of its own, with an object store bound to that environment**. |
| 50 | The declaration is a bool on the instances manifest, written by |
| 51 | `carlos instances enable --app a --private-storage` — a signed-in person's |
| 52 | act, which no service credential can perform. From then on `POST |
| 53 | …/instances` derives the environment from the host's first label, provisions |
| 54 | that environment's store under the account's existing entitlement, and binds |
| 55 | the record to it. An `instance`-role credential gains nothing it could not |
| 56 | already do; it simply lands in a world where the app's policy has already |
| 57 | been decided. |
| 58 | |
| 59 | ## 3. Rulings |
| 60 | |
| 61 | - **(ruling, James 2026-09-22) The policy is declared on the app, and only a |
| 62 | person may declare it.** `handleAPIInstanceEnable` is registered with the |
| 63 | zero Set and gated by `apiPerson`; that is the wall, and it is the same |
| 64 | wall that makes this a better answer than #410. |
| 65 | - **(ruling, James 2026-09-22) Scope is the policy, the derived name and the |
| 66 | ceiling.** The residue listing (#412's point 2) and holding an instance's |
| 67 | first wake until its store is delivered (#412's point 4) are follow-up |
| 68 | work, named in §10. |
| 69 | |
| 70 | ## 4. Non-goals |
| 71 | |
| 72 | - Automatic teardown of anything (#408 §3's ruling, unchanged). |
| 73 | - A residue listing for deleted instances (#412 point 2). §10. |
| 74 | - Gating an instance's first wake on delivery (#412 point 4). §10. |
| 75 | - Per-store usage attribution, scanning environment stores — #408's own |
| 76 | non-goals, unchanged. |
| 77 | - An account-wide (rather than per-app) store ceiling. §8 says why the cap |
| 78 | is shaped per app, what that leaves unbounded, and what a real one costs. |
| 79 | - Per-instance secrets. The derived environment is the primitive they would |
| 80 | hang off; this spec builds the primitive and stops. |
| 81 | |
| 82 | ## 5. The policy |
| 83 | |
| 84 | **(decision) One additive bool on `provision.Manifest`.** |
| 85 | |
| 86 | ```go |
| 87 | // PrivateStorage is the app's standing policy that every instance created |
| 88 | // under this manifest gets a config environment of its own and an object |
| 89 | // store bound to it. Absent/false is what every manifest in a production |
| 90 | // bucket says today, and means exactly today's behaviour. |
| 91 | PrivateStorage bool `json:"private_storage,omitempty"` |
| 92 | ``` |
| 93 | |
| 94 | A plain bool, not a pointer, and the difference from `Record.Hibernate` and |
| 95 | `Record.StatusNamed` is worth stating because those two are pointers for a |
| 96 | reason that does not apply here. Those fields are CONVERGED — a box reads |
| 97 | them every pass, so "absent" has to be distinguishable from "an explicit |
| 98 | false" or the first pass after deploy silently acts on a value nobody wrote. |
| 99 | This field is read at one door, at create time, and false IS the legacy |
| 100 | behaviour byte for byte. There is nothing for absence to mean that false |
| 101 | does not already mean. |
| 102 | |
| 103 | **(decision) The enable door takes it as a POINTER on the wire** — |
| 104 | `private_storage` absent preserves what the manifest carries, present |
| 105 | replaces it — matching `channel` and `seed`, which are pointers on that |
| 106 | door for exactly this reason. `carlos instances enable --private-storage` |
| 107 | sends true; `--private-storage=false` sends false; omitting the flag sends |
| 108 | nothing. |
| 109 | |
| 110 | **(decision) The door refuses a policy it knows cannot be honoured.** Two |
| 111 | checks, both before the manifest is written: |
| 112 | |
| 113 | - The deployment has no object-storage backend (`CARLOS_STORE_BACKEND` |
| 114 | unset) → 503 naming the variable. |
| 115 | - The account's entitlement is not approved → 403 naming `carlos store |
| 116 | apply`. |
| 117 | |
| 118 | This is a departure from the enable door's habit of writing whatever policy |
| 119 | it is handed, and it is deliberate. The policy's entire purpose is that |
| 120 | creates afterwards succeed with no person in the loop. A manifest that |
| 121 | guarantees every future create fails is not policy, it is a trap that fires |
| 122 | once per signup, and the two sentences above are the only two that name the |
| 123 | fix. Turning the policy OFF is never refused — an account whose entitlement |
| 124 | was later declined must be able to get back to a working create door. |
| 125 | |
| 126 | ## 6. The derived name |
| 127 | |
| 128 | **(decision) The environment is the host's first label** — `Record.Slug`, |
| 129 | which the create door already computes as `strings.TrimSuffix(host, "." + |
| 130 | m.Domain)`. It is the one name that is predictable from the host, already |
| 131 | unique within the app (the record key is the host), and readable in `store |
| 132 | status` and the console's Environments view at a hundred of them, which is |
| 133 | #412's point 1 in full. |
| 134 | |
| 135 | **(decision) An explicit `environment` in the request body wins.** The |
| 136 | policy fills a blank; it does not seize the axis. A caller who names an |
| 137 | environment gets a store for THAT environment — which is still the |
| 138 | invariant this spec exists for, since what is promised is that the record's |
| 139 | bundle carries `CARLOS_STORE_*`, not that the bundle has any particular |
| 140 | name. It keeps `carlos canary` working on a private-storage app, and two |
| 141 | instances naming one environment share its store exactly as they share its |
| 142 | config, which #408 §5 already rules is visible rather than refused. |
| 143 | |
| 144 | **(decision) Three names are checked BEFORE anything is written**, because |
| 145 | a DNS label is not an environment name and an environment name is not |
| 146 | always spendable: |
| 147 | |
| 148 | | check | why it can fail | |
| 149 | | --- | --- | |
| 150 | | `appconfig.ValidEnvironmentName(slug)` | a label may be 63 characters, an environment name 32; and `default` is a legal label but names the app's own bundle | |
| 151 | | `objstore.ValidBucketName(MintedBucketIn(...))` | S3's 63 | |
| 152 | | `objstore.ValidUserName(UserNameIn(...))` | IAM's 64 | |
| 153 | |
| 154 | Each refuses with 400 and a sentence naming the part the caller can |
| 155 | shorten. The walk checks the last two itself (#408's implementation gate), |
| 156 | but reaching them there would mean an instance record already written over |
| 157 | an estate that cannot exist. |
| 158 | |
| 159 | ## 7. Provisioning, and the order it happens in |
| 160 | |
| 161 | **(ruling, James 2026-09-22 — implied by the scope choice, stated here |
| 162 | because it is the load-bearing decision of this file) The store is |
| 163 | provisioned BEFORE the record is written, and a provisioning failure means |
| 164 | no record at all.** |
| 165 | |
| 166 | The two orders fail differently, and only one of them fails visibly: |
| 167 | |
| 168 | - *Record first.* Provisioning then fails, and an instance exists whose |
| 169 | bundle has no `CARLOS_STORE_*`. It wakes, finds no coordinates, and runs |
| 170 | on local disk — which is the exact outcome the policy was declared to |
| 171 | prevent, arrived at silently, with no verb to repair it (there is no |
| 172 | `carlos instances create --resume`). |
| 173 | - *Store first.* The record write then fails, and a declared-or-provisioned |
| 174 | store exists for an environment no instance is bound to. It is on the |
| 175 | screen in `store status`, and — because the environment is derived from |
| 176 | the host — **a retry of the same host reuses it exactly**: `GrantIn` is |
| 177 | idempotent, so the second attempt short-circuits and costs nothing. |
| 178 | |
| 179 | So the property this spec asserts is: **no instance record is ever created |
| 180 | under this policy without its store already provisioned and delivered.** |
| 181 | That is stated as such because it is the whole feature; a create that |
| 182 | half-succeeded would be worse than one that refused. |
| 183 | |
| 184 | One window stays open and is named rather than papered over. The create door |
| 185 | already writes the record and then re-reads the sibling-apex check, deleting |
| 186 | the record if a sibling app claimed the label mid-create. On that path the |
| 187 | store survives the record. It is logged in those words, beside the existing |
| 188 | `UNDO FAILED` line, because the alternative — tearing down a bucket from a |
| 189 | code path nobody drove — would make this the platform's first automatic |
| 190 | destroyer of tenant data, which #408 §3 rules out. |
| 191 | |
| 192 | **(decision) The walk runs exactly as `handleAPIStoreProvision` runs it**: |
| 193 | the per-store walk lock (`storeWalks.lock(sqid, app, env)`), the seal key, |
| 194 | `storeGrantTimeout`, `GrantSourceAccountApproval`, and the entitlement |
| 195 | re-read after the lock. Not a second implementation of provisioning — the |
| 196 | same calls in the same order, factored into one helper both doors use, so |
| 197 | the two cannot drift. |
| 198 | |
| 199 | **(decision) The env prefix is inherited from the app's own store.** A |
| 200 | private-storage app that already declared `carlos store create --env-prefix |
| 201 | JELLY_S3` reads those names in its code; an instance store delivering |
| 202 | `CARLOS_STORE_BUCKET` into the same app would be a second vocabulary for one |
| 203 | thing. With no app store declared, the default `CARLOS_STORE` prefix is |
| 204 | taken, which is what a fresh declaration would take. |
| 205 | |
| 206 | ## 8. The ceiling |
| 207 | |
| 208 | **(decision) A cap on how many ENVIRONMENT stores one app may have**, |
| 209 | `CARLOS_STORE_MAX_PER_APP`, default 1000. A bucket plus an IAM user per |
| 210 | instance is a bigger footprint than an instance, and the binding limit is |
| 211 | IAM's: 5,000 users per cloud account, an adjustable quota |
| 212 | ([AWS service quotas](https://docs.aws.amazon.com/general/latest/gr/iam-service.html)). |
| 213 | A signup path that can mint them needs a bar it stops at. |
| 214 | |
| 215 | **(correction, James 2026-09-23) The first draft of this section said 100, |
| 216 | and that was a number picked for tidiness rather than derived from the |
| 217 | quota.** A fifth of one app's fair share would have refused an ordinary |
| 218 | managed product at its hundredth customer — a real refusal in exchange for |
| 219 | no real protection. 1000 is a fifth of the account quota, so one app can run |
| 220 | away without exhausting it while five cannot. |
| 221 | |
| 222 | **What the cap does not bound**, stated because the per-app shape invites the |
| 223 | wrong reading: five apps at the default reach 5,000 between them, and the |
| 224 | deployment's own IAM principals are not in that sum. This is a brake on one |
| 225 | runaway signup path, not a budget for the cloud account. An account-wide |
| 226 | bound needs a counter kept on the side rather than one derived by listing, |
| 227 | and §4 lists it as a non-goal. |
| 228 | |
| 229 | Per app, not per account, for a reason of cost: the count comes from one |
| 230 | `objstore.StoreEnvironments` prefix listing of the app, which the create |
| 231 | door can afford. A deployment-wide count would mean listing every app's |
| 232 | stores on every create. |
| 233 | |
| 234 | Two properties: |
| 235 | |
| 236 | - **It is enforced on both doors**, `POST …/instances` and `POST |
| 237 | …/store?environment=`. A ceiling only the automatic path respected would |
| 238 | be a ceiling anyone could step over by hand. |
| 239 | - **Re-using an environment that already has a store is never capped.** The |
| 240 | cap is on the FOOTPRINT, and a second instance bound to an existing |
| 241 | environment adds no bucket. Without this, a retry of a create that failed |
| 242 | after provisioning would be refused by the cap it had just reached. |
| 243 | |
| 244 | The refusal is a 409 naming the count and the cap. |
| 245 | |
| 246 | ## 9. What a member sees |
| 247 | |
| 248 | ``` |
| 249 | carlos instances enable --app a --private-storage |
| 250 | carlos instances create --app a --host acme.a.<sqid>.<apps-domain> |
| 251 | created instance acme.a.… (channel stable, environment acme) |
| 252 | private storage: bucket carlos-bab-a-acme |
| 253 | ``` |
| 254 | |
| 255 | and `carlos store status --app a` lists the acme store beside every other, |
| 256 | which is #408 §5's listing doing its job unchanged. |
| 257 | |
| 258 | ## 10. Not built, and named so it is not mistaken for shipped |
| 259 | |
| 260 | - **#412's point 2, the residue listing.** `instances delete` leaves the |
| 261 | parked database and blobs today, and after this change it also leaves an |
| 262 | environment bundle and a store. The issue says this "probably should not |
| 263 | ship without at least a way to list what deleted instances left behind", |
| 264 | and that judgement stands: the residue is now created automatically, so it |
| 265 | accrues once per cancelled customer. What exists in the meantime is `store |
| 266 | status`, which lists every store of the app including ones no instance |
| 267 | reads — the honest half of #408 §8, and enough to find an orphan if you go |
| 268 | looking. The follow-up is a verb that does the looking. |
| 269 | - **#412's point 4, holding the wake.** The provisioning walk completes |
| 270 | before the record is written (§7), so the coordinates are in the bundle |
| 271 | before the record a box could reconcile from exists at all. That is |
| 272 | strictly better than the manager's "refuse the seed until the bucket is |
| 273 | seen" workaround, and it is not the same as a guarantee: delivery is a |
| 274 | config-bundle write and the box's sync pass is what carries it to the |
| 275 | instance, so a wake racing the first sync can still find no coordinates. |
| 276 | Closing that needs the box side, and it is not in this branch. |
| 277 | |
amadan