| 1 | # Self-serve object storage: an account applies, an operator approves |
| 2 | |
| 3 | **Date:** 2026-09-02 (revised 2026-09-03 after review) |
| 4 | **Status:** approved in principle (Paul, 2026-09-02, in the brainstorm this |
| 5 | document records). Four decisions were taken there and are marked **(ruling)**; |
| 6 | everything else is marked **(decision)** and can be overturned cheaply. |
| 7 | |
| 8 | **Issue:** #381. Extends the object-storage primitive (#55, spec |
| 9 | `2026-08-10-object-storage-primitive-design.md`, referenced below as *the |
| 10 | primitive spec*). |
| 11 | |
| 12 | **Revision note.** A review on 2026-09-03 found three blockers in the first |
| 13 | draft, all verified in code. §7 described a failure state the walk does not |
| 14 | produce; §5 inherited a membership requirement that would deadlock approval; |
| 15 | and §9 turned a latent concurrency bug into an everyday one. All three are |
| 16 | fixed below, and §9 now carries work this spec did not originally budget for. |
| 17 | |
| 18 | ## 1. The problem, as found |
| 19 | |
| 20 | On 2026-09-02 James reported that carloku had no object storage. He was right, |
| 21 | and the way he came to report it is the bug. |
| 22 | |
| 23 | `james@keymail.dev` (account `bac`, app `jam`) declared a store on 2026-08-29 |
| 24 | under the default `CARLOS_STORE` prefix. `carlos store status` told him the |
| 25 | truth — the declaration is recorded, and this deployment has no backend |
| 26 | configured — and there his information ended. Four days later he raised it in |
| 27 | conversation, out of band, to the one person who could act. |
| 28 | |
| 29 | Two separate operator-side steps stood between him and a bucket, and he could |
| 30 | act on neither: |
| 31 | |
| 32 | 1. **The deployment had no backend at all.** `CARLOS_STORE_BACKEND`, plus the |
| 33 | `CARLOS_STORE_REGION` and `CARLOS_STORE_BUCKET_PREFIX` that |
| 34 | `storeBackend()` makes fatal at boot if the first is set without them |
| 35 | (`cmd/console/main.go:492-524`), live only in the console's root env. That |
| 36 | is #368's problem in its most expensive instance: the primitive has shipped |
| 37 | since #55 and the flagship has never had it switched on, because switching it |
| 38 | on is not a `carlos` verb. |
| 39 | 2. **Every grant is a manual operator verb.** Even with a backend, each app |
| 40 | needs a person on `CARLOS_STORE_OPERATORS` to run `carlos store grant --app |
| 41 | <a>`, once, per app, forever. |
| 42 | |
| 43 | **A third blocker, found 2026-09-08 while trying to switch the flagship on for |
| 44 | another account.** Setting the three env vars is necessary but not sufficient: |
| 45 | the console's own instance role cannot provision. `region-host`'s policy grants |
| 46 | Get/Put/Delete on the deployment and data buckets plus a read-only Describe/List |
| 47 | set for the architecture page (`infra/modules/region-host/main.tf:76-140`), |
| 48 | while the provisioner calls `s3:CreateBucket`/`DeleteBucket` and the whole |
| 49 | `iam:CreateUser`/`PutUserPolicy`/`CreateAccessKey` lifecycle |
| 50 | (`internal/objstore/aws.go:31-46`). A deployment that sets |
| 51 | `CARLOS_STORE_BACKEND` and nothing else fails at its FIRST grant, deep in the |
| 52 | walk, against live AWS — the worst place to discover a missing policy. Flagship |
| 53 | consumes the module from `platform.git//infra/modules/region-host?ref=main`, so |
| 54 | that policy lands in this repo before any deployment can be switched on. |
| 55 | |
| 56 | > **The grant carries a privilege escalation that must be closed with it.** |
| 57 | > `EnsureUser` sets no permissions boundary (`aws.go:282-310`), so a role |
| 58 | > holding `iam:CreateUser` and `iam:PutUserPolicy` scoped only by the |
| 59 | > `/carlos-store/` path can mint a principal with any policy at all — a path |
| 60 | > from the console box to the whole deployment account. The module should ship |
| 61 | > a boundary policy (S3 only, scoped to the bucket prefix), condition both |
| 62 | > actions on `iam:PermissionsBoundary`, and `CreateUserInput` should carry it. |
| 63 | |
| 64 | This spec fixes (2) and makes (1) visible. (1)'s actual fix is #368 and is a |
| 65 | non-goal here (§12). |
| 66 | |
| 67 | ### What is already right, and must stay right |
| 68 | |
| 69 | **The record's shape is right.** Three blocks for three actors, `null`-or-present, |
| 70 | is what makes "declared, waiting" distinguishable from "provisioned" |
| 71 | (`internal/objstore/objstore.go`, `Record`'s doc comment). |
| 72 | |
| 73 | **The status door already answers the right question.** `backend` is in the |
| 74 | reply *specifically* so a member can tell "queued behind an operator" from |
| 75 | "this deployment has no object storage and never will" |
| 76 | (`internal/console/apistore.go:105-112`). That is this spec's reasoning, one |
| 77 | level up. |
| 78 | |
| 79 | **The metering is already built, and this spec adds almost none.** The economics |
| 80 | pass walks every app's store record daily, sizes the provisioned bucket |
| 81 | (`sizeProvisionedStore`, `internal/console/economicspass.go:990`), records |
| 82 | `StoreBytes` and distinguishes "could not measure" from "zero" with |
| 83 | `StoreUnswept`. That rolls into `MeterStorageGBMonth` on |
| 84 | `usage/accounts/<acct>/<month>.json`, which `PriceAccount` already prices at |
| 85 | `StorageGBMonthUSD` against a free tier defaulting to 1 GB |
| 86 | (`internal/economics/scenario.go:162`). **"Unlimited but metered" is a |
| 87 | statement about what already happens.** When the credit card arrives it reads a |
| 88 | statement written every day since. (§11 adds one additive boolean to it.) |
| 89 | |
| 90 | **The money gate is right in principle.** `grant` creates a bucket, an IAM user, |
| 91 | an inline policy and an access key in the *deployment's* cloud account. |
| 92 | `apistore.go:14` calls it the deployment's first explicit operator-identity |
| 93 | gate. Nothing here removes that gate; it moves the decision it guards from |
| 94 | per-app to per-account, so it is made deliberately once instead of being a |
| 95 | queue nobody is watching. |
| 96 | |
| 97 | ## 2. The shape, in one paragraph |
| 98 | |
| 99 | An account applies for object storage. An operator approves the account. From |
| 100 | then on, any app in that account gets a store from `carlos store create` alone |
| 101 | — one member command — and never learns the word "grant". The per-app operator |
| 102 | `grant` verb stays exactly as it is, for unapproved accounts, for `--bucket` |
| 103 | adoption, and for repair. |
| 104 | |
| 105 | ## 3. Rulings taken in the brainstorm |
| 106 | |
| 107 | - **(ruling)** The entitlement is **per account**, not per app. `MintedBucket` |
| 108 | is `d.BucketPrefix + acct + "-" + app` (`internal/objstore/walk.go:62`) and |
| 109 | `UserName` is `"carlos-store-" + acct + "-" + app` (`walk.go:70`), so every |
| 110 | bucket and principal is already named into its account. (§10 records where |
| 111 | that stops being an invariant.) |
| 112 | - **(ruling)** Approval entitles an account to **unlimited stores, metered**. |
| 113 | No quota, no bucket cap. §11 records the risk this accepts. |
| 114 | - **(ruling)** **Account owners only** may apply (`RoleOwner`, |
| 115 | `internal/console/store.go:573`; enforced by the existing `apiOwnerAccount`, |
| 116 | `apiservices.go:103-112`). |
| 117 | - **(ruling)** In an approved account, **`carlos store create` is the whole |
| 118 | member experience** — one command, no second verb, no waiting. §6 records |
| 119 | that this is one CLI command over two doors, which is an implementation |
| 120 | detail of the ruling, not a departure from it. |
| 121 | |
| 122 | ## 4. The entitlement record |
| 123 | |
| 124 | **(decision) Location: `store/entitlements/<sqid>.json`, deliberately NOT under |
| 125 | `accounts/`.** The obvious home was `accounts/<sqid>/store.json`, and the |
| 126 | review caught why it is wrong: `accounts/<sqid>/**` is inside the fleet READ |
| 127 | ACL (`internal/edge/storedoor.go:358-362, 505-510`), so every box credential of |
| 128 | that account could read the applicant's note, the operator's address and the |
| 129 | decline reason. Decline reasons are operator commentary about a tenant; they |
| 130 | must not be tenant-readable. A top-level prefix keeps it console-only. |
| 131 | |
| 132 | > **Implementation gate:** before the record lands, confirm by test that |
| 133 | > `store/entitlements/` is unreachable through the edge store door — read |
| 134 | > `canonicalKey`/`aclRead` rather than trusting the prefix's shape. The review |
| 135 | > flagged this as suspected-not-confirmed (S3). |
| 136 | |
| 137 | ```go |
| 138 | const EntitlementSchema = 1 |
| 139 | |
| 140 | // Entitlement is store/entitlements/<sqid>.json — two blocks for two actors |
| 141 | // (an owner applies, an operator decides), the account-level twin of Record's |
| 142 | // three-for-three. |
| 143 | type Entitlement struct { |
| 144 | Schema int `json:"schema"` |
| 145 | Account string `json:"account"` |
| 146 | Applied *Applied `json:"applied,omitempty"` |
| 147 | Decision *Decision `json:"decision,omitempty"` |
| 148 | } |
| 149 | |
| 150 | type Applied struct { |
| 151 | By string `json:"by"` // the owner's address |
| 152 | At time.Time `json:"at"` |
| 153 | Note string `json:"note,omitempty"` // what they want it for, free text |
| 154 | } |
| 155 | |
| 156 | type Decision struct { |
| 157 | Approved bool `json:"approved"` |
| 158 | By string `json:"by"` // the operator's address |
| 159 | At time.Time `json:"at"` |
| 160 | Reason string `json:"reason,omitempty"` // required on a decline |
| 161 | } |
| 162 | ``` |
| 163 | |
| 164 | **(decision) State is derived, never stored.** No object = never applied. |
| 165 | `applied` with no `decision` = pending. `decision` present = decided, and |
| 166 | `Approved` says which way. Same reasoning as `Record`: a `state string` can |
| 167 | disagree with the blocks beside it, and then two readers disagree about the |
| 168 | account. |
| 169 | |
| 170 | **(decision) `GetEntitlement`/`PutEntitlement` mirror the existing `Get`/`Put`** |
| 171 | (`objstore.go:84`, `:108`), including their `(nil, "", nil)`-means-absent idiom |
| 172 | and CAS-on-etag, so a caller that knows one knows the other. |
| 173 | |
| 174 | **(decision) A declined owner may re-apply.** `apply` overwrites `applied` and |
| 175 | clears `decision`, under CAS. The anti-spam control is the shared `write` |
| 176 | budget keyed on the address, already used by `handleAPIStoreCreate` |
| 177 | (`apistore.go:184`) — not a new rule. |
| 178 | |
| 179 | **Why the bucket and not the console database.** `MaterializeAccounts` is |
| 180 | explicit that the bucket is the control plane and the console is not the only |
| 181 | minter (`internal/console/materialize.go:14-30`). An entitlement held only in |
| 182 | console SQL would be invisible bucket-side and would not survive a console |
| 183 | rebuilt by materialisation. |
| 184 | |
| 185 | **Orphaning is not a risk** (verified in review): apply needs the console to |
| 186 | know the account, so an entitlement cannot precede materialisation; a deleted |
| 187 | account's entitlement lingers inert because resurrection is refused |
| 188 | (`materialize.go:88-105`); and the only `accounts/` key parser cuts on |
| 189 | `/account.json` (`internal/domains/reconcile.go:547`). |
| 190 | |
| 191 | ## 5. Authority |
| 192 | |
| 193 | | Verb | Who | Gate | |
| 194 | |---|---|---| |
| 195 | | apply | account **owner** | `apiOwnerAccount` (member gate, then `RoleOwner`) | |
| 196 | | approve / decline / approvals | deployment operator | fleet-operator bit **and** `CARLOS_STORE_OPERATORS` | |
| 197 | | grant (per-app, unchanged) | deployment operator | unchanged | |
| 198 | |
| 199 | **(decision) Approve/decline/approvals follow the mail PAUSE door's shape, not |
| 200 | `apiAccountApp`'s.** This is blocker 2, and `apimail.go:26-37` already states |
| 201 | the reasoning for the identical case: |
| 202 | |
| 203 | > A pause is applied AGAINST a tenant, not asked for by one — the abusive |
| 204 | > tenant has no reason to add the operator investigating them as a member, and |
| 205 | > only that tenant's own owner can add one, so member-then-allowlist (store's |
| 206 | > grant shape) would deadlock the abuse brake in the exact scenario it exists |
| 207 | > for. |
| 208 | |
| 209 | An operator approving an account has no reason to be a member of it either. |
| 210 | `apiAccountApp` resolves `{account}` against the **caller's own memberships** |
| 211 | (`apicli.go:207`, `AccountsFor`), so the first draft's "member gate first" |
| 212 | would have required every operator to join every account they approve. The |
| 213 | per-app grant door has that property today and is why it is an override rather |
| 214 | than the main path. |
| 215 | |
| 216 | So: `cliUser` → fleet-operator bit → `CARLOS_STORE_OPERATORS` → bare, |
| 217 | byte-identical `http.NotFound`; then resolve the account **viewer-independently** |
| 218 | via `AccountBySqid`, exactly as `mailPauseTarget` does (`apimail.go:212-228`). |
| 219 | |
| 220 | **(decision) Both the fleet bit and the env list are required**, mirroring |
| 221 | mail: `store.IsOperator` (`store.go:1619`) then `s.storeOperator`. The env list |
| 222 | **narrows** the fleet bit, never widens it. This also resolves the review's |
| 223 | finding 8: the operator queue page lives under `/fleet` behind |
| 224 | `operatorOnly`/`IsOperator` (`server.go:2075-2090`), so with this rule the |
| 225 | people who can *see* the queue and the people who can *act* on it are the same |
| 226 | population, minus nobody. |
| 227 | |
| 228 | **`storeOperator`'s exact-match rule is reused, never re-implemented.** Its doc |
| 229 | comment records why `strings.EqualFold` is wrong: U+017F (ſ) folds to "s" and |
| 230 | survives `ToLower`, so `owner@example.teſt` would match `owner@example.test` |
| 231 | and be handed authority to spend money (`apistore.go:360-375`). |
| 232 | |
| 233 | **(decision) Service credentials.** Apply, approve, decline and approvals |
| 234 | register with `svcrole.Set{}` — closed to service principals entirely. Note |
| 235 | explicitly that under this spec an owner-minted `svcrole.Admin` credential's |
| 236 | `create` can mint cloud estate, because `POST .../store` is Admin-reachable |
| 237 | today (`authcli.go:425-440`); that is accepted, and `Declared.By` is set from |
| 238 | `auditActor` so the record names the credential. `TestEveryCLIDoorNamesItsRoles` |
| 239 | and `TestEveryClosedDoorRefusesAServiceCredential` cover the new patterns. |
| 240 | |
| 241 | ## 6. What changes at declare time |
| 242 | |
| 243 | **(decision) One CLI command, two doors.** `carlos store create` calls the |
| 244 | existing declare door, then — when the reply says the account is approved and |
| 245 | the deployment has a backend — a new `POST /api/cli/apps/{account}/{app}/store/provision` |
| 246 | door, member-gated and entitlement-gated. The member still runs one command |
| 247 | (§3's ruling); the wire has two steps. |
| 248 | |
| 249 | This follows `carlos email enable`, which the first draft cited as precedent for |
| 250 | an *inline server walk* and which is in fact client-side orchestration: |
| 251 | `mail_cli.go:635-703` calls declare, ensure, poll and enable as separate doors, |
| 252 | and the server 409s a provision against an undeclared app (`apimail.go:792-848`). |
| 253 | Two doors also fix §8's honesty problem for free: door one's success prints |
| 254 | "declared", door two's failure prints the provisioning error, and neither has to |
| 255 | carry the other's outcome in a single status code. |
| 256 | |
| 257 | The provision door's gates, in order: `apiAccountApp` (member) → entitlement |
| 258 | approved → `storeWalkGates` → `storeSealKey` → `objstore.Grant`. It uses |
| 259 | `storeGrantTimeout` (60s, `apistore.go:339`), **not** `storeTimeout` (15s, |
| 260 | `:212`) — a grant walk makes four backend round trips and the review found the |
| 261 | first draft would have run it under the short one. |
| 262 | |
| 263 | **(decision) The walk runs whenever the account is approved and a backend |
| 264 | exists — there is no "already granted" short-circuit in the door.** This is |
| 265 | blocker 1. `Grant` writes the grant block at step 1, *before* any backend call |
| 266 | (`walk.go:157-162`), so a walk that fails at `EnsureBucket`/`EnsureUser`/ |
| 267 | `MintKey`/`writeEnv` leaves `Grant != nil, Provisioned == nil` — pinned by |
| 268 | `apistore_test.go:1101-1130`. The first draft's "not already granted" condition |
| 269 | would have skipped the retry, and the prefix lock at `apistore.go:241` would |
| 270 | have engaged: **one transient AWS error would have put the member back in the |
| 271 | operator queue this spec exists to remove.** `Grant` is idempotent and |
| 272 | short-circuits itself when the record is provisioned *and* every delivered var |
| 273 | is present (`walk.go:175-200`), which is the correct and only no-op condition. |
| 274 | |
| 275 | **(decision) `Grant` gains a `source` parameter**, written into |
| 276 | `GrantBlock.Source` alongside `By`. This is a signature change touching every |
| 277 | caller — not the "additive, inert" stage-1 item the first draft claimed. |
| 278 | Mirrors `mail.GrantBlock.Source`, whose doc comment established the vocabulary: |
| 279 | |
| 280 | > records WHY the grant exists ("tier:free", "tier:pro", "operator") so a cap |
| 281 | > can be recomputed when a tier changes without silently discarding an |
| 282 | > operator's deliberate override — `internal/mail/mail.go:43-45` |
| 283 | |
| 284 | Values: `"operator"` for the per-app path, `"account-approval"` for this one, |
| 285 | with `By` = the operator who approved the **account**. Not `omitempty`, matching |
| 286 | `mail.GrantBlock.Source` (`mail.go:47`); a record written before the field |
| 287 | existed decodes `""`, which reads correctly as "granted before we recorded why". |
| 288 | |
| 289 | > **`Source` must be additive; renaming `grant` is a fleet-wide silent |
| 290 | > failure.** `release.PurgeApp` parses `grant`/`provisioned` as |
| 291 | > `*json.RawMessage` (`internal/release/trash.go:211-222`) and refuses box-side |
| 292 | > auto-purge when either is present. **Verified in review:** a field *inside* |
| 293 | > `grant` is invisible to it and `TestPurgeGuardSeesOurRecord` stays green. The |
| 294 | > other parsers of these bytes — `appsettings.go:309,808`, |
| 295 | > `economicspass.go:171`, `ownership.go:407`, and the CLI's hand-rolled |
| 296 | > `apiStoreGrantBlock` (`apimode.go:1552`) — all ignore unknown fields. No |
| 297 | > non-Go parser exists. |
| 298 | |
| 299 | **(decision) A same-prefix re-declare becomes a no-op write.** |
| 300 | `apistore.go:257` rewrites `at`/`by` unconditionally today; every such rewrite |
| 301 | can CAS-fail an in-flight `Grant`'s step-1 or step-6 Put. With `create` now |
| 302 | driving walks, that stops being theoretical (see §9). |
| 303 | |
| 304 | ## 7. When provisioning fails |
| 305 | |
| 306 | The provision door answers non-2xx with a fixed sentence, the detail logged — |
| 307 | the pairing `apistore.go` already uses everywhere, because a store error |
| 308 | carries bucket names, endpoints and request ids. `apimode.go:1572-1580` turns |
| 309 | any non-200 into `apiErrorFromBody` and `store.go:150-153` `log.Fatalf`s it, so |
| 310 | the sentence *is* the entire user-visible outcome and must stand alone. |
| 311 | |
| 312 | Because the declaration landed through a **separate, already-succeeded door**, |
| 313 | the CLI has already printed that it landed. There is no single response |
| 314 | pretending to be both a success and a failure. |
| 315 | |
| 316 | The record is left `Grant != nil, Provisioned == nil` — grant block written, |
| 317 | estate incomplete — which `carlos store status` already renders and which a |
| 318 | re-run of `carlos store create` resumes, because §6 removed the short-circuit |
| 319 | that would have blocked it and the walks are idempotent by construction |
| 320 | (`walk.go:144`: "a re-run never re-chooses the bucket"). |
| 321 | |
| 322 | **(decision) Name the post-destroy window in the sentence.** `Teardown` deletes |
| 323 | the record last (`walk.go:641`) and the next `create` re-mints the same |
| 324 | deterministic name (`walk.go:208`); `aws.go:186` swallows only |
| 325 | `BucketAlreadyOwnedByYou`, so a name AWS has not yet released fails the walk in |
| 326 | the member's face — in a destroy/re-create loop they can now drive themselves. |
| 327 | The failure sentence says so, and so does `docs/site/cli.md`. |
| 328 | |
| 329 | ## 8. Surface |
| 330 | |
| 331 | ### CLI |
| 332 | |
| 333 | ``` |
| 334 | carlos store apply --account <a> [--note "..."] owner |
| 335 | carlos store approvals operator |
| 336 | carlos store approve --account <a> operator |
| 337 | carlos store decline --account <a> --reason "..." operator |
| 338 | ``` |
| 339 | |
| 340 | `carlos store status --app <a>` renders four entitlement states: |
| 341 | |
| 342 | - *not applied* — "this account is not approved for object storage — `carlos store apply`" |
| 343 | - *pending* — "applied by <owner> on <date>, waiting for a deployment operator" |
| 344 | - *approved* — no extra line; their store provisions on declare |
| 345 | - *declined* — the reason, and that they may apply again |
| 346 | |
| 347 | **(ruling, 2026-09-08) What a tenant may read of the operator's half.** §4 |
| 348 | says a decline reason must not be tenant-readable; this section says status |
| 349 | renders the reason. Both were written meaning something true, and the |
| 350 | implementation had to pick one. The resolution: |
| 351 | |
| 352 | | | person | machine | |
| 353 | |---|---|---| |
| 354 | | `Decision.By` | withheld | withheld | |
| 355 | | `Decision.Reason` | rendered | withheld | |
| 356 | | `Applied.By` / `Applied.Note` | rendered | withheld | |
| 357 | |
| 358 | Requiring a reason on decline is pointless if the account never learns it — |
| 359 | that is §1's four-day silence in a new hat — so a person reads it. A machine |
| 360 | credential gets the STATE and nothing else: a reason is written for someone to |
| 361 | read and act on, not for an app to branch on. The deciding operator's address |
| 362 | travels to neither; an operator stays answerable to the deployment, not to the |
| 363 | account they just declined. |
| 364 | |
| 365 | §4's sentence therefore means what its placement argument always meant: not |
| 366 | readable by a tenant's BOXES, through the fleet read ACL. The console's own |
| 367 | answer is narrowed by projection (`storeEntitlementWire`), which is a separate |
| 368 | mechanism with a separate test. |
| 369 | |
| 370 | **(decision) Two existing printed sentences must change, not just gain a |
| 371 | neighbour.** `store.go:261` always says "nothing is provisioned until a |
| 372 | deployment operator grants it", and `store.go:348` says "waiting on a |
| 373 | deployment operator (carlos store grant)" — pinned by `store_test.go:491`. |
| 374 | Both are false in an approved account. `printStoreCreate` also needs a |
| 375 | provisioned branch it does not have. |
| 376 | |
| 377 | **(decision) The entitlement rides on the existing `storeReply`** |
| 378 | (`apistore.go:113`) as a new `entitlement` key. That function is deliberately |
| 379 | the ONE answer shape both doors give so they cannot drift; a second shape would |
| 380 | reintroduce exactly the drift it guards against. |
| 381 | |
| 382 | ### Console |
| 383 | |
| 384 | - **Account settings** (`handleAccountSettings`, `server.go:2278-2298`, which |
| 385 | already has `role` in hand) — an Object storage panel: owners see state and an |
| 386 | apply control, other members see state read-only. *(The first draft put this |
| 387 | on app settings, confusing it with the per-app destroy door at |
| 388 | `appsettings.go:829`.)* |
| 389 | - **Operator view** under `/fleet` — pending applications oldest first, with |
| 390 | account, applicant, note, approve/decline, and the §11 bucket count. |
| 391 | |
| 392 | ## 9. Concurrency: a gap this spec must close before it can ship |
| 393 | |
| 394 | **This is blocker 3, and it is a bug that exists today.** `walk.go:662-664` |
| 395 | asserts "the console door serialises these verbs per app". **No such lock |
| 396 | exists** — there is no mutex anywhere in `internal/console` guarding the store |
| 397 | walks. Today two operators would have to run `grant` simultaneously to hit it. |
| 398 | This spec makes `create` drive walks, and `apistore.go:176-179` explicitly says |
| 399 | `create` is re-run "from a second terminal, from a deploy script" — so it turns |
| 400 | a rare race into an ordinary one. |
| 401 | |
| 402 | The failure is not benign. Two concurrent walks on one app: both pass the |
| 403 | handler read, the second's `Get` sees a grant block with `Provisioned == nil` |
| 404 | and runs the full walk in parallel; `ReapKeys` (`walk.go:343`) deletes the |
| 405 | other walk's fresh key; both `writeEnv`; and the loser's final Put CAS-fails |
| 406 | **with no retry** — `Grant` has none, unlike `Rotate`'s guarded one |
| 407 | (`walk.go:303-330`). The record's `KeyID` can then differ from the delivered |
| 408 | key, and the next `Rotate`'s `ReapKeys(keep=[record key])` deletes the key the |
| 409 | app is running on. `walk.go:404-410` calls that outcome "NOT survivable". |
| 410 | |
| 411 | **(decision) Three fixes, all before the §6 branch lands:** |
| 412 | |
| 413 | 1. A keyed per-`(acct, app)` mutex in the console, held across **every** store |
| 414 | walk — provision, grant, rotate, finish, destroy, purge. This makes |
| 415 | `walk.go:662`'s comment true instead of aspirational. |
| 416 | 2. A `Rotate`-style guarded CAS retry on `Grant`'s step-6 Put |
| 417 | (`walk.go:216-219`), re-reading and retrying once when the fresh record |
| 418 | still names the state this walk started from. |
| 419 | 3. The same-prefix re-declare no-op from §6. |
| 420 | |
| 421 | Fixes 1 and 2 are corrections to existing code and are independently |
| 422 | valuable — they should land as their own commits, ahead of the feature, so |
| 423 | they can be reviewed and reverted separately. |
| 424 | |
| 425 | ## 10. Revocation, movement, and what the entitlement does *not* control |
| 426 | |
| 427 | **(decision) `decline` is permitted over an approved decision, and is not a |
| 428 | teardown.** It (a) stops new provisioning, (b) returns drift self-repair |
| 429 | (`store.go:364`'s "re-run grant") to an operator step, and (c) **never** |
| 430 | destroys existing estate — provisioned stores keep their buckets, credentials |
| 431 | and grant blocks. An owner may re-apply afterwards, which clears the decision |
| 432 | back to pending. |
| 433 | |
| 434 | **(decision) The entitlement gates provisioning, never existence.** |
| 435 | `handleAppMoveAccount` (`appsettings.go:~918-988`) moves the console row only, |
| 436 | while bucket and IAM names carry the original sqid (`walk.go:208-215`). So an |
| 437 | app moved into an **unapproved** account keeps its working store, and §3's |
| 438 | "every bucket is named into its account" stops being an invariant after a move. |
| 439 | Metering follows the key prefix (`economicspass.go:161-190`), so the bill |
| 440 | follows the app. All of this is accepted and stated rather than fixed. |
| 441 | |
| 442 | ## 11. The risk this accepts, deliberately |
| 443 | |
| 444 | "Unlimited, metered" means **AWS's per-account bucket limit becomes the |
| 445 | deployment's real ceiling**, hit at provision time as a raw AWS error in front |
| 446 | of whichever tenant declares next. |
| 447 | |
| 448 | **(correction, 2026-09-09)** This section was written against a limit of ~100. |
| 449 | AWS's default is now **10,000** buckets per account, raisable far beyond that |
| 450 | on request, so the ceiling is nowhere near as close as the paragraph below |
| 451 | assumes. The surfacing is still worth having — an operator approving an |
| 452 | account should see the size of the estate they are adding to — but it is now |
| 453 | an inventory figure rather than a hazard light, and nothing here should be |
| 454 | read as urgent. |
| 455 | |
| 456 | **(decision) Do not gate on it. Surface it.** The approvals view shows how many |
| 457 | buckets the deployment has minted, so an operator can see the number before |
| 458 | saying yes. Sourcing it needs one additive field: `usage.AppStorage` |
| 459 | (`internal/usage/storage.go:7-11`) cannot distinguish "provisioned, empty" from |
| 460 | "no store" because `StoreBytes` is `omitempty`, so the daily pass gains |
| 461 | `StoreProvisioned bool` (`economicspass.go:161-190`). A hard cap is a quota by |
| 462 | another name and was ruled out in §3. |
| 463 | |
| 464 | ## 12. Non-goals |
| 465 | |
| 466 | - **Credit-card gating.** Later; the statement it will read is already written |
| 467 | daily. |
| 468 | - **Fixing `CARLOS_STORE_BACKEND` as a root env edit.** That is #368. This spec |
| 469 | adds no new root env vars. |
| 470 | - **The host role's provisioning IAM and its permissions boundary.** Rides with |
| 471 | #368's switch-on rather than with this spec — but neither is complete without |
| 472 | it, and until it lands an approved account still cannot get a bucket (§1). |
| 473 | - **Quotas or bucket caps.** Ruled out in §3. |
| 474 | - **Tearing down estate on decline.** §10. |
| 475 | - **A generalised capability-keyed entitlement system.** Email's tier |
| 476 | vocabulary suggests where this ends up, but building the general machine |
| 477 | before its second caller exists means guessing at email's requirements. |
| 478 | `Entitlement` is shaped so that migration is an addition, not a rewrite. |
| 479 | |
| 480 | ## 13. Testing |
| 481 | |
| 482 | Walks against `MemProvisioner`, doors against the console fixture, printers |
| 483 | against literal replies with no stub server — the primitive spec's §11 split. |
| 484 | |
| 485 | **Record.** Round-trip; four derived states; absent reads `(nil, "", nil)`; CAS |
| 486 | conflict on concurrent apply. **Plus: `store/entitlements/` is not readable |
| 487 | through the edge store door** (§4's implementation gate). |
| 488 | |
| 489 | **Authority.** Non-owner apply → 403 via `apiOwnerAccount`. **A non-member |
| 490 | operator can approve** — the blocker-2 regression test. Non-operator on |
| 491 | approvals → bare 404, not 403. Fleet-bit-without-env-list and |
| 492 | env-list-without-fleet-bit both refused. `storeOperator`'s ſ-vs-s test still |
| 493 | passes and the new doors route through it. Service credentials refused on all |
| 494 | four new doors. |
| 495 | |
| 496 | **Declare/provision branches.** Approved + backend → provisioned, |
| 497 | `Source == "account-approval"`, `By ==` the approving operator. Approved + no |
| 498 | backend → declared, existing backend warning, nothing minted. Not approved → |
| 499 | declared, pointed at `apply`, nothing minted. Provisioned **and delivered** → |
| 500 | no-op, no new key. **Grant-written-but-not-provisioned → the walk RUNS and |
| 501 | completes** (blocker 1's regression). Prefix-change guard still refuses |
| 502 | throughout. |
| 503 | |
| 504 | **Concurrency (§9).** Two simultaneous provisions of one app serialise, mint |
| 505 | one key, and leave the record's `KeyID` equal to the delivered key. Grant's |
| 506 | step-6 CAS retry covers the interleaving that used to lose. A same-prefix |
| 507 | re-declare during an in-flight walk does not CAS-fail it. |
| 508 | |
| 509 | **Failure.** A walk failing after the grant block is written leaves |
| 510 | `Grant != nil, Provisioned == nil`, answers non-2xx, and a re-run converges to |
| 511 | a provisioned store with the same bucket. |
| 512 | |
| 513 | **Regression.** `TestPurgeGuardSeesOurRecord` green with `Source` present. |
| 514 | |
| 515 | **Printers.** Four entitlement states; the two changed sentences |
| 516 | (`store.go:261`, `:348`); `printStoreCreate`'s new provisioned branch. |
| 517 | |
| 518 | ## 14. Rollout order |
| 519 | |
| 520 | Stages 1 and 2 are bug fixes to existing code and land first, separately. |
| 521 | |
| 522 | 1. **Grant's CAS retry** (§9 fix 2) — its own commit. |
| 523 | 2. **The console's per-app store walk mutex** (§9 fix 1) — its own commit. |
| 524 | 3. `Entitlement` record + accessors + edge-ACL test; `GrantBlock.Source` and |
| 525 | `Grant`'s new parameter (signature change, all callers). |
| 526 | 4. Console doors: apply, approvals, approve, decline. No behaviour change yet. |
| 527 | 5. The provision door and the CLI's two-door `create`. **Behaviour changes |
| 528 | here**; everything before is dormant. |
| 529 | 6. CLI verbs, the status line, the two changed sentences. |
| 530 | 7. Console UI, notification, `StoreProvisioned` and the bucket count. |
| 531 | 8. `docs/site/cli.md`. |
| 532 | |
| 533 | ## 15. Notification |
| 534 | |
| 535 | James waited four days because nothing told anyone. This is not optional. |
| 536 | |
| 537 | **It is narrower than it looks, and the review confirmed the mechanism.** |
| 538 | `alertRecipients` (`alertwatch.go:151-168`) uses `store.Operators()` — users |
| 539 | with the fleet-operator bit (`store.go:1634`) — plus `GrantToken` (`:1702`), |
| 540 | and `SendApp` delivers only "to the granting user, using the stored grant |
| 541 | token" (`keymail.go:232-238`). **Mail reaches an address only if that person |
| 542 | completed the console OAuth ceremony.** |
| 543 | |
| 544 | **(decision) The console queue is the primary surface; mail is the nudge.** |
| 545 | |
| 546 | - On apply: for each `cfg.StoreOperators` entry, canonicalise with |
| 547 | `ToLower(TrimSpace)` — matching `storeOperator` (`apistore.go:377`), since |
| 548 | `GrantToken` is keyed on the canonical address — then `GrantToken`, then |
| 549 | `s.sendApp`, guarded for `s.sendApp == nil` (`alertwatch.go:103`). Addresses |
| 550 | with no grant are logged as `alertRun.missing` already is: never a silent |
| 551 | drop, never a failed write. Best-effort off the request goroutine, following |
| 552 | `notifyCLIIssued` (`authcli.go:948-990`). |
| 553 | - On decision: best-effort keymail to the applying owner. |
| 554 | - A send failure never fails the write. The application is durable in the |
| 555 | bucket the moment it is written. |
| 556 | - Because mail may reach nobody, the operator view carries the pending count |
| 557 | where §5's population will actually see it. |
| 558 | |
| 559 | **Open, deliberately deferred to implementation** (review S1): whether |
| 560 | `s.audit(acct, app, …)` (`server.go:2692`) renders an app-less row for these |
| 561 | account-level events. If it does not, apply/approve/decline audit with the app |
| 562 | field empty and the events view gains a case; this is a small question best |
| 563 | answered against the events schema, not guessed at here. |
| 564 | |
amadan