carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1// The app-declared private-storage policy (issue #412, spec
2// 2026-09-22-instance-private-storage-design.md): an app may say once, on
3// its instances manifest, that every instance it creates gets a config
4// environment of its own and an object store bound to that environment.
5//
6// The whole point is the SPLIT. Declaring the policy is
7// `handleAPIInstanceEnable` — person-only, zero Set, no service credential
8// reaches it. Spending it is `handleAPIInstanceCreate`, which an
9// `instance`-role credential can already reach. So a public signup path
10// provisions a tenant's own bucket without ever holding the power to create
11// one, which is what makes this a better answer than widening the role
12// (#410, which this makes unnecessary).
13//
14// Nothing here is a second implementation of provisioning. The walk this
15// file drives is the one `handleAPIStoreProvision` drives, with the same
16// lock, the same seal key, the same timeout and the same
17// GrantSourceAccountApproval — factored so the automatic path and the
18// member's own `carlos store create --environment` cannot drift.
19package console
20
21import (
22 "context"
23 "crypto/ecdh"
24 "errors"
25 "fmt"
26 "log"
27 "net/http"
28 "time"
29
30 "github.com/carlosframework/platform/internal/appconfig"
31 "github.com/carlosframework/platform/internal/objstore"
32 "github.com/carlosframework/platform/internal/release"
33)
34
35// storeMaxPerAppDefault is the ceiling on ENVIRONMENT stores under one app
36// when the deployment names none (spec §8, issue #412's point 3).
37//
38// A bucket plus an IAM user per instance is a bigger footprint than an
39// instance, and the binding limit is IAM's: 5,000 users per cloud account,
40// an adjustable quota
41// (https://docs.aws.amazon.com/general/latest/gr/iam-service.html). So a
42// signup path that can mint them needs a bar it stops at — but the bar has
43// to be sized against that number rather than picked for looking tidy. An
44// earlier draft said 100, which was a fifth of one app's fair share of the
45// quota and would have refused a perfectly ordinary managed product at its
46// hundredth customer (James, 2026-09-23).
47//
48// Per APP rather than per account, and that is a cost decision stated as
49// such: the count is one prefix listing of the app, which the create door
50// can afford on every request. A deployment-wide count would mean listing
51// every app's stores on every create.
52//
53// WHAT THIS DOES NOT BOUND, said plainly because the per-app shape invites
54// the wrong reading: five apps at this default reach 5,000 between them, and
55// the deployment's own IAM principals are not in that sum. This is a brake
56// on ONE runaway signup path, not a budget for the cloud account. A real
57// account-wide bound needs a counter kept on the side rather than one
58// derived by listing, and it is not here.
59const storeMaxPerAppDefault = 1000
60
61// storeMaxPerApp is the ceiling in force. Zero in the config means "not
62// configured", not "no stores allowed": a cap of zero would turn an
63// unset variable into an app that can have no environment store at all.
64func (s *Server) storeMaxPerApp() int {
65 if s.cfg.StoreMaxPerApp > 0 {
66 return s.cfg.StoreMaxPerApp
67 }
68 return storeMaxPerAppDefault
69}
70
71// storeCeiling refuses an app that has reached its environment-store cap.
72//
73// env is the environment ABOUT to be given a store, and a name that already
74// has one is never refused: the cap is on the FOOTPRINT, and re-declaring
75// an existing store adds no bucket and no principal. Without that carve-out
76// a retry of a create that failed just after provisioning would be turned
77// away by the very ceiling it had reached, with no way forward at all.
78//
79// A listing failure is a refusal, not a pass. This is a spend gate; reading
80// "how many are there already" as "none" on a bad round trip is the one
81// error direction that costs money.
82func (s *Server) storeCeiling(ctx context.Context, acct *Account, app, env string) *doorRefusal {
83 if env == "" {
84 return nil // the app's own store is not one of the capped collection
85 }
86 envs, err := objstore.StoreEnvironments(ctx, s.cfg.Releases, acct.Sqid, app)
87 if err != nil {
88 log.Printf("console: store ceiling %s/%s: %v", acct.Sqid, app, err)
89 return &doorRefusal{Status: http.StatusBadGateway,
90 Msg: "could not count this app's stores — try again"}
91 }
92 for _, e := range envs {
93 if e == env {
94 return nil
95 }
96 }
97 max := s.storeMaxPerApp()
98 if len(envs) < max {
99 return nil
100 }
101 return &doorRefusal{Status: http.StatusConflict,
102 Msg: fmt.Sprintf("%s already has %d environment stores, which is this deployment's limit of %d — destroy one in the console, or ask an operator to raise CARLOS_STORE_MAX_PER_APP",
103 app, len(envs), max)}
104}
105
106// instanceStoreNames refuses an environment that cannot carry a store, for
107// an app under the policy (spec §6).
108//
109// derived says where the name came from — the host's first label, or the
110// caller's own `environment` — and it changes only the sentence, never the
111// rails. The first label is the derived name because it is the one
112// predictable from the host, already unique within the app (the record key
113// IS the host), and readable in `store status` at a hundred of them, which
114// is #412's point 1.
115//
116// Three refusals, all BEFORE anything is written, because each of them is
117// otherwise discovered somewhere worse:
118//
119// - a DNS label is not an environment name. Labels run to 63 characters
120// and environment names to 32, and "default" is a perfectly legal label
121// that names the app's own bundle rather than a named overlay.
122// - the bucket name is S3's 63. The walk checks this itself, but reaching
123// it there means an instance record already written over an estate that
124// cannot exist — and the walk's own refusal arrives as a 502 the caller
125// cannot act on, not a 400 naming the part to shorten.
126// - the principal name is IAM's 64, for the same reason. An environment
127// store's name carries three components, so a long enough (account,
128// app, environment) is simply unnameable.
129//
130// Each sentence names the part the caller can shorten, because from a
131// signup path the host label is usually the only one they control.
132func (s *Server) instanceStoreNames(acct *Account, app, env string, derived bool) *doorRefusal {
133 source := "the environment you named"
134 fix := "choose a shorter environment name"
135 if derived {
136 source = "the host's first label"
137 fix = "choose a different host label"
138 }
139 if !appconfig.ValidEnvironmentName(env) {
140 return &doorRefusal{Status: http.StatusBadRequest,
141 Msg: fmt.Sprintf("%s gives every instance its own storage, which needs %s to be a usable environment name — %q is not (want ^[a-z0-9-]{1,32}$, and not \"default\"). Please %s", app, source, env, fix)}
142 }
143 bucket := objstore.MintedBucketIn(s.cfg.StoreDeployment, acct.Sqid, app, env)
144 if !objstore.ValidBucketName(bucket) {
145 return &doorRefusal{Status: http.StatusBadRequest,
146 Msg: fmt.Sprintf("this instance's bucket would be named %q, which is %d characters and not a legal S3 name — %s", bucket, len(bucket), fix)}
147 }
148 user := objstore.UserNameIn(acct.Sqid, app, env)
149 if !objstore.ValidUserName(user) {
150 return &doorRefusal{Status: http.StatusBadRequest,
151 Msg: fmt.Sprintf("this instance's IAM user would be named %q, which is %d characters, over AWS's %d — %s, or shorten the app name", user, len(user), objstore.MaxIAMUserName, fix)}
152 }
153 return nil
154}
155
156// instanceStoreProvision declares and provisions one environment's store
157// under the account's standing entitlement, and answers w itself on every
158// refusal. The caller has already proved membership and spent its write
159// budget; what is left is the store plane's own gates, in the order
160// handleAPIStoreProvision applies them.
161//
162// actor is the create door's principal — a person's address or the service
163// credential's `app:<sqid>/<app>` audit actor — and it lands in the grant
164// block, so a bucket minted by a signup path names the signup path.
165//
166// The env prefix is INHERITED from the app's own store when it has one. An
167// app that declared `carlos store create --env-prefix JELLY_S3` reads those
168// names in its code, and an instance store delivering CARLOS_STORE_BUCKET
169// into the same app would be a second vocabulary for one thing. With no app
170// store declared, the default prefix is taken — what a fresh declaration
171// would have taken anyway.
172//
173// ctx must be the REQUEST context, not a short-deadline derivative of it:
174// the walk below takes its own storeGrantTimeout, which is longer than the
175// create door's own deadline, and a walk that inherited a fifteen-second
176// budget would abandon a bucket it had already paid for.
177func (s *Server) instanceStoreProvision(ctx context.Context, w http.ResponseWriter, acct *Account, app, env, actor string) (*objstore.Record, bool) {
178 out, ok, _ := s.instanceStoreProvisionFor(ctx, w, acct, app, env, actor, "")
179 return out, ok
180}
181
182// errSeedStoreTaken is instanceStoreProvisionFor's one refusal it does not
183// answer itself: the environment already has a store that holds estate and
184// was not made by this seed (seeded-instances spec §4 Storage). The seed
185// door speaks that refusal, naming the store it got back.
186var errSeedStoreTaken = errors.New("console: the environment's store is not this seed's")
187
188// instanceStoreProvisionFor is instanceStoreProvision for a seeded create
189// when seedID is set. With seedID "" it is instanceStoreProvision exactly,
190// and its error is always nil.
191//
192// With a seed, the store must be new and empty, or this seed's own from an
193// earlier attempt. That is re-checked under the store lock, on the record
194// read there, before anything is declared or granted: a check made before
195// the lock (the seed door's own) can be overtaken by a concurrent `carlos
196// store create --environment`, and GrantIn would then reuse that grant.
197// On that refusal it answers nothing on w and returns the record it found,
198// false and errSeedStoreTaken. The environment must also be one no other
199// instance names (final review F6): errSeedStoreTaken again, or
200// errSeedEnvironmentUnread when that cannot be read, also unanswered.
201// Every other refusal is answered on w and returns a nil error, as
202// instanceStoreProvision's do.
203//
204// A seed's declaration carries its id (objstore.Declared.Seed). A record
205// that is declared and holds no estate is declared again for this seed.
206// When the grant then fails before writing its block, the declaration is
207// rolled back under the lock: deleted if this call created the record,
208// else restored to its previous bytes.
209func (s *Server) instanceStoreProvisionFor(ctx context.Context, w http.ResponseWriter, acct *Account, app, env, actor, seedID string) (*objstore.Record, bool, error) {
210 pub, ok := s.instanceStorePreflight(ctx, w, acct, app)
211 if !ok {
212 return nil, false, nil
213 }
214 effectStore, effectProvisioner := claimStoreAdapters(ctx, s.cfg.Releases, s.cfg.StoreProvisioner)
215 // The same lock handleAPIStoreProvision takes, keyed the same way, so a
216 // signup creating an instance and a member running `carlos store create
217 // --environment` against the same name serialise against each other.
218 defer s.storeWalks.lock(acct.Sqid, app, env)()
219 // The walk's deadline starts AFTER the lock, for the reason that door
220 // states: a request that queued behind a full-length walk would
221 // otherwise reach the backend holding a spent context.
222 walkCtx, cancel := context.WithTimeout(ctx, storeGrantTimeout)
223 defer cancel()
224 // Re-read under the lock: a queued request may have waited while an
225 // operator declined the account.
226 e, ok := s.storeProvisionApproval(walkCtx, w, acct)
227 if !ok {
228 return nil, false, nil
229 }
230 // The declaration the walk requires. GrantIn refuses a store nobody
231 // declared (ErrNotDeclared) because the env prefix is the member's
232 // choice and a store with nowhere to be delivered is not one to invent
233 // — under this policy the APP made that choice, once, on the manifest,
234 // so the declaration is written here rather than demanded of a caller
235 // who has no verb to write it with.
236 prefix, ok := s.storeInheritedPrefix(walkCtx, w, acct, app)
237 if !ok {
238 return nil, false, nil
239 }
240 rec, etag, err := objstore.GetIn(walkCtx, effectStore, acct.Sqid, app, env)
241 if err != nil {
242 log.Printf("console: instance store read %s/%s/%s: %v", acct.Sqid, app, env, err)
243 jsonError(w, http.StatusBadGateway, "could not read this instance's store record — try again")
244 return nil, false, nil
245 }
246 if seedID != "" && !seedOwnsStore(rec, seedID) {
247 return rec, false, errSeedStoreTaken
248 }
249 // A seed's environment must also be unused (final review F6), checked
250 // here under the lock an environment binding takes (seedenvbind.go): a
251 // binding either landed before this read, or finds the store marked
252 // and takes it from the seed.
253 if seedID != "" {
254 user, err := s.seedEnvironmentUser(walkCtx, acct.Sqid, app, env, seedID)
255 if err != nil {
256 log.Printf("console: instance store %s/%s/%s: seed %s: who names the environment: %v", acct.Sqid, app, env, seedID, err)
257 return nil, false, errSeedEnvironmentUnread
258 }
259 if user != "" {
260 log.Printf("console: instance store %s/%s/%s: seed %s: instance %s already names the environment", acct.Sqid, app, env, seedID, user)
261 return rec, false, errSeedStoreTaken
262 }
263 }
264 existed := rec != nil
265 if rec == nil {
266 rec = &objstore.Record{}
267 }
268 // Only when absent. A re-run over an existing declaration must not
269 // re-stamp its author and timestamp, and must not move a prefix the app
270 // is already reading — that is the one rewrite handleAPIStoreCreate
271 // refuses with a 409, and this path has no member to show it to.
272 //
273 // A seed is the exception: a declared-only record holds no estate, and
274 // one not marked with this seed is declared again, for this seed, so
275 // the store this seed makes is marked as its own. That replaces the
276 // other writer's EnvPrefix, By and At: a member's chosen --env-prefix
277 // on a store with no estate yet is lost, and the app's inherited prefix
278 // is delivered instead.
279 redeclare := rec.Declared == nil
280 if seedID != "" && rec.Declared != nil && rec.Grant == nil && rec.Provisioned == nil && rec.Declared.Seed != seedID {
281 redeclare = true
282 }
283 // What a seed's failed grant rolls back to (below): the record's exact
284 // bytes before this call declared it, or nil when there was no record.
285 var prevRaw []byte
286 if seedID != "" && redeclare && existed {
287 raw, rawTag, err := effectStore.Get(walkCtx, objstore.KeyIn(acct.Sqid, app, env))
288 if err != nil {
289 log.Printf("console: instance store read %s/%s/%s: %v", acct.Sqid, app, env, err)
290 jsonError(w, http.StatusBadGateway, "could not read this instance's store record — try again")
291 return nil, false, nil
292 }
293 if rawTag != etag {
294 jsonError(w, http.StatusConflict,
295 "this instance's store record changed while it was being created — try again")
296 return nil, false, nil
297 }
298 prevRaw = raw
299 }
300 declaredTag := ""
301 if redeclare {
302 rec.Declared = &objstore.Declared{EnvPrefix: prefix, At: time.Now().UTC(), By: actor, Seed: seedID}
303 if declaredTag, err = objstore.PutIn(walkCtx, effectStore, acct.Sqid, app, env, rec, etag); err != nil {
304 if errors.Is(err, release.ErrCASConflict) {
305 jsonError(w, http.StatusConflict,
306 "this instance's store record changed while it was being created — try again")
307 return nil, false, nil
308 }
309 log.Printf("console: instance store declare %s/%s/%s: %v", acct.Sqid, app, env, err)
310 jsonError(w, http.StatusBadGateway, "could not declare this instance's store — nothing was created; try again")
311 return nil, false, nil
312 }
313 }
314 out, err := objstore.GrantInFor(walkCtx, effectStore, effectProvisioner, s.cfg.StoreDeployment,
315 pub, acct.Sqid, app, env, e.Decision.By, objstore.GrantSourceAccountApproval, "", seedID, time.Now())
316 if err != nil && seedID != "" && redeclare {
317 // A seed's declaration must not outlive a grant that wrote nothing:
318 // another path could grant that record later and keep the mark, and
319 // the seed's own finish would then tear down a store it never made
320 // (spec §4 Storage). Still under the lock, the record goes back to
321 // what it was, conditionally on declaredTag. A grant that did write
322 // its block has moved the record past declaredTag, and that estate
323 // is this seed's to finish.
324 rbCtx, rbCancel := context.WithTimeout(context.WithoutCancel(ctx), storeTimeout)
325 rerr := objstore.RevertIn(rbCtx, effectStore, acct.Sqid, app, env, prevRaw, declaredTag)
326 rbCancel()
327 if errors.Is(rerr, release.ErrCASConflict) {
328 log.Printf("console: instance store %s/%s/%s: seed %s declaration not rolled back: the record was written since (its grant block or another writer), left as it is", acct.Sqid, app, env, seedID)
329 } else if rerr != nil {
330 log.Printf("console: instance store %s/%s/%s: roll back seed %s declaration: %v", acct.Sqid, app, env, seedID, rerr)
331 }
332 }
333 if err == nil && seedID != "" && (out.Declared == nil || out.Declared.Seed != seedID) {
334 // The declare door takes no store lock, so a `carlos store create
335 // --environment` between this declaration and the grant can rewrite
336 // it without the mark. The store is still new, but this seed's
337 // abort can no longer tear it down: said in the log, for an operator.
338 log.Printf("console: instance store %s/%s/%s: granted for seed %s but its declaration no longer names the seed", acct.Sqid, app, env, seedID)
339 }
340 if errors.Is(err, objstore.ErrStoreNameTaken) {
341 log.Printf("console: instance store provision %s/%s/%s: %v", acct.Sqid, app, env, err)
342 jsonError(w, http.StatusConflict,
343 "another store already uses this instance's storage names (app and environment names are joined with \"-\"), so the instance was not created. use a different host name")
344 return nil, false, nil
345 }
346 if err != nil {
347 // Logged, not returned: a walk error carries bucket names, IAM ARNs
348 // and request ids. The sentence the caller gets is the one they can
349 // act on, and it is honest about what is now true — the declaration
350 // may stand, the instance does not, and a retry of the SAME host
351 // resumes this exact store because its name came from the host.
352 log.Printf("console: instance store provision %s/%s/%s: %v", acct.Sqid, app, env, err)
353 jsonError(w, http.StatusBadGateway,
354 "could not provision this instance's storage, so the instance was not created — run the same command again to resume it")
355 return nil, false, nil
356 }
357 // Audited from here as well as from the store doors, because this is the
358 // one path where a bucket is minted by a caller that is not a person:
359 // an account reading its own rows has to be able to see that its signup
360 // path spent money, and by whom it was authorised.
361 s.audit(acct, app, actor, "store-provision",
362 "provisioned "+storeBucketName(out)+" for instance environment "+env+" (instance create)")
363 return out, true, nil
364}
365
366// instanceStorePreflight is the store plane's gates that write nothing and
367// take no lock: a temporary app, no backend, no standing approval, no
368// sealing key. It answers w itself on a refusal. instanceStoreProvisionFor
369// runs it first; a seed door runs it before reserving its record, so these
370// refusals never leave an aborted record behind.
371func (s *Server) instanceStorePreflight(ctx context.Context, w http.ResponseWriter, acct *Account, app string) (*ecdh.PublicKey, bool) {
372 if s.refuseTemporary(w, acct, app, "give it object storage") {
373 return nil, false
374 }
375 if s.cfg.StoreProvisioner == nil {
376 // The manifest carries a policy this deployment cannot honour. The
377 // enable door refuses to write one in this state (§5), so reaching
378 // here means the backend was unwired afterwards — a configuration
379 // fact, named as one.
380 jsonError(w, http.StatusServiceUnavailable,
381 "object storage is not configured for this deployment (CARLOS_STORE_BACKEND), and "+app+" gives every instance its own storage — no instance can be created until one is wired or the policy is turned off (carlos instances enable --private-storage=false)")
382 return nil, false
383 }
384 preCtx, preCancel := context.WithTimeout(ctx, storeTimeout)
385 defer preCancel()
386 if _, ok := s.storeProvisionApproval(preCtx, w, acct); !ok {
387 return nil, false
388 }
389 return s.storeSealKey(w)
390}
391
392// seedOwnsStore reports whether a seeded create may use rec (spec §4
393// Storage): there is no store, the record holds no estate yet, or the
394// estate was made by this very seed.
395func seedOwnsStore(rec *objstore.Record, seedID string) bool {
396 if rec == nil || (rec.Grant == nil && rec.Provisioned == nil) {
397 return true
398 }
399 return rec.Declared != nil && rec.Declared.Seed == seedID
400}
401
402// storeInheritedPrefix reads the app's own store for the variable-name
403// prefix a new environment store should deliver under. Absent is not an
404// error — an app with no store of its own is the ordinary shape for a
405// private-storage app, and takes the default.
406func (s *Server) storeInheritedPrefix(ctx context.Context, w http.ResponseWriter, acct *Account, app string) (string, bool) {
407 rec, _, err := objstore.Get(ctx, s.cfg.Releases, acct.Sqid, app)
408 if err != nil {
409 log.Printf("console: instance store prefix %s/%s: %v", acct.Sqid, app, err)
410 jsonError(w, http.StatusBadGateway, "could not read this app's store settings — try again")
411 return "", false
412 }
413 return storeEffectivePrefix(rec), true
414}
415
416// instancePrivateStorageAllowed is the enable door's half: the two states in
417// which declaring the policy would be a trap rather than a policy (spec §5).
418//
419// A manifest that says "every instance gets a bucket" on a deployment with
420// no object storage, or in an account nobody approved, does not fail when it
421// is written. It fails once per signup, afterwards, in a door whose caller
422// is a machine — and the two sentences below are the only two that name the
423// fix. So they are said here, to the person, at the moment they declare it.
424//
425// Turning the policy OFF is never refused, and that asymmetry is the point:
426// an account whose entitlement was later declined must be able to get its
427// create door working again without an operator.
428func (s *Server) instancePrivateStorageAllowed(ctx context.Context, w http.ResponseWriter, acct *Account) bool {
429 if s.cfg.StoreProvisioner == nil {
430 jsonError(w, http.StatusServiceUnavailable,
431 "object storage is not configured for this deployment (CARLOS_STORE_BACKEND) — an app cannot be given per-instance storage here")
432 return false
433 }
434 e, _, err := objstore.GetEntitlement(ctx, s.cfg.Releases, acct.Sqid)
435 if err != nil {
436 log.Printf("console: instances enable entitlement %s: %v", acct.Sqid, err)
437 jsonError(w, http.StatusBadGateway, "could not read this account's storage approval — try again")
438 return false
439 }
440 if !e.Approved() {
441 jsonError(w, http.StatusForbidden,
442 "this account is not approved for object storage, so its instances cannot be given storage of their own — run `carlos store apply` and ask an operator to approve it first")
443 return false
444 }
445 return true
446}
447