carlos / platform Public

Design doc, 2026-09-04. Follows PR #383 (store-backends-beyond-aws).

1. The problem, as found

CARLOS talks to several external services. Today almost all of them resolve credentials from a single ambient AWS credential chain, and they share a region variable as well. That works when every service is on AWS. It stops working as soon as any one of them is not.

Here is where each client gets its credentials, region and endpoint today.

| site | service | credentials | region | endpoint | |---|---|---|---|---| | release/store.go:440 | deployment bucket (S3) | ambient chain | ambient (AWS_REGION) | CARLOS_S3_ENDPOINT | | lease/endpoint.go:62,104 | hibernation lease, litestream | ambient chain | CARLOS_HIBERNATE_REGION, else ambient | CARLOS_S3_ENDPOINT | | mail/aws.go:105 | SES, IAM, STS | ambient chain | CARLOS_MAIL_REGION | none (real AWS) | | awsinv/awsinv.go:52 | EC2 inventory | ambient chain | ambient, us-east-1 default | none | | console/cepull.go:58 | Cost Explorer | ambient chain | ambient | none | | edge/route53dns.go:60 | Route 53 (ACME) | ambient chain | ambient, us-east-1 fallback | none | | steering/route53.go:116 | Route 53 (steering) | ambient chain | ambient, us-east-1 fallback | none | | dnsrec/route53.go:36 | Route 53 (mail DNS) | ambient chain | ambient, us-east-1 fallback | none | | host/memwatch.go:91 | CloudWatch | ambient chain | ambient, overridden by IMDS | none | | objstore/s3bucket.go:74 | per-app object stores | explicit static, else ambient | CARLOS_STORE_REGION | CARLOS_STORE_ENDPOINT |

Only the last row can be configured independently. That is what PR #383 added, and it was added because the per-app store had to be able to sit on a different provider from the control-plane bucket.

There are three separate couplings here, and they need separating one at a time.

1.1 One credential chain for every AWS-shaped service

AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials is the only credentials-file setting CARLOS has. It is set in infra/remote-host.md:33, printed as an instruction by host/bootstrap.go:171, and documented to operators in templates/fleet_byo.html:29. It is the standard AWS SDK variable, so every LoadDefaultConfig call in the table above picks it up.

The consequence: a deployment whose control-plane bucket is on Hetzner cannot also use SES. mail.NewAWSProvisioner calls STS GetCallerIdentity at construction, which fails with Hetzner keys. Mail construction is deliberately non-fatal (cmd/console/main.go:631), so the console boots with mail silently disabled rather than reporting the real cause.

1.2 AWS_REGION means two different things at once

On a deployment whose bucket is on a compatible store, AWS_REGION is set to that store's location. lease/endpoint.go:37-45 says so directly:

> a compatible-store deployment sets AWS_REGION to its own location (the > private CARLOS uses nbg1), and that is the region release.NewS3Store > already signs for against the same bucket

So on the Hetzner deployment AWS_REGION=nbg1. Every real-AWS client in that process then signs for a region called nbg1. host/memwatch.go:86-90 records the fallout and works around it:

> A compatible-store deployment that points AWS_REGION at its own object > store ... would aim the CloudWatch client at nonsense, which is one more > reason this duty is opt-in per box rather than on wherever credentials > resolve.

Two subsystems have already been made opt-in or IMDS-overridden to avoid a config-model gap. That is the signal that the model is wrong, not the subsystems.

1.3 One S3 endpoint for two subsystems

CARLOS_S3_ENDPOINT is read only by s3compat, and applies only where a site calls s3compat.Options(): release/store.go:444 and lease/endpoint.go:109, plus the litestream replica client. It is not process-wide in effect. But it is a single value, so those subsystems cannot point at different providers.

objstore deliberately does not use it (s3bucket.go:59-73), and threads an endpoint explicitly instead. That is why the per-app store is free. The result is two unrelated vocabularies for the same idea: CARLOS_S3_ENDPOINT and CARLOS_STORE_ENDPOINT.

2. What is already right, and must stay right

The box-side single credential is correct. A BYO-fleet box has one bucket, its own, and templates/fleet_byo.html:44-47 explicitly keeps platform credentials out of that box's env file. One process, one credential is the right answer there. The problem in this document is console-side, where one process talks to several services at once.

A separate credential file per subsystem is already a pattern here. carlos-litestream.service and carlos-restore-verify.service do not read /etc/carlos/host.env like every other unit. They read /data/litestream.env, and carlos-hibernate-verify.service gives the reason:

> those credentials belong to the box-wide replication bucket, and borrowing > them here would sign for the wrong store

This design generalises an existing decision. It does not introduce a new idea.

Credentials are not always in a file. Cloud boxes get them from an IAM instance role (infra/modules/region-host/main.tf:62), resolved through IMDS by the ambient chain. Any model that requires a file per provider would break every cloud box.

AWS-side permissions are already fine-grained. host_route53, host_steering, host_costexplorer and host_metrics are separate, opt-in, per-box IAM policies. The permission model is not the problem. Only the in-process selection of credentials is.

3. The model

Introduce a provider: a named set of coordinates for talking to one vendor. A provider has credentials, a default region, an optional S3 endpoint, and known behaviour.

Each subsystem names the provider it uses, and may override any coordinate.

3.1 Providers

A provider is configured with environment variables prefixed by its name:

AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials.aws
AWS_PROFILE=default
AWS_REGION=eu-central-1

HETZNER_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials.hetzner
HETZNER_REGION=nbg1
HETZNER_S3_ENDPOINT=https://nbg1.your-objectstorage.com

AWS_* keeps its existing SDK meaning. The other prefixes are CARLOS's, but use the same file format, because the AWS SDK can already load an arbitrary credentials file and profile:

• config.WithSharedCredentialsFiles([]string{path})
• config.WithSharedConfigProfile(name)

Both exist in the pinned aws-sdk-go-v2/config@v1.32.6 (load_options.go:449,493). No new INI parser is needed.

Two ways to configure the same thing are supported, because operators run different shapes:

• File per provider. HETZNER_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials.hetzner.
• Profile per provider. One file, sections [aws] and [hetzner], with HETZNER_PROFILE=hetzner. This is the AWS-native idiom and means one mode-0600 file to manage. It is the recommended default in the docs.

A provider with no credentials configured means "use the ambient chain". That is what an EC2 instance role needs, and it is what every existing deployment gets for free.

3.2 Known providers

The registry knows a fixed set of provider names. Each carries more than credentials:

| provider | S3 endpoint | mints store keys | S3 quirks | |---|---|---|---| | aws | none (real AWS) | yes, via IAM | none | | hetzner | required | no | unquoted If-Match | | linode | required | yes, via Linode API | none known | | b2 | required | yes, via B2 API | none known | | minio | required | no | none known | | s3 | required | no | none known |

s3 is the fallback for a compatible store CARLOS has no specific knowledge of. minio exists because CI already stands one up.

This subsumes CARLOS_STORE_BACKEND from PR #383. backend and provider are the same concept under two names. CARLOS_STORE_PROVIDER=hetzner is better than CARLOS_STORE_BACKEND=s3 because the registry knows Hetzner has no key API, instead of the operator having to know to pick the generic backend.

It also fixes a latent bug. s3compat's quirk table matches on endpoint hostname (isHetzner matches .your-objectstorage.com). A Hetzner bucket reached through a CNAME does not match, silently loses the If-Match fix, and every compare-and-swap write starts returning 412. Keyed on provider name, that cannot happen.

3.3 Subsystems

Each subsystem gets a provider and optional overrides:

CARLOS_DEPLOYMENT_PROVIDER=hetzner
CARLOS_DEPLOYMENT_BUCKET=carlos

CARLOS_STORE_PROVIDER=aws
CARLOS_STORE_REGION=eu-central-1
CARLOS_STORE_ACCESS_KEY=AKIA...
CARLOS_STORE_SECRET_KEY=...
CARLOS_STORE_BUCKET_PREFIX=carlos-

CARLOS_MAIL_PROVIDER=aws
CARLOS_MAIL_REGION=eu-west-1

The subsystems are:

| subsystem | prefix | what it covers | |---|---|---| | deployment | CARLOS_DEPLOYMENT_ | control-plane bucket | | store | CARLOS_STORE_ | per-app object stores | | mail | CARLOS_MAIL_ | SES, IAM, STS | | hibernate | CARLOS_HIBERNATE_ | lease and litestream | | dns | CARLOS_DNS_ | Route 53, DNSimple | | inventory | CARLOS_INVENTORY_ | EC2 and friends | | billing | CARLOS_BILLING_ | Cost Explorer | | metrics | CARLOS_METRICS_ | CloudWatch |

3.4 Resolution order

For each coordinate of each subsystem, in order, first match wins:

1. The subsystem's own variable. CARLOS_STORE_REGION, CARLOS_STORE_ACCESS_KEY / CARLOS_STORE_SECRET_KEY, CARLOS_STORE_ENDPOINT.
2. The named provider's variable. AWS_REGION, AWS_SHARED_CREDENTIALS_FILE / AWS_PROFILE, AWS_S3_ENDPOINT.
3. The provider's built-in default, where it has one. For aws, no endpoint. For hetzner, the If-Match quirk.
4. The ambient chain, for credentials only.

An access key set without a secret, or the reverse, is a boot-time error, not a silent fallback to the ambient chain. A half-set pair is a typo, and falling back turns it into a permissions error hours later.

Subsystems with no _PROVIDER set default to aws. That makes every existing deployment behave exactly as it does now.

4. Worked example

The shape this design exists to support:

AWS_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials.aws
AWS_REGION=eu-central-1
HETZNER_SHARED_CREDENTIALS_FILE=/etc/carlos/credentials.hetzner
HETZNER_REGION=nbg1
HETZNER_S3_ENDPOINT=https://nbg1.your-objectstorage.com

# Control-plane bucket on Hetzner, using the Hetzner credentials file.
CARLOS_DEPLOYMENT_PROVIDER=hetzner
CARLOS_DEPLOYMENT_BUCKET=carlos

# Per-app stores on AWS, with their own key rather than the shared file.
CARLOS_STORE_PROVIDER=aws
CARLOS_STORE_REGION=eu-central-1
CARLOS_STORE_ACCESS_KEY=AKIA...
CARLOS_STORE_SECRET_KEY=...
CARLOS_STORE_BUCKET_PREFIX=carlos-

# Mail on AWS, using the shared AWS credentials file.
CARLOS_MAIL_PROVIDER=aws
CARLOS_MAIL_REGION=eu-west-1

Nothing in this example is expressible today.

5. The change, file by file

New: internal/provider. The registry. Resolves a subsystem name to a provider.Config holding credentials, region, endpoint and capabilities. Returns AWS SDK load options so callers do not each hand-roll them. Linked by both binaries, so it cannot depend on console-only packages.

internal/s3compat. Quirks move to keying on provider name rather than endpoint hostname. Endpoint() becomes the deployment subsystem's endpoint rather than a free-floating global. Options() gains a provider argument.

internal/release/store.go. NewS3Store takes a provider.Config instead of reading the ambient chain. The deployment subsystem supplies it.

internal/lease/endpoint.go. Same, for the hibernate subsystem. The three-tier region logic in regionFor collapses into the resolution order above.

internal/mail/aws.go. NewAWSProvisioner takes a provider.Config rather than a bare region.

internal/objstore. S3Config gains credentials-file and profile fields. CARLOS_STORE_BACKEND becomes CARLOS_STORE_PROVIDER, and the capability table moves into the registry.

internal/awsinv, console/cepull.go, the three Route 53 writers, host/memwatch.go. Each takes a provider.Config. The us-east-1 fallbacks become provider defaults.

cmd/console/main.go. storeBackend becomes one call into the registry per subsystem.

Docs. infra/remote-host.md, templates/fleet_byo.html, docs/site/cli.md.

6. Migration

Every rename keeps its old name working for one release, reading the old variable when the new one is unset and logging once at boot:

| old | new | |---|---| | CARLOS_STORE_BACKEND | CARLOS_STORE_PROVIDER | | CARLOS_S3_ENDPOINT | HETZNER_S3_ENDPOINT etc., via the deployment provider | | CARLOS_STORE_ENDPOINT | provider endpoint, or the subsystem override |

A deployment that sets none of the new variables keeps working. Subsystems default to provider aws, and provider aws with nothing configured is the ambient chain, which is exactly today's behaviour.

AWS_REGION keeps working as the aws provider's region. The deployments that currently set it to a Hetzner zone should move that value to HETZNER_REGION and set CARLOS_DEPLOYMENT_PROVIDER=hetzner. Until they do, they behave as they do now. This is the one migration that needs a note in the release announcement, because it is the one that is currently wrong in a way people have worked around.

7. What is not built

• No new providers for non-object-storage services. There is no SES alternative, no Route 53 alternative. Those subsystems get a provider field so their credentials can be separated, not so they can be swapped. Naming CARLOS_MAIL_PROVIDER=postmark is out of scope.
• No credential rotation or fetching. The registry reads configuration. It does not talk to a secret manager.
• No per-account or per-tenant providers. A provider is deployment-wide. Per-app stores are already per-app in bucket and credential; they do not also need to be per-provider.
• No change to the box-side single-credential model. BYO-fleet boxes keep one credentials file.

8. Testing

• Table tests for resolution order: subsystem override beats provider variable beats provider default beats ambient chain, per coordinate.
• A half-set key pair is a boot error, per subsystem.
• Provider-keyed quirks: a Hetzner provider reached through a hostname that does not look like Hetzner still gets the If-Match fix. This is the regression the current hostname matching allows.
• Back-compat: a deployment setting only today's variables resolves to today's clients. Assert on the resolved config, not on behaviour, so the test says which variable was read.
• The existing MinIO contract test (.amadan/ci.d/30-minio-contract) covers the delivered credential path and should keep passing untouched.

9. Rollout

Staged, because the subsystems do not conflict equally.

Phase 1 — the ones that actually conflict. Deployment, store, mail. This is the set that makes "bucket on Hetzner, mail on SES" possible, which is the motivating case. Everything else keeps reading the ambient chain and is unaffected.

Phase 2 — the rest. Hibernate, DNS, inventory, billing, metrics. Mostly mechanical once the registry exists. This phase can remove the two existing workarounds: CloudWatch's opt-in gate and its IMDS region override both exist because of the coupling described in §1.2.

Phase 3 — deprecation. Remove the old variable names one release later.

10. Open questions

1. File per provider, or profile per provider? Both are supported above. The question is which the documentation teaches as the default. Profiles mean one file to manage and match AWS convention. Files are more obvious and easier to permission separately.
2. Coordination with pinfra. The console's systemd unit and env file are not in this repo (cmd/console/sdnotify.go:28-29), and there is no SES IAM grant in this repo's Terraform. Phase 1 changes the console's configuration surface, so the rollout has to land in both repos together.
3. *Does `CARLOSHIBERNATE` become a subsystem or fold into deployment?** The lease and the control-plane bucket are usually the same store, but on BYO-fleet boxes they are deliberately not. Keeping them separate is proposed above; folding them would be simpler and would break that case.

11. Naming note

CARLOS_REGION already exists and means a placement code, not an AWS region. Its values look like fleet/<name> (host/edgesfile.go:82-85, console/fleets.go:535). STATUS_BOX_REGION is a human-readable label, also not a signing region. Neither should be folded into the provider model, and new names should not blur into them.