amadan
Design: fold in the code and AWS reviews
Per-instance tokens become a prerequisite, names get a hash slug inside the IAM and SES bounds, configuration sets stay per account because an identity has only one default, per-instance mode is a CAS flag that deactivates the app key, and SES's own tenant pauses are reconciled.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 file changed,
+411
−307
docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md+411 −307
diff --git a/docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md b/docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md| index d2a99bbb8..5fd37a42d 100644 |
| --- a/docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md |
| +++ b/docs/superpowers/specs/2026-09-15-tenant-email-senders-design.md |
| @@ -6,7 +6,8 @@ address under an SES identity CARLOS does not own) and **#148** |
| customer of a multi-tenant app). |
| Base: `origin/main` at 520fcc04. Author: Paul Campbell with Claude. |
| -Date: 2026-09-15. |
| +Date: 2026-09-15. Revised after two reviews: one against the code, one |
| +against the AWS documentation. |
| ## 1. The problem |
| @@ -19,292 +20,354 @@ default config bundle. Two requests do not fit that shape. |
| both must send as `jelly@tito.io`. `tito.io` is a verified SES identity |
| that Tito Classic has owned for years in the same AWS account. The |
| existing walk would adopt it and then rewrite it: `EnsureIdentity` |
| -moves MAIL FROM to `mail-eu-west-1.tito.io`, `SetIdentityConfigSet` |
| -binds `carlos-<acct>` as the identity's default configuration set, the |
| -domain claim is first-come so the second account is refused, and |
| -`Teardown` calls `DeleteIdentity`. |
| +moves MAIL FROM to `mail-eu-west-1.tito.io` (`aws.go:292`), |
| +`SetIdentityConfigSet` binds `carlos-<acct>` as the identity's default |
| +configuration set (`walk.go:301`), the domain claim is first-come |
| +(`domainclaim.go:119`), and `Teardown` calls `DeleteIdentity` |
| +(`walk.go:1205`). |
| **#148.** titogo runs one instance per customer under a single app. |
| -Each customer needs their own domain, their own credential, their own |
| -cap and their own pause. Today `EnsureUser` writes one policy listing |
| -every domain granted to the app, and that one credential is delivered |
| -to every instance, so code in one customer's instance can send as every |
| -other customer's domain. That is the blocker. |
| +Each customer needs their own domain, credential, cap and pause. Today |
| +`EnsureUser` writes one policy listing every domain granted to the app |
| +(`walk.go:634`), delivered to every instance through the app's default |
| +bundle (`walk.go:818`), so code in one customer's instance can send as |
| +every other customer's domain. |
| ## 2. Decisions taken before this document |
| 1. **Extend the IAM-per-sender model now, and iterate towards a |
| platform SMTP relay.** The relay is wanted anyway, as a layer over |
| - SES that could also front a different SMTP provider. This design |
| - keeps that door open (§9) rather than building it. |
| -2. **Per-instance mode.** Once an app has instance senders, instances |
| - no longer receive the app's credential at all. |
| + SES that could front a different SMTP provider. §10 keeps that door |
| + open rather than building it. |
| +2. **Per-instance mode.** Once an app has instance senders, instances no |
| + longer receive the app's credential at all. |
| 3. **titogo's backend drives domain onboarding through the console |
| - API.** The CLI is a wrapper over the same doors. |
| + API**, with the CLI as a wrapper over the same doors. |
| 4. **For #147, an operator registers the external identity once and may |
| mark it open; accounts then claim addresses under it themselves.** |
| - |
| -## 3. The sender |
| - |
| -A **sender** is the unit that owns a From address, a credential, a cap, |
| -a pause and a set of counts. |
| +5. **Per-instance machine tokens are a prerequisite for #148** (§6.1). |
| + Today `CARLOS_INSTANCE_TOKEN` is delivered through the app-wide |
| + bundle, so every customer's instance holds a token that reaches every |
| + other instance of the app. Opening instance email doors to |
| + `svcrole.Instance` without scoping that token would move #148's leak |
| + from the credential to the API. |
| +6. **Instance backing is mixed today, and moving to the isolating one is |
| + the intent.** So `enable` checks the route's backing rather than |
| + assuming it (§6.4). |
| + |
| +## 3. What the AWS documentation actually says |
| + |
| +The first draft of this design got three of these wrong, so they are |
| +recorded here with their sources. |
| + |
| +- **Condition keys.** The v1 action set, which is what SMTP authorises |
| + against, has `ses:ApiVersion`, `ses:FeedbackAddress`, |
| + `ses:FromAddress`, `ses:FromDisplayName` and `ses:Recipients`. SES v2 |
| + adds `ses:TenantName` and `ses:MultiRegionEndpointId`. **No condition |
| + key names a configuration set.** The configuration set is a *resource* |
| + on the send action, which constrains which set may be named but cannot |
| + force one to be named — so a credential cannot be compelled to publish |
| + events. |
| +- **`ses:TenantName` is documented for `SendEmail` and `SendBulkEmail` |
| + only.** The `SendRawEmail` row, which is what the SMTP interface |
| + authorises against, does not list it, although an AWS blog publishes a |
| + policy that uses it there. This is the first spike (§4). |
| +- **There is no tenant-level default configuration set.** A send on |
| + behalf of a tenant must either name a configuration set associated |
| + with that tenant, or use an identity whose *default* set is associated |
| + with it. So per-sender configuration sets on a shared identity are |
| + impossible, and an address sender under a borrowed identity can only |
| + publish events if the message names CARLOS's set. |
| +- **Associating a resource with a tenant does not modify it**, and one |
| + identity may belong to many tenants. It has one side effect on the |
| + owner: an identity cannot be deleted while an association exists. |
| +- **SES pauses tenants by itself.** Every new tenant gets the Standard |
| + reputation policy, which auto-pauses on high-severity findings, and |
| + Trust & Safety can pause one too. `SetKeyActive(true)` cannot lift |
| + that. Sending quotas remain account-wide per region; the account |
| + suppression list still applies unless a tenant opts into its own. |
| +- **`ses:caller-identity` names the IAM user** on Send, Bounce, |
| + Complaint, Delivery and Reject, but only for events published through |
| + a configuration set, and not on Rendering Failure or DeliveryDelay. |
| + |
| +## 4. Spike first |
| + |
| +Half a day against the Tito deployment's AWS account, on a throwaway |
| +identity and tenant. It is blocking, because two of its answers change |
| +the policy this design writes. |
| + |
| +1. **Send over SMTP with `X-SES-TENANT` set, under a policy conditioned |
| + on `ses:TenantName`.** Then send again without the header. Three |
| + outcomes: both succeed (the key is not populated for SMTP, so the |
| + condition buys nothing); tagged succeeds and untagged is refused (the |
| + design works as written); both are refused (SES does not populate the |
| + key at all and the condition denies everything). In outcome one and |
| + three the tenant condition comes out of the policy, and isolation |
| + rests on `ses:FromAddress` plus SES's own tenant-association |
| + validation. |
| +2. **Send under a tenant without `X-SES-CONFIGURATION-SET`** on an |
| + identity whose default set is not associated with that tenant, to |
| + confirm it is refused, and with the header to confirm events arrive. |
| +3. **Associate an identity created outside CARLOS** and confirm its MAIL |
| + FROM, DKIM and default configuration set are unchanged, and that |
| + sends on it *without* a tenant still work — Classic's mail must not |
| + change behaviour the moment CARLOS associates `tito.io`. |
| +4. **Pause a tenant** with `UpdateReputationEntityCustomerManagedStatus` |
| + and confirm `GetTenant.SendingStatus` reflects it, so §8 can adopt it |
| + as the pause verb. |
| + |
| +Record the outcomes in this document before plans are written. |
| + |
| +## 5. The sender |
| + |
| +A **sender** owns a From address, a credential, a cap, a pause and a set |
| +of counts. |
| ```go |
| // internal/mail/sender.go |
| type Sender struct { |
| Schema int `json:"schema"` // 1 |
| - ID string `json:"id"` // "app", or the instance id |
| + ID string `json:"id"` // "app", or provision.InstanceID(host) |
| + Slug string `json:"slug"` // AWS-safe name component, see below |
| Scope string `json:"scope"` // ScopeApp | ScopeInstance |
| - Host string `json:"host,omitempty"` // instance host, scope=instance |
| + Host string `json:"host,omitempty"` |
| Source string `json:"source"` // SourceDomain | SourceAddress |
| - Declared *Declared `json:"declared,omitempty"` |
| - Grant *GrantBlock `json:"grant,omitempty"` |
| - Provisioned *Provisioned `json:"provisioned,omitempty"` |
| + Declared *Declared `json:"declared,omitempty"` |
| + Grant *GrantBlock `json:"grant,omitempty"` |
| + Provisioned *Provisioned `json:"provisioned,omitempty"` |
| Standalone []*Standalone `json:"standalone,omitempty"` |
| - Identity string `json:"identity,omitempty"` // external identity ARN, source=address |
| - Address string `json:"address,omitempty"` // exact address, source=address |
| + Identity string `json:"identity,omitempty"` // external identity ARN |
| + Address string `json:"address,omitempty"` // exact address, lowercased |
| + TenantARN string `json:"tenant_arn,omitempty"` // from CreateTenant |
| + PerInstance bool `json:"per_instance,omitempty"` // app sender only, §6.3 |
| } |
| ``` |
| -- `ScopeApp` is what exists today; `ScopeInstance` is #148. |
| -- `SourceDomain` is a domain CARLOS verified and may write to; |
| - `SourceAddress` is one address under an identity CARLOS does not own |
| - (#147). |
| -- `Declared`, `GrantBlock`, `Provisioned` and `Standalone` keep their |
| - current shapes and move inside the sender unchanged. |
| +`Declared`, `GrantBlock`, `Provisioned` and `Standalone` keep their |
| +current shapes and move inside the sender unchanged. |
| **Records.** `apps/<acct>/<app>/email/senders/<id>.json`, one object per |
| sender, each with its own CAS etag so two instances never contend. |
| -`<id>` is `app` for the app sender and `provision.InstanceID(host)` for |
| -an instance sender (already bounded at 40 characters, which keeps the |
| -IAM user name inside its 64-character limit). |
| - |
| -**Migration.** `mail.Get(acct, app)` keeps working: it reads |
| -`email.json` when no `senders/app.json` exists and returns it as the app |
| -sender. The first write through a sender-aware path writes |
| -`senders/app.json` and leaves `email.json` in place as a tombstone for |
| -one release. Nothing has to be migrated before a roll, and a console |
| -rollback still finds `email.json`. |
| - |
| -**Per-instance mode.** An app is in per-instance mode when any |
| -`senders/<id>.json` with `Scope == ScopeInstance` exists. Entering it |
| -unsets the app sender's `CARLOS_SMTP_*` vars from the app's default |
| -bundle (§6) and refuses `carlos email enable` for the app sender until |
| -the last instance sender is gone. The app sender's own record and its |
| -domains survive, so leaving per-instance mode is a re-enable, not a |
| -re-claim. |
| - |
| -**Naming.** One IAM user, one SES tenant and one configuration set per |
| -sender: |
| - |
| -| | app sender | instance sender | |
| -|---|---|---| |
| -| IAM user | `carlos-mail-<acct>-<app>` (unchanged) | `carlos-mail-<acct>-<app>+i+<id>` | |
| -| tenant | `carlos-<acct>-<app>` | `carlos-<acct>-<app>-<id>` | |
| -| config set | `carlos-<acct>-<app>` | `carlos-<acct>-<app>-<id>` | |
| - |
| -The account-wide tenant and configuration set (`carlos-<acct>`) stay for |
| -existing app senders until they are re-enabled; `ConfigSetName` gains a |
| -sender argument and the old name remains the value for the app sender |
| -of an account that has not been re-enabled, so no live identity gets |
| -rebound by a deploy. |
| - |
| -## 4. SES tenants carry the isolation |
| - |
| -SES's IAM condition keys are only `ses:Recipients`, `ses:FromAddress`, |
| -`ses:FromDisplayName` and `ses:FeedbackAddress`. There is no condition |
| -key for a configuration set, so a credential cannot be forced to publish |
| -events — which is how a sender could evade its cap. |
| - |
| -SES tenants do have one, `ses:TenantName`, and `StringEquals` on an |
| -absent key fails, so a policy conditioned on the tenant refuses any |
| -message that does not name it. Tenants also give each sender its own |
| -reputation, and `CreateTenantResourceAssociation` associates an existing |
| -identity with a tenant **without modifying the identity**, which is |
| -exactly what #147 needs. |
| - |
| -So each sender's policy is: |
| - |
| -``` |
| -Allow ses:SendRawEmail |
| - Resource <identity ARN(s)> + <sender config set ARN> |
| - Condition StringEquals ses:TenantName = <sender tenant> |
| - StringEquals ses:FromAddress = <address> (source=address) |
| - StringLike ses:FromAddress = ["*@<domain>", …] (source=domain) |
| -``` |
| - |
| -`SendPolicyJSON` grows a sender argument and an exact-address form; its |
| -existing refusals (no identities, no domains, wildcards, malformed ARNs, |
| -an identity that is not a parent of a granted domain) all stay, with the |
| -address form checking that the address's domain is the identity or a |
| -child of it. |
| - |
| -**Spike first (blocking, half a day).** Against the Tito deployment's |
| -AWS account, confirm on a throwaway identity and tenant: |
| - |
| -1. `ses:TenantName` is enforced for SMTP sends, not only the v2 API — a |
| - send without `X-SES-TENANT` must be refused. |
| -2. A tenant's associated configuration set is applied to a message |
| - automatically, so events arrive without the app setting |
| - `X-SES-CONFIGURATION-SET`. |
| -3. `CreateTenantResourceAssociation` against an identity CARLOS did not |
| - create leaves that identity's MAIL FROM, default configuration set |
| - and DKIM untouched. |
| -4. Whether tenant-level sending can be paused directly, which would be a |
| - better pause than `SetKeyActive`. |
| - |
| -If (1) or (2) fails, the isolation still holds through `ses:FromAddress` |
| -and the per-sender IAM user, but counting an address sender depends on |
| -the app's own header. Record that outcome in this document before the |
| -plans are written; it changes §7's fallback, not the record shapes. |
| - |
| -## 5. #147: a sender under an identity CARLOS does not own |
| -**The external identity** is registered once per deployment by an |
| -operator, at `email/external-identities/<domain>.json`: |
| +**Names.** The first draft's names were invalid: SES caps tenant and |
| +configuration-set names at 64 characters of `[A-Za-z0-9_-]`, IAM caps a |
| +user name at 64, and an instance id is the host itself when it is 40 |
| +characters or less, dots included — `carlos-mail-<acct>-<app>+i+<id>` |
| +reaches 104 characters. So each sender gets a `Slug`, computed once at |
| +creation and stored on the record: |
| + |
| +- app sender: `Slug = ""`, names unchanged |
| + (`carlos-mail-<acct>-<app>`, tenant and set `carlos-<acct>`). |
| +- instance sender: `Slug = <first 8 of base32(sha256(acct/app/id))>`, |
| + giving IAM user `carlos-mail-<acct>-<app>+i+<slug>` and tenant |
| + `carlos-<acct>-<app>-<slug>`, both bounded at 12 + 16 + 1 + 32 + 3 + 8 |
| + = 72 — still over 64 for the longest account and app, so the app |
| + component is itself truncated to 24 characters with the hash carrying |
| + uniqueness. The exact rule lives in one function, `SenderNames`, with |
| + a test that enumerates the bounds. |
| + |
| +Because the slug is a hash, nothing parses it back. `SenderOfCaller` |
| +(§8) resolves a caller identity by looking up the recorded `IAMUser`, |
| +not by decomposing the name. |
| + |
| +**Configuration sets stay per account** (`carlos-<acct>`), because an |
| +identity has only one default set and §3 rules out per-sender sets on a |
| +shared identity. Each sender's tenant is associated with that set, and |
| +the set's ARN stays in the sender's policy `Resource`. |
| + |
| +**Migration.** `mail.Get(acct, app)` prefers `senders/app.json` and |
| +falls back to `email.json`; `mail.Put` **always** writes |
| +`senders/app.json`, create-only on first write since the old etag does |
| +not carry over. `email.json` is left in place, unwritten, for one |
| +release and then deleted by a sweep. A console rolled back after the |
| +first sender write loses writes made in between — a pause or a rotation |
| +would have to be redone — which is recorded here and in the roll notes. |
| + |
| +## 6. #148: instance senders |
| + |
| +### 6.1 Prerequisite: per-instance machine tokens |
| + |
| +`handleAPIInstanceToken` mints a token bound to (account, app), and the |
| +documented delivery path is `carlos secrets set -app <app>`, i.e. the |
| +bundle every instance renders. Every customer's instance therefore holds |
| +a credential that reaches every other instance of the app. |
| + |
| +Before any instance email door opens, the platform mints a token bound |
| +to (account, app, instance) and delivers it through the per-instance |
| +config layer (§6.2) as `CARLOS_INSTANCE_TOKEN`, replacing the app-wide |
| +one for instances that have it. `Principal` gains an `Instance` field |
| +alongside `App`, populated from a new `cli_tokens.instance` column; |
| +`apiAccountApp` grows an instance check beside its existing app check, |
| +so a token bound to instance A gets the same byte-identical 404 on |
| +instance B that a stranger gets. Tokens without an instance binding keep |
| +today's reach, so nothing breaks while the fleet rolls. |
| + |
| +This is its own plan, and it lands before the email doors (§11). |
| + |
| +### 6.2 Per-instance config delivery |
| + |
| +A third bundle layer, resolved after the named environment: |
| -```go |
| -type ExternalIdentity struct { |
| - Schema int `json:"schema"` // 1 |
| - Domain string `json:"domain"` // tito.io |
| - Region string `json:"region"` |
| - IdentityARN string `json:"identity_arn"` |
| - Open bool `json:"open"` // any account may claim an address |
| - RegisteredBy string `json:"registered_by"` |
| - At time.Time `json:"at"` |
| -} |
| +``` |
| +apps/<acct>/<app>/config.json default |
| +apps/<acct>/<app>/environments/<name>.json environment |
| +apps/<acct>/<app>/instances-config/<host>.json instance ← new |
| ``` |
| -CARLOS never calls `CreateEmailIdentity`, |
| -`PutEmailIdentityMailFromAttributes`, `SetIdentityConfigSet` or |
| -`DeleteIdentity` for a registered external identity. The only call it |
| -makes is `IdentityStatus` (a `GetEmailIdentity` read) to confirm the |
| -identity is verified and sending-enabled at registration and at enable |
| -time. The walk asserts this: a new `Provisioner` wrapper refuses those |
| -four verbs for any domain that has an external-identity record, so a |
| -future edit cannot reach them by accident, and a test pins it. |
| - |
| -**Address claims.** `email/address-claims/<address>/<acct>.json` records |
| -that an account sends as an address. Unlike the domain claim it is |
| -**not** exclusive: `jam` and `stripe` may both hold `jelly@tito.io`, and |
| -the record names the account, the app, the sender id and who claimed it. |
| -Claiming requires `Open == true` on the external identity; otherwise an |
| -operator grant is needed (the same record, written by an operator). |
| -Releasing a sender deletes its own claim only. |
| - |
| -`ListAddressClaims` gives an operator the roster: who is sending as what |
| -under an identity they registered. `Teardown` of an address sender |
| -deletes the IAM user, the tenant association and the claim, and never |
| -touches the identity. |
| - |
| -**The trade-off you accepted.** With `Open == true`, any account in the |
| -deployment can claim any address under `tito.io`, including |
| -`billing@tito.io`. The mitigations available, in order of cost: leave |
| -`Open` false and grant per address; or add an optional `Local []string` |
| -allowlist of local parts on the external identity record. This design |
| -implements `Open` plus the operator-grant path, and leaves `Local` out |
| -until it is asked for. |
| - |
| -**Enable.** `carlos email enable --app jelly --from jelly@tito.io` |
| -resolves `tito.io` to an external identity, checks or takes the address |
| -claim, creates the sender's tenant, associates the identity and the |
| -configuration set with it, renders the address-form policy and mints the |
| -key. No DNS records are printed, because the identity's DNS belongs to |
| -whoever owns it. |
| - |
| -## 6. #148: instance senders and per-instance delivery |
| - |
| -**Onboarding a customer domain** (titogo's backend, with its existing |
| -app token): |
| +- Same `Manifest` shape, same sealed envelopes, same name and value |
| + rules. `ExecDelivery` is **not** read from this layer, matching the |
| + ruling at `appconfig.go:61-69`. |
| +- `LoadMerged` keeps its signature; `LoadMergedHost(ctx, s, acct, app, |
| + env, host)` adds the third overlay, and `Sync`/`SyncExec` call it per |
| + route after resolving the group base, so a pass costs one extra bucket |
| + read per instance that has an overlay and none for instances that do |
| + not. |
| +- Writing the layer bumps the app's generation object, as |
| + `SetExecDeliveryAt` does, so the fast loop converges. |
| +- Guards that assume a two-layer merge and must be revisited: |
| + `lostKeys`/`defaultMissing` (`sync.go:260`, `:388`), the mass-removal |
| + guard (`syncexec.go:351`), `allShadowed`/`AwaitingRestart` |
| + (`syncexec.go:712`, `:796`). `ConfigRouteFingerprints` |
| + (`routefingerprint.go:58`) digests registry columns only and cannot |
| + see a bucket-side change; the generation bump is that layer's signal. |
| +- Only the platform writes this layer: no CLI verb, no console door. It |
| + exists so a credential can be scoped to one instance. |
| + |
| +### 6.3 Entering per-instance mode |
| + |
| +A prefix listing cannot decide this without racing an app-sender enable, |
| +so the app sender carries `PerInstance bool`, and the transition is one |
| +CAS write on `senders/app.json` that, in order: |
| + |
| +1. Sets `PerInstance`. |
| +2. `SetKeyActive(false)` on the app sender's key, its previous key and |
| + every standalone credential — otherwise the credential already |
| + delivered to every instance keeps authorising sends for every domain, |
| + and addr-shaped routes are never bounced (`syncexec.go:440`), so the |
| + old environment survives in running processes. |
| +3. Unsets the app sender's `CARLOS_SMTP_*` vars from the default bundle. |
| + |
| +App-sender `Enable` refuses when `PerInstance` is set, under that same |
| +etag. Instance `Enable` requires it. Leaving per-instance mode, once the |
| +last instance sender is gone, is a re-enable of the app sender. |
| + |
| +`grantedDomains` filters on the domain record's new `Senders` list, not |
| +`Apps`, and an instance sender's claim never adds the app to `Apps`. So |
| +the app sender's policy can never name a customer domain. |
| + |
| +### 6.4 The doors, and what they check |
| ``` |
| -POST /api/cli/apps/{account}/{app}/instances/{host}/email declare |
| -POST /api/cli/apps/{account}/{app}/instances/{host}/email/domains |
| -POST /api/cli/apps/{account}/{app}/instances/{host}/email/domains/{domain}/verify |
| -POST /api/cli/apps/{account}/{app}/instances/{host}/email/enable |
| -POST /api/cli/apps/{account}/{app}/instances/{host}/email/rotate |
| -GET /api/cli/apps/{account}/{app}/instances/{host}/email status |
| -DELETE /api/cli/apps/{account}/{app}/instances/{host}/email teardown |
| +POST /api/cli/apps/{account}/{app}/instances/{host}/email declare |
| +POST .../instances/{host}/email/domains attach |
| +POST .../instances/{host}/email/domains/{domain}/verify |
| +POST .../instances/{host}/email/enable |
| +POST .../instances/{host}/email/rotate |
| +GET .../instances/{host}/email status |
| +DELETE .../instances/{host}/email teardown |
| ``` |
| -Roles: every door above is `Allow(svcrole.Instance)`, so |
| -`CARLOS_INSTANCE_TOKEN` reaches them, and the widening is recorded in |
| -`exemptFromLegacyPin` alongside `TestInstanceRoleMatchesTodaysAppTokenReach`. |
| -Pause, resume and cap changes stay where they are — operator only, |
| -through the existing `POST …/email/pause` with an added `instance` |
| -field. |
| - |
| -This is a real widening: an app token can now claim a domain, mint a |
| -sending credential for it and deliver it into one of its own instances. |
| -It cannot raise a cap, lift a pause, reach another app, or send as any |
| -domain it has not verified, and the credential it mints lands in the |
| -config of an instance whose environment that app already controls. |
| +All are `Allow(svcrole.Instance)`, recorded in `exemptFromLegacyPin` |
| +beside `TestInstanceRoleMatchesTodaysAppTokenReach`, and reachable only |
| +by a token bound to that instance (§6.1). Pause, resume and cap changes |
| +stay operator-only through the existing `POST …/email/pause`, which |
| +gains an `instance` field. |
| + |
| +Two checks at `enable`, both refusals with an explaining message: |
| + |
| +- **Backing.** The instance's route must isolate its config file from |
| + siblings: unit-backed (systemd reads the 0600 file as root and drops |
| + to a per-unit `DynamicUser`), or exec-backed under an isolated-exec |
| + layout with a distinct uid. Plain exec-backed routes share the agent's |
| + uid, so `<dataDir>/<host>.config.env` is readable by every sibling and |
| + a per-instance credential buys nothing (`admintoken.go:6-12`). This is |
| + the dependency on #126's per-instance uids. |
| +- **Delivery.** The instance's box must ack the new config-ack schema |
| + (§6.5), otherwise its agent cannot render the instance layer and the |
| + credential would never arrive. |
| + |
| +**Domain attach.** `claimDomain` gains a `Sender` field and refuses a |
| +re-claim whose (app, sender) differs from the holder's, *including |
| +within the same account* — today a same-account re-claim is a silent |
| +no-op (`domainclaim.go:137`), which would let one instance attach |
| +another's verified domain. `EnsureDomain` refuses a domain that has an |
| +external-identity record before it claims anything, and the orphan-claim |
| +sweep in `Teardown` matches on sender and skips external domains, so a |
| +refused `EnsureIdentity` can never wedge an app's teardown. |
| + |
| +### 6.5 Confirming delivery |
| + |
| +`ConfigAck` gains `Schema int` and `InstanceConfigETag string`: the etag |
| +of the instance overlay the agent rendered. The console compares it with |
| +the etag its own `Put` returned, which is decidable — unlike a key-id |
| +map, which cannot distinguish an old agent from a new agent rendering a |
| +route with no mail vars. `Schema` is what the enable-time delivery check |
| +reads. `carlos email status --instance <host>` reports delivered, stale |
| +or never delivered. |
| + |
| +## 7. #147: a sender under an identity CARLOS does not own |
| -The domain walk is the existing one with the sender in place of the |
| -app: the claim records (account, app, sender), `DomainRecord` gains |
| -`Senders []string` (`app` or an instance id) beside `Apps`, and an |
| -identity is deleted only when both lists are empty. `Apps` stays |
| -populated for the app sender so a rollback still reads it. |
| - |
| -**Per-instance config delivery.** A third bundle layer, resolved after |
| -the named environment: |
| - |
| -``` |
| -apps/<acct>/<app>/config.json default |
| -apps/<acct>/<app>/environments/<name>.json environment |
| -apps/<acct>/<app>/instances-config/<host>.json instance ← new |
| -``` |
| +**The external identity** is registered once per deployment by an |
| +operator at `email/external-identities/<domain>.json`: domain, region, |
| +identity ARN, `Open bool`, who registered it and when. The region must |
| +match the deployment's, because tenants and identities are per region. |
| -- Same `Manifest` shape, same sealed-secret envelope, same |
| - `ValidVarName`/`ValidVarValue` rules. `ExecDelivery` is **not** read |
| - from this layer, matching the ruling in `appconfig.go:61-69`. |
| -- `LoadMerged` keeps its signature; a new `LoadMergedHost(ctx, s, acct, |
| - app, env, host)` adds the third overlay, and `Sync`/`SyncExec` call it |
| - per route after resolving the group base. The group caches stay, so a |
| - pass costs one extra bucket read per instance that has an overlay, and |
| - none for instances that do not. |
| -- Writing the layer bumps the app's generation object, exactly as |
| - `SetExecDeliveryAt` does, so the fast loop converges. |
| -- The guards that must be revisited, because they assume a two-layer |
| - merge: `lostKeys`/`defaultMissing` (`sync.go:260`, `:388`), the |
| - mass-removal guard (`syncexec.go:351`), `allShadowed`/`AwaitingRestart` |
| - (`syncexec.go:712`, `:796`), and `ConfigRouteFingerprints` |
| - (`routefingerprint.go:58`), which digests registry columns only and so |
| - cannot see a bucket-side overlay change — the generation bump is the |
| - signal for that layer. |
| -- Only the platform writes this layer in this design. No CLI verb, no |
| - console door: it exists so a credential can be scoped to one instance. |
| - |
| -**Delivery, confirmed.** `ConfigAck` has no generation field, so |
| -following the relay's proven pattern, the ack gains |
| -`MailKeyIDs map[string]string` (host → delivered `_USER` value) and the |
| -console compares it exactly against the sender's current `KeyID`, the |
| -way `relayMatchingAck` compares `RelaySuffixes`. `carlos email status |
| ---instance <host>` then says delivered, stale or never delivered rather |
| -than guessing from `missing_vars`. |
| - |
| -**Rolling.** A box running an older agent ignores the instance layer, so |
| -its instances get no `CARLOS_SMTP_*` and the status reads as not |
| -delivered. Boxes must be rolled before instance senders are enabled on |
| -them; the console refuses `enable` for an instance whose box has not |
| -acked the new ack shape. |
| - |
| -## 7. Counting, caps and pauses per sender |
| - |
| -**Attribution by caller, not by domain.** Every sender has its own IAM |
| -user, and SES already puts the caller identity in |
| -`mail.tags["ses:caller-identity"]`, which `mail.AppOfCaller` already |
| -parses (it currently discards the `+s+` suffix). Extend it to |
| -`SenderOfCaller(caller) (acct, app, senderID string, ok bool)`, which |
| -resolves an address sender too — the case where the domain tells you |
| -nothing, because the domain is `tito.io` and belongs to Classic. |
| - |
| -`recordAndEnforceMail` then: |
| - |
| -1. Resolves the sender from the caller identity and counts there, once. |
| -2. Falls back to today's domain-wide loop only when the caller is |
| - unknown, and skips any app in per-instance mode, whose events must |
| - never be charged to the app sender. |
| +CARLOS never calls `CreateEmailIdentity`, |
| +`PutEmailIdentityMailFromAttributes` (both inside `EnsureIdentity`), |
| +`SetIdentityConfigSet` or `DeleteIdentity` for a registered identity. A |
| +provisioner wrapper refuses those verbs for any domain with such a |
| +record, and a test pins the refusal. `EnsureDomain` refuses the domain |
| +before claiming it (§6.4), so the refusal can never strand a claim. |
| + |
| +**Verification.** `Status.Verified()` requires `MailFrom == "SUCCESS"` |
| +(`provisioner.go:45`), which an identity with no custom MAIL FROM never |
| +reaches. External identities are checked with a relaxed predicate: DKIM |
| +and sending enabled, MAIL FROM ignored. |
| + |
| +**Address claims.** `email/address-claims/<address>/<acct>.json`, where |
| +the address is lowercased and validated to the same grammar `Declared` |
| +already enforces. The claim is **not** exclusive: `jam` and `stripe` may |
| +both hold `jelly@tito.io`. It names the account, app, sender and who |
| +claimed it, and `ListAddressClaims` gives the operator the roster. |
| +Claiming requires `Open == true`; otherwise an operator writes the same |
| +record as a grant. The policy condition is |
| +`StringEqualsIgnoreCase ses:FromAddress`, since SES's `StringEquals` is |
| +case-sensitive and a From address is not. |
| + |
| +**Counting an address sender depends on the app.** Because there is no |
| +tenant-level default set (§3) and CARLOS may not touch `tito.io`'s |
| +default, events only reach CARLOS when the message names CARLOS's |
| +configuration set. So the sender's env bundle gains |
| +`<P>_CONFIG_SET`, documented as a header the app must set, and |
| +`carlos email status` says plainly when a sender has sent nothing — |
| +the honour-system limitation you accepted, and one the relay removes. |
| +Enable also reads the external identity's default configuration set: if |
| +it has one that is not associated with the sender's tenant, a |
| +tenant-tagged send is refused outright, so enable refuses first, with |
| +that explanation. |
| + |
| +**The cross-team contract.** While CARLOS's tenant association exists, |
| +Classic cannot delete the `tito.io` identity — `DeleteEmailIdentity` |
| +fails until the association goes. The registration step says so, and |
| +teardown removes the association. |
| + |
| +**The trade-off you accepted.** With `Open`, any account in the |
| +deployment can claim any address under `tito.io`, including |
| +`billing@tito.io`. The cheap mitigation if it bites is an optional |
| +`Local []string` allowlist on the external-identity record; it is not |
| +built now. |
| + |
| +## 8. Counting, caps and pauses per sender |
| + |
| +**Attribution by caller.** Every sender has its own IAM user, and SES |
| +stamps `mail.tags["ses:caller-identity"]` with the user name on Send, |
| +Bounce, Complaint, Delivery and Reject. `SenderOfCaller(caller)` |
| +resolves it to (account, app, sender) through the recorded `IAMUser`, |
| +covering the address-sender case where the domain names Classic, not |
| +CARLOS. `recordAndEnforceMail` counts there once, and falls back to |
| +today's domain-wide loop only when the caller is unknown, skipping any |
| +app in per-instance mode so its events are never charged to the app |
| +sender. |
| **Counts.** A new table, because SQLite cannot widen a primary key: |
| @@ -318,63 +381,104 @@ CREATE TABLE IF NOT EXISTS mail_sender_counts ( |
| ``` |
| Backfilled once from `mail_counts` with `sender = 'app'`; `mail_counts` |
| -is left in place, unwritten, and dropped a release later. The rename and |
| -purge sweeps (`mailAppTables`) gain the new table, which also closes the |
| -existing gap where `mail_counts` rows survive an app rename. |
| +stays, unwritten, and is dropped a release later. The rename and |
| +transfer sweeps key on a timestamp column (`mailAppTimeColumn`), which |
| +this table does not have, so it gets a special case comparing `day` |
| +rather than being added blindly to `mailAppTables` — which also closes |
| +today's gap, where `mail_counts` rows survive an app rename. |
| **Caps and pauses.** `TierCap` and the thresholds are unchanged and now |
| -apply per sender: `Grant.DailyCap` on the sender overrides the tier, an |
| -operator sets it per sender, and a pause calls `SetKeyActive(false)` on |
| -that sender's keys alone. `resumeStaleCapPause` keeps its rule — cap |
| -pauses lift on the UTC day roll, rate and manual pauses never do. One |
| -customer's bounce rate pauses that customer. |
| +apply per sender. A pause calls `SetKeyActive(false)` on that sender's |
| +keys alone, so one customer's bounce rate pauses that customer. |
| +`resumeStaleCapPause` keeps its rule: cap pauses lift on the UTC day |
| +roll, rate and manual pauses never do. |
| + |
| +**SES's own pauses.** A tenant carries the Standard reputation policy |
| +and SES can pause it independently, which `SetKeyActive` cannot lift. |
| +`EnsureTenant` stores `TenantArn`; status reads `GetTenant.SendingStatus` |
| +and reports an SES pause distinctly from a CARLOS pause; and if spike |
| +item 4 confirms it, `UpdateReputationEntityCustomerManagedStatus` |
| +becomes the pause verb for senders that have a tenant, with |
| +`SetKeyActive` as the fallback for those that do not. |
| **Status.** `carlos email status --app <app>` prints the app sender as |
| -today plus a line per instance sender (host, from, domains, cap, today's |
| -counts, pause state, delivery). `--instance <host>` prints one. The API |
| -reply gains a `senders` array; the existing top-level fields keep |
| -describing the app sender so an older CLI still works. |
| - |
| -## 8. Testing |
| - |
| -- `MemProvisioner` grows tenants-with-associations and refuses the four |
| - forbidden verbs for a registered external identity; a test pins that |
| - refusal. |
| -- Walk tests per sender kind: address sender enable/teardown never calls |
| - the forbidden verbs; instance sender enable writes only the instance |
| - layer; entering per-instance mode unsets the app sender's vars. |
| -- Policy tests: exact-address condition, tenant condition, and the |
| - existing refusals. |
| -- Attribution tests: an event whose caller names an instance sender |
| - counts only there; an event on a shared domain in per-instance mode |
| - never charges the app sender; an unknown caller keeps today's |
| - behaviour. |
| -- Limiter tests reuse `mailFixture`, parameterised by sender. |
| -- Config tests: three-layer merge precedence, an old agent ignoring the |
| - layer, the revisited `lostKeys` and mass-removal guards, and the ack |
| - carrying the delivered key id. |
| -- Door tests: the new routes in `svcroutes_test.go`, the `Instance` role |
| - pin exemption, and stranger-404 behaviour on an instance that is not |
| - the caller's. |
| - |
| -## 9. What this leaves for the relay |
| - |
| -The relay (#148's "we'd rather the platform own caps") becomes a third |
| -`Source` on the same sender: `SourceRelay`. The app still reads |
| -`<P>_HOST`, `<P>_PORT`, `<P>_USER`, `<P>_PASS` and `<P>_FROM` from its |
| -config, so switching a sender to the relay is a credential swap and a |
| -host change, invisible to the app's code. The relay enforces the cap |
| -before the send instead of after, which removes the "about 50" |
| -imprecision, and it is where a non-SES SMTP provider would attach. |
| -Nothing in this design blocks that: the cap, the pause and the counts |
| -already live on the sender, and the relay would authenticate a sender |
| -rather than an app. |
| - |
| -## 10. Out of scope |
| +today plus a line per instance sender; `--instance <host>` prints one. |
| +The API reply gains a `senders` array while its existing top-level |
| +fields keep describing the app sender, so an older CLI still works. |
| + |
| +## 9. Testing |
| + |
| +- `MemProvisioner` grows tenants with associations and reputation |
| + status, and refuses the forbidden verbs for a registered external |
| + identity; a test pins that refusal and one pins that `EnsureDomain` |
| + refuses before claiming. |
| +- Walk tests per sender kind: address enable and teardown never call the |
| + forbidden verbs; instance enable writes only the instance layer; |
| + entering per-instance mode deactivates the app key and every |
| + standalone, and unsets the vars in the same pass. |
| +- Race test: instance enable interleaved with app enable leaves no app |
| + credential in the bundle. |
| +- Name tests enumerating the IAM and SES bounds, including the longest |
| + legal account and app. |
| +- Policy tests: address condition (case-insensitive), tenant condition |
| + present and absent, and the existing refusals. |
| +- Attribution tests: caller names an instance sender; per-instance mode |
| + never charges the app sender; unknown caller keeps today's behaviour; |
| + an event type with no caller tag. |
| +- Token tests: a token bound to instance A gets a 404 on instance B, |
| + identical to a stranger's. |
| +- Config tests: three-layer precedence, an old agent ignoring the layer, |
| + the revisited `lostKeys` and mass-removal guards, and the ack carrying |
| + the overlay etag. |
| +- Door tests in `svcroutes_test.go`, the `Instance` role pin exemption, |
| + and the backing and delivery refusals at enable. |
| + |
| +## 10. What this leaves for the relay |
| + |
| +The relay becomes a third `Source` on the same sender: `SourceRelay`. |
| +The app still reads `<P>_HOST`, `<P>_PORT`, `<P>_USER`, `<P>_PASS` and |
| +`<P>_FROM`, so moving a sender onto it is a credential swap and a host |
| +change, invisible to the app's code. It enforces the cap before the send |
| +rather than after, removes the `<P>_CONFIG_SET` honour system for |
| +address senders, spends one IAM user for the fleet rather than one per |
| +sender, and is where a non-SES SMTP provider attaches. |
| + |
| +## 11. Order of work |
| + |
| +1. **Spike** (§4). Blocking; changes the policy and the pause verb. |
| +2. **Senders** (§5): the record, the migration, the naming, per-sender |
| + counts, caps, pauses and status. No new capability, and it lands |
| + `mail_sender_counts` and attribution by caller. |
| +3. **#147** (§7): external identities, address claims, the provisioner |
| + wrapper, `<P>_CONFIG_SET`. Unblocks Jelly without touching #148's |
| + prerequisites. |
| +4. **Per-instance tokens** (§6.1). |
| +5. **Per-instance config layer and acks** (§6.2, §6.5). |
| +6. **#148** (§6.3, §6.4): instance senders, per-instance mode, the |
| + doors, the backing check. |
| + |
| +## 12. Quotas and costs to watch |
| + |
| +- IAM users: 5,000 per AWS account, **not adjustable**. One per instance |
| + sender plus one per standalone credential. At titogo's scale this is |
| + the ceiling that eventually forces the relay. |
| +- IAM inline policy: 2,048 characters per user, all policies aggregated. |
| + An app sender with many domains can reach it; `SendPolicyJSON` should |
| + refuse with a clear error rather than let IAM reject it. |
| +- Configuration sets: 10,000 per region, not adjustable. Per account |
| + here, so not a constraint. |
| +- SES tenants: 10,000 per region, adjustable. Associations per tenant |
| + are undocumented — ask AWS before relying on many. |
| +- The SES control plane is throttled to one request per second for |
| + everything except sends, so an enable that creates a tenant, two |
| + associations and IAM resources needs backoff, and bulk onboarding is |
| + serial. |
| +- SES charges per tenant per month on top of per-message pricing. |
| + |
| +## 13. Out of scope |
| - The relay itself. |
| -- Per-instance sending domains for the platform's own suffix |
| - (`<app>.<sqid>.<suffix>`); an instance sender needs a custom domain or |
| - a registered address. |
| -- Inbound email (#138), event push per sender, and per-sender billing. |
| -- `Local` allowlists on an external identity (§5). |
| +- Per-instance sending domains under the platform suffix; an instance |
| + sender needs a custom domain or a registered address. |
| +- Inbound email (#138), event push per sender, per-sender billing. |
| +- `Local` allowlists on an external identity (§7). |