amadan

carlos / platform Public

Clone 
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

Tenant email: senders

Design for amadan carlos/platform discussions #147 (send as an address under an SES identity CARLOS does not own) and #148 (tenant email per instance: domain, credential, cap and pause for each customer of a multi-tenant app).

Base: origin/main at 520fcc04. Author: Paul Campbell with Claude. Date: 2026-09-15. Revised after two reviews: one against the code, one against the AWS documentation.

1. The problem

Tenant email today is keyed by (account, app) everywhere: email.json, the IAM user carlos-mail-<acct>-<app>, the DailyCap, the pause, the mail_counts rows, and the delivery of CARLOS_SMTP_* into the app's default config bundle. Two requests do not fit that shape.

#147. Jelly runs as an app in two accounts (jam and stripe) and both must send as jelly@tito.io. tito.io is a verified SES identity that Tito Classic has owned for years in the same AWS account. The existing walk would adopt it and then rewrite it: EnsureIdentity moves MAIL FROM to mail-eu-west-1.tito.io (aws.go:292), SetIdentityConfigSet binds carlos-<acct> as the identity's default configuration set (walk.go:301), the domain claim is first-come (domainclaim.go:119), and Teardown calls DeleteIdentity (walk.go:1205).

#148. titogo runs one instance per customer under a single app. Each customer needs their own domain, credential, cap and pause. Today EnsureUser writes one policy listing every domain granted to the app (walk.go:634), delivered to every instance through the app's default bundle (walk.go:818), so code in one customer's instance can send as every other customer's domain.

2. Decisions taken before this document

  1. Extend the IAM-per-sender model now, and iterate towards a platform SMTP relay. The relay is wanted anyway, as a layer over SES that could front a different SMTP provider. §10 keeps that door open rather than building it.
  2. Per-instance mode. Once an app has instance senders, instances no longer receive the app's credential at all.
  3. titogo's backend drives domain onboarding through the console API, with the CLI as a wrapper over the same doors.
  4. For #147, an operator registers the external identity once and may mark it open; accounts then claim addresses under it themselves.
  5. A central hub does the onboarding, holding one credential (§6.1), so no per-instance machine token is needed for this work. The hub's credential carries a new narrow role, svcrole.Mail, rather than Admin.
  6. Instance backing is mixed today, and moving to the isolating one is the intent. So enable checks the route's backing rather than assuming it (§6.4).

3. What the AWS documentation actually says

The first draft of this design got three of these wrong, so they are recorded here with their sources.

  • Condition keys. The v1 action set, which is what SMTP authorises against, has ses:ApiVersion, ses:FeedbackAddress, ses:FromAddress, ses:FromDisplayName and ses:Recipients. SES v2 adds ses:TenantName and ses:MultiRegionEndpointId. No condition key names a configuration set. The configuration set is a resource on the send action, which constrains which set may be named but cannot force one to be named — so a credential cannot be compelled to publish events.
  • ses:TenantName is documented for SendEmail and SendBulkEmail only. The SendRawEmail row, which is what the SMTP interface authorises against, does not list it, although an AWS blog publishes a policy that uses it there. This is the first spike (§4).
  • There is no tenant-level default configuration set. A send on behalf of a tenant must either name a configuration set associated with that tenant, or use an identity whose default set is associated with it. So per-sender configuration sets on a shared identity are impossible, and an address sender under a borrowed identity can only publish events if the message names CARLOS's set.
  • Associating a resource with a tenant does not modify it, and one identity may belong to many tenants. It has one side effect on the owner: an identity cannot be deleted while an association exists.
  • SES pauses tenants by itself. Every new tenant gets the Standard reputation policy, which auto-pauses on high-severity findings, and Trust & Safety can pause one too. SetKeyActive(true) cannot lift that. Sending quotas remain account-wide per region; the account suppression list still applies unless a tenant opts into its own.
  • ses:caller-identity names the IAM user on Send, Bounce, Complaint, Delivery and Reject, but only for events published through a configuration set, and not on Rendering Failure or DeliveryDelay.

4. Spike first

Half a day against the Tito deployment's AWS account, on a throwaway identity and tenant. It is blocking, because two of its answers change the policy this design writes.

  1. Send over SMTP with X-SES-TENANT set, under a policy conditioned on ses:TenantName. Then send again without the header. Three outcomes: both succeed (the key is not populated for SMTP, so the condition buys nothing); tagged succeeds and untagged is refused (the design works as written); both are refused (SES does not populate the key at all and the condition denies everything). In outcome one and three the tenant condition comes out of the policy, and isolation rests on ses:FromAddress plus SES's own tenant-association validation.
  2. Send under a tenant without X-SES-CONFIGURATION-SET on an identity whose default set is not associated with that tenant, to confirm it is refused, and with the header to confirm events arrive.
  3. Associate an identity created outside CARLOS and confirm its MAIL FROM, DKIM and default configuration set are unchanged, and that sends on it without a tenant still work — Classic's mail must not change behaviour the moment CARLOS associates tito.io.
  4. Pause a tenant with UpdateReputationEntityCustomerManagedStatus and confirm GetTenant.SendingStatus reflects it, so §8 can adopt it as the pause verb.

4.1 Results, run 2026-09-15

Run in the Tito deployment's account (665106592142, eu-west-1) against a throwaway identity spike.teamtito.com shaped like tito.io: DKIM verified, no custom MAIL FROM, and a default configuration set (spike-other) standing in for Classic's. Every resource was deleted afterwards.

1. ses:TenantName is enforced over SMTP. Outcome two, the good one. A policy of ses:SendRawEmail on the identity and spike-owned, conditioned StringEquals ses:TenantName = spike-tenant:

| send | result | |---|---| | X-SES-TENANT + X-SES-CONFIGURATION-SET | accepted | | no headers | 554, not authorized on the identity | | configuration set header only | 554, not authorized on the identity |

So the tenant condition stays in the policy, and a credential cannot send without naming its tenant. That is real enforcement, not honour system, and it is what makes one customer's credential useless for another customer's mail.

2. There is no tenant-level default configuration set, as documented. A send with X-SES-TENANT but no configuration-set header was refused with 554 ... not authorized to perform 'ses:SendRawEmail' on resource '...configuration-set/spike-other'. SES applied the identity's default set, which belongs to the identity's owner. So an address sender must set both headers, and the bundle carries <P>_CONFIG_SET and <P>_TENANT.

3. Association is inert for the identity's owner, except for deletion. GetEmailIdentity before and after CreateTenantResourceAssociation was byte-identical: DKIM tokens, MailFromAttributes and ConfigurationSetName all unchanged. A second IAM user with no tenant condition — Classic's shape — still sent on the identity with no tenant header, and its events still published to the identity's own default set, not CARLOS's. DeleteEmailIdentity was refused: "Cannot delete ... because it has tenant associations." That is the cross-team contract §7 records.

4. A tenant can be paused directly, and CARLOS can tell whose pause it is. UpdateReputationEntityCustomerManagedStatus ... DISABLED made the next send fail with 554 Message rejected: Sending disabled for <tenant arn>. GetTenant.SendingStatus read DISABLED, and GetReputationEntity returned SendingStatusAggregate: DISABLED with CustomerManagedStatus: {Status, Cause: "Status manually updated.", LastUpdatedTimestamp} and a separate AwsSesManagedStatus (null here). Re-enabling restored sending. The tenant's policy at creation was arn:aws:ses:eu-west-1:aws:reputation-policy/standard, which is the auto-pausing one.

So §8 adopts the tenant pause as the pause verb for any sender that has a tenant, keeps SetKeyActive for those that do not, and reads the two status fields apart so an SES-imposed pause is reported as SES's rather than silently "resumed" by CARLOS on the next day roll.

Also confirmed: published events carry ses:caller-identity: ["spike-tenant-user"], the bare IAM user name, which is what §8's attribution reads.

4.2 Two prerequisites before any tenant-attributed send

Found while building step 2, and both bite at roll time rather than build time.

  1. Existing domains have no tenant association, and nothing will ever create one. Step 2 associates an identity and the account's configuration set inside EnsureDomain, which is reached only from carlos email domains add. VerifyDomain does not associate. So every domain already in the estate keeps a verified identity with no association, and the first send that carries X-SES-TENANT fails SES's validation. Before step 3 or step 5 rolls, either a backfill pass associates every recorded identity and configuration set, or VerifyDomain learns to associate too. This is the failure the standing warning in aws.go exists to prevent, so it is not optional.
  2. The control-plane role needs two new IAM actions, ses:CreateTenantResourceAssociation and ses:DeleteTenantResourceAssociation. That policy does not live in this repo. If the grant is missing, AccessDenied is neither AlreadyExists nor NotFound, so it is not swallowed: domains add fails after the record and the DNS are already written, and Teardown returns before DeleteIdentity and before releasing the global domain claim, which leaves that claim stranded until the permission lands. Confirm the grant before rolling.

5. The sender

A sender owns a From address, a credential, a cap, a pause and a set of counts.

// internal/mail/sender.go
type Sender struct {
    Schema   int    `json:"schema"`   // 1
    ID       string `json:"id"`       // "app", or provision.InstanceID(host)
    Slug     string `json:"slug"`     // AWS-safe name component, see below
    Scope    string `json:"scope"`    // ScopeApp | ScopeInstance
    Host     string `json:"host,omitempty"`
    Source   string `json:"source"`   // SourceDomain | SourceAddress
    Declared    *Declared     `json:"declared,omitempty"`
    Grant       *GrantBlock   `json:"grant,omitempty"`
    Provisioned *Provisioned  `json:"provisioned,omitempty"`
    Standalone  []*Standalone `json:"standalone,omitempty"`
    Identity  string `json:"identity,omitempty"`   // external identity ARN
    Address   string `json:"address,omitempty"`    // exact address, lowercased
    TenantARN string `json:"tenant_arn,omitempty"` // from CreateTenant
    PerInstance bool `json:"per_instance,omitempty"` // app sender only, §6.3
}

Declared, GrantBlock, Provisioned and Standalone keep their current shapes and move inside the sender unchanged.

Records. apps/<acct>/<app>/email/senders/<id>.json, one object per sender, each with its own CAS etag so two instances never contend.

Names. The first draft's names were invalid: SES caps tenant and configuration-set names at 64 characters of [A-Za-z0-9_-], IAM caps a user name at 64, and an instance id is the host itself when it is 40 characters or less, dots included — carlos-mail-<acct>-<app>+i+<id> reaches 104 characters. So each sender gets a Slug, computed once at creation and stored on the record:

  • app sender: Slug = "", names unchanged (carlos-mail-<acct>-<app>, tenant and set carlos-<acct>).
  • instance sender: Slug = <first 8 of base32(sha256(acct/app/id))>, giving IAM user carlos-mail-<acct>-<app>+i+<slug> and tenant carlos-<acct>-<app>-<slug>, both bounded at 12 + 16 + 1 + 32 + 3 + 8 = 72 — still over 64 for the longest account and app, so the app component is itself truncated to 24 characters with the hash carrying uniqueness. The exact rule lives in one function, SenderNames, with a test that enumerates the bounds.

SenderNames also covers standalone credentials, whose current name (carlos-mail-<acct>-<app>+s+<name>, policy.go:92) already reaches 95 characters against IAM's 64 with nothing checking it. The bounds test enumerates every form.

Because the slug is a hash, nothing parses it back, and a 40-bit slug needs collision detection. Both are served by a reverse index at email/callers/<iamuser>.json → (account, app, sender), with the same rules the domain claim already uses (domainclaim.go:124):

  • Written right after EnsureUser (walk.go:669), which runs on every enable, so a re-enable converges it. Writing it before MintKey would strand a sender forever when a crash landed between the two.
  • The Put is create-only; on conflict it re-reads. The same (account, app, sender) is success — otherwise it is a slug collision, and the slug is recomputed with a salt that is stored on the record.
  • Deleted by RevokeStandalone, Teardown and the account reap. The key sits outside apps/<acct>/<app>/, so release.PurgeApp's prefix sweep never reaches it; without an explicit delete, a stale entry would attribute SES events to a purged app and insert counts for it.
  • Only the slugged names need it. SenderOfCaller (§8) falls back to parsing for carlos-mail-<acct>-<app> and its +s+ standalones.

Configuration sets stay per account (carlos-<acct>), because an identity has only one default set and §3 rules out per-sender sets on a shared identity. Each sender's tenant is associated with that set, and the set's ARN stays in the sender's policy Resource.

SenderNames covers standalone credentials too, but it never changes a name that exists: RefreshPolicies recomputes StandaloneUserName rather than reading sa.IAMUser (walk.go:364), so a changed rule would write a policy for a new principal and leave the live one stale. The rule is: existing shapes are unchanged, and a name that would exceed the bound is refused at create.

Migration. mail.Get(acct, app) prefers senders/app.json and falls back to email.json. When it served the fallback it returns an empty etag, so every writer's Put — and they all pass back the etag they were given (walk.go:522, :736, :909, :993, :1070) — is a create by construction, and a lost race surfaces as ErrCASConflict exactly as it does today. mail.Put always writes senders/app.json. email.json is left in place, unwritten, for one release and then deleted by a sweep. A console rolled back after the first sender write loses writes made in between — a pause or a rotation would have to be redone — which is recorded here and in the roll notes.

6. #148: instance senders

6.1 The caller is a hub, not the customer instances

titogo's own backend onboards every customer, so the instance email doors are Allow(svcrole.Admin) — the same role the app-level email doors carry today — and the hub holds one named service credential with that role, bound to the titogo app.

That credential must not be delivered into customer instances. Today handleAPIInstanceToken mints a token bound to (account, app) and its documented delivery path is carlos secrets set -app <app> (apiinstances.go:1367), the bundle every instance renders, so anything delivered that way reaches every customer. The hub's credential lives wherever the hub runs — a different app, or an instance with its own overlay once §6.2 exists. The console cannot check this: bundle values are sealed and it holds no key (apimail.go:483), so where a credential was delivered is not a decidable question. It is an operator attestation, stated in the enable door's documentation, not a gate.

What an Admin-role service credential reaches. svcrole.Allow adds Admin to every non-empty set (svcrole.go:120), so a titogo-bound Admin credential reaches every app-scoped door that names any role: ship, promote and rollback, pipeline, ledgers, instances create/patch/delete/ move, domains, logs and analytics, features, place, the store's create/provision/grant/rotate, relay, and every email door except credentials and pause. It cannot reach the person-only doors (svcrole.Set{}): instance-token minting, the instances manifest, channel, routes, service minting, fleets, store apply and approve, and system. A compromised hub is therefore a compromised titogo deploy, not only a compromised mailer.

That inverts why those doors are Admin in the first place (apimail.go:460): Admin was chosen to be narrower than Publish or Instance, so that a deploy credential could not acquire a sending identity. For a hub whose only job is onboarding, it means the opposite — it must be able to deploy in order to onboard.

So this design adds one entry to the closed role catalogue, svcrole.Mail, and every email door — the app-level ones that exist and the instance-level ones in §6.4 — becomes Allow(svcrole.Mail). Because Allow always adds Admin (svcrole.go:120), today's Admin credentials keep reaching them, so nothing breaks; what changes is that titogo's hub can hold a credential that onboards customer domains and does nothing else. svcroutes_test.go pins the new role's door list.

Rate budget. mailWriteGates uses the person-sized write limiter keyed on the audit string (apimail.go:136), not allowWrite for service principals (svcprincipal.go:394). Bulk onboarding through the hub will 429 against a person's budget, so the mail write gate takes the service budget when the principal is a service.

Two consequences worth stating plainly. If customer code ever needs to call these doors for itself, tokens bound to (account, app, instance) become a prerequisite again, and this design is shaped so that is a later change to the door's role set and nothing else. And separately, the app-wide CARLOS_INSTANCE_TOKEN that customer instances already hold reaches the existing instance create, patch and delete doors (authcli.go:612, :628), so one customer's code can already delete a sibling instance. That is not introduced here and not fixed here; it is worth its own discussion.

6.2 Per-instance config delivery

A third bundle layer, resolved after the named environment:

apps/<acct>/<app>/config.json                        default
apps/<acct>/<app>/environments/<name>.json           environment
apps/<acct>/<app>/instances-config/<host>.json       instance   ← new
  • Same Manifest shape, same sealed envelopes, same name and value rules. ExecDelivery is not read from this layer, matching the ruling at appconfig.go:61-69.
  • LoadMerged keeps its signature; LoadMergedHost(ctx, s, acct, app, env, host) adds the third overlay, and Sync/SyncExec call it per route after resolving the group base, so a pass costs one extra bucket read per instance that has an overlay and none for instances that do not.
  • Writing the layer bumps the app's generation object, as SetExecDeliveryAt does, so the fast loop converges.
  • Guards that assume a two-layer merge and must be revisited: lostKeys/defaultMissing (sync.go:260, :388), the mass-removal guard (syncexec.go:351), AwaitingRestart (syncexec.go:712) and allShadowed (syncexec.go:813). ConfigRouteFingerprints (routefingerprint.go:58) digests registry columns only and cannot see a bucket-side change; the generation bump is that layer's signal.
  • Only the platform writes this layer: no CLI verb, no console door. It exists so a credential can be scoped to one instance.

6.3 Entering per-instance mode

A prefix listing cannot decide this without racing an app-sender enable, so the app sender carries PerInstance bool, and the transition is one CAS write on senders/app.json that sets PerInstance together with Active = false and PauseKind = PauseKindPerInstance, then:

  1. ReapKeys(user, nil) on the app sender's IAM user and on every standalone user — list-and-delete, not SetKeyActive(false) on the recorded ids. Deactivating recorded ids is not enough, because Enable mints and delivers a key before its own CAS Put (walk.go:717-738): an enable racing the transition leaves an active, unrecorded key in every instance's bundle that nothing would ever reap. Deleting whatever keys exist makes the racing delivery inert. Instance Enable re-runs the same reap.
  2. Unsets the app sender's CARLOS_SMTP_* vars from the default bundle.

PauseKindPerInstance matters because resumeStaleCapPause (mailcounts.go:160) runs on the status read and reactivates every key including standalones (walk.go:973); a cap-paused app sender would otherwise re-arm the credential every instance still holds on the next UTC day roll. SetPaused(false) and resumeStaleCapPause both refuse while PerInstance is set.

App-sender Enable refuses when PerInstance is set, under that same etag. Instance Enable requires it. Leaving per-instance mode, once the last instance sender is gone, is one CAS that clears PerInstance, sets Active = true and clears PauseKind, followed by a normal enable — otherwise Enable carries the false Active forward (walk.go:698) and mints a dead key.

grantedDomains filters on the domain record's new Senders list, not Apps, and an instance sender's claim never adds the app to Apps. So the app sender's policy can never name a customer domain.

Everything that walks rec.Apps must walk Senders instead, or an instance sender's policy is never refreshed when its configuration set is rebound — which is 554 on every send, the failure #271 and #279 already cost us. That means RefreshPolicies (walk.go:324, called from EnsureDomain:297 and VerifyDomain:461) and mailAppDomains (apimail.go:299), which the enable door and status both read.

There is also a door for the transition itself: the first instance enable performs it, and DELETE .../instances/{host}/email on the last instance sender leaves per-instance mode. Nothing else sets the flag.

6.4 The doors, and what they check

POST   /api/cli/apps/{account}/{app}/instances/{host}/email             declare
POST   .../instances/{host}/email/domains                               attach
POST   .../instances/{host}/email/domains/{domain}/verify
POST   .../instances/{host}/email/enable
POST   .../instances/{host}/email/rotate
GET    .../instances/{host}/email                                       status
DELETE .../instances/{host}/email                                       teardown

All are Allow(svcrole.Mail), like the app-level email doors after §6.1, so the Instance role pin is untouched. Pause, resume and cap changes stay operator-only through the existing POST …/email/pause, which gains an instance field.

Two checks at enable, both refusals with an explaining message:

  • A uid of its own. The predicate is not "unit-backed" — a hostrole app runs its units as User=carlos-<app> with DynamicUser=no (caps/render.go:101-112), one uid per app, and ProtectProc hides other users' processes, not a sibling's at the same uid. The check is that this instance's process gets a uid no sibling shares: DynamicUser=yes with no hostrole grant, or an exec route under an isolated-exec layout with a per-instance uid. Plain exec routes share the agent's uid and <dataDir>/<host>.config.env is readable by every sibling (internal/activator/admintoken.go:6-12), so a per-instance credential buys nothing there. This is the dependency on #126.
  • Delivery. The instance's box must be running an agent that renders the instance layer (§6.5).

Domain attach. claimDomain gains a Sender field and refuses a re-claim whose (app, sender) differs from the holder's, including within the same account — today a same-account re-claim is a silent no-op (domainclaim.go:137), which would let one instance attach another's verified domain. EnsureDomain refuses a domain that has an external-identity record before it claims anything, and the orphan-claim sweep in Teardown matches on sender and skips external domains, so a refused EnsureIdentity can never wedge an app's teardown.

6.5 Confirming delivery

ConfigAck gains Schema int and InstanceConfigETag string: the etag of the instance overlay the agent rendered. The console compares it with the etag its own Put returned, which is decidable — unlike a key-id map, which cannot distinguish an old agent from a new agent rendering a route with no mail vars.

An ack is written only when a pass changes a file (configack.go:66), so a fresh instance, or one whose config has not moved since its agent was rolled, has no ack at all — absence cannot mean "old agent".

The capability question is answered by the box record's agent version, not by the ack. An earlier draft had the agent rewrite the ack on every pass so that absence became decidable. That is the wrong trade: it is a Get and a Put per converged route per pass on every box in the fleet, where today it is one write per changed file, and on a fleet-door box it is worse than useless — control/config-ack/ is outside the door's read and write scopes (edge/storedoor.go:389, :489), so every one of those passes would fail, log, and still leave the console with nothing. It also forces the ack's At to stop describing the delivery it sits beside.

So acks stay change-only. The enable-time refusal reads the agent version and names the box and the version. Delivery is still decidable where it matters, because writing an instance's credential always changes that instance's file, so it always produces an ack carrying the InstanceConfigETag to compare. A missing ack reads as "not yet delivered", never as "old agent". carlos email status --instance <host> reports delivered, stale or not yet delivered.

A fleet-door box cannot confirm delivery at all, because it cannot write the ack. Until that door grants control/config-ack/<box>/, an instance sender on such a box can be enabled but its delivery will read as unconfirmed. The roll note says so, and the door grant is its own piece of work.

7. #147: a sender under an identity CARLOS does not own

The external identity is registered once per deployment by an operator at email/external-identities/<domain>.json: domain, region, identity ARN, Open bool, who registered it and when. The region must match the deployment's, because tenants and identities are per region.

CARLOS never calls CreateEmailIdentity, PutEmailIdentityMailFromAttributes (both inside EnsureIdentity), SetIdentityConfigSet or DeleteIdentity for a registered identity. A provisioner wrapper refuses those verbs for any domain with such a record, and a test pins the refusal. EnsureDomain refuses the domain before claiming it (§6.4), so the refusal can never strand a claim.

Enable takes an address branch. Today Enable resolves granted domains and refuses with ErrNoDomains when there are none (walk.go:627), then insists the From address sits at a granted domain name (:647), and SendPolicyJSON requires each identity name to be a granted domain or a parent of one (policy.go:167). An external identity has no DomainRecord in the account by design, so an address sender needs its own branch in both: the identity ARN comes from the external-identity record, the From address from Sender.Address, and the policy renderer gets an address form whose identity check is that the address's domain is the identity or a child of it.

Verification. Status.Verified() requires MailFrom == "SUCCESS" (provisioner.go:45), which an identity with no custom MAIL FROM never reaches. External identities are checked with a relaxed predicate: DKIM and sending enabled, MAIL FROM ignored.

Address claims. email/address-claims/<address>/<acct>.json, where the address is lowercased and validated to the same grammar Declared already enforces. The claim is not exclusive: jam and stripe may both hold jelly@tito.io. It names the account, app, sender and who claimed it, and ListAddressClaims gives the operator the roster. Claiming requires Open == true; otherwise an operator writes the same record as a grant. The policy condition is StringEqualsIgnoreCase ses:FromAddress, since SES's StringEquals is case-sensitive and a From address is not.

An address sender's address is fixed once claimed. Changing it means tearing the sender down and claiming again, and two doors refuse the shortcut: the claim door when the sender already holds a different address, and Declare when a From address would disagree with the claimed one. Without both, carlos email enable --from other@tito.io on a live sender renders a policy for the new address, strands the old claim — which blocks every sibling app in the account and holds the identity association open — and leaves _FROM and the policy naming different addresses while status still reads healthy.

Counting an address sender depends on the app. Because there is no tenant-level default set (§3) and CARLOS may not touch tito.io's default, events only reach CARLOS when the message names CARLOS's configuration set. So the sender's env bundle gains <P>_CONFIG_SET, documented as a header the app must set, and carlos email status says plainly when a sender has sent nothing — the honour-system limitation you accepted, and one the relay removes. The bundle also carries <P>_TENANT, because the spike (§4.1) kept the ses:TenantName condition in the policy and a send that omits it is refused outright. That refusal is the saving grace of the honour system: a message missing either header does not send silently uncounted, it does not send at all. Enable also reads the external identity's default configuration set: if it has one that is not associated with the sender's tenant, a tenant-tagged send is refused outright, so enable refuses first, with that explanation.

Doors. Registration is operator-only, the claim is a member's, and both are new routes:

PUT    /api/cli/email/external-identities/{domain}   register (operator)
DELETE /api/cli/email/external-identities/{domain}   deregister (operator)
GET    /api/cli/email/external-identities            list (operator)
GET    /api/cli/email/external-identities/{domain}/claims   roster (operator)
POST   /api/cli/apps/{account}/{app}/email/address   claim + declare (Admin)

The claim door takes {address} and is what sets Sender.Source = SourceAddress, Sender.Address and Sender.Identity; it is Declare for an address sender, so declaring a From address and claiming it are one call and cannot disagree. Enable, rotate, status and teardown are the existing app-sender doors, which read Source off the record. CLI: carlos email external-identity register|list|claims, and carlos email enable --from jelly@tito.io calls the claim door first when the domain resolves to an external identity.

Refusal scope. EnsureDomain refuses a domain that is a registered external identity or any subdomain of one: a claim on mail.tito.io would otherwise let CARLOS create and later delete an identity underneath one it does not own.

Provisioner verbs. The interface (provisioner.go:49) gains four, each idempotent in the usual way, with MemProvisioner equivalents:

AssociateTenantResource(ctx, tenant, resourceARN string) error
DisassociateTenantResource(ctx, tenant, resourceARN string) error
TenantStatus(ctx, tenant string) (TenantStatus, error)       // GetTenant
SetTenantSending(ctx, tenantARN string, enabled bool) error  // §8's pause

plus IdentityStatus returning the identity's bound configuration set, which it already carries (Status.ConfigSet), so enable can refuse an external identity whose default set is not associated with the tenant.

Associations are wired in step 2, not here. Today EnsureTenant creates the container and nothing associates anything, and aws.go:171 warns that no send may carry X-SES-TENANT until associations exist. Step 2 associates each account's existing tenant with its identities and its configuration set, which changes no behaviour on its own, so that step 3 can rely on them.

The cross-team contract. While CARLOS's tenant association exists, Classic cannot delete the tito.io identity — DeleteEmailIdentity fails until the association goes. The registration step says so, and teardown removes the association.

The trade-off you accepted. With Open, any account in the deployment can claim any address under tito.io, including billing@tito.io. The cheap mitigation if it bites is an optional Local []string allowlist on the external-identity record; it is not built now.

8. Counting, caps and pauses per sender

Attribution by caller. Every sender has its own IAM user, and SES stamps mail.tags["ses:caller-identity"] with the user name on Send, Bounce, Complaint, Delivery and Reject. SenderOfCaller(caller) resolves it to (account, app, sender) in two steps: parse first, exactly as AppOfCaller does today (policy.go:42), which answers for carlos-mail-<acct>-<app> and its +s+ standalones with sender "app" and costs nothing; then, only for a slugged +i+ name, one Get on email/callers/<iamuser>.json. That ordering matters on the roll: every app enabled today short-circuits Enable before MintKey (walk.go:677), so it has no index entry until it next rotates, and a Get-only resolver would silently stop writing the by_sender log rows that sesevents.go:498 writes now.

recordAndEnforceMail counts against the resolved sender once, and falls back to today's domain-wide loop only when the caller is unknown, skipping any app in per-instance mode so its events are never charged to the app sender.

This is a visible change, and it belongs in the roll note. Today every app on a shared domain is charged for every event on that domain (sesevents.go:509); afterwards only the sending app is. A sibling's cap stops being consumed by another app's sends, which is the intent, but a tenant watching its own numbers will see them move.

Counts. A new table, because SQLite cannot widen a primary key:

CREATE TABLE IF NOT EXISTS mail_sender_counts (
  account TEXT NOT NULL, app TEXT NOT NULL, sender TEXT NOT NULL,
  day TEXT NOT NULL, sends INTEGER NOT NULL DEFAULT 0,
  bounces INTEGER NOT NULL DEFAULT 0, complaints INTEGER NOT NULL DEFAULT 0,
  rejects INTEGER NOT NULL DEFAULT 0, deliveries INTEGER NOT NULL DEFAULT 0,
  PRIMARY KEY (account, app, sender, day));

Backfilled once from mail_counts with sender = 'app'; mail_counts stays, unwritten, and is dropped a release later by a sweep that runs once at console start. The rename, transfer, purge and account-reap sweeps (carryMailApp, forgetMailApp, forgetMailAccount) all take the new table, but it cannot join mailAppTables blindly because those sweeps key on a timestamp column (mailAppTimeColumn, mailevents.go:170) and this table has a day string instead — so it gets an explicit case comparing day against the date. That also closes today's gap, where mail_counts rows survive a rename.

Caps and pauses. TierCap and the thresholds are unchanged and now apply per sender. A pause calls SetKeyActive(false) on that sender's keys alone, so one customer's bounce rate pauses that customer. resumeStaleCapPause keeps its rule: cap pauses lift on the UTC day roll, rate and manual pauses never do.

The pause verb is the tenant's. §4.1 confirmed it, so a sender with a tenant is paused with UpdateReputationEntityCustomerManagedStatus(RESOURCE, <tenant ARN>, DISABLED) and SetKeyActive stays the fallback for senders that have none. That is a better pause than deactivating a key: it takes effect on the next send rather than waiting for a credential refresh, and it leaves the credential intact so resuming is one call.

EnsureTenant stores TenantArn. Status reads GetReputationEntity and keeps CustomerManagedStatus and AwsSesManagedStatus apart, so an SES-imposed pause is reported as SES's and is never lifted by resumeStaleCapPause on the day roll — a tenant SES disabled for reputation must not be resumed by a cap timer.

Status. carlos email status --app <app> prints the app sender as today plus a line per instance sender; --instance <host> prints one. The API reply gains a senders array while its existing top-level fields keep describing the app sender, so an older CLI still works.

9. Testing

  • MemProvisioner grows tenants with associations and reputation status, and refuses the forbidden verbs for a registered external identity; a test pins that refusal and one pins that EnsureDomain refuses before claiming.
  • Walk tests per sender kind: address enable and teardown never call the forbidden verbs; instance enable writes only the instance layer; entering per-instance mode deactivates the app key and every standalone, and unsets the vars in the same pass.
  • Race test: an app enable that mints and delivers before its CAS Put, interleaved with the per-instance transition, leaves no usable app credential — the reap deleted the key even though the vars landed.
  • A cap-paused app sender that enters per-instance mode is not reactivated by the next UTC day roll.
  • Name tests enumerating the IAM and SES bounds, including the longest legal account and app.
  • Policy tests: address condition (case-insensitive), tenant condition present and absent, and the existing refusals.
  • Attribution tests: caller names an instance sender; per-instance mode never charges the app sender; unknown caller keeps today's behaviour; an event type with no caller tag.
  • Door tests: an app-scoped Instance-role token is refused at every instance email door, and a stranger's 404 is byte-identical to the refusal for an instance that exists in another account.
  • Config tests: three-layer precedence, an old agent ignoring the layer, the revisited lostKeys and mass-removal guards, and the ack carrying the overlay etag.
  • Door tests in svcroutes_test.go, the Instance role pin exemption, and the backing and delivery refusals at enable.

10. What this leaves for the relay

The relay becomes a third Source on the same sender: SourceRelay. The app still reads <P>_HOST, <P>_PORT, <P>_USER, <P>_PASS and <P>_FROM, so moving a sender onto it is a credential swap and a host change, invisible to the app's code. It enforces the cap before the send rather than after, removes the <P>_CONFIG_SET honour system for address senders, spends one IAM user for the fleet rather than one per sender, and is where a non-SES SMTP provider attaches.

11. Order of work

  1. Spike (§4). Blocking; changes the policy and the pause verb.
  2. Senders (§5): the record, the migration, the naming, per-sender counts, caps, pauses and status. No new capability, and it lands mail_sender_counts and attribution by caller.
  3. #147 (§7): external identities, address claims, the provisioner wrapper, the address branch through enable and the policy renderer, <P>_CONFIG_SET. Unblocks Jelly and depends only on step 2.
  4. Per-instance config layer and acks (§6.2, §6.5).
  5. #148 (§6.3, §6.4): instance senders, per-instance mode, the doors, the uid check.
  6. Per-instance machine tokens, after #148 rather than before it. Not needed while a hub onboards, but the app-wide CARLOS_INSTANCE_TOKEN every customer instance holds already reaches the instance create, patch and delete doors (authcli.go:612, :628), so one customer's code can delete another's instance today. That is filed separately; this is where the fix lands.

12. Quotas and costs to watch

  • IAM users: 5,000 per AWS account, not adjustable. One per instance sender plus one per standalone credential. At titogo's scale this is the ceiling that eventually forces the relay.
  • IAM inline policy: 2,048 characters per user, all policies aggregated. This bites sooner than it looks: every granted domain contributes the same account configuration-set ARN and nothing dedupes it (walk.go:634, :343, policy.go:205), so a twelve-domain app is already over. slices.Compact after the sort in the three callers takes that to about twenty, and SendPolicyJSON refuses with the count rather than letting IAM reject the policy.
  • Configuration sets: 10,000 per region, not adjustable. Per account here, so not a constraint.
  • SES tenants: 10,000 per region, adjustable. Associations per tenant are undocumented — ask AWS before relying on many.
  • The SES control plane is throttled to one request per second for everything except sends, so an enable that creates a tenant, two associations and IAM resources needs backoff, and bulk onboarding is serial.
  • SES charges per tenant per month on top of per-message pricing.

13. Out of scope

  • The relay itself.
  • Per-instance sending domains under the platform suffix; an instance sender needs a custom domain or a registered address.
  • Inbound email (#138), event push per sender, per-sender billing.
  • Local allowlists on an external identity (§7).