carlos / platform Public

Clone
git clone https://amadan.net/carlos/platform

Plain git — no account needed to clone.

Download

Download this file

1# Tenant email: senders
2
3Design for amadan `carlos/platform` discussions **#147** (send as an
4address under an SES identity CARLOS does not own) and **#148**
5(tenant email per instance: domain, credential, cap and pause for each
6customer of a multi-tenant app).
7
8Base: `origin/main` at 520fcc04. Author: Paul Campbell with Claude.
9Date: 2026-09-15. Revised after two reviews: one against the code, one
10against the AWS documentation.
11
12## 1. The problem
13
14Tenant email today is keyed by (account, app) everywhere: `email.json`,
15the IAM user `carlos-mail-<acct>-<app>`, the `DailyCap`, the pause, the
16`mail_counts` rows, and the delivery of `CARLOS_SMTP_*` into the app's
17default config bundle. Two requests do not fit that shape.
18
19**#147.** Jelly runs as an app in two accounts (`jam` and `stripe`) and
20both must send as `jelly@tito.io`. `tito.io` is a verified SES identity
21that Tito Classic has owned for years in the same AWS account. The
22existing walk would adopt it and then rewrite it: `EnsureIdentity`
23moves MAIL FROM to `mail-eu-west-1.tito.io` (`aws.go:292`),
24`SetIdentityConfigSet` binds `carlos-<acct>` as the identity's default
25configuration set (`walk.go:301`), the domain claim is first-come
26(`domainclaim.go:119`), and `Teardown` calls `DeleteIdentity`
27(`walk.go:1205`).
28
29**#148.** titogo runs one instance per customer under a single app.
30Each customer needs their own domain, credential, cap and pause. Today
31`EnsureUser` writes one policy listing every domain granted to the app
32(`walk.go:634`), delivered to every instance through the app's default
33bundle (`walk.go:818`), so code in one customer's instance can send as
34every other customer's domain.
35
36## 2. Decisions taken before this document
37
381. **Extend the IAM-per-sender model now, and iterate towards a
39 platform SMTP relay.** The relay is wanted anyway, as a layer over
40 SES that could front a different SMTP provider. §10 keeps that door
41 open rather than building it.
422. **Per-instance mode.** Once an app has instance senders, instances no
43 longer receive the app's credential at all.
443. **titogo's backend drives domain onboarding through the console
45 API**, with the CLI as a wrapper over the same doors.
464. **For #147, an operator registers the external identity once and may
47 mark it open; accounts then claim addresses under it themselves.**
485. **A central hub does the onboarding, holding one credential** (§6.1),
49 so no per-instance machine token is needed for this work. The hub's
50 credential carries a new narrow role, `svcrole.Mail`, rather than
51 Admin.
526. **Instance backing is mixed today, and moving to the isolating one is
53 the intent.** So `enable` checks the route's backing rather than
54 assuming it (§6.4).
55
56## 3. What the AWS documentation actually says
57
58The first draft of this design got three of these wrong, so they are
59recorded here with their sources.
60
61- **Condition keys.** The v1 action set, which is what SMTP authorises
62 against, has `ses:ApiVersion`, `ses:FeedbackAddress`,
63 `ses:FromAddress`, `ses:FromDisplayName` and `ses:Recipients`. SES v2
64 adds `ses:TenantName` and `ses:MultiRegionEndpointId`. **No condition
65 key names a configuration set.** The configuration set is a *resource*
66 on the send action, which constrains which set may be named but cannot
67 force one to be named — so a credential cannot be compelled to publish
68 events.
69- **`ses:TenantName` is documented for `SendEmail` and `SendBulkEmail`
70 only.** The `SendRawEmail` row, which is what the SMTP interface
71 authorises against, does not list it, although an AWS blog publishes a
72 policy that uses it there. This is the first spike (§4).
73- **There is no tenant-level default configuration set.** A send on
74 behalf of a tenant must either name a configuration set associated
75 with that tenant, or use an identity whose *default* set is associated
76 with it. So per-sender configuration sets on a shared identity are
77 impossible, and an address sender under a borrowed identity can only
78 publish events if the message names CARLOS's set.
79- **Associating a resource with a tenant does not modify it**, and one
80 identity may belong to many tenants. It has one side effect on the
81 owner: an identity cannot be deleted while an association exists.
82- **SES pauses tenants by itself.** Every new tenant gets the Standard
83 reputation policy, which auto-pauses on high-severity findings, and
84 Trust & Safety can pause one too. `SetKeyActive(true)` cannot lift
85 that. Sending quotas remain account-wide per region; the account
86 suppression list still applies unless a tenant opts into its own.
87- **`ses:caller-identity` names the IAM user** on Send, Bounce,
88 Complaint, Delivery and Reject, but only for events published through
89 a configuration set, and not on Rendering Failure or DeliveryDelay.
90
91## 4. Spike first
92
93Half a day against the Tito deployment's AWS account, on a throwaway
94identity and tenant. It is blocking, because two of its answers change
95the policy this design writes.
96
971. **Send over SMTP with `X-SES-TENANT` set, under a policy conditioned
98 on `ses:TenantName`.** Then send again without the header. Three
99 outcomes: both succeed (the key is not populated for SMTP, so the
100 condition buys nothing); tagged succeeds and untagged is refused (the
101 design works as written); both are refused (SES does not populate the
102 key at all and the condition denies everything). In outcome one and
103 three the tenant condition comes out of the policy, and isolation
104 rests on `ses:FromAddress` plus SES's own tenant-association
105 validation.
1062. **Send under a tenant without `X-SES-CONFIGURATION-SET`** on an
107 identity whose default set is not associated with that tenant, to
108 confirm it is refused, and with the header to confirm events arrive.
1093. **Associate an identity created outside CARLOS** and confirm its MAIL
110 FROM, DKIM and default configuration set are unchanged, and that
111 sends on it *without* a tenant still work — Classic's mail must not
112 change behaviour the moment CARLOS associates `tito.io`.
1134. **Pause a tenant** with `UpdateReputationEntityCustomerManagedStatus`
114 and confirm `GetTenant.SendingStatus` reflects it, so §8 can adopt it
115 as the pause verb.
116
117### 4.1 Results, run 2026-09-15
118
119Run in the Tito deployment's account (665106592142, `eu-west-1`) against
120a throwaway identity `spike.teamtito.com` shaped like `tito.io`: DKIM
121verified, no custom MAIL FROM, and a default configuration set
122(`spike-other`) standing in for Classic's. Every resource was deleted
123afterwards.
124
125**1. `ses:TenantName` is enforced over SMTP. Outcome two, the good one.**
126A policy of `ses:SendRawEmail` on the identity and `spike-owned`,
127conditioned `StringEquals ses:TenantName = spike-tenant`:
128
129| send | result |
130|---|---|
131| `X-SES-TENANT` + `X-SES-CONFIGURATION-SET` | accepted |
132| no headers | 554, not authorized on the identity |
133| configuration set header only | 554, not authorized on the identity |
134
135So the tenant condition stays in the policy, and a credential cannot
136send without naming its tenant. That is real enforcement, not honour
137system, and it is what makes one customer's credential useless for
138another customer's mail.
139
140**2. There is no tenant-level default configuration set, as documented.**
141A send with `X-SES-TENANT` but no configuration-set header was refused
142with `554 ... not authorized to perform 'ses:SendRawEmail' on resource
143'...configuration-set/spike-other'`. SES applied the *identity's*
144default set, which belongs to the identity's owner. So an address sender
145must set both headers, and the bundle carries `<P>_CONFIG_SET` and
146`<P>_TENANT`.
147
148**3. Association is inert for the identity's owner, except for deletion.**
149`GetEmailIdentity` before and after `CreateTenantResourceAssociation`
150was byte-identical: DKIM tokens, `MailFromAttributes` and
151`ConfigurationSetName` all unchanged. A second IAM user with no tenant
152condition — Classic's shape — still sent on the identity with no tenant
153header, and its events still published to the identity's own default
154set, not CARLOS's. `DeleteEmailIdentity` was refused: *"Cannot delete
155... because it has tenant associations."* That is the cross-team
156contract §7 records.
157
158**4. A tenant can be paused directly, and CARLOS can tell whose pause it
159is.** `UpdateReputationEntityCustomerManagedStatus ... DISABLED` made
160the next send fail with `554 Message rejected: Sending disabled for
161<tenant arn>`. `GetTenant.SendingStatus` read `DISABLED`, and
162`GetReputationEntity` returned `SendingStatusAggregate: DISABLED` with
163`CustomerManagedStatus: {Status, Cause: "Status manually updated.",
164LastUpdatedTimestamp}` and a separate `AwsSesManagedStatus` (null here).
165Re-enabling restored sending. The tenant's policy at creation was
166`arn:aws:ses:eu-west-1:aws:reputation-policy/standard`, which is the
167auto-pausing one.
168
169So §8 adopts the tenant pause as the pause verb for any sender that has
170a tenant, keeps `SetKeyActive` for those that do not, and reads the two
171status fields apart so an SES-imposed pause is reported as SES's rather
172than silently "resumed" by CARLOS on the next day roll.
173
174**Also confirmed:** published events carry
175`ses:caller-identity: ["spike-tenant-user"]`, the bare IAM user name,
176which is what §8's attribution reads.
177
178### 4.2 Two prerequisites before any tenant-attributed send
179
180Found while building step 2, and both bite at roll time rather than build
181time.
182
1831. **Existing domains have no tenant association, and nothing will ever
184 create one.** Step 2 associates an identity and the account's
185 configuration set inside `EnsureDomain`, which is reached only from
186 `carlos email domains add`. `VerifyDomain` does not associate. So every
187 domain already in the estate keeps a verified identity with no
188 association, and the first send that carries `X-SES-TENANT` fails SES's
189 validation. Before step 3 or step 5 rolls, either a backfill pass
190 associates every recorded identity and configuration set, or
191 `VerifyDomain` learns to associate too. This is the failure the standing
192 warning in `aws.go` exists to prevent, so it is not optional.
1932. **The control-plane role needs two new IAM actions**,
194 `ses:CreateTenantResourceAssociation` and
195 `ses:DeleteTenantResourceAssociation`. That policy does not live in this
196 repo. If the grant is missing, `AccessDenied` is neither `AlreadyExists`
197 nor `NotFound`, so it is not swallowed: `domains add` fails after the
198 record and the DNS are already written, and `Teardown` returns before
199 `DeleteIdentity` and before releasing the global domain claim, which
200 leaves that claim stranded until the permission lands. Confirm the grant
201 before rolling.
202
203## 5. The sender
204
205A **sender** owns a From address, a credential, a cap, a pause and a set
206of counts.
207
208```go
209// internal/mail/sender.go
210type Sender struct {
211 Schema int `json:"schema"` // 1
212 ID string `json:"id"` // "app", or provision.InstanceID(host)
213 Slug string `json:"slug"` // AWS-safe name component, see below
214 Scope string `json:"scope"` // ScopeApp | ScopeInstance
215 Host string `json:"host,omitempty"`
216 Source string `json:"source"` // SourceDomain | SourceAddress
217 Declared *Declared `json:"declared,omitempty"`
218 Grant *GrantBlock `json:"grant,omitempty"`
219 Provisioned *Provisioned `json:"provisioned,omitempty"`
220 Standalone []*Standalone `json:"standalone,omitempty"`
221 Identity string `json:"identity,omitempty"` // external identity ARN
222 Address string `json:"address,omitempty"` // exact address, lowercased
223 TenantARN string `json:"tenant_arn,omitempty"` // from CreateTenant
224 PerInstance bool `json:"per_instance,omitempty"` // app sender only, §6.3
225}
226```
227
228`Declared`, `GrantBlock`, `Provisioned` and `Standalone` keep their
229current shapes and move inside the sender unchanged.
230
231**Records.** `apps/<acct>/<app>/email/senders/<id>.json`, one object per
232sender, each with its own CAS etag so two instances never contend.
233
234**Names.** The first draft's names were invalid: SES caps tenant and
235configuration-set names at 64 characters of `[A-Za-z0-9_-]`, IAM caps a
236user name at 64, and an instance id is the host itself when it is 40
237characters or less, dots included — `carlos-mail-<acct>-<app>+i+<id>`
238reaches 104 characters. So each sender gets a `Slug`, computed once at
239creation and stored on the record:
240
241- app sender: `Slug = ""`, names unchanged
242 (`carlos-mail-<acct>-<app>`, tenant and set `carlos-<acct>`).
243- instance sender: `Slug = <first 8 of base32(sha256(acct/app/id))>`,
244 giving IAM user `carlos-mail-<acct>-<app>+i+<slug>` and tenant
245 `carlos-<acct>-<app>-<slug>`, both bounded at 12 + 16 + 1 + 32 + 3 + 8
246 = 72 — still over 64 for the longest account and app, so the app
247 component is itself truncated to 24 characters with the hash carrying
248 uniqueness. The exact rule lives in one function, `SenderNames`, with
249 a test that enumerates the bounds.
250
251`SenderNames` also covers standalone credentials, whose current name
252(`carlos-mail-<acct>-<app>+s+<name>`, `policy.go:92`) already reaches 95
253characters against IAM's 64 with nothing checking it. The bounds test
254enumerates every form.
255
256Because the slug is a hash, nothing parses it back, and a 40-bit slug
257needs collision detection. Both are served by a reverse index at
258`email/callers/<iamuser>.json` → (account, app, sender), with the same
259rules the domain claim already uses (`domainclaim.go:124`):
260
261- Written **right after `EnsureUser`** (`walk.go:669`), which runs on
262 every enable, so a re-enable converges it. Writing it before `MintKey`
263 would strand a sender forever when a crash landed between the two.
264- The Put is create-only; on conflict it re-reads. The same (account,
265 app, sender) is success — otherwise it is a slug collision, and the
266 slug is recomputed with a salt that is stored on the record.
267- Deleted by `RevokeStandalone`, `Teardown` and the account reap. The
268 key sits outside `apps/<acct>/<app>/`, so `release.PurgeApp`'s prefix
269 sweep never reaches it; without an explicit delete, a stale entry
270 would attribute SES events to a purged app and insert counts for it.
271- Only the slugged names need it. `SenderOfCaller` (§8) falls back to
272 parsing for `carlos-mail-<acct>-<app>` and its `+s+` standalones.
273
274**Configuration sets stay per account** (`carlos-<acct>`), because an
275identity has only one default set and §3 rules out per-sender sets on a
276shared identity. Each sender's tenant is associated with that set, and
277the set's ARN stays in the sender's policy `Resource`.
278
279`SenderNames` covers standalone credentials too, but it never changes a
280name that exists: `RefreshPolicies` recomputes `StandaloneUserName`
281rather than reading `sa.IAMUser` (`walk.go:364`), so a changed rule
282would write a policy for a new principal and leave the live one stale.
283The rule is: existing shapes are unchanged, and a name that would exceed
284the bound is refused at create.
285
286**Migration.** `mail.Get(acct, app)` prefers `senders/app.json` and
287falls back to `email.json`. When it served the fallback it returns an
288empty etag, so every writer's `Put` — and they all pass back the etag
289they were given (`walk.go:522`, `:736`, `:909`, `:993`, `:1070`) — is a
290create by construction, and a lost race surfaces as `ErrCASConflict`
291exactly as it does today. `mail.Put` **always** writes
292`senders/app.json`. `email.json` is left in place, unwritten, for one
293release and then deleted by a sweep. A console rolled back after the
294first sender write loses writes made in between — a pause or a rotation
295would have to be redone — which is recorded here and in the roll notes.
296
297## 6. #148: instance senders
298
299### 6.1 The caller is a hub, not the customer instances
300
301titogo's own backend onboards every customer, so the instance email
302doors are `Allow(svcrole.Admin)` — the same role the app-level email
303doors carry today — and the hub holds one named service credential with
304that role, bound to the titogo app.
305
306That credential must not be delivered into customer instances. Today
307`handleAPIInstanceToken` mints a token bound to (account, app) and its
308documented delivery path is `carlos secrets set -app <app>`
309(`apiinstances.go:1367`), the bundle every instance renders, so anything
310delivered that way reaches every customer. The hub's credential lives
311wherever the hub runs — a different app, or an instance with its own
312overlay once §6.2 exists. The console cannot check this: bundle values
313are sealed and it holds no key (`apimail.go:483`), so where a credential
314was delivered is not a decidable question. It is an operator
315attestation, stated in the enable door's documentation, not a gate.
316
317**What an Admin-role service credential reaches.** `svcrole.Allow` adds
318Admin to every non-empty set (`svcrole.go:120`), so a titogo-bound Admin
319credential reaches every app-scoped door that names any role: ship,
320promote and rollback, pipeline, ledgers, instances create/patch/delete/
321move, domains, logs and analytics, features, place, the store's
322create/provision/grant/rotate, relay, and every email door except
323credentials and pause. It cannot reach the person-only doors
324(`svcrole.Set{}`): instance-token minting, the instances manifest,
325channel, routes, service minting, fleets, store apply and approve, and
326system. A compromised hub is therefore a compromised titogo deploy, not
327only a compromised mailer.
328
329That inverts why those doors are Admin in the first place
330(`apimail.go:460`): Admin was chosen to be *narrower* than Publish or
331Instance, so that a deploy credential could not acquire a sending
332identity. For a hub whose only job is onboarding, it means the opposite
333— it must be able to deploy in order to onboard.
334
335**So this design adds one entry to the closed role catalogue,
336`svcrole.Mail`**, and every email door — the app-level ones that exist
337and the instance-level ones in §6.4 — becomes `Allow(svcrole.Mail)`.
338Because `Allow` always adds Admin (`svcrole.go:120`), today's Admin
339credentials keep reaching them, so nothing breaks; what changes is that
340titogo's hub can hold a credential that onboards customer domains and
341does nothing else. `svcroutes_test.go` pins the new role's door list.
342
343**Rate budget.** `mailWriteGates` uses the person-sized write limiter
344keyed on the audit string (`apimail.go:136`), not `allowWrite` for
345service principals (`svcprincipal.go:394`). Bulk onboarding through the
346hub will 429 against a person's budget, so the mail write gate takes the
347service budget when the principal is a service.
348
349Two consequences worth stating plainly. If customer code ever needs to
350call these doors for itself, tokens bound to (account, app, instance)
351become a prerequisite again, and this design is shaped so that is a
352later change to the door's role set and nothing else. And separately,
353the app-wide `CARLOS_INSTANCE_TOKEN` that customer instances already
354hold reaches the existing instance create, patch and delete doors
355(`authcli.go:612`, `:628`), so one customer's code can already delete a
356sibling instance. That is not introduced here and not fixed here; it is
357worth its own discussion.
358
359### 6.2 Per-instance config delivery
360
361A third bundle layer, resolved after the named environment:
362
363```
364apps/<acct>/<app>/config.json default
365apps/<acct>/<app>/environments/<name>.json environment
366apps/<acct>/<app>/instances-config/<host>.json instance ← new
367```
368
369- Same `Manifest` shape, same sealed envelopes, same name and value
370 rules. `ExecDelivery` is **not** read from this layer, matching the
371 ruling at `appconfig.go:61-69`.
372- `LoadMerged` keeps its signature; `LoadMergedHost(ctx, s, acct, app,
373 env, host)` adds the third overlay, and `Sync`/`SyncExec` call it per
374 route after resolving the group base, so a pass costs one extra bucket
375 read per instance that has an overlay and none for instances that do
376 not.
377- Writing the layer bumps the app's generation object, as
378 `SetExecDeliveryAt` does, so the fast loop converges.
379- Guards that assume a two-layer merge and must be revisited:
380 `lostKeys`/`defaultMissing` (`sync.go:260`, `:388`), the mass-removal
381 guard (`syncexec.go:351`), `AwaitingRestart` (`syncexec.go:712`) and
382 `allShadowed` (`syncexec.go:813`). `ConfigRouteFingerprints`
383 (`routefingerprint.go:58`) digests registry columns only and cannot
384 see a bucket-side change; the generation bump is that layer's signal.
385- Only the platform writes this layer: no CLI verb, no console door. It
386 exists so a credential can be scoped to one instance.
387
388### 6.3 Entering per-instance mode
389
390A prefix listing cannot decide this without racing an app-sender enable,
391so the app sender carries `PerInstance bool`, and the transition is one
392CAS write on `senders/app.json` that sets `PerInstance` together with
393`Active = false` and `PauseKind = PauseKindPerInstance`, then:
394
3951. **`ReapKeys(user, nil)` on the app sender's IAM user and on every
396 standalone user** — list-and-delete, not `SetKeyActive(false)` on the
397 recorded ids. Deactivating recorded ids is not enough, because
398 `Enable` mints and delivers a key *before* its own CAS `Put`
399 (`walk.go:717-738`): an enable racing the transition leaves an
400 active, unrecorded key in every instance's bundle that nothing would
401 ever reap. Deleting whatever keys exist makes the racing delivery
402 inert. Instance `Enable` re-runs the same reap.
4032. Unsets the app sender's `CARLOS_SMTP_*` vars from the default bundle.
404
405`PauseKindPerInstance` matters because `resumeStaleCapPause`
406(`mailcounts.go:160`) runs on the status read and reactivates *every*
407key including standalones (`walk.go:973`); a cap-paused app sender would
408otherwise re-arm the credential every instance still holds on the next
409UTC day roll. `SetPaused(false)` and `resumeStaleCapPause` both refuse
410while `PerInstance` is set.
411
412App-sender `Enable` refuses when `PerInstance` is set, under that same
413etag. Instance `Enable` requires it. Leaving per-instance mode, once the
414last instance sender is gone, is one CAS that clears `PerInstance`, sets
415`Active = true` and clears `PauseKind`, followed by a normal enable —
416otherwise `Enable` carries the false `Active` forward (`walk.go:698`)
417and mints a dead key.
418
419`grantedDomains` filters on the domain record's new `Senders` list, not
420`Apps`, and an instance sender's claim never adds the app to `Apps`. So
421the app sender's policy can never name a customer domain.
422
423Everything that walks `rec.Apps` must walk `Senders` instead, or an
424instance sender's policy is never refreshed when its configuration set
425is rebound — which is 554 on every send, the failure #271 and #279
426already cost us. That means `RefreshPolicies` (`walk.go:324`, called
427from `EnsureDomain:297` and `VerifyDomain:461`) and `mailAppDomains`
428(`apimail.go:299`), which the enable door and status both read.
429
430There is also a door for the transition itself: the first instance
431`enable` performs it, and `DELETE .../instances/{host}/email` on the
432last instance sender leaves per-instance mode. Nothing else sets the
433flag.
434
435### 6.4 The doors, and what they check
436
437```
438POST /api/cli/apps/{account}/{app}/instances/{host}/email declare
439POST .../instances/{host}/email/domains attach
440POST .../instances/{host}/email/domains/{domain}/verify
441POST .../instances/{host}/email/enable
442POST .../instances/{host}/email/rotate
443GET .../instances/{host}/email status
444DELETE .../instances/{host}/email teardown
445```
446
447All are `Allow(svcrole.Mail)`, like the app-level email doors after
448§6.1, so the `Instance` role pin is untouched. Pause,
449resume and cap changes stay operator-only through the existing
450`POST …/email/pause`, which gains an `instance` field.
451
452Two checks at `enable`, both refusals with an explaining message:
453
454- **A uid of its own.** The predicate is not "unit-backed" — a host_role
455 app runs its units as `User=carlos-<app>` with `DynamicUser=no`
456 (`caps/render.go:101-112`), one uid per *app*, and `ProtectProc`
457 hides other users' processes, not a sibling's at the same uid. The
458 check is that this instance's process gets a uid no sibling shares:
459 `DynamicUser=yes` with no host_role grant, or an exec route under an
460 isolated-exec layout with a per-instance uid. Plain exec routes share
461 the agent's uid and `<dataDir>/<host>.config.env` is readable by every
462 sibling (`internal/activator/admintoken.go:6-12`), so a per-instance
463 credential buys nothing there. This is the dependency on #126.
464- **Delivery.** The instance's box must be running an agent that renders
465 the instance layer (§6.5).
466
467**Domain attach.** `claimDomain` gains a `Sender` field and refuses a
468re-claim whose (app, sender) differs from the holder's, *including
469within the same account* — today a same-account re-claim is a silent
470no-op (`domainclaim.go:137`), which would let one instance attach
471another's verified domain. `EnsureDomain` refuses a domain that has an
472external-identity record before it claims anything, and the orphan-claim
473sweep in `Teardown` matches on sender and skips external domains, so a
474refused `EnsureIdentity` can never wedge an app's teardown.
475
476### 6.5 Confirming delivery
477
478`ConfigAck` gains `Schema int` and `InstanceConfigETag string`: the etag
479of the instance overlay the agent rendered. The console compares it with
480the etag its own `Put` returned, which is decidable — unlike a key-id
481map, which cannot distinguish an old agent from a new agent rendering a
482route with no mail vars.
483
484An ack is written only when a pass changes a file (`configack.go:66`),
485so a fresh instance, or one whose config has not moved since its agent
486was rolled, has no ack at all — absence cannot mean "old agent".
487
488**The capability question is answered by the box record's agent version,
489not by the ack.** An earlier draft had the agent rewrite the ack on every
490pass so that absence became decidable. That is the wrong trade: it is a
491Get and a Put per converged route per pass on every box in the fleet,
492where today it is one write per changed file, and on a fleet-door box it
493is worse than useless — `control/config-ack/` is outside the door's read
494and write scopes (`edge/storedoor.go:389`, `:489`), so every one of those
495passes would fail, log, and still leave the console with nothing. It also
496forces the ack's `At` to stop describing the delivery it sits beside.
497
498So acks stay change-only. The enable-time refusal reads the agent version
499and names the box and the version. Delivery is still decidable where it
500matters, because writing an instance's credential always changes that
501instance's file, so it always produces an ack carrying the
502`InstanceConfigETag` to compare. A missing ack reads as "not yet
503delivered", never as "old agent". `carlos email status --instance <host>`
504reports delivered, stale or not yet delivered.
505
506**A fleet-door box cannot confirm delivery at all**, because it cannot
507write the ack. Until that door grants `control/config-ack/<box>/`, an
508instance sender on such a box can be enabled but its delivery will read
509as unconfirmed. The roll note says so, and the door grant is its own
510piece of work.
511
512## 7. #147: a sender under an identity CARLOS does not own
513
514**The external identity** is registered once per deployment by an
515operator at `email/external-identities/<domain>.json`: domain, region,
516identity ARN, `Open bool`, who registered it and when. The region must
517match the deployment's, because tenants and identities are per region.
518
519CARLOS never calls `CreateEmailIdentity`,
520`PutEmailIdentityMailFromAttributes` (both inside `EnsureIdentity`),
521`SetIdentityConfigSet` or `DeleteIdentity` for a registered identity. A
522provisioner wrapper refuses those verbs for any domain with such a
523record, and a test pins the refusal. `EnsureDomain` refuses the domain
524before claiming it (§6.4), so the refusal can never strand a claim.
525
526**Enable takes an address branch.** Today `Enable` resolves granted
527domains and refuses with `ErrNoDomains` when there are none
528(`walk.go:627`), then insists the From address sits at a granted domain
529name (`:647`), and `SendPolicyJSON` requires each identity name to be a
530granted domain or a parent of one (`policy.go:167`). An external
531identity has no `DomainRecord` in the account by design, so an address
532sender needs its own branch in both: the identity ARN comes from the
533external-identity record, the From address from `Sender.Address`, and
534the policy renderer gets an address form whose identity check is that
535the address's domain is the identity or a child of it.
536
537**Verification.** `Status.Verified()` requires `MailFrom == "SUCCESS"`
538(`provisioner.go:45`), which an identity with no custom MAIL FROM never
539reaches. External identities are checked with a relaxed predicate: DKIM
540and sending enabled, MAIL FROM ignored.
541
542**Address claims.** `email/address-claims/<address>/<acct>.json`, where
543the address is lowercased and validated to the same grammar `Declared`
544already enforces. The claim is **not** exclusive: `jam` and `stripe` may
545both hold `jelly@tito.io`. It names the account, app, sender and who
546claimed it, and `ListAddressClaims` gives the operator the roster.
547Claiming requires `Open == true`; otherwise an operator writes the same
548record as a grant. The policy condition is
549`StringEqualsIgnoreCase ses:FromAddress`, since SES's `StringEquals` is
550case-sensitive and a From address is not.
551
552**An address sender's address is fixed once claimed.** Changing it means
553tearing the sender down and claiming again, and two doors refuse the
554shortcut: the claim door when the sender already holds a different
555address, and `Declare` when a From address would disagree with the
556claimed one. Without both, `carlos email enable --from other@tito.io` on
557a live sender renders a policy for the new address, strands the old
558claim — which blocks every sibling app in the account and holds the
559identity association open — and leaves `_FROM` and the policy naming
560different addresses while status still reads healthy.
561
562**Counting an address sender depends on the app.** Because there is no
563tenant-level default set (§3) and CARLOS may not touch `tito.io`'s
564default, events only reach CARLOS when the message names CARLOS's
565configuration set. So the sender's env bundle gains
566`<P>_CONFIG_SET`, documented as a header the app must set, and
567`carlos email status` says plainly when a sender has sent nothing —
568the honour-system limitation you accepted, and one the relay removes.
569The bundle also carries `<P>_TENANT`, because the spike (§4.1) kept the
570`ses:TenantName` condition in the policy and a send that omits it is
571refused outright. That refusal is the saving grace of the honour system:
572a message missing either header does not send silently uncounted, it
573does not send at all.
574Enable also reads the external identity's default configuration set: if
575it has one that is not associated with the sender's tenant, a
576tenant-tagged send is refused outright, so enable refuses first, with
577that explanation.
578
579**Doors.** Registration is operator-only, the claim is a member's, and
580both are new routes:
581
582```
583PUT /api/cli/email/external-identities/{domain} register (operator)
584DELETE /api/cli/email/external-identities/{domain} deregister (operator)
585GET /api/cli/email/external-identities list (operator)
586GET /api/cli/email/external-identities/{domain}/claims roster (operator)
587POST /api/cli/apps/{account}/{app}/email/address claim + declare (Admin)
588```
589
590The claim door takes `{address}` and is what sets
591`Sender.Source = SourceAddress`, `Sender.Address` and
592`Sender.Identity`; it is `Declare` for an address sender, so declaring a
593From address and claiming it are one call and cannot disagree. Enable,
594rotate, status and teardown are the existing app-sender doors, which
595read `Source` off the record. CLI: `carlos email external-identity
596register|list|claims`, and `carlos email enable --from jelly@tito.io`
597calls the claim door first when the domain resolves to an external
598identity.
599
600**Refusal scope.** `EnsureDomain` refuses a domain that is a registered
601external identity **or any subdomain of one**: a claim on
602`mail.tito.io` would otherwise let CARLOS create and later delete an
603identity underneath one it does not own.
604
605**Provisioner verbs.** The interface (`provisioner.go:49`) gains four,
606each idempotent in the usual way, with `MemProvisioner` equivalents:
607
608```go
609AssociateTenantResource(ctx, tenant, resourceARN string) error
610DisassociateTenantResource(ctx, tenant, resourceARN string) error
611TenantStatus(ctx, tenant string) (TenantStatus, error) // GetTenant
612SetTenantSending(ctx, tenantARN string, enabled bool) error // §8's pause
613```
614
615plus `IdentityStatus` returning the identity's bound configuration set,
616which it already carries (`Status.ConfigSet`), so enable can refuse an
617external identity whose default set is not associated with the tenant.
618
619**Associations are wired in step 2, not here.** Today `EnsureTenant`
620creates the container and nothing associates anything, and `aws.go:171`
621warns that no send may carry `X-SES-TENANT` until associations exist.
622Step 2 associates each account's existing tenant with its identities and
623its configuration set, which changes no behaviour on its own, so that
624step 3 can rely on them.
625
626**The cross-team contract.** While CARLOS's tenant association exists,
627Classic cannot delete the `tito.io` identity — `DeleteEmailIdentity`
628fails until the association goes. The registration step says so, and
629teardown removes the association.
630
631**The trade-off you accepted.** With `Open`, any account in the
632deployment can claim any address under `tito.io`, including
633`billing@tito.io`. The cheap mitigation if it bites is an optional
634`Local []string` allowlist on the external-identity record; it is not
635built now.
636
637## 8. Counting, caps and pauses per sender
638
639**Attribution by caller.** Every sender has its own IAM user, and SES
640stamps `mail.tags["ses:caller-identity"]` with the user name on Send,
641Bounce, Complaint, Delivery and Reject. `SenderOfCaller(caller)`
642resolves it to (account, app, sender) in two steps: parse first, exactly
643as `AppOfCaller` does today (`policy.go:42`), which answers for
644`carlos-mail-<acct>-<app>` and its `+s+` standalones with sender
645`"app"` and costs nothing; then, only for a slugged `+i+` name, one Get
646on `email/callers/<iamuser>.json`. That ordering matters on the roll:
647every app enabled today short-circuits `Enable` before `MintKey`
648(`walk.go:677`), so it has no index entry until it next rotates, and a
649Get-only resolver would silently stop writing the `by_sender` log rows
650that `sesevents.go:498` writes now.
651
652`recordAndEnforceMail` counts against the resolved sender once, and
653falls back to today's domain-wide loop only when the caller is unknown,
654skipping any app in per-instance mode so its events are never charged to
655the app sender.
656
657**This is a visible change, and it belongs in the roll note.** Today
658every app on a shared domain is charged for every event on that domain
659(`sesevents.go:509`); afterwards only the sending app is. A sibling's
660cap stops being consumed by another app's sends, which is the intent,
661but a tenant watching its own numbers will see them move.
662
663**Counts.** A new table, because SQLite cannot widen a primary key:
664
665```sql
666CREATE TABLE IF NOT EXISTS mail_sender_counts (
667 account TEXT NOT NULL, app TEXT NOT NULL, sender TEXT NOT NULL,
668 day TEXT NOT NULL, sends INTEGER NOT NULL DEFAULT 0,
669 bounces INTEGER NOT NULL DEFAULT 0, complaints INTEGER NOT NULL DEFAULT 0,
670 rejects INTEGER NOT NULL DEFAULT 0, deliveries INTEGER NOT NULL DEFAULT 0,
671 PRIMARY KEY (account, app, sender, day));
672```
673
674Backfilled once from `mail_counts` with `sender = 'app'`; `mail_counts`
675stays, unwritten, and is dropped a release later by a sweep that runs
676once at console start. The rename, transfer, purge and account-reap
677sweeps (`carryMailApp`, `forgetMailApp`, `forgetMailAccount`) all take
678the new table, but it cannot join `mailAppTables` blindly because those
679sweeps key on a timestamp column (`mailAppTimeColumn`, `mailevents.go:170`)
680and this table has a `day` string instead — so it gets an explicit case
681comparing `day` against the date. That also closes today's gap, where
682`mail_counts` rows survive a rename.
683
684**Caps and pauses.** `TierCap` and the thresholds are unchanged and now
685apply per sender. A pause calls `SetKeyActive(false)` on that sender's
686keys alone, so one customer's bounce rate pauses that customer.
687`resumeStaleCapPause` keeps its rule: cap pauses lift on the UTC day
688roll, rate and manual pauses never do.
689
690**The pause verb is the tenant's.** §4.1 confirmed it, so a sender with
691a tenant is paused with
692`UpdateReputationEntityCustomerManagedStatus(RESOURCE, <tenant ARN>,
693DISABLED)` and `SetKeyActive` stays the fallback for senders that have
694none. That is a better pause than deactivating a key: it takes effect on
695the next send rather than waiting for a credential refresh, and it
696leaves the credential intact so resuming is one call.
697
698`EnsureTenant` stores `TenantArn`. Status reads `GetReputationEntity`
699and keeps `CustomerManagedStatus` and `AwsSesManagedStatus` apart, so an
700SES-imposed pause is reported as SES's and is never lifted by
701`resumeStaleCapPause` on the day roll — a tenant SES disabled for
702reputation must not be resumed by a cap timer.
703
704**Status.** `carlos email status --app <app>` prints the app sender as
705today plus a line per instance sender; `--instance <host>` prints one.
706The API reply gains a `senders` array while its existing top-level
707fields keep describing the app sender, so an older CLI still works.
708
709## 9. Testing
710
711- `MemProvisioner` grows tenants with associations and reputation
712 status, and refuses the forbidden verbs for a registered external
713 identity; a test pins that refusal and one pins that `EnsureDomain`
714 refuses before claiming.
715- Walk tests per sender kind: address enable and teardown never call the
716 forbidden verbs; instance enable writes only the instance layer;
717 entering per-instance mode deactivates the app key and every
718 standalone, and unsets the vars in the same pass.
719- Race test: an app enable that mints and delivers before its CAS `Put`,
720 interleaved with the per-instance transition, leaves no usable app
721 credential — the reap deleted the key even though the vars landed.
722- A cap-paused app sender that enters per-instance mode is not
723 reactivated by the next UTC day roll.
724- Name tests enumerating the IAM and SES bounds, including the longest
725 legal account and app.
726- Policy tests: address condition (case-insensitive), tenant condition
727 present and absent, and the existing refusals.
728- Attribution tests: caller names an instance sender; per-instance mode
729 never charges the app sender; unknown caller keeps today's behaviour;
730 an event type with no caller tag.
731- Door tests: an app-scoped `Instance`-role token is refused at every
732 instance email door, and a stranger's 404 is byte-identical to the
733 refusal for an instance that exists in another account.
734- Config tests: three-layer precedence, an old agent ignoring the layer,
735 the revisited `lostKeys` and mass-removal guards, and the ack carrying
736 the overlay etag.
737- Door tests in `svcroutes_test.go`, the `Instance` role pin exemption,
738 and the backing and delivery refusals at enable.
739
740## 10. What this leaves for the relay
741
742The relay becomes a third `Source` on the same sender: `SourceRelay`.
743The app still reads `<P>_HOST`, `<P>_PORT`, `<P>_USER`, `<P>_PASS` and
744`<P>_FROM`, so moving a sender onto it is a credential swap and a host
745change, invisible to the app's code. It enforces the cap before the send
746rather than after, removes the `<P>_CONFIG_SET` honour system for
747address senders, spends one IAM user for the fleet rather than one per
748sender, and is where a non-SES SMTP provider attaches.
749
750## 11. Order of work
751
7521. **Spike** (§4). Blocking; changes the policy and the pause verb.
7532. **Senders** (§5): the record, the migration, the naming, per-sender
754 counts, caps, pauses and status. No new capability, and it lands
755 `mail_sender_counts` and attribution by caller.
7563. **#147** (§7): external identities, address claims, the provisioner
757 wrapper, the address branch through enable and the policy renderer,
758 `<P>_CONFIG_SET`. Unblocks Jelly and depends only on step 2.
7594. **Per-instance config layer and acks** (§6.2, §6.5).
7605. **#148** (§6.3, §6.4): instance senders, per-instance mode, the
761 doors, the uid check.
7626. **Per-instance machine tokens**, after #148 rather than before it.
763 Not needed while a hub onboards, but the app-wide
764 `CARLOS_INSTANCE_TOKEN` every customer instance holds already reaches
765 the instance create, patch and delete doors (`authcli.go:612`,
766 `:628`), so one customer's code can delete another's instance today.
767 That is filed separately; this is where the fix lands.
768
769## 12. Quotas and costs to watch
770
771- IAM users: 5,000 per AWS account, **not adjustable**. One per instance
772 sender plus one per standalone credential. At titogo's scale this is
773 the ceiling that eventually forces the relay.
774- IAM inline policy: 2,048 characters per user, all policies aggregated.
775 This bites sooner than it looks: every granted domain contributes the
776 *same* account configuration-set ARN and nothing dedupes it
777 (`walk.go:634`, `:343`, `policy.go:205`), so a twelve-domain app is
778 already over. `slices.Compact` after the sort in the three callers
779 takes that to about twenty, and `SendPolicyJSON` refuses with the
780 count rather than letting IAM reject the policy.
781- Configuration sets: 10,000 per region, not adjustable. Per account
782 here, so not a constraint.
783- SES tenants: 10,000 per region, adjustable. Associations per tenant
784 are undocumented — ask AWS before relying on many.
785- The SES control plane is throttled to one request per second for
786 everything except sends, so an enable that creates a tenant, two
787 associations and IAM resources needs backoff, and bulk onboarding is
788 serial.
789- SES charges per tenant per month on top of per-message pricing.
790
791## 13. Out of scope
792
793- The relay itself.
794- Per-instance sending domains under the platform suffix; an instance
795 sender needs a custom domain or a registered address.
796- Inbound email (#138), event push per sender, per-sender billing.
797- `Local` allowlists on an external identity (§7).
798