amadan

carlos / platform Public

spec: inbound email as a first-class platform layer

SES receiving as the mirror of internal/mail, scoped to stage 1
(forwarding-only) for James's shared-inbox app.

The three forced divergences from the sending layer are the substance:
receiving has no IAM isolation to have (one active receipt rule set per
account per region, so the control plane must own it wholesale and never
patch it additively); the MX lives on a platform-owned receiving domain,
which is why the no-record-at-the-apex invariant survives stage 1; and
the cap protects wake budget rather than sending reputation, so it stops
delivery without refusing the mail.

Delivery is a pointer plus a non-stamping wake, reusing #371's
ServeScheduled reasoning — inbound has exactly the shape that finding
described, and a delivery must be able to wake an instance without being
able to hold one awake.

Records SES's verdicts as verified against the developer guide rather
than remembered, in the spirit of internal/mail/aws.go's tenant note, and
states plainly that on the forwarding path they are provenance rather
than a filter: forwarded mail fails SPF by construction, so an
implementation that treats that as suspicious eats real customer mail.

Draft for review — not approved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FKRd4dVyn1Da27uGBj9Skh
Paul Campbell pushed by paul@keymail.dev 231a997615ca2d4d706233a926f91da87ea1a6ea parent f7ac3aa
1 file changed, +492 −0
  • docs/superpowers/specs/2026-09-04-inbound-email-design.md +492 −0
diff --git a/docs/superpowers/specs/2026-09-04-inbound-email-design.md b/docs/superpowers/specs/2026-09-04-inbound-email-design.md
new file mode 100644
index 000000000..c73761526
--- /dev/null
+++ b/docs/superpowers/specs/2026-09-04-inbound-email-design.md
@@ -0,0 +1,492 @@
+# Inbound email — SES receiving as a first-class platform layer
+
+**Date:** 2026-09-04 · **Status:** draft (design conversation 2026-09-04) ·
+**Requested by:** Paul, for James's shared-inbox app (a CARLOS
+implementation of letsjelly.com)
+
+**Builds on:** 2026-08-19-tenant-email-design.md (`internal/mail` — the
+sending layer this one is the mirror of, and whose domain record stage 2
+extends), #151 / 2026-08-12-domain-attach-design.md (the global claim,
+first-writer-wins), 2026-08-27-managed-custom-domains-design.md (managed
+customer-domain connections — where stage 2 lives, not here),
+2026-08-30-scheduled-ticks-and-hibernation-design.md / #371
+(`ServeScheduled`, the non-stamping wake this spec's delivery path reuses).
+
+---
+
+## 0. Scope rulings taken in conversation (Paul, 2026-09-04)
+
+Four rulings shaped this, and each one removed a subsystem:
+
+1. **Stage 1 is forwarding-only.** Customers forward to a platform address;
+ nobody points an MX at CARLOS yet. Direct MX is stage 2, sketched in §10
+ and not specced here.
+2. **Delivery is a pointer plus a wake**, not an inline POST of the message
+ and not a mailbox the app polls. §5.
+3. **Outbound is out of scope entirely.** Customer sending will be some
+ combination of the titogo tenancy model, custom SMTP, and the Gmail API.
+ Nothing in this layer sends. §9 records what that buys.
+4. **Therefore the domain record needs no sub-tenant dimension** — at either
+ stage. Stage 2 needs a *receiving* grant per customer domain, and that is
+ a capability on the `internal/domains` connection, not a new axis on
+ `internal/mail`'s `DomainRecord`. §10.
+
+The driving use case matters to the reader: a shared inbox. It is
+human-attended during a working day, it needs threading (`Message-ID`,
+`In-Reply-To`, `References`) and attachments, and it wants messages in
+seconds rather than on a poll. That is what rules out both alternatives in
+§5, and it is why §7's verdict discussion lands where it does.
+
+---
+
+## 1. What this layer is
+
+An app declares that it *receives* email. The platform mints it addresses on
+a receiving domain the platform owns, arranges for SES to accept mail for
+those addresses, stores each raw message in a per-tenant prefix of a
+deployment-owned bucket, and hands the app a pointer plus the envelope and
+SES's scanning verdicts — waking a hibernating instance to do so.
+
+It is the mirror of `internal/mail` and follows it verb for verb, because it
+is the same problem with the arrows reversed: a declaration block, a grant
+block, a provisioned block, idempotent walks, a console door family, a
+`carlos` verb, a daily cap and a pause. Where it *diverges* from
+`internal/mail` is the subject of §3, and every divergence is forced.
+
+New package: **`internal/inbound`**. Not a second half of `internal/mail`.
+The two share nothing at stage 1 (§10 is where they meet, on the domain
+record), and `internal/mail` is already 6,954 lines across fifteen files.
+
+---
+
+## 2. Records
+
+### 2a. `apps/<acct>/<app>/inbound.json`
+
+Mirrors `mail.Record`'s three-blocks-for-three-actors shape exactly, so a
+reader of one can read the other:
+
+```go
+type Record struct {
+ Schema int `json:"schema"` // 1
+ App string `json:"app"`
+ Declared *Declared `json:"declared,omitempty"` // the member's half
+ Grant *GrantBlock `json:"grant,omitempty"` // the authorisation half
+ Provisioned *Provisioned `json:"provisioned,omitempty"` // the walk's half
+}
+```
+
+- **`Declared`** — `EnvPrefix` (default `CARLOS_INBOUND`), `DeliverPath` (the
+ route path the wake POSTs to, default `/_carlos/inbound`), `At`, `By`.
+- **`GrantBlock`** — `Addresses []string` (the local parts minted for this
+ app on the receiving domain), `DailyCap int`, `Source string`
+ (`tier:free` / `tier:pro` / `operator`, the same recomputable-without-
+ clobbering-an-override discipline `mail.GrantBlock` documents), `At`, `By`.
+- **`Provisioned`** — `Bucket`, `Prefix`, `Region`, `QueueARN`, `Active bool`,
+ `PauseKind string` (`cap` / `rate` / `""` for manual, §8), `At`.
+
+`Key(acct, app) = release.AppPrefix(acct, app) + "inbound.json"`. CAS on
+every write, read-modify-write, `release.ErrCASConflict` surfaced never
+absorbed — `mail.Put`'s rule.
+
+### 2b. The address claim — `email/inbound-addresses/<address>.json`
+
+A **global**, create-only key, for precisely the reason
+`internal/mail/domainclaim.go` gives for its own: the key itself is the
+uniqueness gate, because there is only ever one object to Put. Two apps must
+not both own `support@in.oncarlos.com`, and keying this under an account
+prefix would let each account hold its own claim on the same address —
+which is the whole failure the file exists to prevent.
+
+Address grammar is deliberately narrow: `^[a-z0-9][a-z0-9._-]{2,63}$`, no
+plus-addressing in the claim (a `+tag` suffix routes to the base claim, §4),
+and a reserved list — `postmaster`, `abuse`, `noreply`, `mailer-daemon`,
+`admin`, `root` — that no app may claim, because the deployment needs them
+and RFC 2142 expects two of them to exist.
+
+---
+
+## 3. The three ways this differs from sending, and why
+
+Each of these is a place where copying `internal/mail` would be wrong.
+
+### 3a. There is no IAM isolation to have
+
+Sending isolates tenants with a per-app IAM user carrying an inline policy —
+that is `internal/mail/policy.go`'s entire argument, and its doc comment is
+blunt that the policy, not the SES tenant, is the security boundary.
+
+**Receiving has no equivalent.** SES has exactly one *active receipt rule
+set* per account per region: an ordered list, matched by recipient
+condition, and every tenant's inbound configuration is an entry in that one
+shared mutable global. There is no per-tenant principal to scope, because no
+tenant principal is involved in receiving at all — SES acts on its own.
+
+The consequence is a hard rule, and it is the single most important
+sentence in this spec:
+
+> **The control plane owns the entire receipt rule set and rewrites it
+> wholesale from the records. Nothing ever patches a rule into the live set
+> additively.**
+
+`PutReceiptRuleSet` into a versioned name, then `SetActiveReceiptRuleSet` —
+so the swap is atomic and the previous set survives for rollback. An
+additive patch model would make a mis-ordered or overlapping rule deliver
+one tenant's mail into another tenant's prefix, which is the same class of
+hole `domainclaim.go` was written to close, reached by a different road.
+
+Ordering within the rendered set is fixed and derived, never authored:
+explicit-recipient rules sorted by address (longest match is irrelevant —
+SES matches recipients, not prefixes, so exact addresses cannot overlap
+once §2b guarantees uniqueness), then a terminal catch-all rule whose action
+is a `Stop`. The catch-all is what makes an unknown recipient a **silent
+drop** rather than an accept-then-bounce (§7).
+
+`ScanEnabled` is true on every rule, as a platform invariant rather than a
+per-app knob, because §7's verdicts do not exist without it.
+
+### 3b. The MX record and the apex
+
+`internal/mail/dnsrecords.go` documents that nothing is published at the
+domain apex — the edge owns the A record at an app's host — and
+`TestNoRecordSitsAtTheDomainApex` keeps it true.
+
+**Stage 1 does not touch that invariant**, and this is worth stating so
+nobody goes looking for a conflict that is not there. The receiving domain
+is a distinct label the platform owns — `in.<PlatformSuffix>`, e.g.
+`in.oncarlos.com` — which is not any app's host and carries no A record. Its
+records are:
+
+| Name | Type | Value |
+| --- | --- | --- |
+| `in.<suffix>` | MX | `10 inbound-smtp.<region>.amazonaws.com` |
+| `in.<suffix>` | TXT | `v=spf1 -all` (it never sends) |
+| `_dmarc.in.<suffix>` | TXT | `v=DMARC1; p=reject; rua=<deployment rua>` |
+
+The SPF and DMARC records are not decoration: a receive-only domain that
+publishes neither is an attractive forgery target, and both records cost one
+zone write at deployment setup.
+
+Stage 2 is where the apex collision becomes real, because a customer's
+`theircompany.com` may hold both an MX for us and an A record elsewhere.
+§10 records it; this stage does not solve it.
+
+### 3c. Reputation runs the other way
+
+Outbound's scarce resource is the shared sending reputation, which is why
+`internal/mail` has a daily cap and an automatic pause. Inbound's scarce
+resources are different — S3 spend, and instance wake time (§6) — so the cap
+exists for a different reason and behaves differently on trip (§8). Copying
+the outbound cap semantics unexamined would produce a brake that stops the
+wrong thing.
+
+---
+
+## 4. The receive path
+
+```
+sender → SES (receipt rule set, scan enabled)
+ → S3 action: s3://<deployment-inbound-bucket>/<acct>/<app>/<messageId>
+ → SNS topic → SQS queue (per deployment)
+ → box deliverer (§6) → non-stamping wake → POST to the app's route
+```
+
+Recipient → app resolution happens **once**, in the reconciler that renders
+the rule set, and again in the deliverer as a check. A `+tag` suffix is
+stripped before the claim lookup and preserved verbatim in the delivered
+payload — that is how a shared inbox does per-thread or per-customer
+addressing without a claim per thread.
+
+The S3 object key is SES's `messageId`, which is also the notification's
+`objectKey`. One identifier, two uses, no mapping table.
+
+---
+
+## 5. Delivery contract: pointer plus wake
+
+The app receives an HTTP POST at `Declared.DeliverPath` on its own route,
+carrying JSON:
+
+```json
+{
+ "schema": 1,
+ "message_id": "<SES messageId — also the S3 object key>",
+ "received_at": "<ISO 8601>",
+ "recipient": "support+acme@in.oncarlos.com",
+ "recipient_base": "support@in.oncarlos.com",
+ "tag": "acme",
+ "envelope": { "from": "...", "to": ["..."] },
+ "headers": { "message_id": "...", "from": "...", "subject": "...",
+ "in_reply_to": "...", "references": "..." },
+ "headers_truncated": false,
+ "arrival": "forwarded",
+ "verdicts": { "spam": "PASS", "virus": "PASS", "spf": "FAIL",
+ "dkim": "PASS", "dmarc": "FAIL", "dmarc_policy": "none" },
+ "size": 48213,
+ "raw_url": "<presigned GET, ~1h>"
+}
+```
+
+**Why a pointer and not the message.** An inline POST caps message size at
+whatever the route will take and puts MIME parsing in the platform; SES
+accepts up to 40MB and a shared inbox gets attachments. A mailbox the app
+polls fails the "in seconds" requirement and cannot wake a sleeping
+instance. The pointer is `internal/objstore`'s discipline — hand over
+coordinates, never data.
+
+**Why a presigned URL and not a delivered credential.** A long-lived scoped
+read credential means an IAM principal, a rotation story, and a revocation
+story, all of which `internal/mail` and `internal/objstore` had to build. A
+presigned GET generated at delivery time has no lifecycle at all. If it
+expires before the app fetches, a console door re-mints one for a message
+the app can prove it owns.
+
+**`arrival` is load-bearing, not informational.** It is `"forwarded"` at
+stage 1 and `"direct"` at stage 2, and without it the app cannot interpret
+its own verdicts — see §7.
+
+**`headers` is a triage surface, never the source of truth.** SES truncates
+the header block in the notification above 10KB and sets
+`headersTruncated`; the S3 object is always complete. The field is carried
+so the app can decide whether to fetch at all.
+
+---
+
+## 6. Waking a hibernating instance
+
+The deliverer runs **on the box**, polls the deployment's SQS queue, checks
+the target host against its own registry, and delivers through a
+non-stamping wake — `activator.ServeInbound`, a sibling of `ServeScheduled`
+with the identical signature and the identical `stamp=false`.
+
+**Why SQS and not an SNS HTTP subscription.** #371's `ServeScheduled` doc
+comment makes the argument precisely: a tick never leaves the box, so the
+call site *is* the signal, and there is no header for an outsider to send.
+The same must be true here. An HTTP webhook is a public door whose whole
+function is to wake instances — signature verification would be the only
+thing between a stranger and the fleet's wake budget. Polling a queue keeps
+the wake unspoofable by construction, which is the property worth paying a
+poll for.
+
+**Why non-stamping.** #371's finding was that a 5-minute schedule against a
+15-minute idle window means an instance never sleeps, because a 1ms 204
+buys another full window. Overnight inbound has exactly that shape. A
+delivery must be able to *wake* an instance and must not be able to *hold*
+one awake; the app's own subsequent activity is what earns the idle credit.
+
+**Ownership.** One queue per deployment; each box checks whether the named
+host is its own and releases a message that is not (visibility timeout to
+zero) rather than deleting it. Recommended over per-box queues because the
+fleet is small and the registry already answers "is this host mine"
+authoritatively. Flagged in §11 as the decision most likely to want
+revisiting at fleet scale.
+
+**Coalescing.** A batch of messages for one sleeping host is one wake, all
+pending messages delivered on it. Ten messages must not be ten wakes.
+
+**At-least-once.** SQS redelivers, so the POST is not exactly-once and the
+app must be idempotent on `message_id`. This is one of two dedup keys the
+app needs, and they are not the same key — see §9.
+
+---
+
+## 7. Spam, and what SES actually gives us
+
+Verified against the SES developer guide on 2026-09-04
+(`receiving-email-notifications-contents.html`), in the spirit of
+`internal/mail/aws.go`'s tenant-attribution note — the earlier tenant-email
+spec guessed an `X-SES-TENANT-NAME` header that does not exist, and the cost
+of guessing here is the same.
+
+SES performs five checks and reports each as one of exactly four values —
+`PASS`, `FAIL`, `GRAY`, `PROCESSING_FAILED`:
+
+- On the **S3 action**, results are stamped into the stored message as
+ `X-SES-Spam-Verdict` and `X-SES-Virus-Verdict`.
+- On the **SNS notification**, all five arrive as structured JSON on the
+ `receipt` object: `spamVerdict`, `virusVerdict`, `spfVerdict`,
+ `dkimVerdict`, `dmarcVerdict`, each `{status: ...}`.
+
+**There is no numeric score.** Nothing to threshold, nothing to tune. Any
+design that assumes a score is designing for a different provider.
+
+Two shapes to model correctly:
+
+- `dmarcPolicy` (`none` / `quarantine` / `reject`) **appears only when DMARC
+ fails.** Its absence is not "policy: none" — it is "DMARC did not fail".
+ A struct that decodes a missing field to `"none"` reports the opposite of
+ the truth on every passing message.
+- Scanning is per-rule (`ScanEnabled`) and can be off, which is why §3a
+ makes it a platform invariant.
+
+Taking the verdicts from the notification is the useful part of this: the
+platform gets all five **without parsing the message**, so the app can
+triage before fetching a 40MB object.
+
+### 7a. Why the verdicts are nearly worthless at stage 1
+
+Forwarded mail reaches SES *after* the customer's Gmail or Microsoft 365 has
+already filtered it, so the spam verdict is computed on a pre-cleaned
+stream and will read `PASS` almost always. And forwarding breaks SPF by
+construction — the forwarding server is not authorised to send for the
+original sender's domain — so `spfVerdict: FAIL` is the **normal** case on
+this path, usually taking DMARC with it. DKIM generally survives.
+
+So: at stage 1 the verdicts are **provenance to record, not a filter to act
+on**. An implementation that treats `spf: FAIL` as suspicious will silently
+eat legitimate customer mail on day one. That is the failure this section
+exists to prevent, and it is why §5's `arrival` field is mandatory rather
+than nice to have — at stage 2 the same verdicts on the same wire become
+meaningful, and only `arrival` distinguishes the two.
+
+The exception is **virus**. `virusVerdict: FAIL` is a platform-side drop at
+both stages: counted, visible in `carlos inbound status`, never delivered.
+It is the one verdict whose meaning does not depend on the arrival path.
+
+Junk-versus-ticket classification belongs to the app. A `support@` address
+is published on purpose, so genuine spam volume is high and the call is
+product judgement. The platform hands over the raw message and the
+verdicts, caps abuse, and does not decide.
+
+### 7b. Never bounce after accepting
+
+SES accepts, then delivers. An unknown recipient is therefore a **silent
+drop** (§3a's terminal catch-all `Stop` rule), never an outbound bounce: a
+bounce to a forged sender is backscatter against the deployment's own
+reputation, and a `support@`-shaped surface attracts forged senders
+specifically.
+
+---
+
+## 8. Caps, pause, and retention
+
+**Counting.** Per-app, per-UTC-day, the shape `internal/console/mailcounts.go`
+already uses for sending. SES exposes no per-recipient inbound rate limit,
+so a cap can only be enforced by counting what has already arrived — the
+same reasoning `internal/mail/events.go` records for the outbound side.
+
+**On trip, delivery pauses; receiving does not.** Messages continue to land
+in S3 and count; what stops is the wake and the POST. This is the divergence
+§3c predicted, and the reason is that the two failure modes are not
+symmetrical: a tripped outbound cap protects a shared reputation, so it must
+stop the thing itself, while a tripped inbound cap protects wake budget and
+spend — and refusing the mail would lose a customer's email permanently to
+protect a few cents. When the pause clears, the backlog delivers, coalesced.
+
+`PauseKind` follows `mail.Provisioned.PauseKind` exactly: `cap` clears itself
+on the UTC day roll, `rate` and manual (`""`) never auto-resume, and every
+resume clears the field whatever resumed it.
+
+**Retention.** Inbound objects get a bucket lifecycle expiry — **30 days**,
+deployment-configurable. The platform is a delivery path, not the app's
+archive; an app that wants the message kept copies it into its own store on
+receipt. Without an expiry this bucket only grows, and it grows fastest with
+exactly the spam nobody wants to keep.
+
+---
+
+## 9. What "no outbound" buys, and the one bug it creates
+
+Ruling 3 removes real work: no envelope rewriting, no per-correspondent
+reply cap, no auto-reply loop protection at the platform layer, no risk to
+the shared SES sending reputation, and no sub-tenant dimension on the domain
+record (ruling 4). An auto-reply loop becomes a product bug in the app
+rather than a platform incident.
+
+It creates one bug, and it is the one that bites shared inboxes built this
+way:
+
+> The customer **receives** by forwarding from Gmail and **replies** via the
+> Gmail API — so every reply the app sends is forwarded straight back into
+> the inbox it came from, and every thread doubles.
+
+This is the app's to fix, but the platform must make it *fixable*, which is
+why §5's payload carries the full envelope and the threading headers rather
+than a summary. Two distinct dedup keys are needed and they are not
+interchangeable:
+
+- **`message_id`** (SES's) — deduplicates *SQS redelivery* of one message.
+- **`headers.message_id`** (the mail's own RFC 5322 `Message-ID`) —
+ deduplicates *the same mail arriving twice*, including the app's own sent
+ mail arriving back through the forward.
+
+An implementation that dedups on only the first will double every thread; on
+only the second, it will drop a legitimate resend. Worth designing in before
+James's first pilot rather than discovering during it.
+
+---
+
+## 10. Stage 2, sketched only
+
+Direct MX: the customer points `theircompany.com`'s MX at CARLOS.
+
+It belongs in **`internal/domains`**, extending the managed custom-domain
+connection surface (2026-08-27) with an MX-receiving capability and its own
+verification state — *not* in a parallel claim model inside
+`internal/inbound`. That layer already solves "an app's customers connect
+their own domains and the platform reports verification status" for A
+records and certificates; direct-MX inbound is the same problem with a
+different record type and a different verifier.
+
+What stage 2 must confront and stage 1 does not:
+
+- **The apex.** A customer domain may carry an MX for us and an A record
+ elsewhere. §3b's invariant holds only because stage 1's receiving domain
+ is ours alone.
+- **Rule set at N.** §3a's wholesale rewrite is cheap for a handful of
+ platform addresses and needs re-examining against thousands of customer
+ domains — SES's receipt-rule limits are the binding constraint, and a
+ domain-condition rule per customer may have to replace a recipient
+ condition per address.
+- **Verdicts become meaningful.** `arrival: "direct"` flips §7a: SPF and
+ DMARC now say something true, and the app's classification can use them.
+
+Not specced here. Stage 1 ships without any of it.
+
+---
+
+## 11. Open decisions
+
+1. **Queue topology** (§6) — one deployment queue with an ownership check,
+ versus one per box. Recommended: shared, revisit at fleet scale.
+2. **Receiving domain name** — `in.<PlatformSuffix>` versus a dedicated
+ registrable domain. A separate domain isolates receiving reputation and
+ costs a registration; the subdomain costs nothing and is one zone.
+ Recommended: subdomain.
+3. **Address minting UX** — does `carlos inbound enable` mint one address
+ (`<app>@in.<suffix>`) or does the member choose every local part? The
+ claim model (§2b) supports either.
+4. **Default `DailyCap`** — needs a number from the tier model, not from
+ this spec.
+
+---
+
+## 12. Surfaces to build
+
+- `internal/inbound` — records, claim, `Provisioner` seam + AWS
+ implementation, idempotent walks, rule-set renderer and reconciler.
+- `internal/console/apiinbound.go` — the door family, mirroring
+ `apimail.go`'s registration and its `svcrole` classification: the pause
+ door is a POLICY door, and there is no MINT door here at all (§5's
+ presigned URL is why).
+- `internal/activator` — `ServeInbound`, sibling of `ServeScheduled`.
+- `internal/host` — the deliverer's wiring, mirroring `schedule_wiring.go`'s
+ converger/runner split and its build-once discipline.
+- `inbound_cli.go` — `carlos inbound <enable|status|addresses|pause|resume|test>`.
+ `test` sends a real message to a real address and reports what arrived, on
+ the same argument `carlos email test` makes for existing at all: every
+ other verb reports what the record claims, and one verb must prove the
+ path actually works.
+
+## 13. Repo rules this change is bound by
+
+- **`docs/site/cli.md` must name every `carlos inbound` verb** before the
+ gate passes — `internal/docsite` reads main.go's dispatch switch and fails
+ over a command with no heading.
+- **Wire shapes are additive.** §5's payload is a published contract the
+ moment one app parses it.
+- **The consumer list is everything that PARSES a shape**, not everything
+ that calls the API — learned three times, per AGENTS.md.
+- `go build ./...`, `go vet ./...`, `go test ./... -count=1`, `gofmt -l .`
+ clean; the branch pushed to `amadan` and the run read before it is called
+ green.