amadan
Draft shared producer protocol against current Memoria poll behavior
1 file changed,
+83
−0
docs/superpowers/specs/2026-09-07-producer-protocol-proposal.md+83 −0
CI passed — run details
CI log
=== gate === ok amadan.net/oficina/suite 0.008s ok amadan.net/oficina/suite/conformance 0.004s
diff --git a/docs/superpowers/specs/2026-09-07-producer-protocol-proposal.md b/docs/superpowers/specs/2026-09-07-producer-protocol-proposal.md| new file mode 100644 |
| index 0000000..db53c68 |
| --- /dev/null |
| +++ b/docs/superpowers/specs/2026-09-07-producer-protocol-proposal.md |
| @@ -0,0 +1,83 @@ |
| +# Producer protocol proposal |
| + |
| +Status: draft by the overnight Sheets agent, 2026-09-07. Not a ruling, not reviewed, not authorization to implement new app policy. Independent Fable review queued. Existing doctrine is unchanged. |
| + |
| +## Sources and present behavior |
| + |
| +Authority: suite main `1ac4323`, sharing doctrine §3, §4.2–4.3, §6.2, §10 and §12. Authentication, custody, authorization outcomes and retention requirements remain defined there; this proposal supplies missing wire behavior. |
| + |
| +Implementation witnesses: |
| + |
| +- Memoria main `b9e5de9`, `internal/memoria/siblingsync.go`: Poll commits each page cursor only after applying its rows, stops on an empty page with unchanged cursor, and never infers deletion from absence. A listed S3 404 increments Missing and permits cursor advancement. An empty body increments Empty and leaves prior indexed content untouched. Feed returns a generic error for every non-2xx. No reset protocol exists. |
| +- Memoria `sibling/feed.go`: existing bearer header and suite route constants; S3 bytes become IngestSpec.Body directly. Limits are 4 MiB per feed response and 2 MiB plus 64 KiB per content response. The chunker reads text/headings, not JSON or XLSX. Existing source coordinates are sibling name, opaque item ID and revision; a chunk ordinal is not a spreadsheet cell coordinate. |
| +- Docs main `87662f5`, `grants.go` and `links.go`: ownership, reader/writer grants and public read-only capabilities are implemented. Sharing is present. Explicit team-rung state and sibling producer routes are absent. |
| +- Sheets main `4aea8f4`: completed live workbooks are team-readable, but no explicit rung or producer routes exist. Approved isolated `d35cf94` supplies durable tab versions, not a producer-wide item revision. |
| + |
| +## Proposed configuration and authentication |
| + |
| +Keep Memoria's existing MEMORIA_TEAM, MEMORIA_SIBLINGS and MEMORIA_SIBLING_TOKEN_<APP> sender grammar. Receiver configuration uses app-prefixed `<APP>_TEAM` and `<APP>_SUITE_CONSUMERS`, a strict JSON array. Each element has consumer_id, origin, retention_seconds and tokens; each token has epoch and sha256. The drain supplies SHA-256 hex digests of independent 32-byte random raw bearer tokens; raw sender values never enter receiver DB/config. The same consumer_id survives rotation. |
| + |
| +Example shape, with placeholders rather than credentials: |
| + |
| +```json |
| +[{"consumer_id":"memoria","origin":"https://acme.memoria.oficina.works","retention_seconds":86400,"tokens":[{"epoch":1,"sha256":"<64 lowercase hex characters>"}]}] |
| +``` |
| + |
| +86400 is an example, not a default or ruling. Require an explicit positive retention value from the consumer's reviewed configuration; proposed ceiling 30 days bounds storage and must be reviewed alongside the consumer's availability budget. Reject malformed/unknown/duplicate fields, duplicate identities/origins/digests, more than two tokens, invalid epochs and hosts failing CheckSiblingHost. Require exact HTTPS origin with no path/query/userinfo/port override. Missing list disables routes; partially configured input fails boot. Authentication hashes and constant-time compares the sole Authorization bearer value; no token in query/cookie/body. All auth failures use the same empty 401. Successful matching records consumer identity/epoch, not bearer values, for the existing rotation procedure in doctrine §6.2. Do not claim Origin headers independently authenticate callers. |
| + |
| +No shared server implementation or environment parser is proposed before two actual adopters exercise this shape. The suite module owns wire types/decoders/fixtures; each app owns config loading and storage. |
| + |
| +## Proposed S1 continuation and acknowledgement |
| + |
| +Keep Feed JSON unchanged so existing strict DecodeS1 remains usable. One page has at most 256 rows and at most 4 MiB; reject rather than truncate invalid oversized IDs. Proposed ID bound is 256 UTF-8 bytes, without slash, control characters or empty value, covering both numeric app IDs and future opaque IDs. This is a new limit, subject to review. |
| + |
| +A cursor is an unforgeable opaque handle bound to producer, consumer_id, log generation, exact position and expiry. Token rotation does not invalidate it. The producer must reject a different consumer's cursor. Retention is independent per consumer; activity must not extend an old cursor forever. Cursor issuance records or authenticates the anchor and expiry without storing content. A submitted valid cursor implicitly acknowledges that position; no new ACK endpoint. Returning a cursor does not acknowledge it. An old/replayed request cannot move the stored acknowledged position backwards. Lost responses cause replay, never skipped events. |
| + |
| +Empty initial cursor starts a bounded enumeration at a captured boundary. The producer persists only eligible item/revision metadata for that enumeration; it never materializes plaintext content. Items leaving custody during enumeration become gone or fail S3 according to doctrine; later changes remain in the delta log beyond the boundary. Pages replay deterministically during their retained lifetime. After enumeration, continuation moves through the delta log. At the current tip, return the submitted cursor unchanged and rows []; do not mint a fresh cursor each time, because current Poll would loop until maxFeedPages. |
| + |
| +Mutation, item revision and feed event commit together. Feed anchors use a persisted generation plus collision-resistant per-event identity, not sequence alone: after restoring and branching at the same sequence, the old cursor must fail anchor validation. Cursor beyond the tip, missing/pruned anchor, changed generation or expired enumeration requires reset. A restored prefix that still matches an anchor may replay its suffix; it must never be interpreted as proof that unseen newer content remains current. |
| + |
| +## Proposed explicit reset |
| + |
| +For an authenticated invalid/expired/rewound cursor, answer HTTP409 with exactly {"code":"CURSOR_RESET"}. Do not include the rejected cursor or distinguish reasons on the wire. Invalid bearer still answers401 before parsing the cursor. Other409 bodies remain ordinary failures. Existing Memoria safely stops on409 today; it must gain the typed reset decoder before this protocol is usable through expiry. |
| + |
| +On typed reset, Memoria persists a source reset state before clearing its cursor, and withholds that source until a fresh enumeration completes. Proposed implementation deletes the source's indexed documents and derived dependencies using its existing provenance deletion path, in bounded resumable batches. Only after the source purge completes does it request empty cursor. An interrupted purge resumes; a failed HTTP call never initiates deletion; absence from a page never does. The typed reset is an authenticated explicit command to discard that consumer's source corpus, not an enumeration diff. The product cost is temporary loss of otherwise readable source results during rebuild; review must explicitly accept it. |
| + |
| +No second reset endpoint, background ACK write or new successful Feed fields are needed. Rejected alternative: silently return a fresh snapshot for an invalid cursor. Current Poll would retain items absent from the snapshot indefinitely. Rejected alternative: delete on any non-2xx or short page. Current failure discipline must remain intact. |
| + |
| +## S3 body and coordinate handoff |
| + |
| +Two explicit modes are proposed; the producer never emits structured JSON to a consumer that expects raw text. |
| + |
| +1. Existing consumer: absent Accept profile means text/plain; charset=utf-8. Payload is a deterministic app-owned textual projection, never HTML, JSON, XLSX or a serialized editor state. The item coordinate remains (configured sibling, opaque item_id, revision). Memoria treats it as whole-item search, not cell-level citation. Apps must approve exactly what the projection includes before producer code is written. |
| +2. Typed passages: a future client sends Accept: application/vnd.oficina.passages+json;version=1. A producer either returns that exact Content-Type or406; no silent text fallback. Envelope fields are item_id, revision and passages. Each passage has text and coordinate. Coordinate is a tagged union: document_block {kind, block_id}, sheet_range {kind, tab_id, row_ids, column_ids}, or whole_item {kind}. Values are stable app-owned identifiers; no display names, executable expressions or URLs. The parent item_id/revision binds every coordinate. A sheet range is a rectangular product of the named row/column arrays, not a guessed A1 string after edits. Proposed cap:256 passages,64 row IDs and64 column IDs per range, total response at most the existing content byte ceiling. A producer that cannot fit the approved projection returns413, never a successful prefix; large-item partitioning is a separate app design. |
| + |
| +Memoria's typed decoder must verify item/revision equality before ingestion, validate the closed coordinate union and bounds, seal coordinates with stored passage content, and preserve them through retrieval/citation instead of allowing the general chunker to invent new coordinates. A long passage cannot be split unless each resulting chunk keeps the same valid source coordinate. Current Ingest/Chunk has no such field: typed mode requires a real consumer migration and renderer, not just a producer response. No cross-app deep-link template is settled here; each app supplies resolution of its typed coordinate at the cited revision. |
| + |
| +App-owned choices remain open: Docs block projection from current document state; Sheets workbook vs tab item granularity, display/header/unit context, blank cells, formulas, hidden/filter state, comments and projection byte bounds. Existing grants/public links do not establish team custody in Docs; do not ingest them by assumption. Sheets' current all-team behavior cannot be silently converted to new private defaults. |
| + |
| +## Races current Poll must address |
| + |
| +A non-gone event may be superseded before S3 fetch. Producer can return404 for unavailable requested revision, but must have a later committed event for the current revision or departure. Current Poll's Missing advancement is acceptable only with that guarantee. Returning current bytes for an older requested revision is forbidden by doctrine §4.3. |
| + |
| +When a previously indexed item becomes empty, current Poll increments Empty without removing old content. A usable producer integration therefore requires Memoria to retire the old item and its derived data when it receives a successful exact-revision empty projection. That is separate from S3's ambiguous404, which must not be reinterpreted as deletion. Auth and freshness remain request-scoped; an old body retained for retry is never permission to render it. |
| + |
| +## Implementation and review order |
| + |
| +1. Review this proposal with actual Memoria Poll owner and Docs/Sheets owners; decide reset purge cost, config limits and each first projection. No approved doctrine text changes here. |
| +2. Add shared reset/auth/cursor fixtures and an adaptable conformance harness. Harness seeds real app records and accepts app-specific fixture IDs/body expectations; it must not swap the production router for a fake responder. Existing fixture defaults remain additive-compatible. |
| +3. Implement Memoria typed reset and empty-success retirement against a real producer; current raw-text mode can precede typed passages. Keep typed coordinates explicitly unsupported until decoder/storage/citations are implemented. |
| +4. Implement each approved app projection, custody state and atomic revision/feed. Run repository checks and exact-head CI; independent spec/branch review and deployed same-team HTTPS canary precede merge. Existing bds.oncarlos.com canaries fail the settled sibling hostname guard and are not an end-to-end suite transport substitute. |
| + |
| +## Required gates |
| + |
| +| Area | Evidence required | |
| +|---|---| |
| +| Config/auth | Wrong team, duplicate/retired token, two-epoch handover, no-token disabled mode, browser session cannot authenticate S2, bearer cannot write as a person. Logs contain no raw credential or cursor. | |
| +| Feed | Page retry before/after partial application, response loss, independent consumers, empty tip stable cursor, max-page More, mutation rollback, TTL, restore-ahead and equal-sequence divergent anchors, cross-consumer cursor rejection. | |
| +| Reset |409 typed reset withholds then purges/rebuilds; failure during purge/rebuild resumes.401/403/429/500, malformed409, timeout and truncated page do not purge. Downgrade/deletion during reset never reintroduces custody-ineligible data. | |
| +| Freshness | Concurrent mutation between S1/S3, unavailable old revision followed by guaranteed newer event, source-behind S2 UNKNOWN, successful empty current revision retires old body/dependencies,404 does not. | |
| +| Body | Raw mode enters existing parser as text; typed mode cannot enter raw parser; malformed/unknown coordinate, mismatched revision, oversized payload and silent truncation refused. Stable cell/block citation survives unrelated edits. | |
| +| Real adoption | Shared custody fixtures drive real Docs and Sheets routers/storage, including existing grants and links. Memoria indexes, edits/refetches, withholds and revokes against these servers through suite transport. | |
| + |
| +Mutations must demonstrate failure when cursor acknowledgement precedes applied rows, cursor binding/anchor validation is removed, generic failures trigger purge, old content survives successful empty replacement, or JSON is fed to the raw-text chunker. None of the proposed gates is claimed as implemented by this draft. |