oficina / docs Public

Seal document update and snapshot content at rest open

On branch overnight-content-sealing

Overnight Docs track owns content sealing, implementing the existing design section 2. Current main stores document updates and snapshots as plaintext. This branch will add authenticated encryption with a separately provisioned content key, cover reads, sync, snapshots, and legacy conversion, and exercise recovery and raw-database plaintext checks. Existing import and docs-finish branches remain untouched. Spec and whole branch require Claude Fable review before merge; root coordinates deployment and legacy backup handling. Session oficina-overnight-docs-20260907.

Claude Fable (claude-fable-5-1) reviewed the initial spec and approved its first revision. Required corrections: document GUID/sequence/kind/ABI authenticated together; strict separate key validation; activated storage rejects plaintext; key-check row provides constant-time boot validation; conversion and scrub are separate, and checkpoint results must detect busy readers. Historical Litestream backups remain preserved and are an explicit limitation. Root requested a further rollout amendment so a compatibility release is deployed and retained before sealed writes activate; that amendment still needs review. Fable is now session-limited until 20:50 Europe/Dublin, so no merge or deployment is approved. Commit a04acfd contains only the approved cryptographic primitive and tests. Both new tests were mutation-tested: removing AAD returned unauthenticated cross-row plaintext, and accepting the development key made the production-key test fail; fixes restored and focused tests green. Continuing independent backend work within approved invariants.

Independent rollout amendment review: actual model claude-fable-5-1, session 8aa40554-38b1-4597-9cb8-439ab594e268. Conditional verdict; blocker is being corrected in a follow-up spec, not treated as unqualified approval.

APPROVE with one blocker and four pins.

Blocker: a server process that booted in legacy mode keeps writing plaintext after activation. Mode is decided once at boot from the marker. The spec relies on the operator stopping the server, but nothing in the design defends against a process that was not stopped, or one the platform restarts during the maintenance window. Two cheap corrections, both required:

  • The legacy-mode write path re-reads the marker inside the seq transaction (store.go:841 closure) and refuses the write if it has appeared. One indexed row read per write, only in legacy mode.
  • Activated-mode boot refuses to start if any row has encoding 0, with a message naming the maintenance command. The conversion must therefore be rerunnable on an already-activated database, converting only the remaining encoding-0 rows and leaving the marker untouched. Per-request "reject plaintext rows" is a fallback, not the primary defence; a mixed database should fail at boot, not as scattered 500s.

Pins:

  1. Key required in production in both modes. Legacy mode validates DOCS_CONTENT_KEY and computes the fingerprint even though it does not use it for content. That is what the smoke test of the compatibility release proves: the secret plumbing works before activation depends on it. An instance that cannot load its key must not be activatable.
  1. No deactivate verb. docs-seal has no command that removes or rewrites the marker. "Cannot downgrade" must be true by absence of code, and a test should assert the marker survives every maintenance operation including a failed one.
  1. Marker present with fingerprint mismatch fails boot with a distinct message ("key does not match this database"), separate from the AEAD failure message, so an operator can tell a wrong secret from a corrupt row.
  1. Legacy-mode diagnostic is a boot-time Error log line, not a UI element, and it fires on every boot. Hibernation makes boots frequent, so it will be seen. No new user-facing language, per scope.

One observation for the whole-branch review. The branch already carries commits titled "Prepare atomic content conversion and explicit database scrub" and "Prepare maintenance entry point". If any of that code performs conversion on server boot, it now contradicts this amendment and must be removed rather than left as a dead path. The branch review should read for that specifically.

Existing test list stands. Add: legacy-mode write refused once marker appears; activated boot refused on a mixed database; conversion rerun on an activated database converts only the remaining rows.

Independent final rollout amendment verdict, actual model claude-fable-5-1, session 8aa40554-38b1-4597-9cb8-439ab594e268.

APPROVE. The amended spec closes the blocker and honours all pins. Two implementation notes, neither a condition on approval:

  1. The in-transaction marker re-read must run on the writer connection inside the same gorm.DB.Transaction closure as the insert. dbresolver routes a bare SELECT to the reader pool, which would read a stale snapshot outside the write lock. Reading through tx guarantees the check and the insert share one connection and one transaction, which is the whole point of the check.
  1. The activated-boot "any content_encoding != 1" scan is a full-table read of two tables on every boot. Use SELECT EXISTS(... WHERE content_encoding != 1 LIMIT 1) against both tables so it is index-assisted or at worst a fast scan that stops at the first hit. Consider a partial index on content_encoding if boot latency on large instances becomes visible.

The whole-branch review will check that the prepared helpers are reachable only from docs-seal, that the legacy diagnostic fires in tests on an unactivated database, and that each new test was seen to fail for the right reason. Proceed with runtime wiring.

Accepted amendment: Accepting your blocker and pins, with implementation before any deploy:

  • Both legacy update insertion and legacy snapshot insertion re-read activation marker inside their existing write transaction and refuse if present. This covers a stale preactivation process racing offline conversion; no plaintext row can commit after marker activation. Cipher mode remains fixed per process and cannot auto-switch.
  • Production validates configured content key before DB opening in BOTH modes. Development alone may use fixed dev key.
  • Activated startup verifies marker keycheck then checks for any content_encoding !=1 and refuses before serving. Fingerprint mismatch is a distinct fixed operator error, separate from AEAD failure. No content bytes in errors.
  • Explicit maintenance only (never startup) rerun on activated DB verifies all already-sealed rows, converts remaining encoding0 rows atomically and rejects unknown encodings; keeps original marker/keycheck/fingerprint unchanged even on failure. Marker remains unchanged after scrub except its scrubbed bookkeeping bit. Tests pin these properties. This is an explicit maintenance repair, never runtime downgrade. Mixed-state boot error identifies the docs-seal command.
  • Legacy diagnostic is Error at every boot, with structured encoding and fingerprint fields, no UI strings. No deactivate verb or marker-removal code. docs-seal uses only explicit DB path, production content key env and expected fingerprint guard; it never binds a listener or starts replication. Operator stops server/replicator; the transaction check additionally prevents stale legacy writers from violating activation. The prepared conversion helpers are not called anywhere in server startup. Please confirm APPROVE of this amended spec before runtime wiring. <=200 words. Same independent model identity and scope as prior review.

Runtime integration pushed at 094accf on overnight-content-sealing after explicit Fable spec APPROVE. Both Go packages pass (internal/docs46s; cmd/docs passed). Separate production content key required before DB opening, even in legacy mode; boot logs Error when not activated; activated startup verifies marker/key and refuses mixed encodings. Legacy update and snapshot inserts check marker through tx before writing. All update/snapshot persistence boundaries seal/decrypt, including sync GET before response headers. Explicit docs-seal subcommand takes DB and expected fingerprint, key only from DOCSCONTENTKEY, never starts server/replication. Conversion reruns can repair encoding0 atomically without rewriting keycheck; repair clears scrubbed bookkeeping when needed. No deactivate path.

Eight runtime mutations failed for intended behavior after restoring each: removing update sealing; snapshot sealing; sync read decryption; stale legacy guard; mixed startup check; active repair; Error diagnostic; CLI dispatch. A first sync mutation failed compilation and was discarded; corrected mutation produced the expected behavioral failure. Raw evidence /tmp/oficina-docs-runtime-mutations.txt and /tmp/oficina-docs-runtime-mutations-followup.txt. Added real HTTP roundtrip/tamper refusal, stale Store.ApplyUpdate/snapshot refusal, mixed boot and repair/key mismatch/deleted marker, failed repair marker/row rollback, production key requirement and explicit CLI tests. Prior real process-exit, WAL/scrub and restored-copy tests retained. Whole-branch Fable review requested from root queue; exact-head remote CI pending. No deploy yet.

Exact-head remote CI PASSED at094accf. Three additional runtime mutation proofs also failed as intended: bypassing production key validation, bypassing explicit CLI input checks, and removing conversion transaction causes the failed-repair test to detect partially converted rows. Restored files exactly; worktree clean. Total11 runtime mutations plus earlier crypto/migration/recovery mutations. Extra evidence /tmp/oficina-docs-runtime-extra-mutations.txt. Awaiting allocated whole-branch review.

Independent whole-branch review, actual model claude-fable-5-1, session 864a6877-4e1e-4c69-ba9f-bbdf98f53cb3. The reviewer explicitly lacked git diff; complete diff follow-up is being prepared, and both nonblocking medium test gaps are being closed before final candidate.

Verdict: APPROVE. No blocking defects found. Four non-blocking findings below, two of them test gaps worth closing before or immediately after merge.

Reviewer identity and scope. Model: Claude Fable 5.1 (claude-fable-5-1). Read-only review of the worktree at overnight-content-sealing. I had no shell, so I could not run git diff origin/main or the tests. I read every sealing file end to end, the modified persistence files in full, the updates handler region, migration 0007, the test harness, all six new test files, rastrillo v0.22.0's db package, and the four spec transcripts plus the runtime mutation evidence. Files I did not read may carry changes I have not seen. The coordinator's final CI evidence is the authority on green.

Spec pin compliance, verified by reading.

  • Key validated before the database opens in both modes: internal/docs/app.go:365 precedes app.go:370.
  • Marker re-read inside the writer transaction: content_runtime.go:24 runs on tx, called from store.go:861 and snapshots.go:112. SQLite WAL snapshot isolation also fails a deferred transaction's INSERT with BUSY_SNAPSHOT if the marker committed after the read, so the race closes either way.
  • Activated boot verifies keycheck then EXISTS scans on both tables: content_migration.go:26, content_runtime.go:13. Fingerprint mismatch, AEAD failure, mixed state and missing marker are four distinct errors.
  • Repair leaves marker, fingerprint and keycheck untouched; only the scrubbed bit moves: content_migration.go:78.
  • Scrub validates both checkpoint tuples, VACUUMs on a pinned writer connection, marks scrubbed only on success: content_scrub.go:11.
  • docs-seal never reaches rastrillo.Resolve, New, or a listener: cmd/docs/main.go:138. Stat check refuses to create a database. Dev key refused because the cipher is built as production.
  • No deactivate verb, no marker deletion outside test code. Error text carries row ids only, never bytes.
  • Legacy diagnostic at Error each boot with structured fields: app.go:389.

Findings.

  1. Medium, untested fail-closed branch. content_runtime.go:51 and :61 refuse encoding-1 rows in a legacy-mode Store. No test exercises a stale legacy process reading sealed rows. Deleting those two checks stays green, and a stale legacy server would then serve ciphertext with a 200 and a Docs-Seq. Not a confidentiality leak, but exactly the shape AGENTS.md warns about.
  1. Medium, half-tested startup check. TestContentStartupRejectsMixedState... inserts only a legacy update row. The snapshot clause of requireSealedContent (content_runtime.go:15) has no test; dropping it stays green and one plaintext snapshot could sit undetected at rest until the next docs-seal rerun.
  1. Low, operator diagnostics. MaintainContent returns the same errContentKey for a malformed key and for an expected-fingerprint mismatch (content_maintenance.go:26). An orphan row fails as content authentication failed (content_cipher.go:63 via empty GUID), which reads as a wrong key. docs-seal failures are logged under the label docs: new (main.go:140). Nothing prints the configured key's fingerprint except the legacy boot log.
  1. Low, stale claims in the repo. store.go:578-582 still says content is plaintext and sealing is out of scope. PRODUCT.md:70-76 says no plaintext-grep test exists; TestContentScrubRemovesLegacyBytesFromDatabaseAndWAL is now that test. The approved spec lives only in /tmp transcripts, while AGENTS.md says specs belong in docs/superpowers/specs/. No operator runbook for the phased rollout is in the repo.

Observations, no action required. Every existing test runs in legacy mode because NewForTest never activates. Activated-mode coverage is one HTTP test plus the boot and repair tests. The mutation record shows each activated seam was seen red, so this is adequate, but a harness switch to run the suite activated would be cheap insurance. Random 96-bit GCM nonces are fine at this write volume; key rotation is out of scope as agreed. run-dev.sh will log the legacy Error on every boot since docs-seal refuses the dev key by design.

Complete-diff independent review, actual claude-fable-5-1, session 864a6877-4e1e-4c69-ba9f-bbdf98f53cb3. Exact-head remote CI independently verified PASSED for6d6f6a5 immediately after this verdict. No merge/deploy yet; root coordinates release integration, canary, key and activation.

APPROVE for the complete diff at HEAD (6d6f6a5), subject to the exact-head CI gate passing.

Reviewer identity and scope. Claude Fable 5.1 (claude-fable-5-1). I read all 1793 lines of /tmp/oficina-docs-sealing-complete.diff (git diff origin/main...HEAD), both new mutation records, and had previously read every touched source and test file in the worktree plus rastrillo v0.22.0's db package. The diff touches exactly the files I expected: PRODUCT.md, the new spec, cmd/docs/main.go and its maintenance entry, app.go, models.go, store.go, snapshots.go, handlers.go, migration 0007, and the six content_* source and test files. Nothing else changed. No UI strings, no bundle, no go.mod change.

Both Medium findings from my first pass are closed.

  • TestStaleLegacyStoreRefusesSealedUpdatesAndSnapshots exercises the real stale path: an activated database, then store.content = nil, then newestSnapshot must return errContentEncoding and the raw-updates GET must 500 with no Docs-Seq. The mutation record shows each guard removed separately reddened the matching assertion, not a status side effect.
  • TestContentStartupRejectsPlaintextSnapshotWithoutLegacyUpdates covers the snapshot half of requireSealedContent. Removing that clause reddened it on boot returning nil.

Doc corrections verified. The false PRODUCT.md claim that no plaintext-grep test exists is gone while the "do not describe as encrypted at rest" instruction remains, which is correct given titles and historical backups stay plaintext. Stale out-of-scope comments in store.go and models.go are removed. The approved spec is now in docs/superpowers/specs/2026-09-07-content-sealing.md and matches the implementation I reviewed.

Remaining Low items, non-blocking, unchanged from before. MaintainContent returns the same errContentKey for a malformed key and an expected-fingerprint mismatch. An orphan row surfaces as content authentication failed rather than naming the missing document. docs-seal failures log under the label docs: new. Nothing prints the configured key's fingerprint except the legacy boot log.

Not verified by me. I did not run the tests or CI. Root's exact-head gate remains the required green before merge, per AGENTS.md's shipping cadence.

Rollout prerequisite from current platform source (fresh origin/main df7d1d25dd2d069ed77a6ed11996a31baae079ce): compatibility deployment is supported with an independent strict-base64 32-byte DOCSCONTENTKEY in the app environment. Existing production-style origin/auth/mailer/admission settings remain needed. There is no sealing mode env/flag; activation marker determines mode. A new empty DB boots legacy mode, emits Error with content_encoding0 and key fingerprint, and does not activate. A missing production content key refuses before opening DB.

Existing-db activation command is the binary subcommand docs docs-seal -db EXISTINGDB -expect-fingerprint SHA256HEX [-scrub], with key only in DOCSCONTENTKEY, executed while server and replication are stopped. It intentionally refuses a missing DB. No production command has been run.

Inspected platform member CLI does not expose an app maintenance/exec operation. Host-scoped restart stops an exec-child, then the next request respawns it; that is not an exclusive maintenance fence. internal/activator/activator.go:777 runs Seed only when Restore reports fresh (no local DB and no replica); backend_exec.go appends --db and supplies merged app env before server/replication. Seed cannot run over an existing production database, and docs-seal cannot directly serve as a fresh seed because its database must already exist. No box shell access is presumed or requested.

Consequently production activation needs a supported platform-owned fenced maintenance execution path, with compatible rollback and restore/scrub evidence. A compatibility-only release can still deploy after canary; its success must not be described as content already sealed. A sealed empty-db canary also needs an explicitly prepared database/init path rather than an environment toggle. Root coordinates this dependency and pending bootstrap-fixture design. Backend candidate remains complete-diff Fable-approved/CI-green at6d6f6a5, pending latest-main integration.

Key-preserving compatibility rollout plan (preactivation only).

  1. Pin and retain the exact current production release after label landing as the preactivation rollback target, plus its process /api/version evidence. Keep the candidate compatibility release separately pinned. Do not prune either release or historical backups.
  2. Provision one new independent random32-byte DOCSCONTENTKEY per relevant instance environment, via the supported sealed-secret path. Canary and production keys are distinct. Preserve all existing origin, auth instance key, SMTP and admission values. Do not replace the environment wholesale; record only the new key's public SHA256 fingerprint. Preserve its sealed backup/recovery reference before deployment. Never put key bytes in argv, Amadan, screenshots or logs.
  3. Adding that unused environment name has no content/schema/auth effect on the precompatibility process: its code has no reader for DOCSCONTENTKEY. Keep its existing signing/session settings unchanged. Verify config delivery independently; an adopted env file or promoted pointer is not proof about the running process.
  4. Deploy the reviewed compatibility binary, use the supported host-scoped restart, and verify its real /api/version (not only X-Carlos-Version). Verify boot's legacy Error diagnostic/fingerprint, admission/session behavior and existing document read/write through authorized controlled fixtures when available. Empty-db canary proves key loading and legacy boot only, not sealed activation. No docs-seal command or activation request is made in this phase.
  5. If the compatibility candidate fails before any activation: return the exact production host to the retained precompatibility release, issue scoped restart and verify its process revision and prior behavior. Leave DOCSCONTENTKEY provisioned and preserved; the older process ignores it. Leave additive schema columns/table in place; old writes omit the new fields and retain encoding0 defaults. Do not restore a stale DB backup just to roll back code, do not delete marker/table/columns, and do not rotate or remove either auth or content keys. Restoring old environment wholesale would risk dropping the new content key and is not the rollback method.
  6. This old-binary rollback is valid ONLY while the database remains unactivated and no sealed rows have been written. The deployment record must explicitly state that no maintenance activation was performed. Once a supported maintenance operation exists and is planned, retire the plaintext-only binary as a rollback target BEFORE activation; only the smoke-tested compatibility-capable release may be used after it. A failed or ambiguous maintenance receipt keeps the host fenced until the state is verified.
  7. Key loss after activation is a restore/recovery incident, not an excuse to generate a replacement key or remove the marker. Recover the same key from its preserved secret backup; verify fingerprint and a restored-copy rehearsal. Any restored DB must be explicitly scrubbed through the fenced maintenance path. Historical plaintext backups remain preserved and outside the sealing claim.

Current concrete blocker is carlos/platform discussion117: supported fenced existing-DB maintenance. Therefore the next deploy is compatibility-only. Root owns actual release ids, secret provisioning receipts, scoped restart/process evidence and final canary results; this plan does not claim those steps have already completed.

Integrated release stamping and sign-in label main3d5dfad by clean rebase; new candidate df7db788ab2dfc70e1c7ec8a12e04452f9f5dce8. git range-diff499d200..6d6f6a5 versus3d5dfad..df7db78 marks all six commits = (no feature patch changed). Full local .amadan/ci.d/10-gate PASSED, including Rust/JS/wire/Go checks; exact-head remote CI running. Process-crash, pinned-WAL and restored-copy coverage is unchanged and included in that gate.

Disposable actual-code rollback rehearsal PASSED: compiled a small fixture driver separately against candidate df7db78 and precompatibility main3d5dfad source archives. Candidate creates257 legacy updates and a snapshot; old code reopens, checks exact fixture text and appends; candidate reopens, checks text and appends. Final same DB has259 encoding0 updates, legacy snapshot(s), no activation marker. Same independent key env retained throughout; no auth key changes, no DB restore or schema deletion. Source fixture is synthetic, not private data. Raw /tmp/oficina-docs-rollback-rehearsal.txt; driver /tmp/oficina-docs-rollback-rehearsal.py. This substantiates preactivation code rollback only, not postactivation plaintext-binary compatibility. Root handles actual canary/provisioning/deploy receipts.

Compatibility canary deployed and verified at exact df7db788ab2dfc70e1c7ec8a12e04452f9f5dce8 on overnight-docs.bds.oncarlos.com. Added independent random32-byte DOCSCONTENTKEY only to isolated overnight-release using supported sealed-secret API; existing auth keys unchanged. Supported host-scoped restart followed promotion. Direct process /api/version exact, signin HTTP200. Operator app log reports expected legacy compatibility Error contentencoding=0 and public key fingerprint9cee95052520a25ded6a2e046d1a926ba55c4a195b35c08702701d73e01f8626. No activation performed and this is NOT remote sealed-data evidence. Full branch/diff reviews and exact CI passed; local prior-source -> candidate rollback rehearsal recorded by docstrack. Production compatibility provisioning will preserve current3d5dfad rollback binary and all old keys; platform117 remains activation prerequisite.

Preactivation production prerequisite provisioned: default secret listing first confirmed DOCSCONTENTKEY absent; one independent production key backed up mode0600 at /home/paulca/oficina-docs-content-key.json and set through the supported sealed-secret API. Public fingerprintd091c6b56c29abf527fc88cea16520f180997e4b175d6ec50012174c9d030f7e; distinct from canary. Existing DOCSINSTANCEKEY and SMTP secret preserved. Current supported CLI only accepts secret argv, so a disposable local stdin/file-input bridge invokes its unchanged runSecretsSet implementation in-process; key bytes were not placed in shell commands or OS process argv. No secret values logged or posted. Retain precompatibility3d5dfad release; no activation or schema rollback attempted. Merging the reviewed compatibility release now, rebuilding exact merged main, then verifying legacy-only startup.

Compatibility-only release now LIVE: main87662f5b246004aca604a4b040cb6691cfd7f130 actualCI passed and exact archive-built ARM64 artifact promoted edge. Direct process /api/version now confirms87662f5b246004aca604a4b040cb6691cfd7f130 after scoped restarts; initial restart preceded convergence and still woke old3d5dfad, so root retained honest failure status until fresh process proof. No docs-seal operation performed. Preactivation rollback target3d5dfad remains available; distinct production content key preserved outside repo and through supported sealed-secret service. Platform117 still blocks activation; this release is migration/encoding compatibility groundwork, not a claim that hosted document content is encrypted.