| 1 | # K3 offline durability evidence |
| 2 | |
| 3 | Continuation of native WIP 68a51d7, against reviewed spec 6049830 (§3.3–3.8, |
| 4 | §4.3–4.4 and K3). Work/review records: rastrillo/native #9 and oficina/docs #69. |
| 5 | Nothing has been merged or deployed. |
| 6 | |
| 7 | ## Implementation and review |
| 8 | |
| 9 | - Account-owned sealed outbox, metadata and replicas; Home activates replay. |
| 10 | - CommandsApplied is captured with snapshot/sequence. CommandsDurable advances |
| 11 | only after enqueue; the writer requires equality again immediately at commit. |
| 12 | - Reopening captures the entire lane before reading its replica, applies update |
| 13 | entries in order, and publishes only after replay. Held updates remain visible. |
| 14 | - Editor and headless senders merge with the latest durable replica using a |
| 15 | compare/retry. A shared generation invalidates writers on discard. Sent updates |
| 16 | are cached before ack, so Home replay preserves subsequent offline reopening. |
| 17 | - Metadata keys are registered before metadata writes. Attached identity is |
| 18 | persisted before its repairable sqid alias; sealed paths never move. Creation |
| 19 | tokens survive failed attachment writes and retries remain idempotent. |
| 20 | - Metadata-only outbox markers give titles/drafts the existing drainer's retries. |
| 21 | Activation reconciles the metadata/enqueue crash boundary. A newer rename gets |
| 22 | a successor marker behind an in-flight marker. Failed local title writes and |
| 23 | the draft's initial paragraph are retained for unlock/retry. |
| 24 | - Account-level localized lost-content notice remains until dismissal. Opening a |
| 25 | document loads the outbox before touching the index, preserving key-loss notice. |
| 26 | - Byte-cap pause is recomputed after queue changes and reopening; retry cannot |
| 27 | bypass it. Kit status preserves known state while storage is unavailable and |
| 28 | applies the spec's session-ended priority even for an empty outbox. |
| 29 | - Highest-capability branch review: gpt-6-astra. Its metadata, cache, loss-notice, |
| 30 | cap and draft-seed findings were addressed with regressions. K4/K5/K6 features |
| 31 | remain in their specified slices; no merge approval is implied by code review. |
| 32 | |
| 33 | ## Failing-first evidence |
| 34 | |
| 35 | All mutation controls below were restored before commits. Logs are local /tmp |
| 36 | transcripts; this table records their observed assertions rather than merely |
| 37 | claiming that a command failed. |
| 38 | |
| 39 | | Probe | Failing control / observed failure | Passing evidence | |
| 40 | |---|---|---| |
| 41 | | Original watermark, lane replay, generated history | DocsPersistence/snapshot stubs: 3 tests, 238 assertions (`k3-docs-red.log`) | `k3-docs-green.log` | |
| 42 | | Replace crash boundaries and model reopen | Omit lane replay: 2 tests, 5 missing-text assertions (`k3-docs-integration-red.log`) | `k3-docs-integration-green.log` | |
| 43 | | Unlock resumes queued commands | Existing defect: 3 assertions (`k3-unlock-red.log`) | Same integration green suite | |
| 44 | | Title before replica, title retry, concurrent rename, attachment alias, Home replay | Prior implementation: 5 tests, 15 assertions (`k3-recovery-red.log`) | 15 durability/account tests (`k3-recovery-green.log`) | |
| 45 | | Concurrent cache writes and discard ordering | Omit merge/invalidation; delete lane first: 2 tests, 6 assertions (`k3-cache-discard-red.log`) | 13 durability tests (`k3-cache-discard-green.log`) | |
| 46 | | Locked title, loss notice, cap, live fake stream | Prior behavior / nil loss-notice stub: 4 tests fail (`k3-review-red.log`; the fifth timing probe initially missed its race and was corrected below) | 24 durability/account tests (`k3-review-green.log`) | |
| 47 | | Rename after title POST, before batch ack | Assert both entries reserved; old marker suppresses successor: First != Successor (`k3-trigger-valid-red.log`) | Same 24-test green suite | |
| 48 | | Draft attachment across store operations | Stub attachment sender; alias, token, content, count and empty-lane assertions fail (`k3-attachment-trigger-red.log`) | Same green suite, exhaustive sweep through actual operation count | |
| 49 | | Direct sealed-key loss before index read | Prior code opens index first: no projection/no loss notice (`k3-attachment-trigger-red.log`) | Same green suite | |
| 50 | | Draft paragraph survives metadata failure | Omit retained seed: server text empty instead of Recovered draft (`k3-seed-cap-red.log`) | Both tests pass (`k3-seed-cap-green.log`) | |
| 51 | | Reopen cannot bypass lane cap | Prior unconditional input reset permits editing (`k3-seed-cap-red.log`) | Same green log | |
| 52 | | Parked update snapshot | Replace only snapshot's encode_state_as_update_v1 with encode_diff_v1: healed text missing (`k3-parked-red.log`) | Existing snapshot passes native/crdt test (`k3-parked-green.log`) | |
| 53 | |
| 54 | The generated histories cover 12 deterministic seeds × 60 events. They place |
| 55 | writer ticks between command and enqueue, inject crashes before arbitrary event |
| 56 | store operations, reserve overlapping compaction candidates, integrate then lose |
| 57 | responses, reopen and replay, and compare I1a/I1b and final server convergence. |
| 58 | Separate sweeps cover every replacement/delete boundary, every discard boundary, |
| 59 | and every actual store operation in draft attachment. The fake uses production |
| 60 | WASM replicas, serves since-cursors and publishes subsequent stream sequences. |
| 61 | |
| 62 | ## Real SIGKILL and draft integration |
| 63 | |
| 64 | `native/scripts/test-offline-kill.py` authenticates against actual Go handlers, |
| 65 | opens the native model, verifies initial save, types 12 durable edits, and sends |
| 66 | SIGKILL before the two-second replica writer. Recovery starts offline and checks |
| 67 | exact text before reconnecting and comparing the real server's snapshot. |
| 68 | |
| 69 | The original fixture configured localhost:8080 despite using a random loopback |
| 70 | port. A NativeHTTP-style Origin POST reproduced 403 (`k3-origin-red.log`). It now |
| 71 | uses the listener's actual origin; CSRF is not bypassed. Failed initial-save or |
| 72 | pre-kill typing assertions stop the harness. |
| 73 | |
| 74 | Negative control: omit only lane replay. Initial save and server convergence |
| 75 | still pass, but offline reopen is empty instead of the 12 edits: exactly one |
| 76 | assertion (`k3-kill-valid-red.log`). Restore replay: exact recovery/convergence, |
| 77 | no duplicates (`k3-kill-valid-green.log`). Final-code rerun passed again |
| 78 | (`/tmp/k3-kill-final.log`). |
| 79 | |
| 80 | Existing `test-document-drafts.py` also passed (`k3-draft-integration-green.log`). |
| 81 | Its detail GET used to return Untitled permanently, even after an accepted rename; |
| 82 | it now snapshots the current title at request start, preserving realistic delayed |
| 83 | stale replies. The unchanged title assertion failed before this fixture correction. |
| 84 | |
| 85 | ## Spec differences and slice boundaries |
| 86 | |
| 87 | - Discard removes the replica before the lane. The user accepted this ordering |
| 88 | with crash proof; the exhaustive sweep shows no stranded unsent replica text. |
| 89 | - The index also registers immutable local keys, and the app uses metadata-kind |
| 90 | outbox markers. These minimal additions make metadata-only replay discoverable |
| 91 | after a crash; they introduce no kit update/retarget API. |
| 92 | - Kit load/entries/beginSend/endSend express unavailable storage, held-entry |
| 93 | replay and atomic send reservations. Their rationale is recorded on native #9. |
| 94 | - Equality is used for the durability watermark as explicitly requested by the |
| 95 | user (the spec allows <=). Existing snapshot is used; no new CRDT operation. |
| 96 | - K4 offline library/draft presentation, K5 reauthentication/wipe, and K6 terminal |
| 97 | resolution, first-400 handling, relaxed close and iOS background task remain |
| 98 | outside K3. Existing close/signout protection remains in place. |
| 99 | |
| 100 | ## Inherited WIP problems and gates |
| 101 | |
| 102 | The inherited kit was mostly no-op stubs. Tests included an unchecked empty-array |
| 103 | index and an undeclared AppleTests → TestSupport dependency hidden by incremental |
| 104 | builds. Further implementation checks caught rotation's /var versus /private path |
| 105 | handling, retrying-lane starvation, and early idle from concurrent drain calls. |
| 106 | Their red/green evidence is in the kit's docs/testing/k3-evidence.md. |
| 107 | |
| 108 | Final required gates are serial: native make ci; Docs .amadan/ci.d/10-gate; Docs |
| 109 | native/scripts/check-native.sh. Use rustup cargo first, TMPDIR=/tmp, xcodegen before |
| 110 | xcodebuild. Final head/results are recorded on discussions #9/#69, with logs |
| 111 | `/tmp/k3-kit-release-ci.log`, `/tmp/k3-docs-final-gate.log`, and |
| 112 | `/tmp/k3-docs-final-native.log`. Native hub CI lacks a macOS runner and fails on |
| 113 | Sparkle extraction (missing unzip); the user directed reliance on local make ci. |
| 114 | |
| 115 | Disk is measured on /System/Volumes/Data, with a mandatory stop below 1.5 GiB. |
| 116 | Only outputs created by this session are removed. The kit's inherited .build is |
| 117 | moved aside for full gates, preserved, then restored; Docs targets, framework, |
| 118 | .build and DerivedData created here are removed between stages. |
| 119 | |
| 120 | |
| 121 | ## Fable review follow-up (2026-09-24) |
| 122 | |
| 123 | - Missing, malformed-JSON, and tampered sealed metadata now hold only that lane with `rejected("metadata")`; genuine unavailable I/O still throws. The new test failed all three hold assertions against the inherited implementation, then passed while another document drained. |
| 124 | - Held metadata markers are deduplicated using `entries(lane:excludingInFlight:)`. The new repeated-flush test first found five markers instead of one; it now passes, as does the existing rename-after-title-POST/in-flight-successor regression. |
| 125 | - A durable offline lane uses SyncStatus without setting the generic errorKey. The new test first observed `native.pending` rather than nil, and now observes nil plus `.offline(pending: 1)`. |
| 126 | - All 24 durability tests passed. Logs: `/tmp/k3-docs-review-{red,green}.log` (ephemeral); the red run had five expected assertions across three tests. The existing key-loss test was adjusted to create a fresh sealed store on simulated process restart: reusing the old instance now intentionally retains its cached key. |
| 127 | - Native pins (all five) advance to `0ff1de9d542ce96430e158708e62be04d87d65a4`; native final-head local `make ci` passed (119 Swift tests and remaining platform/Go/Python gates). Its hub runner still fails before tests on missing unzip / absent macOS infrastructure. |
| 128 | - Deferred as directed: review finding 4 (WASM decode per writer tick) belongs to K4/K7; finding 6 (sessionRestored on existing-account/window access) belongs to K5. No new K3 spec departure. |
| 129 | |
| 130 | |
| 131 | Random SIGKILL mode: `native/scripts/test-offline-kill.py --random --seed 17` |
| 132 | leaves typing unpaused and runs concurrent replica writer ticks. The controller |
| 133 | kills after a seeded random delay, without a writer-side sleep/handshake. Recovery |
| 134 | must contain the last atomically recorded durable text as a prefix, contain exactly |
| 135 | the ordered k3 tokens (no gaps/duplicates/torn edits), then equal the real server. |
| 136 | Failing-first control: stub the outbox write in DocsPersistence.enqueue while still |
| 137 | advancing commandsDurable. Seed 17 killed at 0.260992 s; recovery was empty and |
| 138 | failed the durable-prefix and ordered-token assertions (two failures). Restore the |
| 139 | write: the same seed passed offline recovery and exact server convergence. Logs: |
| 140 | `/tmp/k3-docs-review-kill-{red,green}.log`. The stub is not committed. |
| 141 | |
| 142 | Final-head gates and repeated kill runs are reported on #69; local logs use |
| 143 | `/tmp/k3-docs-review-final-{gate,native,kill*}.log`. Build stages run serially, |
| 144 | with rustup cargo first and TMPDIR=/tmp; xcodegen runs before xcodebuild. |
| 145 | |
| 146 | |
| 147 | At 8f25e2f the full local Docs gates passed (82 XCTest cases, four fixture skips, |
| 148 | 16 Swift Testing cases, parked-update and wire checks); deterministic SIGKILL and |
| 149 | random seeds 1/17/2 all passed. Final harness review moved the existing writer-log |
| 150 | assertion check after process termination so random typing cannot emit an assertion |
| 151 | between the check and SIGKILL without failing the script. Final-head reruns and |
| 152 | cleanup are recorded on #69. |
| 153 | |
| 154 | Landing prep (2026-09-30): the native half merged, so all five native pins move |
| 155 | from the K3 branch head `0ff1de9d542ce96430e158708e62be04d87d65a4` to rastrillo/native |
| 156 | main `529f356f476e43174882361511032e0bdefa5317`, after merging fresh Docs origin/main. |
| 157 | Gate results for this head are recorded on #69. |
| 158 | Hub CI at de13d64 failed in the gate's wire-compatibility stage ("JS reads what |
| 159 | Rust wrote": `verify-rust: empty update on stdin`). The runner log shows cargo |
| 160 | stopping partway through replaying cached yrs warnings (40 of 67), then `error`, |
| 161 | before emit_update ran. K3 does not touch core/, native/vendor/, scripts/ or |
| 162 | fixtures/. A fresh clone of de13d64 passed `scripts/wire-gate.sh` locally, and |
| 163 | the JS-reads-Rust pipe passed five times in a row. |
| 164 | |
amadan