| 1 | # 🤖 Native performance instrumentation (iOS + Mac) |
| 2 | |
| 3 | The answer to "the app has lots of little cuts": both native apps carry a |
| 4 | small, always-there instrumentation layer (`ios/LChatCore/Sources/LChatCore/ |
| 5 | Perf.swift`) so a slow moment on any device leaves a record — without a |
| 6 | third-party APM SDK, and without anything leaving the device. Streaming |
| 7 | member telemetry to Datadog/Sentry/Firebase would sit badly with the E2EE |
| 8 | posture (and add the kind of dependency the second rule exists to avoid), so |
| 9 | the stack is Apple-native + self-owned end to end. |
| 10 | |
| 11 | ## The three layers |
| 12 | |
| 13 | 1. **Signposts (always on, every build).** `os_signpost` intervals around |
| 14 | the seams that hurt, via `Perf.begin("name", slowMs:)` → `span.end()`. |
| 15 | Cost is nanoseconds; nothing is stored or sent. Attach Instruments to any |
| 16 | build (subsystem = the bundle id, category `perf`) and the seams below |
| 17 | appear as named intervals; hangs and first-frame land in the |
| 18 | Points of Interest track. |
| 19 | |
| 20 | 2. **The slow-event ledger (consent-gated).** A span that beats its budget, |
| 21 | a main-thread hang (`HangWatchdog` — ClickLab's 200ms ping generalised |
| 22 | into LChatCore for both apps), or a noteworthy event (socket reconnect) |
| 23 | appends one JSON line to `Library/Logs/perf-events.jsonl` in the app's |
| 24 | container. ~512KB cap with one `.prev` rotation. Events carry a fixed |
| 25 | name, a duration, and at most a URL path or row count — never content, |
| 26 | names, or keys. |
| 27 | |
| 28 | 3. **MetricKit (consent-gated).** `MetricKitCollector` subscribes to the |
| 29 | OS's own APM: daily payloads (launch-time histograms, hang totals, |
| 30 | memory) and hang diagnostics **with call trees** — real-device stack |
| 31 | traces for hangs nobody was profiling. Raw payloads are filed under |
| 32 | `Library/Logs/perf-diagnostics/` (newest 24 kept) with a compact ledger |
| 33 | line each. The Mac delivers a subset (no hang diagnostics) — that's why |
| 34 | the watchdog exists on both platforms. |
| 35 | |
| 36 | ## The gate |
| 37 | |
| 38 | - **Debug builds record by default** (opt out: the Settings toggle, or |
| 39 | `defaults write <bundle id> perfDiagnostics -bool NO`). |
| 40 | - **Release builds record only with the consent toggle** — iOS: the server |
| 41 | settings cog → Performance; Mac: Settings → Advanced → Performance |
| 42 | (Paul's call, 2026-07-20). Signposts stay on either way. |
| 43 | - The extensions (NSE, share) never call `Perf.install()`, so for them the |
| 44 | layer is inert. |
| 45 | |
| 46 | ## Instrumented seams |
| 47 | |
| 48 | | Span | Where | Budget | |
| 49 | |---|---|---| |
| 50 | | `boot.firstFrame` | launch hook → RootView first appear (iOS) | recorded always | |
| 51 | | `threads.hydrate` / `threads.refresh` | ThreadStore | 500ms / 2500ms | |
| 52 | | `feed.openThread` | ChatFeedModel.loadHistory (open a chat → rendered) | 1500ms | |
| 53 | | `feed.foldPage` | ChatFeedModel.foldPageDetached (200-row decrypt+fold) | 250ms | |
| 54 | | `api.get` / `api.post` | LChatAPI choke points (detail = path, never query/token) | 1000ms | |
| 55 | | `socket.connect` | ChatSocket connect → handshake (failed attempts unrecorded) | 3000ms | |
| 56 | | `image.load` | BlobImageLoader fetch+decrypt+decode | 1000ms | |
| 57 | | `main.hang` | HangWatchdog (>250ms main-thread wedge — Apple's responsiveness bar) | — | |
| 58 | | `frame.hitch` | HitchRecorder (iOS only: rendered-frame gap ≥120ms; the felt-jank band below the watchdog — the Mac has no CADisplayLink and keeps just the watchdog) | — | |
| 59 | | `socket.reconnect` | note event, 30s same-name floor | — | |
| 60 | |
| 61 | Adding one is two lines (`Perf.begin`/`end`) — budgets are per-call-site, so |
| 62 | pick the number at which you'd actually want a report. |
| 63 | |
| 64 | ## Getting the data off a device |
| 65 | |
| 66 | - **Settings → Performance → Prepare performance report** (iOS shares the |
| 67 | file; the Mac reveals it in Finder): one self-contained |
| 68 | `performance-report.json` — device model, OS, app version, the ledger, |
| 69 | and the newest MetricKit payloads (2MB cap). |
| 70 | - Dev pull without the UI: |
| 71 | `xcrun devicectl device copy from --device <id> --source Library/Logs/perf-events.jsonl --domain-type appDataContainer --domain-identifier com.elevenmessenger.messenger` (same pattern as `hitches.log`). |
| 72 | - Also free, zero code: **Xcode Organizer** (Hangs / Launch / Terminations |
| 73 | from TestFlight + opted-in App Store users) and the App Store Connect |
| 74 | perf-power-metrics + diagnostic-signature APIs (the existing ASC key can |
| 75 | drive a periodic pull). |
| 76 | |
| 77 | ## What this is not (yet) |
| 78 | |
| 79 | The report is the streaming unit by design (`PerfExport` is |
| 80 | self-contained), but nothing streams anywhere today. The intended follow-up |
| 81 | is an **opt-in** upload of that JSON to a webhook-ingest feed |
| 82 | (`docs/webhooks.md`) so reports land as encrypted messages in a team |
| 83 | thread — an Aikido-style stream we own, with the server still seeing only |
| 84 | ciphertext. Third-party APM stays off the table. |
| 85 | |
| 86 | Related: `HitchLogger.swift` (iOS, DEBUG-only frame-drop breadcrumbs — |
| 87 | still marked temporary) and `ClickLab.swift` (Mac click-eating rig) predate |
| 88 | this layer and stay as-is; fold them in only when their bugs close. |
| 89 | |
amadan