Merge branch 'fix/screen-stall-escalation' into main
Screen stalls keyframes cannot mend (spec 15.22) and Reconnecting only for a problem that lasts (15.23). Calls 0.4.5.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
31 files changed,
+2782
−117
Makefile+1 −0docs/superpowers/specs/2026-09-07-meet-design.md+131 −1internal/calls/static/call-window.js+2 −0internal/calls/static/calls.css+3 −0internal/calls/static/room.js+563 −24internal/calls/templates/room.html+6 −0internal/callstest/native_browser_test.go+258 −11native/README.md+37 −0native/apple/Sources/CallsInterop/Interop.swift+41 −1native/apple/Sources/CallsInterop/SyntheticVideo.swift+5 −1native/apple/Sources/CallsKit/BroadcastScreenCapture.swift+73 −11native/apple/Sources/CallsKit/Call.swift+156 −20native/apple/Sources/CallsKit/CallAudio.swift+3 −1native/apple/Sources/CallsKit/Capture.swift+7 −0native/apple/Sources/CallsKit/ChatView.swift+1 −1native/apple/Sources/CallsKit/DecoderWedge.swift+123 −0native/apple/Sources/CallsKit/MacFullWindowCall.swift+1 −1native/apple/Sources/CallsKit/MacScreenCapture.swift+3 −0native/apple/Sources/CallsKit/Media.swift+55 −11native/apple/Sources/CallsKit/ReconnectWatch.swift+125 −0native/apple/Sources/CallsKit/ScreenWatchdog.swift+299 −11native/apple/Sources/CallsMac/Resources/MacIcon.pngbinarynative/apple/Tests/CallsKitTests/ReconnectWatchTests.swift+102 −0native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift+209 −6native/ios/Sources/PhoneFullScreenCall.swift+1 −1native/ios/project.yml+1 −1test/browser/call.mjs+23 −2test/browser/native-screen.mjs+332 −5test/browser/room-reconnect.test.mjs+105 −0test/browser/room-screen.test.mjs+111 −6test/browser/room-share.test.mjs+5 −2
CI passed — run details
CI log
=== vet === go vet ./... === fmt === === test === go test ./... ok amadan.net/oficina/calls/callscore (cached) ? amadan.net/oficina/calls/cmd/calls [no test files] ok amadan.net/oficina/calls/internal/calls 1.427s ok amadan.net/oficina/calls/internal/calls/apns (cached) ok amadan.net/oficina/calls/internal/calls/hub (cached) ? amadan.net/oficina/calls/internal/calls/icons [no test files] ok amadan.net/oficina/calls/internal/calls/relay (cached) ok amadan.net/oficina/calls/internal/calls/seal (cached) ok amadan.net/oficina/calls/internal/callstest 8.442s ? amadan.net/oficina/calls/protocol [no test files] ? amadan.net/oficina/calls/scripts/writeestilo [no test files] === migration-check === go run amadan.net/rastrillo/rastrillo/cmd/rastrillo migration check rastrillo migration check: models and migrations agree === race === CGO_ENABLED=1 go test -race ./internal/... ok amadan.net/oficina/calls/internal/calls 8.679s ok amadan.net/oficina/calls/internal/calls/apns (cached) ok amadan.net/oficina/calls/internal/calls/hub (cached) ? amadan.net/oficina/calls/internal/calls/icons [no test files] ok amadan.net/oficina/calls/internal/calls/relay (cached) ok amadan.net/oficina/calls/internal/calls/seal (cached) ok amadan.net/oficina/calls/internal/callstest 23.758s
diff --git a/Makefile b/Makefile| index 7c5b800..0075e0f 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -151,6 +151,7 @@ browser: |
| node --test test/browser/room-space.test.mjs |
| node --test test/browser/room-hearing.test.mjs |
| node --test test/browser/room-screen.test.mjs |
| + node --test test/browser/room-reconnect.test.mjs |
| node --test test/browser/room-glow.test.mjs |
| node --test test/browser/room-layout.test.mjs |
| node --test test/browser/room-window.test.mjs |
diff --git a/docs/superpowers/specs/2026-09-07-meet-design.md b/docs/superpowers/specs/2026-09-07-meet-design.md| index 58989e6..243ec8e 100644 |
| --- a/docs/superpowers/specs/2026-09-07-meet-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-meet-design.md |
| @@ -2415,7 +2415,7 @@ own screen, and the receiver says when a screen has stopped for it. |
| decoded nothing new here for four seconds while they say they are |
| sharing. Every sender repeats a still screen at least once a second, so |
| that is not a still screen. It is bad news only, omitted when there is |
| - none, held for at most a minute from the last frame, at most fifteen |
| + none, held for at most a minute from the last frame (three since §15.22), at most fifteen |
| distinct well-formed peer ids, validated exactly as `noSoundFrom` is, and |
| re-encoded after it (`muted`, `cameraOff`, `audioBlocked`, `noSoundFrom`, |
| `screenStalled`). The raw payload cap is 2 KB, room for both lists full. |
| @@ -2436,3 +2436,133 @@ mute is never lost to an old server. The sharer's own watchdog works |
| whatever the server is. Deploy the server before, or with, the clients. |
| **Not logged, not stored.** As §15.19. |
| + |
| +### 15.22 §15.21 — when keyframes are not enough (2026-09-30) |
| + |
| +**Why.** Paul, 2026-09-30, sharing from the Mac app (0.4.3 or 0.4.4) to |
| +Samuel: "Still seeing issues with screen sharing after a while." His |
| +screenshot had §15.21's notice, "Samuel's view of your screen stalled — |
| +recovering", under "You are sharing your screen". The receiver's report had |
| +worked; recovery had not. By design, a report alone only ever bought |
| +keyframes, and a stall that keyframes do not mend — a receiver whose decoder |
| +has wedged, or one that has stopped taking this stream — had nothing after |
| +them: the notice sat at "recovering" until the report lapsed a minute later. |
| +Stop and Share, by hand, always mended it. Two more things came up reading the |
| +code: a receiver counted only a *larger* decoded count as movement, so a |
| +screen that had been given a new transceiver (whose count starts again) |
| +stayed "stalled" there for as long as its new count took to pass the old one; |
| +and the iPhone's broadcast sends nothing for a still screen, which any |
| +receiver reported as a stall after four seconds. |
| + |
| +**What the harness measured.** A receiver whose screen decoder is wedged |
| +(the gate's decoder wrapper in the apps, an encoded transform in Chromium) keeps |
| +receiving packets and frames, decodes none, and drops them: packets +75, |
| +frames received +32, decoded +0, dropped +29 in the first four seconds, between |
| +two apps. Keyframes arrive and are dropped with the rest. A new transceiver, |
| +which gets a new receiver and decoder at the far end, mends it at once, in both |
| +directions and between two apps. No long share has yet produced a stall that |
| +keyframes did not mend, so the wedge is a stand-in for the cause, not the cause: |
| + |
| +- Samuel was on the web, so the path was the Mac app to Chromium, where |
| + §15.21 found H.264 held to level 3.1. With the level cap switched off |
| + (`CALLS_SCREEN_UNCAPPED=1`), a Motion share of a 1920×1246 page (a 16:10 |
| + Retina display as the capture fits it) reproduces Paul's picture exactly: |
| + VideoToolbox refuses every frame (-12902, 1,323 times in 150 s), the browser |
| + receives nothing and sends PLIs, freezes of 7–23 s and one that never ended, |
| + which keyframes, fresh encoders and a re-add only partly mend. With the cap as |
| + shipped, the same page starting in Motion and starting in Balanced, modes |
| + cycled every 30 s and the uplink capped from 600 kbit/s to 3 Mbit/s, eleven |
| + ICE restarts from the app and twenty-three at the browser in four minutes |
| + each: no -12902 and no freeze. The real display through ScreenCaptureKit |
| + (3024×1964, sent at 1920×1246) for seven minutes the same way: no -12902; the |
| + only gaps, 4 s each, were Text and detail on a 600 kbit/s link, where a |
| + full-size frame of a still screen takes that long to send, which is the link's |
| + and which the ladder does not take further (nothing was sent). |
| + A screencast source (Balanced, Text and detail) is set up by this WebRTC |
| + build without a fixed level and carries 1920×1246 to Chromium; a source that |
| + is not one (Motion, or a share started in it) is capped, now from the first |
| + frame, since the Mac capture's size is known before it delivers one. |
| +- The browser sharing to the app, modes cycled, renewals every ten seconds: |
| + four and a half minutes, no freeze. |
| +- Considered and not done: making Chromium's H.264 level higher (it answers |
| + 3.1 and room.js does not munge SDP, §5.1), and VP8 or VP9 for screens to |
| + browsers (software VP8 managed 2 fps at 1080p, §5.4's measurements). The |
| + measured cap holds; the ladder covers what it does not. |
| + |
| +What is left untraced, and what the new statistics are for: a receiver's own |
| +decoder going wrong (a hardware decoder in the watcher's browser, which a |
| +headless gate never uses), and the one ICE restart in ten between two apps that |
| +stops the screen for three or four seconds (§15.21). |
| + |
| +**Decision.** The sharer takes a report further, the receiver does its own |
| +share, and both say what they see. |
| + |
| +- **The sharer's ladder**, per reporting peer: keyframes as §15.21. Still |
| + reported six seconds after the first, **that connection's screen is |
| + rebuilt** — its transceiver stopped and a new one added once that has been |
| + negotiated, at the same size and mode; a report never shrinks the screen or |
| + makes a smaller encoder. Still reported ten seconds after that, |
| + **everyone's screen is shared again** from the same capture: every screen |
| + transceiver stopped, a new track on the same source (a clone of the captured |
| + track in the browser), each connection given it as it settles; no picker. |
| + Ten seconds after that, it gives up. A rebuild at most once a minute per |
| + peer, a share again at most once every five minutes; a step that is not |
| + allowed goes to the next, and none is taken unless at least three frames |
| + left for that peer since the last step (a screen with nothing new to send, |
| + or held back by the link, is not a decoder's fault). A report that clears |
| + ends the ladder for that peer. |
| +- **The notice moves on in words**: "Samuel's view of your screen stalled — |
| + recovering", then "Restarting your share for Samuel…", then either nothing |
| + or "Couldn't restore Samuel's view — try Stop and Share again". Worst news |
| + first when people are at different steps. |
| +- **The receiver**: any change in a screen's decoded count is movement. Four |
| + seconds into a stall it hands the screen's picture to its view again, once |
| + per stall (a new renderer in the apps, a fresh stream on the element in the |
| + browser): a picture stopped on this side is this side's to mend. Neither |
| + WebRTC build has a way for a receiver to ask for a keyframe directly; the |
| + report is that request, and WebRTC's own PLIs go out as they always did. A |
| + report is held for up to three minutes from the last frame, not one, so the |
| + sharer's "Couldn't restore" is not taken away while it is still true. |
| +- **The iPhone's broadcast repeats its last frame once a second** while the |
| + screen is still, as the Mac's capture has since §15.21. |
| +- **What is logged.** Every step, both sides, in the screen log (`os_log` |
| + category `screen`; the browser's console), and every stall with what the |
| + receiver's statistics say moved since the screen last did: packets, frames |
| + received, decoded, dropped, keyframes, PLI and NACK, the decoder, and a |
| + verdict (nothing arriving; arriving but not assembled; assembled but not |
| + decoded). "Copy call diagnostics" carries the ladder's counts and stage per |
| + peer, every arriving screen's statistics, and the last 200 lines of the |
| + timeline, on the sharer and the receiver alike. The browser gains the same |
| + button, at the foot of its Devices panel. |
| + |
| +**Compatibility.** Nothing on the wire changes. An older receiver reports as |
| +before and is answered with the new ladder; an older sharer answers a new |
| +receiver's longer report with keyframes as before. |
| + |
| +**Not logged, not stored.** As §15.19; the timeline stays on the device until |
| +someone copies it. |
| + |
| +### 15.23 §4.5, §5 — "Reconnecting…" only for a problem that lasts (2026-09-30) |
| + |
| +**Why.** Paul, 2026-09-30: "During that call I also saw 'Reconnecting' appear |
| +occasionally over the video, but only for a second … it didn't affect the call |
| +quality … I didn't notice any interruption in the call." The apps' core enters |
| +its `reconnecting` phase the moment the signalling stream ends, and the stream |
| +ends on purpose at its TTL (every five minutes, §4.5) and is resumed half a |
| +second later; the apps said "Reconnecting…" every time. The media never |
| +noticed. The browser said nothing at all, even for a real outage. |
| + |
| +**Decision.** The notice is said, on Mac, iPhone and web, only when: |
| + |
| +- the signalling stream has been down for 4 s or more — the whole call; or |
| +- a peer connection's ICE state has been `disconnected` or `failed` for 3 s or |
| + more, or no inbound audio packets have arrived for 2 s or more from a peer |
| + that is not muted — that peer's tile, "Reconnecting to Samuel…"; or the whole |
| + call, when it is every peer (two or more). |
| + |
| +Once said it stays a second past the recovery, then goes. A routine renewal or |
| +resume is never said. The timings are named constants (`ReconnectWatch` in |
| +the apps, `RECONNECT_*` in room.js) with unit tests either side; the browser |
| +gate's `renewal` scenario and the native gate's `resume` and `renewals` watch |
| +for the words through several renewals and fail on any. The core's phase is |
| +unchanged: controls that need an admitted call still follow it. |
diff --git a/internal/calls/static/call-window.js b/internal/calls/static/call-window.js| index a34337d..58c2440 100644 |
| --- a/internal/calls/static/call-window.js |
| +++ b/internal/calls/static/call-window.js |
| @@ -370,6 +370,8 @@ |
| if (s.terminal) return 'Call ended'; |
| if (/full/i.test(status || '')) return 'The room is full'; |
| if (!s.admitted) return /waiting/i.test(status || '') ? 'Waiting to be let in' : 'Connecting'; |
| + // §15.23: only ever a problem that has lasted (room.js's watch). |
| + if (s.reconnect && s.reconnect.call) return 'Reconnecting'; |
| return 'In the meeting'; |
| } |
| const stateLine = $('call-state'); |
diff --git a/internal/calls/static/calls.css b/internal/calls/static/calls.css| index 6766953..4679721 100644 |
| --- a/internal/calls/static/calls.css |
| +++ b/internal/calls/static/calls.css |
| @@ -457,6 +457,9 @@ body:has(> .calls-window) { background: var(--rst-bg); } |
| text-align: center; |
| } |
| .calls-window .calls-device-note { color: #fff; } |
| +/* "Copy call diagnostics" at the foot of the Devices popover (§15.22). */ |
| +.calls-copy-diagnostics { margin-block-start: var(--rst-sp-3); align-self: flex-start; } |
| +.calls-window .calls-copy-diagnostics { color: #fff; } |
| /* The sharing notice (2026-09-28): a dark pill with its own Stop |
| sharing button, and the recovery line under it. */ |
diff --git a/internal/calls/static/room.js b/internal/calls/static/room.js| index 7fdc6da..54a16d4 100644 |
| --- a/internal/calls/static/room.js |
| +++ b/internal/calls/static/room.js |
| @@ -71,6 +71,7 @@ |
| const shareNotice = $('share-notice'); |
| const shareStop = $('share-stop'); |
| const shareRecoveryEl = $('share-recovery'); |
| + const copyDiagnostics = $('copy-diagnostics'); |
| // §15.19: the tile marks, rendered by the server so they are the |
| // vendored badge and the app's own icons rather than a copy of either, |
| // and the polite region that says a change out loud. #status is no |
| @@ -277,10 +278,43 @@ |
| // for a keyframe, at most once every SCREEN_KEYFRAME_EVERY, and rests |
| // SCREEN_KEYFRAME_REST after SCREEN_KEYFRAME_TRIES of them. |
| const SCREEN_STALL_AFTER = 4000; |
| - const SCREEN_STALL_GIVE_UP = 60000; |
| + const SCREEN_STALL_GIVE_UP = 180000; |
| const SCREEN_KEYFRAME_EVERY = 4000; |
| const SCREEN_KEYFRAME_TRIES = 3; |
| const SCREEN_KEYFRAME_REST = 30000; |
| + // §15.22 (2026-09-30), when keyframes do not answer. The receiver |
| + // hands a stalled screen's picture to its element again once, |
| + // SCREEN_REATTACH_AFTER into the stall. The sharer, still named |
| + // SCREEN_REBUILD_AFTER after its first keyframe, rebuilds that |
| + // connection's screen (a new transceiver: a new encoder here, a new |
| + // receiver and decoder there); still named SCREEN_RESHARE_AFTER after |
| + // that, shares again for everyone from the same capture; and |
| + // SCREEN_GIVE_UP_AFTER after that, says it could not. A rebuild at most |
| + // once every SCREEN_REBUILD_EVERY per person, a share again at most |
| + // once every SCREEN_RESHARE_EVERY, and neither unless at least |
| + // SCREEN_ESCALATE_SENT frames left for that person since the last step: |
| + // a receiver cannot be failing to decode frames that were never sent. |
| + const SCREEN_REATTACH_AFTER = 4000; |
| + const SCREEN_REBUILD_AFTER = 6000; |
| + const SCREEN_RESHARE_AFTER = 10000; |
| + const SCREEN_GIVE_UP_AFTER = 10000; |
| + const SCREEN_REBUILD_EVERY = 60000; |
| + const SCREEN_RESHARE_EVERY = 300000; |
| + const SCREEN_ESCALATE_SENT = 3; |
| + // The screen's timeline, for "Copy call diagnostics": its last lines. |
| + const SCREEN_LOG_LINES = 200; |
| + // §15.23 (2026-09-30), "Reconnecting…" only for a problem that lasts: |
| + // the stream down RECONNECT_STREAM_AFTER (the whole call); a connection |
| + // disconnected or failed RECONNECT_CONNECTION_AFTER, or no audio packets |
| + // from someone unmuted RECONNECT_AUDIO_AFTER (that person's tile, or the |
| + // whole call when it is everyone). Said for RECONNECT_HOLD past the |
| + // recovery, so it does not flicker. A stream renewal — ended on purpose |
| + // and resumed in half a second — is never said at all. |
| + const RECONNECT_STREAM_AFTER = 4000; |
| + const RECONNECT_CONNECTION_AFTER = 3000; |
| + const RECONNECT_AUDIO_AFTER = 2000; |
| + const RECONNECT_HOLD = 1000; |
| + const RECONNECT_EVERY = 200; |
| // §5.4, sending: "Optimise shared screen for". What a screen gives up |
| // first when the uplink or the encoder cannot carry all of it. The |
| // first is the default; sharePlan says what each one means. |
| @@ -345,6 +379,20 @@ |
| const screenAsks = new Map(); |
| let shareRecovery = ''; |
| let screenStallTestUntil = 0; |
| + // §15.22: each arriving screen's statistics at the last poll and when |
| + // it last moved (share key -> screenReading); pictures handed back; |
| + // when each peer's screen was last rebuilt, and everyone's last shared |
| + // again; and the timeline itself. |
| + const screenReadings = new Map(); |
| + const screenMoving = new Map(); |
| + let screenReattaches = 0; |
| + const shareRebuiltAt = new Map(); |
| + let shareResharedAt = 0; |
| + const shareCounts = { keyframes: 0, rebuilds: 0, reshares: 0, failures: 0 }; |
| + const screenLines = []; |
| + // §15.23: when the stream went down (0 while it is up), and the watch. |
| + let streamDownSince = 0; |
| + const reconnect = reconnectRecord(); |
| // §15.20: peer id -> what this tab is hearing from that peer, as |
| // pollLevels reads it (hearingRecord). selfHearing is the same record |
| // for our own microphone, read off the sender. |
| @@ -418,6 +466,7 @@ |
| // trying since it opened, and that is the stretch STALL_WINDOW means. |
| let lastHelloAt = Date.now(); |
| let reconnectTimer = 0; |
| + let reconnectWatchTimer = 0; |
| let statsTimer = 0; |
| let fullTimer = 0; |
| let renewTimer = 0; // §5.3: the scheduled renewal, at `renewAt` |
| @@ -1248,6 +1297,7 @@ |
| if (source !== es) return; |
| closeStream(); |
| if (state.stopped) return; |
| + if (!streamDownSince) streamDownSince = Date.now(); |
| // Only a stream that actually lived clears the backoff. One that |
| // died moments after its hello has not proved anything, and |
| // resetting on the hello alone is how a tight reconnect loop |
| @@ -1385,6 +1435,7 @@ |
| // then stayed open. |
| stalls = 0; |
| lastHelloAt = Date.now(); |
| + streamDownSince = 0; |
| const fresh = !state.peer; |
| state.peer = msg.peer || null; |
| state.boot = msg.boot || state.boot; |
| @@ -4303,25 +4354,110 @@ |
| // judgeScreen is the receiver's verdict on one screen, from its decoded |
| // count this poll: stalled, or not. `seen` is its record, updated here. |
| // Pure over its arguments, and lifted by name in the node test. |
| + // |
| + // Any change in the count is movement, a smaller one included: a |
| + // screen the sharer has rebuilt arrives on a new receiver whose count |
| + // starts again, and up to 2026-09-30 only a larger count moved, so a |
| + // rebuilt screen that was playing again stayed "stalled" here — and the |
| + // sharer's notice with it — until the new count passed the old. |
| function judgeScreen(seen, decoded, now, frozen) { |
| - if (!seen.movedAt || (decoded > seen.decoded && !frozen)) { |
| + if (!seen.movedAt || (decoded !== seen.decoded && !frozen)) { |
| seen.decoded = decoded; |
| seen.movedAt = now; |
| + seen.reattached = false; |
| return false; |
| } |
| const quiet = now - seen.movedAt; |
| return quiet >= SCREEN_STALL_AFTER && quiet < SCREEN_STALL_GIVE_UP; |
| } |
| + // reattachDue says, once per stall, when a stalled screen's picture |
| + // should be handed to its element again (§15.22): SCREEN_REATTACH_AFTER |
| + // into the stall. A picture stopped on this side — an element or a |
| + // track Chromium has left muted — is this side's to mend. Pure over |
| + // `seen`, and lifted by name in the node test. |
| + function reattachDue(seen, now) { |
| + const quiet = now - seen.movedAt; |
| + if (seen.reattached || quiet < SCREEN_STALL_AFTER + SCREEN_REATTACH_AFTER || quiet >= SCREEN_STALL_GIVE_UP) { |
| + return false; |
| + } |
| + seen.reattached = true; |
| + return true; |
| + } |
| + |
| + // screenLog is one line of the screen's timeline: the console, and the |
| + // last SCREEN_LOG_LINES kept for "Copy call diagnostics". |
| + function screenLog(line) { |
| + const d = new Date(); |
| + const pad = (n, w) => String(n).padStart(w || 2, '0'); |
| + const stamp = pad(d.getHours()) + ':' + pad(d.getMinutes()) + ':' + pad(d.getSeconds()) + '.' + pad(d.getMilliseconds(), 3); |
| + screenLines.push(stamp + ' ' + line); |
| + if (screenLines.length > SCREEN_LOG_LINES) screenLines.splice(0, screenLines.length - SCREEN_LOG_LINES); |
| + console.info('calls: ' + line); |
| + } |
| + |
| + // screenReading is what one inbound screen report says, and |
| + // readingSummary and readingMoved put it in words. Pure, and lifted by |
| + // name in the node test: packets arriving and frames assembled but none |
| + // decoded is a decoder; nothing arriving is the sender or the path. |
| + function screenReading(r, trackMuted) { |
| + return { |
| + mid: r.mid || '', |
| + packets: r.packetsReceived || 0, |
| + lost: r.packetsLost || 0, |
| + received: r.framesReceived || 0, |
| + decoded: r.framesDecoded || 0, |
| + dropped: r.framesDropped || 0, |
| + keyframes: r.keyFramesDecoded || 0, |
| + pli: r.pliCount || 0, |
| + fir: r.firCount || 0, |
| + nack: r.nackCount || 0, |
| + freezes: r.freezeCount || 0, |
| + width: r.frameWidth || 0, |
| + height: r.frameHeight || 0, |
| + decoder: r.decoderImplementation || '', |
| + muted: !!trackMuted, |
| + }; |
| + } |
| + |
| + function readingSummary(r) { |
| + return 'mid ' + (r.mid || '-') + ', ' + r.width + 'x' + r.height + ', decoder ' + (r.decoder || 'unknown') + |
| + (r.muted ? ', track muted' : '') + '; packets ' + r.packets + ' (lost ' + r.lost + '), frames received ' + |
| + r.received + ', decoded ' + r.decoded + ', dropped ' + r.dropped + ', keyframes ' + r.keyframes + |
| + ', PLI ' + r.pli + ', FIR ' + r.fir + ', NACK ' + r.nack + ', freezes ' + r.freezes; |
| + } |
| + |
| + function readingMoved(r, b) { |
| + if (!b || b.mid !== r.mid) return readingSummary(r) + ' — no earlier reading'; |
| + const d = (a, z) => (a >= z ? '+' + (a - z) : 'new ' + a); |
| + let verdict = 'decoding'; |
| + if (r.packets === b.packets) verdict = 'no packets arriving (the sender or the path)'; |
| + else if (r.received === b.received) verdict = 'packets arriving, no frames assembled (loss, or waiting on a keyframe)'; |
| + else if (r.decoded === b.decoded) verdict = 'frames arriving, none decoded (the decoder)'; |
| + return 'packets ' + d(r.packets, b.packets) + ', frames received ' + d(r.received, b.received) + ', decoded ' + |
| + d(r.decoded, b.decoded) + ', dropped ' + d(r.dropped, b.dropped) + ', keyframes ' + d(r.keyframes, b.keyframes) + |
| + ', PLI ' + d(r.pli, b.pli) + ', NACK ' + d(r.nack, b.nack) + (r.muted ? ', track muted' : '') + ' — ' + verdict; |
| + } |
| + |
| // watchScreens takes one poll's decoded counts, by share key, and |
| // reports the list when it changes. |
| - function watchScreens(decodedByKey) { |
| + // |
| + // `readings` is the same poll's screenReading per share key: kept for |
| + // the diagnostics, and set against the reading from when the screen last |
| + // moved, so a stall is logged with what is and is not still arriving. |
| + function watchScreens(decodedByKey, readings) { |
| const now = Date.now(); |
| const frozen = now < screenStallTestUntil; |
| const stalled = []; |
| for (const key of Array.from(screenSeen.keys())) { |
| if (!decodedByKey.has(key)) screenSeen.delete(key); |
| } |
| + for (const key of Array.from(screenReadings.keys())) { |
| + if (!decodedByKey.has(key)) { |
| + screenReadings.delete(key); |
| + screenMoving.delete(key); |
| + } |
| + } |
| for (const [key, decoded] of decodedByKey) { |
| const id = key.slice(0, -SHARE_SUFFIX.length); |
| const entry = pcs.get(id); |
| @@ -4331,57 +4467,394 @@ |
| } |
| let seen = screenSeen.get(key); |
| if (!seen) { |
| - seen = { decoded: 0, movedAt: 0 }; |
| + seen = { decoded: 0, movedAt: 0, reattached: false }; |
| screenSeen.set(key, seen); |
| } |
| - if (judgeScreen(seen, decoded, now, frozen)) stalled.push(id); |
| + const reading = readings && readings.get(key); |
| + if (reading) screenReadings.set(key, reading); |
| + if (judgeScreen(seen, decoded, now, frozen)) { |
| + stalled.push(id); |
| + if (reattachDue(seen, now)) { |
| + screenLog('screen from ' + id.slice(0, 6) + ' still stalled: handing its picture to the tile again; ' + |
| + (reading ? readingMoved(reading, screenMoving.get(key)) : 'no reading')); |
| + const t = tiles.get(key); |
| + if (t && t.video) { |
| + screenReattaches += 1; |
| + replayTile(key, t, entry, true); |
| + } |
| + } |
| + } else if (reading) { |
| + screenMoving.set(key, reading); |
| + } |
| } |
| stalled.sort(); |
| if (stalled.join() === screenStalled.join()) return; |
| for (const id of stalled) { |
| - if (!screenStalled.includes(id)) console.info('calls: screen from ' + id.slice(0, 6) + ' stalled here, asking for a keyframe'); |
| + if (screenStalled.includes(id)) continue; |
| + const key = shareKey(id); |
| + const reading = screenReadings.get(key); |
| + screenLog('screen from ' + id.slice(0, 6) + ' stalled here: reporting it; since it last moved ' + |
| + (reading ? readingMoved(reading, screenMoving.get(key)) : 'no reading')); |
| + } |
| + for (const id of screenStalled) { |
| + if (stalled.includes(id)) continue; |
| + const reading = screenReadings.get(shareKey(id)); |
| + screenLog('screen from ' + id.slice(0, 6) + ' moving here again' + (reading ? ': ' + readingSummary(reading) : '')); |
| } |
| screenStalled = stalled; |
| reportMedia(); |
| } |
| // answerScreenStalls is the sharer's half: every peer whose report |
| - // names us gets a keyframe from our screen's encoder, and the quiet |
| - // notice says whose view is being put right. |
| + // names us gets a keyframe from our screen's encoder, and — when |
| + // keyframes do not answer (§15.22) — that connection's screen rebuilt, |
| + // then everyone's shared again from the same capture, and then words |
| + // saying it could not be restored. The quiet notice says which. |
| function answerScreenStalls() { |
| const me = state.peer; |
| - const names_ = []; |
| const now = Date.now(); |
| + let reshare = false; |
| for (const [id, m] of remoteMedia) { |
| const asking = !!(share && me && m && m.screenStalled && m.screenStalled.includes(me)); |
| if (!asking) { |
| + const gone = screenAsks.get(id); |
| + if (gone && gone.stage && share) screenLog(id.slice(0, 6) + '\u2019s view restored after ' + stageWords(gone.stage)); |
| screenAsks.delete(id); |
| continue; |
| } |
| - names_.push(names.get(id) || 'Someone'); |
| const entry = pcs.get(id); |
| let ask = screenAsks.get(id); |
| if (!ask) { |
| - ask = { at: 0, tries: 0, restUntil: 0 }; |
| + ask = { at: 0, tries: 0, restUntil: 0, stage: '', stageAt: 0 }; |
| screenAsks.set(id, ask); |
| + if (entry) entry.shareSentSince = 0; |
| + } |
| + if (entry && now >= ask.restUntil && now - ask.at >= SCREEN_KEYFRAME_EVERY) { |
| + ask.at = now; |
| + ask.tries += 1; |
| + if (ask.tries >= SCREEN_KEYFRAME_TRIES) { |
| + ask.tries = 0; |
| + ask.restUntil = now + SCREEN_KEYFRAME_REST; |
| + } |
| + shareCounts.keyframes += 1; |
| + screenLog('keyframe for ' + id.slice(0, 6) + ', whose view of the screen stalled'); |
| + shareKeyframe(entry).catch(() => {}); |
| } |
| - if (!entry || now < ask.restUntil || now - ask.at < SCREEN_KEYFRAME_EVERY) continue; |
| - ask.at = now; |
| - ask.tries += 1; |
| - if (ask.tries >= SCREEN_KEYFRAME_TRIES) { |
| - ask.tries = 0; |
| - ask.restUntil = now + SCREEN_KEYFRAME_REST; |
| + // Frames left for this person since the last step, or no screen |
| + // negotiated with them at all (a rebuild that never came back). |
| + const negotiated = !!(entry && entry.shareTx && entry.shareSend); |
| + const owed = !negotiated || (entry.shareSentSince || 0) >= SCREEN_ESCALATE_SENT; |
| + const rec = { stage: ask.stage, at: ask.stageAt, lastRebuild: shareRebuiltAt.get(id) || 0 }; |
| + const before = rec.stage; |
| + const reshareFree = reshare || !shareResharedAt || now - shareResharedAt >= SCREEN_RESHARE_EVERY; |
| + const action = escalateShare(rec, now, owed, reshareFree); |
| + ask.stage = rec.stage; |
| + ask.stageAt = rec.at; |
| + if (rec.stage !== before && entry) entry.shareSentSince = 0; |
| + if (action === 'rebuild') { |
| + shareRebuiltAt.set(id, now); |
| + shareCounts.rebuilds += 1; |
| + screenLog(id.slice(0, 6) + ' still reports a stall after keyframes: rebuilding its screen (new transceiver, same size and mode)'); |
| + if (entry) rebuildShareFor(entry); |
| + } else if (action === 'reshare') { |
| + reshare = true; |
| + screenLog(id.slice(0, 6) + ' still reports a stall' + (before === 'rebuilt' ? ' after the rebuild' : ', rebuilt under a minute ago') + ': sharing again'); |
| + } else if (action === 'failed') { |
| + shareCounts.failures += 1; |
| + screenLog(id.slice(0, 6) + ' still reports a stall and there is nothing left to try: giving up'); |
| } |
| - console.info('calls: keyframe for ' + id.slice(0, 6) + ', whose view of the screen stalled'); |
| - shareKeyframe(entry).catch(() => {}); |
| } |
| - const text = recoveringWords(names_); |
| + if (reshare) { |
| + shareResharedAt = now; |
| + shareCounts.reshares += 1; |
| + // One share again answers every report in flight. |
| + for (const ask of screenAsks.values()) { |
| + if (ask.stage === 'keyframes' || ask.stage === 'rebuilt') { |
| + ask.stage = 'reshared'; |
| + ask.stageAt = now; |
| + } |
| + } |
| + for (const entry of pcs.values()) entry.shareSentSince = 0; |
| + reshareAll(); |
| + } |
| + const recovering = []; |
| + const restarting = []; |
| + const failed = []; |
| + for (const [id, ask] of screenAsks) { |
| + const who = names.get(id) || 'Someone'; |
| + if (ask.stage === 'failed') failed.push(who); |
| + else if (ask.stage === 'rebuilt' || ask.stage === 'reshared') restarting.push(who); |
| + else recovering.push(who); |
| + } |
| + const text = noticeWords(recovering.sort(), restarting.sort(), failed.sort()); |
| if (text !== shareRecovery) { |
| shareRecovery = text; |
| renderShare(); |
| } |
| } |
| + // callDiagnostics is "Copy call diagnostics" for the browser: the |
| + // connections, and the screen — how ours is leaving and what the ladder |
| + // has done, every screen arriving here as its statistics have it, and |
| + // the timeline. Names, states and counts only: no media, no addresses, |
| + // no credentials. |
| + function callDiagnostics() { |
| + const lines = ['Calls web diagnostics, ' + new Date().toISOString()]; |
| + lines.push('Browser: ' + (navigator.userAgent || 'unknown')); |
| + lines.push(''); |
| + lines.push('## Call'); |
| + lines.push('admitted ' + state.admitted + ', people here ' + (pcs.size + 1) + ', stopped ' + state.stopped); |
| + for (const [id, entry] of pcs) { |
| + lines.push((names.get(id) || 'Someone') + ' (' + id.slice(0, 6) + '): connection ' + entry.pc.connectionState + |
| + ', signalling ' + entry.pc.signalingState + ', ICE ' + entry.pc.iceConnectionState); |
| + } |
| + lines.push(''); |
| + lines.push('## Screen'); |
| + if (share) { |
| + lines.push('Sharing (' + shareMode() + ')'); |
| + for (const [id, entry] of pcs) { |
| + const r = entry.shareSend; |
| + const who = names.get(id) || 'Someone'; |
| + if (!r) { |
| + lines.push('to ' + who + ': ' + (entry.shareTx ? 'negotiating' : 'no screen transceiver')); |
| + continue; |
| + } |
| + lines.push('to ' + who + ': ' + r.width + 'x' + r.height + ' at ' + Math.round(r.fps) + ' fps, encoder ' + |
| + (r.encoder || 'unknown') + ', encoded ' + r.framesEncoded + ', sent ' + r.framesSent + ', keyframes ' + |
| + r.keyFramesEncoded + ', limitation ' + (r.limitation || 'none')); |
| + } |
| + for (const [id, ask] of screenAsks) { |
| + if (ask.stage) lines.push('recovering ' + (names.get(id) || 'Someone') + '\u2019s view: ' + ask.stage); |
| + } |
| + } else { |
| + lines.push('Not sharing'); |
| + } |
| + lines.push('recoveries: ' + shareCounts.keyframes + ' keyframes, ' + shareCounts.rebuilds + ' rebuilds, ' + |
| + shareCounts.reshares + ' shares again, ' + shareCounts.failures + ' given up'); |
| + for (const [key, reading] of screenReadings) { |
| + const id = key.slice(0, -SHARE_SUFFIX.length); |
| + lines.push('from ' + (names.get(id) || 'Someone') + ': ' + readingSummary(reading)); |
| + } |
| + if (screenStalled.length) lines.push('stalled here: ' + screenStalled.map((id) => names.get(id) || 'Someone').join(', ')); |
| + if (screenReattaches) lines.push('pictures handed to their tiles again here: ' + screenReattaches); |
| + lines.push(''); |
| + lines.push('## Screen log (last ' + SCREEN_LOG_LINES + ' lines)'); |
| + if (screenLines.length) lines.push(...screenLines); |
| + else lines.push('(nothing logged)'); |
| + return lines.join('\n'); |
| + } |
| + |
| + // --------------------------------------------------------------- |
| + // "Reconnecting…" (§15.23, 2026-09-30) |
| + // --------------------------------------------------------------- |
| + // |
| + // Paul: "I saw 'Reconnecting' appear occasionally over the video, but |
| + // only for a second … I didn't notice any interruption in the call." |
| + // The Mac app said so the moment its stream ended, and a stream ends on |
| + // purpose every few minutes and is resumed half a second later. Here, |
| + // as there, the words wait for a problem that lasts. |
| + |
| + function reconnectRecord() { |
| + return { streamSince: 0, connection: new Map(), audio: new Map(), call: false, callClear: 0, |
| + peers: new Set(), peerClear: new Map() }; |
| + } |
| + |
| + // reconnectStep is one look: `streamDown`, and `people` (peer id -> |
| + // {connectionBad, audioMissing}) for everyone else. It moves `rec` on: |
| + // rec.call is the whole call's words, rec.peers the tiles that say |
| + // "Reconnecting to …". The native app's ReconnectWatch is its twin. |
| + // Pure, and lifted by name in the node test. |
| + function reconnectStep(rec, now, streamDown, people) { |
| + rec.streamSince = streamDown ? rec.streamSince || now : 0; |
| + for (const id of Array.from(rec.connection.keys())) if (!people.has(id)) rec.connection.delete(id); |
| + for (const id of Array.from(rec.audio.keys())) if (!people.has(id)) rec.audio.delete(id); |
| + const troubled = new Set(); |
| + for (const [id, p] of people) { |
| + if (p.connectionBad) { |
| + if (!rec.connection.has(id)) rec.connection.set(id, now); |
| + } else { |
| + rec.connection.delete(id); |
| + } |
| + if (p.audioMissing) { |
| + if (!rec.audio.has(id)) rec.audio.set(id, now); |
| + } else { |
| + rec.audio.delete(id); |
| + } |
| + const connection = rec.connection.has(id) && now - rec.connection.get(id) >= RECONNECT_CONNECTION_AFTER; |
| + const audio = rec.audio.has(id) && now - rec.audio.get(id) >= RECONNECT_AUDIO_AFTER; |
| + if (connection || audio) troubled.add(id); |
| + } |
| + const stream = !!rec.streamSince && now - rec.streamSince >= RECONNECT_STREAM_AFTER; |
| + const everyone = troubled.size >= 2 && troubled.size === people.size; |
| + const wantCall = stream || everyone; |
| + if (wantCall) { |
| + rec.call = true; |
| + rec.callClear = 0; |
| + } else if (rec.call) { |
| + if (!rec.callClear) rec.callClear = now; |
| + if (now - rec.callClear >= RECONNECT_HOLD) { |
| + rec.call = false; |
| + rec.callClear = 0; |
| + } |
| + } |
| + const shown = new Set(Array.from(rec.peers).filter((id) => people.has(id))); |
| + if (!wantCall) { |
| + for (const id of troubled) { |
| + shown.add(id); |
| + rec.peerClear.delete(id); |
| + } |
| + } |
| + for (const id of Array.from(shown)) { |
| + if (!wantCall && troubled.has(id)) continue; |
| + if (wantCall) { |
| + shown.delete(id); |
| + rec.peerClear.delete(id); |
| + continue; |
| + } |
| + if (!rec.peerClear.has(id)) rec.peerClear.set(id, now); |
| + if (now - rec.peerClear.get(id) >= RECONNECT_HOLD) { |
| + shown.delete(id); |
| + rec.peerClear.delete(id); |
| + } |
| + } |
| + for (const id of Array.from(rec.peerClear.keys())) if (!shown.has(id)) rec.peerClear.delete(id); |
| + rec.peers = shown; |
| + } |
| + |
| + // watchReconnect feeds the watch what this tab can see, five times a |
| + // second, and redraws what changed. |
| + function watchReconnect() { |
| + if (state.stopped) return; |
| + const now = Date.now(); |
| + const people = new Map(); |
| + for (const [id, entry] of pcs) { |
| + const ice = entry.pc.iceConnectionState; |
| + const rec = hearing.get(id); |
| + const media = remoteMedia.get(id); |
| + people.set(id, { |
| + connectionBad: ice === 'disconnected' || ice === 'failed', |
| + // Two level polls without a packet: stopped, as far as a poll can tell. |
| + audioMissing: !!(rec && rec.packetAt && media && !media.muted && now - rec.packetAt > 500), |
| + }); |
| + } |
| + const wasCall = reconnect.call; |
| + const wasPeers = Array.from(reconnect.peers).sort().join(); |
| + reconnectStep(reconnect, now, !!streamDownSince && !!state.peer, people); |
| + if (reconnect.call !== wasCall) { |
| + console.info('calls: reconnecting ' + (reconnect.call ? 'shown (a problem that lasted)' : 'cleared')); |
| + renderStatus(); |
| + } |
| + if (Array.from(reconnect.peers).sort().join() !== wasPeers) { |
| + for (const id of pcs.keys()) drawHearing(id); |
| + } |
| + } |
| + |
| + // noteShareSent counts the frames that left on this connection's |
| + // screen since its escalation last moved on, across a new transceiver's |
| + // counters starting again from nothing. |
| + function noteShareSent(entry) { |
| + if (!entry.shareSend) return; |
| + const sent = entry.shareSend.framesSent || 0; |
| + const last = entry.shareSentLast || 0; |
| + entry.shareSentSince = (entry.shareSentSince || 0) + (sent >= last ? sent - last : sent); |
| + entry.shareSentLast = sent; |
| + } |
| + |
| + // escalateShare is §15.22's ladder for one reporting peer: '' (nothing |
| + // now), 'rebuild', 'reshare' or 'failed', moving `rec` ({stage, at, |
| + // lastRebuild}) on. `owed` is whether frames have left for that person |
| + // since the last step; `reshareFree` whether a share again may happen |
| + // now. The native app's ScreenWatchdog is its twin. Pure, and lifted by |
| + // name in the node test. |
| + function escalateShare(rec, now, owed, reshareFree) { |
| + const move = (stage) => { |
| + rec.stage = stage; |
| + rec.at = now; |
| + }; |
| + if (!rec.stage) { |
| + move('keyframes'); |
| + return ''; |
| + } |
| + if (rec.stage === 'keyframes') { |
| + if (now - rec.at < SCREEN_REBUILD_AFTER || !owed) return ''; |
| + if (!rec.lastRebuild || now - rec.lastRebuild >= SCREEN_REBUILD_EVERY) { |
| + move('rebuilt'); |
| + rec.lastRebuild = now; |
| + return 'rebuild'; |
| + } |
| + } else if (rec.stage === 'rebuilt') { |
| + if (now - rec.at < SCREEN_RESHARE_AFTER || !owed) return ''; |
| + } else if (rec.stage === 'reshared') { |
| + if (now - rec.at < SCREEN_GIVE_UP_AFTER || !owed) return ''; |
| + move('failed'); |
| + return 'failed'; |
| + } else { |
| + return ''; |
| + } |
| + if (reshareFree) { |
| + move('reshared'); |
| + return 'reshare'; |
| + } |
| + move('failed'); |
| + return 'failed'; |
| + } |
| + |
| + function stageWords(stage) { |
| + return { keyframes: 'keyframes', rebuilt: 'the rebuild', reshared: 'sharing again', failed: 'giving up' }[stage] || stage; |
| + } |
| + |
| + // rebuildShareFor is a stop and share again for one person (§15.22): |
| + // this connection's screen transceiver stopped, and a new one added by |
| + // the connection's own signalingstatechange once that has been |
| + // negotiated — a new encoder here, a new receiver and decoder there, at |
| + // the same size and mode. |
| + function rebuildShareFor(entry) { |
| + if (!share || !entry.shareTx) return; |
| + removeShareFrom(entry); |
| + sayShareAgain(); |
| + } |
| + |
| + // reshareAll is Stop and Share for everyone without the picker: every |
| + // connection's screen transceiver stopped, and a clone of the captured |
| + // track — the same capture, a new track — added to each as it settles. |
| + function reshareAll() { |
| + if (!share) return; |
| + const held = share; |
| + let track; |
| + try { |
| + track = held.track.clone(); |
| + } catch { |
| + return; |
| + } |
| + try { |
| + track.contentHint = held.track.contentHint; |
| + } catch { |
| + // A browser without contentHint. |
| + } |
| + // The browser's own Stop sharing bar ends every clone. |
| + track.addEventListener('ended', () => { |
| + if (share && share.track === track) stopSharing(); |
| + }); |
| + for (const entry of pcs.values()) removeShareFrom(entry); |
| + share = { stream: new MediaStream([track]), track }; |
| + // The old one after: stopping it alone does not fire its own |
| + // `ended`, and the clone keeps the capture running. |
| + try { |
| + held.track.stop(); |
| + } catch { |
| + // Already stopped. |
| + } |
| + sayShareAgain(); |
| + } |
| + |
| + // sayShareAgain repeats `share {on:true}` once a rebuilt screen has had |
| + // time to arrive: its new track comes without a label of its own. |
| + function sayShareAgain() { |
| + setTimeout(() => { |
| + if (share) broadcast('share', { on: true }); |
| + }, 2000); |
| + } |
| + |
| // recoveringWords is the sharer's notice. Pure, and lifted by name in |
| // the node test. |
| function recoveringWords(people) { |
| @@ -4393,6 +4866,30 @@ |
| return people.length + ' people\u2019s views of your screen stalled \u2014 recovering'; |
| } |
| + // noticeWords is the notice as §15.22's ladder moves on: recovering, |
| + // then "Restarting your share for Samuel…", then either nothing or |
| + // "Couldn’t restore Samuel’s view — try Stop and Share again". Worst |
| + // news first. The native app's ScreenWords.notice, word for word. Pure, |
| + // and lifted by name in the node test (with recoveringWords). |
| + function noticeWords(recovering, restarting, failed) { |
| + const parts = []; |
| + if (failed.length) { |
| + let whose = failed.length + ' people\u2019s views'; |
| + if (failed.length === 1) whose = failed[0] + '\u2019s view'; |
| + else if (failed.length === 2) whose = failed[0] + '\u2019s and ' + failed[1] + '\u2019s views'; |
| + parts.push('Couldn\u2019t restore ' + whose + ' \u2014 try Stop and Share again'); |
| + } |
| + if (restarting.length) { |
| + let who = restarting.length + ' people'; |
| + if (restarting.length === 1) who = restarting[0]; |
| + else if (restarting.length === 2) who = restarting[0] + ' and ' + restarting[1]; |
| + parts.push('Restarting your share for ' + who + '\u2026'); |
| + } |
| + const stalled = recoveringWords(recovering); |
| + if (stalled) parts.push(stalled); |
| + return parts.join('. '); |
| + } |
| + |
| // shareKeyframe switches this connection's screen encoding off and on |
| // again, which WebRTC answers by starting it on a keyframe: there is |
| // no API that asks for one directly. |
| @@ -4780,7 +5277,9 @@ |
| h.box.hidden = !kind; |
| } |
| if (self) return; |
| - const notice = remoteNotice(name, media, state.peer, localMedia().muted); |
| + const notice = reconnect.peers.has(key) |
| + ? 'Reconnecting to ' + name + '\u2026' |
| + : remoteNotice(name, media, state.peer, localMedia().muted); |
| if (h.notice.textContent !== notice) { |
| h.notice.textContent = notice; |
| h.notice.hidden = !notice; |
| @@ -5103,6 +5602,7 @@ |
| clearTimeout(reconnectTimer); |
| clearTimeout(fullTimer); |
| clearInterval(statsTimer); |
| + clearInterval(reconnectWatchTimer); |
| clearInterval(levelTimer); |
| stopIceTimers(); |
| closeStream(); |
| @@ -5197,6 +5697,10 @@ |
| : 'Connecting to the call.'); |
| return; |
| } |
| + if (reconnect.call) { |
| + say('Reconnecting…'); |
| + return; |
| + } |
| const others = pcs.size; |
| if (!others) { |
| say('You are the only one here.', true); |
| @@ -5694,6 +6198,21 @@ |
| }); |
| } |
| + // "Copy call diagnostics" (§15.22), and a sentence in the device |
| + // panel's own live region either way. |
| + if (copyDiagnostics) { |
| + copyDiagnostics.addEventListener('click', () => { |
| + const text = callDiagnostics(); |
| + const done = () => note('Call diagnostics copied.'); |
| + const failed = () => note('The diagnostics could not be copied.'); |
| + try { |
| + navigator.clipboard.writeText(text).then(done, failed); |
| + } catch { |
| + failed(); |
| + } |
| + }); |
| + } |
| + |
| leaveButton.addEventListener('click', () => { |
| leave(); |
| go('/m/' + encodeURIComponent(sqid) + '/join'); |
| @@ -5716,6 +6235,7 @@ |
| clearTimeout(reconnectTimer); |
| clearTimeout(fullTimer); |
| clearInterval(statsTimer); |
| + clearInterval(reconnectWatchTimer); |
| clearInterval(levelTimer); |
| stopIceTimers(); |
| if (state.peer) postSignal({ peer: state.peer, type: 'leave' }, { keepalive: true }); |
| @@ -5921,6 +6441,8 @@ |
| targetBitrate: r.targetBitrate || 0, |
| encoder: r.encoderImplementation || '', |
| framesEncoded: r.framesEncoded || 0, |
| + framesSent: r.framesSent || 0, |
| + keyFramesEncoded: r.keyFramesEncoded || 0, |
| }; |
| } |
| @@ -6165,8 +6687,10 @@ |
| // callback was dropped is put back on the books (Codex S4-6's gate |
| // found one, and the counter is what every frames assertion reads). |
| rearmFrames(); |
| - // §15.21: every screen's decoded count, for the receiver's watch. |
| + // §15.21: every screen's decoded count, for the receiver's watch, and |
| + // (§15.22) the rest of what its statistics say. |
| const screens = new Map(); |
| + const screenReports = new Map(); |
| for (const [id, entry] of pcs) { |
| try { |
| const stats = await entry.pc.getStats(); |
| @@ -6188,7 +6712,11 @@ |
| const key = frameKey(id, entry, report.mid || ''); |
| if (!key) return; |
| decoded.set(key, report.framesDecoded); |
| - if (key !== id) screens.set(key, report.framesDecoded); |
| + if (key !== id) { |
| + screens.set(key, report.framesDecoded); |
| + const t = tiles.get(key); |
| + screenReports.set(key, screenReading(report, !!(t && t.track && t.track.muted))); |
| + } |
| if (!HAS_RVFC) frames.set(key, report.framesDecoded); |
| }); |
| for (const [key, count] of decoded) { |
| @@ -6199,6 +6727,7 @@ |
| else if (verdict === 'reset') requestReset(id); |
| } |
| entry.shareSend = shareSendReport(outbound, (entry.shareTx && entry.shareTx.mid) || ''); |
| + noteShareSent(entry); |
| const tx = localCameraTx(entry); |
| const verdict = judgeCamera(entry, tx, cameraReport(outbound, (tx && tx.mid) || '')); |
| if (verdict === 'repair') await repairCamera(tx); |
| @@ -6207,7 +6736,7 @@ |
| // getStats on a closing connection: nothing to report. |
| } |
| } |
| - watchScreens(screens); |
| + watchScreens(screens, screenReports); |
| answerScreenStalls(); |
| } |
| @@ -6341,6 +6870,10 @@ |
| // §15.21, for the gate: this tab's receiver watch treats every |
| // screen as frozen for `ms`, so its stall report and the sharer's |
| // keyframe can be driven end to end. |
| + // §15.22: "Copy call diagnostics" as it would be copied. |
| + diagnostics() { |
| + return callDiagnostics(); |
| + }, |
| stallScreenForTest(ms) { |
| screenStallTestUntil = Date.now() + Math.max(0, Math.min(Number(ms) || 0, 60000)); |
| }, |
| @@ -6371,6 +6904,10 @@ |
| // sharer's notice while someone's view of ours is being put right. |
| screenStalled: screenStalled.slice(), |
| shareRecovery, |
| + // §15.22: what the ladder and the receiver's own mending have done. |
| + shareRecoveries: Object.assign({ reattaches: screenReattaches }, shareCounts), |
| + // §15.23: whether "Reconnecting…" is said, for the call and by tile. |
| + reconnect: { call: reconnect.call, peers: Array.from(reconnect.peers), streamDown: !!streamDownSince }, |
| // Targets a 404 has marked stale: nothing is sent to them and |
| // no reset is asked for until a roster settles it. |
| stale: Array.from(stale), |
| @@ -6469,6 +7006,8 @@ |
| // camera that stopped encoding (Codex S4-5), so the walk runs |
| // either way and does whichever halves apply. |
| statsTimer = setInterval(pollFrames, 1000); |
| + // §15.23: no stats, only what the tab already knows. |
| + reconnectWatchTimer = setInterval(watchReconnect, RECONNECT_EVERY); |
| // §15.20's meters, five times a second and cheaper per turn: one |
| // receiver's and one sender's own stats, not the connection's walk. |
| levelTimer = setInterval(() => { |
diff --git a/internal/calls/templates/room.html b/internal/calls/templates/room.html| index ea18507..3bf2521 100644 |
| --- a/internal/calls/templates/room.html |
| +++ b/internal/calls/templates/room.html |
| @@ -172,6 +172,12 @@ |
| render, so a sentence about a camera that would not open has |
| to live somewhere that is not about to be overwritten. */}} |
| <p id="device-note" class="calls-device-note" role="status" hidden></p> |
| + {{/* What the Mac panel's "Copy call diagnostics" copies, for the |
| + browser: the connections, and the screen's recovery timeline and |
| + decode statistics on both sides of a share (spec §15.22). Names, |
| + states and counts only; room.js says in #device-note whether it |
| + was copied. */}} |
| + <button type="button" id="copy-diagnostics" rst-btn="ghost" class="calls-copy-diagnostics">Copy call diagnostics</button> |
| </div> |
| {{/* Ending the meeting for everyone is destructive, so it is a link to |
| its own confirm page (AGENTS.md), opened in a new tab so that |
diff --git a/internal/callstest/native_browser_test.go b/internal/callstest/native_browser_test.go| index e4cadaf..efb643f 100644 |
| --- a/internal/callstest/native_browser_test.go |
| +++ b/internal/callstest/native_browser_test.go |
| @@ -11,6 +11,8 @@ import ( |
| "os" |
| "os/exec" |
| "path/filepath" |
| + "sort" |
| + "strconv" |
| "strings" |
| "sync" |
| "testing" |
| @@ -221,6 +223,8 @@ type nativeSample struct { |
| Stats map[string]uint64 `json:"stats"` |
| ScreenCaptions []string `json:"screenCaptions"` |
| SharingScreen *bool `json:"sharingScreen"` |
| + ScreenRecovery string `json:"screenRecovery"` |
| + Diagnostics string `json:"diagnostics"` |
| } |
| // nativeApp is one CallsInterop process and the last observation it |
| @@ -231,6 +235,7 @@ type nativeApp struct { |
| stdin io.WriteCloser |
| mu sync.Mutex |
| last nativeSample |
| + diag string |
| done chan struct{} |
| } |
| @@ -256,7 +261,16 @@ func startNative(ctx context.Context, t *testing.T, binary string, rig *browserR |
| scanner := bufio.NewScanner(out) |
| for scanner.Scan() { |
| var s nativeSample |
| - if json.Unmarshal(scanner.Bytes(), &s) != nil || s.Stats == nil { |
| + if json.Unmarshal(scanner.Bytes(), &s) != nil { |
| + continue |
| + } |
| + if s.Diagnostics != "" { |
| + app.mu.Lock() |
| + app.diag = s.Diagnostics |
| + app.mu.Unlock() |
| + continue |
| + } |
| + if s.Stats == nil { |
| continue |
| } |
| app.mu.Lock() |
| @@ -278,6 +292,25 @@ func (n *nativeApp) sample() nativeSample { |
| func (n *nativeApp) stat(name string) uint64 { return n.sample().Stats[name] } |
| +// diagnostics asks for "Copy call diagnostics" and returns what came |
| +// back, or "" if nothing did within five seconds. |
| +func (n *nativeApp) diagnostics(t *testing.T) string { |
| + t.Helper() |
| + n.mu.Lock() |
| + n.diag = "" |
| + n.mu.Unlock() |
| + n.command(t, "diagnostics") |
| + for until := time.Now().Add(5 * time.Second); time.Now().Before(until); time.Sleep(100 * time.Millisecond) { |
| + n.mu.Lock() |
| + d := n.diag |
| + n.mu.Unlock() |
| + if d != "" { |
| + return d |
| + } |
| + } |
| + return "" |
| +} |
| + |
| func (n *nativeApp) command(t *testing.T, line string) { |
| t.Helper() |
| if _, err := io.WriteString(n.stdin, line+"\n"); err != nil { |
| @@ -373,24 +406,31 @@ func TestNativeMacScreenRecovery(t *testing.T) { |
| if _, err := os.Stat(binary); err != nil { |
| t.Fatal("build native interop executable with make native-mac first") |
| } |
| - scenarios := []string{"renewals", "resume", "capture", "encoder", "loss"} |
| + scenarios := []string{"renewals", "resume", "capture", "encoder", "loss", "wedge", "wedge-sticky", "web-wedge", "web-wedge-sticky"} |
| if only := os.Getenv("SCREEN_SCENARIOS"); only != "" { |
| scenarios = strings.Split(only, ",") |
| } |
| for _, scenario := range scenarios { |
| t.Run(scenario, func(t *testing.T) { |
| rig := newBrowserRig(t) |
| - if scenario == "renewals" { |
| + soaking := scenario == "soak" || scenario == "web-soak" |
| + if scenario == "renewals" || soaking { |
| if err := rig.ctl.SetRelay(browserRelayContract()); err != nil { |
| t.Fatal(err) |
| } |
| rig.ctl.SetIceLife(20 * time.Second) |
| } |
| - if scenario == "renewals" || scenario == "resume" { |
| + if scenario == "renewals" || scenario == "resume" || soaking { |
| rig.ctl.SetStreamTimings(time.Second, 10*time.Second) |
| } |
| room, _ := rig.createMeeting(t) |
| - ctx, cancel := context.WithTimeout(t.Context(), 240*time.Second) |
| + limit := 240 * time.Second |
| + if soaking { |
| + // SCREEN_SOAK_SECONDS of soak, and the scenario around it. |
| + seconds, _ := strconv.Atoi(envOr("SCREEN_SOAK_SECONDS", "60")) |
| + limit = time.Duration(seconds)*time.Second + 180*time.Second |
| + } |
| + ctx, cancel := context.WithTimeout(t.Context(), limit) |
| defer cancel() |
| cmd := exec.CommandContext(ctx, "node", filepath.Join(root, "test/browser/native-screen.mjs"), rig.origin, room, binary, scenario) |
| cookies, _ := json.Marshal([]map[string]string{{"name": rig.cookieName, "value": rig.cookieB}}) |
| @@ -454,7 +494,7 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| if v, err := time.ParseDuration(os.Getenv("SCREEN_PAIR_SOAK")); err == nil { |
| soak = v |
| } |
| - scenarios := []string{"renewals", "capture", "encoder", "loss"} |
| + scenarios := []string{"renewals", "capture", "encoder", "loss", "wedge", "wedge-sticky", "reshare"} |
| if only := os.Getenv("SCREEN_SCENARIOS"); only != "" { |
| scenarios = strings.Split(only, ",") |
| } |
| @@ -466,9 +506,15 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| t.Skip("the sharing app predates the stall commands") |
| case scenario == "loss" && (!newA || !newB): |
| t.Skip("a stall report needs the new receiver and the new sharer") |
| + case strings.HasPrefix(scenario, "wedge") || scenario == "reshare": |
| + if !newA || !newB { |
| + t.Skip("the escalation needs the new receiver and the new sharer") |
| + } |
| + // Decoders the receiver can wedge (spec §15.22). |
| + t.Setenv("CALLS_DECODER_WEDGE", "1") |
| } |
| rig := newBrowserRig(t) |
| - if scenario == "renewals" { |
| + if scenario == "renewals" || scenario == "soak" { |
| if err := rig.ctl.SetRelay(browserRelayContract()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -501,6 +547,9 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| a.command(t, "screen-mode "+envOr("SCREEN_MODE", "balanced")) |
| if os.Getenv("SCREEN_PAIR_DISPLAY") == "1" && newA { |
| a.command(t, "screen-display 6") |
| + } else if os.Getenv("SCREEN_SOURCE") == "motion" { |
| + // The 1920-wide scrolling page (CALLS_SYNTHETIC_SCREEN_HEIGHT). |
| + a.command(t, "screen-motion") |
| } else { |
| a.command(t, "screen-on") |
| } |
| @@ -510,11 +559,11 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| // The watch: B's screen frames, once a second. |
| freezes, longest := 0, time.Duration(0) |
| watch := func(d time.Duration, label string) { |
| - last, moved, frozen := b.stat("screenFramesDecoded"), time.Now(), false |
| + last, moved, frozen := b.stat("screenFramesDecodedTotal"), time.Now(), false |
| camera := b.stat("framesDecoded") |
| for until := time.Now().Add(d); time.Now().Before(until); { |
| time.Sleep(time.Second) |
| - now := b.stat("screenFramesDecoded") |
| + now := b.stat("screenFramesDecodedTotal") |
| if now > last { |
| if frozen { |
| gap := time.Since(moved) |
| @@ -529,9 +578,10 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| } else if !frozen && time.Since(moved) > 3*time.Second { |
| frozen = true |
| s := a.sample().Stats |
| - t.Logf("FROZEN %s: no screen frame at b for %v (a: captured %d encoded %d sent %d, watchdog k%d e%d r%d a%d)", label, |
| + t.Logf("FROZEN %s: no screen frame at b for %v (a: captured %d encoded %d sent %d, watchdog k%d e%d r%d a%d rebuilds %d shares again %d)", label, |
| time.Since(moved).Round(time.Millisecond), s["screenCaptureDelivered"], s["screenFramesEncoded"], s["screenFramesSent"], |
| - s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], s["screenWatchdogCaptureRestarts"], s["screenWatchdogReAdds"]) |
| + s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], s["screenWatchdogCaptureRestarts"], s["screenWatchdogReAdds"], |
| + s["screenWatchdogRebuilds"], s["screenWatchdogReshares"]) |
| } |
| last = now |
| } |
| @@ -551,6 +601,61 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| t.Fatalf("%s: a=%+v b=%+v", label, a.sample(), b.sample()) |
| } |
| switch scenario { |
| + case "soak": |
| + // §15.22's search for the cause, between two apps: renewals |
| + // and resumes as `renewals`, with A's mode cycled and its |
| + // uplink capped and lifted every 45 s, for SCREEN_PAIR_SOAK. |
| + modes := []string{"balanced", "motion", "detail"} |
| + uplinks := []string{"0", "1500000", "600000", "0", "3000000", "900000"} |
| + done := make(chan struct{}) |
| + notices := map[string]bool{} |
| + var mu sync.Mutex |
| + go func() { |
| + step := 0 |
| + tick := time.NewTicker(45 * time.Second) |
| + defer tick.Stop() |
| + for { |
| + select { |
| + case <-done: |
| + return |
| + case <-tick.C: |
| + step++ |
| + a.command(t, "screen-mode "+modes[step%len(modes)]) |
| + a.command(t, "uplink "+uplinks[step%len(uplinks)]) |
| + t.Logf("soak step %d: mode %s, uplink %s", step, modes[step%len(modes)], uplinks[step%len(uplinks)]) |
| + } |
| + } |
| + }() |
| + go func() { |
| + for { |
| + select { |
| + case <-done: |
| + return |
| + case <-time.After(500 * time.Millisecond): |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + mu.Lock() |
| + notices[n] = true |
| + mu.Unlock() |
| + } |
| + } |
| + } |
| + }() |
| + watch(soak, "soak") |
| + close(done) |
| + a.command(t, "uplink 0") |
| + s := a.sample().Stats |
| + t.Logf("SOAK a: ICE restarts %d (b %d), stall reports %d, keyframes %d, fresh encoders %d, re-adds %d, rebuilds %d, shares again %d, given up %d, capture restarts %d; b's pictures attached again %d", |
| + s["iceRestarts"], b.stat("iceRestarts"), s["screenStallReports"], s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], |
| + s["screenWatchdogReAdds"], s["screenWatchdogRebuilds"], s["screenWatchdogReshares"], s["screenWatchdogFailures"], |
| + s["screenWatchdogCaptureRestarts"], b.stat("screenReattaches")) |
| + mu.Lock() |
| + t.Logf("SOAK notices at a: %q; %d freezes, longest %v", keys(notices), freezes, longest) |
| + mu.Unlock() |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b's diagnostics:\n%s", screenSections(b.diagnostics(t))) |
| + if s["screenWatchdogFailures"] > 0 { |
| + t.Error("a gave up on b's view") |
| + } |
| case "renewals": |
| watch(soak, "renewals") |
| t.Logf("ICE restarts: a started %d, b started %d", a.stat("iceRestarts"), b.stat("iceRestarts")) |
| @@ -592,12 +697,154 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| }, a, b) |
| waitNative(t, "b clears its report once frames move", func() bool { return b.stat("screenStalledHere") == 0 }, a, b) |
| recovered("b decodes after the keyframe") |
| + case "wedge": |
| + // B's screen decoder stops putting out pictures while packets |
| + // keep arriving (spec §15.22): keyframes cannot mend it and a |
| + // new transceiver can. Without a manual share again, B has to |
| + // be decoding again within 30 s, and A's notice has to have |
| + // said so honestly on the way. |
| + watch(5*time.Second, "before") |
| + keyframes := a.stat("screenWatchdogKeyframes") |
| + wedged := time.Now() |
| + b.command(t, "screen-decoder-wedge") |
| + waitNative(t, "b reports a's screen stalled", func() bool { return b.stat("screenStalledHere") == 1 }, a, b) |
| + notices := map[string]bool{} |
| + base := b.stat("screenFramesDecodedTotal") |
| + back := false |
| + for until := wedged.Add(30 * time.Second); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + notices[n] = true |
| + } |
| + if b.stat("screenFramesDecodedTotal") > base+15 { |
| + back = true |
| + break |
| + } |
| + } |
| + took := time.Since(wedged) |
| + t.Logf("b decoding again after %v: rebuilds %d, shares again %d, keyframes %d, b's pictures attached again %d", |
| + took.Round(time.Millisecond), a.stat("screenWatchdogRebuilds"), a.stat("screenWatchdogReshares"), |
| + a.stat("screenWatchdogKeyframes")-keyframes, b.stat("screenReattaches")) |
| + if !back { |
| + t.Logf("a:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b:\n%s", screenSections(b.diagnostics(t))) |
| + t.Fatalf("b's wedged screen was not restored within 30 s: a=%+v b=%+v", a.sample(), b.sample()) |
| + } |
| + if a.stat("screenWatchdogKeyframes") == keyframes { |
| + t.Error("keyframes were not tried first") |
| + } |
| + if a.stat("screenWatchdogRebuilds") == 0 { |
| + t.Error("b decoded again without a rebuild: the wedge did not hold") |
| + } |
| + waitNative(t, "b clears its report", func() bool { return b.stat("screenStalledHere") == 0 }, a, b) |
| + waitNative(t, "a's notice clears", func() bool { |
| + n := a.sample().ScreenRecovery |
| + if n != "" { |
| + notices[n] = true |
| + } |
| + return n == "" |
| + }, a, b) |
| + t.Logf("notices seen at a: %q", keys(notices)) |
| + if !anyHas(notices, "recovering") || !anyHas(notices, "Restarting your share for") { |
| + t.Errorf("the notice did not move from recovering to restarting: %q", keys(notices)) |
| + } |
| + watch(10*time.Second, "after the rebuild") |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b's diagnostics:\n%s", screenSections(b.diagnostics(t))) |
| + case "wedge-sticky": |
| + // Every screen decoder B makes is wedged, the new ones too: |
| + // nothing A does can mend it, and A has to say so rather than |
| + // sit at "recovering". |
| + watch(5*time.Second, "before") |
| + b.command(t, "screen-decoder-wedge sticky") |
| + notices := map[string]bool{} |
| + gaveUp := false |
| + for until := time.Now().Add(60 * time.Second); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + notices[n] = true |
| + } |
| + if a.stat("screenWatchdogFailures") > 0 && anyHas(map[string]bool{a.sample().ScreenRecovery: true}, "Couldn\u2019t restore") { |
| + gaveUp = true |
| + break |
| + } |
| + } |
| + t.Logf("notices seen at a: %q", keys(notices)) |
| + if !gaveUp { |
| + t.Fatalf("a never said it could not restore b's view: a=%+v", a.sample()) |
| + } |
| + if a.stat("screenWatchdogRebuilds") == 0 || a.stat("screenWatchdogReshares") == 0 { |
| + t.Errorf("a gave up without trying a rebuild and a share again: %+v", a.sample().Stats) |
| + } |
| + if !anyHas(notices, "Restarting your share for") { |
| + t.Errorf("the notice never said it was restarting: %q", keys(notices)) |
| + } |
| + // The wedge lifted, the view comes back and the notice goes. |
| + b.command(t, "screen-decoder-release") |
| + recoveredTotal := func() bool { |
| + base := b.stat("screenFramesDecodedTotal") |
| + return pollNative(20*time.Second, func() bool { return b.stat("screenFramesDecodedTotal") > base+10 }) |
| + } |
| + if !recoveredTotal() { |
| + t.Fatalf("b did not decode again once the wedge was lifted: %+v", b.sample()) |
| + } |
| + waitNative(t, "a's notice clears once b's view is back", func() bool { return a.sample().ScreenRecovery == "" }, a, b) |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + case "reshare": |
| + // The watchdog's share again, on its own: every connection's |
| + // screen stopped and added again from the same capture, and |
| + // the watcher decoding again within a bound, one tile. |
| + watch(5*time.Second, "before") |
| + base := b.stat("screenFramesDecodedTotal") |
| + started := time.Now() |
| + a.command(t, "screen-reshare-test") |
| + if !pollNative(15*time.Second, func() bool { return b.stat("screenFramesDecodedTotal") > base+30 }) { |
| + t.Fatalf("b did not decode again after the share again: a=%+v b=%+v", a.sample(), b.sample()) |
| + } |
| + t.Logf("b decoding again %v after the share again", time.Since(started).Round(time.Millisecond)) |
| + waitNative(t, "one screen at b", func() bool { return b.stat("remoteScreens") == 1 && len(b.sample().ScreenCaptions) == 1 }, a, b) |
| + if a.sample().SharingScreen == nil || !*a.sample().SharingScreen { |
| + t.Error("a stopped sharing") |
| + } |
| + watch(10*time.Second, "after the share again") |
| } |
| t.Logf("PASS %s: %d freezes, longest %v", scenario, freezes, longest) |
| }) |
| } |
| } |
| +// screenSections is the screen's part of "Copy call diagnostics": the |
| +// "## Screen" sections and nothing else. |
| +func screenSections(diag string) string { |
| + var out []string |
| + in := false |
| + for _, line := range strings.Split(diag, "\n") { |
| + if strings.HasPrefix(line, "## ") { |
| + in = strings.HasPrefix(line, "## Screen") |
| + } |
| + if in { |
| + out = append(out, line) |
| + } |
| + } |
| + return strings.Join(out, "\n") |
| +} |
| + |
| +func keys(m map[string]bool) []string { |
| + out := make([]string, 0, len(m)) |
| + for k := range m { |
| + out = append(out, k) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
| + |
| +func anyHas(m map[string]bool, part string) bool { |
| + for k := range m { |
| + if strings.Contains(k, part) { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| // pollNative is waitNative without the failure: whether ok held in time. |
| func pollNative(d time.Duration, ok func() bool) bool { |
| for until := time.Now().Add(d); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
diff --git a/native/README.md b/native/README.md| index 2f6f916..bfefb36 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -546,6 +546,43 @@ steps only if its own screen has stopped leaving too), and shows |
| **Copy audio diagnostics**). The apps restart ICE only when a relay credential is |
| due for renewal, not on every stream resume as 0.4.x did. |
| +**When keyframes are not enough** (spec §15.22, after Paul's "still seeing issues |
| +with screen sharing after a while", 30 Sep, the notice sitting at "recovering"): |
| +a receiver still reporting six seconds after the first keyframe gets its screen |
| +rebuilt — a new transceiver for that person alone, so a new encoder here and a new |
| +receiver and decoder there, at the same size and mode (a report never shrinks a |
| +Text and detail share) — and the notice says "Restarting your share for Samuel…". |
| +Still reported ten seconds later, everyone's screen is shared again from the same |
| +capture, with no picker: what Stop and Share did by hand. Ten seconds after that |
| +the notice says "Couldn't restore Samuel's view — try Stop and Share again". A |
| +rebuild at most once a minute per person, a share again at most once every five |
| +minutes, and neither unless frames actually left for that person since the last |
| +step. The receiver hands a stalled screen's picture to its view again four seconds |
| +in, logs what its statistics say is still arriving (packets, frames received, |
| +decoded, dropped, keyframes, PLI, the decoder), holds its report for up to three |
| +minutes, and counts any change in the decoded count as movement (a rebuilt screen's |
| +count starts again from zero, which 0.4.3 read as still stalled). The iPhone's |
| +broadcast repeats its last frame once a second while the screen is still, as the |
| +Mac's capture does, so a still iPhone screen is no longer reported as stalled. The |
| +browser does all of this too (its sharer's rebuild and share again included), and |
| +has **Copy call diagnostics** at the foot of its Devices panel. |
| + |
| +"Reconnecting…" is said only for a problem that lasts (spec §15.23): the |
| +signalling stream down 4 s, a connection `disconnected`/`failed` 3 s, or no |
| +audio for 2 s from someone unmuted — that one on their tile, "Reconnecting to |
| +Samuel…" — and it stays a second past the recovery. The stream's own renewal |
| +every few minutes is never said. `ReconnectWatch`; the `connection` log |
| +category. |
| + |
| +The gate for it wedges a receiver's screen decoder, which keyframes cannot mend: |
| +`screen-decoder-wedge` in `CallsInterop` (run with `CALLS_DECODER_WEDGE=1`; a |
| +wrapping decoder factory that only the gate turns on), and in Chromium an encoded |
| +transform that swallows every frame. `wedge`, `wedge-sticky` and `reshare` in |
| +`make native-pair`, and `wedge`, `wedge-sticky`, `web-wedge` and `web-wedge-sticky` |
| +in `make native-browser`, hold the ladder to restoring the view within 30 s, or to |
| +saying it cannot. `soak` and `web-soak` (`SCREEN_SOAK_SECONDS`) and the pair's |
| +`soak` (`SCREEN_PAIR_SOAK`) are the long shares, run by hand. |
| + |
| The numbers behind the modes come from `TestNativeMacScreenMotion` (in |
| `make native-browser`): the app shares a 1080p page scrolling at 30 fps with a |
| window dragged across it, and Chromium measures what arrives, with the app's |
diff --git a/native/apple/Sources/CallsInterop/Interop.swift b/native/apple/Sources/CallsInterop/Interop.swift| index fe9013d..21ef045 100644 |
| --- a/native/apple/Sources/CallsInterop/Interop.swift |
| +++ b/native/apple/Sources/CallsInterop/Interop.swift |
| @@ -56,6 +56,10 @@ struct Input: Decodable { |
| let input = try? JSONDecoder().decode(Input.self, from: Data(line.utf8)), |
| let url = URL(string: input.origin), url.scheme == "http", url.host == "127.0.0.1" |
| else { exit(2) } |
| + // Decoders the gate can wedge (spec §15.22), set before the call's |
| + // factory is made; only for the scenarios that ask, so every other |
| + // gate decodes exactly as the apps do. |
| + if ProcessInfo.processInfo.environment["CALLS_DECODER_WEDGE"] == "1" { DecoderWedge.enabled = true } |
| let call = Call(playAudio: false, audioInput: SyntheticVoice()) |
| if ProcessInfo.processInfo.environment["CALLS_SCREEN_WATCHDOG"] == "0" { Call.screenWatchdogEnabled = false } |
| // WebRTC's own log, filtered to the video encoder, into a file: for |
| @@ -216,6 +220,34 @@ struct Input: Decodable { |
| print("{\"screenFrozenForTest\":true}") |
| fflush(stdout) |
| } |
| + // §15.22: this app's screen decoders stop putting out pictures |
| + // while packets keep arriving — a wedge no keyframe mends and a |
| + // new transceiver does. `sticky` wedges the new ones too. |
| + if command == "screen-decoder-wedge" || command == "screen-decoder-wedge sticky" { |
| + let n = DecoderWedge.wedgeScreens(sticky: command.hasSuffix("sticky")) |
| + print("{\"decoderWedged\":\(n)}") |
| + fflush(stdout) |
| + } |
| + if command == "screen-decoder-release" { |
| + DecoderWedge.release() |
| + print("{\"decoderReleased\":true}") |
| + fflush(stdout) |
| + } |
| + // "Copy call diagnostics", as the panel would copy it. |
| + if command == "diagnostics" { |
| + let text = await call.audioDiagnosticsText() |
| + if let data = try? JSONSerialization.data(withJSONObject: ["diagnostics": text]), |
| + let line = String(data: data, encoding: .utf8) |
| + { |
| + print(line) |
| + fflush(stdout) |
| + } |
| + } |
| + if command == "screen-reshare-test" { |
| + await call.reshareScreenForTest() |
| + print("{\"reshared\":true}") |
| + fflush(stdout) |
| + } |
| if command == "screen-end" { |
| await Interop.screen?.end() |
| let on = await call.sharingScreen |
| @@ -232,7 +264,9 @@ struct Input: Decodable { |
| // Long enough for the member scenario, screens and all, and for the |
| // motion gate's three modes: the gate always says `leave` long before |
| // this runs out. |
| - for _ in 0..<400 { |
| + // CALLS_INTEROP_SECONDS for the soaks, which outlast that (§15.22). |
| + let life = Int(ProcessInfo.processInfo.environment["CALLS_INTEROP_SECONDS"] ?? "") ?? 400 |
| + for _ in 0..<life { |
| try? await Task.sleep(nanoseconds: 1_000_000_000) |
| let stats = await call.sampleStats() |
| if let data = try? JSONSerialization.data(withJSONObject: [ |
| @@ -249,6 +283,11 @@ struct Input: Decodable { |
| // place a screen says whose it is. |
| "screenCaptions": call.screens.map { screenCaption(call.name(of: $0)) }, |
| "sharingScreen": call.sharingScreen, |
| + // §15.22: the sharer's notice, word for word. |
| + "screenRecovery": call.screenRecovery, |
| + // §15.23: "Reconnecting…" said for the call, and on how many tiles. |
| + "reconnecting": call.reconnecting, |
| + "reconnectingPeers": call.reconnectingPeers.count, |
| "syntheticScreenFrames": Interop.screen?.produced ?? 0, |
| "cameraOpens": Interop.camera.opens, |
| "cameraStatus": { |
| @@ -285,6 +324,7 @@ struct Input: Decodable { |
| func stop() { mac.stop() } |
| var counts: ScreenCaptureCounts { mac.counts } |
| func restart() async -> Bool { await mac.restart() } |
| + var frameSize: (width: Int, height: Int)? { mac.frameSize } |
| } |
| // Render the shipping chat view against real received messages. This fixture |
diff --git a/native/apple/Sources/CallsInterop/SyntheticVideo.swift b/native/apple/Sources/CallsInterop/SyntheticVideo.swift| index 845c2c4..1ddf271 100644 |
| --- a/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| +++ b/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| @@ -129,6 +129,7 @@ import LiveKitWebRTC |
| let n = page.map { UInt64($0.produced) } ?? made |
| return ScreenCaptureCounts(heartbeats: n, delivered: n) |
| } |
| + var frameSize: (width: Int, height: Int)? { page != nil ? (ScrollingPage.width, ScrollingPage.height) : (640, 360) } |
| func restart() async -> Bool { |
| guard source != nil else { return false } |
| restarts += 1 |
| @@ -186,7 +187,10 @@ import LiveKitWebRTC |
| /// each frame is row copies into a pooled buffer, on a queue of its own. |
| final class ScrollingPage: @unchecked Sendable { |
| static let width = 1920 |
| - static let height = 1080 |
| + /// 1080 unless `CALLS_SYNTHETIC_SCREEN_HEIGHT` says otherwise: 1246 is a |
| + /// 16:10 Retina display as the Mac's capture fits it to 1920 wide, the |
| + /// size that meets H.264 level 3.1's block limit (spec §15.21, §15.22). |
| + static let height = Int(ProcessInfo.processInfo.environment["CALLS_SYNTHETIC_SCREEN_HEIGHT"] ?? "") ?? 1080 |
| private let rows = ScrollingPage.height * 2 |
| private let texture: [UInt8] |
| private var offset = 0 |
diff --git a/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift| index 01ff1a8..0ce8d13 100644 |
| --- a/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift |
| +++ b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift |
| @@ -1,6 +1,7 @@ |
| import CallsBroadcastKit |
| import CoreVideo |
| import Foundation |
| +import os |
| import LiveKitWebRTC |
| /// The iPhone's whole screen, arriving from the broadcast extension over |
| @@ -17,6 +18,7 @@ import LiveKitWebRTC |
| @MainActor public final class BroadcastScreenCapture: ScreenCapture { |
| private let connection: BroadcastConnection |
| private var stopped = false |
| + private var feed: StillScreenFeed? |
| /// Called after the share has ended on its own, once the call has heard. |
| /// The app uses it to leave a call it is only still in for the screen. |
| public var afterEnded: (@MainActor () -> Void)? |
| @@ -24,17 +26,9 @@ import LiveKitWebRTC |
| public init(_ connection: BroadcastConnection) { self.connection = connection } |
| public func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws { |
| - let capturer = RTCVideoCapturer(delegate: source) |
| - connection.onFrame = { pixels, orientation in |
| - // Stamped on arrival, on this device's clock, rather than with the |
| - // extension's: the two are separate processes, and the encoder only |
| - // needs the frames in order and spaced as they came. |
| - source.capturer( |
| - capturer, |
| - didCapture: RTCVideoFrame( |
| - buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: broadcastRotation(orientation), |
| - timeStampNs: Int64(DispatchTime.now().uptimeNanoseconds))) |
| - } |
| + let feed = StillScreenFeed(source: source) |
| + self.feed = feed |
| + connection.onFrame = { pixels, orientation in feed.frame(pixels, rotation: broadcastRotation(orientation)) } |
| connection.onClose = { [weak self] in |
| Task { @MainActor [weak self] in |
| guard let self, !self.stopped else { return } |
| @@ -46,8 +40,12 @@ import LiveKitWebRTC |
| connection.resume() |
| } |
| + public var counts: ScreenCaptureCounts { feed?.counts ?? ScreenCaptureCounts() } |
| + |
| public func stop() { |
| stopped = true |
| + feed?.stop() |
| + feed = nil |
| connection.onClose = nil |
| connection.onFrame = nil |
| connection.close() |
| @@ -68,3 +66,67 @@ func broadcastRotation(_ orientation: UInt16) -> RTCVideoRotation { |
| default: return ._0 |
| } |
| } |
| + |
| +/// The broadcast's frames into the call's source, and the last of them |
| +/// again once a second while the screen is still (2026-09-30). ReplayKit |
| +/// sends nothing at all for a screen that is not changing, and a receiver |
| +/// that sees no frame for four seconds reports the screen stalled (spec |
| +/// §15.21) — which, for a still iPhone screen, was a false alarm the |
| +/// sharer then tried to mend. The Mac's capture has repeated its last |
| +/// frame since 2026-09-28 (MacScreenCapture); this is the same rule. |
| +final class StillScreenFeed: @unchecked Sendable { |
| + static let repeatAfter: UInt64 = 1_000_000_000 |
| + |
| + private let source: RTCVideoSource |
| + private let capturer: RTCVideoCapturer |
| + private let queue = DispatchQueue(label: "calls.broadcast.feed", qos: .userInteractive) |
| + private var timer: DispatchSourceTimer? |
| + // Only touched on `queue`. |
| + private var last: (CVPixelBuffer, RTCVideoRotation)? |
| + private var lastSent: UInt64 = 0 |
| + private let tally = OSAllocatedUnfairLock(initialState: ScreenCaptureCounts(heartbeats: nil, delivered: 0)) |
| + var counts: ScreenCaptureCounts { tally.withLock { $0 } } |
| + |
| + init(source: RTCVideoSource) { |
| + self.source = source |
| + capturer = RTCVideoCapturer(delegate: source) |
| + let timer = DispatchSource.makeTimerSource(queue: queue) |
| + timer.schedule(deadline: .now() + .milliseconds(500), repeating: .milliseconds(500), leeway: .milliseconds(50)) |
| + timer.setEventHandler { [weak self] in self?.repeatIfIdle() } |
| + timer.resume() |
| + self.timer = timer |
| + } |
| + |
| + /// One frame from the extension. Stamped on arrival, on this device's |
| + /// clock, rather than with the extension's: the two are separate |
| + /// processes, and the encoder only needs the frames in order and spaced |
| + /// as they came. |
| + func frame(_ pixels: CVPixelBuffer, rotation: RTCVideoRotation) { |
| + queue.async { |
| + self.last = (pixels, rotation) |
| + self.deliver(pixels, rotation) |
| + } |
| + } |
| + |
| + func stop() { |
| + timer?.cancel() |
| + timer = nil |
| + queue.async { self.last = nil } |
| + } |
| + |
| + private func repeatIfIdle() { |
| + guard let (pixels, rotation) = last else { return } |
| + let now = DispatchTime.now().uptimeNanoseconds |
| + guard now >= lastSent, now - lastSent >= Self.repeatAfter else { return } |
| + deliver(pixels, rotation) |
| + } |
| + |
| + private func deliver(_ pixels: CVPixelBuffer, _ rotation: RTCVideoRotation) { |
| + let ns = max(DispatchTime.now().uptimeNanoseconds, lastSent + 1) |
| + lastSent = ns |
| + tally.withLock { $0.delivered += 1 } |
| + source.capturer( |
| + capturer, |
| + didCapture: RTCVideoFrame(buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: rotation, timeStampNs: Int64(ns))) |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index 80d24aa..56f4107 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -75,6 +75,26 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| /// `screenStalled` (spec §15.21). |
| private var screenReceiveWatch = ScreenReceiveWatch() |
| private var reportedStalled: [String] = [] |
| + /// Each arriving screen's statistics at the last look, for the stall |
| + /// log's "what moved" and for "Copy call diagnostics". |
| + private var screenReceived: [String: ScreenReceiveReport] = [:] |
| + /// Where each arriving screen stood when it last moved, so a stall can |
| + /// be described against it. |
| + private var screenReceivedMoving: [String: ScreenReceiveReport] = [:] |
| + private var screenReattaches = 0 |
| + /// "Reconnecting…", said only for a problem that lasts (spec §15.23): |
| + /// the whole call's, and the people whose tiles say it on their own. |
| + private var reconnectWatch = ReconnectWatch() |
| + @Published public private(set) var reconnecting = false |
| + @Published public private(set) var reconnectingPeers: Set<String> = [] |
| + /// The phase the notices are drawn for: the core's, except that its |
| + /// `reconnecting` is only said once ReconnectWatch says so, and is said |
| + /// for a second after it is over. |
| + public var noticePhase: String? { |
| + if reconnecting { return "reconnecting" } |
| + if state?.phase == "reconnecting" { return "admitted" } |
| + return state?.phase |
| + } |
| private var screenWatchFrozenUntil = 0.0 |
| @Published public private(set) var localTrack: RTCVideoTrack? |
| @Published public private(set) var muted = false |
| @@ -202,11 +222,28 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| } else { |
| lines.append("Not sharing") |
| } |
| + if sharingScreen { |
| + for peer in (state?.peers ?? []).map(\.peer) where screenWatchdog.escalation(of: peer) != .none { |
| + lines.append("recovering \(name(of: peer))'s view: \(screenWatchdog.escalation(of: peer))") |
| + } |
| + lines.append( |
| + "recoveries: \(screenWatchdog.keyframes) keyframes, \(screenWatchdog.encoderRecoveries) fresh encoders, \(screenWatchdog.reAdds) re-adds, \(screenWatchdog.rebuilds) rebuilds, \(screenWatchdog.reshares) shares again, \(screenWatchdog.failures) given up") |
| + } |
| + // The receiver's half: every screen arriving here, as its statistics |
| + // have it, whether or not it has stalled. |
| + for (peer, report) in screenReceived.sorted(by: { $0.key < $1.key }) { |
| + lines.append("from \(name(of: peer)): \(report.summary)") |
| + } |
| if !reportedStalled.isEmpty { |
| lines.append("stalled here: " + reportedStalled.map { name(of: $0) }.joined(separator: ", ")) |
| } |
| - let log = AudioDiagnosticsStore.screen.lines(last: 50) |
| - return [("Screen", lines), ("Screen log (last 50 lines)", log.isEmpty ? ["(nothing logged)"] : log)] |
| + if screenReattaches > 0 { lines.append("pictures attached again here: \(screenReattaches)") } |
| + let log = AudioDiagnosticsStore.screen.lines(last: AudioDiagnosticsStore.lineLimit) |
| + let connection = AudioDiagnosticsStore.connection.lines(last: 50) |
| + return [ |
| + ("Screen", lines), ("Screen log (last \(AudioDiagnosticsStore.lineLimit) lines)", log.isEmpty ? ["(nothing logged)"] : log), |
| + ("Connection log (last 50 lines)", connection.isEmpty ? ["(nothing logged)"] : connection), |
| + ] |
| } |
| private var ice: ICEConfig? |
| private var pendingMedia: [Message] = [] |
| @@ -504,6 +541,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| levelWatch = nil |
| screenReceiveWatch = ScreenReceiveWatch() |
| reportedStalled = [] |
| + screenReceived = [:] |
| + screenReceivedMoving = [:] |
| levels.reset() |
| reportedNoSound = [] |
| audioBytes = [:] |
| @@ -540,6 +579,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| guard levelWatch != nil, let state else { return } |
| let me = state.peer |
| var present = Set<String>() |
| + var people: [String: ReconnectWatch.Peer] = [:] |
| for person in state.peers where person.peer != me { |
| guard let reading = readings[person.peer] else { continue } |
| present.insert(person.peer) |
| @@ -547,15 +587,21 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| audioArriving[person.peer] = bytes > (audioBytes[person.peer] ?? 0) |
| audioBytes[person.peer] = bytes |
| } |
| + people[person.peer] = ReconnectWatch.Peer( |
| + connectionBad: reading.connectionBad, |
| + audioMissing: audioArriving[person.peer] == false && person.media?.muted != true) |
| + var notices = AudioWords.notices(name: person.name, media: person.media, me: me, muted: muted) |
| + if reconnectingPeers.contains(person.peer) { notices.insert(ReconnectWords.peer(person.name), at: 0) } |
| remotePeakSinceSample = max(remotePeakSinceSample, reading.peak) |
| levels.updatePeer( |
| person.peer, name: person.name, |
| sample: PeerAudioSample( |
| reported: person.media, connected: reading.connected, |
| arriving: audioArriving[person.peer] ?? false, level: reading.peak), |
| - notices: AudioWords.notices(name: person.name, media: person.media, me: me, muted: muted), |
| + notices: notices, |
| now: now) |
| } |
| + watchReconnect(now: now, people: people) |
| levels.settle(present: present) |
| for id in Array(audioBytes.keys) where !present.contains(id) { |
| audioBytes.removeValue(forKey: id) |
| @@ -564,6 +610,23 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| reportNoSound(levels.noSoundFrom) |
| } |
| + /// One look by ReconnectWatch (spec §15.23), and the words it says. |
| + private func watchReconnect(now: Double, people: [String: ReconnectWatch.Peer]) { |
| + reconnectWatch.observe(at: now, streamDown: state?.phase == "reconnecting", people: people) |
| + if reconnectingPeers != reconnectWatch.peers { |
| + for id in reconnectWatch.peers.subtracting(reconnectingPeers) { CallLog.info("reconnecting to \(id.prefix(6)): shown") } |
| + reconnectingPeers = reconnectWatch.peers |
| + } |
| + guard reconnecting != reconnectWatch.call else { return } |
| + reconnecting = reconnectWatch.call |
| + CallLog.info(reconnecting ? "reconnecting: shown (a problem that lasted)" : "reconnecting: cleared") |
| + switch state?.phase { |
| + case "reconnecting": status = reconnecting ? ReconnectWords.call : "In the meeting" |
| + case "admitted": status = reconnecting ? ReconnectWords.call : "In the meeting" |
| + default: break |
| + } |
| + } |
| + |
| /// §15.20's receiving half: who is reaching this app as silence. The |
| /// core keeps only roster peers and reports on a change; this sends only |
| /// on a change too, in order, off the main thread. |
| @@ -640,10 +703,12 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| KnockNotice.announce( |
| knocks.arrivals(waiting: state.waiting, visible: KnockNotice.appIsFrontmost)) |
| switch state.phase { |
| - case "admitted": status = "In the meeting" |
| + case "admitted": status = reconnecting ? ReconnectWords.call : "In the meeting" |
| case "waiting": status = "Waiting to be let in" |
| case "full": status = "The meeting is full. Try joining again shortly." |
| - case "reconnecting": status = "Reconnecting…" |
| + // Said only once it has lasted (ReconnectWatch, spec §15.23): a |
| + // stream renewal is a reconnect of half a second nobody should see. |
| + case "reconnecting": if reconnecting { status = ReconnectWords.call } |
| case "ended": |
| status = |
| [ |
| @@ -823,20 +888,41 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| case .reAdd(let peer): |
| ScreenLog.info("screen re-added for \(ScreenWatchdog.short(peer))") |
| media.reAddScreen(peer) |
| - // The new transceiver's track arrives unlabelled; say it again. |
| - Task { @MainActor [weak self] in |
| - try? await Task.sleep(nanoseconds: 2_000_000_000) |
| - guard let self, self.sharingScreen else { return } |
| - self.send(type: "share", to: "", gen: 0, payload: json(["on": true]) ?? "") |
| - } |
| + sayShareAgain() |
| + case .rebuild(let peer): |
| + ScreenLog.info("screen rebuilt for \(ScreenWatchdog.short(peer)): new transceiver, same size and mode") |
| + media.reAddScreen(peer) |
| + sayShareAgain() |
| + case .reshare: |
| + ScreenLog.info("sharing again for everyone, from the same capture") |
| + await media.reshareScreen() |
| + sayShareAgain() |
| } |
| } |
| screenWatchStatus = screenWatchdog.status |
| if let (peer, worst) = reports.min(by: { $0.value.framesPerSecond < $1.value.framesPerSecond }) { |
| screenDetail = worst.detail(since: previousReports[peer], watchdog: screenWatchdog.status) |
| } |
| - let names = stalledBy.map { name(of: $0) }.sorted() |
| - screenRecovery = ScreenWords.recovering(names: names) |
| + var recovering: [String] = [] |
| + var restarting: [String] = [] |
| + var failed: [String] = [] |
| + for peer in stalledBy.sorted(by: { name(of: $0) < name(of: $1) }) { |
| + switch screenWatchdog.escalation(of: peer) { |
| + case .rebuilt, .reshared: restarting.append(name(of: peer)) |
| + case .failed: failed.append(name(of: peer)) |
| + case .none, .keyframes: recovering.append(name(of: peer)) |
| + } |
| + } |
| + screenRecovery = ScreenWords.notice(recovering: recovering, restarting: restarting, failed: failed) |
| + } |
| + |
| + /// A new transceiver's track arrives unlabelled; say it again. |
| + private func sayShareAgain() { |
| + Task { @MainActor [weak self] in |
| + try? await Task.sleep(nanoseconds: 2_000_000_000) |
| + guard let self, self.sharingScreen else { return } |
| + self.send(type: "share", to: "", gen: 0, payload: json(["on": true]) ?? "") |
| + } |
| } |
| private func endScreenWatch() { |
| screenWatch?.cancel() |
| @@ -851,30 +937,67 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| /// The receiver's watchdog, once a second from the level watch: whose |
| /// screens have stopped moving here, reported as `screenStalled` when |
| - /// that changes (spec §15.21). |
| + /// that changes (spec §15.21), each stall logged with what the |
| + /// statistics say is still arriving, and the picture attached again |
| + /// four seconds in (spec §15.22). |
| private func watchRemoteScreens(now: Double) async { |
| var stalled: [String] = [] |
| - if !remoteScreens.isEmpty { |
| - var frames = await media.remoteScreenFrames() |
| + if !screens.isEmpty { |
| + let reports = await media.remoteScreenReports() |
| + var frames = reports.mapValues(\.framesDecoded) |
| // The gate's frozen picture: the counts held where they were. |
| if now < screenWatchFrozenUntil { frames = frames.mapValues { _ in 0 } } |
| let sharing = Set(screens) |
| + let before = screenReceived |
| + screenReceived = reports.filter { sharing.contains($0.key) } |
| stalled = screenReceiveWatch.observe(at: now, decoded: frames.filter { sharing.contains($0.key) }) |
| + for (peer, report) in screenReceived where !stalled.contains(peer) { |
| + if before[peer].map({ $0.framesDecoded != report.framesDecoded || $0.mid != report.mid }) ?? true { |
| + screenReceivedMoving[peer] = report |
| + } |
| + } |
| + for peer in stalled where !reportedStalled.contains(peer) { |
| + let report = screenReceived[peer] ?? ScreenReceiveReport() |
| + let moving = screenReceivedMoving[peer] |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) stalled here: reporting it; since it last moved \(report.moved(since: moving)) — \(report.diagnosis(since: moving))") |
| + ScreenLog.info("screen from \(ScreenWatchdog.short(peer)): \(report.summary)") |
| + } |
| + for peer in screenReceiveWatch.reattach { |
| + let report = screenReceived[peer] ?? ScreenReceiveReport() |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) still stalled: attaching its picture again; \(report.moved(since: screenReceivedMoving[peer]))") |
| + reattachScreen(peer) |
| + } |
| } else { |
| screenReceiveWatch = ScreenReceiveWatch() |
| + screenReceived = [:] |
| + screenReceivedMoving = [:] |
| } |
| guard stalled != reportedStalled, let client = bridge else { return } |
| - for peer in stalled where !reportedStalled.contains(peer) { |
| - ScreenLog.info("screen from \(ScreenWatchdog.short(peer)) stalled here: asking for a keyframe") |
| - } |
| for peer in reportedStalled where !stalled.contains(peer) { |
| - ScreenLog.info("screen from \(ScreenWatchdog.short(peer)) moving here again") |
| + let report = screenReceived[peer] |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) moving here again\(report.map { ": " + $0.summary } ?? "")") |
| } |
| reportedStalled = stalled |
| let body = json(stalled) ?? "[]" |
| noSoundQueue.async { try? client.setScreenStalled(body) } |
| } |
| + /// A stalled screen's picture dropped and attached again: a new renderer |
| + /// on the same track, for a picture stopped on this side. |
| + private func reattachScreen(_ peer: String) { |
| + guard let track = remoteScreens[peer] else { return } |
| + screenReattaches += 1 |
| + remoteScreens.removeValue(forKey: peer) |
| + Task { @MainActor [weak self] in |
| + try? await Task.sleep(nanoseconds: 150_000_000) |
| + guard let self, self.screens.contains(peer), self.remoteScreens[peer] == nil else { return } |
| + self.remoteScreens[peer] = track |
| + } |
| + } |
| + |
| /// Stops sharing, from every way out: the button, the system's own stop |
| /// control, and the capture ending under us. |
| public func stopSharingScreen() { |
| @@ -915,6 +1038,13 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| } |
| /// For the gate's encoder stall (see MediaEngine). |
| @_spi(Fixtures) public func stallScreenEncoderForTest() { media.stallScreenEncoderForTest() } |
| + /// For the gate: the watchdog's share again (spec §15.22), on demand. |
| + @_spi(Fixtures) public func reshareScreenForTest() async { |
| + guard sharingScreen else { return } |
| + ScreenLog.info("sharing again for everyone, from the same capture (asked for by the gate)") |
| + await media.reshareScreen() |
| + sayShareAgain() |
| + } |
| public func sampleStats() async -> [String: UInt64] { |
| var values = await media.stats() |
| values["hasICE"] = ice == nil ? 0 : 1 |
| @@ -952,6 +1082,10 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| values["screenWatchdogReAdds"] = UInt64(screenWatchdog.reAdds) |
| values["screenStallReports"] = UInt64(screenWatchdog.stallReports) |
| values["screenStalledHere"] = UInt64(reportedStalled.count) |
| + values["screenWatchdogRebuilds"] = UInt64(screenWatchdog.rebuilds) |
| + values["screenWatchdogReshares"] = UInt64(screenWatchdog.reshares) |
| + values["screenWatchdogFailures"] = UInt64(screenWatchdog.failures) |
| + values["screenReattaches"] = UInt64(screenReattaches) |
| values["iceRestarts"] = media.iceRestarts |
| values["screenKeyframes"] = screenReport?.keyFramesEncoded ?? 0 |
| if let capture = media.screenCaptureCounts { |
| @@ -1006,6 +1140,8 @@ extension Call { |
| self.standing = standing |
| self.state = try? JSONDecoder().decode(RoomState.self, from: Data(json.utf8)) |
| self.status = status |
| + // A fixture's reconnecting is one that has lasted (spec §15.23). |
| + self.reconnecting = self.state?.phase == "reconnecting" |
| self.screens = screens |
| self.sharingScreen = sharingScreen |
| self.muted = muted |
diff --git a/native/apple/Sources/CallsKit/CallAudio.swift b/native/apple/Sources/CallsKit/CallAudio.swift| index f3d53df..dd281b5 100644 |
| --- a/native/apple/Sources/CallsKit/CallAudio.swift |
| +++ b/native/apple/Sources/CallsKit/CallAudio.swift |
| @@ -158,7 +158,9 @@ public struct ProcessingReport: Equatable, Sendable { |
| denoiser: CaptureDenoiser? |
| ) -> Parts { |
| let encoder = RTCDefaultVideoEncoderFactory() |
| - let decoder = RTCDefaultVideoDecoderFactory() |
| + // WebRTC's own, except in the gate's decoder-wedge scenarios. |
| + let decoder: RTCVideoDecoderFactory = |
| + DecoderWedge.enabled ? DecoderWedge.factory() : RTCDefaultVideoDecoderFactory() |
| let denoiser = denoiser ?? CaptureDenoiser(suppressor: CallAudio.makeSuppressor()) |
| #if os(iOS) |
| if let audioDevice { |
diff --git a/native/apple/Sources/CallsKit/Capture.swift b/native/apple/Sources/CallsKit/Capture.swift| index 3b773c2..15ec995 100644 |
| --- a/native/apple/Sources/CallsKit/Capture.swift |
| +++ b/native/apple/Sources/CallsKit/Capture.swift |
| @@ -104,11 +104,18 @@ extension VideoCapture { |
| /// source — after the watchdog has seen it stall. False when this kind |
| /// of capture cannot, or the attempt failed. |
| func restart() async -> Bool |
| + /// The size the capture delivers frames at, when it knows before the |
| + /// first frame: the send policy's H.264 level cap (ScreenSendPolicy) |
| + /// then holds from the first frame rather than from the first |
| + /// statistics a second later, a second in which a browser's level-3.1 |
| + /// encoder refuses every frame of a Motion share (spec §15.22). |
| + var frameSize: (width: Int, height: Int)? { get } |
| } |
| extension ScreenCapture { |
| public var counts: ScreenCaptureCounts { ScreenCaptureCounts() } |
| public func restart() async -> Bool { false } |
| + public var frameSize: (width: Int, height: Int)? { nil } |
| } |
| /// A screen capture's running totals (see ScreenWatchdog). |
diff --git a/native/apple/Sources/CallsKit/ChatView.swift b/native/apple/Sources/CallsKit/ChatView.swift| index 8f99bd6..814b481 100644 |
| --- a/native/apple/Sources/CallsKit/ChatView.swift |
| +++ b/native/apple/Sources/CallsKit/ChatView.swift |
| @@ -35,7 +35,7 @@ import SwiftUI |
| proxy.scrollTo("chat-bottom", anchor: .bottom) |
| } |
| } |
| - if call.state?.phase == "reconnecting" { |
| + if call.reconnecting { |
| Text("Reconnecting. You can send again when the call reconnects.") |
| .font(.footnote).foregroundStyle(.secondary) |
| } |
diff --git a/native/apple/Sources/CallsKit/DecoderWedge.swift b/native/apple/Sources/CallsKit/DecoderWedge.swift| new file mode 100644 |
| index 0000000..bb775a0 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/DecoderWedge.swift |
| @@ -0,0 +1,123 @@ |
| +import Foundation |
| +import LiveKitWebRTC |
| +import os |
| + |
| +/// For the gate only (spec §15.22): a receiver whose screen decoder has |
| +/// wedged. Packets keep arriving, frames keep being handed to the decoder, |
| +/// and nothing comes out — which no keyframe mends, because it is this |
| +/// decoder instance that is stuck, and which a new transceiver does mend, |
| +/// because the far end's new stream gets a decoder of its own. That is the |
| +/// shape of a stall that "keyframes only" (0.4.3) left sitting at |
| +/// "recovering". |
| +/// |
| +/// Off unless the interop executable turns it on before the call is |
| +/// built; the apps never do, and then the call's factory is WebRTC's own. |
| +@_spi(Fixtures) public enum DecoderWedge { |
| + /// Wrap the call's decoders so they can be wedged. Read once, when the |
| + /// call's factory is made. |
| + nonisolated(unsafe) public static var enabled = false |
| + |
| + |
| + private static let live = OSAllocatedUnfairLock(initialState: [WeakDecoder]()) |
| + private static let sticky = OSAllocatedUnfairLock(initialState: false) |
| + |
| + static func register(_ decoder: WedgeableDecoder) { |
| + live.withLock { list in |
| + list.removeAll { $0.decoder == nil } |
| + list.append(WeakDecoder(decoder: decoder)) |
| + } |
| + } |
| + |
| + /// Wedges the decoder now putting out the largest picture, which in the |
| + /// gate is always the screen (its cameras are 320×240 to 640×480, and a |
| + /// screen even at its smallest under a thin link is wider). `sticky` |
| + /// wedges every decoder made afterwards too — only a screen's rebuild or |
| + /// share again makes one mid-call — for a receiver no rebuild can mend, |
| + /// the gate's "could not restore". Returns how many were wedged. |
| + @discardableResult public static func wedgeScreens(sticky: Bool = false) -> Int { |
| + self.sticky.withLock { $0 = sticky } |
| + let decoders = live.withLock { $0.compactMap(\.decoder) } |
| + guard let largest = decoders.map(\.area).max(), largest > 0 else { return 0 } |
| + var count = 0 |
| + for d in decoders where d.area == largest { |
| + d.wedge() |
| + count += 1 |
| + } |
| + return count |
| + } |
| + |
| + /// Every wedge lifted, and nothing wedged from now on. |
| + public static func release() { |
| + sticky.withLock { $0 = false } |
| + for d in live.withLock({ $0.compactMap(\.decoder) }) { d.unwedge() } |
| + } |
| + |
| + static var isSticky: Bool { sticky.withLock { $0 } } |
| + |
| + static func factory() -> RTCVideoDecoderFactory { WedgeableDecoderFactory() } |
| +} |
| + |
| +private struct WeakDecoder { |
| + weak var decoder: WedgeableDecoder? |
| +} |
| + |
| +final class WedgeableDecoderFactory: NSObject, RTCVideoDecoderFactory { |
| + private let inner = RTCDefaultVideoDecoderFactory() |
| + |
| + func createDecoder(_ info: RTCVideoCodecInfo) -> RTCVideoDecoder? { |
| + guard let decoder = inner.createDecoder(info) else { return nil } |
| + // Only the VideoToolbox decoders are Objective-C all the way down. |
| + // WebRTC's own VP8, VP9 and AV1 come back as wrapped native decoders, |
| + // which the factory unwraps and never calls through this protocol, so |
| + // they are handed over as they are and cannot be wedged. |
| + guard ["H264", "H265"].contains(info.name.uppercased()) else { return decoder } |
| + let wrapped = WedgeableDecoder(decoder) |
| + if DecoderWedge.isSticky { wrapped.wedge() } |
| + DecoderWedge.register(wrapped) |
| + return wrapped |
| + } |
| + |
| + func supportedCodecs() -> [RTCVideoCodecInfo] { inner.supportedCodecs() } |
| +} |
| + |
| +final class WedgeableDecoder: NSObject, RTCVideoDecoder, @unchecked Sendable { |
| + private let inner: RTCVideoDecoder |
| + private struct State { |
| + var wedged = false |
| + var area = 0 |
| + } |
| + private let state = OSAllocatedUnfairLock(initialState: State()) |
| + |
| + init(_ inner: RTCVideoDecoder) { self.inner = inner } |
| + |
| + var area: Int { state.withLock { $0.area } } |
| + func wedge() { state.withLock { $0.wedged = true } } |
| + func unwedge() { state.withLock { $0.wedged = false } } |
| + |
| + func setCallback(_ callback: @escaping RTCVideoDecoderCallback) { |
| + inner.setCallback { [weak self] frame in |
| + guard let self else { return } |
| + let area = Int(frame.width) * Int(frame.height) |
| + let wedged = self.state.withLock { s -> Bool in |
| + s.area = area |
| + return s.wedged |
| + } |
| + if !wedged { callback(frame) } |
| + } |
| + } |
| + |
| + func startDecode(withNumberOfCores numberOfCores: Int32) -> Int { |
| + inner.startDecode(withNumberOfCores: numberOfCores) |
| + } |
| + |
| + func release() -> Int { inner.release() } |
| + |
| + func decode( |
| + _ encodedImage: RTCEncodedImage, missingFrames: Bool, codecSpecificInfo info: RTCCodecSpecificInfo?, |
| + renderTimeMs: Int64 |
| + ) -> Int { |
| + inner.decode(encodedImage, missingFrames: missingFrames, codecSpecificInfo: info, renderTimeMs: renderTimeMs) |
| + } |
| + |
| + func implementationName() -> String { inner.implementationName() } |
| +} |
diff --git a/native/apple/Sources/CallsKit/MacFullWindowCall.swift b/native/apple/Sources/CallsKit/MacFullWindowCall.swift| index 1e79c08..cdfc3f6 100644 |
| --- a/native/apple/Sources/CallsKit/MacFullWindowCall.swift |
| +++ b/native/apple/Sources/CallsKit/MacFullWindowCall.swift |
| @@ -304,7 +304,7 @@ |
| private var notices: [CallNotice] { |
| CallNotices.notices( |
| - phase: call.state?.phase, status: call.status, sharingScreen: call.sharingScreen, |
| + phase: call.noticePhase, status: call.status, sharingScreen: call.sharingScreen, |
| screenNote: call.screenNote, floor: call.floor, screenRecovery: call.screenRecovery) |
| } |
diff --git a/native/apple/Sources/CallsKit/MacScreenCapture.swift b/native/apple/Sources/CallsKit/MacScreenCapture.swift| index 2ee3714..b5b38d8 100644 |
| --- a/native/apple/Sources/CallsKit/MacScreenCapture.swift |
| +++ b/native/apple/Sources/CallsKit/MacScreenCapture.swift |
| @@ -108,7 +108,10 @@ |
| self.frames = frames |
| self.filter = filter |
| self.source = source |
| + let configured = Self.configuration(for: filter) |
| + frameSize = (configured.width, configured.height) |
| } |
| + public private(set) var frameSize: (width: Int, height: Int)? |
| /// The same content into the same source, on a capture of its own: |
| /// a fresh stream with a fresh pool of surfaces. What stopping and |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 609fe81..92857a4 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -139,9 +139,32 @@ import LiveKitWebRTC |
| throw CancellationError() |
| } |
| screenCapture = capture |
| + screenSourceInUse = source |
| + if let size = capture.frameSize, size.width > 0, size.height > 0 { screenSize = size } |
| localScreen = factory.videoTrack(with: source, trackId: "screen") |
| for peer in peers.values { peer.addScreen() } |
| } |
| + /// The source the screen's capture is writing into, for a share again |
| + /// from the same capture. |
| + private var screenSourceInUse: RTCVideoSource? |
| + private var reshares = 0 |
| + |
| + /// Stop and Share, without the picker (spec §15.22): every connection's |
| + /// screen transceiver stopped and renegotiated, a fresh track on the |
| + /// same source, the capture started again on the same content, and each |
| + /// connection given the new track as soon as it settles. New |
| + /// transceivers mean new encoders here and new receivers and decoders at |
| + /// every far end, which is what the hand-made stop and share did. |
| + func reshareScreen() async { |
| + guard let capture = screenCapture, let source = screenSourceInUse, localScreen != nil else { return } |
| + reshares += 1 |
| + for peer in peers.values { peer.removeScreen() } |
| + localScreen = factory.videoTrack(with: source, trackId: "screen-\(reshares)") |
| + if !(await capture.restart()) { ScreenLog.info("share again: the capture carries on as it was") } |
| + guard screenCapture === capture else { return } |
| + // Any connection already stable takes it now; the rest as they settle. |
| + for peer in peers.values { peer.addScreen() } |
| + } |
| /// Stops sending the screen: every connection's screen transceiver is |
| /// stopped and renegotiated, so the receivers' tracks *end*, and the |
| @@ -150,6 +173,7 @@ import LiveKitWebRTC |
| screenAttempt += 1 |
| guard let capture = screenCapture else { return } |
| screenCapture = nil |
| + screenSourceInUse = nil |
| localScreen = nil |
| for peer in peers.values { |
| peer.removeScreen() |
| @@ -181,24 +205,36 @@ import LiveKitWebRTC |
| for peer in peers.values { peer.setScreenActive(false) } |
| } |
| - /// Frames decoded so far from each peer's screen, for the peers whose |
| - /// screen is arriving here (the receiver's watchdog). |
| - func remoteScreenFrames() async -> [String: UInt64] { |
| - var out: [String: UInt64] = [:] |
| + /// Each peer's screen as it arrives here, for the peers whose screen is |
| + /// arriving (the receiver's watchdog, and "Copy call diagnostics"). |
| + func remoteScreenReports() async -> [String: ScreenReceiveReport] { |
| + var out: [String: ScreenReceiveReport] = [:] |
| for (id, peer) in peers { |
| guard let pc = peer.pc, peer.receivingScreen else { continue } |
| let report = await PeerCalls.statistics(pc) |
| - var decoded: UInt64 = 0 |
| + var reading = ScreenReceiveReport() |
| for stat in report.statistics.values where stat.type == "inbound-rtp" { |
| let mid = stat.values["mid"] as? String ?? "" |
| guard peer.isScreen(mid: mid) else { continue } |
| - decoded += (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + reading.add(stat.values) |
| } |
| - out[id] = decoded |
| + reading.mid = peer.screenMidHere |
| + out[id] = reading |
| } |
| return out |
| } |
| + /// Frames decoded from screens here, never going backwards: a rebuilt |
| + /// screen arrives on a new receiver whose own count starts again, and |
| + /// the gate's "decoding again" has to see its frames as more. |
| + private var screenDecodedSeen: [String: UInt64] = [:] |
| + private(set) var screenDecodedTotal: UInt64 = 0 |
| + private func tallyScreenDecoded(_ peer: String, _ now: UInt64) { |
| + let last = screenDecodedSeen[peer] ?? 0 |
| + screenDecodedTotal += now >= last ? now - last : now |
| + screenDecodedSeen[peer] = now |
| + } |
| + |
| /// Puts every connection's screen and camera senders where the policy |
| /// wants them; a no-op for any that already are. |
| func applySendPolicy() { |
| @@ -296,10 +332,12 @@ import LiveKitWebRTC |
| "insertedSamplesForDeceleration": 0, "removedSamplesForAcceleration": 0, |
| "jitterBufferDelayMs": 0, |
| ] |
| - for peer in peers.values { |
| + for (id, peer) in peers { |
| guard let pc = peer.pc else { continue } |
| let ownScreen = await peer.ownScreenMid() |
| let report = await PeerCalls.statistics(pc) |
| + var screenDecoded: UInt64 = 0 |
| + defer { tallyScreenDecoded(id, screenDecoded) } |
| for stat in report.statistics.values { |
| let kind = (stat.values["kind"] as? String) ?? (stat.values["mediaType"] as? String) ?? "" |
| let mid = stat.values["mid"] as? String ?? "" |
| @@ -310,8 +348,9 @@ import LiveKitWebRTC |
| // for a camera that had stopped. Told apart by mid, which is |
| // what the browser's own gate does. |
| let field = peer.isScreen(mid: mid) ? "screenFramesDecoded" : "framesDecoded" |
| - values[field, default: 0] += |
| - (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + let decoded = (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + values[field, default: 0] += decoded |
| + if peer.isScreen(mid: mid) { screenDecoded += decoded } |
| } |
| if kind == "audio" { |
| values["audioBytesReceived", default: 0] += |
| @@ -341,6 +380,7 @@ import LiveKitWebRTC |
| } |
| } |
| } |
| + values["screenFramesDecodedTotal"] = screenDecodedTotal |
| return values |
| } |
| @@ -361,7 +401,8 @@ import LiveKitWebRTC |
| (pc.iceConnectionState, stats ? pc.receivers.filter { $0.track?.kind == "audio" } : []) |
| } |
| var reading = PeerAudioReading( |
| - peak: peak, buffers: buffers, connected: state == .connected || state == .completed) |
| + peak: peak, buffers: buffers, connected: state == .connected || state == .completed, |
| + connectionBad: state == .disconnected || state == .failed) |
| if stats { |
| var bytes: UInt64 = 0 |
| for receiver in receivers { |
| @@ -421,6 +462,8 @@ struct PeerAudioReading: Equatable, Sendable { |
| var peak: Float |
| var buffers: Int |
| var connected: Bool |
| + /// ICE `disconnected` or `failed` (spec §15.23's three seconds). |
| + var connectionBad = false |
| var bytes: UInt64? |
| } |
| @@ -640,6 +683,7 @@ struct PeerAudioReading: Equatable, Sendable { |
| } |
| } |
| func isScreen(mid: String) -> Bool { !mid.isEmpty && mid == screenMid } |
| + var screenMidHere: String { screenMid } |
| /// The mid our screen went out on, once negotiated: read on the audio |
| /// queue (a transceiver's mid is a hop to WebRTC's signalling thread), |
| /// and kept once known. |
diff --git a/native/apple/Sources/CallsKit/ReconnectWatch.swift b/native/apple/Sources/CallsKit/ReconnectWatch.swift| new file mode 100644 |
| index 0000000..023c902 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/ReconnectWatch.swift |
| @@ -0,0 +1,125 @@ |
| +import Foundation |
| +import os |
| + |
| +/// When "Reconnecting…" is said, and to whom (spec §15.23). |
| +/// |
| +/// Paul, 2026-09-30: "During that call I also saw 'Reconnecting' appear |
| +/// occasionally over the video, but only for a second … I didn't notice any |
| +/// interruption in the call." The core puts the call in its `reconnecting` |
| +/// phase the moment its signalling stream ends, and the stream ends on |
| +/// purpose every few minutes (its renewal) and is resumed half a second |
| +/// later; the media never notices. So the words wait until a problem has |
| +/// lasted: |
| +/// |
| +/// - the signalling stream down for `streamAfter` (4 s) — the whole call; |
| +/// - a connection `disconnected` or `failed` for `connectionAfter` (3 s), |
| +/// or no audio arriving from someone who is not muted for `audioAfter` |
| +/// (2 s) — that person's tile, "Reconnecting to Samuel…", unless it is |
| +/// everyone (two or more), which is the whole call again. |
| +/// |
| +/// Once said, the words stay `holdAfter` (1 s) past the recovery, so they |
| +/// do not flicker. Pure: fed the facts as they are, with the time. |
| +struct ReconnectWatch: Equatable { |
| + static let streamAfter = 4.0 |
| + static let connectionAfter = 3.0 |
| + static let audioAfter = 2.0 |
| + static let holdAfter = 1.0 |
| + |
| + /// One person's connection as it is now. |
| + struct Peer: Equatable { |
| + /// ICE `disconnected` or `failed`. |
| + var connectionBad = false |
| + /// No audio packets arrived at the last look, and they are not muted. |
| + var audioMissing = false |
| + } |
| + |
| + private var streamDownSince: Double? |
| + private var connectionBadSince: [String: Double] = [:] |
| + private var audioMissingSince: [String: Double] = [:] |
| + private var callClearSince: Double? |
| + private var peerClearSince: [String: Double] = [:] |
| + |
| + /// Whether the whole call is said to be reconnecting. |
| + private(set) var call = false |
| + /// The people whose tiles say "Reconnecting to …". |
| + private(set) var peers: Set<String> = [] |
| + |
| + init() {} |
| + |
| + /// One look. `streamDown` is the core's `reconnecting` phase; `people` |
| + /// everyone in the call but you. |
| + mutating func observe(at now: Double, streamDown: Bool, people: [String: Peer]) { |
| + streamDownSince = streamDown ? (streamDownSince ?? now) : nil |
| + for id in Array(connectionBadSince.keys) where people[id] == nil { connectionBadSince.removeValue(forKey: id) } |
| + for id in Array(audioMissingSince.keys) where people[id] == nil { audioMissingSince.removeValue(forKey: id) } |
| + var troubled: Set<String> = [] |
| + for (id, p) in people { |
| + connectionBadSince[id] = p.connectionBad ? (connectionBadSince[id] ?? now) : nil |
| + audioMissingSince[id] = p.audioMissing ? (audioMissingSince[id] ?? now) : nil |
| + let connection = connectionBadSince[id].map { now - $0 >= Self.connectionAfter } ?? false |
| + let audio = audioMissingSince[id].map { now - $0 >= Self.audioAfter } ?? false |
| + if connection || audio { troubled.insert(id) } |
| + } |
| + let stream = streamDownSince.map { now - $0 >= Self.streamAfter } ?? false |
| + let everyone = troubled.count >= 2 && troubled.count == people.count |
| + let wantCall = stream || everyone |
| + let wantPeers = wantCall ? [] : troubled |
| + |
| + if wantCall { |
| + call = true |
| + callClearSince = nil |
| + } else if call { |
| + let since = callClearSince ?? now |
| + callClearSince = since |
| + if now - since >= Self.holdAfter { |
| + call = false |
| + callClearSince = nil |
| + } |
| + } |
| + |
| + var shown = peers.filter { people[$0] != nil } |
| + for id in wantPeers { |
| + shown.insert(id) |
| + peerClearSince.removeValue(forKey: id) |
| + } |
| + for id in shown where !wantPeers.contains(id) { |
| + // Covered by the whole call's words, or better: held, then gone. |
| + if wantCall { |
| + shown.remove(id) |
| + peerClearSince.removeValue(forKey: id) |
| + continue |
| + } |
| + let since = peerClearSince[id] ?? now |
| + peerClearSince[id] = since |
| + if now - since >= Self.holdAfter { |
| + shown.remove(id) |
| + peerClearSince.removeValue(forKey: id) |
| + } |
| + } |
| + for id in Array(peerClearSince.keys) where !shown.contains(id) { peerClearSince.removeValue(forKey: id) } |
| + peers = shown |
| + } |
| +} |
| + |
| +/// The words (spec §15.23). |
| +public enum ReconnectWords { |
| + public static let call = "Reconnecting…" |
| + public static func peer(_ name: String) -> String { "Reconnecting to \(name)\u{2026}" } |
| +} |
| + |
| +/// `log show --predicate 'subsystem == "net.amadan.oficina.calls" && category == "connection"' --info`, |
| +/// and the last lines in "Copy call diagnostics": when "Reconnecting…" was |
| +/// said and cleared. |
| +enum CallLog { |
| + static let logger = Logger(subsystem: "net.amadan.oficina.calls", category: "connection") |
| + |
| + static func info(_ message: String) { |
| + logger.info("\(message, privacy: .public)") |
| + AudioDiagnosticsStore.connection.log("info", message) |
| + } |
| +} |
| + |
| +extension AudioDiagnosticsStore { |
| + /// The connection log's lines, kept apart from the others. |
| + public static let connection = AudioDiagnosticsStore() |
| +} |
diff --git a/native/apple/Sources/CallsKit/ScreenWatchdog.swift b/native/apple/Sources/CallsKit/ScreenWatchdog.swift| index 8286145..dbcd5db 100644 |
| --- a/native/apple/Sources/CallsKit/ScreenWatchdog.swift |
| +++ b/native/apple/Sources/CallsKit/ScreenWatchdog.swift |
| @@ -60,6 +60,29 @@ enum ScreenRecovery: Equatable, Sendable { |
| /// What stopping and sharing again did: this connection's screen |
| /// transceiver stopped and a new one added, renegotiated. |
| case reAdd(String) |
| + /// A receiver's report that keyframes did not answer: this connection's |
| + /// screen sending path built again — a new transceiver, so a new encoder |
| + /// and, at the far end, a new receiver and decoder — at the same size |
| + /// and mode. What stopping and sharing again does, for that one person. |
| + case rebuild(String) |
| + /// Still reported after that: every connection's screen stopped and |
| + /// added again from the same capture (no picker), which is exactly what |
| + /// Stop and Share did by hand. |
| + case reshare |
| +} |
| + |
| +/// Where the answer to one receiver's report has got to (the sharer's |
| +/// notice says it in words: see ScreenWords.notice). |
| +enum ScreenEscalation: Equatable, Sendable { |
| + case none |
| + /// Keyframes asked for since this time. |
| + case keyframes(since: Double) |
| + /// That peer's screen rebuilt at this time. |
| + case rebuilt(at: Double) |
| + /// Everyone's screen shared again at this time. |
| + case reshared(at: Double) |
| + /// Nothing more to try: the person is told to stop and share again. |
| + case failed |
| } |
| /// The sharer's watchdog: pure, fed once a second while sharing. |
| @@ -83,6 +106,17 @@ enum ScreenRecovery: Equatable, Sendable { |
| /// (the report is a few seconds old when it lands), three times and then |
| /// a rest — and the fuller steps only if the screen has stopped leaving |
| /// here too. |
| +/// - **A receiver keeps saying so** (spec §15.22, Paul 2026-09-30: "still |
| +/// seeing issues with screen sharing after a while", the notice sat at |
| +/// "recovering"). Keyframes cannot mend a decoder that has wedged at the |
| +/// far end, or a receiver that has stopped taking this stream at all; |
| +/// a new transceiver can. Still reported six seconds after the first |
| +/// keyframe: that connection's screen is rebuilt, at the same size and |
| +/// mode (a report alone never shrinks a Text and detail share). Still |
| +/// reported ten seconds after that: everyone's screen is shared again |
| +/// from the same capture. Ten seconds more and the sharer is told it |
| +/// could not be restored. A rebuild at most once a minute per person, |
| +/// a share again at most once every five minutes. |
| struct ScreenWatchdog: Equatable { |
| static let captureStallAfter = 3.0 |
| static let captureBackoffFirst = 5.0 |
| @@ -100,6 +134,14 @@ struct ScreenWatchdog: Equatable { |
| /// and that is not a stall (the motion gate, 2026-09-28). A receiver's |
| /// report is still answered then. |
| static let warmUp = 10.0 |
| + /// §15.22's escalation: how long a report may outlast the first |
| + /// keyframe, then the rebuild, then the share again; and how often each |
| + /// of the last two may happen. |
| + static let rebuildAfter = 6.0 |
| + static let reshareAfter = 10.0 |
| + static let giveUpAfter = 10.0 |
| + static let rebuildEvery = 60.0 |
| + static let reshareEvery = 300.0 |
| struct Peer: Equatable { |
| var last = ScreenSendCounts() |
| @@ -112,7 +154,17 @@ struct ScreenWatchdog: Equatable { |
| var reported = false |
| var restingUntil = 0.0 |
| var since = 0.0 |
| + var escalation = ScreenEscalation.none |
| + var lastRebuild: Double? |
| + /// Frames sent on this connection since the escalation last moved on: |
| + /// a receiver can only be failing to decode frames that were sent. A |
| + /// screen the link is holding back, or one with nothing new to send, |
| + /// is not taken further. |
| + var sentSinceStage: UInt64 = 0 |
| } |
| + /// Frames that must leave for a receiver before its report is taken |
| + /// further than keyframes. |
| + static let escalateAfterSent: UInt64 = 3 |
| private(set) var peers: [String: Peer] = [:] |
| private var heartbeats: UInt64? |
| @@ -128,6 +180,10 @@ struct ScreenWatchdog: Equatable { |
| private(set) var encoderRecoveries = 0 |
| private(set) var reAdds = 0 |
| private(set) var stallReports = 0 |
| + private(set) var rebuilds = 0 |
| + private(set) var reshares = 0 |
| + private(set) var failures = 0 |
| + private var lastReshare: Double? |
| /// One line on what the watchdog is doing, for the device panel. |
| private(set) var status = "watching" |
| @@ -177,6 +233,7 @@ struct ScreenWatchdog: Equatable { |
| // all, so the steps do not start again from nothing on the new one. |
| for (id, counts) in sending { |
| var peer = peers[id] ?? Peer(last: counts, movedAt: now, deliveredAt: delivered, since: now) |
| + peer.sentSinceStage += counts.sent >= peer.last.sent ? counts.sent - peer.last.sent : counts.sent |
| if counts.encoded < peer.last.encoded || counts.sent < peer.last.sent { |
| // A new transceiver's counters, starting again from nothing. |
| peer.last = counts |
| @@ -238,20 +295,123 @@ struct ScreenWatchdog: Equatable { |
| } |
| peers[id] = peer |
| } |
| + out += escalate(at: now, stalledBy: stalledBy, negotiated: Set(sending.keys)) |
| status = describe() |
| return out |
| } |
| + /// §15.22: a report that keyframes have not answered, taken further. Run |
| + /// over every peer with a record, whether or not its screen is |
| + /// negotiated this second — a rebuilt connection has none for a moment, |
| + /// and a rebuild that never comes back must still end in words. |
| + private mutating func escalate( |
| + at now: Double, stalledBy: Set<String>, negotiated: Set<String> |
| + ) -> [ScreenRecovery] { |
| + var out: [ScreenRecovery] = [] |
| + var reshare = false |
| + for id in peers.keys.sorted() { |
| + guard var peer = peers[id] else { continue } |
| + let reported = stalledBy.contains(id) |
| + // Frames went to this receiver since the last step, or there is no |
| + // screen negotiated with it at all (a rebuild that never came back). |
| + let owed = peer.sentSinceStage >= Self.escalateAfterSent || !negotiated.contains(id) |
| + let before = peer.escalation |
| + defer { |
| + if peers[id]?.escalation != before { peers[id]?.sentSinceStage = 0 } |
| + } |
| + if !reported { |
| + if peer.escalation != .none { |
| + switch peer.escalation { |
| + case .keyframes: ScreenLog.info("\(Self.short(id))'s view restored after keyframes") |
| + case .rebuilt: ScreenLog.info("\(Self.short(id))'s view restored after the rebuild") |
| + case .reshared: ScreenLog.info("\(Self.short(id))'s view restored after sharing again") |
| + case .failed: ScreenLog.info("\(Self.short(id))'s view came back after all") |
| + case .none: break |
| + } |
| + peer.escalation = .none |
| + } |
| + peers[id] = peer |
| + continue |
| + } |
| + let canRebuild = peer.lastRebuild.map { now - $0 >= Self.rebuildEvery } ?? true |
| + let canReshare = reshare || (lastReshare.map { now - $0 >= Self.reshareEvery } ?? true) |
| + switch peer.escalation { |
| + case .none: |
| + peer.escalation = .keyframes(since: now) |
| + case .keyframes(let since) where now - since >= Self.rebuildAfter && owed: |
| + if canRebuild { |
| + out.append(.rebuild(id)) |
| + rebuilds += 1 |
| + peer.lastRebuild = now |
| + peer.escalation = .rebuilt(at: now) |
| + ScreenLog.info( |
| + "\(Self.short(id)) still reports a stall \(Int(now - since)) s after keyframes: rebuilding its screen") |
| + } else if canReshare { |
| + reshare = true |
| + peer.escalation = .reshared(at: now) |
| + ScreenLog.info("\(Self.short(id)) still reports a stall, rebuilt under a minute ago: sharing again") |
| + } else { |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall and every step is resting: giving up") |
| + } |
| + case .rebuilt(let at) where now - at >= Self.reshareAfter && owed: |
| + if canReshare { |
| + reshare = true |
| + peer.escalation = .reshared(at: now) |
| + ScreenLog.info("\(Self.short(id)) still reports a stall \(Int(now - at)) s after the rebuild: sharing again") |
| + } else { |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall after the rebuild, shared again under five minutes ago: giving up") |
| + } |
| + case .reshared(let at) where now - at >= Self.giveUpAfter && owed: |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall \(Int(now - at)) s after sharing again: giving up") |
| + default: |
| + break |
| + } |
| + peers[id] = peer |
| + } |
| + if reshare { |
| + out.append(.reshare) |
| + reshares += 1 |
| + lastReshare = now |
| + // One share again answers every report in flight. |
| + for (id, peer) in peers where stalledBy.contains(id) { |
| + switch peer.escalation { |
| + case .keyframes, .rebuilt: |
| + peers[id]?.escalation = .reshared(at: now) |
| + peers[id]?.sentSinceStage = 0 |
| + default: break |
| + } |
| + } |
| + } |
| + return out |
| + } |
| + |
| + /// Where the answer to each reporting peer has got to, for the notice. |
| + func escalation(of peer: String) -> ScreenEscalation { peers[peer]?.escalation ?? .none } |
| + |
| /// Whether anyone's view of the screen is being recovered right now. |
| var recovering: Bool { captureStalled || peers.values.contains { $0.step > 0 || $0.reported } } |
| private func describe() -> String { |
| if captureStalled { return "capture stalled, restarting (\(captureRestarts))" } |
| + let stages = peers.values.map(\.escalation) |
| + if stages.contains(.failed) { return "a receiver's view could not be restored" } |
| + if stages.contains(where: { if case .reshared = $0 { return true } else { return false } }) { |
| + return "shared again for everyone" |
| + } |
| + if stages.contains(where: { if case .rebuilt = $0 { return true } else { return false } }) { |
| + return "rebuilt for a receiver" |
| + } |
| if let (_, p) = peers.first(where: { $0.value.step > 0 }) { |
| return ["", "keyframe asked for", "encoder recreated", "re-added"][min(p.step, 3)] |
| } |
| if peers.values.contains(where: \.reported) { return "a receiver reports a stall" } |
| - let total = captureRestarts + keyframes + encoderRecoveries + reAdds |
| + let total = captureRestarts + keyframes + encoderRecoveries + reAdds + rebuilds + reshares |
| return total == 0 ? "watching" : "watching, \(total) recover\(total == 1 ? "y" : "ies") so far" |
| } |
| @@ -261,21 +421,37 @@ struct ScreenWatchdog: Equatable { |
| /// The receiver's watchdog: whose screens have stopped moving here. |
| /// |
| /// A peer's screen is stalled when nothing new has been decoded from it |
| -/// for four seconds while it says it is sharing. Both apps and the browser |
| -/// repeat a still screen at least once a second, so four seconds without |
| -/// a frame is not a still screen. The list is reported as `screenStalled` |
| -/// (spec §15.21) and the sharer recovers; it clears as soon as frames |
| -/// move, and a report is never held for more than a minute — a screen |
| -/// the sharer cannot recover is not made into a report that never ends. |
| +/// for four seconds while it says it is sharing. Every sender repeats a |
| +/// still screen at least once a second (the iPhone's broadcast too, since |
| +/// 2026-09-30), so four seconds without a frame is not a still screen. The |
| +/// list is reported as `screenStalled` (spec §15.21) and the sharer |
| +/// recovers; it clears as soon as frames move. |
| +/// |
| +/// Any change in the count is movement, a smaller one included: a screen |
| +/// the sharer has rebuilt arrives on a new receiver whose count starts |
| +/// again from nothing (2026-09-30 — before this, a rebuilt screen that was |
| +/// moving again stayed "stalled" here until its new count passed the old |
| +/// one, and the sharer's notice with it). |
| +/// |
| +/// Four seconds into a stall the tile's picture is also dropped and |
| +/// attached again, once (spec §15.22): a renderer that has stopped, on |
| +/// this side, is this side's to mend. A report is held for at most three |
| +/// minutes from the last frame, long enough for the sharer to try |
| +/// everything and say so, and no longer. |
| struct ScreenReceiveWatch: Equatable { |
| static let stallAfter = 4.0 |
| - static let giveUpAfter = 60.0 |
| + static let reattachAfter = 4.0 |
| + static let giveUpAfter = 180.0 |
| struct Seen: Equatable { |
| var decoded: UInt64 |
| var movedAt: Double |
| + var reattached = false |
| } |
| private(set) var seen: [String: Seen] = [:] |
| + /// The peers whose picture should be attached again now: set by each |
| + /// look, and each stall's only once. |
| + private(set) var reattach: [String] = [] |
| init() {} |
| @@ -284,18 +460,26 @@ struct ScreenReceiveWatch: Equatable { |
| mutating func observe(at now: Double, decoded: [String: UInt64]) -> [String] { |
| for id in seen.keys where decoded[id] == nil { seen.removeValue(forKey: id) } |
| var stalled: [String] = [] |
| + reattach = [] |
| for (id, count) in decoded { |
| - guard var s = seen[id], count <= s.decoded else { |
| + guard var s = seen[id], count == s.decoded else { |
| seen[id] = Seen(decoded: count, movedAt: now) |
| continue |
| } |
| - s.decoded = count |
| - seen[id] = s |
| let quiet = now - s.movedAt |
| if quiet >= Self.stallAfter && quiet < Self.giveUpAfter { stalled.append(id) } |
| + if quiet >= Self.stallAfter + Self.reattachAfter && !s.reattached && quiet < Self.giveUpAfter { |
| + s.reattached = true |
| + reattach.append(id) |
| + } |
| + seen[id] = s |
| } |
| + reattach.sort() |
| return stalled.sorted() |
| } |
| + |
| + /// How long each stalled peer's screen has gone without a frame. |
| + func quiet(at now: Double) -> [String: Double] { seen.mapValues { now - $0.movedAt } } |
| } |
| /// The words for the sharer's quiet notice. |
| @@ -310,4 +494,108 @@ public enum ScreenWords { |
| default: return "\(names.count) people\u{2019}s views of your screen stalled \u{2014} recovering" |
| } |
| } |
| + |
| + /// The notice as the escalation moves on (spec §15.22): "recovering", |
| + /// then "Restarting your share for Samuel…", then either nothing or |
| + /// "Couldn’t restore Samuel’s view — try Stop and Share again". Worst |
| + /// news first when several people are at different steps. |
| + public static func notice(recovering: [String], restarting: [String], failed: [String]) -> String { |
| + var parts: [String] = [] |
| + if !failed.isEmpty { |
| + let whose: String |
| + switch failed.count { |
| + case 1: whose = "\(failed[0])\u{2019}s view" |
| + case 2: whose = "\(failed[0])\u{2019}s and \(failed[1])\u{2019}s views" |
| + default: whose = "\(failed.count) people\u{2019}s views" |
| + } |
| + parts.append("Couldn\u{2019}t restore \(whose) \u{2014} try Stop and Share again") |
| + } |
| + if !restarting.isEmpty { |
| + let who: String |
| + switch restarting.count { |
| + case 1: who = restarting[0] |
| + case 2: who = "\(restarting[0]) and \(restarting[1])" |
| + default: who = "\(restarting.count) people" |
| + } |
| + parts.append("Restarting your share for \(who)\u{2026}") |
| + } |
| + let stalled = Self.recovering(names: recovering) |
| + if !stalled.isEmpty { parts.append(stalled) } |
| + return parts.joined(separator: ". ") |
| + } |
| +} |
| + |
| +/// One peer's screen as it arrives here, from the receiver's statistics |
| +/// (spec §15.22): enough to tell the ways a screen stops apart. Packets |
| +/// arriving and frames assembled but none decoded is a decoder that has |
| +/// wedged; nothing arriving at all is the sender, or the path; frames |
| +/// decoded and a picture that does not move is the renderer. |
| +struct ScreenReceiveReport: Equatable, Sendable { |
| + var packetsReceived: UInt64 = 0 |
| + var packetsLost: Int64 = 0 |
| + var bytesReceived: UInt64 = 0 |
| + var framesReceived: UInt64 = 0 |
| + var framesDecoded: UInt64 = 0 |
| + var framesDropped: UInt64 = 0 |
| + var keyFramesDecoded: UInt64 = 0 |
| + var pliCount: UInt64 = 0 |
| + var firCount: UInt64 = 0 |
| + var nackCount: UInt64 = 0 |
| + var freezeCount: UInt64 = 0 |
| + var width = 0 |
| + var height = 0 |
| + var decoder = "" |
| + /// When the last packet arrived, in the statistics' own milliseconds. |
| + var lastPacketAt = 0.0 |
| + var mid = "" |
| + |
| + init() {} |
| + |
| + /// One inbound-rtp report's values added in (a screen has one). |
| + mutating func add(_ v: [String: NSObject]) { |
| + func n(_ key: String) -> UInt64 { (v[key] as? NSNumber)?.uint64Value ?? 0 } |
| + packetsReceived += n("packetsReceived") |
| + packetsLost += (v["packetsLost"] as? NSNumber)?.int64Value ?? 0 |
| + bytesReceived += n("bytesReceived") |
| + framesReceived += n("framesReceived") |
| + framesDecoded += n("framesDecoded") |
| + framesDropped += n("framesDropped") |
| + keyFramesDecoded += n("keyFramesDecoded") |
| + pliCount += n("pliCount") |
| + firCount += n("firCount") |
| + nackCount += n("nackCount") |
| + freezeCount += n("freezeCount") |
| + width = max(width, Int(n("frameWidth"))) |
| + height = max(height, Int(n("frameHeight"))) |
| + if let name = v["decoderImplementation"] as? String, !name.isEmpty { decoder = name } |
| + lastPacketAt = max(lastPacketAt, (v["lastPacketReceivedTimestamp"] as? NSNumber)?.doubleValue ?? 0) |
| + } |
| + |
| + /// What changed between two looks, in words: "packets +120, frames |
| + /// received +30, decoded +0, dropped +0, keyframes +0, PLI +2". |
| + func moved(since before: ScreenReceiveReport?) -> String { |
| + guard let b = before, b.mid == mid else { return summary } |
| + func d(_ a: UInt64, _ z: UInt64) -> String { a >= z ? "+\(a - z)" : "new \(a)" } |
| + return "packets \(d(packetsReceived, b.packetsReceived)), frames received \(d(framesReceived, b.framesReceived)), " |
| + + "decoded \(d(framesDecoded, b.framesDecoded)), dropped \(d(framesDropped, b.framesDropped)), " |
| + + "keyframes \(d(keyFramesDecoded, b.keyFramesDecoded)), PLI \(d(pliCount, b.pliCount)), " |
| + + "NACK \(d(nackCount, b.nackCount))" |
| + } |
| + |
| + /// The running totals, one line. |
| + var summary: String { |
| + "mid \(mid.isEmpty ? "-" : mid), \(width)x\(height), decoder \(decoder.isEmpty ? "unknown" : decoder); " |
| + + "packets \(packetsReceived) (lost \(packetsLost)), frames received \(framesReceived), " |
| + + "decoded \(framesDecoded), dropped \(framesDropped), keyframes \(keyFramesDecoded), " |
| + + "PLI \(pliCount), FIR \(firCount), NACK \(nackCount), freezes \(freezeCount)" |
| + } |
| + |
| + /// The verdict the numbers point to, for a screen that has stopped. |
| + func diagnosis(since before: ScreenReceiveReport?) -> String { |
| + guard let b = before, b.mid == mid else { return "no earlier reading" } |
| + if packetsReceived == b.packetsReceived { return "no packets arriving (the sender or the path)" } |
| + if framesReceived == b.framesReceived { return "packets arriving, no frames assembled (loss, or waiting on a keyframe)" } |
| + if framesDecoded == b.framesDecoded { return "frames arriving, none decoded (the decoder)" } |
| + return "decoding" |
| + } |
| } |
diff --git a/native/apple/Sources/CallsMac/Resources/MacIcon.png b/native/apple/Sources/CallsMac/Resources/MacIcon.png| index f0de939..1ba703e 100644 |
| Binary files a/native/apple/Sources/CallsMac/Resources/MacIcon.png and b/native/apple/Sources/CallsMac/Resources/MacIcon.png differ |
diff --git a/native/apple/Tests/CallsKitTests/ReconnectWatchTests.swift b/native/apple/Tests/CallsKitTests/ReconnectWatchTests.swift| new file mode 100644 |
| index 0000000..8d656ff |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/ReconnectWatchTests.swift |
| @@ -0,0 +1,102 @@ |
| +import XCTest |
| + |
| +@testable import CallsKit |
| + |
| +/// "Reconnecting…" only for a problem that lasts (spec §15.23). Paul, |
| +/// 2026-09-30: "I saw 'Reconnecting' appear occasionally over the video, |
| +/// but only for a second … I didn't notice any interruption." |
| +final class ReconnectWatchTests: XCTestCase { |
| + private typealias Peer = ReconnectWatch.Peer |
| + private let fine = Peer() |
| + |
| + /// Runs the watch at 5 Hz from `from` to `to` seconds, and returns the |
| + /// times at which the whole call's words were showing. |
| + private func run( |
| + _ watch: inout ReconnectWatch, from: Double, to: Double, |
| + stream: (Double) -> Bool, people: (Double) -> [String: Peer] |
| + ) -> (call: [Double], peers: [Double: Set<String>]) { |
| + var call: [Double] = [] |
| + var peers: [Double: Set<String>] = [:] |
| + var t = from |
| + while t < to { |
| + watch.observe(at: t, streamDown: stream(t), people: people(t)) |
| + if watch.call { call.append(t) } |
| + if !watch.peers.isEmpty { peers[t] = watch.peers } |
| + t = (t * 10 + 2).rounded() / 10 |
| + } |
| + return (call, peers) |
| + } |
| + |
| + func testARoutineStreamRenewalIsNeverShown() { |
| + // The stream ends on purpose and is back half a second later, over and |
| + // over, and a slower resume of three and a half seconds as well. |
| + var watch = ReconnectWatch() |
| + let seen = run( |
| + &watch, from: 0, to: 60, |
| + stream: { t in (10..<10.6).contains(t) || (30..<33.5).contains(t) || (50..<50.4).contains(t) }, |
| + people: { _ in ["sam": self.fine] }) |
| + XCTAssertEqual(seen.call, []) |
| + XCTAssertTrue(seen.peers.isEmpty) |
| + } |
| + |
| + func testAStreamDownFourSecondsIsShownAndHeldASecondPastRecovery() { |
| + var watch = ReconnectWatch() |
| + let seen = run(&watch, from: 0, to: 20, stream: { (5..<12).contains($0) }, people: { _ in ["sam": self.fine] }) |
| + XCTAssertEqual(seen.call.first, 9, "down at 5, said at 9") |
| + XCTAssertEqual(seen.call.last, 12.8, "back at 12, said until a second after") |
| + } |
| + |
| + func testShortBlipsOnOnePersonAreNeverShown() { |
| + var watch = ReconnectWatch() |
| + let seen = run( |
| + &watch, from: 0, to: 30, stream: { _ in false }, |
| + people: { t in |
| + ["sam": Peer(connectionBad: (5..<7.5).contains(t), audioMissing: (15..<16.5).contains(t))] |
| + }) |
| + XCTAssertEqual(seen.call, []) |
| + XCTAssertTrue(seen.peers.isEmpty, "\(seen.peers)") |
| + } |
| + |
| + func testOnePersonsLastingTroubleIsSaidOnTheirTileAlone() { |
| + var watch = ReconnectWatch() |
| + let seen = run( |
| + &watch, from: 0, to: 30, stream: { _ in false }, |
| + people: { t in ["sam": Peer(connectionBad: (5..<15).contains(t)), "ana": self.fine] }) |
| + XCTAssertEqual(seen.call, [], "not the whole call") |
| + let times = seen.peers.keys.sorted() |
| + XCTAssertEqual(times.first, 8, "disconnected at 5, said at 8") |
| + XCTAssertEqual(times.last, 15.8, "held a second past recovery") |
| + XCTAssertTrue(seen.peers.values.allSatisfy { $0 == ["sam"] }) |
| + } |
| + |
| + func testAudioThatStopsFromSomeoneUnmutedIsSaidAfterTwoSeconds() { |
| + var watch = ReconnectWatch() |
| + let seen = run( |
| + &watch, from: 0, to: 20, stream: { _ in false }, |
| + people: { t in ["sam": Peer(audioMissing: (5..<10).contains(t)), "ana": self.fine] }) |
| + XCTAssertEqual(seen.peers.keys.sorted().first, 7) |
| + } |
| + |
| + func testEveryonesTroubleIsTheWholeCall() { |
| + var watch = ReconnectWatch() |
| + let seen = run( |
| + &watch, from: 0, to: 20, stream: { _ in false }, |
| + people: { t in |
| + let bad = (5..<12).contains(t) |
| + return ["sam": Peer(connectionBad: bad), "ana": Peer(connectionBad: bad)] |
| + }) |
| + XCTAssertEqual(seen.call.first, 8) |
| + XCTAssertTrue(seen.peers.isEmpty, "not two tiles as well") |
| + } |
| + |
| + func testTheWords() { |
| + XCTAssertEqual(ReconnectWords.call, "Reconnecting…") |
| + XCTAssertEqual(ReconnectWords.peer("Samuel"), "Reconnecting to Samuel…") |
| + } |
| + |
| + func testTheNoticeFollowsTheWatchNotThePhase() { |
| + let notices = CallNotices.notices( |
| + phase: "admitted", status: "In the meeting", sharingScreen: false, screenNote: "", floor: false) |
| + XCTAssertFalse(notices.contains { $0.kind == .reconnecting }) |
| + } |
| +} |
diff --git a/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift b/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift| index 3e125e3..85aa5c4 100644 |
| --- a/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift |
| +++ b/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift |
| @@ -83,6 +83,25 @@ final class ScreenWatchdogTests: XCTestCase { |
| XCTAssertEqual(dog.reAdds, 1) |
| } |
| + func testAnEncoderStoppedForOnePersonIsMendedForThemAlone() { |
| + // Each connection has an encoder of its own: the browser's (H.264 at |
| + // the level Chromium answers) can refuse frames the app's (High 5.2) |
| + // takes. The steps go to the stopped one only. |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<20 { |
| + let delivered = UInt64(t * 30) |
| + let sending = [ |
| + "browser": ScreenSendCounts(encoded: t < 5 ? delivered : 120, sent: t < 5 ? delivered : 120, keyframes: 1), |
| + "app": ScreenSendCounts(encoded: delivered, sent: delivered, keyframes: 1), |
| + ] |
| + for a in dog.observe(at: Double(t), capture: counts(delivered, delivered), sending: sending, stalledBy: []) { |
| + log.append((t, a)) |
| + } |
| + } |
| + XCTAssertEqual(log.map(\.1), [.keyframe("browser"), .recreateEncoder("browser"), .reAdd("browser")]) |
| + } |
| + |
| func testAnEncoderThatRecoversAfterTheKeyframeIsLeftAlone() { |
| var dog = ScreenWatchdog() |
| var actions: [ScreenRecovery] = [] |
| @@ -126,7 +145,7 @@ final class ScreenWatchdogTests: XCTestCase { |
| } |
| } |
| - func testAReceiversReportWhileTheScreenLeavesNormallyGetsKeyframesOnly() { |
| + func testAReceiversReportWhileTheScreenLeavesNormallyGetsKeyframesFirstAndNeverAShrink() { |
| var dog = ScreenWatchdog() |
| var log: [(Int, ScreenRecovery)] = [] |
| for t in 0..<20 { |
| @@ -136,8 +155,12 @@ final class ScreenWatchdogTests: XCTestCase { |
| log.append((t, a)) |
| } |
| } |
| - XCTAssertEqual(log.map(\.1), [.keyframe("vicky"), .keyframe("vicky"), .keyframe("vicky")]) |
| - XCTAssertEqual(log.map(\.0), [3, 7, 11]) |
| + let keyframes = log.filter { $0.1 == .keyframe("vicky") } |
| + XCTAssertEqual(keyframes.map(\.0), [3, 7, 11]) |
| + // What comes after the keyframes (spec §15.22) is a rebuild at the |
| + // same size, never a smaller encoder. |
| + XCTAssertFalse(log.contains { $0.1 == .recreateEncoder("vicky") }) |
| + XCTAssertEqual(log.first?.1, .keyframe("vicky")) |
| XCTAssertEqual(dog.stallReports, 1, "one report, however long it lasts") |
| XCTAssertTrue(dog.recovering) |
| } |
| @@ -182,17 +205,197 @@ final class ScreenWatchdogTests: XCTestCase { |
| } |
| } |
| + // MARK: - A receiver that keeps reporting (spec §15.22) |
| + |
| + private func escalating( |
| + _ range: Range<Int>, stalledBy: (Int) -> Set<String>, sent: (Int) -> [String: ScreenSendCounts]? = { _ in nil }, |
| + dog: inout ScreenWatchdog |
| + ) -> [(Int, ScreenRecovery)] { |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in range { |
| + let n = UInt64(t * 30) |
| + for a in dog.observe(at: Double(t), capture: counts(n, n), sending: sent(t) ?? sending(n), stalledBy: stalledBy(t)) { |
| + log.append((t, a)) |
| + } |
| + } |
| + return log |
| + } |
| + |
| + func testAReportKeyframesDoNotAnswerIsRebuiltThenSharedAgainThenGivenUp() { |
| + var dog = ScreenWatchdog() |
| + let log = escalating(0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, dog: &dog) |
| + let steps = log.filter { if case .keyframe = $0.1 { return false } else { return true } } |
| + // Reported from 3: keyframes at once; six seconds on, still reported, |
| + // the connection is rebuilt; ten more, everyone's screen shared again; |
| + // ten more, given up. |
| + XCTAssertEqual(steps.map(\.1), [.rebuild("vicky"), .reshare]) |
| + XCTAssertEqual(steps.map(\.0), [9, 19]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + XCTAssertEqual(dog.rebuilds, 1) |
| + XCTAssertEqual(dog.reshares, 1) |
| + XCTAssertEqual(dog.failures, 1) |
| + // A report alone never shrinks the screen or recreates its encoder: |
| + // Text and detail stays Text and detail. |
| + XCTAssertFalse(log.contains { if case .recreateEncoder = $0.1 { return true } else { return false } }) |
| + XCTAssertEqual(dog.status, "a receiver's view could not be restored") |
| + } |
| + |
| + func testAFailureStaysUntilTheReportClears() { |
| + var dog = ScreenWatchdog() |
| + _ = escalating(0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, dog: &dog) |
| + let later = escalating(45..<120, stalledBy: { $0 < 100 ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertFalse(later.contains { $0.1 == .reshare || $0.1 == .rebuild("vicky") }, "nothing more is tried") |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .none, "and a view that comes back is forgotten") |
| + } |
| + |
| + func testARebuildThatWorksEndsIt() { |
| + var dog = ScreenWatchdog() |
| + let log = escalating(0..<60, stalledBy: { (3..<12).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + let steps = log.filter { if case .keyframe = $0.1 { return false } else { return true } } |
| + XCTAssertEqual(steps.map(\.1), [.rebuild("vicky")]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .none) |
| + XCTAssertEqual(dog.reshares, 0) |
| + } |
| + |
| + func testRebuildsAndSharesAgainAreRationed() { |
| + var dog = ScreenWatchdog() |
| + // First stall: rebuilt at 9, and that works. |
| + _ = escalating(0..<20, stalledBy: { (3..<12).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + // A second, within the minute: no rebuild so soon, so straight to a |
| + // share again; that works too. |
| + let second = escalating(20..<50, stalledBy: { (25..<34).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertEqual(second.filter { $0.1 == .reshare || $0.1 == .rebuild("vicky") }.map(\.1), [.reshare]) |
| + // A third, past the minute but inside five: rebuilt, and then there is |
| + // no share again to be had, so it is given up. |
| + let third = escalating(80..<120, stalledBy: { $0 >= 85 ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertEqual(third.filter { $0.1 == .reshare || $0.1 == .rebuild("vicky") }.map(\.1), [.rebuild("vicky")]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + } |
| + |
| + func testNothingSentIsNothingToEscalate() { |
| + // A still screen with nothing new to send, or a link holding it all |
| + // back: the receiver cannot be failing to decode frames that never |
| + // left, so keyframes are all it gets. |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<40 { |
| + // Reporting all along (idle reports), nothing new delivered or sent |
| + // after t=3. |
| + let n = UInt64(min(t, 3) * 30) |
| + let still = ["vicky": ScreenSendCounts(encoded: n, sent: n, keyframes: 1)] |
| + for a in dog.observe( |
| + at: Double(t), capture: counts(UInt64(t * 30), n), sending: still, stalledBy: t >= 3 ? ["vicky"] : []) |
| + { |
| + log.append((t, a)) |
| + } |
| + } |
| + XCTAssertFalse(log.isEmpty) |
| + XCTAssertTrue(log.allSatisfy { if case .keyframe = $0.1 { return true } else { return false } }, "\(log)") |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .keyframes(since: 3)) |
| + } |
| + |
| + func testARebuildThatNeverComesBackStillEndsInWords() { |
| + // The rebuilt connection never negotiates a screen again: there is |
| + // nothing to count, and the steps go on regardless. |
| + var dog = ScreenWatchdog() |
| + let log = escalating( |
| + 0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, sent: { $0 > 9 ? [:] : nil }, dog: &dog) |
| + XCTAssertTrue(log.contains { $0.1 == .reshare }) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + } |
| + |
| + func testOneShareAgainAnswersEveryReport() { |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<30 { |
| + let n = UInt64(t * 30) |
| + let both = ["vicky": ScreenSendCounts(encoded: n, sent: n, keyframes: 1), "tony": ScreenSendCounts(encoded: n, sent: n, keyframes: 1)] |
| + let stalled: Set<String> = t >= 3 ? ["vicky", "tony"] : [] |
| + for a in dog.observe(at: Double(t), capture: counts(n, n), sending: both, stalledBy: stalled) { log.append((t, a)) } |
| + } |
| + XCTAssertEqual(log.filter { $0.1 == .reshare }.count, 1) |
| + XCTAssertEqual(log.filter { if case .rebuild = $0.1 { return true } else { return false } }.count, 2) |
| + } |
| + |
| + func testTheNoticeMovesOnInWords() { |
| + XCTAssertEqual(ScreenWords.notice(recovering: [], restarting: [], failed: []), "") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: ["Samuel"], restarting: [], failed: []), |
| + "Samuel\u{2019}s view of your screen stalled \u{2014} recovering") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: ["Samuel"], failed: []), "Restarting your share for Samuel\u{2026}") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: [], failed: ["Samuel"]), |
| + "Couldn\u{2019}t restore Samuel\u{2019}s view \u{2014} try Stop and Share again") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: ["Tony"], restarting: [], failed: ["Samuel"]), |
| + "Couldn\u{2019}t restore Samuel\u{2019}s view \u{2014} try Stop and Share again. Tony\u{2019}s view of your screen stalled \u{2014} recovering") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: ["A", "B", "C"], failed: []), "Restarting your share for 3 people\u{2026}") |
| + } |
| + |
| + func testTheReceiversNumbersNameTheStall() { |
| + var before = ScreenReceiveReport() |
| + before.mid = "3" |
| + before.packetsReceived = 1000 |
| + before.framesReceived = 300 |
| + before.framesDecoded = 300 |
| + var wedged = before |
| + wedged.packetsReceived = 1400 |
| + wedged.framesReceived = 420 |
| + XCTAssertEqual(wedged.diagnosis(since: before), "frames arriving, none decoded (the decoder)") |
| + XCTAssertTrue(wedged.moved(since: before).hasPrefix("packets +400, frames received +120, decoded +0")) |
| + XCTAssertEqual(before.diagnosis(since: before), "no packets arriving (the sender or the path)") |
| + var rebuilt = wedged |
| + rebuilt.mid = "5" |
| + XCTAssertEqual(rebuilt.diagnosis(since: before), "no earlier reading") |
| + } |
| + |
| // MARK: - The receiver |
| - func testAScreenThatStopsMovingIsReportedAfterFourSecondsAndForAMinuteAtMost() { |
| + func testAScreenThatStopsMovingIsReportedAfterFourSecondsAndForThreeMinutesAtMost() { |
| var watch = ScreenReceiveWatch() |
| var reported: [Int] = [] |
| - for t in 0..<80 { |
| + for t in 0..<200 { |
| let decoded: UInt64 = t < 10 ? UInt64(t * 10) : 90 |
| if !watch.observe(at: Double(t), decoded: ["paul": decoded]).isEmpty { reported.append(t) } |
| } |
| XCTAssertEqual(reported.first, 13, "last moved at 9: stalled from 13") |
| - XCTAssertEqual(reported.last, 68, "and no longer than a minute after it last moved") |
| + XCTAssertEqual(reported.last, 188, "and no longer than three minutes after it last moved") |
| + } |
| + |
| + func testARebuiltScreenStartingFromZeroIsMovingNotStalled() { |
| + // 2026-09-30: a screen the sharer rebuilt arrives on a new receiver |
| + // whose count starts again. Up to 0.4.4 only a larger count was |
| + // movement, so a rebuilt screen that was moving again stayed stalled |
| + // here — and the sharer's notice with it — until the new count passed |
| + // the old. |
| + var watch = ScreenReceiveWatch() |
| + for t in 0..<10 { _ = watch.observe(at: Double(t), decoded: ["paul": 900 + UInt64(t * 30)]) } |
| + for t in 10..<16 { _ = watch.observe(at: Double(t), decoded: ["paul": 1170]) } |
| + XCTAssertEqual(watch.observe(at: 16, decoded: ["paul": 1170]), ["paul"]) |
| + for t in 17..<40 { |
| + XCTAssertEqual(watch.observe(at: Double(t), decoded: ["paul": UInt64((t - 17) * 30)]), [], "t=\(t)") |
| + } |
| + } |
| + |
| + func testAStalledPictureIsAttachedAgainOnceFourSecondsIn() { |
| + var watch = ScreenReceiveWatch() |
| + var reattached: [Int] = [] |
| + for t in 0..<60 { |
| + let decoded: UInt64 = t < 10 ? UInt64(t * 10) : 90 |
| + _ = watch.observe(at: Double(t), decoded: ["paul": decoded]) |
| + if watch.reattach == ["paul"] { reattached.append(t) } |
| + } |
| + XCTAssertEqual(reattached, [17], "stalled from 13, attached again four seconds later, once") |
| + // Moving again and stalled again: a new stall, a new try. |
| + for t in 60..<63 { _ = watch.observe(at: Double(t), decoded: ["paul": UInt64(t)]) } |
| + var again: [Int] = [] |
| + for t in 63..<80 { |
| + _ = watch.observe(at: Double(t), decoded: ["paul": 62]) |
| + if !watch.reattach.isEmpty { again.append(t) } |
| + } |
| + XCTAssertEqual(again, [70]) |
| } |
| func testAMovingScreenIsNeverReported() { |
diff --git a/native/ios/Sources/PhoneFullScreenCall.swift b/native/ios/Sources/PhoneFullScreenCall.swift| index 4676f9b..6640a90 100644 |
| --- a/native/ios/Sources/PhoneFullScreenCall.swift |
| +++ b/native/ios/Sources/PhoneFullScreenCall.swift |
| @@ -266,7 +266,7 @@ extension PhoneCall { |
| private var notices: [CallNotice] { |
| CallNotices.notices( |
| - phase: call.state?.phase, status: call.status, sharingScreen: call.sharingScreen, |
| + phase: call.noticePhase, status: call.status, sharingScreen: call.sharingScreen, |
| screenNote: call.screenNote, floor: call.floor, screenRecovery: call.screenRecovery) |
| } |
diff --git a/native/ios/project.yml b/native/ios/project.yml| index c72fb95..295a52a 100644 |
| --- a/native/ios/project.yml |
| +++ b/native/ios/project.yml |
| @@ -10,7 +10,7 @@ settings: |
| base: |
| SWIFT_VERSION: '5.0' |
| CODE_SIGN_STYLE: Automatic |
| - MARKETING_VERSION: '0.4.4' |
| + MARKETING_VERSION: '0.4.5' |
| CURRENT_PROJECT_VERSION: '1' |
| # The App Group the app and its broadcast extension share: the socket |
| # the iPhone's screen crosses lives in its container. One setting |
diff --git a/test/browser/call.mjs b/test/browser/call.mjs| index 204a71e..59058f8 100644 |
| --- a/test/browser/call.mjs |
| +++ b/test/browser/call.mjs |
| @@ -2607,9 +2607,30 @@ const run = { |
| const before = { a: await state(pageA), b: await state(pageB) }; |
| // Long enough for at least one renewal each at a six-second life, |
| - // and in practice two. |
| - await sleep(16000); |
| + // and in practice two. Watched four times a second the whole way: |
| + // a renewal is a stream ended on purpose and resumed, and §15.23 says |
| + // nobody sees it — not "Reconnecting…" over the call, not on a tile, |
| + // not in the title (Paul, 2026-09-30: "I saw 'Reconnecting' appear |
| + // occasionally over the video, but only for a second"). |
| + const said = []; |
| + let streamDowns = 0; |
| + for (const until = Date.now() + 16000; Date.now() < until;) { |
| + for (const [page, who] of [[pageA, 'A'], [pageB, 'B']]) { |
| + const r = await page.evaluate(() => ({ |
| + reconnect: window.meet.state().reconnect, |
| + status: (document.getElementById('status') || {}).textContent || '', |
| + pill: (document.getElementById('call-state') || {}).textContent || '', |
| + tiles: Array.from(document.querySelectorAll('.calls-tile-notice, [data-hear-notice]')).map((n) => n.textContent), |
| + })); |
| + if (r.reconnect.streamDown) streamDowns += 1; |
| + const text = [r.status, r.pill, ...r.tiles].join(' | '); |
| + if (r.reconnect.call || r.reconnect.peers.length || /Reconnecting/.test(text)) said.push(who + ': ' + text); |
| + } |
| + await sleep(250); |
| + } |
| const after = { a: await state(pageA), b: await state(pageB) }; |
| + check(said.length === 0, 'no renewal was said as "Reconnecting" (' + said.slice(0, 3).join('; ') + ')'); |
| + console.log('renewal: a stream was seen down (between its end and its resume) at ' + streamDowns + ' of the looks'); |
| check(after.a.peer === before.a.peer, |
| 'A kept its participation across the renewals (' + before.a.peer + ' -> ' + after.a.peer + ')'); |
diff --git a/test/browser/native-screen.mjs b/test/browser/native-screen.mjs| index 637b9d0..c4551d3 100644 |
| --- a/test/browser/native-screen.mjs |
| +++ b/test/browser/native-screen.mjs |
| @@ -21,6 +21,24 @@ |
| // sender deactivated under it): the watchdog must recover it |
| // loss the browser stops painting and asks for a keyframe |
| // (screenStalled): the app must answer with one |
| +// wedge the browser's screen receiver stops passing anything to its |
| +// decoder (an encoded transform that drops every frame), which |
| +// no keyframe mends: the app must escalate to a rebuild and |
| +// the browser decode again within 30 s, the app's notice going |
| +// "recovering", "Restarting your share for …", then nothing |
| +// (spec §15.22) |
| +// wedge-sticky every screen receiver the browser gets is wedged, new |
| +// ones too: the app must end at "Couldn’t restore …", and |
| +// clear once the wedge is lifted |
| +// soak renewals and resumes as `renewals`, for SCREEN_SOAK_SECONDS, |
| +// with the mode cycled (balanced, motion, detail) and the |
| +// app's uplink capped and lifted every SCREEN_SOAK_STEP_MS: |
| +// every freeze is logged with what the ladder did about it |
| +// web-soak the same with the browser sharing and the app watching, the |
| +// browser's mode cycled |
| +// web-wedge, web-wedge-sticky the same the other way round: the |
| +// browser shares, the app's screen decoder wedges, and the |
| +// browser's ladder has to restore it (or say it cannot) |
| // |
| // Run by TestNativeMacScreenRecovery (make native-browser). |
| import { chromium } from 'playwright'; |
| @@ -41,23 +59,104 @@ const browser = await chromium.launch({ headless: true, args: [ |
| '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream', |
| '--autoplay-policy=no-user-gesture-required', '--disable-features=WebRtcHideLocalIpsWithMdns', |
| '--mute-audio', |
| + // getDisplayMedia answered with a fake 1280x720 monitor and no picker, |
| + // for the web-* scenarios (see call.mjs). |
| + '--auto-accept-this-tab-capture', |
| ] }); |
| -const context = await browser.newContext({ permissions: ['camera', 'microphone'] }); |
| +// bypassCSP: the wedge's encoded transform runs in a worker made from a |
| +// blob, which the page's own policy (default-src 'self') would refuse. |
| +// The gate's, never the page's. |
| +const context = await browser.newContext({ permissions: ['camera', 'microphone'], bypassCSP: true }); |
| await context.addCookies([{ name: process.env.CALLS_COOKIE_NAME, value: process.env.CALLS_COOKIE_A, url: origin }]); |
| const page = await context.newPage(); |
| await page.addInitScript(() => { |
| const Native = window.RTCPeerConnection; |
| window.__nativeInteropPCs = new Set(); |
| + // §15.22's wedge: a receiver whose frames never reach its decoder. |
| + // Chromium takes an encoded transform only on a receiver that has not |
| + // started, so in the wedge scenarios every screen receiver gets one as |
| + // it arrives — passing everything through — and __wedgeScreens tells |
| + // the ones there now to swallow every frame instead. `sticky` has each |
| + // one that arrives after start out swallowing. |
| + let worker = null; |
| + let serial = 0; |
| + const wedgeWorker = () => { |
| + if (!worker) { |
| + const code = [ |
| + 'const drops = new Map();', |
| + 'onmessage = (m) => { for (const id of m.data.ids) drops.set(id, m.data.drop); };', |
| + 'onrtctransform = (e) => {', |
| + ' const id = e.transformer.options.id;', |
| + ' if (!drops.has(id)) drops.set(id, !!e.transformer.options.drop);', |
| + ' e.transformer.readable.pipeThrough(new TransformStream({ transform(f, c) { if (!drops.get(id)) c.enqueue(f); } }))', |
| + ' .pipeTo(e.transformer.writable);', |
| + '};', |
| + ].join('\n'); |
| + worker = new Worker(URL.createObjectURL(new Blob([code], { type: 'text/javascript' }))); |
| + } |
| + return worker; |
| + }; |
| + window.__wedge = { sticky: false, wedged: 0, ids: new Map() }; |
| + const screenReceiver = (tx) => tx && tx.receiver && tx.receiver.track && tx.receiver.track.kind === 'video' && |
| + tx.direction === 'recvonly' && tx.currentDirection !== 'stopped'; |
| + const watchReceiver = (tx) => { |
| + if (!window.__wedgeReady || !screenReceiver(tx) || window.__wedge.ids.has(tx.receiver)) return; |
| + const id = ++serial; |
| + window.__wedge.ids.set(tx.receiver, id); |
| + tx.receiver.transform = new RTCRtpScriptTransform(wedgeWorker(), { id, drop: window.__wedge.sticky }); |
| + if (window.__wedge.sticky) window.__wedge.wedged += 1; |
| + }; |
| + window.__wedgeScreens = (sticky) => { |
| + window.__wedge.sticky = !!sticky; |
| + const ids = []; |
| + for (const pc of window.__nativeInteropPCs) { |
| + for (const tx of pc.getTransceivers()) { |
| + const id = screenReceiver(tx) && window.__wedge.ids.get(tx.receiver); |
| + if (id) ids.push(id); |
| + } |
| + } |
| + wedgeWorker().postMessage({ ids, drop: true }); |
| + window.__wedge.wedged += ids.length; |
| + return ids.length; |
| + }; |
| + window.__releaseWedge = () => { |
| + window.__wedge.sticky = false; |
| + wedgeWorker().postMessage({ ids: [...window.__wedge.ids.values()], drop: false }); |
| + }; |
| window.RTCPeerConnection = class extends Native { |
| - constructor(...args) { super(...args); window.__nativeInteropPCs.add(this); } |
| + // The web-* scenarios' screen goes out as H.264: the app can only |
| + // wedge a VideoToolbox decoder (DecoderWedge.swift), and the browser |
| + // left to itself sends VP8. What is under test, the ladder, does not |
| + // turn on the codec. |
| + addTransceiver(track, init) { |
| + const tx = super.addTransceiver(track, init); |
| + if (window.__h264Screen && init && init.direction === 'sendonly' && track && track.kind === 'video') { |
| + const codecs = RTCRtpSender.getCapabilities('video').codecs; |
| + const h264 = codecs.filter((c) => /h264/i.test(c.mimeType)); |
| + try { |
| + tx.setCodecPreferences([...h264, ...codecs.filter((c) => !/h264/i.test(c.mimeType))]); |
| + } catch {} |
| + } |
| + return tx; |
| + } |
| + constructor(...args) { |
| + super(...args); |
| + window.__nativeInteropPCs.add(this); |
| + this.addEventListener('track', (e) => watchReceiver(e.transceiver)); |
| + } |
| }; |
| }); |
| +// Receivers made ready to wedge only in the scenarios that wedge them, so |
| +// every other scenario's browser decodes exactly as a person's would. |
| +await page.addInitScript((ready) => { window.__wedgeReady = ready; }, scenario === 'wedge' || scenario === 'wedge-sticky'); |
| let native; |
| let failed = false; |
| try { |
| await page.goto(`${origin}/m/${room}`); |
| await page.waitForFunction(() => !!window.meet); |
| - native = spawn(binary, [], { stdio: ['pipe', 'pipe', 'pipe'] }); |
| + // The app's decoders can be wedged in the web-* scenarios only. |
| + const env = scenario.startsWith('web-') ? Object.assign({}, process.env, { CALLS_DECODER_WEDGE: '1' }) : process.env; |
| + native = spawn(binary, [], { stdio: ['pipe', 'pipe', 'pipe'], env }); |
| native.stdin.write(process.env.CALLS_NATIVE_INPUT + '\n'); |
| let sample = null; |
| let exited = false; |
| @@ -82,6 +181,17 @@ try { |
| }; |
| await wait(() => sample?.stats.framesDecoded > 15, 'native receives the browser'); |
| const nativePeer = sample.peer; |
| + if (scenario === 'web-soak') { |
| + await webSoak(page, native, () => sample, wait); |
| + console.log(`PASS ${scenario}`); |
| + throw new Done(); |
| + } |
| + if (scenario.startsWith('web-')) { |
| + await webShares(page, native, () => sample, replies, wait, scenario.endsWith('sticky')); |
| + console.log(`PASS ${scenario}`); |
| + process.exitCode = 0; |
| + throw new Done(); |
| + } |
| const key = nativePeer + '#share'; |
| // What the browser is receiving on the app's screen, from the |
| // connection's own statistics: the screen is the inbound video that is |
| @@ -110,7 +220,12 @@ try { |
| // The mode first: it is remembered across runs of the interop binary, |
| // and the screen's source is made for the mode the share starts in. |
| native.stdin.write(`screen-mode ${process.env.SCREEN_MODE || 'balanced'}\n`); |
| + // SCREEN_SOURCE=motion shares the 1920-wide scrolling page (its height |
| + // CALLS_SYNTHETIC_SCREEN_HEIGHT) rather than the small still screen: |
| + // what meets H.264 level 3.1's limit (§15.22). |
| if (scenario === 'display') native.stdin.write(`screen-display ${process.env.SCREEN_DEPTH || 6}\n`); |
| + else if (process.env.SCREEN_SOURCE === 'motion') native.stdin.write('screen-motion\n'); |
| + else if (process.env.SCREEN_SOURCE === 'display') native.stdin.write(`screen-display ${process.env.SCREEN_DEPTH || 6}\n`); |
| else native.stdin.write('screen-on\n'); |
| await wait(() => sample?.sharingScreen === true, 'native says it is sharing'); |
| await wait(async () => (await received()).frames > 15, 'the browser decodes the app’s screen'); |
| @@ -172,7 +287,22 @@ try { |
| await watchFor(SOAK, 'display'); |
| clearInterval(tick); |
| } else if (scenario === 'renewals' || scenario === 'resume') { |
| + // §15.23: a renewal or a resume is never said as "Reconnecting…", by |
| + // the app or the browser, while the core's phase does go through it. |
| + const said = []; |
| + let phases = 0; |
| + const saidTick = setInterval(async () => { |
| + if (sample?.phase === 'reconnecting') phases += 1; |
| + if (sample?.reconnecting || sample?.reconnectingPeers) said.push('app'); |
| + try { |
| + const r = (await page.evaluate(() => window.meet.state())).reconnect; |
| + if (r.call || r.peers.length) said.push('browser'); |
| + } catch {} |
| + }, 250); |
| await watchFor(SOAK, scenario); |
| + clearInterval(saidTick); |
| + console.log(`reconnecting: the app's core was mid-resume at ${phases} samples; said by ${JSON.stringify([...new Set(said)])}`); |
| + if (asserting) assert.equal(said.length, 0, 'a renewal or resume was said as Reconnecting: ' + said.join(',')); |
| console.log(`ICE restarts seen by the browser: ${restarts}`); |
| const ours = sample?.stats.iceRestarts || 0; |
| console.log(`ICE restarts the app started: ${ours}`); |
| @@ -199,6 +329,76 @@ try { |
| await wait(() => (sample?.stats.screenWatchdogKeyframes || 0) + (sample?.stats.screenWatchdogEncoderRecoveries || 0) > 0, |
| 'the watchdog says it recovered the encoder', 5); |
| console.log('PASS a stalled encoder was recovered without a re-share'); |
| + } else if (scenario === 'soak') { |
| + // §15.22's search for the cause: a long share with everything that |
| + // moves under it moving. What matters is not whether a freeze |
| + // happened but what it took to end it — the ladder's counts at each |
| + // freeze say whether keyframes were enough. |
| + const modes = ['balanced', 'motion', 'detail']; |
| + const uplinks = [0, 1500000, 600000, 0, 3000000, 900000]; |
| + const started = Date.now(); |
| + const notices = new Set(); |
| + let step = 0; |
| + const tick = setInterval(() => { |
| + step += 1; |
| + const mode = modes[step % modes.length]; |
| + const up = uplinks[step % uplinks.length]; |
| + native.stdin.write(`screen-mode ${mode}\n`); |
| + native.stdin.write(`uplink ${up}\n`); |
| + const st = sample?.stats || {}; |
| + console.log(`soak ${Math.round((Date.now() - started) / 1000)} s: mode ${mode}, uplink ${up || 'open'} (was sending ${st.screenSendWidth}x${st.screenSendHeight} at ${(st.screenSendFpsTenths || 0) / 10} fps, limitation ${st.screenLimitation})`); |
| + }, Number(process.env.SCREEN_SOAK_STEP_MS || 45000)); |
| + const noticeTick = setInterval(() => { if (sample?.screenRecovery) notices.add(sample.screenRecovery); }, 500); |
| + await watchFor(SOAK, 'soak'); |
| + clearInterval(tick); |
| + clearInterval(noticeTick); |
| + native.stdin.write('uplink 0\n'); |
| + const st = sample?.stats || {}; |
| + console.log(`SOAK app: ICE restarts ${st.iceRestarts}, stall reports ${st.screenStallReports}, keyframes ${st.screenWatchdogKeyframes}, fresh encoders ${st.screenWatchdogEncoderRecoveries}, re-adds ${st.screenWatchdogReAdds}, rebuilds ${st.screenWatchdogRebuilds}, shares again ${st.screenWatchdogReshares}, given up ${st.screenWatchdogFailures}, capture restarts ${st.screenWatchdogCaptureRestarts}`); |
| + console.log(`SOAK browser ICE restarts ${restarts}; notices at the app: ${JSON.stringify([...notices])}`); |
| + console.log('the browser\u2019s screen log:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| + if (asserting) { |
| + assert.equal(st.screenWatchdogFailures || 0, 0, 'the app gave up on the browser\u2019s view'); |
| + assert.equal(freezes.filter((f) => f.open).length, 0, 'the screen froze and never came back'); |
| + } |
| + } else if (scenario === 'wedge' || scenario === 'wedge-sticky') { |
| + await watchFor(5, 'before the wedge'); |
| + const sticky = scenario === 'wedge-sticky'; |
| + const keyframes = sample?.stats.screenWatchdogKeyframes || 0; |
| + const notices = new Set(); |
| + const note = () => { if (sample?.screenRecovery) notices.add(sample.screenRecovery); }; |
| + const wedgedAt = Date.now(); |
| + const n = await page.evaluate((s) => window.__wedgeScreens(s), sticky); |
| + assert.ok(n >= 1, 'the browser\u2019s screen receiver was wedged'); |
| + await wait(async () => (await page.evaluate(() => window.meet.state().screenStalled)).length === 1, |
| + 'the browser reports the app\u2019s screen stalled', 15); |
| + if (!sticky) { |
| + // Painted frames on the share tile, counted across a new tile. |
| + const progress = await progressOf(() => received().then((r) => r.frames)); |
| + await wait(async () => { note(); return (await progress()) > 15; }, |
| + 'the browser decodes the app\u2019s screen again, without a share again', 30); |
| + console.log(`the browser decoding again ${Date.now() - wedgedAt} ms after the wedge: rebuilds ${sample?.stats.screenWatchdogRebuilds}, shares again ${sample?.stats.screenWatchdogReshares}, keyframes ${(sample?.stats.screenWatchdogKeyframes || 0) - keyframes}`); |
| + assert.ok((sample?.stats.screenWatchdogKeyframes || 0) > keyframes, 'keyframes were tried first'); |
| + assert.ok((sample?.stats.screenWatchdogRebuilds || 0) > 0, 'the rebuild is what mended it'); |
| + await wait(async () => (await page.evaluate(() => window.meet.state().screenStalled)).length === 0, |
| + 'the browser clears its report', 20); |
| + await wait(() => { note(); return sample?.screenRecovery === ''; }, 'the app\u2019s notice clears', 20); |
| + console.log('notices seen at the app: ' + JSON.stringify([...notices])); |
| + assert.ok([...notices].some((x) => x.includes('recovering')), 'the notice said recovering'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + console.log('PASS a wedged browser receiver was restored by the app\u2019s rebuild'); |
| + } else { |
| + await wait(() => { note(); return (sample?.screenRecovery || '').includes('Couldn\u2019t restore'); }, |
| + 'the app says it could not restore the browser\u2019s view', 60); |
| + console.log('notices seen at the app: ' + JSON.stringify([...notices])); |
| + assert.ok((sample?.stats.screenWatchdogRebuilds || 0) > 0 && (sample?.stats.screenWatchdogReshares || 0) > 0, |
| + 'a rebuild and a share again were tried first'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + await page.evaluate(() => window.__releaseWedge()); |
| + await recovered('the browser decodes again once the wedge is lifted'); |
| + await wait(() => sample?.screenRecovery === '', 'the app\u2019s notice clears once the view is back', 30); |
| + console.log('PASS the app said it could not restore a view nothing could, and cleared when it came back'); |
| + } |
| } else if (scenario === 'loss') { |
| await watchFor(5, 'before loss'); |
| // The browser's watch treats the app's screen as frozen, as it would |
| @@ -231,8 +431,10 @@ try { |
| } |
| console.log(`PASS ${scenario}: the app’s screen kept arriving`); |
| } catch (err) { |
| - failed = true; |
| - console.log('FAIL', err.message); |
| + if (!(err instanceof Done)) { |
| + failed = true; |
| + console.log('FAIL', err.message); |
| + } |
| } finally { |
| if (native) { |
| native.stdin.write('leave\n'); |
| @@ -242,3 +444,128 @@ try { |
| await browser.close(); |
| if (failed) process.exit(1); |
| } |
| + |
| +// Done ends a scenario that has passed before the app-shares path below |
| +// it: thrown, and caught as not a failure. |
| +function Done() {} |
| + |
| +// progressOf counts how far a counter has moved since now, across a |
| +// restart from zero (a new tile, a new receiver): a fall is taken as a |
| +// new count starting. |
| +async function progressOf(read) { |
| + let last = await read(); |
| + let total = 0; |
| + return async () => { |
| + const now = await read(); |
| + total += now >= last ? now - last : now; |
| + last = now; |
| + return total; |
| + }; |
| +} |
| + |
| +// webShares is the web-* scenarios: the browser shares its screen (a fake |
| +// monitor), the app watches, the app's screen decoder is wedged, and the |
| +// browser's own ladder (§15.22) has to put it right — or, sticky, say it |
| +// cannot, and clear once the wedge is lifted. |
| +async function webShares(page, native, current, replies, wait, sticky) { |
| + const stat = (k) => current()?.stats[k] || 0; |
| + await page.evaluate(() => { window.__h264Screen = true; }); |
| + await page.hover('#share'); |
| + await page.click('#share'); |
| + await wait(() => stat('remoteScreens') === 1 && stat('screenFramesDecoded') > 15, 'the app decodes the browser\u2019s screen', 30); |
| + console.log('PASS the app decodes the browser\u2019s screen'); |
| + await new Promise((r) => setTimeout(r, 5000)); |
| + const s0 = () => page.evaluate(() => window.meet.state()); |
| + const keyframes = (await s0()).shareRecoveries.keyframes; |
| + const wedgedAt = Date.now(); |
| + native.stdin.write(sticky ? 'screen-decoder-wedge sticky\n' : 'screen-decoder-wedge\n'); |
| + await wait(() => replies.some((r) => (r.decoderWedged || 0) >= 1), 'the app\u2019s screen decoder is wedged', 10); |
| + await wait(() => stat('screenStalledHere') === 1, 'the app reports the browser\u2019s screen stalled (the wedge hit the screen)', 15); |
| + const notices = new Set(); |
| + const note = async () => { |
| + const s = await s0(); |
| + if (s.shareRecovery) notices.add(s.shareRecovery); |
| + return s; |
| + }; |
| + if (!sticky) { |
| + const base = stat('screenFramesDecodedTotal'); |
| + await wait(async () => { await note(); return stat('screenFramesDecodedTotal') > base + 15; }, |
| + 'the app decodes the browser\u2019s screen again, without a share again', 30); |
| + const s = await note(); |
| + console.log(`the app decoding again ${Date.now() - wedgedAt} ms after the wedge: ${JSON.stringify(s.shareRecoveries)}`); |
| + assert.ok(s.shareRecoveries.keyframes > keyframes, 'keyframes were tried first'); |
| + assert.ok(s.shareRecoveries.rebuilds > 0, 'the rebuild is what mended it'); |
| + await wait(() => stat('screenStalledHere') === 0, 'the app clears its report', 20); |
| + await wait(async () => (await note()).shareRecovery === '', 'the browser\u2019s notice clears', 20); |
| + console.log('notices seen at the browser: ' + JSON.stringify([...notices])); |
| + assert.ok([...notices].some((x) => x.includes('recovering')), 'the notice said recovering'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + } else { |
| + await wait(async () => (await note()).shareRecovery.includes('Couldn\u2019t restore'), |
| + 'the browser says it could not restore the app\u2019s view', 60); |
| + const s = await note(); |
| + console.log('notices seen at the browser: ' + JSON.stringify([...notices]) + ' ' + JSON.stringify(s.shareRecoveries)); |
| + assert.ok(s.shareRecoveries.rebuilds > 0 && s.shareRecoveries.reshares > 0, 'a rebuild and a share again were tried first'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + native.stdin.write('screen-decoder-release\n'); |
| + const base = stat('screenFramesDecodedTotal'); |
| + await wait(() => stat('screenFramesDecodedTotal') > base + 10, 'the app decodes again once the wedge is lifted', 30); |
| + await wait(async () => (await note()).shareRecovery === '', 'the browser\u2019s notice clears once the view is back', 30); |
| + } |
| + console.log('the browser\u2019s diagnostics:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| +} |
| + |
| +// webSoak is `soak` with the browser sharing and the app watching: the |
| +// app's received screen, counted across rebuilds, watched for freezes |
| +// while the renewals run and the browser's mode is cycled. |
| +async function webSoak(page, native, current, wait) { |
| + const stat = (k) => current()?.stats[k] || 0; |
| + await page.hover('#share'); |
| + await page.click('#share'); |
| + await wait(() => stat('remoteScreens') === 1 && stat('screenFramesDecoded') > 15, 'the app decodes the browser\u2019s screen', 30); |
| + const modes = ['balanced', 'motion', 'detail']; |
| + const started = Date.now(); |
| + let step = 0; |
| + const notices = new Set(); |
| + const freezes = []; |
| + let last = stat('screenFramesDecodedTotal'); |
| + let moved = Date.now(); |
| + let frozen = null; |
| + let nextStep = Date.now() + Number(process.env.SCREEN_SOAK_STEP_MS || 45000); |
| + while (Date.now() - started < SOAK * 1000) { |
| + await new Promise((r) => setTimeout(r, 1000)); |
| + const s = await page.evaluate(() => window.meet.state()); |
| + if (s.shareRecovery) notices.add(s.shareRecovery); |
| + if (Date.now() >= nextStep) { |
| + step += 1; |
| + const mode = modes[step % modes.length]; |
| + await page.evaluate((m) => { |
| + const el = document.getElementById('pick-share-mode'); |
| + if (el) { el.value = m; el.dispatchEvent(new Event('change', { bubbles: true })); } |
| + }, mode); |
| + console.log(`web-soak ${Math.round((Date.now() - started) / 1000)} s: the browser\u2019s mode ${mode}`); |
| + nextStep = Date.now() + Number(process.env.SCREEN_SOAK_STEP_MS || 45000); |
| + } |
| + const now = stat('screenFramesDecodedTotal'); |
| + if (now > last) { |
| + if (frozen) { |
| + frozen.ms = Date.now() - frozen.since; |
| + freezes.push(frozen); |
| + console.log(`FREEZE web-soak: ${frozen.ms} ms, then frames again; browser ladder ${JSON.stringify(s.shareRecoveries)}`); |
| + frozen = null; |
| + } |
| + moved = Date.now(); |
| + } else if (!frozen && Date.now() - moved > FREEZE_MS) { |
| + frozen = { since: moved }; |
| + console.log(`FROZEN web-soak: no screen frame at the app for ${Date.now() - moved} ms; app stalled-here ${stat('screenStalledHere')}; browser ladder ${JSON.stringify(s.shareRecoveries)}`); |
| + } |
| + last = now; |
| + } |
| + const s = await page.evaluate(() => window.meet.state()); |
| + console.log(`SUMMARY web-soak: ${freezes.length} freezes, longest ${Math.max(0, ...freezes.map((f) => f.ms))} ms, open ${!!frozen}; app ICE restarts ${stat('iceRestarts')}, pictures attached again ${stat('screenReattaches')}; browser ladder ${JSON.stringify(s.shareRecoveries)}; notices ${JSON.stringify([...notices])}`); |
| + console.log('the browser\u2019s screen log:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| + if (asserting) { |
| + assert.equal(s.shareRecoveries.failures, 0, 'the browser gave up on the app\u2019s view'); |
| + assert.ok(!frozen, 'the app\u2019s screen froze and never came back'); |
| + } |
| +} |
diff --git a/test/browser/room-reconnect.test.mjs b/test/browser/room-reconnect.test.mjs| new file mode 100644 |
| index 0000000..e5a3455 |
| --- /dev/null |
| +++ b/test/browser/room-reconnect.test.mjs |
| @@ -0,0 +1,105 @@ |
| +// room-reconnect.test.mjs — "Reconnecting…" only for a problem that |
| +// lasts (spec §15.23, 2026-09-30), gated without a browser. |
| +// |
| +// node --test test/browser/room-reconnect.test.mjs |
| +// |
| +// reconnectStep and reconnectRecord lifted out of the shipped room.js by |
| +// name, with the constants they read, as room-screen.test.mjs does. |
| +import { test } from 'node:test'; |
| +import assert from 'node:assert/strict'; |
| +import { readFileSync } from 'node:fs'; |
| +import { fileURLToPath } from 'node:url'; |
| + |
| +const src = readFileSync(fileURLToPath(new URL('../../internal/calls/static/room.js', import.meta.url)), 'utf8'); |
| + |
| +function lift(name) { |
| + const start = src.indexOf('function ' + name + '('); |
| + assert.notEqual(start, -1, name + ' is not in room.js any more — this test is stale, not passing'); |
| + let depth = 0; |
| + for (let i = src.indexOf('{', start); i < src.length; i++) { |
| + if (src[i] === '{') depth += 1; |
| + else if (src[i] === '}') { |
| + depth -= 1; |
| + if (depth === 0) return src.slice(start, i + 1); |
| + } |
| + } |
| + assert.fail('unbalanced braces reading ' + name); |
| +} |
| +function liftConst(name) { |
| + const m = src.match(new RegExp('const ' + name + ' = ([0-9.]+);')); |
| + assert.ok(m, name + ' is not a numeric const in room.js any more'); |
| + return 'const ' + name + ' = ' + m[1] + ';'; |
| +} |
| +const CONSTS = ['RECONNECT_STREAM_AFTER', 'RECONNECT_CONNECTION_AFTER', 'RECONNECT_AUDIO_AFTER', 'RECONNECT_HOLD'] |
| + .map(liftConst).join('\n'); |
| +// eslint-disable-next-line no-new-func |
| +const { reconnectRecord, reconnectStep } = new Function(CONSTS + '\n' + lift('reconnectRecord') + '\n' + |
| + lift('reconnectStep') + '; return { reconnectRecord, reconnectStep };')(); |
| + |
| +const fine = { connectionBad: false, audioMissing: false }; |
| + |
| +// Five looks a second from `from` to `to` seconds; the times the whole |
| +// call's words showed, and the tiles' at each time. |
| +function run(rec, from, to, stream, people) { |
| + const call = []; |
| + const peers = new Map(); |
| + for (let t = from * 10; t < to * 10; t += 2) { |
| + const s = t / 10; |
| + reconnectStep(rec, s * 1000, stream(s), people(s)); |
| + if (rec.call) call.push(s); |
| + if (rec.peers.size) peers.set(s, Array.from(rec.peers).sort().join()); |
| + } |
| + return { call, peers }; |
| +} |
| + |
| +test('a routine stream renewal, and a slow one, are never said', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 60, (t) => (t >= 10 && t < 10.6) || (t >= 30 && t < 33.5), |
| + () => new Map([['sam', fine]])); |
| + assert.deepEqual(seen.call, []); |
| + assert.equal(seen.peers.size, 0); |
| +}); |
| + |
| +test('a stream down four seconds is said, and held a second past recovery', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 20, (t) => t >= 5 && t < 12, () => new Map([['sam', fine]])); |
| + assert.equal(seen.call[0], 9); |
| + assert.equal(seen.call[seen.call.length - 1], 12.8); |
| +}); |
| + |
| +test('short blips on one person are never said', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 30, () => false, (t) => new Map([['sam', { |
| + connectionBad: t >= 5 && t < 7.5, audioMissing: t >= 15 && t < 16.5, |
| + }]])); |
| + assert.deepEqual(seen.call, []); |
| + assert.equal(seen.peers.size, 0); |
| +}); |
| + |
| +test('one person\'s lasting trouble is said on their tile alone', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 30, () => false, |
| + (t) => new Map([['sam', { connectionBad: t >= 5 && t < 15, audioMissing: false }], ['ana', fine]])); |
| + assert.deepEqual(seen.call, []); |
| + const times = Array.from(seen.peers.keys()); |
| + assert.equal(times[0], 8); |
| + assert.equal(times[times.length - 1], 15.8); |
| + assert.ok(Array.from(seen.peers.values()).every((v) => v === 'sam')); |
| +}); |
| + |
| +test('audio that stops from someone unmuted is said after two seconds', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 20, () => false, |
| + (t) => new Map([['sam', { connectionBad: false, audioMissing: t >= 5 && t < 10 }], ['ana', fine]])); |
| + assert.equal(Array.from(seen.peers.keys())[0], 7); |
| +}); |
| + |
| +test('everyone\'s trouble is the whole call, not a tile each', () => { |
| + const rec = reconnectRecord(); |
| + const seen = run(rec, 0, 20, () => false, (t) => { |
| + const bad = { connectionBad: t >= 5 && t < 12, audioMissing: false }; |
| + return new Map([['sam', bad], ['ana', bad]]); |
| + }); |
| + assert.equal(seen.call[0], 8); |
| + assert.equal(seen.peers.size, 0); |
| +}); |
diff --git a/test/browser/room-screen.test.mjs b/test/browser/room-screen.test.mjs| index 652228d..1aa5403 100644 |
| --- a/test/browser/room-screen.test.mjs |
| +++ b/test/browser/room-screen.test.mjs |
| @@ -7,8 +7,12 @@ |
| // does, with the constants they read. |
| // |
| // judgeScreen the receiver's verdict on one screen: stalled after |
| -// four seconds without a new frame, for a minute at most |
| -// recoveringWords the sharer's quiet notice |
| +// four seconds without a new frame, for three minutes at |
| +// most; any change in the count is movement |
| +// reattachDue the receiver's own mending, once per stall (§15.22) |
| +// escalateShare the sharer's ladder when keyframes do not answer |
| +// recoveringWords, noticeWords the sharer's notice |
| +// readingMoved what a stalled screen's statistics say |
| // mediaFrom, mediaKey, mediaPayload the report's screenStalled list |
| import { test } from 'node:test'; |
| import assert from 'node:assert/strict'; |
| @@ -37,13 +41,21 @@ function liftConst(name) { |
| assert.ok(m, name + ' is not a numeric const in room.js any more'); |
| return 'const ' + name + ' = ' + m[1] + ';'; |
| } |
| -const CONSTS = ['SCREEN_STALL_AFTER', 'SCREEN_STALL_GIVE_UP'].map(liftConst).join('\n'); |
| +const CONSTS = [ |
| + 'SCREEN_STALL_AFTER', 'SCREEN_STALL_GIVE_UP', 'SCREEN_REATTACH_AFTER', 'SCREEN_REBUILD_AFTER', |
| + 'SCREEN_RESHARE_AFTER', 'SCREEN_GIVE_UP_AFTER', 'SCREEN_REBUILD_EVERY', |
| +].map(liftConst).join('\n'); |
| const build = (name) => |
| // eslint-disable-next-line no-new-func |
| new Function(CONSTS + '\n' + lift(src, name) + '; return ' + name + ';')(); |
| const judgeScreen = build('judgeScreen'); |
| const recoveringWords = build('recoveringWords'); |
| +const reattachDue = build('reattachDue'); |
| +const escalateShare = build('escalateShare'); |
| +const readingMoved = new Function(lift(src, 'readingSummary') + '\n' + lift(src, 'readingMoved') + '; return readingMoved;')(); |
| +// eslint-disable-next-line no-new-func |
| +const noticeWords = new Function(lift(src, 'recoveringWords') + '\n' + lift(src, 'noticeWords') + '; return noticeWords;')(); |
| const mediaFrom = build('mediaFrom'); |
| const mediaPayload = build('mediaPayload'); |
| // eslint-disable-next-line no-new-func |
| @@ -51,15 +63,108 @@ const mediaKey = new Function(lift(src, 'mediaKey') + '; return mediaKey;')(); |
| const ID = '0123456789abcdef0123456789abcdef'; |
| -test('a screen is stalled four seconds after its last frame, and for a minute at most', () => { |
| +test('a screen is stalled four seconds after its last frame, and for three minutes at most', () => { |
| const seen = { decoded: 0, movedAt: 0 }; |
| const stalledAt = []; |
| - for (let t = 0; t <= 80; t++) { |
| + for (let t = 0; t <= 200; t++) { |
| const decoded = t < 10 ? t * 10 : 90; |
| if (judgeScreen(seen, decoded, t * 1000, false)) stalledAt.push(t); |
| } |
| assert.equal(stalledAt[0], 13); |
| - assert.equal(stalledAt[stalledAt.length - 1], 68); |
| + assert.equal(stalledAt[stalledAt.length - 1], 188); |
| +}); |
| + |
| +test('a rebuilt screen whose count starts again is moving, not stalled', () => { |
| + const seen = { decoded: 0, movedAt: 0 }; |
| + for (let t = 0; t < 10; t++) judgeScreen(seen, 900 + t * 30, t * 1000, false); |
| + for (let t = 10; t < 16; t++) judgeScreen(seen, 1170, t * 1000, false); |
| + assert.equal(judgeScreen(seen, 1170, 16000, false), true); |
| + for (let t = 17; t < 40; t++) { |
| + assert.equal(judgeScreen(seen, (t - 17) * 30, t * 1000, false), false, 't=' + t); |
| + } |
| +}); |
| + |
| +test('a stalled picture is handed to its tile again once, four seconds into the stall', () => { |
| + const seen = { decoded: 0, movedAt: 0 }; |
| + const due = []; |
| + for (let t = 0; t < 60; t++) { |
| + const decoded = t < 10 ? t * 10 : 90; |
| + judgeScreen(seen, decoded, t * 1000, false); |
| + if (reattachDue(seen, t * 1000)) due.push(t); |
| + } |
| + assert.deepEqual(due, [17], 'last moved at 9, stalled from 13, handed back at 17, once'); |
| + judgeScreen(seen, 91, 60000, false); |
| + const again = []; |
| + for (let t = 61; t < 80; t++) { |
| + judgeScreen(seen, 91, t * 1000, false); |
| + if (reattachDue(seen, t * 1000)) again.push(t); |
| + } |
| + assert.deepEqual(again, [68], 'a new stall is a new try'); |
| +}); |
| + |
| +// The ladder, driven once a second the way answerScreenStalls drives it. |
| +function ladder(from, to, reported, opts = {}) { |
| + const rec = opts.rec || { stage: '', at: 0, lastRebuild: 0 }; |
| + let resharedAt = opts.resharedAt || 0; |
| + const out = []; |
| + for (let t = from; t < to; t++) { |
| + const now = t * 1000; |
| + if (!reported(t)) { |
| + rec.stage = ''; |
| + continue; |
| + } |
| + const owed = opts.owed ? opts.owed(t) : true; |
| + const free = !resharedAt || now - resharedAt >= 300000; |
| + const action = escalateShare(rec, now, owed, free); |
| + if (action === 'reshare') resharedAt = now; |
| + if (action) out.push([t, action]); |
| + } |
| + return { out, rec, resharedAt }; |
| +} |
| + |
| +test('a report keyframes do not answer is rebuilt, then shared again, then given up', () => { |
| + const { out, rec } = ladder(0, 45, (t) => t >= 3); |
| + assert.deepEqual(out, [[9, 'rebuild'], [19, 'reshare'], [29, 'failed']]); |
| + assert.equal(rec.stage, 'failed'); |
| +}); |
| + |
| +test('a rebuild that works ends the ladder', () => { |
| + const { out, rec } = ladder(0, 60, (t) => t >= 3 && t < 12); |
| + assert.deepEqual(out, [[9, 'rebuild']]); |
| + assert.equal(rec.stage, ''); |
| +}); |
| + |
| +test('rebuilds and shares again are rationed', () => { |
| + const first = ladder(0, 20, (t) => t >= 3 && t < 12); |
| + const second = ladder(20, 50, (t) => t >= 25 && t < 34, { rec: first.rec, resharedAt: first.resharedAt }); |
| + assert.deepEqual(second.out, [[31, 'reshare']], 'no second rebuild inside a minute'); |
| + const third = ladder(80, 120, (t) => t >= 85, { rec: second.rec, resharedAt: second.resharedAt }); |
| + assert.deepEqual(third.out, [[91, 'rebuild'], [101, 'failed']], 'and no second share again inside five'); |
| +}); |
| + |
| +test('nothing sent is nothing to escalate', () => { |
| + const { out, rec } = ladder(0, 60, (t) => t >= 3, { owed: () => false }); |
| + assert.deepEqual(out, []); |
| + assert.equal(rec.stage, 'keyframes'); |
| +}); |
| + |
| +test('the notice moves on in words, the app\'s own', () => { |
| + assert.equal(noticeWords([], [], []), ''); |
| + assert.equal(noticeWords(['Samuel'], [], []), 'Samuel’s view of your screen stalled — recovering'); |
| + assert.equal(noticeWords([], ['Samuel'], []), 'Restarting your share for Samuel…'); |
| + assert.equal(noticeWords([], [], ['Samuel']), 'Couldn’t restore Samuel’s view — try Stop and Share again'); |
| + assert.equal(noticeWords(['Tony'], [], ['Samuel']), |
| + 'Couldn’t restore Samuel’s view — try Stop and Share again. Tony’s view of your screen stalled — recovering'); |
| + assert.equal(noticeWords([], ['A', 'B', 'C'], []), 'Restarting your share for 3 people…'); |
| +}); |
| + |
| +test('a stalled screen\'s statistics name what stopped', () => { |
| + const base = { mid: '3', packets: 1000, lost: 0, received: 300, decoded: 300, dropped: 0, keyframes: 2, pli: 0, fir: 0, |
| + nack: 0, freezes: 0, width: 1920, height: 1080, decoder: 'ExternalDecoder', muted: false }; |
| + const wedged = Object.assign({}, base, { packets: 1400, received: 420, dropped: 110 }); |
| + assert.match(readingMoved(wedged, base), /^packets \+400, frames received \+120, decoded \+0, dropped \+110.* — frames arriving, none decoded \(the decoder\)$/); |
| + assert.match(readingMoved(base, base), /no packets arriving/); |
| + assert.match(readingMoved(Object.assign({}, base, { mid: '5' }), base), /no earlier reading$/); |
| }); |
| test('a moving screen is never stalled, and a frozen test window is', () => { |
diff --git a/test/browser/room-share.test.mjs b/test/browser/room-share.test.mjs| index 7be3622..b12fd74 100644 |
| --- a/test/browser/room-share.test.mjs |
| +++ b/test/browser/room-share.test.mjs |
| @@ -1001,11 +1001,14 @@ test('the screen is read off its own outbound report, by mid', () => { |
| const reports = [ |
| { mid: '1', frameWidth: 640, frameHeight: 480, framesPerSecond: 30 }, |
| { mid: '2', frameWidth: 1920, frameHeight: 1080, framesPerSecond: 8, qualityLimitationReason: 'bandwidth', |
| - targetBitrate: 1200000, encoderImplementation: 'OpenH264', framesEncoded: 400 }, |
| + targetBitrate: 1200000, encoderImplementation: 'OpenH264', framesEncoded: 400, framesSent: 398, |
| + keyFramesEncoded: 3 }, |
| ]; |
| + // framesSent and keyFramesEncoded (§15.22): what the ladder counts as |
| + // having left for this person, and the diagnostics' keyframes. |
| assert.deepEqual(shareSendReport(reports, '2'), { |
| width: 1920, height: 1080, fps: 8, limitation: 'bandwidth', targetBitrate: 1200000, |
| - encoder: 'OpenH264', framesEncoded: 400, |
| + encoder: 'OpenH264', framesEncoded: 400, framesSent: 398, keyFramesEncoded: 3, |
| }); |
| assert.equal(shareSendReport(reports, ''), null, 'not sharing on this connection'); |
| assert.equal(shareSendReport(reports, '9'), null, 'not negotiated yet'); |