amadan
Screen stalls keyframes cannot mend: the sharer escalates, the receiver mends its own
Paul, 30 Sep (Mac 0.4.3/0.4.4 to Samuel on the web): "Still seeing issues with screen sharing after a while." The notice sat at "Samuel's view of your screen stalled - recovering": a receiver's report only ever bought keyframes (spec 15.21), and nothing came after them. Spec 15.22: - The sharer's ladder, per reporting peer, apps and browser alike: keyframes; still reported 6 s after the first, that connection's screen rebuilt (new transceiver, same size and mode - a report never shrinks it); 10 s more, everyone's screen shared again from the same capture (no picker); 10 s more, "Couldn't restore Samuel's view - try Stop and Share again". The notice goes recovering -> "Restarting your share for Samuel..." -> clear or couldn't restore. A rebuild at most once a minute per peer, a share again at most once every five, and neither unless frames actually left for that peer since the last step. - The receiver: any change in the decoded count is movement (a rebuilt screen's count starts again, which 0.4.3 read as still stalled, keeping the sharer's notice up); the picture handed to its view again 4 s into a stall; reports held up to 3 min; every stall logged with what its statistics say moved (packets, frames received/decoded/dropped, keyframes, PLI/NACK, decoder) and a verdict. - The iPhone broadcast repeats a still screen once a second, as the Mac capture does, so a still iPhone screen is not reported stalled. - The Mac capture's size is known before the first frame, so the H.264 level-3.1 cap on a Motion share holds from the first frame. - "Copy call diagnostics": the ladder's counts and stage per peer, every arriving screen's statistics, the last 200 lines of the screen log. The browser gains the button, at the foot of its Devices panel. Gates: a receiver decoder wedge keyframes cannot mend - CallsInterop's wrapping decoder factory (CALLS_DECODER_WEDGE=1, gate only) and a Chromium encoded transform that swallows every frame. native-pair wedge, wedge-sticky, reshare; native-browser wedge, wedge-sticky, web-wedge, web-wedge-sticky. Soaks (soak, web-soak, the pair's soak) by hand.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
19 files changed,
+2144
−103
docs/superpowers/specs/2026-09-07-meet-design.md+76 −1internal/calls/static/calls.css+3 −0internal/calls/static/room.js+427 −23internal/calls/templates/room.html+6 −0internal/callstest/native_browser_test.go+258 −11native/README.md+30 −0native/apple/Sources/CallsInterop/Interop.swift+35 −0native/apple/Sources/CallsInterop/SyntheticVideo.swift+5 −1native/apple/Sources/CallsKit/BroadcastScreenCapture.swift+73 −11native/apple/Sources/CallsKit/Call.swift+108 −17native/apple/Sources/CallsKit/CallAudio.swift+3 −1native/apple/Sources/CallsKit/Capture.swift+7 −0native/apple/Sources/CallsKit/DecoderWedge.swift+123 −0native/apple/Sources/CallsKit/MacScreenCapture.swift+3 −0native/apple/Sources/CallsKit/Media.swift+51 −10native/apple/Sources/CallsKit/ScreenWatchdog.swift+299 −11native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift+209 −6test/browser/native-screen.mjs+317 −5test/browser/room-screen.test.mjs+111 −6
diff --git a/docs/superpowers/specs/2026-09-07-meet-design.md b/docs/superpowers/specs/2026-09-07-meet-design.md| index 58989e6..a522d98 100644 |
| --- a/docs/superpowers/specs/2026-09-07-meet-design.md |
| +++ b/docs/superpowers/specs/2026-09-07-meet-design.md |
| @@ -2415,7 +2415,7 @@ own screen, and the receiver says when a screen has stopped for it. |
| decoded nothing new here for four seconds while they say they are |
| sharing. Every sender repeats a still screen at least once a second, so |
| that is not a still screen. It is bad news only, omitted when there is |
| - none, held for at most a minute from the last frame, at most fifteen |
| + none, held for at most a minute from the last frame (three since §15.22), at most fifteen |
| distinct well-formed peer ids, validated exactly as `noSoundFrom` is, and |
| re-encoded after it (`muted`, `cameraOff`, `audioBlocked`, `noSoundFrom`, |
| `screenStalled`). The raw payload cap is 2 KB, room for both lists full. |
| @@ -2436,3 +2436,78 @@ mute is never lost to an old server. The sharer's own watchdog works |
| whatever the server is. Deploy the server before, or with, the clients. |
| **Not logged, not stored.** As §15.19. |
| + |
| +### 15.22 §15.21 — when keyframes are not enough (2026-09-30) |
| + |
| +**Why.** Paul, 2026-09-30, sharing from the Mac app (0.4.3 or 0.4.4) to |
| +Samuel: "Still seeing issues with screen sharing after a while." His |
| +screenshot had §15.21's notice, "Samuel's view of your screen stalled — |
| +recovering", under "You are sharing your screen". The receiver's report had |
| +worked; recovery had not. By design, a report alone only ever bought |
| +keyframes, and a stall that keyframes do not mend — a receiver whose decoder |
| +has wedged, or one that has stopped taking this stream — had nothing after |
| +them: the notice sat at "recovering" until the report lapsed a minute later. |
| +Stop and Share, by hand, always mended it. Two more things came up reading the |
| +code: a receiver counted only a *larger* decoded count as movement, so a |
| +screen that had been given a new transceiver (whose count starts again) |
| +stayed "stalled" there for as long as its new count took to pass the old one; |
| +and the iPhone's broadcast sends nothing for a still screen, which any |
| +receiver reported as a stall after four seconds. |
| + |
| +**What the harness measured.** A receiver whose screen decoder is wedged |
| +(the gate's decoder wrapper in the apps, an encoded transform in Chromium) keeps |
| +receiving packets and frames, decodes none, and drops them: packets +75, |
| +frames received +32, decoded +0, dropped +29 in the first four seconds, between |
| +two apps. Keyframes arrive and are dropped with the rest. A new transceiver, |
| +which gets a new receiver and decoder at the far end, mends it at once, in both |
| +directions and between two apps. No long share (§15.22's soaks, below) has yet |
| +produced a stall that keyframes did not mend, so the wedge is a stand-in for the |
| +cause, not the cause. |
| + |
| +**Decision.** The sharer takes a report further, the receiver does its own |
| +share, and both say what they see. |
| + |
| +- **The sharer's ladder**, per reporting peer: keyframes as §15.21. Still |
| + reported six seconds after the first, **that connection's screen is |
| + rebuilt** — its transceiver stopped and a new one added once that has been |
| + negotiated, at the same size and mode; a report never shrinks the screen or |
| + makes a smaller encoder. Still reported ten seconds after that, |
| + **everyone's screen is shared again** from the same capture: every screen |
| + transceiver stopped, a new track on the same source (a clone of the captured |
| + track in the browser), each connection given it as it settles; no picker. |
| + Ten seconds after that, it gives up. A rebuild at most once a minute per |
| + peer, a share again at most once every five minutes; a step that is not |
| + allowed goes to the next, and none is taken unless at least three frames |
| + left for that peer since the last step (a screen with nothing new to send, |
| + or held back by the link, is not a decoder's fault). A report that clears |
| + ends the ladder for that peer. |
| +- **The notice moves on in words**: "Samuel's view of your screen stalled — |
| + recovering", then "Restarting your share for Samuel…", then either nothing |
| + or "Couldn't restore Samuel's view — try Stop and Share again". Worst news |
| + first when people are at different steps. |
| +- **The receiver**: any change in a screen's decoded count is movement. Four |
| + seconds into a stall it hands the screen's picture to its view again, once |
| + per stall (a new renderer in the apps, a fresh stream on the element in the |
| + browser): a picture stopped on this side is this side's to mend. Neither |
| + WebRTC build has a way for a receiver to ask for a keyframe directly; the |
| + report is that request, and WebRTC's own PLIs go out as they always did. A |
| + report is held for up to three minutes from the last frame, not one, so the |
| + sharer's "Couldn't restore" is not taken away while it is still true. |
| +- **The iPhone's broadcast repeats its last frame once a second** while the |
| + screen is still, as the Mac's capture has since §15.21. |
| +- **What is logged.** Every step, both sides, in the screen log (`os_log` |
| + category `screen`; the browser's console), and every stall with what the |
| + receiver's statistics say moved since the screen last did: packets, frames |
| + received, decoded, dropped, keyframes, PLI and NACK, the decoder, and a |
| + verdict (nothing arriving; arriving but not assembled; assembled but not |
| + decoded). "Copy call diagnostics" carries the ladder's counts and stage per |
| + peer, every arriving screen's statistics, and the last 200 lines of the |
| + timeline, on the sharer and the receiver alike. The browser gains the same |
| + button, at the foot of its Devices panel. |
| + |
| +**Compatibility.** Nothing on the wire changes. An older receiver reports as |
| +before and is answered with the new ladder; an older sharer answers a new |
| +receiver's longer report with keyframes as before. |
| + |
| +**Not logged, not stored.** As §15.19; the timeline stays on the device until |
| +someone copies it. |
diff --git a/internal/calls/static/calls.css b/internal/calls/static/calls.css| index 6766953..4679721 100644 |
| --- a/internal/calls/static/calls.css |
| +++ b/internal/calls/static/calls.css |
| @@ -457,6 +457,9 @@ body:has(> .calls-window) { background: var(--rst-bg); } |
| text-align: center; |
| } |
| .calls-window .calls-device-note { color: #fff; } |
| +/* "Copy call diagnostics" at the foot of the Devices popover (§15.22). */ |
| +.calls-copy-diagnostics { margin-block-start: var(--rst-sp-3); align-self: flex-start; } |
| +.calls-window .calls-copy-diagnostics { color: #fff; } |
| /* The sharing notice (2026-09-28): a dark pill with its own Stop |
| sharing button, and the recovery line under it. */ |
diff --git a/internal/calls/static/room.js b/internal/calls/static/room.js| index 7fdc6da..7aa2c24 100644 |
| --- a/internal/calls/static/room.js |
| +++ b/internal/calls/static/room.js |
| @@ -71,6 +71,7 @@ |
| const shareNotice = $('share-notice'); |
| const shareStop = $('share-stop'); |
| const shareRecoveryEl = $('share-recovery'); |
| + const copyDiagnostics = $('copy-diagnostics'); |
| // §15.19: the tile marks, rendered by the server so they are the |
| // vendored badge and the app's own icons rather than a copy of either, |
| // and the polite region that says a change out loud. #status is no |
| @@ -277,10 +278,31 @@ |
| // for a keyframe, at most once every SCREEN_KEYFRAME_EVERY, and rests |
| // SCREEN_KEYFRAME_REST after SCREEN_KEYFRAME_TRIES of them. |
| const SCREEN_STALL_AFTER = 4000; |
| - const SCREEN_STALL_GIVE_UP = 60000; |
| + const SCREEN_STALL_GIVE_UP = 180000; |
| const SCREEN_KEYFRAME_EVERY = 4000; |
| const SCREEN_KEYFRAME_TRIES = 3; |
| const SCREEN_KEYFRAME_REST = 30000; |
| + // §15.22 (2026-09-30), when keyframes do not answer. The receiver |
| + // hands a stalled screen's picture to its element again once, |
| + // SCREEN_REATTACH_AFTER into the stall. The sharer, still named |
| + // SCREEN_REBUILD_AFTER after its first keyframe, rebuilds that |
| + // connection's screen (a new transceiver: a new encoder here, a new |
| + // receiver and decoder there); still named SCREEN_RESHARE_AFTER after |
| + // that, shares again for everyone from the same capture; and |
| + // SCREEN_GIVE_UP_AFTER after that, says it could not. A rebuild at most |
| + // once every SCREEN_REBUILD_EVERY per person, a share again at most |
| + // once every SCREEN_RESHARE_EVERY, and neither unless at least |
| + // SCREEN_ESCALATE_SENT frames left for that person since the last step: |
| + // a receiver cannot be failing to decode frames that were never sent. |
| + const SCREEN_REATTACH_AFTER = 4000; |
| + const SCREEN_REBUILD_AFTER = 6000; |
| + const SCREEN_RESHARE_AFTER = 10000; |
| + const SCREEN_GIVE_UP_AFTER = 10000; |
| + const SCREEN_REBUILD_EVERY = 60000; |
| + const SCREEN_RESHARE_EVERY = 300000; |
| + const SCREEN_ESCALATE_SENT = 3; |
| + // The screen's timeline, for "Copy call diagnostics": its last lines. |
| + const SCREEN_LOG_LINES = 200; |
| // §5.4, sending: "Optimise shared screen for". What a screen gives up |
| // first when the uplink or the encoder cannot carry all of it. The |
| // first is the default; sharePlan says what each one means. |
| @@ -345,6 +367,17 @@ |
| const screenAsks = new Map(); |
| let shareRecovery = ''; |
| let screenStallTestUntil = 0; |
| + // §15.22: each arriving screen's statistics at the last poll and when |
| + // it last moved (share key -> screenReading); pictures handed back; |
| + // when each peer's screen was last rebuilt, and everyone's last shared |
| + // again; and the timeline itself. |
| + const screenReadings = new Map(); |
| + const screenMoving = new Map(); |
| + let screenReattaches = 0; |
| + const shareRebuiltAt = new Map(); |
| + let shareResharedAt = 0; |
| + const shareCounts = { keyframes: 0, rebuilds: 0, reshares: 0, failures: 0 }; |
| + const screenLines = []; |
| // §15.20: peer id -> what this tab is hearing from that peer, as |
| // pollLevels reads it (hearingRecord). selfHearing is the same record |
| // for our own microphone, read off the sender. |
| @@ -4303,25 +4336,110 @@ |
| // judgeScreen is the receiver's verdict on one screen, from its decoded |
| // count this poll: stalled, or not. `seen` is its record, updated here. |
| // Pure over its arguments, and lifted by name in the node test. |
| + // |
| + // Any change in the count is movement, a smaller one included: a |
| + // screen the sharer has rebuilt arrives on a new receiver whose count |
| + // starts again, and up to 2026-09-30 only a larger count moved, so a |
| + // rebuilt screen that was playing again stayed "stalled" here — and the |
| + // sharer's notice with it — until the new count passed the old. |
| function judgeScreen(seen, decoded, now, frozen) { |
| - if (!seen.movedAt || (decoded > seen.decoded && !frozen)) { |
| + if (!seen.movedAt || (decoded !== seen.decoded && !frozen)) { |
| seen.decoded = decoded; |
| seen.movedAt = now; |
| + seen.reattached = false; |
| return false; |
| } |
| const quiet = now - seen.movedAt; |
| return quiet >= SCREEN_STALL_AFTER && quiet < SCREEN_STALL_GIVE_UP; |
| } |
| + // reattachDue says, once per stall, when a stalled screen's picture |
| + // should be handed to its element again (§15.22): SCREEN_REATTACH_AFTER |
| + // into the stall. A picture stopped on this side — an element or a |
| + // track Chromium has left muted — is this side's to mend. Pure over |
| + // `seen`, and lifted by name in the node test. |
| + function reattachDue(seen, now) { |
| + const quiet = now - seen.movedAt; |
| + if (seen.reattached || quiet < SCREEN_STALL_AFTER + SCREEN_REATTACH_AFTER || quiet >= SCREEN_STALL_GIVE_UP) { |
| + return false; |
| + } |
| + seen.reattached = true; |
| + return true; |
| + } |
| + |
| + // screenLog is one line of the screen's timeline: the console, and the |
| + // last SCREEN_LOG_LINES kept for "Copy call diagnostics". |
| + function screenLog(line) { |
| + const d = new Date(); |
| + const pad = (n, w) => String(n).padStart(w || 2, '0'); |
| + const stamp = pad(d.getHours()) + ':' + pad(d.getMinutes()) + ':' + pad(d.getSeconds()) + '.' + pad(d.getMilliseconds(), 3); |
| + screenLines.push(stamp + ' ' + line); |
| + if (screenLines.length > SCREEN_LOG_LINES) screenLines.splice(0, screenLines.length - SCREEN_LOG_LINES); |
| + console.info('calls: ' + line); |
| + } |
| + |
| + // screenReading is what one inbound screen report says, and |
| + // readingSummary and readingMoved put it in words. Pure, and lifted by |
| + // name in the node test: packets arriving and frames assembled but none |
| + // decoded is a decoder; nothing arriving is the sender or the path. |
| + function screenReading(r, trackMuted) { |
| + return { |
| + mid: r.mid || '', |
| + packets: r.packetsReceived || 0, |
| + lost: r.packetsLost || 0, |
| + received: r.framesReceived || 0, |
| + decoded: r.framesDecoded || 0, |
| + dropped: r.framesDropped || 0, |
| + keyframes: r.keyFramesDecoded || 0, |
| + pli: r.pliCount || 0, |
| + fir: r.firCount || 0, |
| + nack: r.nackCount || 0, |
| + freezes: r.freezeCount || 0, |
| + width: r.frameWidth || 0, |
| + height: r.frameHeight || 0, |
| + decoder: r.decoderImplementation || '', |
| + muted: !!trackMuted, |
| + }; |
| + } |
| + |
| + function readingSummary(r) { |
| + return 'mid ' + (r.mid || '-') + ', ' + r.width + 'x' + r.height + ', decoder ' + (r.decoder || 'unknown') + |
| + (r.muted ? ', track muted' : '') + '; packets ' + r.packets + ' (lost ' + r.lost + '), frames received ' + |
| + r.received + ', decoded ' + r.decoded + ', dropped ' + r.dropped + ', keyframes ' + r.keyframes + |
| + ', PLI ' + r.pli + ', FIR ' + r.fir + ', NACK ' + r.nack + ', freezes ' + r.freezes; |
| + } |
| + |
| + function readingMoved(r, b) { |
| + if (!b || b.mid !== r.mid) return readingSummary(r) + ' — no earlier reading'; |
| + const d = (a, z) => (a >= z ? '+' + (a - z) : 'new ' + a); |
| + let verdict = 'decoding'; |
| + if (r.packets === b.packets) verdict = 'no packets arriving (the sender or the path)'; |
| + else if (r.received === b.received) verdict = 'packets arriving, no frames assembled (loss, or waiting on a keyframe)'; |
| + else if (r.decoded === b.decoded) verdict = 'frames arriving, none decoded (the decoder)'; |
| + return 'packets ' + d(r.packets, b.packets) + ', frames received ' + d(r.received, b.received) + ', decoded ' + |
| + d(r.decoded, b.decoded) + ', dropped ' + d(r.dropped, b.dropped) + ', keyframes ' + d(r.keyframes, b.keyframes) + |
| + ', PLI ' + d(r.pli, b.pli) + ', NACK ' + d(r.nack, b.nack) + (r.muted ? ', track muted' : '') + ' — ' + verdict; |
| + } |
| + |
| // watchScreens takes one poll's decoded counts, by share key, and |
| // reports the list when it changes. |
| - function watchScreens(decodedByKey) { |
| + // |
| + // `readings` is the same poll's screenReading per share key: kept for |
| + // the diagnostics, and set against the reading from when the screen last |
| + // moved, so a stall is logged with what is and is not still arriving. |
| + function watchScreens(decodedByKey, readings) { |
| const now = Date.now(); |
| const frozen = now < screenStallTestUntil; |
| const stalled = []; |
| for (const key of Array.from(screenSeen.keys())) { |
| if (!decodedByKey.has(key)) screenSeen.delete(key); |
| } |
| + for (const key of Array.from(screenReadings.keys())) { |
| + if (!decodedByKey.has(key)) { |
| + screenReadings.delete(key); |
| + screenMoving.delete(key); |
| + } |
| + } |
| for (const [key, decoded] of decodedByKey) { |
| const id = key.slice(0, -SHARE_SUFFIX.length); |
| const entry = pcs.get(id); |
| @@ -4331,57 +4449,289 @@ |
| } |
| let seen = screenSeen.get(key); |
| if (!seen) { |
| - seen = { decoded: 0, movedAt: 0 }; |
| + seen = { decoded: 0, movedAt: 0, reattached: false }; |
| screenSeen.set(key, seen); |
| } |
| - if (judgeScreen(seen, decoded, now, frozen)) stalled.push(id); |
| + const reading = readings && readings.get(key); |
| + if (reading) screenReadings.set(key, reading); |
| + if (judgeScreen(seen, decoded, now, frozen)) { |
| + stalled.push(id); |
| + if (reattachDue(seen, now)) { |
| + screenLog('screen from ' + id.slice(0, 6) + ' still stalled: handing its picture to the tile again; ' + |
| + (reading ? readingMoved(reading, screenMoving.get(key)) : 'no reading')); |
| + const t = tiles.get(key); |
| + if (t && t.video) { |
| + screenReattaches += 1; |
| + replayTile(key, t, entry, true); |
| + } |
| + } |
| + } else if (reading) { |
| + screenMoving.set(key, reading); |
| + } |
| } |
| stalled.sort(); |
| if (stalled.join() === screenStalled.join()) return; |
| for (const id of stalled) { |
| - if (!screenStalled.includes(id)) console.info('calls: screen from ' + id.slice(0, 6) + ' stalled here, asking for a keyframe'); |
| + if (screenStalled.includes(id)) continue; |
| + const key = shareKey(id); |
| + const reading = screenReadings.get(key); |
| + screenLog('screen from ' + id.slice(0, 6) + ' stalled here: reporting it; since it last moved ' + |
| + (reading ? readingMoved(reading, screenMoving.get(key)) : 'no reading')); |
| + } |
| + for (const id of screenStalled) { |
| + if (stalled.includes(id)) continue; |
| + const reading = screenReadings.get(shareKey(id)); |
| + screenLog('screen from ' + id.slice(0, 6) + ' moving here again' + (reading ? ': ' + readingSummary(reading) : '')); |
| } |
| screenStalled = stalled; |
| reportMedia(); |
| } |
| // answerScreenStalls is the sharer's half: every peer whose report |
| - // names us gets a keyframe from our screen's encoder, and the quiet |
| - // notice says whose view is being put right. |
| + // names us gets a keyframe from our screen's encoder, and — when |
| + // keyframes do not answer (§15.22) — that connection's screen rebuilt, |
| + // then everyone's shared again from the same capture, and then words |
| + // saying it could not be restored. The quiet notice says which. |
| function answerScreenStalls() { |
| const me = state.peer; |
| - const names_ = []; |
| const now = Date.now(); |
| + let reshare = false; |
| for (const [id, m] of remoteMedia) { |
| const asking = !!(share && me && m && m.screenStalled && m.screenStalled.includes(me)); |
| if (!asking) { |
| + const gone = screenAsks.get(id); |
| + if (gone && gone.stage && share) screenLog(id.slice(0, 6) + '\u2019s view restored after ' + stageWords(gone.stage)); |
| screenAsks.delete(id); |
| continue; |
| } |
| - names_.push(names.get(id) || 'Someone'); |
| const entry = pcs.get(id); |
| let ask = screenAsks.get(id); |
| if (!ask) { |
| - ask = { at: 0, tries: 0, restUntil: 0 }; |
| + ask = { at: 0, tries: 0, restUntil: 0, stage: '', stageAt: 0 }; |
| screenAsks.set(id, ask); |
| + if (entry) entry.shareSentSince = 0; |
| + } |
| + if (entry && now >= ask.restUntil && now - ask.at >= SCREEN_KEYFRAME_EVERY) { |
| + ask.at = now; |
| + ask.tries += 1; |
| + if (ask.tries >= SCREEN_KEYFRAME_TRIES) { |
| + ask.tries = 0; |
| + ask.restUntil = now + SCREEN_KEYFRAME_REST; |
| + } |
| + shareCounts.keyframes += 1; |
| + screenLog('keyframe for ' + id.slice(0, 6) + ', whose view of the screen stalled'); |
| + shareKeyframe(entry).catch(() => {}); |
| } |
| - if (!entry || now < ask.restUntil || now - ask.at < SCREEN_KEYFRAME_EVERY) continue; |
| - ask.at = now; |
| - ask.tries += 1; |
| - if (ask.tries >= SCREEN_KEYFRAME_TRIES) { |
| - ask.tries = 0; |
| - ask.restUntil = now + SCREEN_KEYFRAME_REST; |
| + // Frames left for this person since the last step, or no screen |
| + // negotiated with them at all (a rebuild that never came back). |
| + const negotiated = !!(entry && entry.shareTx && entry.shareSend); |
| + const owed = !negotiated || (entry.shareSentSince || 0) >= SCREEN_ESCALATE_SENT; |
| + const rec = { stage: ask.stage, at: ask.stageAt, lastRebuild: shareRebuiltAt.get(id) || 0 }; |
| + const before = rec.stage; |
| + const reshareFree = reshare || !shareResharedAt || now - shareResharedAt >= SCREEN_RESHARE_EVERY; |
| + const action = escalateShare(rec, now, owed, reshareFree); |
| + ask.stage = rec.stage; |
| + ask.stageAt = rec.at; |
| + if (rec.stage !== before && entry) entry.shareSentSince = 0; |
| + if (action === 'rebuild') { |
| + shareRebuiltAt.set(id, now); |
| + shareCounts.rebuilds += 1; |
| + screenLog(id.slice(0, 6) + ' still reports a stall after keyframes: rebuilding its screen (new transceiver, same size and mode)'); |
| + if (entry) rebuildShareFor(entry); |
| + } else if (action === 'reshare') { |
| + reshare = true; |
| + screenLog(id.slice(0, 6) + ' still reports a stall' + (before === 'rebuilt' ? ' after the rebuild' : ', rebuilt under a minute ago') + ': sharing again'); |
| + } else if (action === 'failed') { |
| + shareCounts.failures += 1; |
| + screenLog(id.slice(0, 6) + ' still reports a stall and there is nothing left to try: giving up'); |
| } |
| - console.info('calls: keyframe for ' + id.slice(0, 6) + ', whose view of the screen stalled'); |
| - shareKeyframe(entry).catch(() => {}); |
| } |
| - const text = recoveringWords(names_); |
| + if (reshare) { |
| + shareResharedAt = now; |
| + shareCounts.reshares += 1; |
| + // One share again answers every report in flight. |
| + for (const ask of screenAsks.values()) { |
| + if (ask.stage === 'keyframes' || ask.stage === 'rebuilt') { |
| + ask.stage = 'reshared'; |
| + ask.stageAt = now; |
| + } |
| + } |
| + for (const entry of pcs.values()) entry.shareSentSince = 0; |
| + reshareAll(); |
| + } |
| + const recovering = []; |
| + const restarting = []; |
| + const failed = []; |
| + for (const [id, ask] of screenAsks) { |
| + const who = names.get(id) || 'Someone'; |
| + if (ask.stage === 'failed') failed.push(who); |
| + else if (ask.stage === 'rebuilt' || ask.stage === 'reshared') restarting.push(who); |
| + else recovering.push(who); |
| + } |
| + const text = noticeWords(recovering.sort(), restarting.sort(), failed.sort()); |
| if (text !== shareRecovery) { |
| shareRecovery = text; |
| renderShare(); |
| } |
| } |
| + // callDiagnostics is "Copy call diagnostics" for the browser: the |
| + // connections, and the screen — how ours is leaving and what the ladder |
| + // has done, every screen arriving here as its statistics have it, and |
| + // the timeline. Names, states and counts only: no media, no addresses, |
| + // no credentials. |
| + function callDiagnostics() { |
| + const lines = ['Calls web diagnostics, ' + new Date().toISOString()]; |
| + lines.push('Browser: ' + (navigator.userAgent || 'unknown')); |
| + lines.push(''); |
| + lines.push('## Call'); |
| + lines.push('admitted ' + state.admitted + ', people here ' + (pcs.size + 1) + ', stopped ' + state.stopped); |
| + for (const [id, entry] of pcs) { |
| + lines.push((names.get(id) || 'Someone') + ' (' + id.slice(0, 6) + '): connection ' + entry.pc.connectionState + |
| + ', signalling ' + entry.pc.signalingState + ', ICE ' + entry.pc.iceConnectionState); |
| + } |
| + lines.push(''); |
| + lines.push('## Screen'); |
| + if (share) { |
| + lines.push('Sharing (' + shareMode() + ')'); |
| + for (const [id, entry] of pcs) { |
| + const r = entry.shareSend; |
| + const who = names.get(id) || 'Someone'; |
| + if (!r) { |
| + lines.push('to ' + who + ': ' + (entry.shareTx ? 'negotiating' : 'no screen transceiver')); |
| + continue; |
| + } |
| + lines.push('to ' + who + ': ' + r.width + 'x' + r.height + ' at ' + Math.round(r.fps) + ' fps, encoder ' + |
| + (r.encoder || 'unknown') + ', encoded ' + r.framesEncoded + ', sent ' + r.framesSent + ', keyframes ' + |
| + r.keyFramesEncoded + ', limitation ' + (r.limitation || 'none')); |
| + } |
| + for (const [id, ask] of screenAsks) { |
| + if (ask.stage) lines.push('recovering ' + (names.get(id) || 'Someone') + '\u2019s view: ' + ask.stage); |
| + } |
| + } else { |
| + lines.push('Not sharing'); |
| + } |
| + lines.push('recoveries: ' + shareCounts.keyframes + ' keyframes, ' + shareCounts.rebuilds + ' rebuilds, ' + |
| + shareCounts.reshares + ' shares again, ' + shareCounts.failures + ' given up'); |
| + for (const [key, reading] of screenReadings) { |
| + const id = key.slice(0, -SHARE_SUFFIX.length); |
| + lines.push('from ' + (names.get(id) || 'Someone') + ': ' + readingSummary(reading)); |
| + } |
| + if (screenStalled.length) lines.push('stalled here: ' + screenStalled.map((id) => names.get(id) || 'Someone').join(', ')); |
| + if (screenReattaches) lines.push('pictures handed to their tiles again here: ' + screenReattaches); |
| + lines.push(''); |
| + lines.push('## Screen log (last ' + SCREEN_LOG_LINES + ' lines)'); |
| + if (screenLines.length) lines.push(...screenLines); |
| + else lines.push('(nothing logged)'); |
| + return lines.join('\n'); |
| + } |
| + |
| + // noteShareSent counts the frames that left on this connection's |
| + // screen since its escalation last moved on, across a new transceiver's |
| + // counters starting again from nothing. |
| + function noteShareSent(entry) { |
| + if (!entry.shareSend) return; |
| + const sent = entry.shareSend.framesSent || 0; |
| + const last = entry.shareSentLast || 0; |
| + entry.shareSentSince = (entry.shareSentSince || 0) + (sent >= last ? sent - last : sent); |
| + entry.shareSentLast = sent; |
| + } |
| + |
| + // escalateShare is §15.22's ladder for one reporting peer: '' (nothing |
| + // now), 'rebuild', 'reshare' or 'failed', moving `rec` ({stage, at, |
| + // lastRebuild}) on. `owed` is whether frames have left for that person |
| + // since the last step; `reshareFree` whether a share again may happen |
| + // now. The native app's ScreenWatchdog is its twin. Pure, and lifted by |
| + // name in the node test. |
| + function escalateShare(rec, now, owed, reshareFree) { |
| + const move = (stage) => { |
| + rec.stage = stage; |
| + rec.at = now; |
| + }; |
| + if (!rec.stage) { |
| + move('keyframes'); |
| + return ''; |
| + } |
| + if (rec.stage === 'keyframes') { |
| + if (now - rec.at < SCREEN_REBUILD_AFTER || !owed) return ''; |
| + if (!rec.lastRebuild || now - rec.lastRebuild >= SCREEN_REBUILD_EVERY) { |
| + move('rebuilt'); |
| + rec.lastRebuild = now; |
| + return 'rebuild'; |
| + } |
| + } else if (rec.stage === 'rebuilt') { |
| + if (now - rec.at < SCREEN_RESHARE_AFTER || !owed) return ''; |
| + } else if (rec.stage === 'reshared') { |
| + if (now - rec.at < SCREEN_GIVE_UP_AFTER || !owed) return ''; |
| + move('failed'); |
| + return 'failed'; |
| + } else { |
| + return ''; |
| + } |
| + if (reshareFree) { |
| + move('reshared'); |
| + return 'reshare'; |
| + } |
| + move('failed'); |
| + return 'failed'; |
| + } |
| + |
| + function stageWords(stage) { |
| + return { keyframes: 'keyframes', rebuilt: 'the rebuild', reshared: 'sharing again', failed: 'giving up' }[stage] || stage; |
| + } |
| + |
| + // rebuildShareFor is a stop and share again for one person (§15.22): |
| + // this connection's screen transceiver stopped, and a new one added by |
| + // the connection's own signalingstatechange once that has been |
| + // negotiated — a new encoder here, a new receiver and decoder there, at |
| + // the same size and mode. |
| + function rebuildShareFor(entry) { |
| + if (!share || !entry.shareTx) return; |
| + removeShareFrom(entry); |
| + sayShareAgain(); |
| + } |
| + |
| + // reshareAll is Stop and Share for everyone without the picker: every |
| + // connection's screen transceiver stopped, and a clone of the captured |
| + // track — the same capture, a new track — added to each as it settles. |
| + function reshareAll() { |
| + if (!share) return; |
| + const held = share; |
| + let track; |
| + try { |
| + track = held.track.clone(); |
| + } catch { |
| + return; |
| + } |
| + try { |
| + track.contentHint = held.track.contentHint; |
| + } catch { |
| + // A browser without contentHint. |
| + } |
| + // The browser's own Stop sharing bar ends every clone. |
| + track.addEventListener('ended', () => { |
| + if (share && share.track === track) stopSharing(); |
| + }); |
| + for (const entry of pcs.values()) removeShareFrom(entry); |
| + share = { stream: new MediaStream([track]), track }; |
| + // The old one after: stopping it alone does not fire its own |
| + // `ended`, and the clone keeps the capture running. |
| + try { |
| + held.track.stop(); |
| + } catch { |
| + // Already stopped. |
| + } |
| + sayShareAgain(); |
| + } |
| + |
| + // sayShareAgain repeats `share {on:true}` once a rebuilt screen has had |
| + // time to arrive: its new track comes without a label of its own. |
| + function sayShareAgain() { |
| + setTimeout(() => { |
| + if (share) broadcast('share', { on: true }); |
| + }, 2000); |
| + } |
| + |
| // recoveringWords is the sharer's notice. Pure, and lifted by name in |
| // the node test. |
| function recoveringWords(people) { |
| @@ -4393,6 +4743,30 @@ |
| return people.length + ' people\u2019s views of your screen stalled \u2014 recovering'; |
| } |
| + // noticeWords is the notice as §15.22's ladder moves on: recovering, |
| + // then "Restarting your share for Samuel…", then either nothing or |
| + // "Couldn’t restore Samuel’s view — try Stop and Share again". Worst |
| + // news first. The native app's ScreenWords.notice, word for word. Pure, |
| + // and lifted by name in the node test (with recoveringWords). |
| + function noticeWords(recovering, restarting, failed) { |
| + const parts = []; |
| + if (failed.length) { |
| + let whose = failed.length + ' people\u2019s views'; |
| + if (failed.length === 1) whose = failed[0] + '\u2019s view'; |
| + else if (failed.length === 2) whose = failed[0] + '\u2019s and ' + failed[1] + '\u2019s views'; |
| + parts.push('Couldn\u2019t restore ' + whose + ' \u2014 try Stop and Share again'); |
| + } |
| + if (restarting.length) { |
| + let who = restarting.length + ' people'; |
| + if (restarting.length === 1) who = restarting[0]; |
| + else if (restarting.length === 2) who = restarting[0] + ' and ' + restarting[1]; |
| + parts.push('Restarting your share for ' + who + '\u2026'); |
| + } |
| + const stalled = recoveringWords(recovering); |
| + if (stalled) parts.push(stalled); |
| + return parts.join('. '); |
| + } |
| + |
| // shareKeyframe switches this connection's screen encoding off and on |
| // again, which WebRTC answers by starting it on a keyframe: there is |
| // no API that asks for one directly. |
| @@ -5694,6 +6068,21 @@ |
| }); |
| } |
| + // "Copy call diagnostics" (§15.22), and a sentence in the device |
| + // panel's own live region either way. |
| + if (copyDiagnostics) { |
| + copyDiagnostics.addEventListener('click', () => { |
| + const text = callDiagnostics(); |
| + const done = () => note('Call diagnostics copied.'); |
| + const failed = () => note('The diagnostics could not be copied.'); |
| + try { |
| + navigator.clipboard.writeText(text).then(done, failed); |
| + } catch { |
| + failed(); |
| + } |
| + }); |
| + } |
| + |
| leaveButton.addEventListener('click', () => { |
| leave(); |
| go('/m/' + encodeURIComponent(sqid) + '/join'); |
| @@ -5921,6 +6310,8 @@ |
| targetBitrate: r.targetBitrate || 0, |
| encoder: r.encoderImplementation || '', |
| framesEncoded: r.framesEncoded || 0, |
| + framesSent: r.framesSent || 0, |
| + keyFramesEncoded: r.keyFramesEncoded || 0, |
| }; |
| } |
| @@ -6165,8 +6556,10 @@ |
| // callback was dropped is put back on the books (Codex S4-6's gate |
| // found one, and the counter is what every frames assertion reads). |
| rearmFrames(); |
| - // §15.21: every screen's decoded count, for the receiver's watch. |
| + // §15.21: every screen's decoded count, for the receiver's watch, and |
| + // (§15.22) the rest of what its statistics say. |
| const screens = new Map(); |
| + const screenReports = new Map(); |
| for (const [id, entry] of pcs) { |
| try { |
| const stats = await entry.pc.getStats(); |
| @@ -6188,7 +6581,11 @@ |
| const key = frameKey(id, entry, report.mid || ''); |
| if (!key) return; |
| decoded.set(key, report.framesDecoded); |
| - if (key !== id) screens.set(key, report.framesDecoded); |
| + if (key !== id) { |
| + screens.set(key, report.framesDecoded); |
| + const t = tiles.get(key); |
| + screenReports.set(key, screenReading(report, !!(t && t.track && t.track.muted))); |
| + } |
| if (!HAS_RVFC) frames.set(key, report.framesDecoded); |
| }); |
| for (const [key, count] of decoded) { |
| @@ -6199,6 +6596,7 @@ |
| else if (verdict === 'reset') requestReset(id); |
| } |
| entry.shareSend = shareSendReport(outbound, (entry.shareTx && entry.shareTx.mid) || ''); |
| + noteShareSent(entry); |
| const tx = localCameraTx(entry); |
| const verdict = judgeCamera(entry, tx, cameraReport(outbound, (tx && tx.mid) || '')); |
| if (verdict === 'repair') await repairCamera(tx); |
| @@ -6207,7 +6605,7 @@ |
| // getStats on a closing connection: nothing to report. |
| } |
| } |
| - watchScreens(screens); |
| + watchScreens(screens, screenReports); |
| answerScreenStalls(); |
| } |
| @@ -6341,6 +6739,10 @@ |
| // §15.21, for the gate: this tab's receiver watch treats every |
| // screen as frozen for `ms`, so its stall report and the sharer's |
| // keyframe can be driven end to end. |
| + // §15.22: "Copy call diagnostics" as it would be copied. |
| + diagnostics() { |
| + return callDiagnostics(); |
| + }, |
| stallScreenForTest(ms) { |
| screenStallTestUntil = Date.now() + Math.max(0, Math.min(Number(ms) || 0, 60000)); |
| }, |
| @@ -6371,6 +6773,8 @@ |
| // sharer's notice while someone's view of ours is being put right. |
| screenStalled: screenStalled.slice(), |
| shareRecovery, |
| + // §15.22: what the ladder and the receiver's own mending have done. |
| + shareRecoveries: Object.assign({ reattaches: screenReattaches }, shareCounts), |
| // Targets a 404 has marked stale: nothing is sent to them and |
| // no reset is asked for until a roster settles it. |
| stale: Array.from(stale), |
diff --git a/internal/calls/templates/room.html b/internal/calls/templates/room.html| index ea18507..3bf2521 100644 |
| --- a/internal/calls/templates/room.html |
| +++ b/internal/calls/templates/room.html |
| @@ -172,6 +172,12 @@ |
| render, so a sentence about a camera that would not open has |
| to live somewhere that is not about to be overwritten. */}} |
| <p id="device-note" class="calls-device-note" role="status" hidden></p> |
| + {{/* What the Mac panel's "Copy call diagnostics" copies, for the |
| + browser: the connections, and the screen's recovery timeline and |
| + decode statistics on both sides of a share (spec §15.22). Names, |
| + states and counts only; room.js says in #device-note whether it |
| + was copied. */}} |
| + <button type="button" id="copy-diagnostics" rst-btn="ghost" class="calls-copy-diagnostics">Copy call diagnostics</button> |
| </div> |
| {{/* Ending the meeting for everyone is destructive, so it is a link to |
| its own confirm page (AGENTS.md), opened in a new tab so that |
diff --git a/internal/callstest/native_browser_test.go b/internal/callstest/native_browser_test.go| index e4cadaf..efb643f 100644 |
| --- a/internal/callstest/native_browser_test.go |
| +++ b/internal/callstest/native_browser_test.go |
| @@ -11,6 +11,8 @@ import ( |
| "os" |
| "os/exec" |
| "path/filepath" |
| + "sort" |
| + "strconv" |
| "strings" |
| "sync" |
| "testing" |
| @@ -221,6 +223,8 @@ type nativeSample struct { |
| Stats map[string]uint64 `json:"stats"` |
| ScreenCaptions []string `json:"screenCaptions"` |
| SharingScreen *bool `json:"sharingScreen"` |
| + ScreenRecovery string `json:"screenRecovery"` |
| + Diagnostics string `json:"diagnostics"` |
| } |
| // nativeApp is one CallsInterop process and the last observation it |
| @@ -231,6 +235,7 @@ type nativeApp struct { |
| stdin io.WriteCloser |
| mu sync.Mutex |
| last nativeSample |
| + diag string |
| done chan struct{} |
| } |
| @@ -256,7 +261,16 @@ func startNative(ctx context.Context, t *testing.T, binary string, rig *browserR |
| scanner := bufio.NewScanner(out) |
| for scanner.Scan() { |
| var s nativeSample |
| - if json.Unmarshal(scanner.Bytes(), &s) != nil || s.Stats == nil { |
| + if json.Unmarshal(scanner.Bytes(), &s) != nil { |
| + continue |
| + } |
| + if s.Diagnostics != "" { |
| + app.mu.Lock() |
| + app.diag = s.Diagnostics |
| + app.mu.Unlock() |
| + continue |
| + } |
| + if s.Stats == nil { |
| continue |
| } |
| app.mu.Lock() |
| @@ -278,6 +292,25 @@ func (n *nativeApp) sample() nativeSample { |
| func (n *nativeApp) stat(name string) uint64 { return n.sample().Stats[name] } |
| +// diagnostics asks for "Copy call diagnostics" and returns what came |
| +// back, or "" if nothing did within five seconds. |
| +func (n *nativeApp) diagnostics(t *testing.T) string { |
| + t.Helper() |
| + n.mu.Lock() |
| + n.diag = "" |
| + n.mu.Unlock() |
| + n.command(t, "diagnostics") |
| + for until := time.Now().Add(5 * time.Second); time.Now().Before(until); time.Sleep(100 * time.Millisecond) { |
| + n.mu.Lock() |
| + d := n.diag |
| + n.mu.Unlock() |
| + if d != "" { |
| + return d |
| + } |
| + } |
| + return "" |
| +} |
| + |
| func (n *nativeApp) command(t *testing.T, line string) { |
| t.Helper() |
| if _, err := io.WriteString(n.stdin, line+"\n"); err != nil { |
| @@ -373,24 +406,31 @@ func TestNativeMacScreenRecovery(t *testing.T) { |
| if _, err := os.Stat(binary); err != nil { |
| t.Fatal("build native interop executable with make native-mac first") |
| } |
| - scenarios := []string{"renewals", "resume", "capture", "encoder", "loss"} |
| + scenarios := []string{"renewals", "resume", "capture", "encoder", "loss", "wedge", "wedge-sticky", "web-wedge", "web-wedge-sticky"} |
| if only := os.Getenv("SCREEN_SCENARIOS"); only != "" { |
| scenarios = strings.Split(only, ",") |
| } |
| for _, scenario := range scenarios { |
| t.Run(scenario, func(t *testing.T) { |
| rig := newBrowserRig(t) |
| - if scenario == "renewals" { |
| + soaking := scenario == "soak" || scenario == "web-soak" |
| + if scenario == "renewals" || soaking { |
| if err := rig.ctl.SetRelay(browserRelayContract()); err != nil { |
| t.Fatal(err) |
| } |
| rig.ctl.SetIceLife(20 * time.Second) |
| } |
| - if scenario == "renewals" || scenario == "resume" { |
| + if scenario == "renewals" || scenario == "resume" || soaking { |
| rig.ctl.SetStreamTimings(time.Second, 10*time.Second) |
| } |
| room, _ := rig.createMeeting(t) |
| - ctx, cancel := context.WithTimeout(t.Context(), 240*time.Second) |
| + limit := 240 * time.Second |
| + if soaking { |
| + // SCREEN_SOAK_SECONDS of soak, and the scenario around it. |
| + seconds, _ := strconv.Atoi(envOr("SCREEN_SOAK_SECONDS", "60")) |
| + limit = time.Duration(seconds)*time.Second + 180*time.Second |
| + } |
| + ctx, cancel := context.WithTimeout(t.Context(), limit) |
| defer cancel() |
| cmd := exec.CommandContext(ctx, "node", filepath.Join(root, "test/browser/native-screen.mjs"), rig.origin, room, binary, scenario) |
| cookies, _ := json.Marshal([]map[string]string{{"name": rig.cookieName, "value": rig.cookieB}}) |
| @@ -454,7 +494,7 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| if v, err := time.ParseDuration(os.Getenv("SCREEN_PAIR_SOAK")); err == nil { |
| soak = v |
| } |
| - scenarios := []string{"renewals", "capture", "encoder", "loss"} |
| + scenarios := []string{"renewals", "capture", "encoder", "loss", "wedge", "wedge-sticky", "reshare"} |
| if only := os.Getenv("SCREEN_SCENARIOS"); only != "" { |
| scenarios = strings.Split(only, ",") |
| } |
| @@ -466,9 +506,15 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| t.Skip("the sharing app predates the stall commands") |
| case scenario == "loss" && (!newA || !newB): |
| t.Skip("a stall report needs the new receiver and the new sharer") |
| + case strings.HasPrefix(scenario, "wedge") || scenario == "reshare": |
| + if !newA || !newB { |
| + t.Skip("the escalation needs the new receiver and the new sharer") |
| + } |
| + // Decoders the receiver can wedge (spec §15.22). |
| + t.Setenv("CALLS_DECODER_WEDGE", "1") |
| } |
| rig := newBrowserRig(t) |
| - if scenario == "renewals" { |
| + if scenario == "renewals" || scenario == "soak" { |
| if err := rig.ctl.SetRelay(browserRelayContract()); err != nil { |
| t.Fatal(err) |
| } |
| @@ -501,6 +547,9 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| a.command(t, "screen-mode "+envOr("SCREEN_MODE", "balanced")) |
| if os.Getenv("SCREEN_PAIR_DISPLAY") == "1" && newA { |
| a.command(t, "screen-display 6") |
| + } else if os.Getenv("SCREEN_SOURCE") == "motion" { |
| + // The 1920-wide scrolling page (CALLS_SYNTHETIC_SCREEN_HEIGHT). |
| + a.command(t, "screen-motion") |
| } else { |
| a.command(t, "screen-on") |
| } |
| @@ -510,11 +559,11 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| // The watch: B's screen frames, once a second. |
| freezes, longest := 0, time.Duration(0) |
| watch := func(d time.Duration, label string) { |
| - last, moved, frozen := b.stat("screenFramesDecoded"), time.Now(), false |
| + last, moved, frozen := b.stat("screenFramesDecodedTotal"), time.Now(), false |
| camera := b.stat("framesDecoded") |
| for until := time.Now().Add(d); time.Now().Before(until); { |
| time.Sleep(time.Second) |
| - now := b.stat("screenFramesDecoded") |
| + now := b.stat("screenFramesDecodedTotal") |
| if now > last { |
| if frozen { |
| gap := time.Since(moved) |
| @@ -529,9 +578,10 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| } else if !frozen && time.Since(moved) > 3*time.Second { |
| frozen = true |
| s := a.sample().Stats |
| - t.Logf("FROZEN %s: no screen frame at b for %v (a: captured %d encoded %d sent %d, watchdog k%d e%d r%d a%d)", label, |
| + t.Logf("FROZEN %s: no screen frame at b for %v (a: captured %d encoded %d sent %d, watchdog k%d e%d r%d a%d rebuilds %d shares again %d)", label, |
| time.Since(moved).Round(time.Millisecond), s["screenCaptureDelivered"], s["screenFramesEncoded"], s["screenFramesSent"], |
| - s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], s["screenWatchdogCaptureRestarts"], s["screenWatchdogReAdds"]) |
| + s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], s["screenWatchdogCaptureRestarts"], s["screenWatchdogReAdds"], |
| + s["screenWatchdogRebuilds"], s["screenWatchdogReshares"]) |
| } |
| last = now |
| } |
| @@ -551,6 +601,61 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| t.Fatalf("%s: a=%+v b=%+v", label, a.sample(), b.sample()) |
| } |
| switch scenario { |
| + case "soak": |
| + // §15.22's search for the cause, between two apps: renewals |
| + // and resumes as `renewals`, with A's mode cycled and its |
| + // uplink capped and lifted every 45 s, for SCREEN_PAIR_SOAK. |
| + modes := []string{"balanced", "motion", "detail"} |
| + uplinks := []string{"0", "1500000", "600000", "0", "3000000", "900000"} |
| + done := make(chan struct{}) |
| + notices := map[string]bool{} |
| + var mu sync.Mutex |
| + go func() { |
| + step := 0 |
| + tick := time.NewTicker(45 * time.Second) |
| + defer tick.Stop() |
| + for { |
| + select { |
| + case <-done: |
| + return |
| + case <-tick.C: |
| + step++ |
| + a.command(t, "screen-mode "+modes[step%len(modes)]) |
| + a.command(t, "uplink "+uplinks[step%len(uplinks)]) |
| + t.Logf("soak step %d: mode %s, uplink %s", step, modes[step%len(modes)], uplinks[step%len(uplinks)]) |
| + } |
| + } |
| + }() |
| + go func() { |
| + for { |
| + select { |
| + case <-done: |
| + return |
| + case <-time.After(500 * time.Millisecond): |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + mu.Lock() |
| + notices[n] = true |
| + mu.Unlock() |
| + } |
| + } |
| + } |
| + }() |
| + watch(soak, "soak") |
| + close(done) |
| + a.command(t, "uplink 0") |
| + s := a.sample().Stats |
| + t.Logf("SOAK a: ICE restarts %d (b %d), stall reports %d, keyframes %d, fresh encoders %d, re-adds %d, rebuilds %d, shares again %d, given up %d, capture restarts %d; b's pictures attached again %d", |
| + s["iceRestarts"], b.stat("iceRestarts"), s["screenStallReports"], s["screenWatchdogKeyframes"], s["screenWatchdogEncoderRecoveries"], |
| + s["screenWatchdogReAdds"], s["screenWatchdogRebuilds"], s["screenWatchdogReshares"], s["screenWatchdogFailures"], |
| + s["screenWatchdogCaptureRestarts"], b.stat("screenReattaches")) |
| + mu.Lock() |
| + t.Logf("SOAK notices at a: %q; %d freezes, longest %v", keys(notices), freezes, longest) |
| + mu.Unlock() |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b's diagnostics:\n%s", screenSections(b.diagnostics(t))) |
| + if s["screenWatchdogFailures"] > 0 { |
| + t.Error("a gave up on b's view") |
| + } |
| case "renewals": |
| watch(soak, "renewals") |
| t.Logf("ICE restarts: a started %d, b started %d", a.stat("iceRestarts"), b.stat("iceRestarts")) |
| @@ -592,12 +697,154 @@ func TestNativeMacScreenPairRecovery(t *testing.T) { |
| }, a, b) |
| waitNative(t, "b clears its report once frames move", func() bool { return b.stat("screenStalledHere") == 0 }, a, b) |
| recovered("b decodes after the keyframe") |
| + case "wedge": |
| + // B's screen decoder stops putting out pictures while packets |
| + // keep arriving (spec §15.22): keyframes cannot mend it and a |
| + // new transceiver can. Without a manual share again, B has to |
| + // be decoding again within 30 s, and A's notice has to have |
| + // said so honestly on the way. |
| + watch(5*time.Second, "before") |
| + keyframes := a.stat("screenWatchdogKeyframes") |
| + wedged := time.Now() |
| + b.command(t, "screen-decoder-wedge") |
| + waitNative(t, "b reports a's screen stalled", func() bool { return b.stat("screenStalledHere") == 1 }, a, b) |
| + notices := map[string]bool{} |
| + base := b.stat("screenFramesDecodedTotal") |
| + back := false |
| + for until := wedged.Add(30 * time.Second); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + notices[n] = true |
| + } |
| + if b.stat("screenFramesDecodedTotal") > base+15 { |
| + back = true |
| + break |
| + } |
| + } |
| + took := time.Since(wedged) |
| + t.Logf("b decoding again after %v: rebuilds %d, shares again %d, keyframes %d, b's pictures attached again %d", |
| + took.Round(time.Millisecond), a.stat("screenWatchdogRebuilds"), a.stat("screenWatchdogReshares"), |
| + a.stat("screenWatchdogKeyframes")-keyframes, b.stat("screenReattaches")) |
| + if !back { |
| + t.Logf("a:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b:\n%s", screenSections(b.diagnostics(t))) |
| + t.Fatalf("b's wedged screen was not restored within 30 s: a=%+v b=%+v", a.sample(), b.sample()) |
| + } |
| + if a.stat("screenWatchdogKeyframes") == keyframes { |
| + t.Error("keyframes were not tried first") |
| + } |
| + if a.stat("screenWatchdogRebuilds") == 0 { |
| + t.Error("b decoded again without a rebuild: the wedge did not hold") |
| + } |
| + waitNative(t, "b clears its report", func() bool { return b.stat("screenStalledHere") == 0 }, a, b) |
| + waitNative(t, "a's notice clears", func() bool { |
| + n := a.sample().ScreenRecovery |
| + if n != "" { |
| + notices[n] = true |
| + } |
| + return n == "" |
| + }, a, b) |
| + t.Logf("notices seen at a: %q", keys(notices)) |
| + if !anyHas(notices, "recovering") || !anyHas(notices, "Restarting your share for") { |
| + t.Errorf("the notice did not move from recovering to restarting: %q", keys(notices)) |
| + } |
| + watch(10*time.Second, "after the rebuild") |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + t.Logf("b's diagnostics:\n%s", screenSections(b.diagnostics(t))) |
| + case "wedge-sticky": |
| + // Every screen decoder B makes is wedged, the new ones too: |
| + // nothing A does can mend it, and A has to say so rather than |
| + // sit at "recovering". |
| + watch(5*time.Second, "before") |
| + b.command(t, "screen-decoder-wedge sticky") |
| + notices := map[string]bool{} |
| + gaveUp := false |
| + for until := time.Now().Add(60 * time.Second); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
| + if n := a.sample().ScreenRecovery; n != "" { |
| + notices[n] = true |
| + } |
| + if a.stat("screenWatchdogFailures") > 0 && anyHas(map[string]bool{a.sample().ScreenRecovery: true}, "Couldn\u2019t restore") { |
| + gaveUp = true |
| + break |
| + } |
| + } |
| + t.Logf("notices seen at a: %q", keys(notices)) |
| + if !gaveUp { |
| + t.Fatalf("a never said it could not restore b's view: a=%+v", a.sample()) |
| + } |
| + if a.stat("screenWatchdogRebuilds") == 0 || a.stat("screenWatchdogReshares") == 0 { |
| + t.Errorf("a gave up without trying a rebuild and a share again: %+v", a.sample().Stats) |
| + } |
| + if !anyHas(notices, "Restarting your share for") { |
| + t.Errorf("the notice never said it was restarting: %q", keys(notices)) |
| + } |
| + // The wedge lifted, the view comes back and the notice goes. |
| + b.command(t, "screen-decoder-release") |
| + recoveredTotal := func() bool { |
| + base := b.stat("screenFramesDecodedTotal") |
| + return pollNative(20*time.Second, func() bool { return b.stat("screenFramesDecodedTotal") > base+10 }) |
| + } |
| + if !recoveredTotal() { |
| + t.Fatalf("b did not decode again once the wedge was lifted: %+v", b.sample()) |
| + } |
| + waitNative(t, "a's notice clears once b's view is back", func() bool { return a.sample().ScreenRecovery == "" }, a, b) |
| + t.Logf("a's diagnostics:\n%s", screenSections(a.diagnostics(t))) |
| + case "reshare": |
| + // The watchdog's share again, on its own: every connection's |
| + // screen stopped and added again from the same capture, and |
| + // the watcher decoding again within a bound, one tile. |
| + watch(5*time.Second, "before") |
| + base := b.stat("screenFramesDecodedTotal") |
| + started := time.Now() |
| + a.command(t, "screen-reshare-test") |
| + if !pollNative(15*time.Second, func() bool { return b.stat("screenFramesDecodedTotal") > base+30 }) { |
| + t.Fatalf("b did not decode again after the share again: a=%+v b=%+v", a.sample(), b.sample()) |
| + } |
| + t.Logf("b decoding again %v after the share again", time.Since(started).Round(time.Millisecond)) |
| + waitNative(t, "one screen at b", func() bool { return b.stat("remoteScreens") == 1 && len(b.sample().ScreenCaptions) == 1 }, a, b) |
| + if a.sample().SharingScreen == nil || !*a.sample().SharingScreen { |
| + t.Error("a stopped sharing") |
| + } |
| + watch(10*time.Second, "after the share again") |
| } |
| t.Logf("PASS %s: %d freezes, longest %v", scenario, freezes, longest) |
| }) |
| } |
| } |
| +// screenSections is the screen's part of "Copy call diagnostics": the |
| +// "## Screen" sections and nothing else. |
| +func screenSections(diag string) string { |
| + var out []string |
| + in := false |
| + for _, line := range strings.Split(diag, "\n") { |
| + if strings.HasPrefix(line, "## ") { |
| + in = strings.HasPrefix(line, "## Screen") |
| + } |
| + if in { |
| + out = append(out, line) |
| + } |
| + } |
| + return strings.Join(out, "\n") |
| +} |
| + |
| +func keys(m map[string]bool) []string { |
| + out := make([]string, 0, len(m)) |
| + for k := range m { |
| + out = append(out, k) |
| + } |
| + sort.Strings(out) |
| + return out |
| +} |
| + |
| +func anyHas(m map[string]bool, part string) bool { |
| + for k := range m { |
| + if strings.Contains(k, part) { |
| + return true |
| + } |
| + } |
| + return false |
| +} |
| + |
| // pollNative is waitNative without the failure: whether ok held in time. |
| func pollNative(d time.Duration, ok func() bool) bool { |
| for until := time.Now().Add(d); time.Now().Before(until); time.Sleep(250 * time.Millisecond) { |
diff --git a/native/README.md b/native/README.md| index 2f6f916..e402337 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -546,6 +546,36 @@ steps only if its own screen has stopped leaving too), and shows |
| **Copy audio diagnostics**). The apps restart ICE only when a relay credential is |
| due for renewal, not on every stream resume as 0.4.x did. |
| +**When keyframes are not enough** (spec §15.22, after Paul's "still seeing issues |
| +with screen sharing after a while", 30 Sep, the notice sitting at "recovering"): |
| +a receiver still reporting six seconds after the first keyframe gets its screen |
| +rebuilt — a new transceiver for that person alone, so a new encoder here and a new |
| +receiver and decoder there, at the same size and mode (a report never shrinks a |
| +Text and detail share) — and the notice says "Restarting your share for Samuel…". |
| +Still reported ten seconds later, everyone's screen is shared again from the same |
| +capture, with no picker: what Stop and Share did by hand. Ten seconds after that |
| +the notice says "Couldn't restore Samuel's view — try Stop and Share again". A |
| +rebuild at most once a minute per person, a share again at most once every five |
| +minutes, and neither unless frames actually left for that person since the last |
| +step. The receiver hands a stalled screen's picture to its view again four seconds |
| +in, logs what its statistics say is still arriving (packets, frames received, |
| +decoded, dropped, keyframes, PLI, the decoder), holds its report for up to three |
| +minutes, and counts any change in the decoded count as movement (a rebuilt screen's |
| +count starts again from zero, which 0.4.3 read as still stalled). The iPhone's |
| +broadcast repeats its last frame once a second while the screen is still, as the |
| +Mac's capture does, so a still iPhone screen is no longer reported as stalled. The |
| +browser does all of this too (its sharer's rebuild and share again included), and |
| +has **Copy call diagnostics** at the foot of its Devices panel. |
| + |
| +The gate for it wedges a receiver's screen decoder, which keyframes cannot mend: |
| +`screen-decoder-wedge` in `CallsInterop` (run with `CALLS_DECODER_WEDGE=1`; a |
| +wrapping decoder factory that only the gate turns on), and in Chromium an encoded |
| +transform that swallows every frame. `wedge`, `wedge-sticky` and `reshare` in |
| +`make native-pair`, and `wedge`, `wedge-sticky`, `web-wedge` and `web-wedge-sticky` |
| +in `make native-browser`, hold the ladder to restoring the view within 30 s, or to |
| +saying it cannot. `soak` and `web-soak` (`SCREEN_SOAK_SECONDS`) and the pair's |
| +`soak` (`SCREEN_PAIR_SOAK`) are the long shares, run by hand. |
| + |
| The numbers behind the modes come from `TestNativeMacScreenMotion` (in |
| `make native-browser`): the app shares a 1080p page scrolling at 30 fps with a |
| window dragged across it, and Chromium measures what arrives, with the app's |
diff --git a/native/apple/Sources/CallsInterop/Interop.swift b/native/apple/Sources/CallsInterop/Interop.swift| index fe9013d..f027765 100644 |
| --- a/native/apple/Sources/CallsInterop/Interop.swift |
| +++ b/native/apple/Sources/CallsInterop/Interop.swift |
| @@ -56,6 +56,10 @@ struct Input: Decodable { |
| let input = try? JSONDecoder().decode(Input.self, from: Data(line.utf8)), |
| let url = URL(string: input.origin), url.scheme == "http", url.host == "127.0.0.1" |
| else { exit(2) } |
| + // Decoders the gate can wedge (spec §15.22), set before the call's |
| + // factory is made; only for the scenarios that ask, so every other |
| + // gate decodes exactly as the apps do. |
| + if ProcessInfo.processInfo.environment["CALLS_DECODER_WEDGE"] == "1" { DecoderWedge.enabled = true } |
| let call = Call(playAudio: false, audioInput: SyntheticVoice()) |
| if ProcessInfo.processInfo.environment["CALLS_SCREEN_WATCHDOG"] == "0" { Call.screenWatchdogEnabled = false } |
| // WebRTC's own log, filtered to the video encoder, into a file: for |
| @@ -216,6 +220,34 @@ struct Input: Decodable { |
| print("{\"screenFrozenForTest\":true}") |
| fflush(stdout) |
| } |
| + // §15.22: this app's screen decoders stop putting out pictures |
| + // while packets keep arriving — a wedge no keyframe mends and a |
| + // new transceiver does. `sticky` wedges the new ones too. |
| + if command == "screen-decoder-wedge" || command == "screen-decoder-wedge sticky" { |
| + let n = DecoderWedge.wedgeScreens(sticky: command.hasSuffix("sticky")) |
| + print("{\"decoderWedged\":\(n)}") |
| + fflush(stdout) |
| + } |
| + if command == "screen-decoder-release" { |
| + DecoderWedge.release() |
| + print("{\"decoderReleased\":true}") |
| + fflush(stdout) |
| + } |
| + // "Copy call diagnostics", as the panel would copy it. |
| + if command == "diagnostics" { |
| + let text = await call.audioDiagnosticsText() |
| + if let data = try? JSONSerialization.data(withJSONObject: ["diagnostics": text]), |
| + let line = String(data: data, encoding: .utf8) |
| + { |
| + print(line) |
| + fflush(stdout) |
| + } |
| + } |
| + if command == "screen-reshare-test" { |
| + await call.reshareScreenForTest() |
| + print("{\"reshared\":true}") |
| + fflush(stdout) |
| + } |
| if command == "screen-end" { |
| await Interop.screen?.end() |
| let on = await call.sharingScreen |
| @@ -249,6 +281,8 @@ struct Input: Decodable { |
| // place a screen says whose it is. |
| "screenCaptions": call.screens.map { screenCaption(call.name(of: $0)) }, |
| "sharingScreen": call.sharingScreen, |
| + // §15.22: the sharer's notice, word for word. |
| + "screenRecovery": call.screenRecovery, |
| "syntheticScreenFrames": Interop.screen?.produced ?? 0, |
| "cameraOpens": Interop.camera.opens, |
| "cameraStatus": { |
| @@ -285,6 +319,7 @@ struct Input: Decodable { |
| func stop() { mac.stop() } |
| var counts: ScreenCaptureCounts { mac.counts } |
| func restart() async -> Bool { await mac.restart() } |
| + var frameSize: (width: Int, height: Int)? { mac.frameSize } |
| } |
| // Render the shipping chat view against real received messages. This fixture |
diff --git a/native/apple/Sources/CallsInterop/SyntheticVideo.swift b/native/apple/Sources/CallsInterop/SyntheticVideo.swift| index 845c2c4..1ddf271 100644 |
| --- a/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| +++ b/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| @@ -129,6 +129,7 @@ import LiveKitWebRTC |
| let n = page.map { UInt64($0.produced) } ?? made |
| return ScreenCaptureCounts(heartbeats: n, delivered: n) |
| } |
| + var frameSize: (width: Int, height: Int)? { page != nil ? (ScrollingPage.width, ScrollingPage.height) : (640, 360) } |
| func restart() async -> Bool { |
| guard source != nil else { return false } |
| restarts += 1 |
| @@ -186,7 +187,10 @@ import LiveKitWebRTC |
| /// each frame is row copies into a pooled buffer, on a queue of its own. |
| final class ScrollingPage: @unchecked Sendable { |
| static let width = 1920 |
| - static let height = 1080 |
| + /// 1080 unless `CALLS_SYNTHETIC_SCREEN_HEIGHT` says otherwise: 1246 is a |
| + /// 16:10 Retina display as the Mac's capture fits it to 1920 wide, the |
| + /// size that meets H.264 level 3.1's block limit (spec §15.21, §15.22). |
| + static let height = Int(ProcessInfo.processInfo.environment["CALLS_SYNTHETIC_SCREEN_HEIGHT"] ?? "") ?? 1080 |
| private let rows = ScrollingPage.height * 2 |
| private let texture: [UInt8] |
| private var offset = 0 |
diff --git a/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift| index 01ff1a8..0ce8d13 100644 |
| --- a/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift |
| +++ b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift |
| @@ -1,6 +1,7 @@ |
| import CallsBroadcastKit |
| import CoreVideo |
| import Foundation |
| +import os |
| import LiveKitWebRTC |
| /// The iPhone's whole screen, arriving from the broadcast extension over |
| @@ -17,6 +18,7 @@ import LiveKitWebRTC |
| @MainActor public final class BroadcastScreenCapture: ScreenCapture { |
| private let connection: BroadcastConnection |
| private var stopped = false |
| + private var feed: StillScreenFeed? |
| /// Called after the share has ended on its own, once the call has heard. |
| /// The app uses it to leave a call it is only still in for the screen. |
| public var afterEnded: (@MainActor () -> Void)? |
| @@ -24,17 +26,9 @@ import LiveKitWebRTC |
| public init(_ connection: BroadcastConnection) { self.connection = connection } |
| public func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws { |
| - let capturer = RTCVideoCapturer(delegate: source) |
| - connection.onFrame = { pixels, orientation in |
| - // Stamped on arrival, on this device's clock, rather than with the |
| - // extension's: the two are separate processes, and the encoder only |
| - // needs the frames in order and spaced as they came. |
| - source.capturer( |
| - capturer, |
| - didCapture: RTCVideoFrame( |
| - buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: broadcastRotation(orientation), |
| - timeStampNs: Int64(DispatchTime.now().uptimeNanoseconds))) |
| - } |
| + let feed = StillScreenFeed(source: source) |
| + self.feed = feed |
| + connection.onFrame = { pixels, orientation in feed.frame(pixels, rotation: broadcastRotation(orientation)) } |
| connection.onClose = { [weak self] in |
| Task { @MainActor [weak self] in |
| guard let self, !self.stopped else { return } |
| @@ -46,8 +40,12 @@ import LiveKitWebRTC |
| connection.resume() |
| } |
| + public var counts: ScreenCaptureCounts { feed?.counts ?? ScreenCaptureCounts() } |
| + |
| public func stop() { |
| stopped = true |
| + feed?.stop() |
| + feed = nil |
| connection.onClose = nil |
| connection.onFrame = nil |
| connection.close() |
| @@ -68,3 +66,67 @@ func broadcastRotation(_ orientation: UInt16) -> RTCVideoRotation { |
| default: return ._0 |
| } |
| } |
| + |
| +/// The broadcast's frames into the call's source, and the last of them |
| +/// again once a second while the screen is still (2026-09-30). ReplayKit |
| +/// sends nothing at all for a screen that is not changing, and a receiver |
| +/// that sees no frame for four seconds reports the screen stalled (spec |
| +/// §15.21) — which, for a still iPhone screen, was a false alarm the |
| +/// sharer then tried to mend. The Mac's capture has repeated its last |
| +/// frame since 2026-09-28 (MacScreenCapture); this is the same rule. |
| +final class StillScreenFeed: @unchecked Sendable { |
| + static let repeatAfter: UInt64 = 1_000_000_000 |
| + |
| + private let source: RTCVideoSource |
| + private let capturer: RTCVideoCapturer |
| + private let queue = DispatchQueue(label: "calls.broadcast.feed", qos: .userInteractive) |
| + private var timer: DispatchSourceTimer? |
| + // Only touched on `queue`. |
| + private var last: (CVPixelBuffer, RTCVideoRotation)? |
| + private var lastSent: UInt64 = 0 |
| + private let tally = OSAllocatedUnfairLock(initialState: ScreenCaptureCounts(heartbeats: nil, delivered: 0)) |
| + var counts: ScreenCaptureCounts { tally.withLock { $0 } } |
| + |
| + init(source: RTCVideoSource) { |
| + self.source = source |
| + capturer = RTCVideoCapturer(delegate: source) |
| + let timer = DispatchSource.makeTimerSource(queue: queue) |
| + timer.schedule(deadline: .now() + .milliseconds(500), repeating: .milliseconds(500), leeway: .milliseconds(50)) |
| + timer.setEventHandler { [weak self] in self?.repeatIfIdle() } |
| + timer.resume() |
| + self.timer = timer |
| + } |
| + |
| + /// One frame from the extension. Stamped on arrival, on this device's |
| + /// clock, rather than with the extension's: the two are separate |
| + /// processes, and the encoder only needs the frames in order and spaced |
| + /// as they came. |
| + func frame(_ pixels: CVPixelBuffer, rotation: RTCVideoRotation) { |
| + queue.async { |
| + self.last = (pixels, rotation) |
| + self.deliver(pixels, rotation) |
| + } |
| + } |
| + |
| + func stop() { |
| + timer?.cancel() |
| + timer = nil |
| + queue.async { self.last = nil } |
| + } |
| + |
| + private func repeatIfIdle() { |
| + guard let (pixels, rotation) = last else { return } |
| + let now = DispatchTime.now().uptimeNanoseconds |
| + guard now >= lastSent, now - lastSent >= Self.repeatAfter else { return } |
| + deliver(pixels, rotation) |
| + } |
| + |
| + private func deliver(_ pixels: CVPixelBuffer, _ rotation: RTCVideoRotation) { |
| + let ns = max(DispatchTime.now().uptimeNanoseconds, lastSent + 1) |
| + lastSent = ns |
| + tally.withLock { $0.delivered += 1 } |
| + source.capturer( |
| + capturer, |
| + didCapture: RTCVideoFrame(buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: rotation, timeStampNs: Int64(ns))) |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index 80d24aa..d1ce3a6 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -75,6 +75,13 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| /// `screenStalled` (spec §15.21). |
| private var screenReceiveWatch = ScreenReceiveWatch() |
| private var reportedStalled: [String] = [] |
| + /// Each arriving screen's statistics at the last look, for the stall |
| + /// log's "what moved" and for "Copy call diagnostics". |
| + private var screenReceived: [String: ScreenReceiveReport] = [:] |
| + /// Where each arriving screen stood when it last moved, so a stall can |
| + /// be described against it. |
| + private var screenReceivedMoving: [String: ScreenReceiveReport] = [:] |
| + private var screenReattaches = 0 |
| private var screenWatchFrozenUntil = 0.0 |
| @Published public private(set) var localTrack: RTCVideoTrack? |
| @Published public private(set) var muted = false |
| @@ -202,11 +209,24 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| } else { |
| lines.append("Not sharing") |
| } |
| + if sharingScreen { |
| + for peer in (state?.peers ?? []).map(\.peer) where screenWatchdog.escalation(of: peer) != .none { |
| + lines.append("recovering \(name(of: peer))'s view: \(screenWatchdog.escalation(of: peer))") |
| + } |
| + lines.append( |
| + "recoveries: \(screenWatchdog.keyframes) keyframes, \(screenWatchdog.encoderRecoveries) fresh encoders, \(screenWatchdog.reAdds) re-adds, \(screenWatchdog.rebuilds) rebuilds, \(screenWatchdog.reshares) shares again, \(screenWatchdog.failures) given up") |
| + } |
| + // The receiver's half: every screen arriving here, as its statistics |
| + // have it, whether or not it has stalled. |
| + for (peer, report) in screenReceived.sorted(by: { $0.key < $1.key }) { |
| + lines.append("from \(name(of: peer)): \(report.summary)") |
| + } |
| if !reportedStalled.isEmpty { |
| lines.append("stalled here: " + reportedStalled.map { name(of: $0) }.joined(separator: ", ")) |
| } |
| - let log = AudioDiagnosticsStore.screen.lines(last: 50) |
| - return [("Screen", lines), ("Screen log (last 50 lines)", log.isEmpty ? ["(nothing logged)"] : log)] |
| + if screenReattaches > 0 { lines.append("pictures attached again here: \(screenReattaches)") } |
| + let log = AudioDiagnosticsStore.screen.lines(last: AudioDiagnosticsStore.lineLimit) |
| + return [("Screen", lines), ("Screen log (last \(AudioDiagnosticsStore.lineLimit) lines)", log.isEmpty ? ["(nothing logged)"] : log)] |
| } |
| private var ice: ICEConfig? |
| private var pendingMedia: [Message] = [] |
| @@ -504,6 +524,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| levelWatch = nil |
| screenReceiveWatch = ScreenReceiveWatch() |
| reportedStalled = [] |
| + screenReceived = [:] |
| + screenReceivedMoving = [:] |
| levels.reset() |
| reportedNoSound = [] |
| audioBytes = [:] |
| @@ -823,20 +845,41 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| case .reAdd(let peer): |
| ScreenLog.info("screen re-added for \(ScreenWatchdog.short(peer))") |
| media.reAddScreen(peer) |
| - // The new transceiver's track arrives unlabelled; say it again. |
| - Task { @MainActor [weak self] in |
| - try? await Task.sleep(nanoseconds: 2_000_000_000) |
| - guard let self, self.sharingScreen else { return } |
| - self.send(type: "share", to: "", gen: 0, payload: json(["on": true]) ?? "") |
| - } |
| + sayShareAgain() |
| + case .rebuild(let peer): |
| + ScreenLog.info("screen rebuilt for \(ScreenWatchdog.short(peer)): new transceiver, same size and mode") |
| + media.reAddScreen(peer) |
| + sayShareAgain() |
| + case .reshare: |
| + ScreenLog.info("sharing again for everyone, from the same capture") |
| + await media.reshareScreen() |
| + sayShareAgain() |
| } |
| } |
| screenWatchStatus = screenWatchdog.status |
| if let (peer, worst) = reports.min(by: { $0.value.framesPerSecond < $1.value.framesPerSecond }) { |
| screenDetail = worst.detail(since: previousReports[peer], watchdog: screenWatchdog.status) |
| } |
| - let names = stalledBy.map { name(of: $0) }.sorted() |
| - screenRecovery = ScreenWords.recovering(names: names) |
| + var recovering: [String] = [] |
| + var restarting: [String] = [] |
| + var failed: [String] = [] |
| + for peer in stalledBy.sorted(by: { name(of: $0) < name(of: $1) }) { |
| + switch screenWatchdog.escalation(of: peer) { |
| + case .rebuilt, .reshared: restarting.append(name(of: peer)) |
| + case .failed: failed.append(name(of: peer)) |
| + case .none, .keyframes: recovering.append(name(of: peer)) |
| + } |
| + } |
| + screenRecovery = ScreenWords.notice(recovering: recovering, restarting: restarting, failed: failed) |
| + } |
| + |
| + /// A new transceiver's track arrives unlabelled; say it again. |
| + private func sayShareAgain() { |
| + Task { @MainActor [weak self] in |
| + try? await Task.sleep(nanoseconds: 2_000_000_000) |
| + guard let self, self.sharingScreen else { return } |
| + self.send(type: "share", to: "", gen: 0, payload: json(["on": true]) ?? "") |
| + } |
| } |
| private func endScreenWatch() { |
| screenWatch?.cancel() |
| @@ -851,30 +894,67 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| /// The receiver's watchdog, once a second from the level watch: whose |
| /// screens have stopped moving here, reported as `screenStalled` when |
| - /// that changes (spec §15.21). |
| + /// that changes (spec §15.21), each stall logged with what the |
| + /// statistics say is still arriving, and the picture attached again |
| + /// four seconds in (spec §15.22). |
| private func watchRemoteScreens(now: Double) async { |
| var stalled: [String] = [] |
| - if !remoteScreens.isEmpty { |
| - var frames = await media.remoteScreenFrames() |
| + if !screens.isEmpty { |
| + let reports = await media.remoteScreenReports() |
| + var frames = reports.mapValues(\.framesDecoded) |
| // The gate's frozen picture: the counts held where they were. |
| if now < screenWatchFrozenUntil { frames = frames.mapValues { _ in 0 } } |
| let sharing = Set(screens) |
| + let before = screenReceived |
| + screenReceived = reports.filter { sharing.contains($0.key) } |
| stalled = screenReceiveWatch.observe(at: now, decoded: frames.filter { sharing.contains($0.key) }) |
| + for (peer, report) in screenReceived where !stalled.contains(peer) { |
| + if before[peer].map({ $0.framesDecoded != report.framesDecoded || $0.mid != report.mid }) ?? true { |
| + screenReceivedMoving[peer] = report |
| + } |
| + } |
| + for peer in stalled where !reportedStalled.contains(peer) { |
| + let report = screenReceived[peer] ?? ScreenReceiveReport() |
| + let moving = screenReceivedMoving[peer] |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) stalled here: reporting it; since it last moved \(report.moved(since: moving)) — \(report.diagnosis(since: moving))") |
| + ScreenLog.info("screen from \(ScreenWatchdog.short(peer)): \(report.summary)") |
| + } |
| + for peer in screenReceiveWatch.reattach { |
| + let report = screenReceived[peer] ?? ScreenReceiveReport() |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) still stalled: attaching its picture again; \(report.moved(since: screenReceivedMoving[peer]))") |
| + reattachScreen(peer) |
| + } |
| } else { |
| screenReceiveWatch = ScreenReceiveWatch() |
| + screenReceived = [:] |
| + screenReceivedMoving = [:] |
| } |
| guard stalled != reportedStalled, let client = bridge else { return } |
| - for peer in stalled where !reportedStalled.contains(peer) { |
| - ScreenLog.info("screen from \(ScreenWatchdog.short(peer)) stalled here: asking for a keyframe") |
| - } |
| for peer in reportedStalled where !stalled.contains(peer) { |
| - ScreenLog.info("screen from \(ScreenWatchdog.short(peer)) moving here again") |
| + let report = screenReceived[peer] |
| + ScreenLog.info( |
| + "screen from \(ScreenWatchdog.short(peer)) moving here again\(report.map { ": " + $0.summary } ?? "")") |
| } |
| reportedStalled = stalled |
| let body = json(stalled) ?? "[]" |
| noSoundQueue.async { try? client.setScreenStalled(body) } |
| } |
| + /// A stalled screen's picture dropped and attached again: a new renderer |
| + /// on the same track, for a picture stopped on this side. |
| + private func reattachScreen(_ peer: String) { |
| + guard let track = remoteScreens[peer] else { return } |
| + screenReattaches += 1 |
| + remoteScreens.removeValue(forKey: peer) |
| + Task { @MainActor [weak self] in |
| + try? await Task.sleep(nanoseconds: 150_000_000) |
| + guard let self, self.screens.contains(peer), self.remoteScreens[peer] == nil else { return } |
| + self.remoteScreens[peer] = track |
| + } |
| + } |
| + |
| /// Stops sharing, from every way out: the button, the system's own stop |
| /// control, and the capture ending under us. |
| public func stopSharingScreen() { |
| @@ -915,6 +995,13 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| } |
| /// For the gate's encoder stall (see MediaEngine). |
| @_spi(Fixtures) public func stallScreenEncoderForTest() { media.stallScreenEncoderForTest() } |
| + /// For the gate: the watchdog's share again (spec §15.22), on demand. |
| + @_spi(Fixtures) public func reshareScreenForTest() async { |
| + guard sharingScreen else { return } |
| + ScreenLog.info("sharing again for everyone, from the same capture (asked for by the gate)") |
| + await media.reshareScreen() |
| + sayShareAgain() |
| + } |
| public func sampleStats() async -> [String: UInt64] { |
| var values = await media.stats() |
| values["hasICE"] = ice == nil ? 0 : 1 |
| @@ -952,6 +1039,10 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| values["screenWatchdogReAdds"] = UInt64(screenWatchdog.reAdds) |
| values["screenStallReports"] = UInt64(screenWatchdog.stallReports) |
| values["screenStalledHere"] = UInt64(reportedStalled.count) |
| + values["screenWatchdogRebuilds"] = UInt64(screenWatchdog.rebuilds) |
| + values["screenWatchdogReshares"] = UInt64(screenWatchdog.reshares) |
| + values["screenWatchdogFailures"] = UInt64(screenWatchdog.failures) |
| + values["screenReattaches"] = UInt64(screenReattaches) |
| values["iceRestarts"] = media.iceRestarts |
| values["screenKeyframes"] = screenReport?.keyFramesEncoded ?? 0 |
| if let capture = media.screenCaptureCounts { |
diff --git a/native/apple/Sources/CallsKit/CallAudio.swift b/native/apple/Sources/CallsKit/CallAudio.swift| index f3d53df..dd281b5 100644 |
| --- a/native/apple/Sources/CallsKit/CallAudio.swift |
| +++ b/native/apple/Sources/CallsKit/CallAudio.swift |
| @@ -158,7 +158,9 @@ public struct ProcessingReport: Equatable, Sendable { |
| denoiser: CaptureDenoiser? |
| ) -> Parts { |
| let encoder = RTCDefaultVideoEncoderFactory() |
| - let decoder = RTCDefaultVideoDecoderFactory() |
| + // WebRTC's own, except in the gate's decoder-wedge scenarios. |
| + let decoder: RTCVideoDecoderFactory = |
| + DecoderWedge.enabled ? DecoderWedge.factory() : RTCDefaultVideoDecoderFactory() |
| let denoiser = denoiser ?? CaptureDenoiser(suppressor: CallAudio.makeSuppressor()) |
| #if os(iOS) |
| if let audioDevice { |
diff --git a/native/apple/Sources/CallsKit/Capture.swift b/native/apple/Sources/CallsKit/Capture.swift| index 3b773c2..15ec995 100644 |
| --- a/native/apple/Sources/CallsKit/Capture.swift |
| +++ b/native/apple/Sources/CallsKit/Capture.swift |
| @@ -104,11 +104,18 @@ extension VideoCapture { |
| /// source — after the watchdog has seen it stall. False when this kind |
| /// of capture cannot, or the attempt failed. |
| func restart() async -> Bool |
| + /// The size the capture delivers frames at, when it knows before the |
| + /// first frame: the send policy's H.264 level cap (ScreenSendPolicy) |
| + /// then holds from the first frame rather than from the first |
| + /// statistics a second later, a second in which a browser's level-3.1 |
| + /// encoder refuses every frame of a Motion share (spec §15.22). |
| + var frameSize: (width: Int, height: Int)? { get } |
| } |
| extension ScreenCapture { |
| public var counts: ScreenCaptureCounts { ScreenCaptureCounts() } |
| public func restart() async -> Bool { false } |
| + public var frameSize: (width: Int, height: Int)? { nil } |
| } |
| /// A screen capture's running totals (see ScreenWatchdog). |
diff --git a/native/apple/Sources/CallsKit/DecoderWedge.swift b/native/apple/Sources/CallsKit/DecoderWedge.swift| new file mode 100644 |
| index 0000000..bb775a0 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/DecoderWedge.swift |
| @@ -0,0 +1,123 @@ |
| +import Foundation |
| +import LiveKitWebRTC |
| +import os |
| + |
| +/// For the gate only (spec §15.22): a receiver whose screen decoder has |
| +/// wedged. Packets keep arriving, frames keep being handed to the decoder, |
| +/// and nothing comes out — which no keyframe mends, because it is this |
| +/// decoder instance that is stuck, and which a new transceiver does mend, |
| +/// because the far end's new stream gets a decoder of its own. That is the |
| +/// shape of a stall that "keyframes only" (0.4.3) left sitting at |
| +/// "recovering". |
| +/// |
| +/// Off unless the interop executable turns it on before the call is |
| +/// built; the apps never do, and then the call's factory is WebRTC's own. |
| +@_spi(Fixtures) public enum DecoderWedge { |
| + /// Wrap the call's decoders so they can be wedged. Read once, when the |
| + /// call's factory is made. |
| + nonisolated(unsafe) public static var enabled = false |
| + |
| + |
| + private static let live = OSAllocatedUnfairLock(initialState: [WeakDecoder]()) |
| + private static let sticky = OSAllocatedUnfairLock(initialState: false) |
| + |
| + static func register(_ decoder: WedgeableDecoder) { |
| + live.withLock { list in |
| + list.removeAll { $0.decoder == nil } |
| + list.append(WeakDecoder(decoder: decoder)) |
| + } |
| + } |
| + |
| + /// Wedges the decoder now putting out the largest picture, which in the |
| + /// gate is always the screen (its cameras are 320×240 to 640×480, and a |
| + /// screen even at its smallest under a thin link is wider). `sticky` |
| + /// wedges every decoder made afterwards too — only a screen's rebuild or |
| + /// share again makes one mid-call — for a receiver no rebuild can mend, |
| + /// the gate's "could not restore". Returns how many were wedged. |
| + @discardableResult public static func wedgeScreens(sticky: Bool = false) -> Int { |
| + self.sticky.withLock { $0 = sticky } |
| + let decoders = live.withLock { $0.compactMap(\.decoder) } |
| + guard let largest = decoders.map(\.area).max(), largest > 0 else { return 0 } |
| + var count = 0 |
| + for d in decoders where d.area == largest { |
| + d.wedge() |
| + count += 1 |
| + } |
| + return count |
| + } |
| + |
| + /// Every wedge lifted, and nothing wedged from now on. |
| + public static func release() { |
| + sticky.withLock { $0 = false } |
| + for d in live.withLock({ $0.compactMap(\.decoder) }) { d.unwedge() } |
| + } |
| + |
| + static var isSticky: Bool { sticky.withLock { $0 } } |
| + |
| + static func factory() -> RTCVideoDecoderFactory { WedgeableDecoderFactory() } |
| +} |
| + |
| +private struct WeakDecoder { |
| + weak var decoder: WedgeableDecoder? |
| +} |
| + |
| +final class WedgeableDecoderFactory: NSObject, RTCVideoDecoderFactory { |
| + private let inner = RTCDefaultVideoDecoderFactory() |
| + |
| + func createDecoder(_ info: RTCVideoCodecInfo) -> RTCVideoDecoder? { |
| + guard let decoder = inner.createDecoder(info) else { return nil } |
| + // Only the VideoToolbox decoders are Objective-C all the way down. |
| + // WebRTC's own VP8, VP9 and AV1 come back as wrapped native decoders, |
| + // which the factory unwraps and never calls through this protocol, so |
| + // they are handed over as they are and cannot be wedged. |
| + guard ["H264", "H265"].contains(info.name.uppercased()) else { return decoder } |
| + let wrapped = WedgeableDecoder(decoder) |
| + if DecoderWedge.isSticky { wrapped.wedge() } |
| + DecoderWedge.register(wrapped) |
| + return wrapped |
| + } |
| + |
| + func supportedCodecs() -> [RTCVideoCodecInfo] { inner.supportedCodecs() } |
| +} |
| + |
| +final class WedgeableDecoder: NSObject, RTCVideoDecoder, @unchecked Sendable { |
| + private let inner: RTCVideoDecoder |
| + private struct State { |
| + var wedged = false |
| + var area = 0 |
| + } |
| + private let state = OSAllocatedUnfairLock(initialState: State()) |
| + |
| + init(_ inner: RTCVideoDecoder) { self.inner = inner } |
| + |
| + var area: Int { state.withLock { $0.area } } |
| + func wedge() { state.withLock { $0.wedged = true } } |
| + func unwedge() { state.withLock { $0.wedged = false } } |
| + |
| + func setCallback(_ callback: @escaping RTCVideoDecoderCallback) { |
| + inner.setCallback { [weak self] frame in |
| + guard let self else { return } |
| + let area = Int(frame.width) * Int(frame.height) |
| + let wedged = self.state.withLock { s -> Bool in |
| + s.area = area |
| + return s.wedged |
| + } |
| + if !wedged { callback(frame) } |
| + } |
| + } |
| + |
| + func startDecode(withNumberOfCores numberOfCores: Int32) -> Int { |
| + inner.startDecode(withNumberOfCores: numberOfCores) |
| + } |
| + |
| + func release() -> Int { inner.release() } |
| + |
| + func decode( |
| + _ encodedImage: RTCEncodedImage, missingFrames: Bool, codecSpecificInfo info: RTCCodecSpecificInfo?, |
| + renderTimeMs: Int64 |
| + ) -> Int { |
| + inner.decode(encodedImage, missingFrames: missingFrames, codecSpecificInfo: info, renderTimeMs: renderTimeMs) |
| + } |
| + |
| + func implementationName() -> String { inner.implementationName() } |
| +} |
diff --git a/native/apple/Sources/CallsKit/MacScreenCapture.swift b/native/apple/Sources/CallsKit/MacScreenCapture.swift| index 2ee3714..b5b38d8 100644 |
| --- a/native/apple/Sources/CallsKit/MacScreenCapture.swift |
| +++ b/native/apple/Sources/CallsKit/MacScreenCapture.swift |
| @@ -108,7 +108,10 @@ |
| self.frames = frames |
| self.filter = filter |
| self.source = source |
| + let configured = Self.configuration(for: filter) |
| + frameSize = (configured.width, configured.height) |
| } |
| + public private(set) var frameSize: (width: Int, height: Int)? |
| /// The same content into the same source, on a capture of its own: |
| /// a fresh stream with a fresh pool of surfaces. What stopping and |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 609fe81..262c256 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -139,9 +139,32 @@ import LiveKitWebRTC |
| throw CancellationError() |
| } |
| screenCapture = capture |
| + screenSourceInUse = source |
| + if let size = capture.frameSize, size.width > 0, size.height > 0 { screenSize = size } |
| localScreen = factory.videoTrack(with: source, trackId: "screen") |
| for peer in peers.values { peer.addScreen() } |
| } |
| + /// The source the screen's capture is writing into, for a share again |
| + /// from the same capture. |
| + private var screenSourceInUse: RTCVideoSource? |
| + private var reshares = 0 |
| + |
| + /// Stop and Share, without the picker (spec §15.22): every connection's |
| + /// screen transceiver stopped and renegotiated, a fresh track on the |
| + /// same source, the capture started again on the same content, and each |
| + /// connection given the new track as soon as it settles. New |
| + /// transceivers mean new encoders here and new receivers and decoders at |
| + /// every far end, which is what the hand-made stop and share did. |
| + func reshareScreen() async { |
| + guard let capture = screenCapture, let source = screenSourceInUse, localScreen != nil else { return } |
| + reshares += 1 |
| + for peer in peers.values { peer.removeScreen() } |
| + localScreen = factory.videoTrack(with: source, trackId: "screen-\(reshares)") |
| + if !(await capture.restart()) { ScreenLog.info("share again: the capture carries on as it was") } |
| + guard screenCapture === capture else { return } |
| + // Any connection already stable takes it now; the rest as they settle. |
| + for peer in peers.values { peer.addScreen() } |
| + } |
| /// Stops sending the screen: every connection's screen transceiver is |
| /// stopped and renegotiated, so the receivers' tracks *end*, and the |
| @@ -150,6 +173,7 @@ import LiveKitWebRTC |
| screenAttempt += 1 |
| guard let capture = screenCapture else { return } |
| screenCapture = nil |
| + screenSourceInUse = nil |
| localScreen = nil |
| for peer in peers.values { |
| peer.removeScreen() |
| @@ -181,24 +205,36 @@ import LiveKitWebRTC |
| for peer in peers.values { peer.setScreenActive(false) } |
| } |
| - /// Frames decoded so far from each peer's screen, for the peers whose |
| - /// screen is arriving here (the receiver's watchdog). |
| - func remoteScreenFrames() async -> [String: UInt64] { |
| - var out: [String: UInt64] = [:] |
| + /// Each peer's screen as it arrives here, for the peers whose screen is |
| + /// arriving (the receiver's watchdog, and "Copy call diagnostics"). |
| + func remoteScreenReports() async -> [String: ScreenReceiveReport] { |
| + var out: [String: ScreenReceiveReport] = [:] |
| for (id, peer) in peers { |
| guard let pc = peer.pc, peer.receivingScreen else { continue } |
| let report = await PeerCalls.statistics(pc) |
| - var decoded: UInt64 = 0 |
| + var reading = ScreenReceiveReport() |
| for stat in report.statistics.values where stat.type == "inbound-rtp" { |
| let mid = stat.values["mid"] as? String ?? "" |
| guard peer.isScreen(mid: mid) else { continue } |
| - decoded += (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + reading.add(stat.values) |
| } |
| - out[id] = decoded |
| + reading.mid = peer.screenMidHere |
| + out[id] = reading |
| } |
| return out |
| } |
| + /// Frames decoded from screens here, never going backwards: a rebuilt |
| + /// screen arrives on a new receiver whose own count starts again, and |
| + /// the gate's "decoding again" has to see its frames as more. |
| + private var screenDecodedSeen: [String: UInt64] = [:] |
| + private(set) var screenDecodedTotal: UInt64 = 0 |
| + private func tallyScreenDecoded(_ peer: String, _ now: UInt64) { |
| + let last = screenDecodedSeen[peer] ?? 0 |
| + screenDecodedTotal += now >= last ? now - last : now |
| + screenDecodedSeen[peer] = now |
| + } |
| + |
| /// Puts every connection's screen and camera senders where the policy |
| /// wants them; a no-op for any that already are. |
| func applySendPolicy() { |
| @@ -296,10 +332,12 @@ import LiveKitWebRTC |
| "insertedSamplesForDeceleration": 0, "removedSamplesForAcceleration": 0, |
| "jitterBufferDelayMs": 0, |
| ] |
| - for peer in peers.values { |
| + for (id, peer) in peers { |
| guard let pc = peer.pc else { continue } |
| let ownScreen = await peer.ownScreenMid() |
| let report = await PeerCalls.statistics(pc) |
| + var screenDecoded: UInt64 = 0 |
| + defer { tallyScreenDecoded(id, screenDecoded) } |
| for stat in report.statistics.values { |
| let kind = (stat.values["kind"] as? String) ?? (stat.values["mediaType"] as? String) ?? "" |
| let mid = stat.values["mid"] as? String ?? "" |
| @@ -310,8 +348,9 @@ import LiveKitWebRTC |
| // for a camera that had stopped. Told apart by mid, which is |
| // what the browser's own gate does. |
| let field = peer.isScreen(mid: mid) ? "screenFramesDecoded" : "framesDecoded" |
| - values[field, default: 0] += |
| - (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + let decoded = (stat.values["framesDecoded"] as? NSNumber)?.uint64Value ?? 0 |
| + values[field, default: 0] += decoded |
| + if peer.isScreen(mid: mid) { screenDecoded += decoded } |
| } |
| if kind == "audio" { |
| values["audioBytesReceived", default: 0] += |
| @@ -341,6 +380,7 @@ import LiveKitWebRTC |
| } |
| } |
| } |
| + values["screenFramesDecodedTotal"] = screenDecodedTotal |
| return values |
| } |
| @@ -640,6 +680,7 @@ struct PeerAudioReading: Equatable, Sendable { |
| } |
| } |
| func isScreen(mid: String) -> Bool { !mid.isEmpty && mid == screenMid } |
| + var screenMidHere: String { screenMid } |
| /// The mid our screen went out on, once negotiated: read on the audio |
| /// queue (a transceiver's mid is a hop to WebRTC's signalling thread), |
| /// and kept once known. |
diff --git a/native/apple/Sources/CallsKit/ScreenWatchdog.swift b/native/apple/Sources/CallsKit/ScreenWatchdog.swift| index 8286145..dbcd5db 100644 |
| --- a/native/apple/Sources/CallsKit/ScreenWatchdog.swift |
| +++ b/native/apple/Sources/CallsKit/ScreenWatchdog.swift |
| @@ -60,6 +60,29 @@ enum ScreenRecovery: Equatable, Sendable { |
| /// What stopping and sharing again did: this connection's screen |
| /// transceiver stopped and a new one added, renegotiated. |
| case reAdd(String) |
| + /// A receiver's report that keyframes did not answer: this connection's |
| + /// screen sending path built again — a new transceiver, so a new encoder |
| + /// and, at the far end, a new receiver and decoder — at the same size |
| + /// and mode. What stopping and sharing again does, for that one person. |
| + case rebuild(String) |
| + /// Still reported after that: every connection's screen stopped and |
| + /// added again from the same capture (no picker), which is exactly what |
| + /// Stop and Share did by hand. |
| + case reshare |
| +} |
| + |
| +/// Where the answer to one receiver's report has got to (the sharer's |
| +/// notice says it in words: see ScreenWords.notice). |
| +enum ScreenEscalation: Equatable, Sendable { |
| + case none |
| + /// Keyframes asked for since this time. |
| + case keyframes(since: Double) |
| + /// That peer's screen rebuilt at this time. |
| + case rebuilt(at: Double) |
| + /// Everyone's screen shared again at this time. |
| + case reshared(at: Double) |
| + /// Nothing more to try: the person is told to stop and share again. |
| + case failed |
| } |
| /// The sharer's watchdog: pure, fed once a second while sharing. |
| @@ -83,6 +106,17 @@ enum ScreenRecovery: Equatable, Sendable { |
| /// (the report is a few seconds old when it lands), three times and then |
| /// a rest — and the fuller steps only if the screen has stopped leaving |
| /// here too. |
| +/// - **A receiver keeps saying so** (spec §15.22, Paul 2026-09-30: "still |
| +/// seeing issues with screen sharing after a while", the notice sat at |
| +/// "recovering"). Keyframes cannot mend a decoder that has wedged at the |
| +/// far end, or a receiver that has stopped taking this stream at all; |
| +/// a new transceiver can. Still reported six seconds after the first |
| +/// keyframe: that connection's screen is rebuilt, at the same size and |
| +/// mode (a report alone never shrinks a Text and detail share). Still |
| +/// reported ten seconds after that: everyone's screen is shared again |
| +/// from the same capture. Ten seconds more and the sharer is told it |
| +/// could not be restored. A rebuild at most once a minute per person, |
| +/// a share again at most once every five minutes. |
| struct ScreenWatchdog: Equatable { |
| static let captureStallAfter = 3.0 |
| static let captureBackoffFirst = 5.0 |
| @@ -100,6 +134,14 @@ struct ScreenWatchdog: Equatable { |
| /// and that is not a stall (the motion gate, 2026-09-28). A receiver's |
| /// report is still answered then. |
| static let warmUp = 10.0 |
| + /// §15.22's escalation: how long a report may outlast the first |
| + /// keyframe, then the rebuild, then the share again; and how often each |
| + /// of the last two may happen. |
| + static let rebuildAfter = 6.0 |
| + static let reshareAfter = 10.0 |
| + static let giveUpAfter = 10.0 |
| + static let rebuildEvery = 60.0 |
| + static let reshareEvery = 300.0 |
| struct Peer: Equatable { |
| var last = ScreenSendCounts() |
| @@ -112,7 +154,17 @@ struct ScreenWatchdog: Equatable { |
| var reported = false |
| var restingUntil = 0.0 |
| var since = 0.0 |
| + var escalation = ScreenEscalation.none |
| + var lastRebuild: Double? |
| + /// Frames sent on this connection since the escalation last moved on: |
| + /// a receiver can only be failing to decode frames that were sent. A |
| + /// screen the link is holding back, or one with nothing new to send, |
| + /// is not taken further. |
| + var sentSinceStage: UInt64 = 0 |
| } |
| + /// Frames that must leave for a receiver before its report is taken |
| + /// further than keyframes. |
| + static let escalateAfterSent: UInt64 = 3 |
| private(set) var peers: [String: Peer] = [:] |
| private var heartbeats: UInt64? |
| @@ -128,6 +180,10 @@ struct ScreenWatchdog: Equatable { |
| private(set) var encoderRecoveries = 0 |
| private(set) var reAdds = 0 |
| private(set) var stallReports = 0 |
| + private(set) var rebuilds = 0 |
| + private(set) var reshares = 0 |
| + private(set) var failures = 0 |
| + private var lastReshare: Double? |
| /// One line on what the watchdog is doing, for the device panel. |
| private(set) var status = "watching" |
| @@ -177,6 +233,7 @@ struct ScreenWatchdog: Equatable { |
| // all, so the steps do not start again from nothing on the new one. |
| for (id, counts) in sending { |
| var peer = peers[id] ?? Peer(last: counts, movedAt: now, deliveredAt: delivered, since: now) |
| + peer.sentSinceStage += counts.sent >= peer.last.sent ? counts.sent - peer.last.sent : counts.sent |
| if counts.encoded < peer.last.encoded || counts.sent < peer.last.sent { |
| // A new transceiver's counters, starting again from nothing. |
| peer.last = counts |
| @@ -238,20 +295,123 @@ struct ScreenWatchdog: Equatable { |
| } |
| peers[id] = peer |
| } |
| + out += escalate(at: now, stalledBy: stalledBy, negotiated: Set(sending.keys)) |
| status = describe() |
| return out |
| } |
| + /// §15.22: a report that keyframes have not answered, taken further. Run |
| + /// over every peer with a record, whether or not its screen is |
| + /// negotiated this second — a rebuilt connection has none for a moment, |
| + /// and a rebuild that never comes back must still end in words. |
| + private mutating func escalate( |
| + at now: Double, stalledBy: Set<String>, negotiated: Set<String> |
| + ) -> [ScreenRecovery] { |
| + var out: [ScreenRecovery] = [] |
| + var reshare = false |
| + for id in peers.keys.sorted() { |
| + guard var peer = peers[id] else { continue } |
| + let reported = stalledBy.contains(id) |
| + // Frames went to this receiver since the last step, or there is no |
| + // screen negotiated with it at all (a rebuild that never came back). |
| + let owed = peer.sentSinceStage >= Self.escalateAfterSent || !negotiated.contains(id) |
| + let before = peer.escalation |
| + defer { |
| + if peers[id]?.escalation != before { peers[id]?.sentSinceStage = 0 } |
| + } |
| + if !reported { |
| + if peer.escalation != .none { |
| + switch peer.escalation { |
| + case .keyframes: ScreenLog.info("\(Self.short(id))'s view restored after keyframes") |
| + case .rebuilt: ScreenLog.info("\(Self.short(id))'s view restored after the rebuild") |
| + case .reshared: ScreenLog.info("\(Self.short(id))'s view restored after sharing again") |
| + case .failed: ScreenLog.info("\(Self.short(id))'s view came back after all") |
| + case .none: break |
| + } |
| + peer.escalation = .none |
| + } |
| + peers[id] = peer |
| + continue |
| + } |
| + let canRebuild = peer.lastRebuild.map { now - $0 >= Self.rebuildEvery } ?? true |
| + let canReshare = reshare || (lastReshare.map { now - $0 >= Self.reshareEvery } ?? true) |
| + switch peer.escalation { |
| + case .none: |
| + peer.escalation = .keyframes(since: now) |
| + case .keyframes(let since) where now - since >= Self.rebuildAfter && owed: |
| + if canRebuild { |
| + out.append(.rebuild(id)) |
| + rebuilds += 1 |
| + peer.lastRebuild = now |
| + peer.escalation = .rebuilt(at: now) |
| + ScreenLog.info( |
| + "\(Self.short(id)) still reports a stall \(Int(now - since)) s after keyframes: rebuilding its screen") |
| + } else if canReshare { |
| + reshare = true |
| + peer.escalation = .reshared(at: now) |
| + ScreenLog.info("\(Self.short(id)) still reports a stall, rebuilt under a minute ago: sharing again") |
| + } else { |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall and every step is resting: giving up") |
| + } |
| + case .rebuilt(let at) where now - at >= Self.reshareAfter && owed: |
| + if canReshare { |
| + reshare = true |
| + peer.escalation = .reshared(at: now) |
| + ScreenLog.info("\(Self.short(id)) still reports a stall \(Int(now - at)) s after the rebuild: sharing again") |
| + } else { |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall after the rebuild, shared again under five minutes ago: giving up") |
| + } |
| + case .reshared(let at) where now - at >= Self.giveUpAfter && owed: |
| + peer.escalation = .failed |
| + failures += 1 |
| + ScreenLog.error("\(Self.short(id)) still reports a stall \(Int(now - at)) s after sharing again: giving up") |
| + default: |
| + break |
| + } |
| + peers[id] = peer |
| + } |
| + if reshare { |
| + out.append(.reshare) |
| + reshares += 1 |
| + lastReshare = now |
| + // One share again answers every report in flight. |
| + for (id, peer) in peers where stalledBy.contains(id) { |
| + switch peer.escalation { |
| + case .keyframes, .rebuilt: |
| + peers[id]?.escalation = .reshared(at: now) |
| + peers[id]?.sentSinceStage = 0 |
| + default: break |
| + } |
| + } |
| + } |
| + return out |
| + } |
| + |
| + /// Where the answer to each reporting peer has got to, for the notice. |
| + func escalation(of peer: String) -> ScreenEscalation { peers[peer]?.escalation ?? .none } |
| + |
| /// Whether anyone's view of the screen is being recovered right now. |
| var recovering: Bool { captureStalled || peers.values.contains { $0.step > 0 || $0.reported } } |
| private func describe() -> String { |
| if captureStalled { return "capture stalled, restarting (\(captureRestarts))" } |
| + let stages = peers.values.map(\.escalation) |
| + if stages.contains(.failed) { return "a receiver's view could not be restored" } |
| + if stages.contains(where: { if case .reshared = $0 { return true } else { return false } }) { |
| + return "shared again for everyone" |
| + } |
| + if stages.contains(where: { if case .rebuilt = $0 { return true } else { return false } }) { |
| + return "rebuilt for a receiver" |
| + } |
| if let (_, p) = peers.first(where: { $0.value.step > 0 }) { |
| return ["", "keyframe asked for", "encoder recreated", "re-added"][min(p.step, 3)] |
| } |
| if peers.values.contains(where: \.reported) { return "a receiver reports a stall" } |
| - let total = captureRestarts + keyframes + encoderRecoveries + reAdds |
| + let total = captureRestarts + keyframes + encoderRecoveries + reAdds + rebuilds + reshares |
| return total == 0 ? "watching" : "watching, \(total) recover\(total == 1 ? "y" : "ies") so far" |
| } |
| @@ -261,21 +421,37 @@ struct ScreenWatchdog: Equatable { |
| /// The receiver's watchdog: whose screens have stopped moving here. |
| /// |
| /// A peer's screen is stalled when nothing new has been decoded from it |
| -/// for four seconds while it says it is sharing. Both apps and the browser |
| -/// repeat a still screen at least once a second, so four seconds without |
| -/// a frame is not a still screen. The list is reported as `screenStalled` |
| -/// (spec §15.21) and the sharer recovers; it clears as soon as frames |
| -/// move, and a report is never held for more than a minute — a screen |
| -/// the sharer cannot recover is not made into a report that never ends. |
| +/// for four seconds while it says it is sharing. Every sender repeats a |
| +/// still screen at least once a second (the iPhone's broadcast too, since |
| +/// 2026-09-30), so four seconds without a frame is not a still screen. The |
| +/// list is reported as `screenStalled` (spec §15.21) and the sharer |
| +/// recovers; it clears as soon as frames move. |
| +/// |
| +/// Any change in the count is movement, a smaller one included: a screen |
| +/// the sharer has rebuilt arrives on a new receiver whose count starts |
| +/// again from nothing (2026-09-30 — before this, a rebuilt screen that was |
| +/// moving again stayed "stalled" here until its new count passed the old |
| +/// one, and the sharer's notice with it). |
| +/// |
| +/// Four seconds into a stall the tile's picture is also dropped and |
| +/// attached again, once (spec §15.22): a renderer that has stopped, on |
| +/// this side, is this side's to mend. A report is held for at most three |
| +/// minutes from the last frame, long enough for the sharer to try |
| +/// everything and say so, and no longer. |
| struct ScreenReceiveWatch: Equatable { |
| static let stallAfter = 4.0 |
| - static let giveUpAfter = 60.0 |
| + static let reattachAfter = 4.0 |
| + static let giveUpAfter = 180.0 |
| struct Seen: Equatable { |
| var decoded: UInt64 |
| var movedAt: Double |
| + var reattached = false |
| } |
| private(set) var seen: [String: Seen] = [:] |
| + /// The peers whose picture should be attached again now: set by each |
| + /// look, and each stall's only once. |
| + private(set) var reattach: [String] = [] |
| init() {} |
| @@ -284,18 +460,26 @@ struct ScreenReceiveWatch: Equatable { |
| mutating func observe(at now: Double, decoded: [String: UInt64]) -> [String] { |
| for id in seen.keys where decoded[id] == nil { seen.removeValue(forKey: id) } |
| var stalled: [String] = [] |
| + reattach = [] |
| for (id, count) in decoded { |
| - guard var s = seen[id], count <= s.decoded else { |
| + guard var s = seen[id], count == s.decoded else { |
| seen[id] = Seen(decoded: count, movedAt: now) |
| continue |
| } |
| - s.decoded = count |
| - seen[id] = s |
| let quiet = now - s.movedAt |
| if quiet >= Self.stallAfter && quiet < Self.giveUpAfter { stalled.append(id) } |
| + if quiet >= Self.stallAfter + Self.reattachAfter && !s.reattached && quiet < Self.giveUpAfter { |
| + s.reattached = true |
| + reattach.append(id) |
| + } |
| + seen[id] = s |
| } |
| + reattach.sort() |
| return stalled.sorted() |
| } |
| + |
| + /// How long each stalled peer's screen has gone without a frame. |
| + func quiet(at now: Double) -> [String: Double] { seen.mapValues { now - $0.movedAt } } |
| } |
| /// The words for the sharer's quiet notice. |
| @@ -310,4 +494,108 @@ public enum ScreenWords { |
| default: return "\(names.count) people\u{2019}s views of your screen stalled \u{2014} recovering" |
| } |
| } |
| + |
| + /// The notice as the escalation moves on (spec §15.22): "recovering", |
| + /// then "Restarting your share for Samuel…", then either nothing or |
| + /// "Couldn’t restore Samuel’s view — try Stop and Share again". Worst |
| + /// news first when several people are at different steps. |
| + public static func notice(recovering: [String], restarting: [String], failed: [String]) -> String { |
| + var parts: [String] = [] |
| + if !failed.isEmpty { |
| + let whose: String |
| + switch failed.count { |
| + case 1: whose = "\(failed[0])\u{2019}s view" |
| + case 2: whose = "\(failed[0])\u{2019}s and \(failed[1])\u{2019}s views" |
| + default: whose = "\(failed.count) people\u{2019}s views" |
| + } |
| + parts.append("Couldn\u{2019}t restore \(whose) \u{2014} try Stop and Share again") |
| + } |
| + if !restarting.isEmpty { |
| + let who: String |
| + switch restarting.count { |
| + case 1: who = restarting[0] |
| + case 2: who = "\(restarting[0]) and \(restarting[1])" |
| + default: who = "\(restarting.count) people" |
| + } |
| + parts.append("Restarting your share for \(who)\u{2026}") |
| + } |
| + let stalled = Self.recovering(names: recovering) |
| + if !stalled.isEmpty { parts.append(stalled) } |
| + return parts.joined(separator: ". ") |
| + } |
| +} |
| + |
| +/// One peer's screen as it arrives here, from the receiver's statistics |
| +/// (spec §15.22): enough to tell the ways a screen stops apart. Packets |
| +/// arriving and frames assembled but none decoded is a decoder that has |
| +/// wedged; nothing arriving at all is the sender, or the path; frames |
| +/// decoded and a picture that does not move is the renderer. |
| +struct ScreenReceiveReport: Equatable, Sendable { |
| + var packetsReceived: UInt64 = 0 |
| + var packetsLost: Int64 = 0 |
| + var bytesReceived: UInt64 = 0 |
| + var framesReceived: UInt64 = 0 |
| + var framesDecoded: UInt64 = 0 |
| + var framesDropped: UInt64 = 0 |
| + var keyFramesDecoded: UInt64 = 0 |
| + var pliCount: UInt64 = 0 |
| + var firCount: UInt64 = 0 |
| + var nackCount: UInt64 = 0 |
| + var freezeCount: UInt64 = 0 |
| + var width = 0 |
| + var height = 0 |
| + var decoder = "" |
| + /// When the last packet arrived, in the statistics' own milliseconds. |
| + var lastPacketAt = 0.0 |
| + var mid = "" |
| + |
| + init() {} |
| + |
| + /// One inbound-rtp report's values added in (a screen has one). |
| + mutating func add(_ v: [String: NSObject]) { |
| + func n(_ key: String) -> UInt64 { (v[key] as? NSNumber)?.uint64Value ?? 0 } |
| + packetsReceived += n("packetsReceived") |
| + packetsLost += (v["packetsLost"] as? NSNumber)?.int64Value ?? 0 |
| + bytesReceived += n("bytesReceived") |
| + framesReceived += n("framesReceived") |
| + framesDecoded += n("framesDecoded") |
| + framesDropped += n("framesDropped") |
| + keyFramesDecoded += n("keyFramesDecoded") |
| + pliCount += n("pliCount") |
| + firCount += n("firCount") |
| + nackCount += n("nackCount") |
| + freezeCount += n("freezeCount") |
| + width = max(width, Int(n("frameWidth"))) |
| + height = max(height, Int(n("frameHeight"))) |
| + if let name = v["decoderImplementation"] as? String, !name.isEmpty { decoder = name } |
| + lastPacketAt = max(lastPacketAt, (v["lastPacketReceivedTimestamp"] as? NSNumber)?.doubleValue ?? 0) |
| + } |
| + |
| + /// What changed between two looks, in words: "packets +120, frames |
| + /// received +30, decoded +0, dropped +0, keyframes +0, PLI +2". |
| + func moved(since before: ScreenReceiveReport?) -> String { |
| + guard let b = before, b.mid == mid else { return summary } |
| + func d(_ a: UInt64, _ z: UInt64) -> String { a >= z ? "+\(a - z)" : "new \(a)" } |
| + return "packets \(d(packetsReceived, b.packetsReceived)), frames received \(d(framesReceived, b.framesReceived)), " |
| + + "decoded \(d(framesDecoded, b.framesDecoded)), dropped \(d(framesDropped, b.framesDropped)), " |
| + + "keyframes \(d(keyFramesDecoded, b.keyFramesDecoded)), PLI \(d(pliCount, b.pliCount)), " |
| + + "NACK \(d(nackCount, b.nackCount))" |
| + } |
| + |
| + /// The running totals, one line. |
| + var summary: String { |
| + "mid \(mid.isEmpty ? "-" : mid), \(width)x\(height), decoder \(decoder.isEmpty ? "unknown" : decoder); " |
| + + "packets \(packetsReceived) (lost \(packetsLost)), frames received \(framesReceived), " |
| + + "decoded \(framesDecoded), dropped \(framesDropped), keyframes \(keyFramesDecoded), " |
| + + "PLI \(pliCount), FIR \(firCount), NACK \(nackCount), freezes \(freezeCount)" |
| + } |
| + |
| + /// The verdict the numbers point to, for a screen that has stopped. |
| + func diagnosis(since before: ScreenReceiveReport?) -> String { |
| + guard let b = before, b.mid == mid else { return "no earlier reading" } |
| + if packetsReceived == b.packetsReceived { return "no packets arriving (the sender or the path)" } |
| + if framesReceived == b.framesReceived { return "packets arriving, no frames assembled (loss, or waiting on a keyframe)" } |
| + if framesDecoded == b.framesDecoded { return "frames arriving, none decoded (the decoder)" } |
| + return "decoding" |
| + } |
| } |
diff --git a/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift b/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift| index 3e125e3..85aa5c4 100644 |
| --- a/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift |
| +++ b/native/apple/Tests/CallsKitTests/ScreenWatchdogTests.swift |
| @@ -83,6 +83,25 @@ final class ScreenWatchdogTests: XCTestCase { |
| XCTAssertEqual(dog.reAdds, 1) |
| } |
| + func testAnEncoderStoppedForOnePersonIsMendedForThemAlone() { |
| + // Each connection has an encoder of its own: the browser's (H.264 at |
| + // the level Chromium answers) can refuse frames the app's (High 5.2) |
| + // takes. The steps go to the stopped one only. |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<20 { |
| + let delivered = UInt64(t * 30) |
| + let sending = [ |
| + "browser": ScreenSendCounts(encoded: t < 5 ? delivered : 120, sent: t < 5 ? delivered : 120, keyframes: 1), |
| + "app": ScreenSendCounts(encoded: delivered, sent: delivered, keyframes: 1), |
| + ] |
| + for a in dog.observe(at: Double(t), capture: counts(delivered, delivered), sending: sending, stalledBy: []) { |
| + log.append((t, a)) |
| + } |
| + } |
| + XCTAssertEqual(log.map(\.1), [.keyframe("browser"), .recreateEncoder("browser"), .reAdd("browser")]) |
| + } |
| + |
| func testAnEncoderThatRecoversAfterTheKeyframeIsLeftAlone() { |
| var dog = ScreenWatchdog() |
| var actions: [ScreenRecovery] = [] |
| @@ -126,7 +145,7 @@ final class ScreenWatchdogTests: XCTestCase { |
| } |
| } |
| - func testAReceiversReportWhileTheScreenLeavesNormallyGetsKeyframesOnly() { |
| + func testAReceiversReportWhileTheScreenLeavesNormallyGetsKeyframesFirstAndNeverAShrink() { |
| var dog = ScreenWatchdog() |
| var log: [(Int, ScreenRecovery)] = [] |
| for t in 0..<20 { |
| @@ -136,8 +155,12 @@ final class ScreenWatchdogTests: XCTestCase { |
| log.append((t, a)) |
| } |
| } |
| - XCTAssertEqual(log.map(\.1), [.keyframe("vicky"), .keyframe("vicky"), .keyframe("vicky")]) |
| - XCTAssertEqual(log.map(\.0), [3, 7, 11]) |
| + let keyframes = log.filter { $0.1 == .keyframe("vicky") } |
| + XCTAssertEqual(keyframes.map(\.0), [3, 7, 11]) |
| + // What comes after the keyframes (spec §15.22) is a rebuild at the |
| + // same size, never a smaller encoder. |
| + XCTAssertFalse(log.contains { $0.1 == .recreateEncoder("vicky") }) |
| + XCTAssertEqual(log.first?.1, .keyframe("vicky")) |
| XCTAssertEqual(dog.stallReports, 1, "one report, however long it lasts") |
| XCTAssertTrue(dog.recovering) |
| } |
| @@ -182,17 +205,197 @@ final class ScreenWatchdogTests: XCTestCase { |
| } |
| } |
| + // MARK: - A receiver that keeps reporting (spec §15.22) |
| + |
| + private func escalating( |
| + _ range: Range<Int>, stalledBy: (Int) -> Set<String>, sent: (Int) -> [String: ScreenSendCounts]? = { _ in nil }, |
| + dog: inout ScreenWatchdog |
| + ) -> [(Int, ScreenRecovery)] { |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in range { |
| + let n = UInt64(t * 30) |
| + for a in dog.observe(at: Double(t), capture: counts(n, n), sending: sent(t) ?? sending(n), stalledBy: stalledBy(t)) { |
| + log.append((t, a)) |
| + } |
| + } |
| + return log |
| + } |
| + |
| + func testAReportKeyframesDoNotAnswerIsRebuiltThenSharedAgainThenGivenUp() { |
| + var dog = ScreenWatchdog() |
| + let log = escalating(0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, dog: &dog) |
| + let steps = log.filter { if case .keyframe = $0.1 { return false } else { return true } } |
| + // Reported from 3: keyframes at once; six seconds on, still reported, |
| + // the connection is rebuilt; ten more, everyone's screen shared again; |
| + // ten more, given up. |
| + XCTAssertEqual(steps.map(\.1), [.rebuild("vicky"), .reshare]) |
| + XCTAssertEqual(steps.map(\.0), [9, 19]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + XCTAssertEqual(dog.rebuilds, 1) |
| + XCTAssertEqual(dog.reshares, 1) |
| + XCTAssertEqual(dog.failures, 1) |
| + // A report alone never shrinks the screen or recreates its encoder: |
| + // Text and detail stays Text and detail. |
| + XCTAssertFalse(log.contains { if case .recreateEncoder = $0.1 { return true } else { return false } }) |
| + XCTAssertEqual(dog.status, "a receiver's view could not be restored") |
| + } |
| + |
| + func testAFailureStaysUntilTheReportClears() { |
| + var dog = ScreenWatchdog() |
| + _ = escalating(0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, dog: &dog) |
| + let later = escalating(45..<120, stalledBy: { $0 < 100 ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertFalse(later.contains { $0.1 == .reshare || $0.1 == .rebuild("vicky") }, "nothing more is tried") |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .none, "and a view that comes back is forgotten") |
| + } |
| + |
| + func testARebuildThatWorksEndsIt() { |
| + var dog = ScreenWatchdog() |
| + let log = escalating(0..<60, stalledBy: { (3..<12).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + let steps = log.filter { if case .keyframe = $0.1 { return false } else { return true } } |
| + XCTAssertEqual(steps.map(\.1), [.rebuild("vicky")]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .none) |
| + XCTAssertEqual(dog.reshares, 0) |
| + } |
| + |
| + func testRebuildsAndSharesAgainAreRationed() { |
| + var dog = ScreenWatchdog() |
| + // First stall: rebuilt at 9, and that works. |
| + _ = escalating(0..<20, stalledBy: { (3..<12).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + // A second, within the minute: no rebuild so soon, so straight to a |
| + // share again; that works too. |
| + let second = escalating(20..<50, stalledBy: { (25..<34).contains($0) ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertEqual(second.filter { $0.1 == .reshare || $0.1 == .rebuild("vicky") }.map(\.1), [.reshare]) |
| + // A third, past the minute but inside five: rebuilt, and then there is |
| + // no share again to be had, so it is given up. |
| + let third = escalating(80..<120, stalledBy: { $0 >= 85 ? ["vicky"] : [] }, dog: &dog) |
| + XCTAssertEqual(third.filter { $0.1 == .reshare || $0.1 == .rebuild("vicky") }.map(\.1), [.rebuild("vicky")]) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + } |
| + |
| + func testNothingSentIsNothingToEscalate() { |
| + // A still screen with nothing new to send, or a link holding it all |
| + // back: the receiver cannot be failing to decode frames that never |
| + // left, so keyframes are all it gets. |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<40 { |
| + // Reporting all along (idle reports), nothing new delivered or sent |
| + // after t=3. |
| + let n = UInt64(min(t, 3) * 30) |
| + let still = ["vicky": ScreenSendCounts(encoded: n, sent: n, keyframes: 1)] |
| + for a in dog.observe( |
| + at: Double(t), capture: counts(UInt64(t * 30), n), sending: still, stalledBy: t >= 3 ? ["vicky"] : []) |
| + { |
| + log.append((t, a)) |
| + } |
| + } |
| + XCTAssertFalse(log.isEmpty) |
| + XCTAssertTrue(log.allSatisfy { if case .keyframe = $0.1 { return true } else { return false } }, "\(log)") |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .keyframes(since: 3)) |
| + } |
| + |
| + func testARebuildThatNeverComesBackStillEndsInWords() { |
| + // The rebuilt connection never negotiates a screen again: there is |
| + // nothing to count, and the steps go on regardless. |
| + var dog = ScreenWatchdog() |
| + let log = escalating( |
| + 0..<45, stalledBy: { $0 >= 3 ? ["vicky"] : [] }, sent: { $0 > 9 ? [:] : nil }, dog: &dog) |
| + XCTAssertTrue(log.contains { $0.1 == .reshare }) |
| + XCTAssertEqual(dog.escalation(of: "vicky"), .failed) |
| + } |
| + |
| + func testOneShareAgainAnswersEveryReport() { |
| + var dog = ScreenWatchdog() |
| + var log: [(Int, ScreenRecovery)] = [] |
| + for t in 0..<30 { |
| + let n = UInt64(t * 30) |
| + let both = ["vicky": ScreenSendCounts(encoded: n, sent: n, keyframes: 1), "tony": ScreenSendCounts(encoded: n, sent: n, keyframes: 1)] |
| + let stalled: Set<String> = t >= 3 ? ["vicky", "tony"] : [] |
| + for a in dog.observe(at: Double(t), capture: counts(n, n), sending: both, stalledBy: stalled) { log.append((t, a)) } |
| + } |
| + XCTAssertEqual(log.filter { $0.1 == .reshare }.count, 1) |
| + XCTAssertEqual(log.filter { if case .rebuild = $0.1 { return true } else { return false } }.count, 2) |
| + } |
| + |
| + func testTheNoticeMovesOnInWords() { |
| + XCTAssertEqual(ScreenWords.notice(recovering: [], restarting: [], failed: []), "") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: ["Samuel"], restarting: [], failed: []), |
| + "Samuel\u{2019}s view of your screen stalled \u{2014} recovering") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: ["Samuel"], failed: []), "Restarting your share for Samuel\u{2026}") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: [], failed: ["Samuel"]), |
| + "Couldn\u{2019}t restore Samuel\u{2019}s view \u{2014} try Stop and Share again") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: ["Tony"], restarting: [], failed: ["Samuel"]), |
| + "Couldn\u{2019}t restore Samuel\u{2019}s view \u{2014} try Stop and Share again. Tony\u{2019}s view of your screen stalled \u{2014} recovering") |
| + XCTAssertEqual( |
| + ScreenWords.notice(recovering: [], restarting: ["A", "B", "C"], failed: []), "Restarting your share for 3 people\u{2026}") |
| + } |
| + |
| + func testTheReceiversNumbersNameTheStall() { |
| + var before = ScreenReceiveReport() |
| + before.mid = "3" |
| + before.packetsReceived = 1000 |
| + before.framesReceived = 300 |
| + before.framesDecoded = 300 |
| + var wedged = before |
| + wedged.packetsReceived = 1400 |
| + wedged.framesReceived = 420 |
| + XCTAssertEqual(wedged.diagnosis(since: before), "frames arriving, none decoded (the decoder)") |
| + XCTAssertTrue(wedged.moved(since: before).hasPrefix("packets +400, frames received +120, decoded +0")) |
| + XCTAssertEqual(before.diagnosis(since: before), "no packets arriving (the sender or the path)") |
| + var rebuilt = wedged |
| + rebuilt.mid = "5" |
| + XCTAssertEqual(rebuilt.diagnosis(since: before), "no earlier reading") |
| + } |
| + |
| // MARK: - The receiver |
| - func testAScreenThatStopsMovingIsReportedAfterFourSecondsAndForAMinuteAtMost() { |
| + func testAScreenThatStopsMovingIsReportedAfterFourSecondsAndForThreeMinutesAtMost() { |
| var watch = ScreenReceiveWatch() |
| var reported: [Int] = [] |
| - for t in 0..<80 { |
| + for t in 0..<200 { |
| let decoded: UInt64 = t < 10 ? UInt64(t * 10) : 90 |
| if !watch.observe(at: Double(t), decoded: ["paul": decoded]).isEmpty { reported.append(t) } |
| } |
| XCTAssertEqual(reported.first, 13, "last moved at 9: stalled from 13") |
| - XCTAssertEqual(reported.last, 68, "and no longer than a minute after it last moved") |
| + XCTAssertEqual(reported.last, 188, "and no longer than three minutes after it last moved") |
| + } |
| + |
| + func testARebuiltScreenStartingFromZeroIsMovingNotStalled() { |
| + // 2026-09-30: a screen the sharer rebuilt arrives on a new receiver |
| + // whose count starts again. Up to 0.4.4 only a larger count was |
| + // movement, so a rebuilt screen that was moving again stayed stalled |
| + // here — and the sharer's notice with it — until the new count passed |
| + // the old. |
| + var watch = ScreenReceiveWatch() |
| + for t in 0..<10 { _ = watch.observe(at: Double(t), decoded: ["paul": 900 + UInt64(t * 30)]) } |
| + for t in 10..<16 { _ = watch.observe(at: Double(t), decoded: ["paul": 1170]) } |
| + XCTAssertEqual(watch.observe(at: 16, decoded: ["paul": 1170]), ["paul"]) |
| + for t in 17..<40 { |
| + XCTAssertEqual(watch.observe(at: Double(t), decoded: ["paul": UInt64((t - 17) * 30)]), [], "t=\(t)") |
| + } |
| + } |
| + |
| + func testAStalledPictureIsAttachedAgainOnceFourSecondsIn() { |
| + var watch = ScreenReceiveWatch() |
| + var reattached: [Int] = [] |
| + for t in 0..<60 { |
| + let decoded: UInt64 = t < 10 ? UInt64(t * 10) : 90 |
| + _ = watch.observe(at: Double(t), decoded: ["paul": decoded]) |
| + if watch.reattach == ["paul"] { reattached.append(t) } |
| + } |
| + XCTAssertEqual(reattached, [17], "stalled from 13, attached again four seconds later, once") |
| + // Moving again and stalled again: a new stall, a new try. |
| + for t in 60..<63 { _ = watch.observe(at: Double(t), decoded: ["paul": UInt64(t)]) } |
| + var again: [Int] = [] |
| + for t in 63..<80 { |
| + _ = watch.observe(at: Double(t), decoded: ["paul": 62]) |
| + if !watch.reattach.isEmpty { again.append(t) } |
| + } |
| + XCTAssertEqual(again, [70]) |
| } |
| func testAMovingScreenIsNeverReported() { |
diff --git a/test/browser/native-screen.mjs b/test/browser/native-screen.mjs| index 637b9d0..44b4ec8 100644 |
| --- a/test/browser/native-screen.mjs |
| +++ b/test/browser/native-screen.mjs |
| @@ -21,6 +21,24 @@ |
| // sender deactivated under it): the watchdog must recover it |
| // loss the browser stops painting and asks for a keyframe |
| // (screenStalled): the app must answer with one |
| +// wedge the browser's screen receiver stops passing anything to its |
| +// decoder (an encoded transform that drops every frame), which |
| +// no keyframe mends: the app must escalate to a rebuild and |
| +// the browser decode again within 30 s, the app's notice going |
| +// "recovering", "Restarting your share for …", then nothing |
| +// (spec §15.22) |
| +// wedge-sticky every screen receiver the browser gets is wedged, new |
| +// ones too: the app must end at "Couldn’t restore …", and |
| +// clear once the wedge is lifted |
| +// soak renewals and resumes as `renewals`, for SCREEN_SOAK_SECONDS, |
| +// with the mode cycled (balanced, motion, detail) and the |
| +// app's uplink capped and lifted every SCREEN_SOAK_STEP_MS: |
| +// every freeze is logged with what the ladder did about it |
| +// web-soak the same with the browser sharing and the app watching, the |
| +// browser's mode cycled |
| +// web-wedge, web-wedge-sticky the same the other way round: the |
| +// browser shares, the app's screen decoder wedges, and the |
| +// browser's ladder has to restore it (or say it cannot) |
| // |
| // Run by TestNativeMacScreenRecovery (make native-browser). |
| import { chromium } from 'playwright'; |
| @@ -41,23 +59,104 @@ const browser = await chromium.launch({ headless: true, args: [ |
| '--use-fake-device-for-media-stream', '--use-fake-ui-for-media-stream', |
| '--autoplay-policy=no-user-gesture-required', '--disable-features=WebRtcHideLocalIpsWithMdns', |
| '--mute-audio', |
| + // getDisplayMedia answered with a fake 1280x720 monitor and no picker, |
| + // for the web-* scenarios (see call.mjs). |
| + '--auto-accept-this-tab-capture', |
| ] }); |
| -const context = await browser.newContext({ permissions: ['camera', 'microphone'] }); |
| +// bypassCSP: the wedge's encoded transform runs in a worker made from a |
| +// blob, which the page's own policy (default-src 'self') would refuse. |
| +// The gate's, never the page's. |
| +const context = await browser.newContext({ permissions: ['camera', 'microphone'], bypassCSP: true }); |
| await context.addCookies([{ name: process.env.CALLS_COOKIE_NAME, value: process.env.CALLS_COOKIE_A, url: origin }]); |
| const page = await context.newPage(); |
| await page.addInitScript(() => { |
| const Native = window.RTCPeerConnection; |
| window.__nativeInteropPCs = new Set(); |
| + // §15.22's wedge: a receiver whose frames never reach its decoder. |
| + // Chromium takes an encoded transform only on a receiver that has not |
| + // started, so in the wedge scenarios every screen receiver gets one as |
| + // it arrives — passing everything through — and __wedgeScreens tells |
| + // the ones there now to swallow every frame instead. `sticky` has each |
| + // one that arrives after start out swallowing. |
| + let worker = null; |
| + let serial = 0; |
| + const wedgeWorker = () => { |
| + if (!worker) { |
| + const code = [ |
| + 'const drops = new Map();', |
| + 'onmessage = (m) => { for (const id of m.data.ids) drops.set(id, m.data.drop); };', |
| + 'onrtctransform = (e) => {', |
| + ' const id = e.transformer.options.id;', |
| + ' if (!drops.has(id)) drops.set(id, !!e.transformer.options.drop);', |
| + ' e.transformer.readable.pipeThrough(new TransformStream({ transform(f, c) { if (!drops.get(id)) c.enqueue(f); } }))', |
| + ' .pipeTo(e.transformer.writable);', |
| + '};', |
| + ].join('\n'); |
| + worker = new Worker(URL.createObjectURL(new Blob([code], { type: 'text/javascript' }))); |
| + } |
| + return worker; |
| + }; |
| + window.__wedge = { sticky: false, wedged: 0, ids: new Map() }; |
| + const screenReceiver = (tx) => tx && tx.receiver && tx.receiver.track && tx.receiver.track.kind === 'video' && |
| + tx.direction === 'recvonly' && tx.currentDirection !== 'stopped'; |
| + const watchReceiver = (tx) => { |
| + if (!window.__wedgeReady || !screenReceiver(tx) || window.__wedge.ids.has(tx.receiver)) return; |
| + const id = ++serial; |
| + window.__wedge.ids.set(tx.receiver, id); |
| + tx.receiver.transform = new RTCRtpScriptTransform(wedgeWorker(), { id, drop: window.__wedge.sticky }); |
| + if (window.__wedge.sticky) window.__wedge.wedged += 1; |
| + }; |
| + window.__wedgeScreens = (sticky) => { |
| + window.__wedge.sticky = !!sticky; |
| + const ids = []; |
| + for (const pc of window.__nativeInteropPCs) { |
| + for (const tx of pc.getTransceivers()) { |
| + const id = screenReceiver(tx) && window.__wedge.ids.get(tx.receiver); |
| + if (id) ids.push(id); |
| + } |
| + } |
| + wedgeWorker().postMessage({ ids, drop: true }); |
| + window.__wedge.wedged += ids.length; |
| + return ids.length; |
| + }; |
| + window.__releaseWedge = () => { |
| + window.__wedge.sticky = false; |
| + wedgeWorker().postMessage({ ids: [...window.__wedge.ids.values()], drop: false }); |
| + }; |
| window.RTCPeerConnection = class extends Native { |
| - constructor(...args) { super(...args); window.__nativeInteropPCs.add(this); } |
| + // The web-* scenarios' screen goes out as H.264: the app can only |
| + // wedge a VideoToolbox decoder (DecoderWedge.swift), and the browser |
| + // left to itself sends VP8. What is under test, the ladder, does not |
| + // turn on the codec. |
| + addTransceiver(track, init) { |
| + const tx = super.addTransceiver(track, init); |
| + if (window.__h264Screen && init && init.direction === 'sendonly' && track && track.kind === 'video') { |
| + const codecs = RTCRtpSender.getCapabilities('video').codecs; |
| + const h264 = codecs.filter((c) => /h264/i.test(c.mimeType)); |
| + try { |
| + tx.setCodecPreferences([...h264, ...codecs.filter((c) => !/h264/i.test(c.mimeType))]); |
| + } catch {} |
| + } |
| + return tx; |
| + } |
| + constructor(...args) { |
| + super(...args); |
| + window.__nativeInteropPCs.add(this); |
| + this.addEventListener('track', (e) => watchReceiver(e.transceiver)); |
| + } |
| }; |
| }); |
| +// Receivers made ready to wedge only in the scenarios that wedge them, so |
| +// every other scenario's browser decodes exactly as a person's would. |
| +await page.addInitScript((ready) => { window.__wedgeReady = ready; }, scenario === 'wedge' || scenario === 'wedge-sticky'); |
| let native; |
| let failed = false; |
| try { |
| await page.goto(`${origin}/m/${room}`); |
| await page.waitForFunction(() => !!window.meet); |
| - native = spawn(binary, [], { stdio: ['pipe', 'pipe', 'pipe'] }); |
| + // The app's decoders can be wedged in the web-* scenarios only. |
| + const env = scenario.startsWith('web-') ? Object.assign({}, process.env, { CALLS_DECODER_WEDGE: '1' }) : process.env; |
| + native = spawn(binary, [], { stdio: ['pipe', 'pipe', 'pipe'], env }); |
| native.stdin.write(process.env.CALLS_NATIVE_INPUT + '\n'); |
| let sample = null; |
| let exited = false; |
| @@ -82,6 +181,17 @@ try { |
| }; |
| await wait(() => sample?.stats.framesDecoded > 15, 'native receives the browser'); |
| const nativePeer = sample.peer; |
| + if (scenario === 'web-soak') { |
| + await webSoak(page, native, () => sample, wait); |
| + console.log(`PASS ${scenario}`); |
| + throw new Done(); |
| + } |
| + if (scenario.startsWith('web-')) { |
| + await webShares(page, native, () => sample, replies, wait, scenario.endsWith('sticky')); |
| + console.log(`PASS ${scenario}`); |
| + process.exitCode = 0; |
| + throw new Done(); |
| + } |
| const key = nativePeer + '#share'; |
| // What the browser is receiving on the app's screen, from the |
| // connection's own statistics: the screen is the inbound video that is |
| @@ -110,7 +220,12 @@ try { |
| // The mode first: it is remembered across runs of the interop binary, |
| // and the screen's source is made for the mode the share starts in. |
| native.stdin.write(`screen-mode ${process.env.SCREEN_MODE || 'balanced'}\n`); |
| + // SCREEN_SOURCE=motion shares the 1920-wide scrolling page (its height |
| + // CALLS_SYNTHETIC_SCREEN_HEIGHT) rather than the small still screen: |
| + // what meets H.264 level 3.1's limit (§15.22). |
| if (scenario === 'display') native.stdin.write(`screen-display ${process.env.SCREEN_DEPTH || 6}\n`); |
| + else if (process.env.SCREEN_SOURCE === 'motion') native.stdin.write('screen-motion\n'); |
| + else if (process.env.SCREEN_SOURCE === 'display') native.stdin.write(`screen-display ${process.env.SCREEN_DEPTH || 6}\n`); |
| else native.stdin.write('screen-on\n'); |
| await wait(() => sample?.sharingScreen === true, 'native says it is sharing'); |
| await wait(async () => (await received()).frames > 15, 'the browser decodes the app’s screen'); |
| @@ -199,6 +314,76 @@ try { |
| await wait(() => (sample?.stats.screenWatchdogKeyframes || 0) + (sample?.stats.screenWatchdogEncoderRecoveries || 0) > 0, |
| 'the watchdog says it recovered the encoder', 5); |
| console.log('PASS a stalled encoder was recovered without a re-share'); |
| + } else if (scenario === 'soak') { |
| + // §15.22's search for the cause: a long share with everything that |
| + // moves under it moving. What matters is not whether a freeze |
| + // happened but what it took to end it — the ladder's counts at each |
| + // freeze say whether keyframes were enough. |
| + const modes = ['balanced', 'motion', 'detail']; |
| + const uplinks = [0, 1500000, 600000, 0, 3000000, 900000]; |
| + const started = Date.now(); |
| + const notices = new Set(); |
| + let step = 0; |
| + const tick = setInterval(() => { |
| + step += 1; |
| + const mode = modes[step % modes.length]; |
| + const up = uplinks[step % uplinks.length]; |
| + native.stdin.write(`screen-mode ${mode}\n`); |
| + native.stdin.write(`uplink ${up}\n`); |
| + const st = sample?.stats || {}; |
| + console.log(`soak ${Math.round((Date.now() - started) / 1000)} s: mode ${mode}, uplink ${up || 'open'} (was sending ${st.screenSendWidth}x${st.screenSendHeight} at ${(st.screenSendFpsTenths || 0) / 10} fps, limitation ${st.screenLimitation})`); |
| + }, Number(process.env.SCREEN_SOAK_STEP_MS || 45000)); |
| + const noticeTick = setInterval(() => { if (sample?.screenRecovery) notices.add(sample.screenRecovery); }, 500); |
| + await watchFor(SOAK, 'soak'); |
| + clearInterval(tick); |
| + clearInterval(noticeTick); |
| + native.stdin.write('uplink 0\n'); |
| + const st = sample?.stats || {}; |
| + console.log(`SOAK app: ICE restarts ${st.iceRestarts}, stall reports ${st.screenStallReports}, keyframes ${st.screenWatchdogKeyframes}, fresh encoders ${st.screenWatchdogEncoderRecoveries}, re-adds ${st.screenWatchdogReAdds}, rebuilds ${st.screenWatchdogRebuilds}, shares again ${st.screenWatchdogReshares}, given up ${st.screenWatchdogFailures}, capture restarts ${st.screenWatchdogCaptureRestarts}`); |
| + console.log(`SOAK browser ICE restarts ${restarts}; notices at the app: ${JSON.stringify([...notices])}`); |
| + console.log('the browser\u2019s screen log:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| + if (asserting) { |
| + assert.equal(st.screenWatchdogFailures || 0, 0, 'the app gave up on the browser\u2019s view'); |
| + assert.equal(freezes.filter((f) => f.open).length, 0, 'the screen froze and never came back'); |
| + } |
| + } else if (scenario === 'wedge' || scenario === 'wedge-sticky') { |
| + await watchFor(5, 'before the wedge'); |
| + const sticky = scenario === 'wedge-sticky'; |
| + const keyframes = sample?.stats.screenWatchdogKeyframes || 0; |
| + const notices = new Set(); |
| + const note = () => { if (sample?.screenRecovery) notices.add(sample.screenRecovery); }; |
| + const wedgedAt = Date.now(); |
| + const n = await page.evaluate((s) => window.__wedgeScreens(s), sticky); |
| + assert.ok(n >= 1, 'the browser\u2019s screen receiver was wedged'); |
| + await wait(async () => (await page.evaluate(() => window.meet.state().screenStalled)).length === 1, |
| + 'the browser reports the app\u2019s screen stalled', 15); |
| + if (!sticky) { |
| + // Painted frames on the share tile, counted across a new tile. |
| + const progress = await progressOf(() => received().then((r) => r.frames)); |
| + await wait(async () => { note(); return (await progress()) > 15; }, |
| + 'the browser decodes the app\u2019s screen again, without a share again', 30); |
| + console.log(`the browser decoding again ${Date.now() - wedgedAt} ms after the wedge: rebuilds ${sample?.stats.screenWatchdogRebuilds}, shares again ${sample?.stats.screenWatchdogReshares}, keyframes ${(sample?.stats.screenWatchdogKeyframes || 0) - keyframes}`); |
| + assert.ok((sample?.stats.screenWatchdogKeyframes || 0) > keyframes, 'keyframes were tried first'); |
| + assert.ok((sample?.stats.screenWatchdogRebuilds || 0) > 0, 'the rebuild is what mended it'); |
| + await wait(async () => (await page.evaluate(() => window.meet.state().screenStalled)).length === 0, |
| + 'the browser clears its report', 20); |
| + await wait(() => { note(); return sample?.screenRecovery === ''; }, 'the app\u2019s notice clears', 20); |
| + console.log('notices seen at the app: ' + JSON.stringify([...notices])); |
| + assert.ok([...notices].some((x) => x.includes('recovering')), 'the notice said recovering'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + console.log('PASS a wedged browser receiver was restored by the app\u2019s rebuild'); |
| + } else { |
| + await wait(() => { note(); return (sample?.screenRecovery || '').includes('Couldn\u2019t restore'); }, |
| + 'the app says it could not restore the browser\u2019s view', 60); |
| + console.log('notices seen at the app: ' + JSON.stringify([...notices])); |
| + assert.ok((sample?.stats.screenWatchdogRebuilds || 0) > 0 && (sample?.stats.screenWatchdogReshares || 0) > 0, |
| + 'a rebuild and a share again were tried first'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + await page.evaluate(() => window.__releaseWedge()); |
| + await recovered('the browser decodes again once the wedge is lifted'); |
| + await wait(() => sample?.screenRecovery === '', 'the app\u2019s notice clears once the view is back', 30); |
| + console.log('PASS the app said it could not restore a view nothing could, and cleared when it came back'); |
| + } |
| } else if (scenario === 'loss') { |
| await watchFor(5, 'before loss'); |
| // The browser's watch treats the app's screen as frozen, as it would |
| @@ -231,8 +416,10 @@ try { |
| } |
| console.log(`PASS ${scenario}: the app’s screen kept arriving`); |
| } catch (err) { |
| - failed = true; |
| - console.log('FAIL', err.message); |
| + if (!(err instanceof Done)) { |
| + failed = true; |
| + console.log('FAIL', err.message); |
| + } |
| } finally { |
| if (native) { |
| native.stdin.write('leave\n'); |
| @@ -242,3 +429,128 @@ try { |
| await browser.close(); |
| if (failed) process.exit(1); |
| } |
| + |
| +// Done ends a scenario that has passed before the app-shares path below |
| +// it: thrown, and caught as not a failure. |
| +function Done() {} |
| + |
| +// progressOf counts how far a counter has moved since now, across a |
| +// restart from zero (a new tile, a new receiver): a fall is taken as a |
| +// new count starting. |
| +async function progressOf(read) { |
| + let last = await read(); |
| + let total = 0; |
| + return async () => { |
| + const now = await read(); |
| + total += now >= last ? now - last : now; |
| + last = now; |
| + return total; |
| + }; |
| +} |
| + |
| +// webShares is the web-* scenarios: the browser shares its screen (a fake |
| +// monitor), the app watches, the app's screen decoder is wedged, and the |
| +// browser's own ladder (§15.22) has to put it right — or, sticky, say it |
| +// cannot, and clear once the wedge is lifted. |
| +async function webShares(page, native, current, replies, wait, sticky) { |
| + const stat = (k) => current()?.stats[k] || 0; |
| + await page.evaluate(() => { window.__h264Screen = true; }); |
| + await page.hover('#share'); |
| + await page.click('#share'); |
| + await wait(() => stat('remoteScreens') === 1 && stat('screenFramesDecoded') > 15, 'the app decodes the browser\u2019s screen', 30); |
| + console.log('PASS the app decodes the browser\u2019s screen'); |
| + await new Promise((r) => setTimeout(r, 5000)); |
| + const s0 = () => page.evaluate(() => window.meet.state()); |
| + const keyframes = (await s0()).shareRecoveries.keyframes; |
| + const wedgedAt = Date.now(); |
| + native.stdin.write(sticky ? 'screen-decoder-wedge sticky\n' : 'screen-decoder-wedge\n'); |
| + await wait(() => replies.some((r) => (r.decoderWedged || 0) >= 1), 'the app\u2019s screen decoder is wedged', 10); |
| + await wait(() => stat('screenStalledHere') === 1, 'the app reports the browser\u2019s screen stalled (the wedge hit the screen)', 15); |
| + const notices = new Set(); |
| + const note = async () => { |
| + const s = await s0(); |
| + if (s.shareRecovery) notices.add(s.shareRecovery); |
| + return s; |
| + }; |
| + if (!sticky) { |
| + const base = stat('screenFramesDecodedTotal'); |
| + await wait(async () => { await note(); return stat('screenFramesDecodedTotal') > base + 15; }, |
| + 'the app decodes the browser\u2019s screen again, without a share again', 30); |
| + const s = await note(); |
| + console.log(`the app decoding again ${Date.now() - wedgedAt} ms after the wedge: ${JSON.stringify(s.shareRecoveries)}`); |
| + assert.ok(s.shareRecoveries.keyframes > keyframes, 'keyframes were tried first'); |
| + assert.ok(s.shareRecoveries.rebuilds > 0, 'the rebuild is what mended it'); |
| + await wait(() => stat('screenStalledHere') === 0, 'the app clears its report', 20); |
| + await wait(async () => (await note()).shareRecovery === '', 'the browser\u2019s notice clears', 20); |
| + console.log('notices seen at the browser: ' + JSON.stringify([...notices])); |
| + assert.ok([...notices].some((x) => x.includes('recovering')), 'the notice said recovering'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + } else { |
| + await wait(async () => (await note()).shareRecovery.includes('Couldn\u2019t restore'), |
| + 'the browser says it could not restore the app\u2019s view', 60); |
| + const s = await note(); |
| + console.log('notices seen at the browser: ' + JSON.stringify([...notices]) + ' ' + JSON.stringify(s.shareRecoveries)); |
| + assert.ok(s.shareRecoveries.rebuilds > 0 && s.shareRecoveries.reshares > 0, 'a rebuild and a share again were tried first'); |
| + assert.ok([...notices].some((x) => x.includes('Restarting your share for')), 'the notice said restarting'); |
| + native.stdin.write('screen-decoder-release\n'); |
| + const base = stat('screenFramesDecodedTotal'); |
| + await wait(() => stat('screenFramesDecodedTotal') > base + 10, 'the app decodes again once the wedge is lifted', 30); |
| + await wait(async () => (await note()).shareRecovery === '', 'the browser\u2019s notice clears once the view is back', 30); |
| + } |
| + console.log('the browser\u2019s diagnostics:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| +} |
| + |
| +// webSoak is `soak` with the browser sharing and the app watching: the |
| +// app's received screen, counted across rebuilds, watched for freezes |
| +// while the renewals run and the browser's mode is cycled. |
| +async function webSoak(page, native, current, wait) { |
| + const stat = (k) => current()?.stats[k] || 0; |
| + await page.hover('#share'); |
| + await page.click('#share'); |
| + await wait(() => stat('remoteScreens') === 1 && stat('screenFramesDecoded') > 15, 'the app decodes the browser\u2019s screen', 30); |
| + const modes = ['balanced', 'motion', 'detail']; |
| + const started = Date.now(); |
| + let step = 0; |
| + const notices = new Set(); |
| + const freezes = []; |
| + let last = stat('screenFramesDecodedTotal'); |
| + let moved = Date.now(); |
| + let frozen = null; |
| + let nextStep = Date.now() + Number(process.env.SCREEN_SOAK_STEP_MS || 45000); |
| + while (Date.now() - started < SOAK * 1000) { |
| + await new Promise((r) => setTimeout(r, 1000)); |
| + const s = await page.evaluate(() => window.meet.state()); |
| + if (s.shareRecovery) notices.add(s.shareRecovery); |
| + if (Date.now() >= nextStep) { |
| + step += 1; |
| + const mode = modes[step % modes.length]; |
| + await page.evaluate((m) => { |
| + const el = document.getElementById('pick-share-mode'); |
| + if (el) { el.value = m; el.dispatchEvent(new Event('change', { bubbles: true })); } |
| + }, mode); |
| + console.log(`web-soak ${Math.round((Date.now() - started) / 1000)} s: the browser\u2019s mode ${mode}`); |
| + nextStep = Date.now() + Number(process.env.SCREEN_SOAK_STEP_MS || 45000); |
| + } |
| + const now = stat('screenFramesDecodedTotal'); |
| + if (now > last) { |
| + if (frozen) { |
| + frozen.ms = Date.now() - frozen.since; |
| + freezes.push(frozen); |
| + console.log(`FREEZE web-soak: ${frozen.ms} ms, then frames again; browser ladder ${JSON.stringify(s.shareRecoveries)}`); |
| + frozen = null; |
| + } |
| + moved = Date.now(); |
| + } else if (!frozen && Date.now() - moved > FREEZE_MS) { |
| + frozen = { since: moved }; |
| + console.log(`FROZEN web-soak: no screen frame at the app for ${Date.now() - moved} ms; app stalled-here ${stat('screenStalledHere')}; browser ladder ${JSON.stringify(s.shareRecoveries)}`); |
| + } |
| + last = now; |
| + } |
| + const s = await page.evaluate(() => window.meet.state()); |
| + console.log(`SUMMARY web-soak: ${freezes.length} freezes, longest ${Math.max(0, ...freezes.map((f) => f.ms))} ms, open ${!!frozen}; app ICE restarts ${stat('iceRestarts')}, pictures attached again ${stat('screenReattaches')}; browser ladder ${JSON.stringify(s.shareRecoveries)}; notices ${JSON.stringify([...notices])}`); |
| + console.log('the browser\u2019s screen log:\n' + (await page.evaluate(() => window.meet.diagnostics())).split('## Screen')[1]); |
| + if (asserting) { |
| + assert.equal(s.shareRecoveries.failures, 0, 'the browser gave up on the app\u2019s view'); |
| + assert.ok(!frozen, 'the app\u2019s screen froze and never came back'); |
| + } |
| +} |
diff --git a/test/browser/room-screen.test.mjs b/test/browser/room-screen.test.mjs| index 652228d..1aa5403 100644 |
| --- a/test/browser/room-screen.test.mjs |
| +++ b/test/browser/room-screen.test.mjs |
| @@ -7,8 +7,12 @@ |
| // does, with the constants they read. |
| // |
| // judgeScreen the receiver's verdict on one screen: stalled after |
| -// four seconds without a new frame, for a minute at most |
| -// recoveringWords the sharer's quiet notice |
| +// four seconds without a new frame, for three minutes at |
| +// most; any change in the count is movement |
| +// reattachDue the receiver's own mending, once per stall (§15.22) |
| +// escalateShare the sharer's ladder when keyframes do not answer |
| +// recoveringWords, noticeWords the sharer's notice |
| +// readingMoved what a stalled screen's statistics say |
| // mediaFrom, mediaKey, mediaPayload the report's screenStalled list |
| import { test } from 'node:test'; |
| import assert from 'node:assert/strict'; |
| @@ -37,13 +41,21 @@ function liftConst(name) { |
| assert.ok(m, name + ' is not a numeric const in room.js any more'); |
| return 'const ' + name + ' = ' + m[1] + ';'; |
| } |
| -const CONSTS = ['SCREEN_STALL_AFTER', 'SCREEN_STALL_GIVE_UP'].map(liftConst).join('\n'); |
| +const CONSTS = [ |
| + 'SCREEN_STALL_AFTER', 'SCREEN_STALL_GIVE_UP', 'SCREEN_REATTACH_AFTER', 'SCREEN_REBUILD_AFTER', |
| + 'SCREEN_RESHARE_AFTER', 'SCREEN_GIVE_UP_AFTER', 'SCREEN_REBUILD_EVERY', |
| +].map(liftConst).join('\n'); |
| const build = (name) => |
| // eslint-disable-next-line no-new-func |
| new Function(CONSTS + '\n' + lift(src, name) + '; return ' + name + ';')(); |
| const judgeScreen = build('judgeScreen'); |
| const recoveringWords = build('recoveringWords'); |
| +const reattachDue = build('reattachDue'); |
| +const escalateShare = build('escalateShare'); |
| +const readingMoved = new Function(lift(src, 'readingSummary') + '\n' + lift(src, 'readingMoved') + '; return readingMoved;')(); |
| +// eslint-disable-next-line no-new-func |
| +const noticeWords = new Function(lift(src, 'recoveringWords') + '\n' + lift(src, 'noticeWords') + '; return noticeWords;')(); |
| const mediaFrom = build('mediaFrom'); |
| const mediaPayload = build('mediaPayload'); |
| // eslint-disable-next-line no-new-func |
| @@ -51,15 +63,108 @@ const mediaKey = new Function(lift(src, 'mediaKey') + '; return mediaKey;')(); |
| const ID = '0123456789abcdef0123456789abcdef'; |
| -test('a screen is stalled four seconds after its last frame, and for a minute at most', () => { |
| +test('a screen is stalled four seconds after its last frame, and for three minutes at most', () => { |
| const seen = { decoded: 0, movedAt: 0 }; |
| const stalledAt = []; |
| - for (let t = 0; t <= 80; t++) { |
| + for (let t = 0; t <= 200; t++) { |
| const decoded = t < 10 ? t * 10 : 90; |
| if (judgeScreen(seen, decoded, t * 1000, false)) stalledAt.push(t); |
| } |
| assert.equal(stalledAt[0], 13); |
| - assert.equal(stalledAt[stalledAt.length - 1], 68); |
| + assert.equal(stalledAt[stalledAt.length - 1], 188); |
| +}); |
| + |
| +test('a rebuilt screen whose count starts again is moving, not stalled', () => { |
| + const seen = { decoded: 0, movedAt: 0 }; |
| + for (let t = 0; t < 10; t++) judgeScreen(seen, 900 + t * 30, t * 1000, false); |
| + for (let t = 10; t < 16; t++) judgeScreen(seen, 1170, t * 1000, false); |
| + assert.equal(judgeScreen(seen, 1170, 16000, false), true); |
| + for (let t = 17; t < 40; t++) { |
| + assert.equal(judgeScreen(seen, (t - 17) * 30, t * 1000, false), false, 't=' + t); |
| + } |
| +}); |
| + |
| +test('a stalled picture is handed to its tile again once, four seconds into the stall', () => { |
| + const seen = { decoded: 0, movedAt: 0 }; |
| + const due = []; |
| + for (let t = 0; t < 60; t++) { |
| + const decoded = t < 10 ? t * 10 : 90; |
| + judgeScreen(seen, decoded, t * 1000, false); |
| + if (reattachDue(seen, t * 1000)) due.push(t); |
| + } |
| + assert.deepEqual(due, [17], 'last moved at 9, stalled from 13, handed back at 17, once'); |
| + judgeScreen(seen, 91, 60000, false); |
| + const again = []; |
| + for (let t = 61; t < 80; t++) { |
| + judgeScreen(seen, 91, t * 1000, false); |
| + if (reattachDue(seen, t * 1000)) again.push(t); |
| + } |
| + assert.deepEqual(again, [68], 'a new stall is a new try'); |
| +}); |
| + |
| +// The ladder, driven once a second the way answerScreenStalls drives it. |
| +function ladder(from, to, reported, opts = {}) { |
| + const rec = opts.rec || { stage: '', at: 0, lastRebuild: 0 }; |
| + let resharedAt = opts.resharedAt || 0; |
| + const out = []; |
| + for (let t = from; t < to; t++) { |
| + const now = t * 1000; |
| + if (!reported(t)) { |
| + rec.stage = ''; |
| + continue; |
| + } |
| + const owed = opts.owed ? opts.owed(t) : true; |
| + const free = !resharedAt || now - resharedAt >= 300000; |
| + const action = escalateShare(rec, now, owed, free); |
| + if (action === 'reshare') resharedAt = now; |
| + if (action) out.push([t, action]); |
| + } |
| + return { out, rec, resharedAt }; |
| +} |
| + |
| +test('a report keyframes do not answer is rebuilt, then shared again, then given up', () => { |
| + const { out, rec } = ladder(0, 45, (t) => t >= 3); |
| + assert.deepEqual(out, [[9, 'rebuild'], [19, 'reshare'], [29, 'failed']]); |
| + assert.equal(rec.stage, 'failed'); |
| +}); |
| + |
| +test('a rebuild that works ends the ladder', () => { |
| + const { out, rec } = ladder(0, 60, (t) => t >= 3 && t < 12); |
| + assert.deepEqual(out, [[9, 'rebuild']]); |
| + assert.equal(rec.stage, ''); |
| +}); |
| + |
| +test('rebuilds and shares again are rationed', () => { |
| + const first = ladder(0, 20, (t) => t >= 3 && t < 12); |
| + const second = ladder(20, 50, (t) => t >= 25 && t < 34, { rec: first.rec, resharedAt: first.resharedAt }); |
| + assert.deepEqual(second.out, [[31, 'reshare']], 'no second rebuild inside a minute'); |
| + const third = ladder(80, 120, (t) => t >= 85, { rec: second.rec, resharedAt: second.resharedAt }); |
| + assert.deepEqual(third.out, [[91, 'rebuild'], [101, 'failed']], 'and no second share again inside five'); |
| +}); |
| + |
| +test('nothing sent is nothing to escalate', () => { |
| + const { out, rec } = ladder(0, 60, (t) => t >= 3, { owed: () => false }); |
| + assert.deepEqual(out, []); |
| + assert.equal(rec.stage, 'keyframes'); |
| +}); |
| + |
| +test('the notice moves on in words, the app\'s own', () => { |
| + assert.equal(noticeWords([], [], []), ''); |
| + assert.equal(noticeWords(['Samuel'], [], []), 'Samuel’s view of your screen stalled — recovering'); |
| + assert.equal(noticeWords([], ['Samuel'], []), 'Restarting your share for Samuel…'); |
| + assert.equal(noticeWords([], [], ['Samuel']), 'Couldn’t restore Samuel’s view — try Stop and Share again'); |
| + assert.equal(noticeWords(['Tony'], [], ['Samuel']), |
| + 'Couldn’t restore Samuel’s view — try Stop and Share again. Tony’s view of your screen stalled — recovering'); |
| + assert.equal(noticeWords([], ['A', 'B', 'C'], []), 'Restarting your share for 3 people…'); |
| +}); |
| + |
| +test('a stalled screen\'s statistics name what stopped', () => { |
| + const base = { mid: '3', packets: 1000, lost: 0, received: 300, decoded: 300, dropped: 0, keyframes: 2, pli: 0, fir: 0, |
| + nack: 0, freezes: 0, width: 1920, height: 1080, decoder: 'ExternalDecoder', muted: false }; |
| + const wedged = Object.assign({}, base, { packets: 1400, received: 420, dropped: 110 }); |
| + assert.match(readingMoved(wedged, base), /^packets \+400, frames received \+120, decoded \+0, dropped \+110.* — frames arriving, none decoded \(the decoder\)$/); |
| + assert.match(readingMoved(base, base), /no packets arriving/); |
| + assert.match(readingMoved(Object.assign({}, base, { mid: '5' }), base), /no earlier reading$/); |
| }); |
| test('a moving screen is never stalled, and a frozen test window is', () => { |