Share a screen from the Mac app
The browser has shared screens since slice 4. The Mac app now does too,
and sends one exactly as the browser does, because the browser is who
receives most of them: one sendonly transceiver per connection, in a
stream of its own, added last so it takes an m-line of its own after the
camera's, and announced with `share {on:true}` once it is on every
connection. Stopping stops the transceiver and renegotiates, so the far
side's track ends rather than freezing on its last frame, and then says
`share {on:false}`.
The framework's transceiver stop is the immediate kind: it stops, and
unlike the browser's stop() it does not raise negotiation-needed. So the
offer that tells the far side is asked for by hand, and owed until the
connection is next stable if it is mid-exchange when asked. A screen
transceiver is only ever added to a stable connection, and the deferred
ones are added when it settles, for the reason the browser's mutualshare
gate found: added mid-negotiation, it is unassociated when the far
side's offer arrives and gets handed an m-line meant for something else.
The picker is SCContentSharingPicker, which is macOS 14 and the app's
floor. What a person picks there is content they have agreed to share,
so the app asks for no Screen Recording permission, needs no Info.plist
key (there is no usage string for screen capture), sends nobody to
System Settings, and keeps working in the sandbox the App Store build
will need. Calls' own windows are kept out of the picker, and out of a
shared display too on macOS 14.4 and later: the display is rebuilt as
the same display excluding this application, found through the current
process's own shareable content — the one query that needs no
permission. Excluding the application rather than a list of windows
keeps out the ones that open later, the floating mini window on
feat/mac-mini-window among them. Before 14.4 the display is shared as
picked, and says so.
The menu bar's stop control and a shared window closing end the share
through the same path as the button. The source is marked a screencast,
so WebRTC gives up frame rate before resolution; frames are capped at
1920 on the long side and thirty a second.
The native gate now shares both ways against a real browser: the app
shares a synthetic screen, the browser must caption it with the app's
name and decode it beside the app's camera, and then see its received
track *end* — once by the button and once by the capture ending on its
own. Its timeout goes from three minutes to five, the interop's lifetime
and the test's context with it; chat moved ahead of the screen
scenarios, because it is not replayed across the ten-second renewals
and a minute of screens put it among them.
`make native-pair` is new: two apps in one call and no browser, with a
screen shared before the other app joined, both endings, and both apps
sharing at once. It is not in native-browser because two apps on one Mac
do not reliably connect at all, before anything is shared and with
main's binary as well as this one's: about two runs in five fail with
signalling complete and in order, every candidate added on both sides,
and no connectivity check answered. The macOS firewall is on here and
the interop binary is unsigned, which fits; allowing it through needs an
administrator, so that is a reading and not a proof. Every run that
connected passed.
Gates: swift test (41), make native-mac, the iPhone simulator build,
make native-browser (all three tests pass), make native-pair (3 of 4
full passes, the fourth the connection failure above).
Not verified: the real picker, a real display or window, the menu bar's
stop control, and the exclusion of Calls' windows from a real display —
none of which a headless gate can reach.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
13 files changed,
+893
−15
Makefile+8 −2internal/callstest/native_browser_test.go+216 −1native/README.md+28 −0native/apple/Sources/CallsInterop/Interop.swift+26 −1native/apple/Sources/CallsInterop/SyntheticVideo.swift+60 −0native/apple/Sources/CallsKit/Call.swift+55 −0native/apple/Sources/CallsKit/Capture.swift+14 −0native/apple/Sources/CallsKit/MacCallView.swift+23 −0native/apple/Sources/CallsKit/MacScreenCapture.swift+230 −0native/apple/Sources/CallsKit/Media.swift+132 −2native/apple/Sources/CallsKit/ScreenShare.swift+11 −0native/apple/Tests/CallsKitTests/ScreenShareTests.swift+17 −0test/browser/native.mjs+73 −9
diff --git a/Makefile b/Makefile| index a15a497..7948a82 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -156,7 +156,7 @@ browser: |
| go test -tags browser -count=1 -timeout 20m -run TestCallInARealBrowser -v ./internal/callstest/ |
| # Native builds isolate gomobile's cgo bridge from the cgo-free server. |
| -.PHONY: native-mac native-test native-browser |
| +.PHONY: native-mac native-test native-browser native-pair |
| native-mac: |
| ./native/scripts/build-mac.sh |
| @@ -166,7 +166,13 @@ native-test: |
| CGO_ENABLED=1 go test -race ./callscore |
| native-browser: native-mac |
| - CGO_ENABLED=0 go test -tags browser -count=1 -timeout 3m -run 'TestNative(MacCallsBrowser|MacChat|BrowserAuthentication)' -v ./internal/callstest/ |
| + CGO_ENABLED=0 go test -tags browser -count=1 -timeout 5m -run 'TestNative(MacCallsBrowser|MacChat|BrowserAuthentication)' -v ./internal/callstest/ |
| + |
| +# Two apps in one call, sharing screens to each other. Separate from |
| +# native-browser because two unsigned apps on one Mac do not reliably |
| +# connect, which looks like the macOS firewall: see TestNativeMacScreenShare. |
| +native-pair: native-mac |
| + CGO_ENABLED=0 go test -tags browser -count=1 -timeout 3m -run 'TestNativeMacScreenShare' -v ./internal/callstest/ |
| .PHONY: iphone iphone-device iphone-browser |
| iphone: |
diff --git a/internal/callstest/native_browser_test.go b/internal/callstest/native_browser_test.go| index 51d90e9..53da5c7 100644 |
| --- a/internal/callstest/native_browser_test.go |
| +++ b/internal/callstest/native_browser_test.go |
| @@ -6,10 +6,12 @@ import ( |
| "bufio" |
| "context" |
| "encoding/json" |
| + "io" |
| "net/http" |
| "os" |
| "os/exec" |
| "path/filepath" |
| + "sync" |
| "testing" |
| "time" |
| ) |
| @@ -69,7 +71,7 @@ func nativeMacCallsBrowser(t *testing.T, guest, chat bool) { |
| if _, err := os.Stat(binary); err != nil { |
| t.Fatal("build native interop executable with make native-mac first") |
| } |
| - ctx, cancel := context.WithTimeout(t.Context(), 100*time.Second) |
| + ctx, cancel := context.WithTimeout(t.Context(), 160*time.Second) |
| defer cancel() |
| cmd := exec.CommandContext(ctx, "node", filepath.Join(root, "test/browser/native.mjs"), rig.origin, room, binary, mode) |
| cookies, _ := json.Marshal([]map[string]string{{"name": rig.cookieName, "value": rig.cookieB}}) |
| @@ -95,3 +97,216 @@ func nativeMacCallsBrowser(t *testing.T, guest, chat bool) { |
| t.Fatal(err) |
| } |
| } |
| + |
| +// TestNativeMacScreenShare is §5.4 between two apps, with no browser in |
| +// the call: each side is the shipping Go bridge and WebRTC build driving a |
| +// synthetic camera and a synthetic screen. The browser gate above holds |
| +// the app to the browser's rules in both directions; this holds two apps |
| +// to each other, which is the pairing nothing else would catch drifting — |
| +// and it shares from both sides at once, which is the negotiation |
| +// collision the browser's mutualshare scenario was written for. |
| +// |
| +// It is `make native-pair`, not part of `make native-browser`, because |
| +// two apps on one Mac do not reliably connect at all, screens or no |
| +// screens — measured with main's interop binary as well as this branch's, |
| +// before anything was shared. About two runs in five fail with signalling |
| +// complete and correctly ordered, every candidate added on both sides, and |
| +// no connectivity check ever answered. What the evidence points at is the |
| +// Mac rather than the code: the macOS application firewall was on, it |
| +// admits signed software automatically, and the interop binary is |
| +// unsigned, so inbound UDP reaches it only as a reply to its own outbound |
| +// on the same address pair — which two such binaries achieve only when |
| +// their first checks happen to cross in the right order. Chromium is |
| +// signed, so the browser gates never meet it, and a Developer ID Calls.app |
| +// would not either. Not proved: allowing the binary through the firewall |
| +// needs an administrator, and was not done. |
| +func TestNativeMacScreenShare(t *testing.T) { |
| + rig := newBrowserRig(t) |
| + room, _ := rig.createMeeting(t) |
| + root, err := filepath.Abs("../..") |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + binary := filepath.Join(root, "native/apple/.build/debug/CallsInterop") |
| + if _, err := os.Stat(binary); err != nil { |
| + t.Fatal("build native interop executable with make native-mac first") |
| + } |
| + ctx, cancel := context.WithTimeout(t.Context(), 140*time.Second) |
| + defer cancel() |
| + // The person behind cookie A is paul@example.com, and the hub names |
| + // a person by the local part of their address. |
| + const paulsScreen = "paul\u2019s screen" |
| + |
| + // The first app shares before the second has joined, so the `share` |
| + // message goes to nobody and is never replayed: the second app has |
| + // only the m-line to go on, and has to caption the screen anyway. |
| + a := startNative(ctx, t, binary, rig, room, rig.cookieA, "a") |
| + defer a.leave(t) |
| + waitNative(t, "the first app is in the meeting", func() bool { return a.sample().Phase == "admitted" }, a) |
| + a.command(t, "screen-on") |
| + waitNative(t, "the first app is sharing to an empty room", func() bool { |
| + s := a.sample() |
| + return s.SharingScreen != nil && *s.SharingScreen |
| + }, a) |
| + b := startNative(ctx, t, binary, rig, room, rig.cookieB, "b") |
| + defer b.leave(t) |
| + waitNative(t, "both apps decode each other's camera", func() bool { |
| + return a.stat("framesDecoded") > 15 && b.stat("framesDecoded") > 15 |
| + }, a, b) |
| + waitNative(t, "the second app captions the first app's screen", func() bool { |
| + c := b.sample().ScreenCaptions |
| + return len(c) == 1 && c[0] == paulsScreen && b.stat("remoteScreens") == 1 |
| + }, a, b) |
| + base, camera := b.stat("screenFramesDecoded"), b.stat("framesDecoded") |
| + waitNative(t, "the second app decodes the screen and the camera beside it", func() bool { |
| + return b.stat("screenFramesDecoded") > base+15 && b.stat("framesDecoded") > camera+15 |
| + }, a, b) |
| + if b.stat("remoteTracks") != 1 { |
| + t.Fatalf("the screen was taken for a second camera: %+v", b.sample()) |
| + } |
| + a.command(t, "screen-off") |
| + waitNative(t, "the screen goes when the first app stops", func() bool { |
| + return len(b.sample().ScreenCaptions) == 0 && b.stat("remoteScreens") == 0 |
| + }, a, b) |
| + camera = b.stat("framesDecoded") |
| + waitNative(t, "the camera survives the screen stopping", func() bool { |
| + return b.stat("framesDecoded") > camera+15 |
| + }, a, b) |
| + t.Log("PASS app to app: a screen shared before the other app joined is captioned, decoded and stopped by the button") |
| + |
| + // The other way, ended by the capture itself. |
| + b.command(t, "screen-on") |
| + waitNative(t, "the first app shows the second app's screen", func() bool { |
| + return len(a.sample().ScreenCaptions) == 1 && a.stat("remoteScreens") == 1 |
| + }, a, b) |
| + base = a.stat("screenFramesDecoded") |
| + waitNative(t, "the first app decodes it", func() bool { return a.stat("screenFramesDecoded") > base+15 }, a, b) |
| + b.command(t, "screen-end") |
| + waitNative(t, "a capture ending on its own ends the share", func() bool { |
| + s := b.sample() |
| + return s.SharingScreen != nil && !*s.SharingScreen && len(a.sample().ScreenCaptions) == 0 && a.stat("remoteScreens") == 0 |
| + }, a, b) |
| + t.Log("PASS app to app the other way, ended by the capture") |
| + |
| + // Both at once: two transceivers added in the same moment, and a |
| + // collision for perfect negotiation to settle without either screen |
| + // landing on the m-line the other side built for something else. |
| + a.command(t, "screen-on") |
| + b.command(t, "screen-on") |
| + waitNative(t, "each app shows the other's screen", func() bool { |
| + return a.stat("remoteScreens") == 1 && b.stat("remoteScreens") == 1 |
| + }, a, b) |
| + baseA, baseB := a.stat("screenFramesDecoded"), b.stat("screenFramesDecoded") |
| + camA, camB := a.stat("framesDecoded"), b.stat("framesDecoded") |
| + waitNative(t, "both screens and both cameras decode through the collision", func() bool { |
| + return a.stat("screenFramesDecoded") > baseA+15 && b.stat("screenFramesDecoded") > baseB+15 && |
| + a.stat("framesDecoded") > camA+15 && b.stat("framesDecoded") > camB+15 |
| + }, a, b) |
| + if a.stat("remoteTracks") != 1 || b.stat("remoteTracks") != 1 { |
| + t.Fatalf("a screen was taken for a camera: a=%+v b=%+v", a.sample(), b.sample()) |
| + } |
| + a.command(t, "screen-off") |
| + b.command(t, "screen-off") |
| + waitNative(t, "both screens go", func() bool { |
| + return a.stat("remoteScreens") == 0 && b.stat("remoteScreens") == 0 && |
| + len(a.sample().ScreenCaptions) == 0 && len(b.sample().ScreenCaptions) == 0 |
| + }, a, b) |
| + t.Log("PASS app to app, both sharing at once") |
| +} |
| + |
| +type nativeSample struct { |
| + Phase string `json:"phase"` |
| + Peer string `json:"peer"` |
| + Stats map[string]uint64 `json:"stats"` |
| + ScreenCaptions []string `json:"screenCaptions"` |
| + SharingScreen *bool `json:"sharingScreen"` |
| +} |
| + |
| +// nativeApp is one CallsInterop process and the last observation it |
| +// printed. Only numeric samples are kept; nothing else it prints is read. |
| +type nativeApp struct { |
| + label string |
| + cmd *exec.Cmd |
| + stdin io.WriteCloser |
| + mu sync.Mutex |
| + last nativeSample |
| + done chan struct{} |
| +} |
| + |
| +func startNative(ctx context.Context, t *testing.T, binary string, rig *browserRig, room, cookie, label string) *nativeApp { |
| + t.Helper() |
| + cookies, _ := json.Marshal([]map[string]string{{"name": rig.cookieName, "value": cookie}}) |
| + input, _ := json.Marshal(map[string]string{"origin": rig.origin, "room": room, "cookies": string(cookies)}) |
| + cmd := exec.CommandContext(ctx, binary) |
| + stdin, err := cmd.StdinPipe() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + out, err := cmd.StdoutPipe() |
| + if err != nil { |
| + t.Fatal(err) |
| + } |
| + if err := cmd.Start(); err != nil { |
| + t.Fatal(err) |
| + } |
| + app := &nativeApp{label: label, cmd: cmd, stdin: stdin, done: make(chan struct{})} |
| + go func() { |
| + defer close(app.done) |
| + scanner := bufio.NewScanner(out) |
| + for scanner.Scan() { |
| + var s nativeSample |
| + if json.Unmarshal(scanner.Bytes(), &s) != nil || s.Stats == nil { |
| + continue |
| + } |
| + app.mu.Lock() |
| + app.last = s |
| + app.mu.Unlock() |
| + } |
| + }() |
| + if _, err := io.WriteString(stdin, string(input)+"\n"); err != nil { |
| + t.Fatal(err) |
| + } |
| + return app |
| +} |
| + |
| +func (n *nativeApp) sample() nativeSample { |
| + n.mu.Lock() |
| + defer n.mu.Unlock() |
| + return n.last |
| +} |
| + |
| +func (n *nativeApp) stat(name string) uint64 { return n.sample().Stats[name] } |
| + |
| +func (n *nativeApp) command(t *testing.T, line string) { |
| + t.Helper() |
| + if _, err := io.WriteString(n.stdin, line+"\n"); err != nil { |
| + t.Fatalf("%s: %s: %v", n.label, line, err) |
| + } |
| +} |
| + |
| +func (n *nativeApp) leave(t *testing.T) { |
| + _, _ = io.WriteString(n.stdin, "leave\n") |
| + select { |
| + case <-n.done: |
| + case <-time.After(5 * time.Second): |
| + _ = n.cmd.Process.Kill() |
| + } |
| + _ = n.cmd.Wait() |
| +} |
| + |
| +// waitNative polls until ok holds, and on failure says what each app |
| +// last reported, which is the only account of the failure there is. |
| +func waitNative(t *testing.T, label string, ok func() bool, apps ...*nativeApp) { |
| + t.Helper() |
| + until := time.Now().Add(35 * time.Second) |
| + for time.Now().Before(until) { |
| + if ok() { |
| + return |
| + } |
| + time.Sleep(250 * time.Millisecond) |
| + } |
| + for _, app := range apps { |
| + t.Logf("%s: %+v", app.label, app.sample()) |
| + } |
| + t.Fatal(label) |
| +} |
diff --git a/native/README.md b/native/README.md| index fb18de2..dbe913e 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -78,6 +78,22 @@ message, or the stopped transceiver taking its receiver away. Each was |
| checked alone against a browser when this was written (the other one |
| disabled); the native gate checks them together. |
| +**Share your screen** on the Mac opens the system's own picker |
| +(`SCContentSharingPicker`, macOS 14) for a display or a window. What a person |
| +picks there is what they have agreed to share, so Calls asks for no Screen |
| +Recording permission, sends nobody to System Settings, and needs no Info.plist |
| +key — macOS has no usage string for screen capture — and the same path works in |
| +the App Sandbox. Calls' own windows are left out of the picker, and from a shared |
| +display too on macOS 14.4 and later, where the display is rebuilt excluding the |
| +app so that windows opening later (the floating mini window among them) stay |
| +out as well. The screen goes to every connection on its own `sendonly` |
| +transceiver in its own stream, exactly as the browser sends one, announced with |
| +`share {on:true}`; stopping stops the transceiver and renegotiates, so the far |
| +side's track ends rather than freezing, and says `share {on:false}`. The menu |
| +bar's stop control and a shared window closing end it the same way as the |
| +button. The source is a screencast, so WebRTC gives up frame rate before |
| +resolution, and frames are capped at 1920 on the long side and 30 a second. |
| + |
| **Knocks** are announced when someone asks to be let in while the app is |
| not frontmost. The admission queue is already on screen for a member who is |
| looking at the call; this is for the member who is in another Space, |
| @@ -152,6 +168,7 @@ make race |
| CGO_ENABLED=0 make browser |
| make native-test |
| make native-browser |
| +make native-pair |
| CGO_ENABLED=0 go test -tags browser -run TestNativeBrowserAuthentication ./internal/callstest |
| ``` |
| @@ -165,6 +182,17 @@ counters, not audible output. It never needs production credentials or hardware |
| camera access. The Go Home test separately exercises a session minted through |
| Home assertions and explicit identity linking, including resume and meeting end. |
| +The native/browser gate also shares screens both ways: the browser shares (a |
| +fake monitor, `--auto-accept-this-tab-capture`) and the app must caption and |
| +decode it beside the camera, then drop it when it stops; the app shares a |
| +synthetic screen and the browser must show it under the app's name, then see its |
| +received track *end* — once by the button and once by the capture ending on its |
| +own. `make native-pair` puts two apps in one call with no browser: a screen |
| +shared before the other app joined, both endings, and both apps sharing at once. |
| +It is separate because two unsigned apps on one Mac do not reliably connect at |
| +all (see `TestNativeMacScreenShare`), which looks like the machine rather than |
| +the feature. |
| + |
| The native chat gate sends a sequence of browser messages and renders the shared |
| native chat view to `native/build/chat-incoming.png` for visual inspection of the |
| latest message. The window stays off-screen. Native chat follows new messages |
diff --git a/native/apple/Sources/CallsInterop/Interop.swift b/native/apple/Sources/CallsInterop/Interop.swift| index 84a3e92..8fc447f 100644 |
| --- a/native/apple/Sources/CallsInterop/Interop.swift |
| +++ b/native/apple/Sources/CallsInterop/Interop.swift |
| @@ -13,6 +13,7 @@ struct Input: Decodable { |
| let invitation: String? |
| } |
| @main struct Interop { |
| + @MainActor static var screen: SyntheticScreen? |
| @MainActor static func main() async { |
| if CommandLine.arguments.count == 3, CommandLine.arguments[1] == "--lobby-snapshots" { |
| do { try await LobbySnapshots.save(to: CommandLine.arguments[2]) } catch { exit(1) } |
| @@ -64,6 +65,27 @@ struct Input: Decodable { |
| print("{\"camera\":\"\(now)\",\"cameraRefused\":\(note.isEmpty ? "false" : "true")}") |
| fflush(stdout) |
| } |
| + // §5.4, sending. `screen-end` is the capture ending on its own, |
| + // which is the path the system's stop control takes. |
| + if command == "screen-on" { |
| + let screen = await SyntheticScreen() |
| + await MainActor.run { Interop.screen = screen } |
| + await call.shareScreen(screen) |
| + let on = await call.sharingScreen |
| + print("{\"screenSharing\":\(on)}") |
| + fflush(stdout) |
| + } |
| + if command == "screen-off" { |
| + await call.stopSharingScreen() |
| + print("{\"screenSharing\":false}") |
| + fflush(stdout) |
| + } |
| + if command == "screen-end" { |
| + await Interop.screen?.end() |
| + let on = await call.sharingScreen |
| + print("{\"screenSharing\":\(on)}") |
| + fflush(stdout) |
| + } |
| if command == "share-snapshot" { |
| let saved = await preview?.saveInvitation(call: call) ?? false |
| print("{\"invitationSnapshotSaved\":\(saved)}") |
| @@ -71,7 +93,9 @@ struct Input: Decodable { |
| } |
| } |
| } |
| - for _ in 0..<90 { |
| + // Long enough for the member scenario, screens and all, with room to |
| + // spare: the gate always says `leave` long before this runs out. |
| + for _ in 0..<150 { |
| try? await Task.sleep(nanoseconds: 1_000_000_000) |
| let stats = await call.sampleStats() |
| if let data = try? JSONSerialization.data(withJSONObject: [ |
| @@ -83,6 +107,7 @@ struct Input: Decodable { |
| // they started. The words, not a count: the caption is the only |
| // place a screen says whose it is. |
| "screenCaptions": call.screens.map { screenCaption(call.name(of: $0)) }, |
| + "sharingScreen": call.sharingScreen, |
| ]), let text = String(data: data, encoding: .utf8) { |
| print(text) |
| fflush(stdout) |
diff --git a/native/apple/Sources/CallsInterop/SyntheticVideo.swift b/native/apple/Sources/CallsInterop/SyntheticVideo.swift| index 2eba113..0cdead0 100644 |
| --- a/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| +++ b/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| @@ -70,3 +70,63 @@ import WebRTC |
| timeStampNs: Int64(ProcessInfo.processInfo.systemUptime * 1_000_000_000))) |
| } |
| } |
| + |
| +/// A screen with nothing on it but frames, for the gates: there is no |
| +/// display to share in CI and no person to answer a picker. What it can |
| +/// still hold is everything after the picker — the source, the sendonly |
| +/// transceiver, the `share` message, and both of §5.4's endings, the |
| +/// button's and the capture's own (`end`, which is the system's stop |
| +/// control as far as the call can tell). |
| +@MainActor final class SyntheticScreen: ScreenCapture { |
| + private var source: RTCVideoSource? |
| + private var capturer: RTCVideoCapturer? |
| + private var task: Task<Void, Never>? |
| + private var ended: (@MainActor () -> Void)? |
| + |
| + func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws { |
| + self.source = source |
| + self.ended = ended |
| + capturer = RTCVideoCapturer(delegate: source) |
| + task = Task { @MainActor [weak self] in |
| + while !Task.isCancelled { |
| + self?.frame() |
| + try? await Task.sleep(nanoseconds: 100_000_000) |
| + } |
| + } |
| + } |
| + func stop() { |
| + task?.cancel() |
| + task = nil |
| + source = nil |
| + capturer = nil |
| + ended = nil |
| + } |
| + /// The capture ending on its own, as a shared window closing would. |
| + func end() { |
| + let ended = self.ended |
| + stop() |
| + ended?() |
| + } |
| + private func frame() { |
| + guard let source, let capturer else { return } |
| + // A 16:9 screen, wider than either synthetic camera, so the far side |
| + // can tell by size alone which one it is decoding. |
| + var pixel: CVPixelBuffer? |
| + CVPixelBufferCreate( |
| + kCFAllocatorDefault, 640, 360, kCVPixelFormatType_32BGRA, |
| + [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary, &pixel) |
| + guard let pixel else { return } |
| + CVPixelBufferLockBaseAddress(pixel, []) |
| + if let base = CVPixelBufferGetBaseAddress(pixel) { |
| + memset( |
| + base, Int32(Int(Date().timeIntervalSince1970 * 10) % 100 + 120), |
| + CVPixelBufferGetDataSize(pixel)) |
| + } |
| + CVPixelBufferUnlockBaseAddress(pixel, []) |
| + source.capturer( |
| + capturer, |
| + didCapture: RTCVideoFrame( |
| + buffer: RTCCVPixelBuffer(pixelBuffer: pixel), rotation: ._0, |
| + timeStampNs: Int64(ProcessInfo.processInfo.systemUptime * 1_000_000_000))) |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index e807ba7..33d94e4 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -38,6 +38,11 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| /// carries its caption before there is anything to show in it. |
| @Published public private(set) var screens: [String] = [] |
| @Published public private(set) var remoteScreens: [String: RTCVideoTrack] = [:] |
| + /// Whether this app is sending a screen, and what the last attempt to |
| + /// had to say — which is usually nothing. |
| + @Published public private(set) var sharingScreen = false |
| + @Published public private(set) var screenNote = "" |
| + private var screenPending = false |
| @Published public private(set) var localTrack: RTCVideoTrack? |
| @Published public private(set) var muted = false |
| @Published public private(set) var cameraOff = false |
| @@ -122,6 +127,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| remoteTracks = [:] |
| screens = [] |
| remoteScreens = [:] |
| + sharingScreen = false |
| + screenNote = "" |
| localTrack = nil |
| ice = nil |
| state = nil |
| @@ -298,6 +305,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| remoteTracks = [:] |
| screens = [] |
| remoteScreens = [:] |
| + sharingScreen = false |
| localTrack = nil |
| default: status = "Connecting…" |
| } |
| @@ -358,6 +366,53 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| state?.peers.first { $0.peer == peer }?.name ?? "Participant" |
| } |
| + /// Shares a screen (§5.4): the capture asks the person what to share, |
| + /// every connection gets it on a transceiver of its own, and then the |
| + /// room is told — the label, not the media, so peers can caption the |
| + /// tile before the renegotiation lands. |
| + /// |
| + /// A refusal at the picker says one sentence, the browser's, and does |
| + /// not diagnose which kind of refusal it was. |
| + public func shareScreen(_ capture: ScreenCapture) async { |
| + // The picker stays open for as long as a person takes to answer it, |
| + // and the button is live the whole time. Without this a second press |
| + // starts a second capture that nothing would ever stop. |
| + guard bridge != nil, !sharingScreen, !screenPending else { return } |
| + screenPending = true |
| + defer { screenPending = false } |
| + let joined = epoch |
| + screenNote = "" |
| + do { |
| + try await media.startScreen(capture) { [weak self] in |
| + // The capture ended on its own: the system's stop control, or |
| + // the shared window closing. The same ending as the button. |
| + guard let self, self.epoch == joined else { return } |
| + self.stopSharingScreen() |
| + } |
| + } catch is CancellationError { |
| + return |
| + } catch { |
| + guard epoch == joined else { return } |
| + screenNote = "Screen sharing was not allowed." |
| + return |
| + } |
| + guard epoch == joined, bridge != nil else { |
| + media.stopScreen() |
| + return |
| + } |
| + sharingScreen = true |
| + send(type: "share", to: "", gen: 0, payload: json(["on": true]) ?? "") |
| + } |
| + |
| + /// Stops sharing, from every way out: the button, the system's own stop |
| + /// control, and the capture ending under us. |
| + public func stopSharingScreen() { |
| + guard sharingScreen else { return } |
| + sharingScreen = false |
| + media.stopScreen() |
| + send(type: "share", to: "", gen: 0, payload: json(["on": false]) ?? "") |
| + } |
| + |
| /// A peer has said it is sharing, or its screen has arrived. Either one |
| /// is enough to draw the tile. |
| func announceScreen(_ peer: String) { |
diff --git a/native/apple/Sources/CallsKit/Capture.swift b/native/apple/Sources/CallsKit/Capture.swift| index 8f4e51f..b66fca0 100644 |
| --- a/native/apple/Sources/CallsKit/Capture.swift |
| +++ b/native/apple/Sources/CallsKit/Capture.swift |
| @@ -37,6 +37,20 @@ extension VideoCapture { |
| public func use(camera id: String) async throws {} |
| } |
| +/// A screen, as §5.4 sends one: frames into a source of its own, which |
| +/// the call carries on a `sendonly` transceiver beside the camera's. |
| +/// |
| +/// `start` returns once the person has chosen what to share and frames |
| +/// are on their way; a refusal at the system's own picker is a throw, and |
| +/// nothing has been announced. `ended` is for the capture ending on its |
| +/// own — the system's stop control, a shared window closing, a broadcast |
| +/// stopped from Control Center — and is called at most once, on the main |
| +/// actor. `stop` is the call ending it, and calls nothing back. |
| +@MainActor public protocol ScreenCapture: AnyObject { |
| + func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws |
| + func stop() |
| +} |
| + |
| @MainActor final class CameraCapture: VideoCapture { |
| private var capturer: RTCCameraVideoCapturer? |
| private var source: RTCVideoSource? |
diff --git a/native/apple/Sources/CallsKit/MacCallView.swift b/native/apple/Sources/CallsKit/MacCallView.swift| index 4389adc..788351e 100644 |
| --- a/native/apple/Sources/CallsKit/MacCallView.swift |
| +++ b/native/apple/Sources/CallsKit/MacCallView.swift |
| @@ -27,6 +27,15 @@ |
| Text(call.title).font(.title2.bold()).lineLimit(1) |
| Text(call.status).font(.callout).foregroundStyle(.secondary) |
| .accessibilityAddTraits(.updatesFrequently) |
| + // Said in words as well as by the button's state: a person |
| + // sharing a screen should never have to wonder whether they |
| + // still are. |
| + if call.sharingScreen { |
| + Label("You are sharing your screen", systemImage: "rectangle.on.rectangle") |
| + .font(.callout.bold()).foregroundStyle(.orange) |
| + } else if !call.screenNote.isEmpty { |
| + Text(call.screenNote).font(.callout).foregroundStyle(.secondary) |
| + } |
| } |
| Spacer() |
| if !grid, peers.count > 1 || !call.screens.isEmpty { |
| @@ -216,6 +225,20 @@ |
| emphasized: call.cameraOff) |
| ) |
| .help(call.cameraOff ? "Turn camera on" : "Turn camera off") |
| + Button( |
| + call.sharingScreen ? "Stop sharing your screen" : "Share your screen", |
| + systemImage: call.sharingScreen ? "rectangle.on.rectangle.slash" : "rectangle.on.rectangle" |
| + ) { |
| + if call.sharingScreen { |
| + call.stopSharingScreen() |
| + } else { |
| + Task { await call.shareScreen(MacScreenCapture()) } |
| + } |
| + } |
| + .buttonStyle(CallActionStyle(color: .orange, emphasized: call.sharingScreen)) |
| + .help(call.sharingScreen ? "Stop sharing your screen" : "Share a window or your whole screen") |
| + .disabled(call.state?.phase != "admitted") |
| + .accessibilityIdentifier("shareScreen") |
| Button("Devices", systemImage: "slider.horizontal.3") { showingDevices.toggle() } |
| .buttonStyle(CallActionStyle(color: .teal, emphasized: showingDevices)) |
| .help("Choose your camera, and see which microphone and speaker this call is on") |
diff --git a/native/apple/Sources/CallsKit/MacScreenCapture.swift b/native/apple/Sources/CallsKit/MacScreenCapture.swift| new file mode 100644 |
| index 0000000..2017e38 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/MacScreenCapture.swift |
| @@ -0,0 +1,230 @@ |
| +#if os(macOS) |
| + import CoreMedia |
| + import Foundation |
| + import ScreenCaptureKit |
| + import WebRTC |
| + |
| + /// A display or a window, chosen with the system's own picker and |
| + /// delivered into the call's screen source. |
| + /// |
| + /// The picker is SCContentSharingPicker (macOS 14, the app's floor), and |
| + /// that choice does more than save drawing one: content a person picks |
| + /// there is content they have consented to share, so the app needs no |
| + /// Screen Recording permission of its own and never sends anyone to |
| + /// System Settings to grant one. There is nothing in Info.plist for it |
| + /// either — macOS has no usage string for screen capture. The same holds |
| + /// in the App Sandbox, which is why this is also the path a sandboxed |
| + /// build can keep. |
| + /// |
| + /// The capture ends on its own when the person stops it from the menu |
| + /// bar's sharing indicator, or the shared window closes; either one |
| + /// reaches `ended`, which the call treats exactly like its own button. |
| + @MainActor public final class MacScreenCapture: NSObject, ScreenCapture { |
| + private var stream: SCStream? |
| + private var frames: ScreenFrames? |
| + private var ended: (@MainActor () -> Void)? |
| + private var choosing: CheckedContinuation<SCContentFilter, Error>? |
| + private let queue = DispatchQueue(label: "calls.screen", qos: .userInteractive) |
| + |
| + public override init() {} |
| + |
| + public func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws { |
| + let picker = SCContentSharingPicker.shared |
| + var configuration = SCContentSharingPickerConfiguration() |
| + configuration.allowedPickerModes = [.singleDisplay, .singleWindow] |
| + // The call's own window is the one thing nobody in it needs to see: |
| + // shared, it is a hall of mirrors of their own faces. |
| + if let own = Bundle.main.bundleIdentifier { |
| + configuration.excludedBundleIDs = [own] |
| + } |
| + picker.defaultConfiguration = configuration |
| + picker.maximumStreamCount = 1 |
| + picker.add(self) |
| + picker.isActive = true |
| + let filter: SCContentFilter |
| + do { |
| + let picked = try await withCheckedThrowingContinuation { continuation in |
| + choosing = continuation |
| + picker.present() |
| + } |
| + filter = await Self.withoutCalls(picked) |
| + } catch { |
| + release() |
| + throw error |
| + } |
| + let frames = ScreenFrames(source: source) |
| + let stream = SCStream(filter: filter, configuration: Self.configuration(for: filter), delegate: self) |
| + do { |
| + try stream.addStreamOutput(frames, type: .screen, sampleHandlerQueue: queue) |
| + try await stream.startCapture() |
| + } catch { |
| + release() |
| + throw error |
| + } |
| + self.stream = stream |
| + self.frames = frames |
| + self.ended = ended |
| + } |
| + |
| + public func stop() { |
| + ended = nil |
| + let stream = self.stream |
| + self.stream = nil |
| + frames = nil |
| + release() |
| + stream?.stopCapture { _ in } |
| + } |
| + |
| + private func release() { |
| + let picker = SCContentSharingPicker.shared |
| + picker.remove(self) |
| + picker.isActive = false |
| + choosing?.resume(throwing: CancellationError()) |
| + choosing = nil |
| + } |
| + |
| + /// The capture ending under the call, rather than by it. |
| + private func finished() { |
| + let ended = self.ended |
| + stop() |
| + ended?() |
| + } |
| + |
| + /// The picked content with Calls taken out of it. |
| + /// |
| + /// A whole display, as the picker hands it back, includes this app's |
| + /// own windows: the call window, and the small floating one it shows |
| + /// while it is in the background. Shared, they are a hall of mirrors |
| + /// of everyone's faces — and the floating one is always on top of |
| + /// whatever the person meant to show. So a display is rebuilt as the |
| + /// same display excluding this application, which also covers windows |
| + /// that open after sharing starts. A single window needs nothing: this |
| + /// app's are left out of the picker, so the window is someone else's. |
| + /// |
| + /// The display comes from the filter itself on macOS 15.2 and later, |
| + /// and before that from matching its frame. The application comes from |
| + /// the current process's own shareable content, which is the one query |
| + /// that asks for no Screen Recording permission. On a macOS too old |
| + /// for that (before 14.4) the display is shared as picked; the |
| + /// floating window's own `sharingType` is then the only thing keeping |
| + /// it out, and the call window is in the share. |
| + static func withoutCalls(_ filter: SCContentFilter) async -> SCContentFilter { |
| + guard filter.style == .display, #available(macOS 14.4, *), |
| + let content = try? await SCShareableContent.currentProcess, |
| + let calls = content.applications.first(where: { $0.processID == getpid() }) |
| + else { return filter } |
| + var display: SCDisplay? |
| + if #available(macOS 15.2, *) { display = filter.includedDisplays.first } |
| + if display == nil { display = content.displays.first { $0.frame == filter.contentRect } } |
| + guard let display else { return filter } |
| + return SCContentFilter(display: display, excludingApplications: [calls], exceptingWindows: []) |
| + } |
| + |
| + /// Full resolution up to 1920 on the long side, at up to 30 frames a |
| + /// second. Larger is sharper than any receiver will be shown it and |
| + /// costs the encoder for nothing; the source is a screencast, so |
| + /// under pressure WebRTC gives up frames before it gives up pixels. |
| + static func configuration(for filter: SCContentFilter) -> SCStreamConfiguration { |
| + let configuration = SCStreamConfiguration() |
| + let rect = filter.contentRect |
| + let scale = CGFloat(filter.pointPixelScale) |
| + let size = fitted(width: rect.width * scale, height: rect.height * scale, longest: 1920) |
| + configuration.width = size.width |
| + configuration.height = size.height |
| + configuration.minimumFrameInterval = CMTime(value: 1, timescale: 30) |
| + configuration.pixelFormat = kCVPixelFormatType_420YpCbCr8BiPlanarFullRange |
| + configuration.showsCursor = true |
| + configuration.queueDepth = 5 |
| + return configuration |
| + } |
| + } |
| + |
| + extension MacScreenCapture: SCContentSharingPickerObserver { |
| + nonisolated public func contentSharingPicker( |
| + _ picker: SCContentSharingPicker, didCancelFor stream: SCStream? |
| + ) { |
| + Task { @MainActor [weak self] in |
| + guard let self else { return } |
| + if stream == nil || stream === self.stream { |
| + if let choosing = self.choosing { |
| + self.choosing = nil |
| + choosing.resume(throwing: ScreenShareRefused()) |
| + } |
| + } |
| + } |
| + } |
| + |
| + nonisolated public func contentSharingPicker( |
| + _ picker: SCContentSharingPicker, didUpdateWith filter: SCContentFilter, for stream: SCStream? |
| + ) { |
| + Task { @MainActor [weak self] in |
| + guard let self else { return } |
| + if let choosing = self.choosing { |
| + self.choosing = nil |
| + choosing.resume(returning: filter) |
| + return |
| + } |
| + // Chosen again from the menu bar mid-share: the same stream, now |
| + // on something else. The call does not need to hear about it. |
| + guard let current = self.stream, stream == nil || stream === current else { return } |
| + let next = await Self.withoutCalls(filter) |
| + guard current === self.stream else { return } |
| + current.updateContentFilter(next) { _ in } |
| + current.updateConfiguration(Self.configuration(for: next)) { _ in } |
| + } |
| + } |
| + |
| + nonisolated public func contentSharingPickerStartDidFailWithError(_ error: Error) { |
| + Task { @MainActor [weak self] in |
| + guard let self, let choosing = self.choosing else { return } |
| + self.choosing = nil |
| + choosing.resume(throwing: error) |
| + } |
| + } |
| + } |
| + |
| + extension MacScreenCapture: SCStreamDelegate { |
| + nonisolated public func stream(_ stream: SCStream, didStopWithError error: Error) { |
| + Task { @MainActor [weak self] in |
| + guard let self, stream === self.stream else { return } |
| + self.finished() |
| + } |
| + } |
| + } |
| + |
| + struct ScreenShareRefused: Error {} |
| + |
| + /// Hands each complete frame to the source. Called on the capture's own |
| + /// queue, never the main actor: a screen is thirty frames a second and |
| + /// the main actor has an interface to draw. |
| + private final class ScreenFrames: NSObject, SCStreamOutput, @unchecked Sendable { |
| + private let source: RTCVideoSource |
| + private let capturer: RTCVideoCapturer |
| + |
| + init(source: RTCVideoSource) { |
| + self.source = source |
| + capturer = RTCVideoCapturer(delegate: source) |
| + } |
| + |
| + func stream( |
| + _ stream: SCStream, didOutputSampleBuffer sample: CMSampleBuffer, of type: SCStreamOutputType |
| + ) { |
| + guard type == .screen, sample.isValid, let pixels = sample.imageBuffer else { return } |
| + // Only complete frames carry pixels. An idle screen sends status |
| + // frames and nothing else, and the receiver keeps the last picture, |
| + // which is what an idle screen looks like. |
| + guard |
| + let attachments = CMSampleBufferGetSampleAttachmentsArray(sample, createIfNecessary: false) |
| + as? [[SCStreamFrameInfo: Any]], |
| + let raw = attachments.first?[.status] as? Int, |
| + SCFrameStatus(rawValue: raw) == .complete |
| + else { return } |
| + let time = CMTimeGetSeconds(sample.presentationTimeStamp) |
| + source.capturer( |
| + capturer, |
| + didCapture: RTCVideoFrame( |
| + buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: ._0, |
| + timeStampNs: Int64(time * 1_000_000_000))) |
| + } |
| + } |
| +#endif |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 15d358a..dacff27 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -8,6 +8,14 @@ import WebRTC |
| let factory: RTCPeerConnectionFactory |
| var localVideo: RTCVideoTrack? |
| var localAudio: RTCAudioTrack? |
| + /// §5.4: the display track while this app is sharing, and nothing |
| + /// otherwise. Every connection carries it on a transceiver of its own. |
| + private(set) var localScreen: RTCVideoTrack? |
| + private var screenCapture: ScreenCapture? |
| + // Bumped by everything that ends a share, so a capture whose picker |
| + // was still open when the share or the call ended is stopped when it |
| + // finally answers, rather than sent. |
| + private var screenAttempt = 0 |
| private var capture: VideoCapture? |
| private var peers: [String: MediaPeer] = [:] |
| @@ -62,7 +70,44 @@ import WebRTC |
| for (_, p) in peers { p.close() } |
| peers.removeAll() |
| } |
| + |
| + /// Starts sending a screen (§5.4). The capture is asked first: a person |
| + /// who changes their mind at the system's picker has shared nothing, |
| + /// and nothing is added to any connection. |
| + /// |
| + /// The source is marked as a screencast, which is what tells WebRTC to |
| + /// keep resolution and give up frame rate under pressure — text that |
| + /// goes soft is unreadable, where text that updates a little late is |
| + /// not. |
| + func startScreen(_ capture: ScreenCapture, ended: @escaping @MainActor () -> Void) async throws { |
| + guard localScreen == nil else { return } |
| + screenAttempt += 1 |
| + let attempt = screenAttempt |
| + let source = factory.videoSource(forScreenCast: true) |
| + try await capture.start(source: source, ended: ended) |
| + guard attempt == screenAttempt, localScreen == nil else { |
| + capture.stop() |
| + throw CancellationError() |
| + } |
| + screenCapture = capture |
| + localScreen = factory.videoTrack(with: source, trackId: "screen") |
| + for peer in peers.values { peer.addScreen() } |
| + } |
| + |
| + /// Stops sending the screen: every connection's screen transceiver is |
| + /// stopped and renegotiated, so the receivers' tracks *end*, and the |
| + /// capture goes last. |
| + func stopScreen() { |
| + screenAttempt += 1 |
| + guard let capture = screenCapture else { return } |
| + screenCapture = nil |
| + localScreen = nil |
| + for peer in peers.values { peer.removeScreen() } |
| + capture.stop() |
| + } |
| + |
| func close() { |
| + stopScreen() |
| closePeers() |
| capture?.stop() |
| capture = nil |
| @@ -217,6 +262,11 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| // Whether this connection carries a camera of ours: the classifier's |
| // direction rule turns on it. |
| private var sentVideo = false |
| + // §5.4, sending: the transceiver this connection carries our screen on, |
| + // while we are sharing. And an offer this connection owes the far side |
| + // but could not make yet, because it was in the middle of another. |
| + private var screenTx: RTCRtpTransceiver? |
| + private var owesOffer = false |
| init(id: String, generation: Int64, polite: Bool, engine: MediaEngine) { |
| self.id = id |
| @@ -244,9 +294,85 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| if let video = engine.localVideo { |
| sentVideo = pc?.add(video, streamIds: ["calls"]) != nil |
| } |
| + // A connection built while this app is already sharing gets the |
| + // screen with everything else: someone joining mid-share sees it. |
| + // Last, as in the browser, so the camera and microphone take the |
| + // m-lines the far side built for them and the screen a place of its |
| + // own after them. |
| + addScreen() |
| + } |
| + |
| + /// Gives this connection the screen on a `sendonly` transceiver in a |
| + /// stream of its own, which is what keeps it clear of the camera at the |
| + /// far end. Adding it asks for negotiation, and perfect negotiation |
| + /// carries the offer with no help from here. |
| + /// |
| + /// Only on a stable connection. A transceiver added mid-negotiation is |
| + /// unassociated when the far side's offer arrives, and m-line matching |
| + /// will hand it that offer's m-line — so two people sharing at once end |
| + /// up each sending a screen in the place the other built for something |
| + /// else. The browser's mutualshare gate found that; the deferred ones |
| + /// are added when the connection settles. |
| + func addScreen() { |
| + guard let pc, !closed, screenTx == nil, let track = engine?.localScreen, |
| + pc.signalingState == .stable |
| + else { return } |
| + let setup = RTCRtpTransceiverInit() |
| + setup.direction = .sendOnly |
| + setup.streamIds = ["screen"] |
| + screenTx = pc.addTransceiver(with: track, init: setup) |
| + } |
| + |
| + /// Ends this connection's screen m-line. Stopping the transceiver is |
| + /// §5.4's own instruction and the only thing that makes the receiver's |
| + /// track end rather than freeze on its last frame. |
| + /// |
| + /// The framework's stop is the immediate one: it stops the transceiver |
| + /// and asks for nothing, where the browser's `stop()` also raises |
| + /// negotiation-needed. So the offer that tells the far side is asked |
| + /// for here, by hand. |
| + func removeScreen() { |
| + guard let tx = screenTx else { return } |
| + screenTx = nil |
| + tx.stopInternal() |
| + renegotiate() |
| + } |
| + |
| + /// An offer this side has to make for a reason negotiation-needed will |
| + /// not announce. Made now if the connection is quiet, and otherwise |
| + /// owed, and made the next time it settles. |
| + private func renegotiate() { |
| + guard let pc, !closed else { return } |
| + guard pc.signalingState == .stable else { |
| + owesOffer = true |
| + return |
| + } |
| + Task { @MainActor [weak self] in |
| + guard let self else { return } |
| + if self.polite { try? await Task.sleep(nanoseconds: 200_000_000) } |
| + self.enqueue { [weak self] in |
| + guard let self, let pc = self.pc else { return } |
| + if pc.signalingState != .stable { |
| + self.owesOffer = true |
| + return |
| + } |
| + try await self.offer() |
| + } |
| + } |
| + } |
| + |
| + /// The connection is back at stable: the one moment a deferred screen |
| + /// can be added, and an owed offer made. |
| + private func settled() { |
| + guard let pc, !closed, pc.signalingState == .stable else { return } |
| + addScreen() |
| + if owesOffer { |
| + owesOffer = false |
| + renegotiate() |
| + } |
| } |
| func isScreen(mid: String) -> Bool { !mid.isEmpty && mid == screenMid } |
| - func isOwnScreen(mid: String) -> Bool { false } |
| + func isOwnScreen(mid: String) -> Bool { !mid.isEmpty && mid == screenTx?.mid } |
| /// The `share` message: on draws the captioned tile ahead of the media, |
| /// off is one of §5.4's two endings. |
| @@ -293,6 +419,7 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| func close() { |
| closed = true |
| + screenTx = nil |
| work?.cancel() |
| work = nil |
| pc?.close() |
| @@ -486,7 +613,10 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| } |
| nonisolated func peerConnection( |
| _ peerConnection: RTCPeerConnection, didChange stateChanged: RTCSignalingState |
| - ) {} |
| + ) { |
| + guard stateChanged == .stable else { return } |
| + Task { @MainActor [weak self] in self?.settled() } |
| + } |
| nonisolated func peerConnection( |
| _ peerConnection: RTCPeerConnection, didAdd stream: RTCMediaStream |
| ) {} |
diff --git a/native/apple/Sources/CallsKit/ScreenShare.swift b/native/apple/Sources/CallsKit/ScreenShare.swift| index c3661f1..9d293b9 100644 |
| --- a/native/apple/Sources/CallsKit/ScreenShare.swift |
| +++ b/native/apple/Sources/CallsKit/ScreenShare.swift |
| @@ -1,3 +1,4 @@ |
| +import CoreGraphics |
| import Foundation |
| // --------------------------------------------------------------- |
| @@ -118,3 +119,13 @@ public enum Stage { |
| return peers.first.map(StageItem.camera) |
| } |
| } |
| + |
| +/// A frame size that fits within `longest` on its long side, keeps its |
| +/// shape, and is even in both directions, which is what the encoders' |
| +/// 4:2:0 chroma needs. |
| +func fitted(width: CGFloat, height: CGFloat, longest: CGFloat) -> (width: Int, height: Int) { |
| + guard width > 0, height > 0 else { return (2, 2) } |
| + let scale = min(1, longest / max(width, height)) |
| + func even(_ value: CGFloat) -> Int { max(2, Int((value * scale).rounded(.down)) & ~1) } |
| + return (even(width), even(height)) |
| +} |
diff --git a/native/apple/Tests/CallsKitTests/ScreenShareTests.swift b/native/apple/Tests/CallsKitTests/ScreenShareTests.swift| index bc2d784..9c5ebb6 100644 |
| --- a/native/apple/Tests/CallsKitTests/ScreenShareTests.swift |
| +++ b/native/apple/Tests/CallsKitTests/ScreenShareTests.swift |
| @@ -91,6 +91,23 @@ final class ScreenShareTests: XCTestCase { |
| XCTAssertEqual(screenCaption("Bea"), "Bea\u{2019}s screen") |
| } |
| + func testAScreenIsFittedEvenAndNeverEnlarged() { |
| + // A 5K display at 2x comes down to 1920 on its long side, keeping its |
| + // shape; both sides even, for the encoders' 4:2:0 chroma. |
| + let big = fitted(width: 5120, height: 2880, longest: 1920) |
| + XCTAssertEqual(big.width, 1920) |
| + XCTAssertEqual(big.height, 1080) |
| + // A small window is sent at its own size, rounded down to even. |
| + let small = fitted(width: 801, height: 601, longest: 1920) |
| + XCTAssertEqual(small.width, 800) |
| + XCTAssertEqual(small.height, 600) |
| + // A portrait phone screen is fitted on its height. |
| + let phone = fitted(width: 1179, height: 2556, longest: 1280) |
| + XCTAssertEqual(phone.height, 1280) |
| + XCTAssertEqual(phone.width % 2, 0) |
| + XCTAssertEqual(fitted(width: 0, height: 100, longest: 1920).width, 2) |
| + } |
| + |
| // --------------------------------------------------------------- |
| // The stage |
| // --------------------------------------------------------------- |
diff --git a/test/browser/native.mjs b/test/browser/native.mjs| index 10c6507..12e09d5 100644 |
| --- a/test/browser/native.mjs |
| +++ b/test/browser/native.mjs |
| @@ -128,13 +128,27 @@ try { |
| await wait(async () => await remoteFrames() > refusedFrames + 15, 'media survives a refused camera switch'); |
| console.log('PASS camera switched mid-call without disturbing the connection'); |
| } |
| + await page.locator('[data-chat-text]').fill('browser chat probe'); |
| + await page.locator('[data-chat-send]').click(); |
| + await wait(() => sample?.receivedChatProbe === true, 'native displays browser chat'); |
| + native.stdin.write('chat\n'); |
| + await wait(() => chatCommandHandled, 'native handles send-chat command'); |
| + await page.locator('[data-chat-log]').getByText('native chat probe', { exact: true }).waitFor(); |
| + await wait(() => sample?.sentChatProbe === true, 'native displays accepted outgoing chat'); |
| + console.log('PASS chat delivered from browser to native and native to browser'); |
| + // The screen scenarios run after chat and not before: they take the |
| + // better part of a minute, and chat is not replayed to a recipient |
| + // whose stream is renewing, which in this mode is every ten seconds. |
| + // Screen share does not depend on its message arriving — the m-line |
| + // is classified on its own, and a stopped transceiver ends the track — |
| + // so it can share the call with renewals where chat cannot. |
| // §5.4, receiving: the browser shares its screen and the app shows it |
| // as a tile of its own, captioned with the sharer's name, while the |
| // browser's camera keeps arriving on the tile it always had. Then the |
| // browser stops, and the tile goes — by the `share {on:false}` or the |
| // stopped transceiver, whichever lands first — with the camera still |
| // arriving afterwards. |
| - if (mode !== 'guest') { |
| + if (mode === 'member') { |
| const name = await page.evaluate(() => document.getElementById('room')?.dataset.selfName || ''); |
| assert.notEqual(name, '', 'the browser has a name for its screen to be captioned with'); |
| const caption = name + '\u2019s screen'; |
| @@ -155,14 +169,64 @@ try { |
| await wait(() => sample.stats.framesDecoded > afterStop + 15, 'the browser\u2019s camera survives its screen stopping'); |
| console.log('PASS native removes a stopped screen and keeps the camera'); |
| } |
| - await page.locator('[data-chat-text]').fill('browser chat probe'); |
| - await page.locator('[data-chat-send]').click(); |
| - await wait(() => sample?.receivedChatProbe === true, 'native displays browser chat'); |
| - native.stdin.write('chat\n'); |
| - await wait(() => chatCommandHandled, 'native handles send-chat command'); |
| - await page.locator('[data-chat-log]').getByText('native chat probe', { exact: true }).waitFor(); |
| - await wait(() => sample?.sentChatProbe === true, 'native displays accepted outgoing chat'); |
| - console.log('PASS chat delivered from browser to native and native to browser'); |
| + // §5.4, sending: the app shares a (synthetic) screen and the browser |
| + // shows it exactly as it shows another browser's — a tile of its own |
| + // under shareKey, captioned with the app's name, decoding real frames |
| + // while the app's camera keeps arriving. Stopping must *end* the |
| + // browser's received track, which only a stopped transceiver and a |
| + // renegotiation do; a message alone would leave a frozen frame behind |
| + // on a browser that never got it. Then the capture ending on its own, |
| + // the path the system's stop control takes, ends the share the same |
| + // way. |
| + if (mode === 'member') { |
| + const nativePeer = sample.peer; |
| + const key = nativePeer + '#share'; |
| + const browserState = () => page.evaluate(() => { |
| + const s = window.meet.state(); |
| + return { shares: s.shares, frames: Object.fromEntries(s.frames) }; |
| + }); |
| + const tiles = () => page.evaluate(() => ({ |
| + camera: [...document.querySelectorAll('#tiles figure.calls-tile:not(.calls-tile-self):not(.calls-tile-share) figcaption span')] |
| + .map(e => e.textContent.trim()).filter(Boolean)[0] || '', |
| + screens: [...document.querySelectorAll('#tiles figure.calls-tile-share figcaption span')] |
| + .map(e => e.textContent.trim()).filter(Boolean), |
| + })); |
| + for (const ending of ['screen-off', 'screen-end']) { |
| + native.stdin.write('screen-on\n'); |
| + await wait(() => sample?.sharingScreen === true, 'native says it is sharing'); |
| + await wait(async () => (await browserState()).shares.includes(nativePeer), 'the browser is told whose screen is coming'); |
| + const { camera } = await tiles(); |
| + assert.notEqual(camera, '', 'the browser captions the app\u2019s camera'); |
| + await wait(async () => { const t = await tiles(); return t.screens.length === 1 && t.screens[0] === camera + '\u2019s screen'; }, |
| + 'the browser captions the app\u2019s screen with its name'); |
| + const screenBase = (await browserState()).frames[key] || 0; |
| + await wait(async () => ((await browserState()).frames[key] || 0) > screenBase + 15, 'the browser decodes the app\u2019s screen'); |
| + const cameraBase = (await browserState()).frames[nativePeer] || 0; |
| + await wait(async () => ((await browserState()).frames[nativePeer] || 0) > cameraBase + 15, |
| + 'the app\u2019s camera keeps arriving beside its screen'); |
| + await wait(() => sample.stats.screenBytesSent > 0, 'the app counts its screen apart from its camera'); |
| + console.log('PASS the browser shows the app\u2019s screen as its own captioned tile'); |
| + const armed = await page.evaluate(() => { |
| + const v = document.querySelector('#tiles figure.calls-tile-share video'); |
| + const track = v && v.srcObject ? v.srcObject.getVideoTracks()[0] : null; |
| + if (!track) return null; |
| + window.__nativeShareEnded = false; |
| + track.addEventListener('ended', () => { window.__nativeShareEnded = true; }); |
| + return track.readyState; |
| + }); |
| + assert.equal(armed, 'live', 'the received screen track is live and watched for its ending'); |
| + native.stdin.write(ending + '\n'); |
| + await wait(() => sample?.sharingScreen === false, 'native says it has stopped'); |
| + await wait(async () => (await browserState()).shares.length === 0 && (await tiles()).screens.length === 0, |
| + 'the browser removes the app\u2019s screen'); |
| + await wait(() => page.evaluate(() => window.__nativeShareEnded === true), |
| + 'the browser\u2019s received screen track ended rather than froze'); |
| + const afterStop = (await browserState()).frames[nativePeer] || 0; |
| + await wait(async () => ((await browserState()).frames[nativePeer] || 0) > afterStop + 15, |
| + 'the app\u2019s camera survives its screen stopping'); |
| + console.log('PASS the app\u2019s screen ended in the browser (' + ending + ') and its camera continued'); |
| + } |
| + } |
| if (mode === 'chat') { |
| native.stdin.write('share\n'); |
| await wait(() => invitationReady, 'native member retrieves the original invitation'); |