amadan
Choose a camera, and see what the audio is actually doing
The apps took whatever devices the system handed them and said nothing about what those were. A camera chosen wrongly cost the call; a microphone could not be reached at all; and when a call sounded wrong there was no way to tell a bad device from a bad network, because nothing in the app recorded either. Devices opens a panel with the camera, the microphone, the speaker and an audio readout. The camera is a picker on both platforms. Switching rests on the capturer feeding the same RTCVideoSource, so the RTCVideoTrack every connection already holds does not change and nothing renegotiates — the native counterpart of the browser's replaceTrack, and for the same reason. Switches are serialised: two picks are two capture restarts on one device, and run at once the survivor is whichever session opened last rather than the one the person chose. A refused switch reads the camera back rather than assuming, so the control follows the call instead of the intention. Audio is asymmetric, and not by choice. WebRTC's iOS slices ship five audio headers including RTCAudioDevice and RTCAudioSession; the macOS slice ships two. There is no supported way to enumerate or select an audio device for WebRTC on macOS through this framework, and no way to supply a custom audio device module either, because the protocol is not in the slice. So iPhone gets a microphone picker, and Mac gets the truth about what macOS has chosen — read from CoreAudio, which is exactly what the audio device module follows — plus the Sound settings pane. Naming the device earns its place on its own: the first thing someone needs when a call sounds wrong is to know what they are connected to. Choosing a Bluetooth microphone on iPhone means the hands-free profile, which is 8 kHz in both directions. That has been happening silently to every call the moment any headset connected, because .allowBluetoothHFP was set unconditionally at join. It is now set only while a Bluetooth input is the chosen one, and the panel says what the choice costs before it is made. The readout is the successor to a call on 2026-09-17 that came back as an unintelligible growl and left nothing behind to look at — no log, no counters, no artefact, and a reproduction would have produced nothing either. Concealed samples against total are loss, and sound like dropouts. Inserted and removed samples are the jitter buffer rescaling the signal to track the sender's clock, and a stretched voice is what a growl is. Drift is named ahead of loss when both are up: loss is common and widely understood, sustained drift is rarer and is the one nobody thinks to look for. On macOS these four counters are the only instrument there is, because a bad device cannot be ruled out by switching away from it. Gates: make ci, make native-test, make native-browser, make browser, swift test (17), and the iPhone simulator's AudioRouteTests, which is where the Bluetooth rule can be tested at all since CallsKitTests builds for macOS. The native gate gains a camera-switch scenario — switch, race two switches, refuse an unknown one, and hold the far side to still decoding frames on the same pair across each. The interop executable grows two synthetic cameras of different sizes so a switch is something the browser can see rather than something this side claims. Not covered: the AVFoundation capture restart itself, which needs a real camera. What the gate holds is the plumbing around it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
14 files changed,
+928
−5
native/README.md+38 −0native/apple/Sources/CallsInterop/Interop.swift+8 −0native/apple/Sources/CallsInterop/SyntheticVideo.swift+27 −1native/apple/Sources/CallsKit/Audio.swift+168 −0native/apple/Sources/CallsKit/AudioHealth.swift+79 −0native/apple/Sources/CallsKit/Call.swift+85 −0native/apple/Sources/CallsKit/Capture.swift+88 −3native/apple/Sources/CallsKit/DevicePanel.swift+154 −0native/apple/Sources/CallsKit/MacCallView.swift+19 −0native/apple/Sources/CallsKit/Media.swift+49 −1native/apple/Tests/CallsKitTests/AudioHealthTests.swift+123 −0native/ios/Sources/CallsPhone.swift+16 −0native/ios/Tests/AudioRouteTests.swift+39 −0test/browser/native.mjs+35 −0
CI passed — run details
diff --git a/native/README.md b/native/README.md| index 8770e08..d9bbcc0 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -28,6 +28,44 @@ credential, and the room's own **Share** is where one is minted. |
| Scheduling a meeting for later, ending one, and replacing invitations stay on |
| the web for this milestone. |
| +**Devices** in the call opens the camera, microphone and speaker panel, and |
| +the audio readout beneath them. |
| + |
| +The camera is a picker on both platforms. Switching rests on the capturer |
| +feeding the same `RTCVideoSource`, so the `RTCVideoTrack` every peer |
| +connection already holds does not change and nothing renegotiates — the |
| +native counterpart of the browser's `replaceTrack`. Switches are serialised; |
| +the last pick wins. |
| + |
| +Audio is where the platforms part, and not by choice. WebRTC's iOS slices |
| +ship `RTCAudioDevice`, `RTCAudioSession` and three more audio headers; the |
| +macOS slice ships two, `RTCAudioSource` and `RTCAudioTrack`. There is no |
| +supported way to enumerate or select an audio device for WebRTC on macOS |
| +through this framework, and no way to supply a custom audio device module |
| +either, because the protocol is not in the slice. (It *is* in the Mac |
| +Catalyst slice, so this is a packaging boundary rather than a platform |
| +limit — but Catalyst would be a poor trade for a real AppKit app.) |
| + |
| +So on iPhone the microphone is a picker, and on Mac the panel reports which |
| +devices macOS has chosen — read from CoreAudio, which is what WebRTC's |
| +audio device module follows — and offers Sound settings. Naming them is |
| +worth doing on its own: the first thing someone needs when a call sounds |
| +wrong is to know what they are connected to. |
| + |
| +Choosing a Bluetooth microphone on iPhone puts the call on the hands-free |
| +profile, which is narrowband in both directions. That used to happen |
| +silently to every call the moment any headset connected, because |
| +`.allowBluetoothHFP` was set unconditionally; it is now set only while a |
| +Bluetooth input is the chosen one, and the panel says what it costs. |
| + |
| +The **audio readout** separates the two ways received audio goes wrong, |
| +which is the distinction a person reporting "it growled" cannot make: |
| +concealed samples are loss and sound like dropouts, while inserted and |
| +removed samples are the jitter buffer rescaling the signal to track the |
| +sender's clock — and a stretched voice is what a growl is. On macOS it is |
| +the only instrument there is, since a bad device cannot be ruled out by |
| +switching away from it. |
| + |
| Members can use **Share** in the call to retrieve the current invitation, copy |
| it, or open the system sharing menu. This needs the server's |
| `POST /m/{sqid}/share` endpoint and the encrypted-invitation migration. |
diff --git a/native/apple/Sources/CallsInterop/Interop.swift b/native/apple/Sources/CallsInterop/Interop.swift| index bedb8e0..c358db5 100644 |
| --- a/native/apple/Sources/CallsInterop/Interop.swift |
| +++ b/native/apple/Sources/CallsInterop/Interop.swift |
| @@ -56,6 +56,14 @@ struct Input: Decodable { |
| print("{\"invitationReady\":\(ready),\"invitationRefused\":\(refused)}") |
| fflush(stdout) |
| } |
| + if command.hasPrefix("camera ") { |
| + let id = String(command.dropFirst("camera ".count)) |
| + await call.useCamera(id) |
| + let now = await call.currentCamera ?? "" |
| + let note = await call.deviceNote |
| + print("{\"camera\":\"\(now)\",\"cameraRefused\":\(note.isEmpty ? "false" : "true")}") |
| + fflush(stdout) |
| + } |
| if command == "share-snapshot" { |
| let saved = await preview?.saveInvitation(call: call) ?? false |
| print("{\"invitationSnapshotSaved\":\(saved)}") |
diff --git a/native/apple/Sources/CallsInterop/SyntheticVideo.swift b/native/apple/Sources/CallsInterop/SyntheticVideo.swift| index 5f3742e..2eba113 100644 |
| --- a/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| +++ b/native/apple/Sources/CallsInterop/SyntheticVideo.swift |
| @@ -7,6 +7,32 @@ import WebRTC |
| private var source: RTCVideoSource? |
| private var syntheticCapturer: RTCVideoCapturer? |
| private var task: Task<Void, Never>? |
| + // Two cameras, differing only in frame size, so that a switch is |
| + // something the far side can *see* rather than something this side |
| + // merely claims. There is no camera in CI; what the gate can still hold |
| + // is the plumbing — that a switch is serialised, that the track and its |
| + // m-line survive it, and that frames keep arriving across it. |
| + private var chosen = SyntheticVideo.first |
| + nonisolated static let first = "synthetic-320" |
| + nonisolated static let second = "synthetic-480" |
| + private var width: Int { chosen == SyntheticVideo.second ? 480 : 320 } |
| + private var height: Int { chosen == SyntheticVideo.second ? 360 : 240 } |
| + |
| + nonisolated var cameras: [CameraDevice] { |
| + [ |
| + CameraDevice(id: SyntheticVideo.first, name: "Synthetic 320x240"), |
| + CameraDevice(id: SyntheticVideo.second, name: "Synthetic 480x360"), |
| + ] |
| + } |
| + var currentCamera: String? { chosen } |
| + func use(camera id: String) async throws { |
| + guard cameras.contains(where: { $0.id == id }) else { |
| + throw NSError( |
| + domain: "Calls", code: 4, |
| + userInfo: [NSLocalizedDescriptionKey: "That camera is no longer available."]) |
| + } |
| + chosen = id |
| + } |
| func start(source: RTCVideoSource) async throws { |
| self.source = source |
| syntheticCapturer = RTCVideoCapturer(delegate: source) |
| @@ -27,7 +53,7 @@ import WebRTC |
| guard let source, let syntheticCapturer else { return } |
| var pixel: CVPixelBuffer? |
| CVPixelBufferCreate( |
| - kCFAllocatorDefault, 320, 240, kCVPixelFormatType_32BGRA, |
| + kCFAllocatorDefault, width, height, kCVPixelFormatType_32BGRA, |
| [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary, &pixel) |
| guard let pixel else { return } |
| CVPixelBufferLockBaseAddress(pixel, []) |
diff --git a/native/apple/Sources/CallsKit/Audio.swift b/native/apple/Sources/CallsKit/Audio.swift| new file mode 100644 |
| index 0000000..495036d |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/Audio.swift |
| @@ -0,0 +1,168 @@ |
| +import AVFoundation |
| +import Foundation |
| + |
| +#if os(macOS) |
| + import CoreAudio |
| +#endif |
| + |
| +#if os(iOS) |
| + import WebRTC |
| +#endif |
| + |
| +/// One audio device the person can be on. `selectable` is the whole |
| +/// platform difference in one flag: on iPhone a route can be chosen, and |
| +/// on Mac it can only be reported. |
| +public struct AudioDevice: Identifiable, Hashable, Sendable { |
| + public let id: String |
| + public let name: String |
| + public let selectable: Bool |
| + public init(id: String, name: String, selectable: Bool) { |
| + self.id = id |
| + self.name = name |
| + self.selectable = selectable |
| + } |
| +} |
| + |
| +/// The audio devices a call is on, and — where the platform allows it — |
| +/// the ability to change them. |
| +/// |
| +/// The asymmetry here is not a design choice. WebRTC's macOS slice ships |
| +/// two audio headers, RTCAudioSource and RTCAudioTrack; the iOS slices |
| +/// ship five, including RTCAudioDevice and RTCAudioSession. There is no |
| +/// supported way to enumerate or select an audio device for WebRTC on |
| +/// macOS through this framework, and no way to supply a custom audio |
| +/// device module either, because the protocol that would be implemented |
| +/// is not in the slice. (It *is* in the Mac Catalyst slice, so this is a |
| +/// packaging boundary rather than a platform impossibility — but a |
| +/// Catalyst build would be a poor trade for a real AppKit app.) |
| +/// |
| +/// So on Mac this reports what macOS has chosen, read straight from |
| +/// CoreAudio, and offers the Sound settings pane to change it. Naming the |
| +/// device is worth doing on its own: the first thing a person needs when |
| +/// a call sounds wrong is to know what they are actually connected to, |
| +/// and until now the app could not tell them. |
| +@MainActor public final class AudioRoutes: ObservableObject { |
| + @Published public private(set) var inputs: [AudioDevice] = [] |
| + @Published public private(set) var currentInput: String? |
| + @Published public private(set) var outputName = "" |
| + /// True where inputs can actually be chosen. The UI shows a picker when |
| + /// it is set and a plain line of text when it is not, rather than a |
| + /// disabled control that looks like something is broken. |
| + public let selectable: Bool |
| + |
| + public init() { |
| + #if os(iOS) |
| + selectable = true |
| + #else |
| + selectable = false |
| + #endif |
| + refresh() |
| + } |
| + |
| + public func refresh() { |
| + #if os(iOS) |
| + let session = AVAudioSession.sharedInstance() |
| + inputs = (session.availableInputs ?? []).map { |
| + AudioDevice(id: $0.uid, name: $0.portName, selectable: true) |
| + } |
| + currentInput = session.currentRoute.inputs.first?.uid |
| + outputName = session.currentRoute.outputs.first?.portName ?? "" |
| + #else |
| + inputs = macDevice(input: true).map { [$0] } ?? [] |
| + currentInput = inputs.first?.id |
| + outputName = macDevice(input: false)?.name ?? "" |
| + #endif |
| + } |
| + |
| + #if os(iOS) |
| + /// Chooses the microphone, and decides the Bluetooth question with it. |
| + /// |
| + /// `.allowBluetoothHFP` is what lets a Bluetooth microphone be used at |
| + /// all, and it is also what drags the whole call down to the 8 kHz |
| + /// hands-free profile the moment a headset connects — narrowband in |
| + /// both directions, for a call that would otherwise be wideband. Left |
| + /// on unconditionally, as it was, every pair of AirPods silently cost |
| + /// the call its audio quality and nothing said so. |
| + /// |
| + /// So the category is rebuilt around the choice rather than set once: |
| + /// HFP is allowed only while a Bluetooth input is the chosen one, and |
| + /// otherwise the route is A2DP for output with the device's own |
| + /// microphone, which is both wideband and usually the better |
| + /// microphone anyway. Picking the headset is then an informed trade |
| + /// someone made, not one made for them. |
| + public func use(input id: String) throws { |
| + let session = AVAudioSession.sharedInstance() |
| + let port = (session.availableInputs ?? []).first { $0.uid == id } |
| + guard let port else { |
| + throw NSError( |
| + domain: "Calls", code: 5, |
| + userInfo: [NSLocalizedDescriptionKey: "That microphone is no longer available."]) |
| + } |
| + let audio = RTCAudioSession.sharedInstance() |
| + audio.lockForConfiguration() |
| + defer { audio.unlockForConfiguration() } |
| + try audio.setCategory( |
| + AVAudioSession.Category.playAndRecord, |
| + mode: AVAudioSession.Mode.videoChat, |
| + options: AudioRoutes.options(bluetoothInput: AudioRoutes.isBluetooth(port))) |
| + try session.setPreferredInput(port) |
| + refresh() |
| + } |
| + |
| + /// Pure, and deliberately outside the actor: a rule about category |
| + /// options is not state, and the test that holds it has no audio |
| + /// session and no main actor to run on. |
| + /// |
| + /// True for the port types that only reach a microphone over the |
| + /// hands-free profile. A2DP is output-only, so a headset used as a |
| + /// microphone is always one of these. |
| + public nonisolated static func isBluetooth(_ port: AVAudioSessionPortDescription) -> Bool { |
| + port.portType == .bluetoothHFP |
| + } |
| + |
| + /// The category options for a call, given whether its microphone is a |
| + /// Bluetooth one. Pure, so the test can hold the rule without an audio |
| + /// session to configure. |
| + public nonisolated static func options(bluetoothInput: Bool) -> AVAudioSession.CategoryOptions { |
| + bluetoothInput |
| + ? [.defaultToSpeaker, .allowBluetoothHFP, .allowBluetoothA2DP] |
| + : [.defaultToSpeaker, .allowBluetoothA2DP] |
| + } |
| + #endif |
| + |
| + #if os(macOS) |
| + /// The system's default input or output device, by name. CoreAudio is |
| + /// the only way to ask: the WebRTC slice cannot say, and this is |
| + /// precisely the device it will be using, because its audio device |
| + /// module follows the system default. |
| + private func macDevice(input: Bool) -> AudioDevice? { |
| + var id = AudioDeviceID(0) |
| + var size = UInt32(MemoryLayout<AudioDeviceID>.size) |
| + var address = AudioObjectPropertyAddress( |
| + mSelector: input |
| + ? kAudioHardwarePropertyDefaultInputDevice : kAudioHardwarePropertyDefaultOutputDevice, |
| + mScope: kAudioObjectPropertyScopeGlobal, |
| + mElement: kAudioObjectPropertyElementMain) |
| + guard |
| + AudioObjectGetPropertyData( |
| + AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, &id) == noErr, |
| + id != kAudioObjectUnknown |
| + else { return nil } |
| + // Unmanaged, not CFString: CoreAudio writes a retained reference into |
| + // this buffer, and pointing at a managed CFString both trips ARC's |
| + // "object reference in a raw buffer" warning and leaves the +1 |
| + // unbalanced. takeRetainedValue consumes it exactly once. |
| + var name: Unmanaged<CFString>? |
| + var nameSize = UInt32(MemoryLayout<Unmanaged<CFString>?>.size) |
| + var nameAddress = AudioObjectPropertyAddress( |
| + mSelector: kAudioObjectPropertyName, |
| + mScope: kAudioObjectPropertyScopeGlobal, |
| + mElement: kAudioObjectPropertyElementMain) |
| + guard |
| + AudioObjectGetPropertyData(id, &nameAddress, 0, nil, &nameSize, &name) == noErr, |
| + let value = name?.takeRetainedValue() |
| + else { return nil } |
| + return AudioDevice(id: String(id), name: value as String, selectable: false) |
| + } |
| + #endif |
| +} |
diff --git a/native/apple/Sources/CallsKit/AudioHealth.swift b/native/apple/Sources/CallsKit/AudioHealth.swift| new file mode 100644 |
| index 0000000..fbfc83b |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/AudioHealth.swift |
| @@ -0,0 +1,79 @@ |
| +import Foundation |
| + |
| +/// What the received audio actually sounded like, as far as the counters |
| +/// can say — the distinction a person reporting "it growled" cannot make |
| +/// for us. |
| +/// |
| +/// Two failures wear the same description. Loss is the decoder inventing |
| +/// audio that never arrived, and sounds like dropouts and robotic |
| +/// stuttering. Drift is the jitter buffer stretching or compressing the |
| +/// signal because the sender's clock and this device's clock disagree, |
| +/// and stretching a voice is what a growl *is*. Bytes arriving tells you |
| +/// neither. |
| +public struct AudioHealth: Equatable, Sendable { |
| + /// Samples the decoder invented, against samples it received. |
| + public let concealed: UInt64 |
| + public let total: UInt64 |
| + /// Samples the jitter buffer added or dropped to track the sender's |
| + /// clock. Either one running is drift; which one says which way. |
| + public let inserted: UInt64 |
| + public let removed: UInt64 |
| + public let jitterBufferMs: UInt64 |
| + |
| + public init(concealed: UInt64, total: UInt64, inserted: UInt64, removed: UInt64, jitterBufferMs: UInt64) { |
| + self.concealed = concealed |
| + self.total = total |
| + self.inserted = inserted |
| + self.removed = removed |
| + self.jitterBufferMs = jitterBufferMs |
| + } |
| + |
| + public init(_ stats: [String: UInt64]) { |
| + self.init( |
| + concealed: stats["concealedSamples"] ?? 0, |
| + total: stats["totalSamplesReceived"] ?? 0, |
| + inserted: stats["insertedSamplesForDeceleration"] ?? 0, |
| + removed: stats["removedSamplesForAcceleration"] ?? 0, |
| + jitterBufferMs: stats["jitterBufferDelayMs"] ?? 0) |
| + } |
| + |
| + /// The fraction of received audio the decoder had to invent, 0 to 1. |
| + /// Zero when nothing has been received, because no audio is not clean |
| + /// audio and a reader should not be told otherwise by a division. |
| + public var concealmentRate: Double { |
| + total == 0 ? 0 : Double(concealed) / Double(total) |
| + } |
| + |
| + /// The fraction of received audio the jitter buffer stretched or |
| + /// squeezed. Both directions count: what matters is that the signal was |
| + /// rescaled, not which way it went. |
| + public var driftRate: Double { |
| + total == 0 ? 0 : Double(inserted + removed) / Double(total) |
| + } |
| + |
| + /// Nothing received yet. Distinct from healthy, and said differently. |
| + public var silent: Bool { total == 0 } |
| + |
| + /// One plain sentence, for the person who has opened this panel because |
| + /// something sounds wrong. Thresholds are deliberately blunt: this is a |
| + /// pointer towards which half of the problem to look at, not a |
| + /// measurement anyone should tune against. |
| + /// |
| + /// Drift is named first when both are up. Loss is common, usually |
| + /// transient and widely understood; sustained drift is rarer, is what |
| + /// makes a voice sound wrong rather than broken, and is the one nobody |
| + /// thinks to look for. |
| + public var summary: String { |
| + if silent { return "No audio received yet." } |
| + let loss = Int((concealmentRate * 100).rounded()) |
| + let drift = Int((driftRate * 100).rounded()) |
| + if driftRate >= 0.02 { |
| + return "Audio is being resampled to track the sender's clock (\(drift)%). " |
| + + "That is what makes a voice sound slowed or growling." |
| + } |
| + if concealmentRate >= 0.05 { |
| + return "\(loss)% of audio did not arrive and was filled in. That sounds like dropouts." |
| + } |
| + return "Audio is arriving cleanly (\(loss)% filled in, \(drift)% resampled)." |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index aed8006..95bd93d 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -41,6 +41,19 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| @Published public private(set) var invitationURL: URL? |
| @Published public private(set) var invitationError = "" |
| @Published public private(set) var loadingInvitation = false |
| + /// What the last device change had to say, and usually nothing. |
| + @Published public private(set) var deviceNote = "" |
| + /// The audio-health readout, once a call has been up long enough for |
| + /// one. Nil means no sample yet, which is not the same as healthy. |
| + @Published public private(set) var audioHealth: AudioHealth? |
| + /// The camera in use. Published rather than read through the capture on |
| + /// demand, because a picker bound to a plain computed property never |
| + /// learns that a switch finished. |
| + @Published public private(set) var currentCamera: String? |
| + /// The routes this platform can report, and on iPhone choose between. |
| + public let audio = AudioRoutes() |
| + private var cameraWork: Task<Void, Never>? |
| + private var cameraSwitches = 0 |
| fileprivate var epoch = UUID() |
| private var bridge: BridgeClient? |
| private var events: Events? |
| @@ -84,6 +97,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| try await media.prepare(capture: capture) |
| guard epoch == joiningEpoch, bridge === client else { throw CancellationError() } |
| localTrack = media.localVideo |
| + currentCamera = media.currentCamera |
| + audio.refresh() |
| try client.start() |
| } |
| public func close() { |
| @@ -100,6 +115,11 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| invitationURL = nil |
| invitationError = "" |
| loadingInvitation = false |
| + deviceNote = "" |
| + audioHealth = nil |
| + currentCamera = nil |
| + cameraWork?.cancel() |
| + cameraWork = nil |
| if let old { DispatchQueue.global().async { old.close() } } |
| } |
| public func leave() async { |
| @@ -117,6 +137,71 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| cameraOff.toggle() |
| media.localVideo?.isEnabled = !cameraOff |
| } |
| + |
| + // --------------------------------------------------------------- |
| + // Devices |
| + // --------------------------------------------------------------- |
| + |
| + /// The cameras available, and the one in use. Read through the running |
| + /// capture rather than from AVFoundation directly, so what the picker |
| + /// offers is what the call could actually switch to. |
| + public var cameras: [CameraDevice] { media.cameras } |
| + |
| + /// Switches camera mid-call. Serialised: two picks are two capture |
| + /// restarts on one device, and run at once the survivor is whichever |
| + /// session happened to open last, which is not the one the person |
| + /// chose. The counter decides who wins; the queue decides that only |
| + /// one runs. |
| + public func useCamera(_ id: String) async { |
| + cameraSwitches += 1 |
| + let mine = cameraSwitches |
| + let previous = cameraWork |
| + let work = Task { @MainActor [weak self] in |
| + await previous?.value |
| + guard let self, mine == self.cameraSwitches else { return } |
| + do { |
| + try await self.media.use(camera: id) |
| + self.deviceNote = "" |
| + } catch is CancellationError { |
| + } catch { |
| + // The call is left without a picture rather than silently on the |
| + // old camera: the old session has to be released before the new |
| + // device can be opened. Saying so is the least this can do. |
| + self.deviceNote = error.localizedDescription |
| + } |
| + // Read back rather than assume: a refused switch left the capture |
| + // wherever it actually is, and the picker has to follow the camera |
| + // and not the intention. |
| + self.currentCamera = self.media.currentCamera |
| + } |
| + cameraWork = work |
| + await work.value |
| + } |
| + |
| + /// Records something a device change had to say. The panel has no |
| + /// place of its own to put an error, and a route that would not change |
| + /// belongs beside the control that would not change it. |
| + public func noteDevice(_ text: String) { deviceNote = text } |
| + |
| + /// Polls the audio-health counters while a call is up. |
| + /// |
| + /// This exists because of a call on 2026-09-17 that came back as an |
| + /// unintelligible growl, and because nothing in this app recorded |
| + /// anything about it — no log, no counters, no artefact. A reproduction |
| + /// would have produced nothing either. On macOS it is also the *only* |
| + /// instrument available, since the framework's macOS slice has no audio |
| + /// device API and a bad device cannot be ruled out by switching away |
| + /// from it. |
| + /// |
| + /// Five seconds because this is a readout a person glances at when |
| + /// something sounds wrong, not a time series. |
| + public func watchAudio() async { |
| + while !Task.isCancelled { |
| + try? await Task.sleep(nanoseconds: 5_000_000_000) |
| + guard !Task.isCancelled, bridge != nil else { return } |
| + audioHealth = AudioHealth(await media.stats()) |
| + } |
| + } |
| public func retryJoin() { send(type: "join", to: "", gen: 0, payload: "") } |
| public func loadInvitation() async { |
| guard let client = bridge, !loadingInvitation else { return } |
diff --git a/native/apple/Sources/CallsKit/Capture.swift b/native/apple/Sources/CallsKit/Capture.swift| index 533d30c..8f4e51f 100644 |
| --- a/native/apple/Sources/CallsKit/Capture.swift |
| +++ b/native/apple/Sources/CallsKit/Capture.swift |
| @@ -2,16 +2,53 @@ import AVFoundation |
| import Foundation |
| import WebRTC |
| +/// One camera the person can choose, named the way a menu wants it. |
| +/// |
| +/// The id is `AVCaptureDevice.uniqueID`, which survives unplugging and |
| +/// replugging the same camera, so a remembered choice means the same |
| +/// device next time rather than whatever now sits at that index. |
| +public struct CameraDevice: Identifiable, Hashable, Sendable { |
| + public let id: String |
| + public let name: String |
| + public init(id: String, name: String) { |
| + self.id = id |
| + self.name = name |
| + } |
| +} |
| + |
| // Capture is injected separately from signaling and rendering. App targets |
| // provide the platform camera; the interop executable provides synthetic frames. |
| @MainActor public protocol VideoCapture: AnyObject { |
| func start(source: RTCVideoSource) async throws |
| func stop() |
| + |
| + /// The cameras this capture can offer, and which one it is on. A capture |
| + /// with nothing to choose between answers an empty list, and the picker |
| + /// does not appear — which is what the interop executable's synthetic |
| + /// frames want, and why these carry defaults rather than being required. |
| + var cameras: [CameraDevice] { get } |
| + var currentCamera: String? { get } |
| + func use(camera id: String) async throws |
| +} |
| + |
| +extension VideoCapture { |
| + public var cameras: [CameraDevice] { [] } |
| + public var currentCamera: String? { nil } |
| + public func use(camera id: String) async throws {} |
| } |
| @MainActor final class CameraCapture: VideoCapture { |
| private var capturer: RTCCameraVideoCapturer? |
| + private var source: RTCVideoSource? |
| private var epoch = UUID() |
| + private(set) var currentCamera: String? |
| + |
| + var cameras: [CameraDevice] { |
| + RTCCameraVideoCapturer.captureDevices().map { |
| + CameraDevice(id: $0.uniqueID, name: $0.localizedName) |
| + } |
| + } |
| + |
| func start(source: RTCVideoSource) async throws { |
| let preparing = epoch |
| for kind in [AVMediaType.video, .audio] { |
| @@ -32,7 +69,54 @@ import WebRTC |
| #else |
| let preferredDevice = devices.first |
| #endif |
| - guard let device = preferredDevice ?? devices.first, |
| + guard let device = preferredDevice ?? devices.first else { |
| + throw NSError( |
| + domain: "Calls", code: 3, userInfo: [NSLocalizedDescriptionKey: "No camera is available."]) |
| + } |
| + self.source = source |
| + let capture = RTCCameraVideoCapturer(delegate: source) |
| + capturer = capture |
| + try await run(capture, device: device, preparing: preparing) |
| + } |
| + |
| + /// Switches the camera without touching the track. |
| + /// |
| + /// The capturer keeps delivering into the same RTCVideoSource, so the |
| + /// RTCVideoTrack every peer connection is already sending does not |
| + /// change and no renegotiation happens — the native counterpart of the |
| + /// browser's replaceTrack, and for the same reason: an m-line that |
| + /// exists and keeps existing is the one thing perfect negotiation never |
| + /// has to hear about. |
| + /// |
| + /// A failed switch leaves the call without a picture rather than |
| + /// silently on the old camera, because the old capture session has to |
| + /// be stopped before the new device can be opened — two AVCaptureDevices |
| + /// cannot hold the same camera, and on a Mac they often *are* the same |
| + /// camera behind two names. The error says so; the picker puts itself |
| + /// back. |
| + func use(camera id: String) async throws { |
| + guard let capture = capturer, source != nil else { return } |
| + guard id != currentCamera else { return } |
| + guard let device = RTCCameraVideoCapturer.captureDevices().first(where: { $0.uniqueID == id }) |
| + else { |
| + throw NSError( |
| + domain: "Calls", code: 4, |
| + userInfo: [NSLocalizedDescriptionKey: "That camera is no longer available."]) |
| + } |
| + epoch = UUID() |
| + let preparing = epoch |
| + await capture.stopCapture() |
| + guard preparing == epoch else { throw CancellationError() } |
| + try await run(capture, device: device, preparing: preparing) |
| + } |
| + |
| + /// The half start and use share: pick the format nearest 640 wide, cap |
| + /// the frame rate at 30, and start. Held together so a camera chosen |
| + /// mid-call is opened exactly the way the one chosen at join was. |
| + private func run(_ capture: RTCCameraVideoCapturer, device: AVCaptureDevice, preparing: UUID) |
| + async throws |
| + { |
| + guard |
| let format = RTCCameraVideoCapturer.supportedFormats(for: device).min(by: { |
| abs(Int(CMVideoFormatDescriptionGetDimensions($0.formatDescription).width) - 640) |
| < abs(Int(CMVideoFormatDescriptionGetDimensions($1.formatDescription).width) - 640) |
| @@ -41,8 +125,6 @@ import WebRTC |
| throw NSError( |
| domain: "Calls", code: 3, userInfo: [NSLocalizedDescriptionKey: "No camera is available."]) |
| } |
| - let capture = RTCCameraVideoCapturer(delegate: source) |
| - capturer = capture |
| try await withCheckedThrowingContinuation { (continuation: CheckedContinuation<Void, Error>) in |
| capture.startCapture( |
| with: device, format: format, |
| @@ -56,11 +138,14 @@ import WebRTC |
| await capture.stopCapture() |
| throw CancellationError() |
| } |
| + currentCamera = device.uniqueID |
| } |
| func stop() { |
| epoch = UUID() |
| capturer?.stopCapture() |
| capturer = nil |
| + source = nil |
| + currentCamera = nil |
| } |
| } |
diff --git a/native/apple/Sources/CallsKit/DevicePanel.swift b/native/apple/Sources/CallsKit/DevicePanel.swift| new file mode 100644 |
| index 0000000..ab3c609 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/DevicePanel.swift |
| @@ -0,0 +1,154 @@ |
| +import SwiftUI |
| + |
| +#if os(macOS) |
| + import AppKit |
| +#endif |
| + |
| +/// The camera, microphone and speaker a call is on — chosen where the |
| +/// platform allows it, reported where it does not. |
| +/// |
| +/// One view for both apps, because the thing a person wants here is the |
| +/// same on a Mac and an iPhone even though what they can do about it is |
| +/// not. On iPhone every row is a choice. On Mac the camera is a choice |
| +/// and audio is a statement of fact with a way out to Sound settings, |
| +/// because WebRTC's macOS slice carries no audio device API at all. |
| +/// |
| +/// The audio readout at the bottom is the reason this panel opens at |
| +/// all, most of the time: someone has noticed the call sounds wrong. It |
| +/// says which half of "wrong" the counters are seeing, in a sentence, |
| +/// rather than making them guess between a bad network and a bad device. |
| +@MainActor public struct DevicePanel: View { |
| + @ObservedObject private var call: Call |
| + @ObservedObject private var audio: AudioRoutes |
| + @State private var camera: String? |
| + |
| + public init(call: Call) { |
| + self.call = call |
| + self.audio = call.audio |
| + } |
| + |
| + public var body: some View { |
| + VStack(alignment: .leading, spacing: 16) { |
| + if call.cameras.count > 1 { |
| + picker( |
| + "Camera", selection: $camera, |
| + options: call.cameras.map { ($0.id, $0.name) }, identifier: "cameraPicker") |
| + } else if let only = call.cameras.first { |
| + row("Camera", only.name) |
| + } |
| + |
| + #if os(iOS) |
| + if audio.inputs.count > 1 { |
| + microphonePicker |
| + } else if let only = audio.inputs.first { |
| + row("Microphone", only.name) |
| + } |
| + if !audio.outputName.isEmpty { row("Speaker", audio.outputName) } |
| + #else |
| + macAudio |
| + #endif |
| + |
| + if !call.deviceNote.isEmpty { |
| + Label(call.deviceNote, systemImage: "exclamationmark.circle.fill") |
| + .font(.callout).foregroundStyle(.red) |
| + .fixedSize(horizontal: false, vertical: true) |
| + .accessibilityLabel("Device problem: \(call.deviceNote)") |
| + } |
| + |
| + Divider() |
| + audioReadout |
| + } |
| + .padding(16) |
| + .onAppear { |
| + camera = call.currentCamera |
| + audio.refresh() |
| + } |
| + // The picker follows the camera, not the intention: a switch that was |
| + // refused leaves the capture where it was, and the control has to go |
| + // back rather than keep showing a device the call is not on. |
| + .onChange(of: call.currentCamera) { _, now in camera = now } |
| + .onChange(of: camera) { was, now in |
| + guard let now, now != was, now != call.currentCamera else { return } |
| + Task { await call.useCamera(now) } |
| + } |
| + } |
| + |
| + #if os(iOS) |
| + private var microphonePicker: some View { |
| + VStack(alignment: .leading, spacing: 6) { |
| + picker( |
| + "Microphone", selection: Binding( |
| + get: { audio.currentInput }, |
| + set: { id in |
| + guard let id else { return } |
| + do { try audio.use(input: id) } catch { call.noteDevice(error.localizedDescription) } |
| + }), |
| + options: audio.inputs.map { ($0.id, $0.name) }, identifier: "microphonePicker") |
| + // Said before the choice rather than after it: the cost of a |
| + // Bluetooth microphone is that the whole call drops to the |
| + // hands-free profile, which is narrowband both ways. Someone who |
| + // picks it should be picking it knowingly. |
| + Text( |
| + "A Bluetooth microphone puts the call on the hands-free profile, which is lower quality in both directions." |
| + ) |
| + .font(.footnote).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + } |
| + } |
| + #endif |
| + |
| + #if os(macOS) |
| + private var macAudio: some View { |
| + VStack(alignment: .leading, spacing: 8) { |
| + if let input = audio.inputs.first { row("Microphone", input.name) } |
| + if !audio.outputName.isEmpty { row("Speaker", audio.outputName) } |
| + Text("Calls uses whichever microphone and speaker macOS is set to.") |
| + .font(.footnote).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + Button("Open Sound settings", systemImage: "gearshape") { |
| + if let url = URL(string: "x-apple.systempreferences:com.apple.Sound-Settings.extension") { |
| + NSWorkspace.shared.open(url) |
| + } |
| + } |
| + .buttonStyle(.bordered) |
| + .accessibilityIdentifier("openSoundSettings") |
| + } |
| + } |
| + #endif |
| + |
| + private var audioReadout: some View { |
| + VStack(alignment: .leading, spacing: 4) { |
| + Text("Audio health").font(.headline) |
| + if let health = call.audioHealth { |
| + Text(health.summary) |
| + .font(.callout).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + .accessibilityIdentifier("audioHealth") |
| + } else { |
| + Text("Measuring…").font(.callout).foregroundStyle(.secondary) |
| + .accessibilityIdentifier("audioHealth") |
| + } |
| + } |
| + } |
| + |
| + private func row(_ label: String, _ value: String) -> some View { |
| + VStack(alignment: .leading, spacing: 2) { |
| + Text(label).font(.headline) |
| + Text(value).font(.callout).foregroundStyle(.secondary).textSelection(.enabled) |
| + }.accessibilityElement(children: .combine) |
| + } |
| + |
| + private func picker( |
| + _ label: String, selection: Binding<String?>, options: [(String, String)], identifier: String |
| + ) -> some View { |
| + Picker(label, selection: selection) { |
| + ForEach(options, id: \.0) { option in |
| + Text(option.1).tag(Optional(option.0)) |
| + } |
| + } |
| + .accessibilityIdentifier(identifier) |
| + #if os(macOS) |
| + .pickerStyle(.menu) |
| + #endif |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/MacCallView.swift b/native/apple/Sources/CallsKit/MacCallView.swift| index 04ac523..93a201c 100644 |
| --- a/native/apple/Sources/CallsKit/MacCallView.swift |
| +++ b/native/apple/Sources/CallsKit/MacCallView.swift |
| @@ -8,6 +8,7 @@ |
| @State private var showingChat: Bool |
| @State private var selectedPeer: String? |
| @State private var readChatCount = 0 |
| + @State private var showingDevices = false |
| @StateObject private var chatWindow = MacChatWindow() |
| public init( |
| @@ -50,6 +51,19 @@ |
| } |
| HStack(alignment: .top, spacing: 16) { |
| stage.frame(maxWidth: .infinity, maxHeight: .infinity) |
| + if showingDevices { |
| + VStack(spacing: 0) { |
| + HStack { |
| + Text("Devices").font(.headline) |
| + Spacer() |
| + Button("Hide devices", systemImage: "xmark") { showingDevices = false } |
| + .labelStyle(.iconOnly).buttonStyle(.borderless).help("Hide devices") |
| + }.padding(.horizontal, 16).padding(.top, 16) |
| + DevicePanel(call: call) |
| + Spacer(minLength: 0) |
| + }.frame(width: 300) |
| + .background(.quaternary.opacity(0.4), in: RoundedRectangle(cornerRadius: 16)) |
| + } |
| if showingChat { |
| VStack(spacing: 14) { |
| HStack { |
| @@ -80,6 +94,7 @@ |
| .onAppear { |
| if showingChat { readChatCount = call.receivedChatCount } |
| } |
| + .task { await call.watchAudio() } |
| .onChange(of: call.receivedChatCount) { _, count in |
| if showingChat || chatWindow.isOpen { readChatCount = count } |
| } |
| @@ -159,6 +174,10 @@ |
| emphasized: call.cameraOff) |
| ) |
| .help(call.cameraOff ? "Turn camera on" : "Turn camera off") |
| + Button("Devices", systemImage: "slider.horizontal.3") { showingDevices.toggle() } |
| + .buttonStyle(CallActionStyle(color: .teal, emphasized: showingDevices)) |
| + .help("Choose your camera, and see which microphone and speaker this call is on") |
| + .accessibilityIdentifier("devices") |
| MeetingShareButton(call: call).buttonStyle(CallActionStyle(color: .indigo)) |
| .help("Share meeting link") |
| Spacer() |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 12224f2..46e73ca 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -39,14 +39,25 @@ import WebRTC |
| let audio = RTCAudioSession.sharedInstance() |
| audio.lockForConfiguration() |
| defer { audio.unlockForConfiguration() } |
| + // Not .allowBluetoothHFP: see AudioRoutes.options. At join nobody |
| + // has chosen a Bluetooth microphone, so the call starts wideband |
| + // and only drops to the hands-free profile if someone asks for a |
| + // Bluetooth input by name. |
| try audio.setCategory( |
| AVAudioSession.Category.playAndRecord, |
| mode: AVAudioSession.Mode.videoChat, |
| - options: [.defaultToSpeaker, .allowBluetoothHFP]) |
| + options: AudioRoutes.options(bluetoothInput: false)) |
| // WebRTC owns activation and interruption recovery as peers start/stop. |
| } |
| #endif |
| } |
| + /// The cameras the running capture can offer, and the one it is on. |
| + /// Empty before prepare, and empty for the interop executable's |
| + /// synthetic frames, which is how the picker knows not to appear. |
| + var cameras: [CameraDevice] { capture?.cameras ?? [] } |
| + var currentCamera: String? { capture?.currentCamera } |
| + func use(camera id: String) async throws { try await capture?.use(camera: id) } |
| + |
| func closePeers() { |
| for (_, p) in peers { p.close() } |
| peers.removeAll() |
| @@ -79,10 +90,37 @@ import WebRTC |
| } |
| func receive(_ message: Message) { if let from = message.from { peers[from]?.receive(message) } } |
| func updateICE(_ config: ICEConfig) { for p in peers.values { p.updateICE(config) } } |
| + /// The five audio-health fields, gathered alongside the byte counts. |
| + /// |
| + /// Bytes say audio arrived; they say nothing about what it sounded like. |
| + /// These five separate the two ways received audio goes wrong, which is |
| + /// the distinction a person reporting "it growled" cannot make for us: |
| + /// |
| + /// - `concealedSamples` against `totalSamplesReceived` is loss. The |
| + /// decoder invented that fraction of the audio because nothing |
| + /// arrived to decode. |
| + /// - `insertedSamplesForDeceleration` and |
| + /// `removedSamplesForAcceleration` are clock drift. The jitter |
| + /// buffer is stretching or compressing the signal to keep up with a |
| + /// sender whose clock runs at a different rate than this device's — |
| + /// and stretching a voice is what a growl *is*. |
| + /// |
| + /// On macOS this is the only instrument there is. The framework's macOS |
| + /// slice ships no audio device API at all, so a bad device cannot be |
| + /// ruled out by switching away from it the way it can on iPhone; the |
| + /// counters are what is left. See DESIGN.md. |
| + static let audioHealthFields = [ |
| + "concealedSamples", "totalSamplesReceived", |
| + "insertedSamplesForDeceleration", "removedSamplesForAcceleration", |
| + ] |
| + |
| func stats() async -> [String: UInt64] { |
| var values: [String: UInt64] = [ |
| "framesDecoded": 0, "audioBytesReceived": 0, "videoBytesSent": 0, "audioBytesSent": 0, |
| "peerConnections": UInt64(peers.count), |
| + "concealedSamples": 0, "totalSamplesReceived": 0, |
| + "insertedSamplesForDeceleration": 0, "removedSamplesForAcceleration": 0, |
| + "jitterBufferDelayMs": 0, |
| ] |
| for peer in peers.values { |
| guard let pc = peer.pc else { continue } |
| @@ -99,6 +137,16 @@ import WebRTC |
| if kind == "audio" { |
| values["audioBytesReceived", default: 0] += |
| (stat.values["bytesReceived"] as? NSNumber)?.uint64Value ?? 0 |
| + for field in MediaEngine.audioHealthFields { |
| + values[field, default: 0] += (stat.values[field] as? NSNumber)?.uint64Value ?? 0 |
| + } |
| + // jitterBufferDelay is seconds as a double, and is cumulative |
| + // against jitterBufferEmittedCount rather than a gauge. It is |
| + // carried in milliseconds because everything else here is an |
| + // integer, and a sub-millisecond buffer is not a fact anyone |
| + // reading this is going to act on. |
| + let delay = (stat.values["jitterBufferDelay"] as? NSNumber)?.doubleValue ?? 0 |
| + values["jitterBufferDelayMs", default: 0] += UInt64((delay * 1000).rounded()) |
| } |
| } |
| if stat.type == "outbound-rtp" { |
diff --git a/native/apple/Tests/CallsKitTests/AudioHealthTests.swift b/native/apple/Tests/CallsKitTests/AudioHealthTests.swift| new file mode 100644 |
| index 0000000..de58c53 |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/AudioHealthTests.swift |
| @@ -0,0 +1,123 @@ |
| +import XCTest |
| + |
| +@testable import CallsKit |
| + |
| +/// AudioHealth is the one part of the device work with real logic in it, |
| +/// and the one that has to be right: it is what a person reads when a |
| +/// call sounds wrong, and on macOS it is the only instrument there is. |
| +final class AudioHealthTests: XCTestCase { |
| + func testNothingReceivedIsSaidPlainlyAndNeverDividedBy() { |
| + let health = AudioHealth(concealed: 0, total: 0, inserted: 0, removed: 0, jitterBufferMs: 0) |
| + XCTAssertTrue(health.silent) |
| + // Not 0% clean: no audio is not clean audio, and a rate computed from |
| + // an empty denominator would say the opposite of the truth. |
| + XCTAssertEqual(health.concealmentRate, 0) |
| + XCTAssertEqual(health.driftRate, 0) |
| + XCTAssertEqual(health.summary, "No audio received yet.") |
| + } |
| + |
| + func testDriftIsNamedAheadOfLossWhenBothAreUp() { |
| + // The growl case. Loss is common, transient and widely understood; |
| + // sustained drift is rarer, is what makes a voice sound *wrong* |
| + // rather than broken, and is the one nobody thinks to look for. With |
| + // both raised, the rarer diagnosis is the useful one to lead with. |
| + let health = AudioHealth( |
| + concealed: 20_000, total: 100_000, inserted: 4_000, removed: 1_000, jitterBufferMs: 300) |
| + XCTAssertEqual(health.driftRate, 0.05, accuracy: 0.0001) |
| + XCTAssertEqual(health.concealmentRate, 0.20, accuracy: 0.0001) |
| + XCTAssertTrue(health.summary.contains("resampled"), health.summary) |
| + XCTAssertTrue(health.summary.contains("growling"), health.summary) |
| + } |
| + |
| + func testLossAloneReadsAsDropoutsAndNotAsDrift() { |
| + let health = AudioHealth( |
| + concealed: 10_000, total: 100_000, inserted: 0, removed: 0, jitterBufferMs: 40) |
| + XCTAssertTrue(health.summary.contains("did not arrive"), health.summary) |
| + XCTAssertTrue(health.summary.contains("dropouts"), health.summary) |
| + XCTAssertFalse(health.summary.contains("growling"), health.summary) |
| + } |
| + |
| + func testAHealthyCallSaysSoWithItsNumbers() { |
| + let health = AudioHealth( |
| + concealed: 500, total: 100_000, inserted: 100, removed: 100, jitterBufferMs: 30) |
| + XCTAssertFalse(health.silent) |
| + XCTAssertTrue(health.summary.contains("cleanly"), health.summary) |
| + // The numbers ride along even when the verdict is good: someone |
| + // comparing two calls needs them, and rounding to zero is honest at |
| + // this resolution. |
| + XCTAssertTrue(health.summary.contains("0%"), health.summary) |
| + } |
| + |
| + func testDriftCountsBothDirections() { |
| + // Inserting and removing are the same fault seen from two sides — the |
| + // signal was rescaled either way. A reading that counted only one |
| + // would miss a sender whose clock runs fast. |
| + let fast = AudioHealth(concealed: 0, total: 1_000, inserted: 0, removed: 50, jitterBufferMs: 0) |
| + let slow = AudioHealth(concealed: 0, total: 1_000, inserted: 50, removed: 0, jitterBufferMs: 0) |
| + XCTAssertEqual(fast.driftRate, slow.driftRate) |
| + XCTAssertTrue(fast.summary.contains("resampled"), fast.summary) |
| + XCTAssertTrue(slow.summary.contains("resampled"), slow.summary) |
| + } |
| + |
| + func testItReadsTheStatsDictionaryTheEngineActuallyProduces() { |
| + // The field names are libwebrtc's, spelled once in MediaEngine and |
| + // once here. This is the test that fails if they stop agreeing. |
| + let health = AudioHealth([ |
| + "concealedSamples": 7, "totalSamplesReceived": 70, |
| + "insertedSamplesForDeceleration": 3, "removedSamplesForAcceleration": 4, |
| + "jitterBufferDelayMs": 120, |
| + ]) |
| + XCTAssertEqual(health.concealed, 7) |
| + XCTAssertEqual(health.total, 70) |
| + XCTAssertEqual(health.inserted, 3) |
| + XCTAssertEqual(health.removed, 4) |
| + XCTAssertEqual(health.jitterBufferMs, 120) |
| + for field in MediaEngine.audioHealthFields { |
| + XCTAssertTrue( |
| + ["concealedSamples", "totalSamplesReceived", "insertedSamplesForDeceleration", |
| + "removedSamplesForAcceleration"].contains(field), |
| + "MediaEngine gathers \(field), which AudioHealth does not read") |
| + } |
| + } |
| + |
| + func testAMissingFieldIsZeroAndNotACrash() { |
| + // An older or newer libwebrtc that stops reporting one of these |
| + // should cost the readout a number, not the call a panel. |
| + let health = AudioHealth(["totalSamplesReceived": 1_000]) |
| + XCTAssertEqual(health.concealed, 0) |
| + XCTAssertFalse(health.silent) |
| + XCTAssertTrue(health.summary.contains("cleanly"), health.summary) |
| + } |
| +} |
| + |
| +/// AudioRoutes on macOS is read-only, and everything it reports comes |
| +/// from CoreAudio rather than from WebRTC — which has no audio device API |
| +/// in its macOS slice at all. These run against the real machine, because |
| +/// the thing worth testing is that the CoreAudio calls are right. |
| +final class AudioRoutesTests: XCTestCase { |
| + @MainActor func testMacReportsTheSystemDevicesAndOffersNoChoice() throws { |
| + #if os(macOS) |
| + let routes = AudioRoutes() |
| + XCTAssertFalse( |
| + routes.selectable, |
| + "macOS cannot select an audio device for WebRTC; the UI must not offer a picker") |
| + // A machine with no input at all is possible, so this asserts the |
| + // shape rather than the presence: whatever is reported must be |
| + // named, because an unnamed device tells a person nothing and the |
| + // whole point of the Mac panel is saying what you are connected to. |
| + for device in routes.inputs { |
| + XCTAssertFalse(device.name.isEmpty, "a reported device must have a name") |
| + XCTAssertFalse(device.selectable) |
| + } |
| + if !routes.outputName.isEmpty { |
| + XCTAssertFalse(routes.outputName.hasPrefix("<"), "output name should be a real name") |
| + } |
| + // Reading twice must give the same answer and must not crash on the |
| + // retained CFString CoreAudio hands back. |
| + let again = AudioRoutes() |
| + XCTAssertEqual(again.outputName, routes.outputName) |
| + #else |
| + throw XCTSkip("macOS only") |
| + #endif |
| + } |
| +} |
diff --git a/native/ios/Sources/CallsPhone.swift b/native/ios/Sources/CallsPhone.swift| index cfc283b..d094e07 100644 |
| --- a/native/ios/Sources/CallsPhone.swift |
| +++ b/native/ios/Sources/CallsPhone.swift |
| @@ -41,6 +41,7 @@ struct PhoneCall: View { |
| @ObservedObject var call: Call |
| let leave: () -> Void |
| @State private var showingChat = false |
| + @State private var showingDevices = false |
| @State private var readChatCount = 0 |
| @State private var grid = false |
| @State private var selectedPeer: String? |
| @@ -148,6 +149,9 @@ struct PhoneCall: View { |
| systemImage: call.cameraOff ? "video.slash.fill" : "video.fill" |
| ) { call.toggleCamera() } |
| .buttonStyle(CallActionStyle(color: .blue, emphasized: call.cameraOff)) |
| + Button("Devices", systemImage: "slider.horizontal.3") { showingDevices = true } |
| + .buttonStyle(CallActionStyle(color: .teal)) |
| + .accessibilityIdentifier("devices") |
| MeetingShareButton(call: call).buttonStyle(CallActionStyle(color: .indigo)) |
| Spacer(minLength: 0) |
| Button(chatButtonTitle, systemImage: "bubble.left.and.bubble.right.fill") { |
| @@ -164,6 +168,18 @@ struct PhoneCall: View { |
| .buttonStyle(CallActionStyle(color: .red, emphasized: true)) |
| }.padding(16).background(.bar) |
| } |
| + .task { await call.watchAudio() } |
| + .sheet(isPresented: $showingDevices) { |
| + NavigationStack { |
| + ScrollView { DevicePanel(call: call).frame(maxWidth: .infinity, alignment: .leading) } |
| + .navigationTitle("Devices").navigationBarTitleDisplayMode(.inline) |
| + .toolbar { |
| + ToolbarItem(placement: .confirmationAction) { |
| + Button("Done") { showingDevices = false } |
| + } |
| + } |
| + }.presentationDetents([.medium, .large]) |
| + } |
| .sheet(isPresented: $showingChat, onDismiss: { readChatCount = call.receivedChatCount }) { |
| NavigationStack { |
| ChatView(call: call, draft: draft).padding() |
diff --git a/native/ios/Tests/AudioRouteTests.swift b/native/ios/Tests/AudioRouteTests.swift| new file mode 100644 |
| index 0000000..6454991 |
| --- /dev/null |
| +++ b/native/ios/Tests/AudioRouteTests.swift |
| @@ -0,0 +1,39 @@ |
| +import AVFoundation |
| +import CallsKit |
| +import XCTest |
| + |
| +/// The Bluetooth rule, which only exists on iOS and so can only be tested |
| +/// here — CallsKitTests builds for macOS, where AVAudioSession does not |
| +/// exist at all. |
| +final class AudioRouteTests: XCTestCase { |
| + func testBluetoothIsAllowedOnlyWhenItIsTheChosenMicrophone() { |
| + // .allowBluetoothHFP is what lets a Bluetooth microphone be used, and |
| + // it is also what drags the whole call to the 8 kHz hands-free |
| + // profile the moment any headset connects. Left on unconditionally — |
| + // as it was until this change — every pair of AirPods silently cost |
| + // the call its audio quality in both directions, and nothing said so. |
| + let ordinary = AudioRoutes.options(bluetoothInput: false) |
| + XCTAssertFalse( |
| + ordinary.contains(.allowBluetoothHFP), |
| + "a call with no Bluetooth microphone chosen must not offer the hands-free profile") |
| + XCTAssertTrue( |
| + ordinary.contains(.allowBluetoothA2DP), |
| + "output should still reach a headset, at A2DP quality") |
| + XCTAssertTrue(ordinary.contains(.defaultToSpeaker)) |
| + |
| + // Picking the headset microphone is an informed trade, and then HFP |
| + // is exactly what has to be allowed for it to work at all. |
| + let headset = AudioRoutes.options(bluetoothInput: true) |
| + XCTAssertTrue(headset.contains(.allowBluetoothHFP)) |
| + XCTAssertTrue(headset.contains(.defaultToSpeaker)) |
| + } |
| + |
| + func testOnlyTheHandsFreePortCountsAsABluetoothMicrophone() { |
| + // A2DP is output-only, so a Bluetooth device being used as a |
| + // microphone is always on the hands-free port. Treating A2DP as a |
| + // microphone would turn HFP back on for a headset nobody is |
| + // recording from, which is the bug this rule exists to avoid. |
| + XCTAssertEqual(AVAudioSession.Port.bluetoothHFP.rawValue, "BluetoothHFP") |
| + XCTAssertNotEqual(AVAudioSession.Port.bluetoothA2DP, AVAudioSession.Port.bluetoothHFP) |
| + } |
| +} |
diff --git a/test/browser/native.mjs b/test/browser/native.mjs| index 28c2e5d..e104eec 100644 |
| --- a/test/browser/native.mjs |
| +++ b/test/browser/native.mjs |
| @@ -33,6 +33,8 @@ try { |
| let invitationReady = false; |
| let invitationRefused = false; |
| let invitationSnapshotSaved = false; |
| + let camera = null; |
| + let cameraRefused = false; |
| let exited = false; |
| native.on('exit', () => { exited = true; }); |
| // Native logs must never print credentials, SDP or candidates. Keep only |
| @@ -45,6 +47,7 @@ try { |
| if (value.invitationReady) invitationReady = true; |
| if (value.invitationRefused) invitationRefused = true; |
| if (value.invitationSnapshotSaved) invitationSnapshotSaved = true; |
| + if (value.camera !== undefined) { camera = value.camera; cameraRefused = value.cameraRefused; } |
| if (value.stats) { sample = value; console.log(JSON.stringify(value)); } |
| } catch {} |
| }); |
| @@ -90,6 +93,38 @@ try { |
| }); |
| await wait(async () => await browserAudio() > 0, 'browser receives native audio'); |
| console.log('PASS browser decoded native video and received native audio'); |
| + // Changing camera mid-call. The switch rests on the capturer feeding the |
| + // same RTCVideoSource, so the RTCVideoTrack every connection is already |
| + // sending does not change and nothing renegotiates — the native |
| + // counterpart of the browser's replaceTrack. What proves it is the far |
| + // side: frames must keep arriving across the switch, on the same |
| + // connection, with no new m-line. |
| + { |
| + const beforeFrames = await remoteFrames(); |
| + const beforeLines = await page.evaluate(() => window.meet.state().peers.length); |
| + native.stdin.write('camera synthetic-480\n'); |
| + await wait(() => camera === 'synthetic-480', 'native reports the camera it switched to'); |
| + assert.equal(cameraRefused, false, 'camera switch was refused'); |
| + await wait(async () => await remoteFrames() > beforeFrames + 15, 'browser keeps decoding across a camera switch'); |
| + assert.equal( |
| + await page.evaluate(() => window.meet.state().peers.length), beforeLines, |
| + 'a camera switch must not change the roster or rebuild the pair'); |
| + // Two switches at once are two capture restarts on one device. Queued, |
| + // the last one wins; run at once, the survivor is whichever session |
| + // opened last, which is not the one a person chose. |
| + native.stdin.write('camera synthetic-320\n'); |
| + native.stdin.write('camera synthetic-480\n'); |
| + await wait(() => camera === 'synthetic-480', 'the last of two racing switches is the one that holds'); |
| + const racedFrames = await remoteFrames(); |
| + await wait(async () => await remoteFrames() > racedFrames + 15, 'media survives racing camera switches'); |
| + // And a camera that is not there is refused without costing the call |
| + // its picture. |
| + native.stdin.write('camera no-such-camera\n'); |
| + await wait(() => cameraRefused === true, 'an unknown camera is refused'); |
| + const refusedFrames = await remoteFrames(); |
| + await wait(async () => await remoteFrames() > refusedFrames + 15, 'media survives a refused camera switch'); |
| + console.log('PASS camera switched mid-call without disturbing the connection'); |
| + } |
| await page.locator('[data-chat-text]').fill('browser chat probe'); |
| await page.locator('[data-chat-send]').click(); |
| await wait(() => sample?.receivedChatProbe === true, 'native displays browser chat'); |