amadan
Calls native: meters on every tile, and the input a multi-channel mic sends
Spec §15.20 on the Mac and iPhone apps.
- Your tile and the panel meter what is sent (the capture hook after noise
reduction, 10 Hz). Unmuted and capturing with nothing above -90 dBFS
before the suppressor for five seconds: "Your microphone isn't picking
anything up", its ⓘ naming the device and input ("Scarlett 18i8 USB,
input 1").
- Every other tile meters what arrives: a renderer on the remote audio
track (RemoteAudioTap), which sees decoded frames before the output
volume, at 5 Hz; packets arriving from inbound-rtp bytes, one
statistics(for:) per receiver a second. Muted wins; reporting unmuted,
connected, bytes rising and flat under -85 dBFS for five seconds is "No
sound from <name>", reported to the core with setNoSoundFrom. A speaking
ring with hysteresis, no fade under Reduce Motion, and a waveform on the
mini window's tab.
- Their report about you on their tile: blocked browser, or not getting
sound from you while you are unmuted; in the accessibility label and
announced. PeerMedia decodes audioBlocked and noSoundFrom, optional.
- Mac, the likely cause of 2026-09-25: LiveKit's engine module wires the
mic in as mono and keeps a multi-channel interface's first input only.
The observer now sets the input unit's channel map to the chosen input
(every client channel carries it) and connects nothing, so the module's
default connection stands; the map is applied again before the engine
starts. "Input" picker for a mic with more than one input, remembered
per device UID; changing it mid-call rebuilds the module's input with
setEngineAvailability. Not verified on hardware.
- Levels live in per-tile observable objects, published only when a step
or a line changes. Rules in AudioLevelRules.swift, tested on an
injected clock; the native gate checks the browser measures energy in
what the app sends and the app's meter hears the browser.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
22 files changed,
+1843
−47
native/README.md+42 −0native/apple/Sources/CallsKit/Audio.swift+60 −2native/apple/Sources/CallsKit/AudioLevelRules.swift+253 −0native/apple/Sources/CallsKit/AudioMeter.swift+365 −0native/apple/Sources/CallsKit/Call.swift+130 −0native/apple/Sources/CallsKit/CallAudio.swift+34 −7native/apple/Sources/CallsKit/DevicePanel.swift+3 −0native/apple/Sources/CallsKit/DevicePanelAudio.swift+74 −0native/apple/Sources/CallsKit/InputChannel.swift+241 −0native/apple/Sources/CallsKit/MacCallView.swift+8 −5native/apple/Sources/CallsKit/MacFullWindowCall.swift+7 −5native/apple/Sources/CallsKit/MacVideoTile.swift+24 −7native/apple/Sources/CallsKit/Media.swift+48 −0native/apple/Sources/CallsKit/MiniWindow.swift+7 −5native/apple/Sources/CallsKit/RemoteAudioTap.swift+61 −0native/apple/Sources/CallsKit/Wire.swift+16 −1native/apple/Tests/CallsKitTests/AudioLevelTests.swift+235 −0native/apple/Tests/CallsKitTests/InputChannelTests.swift+125 −0native/apple/Tests/CallsKitTests/PeerMediaHearingTests.swift+43 −0native/ios/Sources/CallsPhone.swift+29 −11native/ios/Sources/PhoneFullScreenCall.swift+5 −4test/browser/native.mjs+33 −0
diff --git a/native/README.md b/native/README.md| index 000b175..7d8233e 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -107,6 +107,36 @@ while sharing; the panel itself shows one warning line only when something is |
| wrong (a stretched or broken-up signal, echo cancellation off, noise reduction |
| bypassed, processing refused — `AudioWarning`). |
| +**Sound is measured, not inferred** (spec §15.20). On 2026-09-25 Tony, in a |
| +browser, could not hear Paul on the Mac app, and Paul's panel looked fine |
| +because nothing on it measured sound. Now: |
| + |
| +- **Your tile** and the panel's **Microphone level** meter what is sent — the |
| + capture hook's level after noise reduction (`CaptureLevel`, read at 10 Hz). |
| + Unmuted and capturing, with nothing reaching the hook above -90 dBFS |
| + *before* the suppressor for five seconds, the tile says "Your microphone |
| + isn't picking anything up", and its ⓘ names the device and input being sent |
| + ("Scarlett 18i8 USB, input 1"). |
| +- **Everyone else's tile** meters what arrives from them: a renderer on their |
| + remote audio track (`RemoteAudioTap`), which sees each decoded frame before |
| + the output volume, read at 5 Hz. Muted, the tile shows the Muted badge and no |
| + meter. Reporting themselves unmuted, connected, with audio bytes still rising |
| + (inbound-rtp, one `statistics(for:)` per receiver a second) and the decoded |
| + peak under -85 dBFS for five seconds, it says "No sound from Bea" instead, |
| + and the app reports those people as `noSoundFrom` through the bridge. A thin |
| + white ring marks whoever is speaking (with hysteresis; no fade under Reduce |
| + Motion), and the mini window's tab shows a waveform while anyone is. |
| +- **Their report about you** shows on their tile: "Tony's browser is blocking |
| + sound — they need to click to hear you", or, while you are unmuted, "Tony |
| + isn't getting any sound from you". Each line is in the tile's accessibility |
| + label and announced when it appears. The app itself is never blocked from |
| + playing sound, so it never reports `audioBlocked`. |
| + |
| +The rules (`AudioLevelRules.swift`) are held by `AudioLevelTests` on an |
| +injected clock; `TestNativeMacCallsBrowser` checks that the browser measures |
| +energy, not only bytes, in what the app sends, and that the app's receive meter |
| +hears the browser. |
| + |
| The camera is a picker on both platforms. Switching rests on the capturer |
| feeding the same `RTCVideoSource`, so the `RTCVideoTrack` every peer |
| connection already holds does not change and nothing renegotiates — the |
| @@ -183,6 +213,18 @@ The Mac's choices follow these rules (`AudioRoutes`, held by |
| is listened to for that as well as the module's own report. No poller. |
| - Nothing changes Sound settings. Mute is not remembered: every call starts |
| unmuted. |
| +- **Input**, for a microphone with more than one input (an audio interface): |
| + a picker under Microphone, "Input 1" to "Input 18" with the driver's channel |
| + names where it gives them, remembered per device UID. LiveKit's engine module |
| + wires the microphone in as mono, and the conversion from a multi-channel |
| + interface keeps its first input only, so a microphone on input 2 used to send |
| + silence. The module's observer now sets the input unit's channel map |
| + (`kAudioOutputUnitProperty_ChannelMap`) so every channel it hands the engine |
| + is the chosen input, and connects nothing, leaving the module's own mono |
| + connection in place (`InputChannel.swift`). Changing it mid-call takes the |
| + module's input away and gives it back (`setEngineAvailability`), which |
| + rewires the microphone with the new map — a moment's gap, as a device switch |
| + has. Input 1 on a device never mapped is left exactly as the module had it. |
| The app keeps the choice itself rather than asking the module, because |
| LiveKit's engine module (M150) cannot say which device it is on — |
diff --git a/native/apple/Sources/CallsKit/Audio.swift b/native/apple/Sources/CallsKit/Audio.swift| index ca07f56..441b2f4 100644 |
| --- a/native/apple/Sources/CallsKit/Audio.swift |
| +++ b/native/apple/Sources/CallsKit/Audio.swift |
| @@ -73,8 +73,18 @@ public struct AudioDevice: Identifiable, Hashable, Sendable { |
| /// Said while a remembered device is missing and the call is on the |
| /// system default instead. Clears itself when the device is back. |
| @Published public private(set) var routeNote = "" |
| + /// The microphone's input channels, labelled, when it has more than |
| + /// one (an audio interface); empty for an ordinary microphone. See |
| + /// InputChannel.swift for why this is a choice at all. |
| + @Published public private(set) var inputChannels: [String] = [] |
| + /// The input the call sends from, 0-based. Remembered per device. |
| + @Published public private(set) var inputChannel = 0 |
| + /// The microphone and, on an interface, its input: "Scarlett 18i8 USB, |
| + /// input 1". Empty before a call has a module. |
| + @Published public private(set) var inputSource = "" |
| + private var inputDevice: InputDeviceChannels? |
| private var control: AudioModuleControl? |
| - private let defaults: UserDefaults |
| + let defaults: UserDefaults |
| /// The device each side of the module was last moved to. Nil is where |
| /// the module starts: following the system default. |
| private var appliedInput: String? |
| @@ -120,6 +130,7 @@ public struct AudioDevice: Identifiable, Hashable, Sendable { |
| update(\.currentInput, input.current) |
| update(\.currentOutput, output.current) |
| update(\.outputName, outputs.first { $0.id == currentOutput }?.name ?? "") |
| + readChannels(input.current) |
| #endif |
| } |
| @@ -222,6 +233,29 @@ public struct AudioDevice: Identifiable, Hashable, Sendable { |
| refresh() |
| } |
| + /// The current microphone's channels and the remembered one of them. |
| + private func readChannels(_ id: String?) { |
| + let device = control.flatMap { $0.channels(id) } |
| + inputDevice = device |
| + let channel = device.map { InputChannelPreference.load(uid: $0.uid, channels: $0.count, defaults) } ?? 0 |
| + update(\.inputChannels, device.map { d in d.count > 1 ? (0..<d.count).map(d.label) : [] } ?? []) |
| + update(\.inputChannel, channel) |
| + update(\.inputSource, device?.describe(channel: channel) ?? "") |
| + } |
| + |
| + /// Chooses which input of a multi-channel microphone the call sends, |
| + /// remembers it for that device, and rebuilds the module's input so it |
| + /// takes effect now — a moment's gap in both directions, as a device |
| + /// switch has. Outside a call it is only remembered. |
| + public func use(inputChannel channel: Int) { |
| + guard let device = inputDevice, channel >= 0, channel < max(device.count, 1), |
| + channel != inputChannel |
| + else { return } |
| + InputChannelPreference.save(channel, uid: device.uid, defaults) |
| + control?.rebuildInput() |
| + refresh() |
| + } |
| + |
| /// Leaving the call. The choices are remembered, so there is nothing |
| /// to undo; what is left is to look again at what is connected. |
| func reset() { |
| @@ -331,13 +365,23 @@ public struct AudioDevice: Identifiable, Hashable, Sendable { |
| /// False when the device is not listed. See AudioRoutes for why the |
| /// module's own answer is not asked. |
| var select: (_ id: String, _ input: Bool) -> Bool |
| + /// The input channels of the device behind a module device id. |
| + var channels: (_ id: String?) -> InputDeviceChannels? |
| + /// Makes the module wire its microphone in again, so a new channel |
| + /// map is applied. A no-op while it is not capturing: the map is read |
| + /// whenever the input is next wired in. |
| + var rebuildInput: () -> Void |
| init( |
| devices: @escaping (_ input: Bool) -> [ModuleDevice], |
| - select: @escaping (_ id: String, _ input: Bool) -> Bool |
| + select: @escaping (_ id: String, _ input: Bool) -> Bool, |
| + channels: @escaping (_ id: String?) -> InputDeviceChannels? = { _ in nil }, |
| + rebuildInput: @escaping () -> Void = {} |
| ) { |
| self.devices = devices |
| self.select = select |
| + self.channels = channels |
| + self.rebuildInput = rebuildInput |
| } |
| init(_ module: RTCAudioDeviceModule) { |
| @@ -356,6 +400,20 @@ public struct AudioDevice: Identifiable, Hashable, Sendable { |
| if input { module.inputDevice = device } else { module.outputDevice = device } |
| return true |
| } |
| + channels = { id in CoreAudioInput.device(forModuleID: id).flatMap(CoreAudioInput.describe) } |
| + // The module has no "rewire the input" call. Taking the input away |
| + // and giving it back does it: the module recreates its engine for |
| + // output alone, then wires the microphone in again, asking the |
| + // observer first — which is where the map is applied. (Re-selecting |
| + // the same device does nothing: the module ignores a state that has |
| + // not changed.) |
| + rebuildInput = { [weak module] in |
| + guard let module, module.isRecording, !module.isManualRenderingMode else { return } |
| + _ = module.setEngineAvailability( |
| + RTCAudioEngineAvailability(isInputAvailable: false, isOutputAvailable: true)) |
| + _ = module.setEngineAvailability( |
| + RTCAudioEngineAvailability(isInputAvailable: true, isOutputAvailable: true)) |
| + } |
| } |
| } |
| #endif |
diff --git a/native/apple/Sources/CallsKit/AudioLevelRules.swift b/native/apple/Sources/CallsKit/AudioLevelRules.swift| new file mode 100644 |
| index 0000000..077d849 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/AudioLevelRules.swift |
| @@ -0,0 +1,253 @@ |
| +import Foundation |
| + |
| +// Spec §15.20: what the meters show and when the "no sound" lines appear, |
| +// as rules the tests can hold with an injected clock. Nothing here touches |
| +// WebRTC or the main actor; `Call` feeds these from the capture hook |
| +// (CaptureLevel) and the per-peer receive tap (RemoteAudioTap). |
| +// |
| +// Levels are 0...1 of full scale throughout, as CaptureLevel reports |
| +// them, and the thresholds are in dBFS. |
| + |
| +/// A level as one of a few steps, for a meter that should move only when |
| +/// the picture changes — not sixty times a second. |
| +public enum MeterScale { |
| + /// Bars on a meter. Step 0 is none lit. |
| + public static let steps = 5 |
| + /// The dBFS at which each bar lights. Speech peaks sit around -20 to |
| + /// -6; a quiet room's floor around -60. The bottom bar lights for a |
| + /// room tone, so a live microphone always shows *something*. |
| + public static let thresholds: [Float] = [-60, -45, -32, -20, -10] |
| + |
| + public static func decibels(_ level: Float) -> Float { |
| + level > 0 ? 20 * log10(level) : -.infinity |
| + } |
| + |
| + /// 0 (nothing) to `steps` (loud). |
| + public static func step(_ level: Float) -> Int { |
| + let db = decibels(level) |
| + return thresholds.filter { db >= $0 }.count |
| + } |
| +} |
| + |
| +/// How long a condition has held, on the caller's clock. |
| +public struct Held: Equatable, Sendable { |
| + private var since: Double? |
| + public init() {} |
| + /// Seconds the condition has held as of `now`, or 0 when it does not. |
| + public mutating func update(_ holding: Bool, now: Double) -> Double { |
| + guard holding else { |
| + since = nil |
| + return 0 |
| + } |
| + if since == nil { since = now } |
| + return now - (since ?? now) |
| + } |
| +} |
| + |
| +/// Whether someone is speaking, for the tile's highlight: on at a |
| +/// speech-like level, off only once the level has fallen well below it |
| +/// and stayed there a moment, so the highlight does not flicker between |
| +/// syllables. |
| +public struct SpeakingGate: Equatable, Sendable { |
| + public static let onDb: Float = -30 |
| + public static let offDb: Float = -42 |
| + /// How long below `offDb` before the highlight goes. |
| + public static let hangover: Double = 0.6 |
| + public private(set) var speaking = false |
| + private var quiet = Held() |
| + public init() {} |
| + |
| + public mutating func update(level: Float, now: Double) -> Bool { |
| + let db = MeterScale.decibels(level) |
| + if db >= Self.onDb { |
| + speaking = true |
| + _ = quiet.update(false, now: now) |
| + } else if speaking { |
| + if quiet.update(db < Self.offDb, now: now) >= Self.hangover { speaking = false } |
| + } |
| + return speaking |
| + } |
| + |
| + public mutating func reset() { |
| + speaking = false |
| + quiet = Held() |
| + } |
| +} |
| + |
| +/// What a remote person's tile says about their sound. |
| +public enum PeerAudioState: Equatable, Sendable { |
| + /// They report themselves muted: the tile's existing Muted badge, and |
| + /// no meter. |
| + case muted |
| + /// Their audio is arriving, they report themselves unmuted, and it has |
| + /// been flat for `PeerAudioWatch.noSoundAfter` seconds. |
| + case noSound |
| + /// A level meter at this step. |
| + case meter(Int) |
| +} |
| + |
| +/// One reading of a remote person's audio, as the call measures it. |
| +public struct PeerAudioSample: Equatable, Sendable { |
| + /// Their own report (§15.19). Nil is unknown, which is not unmuted: |
| + /// "No sound from" needs them to have said they are unmuted. |
| + public var reported: PeerMedia? |
| + /// The connection to them is up. |
| + public var connected: Bool |
| + /// Audio packets have arrived since the last look (bytes received rose). |
| + public var arriving: Bool |
| + /// The loudest decoded sample since the last reading, 0...1. |
| + public var level: Float |
| + |
| + public init(reported: PeerMedia?, connected: Bool, arriving: Bool, level: Float) { |
| + self.reported = reported |
| + self.connected = connected |
| + self.arriving = arriving |
| + self.level = level |
| + } |
| +} |
| + |
| +/// The receive side's rule, one per remote person. Precedence: muted, then |
| +/// "No sound from", then the meter. |
| +public struct PeerAudioWatch: Equatable, Sendable { |
| + /// Below this peak the decoded audio is digital silence: a live |
| + /// microphone in a quiet room, after a browser's own noise suppression, |
| + /// still sits tens of dB above it. -85 dBFS is under two steps of a |
| + /// 16-bit sample. |
| + public static let silentDb: Float = -85 |
| + /// "About five seconds" (spec §15.20). |
| + public static let noSoundAfter: Double = 5 |
| + |
| + private var flat = Held() |
| + private var gate = SpeakingGate() |
| + public private(set) var state: PeerAudioState = .meter(0) |
| + public private(set) var speaking = false |
| + |
| + public init() {} |
| + |
| + @discardableResult |
| + public mutating func update(_ sample: PeerAudioSample, now: Double) -> PeerAudioState { |
| + if sample.reported?.muted == true { |
| + _ = flat.update(false, now: now) |
| + gate.reset() |
| + speaking = false |
| + state = .muted |
| + return state |
| + } |
| + let silent = MeterScale.decibels(sample.level) < Self.silentDb |
| + let watching = sample.reported != nil && sample.connected && sample.arriving && silent |
| + let held = flat.update(watching, now: now) |
| + speaking = gate.update(level: sample.level, now: now) |
| + state = held >= Self.noSoundAfter ? .noSound : .meter(MeterScale.step(sample.level)) |
| + return state |
| + } |
| +} |
| + |
| +/// One reading of this app's own microphone. |
| +public struct MicSample: Equatable, Sendable { |
| + public var muted: Bool |
| + /// The microphone is open (see `CallAudio.capturing`). WebRTC opens it |
| + /// only once there is someone to send to. |
| + public var capturing: Bool |
| + /// The loudest sample reaching the capture hook, before the noise |
| + /// suppressor, since the last reading. The question "is the microphone |
| + /// picking anything up" is asked of this: a quiet room the suppressor |
| + /// has taken down can look dead afterwards and is not. |
| + public var inputPeak: Float |
| + /// Buffers through the hook since the last reading; none means nothing |
| + /// flowed at all. |
| + public var inputBuffers: Int |
| + /// The loudest sample leaving the hook — what is sent — for the meter. |
| + public var sentPeak: Float |
| + |
| + public init(muted: Bool, capturing: Bool, inputPeak: Float, inputBuffers: Int, sentPeak: Float) { |
| + self.muted = muted |
| + self.capturing = capturing |
| + self.inputPeak = inputPeak |
| + self.inputBuffers = inputBuffers |
| + self.sentPeak = sentPeak |
| + } |
| +} |
| + |
| +/// The sending side's rule: the meter on your own tile, and "Your |
| +/// microphone isn't picking anything up". |
| +public struct MicWatch: Equatable, Sendable { |
| + /// A microphone that is plugged in and on hears its own electrical |
| + /// noise well above this; an empty input on an audio interface, or a |
| + /// channel with nothing on it, does not. About one step of a 16-bit |
| + /// sample. |
| + public static let silentDb: Float = -90 |
| + public static let silentAfter: Double = 5 |
| + |
| + private var flat = Held() |
| + private var gate = SpeakingGate() |
| + public private(set) var step = 0 |
| + public private(set) var silent = false |
| + public private(set) var speaking = false |
| + |
| + public init() {} |
| + |
| + public mutating func update(_ sample: MicSample, now: Double) { |
| + // Muted, WebRTC skips the hook, so there is nothing to measure and |
| + // nothing to warn about: the Muted state already says it. |
| + guard !sample.muted, sample.capturing else { |
| + _ = flat.update(false, now: now) |
| + gate.reset() |
| + step = 0 |
| + silent = false |
| + speaking = false |
| + return |
| + } |
| + let dead = |
| + sample.inputBuffers == 0 || MeterScale.decibels(sample.inputPeak) < Self.silentDb |
| + silent = flat.update(dead, now: now) >= Self.silentAfter |
| + step = MeterScale.step(sample.sentPeak) |
| + speaking = gate.update(level: sample.sentPeak, now: now) |
| + } |
| +} |
| + |
| +/// The words, in one place: tiles, accessibility and the tests read them |
| +/// from here. |
| +public enum AudioWords { |
| + public static func noSound(from name: String) -> String { "No sound from \(name)" } |
| + public static let noSoundDetail = |
| + "Their audio is arriving, but it has been silent for a few seconds while they are unmuted. Their microphone may be on the wrong input, turned down, or not the one they think they chose." |
| + |
| + public static let micSilent = "Your microphone isn’t picking anything up" |
| + /// The ⓘ under "Your microphone isn't picking anything up": what is |
| + /// being sent, from what, and where to change it. |
| + public static func micSilentDetail(source: String, channels: Bool) -> String { |
| + let what = source.isEmpty ? "your microphone" : source |
| + var text = "Calls is sending what it hears from \(what), and that has been silent for five seconds, so others hear nothing." |
| + if channels { |
| + text += " If your microphone is plugged into a different input, choose that input under Devices." |
| + } else { |
| + text += " Check the microphone chosen under Devices, and that it is switched on and turned up." |
| + } |
| + return text |
| + } |
| + |
| + public static func blocked(_ name: String) -> String { |
| + "\(name)’s browser is blocking sound — they need to click to hear you." |
| + } |
| + public static func notHearingYou(_ name: String) -> String { |
| + "\(name) isn’t getting any sound from you." |
| + } |
| + |
| + /// What a person's tile tells you about their report on *you* (spec |
| + /// §15.20): their browser refusing sound, and their not hearing you. |
| + /// The second only while you are unmuted: muted, silence is expected. |
| + public static func notices(name: String, media: PeerMedia?, me: String, muted: Bool) -> [String] { |
| + guard let media else { return [] } |
| + var out: [String] = [] |
| + if media.audioBlocked == true { out.append(blocked(name)) } |
| + if !muted, !me.isEmpty, media.noSoundFrom?.contains(me) == true { |
| + out.append(notHearingYou(name)) |
| + } |
| + return out |
| + } |
| + |
| + /// The sentence a tile adds to its accessibility label for its sound. |
| + public static func accessibility(_ state: PeerAudioState, name: String) -> String? { |
| + state == .noSound ? noSound(from: name) : nil |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/AudioMeter.swift b/native/apple/Sources/CallsKit/AudioMeter.swift| new file mode 100644 |
| index 0000000..bc62e7c |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/AudioMeter.swift |
| @@ -0,0 +1,365 @@ |
| +import Combine |
| +import SwiftUI |
| + |
| +// Spec §15.20 on the tiles: a meter for every voice, measured where it can |
| +// be — what you send on your own tile, what arrives on everyone else's — |
| +// and a sentence in place of the meter when the sound has gone flat. |
| +// |
| +// The levels live in small observable objects of their own, one per tile, |
| +// rather than on `Call`: a meter that changes a few times a second must not |
| +// re-render the whole call view each time. Each object publishes only when |
| +// what it shows changes — a meter step, a line appearing — not per reading. |
| + |
| +/// Your own microphone, for your tile. |
| +@MainActor public final class MicAudio: ObservableObject { |
| + @Published public private(set) var step = 0 |
| + @Published public private(set) var muted = false |
| + @Published public private(set) var speaking = false |
| + /// "Your microphone isn't picking anything up". |
| + @Published public private(set) var silent = false |
| + /// What the ⓘ says is being sent from. |
| + @Published public private(set) var source = "" |
| + /// The microphone has more than one input, so the ⓘ points at the |
| + /// Input picker. |
| + @Published public private(set) var channels = false |
| + |
| + public init() {} |
| + |
| + func show(step: Int, muted: Bool, speaking: Bool, silent: Bool, source: String, channels: Bool) { |
| + if self.step != step { self.step = step } |
| + if self.muted != muted { self.muted = muted } |
| + if self.speaking != speaking { self.speaking = speaking } |
| + if self.silent != silent { self.silent = silent } |
| + if self.source != source { self.source = source } |
| + if self.channels != channels { self.channels = channels } |
| + } |
| +} |
| + |
| +/// One remote person's sound, for their tile. |
| +@MainActor public final class PeerAudio: ObservableObject { |
| + @Published public private(set) var state: PeerAudioState = .meter(0) |
| + @Published public private(set) var speaking = false |
| + /// Their report about you: blocked, or not hearing you. |
| + @Published public private(set) var notices: [String] = [] |
| + |
| + public init() {} |
| + |
| + func show(state: PeerAudioState, speaking: Bool, notices: [String]) { |
| + if self.state != state { self.state = state } |
| + if self.speaking != speaking { self.speaking = speaking } |
| + if self.notices != notices { self.notices = notices } |
| + } |
| +} |
| + |
| +/// Every meter in a call, fed by `Call` (see `Call.watchLevels`). |
| +@MainActor public final class AudioLevels { |
| + public let mic = MicAudio() |
| + private var peers: [String: PeerAudio] = [:] |
| + private var watches: [String: PeerAudioWatch] = [:] |
| + private var micWatch = MicWatch() |
| + /// Anyone other than you speaking, for the mini window's tab. |
| + public let anyoneSpeaking = SpeakingNow() |
| + |
| + public init() {} |
| + |
| + /// The model a remote person's tile reads. Made on first asking, so a |
| + /// tile can be drawn before its first reading. |
| + public func peer(_ id: String) -> PeerAudio { |
| + if let existing = peers[id] { return existing } |
| + let made = PeerAudio() |
| + peers[id] = made |
| + return made |
| + } |
| + |
| + func updateMic(_ sample: MicSample, now: Double, source: String, channels: Bool) { |
| + let wasSilent = mic.silent |
| + micWatch.update(sample, now: now) |
| + mic.show( |
| + step: micWatch.step, muted: sample.muted, speaking: micWatch.speaking, |
| + silent: micWatch.silent, source: source, channels: channels) |
| + if micWatch.silent, !wasSilent { AudioAnnouncer.say(AudioWords.micSilent) } |
| + } |
| + |
| + /// Returns the state the rule arrived at. |
| + @discardableResult |
| + func updatePeer( |
| + _ id: String, name: String, sample: PeerAudioSample, notices: [String], now: Double |
| + ) -> PeerAudioState { |
| + var watch = watches[id] ?? PeerAudioWatch() |
| + let before = watch.state |
| + let state = watch.update(sample, now: now) |
| + watches[id] = watch |
| + let model = peer(id) |
| + let fresh = notices.filter { !model.notices.contains($0) } |
| + model.show(state: state, speaking: watch.speaking, notices: notices) |
| + if state == .noSound, before != .noSound { AudioAnnouncer.say(AudioWords.noSound(from: name)) } |
| + for line in fresh { AudioAnnouncer.say(line) } |
| + return state |
| + } |
| + |
| + /// Forgets everyone not in `present`, and says whether anyone left is |
| + /// speaking. |
| + func settle(present: Set<String>) { |
| + for id in Array(watches.keys) where !present.contains(id) { |
| + watches.removeValue(forKey: id) |
| + peers[id]?.show(state: .meter(0), speaking: false, notices: []) |
| + } |
| + anyoneSpeaking.set(watches.values.contains { $0.speaking }) |
| + } |
| + |
| + /// The people whose sound has gone flat, in a stable order: what this |
| + /// app reports as `noSoundFrom`. |
| + var noSoundFrom: [String] { |
| + watches.filter { $0.value.state == .noSound }.keys.sorted() |
| + } |
| + |
| + func reset() { |
| + micWatch = MicWatch() |
| + mic.show(step: 0, muted: false, speaking: false, silent: false, source: "", channels: false) |
| + for id in Array(watches.keys) { peers[id]?.show(state: .meter(0), speaking: false, notices: []) } |
| + watches = [:] |
| + anyoneSpeaking.set(false) |
| + } |
| +} |
| + |
| +/// A single flag, observed alone (the mini window's tab). |
| +@MainActor public final class SpeakingNow: ObservableObject { |
| + @Published public private(set) var value = false |
| + func set(_ on: Bool) { if value != on { value = on } } |
| +} |
| + |
| +/// Lines a person needs to hear when they appear, not only see. |
| +@MainActor enum AudioAnnouncer { |
| + static func say(_ text: String) { |
| + AccessibilityNotification.Announcement(text).post() |
| + } |
| +} |
| + |
| +// MARK: - Views |
| + |
| +/// What a tile is told about its sound. |
| +public enum TileAudio { |
| + case you(MicAudio) |
| + case peer(PeerAudio, name: String) |
| +} |
| + |
| +/// A tile's sound, read out of its model as plain values. |
| +public struct TileAudioInfo: Equatable { |
| + public enum Status: Equatable { |
| + case none |
| + case meter(Int) |
| + case noSound(String) |
| + } |
| + public var status: Status |
| + public var speaking: Bool |
| + /// Lines above the name: their notices about you, or your microphone |
| + /// warning. |
| + public var notices: [String] |
| + /// The ⓘ's text, when a line has one. |
| + public var detail: String? |
| + |
| + /// What the tile's accessibility label adds, after the name. |
| + public var spoken: [String] { |
| + var out: [String] = [] |
| + if case .noSound(let name) = status { out.append(AudioWords.noSound(from: name)) } |
| + out.append(contentsOf: notices) |
| + return out |
| + } |
| + |
| + @MainActor static func you(_ mic: MicAudio) -> TileAudioInfo { |
| + TileAudioInfo( |
| + status: mic.muted ? .none : .meter(mic.step), speaking: mic.speaking, |
| + notices: mic.silent ? [AudioWords.micSilent] : [], |
| + detail: mic.silent |
| + ? AudioWords.micSilentDetail(source: mic.source, channels: mic.channels) : nil) |
| + } |
| + |
| + @MainActor static func peer(_ audio: PeerAudio, name: String) -> TileAudioInfo { |
| + let status: Status |
| + switch audio.state { |
| + case .muted: status = .none |
| + case .noSound: status = .noSound(name) |
| + case .meter(let step): status = .meter(step) |
| + } |
| + return TileAudioInfo( |
| + status: status, speaking: audio.speaking, notices: audio.notices, |
| + detail: audio.state == .noSound ? AudioWords.noSoundDetail : nil) |
| + } |
| +} |
| + |
| +/// Reads a tile's model and hands the tile plain values, so the tile |
| +/// itself need not observe anything and a tile without sound (a screen) |
| +/// observes nothing at all. |
| +public struct TileAudioReader<Content: View>: View { |
| + let audio: TileAudio? |
| + let content: (TileAudioInfo?) -> Content |
| + public init(_ audio: TileAudio?, @ViewBuilder content: @escaping (TileAudioInfo?) -> Content) { |
| + self.audio = audio |
| + self.content = content |
| + } |
| + public var body: some View { |
| + switch audio { |
| + case .you(let mic)?: MicReader(mic: mic, content: content) |
| + case .peer(let peer, let name)?: PeerReader(peer: peer, name: name, content: content) |
| + case nil: content(nil) |
| + } |
| + } |
| + private struct MicReader: View { |
| + @ObservedObject var mic: MicAudio |
| + let content: (TileAudioInfo?) -> Content |
| + var body: some View { content(TileAudioInfo.you(mic)) } |
| + } |
| + private struct PeerReader: View { |
| + @ObservedObject var peer: PeerAudio |
| + let name: String |
| + let content: (TileAudioInfo?) -> Content |
| + var body: some View { content(TileAudioInfo.peer(peer, name: name)) } |
| + } |
| +} |
| + |
| +/// A small level meter: bars that light from the left. Steps, not a |
| +/// continuous level, so it never animates frame by frame, and with Reduce |
| +/// Motion it does not animate at all. Hidden from assistive technology: |
| +/// the words beside it carry anything worth saying. |
| +public struct AudioLevelBars: View { |
| + let step: Int |
| + var height: CGFloat = 12 |
| + @Environment(\.accessibilityReduceMotion) private var reduceMotion |
| + public init(step: Int, height: CGFloat = 12) { |
| + self.step = step |
| + self.height = height |
| + } |
| + public var body: some View { |
| + HStack(alignment: .bottom, spacing: 2) { |
| + ForEach(0..<MeterScale.steps, id: \.self) { bar in |
| + RoundedRectangle(cornerRadius: 1) |
| + .fill(bar < step ? Color.white : Color.white.opacity(0.3)) |
| + .frame(width: 3, height: height * CGFloat(bar + 2) / CGFloat(MeterScale.steps + 1)) |
| + } |
| + } |
| + .frame(height: height, alignment: .bottom) |
| + .animation(reduceMotion ? nil : .easeOut(duration: 0.08), value: step) |
| + .accessibilityHidden(true) |
| + } |
| +} |
| + |
| +/// The sound beside a tile's name: the meter, or "No sound from Bea" with |
| +/// its ⓘ. Nothing while muted — the name label already says Muted. |
| +public struct TileAudioStatus: View { |
| + let info: TileAudioInfo? |
| + public init(_ info: TileAudioInfo?) { self.info = info } |
| + public var body: some View { |
| + switch info?.status { |
| + case .meter(let step)?: |
| + AudioLevelBars(step: step) |
| + .padding(.horizontal, 7).padding(.vertical, 5) |
| + .background(.black.opacity(0.7), in: Capsule()) |
| + case .noSound(let name)?: |
| + AudioLine( |
| + text: AudioWords.noSound(from: name), systemImage: "speaker.slash.fill", |
| + detail: info?.detail) |
| + default: |
| + EmptyView() |
| + } |
| + } |
| +} |
| + |
| +/// The lines above a tile's name. |
| +public struct TileAudioNotices: View { |
| + let info: TileAudioInfo? |
| + public init(_ info: TileAudioInfo?) { self.info = info } |
| + public var body: some View { |
| + if let info, !info.notices.isEmpty { |
| + VStack(alignment: .leading, spacing: 4) { |
| + ForEach(info.notices, id: \.self) { line in |
| + AudioLine( |
| + text: line, systemImage: "exclamationmark.triangle.fill", |
| + detail: line == AudioWords.micSilent ? info.detail : nil) |
| + } |
| + } |
| + } |
| + } |
| +} |
| + |
| +/// One line over the picture: light glyph and text on the dark overlay |
| +/// surface, legible over any video, with an optional ⓘ that opens the |
| +/// explanation. |
| +struct AudioLine: View { |
| + let text: String |
| + let systemImage: String |
| + var detail: String? |
| + @State private var showing = false |
| + var body: some View { |
| + HStack(spacing: 5) { |
| + Image(systemName: systemImage) |
| + Text(text).lineLimit(2).multilineTextAlignment(.leading) |
| + .fixedSize(horizontal: false, vertical: true) |
| + if let detail { |
| + Button { |
| + showing.toggle() |
| + } label: { |
| + Image(systemName: "info.circle") |
| + } |
| + .buttonStyle(.plain) |
| + .help(detail) |
| + .accessibilityLabel("About this") |
| + .popover(isPresented: $showing) { |
| + Text(detail).font(.callout).padding(14) |
| + .frame(idealWidth: 280, maxWidth: 320) |
| + .fixedSize(horizontal: false, vertical: true) |
| + #if os(iOS) |
| + .presentationCompactAdaptation(.popover) |
| + #endif |
| + } |
| + } |
| + } |
| + .font(.caption.weight(.semibold)).foregroundStyle(.white) |
| + .padding(.horizontal, 8).padding(.vertical, 4) |
| + .background(.black.opacity(0.7), in: RoundedRectangle(cornerRadius: 8, style: .continuous)) |
| + .overlay( |
| + RoundedRectangle(cornerRadius: 8, style: .continuous).strokeBorder( |
| + .white.opacity(0.18), lineWidth: 1)) |
| + } |
| +} |
| + |
| +extension View { |
| + /// The speaking highlight round a tile: a white ring, on and off in |
| + /// steps (see SpeakingGate), and without a fade under Reduce Motion. |
| + public func speakingRing(_ on: Bool, cornerRadius: CGFloat) -> some View { |
| + modifier(SpeakingRing(on: on, cornerRadius: cornerRadius)) |
| + } |
| +} |
| + |
| +private struct SpeakingRing: ViewModifier { |
| + let on: Bool |
| + let cornerRadius: CGFloat |
| + @Environment(\.accessibilityReduceMotion) private var reduceMotion |
| + func body(content: Content) -> some View { |
| + content.overlay { |
| + RoundedRectangle(cornerRadius: cornerRadius) |
| + .strokeBorder(Color.white.opacity(on ? 0.85 : 0), lineWidth: 3) |
| + .animation(reduceMotion ? nil : .easeOut(duration: 0.15), value: on) |
| + .allowsHitTesting(false) |
| + } |
| + } |
| +} |
| + |
| +/// A tile's accessibility label with its sound said after the name. |
| +public func tileAccessibilityLabel(_ label: String, muted: Bool, audio: TileAudioInfo?) -> String { |
| + ([tileAccessibilityLabel(label, muted: muted)] + (audio?.spoken ?? [])).joined(separator: ", ") |
| +} |
| + |
| +/// A waveform while someone is speaking, and nothing otherwise: the mini |
| +/// window's tab. Hidden from assistive technology, which would hear it |
| +/// flicker on and off with every sentence. |
| +public struct SpeakingGlyph: View { |
| + @ObservedObject var speaking: SpeakingNow |
| + public init(speaking: SpeakingNow) { self.speaking = speaking } |
| + public var body: some View { |
| + Image(systemName: "waveform") |
| + .font(.caption.bold()).foregroundStyle(.white) |
| + .opacity(speaking.value ? 1 : 0) |
| + .help("Someone is speaking") |
| + .accessibilityHidden(true) |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index 1fce599..588be33 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -102,6 +102,21 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| static let reduceNoiseKey = "reduceBackgroundNoise" |
| /// Which processing is running on the microphone, for the readout. |
| @Published public private(set) var audioProcessing: ProcessingReport? |
| + /// The meters (spec §15.20): your microphone and everyone's sound, in |
| + /// objects of their own so a meter moving does not redraw the call. |
| + public let levels = AudioLevels() |
| + private var levelWatch: Task<Void, Never>? |
| + /// What was last reported as `noSoundFrom`, and the queue that reports |
| + /// it: the bridge's send is a request, and not for the main thread. |
| + private var reportedNoSound: [String] = [] |
| + private let noSoundQueue = DispatchQueue(label: "calls.no-sound-from") |
| + /// Received bytes per person at the last statistics read, and whether |
| + /// they had risen: "arriving" for the no-sound rule. |
| + private var audioBytes: [String: UInt64] = [:] |
| + private var audioArriving: [String: Bool] = [:] |
| + /// The loudest remote sound since the gate last asked (sampleStats). |
| + private var remotePeakSinceSample: Float = 0 |
| + private var micPeakSinceSample: Float = 0 |
| private var cameraWork: Task<Void, Never>? |
| private var cameraSwitches = 0 |
| private let knocks = KnockWatch() |
| @@ -161,6 +176,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| reportMedia() |
| media.audio.reduceNoise = reduceNoise |
| audioProcessing = media.audio.report() |
| + watchLevels() |
| KnockNotice.requestPermission() |
| // Asking Apple at join, once there is a session to register the |
| // answer against. Apple replies on its own schedule, at the app |
| @@ -195,6 +211,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| deviceNote = "" |
| audioHealth = nil |
| audioProcessing = nil |
| + endLevelWatch() |
| currentCamera = nil |
| cameraStatus = nil |
| cameraNote = "" |
| @@ -352,6 +369,108 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| audioProcessing = media.audio.report() |
| } |
| } |
| + // --------------------------------------------------------------- |
| + // Levels (§15.20) |
| + // --------------------------------------------------------------- |
| + |
| + /// What a tile shows about its sound: yours, or a remote person's. |
| + public var selfAudio: TileAudio { .you(levels.mic) } |
| + public func tileAudio(_ peer: String) -> TileAudio { |
| + .peer(levels.peer(peer), name: name(of: peer)) |
| + } |
| + |
| + /// Reads the meters ten times a second while the call is up: your |
| + /// microphone every tick, everyone else's every other (5 Hz), and the |
| + /// received byte counts once a second. What is published only changes |
| + /// when a meter step or a line does (see AudioLevels). |
| + private func watchLevels() { |
| + levelWatch?.cancel() |
| + levels.reset() |
| + reportedNoSound = [] |
| + audioBytes = [:] |
| + audioArriving = [:] |
| + levelWatch = Task { @MainActor [weak self] in |
| + var tick = 0 |
| + while !Task.isCancelled { |
| + try? await Task.sleep(nanoseconds: 100_000_000) |
| + guard let self, !Task.isCancelled, self.bridge != nil else { return } |
| + tick += 1 |
| + let now = ProcessInfo.processInfo.systemUptime |
| + self.sampleMic(now: now) |
| + if tick % 2 == 0 { await self.samplePeers(stats: tick % 10 == 0, now: now) } |
| + } |
| + } |
| + } |
| + |
| + private func endLevelWatch() { |
| + levelWatch?.cancel() |
| + levelWatch = nil |
| + levels.reset() |
| + reportedNoSound = [] |
| + audioBytes = [:] |
| + audioArriving = [:] |
| + } |
| + |
| + private func sampleMic(now: Double) { |
| + let denoiser = media.audio.denoiser |
| + let sent = denoiser.level.take() |
| + let heard = denoiser.inputLevel.take() |
| + micPeakSinceSample = max(micPeakSinceSample, sent.peak) |
| + #if os(macOS) |
| + let source = audio.inputSource |
| + let channels = audio.inputChannels.count > 1 |
| + let capturing = media.audio.capturing |
| + #else |
| + let source = audio.inputs.first { $0.id == audio.currentInput }?.name ?? "" |
| + let channels = false |
| + let capturing = media.audio.hookFlowing |
| + #endif |
| + levels.updateMic( |
| + MicSample( |
| + muted: muted || media.localAudio == nil, capturing: capturing, |
| + inputPeak: heard.peak, inputBuffers: heard.buffers, sentPeak: sent.peak), |
| + now: now, source: source, channels: channels) |
| + } |
| + |
| + private func samplePeers(stats: Bool, now: Double) async { |
| + let readings = await media.peerAudio(stats: stats) |
| + guard levelWatch != nil, let state else { return } |
| + let me = state.peer |
| + var present = Set<String>() |
| + for person in state.peers where person.peer != me { |
| + guard let reading = readings[person.peer] else { continue } |
| + present.insert(person.peer) |
| + if let bytes = reading.bytes { |
| + audioArriving[person.peer] = bytes > (audioBytes[person.peer] ?? 0) |
| + audioBytes[person.peer] = bytes |
| + } |
| + remotePeakSinceSample = max(remotePeakSinceSample, reading.peak) |
| + levels.updatePeer( |
| + person.peer, name: person.name, |
| + sample: PeerAudioSample( |
| + reported: person.media, connected: reading.connected, |
| + arriving: audioArriving[person.peer] ?? false, level: reading.peak), |
| + notices: AudioWords.notices(name: person.name, media: person.media, me: me, muted: muted), |
| + now: now) |
| + } |
| + levels.settle(present: present) |
| + for id in Array(audioBytes.keys) where !present.contains(id) { |
| + audioBytes.removeValue(forKey: id) |
| + audioArriving.removeValue(forKey: id) |
| + } |
| + reportNoSound(levels.noSoundFrom) |
| + } |
| + |
| + /// §15.20's receiving half: who is reaching this app as silence. The |
| + /// core keeps only roster peers and reports on a change; this sends only |
| + /// on a change too, in order, off the main thread. |
| + private func reportNoSound(_ peers: [String]) { |
| + guard peers != reportedNoSound, let client = bridge else { return } |
| + reportedNoSound = peers |
| + let body = json(peers) ?? "[]" |
| + noSoundQueue.async { try? client.setNoSoundFrom(body) } |
| + } |
| + |
| public func retryJoin() { send(type: "join", to: "", gen: 0, payload: "") } |
| public func loadInvitation() async { |
| guard let client = bridge, !loadingInvitation else { return } |
| @@ -437,6 +556,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| remoteScreens = [:] |
| sharingScreen = false |
| endScreenWatch() |
| + endLevelWatch() |
| localTrack = nil |
| cameraChanged() |
| default: status = "Connecting…" |
| @@ -639,6 +759,16 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| values["webrtcGainControl"] = running.gainControl.hasPrefix("Calls") || running.gainControl == "off" ? 0 : 1 |
| // Whether the readout would say the microphone has started. |
| values["audioReadoutStarted"] = running.started ? 1 : 0 |
| + // §15.20's meters, as the tiles see them: the loudest remote sound and |
| + // the loudest sent sound since the last sample, in thousandths of full |
| + // scale; who is in "No sound from"; and whether your own tile says |
| + // the microphone is picking nothing up. |
| + values["remoteAudioPeakPermille"] = UInt64((remotePeakSinceSample * 1000).rounded()) |
| + values["micPeakPermille"] = UInt64((micPeakSinceSample * 1000).rounded()) |
| + remotePeakSinceSample = 0 |
| + micPeakSinceSample = 0 |
| + values["noSoundFrom"] = UInt64(levels.noSoundFrom.count) |
| + values["micSilent"] = levels.mic.silent ? 1 : 0 |
| return values |
| } |
| } |
diff --git a/native/apple/Sources/CallsKit/CallAudio.swift b/native/apple/Sources/CallsKit/CallAudio.swift| index 5489cf0..05772fa 100644 |
| --- a/native/apple/Sources/CallsKit/CallAudio.swift |
| +++ b/native/apple/Sources/CallsKit/CallAudio.swift |
| @@ -171,7 +171,7 @@ public struct ProcessingReport: Equatable, Sendable { |
| // canceller. WebRTC's software processing is what ran before this |
| // module, and is what runs now. |
| _ = module.setPlatformVoiceProcessingAllowed(false) |
| - devices = EngineWatch(routes: routes, input: input) |
| + devices = EngineWatch(routes: routes, input: input, channels: InputChannelMapper(defaults: routes.defaults)) |
| module.observer = devices |
| if input != nil { _ = module.setManualRenderingMode(true) } |
| // Attached in manual rendering too, so the interop harness drives the |
| @@ -212,6 +212,13 @@ public struct ProcessingReport: Equatable, Sendable { |
| #endif |
| } |
| + /// Whether the capture hook has been handed audio since this call's |
| + /// track was made: the microphone meter's "capturing" on iPhone, where |
| + /// `capturing` cannot tell a microphone that has not started (alone in a |
| + /// call) from one that has, and a warning about silence from a |
| + /// microphone nobody has opened would be wrong. |
| + var hookFlowing: Bool { denoiser.buffersSeen > buffersAtStart } |
| + |
| /// The suppressor a call starts on. See `upgrade`. |
| static func makeSuppressor() -> NoiseSuppressor { RNNoiseSuppressor() } |
| @@ -311,15 +318,20 @@ public struct ProcessingReport: Equatable, Sendable { |
| #if os(macOS) |
| /// The module's observer. The device-list callback keeps the pickers |
| /// current; configuring the input is where a synthetic source is wired |
| - /// in. The rest of the protocol is the engine lifecycle, left alone |
| - /// (every one of those answers 0, "carry on"). |
| + /// in, or — for a real microphone — where the chosen input of a |
| + /// multi-channel one is mapped (InputChannel.swift). The rest of the |
| + /// protocol is the engine lifecycle, left alone (every one of those |
| + /// answers 0, "carry on") except that the map is applied once more just |
| + /// before the engine starts. |
| private final class EngineWatch: NSObject, RTCAudioDeviceModuleDelegate, @unchecked Sendable { |
| weak var routes: AudioRoutes? |
| let input: AudioInputSource? |
| + let channels: InputChannelMapper |
| private var node: AVAudioSourceNode? |
| - init(routes: AudioRoutes, input: AudioInputSource?) { |
| + init(routes: AudioRoutes, input: AudioInputSource?, channels: InputChannelMapper) { |
| self.routes = routes |
| self.input = input |
| + self.channels = channels |
| } |
| func audioDeviceModuleDidUpdateDevices(_ audioDeviceModule: RTCAudioDeviceModule) { |
| @@ -338,7 +350,15 @@ public struct ProcessingReport: Equatable, Sendable { |
| func audioDeviceModule( |
| _ audioDeviceModule: RTCAudioDeviceModule, willStartEngine engine: AVAudioEngine, |
| isPlayoutEnabled: Bool, isRecordingEnabled: Bool |
| - ) -> Int { 0 } |
| + ) -> Int { |
| + // By now the engine has connected the input and set the unit's |
| + // client format, so the map is sized to what it will run with. |
| + // Only with the microphone in use: asking for `inputNode` makes one. |
| + if input == nil, isRecordingEnabled, !engine.isInManualRenderingMode { |
| + channels.apply(to: engine.inputNode) |
| + } |
| + return 0 |
| + } |
| func audioDeviceModule( |
| _ audioDeviceModule: RTCAudioDeviceModule, didStopEngine engine: AVAudioEngine, |
| isPlayoutEnabled: Bool, isRecordingEnabled: Bool |
| @@ -355,8 +375,15 @@ public struct ProcessingReport: Equatable, Sendable { |
| configureInputFromSource source: AVAudioNode?, toDestination destination: AVAudioNode, |
| format: AVAudioFormat, context: [AnyHashable: Any] |
| ) -> Int { |
| - guard let input, |
| - let mono = AVAudioFormat(standardFormatWithSampleRate: format.sampleRate, channels: 1) |
| + // A real microphone: map the chosen input and connect nothing. With |
| + // nothing connected to the mixer the module makes its own default |
| + // connection, input node to mixer in mono (audio_engine_device.mm, |
| + // "Nothing connected to input mixer, connecting input node"). |
| + guard let input else { |
| + channels.apply(to: source) |
| + return 0 |
| + } |
| + guard let mono = AVAudioFormat(standardFormatWithSampleRate: format.sampleRate, channels: 1) |
| else { return 0 } |
| if let node { engine.detach(node) } |
| let rate = format.sampleRate |
diff --git a/native/apple/Sources/CallsKit/DevicePanel.swift b/native/apple/Sources/CallsKit/DevicePanel.swift| index 74c018f..f3b06a2 100644 |
| --- a/native/apple/Sources/CallsKit/DevicePanel.swift |
| +++ b/native/apple/Sources/CallsKit/DevicePanel.swift |
| @@ -43,6 +43,8 @@ import SwiftUI |
| macAudio |
| #endif |
| + MicLevelRow(mic: call.levels.mic) |
| + |
| if !call.deviceNote.isEmpty { |
| Label(call.deviceNote, systemImage: "exclamationmark.circle.fill") |
| .font(.callout).foregroundStyle(.red) |
| @@ -210,6 +212,7 @@ import SwiftUI |
| }), |
| options: audio.inputs.map { ($0.id, $0.name) }, identifier: "microphonePicker") |
| } |
| + InputChannelPicker(audio: audio) |
| if !audio.outputs.isEmpty { |
| picker( |
| "Speaker", |
diff --git a/native/apple/Sources/CallsKit/DevicePanelAudio.swift b/native/apple/Sources/CallsKit/DevicePanelAudio.swift| new file mode 100644 |
| index 0000000..2c06cee |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/DevicePanelAudio.swift |
| @@ -0,0 +1,74 @@ |
| +import SwiftUI |
| + |
| +// The device panel's two §15.20 rows, apart from DevicePanel so the panel |
| +// changes by a line each: what your microphone is sending, measured, and |
| +// on a Mac with an audio interface, which of its inputs. |
| + |
| +/// "Microphone level": the meter from your own tile, in the panel where a |
| +/// person looks when they are told they cannot be heard — which, before |
| +/// this, showed a microphone name and nothing measured at all. |
| +public struct MicLevelRow: View { |
| + @ObservedObject var mic: MicAudio |
| + public init(mic: MicAudio) { self.mic = mic } |
| + public var body: some View { |
| + VStack(alignment: .leading, spacing: 4) { |
| + HStack(spacing: 8) { |
| + Text("Microphone level").font(.subheadline) |
| + AudioLevelBars(step: mic.muted ? 0 : mic.step, height: 14) |
| + .padding(.horizontal, 6).padding(.vertical, 4) |
| + .background(Color.black.opacity(0.75), in: Capsule()) |
| + Text(words).font(.footnote).foregroundStyle(.secondary) |
| + } |
| + if mic.silent { |
| + Label(AudioWords.micSilent, systemImage: "exclamationmark.triangle.fill") |
| + .font(.callout).foregroundStyle(.orange) |
| + .fixedSize(horizontal: false, vertical: true) |
| + .accessibilityIdentifier("micSilent") |
| + Text(AudioWords.micSilentDetail(source: mic.source, channels: mic.channels)) |
| + .font(.footnote).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + } |
| + } |
| + .accessibilityElement(children: .combine) |
| + .accessibilityIdentifier("micLevel") |
| + } |
| + |
| + /// The meter in words, so it is never only a picture. |
| + private var words: String { |
| + if mic.muted { return "Muted" } |
| + if mic.silent { return "Silent" } |
| + return mic.step == 0 ? "Quiet" : mic.speaking ? "Hearing you" : "Picking up sound" |
| + } |
| +} |
| + |
| +#if os(macOS) |
| + /// "Input", for a microphone with more than one: an audio interface's |
| + /// inputs by number and, where the driver names them, by name. The call |
| + /// sends the chosen one, and the choice is remembered for the device. |
| + public struct InputChannelPicker: View { |
| + @ObservedObject var audio: AudioRoutes |
| + public init(audio: AudioRoutes) { self.audio = audio } |
| + public var body: some View { |
| + if audio.inputChannels.count > 1 { |
| + VStack(alignment: .leading, spacing: 4) { |
| + Picker( |
| + "Input", |
| + selection: Binding( |
| + get: { audio.inputChannel }, set: { audio.use(inputChannel: $0) }) |
| + ) { |
| + ForEach(Array(audio.inputChannels.enumerated()), id: \.offset) { index, label in |
| + Text(label).tag(index) |
| + } |
| + } |
| + .pickerStyle(.menu) |
| + .accessibilityIdentifier("inputChannelPicker") |
| + Text( |
| + "This microphone has \(audio.inputChannels.count) inputs, and the call sends one. Choose the one your microphone is plugged into; changing it restarts the microphone for a moment." |
| + ) |
| + .font(.footnote).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + } |
| + } |
| + } |
| + } |
| +#endif |
diff --git a/native/apple/Sources/CallsKit/InputChannel.swift b/native/apple/Sources/CallsKit/InputChannel.swift| new file mode 100644 |
| index 0000000..a50db7a |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/InputChannel.swift |
| @@ -0,0 +1,241 @@ |
| +#if os(macOS) |
| + import AudioToolbox |
| + import AVFoundation |
| + import CoreAudio |
| + import Foundation |
| + |
| + // Spec §15.20: which input of a multi-channel microphone the call sends. |
| + // |
| + // LiveKit's AVAudioEngine module connects the microphone to its mixer in |
| + // mono, at the hardware rate, and the conversion from a multi-channel |
| + // interface to mono keeps the first input only. On a Scarlett 18i8 with |
| + // the microphone on input 2 the call sent silence, and nothing on the |
| + // Mac said so (Paul and Tony, 2026-09-25). |
| + // |
| + // The fix is the input unit's own channel map: every channel it hands the |
| + // engine is set to the chosen device input, so whatever the engine does |
| + // to make mono of them, mono is that input. The module is told nothing; |
| + // its default connection stays in place. |
| + |
| + /// The device a call's microphone is on, as CoreAudio describes it. |
| + public struct InputDeviceChannels: Equatable, Sendable { |
| + public var uid: String |
| + public var name: String |
| + /// Input channels on the device, 1 for an ordinary microphone. |
| + public var count: Int |
| + /// CoreAudio's name for each input channel, where the driver gives |
| + /// one ("Analogue 1"); empty strings where it does not. |
| + public var names: [String] |
| + |
| + public init(uid: String, name: String, count: Int, names: [String] = []) { |
| + self.uid = uid |
| + self.name = name |
| + self.count = count |
| + self.names = names |
| + } |
| + |
| + /// "Input 2", or "Input 2 (Analogue 2)" where the channel has a name |
| + /// that says more than its number. `index` is 0-based. |
| + public func label(_ index: Int) -> String { |
| + let base = "Input \(index + 1)" |
| + guard index < names.count else { return base } |
| + let name = names[index].trimmingCharacters(in: .whitespaces) |
| + if name.isEmpty || name == "\(index + 1)" { return base } |
| + return "\(base) (\(name))" |
| + } |
| + |
| + /// What the ⓘ and the readout call it: "Scarlett 18i8 USB, input 1", |
| + /// or just the name for a one-channel microphone. |
| + public func describe(channel: Int) -> String { |
| + count > 1 ? "\(name), input \(channel + 1)" : name |
| + } |
| + } |
| + |
| + /// The remembered input for each device, by CoreAudio UID. Input 1 (0) |
| + /// is where every device starts, which is what the module always did. |
| + public struct InputChannelPreference { |
| + static func key(_ uid: String) -> String { "audioInputChannel.\(uid)" } |
| + |
| + /// The remembered channel, 0-based, clamped to the channels the |
| + /// device has now (an interface can be reconfigured with fewer). |
| + public static func load(uid: String, channels: Int, _ defaults: UserDefaults) -> Int { |
| + let saved = defaults.integer(forKey: key(uid)) |
| + return saved >= 0 && saved < max(channels, 1) ? saved : 0 |
| + } |
| + |
| + public static func save(_ channel: Int, uid: String, _ defaults: UserDefaults) { |
| + if channel <= 0 { |
| + defaults.removeObject(forKey: key(uid)) |
| + } else { |
| + defaults.set(channel, forKey: key(uid)) |
| + } |
| + } |
| + } |
| + |
| + /// The pure parts of the channel map, apart from any device. |
| + public enum InputChannelMap { |
| + /// The map for an input unit whose client side has `clientChannels` |
| + /// channels: every one of them carries device input `channel` |
| + /// (0-based), so a mono mixdown of them — or the first of them — is |
| + /// that input. Nil means leave the unit alone: input 1 on a unit this |
| + /// app has never mapped is exactly what the module does already. |
| + public static func map(channel: Int, clientChannels: Int, mappedBefore: Bool) -> [Int32]? { |
| + let width = max(clientChannels, 1) |
| + if channel <= 0 { |
| + // Undo an earlier choice on the same unit: straight through. |
| + return mappedBefore ? (0..<width).map { Int32($0) } : nil |
| + } |
| + return Array(repeating: Int32(channel), count: width) |
| + } |
| + |
| + /// Input channels in a stream configuration: the sum over its buffers. |
| + public static func count(_ list: UnsafeMutableAudioBufferListPointer) -> Int { |
| + list.reduce(0) { $0 + Int($1.mNumberChannels) } |
| + } |
| + } |
| + |
| + /// CoreAudio, read. Every call here is a property query on the HAL; none |
| + /// opens a device or needs microphone permission. |
| + enum CoreAudioInput { |
| + /// The device for a module device id: its CoreAudio UID, or "default" |
| + /// (the module's own name for the system default). |
| + static func device(forModuleID id: String?) -> AudioDeviceID? { |
| + if let id, id != "default", !id.isEmpty, let device = device(uid: id) { return device } |
| + return defaultInput() |
| + } |
| + |
| + static func device(uid: String) -> AudioDeviceID? { |
| + var address = AudioObjectPropertyAddress( |
| + mSelector: kAudioHardwarePropertyTranslateUIDToDevice, |
| + mScope: kAudioObjectPropertyScopeGlobal, mElement: kAudioObjectPropertyElementMain) |
| + var cf = uid as CFString |
| + var device = AudioDeviceID(kAudioObjectUnknown) |
| + var size = UInt32(MemoryLayout<AudioDeviceID>.size) |
| + let status = withUnsafeMutablePointer(to: &cf) { pointer in |
| + AudioObjectGetPropertyData( |
| + AudioObjectID(kAudioObjectSystemObject), &address, |
| + UInt32(MemoryLayout<CFString>.size), pointer, &size, &device) |
| + } |
| + return status == noErr && device != kAudioObjectUnknown ? device : nil |
| + } |
| + |
| + static func defaultInput() -> AudioDeviceID? { |
| + var address = AudioObjectPropertyAddress( |
| + mSelector: kAudioHardwarePropertyDefaultInputDevice, |
| + mScope: kAudioObjectPropertyScopeGlobal, mElement: kAudioObjectPropertyElementMain) |
| + var device = AudioDeviceID(kAudioObjectUnknown) |
| + var size = UInt32(MemoryLayout<AudioDeviceID>.size) |
| + let status = AudioObjectGetPropertyData( |
| + AudioObjectID(kAudioObjectSystemObject), &address, 0, nil, &size, &device) |
| + return status == noErr && device != kAudioObjectUnknown ? device : nil |
| + } |
| + |
| + static func string(_ device: AudioDeviceID, _ selector: AudioObjectPropertySelector, |
| + scope: AudioObjectPropertyScope = kAudioObjectPropertyScopeGlobal, |
| + element: AudioObjectPropertyElement = kAudioObjectPropertyElementMain |
| + ) -> String? { |
| + var address = AudioObjectPropertyAddress(mSelector: selector, mScope: scope, mElement: element) |
| + guard AudioObjectHasProperty(device, &address) else { return nil } |
| + var value: Unmanaged<CFString>? |
| + var size = UInt32(MemoryLayout<Unmanaged<CFString>?>.size) |
| + let status = AudioObjectGetPropertyData(device, &address, 0, nil, &size, &value) |
| + guard status == noErr, let value else { return nil } |
| + return value.takeRetainedValue() as String |
| + } |
| + |
| + static func inputChannelCount(_ device: AudioDeviceID) -> Int { |
| + var address = AudioObjectPropertyAddress( |
| + mSelector: kAudioDevicePropertyStreamConfiguration, |
| + mScope: kAudioObjectPropertyScopeInput, mElement: kAudioObjectPropertyElementMain) |
| + var size: UInt32 = 0 |
| + guard AudioObjectGetPropertyDataSize(device, &address, 0, nil, &size) == noErr, size > 0 |
| + else { return 0 } |
| + let raw = UnsafeMutableRawPointer.allocate( |
| + byteCount: Int(size), alignment: MemoryLayout<AudioBufferList>.alignment) |
| + defer { raw.deallocate() } |
| + let list = raw.bindMemory(to: AudioBufferList.self, capacity: 1) |
| + guard AudioObjectGetPropertyData(device, &address, 0, nil, &size, list) == noErr else { |
| + return 0 |
| + } |
| + return InputChannelMap.count(UnsafeMutableAudioBufferListPointer(list)) |
| + } |
| + |
| + static func describe(_ device: AudioDeviceID) -> InputDeviceChannels? { |
| + guard let uid = string(device, kAudioDevicePropertyDeviceUID) else { return nil } |
| + let name = string(device, kAudioObjectPropertyName) ?? "Microphone" |
| + let count = inputChannelCount(device) |
| + // Names only matter for a picker, which only a multi-channel device |
| + // gets; a USB interface's driver usually names its inputs. |
| + let names = |
| + count > 1 |
| + ? (1...count).map { |
| + string( |
| + device, kAudioObjectPropertyElementName, scope: kAudioObjectPropertyScopeInput, |
| + element: AudioObjectPropertyElement($0)) ?? "" |
| + } : [] |
| + return InputDeviceChannels(uid: uid, name: name, count: count, names: names) |
| + } |
| + |
| + /// The device an input unit is on. |
| + static func device(of unit: AudioUnit) -> AudioDeviceID? { |
| + var device = AudioDeviceID(kAudioObjectUnknown) |
| + var size = UInt32(MemoryLayout<AudioDeviceID>.size) |
| + let status = AudioUnitGetProperty( |
| + unit, kAudioOutputUnitProperty_CurrentDevice, kAudioUnitScope_Global, 0, &device, &size) |
| + return status == noErr && device != kAudioObjectUnknown ? device : nil |
| + } |
| + |
| + /// Channels on the input unit's client side: what the engine is handed. |
| + static func clientChannels(of unit: AudioUnit) -> Int { |
| + var format = AudioStreamBasicDescription() |
| + var size = UInt32(MemoryLayout<AudioStreamBasicDescription>.size) |
| + let status = AudioUnitGetProperty( |
| + unit, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &format, &size) |
| + return status == noErr ? Int(format.mChannelsPerFrame) : 0 |
| + } |
| + |
| + static func setChannelMap(_ map: [Int32], on unit: AudioUnit) -> OSStatus { |
| + var map = map |
| + return AudioUnitSetProperty( |
| + unit, kAudioOutputUnitProperty_ChannelMap, kAudioUnitScope_Output, 1, &map, |
| + UInt32(MemoryLayout<Int32>.size * map.count)) |
| + } |
| + } |
| + |
| + /// Applies the remembered channel to the engine module's input unit. |
| + /// Called from the module's observer on its own thread, when it wires the |
| + /// microphone in and again just before it starts the engine (by which |
| + /// time the unit's client format is the one it will run with). |
| + final class InputChannelMapper: @unchecked Sendable { |
| + private let defaults: UserDefaults |
| + private let lock = NSLock() |
| + private var mapped = false |
| + /// What was last applied, for the readout: the device and the channel. |
| + private(set) var applied: (uid: String, channel: Int)? |
| + |
| + init(defaults: UserDefaults) { self.defaults = defaults } |
| + |
| + func apply(to node: AVAudioNode?) { |
| + guard let node = node as? AVAudioInputNode, let unit = node.audioUnit, |
| + let device = CoreAudioInput.device(of: unit), |
| + let uid = CoreAudioInput.string(device, kAudioDevicePropertyDeviceUID) |
| + else { return } |
| + let count = CoreAudioInput.inputChannelCount(device) |
| + let channel = InputChannelPreference.load(uid: uid, channels: count, defaults) |
| + lock.lock() |
| + defer { lock.unlock() } |
| + guard |
| + let map = InputChannelMap.map( |
| + channel: channel, clientChannels: CoreAudioInput.clientChannels(of: unit), |
| + mappedBefore: mapped) |
| + else { |
| + applied = (uid, 0) |
| + return |
| + } |
| + if CoreAudioInput.setChannelMap(map, on: unit) == noErr { |
| + mapped = channel > 0 |
| + applied = (uid, channel) |
| + } |
| + } |
| + } |
| +#endif |
diff --git a/native/apple/Sources/CallsKit/MacCallView.swift b/native/apple/Sources/CallsKit/MacCallView.swift| index e3607bf..3f86267 100644 |
| --- a/native/apple/Sources/CallsKit/MacCallView.swift |
| +++ b/native/apple/Sources/CallsKit/MacCallView.swift |
| @@ -200,13 +200,14 @@ |
| } |
| MacVideoTile( |
| track: call.localTrack, label: "You", mirrored: true, cameraOff: call.cameraOff, |
| - placeholder: call.selfPlaceholder |
| + placeholder: call.selfPlaceholder, audio: call.selfAudio |
| ) |
| .aspectRatio(16 / 9, contentMode: .fit) |
| ForEach(peers) { peer in |
| MacVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| - muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer) |
| + muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .aspectRatio(16 / 9, contentMode: .fit) |
| } |
| @@ -223,7 +224,8 @@ |
| case .camera(let peer): |
| MacVideoTile( |
| track: call.remoteTracks[peer], label: call.name(of: peer), |
| - muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer)) |
| + muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer), |
| + audio: call.tileAudio(peer)) |
| case nil: |
| ContentUnavailableView( |
| "Waiting for others", systemImage: "person.2", |
| @@ -240,13 +242,14 @@ |
| ForEach(others) { peer in |
| MacVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| - muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer) |
| + muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .frame(width: size.width, height: size.height) |
| } |
| MacVideoTile( |
| track: call.localTrack, label: "You", mirrored: true, cameraOff: call.cameraOff, |
| - placeholder: call.selfPlaceholder |
| + placeholder: call.selfPlaceholder, audio: call.selfAudio |
| ) |
| .frame(width: size.width, height: size.height) |
| } |
diff --git a/native/apple/Sources/CallsKit/MacFullWindowCall.swift b/native/apple/Sources/CallsKit/MacFullWindowCall.swift| index 3e2a2cf..fd0f26f 100644 |
| --- a/native/apple/Sources/CallsKit/MacFullWindowCall.swift |
| +++ b/native/apple/Sources/CallsKit/MacFullWindowCall.swift |
| @@ -98,7 +98,7 @@ |
| MacVideoTile( |
| track: call.remoteTracks[peer], label: call.name(of: peer), |
| muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer), |
| - cornerRadius: 0, compact: true, labelLift: lift) |
| + cornerRadius: 0, compact: true, labelLift: lift, audio: call.tileAudio(peer)) |
| case nil: |
| ContentUnavailableView( |
| "Waiting for others", systemImage: "person.2", |
| @@ -118,7 +118,8 @@ |
| MacVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| - cornerRadius: 10, compact: true, backdrop: Self.tileBackdrop |
| + cornerRadius: 10, compact: true, backdrop: Self.tileBackdrop, |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .frame(width: thumb.width, height: thumb.height) |
| } |
| @@ -162,14 +163,15 @@ |
| MacVideoTile( |
| track: call.localTrack, label: "You", muted: call.muted, mirrored: true, |
| cameraOff: call.cameraOff, placeholder: call.selfPlaceholder, cornerRadius: 6, |
| - compact: true, backdrop: Self.tileBackdrop |
| + compact: true, backdrop: Self.tileBackdrop, audio: call.selfAudio |
| ) |
| .frame(width: tile.width, height: tile.height) |
| ForEach(peers) { peer in |
| MacVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| - cornerRadius: 6, compact: true, backdrop: Self.tileBackdrop |
| + cornerRadius: 6, compact: true, backdrop: Self.tileBackdrop, |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .frame(width: tile.width, height: tile.height) |
| } |
| @@ -192,7 +194,7 @@ |
| return MacVideoTile( |
| track: call.localTrack, label: "You", muted: call.muted, mirrored: true, |
| cameraOff: call.cameraOff, placeholder: call.selfPlaceholder, cornerRadius: 10, |
| - compact: true, backdrop: Self.tileBackdrop |
| + compact: true, backdrop: Self.tileBackdrop, audio: call.selfAudio |
| ) |
| .frame(width: pip.width, height: pip.height) |
| .overlay(RoundedRectangle(cornerRadius: 10).strokeBorder(.white.opacity(0.18))) |
diff --git a/native/apple/Sources/CallsKit/MacVideoTile.swift b/native/apple/Sources/CallsKit/MacVideoTile.swift| index 773ecf8..38f9380 100644 |
| --- a/native/apple/Sources/CallsKit/MacVideoTile.swift |
| +++ b/native/apple/Sources/CallsKit/MacVideoTile.swift |
| @@ -27,8 +27,16 @@ |
| /// does not fill the tile. A tile among others on the full-window |
| /// stage is a shade lighter than the stage, so it reads as a tile. |
| var backdrop = Color.black |
| + /// The sound on this tile (spec §15.20): your microphone's meter, or a |
| + /// remote person's with "No sound from" and their notices. Nil for a |
| + /// screen. |
| + var audio: TileAudio? = nil |
| var body: some View { |
| + TileAudioReader(audio) { info in tile(info) } |
| + } |
| + |
| + private func tile(_ info: TileAudioInfo?) -> some View { |
| ZStack { |
| backdrop |
| if let track, !cameraOff, placeholder == nil { |
| @@ -45,18 +53,27 @@ |
| } |
| } |
| .overlay(alignment: .bottomLeading) { |
| - if compact { |
| - TileNameLabel(label: label, muted: muted).padding(8).padding(.bottom, labelLift) |
| - } else { |
| - Text(label).font(.callout.bold()).foregroundStyle(.white) |
| - .lineLimit(1).padding(.horizontal, 10).padding(.vertical, 6) |
| - .background(.black.opacity(0.65), in: Capsule()).padding(10) |
| + VStack(alignment: .leading, spacing: 6) { |
| + TileAudioNotices(info) |
| + HStack(spacing: 6) { |
| + if compact { |
| + TileNameLabel(label: label, muted: muted) |
| + } else { |
| + Text(label).font(.callout.bold()).foregroundStyle(.white) |
| + .lineLimit(1).padding(.horizontal, 10).padding(.vertical, 6) |
| + .background(.black.opacity(0.65), in: Capsule()) |
| + } |
| + TileAudioStatus(info) |
| + } |
| } |
| + .padding(compact ? 8 : 10).padding(.bottom, compact ? labelLift : 0) |
| } |
| .overlay(alignment: .topLeading) { if muted, !compact { MutedBadge().padding(10) } } |
| + .speakingRing(info?.speaking == true, cornerRadius: cornerRadius) |
| .clipShape(RoundedRectangle(cornerRadius: cornerRadius)) |
| .accessibilityElement(children: .ignore) |
| - .accessibilityLabel(tileAccessibilityLabel(label, muted: muted)) |
| + .accessibilityLabel(tileAccessibilityLabel(label, muted: muted, audio: info)) |
| + .accessibilityHint(info?.detail ?? "") |
| .accessibilityValue(cameraOff ? "Camera off" : track == nil || placeholder != nil ? waiting : "Video on") |
| } |
| private var waiting: String { |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 47c45eb..06b45c0 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -267,6 +267,36 @@ import LiveKitWebRTC |
| return values |
| } |
| + /// Each remote person's sound, for the meters (spec §15.20): the level |
| + /// their receive tap has seen since the last look, whether the |
| + /// connection is up, and — when `stats` is set — the audio bytes |
| + /// received so far, one `statistics(for:)` per audio receiver rather |
| + /// than the whole report. Nil bytes when not asked. |
| + func peerAudio(stats: Bool) async -> [String: PeerAudioReading] { |
| + var out: [String: PeerAudioReading] = [:] |
| + for (id, peer) in peers { |
| + guard let pc = peer.pc else { continue } |
| + let (peak, buffers) = peer.tap.take() |
| + let state = pc.iceConnectionState |
| + var reading = PeerAudioReading( |
| + peak: peak, buffers: buffers, connected: state == .connected || state == .completed) |
| + if stats { |
| + var bytes: UInt64 = 0 |
| + for receiver in pc.receivers where receiver.track?.kind == "audio" { |
| + let report: RTCStatisticsReport = await withCheckedContinuation { c in |
| + pc.statistics(for: receiver) { c.resume(returning: $0) } |
| + } |
| + for stat in report.statistics.values where stat.type == "inbound-rtp" { |
| + bytes += (stat.values["bytesReceived"] as? NSNumber)?.uint64Value ?? 0 |
| + } |
| + } |
| + reading.bytes = bytes |
| + } |
| + out[id] = reading |
| + } |
| + return out |
| + } |
| + |
| /// Caps every connection's bandwidth estimate, which is what a thin |
| /// uplink does to it. For the gate, which has no thin uplink to test on; |
| /// nil takes the cap off. |
| @@ -308,8 +338,20 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| } |
| } |
| +/// One look at a remote person's audio (see MediaEngine.peerAudio). |
| +struct PeerAudioReading: Equatable, Sendable { |
| + var peak: Float |
| + var buffers: Int |
| + var connected: Bool |
| + var bytes: UInt64? |
| +} |
| + |
| @MainActor private final class MediaPeer: NSObject, RTCPeerConnectionDelegate { |
| let id: String |
| + /// Measures every audio track that arrives on this connection (their |
| + /// microphone; a browser's shared tab audio would add to it). |
| + nonisolated let tap = RemoteAudioTap() |
| + private var tapped: [RTCAudioTrack] = [] |
| let generation: Int64 |
| let polite: Bool |
| nonisolated let playAudio: Bool |
| @@ -509,10 +551,16 @@ private func sdpDirection(_ direction: RTCRtpTransceiverDirection) -> String { |
| if role == .camera, let video = track as? RTCVideoTrack { |
| engine?.call?.setRemote(video, peer: id) |
| } |
| + if let audio = track as? RTCAudioTrack, !tapped.contains(where: { $0 === audio }) { |
| + audio.add(tap) |
| + tapped.append(audio) |
| + } |
| } |
| func close() { |
| closed = true |
| + for audio in tapped { audio.remove(tap) } |
| + tapped = [] |
| screenTx = nil |
| work?.cancel() |
| work = nil |
diff --git a/native/apple/Sources/CallsKit/MiniWindow.swift b/native/apple/Sources/CallsKit/MiniWindow.swift| index 841a696..d0ac85c 100644 |
| --- a/native/apple/Sources/CallsKit/MiniWindow.swift |
| +++ b/native/apple/Sources/CallsKit/MiniWindow.swift |
| @@ -428,11 +428,12 @@ |
| case .camera(let peer): |
| MacVideoTile( |
| track: call.remoteTracks[peer], label: call.name(of: peer), |
| - muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer)) |
| + muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer), |
| + audio: call.tileAudio(peer)) |
| case nil: |
| MacVideoTile( |
| track: call.localTrack, label: "You", mirrored: true, cameraOff: call.cameraOff, |
| - placeholder: call.selfPlaceholder) |
| + placeholder: call.selfPlaceholder, audio: call.selfAudio) |
| } |
| } |
| // The tile is a picture, not a target: clicks and drags go through |
| @@ -484,9 +485,9 @@ |
| /// the call is with. Click it to open the mini window again, drag it out |
| /// to open it there, double-click it for the whole call. |
| /// |
| - /// There is no speaking indicator: nothing in the app measures audio |
| - /// level live (the audio readout polls every five seconds), and a light |
| - /// that lags speech by seconds says the wrong thing. |
| + /// A small waveform shows while someone else is speaking, from the |
| + /// receive meters (spec §15.20), which read five times a second — quick |
| + /// enough to follow speech, where the five-second audio readout was not. |
| struct MiniTabView: View { |
| @ObservedObject var call: Call |
| let actions: MiniWindowActions |
| @@ -509,6 +510,7 @@ |
| Text(caption).font(.callout.weight(.medium)).lineLimit(1).truncationMode(.tail) |
| .foregroundStyle(.white) |
| Spacer(minLength: 0) |
| + SpeakingGlyph(speaking: call.levels.anyoneSpeaking) |
| } |
| .padding(.horizontal, 7) |
| .frame(width: MiniWindowTab.size.width, height: MiniWindowTab.size.height) |
diff --git a/native/apple/Sources/CallsKit/RemoteAudioTap.swift b/native/apple/Sources/CallsKit/RemoteAudioTap.swift| new file mode 100644 |
| index 0000000..4ad0b29 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/RemoteAudioTap.swift |
| @@ -0,0 +1,61 @@ |
| +import AVFoundation |
| +import Foundation |
| +import LiveKitWebRTC |
| +import os |
| + |
| +/// The level of one person's decoded audio, for the meter on their tile. |
| +/// |
| +/// A renderer on their remote audio track: WebRTC hands it every 10 ms |
| +/// frame the moment it is decoded for playout, before the output volume |
| +/// is applied (the track's sink sits ahead of the gain in the receive |
| +/// channel). So it measures what arrived, whatever the app then does with |
| +/// the volume — including the interop harness, which plays at zero. |
| +/// |
| +/// Chosen over inbound-rtp `audioLevel`, which is measured *after* that |
| +/// gain and would read silence at volume zero, and which would take a |
| +/// statistics walk per peer five times a second. The statistics are still |
| +/// read, once a second, for the other half of "No sound from": whether |
| +/// packets are arriving at all (MediaEngine.peerAudio). |
| +/// |
| +/// Called on WebRTC's audio thread: nothing here blocks or waits. |
| +final class RemoteAudioTap: NSObject, RTCAudioRenderer, @unchecked Sendable { |
| + private struct Reading { |
| + var peak: Float = 0 |
| + var buffers = 0 |
| + } |
| + private let lock = OSAllocatedUnfairLock(initialState: Reading()) |
| + |
| + func render(pcmBuffer: AVAudioPCMBuffer) { |
| + let frames = Int(pcmBuffer.frameLength) |
| + guard frames > 0 else { return } |
| + var top: Float = 0 |
| + if let int16 = pcmBuffer.int16ChannelData { |
| + // The adapter's buffers are interleaved int16: one pointer, every |
| + // channel's samples in it. |
| + let channels = pcmBuffer.format.isInterleaved ? Int(pcmBuffer.format.channelCount) : 1 |
| + let samples = int16[0] |
| + var peak: Int32 = 0 |
| + for i in 0..<(frames * channels) { peak = max(peak, abs(Int32(samples[i]))) } |
| + top = Float(peak) / 32768 |
| + } else if let float = pcmBuffer.floatChannelData { |
| + let samples = float[0] |
| + for i in 0..<frames { top = max(top, abs(samples[i])) } |
| + } else { |
| + return |
| + } |
| + let level = min(top, 1) |
| + lock.withLock { r in |
| + r.peak = max(r.peak, level) |
| + r.buffers += 1 |
| + } |
| + } |
| + |
| + /// The loudest sample since the last call, and how many frames came. |
| + func take() -> (peak: Float, buffers: Int) { |
| + lock.withLock { r in |
| + let out = (r.peak, r.buffers) |
| + r = Reading() |
| + return out |
| + } |
| + } |
| +} |
diff --git a/native/apple/Sources/CallsKit/Wire.swift b/native/apple/Sources/CallsKit/Wire.swift| index fd9d5a1..2b18c25 100644 |
| --- a/native/apple/Sources/CallsKit/Wire.swift |
| +++ b/native/apple/Sources/CallsKit/Wire.swift |
| @@ -11,10 +11,25 @@ public struct RosterPeer: Decodable, Identifiable { |
| public let media: PeerMedia? |
| public var id: String { peer } |
| } |
| -/// What a peer says about its own outgoing media (spec §15.19). |
| +/// What a peer says about its own outgoing media (spec §15.19), and about |
| +/// the sound reaching it (§15.20). |
| public struct PeerMedia: Decodable, Equatable, Sendable { |
| public let muted: Bool |
| public let cameraOff: Bool |
| + /// Its browser refused to play sound and is waiting for a click. Nil |
| + /// (absent) is none; the key only travels with bad news. |
| + public let audioBlocked: Bool? |
| + /// The peers whose audio reaches it as silence. Nil (absent) is none. |
| + public let noSoundFrom: [String]? |
| + |
| + public init( |
| + muted: Bool, cameraOff: Bool, audioBlocked: Bool? = nil, noSoundFrom: [String]? = nil |
| + ) { |
| + self.muted = muted |
| + self.cameraOff = cameraOff |
| + self.audioBlocked = audioBlocked |
| + self.noSoundFrom = noSoundFrom |
| + } |
| } |
| public struct RoomState: Decodable { |
| public let phase: String |
diff --git a/native/apple/Tests/CallsKitTests/AudioLevelTests.swift b/native/apple/Tests/CallsKitTests/AudioLevelTests.swift| new file mode 100644 |
| index 0000000..61ff778 |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/AudioLevelTests.swift |
| @@ -0,0 +1,235 @@ |
| +import XCTest |
| + |
| +@testable import CallsKit |
| + |
| +/// Spec §15.20's rules on the native side, on a clock the tests own: the |
| +/// meter's steps, the five-second "no sound" timers, what wins when a |
| +/// person is muted, silent and reporting on you all at once, and the words. |
| +final class AudioLevelTests: XCTestCase { |
| + private func level(db: Float) -> Float { pow(10, db / 20) } |
| + |
| + // MARK: Meter |
| + |
| + func testTheMeterStepsByDecibels() { |
| + XCTAssertEqual(MeterScale.step(0), 0) |
| + XCTAssertEqual(MeterScale.step(level(db: -90)), 0, "digital silence lights nothing") |
| + XCTAssertEqual(MeterScale.step(level(db: -55)), 1, "a room's tone lights the bottom bar") |
| + XCTAssertEqual(MeterScale.step(level(db: -40)), 2) |
| + XCTAssertEqual(MeterScale.step(level(db: -25)), 3) |
| + XCTAssertEqual(MeterScale.step(level(db: -15)), 4) |
| + XCTAssertEqual(MeterScale.step(1), MeterScale.steps, "full scale lights every bar") |
| + // Monotonic: louder never shows fewer bars. |
| + var last = 0 |
| + for db in stride(from: Float(-100), through: 0, by: 1) { |
| + let step = MeterScale.step(level(db: db)) |
| + XCTAssertGreaterThanOrEqual(step, last) |
| + last = step |
| + } |
| + } |
| + |
| + func testHeldCountsFromWhenTheConditionStarted() { |
| + var held = Held() |
| + XCTAssertEqual(held.update(true, now: 10), 0) |
| + XCTAssertEqual(held.update(true, now: 13), 3) |
| + XCTAssertEqual(held.update(false, now: 14), 0) |
| + XCTAssertEqual(held.update(true, now: 20), 0, "a break starts the count again") |
| + } |
| + |
| + func testSpeakingHasHysteresisAndAHangover() { |
| + var gate = SpeakingGate() |
| + XCTAssertFalse(gate.update(level: level(db: -36), now: 0), "between the thresholds: not yet") |
| + XCTAssertTrue(gate.update(level: level(db: -20), now: 0.2)) |
| + XCTAssertTrue(gate.update(level: level(db: -36), now: 0.4), "between the thresholds: still") |
| + XCTAssertTrue(gate.update(level: level(db: -60), now: 0.6), "a gap between syllables") |
| + XCTAssertTrue(gate.update(level: level(db: -60), now: 1.0)) |
| + XCTAssertFalse(gate.update(level: level(db: -60), now: 1.3), "quiet past the hangover") |
| + } |
| + |
| + // MARK: Receive side |
| + |
| + private let unmuted = PeerMedia(muted: false, cameraOff: false) |
| + |
| + private func sample( |
| + _ media: PeerMedia?, db: Float = -120, connected: Bool = true, arriving: Bool = true |
| + ) -> PeerAudioSample { |
| + PeerAudioSample(reported: media, connected: connected, arriving: arriving, level: level(db: db)) |
| + } |
| + |
| + func testNoSoundAfterFiveFlatSecondsWhileTheyAreUnmuted() { |
| + var watch = PeerAudioWatch() |
| + for t in stride(from: 0.0, through: 4.8, by: 0.2) { |
| + XCTAssertEqual(watch.update(sample(unmuted), now: t), .meter(0), "at \(t)s") |
| + } |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 5.0), .noSound) |
| + XCTAssertEqual(watch.update(sample(unmuted, db: -30), now: 5.2), .meter(3), "sound clears it at once") |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 5.4), .meter(0), "and the count starts again") |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 10.2), .meter(0)) |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 10.5), .noSound) |
| + } |
| + |
| + func testMutedWinsOverEverything() { |
| + var watch = PeerAudioWatch() |
| + let muted = PeerMedia(muted: true, cameraOff: false) |
| + for t in stride(from: 0.0, through: 10, by: 0.5) { |
| + XCTAssertEqual(watch.update(sample(muted), now: t), .muted) |
| + } |
| + XCTAssertFalse(watch.speaking) |
| + // Unmuting starts the five seconds from then, not from before. |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 10.5), .meter(0)) |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 14), .meter(0)) |
| + XCTAssertEqual(watch.update(sample(unmuted), now: 15.5), .noSound) |
| + } |
| + |
| + func testNoSoundNeedsAReportConnectionAndPackets() { |
| + // Unknown is not "reporting themselves unmuted" (a client from before |
| + // §15.19): a meter, never the sentence. |
| + var unknown = PeerAudioWatch() |
| + var down = PeerAudioWatch() |
| + var stalled = PeerAudioWatch() |
| + for t in stride(from: 0.0, through: 8, by: 0.2) { |
| + XCTAssertEqual(unknown.update(sample(nil), now: t), .meter(0)) |
| + XCTAssertEqual(down.update(sample(unmuted, connected: false), now: t), .meter(0)) |
| + XCTAssertEqual(stalled.update(sample(unmuted, arriving: false), now: t), .meter(0)) |
| + } |
| + } |
| + |
| + func testQuietButLiveIsNotSilence() { |
| + // A live microphone in a quiet room, after a browser's suppression: |
| + // far above the silence threshold, however long it lasts. |
| + var watch = PeerAudioWatch() |
| + for t in stride(from: 0.0, through: 20, by: 0.2) { |
| + XCTAssertNotEqual(watch.update(sample(unmuted, db: -70), now: t), .noSound) |
| + } |
| + } |
| + |
| + // MARK: Send side |
| + |
| + private func mic( |
| + muted: Bool = false, capturing: Bool = true, inputDb: Float = -120, buffers: Int = 10, |
| + sentDb: Float = -120 |
| + ) -> MicSample { |
| + MicSample( |
| + muted: muted, capturing: capturing, inputPeak: level(db: inputDb), inputBuffers: buffers, |
| + sentPeak: level(db: sentDb)) |
| + } |
| + |
| + func testTheMicrophoneWarningComesAfterFiveDeadSeconds() { |
| + var watch = MicWatch() |
| + for t in stride(from: 0.0, through: 4.9, by: 0.1) { |
| + watch.update(mic(), now: t) |
| + XCTAssertFalse(watch.silent, "at \(t)s") |
| + } |
| + watch.update(mic(), now: 5.0) |
| + XCTAssertTrue(watch.silent) |
| + watch.update(mic(inputDb: -60, sentDb: -75), now: 5.1) |
| + XCTAssertFalse(watch.silent, "any sound on the input clears it") |
| + } |
| + |
| + func testNoBuffersAtAllIsDeadToo() { |
| + var watch = MicWatch() |
| + for t in stride(from: 0.0, through: 5.2, by: 0.1) { |
| + watch.update(mic(inputDb: -20, buffers: 0), now: t) |
| + } |
| + XCTAssertTrue(watch.silent) |
| + } |
| + |
| + func testTheWarningAsksThePreSuppressorLevel() { |
| + // The suppressor can take a quiet room to near silence; the input |
| + // before it is what says whether the microphone hears anything. |
| + var watch = MicWatch() |
| + for t in stride(from: 0.0, through: 8, by: 0.1) { |
| + watch.update(mic(inputDb: -75, sentDb: -120), now: t) |
| + } |
| + XCTAssertFalse(watch.silent) |
| + XCTAssertEqual(watch.step, 0) |
| + } |
| + |
| + func testMutedOrNotCapturingNeverWarns() { |
| + var muted = MicWatch() |
| + var idle = MicWatch() |
| + for t in stride(from: 0.0, through: 10, by: 0.1) { |
| + muted.update(mic(muted: true, buffers: 0), now: t) |
| + idle.update(mic(capturing: false, buffers: 0), now: t) |
| + } |
| + XCTAssertFalse(muted.silent) |
| + XCTAssertFalse(idle.silent) |
| + // Unmuting starts the count afresh. |
| + muted.update(mic(), now: 10.1) |
| + muted.update(mic(), now: 14) |
| + XCTAssertFalse(muted.silent) |
| + muted.update(mic(), now: 15.2) |
| + XCTAssertTrue(muted.silent) |
| + } |
| + |
| + func testTheSelfMeterShowsWhatIsSent() { |
| + var watch = MicWatch() |
| + watch.update(mic(inputDb: -20, sentDb: -15), now: 0) |
| + XCTAssertEqual(watch.step, 4) |
| + XCTAssertTrue(watch.speaking) |
| + } |
| + |
| + // MARK: Words |
| + |
| + func testNoticesSayWhoAndWhat() { |
| + let both = PeerMedia(muted: false, cameraOff: false, audioBlocked: true, noSoundFrom: ["me", "x"]) |
| + XCTAssertEqual( |
| + AudioWords.notices(name: "Tony", media: both, me: "me", muted: false), |
| + [ |
| + "Tony’s browser is blocking sound — they need to click to hear you.", |
| + "Tony isn’t getting any sound from you.", |
| + ]) |
| + XCTAssertEqual( |
| + AudioWords.notices(name: "Tony", media: both, me: "me", muted: true), |
| + ["Tony’s browser is blocking sound — they need to click to hear you."], |
| + "muted, silence from you is expected") |
| + XCTAssertEqual( |
| + AudioWords.notices(name: "Tony", media: both, me: "other", muted: false), |
| + ["Tony’s browser is blocking sound — they need to click to hear you."], |
| + "someone else's silence is not yours") |
| + XCTAssertEqual(AudioWords.notices(name: "Tony", media: unmuted, me: "me", muted: false), []) |
| + XCTAssertEqual(AudioWords.notices(name: "Tony", media: nil, me: "me", muted: false), []) |
| + } |
| + |
| + func testTheWarningNamesTheDeviceAndChannel() { |
| + let text = AudioWords.micSilentDetail(source: "Scarlett 18i8 USB, input 1", channels: true) |
| + XCTAssertTrue(text.contains("Scarlett 18i8 USB, input 1")) |
| + XCTAssertTrue(text.contains("choose that input under Devices")) |
| + XCTAssertFalse( |
| + AudioWords.micSilentDetail(source: "MacBook Pro Microphone", channels: false).contains("input")) |
| + } |
| + |
| + @MainActor func testTheTileSaysItsSoundInWords() { |
| + let peer = PeerAudio() |
| + peer.show(state: .noSound, speaking: false, notices: [AudioWords.notHearingYou("Bea")]) |
| + let info = TileAudioInfo.peer(peer, name: "Bea") |
| + XCTAssertEqual(info.status, .noSound("Bea")) |
| + XCTAssertEqual( |
| + tileAccessibilityLabel("Bea", muted: false, audio: info), |
| + "Bea, No sound from Bea, Bea isn’t getting any sound from you.") |
| + XCTAssertEqual(info.detail, AudioWords.noSoundDetail) |
| + |
| + peer.show(state: .muted, speaking: false, notices: []) |
| + let muted = TileAudioInfo.peer(peer, name: "Bea") |
| + XCTAssertEqual(muted.status, .none, "muted: the badge, and no meter") |
| + XCTAssertEqual(tileAccessibilityLabel("Bea", muted: true, audio: muted), "Bea, muted") |
| + |
| + let you = MicAudio() |
| + you.show(step: 0, muted: false, speaking: false, silent: true, source: "Scarlett", channels: true) |
| + XCTAssertEqual(TileAudioInfo.you(you).notices, [AudioWords.micSilent]) |
| + } |
| + |
| + @MainActor func testLevelsReportWhoHasGoneQuietAndForgetWhoLeft() { |
| + let levels = AudioLevels() |
| + for t in stride(from: 0.0, through: 5.2, by: 0.2) { |
| + levels.updatePeer("bea", name: "Bea", sample: sample(unmuted), notices: [], now: t) |
| + levels.updatePeer("cy", name: "Cy", sample: sample(unmuted, db: -20), notices: [], now: t) |
| + levels.settle(present: ["bea", "cy"]) |
| + } |
| + XCTAssertEqual(levels.noSoundFrom, ["bea"]) |
| + XCTAssertEqual(levels.peer("bea").state, .noSound) |
| + XCTAssertTrue(levels.anyoneSpeaking.value) |
| + levels.settle(present: ["cy"]) |
| + XCTAssertEqual(levels.noSoundFrom, [], "someone who left is no longer reported") |
| + XCTAssertEqual(levels.peer("bea").state, .meter(0)) |
| + } |
| +} |
diff --git a/native/apple/Tests/CallsKitTests/InputChannelTests.swift b/native/apple/Tests/CallsKitTests/InputChannelTests.swift| new file mode 100644 |
| index 0000000..cfafd6a |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/InputChannelTests.swift |
| @@ -0,0 +1,125 @@ |
| +#if os(macOS) |
| + import CoreAudio |
| + import XCTest |
| + |
| + @testable import CallsKit |
| + |
| + /// The Mac's input channel choice (spec §15.20), apart from any device: |
| + /// the remembered choice, the channel map it becomes, the channel count |
| + /// read from a stream configuration, and the routes' picker state. |
| + @MainActor final class InputChannelTests: XCTestCase { |
| + private var store: UserDefaults! |
| + private var storeName = "" |
| + |
| + override func setUp() { |
| + super.setUp() |
| + storeName = "calls-channel-tests-\(UUID().uuidString)" |
| + store = UserDefaults(suiteName: storeName) |
| + } |
| + override func tearDown() { |
| + store.removePersistentDomain(forName: storeName) |
| + super.tearDown() |
| + } |
| + |
| + func testInputOneIsWhereEveryDeviceStarts() { |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett", channels: 18, store), 0) |
| + } |
| + |
| + func testTheChoiceIsRememberedPerDevice() { |
| + InputChannelPreference.save(1, uid: "scarlett", store) |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett", channels: 18, store), 1) |
| + XCTAssertEqual(InputChannelPreference.load(uid: "motu", channels: 8, store), 0) |
| + InputChannelPreference.save(0, uid: "scarlett", store) |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett", channels: 18, store), 0) |
| + } |
| + |
| + func testAChoiceTheDeviceNoLongerHasFallsBackToInputOne() { |
| + InputChannelPreference.save(9, uid: "scarlett", store) |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett", channels: 18, store), 9) |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett", channels: 2, store), 0) |
| + } |
| + |
| + func testTheMapCarriesTheChosenInputOnEveryClientChannel() { |
| + // A mono client (the engine's mono connection): one entry. |
| + XCTAssertEqual(InputChannelMap.map(channel: 1, clientChannels: 1, mappedBefore: false), [1]) |
| + // The hardware's width: every channel is input 2, so any mixdown or |
| + // "first channel" of them is input 2. |
| + XCTAssertEqual( |
| + InputChannelMap.map(channel: 1, clientChannels: 4, mappedBefore: false), [1, 1, 1, 1]) |
| + // Input 1 on a unit never mapped: left alone, today's behaviour. |
| + XCTAssertNil(InputChannelMap.map(channel: 0, clientChannels: 18, mappedBefore: false)) |
| + // Back to input 1 after a choice: straight through. |
| + XCTAssertEqual( |
| + InputChannelMap.map(channel: 0, clientChannels: 3, mappedBefore: true), [0, 1, 2]) |
| + // A unit that has not said its width still gets a map. |
| + XCTAssertEqual(InputChannelMap.map(channel: 2, clientChannels: 0, mappedBefore: false), [2]) |
| + } |
| + |
| + func testChannelsAreCountedAcrossAStreamConfiguration() { |
| + let list = AudioBufferList.allocate(maximumBuffers: 3) |
| + defer { free(list.unsafeMutablePointer) } |
| + list[0] = AudioBuffer(mNumberChannels: 8, mDataByteSize: 0, mData: nil) |
| + list[1] = AudioBuffer(mNumberChannels: 8, mDataByteSize: 0, mData: nil) |
| + list[2] = AudioBuffer(mNumberChannels: 2, mDataByteSize: 0, mData: nil) |
| + XCTAssertEqual(InputChannelMap.count(list), 18) |
| + } |
| + |
| + func testChannelsAreLabelledByNumberAndName() { |
| + let device = InputDeviceChannels( |
| + uid: "scarlett", name: "Scarlett 18i8 USB", count: 3, names: ["Analogue 1", "", "3"]) |
| + XCTAssertEqual(device.label(0), "Input 1 (Analogue 1)") |
| + XCTAssertEqual(device.label(1), "Input 2") |
| + XCTAssertEqual(device.label(2), "Input 3", "a name that is only the number adds nothing") |
| + XCTAssertEqual(device.describe(channel: 0), "Scarlett 18i8 USB, input 1") |
| + XCTAssertEqual( |
| + InputDeviceChannels(uid: "mbp", name: "MacBook Pro Microphone", count: 1).describe(channel: 0), |
| + "MacBook Pro Microphone") |
| + } |
| + |
| + private func routes(channels: InputDeviceChannels?, rebuilt: @escaping () -> Void) -> AudioRoutes { |
| + let routes = AudioRoutes(defaults: store) |
| + routes.attach( |
| + AudioModuleControl( |
| + devices: { input in |
| + [ModuleDevice(id: "default", name: "Default", isDefault: true)] |
| + + (input ? [ModuleDevice(id: "scarlett", name: "Scarlett 18i8 USB", isDefault: false)] : []) |
| + }, |
| + select: { _, _ in true }, |
| + channels: { id in id == "scarlett" ? channels : nil }, |
| + rebuildInput: rebuilt)) |
| + return routes |
| + } |
| + |
| + func testThePickerAppearsForAMultiChannelMicrophoneOnly() throws { |
| + var rebuilds = 0 |
| + let scarlett = InputDeviceChannels(uid: "scarlett-uid", name: "Scarlett 18i8 USB", count: 4) |
| + let routes = routes(channels: scarlett) { rebuilds += 1 } |
| + XCTAssertEqual(routes.inputChannels, [], "on the default, which is not the interface here") |
| + try routes.use(input: "scarlett") |
| + XCTAssertEqual(routes.inputChannels, ["Input 1", "Input 2", "Input 3", "Input 4"]) |
| + XCTAssertEqual(routes.inputChannel, 0) |
| + XCTAssertEqual(routes.inputSource, "Scarlett 18i8 USB, input 1") |
| + |
| + routes.use(inputChannel: 1) |
| + XCTAssertEqual(routes.inputChannel, 1) |
| + XCTAssertEqual(routes.inputSource, "Scarlett 18i8 USB, input 2") |
| + XCTAssertEqual(rebuilds, 1, "the module's input is rebuilt so the map applies now") |
| + XCTAssertEqual(InputChannelPreference.load(uid: "scarlett-uid", channels: 4, store), 1) |
| + |
| + routes.use(inputChannel: 1) |
| + routes.use(inputChannel: 7) |
| + XCTAssertEqual(rebuilds, 1, "the same choice, or one the device lacks, does nothing") |
| + |
| + // Remembered: a new launch on the same defaults comes back on input 2. |
| + let again = self.routes(channels: scarlett) {} |
| + XCTAssertEqual(again.inputChannel, 1) |
| + } |
| + |
| + func testAnOrdinaryMicrophoneHasNoPicker() throws { |
| + let routes = routes(channels: InputDeviceChannels(uid: "usb", name: "USB Mic", count: 1)) {} |
| + try routes.use(input: "scarlett") |
| + XCTAssertEqual(routes.inputChannels, []) |
| + XCTAssertEqual(routes.inputSource, "USB Mic") |
| + } |
| + } |
| +#endif |
diff --git a/native/apple/Tests/CallsKitTests/PeerMediaHearingTests.swift b/native/apple/Tests/CallsKitTests/PeerMediaHearingTests.swift| new file mode 100644 |
| index 0000000..b33cbd2 |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/PeerMediaHearingTests.swift |
| @@ -0,0 +1,43 @@ |
| +import XCTest |
| + |
| +@testable import CallsKit |
| + |
| +/// Spec §15.20's two optional members on the roster, as the native core |
| +/// forwards them: absent is none, and a report without them decodes |
| +/// exactly as §15.19's did. |
| +final class PeerMediaHearingTests: XCTestCase { |
| + private func media(_ json: String) throws -> PeerMedia? { |
| + let state = try JSONDecoder().decode( |
| + RoomState.self, |
| + from: Data( |
| + #"{"phase":"admitted","peer":"me","boot":"b","waiting":[],"peers":[{"peer":"tony","name":"Tony","kind":"member","gen":1,"media":\#(json)}]}"# |
| + .utf8)) |
| + return Call.peerMedia("tony", in: state) |
| + } |
| + |
| + func testTheTwoKeyReportStillDecodes() throws { |
| + let m = try XCTUnwrap(media(#"{"muted":false,"cameraOff":true}"#)) |
| + XCTAssertEqual(m, PeerMedia(muted: false, cameraOff: true)) |
| + XCTAssertNil(m.audioBlocked) |
| + XCTAssertNil(m.noSoundFrom) |
| + } |
| + |
| + func testTheReceivingHalfDecodes() throws { |
| + let m = try XCTUnwrap( |
| + media(#"{"muted":false,"cameraOff":false,"audioBlocked":true,"noSoundFrom":["me","bea"]}"#)) |
| + XCTAssertEqual(m.audioBlocked, true) |
| + XCTAssertEqual(m.noSoundFrom, ["me", "bea"]) |
| + XCTAssertEqual( |
| + m, PeerMedia(muted: false, cameraOff: false, audioBlocked: true, noSoundFrom: ["me", "bea"])) |
| + } |
| + |
| + func testEitherMemberAlone() throws { |
| + XCTAssertEqual(try media(#"{"muted":true,"cameraOff":false,"audioBlocked":true}"#)?.noSoundFrom, nil) |
| + XCTAssertEqual(try media(#"{"muted":true,"cameraOff":false,"noSoundFrom":["me"]}"#)?.audioBlocked, nil) |
| + } |
| + |
| + func testAMalformedMemberIsRefusedNotGuessed() { |
| + XCTAssertThrowsError(try media(#"{"muted":false,"cameraOff":false,"noSoundFrom":"me"}"#)) |
| + XCTAssertThrowsError(try media(#"{"muted":false,"cameraOff":false,"audioBlocked":"yes"}"#)) |
| + } |
| +} |
diff --git a/native/ios/Sources/CallsPhone.swift b/native/ios/Sources/CallsPhone.swift| index 0a8521c..c7214a5 100644 |
| --- a/native/ios/Sources/CallsPhone.swift |
| +++ b/native/ios/Sources/CallsPhone.swift |
| @@ -256,13 +256,14 @@ struct PhoneCall: View { |
| } |
| PhoneVideoTile( |
| track: call.localTrack, label: "You", mirrored: true, cameraOff: call.cameraOff, |
| - placeholder: call.selfPlaceholder |
| + placeholder: call.selfPlaceholder, audio: call.selfAudio |
| ) |
| .aspectRatio(4 / 3, contentMode: .fit) |
| ForEach(peers) { peer in |
| PhoneVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| - muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer) |
| + muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .aspectRatio(4 / 3, contentMode: .fit) |
| } |
| @@ -279,7 +280,8 @@ struct PhoneCall: View { |
| case .camera(let peer): |
| PhoneVideoTile( |
| track: call.remoteTracks[peer], label: call.name(of: peer), |
| - muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer)) |
| + muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer), |
| + audio: call.tileAudio(peer)) |
| case nil: |
| ContentUnavailableView( |
| "Waiting for others", systemImage: "person.2", |
| @@ -290,7 +292,7 @@ struct PhoneCall: View { |
| } |
| PhoneVideoTile( |
| track: call.localTrack, label: "You", mirrored: true, cameraOff: call.cameraOff, |
| - placeholder: call.selfPlaceholder |
| + placeholder: call.selfPlaceholder, audio: call.selfAudio |
| ) |
| .frame( |
| width: min(160, geometry.size.width * 0.32), |
| @@ -393,7 +395,14 @@ struct PhoneVideoTile: View { |
| /// Behind a picture that is not there: a shade lighter than the stage |
| /// for a tile among others, so it reads as a tile. |
| var backdrop = Color.black |
| + /// The sound on this tile (spec §15.20): your microphone's meter, or a |
| + /// remote person's with "No sound from" and their notices. Nil for a |
| + /// screen. |
| + var audio: TileAudio? = nil |
| var body: some View { |
| + TileAudioReader(audio) { info in tile(info) } |
| + } |
| + private func tile(_ info: TileAudioInfo?) -> some View { |
| ZStack { |
| backdrop |
| if let track, !cameraOff, placeholder == nil { |
| @@ -412,18 +421,27 @@ struct PhoneVideoTile: View { |
| } |
| } |
| .overlay(alignment: .bottomLeading) { |
| - if compact { |
| - TileNameLabel(label: label, muted: muted).padding(6).padding(labelInsets) |
| - } else { |
| - Text(label).font(.callout.bold()).foregroundStyle(.white).lineLimit(1) |
| - .padding(.horizontal, 10).padding(.vertical, 6) |
| - .background(.black.opacity(0.65), in: Capsule()).padding(8) |
| + VStack(alignment: .leading, spacing: 4) { |
| + TileAudioNotices(info) |
| + HStack(spacing: 5) { |
| + if compact { |
| + TileNameLabel(label: label, muted: muted) |
| + } else { |
| + Text(label).font(.callout.bold()).foregroundStyle(.white).lineLimit(1) |
| + .padding(.horizontal, 10).padding(.vertical, 6) |
| + .background(.black.opacity(0.65), in: Capsule()) |
| + } |
| + TileAudioStatus(info) |
| + } |
| } |
| + .padding(compact ? 6 : 8).padding(compact ? labelInsets : EdgeInsets()) |
| } |
| .overlay(alignment: .topLeading) { if muted, !compact { MutedBadge().padding(8) } } |
| + .speakingRing(info?.speaking == true, cornerRadius: cornerRadius) |
| .clipShape(RoundedRectangle(cornerRadius: cornerRadius)) |
| .accessibilityElement(children: .ignore) |
| - .accessibilityLabel(tileAccessibilityLabel(label, muted: muted)) |
| + .accessibilityLabel(tileAccessibilityLabel(label, muted: muted, audio: info)) |
| + .accessibilityHint(info?.detail ?? "") |
| .accessibilityValue( |
| cameraOff |
| ? "Camera off" |
diff --git a/native/ios/Sources/PhoneFullScreenCall.swift b/native/ios/Sources/PhoneFullScreenCall.swift| index d904ca1..978d478 100644 |
| --- a/native/ios/Sources/PhoneFullScreenCall.swift |
| +++ b/native/ios/Sources/PhoneFullScreenCall.swift |
| @@ -105,7 +105,7 @@ extension PhoneCall { |
| PhoneVideoTile( |
| track: call.remoteTracks[peer], label: call.name(of: peer), |
| muted: call.isMuted(peer), cameraOff: call.isCameraOff(peer), |
| - cornerRadius: 0, compact: true, labelInsets: lift) |
| + cornerRadius: 0, compact: true, labelInsets: lift, audio: call.tileAudio(peer)) |
| case nil: |
| ContentUnavailableView( |
| "Waiting for others", systemImage: "person.2", |
| @@ -145,14 +145,15 @@ extension PhoneCall { |
| PhoneVideoTile( |
| track: call.localTrack, label: "You", muted: call.muted, mirrored: true, |
| cameraOff: call.cameraOff, placeholder: call.selfPlaceholder, cornerRadius: 8, |
| - compact: true, backdrop: Self.tileBackdrop |
| + compact: true, backdrop: Self.tileBackdrop, audio: call.selfAudio |
| ) |
| .frame(width: tile.width, height: tile.height) |
| ForEach(peers) { peer in |
| PhoneVideoTile( |
| track: call.remoteTracks[peer.peer], label: peer.name, |
| muted: call.isMuted(peer.peer), cameraOff: call.isCameraOff(peer.peer), |
| - cornerRadius: 8, compact: true, backdrop: Self.tileBackdrop |
| + cornerRadius: 8, compact: true, backdrop: Self.tileBackdrop, |
| + audio: call.tileAudio(peer.peer) |
| ) |
| .frame(width: tile.width, height: tile.height) |
| } |
| @@ -179,7 +180,7 @@ extension PhoneCall { |
| return PhoneVideoTile( |
| track: call.localTrack, label: "You", muted: call.muted, mirrored: true, |
| cameraOff: call.cameraOff, placeholder: call.selfPlaceholder, cornerRadius: 14, |
| - compact: true, backdrop: Self.tileBackdrop |
| + compact: true, backdrop: Self.tileBackdrop, audio: call.selfAudio |
| ) |
| .frame(width: pip.width, height: pip.height) |
| .overlay(RoundedRectangle(cornerRadius: 14).strokeBorder(.white.opacity(0.2))) |
diff --git a/test/browser/native.mjs b/test/browser/native.mjs| index e64991e..abd8d74 100644 |
| --- a/test/browser/native.mjs |
| +++ b/test/browser/native.mjs |
| @@ -111,6 +111,39 @@ try { |
| }); |
| await wait(async () => await browserAudio() > 0, 'browser receives native audio'); |
| console.log('PASS browser decoded native video and received native audio'); |
| + // Spec §15.20: bytes are not sound. On 2026-09-25 a call carried audio |
| + // packets both ways while one side sent silence, and every counter said |
| + // all was well. So the browser has to measure *energy* in what the app |
| + // sends (inbound-rtp totalAudioEnergy and audioLevel), and the app's own |
| + // receive meter has to see the browser's fake microphone. |
| + { |
| + const browserHearing = () => page.evaluate(async () => { |
| + let energy = 0, level = 0; |
| + for (const pc of window.__nativeInteropPCs) { |
| + if (pc.connectionState === 'closed') continue; |
| + for (const report of (await pc.getStats()).values()) { |
| + if (report.type === 'inbound-rtp' && report.kind === 'audio') { |
| + energy += report.totalAudioEnergy || 0; |
| + level = Math.max(level, report.audioLevel || 0); |
| + } |
| + } |
| + } |
| + return { energy, level }; |
| + }); |
| + const before = await browserHearing(); |
| + let loudest = 0; |
| + await wait(async () => { |
| + const now = await browserHearing(); |
| + loudest = Math.max(loudest, now.level); |
| + return now.energy > before.energy + 0.001 && loudest > 0.01; |
| + }, 'the browser hears non-silent audio from the app'); |
| + console.log(`PASS the browser measures sound, not only bytes, from the app (audioLevel up to ${loudest.toFixed(3)})`); |
| + await wait(() => sample?.stats.remoteAudioPeakPermille > 10, "the app's receive meter hears the browser"); |
| + console.log(`PASS the app's receive meter hears the browser (peak ${sample.stats.remoteAudioPeakPermille}\u2030)`); |
| + await wait(() => sample?.stats.micPeakPermille > 10, "the app's own meter sees what it sends"); |
| + assert.equal(sample.stats.micSilent, 0, "a live synthetic microphone is not reported as picking nothing up"); |
| + console.log(`PASS the app's microphone meter reads what it sends, with no silence warning (noSoundFrom ${sample.stats.noSoundFrom})`); |
| + } |
| // Choose the system default on the audio device module during the live |
| // call, then refuse an unavailable microphone. Both must leave media and |
| // the peer connection intact. (The harness's module is in manual |