amadan
Reduce background noise with RNNoise in WebRTC's capture hook
WebRTC's classic noise suppressor is switched off and RNNoise runs in its capture post-processing hook instead, after the echo canceller: 480-sample frames at 48 kHz, in WebRTC's own ±32768 scale. A "Reduce background noise" toggle in the device panel, on by default and remembered, turns it off. ProcessingPlan holds the rules and is tested across every combination: ours never runs beside WebRTC's suppressor or Apple's Voice Isolation; WebRTC's gain control, which runs before the hook and would pump the noise, is off while ours runs, and SpeechGain lifts the voice after the noise is gone (upward only, at most 12 dB, speech frames only, about 1 dB/s, peak-limited). At any rate but 48 kHz frames pass through and WebRTC's own suppressor and gain return. The iPhone keeps Apple's coupled echo canceller and suppressor, as it always has, and drops only Apple's gain. The delegate is set after the factory exists: handed to the module's initialiser it is never called. It is made once and survives WebRTC's initialise/release cycles. RNNoise's weights are a generated 78 MB C file, so build-rnnoise.sh builds a pinned, checksummed static xcframework into native/build, as CallsBridge is built. Its BSD notice ships in both apps. The audio readout names the suppressor, its share of a core, and what WebRTC reports running for echo cancellation, suppression and gain. The native/browser gate now asserts the suppressor processed the synthetic microphone's frames and that WebRTC reports its own suppressor and gain off and echo cancellation on. It also builds CallsInterop, which it previously ran stale from whatever build came before. A fixture test mixes seeded hiss and key clicks into synthesised speech: RNNoise takes the noise-only lead-in from -41 to -116 dBFS, holds speech at -24.5 -> -24.6 dBFS and keeps 96% of voiced frames, at 0.70 ms per 10 ms frame on an M1 Max.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
17 files changed,
+1034
−9
Makefile+3 −0native/LICENSE.rnnoise+32 −0native/README.md+54 −1native/apple/Package.swift+7 −2native/apple/Sources/CallsKit/Call.swift+23 −0native/apple/Sources/CallsKit/CallAudio.swift+194 −4native/apple/Sources/CallsKit/DevicePanel.swift+19 −0native/apple/Sources/CallsKit/Media.swift+1 −1native/apple/Sources/CallsKit/NoiseSuppression.swift+238 −0native/apple/Tests/CallsKitTests/Fixtures/speech-48k.wavbinarynative/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift+362 −0native/ios/project.yml+2 −0native/scripts/build-iphone.sh+1 −0native/scripts/build-mac.sh+1 −1native/scripts/build-rnnoise.sh+87 −0native/scripts/bundle-mac.py+1 −0test/browser/native.mjs+9 −0
diff --git a/Makefile b/Makefile| index a15a497..04938d4 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -166,6 +166,9 @@ native-test: |
| CGO_ENABLED=1 go test -race ./callscore |
| native-browser: native-mac |
| + # The gate runs CallsInterop, which native-mac does not build; without |
| + # this it runs whatever binary a previous build left behind. |
| + swift build --package-path native/apple --product CallsInterop |
| CGO_ENABLED=0 go test -tags browser -count=1 -timeout 3m -run 'TestNative(MacCallsBrowser|MacChat|BrowserAuthentication)' -v ./internal/callstest/ |
| .PHONY: iphone iphone-device iphone-browser |
diff --git a/native/LICENSE.rnnoise b/native/LICENSE.rnnoise| new file mode 100644 |
| index 0000000..59d7d09 |
| --- /dev/null |
| +++ b/native/LICENSE.rnnoise |
| @@ -0,0 +1,32 @@ |
| +Copyright (c) 2007-2017, 2024 Jean-Marc Valin |
| +Copyright (c) 2023 Amazon |
| +Copyright (c) 2017, Mozilla |
| +Copyright (c) 2005-2017, Xiph.Org Foundation |
| +Copyright (c) 2003-2004, Mark Borgerding |
| + |
| +Redistribution and use in source and binary forms, with or without |
| +modification, are permitted provided that the following conditions |
| +are met: |
| + |
| +- Redistributions of source code must retain the above copyright |
| +notice, this list of conditions and the following disclaimer. |
| + |
| +- Redistributions in binary form must reproduce the above copyright |
| +notice, this list of conditions and the following disclaimer in the |
| +documentation and/or other materials provided with the distribution. |
| + |
| +- Neither the name of the Xiph.Org Foundation nor the names of its |
| +contributors may be used to endorse or promote products derived from |
| +this software without specific prior written permission. |
| + |
| +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS |
| +``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT |
| +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR |
| +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE FOUNDATION |
| +OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, |
| +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT |
| +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, |
| +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY |
| +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT |
| +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE |
| +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. |
diff --git a/native/README.md b/native/README.md| index 8991207..fb274a3 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -59,6 +59,45 @@ permission to), so `CallsInterop` gives the engine module a synthetic voice |
| through manual rendering mode, as it gives the capturer synthetic frames; no |
| device is opened on either side of it. |
| +**Reduce background noise** (in the device panel, on by default, remembered) |
| +replaces WebRTC's classic noise suppressor with a neural one, for the noises |
| +people actually hear: typing, wind, a room's echo. WebRTC's capture |
| +processing runs high-pass filter, echo canceller, noise suppressor and gain |
| +control, then hands each 10 ms frame to a capture post-processing hook |
| +(`RTCDefaultAudioProcessingModule`); the suppressor runs there, after the |
| +echo canceller has removed the far side. `ProcessingPlan` holds the rules: |
| + |
| +- WebRTC's noise suppressor is off while ours runs, and ours never runs |
| + beside Apple's Voice Isolation (it stands aside when that is the active |
| + mic mode). |
| +- WebRTC's gain control runs *before* the hook, so it would lift the room |
| + with a quiet voice and hand the suppressor more noise to remove. It is off |
| + while ours runs; `SpeechGain` lifts the voice after the noise is gone |
| + instead — only upward, at most 12 dB, learning only from frames the model |
| + calls speech, about 1 dB a second, with a peak limit. |
| +- The models run at 48 kHz only. At any other capture rate (a Bluetooth |
| + headset's 16 kHz) frames pass through untouched and WebRTC's own |
| + suppressor and gain come back on the Mac. |
| +- Off means no suppression at all; WebRTC's gain control comes back. |
| +- On iPhone Apple's echo canceller and its noise suppressor are one switch |
| + in the voice-processing unit, and the phone has always run both; that |
| + pairing stays. Apple's gain control is separate and comes off while ours |
| + runs. |
| + |
| +The audio readout says which suppressor is running and at what share of a |
| +core, and what WebRTC reports as running for echo cancellation, noise |
| +suppression and gain — what it reports, not what was asked. WebRTC skips the |
| +hook while the microphone is muted, and the readout says so. |
| + |
| +The suppressor is RNNoise (xiph, BSD-3; notice in `LICENSE.rnnoise`, bundled |
| +in both apps): 480-sample frames at 48 kHz in WebRTC's own ±32768 scale. |
| +Its model weights arrive as a generated C file of 78 MB, so it is not a |
| +source target: `scripts/build-rnnoise.sh` fetches the pinned commit and the |
| +checksummed model and builds `native/build/RNNoise.xcframework` (static, Mac, |
| +iPhone and simulator), which is not committed — the CallsBridge precedent. |
| +`build-mac.sh` and `build-iphone.sh` run it; run it yourself before |
| +`swift test`. It is cached against the pins. |
| + |
| Choosing a Bluetooth microphone on iPhone puts the call on the hands-free |
| profile, which is narrowband in both directions. That used to happen |
| silently to every call the moment any headset connected, because |
| @@ -156,13 +195,27 @@ make native-browser |
| CGO_ENABLED=0 go test -tags browser -run TestNativeBrowserAuthentication ./internal/callstest |
| ``` |
| +`swift test` needs `native/scripts/build-bridge.sh` and |
| +`native/scripts/build-rnnoise.sh` to have run. Its suppressor tests feed |
| +`Tests/CallsKitTests/Fixtures/speech-48k.wav` (macOS's own speech |
| +synthesiser, normalised to -23 dBFS, a second of silence ahead) mixed with |
| +seeded hiss and key clicks through each real suppressor, and require the |
| +noise-only lead-in to fall by at least 10 dB, the speech level to hold within |
| +6 dB, and 80% of voiced frames to survive. The frame adapter, the gain stage |
| +and the processing plan have unit tests of their own. |
| + |
| The native/browser gate requires the same Playwright/Chromium setup as |
| `make browser`, and working UDP. It creates two real magic-link sessions and a |
| meeting on an isolated server, starts Chromium and the separate native test |
| executable, then checks guest admission, actual incoming/outgoing media, clean departure and continuity through |
| ten-second SSE renewals, bidirectional chat and the native sender's accepted-message |
| history. Test speaker playback is disabled in both clients; audio checks use RTP |
| -counters, not audible output. It never needs production credentials or hardware |
| +counters, not audible output. The native side's microphone is a synthetic |
| +voice through the audio device module's manual rendering mode, so its capture |
| +passes through WebRTC's real processing: the gate asserts the noise suppressor |
| +processed frames, and that WebRTC reports its own suppressor and gain off and |
| +its echo canceller on. `make native-browser` builds `CallsInterop` itself; |
| +it used to run whichever binary an earlier build had left. It never needs production credentials or hardware |
| camera access. The Go Home test separately exercises a session minted through |
| Home assertions and explicit identity linking, including resume and meeting end. |
diff --git a/native/apple/Package.swift b/native/apple/Package.swift| index 20748e5..d4204b5 100644 |
| --- a/native/apple/Package.swift |
| +++ b/native/apple/Package.swift |
| @@ -19,10 +19,15 @@ let package = Package( |
| ], |
| targets: [ |
| .binaryTarget(name: "CallsBridge", path: "../build/CallsBridge.xcframework"), |
| + // Built by scripts/build-rnnoise.sh, as CallsBridge is by build-bridge.sh. |
| + .binaryTarget(name: "RNNoise", path: "../build/RNNoise.xcframework"), |
| .target( |
| - name: "CallsKit", dependencies: ["CallsBridge", .product(name: "LiveKitWebRTC", package: "webrtc-xcframework")]), |
| + name: "CallsKit", |
| + dependencies: [ |
| + "CallsBridge", "RNNoise", .product(name: "LiveKitWebRTC", package: "webrtc-xcframework"), |
| + ]), |
| .executableTarget(name: "CallsMac", dependencies: [.product(name: "RastrilloMacUpdates", package: "native", condition: .when(platforms: [.macOS])), "CallsKit"], resources: [.copy("Resources/AppIcon.png"), .copy("Resources/MacIcon.png")]), |
| .executableTarget(name: "CallsInterop", dependencies: ["CallsKit"]), |
| - .testTarget(name: "CallsKitTests", dependencies: ["CallsKit"]), |
| + .testTarget(name: "CallsKitTests", dependencies: ["CallsKit"], resources: [.copy("Fixtures")]), |
| ] |
| ) |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index e3605e4..d49066f 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -52,6 +52,18 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| @Published public private(set) var currentCamera: String? |
| /// The routes this platform can report, and on iPhone choose between. |
| public let audio = AudioRoutes() |
| + /// "Reduce background noise": the neural suppressor in place of WebRTC's |
| + /// own. On by default, and remembered. |
| + @Published public var reduceNoise: Bool = UserDefaults.standard.object(forKey: Call.reduceNoiseKey) as? Bool ?? true { |
| + didSet { |
| + UserDefaults.standard.set(reduceNoise, forKey: Call.reduceNoiseKey) |
| + media.audio.reduceNoise = reduceNoise |
| + audioProcessing = media.audio.report() |
| + } |
| + } |
| + static let reduceNoiseKey = "reduceBackgroundNoise" |
| + /// Which processing is running on the microphone, for the readout. |
| + @Published public private(set) var audioProcessing: ProcessingReport? |
| private var cameraWork: Task<Void, Never>? |
| private var cameraSwitches = 0 |
| private let knocks = KnockWatch() |
| @@ -105,6 +117,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| localTrack = media.localVideo |
| currentCamera = media.currentCamera |
| audio.refresh() |
| + media.audio.reduceNoise = reduceNoise |
| + audioProcessing = media.audio.report() |
| KnockNotice.requestPermission() |
| // Asking Apple at join, once there is a session to register the |
| // answer against. Apple replies on its own schedule, at the app |
| @@ -128,6 +142,7 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| loadingInvitation = false |
| deviceNote = "" |
| audioHealth = nil |
| + audioProcessing = nil |
| currentCamera = nil |
| knocks.reset() |
| cameraWork?.cancel() |
| @@ -211,6 +226,8 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| try? await Task.sleep(nanoseconds: 5_000_000_000) |
| guard !Task.isCancelled, bridge != nil else { return } |
| audioHealth = AudioHealth(await media.stats()) |
| + media.audio.refresh() |
| + audioProcessing = media.audio.report() |
| } |
| } |
| public func retryJoin() { send(type: "join", to: "", gen: 0, payload: "") } |
| @@ -339,6 +356,12 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| values["remoteTracks"] = UInt64(remoteTracks.count) |
| values["invalidEvents"] = invalidEvents |
| values["rosterPeers"] = UInt64(state?.peers.count ?? 0) |
| + values["noiseFramesProcessed"] = media.audio.denoiser.framesProcessed |
| + // What WebRTC reports running, not what was asked of it. |
| + let running = media.audio.report() |
| + values["webrtcEchoCancellation"] = running.echoCancellation == "off" ? 0 : 1 |
| + values["webrtcNoiseSuppression"] = running.noiseSuppression == "off" ? 0 : 1 |
| + values["webrtcGainControl"] = running.gainControl.hasPrefix("Calls") || running.gainControl == "off" ? 0 : 1 |
| return values |
| } |
| } |
diff --git a/native/apple/Sources/CallsKit/CallAudio.swift b/native/apple/Sources/CallsKit/CallAudio.swift| index 2555d06..dd3cfe3 100644 |
| --- a/native/apple/Sources/CallsKit/CallAudio.swift |
| +++ b/native/apple/Sources/CallsKit/CallAudio.swift |
| @@ -10,13 +10,104 @@ public protocol AudioInputSource: AnyObject, Sendable { |
| func render(_ samples: UnsafeMutablePointer<Float>, count: Int, sampleRate: Double) |
| } |
| +/// Which of WebRTC's processing stages run on the microphone, decided in |
| +/// one place so the rule can be read and tested apart from WebRTC. |
| +/// |
| +/// The rule the plan holds: our suppressor never runs beside another one. |
| +/// Not beside WebRTC's own noise suppressor, and not beside Apple's Voice |
| +/// Isolation. And WebRTC's gain control, which runs *before* our |
| +/// suppressor, is off whenever ours is on; SpeechGain lifts the voice |
| +/// after the noise has gone instead. |
| +public struct ProcessingPlan: Equatable, Sendable { |
| + public enum Where: Equatable, Sendable { case off, software, platform } |
| + /// Our neural suppressor, in the capture post-processing hook. |
| + public var suppressor: Bool |
| + public var echoCancellation: Where |
| + public var noiseSuppression: Where |
| + public var gainControl: Where |
| + /// WebRTC's high-pass filter, always: it is cheap and only takes out |
| + /// rumble below the voice. |
| + public var highPass = true |
| + |
| + /// - Parameters: |
| + /// - iPhone: the phone runs on Apple's voice-processing unit, whose |
| + /// echo canceller and noise suppressor are one switch; the Mac runs |
| + /// WebRTC's software processing. |
| + /// - reduceNoise: the person's toggle. |
| + /// - rate: the capture rate WebRTC reported, or nil before it has. |
| + /// - voiceIsolation: Apple's Voice Isolation is the active mic mode. |
| + public static func plan(iPhone: Bool, reduceNoise: Bool, rate: Int?, voiceIsolation: Bool) |
| + -> ProcessingPlan |
| + { |
| + let rateOK = rate.map { $0 == CaptureDenoiser.supportedRate } ?? true |
| + let ours = reduceNoise && rateOK && !voiceIsolation |
| + if iPhone { |
| + // Apple's echo canceller cannot be had without Apple's suppressor |
| + // beside it; that pairing is what the phone has always run. Its gain |
| + // control is separate, and is what comes off. |
| + return ProcessingPlan( |
| + suppressor: ours, echoCancellation: .platform, noiseSuppression: .platform, |
| + gainControl: ours ? .off : .platform) |
| + } |
| + let classicNS = reduceNoise && !rateOK |
| + return ProcessingPlan( |
| + suppressor: ours, echoCancellation: .software, |
| + noiseSuppression: classicNS ? .software : .off, |
| + gainControl: ours ? .off : .software) |
| + } |
| +} |
| + |
| +/// What the audio readout says about processing. |
| +public struct ProcessingReport: Equatable, Sendable { |
| + public var suppression: SuppressionState |
| + public var echoCancellation: String |
| + public var noiseSuppression: String |
| + public var gainControl: String |
| + /// Set when WebRTC refused the options asked of it. |
| + public var refused: String? |
| + |
| + /// One or two plain sentences. |
| + public var summary: String { |
| + var lines: [String] = [] |
| + switch suppression { |
| + case .off: lines.append("Background noise reduction is off.") |
| + case .running(let engine, let cpu): |
| + lines.append( |
| + "Reducing background noise with \(engine) (\(String(format: "%.1f", cpu * 100))% of one core).") |
| + case .idle(let engine): lines.append("Reducing background noise with \(engine) when you speak.") |
| + case .unsupportedRate(let rate): |
| + lines.append( |
| + "The microphone runs at \(rate / 1000) kHz, which the noise reducer cannot; WebRTC's own is standing in.") |
| + case .standingAside: |
| + lines.append("Voice Isolation is on, so Calls leaves noise reduction to it.") |
| + } |
| + lines.append( |
| + "Echo cancellation: \(echoCancellation). Noise suppression: \(noiseSuppression). Gain: \(gainControl).") |
| + if let refused { lines.append("WebRTC refused the requested processing: \(refused)") } |
| + return lines.joined(separator: " ") |
| + } |
| +} |
| + |
| /// How a call's audio is built: which audio device module WebRTC runs on, |
| -/// and — on Mac — what the app is told about the devices it can use. |
| +/// what processing runs on the microphone, and — on Mac — what the app is |
| +/// told about the devices it can use. |
| /// |
| /// Kept out of Media.swift so that file only changes where the factory is |
| /// made. |
| @MainActor final class CallAudio { |
| let factory: RTCPeerConnectionFactory |
| + /// Nil for the iPhone tests' silent device, which takes the constructor |
| + /// that has no processing module. |
| + let processing: RTCDefaultAudioProcessingModule? |
| + let denoiser: CaptureDenoiser |
| + private weak var track: RTCAudioTrack? |
| + private var rate: Int? |
| + private var voiceIsolation = false |
| + private var refused: String? |
| + private(set) var plan: ProcessingPlan |
| + var reduceNoise: Bool { |
| + didSet { if reduceNoise != oldValue { apply() } } |
| + } |
| #if os(macOS) |
| private let devices: EngineWatch |
| @@ -27,26 +118,40 @@ public protocol AudioInputSource: AnyObject, Sendable { |
| /// |
| /// `input`, on Mac, replaces the microphone and the speaker both: the |
| /// engine runs in manual rendering mode, touching no device at all. |
| - init(routes: AudioRoutes, audioDevice: RTCAudioDevice?, input: AudioInputSource? = nil) { |
| + init( |
| + routes: AudioRoutes, audioDevice: RTCAudioDevice?, input: AudioInputSource? = nil, |
| + reduceNoise: Bool = true |
| + ) { |
| let encoder = RTCDefaultVideoEncoderFactory() |
| let decoder = RTCDefaultVideoDecoderFactory() |
| + self.reduceNoise = reduceNoise |
| + denoiser = CaptureDenoiser(suppressor: CallAudio.makeSuppressor()) |
| #if os(iOS) |
| + plan = ProcessingPlan.plan(iPhone: true, reduceNoise: reduceNoise, rate: nil, voiceIsolation: false) |
| if let audioDevice { |
| + processing = nil |
| factory = RTCPeerConnectionFactory( |
| encoderFactory: encoder, decoderFactory: decoder, audioDevice: audioDevice) |
| return |
| } |
| + let processing = RTCDefaultAudioProcessingModule( |
| + config: nil, capturePostProcessingDelegate: nil, renderPreProcessingDelegate: nil) |
| + self.processing = processing |
| // The platform default on iPhone is the voice-processing I/O unit it |
| // has always been. |
| factory = RTCPeerConnectionFactory( |
| audioDeviceModuleType: .platformDefault, bypassVoiceProcessing: false, |
| - encoderFactory: encoder, decoderFactory: decoder, audioProcessingModule: nil) |
| + encoderFactory: encoder, decoderFactory: decoder, audioProcessingModule: processing) |
| #else |
| + plan = ProcessingPlan.plan(iPhone: false, reduceNoise: reduceNoise, rate: nil, voiceIsolation: false) |
| + let processing = RTCDefaultAudioProcessingModule( |
| + config: nil, capturePostProcessingDelegate: nil, renderPreProcessingDelegate: nil) |
| + self.processing = processing |
| // AVAudioEngine rather than the CoreAudio module: it is the one with a |
| // device list and a way to choose from it. |
| factory = RTCPeerConnectionFactory( |
| audioDeviceModuleType: .audioEngine, bypassVoiceProcessing: false, |
| - encoderFactory: encoder, decoderFactory: decoder, audioProcessingModule: nil) |
| + encoderFactory: encoder, decoderFactory: decoder, audioProcessingModule: processing) |
| let module = factory.audioDeviceModule |
| // Apple's voice processing stays off on Mac. Left allowed, the engine |
| // module resolves echo cancellation, noise suppression and gain to |
| @@ -64,6 +169,91 @@ public protocol AudioInputSource: AnyObject, Sendable { |
| routes.attach(module) |
| } |
| #endif |
| + // Set after the factory exists: handed to the module's initialiser, the |
| + // delegate is never called (seen in the spike, M150). The property is |
| + // weak; `denoiser` is what keeps it alive. |
| + processing.capturePostProcessingDelegate = denoiser |
| + denoiser.enabled = plan.suppressor |
| + denoiser.rateChanged = { [weak self] rate in |
| + DispatchQueue.main.async { |
| + MainActor.assumeIsolated { |
| + guard let self, self.rate != rate else { return } |
| + self.rate = rate |
| + self.apply() |
| + } |
| + } |
| + } |
| + } |
| + |
| + /// The suppressor this platform runs. RNNoise on both, for now. |
| + static func makeSuppressor() -> NoiseSuppressor { RNNoiseSuppressor() } |
| + |
| + /// The microphone track, made here so its processing options are set |
| + /// before it is ever sent. |
| + func makeTrack() -> RTCAudioTrack { |
| + let track = factory.audioTrack(withTrackId: "microphone") |
| + self.track = track |
| + apply() |
| + return track |
| + } |
| + |
| + /// Re-reads Apple's mic mode and reapplies the plan if anything moved. |
| + /// Called on the audio-health tick. |
| + func refresh() { |
| + let isolating = AVCaptureDevice.activeMicrophoneMode == .voiceIsolation |
| + if isolating != voiceIsolation { |
| + voiceIsolation = isolating |
| + apply() |
| + } |
| + } |
| + |
| + private func apply() { |
| + #if os(iOS) |
| + let iPhone = true |
| + #else |
| + let iPhone = false |
| + #endif |
| + plan = ProcessingPlan.plan( |
| + iPhone: iPhone, reduceNoise: reduceNoise, rate: rate, voiceIsolation: voiceIsolation) |
| + denoiser.enabled = reduceNoise |
| + denoiser.standAside = voiceIsolation |
| + guard let track, processing != nil else { return } |
| + let result = track.setAudioProcessingOptions(CallAudio.options(plan)) |
| + refused = result.isSuccess ? nil : result.message |
| + } |
| + |
| + static func options(_ plan: ProcessingPlan) -> RTCAudioProcessingOptions { |
| + func component(_ place: ProcessingPlan.Where) -> RTCAudioProcessingComponentOptions { |
| + switch place { |
| + case .off: return RTCAudioProcessingComponentOptions(enabled: false) |
| + case .software: return RTCAudioProcessingComponentOptions(enabled: true, mode: .software) |
| + case .platform: return RTCAudioProcessingComponentOptions(enabled: true, mode: .platform) |
| + } |
| + } |
| + return RTCAudioProcessingOptions( |
| + echoCancellationOptions: component(plan.echoCancellation), |
| + noiseSuppressionOptions: component(plan.noiseSuppression), |
| + autoGainControlOptions: component(plan.gainControl), |
| + highPassFilterOptions: component(plan.highPass ? .software : .off)) |
| + } |
| + |
| + /// What is actually running, as WebRTC reports it — not what was asked. |
| + func report() -> ProcessingReport { |
| + let state = factory.audioProcessingState |
| + func says(_ component: RTCAudioProcessingComponentState) -> String { |
| + switch component.effective { |
| + case .disabled: return "off" |
| + case .software: return "WebRTC" |
| + case .platform: return "Apple" |
| + case .softwareAndPlatform: return "WebRTC and Apple" |
| + default: return "unknown" |
| + } |
| + } |
| + var gain = says(state.autoGainControl) |
| + if plan.suppressor, gain == "off" { gain = "Calls, after noise reduction" } |
| + return ProcessingReport( |
| + suppression: denoiser.state, echoCancellation: says(state.echoCancellation), |
| + noiseSuppression: says(state.noiseSuppression), gainControl: gain, refused: refused) |
| } |
| } |
diff --git a/native/apple/Sources/CallsKit/DevicePanel.swift b/native/apple/Sources/CallsKit/DevicePanel.swift| index 3f9f85b..5324ff7 100644 |
| --- a/native/apple/Sources/CallsKit/DevicePanel.swift |
| +++ b/native/apple/Sources/CallsKit/DevicePanel.swift |
| @@ -50,6 +50,8 @@ import SwiftUI |
| .accessibilityLabel("Device problem: \(call.deviceNote)") |
| } |
| + noiseToggle |
| + |
| Divider() |
| audioReadout |
| } |
| @@ -128,6 +130,17 @@ import SwiftUI |
| } |
| #endif |
| + /// Said as what it does for the far side, since that is who hears it. |
| + private var noiseToggle: some View { |
| + VStack(alignment: .leading, spacing: 4) { |
| + Toggle("Reduce background noise", isOn: $call.reduceNoise) |
| + .accessibilityIdentifier("reduceNoise") |
| + Text("Takes out typing, wind and room noise before others hear you.") |
| + .font(.footnote).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + } |
| + } |
| + |
| private var audioReadout: some View { |
| VStack(alignment: .leading, spacing: 4) { |
| Text("Audio health").font(.headline) |
| @@ -140,6 +153,12 @@ import SwiftUI |
| Text("Measuring…").font(.callout).foregroundStyle(.secondary) |
| .accessibilityIdentifier("audioHealth") |
| } |
| + if let processing = call.audioProcessing { |
| + Text(processing.summary) |
| + .font(.callout).foregroundStyle(.secondary) |
| + .fixedSize(horizontal: false, vertical: true) |
| + .accessibilityIdentifier("audioProcessing") |
| + } |
| } |
| } |
diff --git a/native/apple/Sources/CallsKit/Media.swift b/native/apple/Sources/CallsKit/Media.swift| index 1562123..6d72d83 100644 |
| --- a/native/apple/Sources/CallsKit/Media.swift |
| +++ b/native/apple/Sources/CallsKit/Media.swift |
| @@ -28,7 +28,7 @@ import LiveKitWebRTC |
| close() |
| let source = factory.videoSource() |
| localVideo = factory.videoTrack(with: source, trackId: "camera") |
| - localAudio = factory.audioTrack(withTrackId: "microphone") |
| + localAudio = audio.makeTrack() |
| let capture = supplied ?? CameraCapture() |
| self.capture = capture |
| try await capture.start(source: source) |
diff --git a/native/apple/Sources/CallsKit/NoiseSuppression.swift b/native/apple/Sources/CallsKit/NoiseSuppression.swift| new file mode 100644 |
| index 0000000..83935a1 |
| --- /dev/null |
| +++ b/native/apple/Sources/CallsKit/NoiseSuppression.swift |
| @@ -0,0 +1,238 @@ |
| +import Foundation |
| +import LiveKitWebRTC |
| +import RNNoise |
| +import os |
| + |
| +// Noise suppression for the microphone, run inside WebRTC's capture |
| +// processing. |
| +// |
| +// WebRTC's audio processing runs high-pass filter → echo canceller → |
| +// noise suppressor → gain control, then hands each 10 ms frame to a |
| +// "capture post-processing" delegate before encoding. Its own noise |
| +// suppressor is the classic statistical one: good on steady hiss, poor on |
| +// wind, typing and a room's echo, which are what people actually hear. |
| +// So WebRTC's is switched off, and a neural suppressor runs in the |
| +// delegate instead, after the echo canceller has taken out the far side. |
| +// |
| +// The delegate is skipped by WebRTC while the microphone is muted (its |
| +// output is unused), and runs on WebRTC's audio thread: nothing here may |
| +// block, allocate per frame, or wait for the main actor. |
| + |
| +/// A mono, frame-at-a-time neural noise suppressor. |
| +/// |
| +/// `process` works in place on one frame of `frameLength` samples in |
| +/// WebRTC's float scale — ±32768 for full scale, int16's range as floats — |
| +/// and returns how sure it is that the frame is speech, 0 to 1. Each |
| +/// implementation owns its own conversion to whatever scale its model |
| +/// wants. |
| +public protocol NoiseSuppressor: AnyObject { |
| + /// Shown in the audio readout. |
| + var name: String { get } |
| + /// Samples per frame at 48 kHz, the only rate either model runs at. |
| + var frameLength: Int { get } |
| + func process(_ frame: UnsafeMutablePointer<Float>) -> Float |
| +} |
| + |
| +/// xiph's RNNoise: a small recurrent network, BSD-3, 480-sample frames at |
| +/// 48 kHz, and — unusually — in WebRTC's own ±32768 scale already, so the |
| +/// samples go in and come out untouched by any conversion. |
| +public final class RNNoiseSuppressor: NoiseSuppressor { |
| + public let name = "RNNoise" |
| + public let frameLength: Int |
| + private let state: OpaquePointer |
| + private let scratch: UnsafeMutablePointer<Float> |
| + |
| + public init() { |
| + state = rnnoise_create(nil) |
| + frameLength = Int(rnnoise_get_frame_size()) |
| + scratch = .allocate(capacity: frameLength) |
| + } |
| + deinit { |
| + rnnoise_destroy(state) |
| + scratch.deallocate() |
| + } |
| + public func process(_ frame: UnsafeMutablePointer<Float>) -> Float { |
| + scratch.update(from: frame, count: frameLength) |
| + return rnnoise_process_frame(state, frame, scratch) |
| + } |
| +} |
| + |
| +/// A gentle gain after the suppressor, instead of WebRTC's gain control. |
| +/// |
| +/// WebRTC's gain control runs *before* the capture post-processing hook, |
| +/// so left on it would raise the room along with a quiet voice and then |
| +/// hand the suppressor louder noise to remove — the pumping a denoised |
| +/// call is known for. It is switched off while a suppressor runs, and this |
| +/// takes its place, after the noise is gone. |
| +/// |
| +/// Conservative on purpose: it only ever lifts (0 to +12 dB), it only |
| +/// learns from frames the suppressor calls speech, it rises slowly |
| +/// (about 1 dB a second of speech) and falls fast, and a per-frame peak |
| +/// limit keeps it from ever clipping. A voice already at a sensible level |
| +/// is left alone. |
| +public struct SpeechGain { |
| + /// Speech level aimed for, dBFS RMS. -26 is the usual target for |
| + /// conversational speech (ITU-T P.56 practice). |
| + public static let targetDb: Float = -26 |
| + public static let maxGainDb: Float = 12 |
| + public static let voiced: Float = 0.6 |
| + public private(set) var gainDb: Float = 0 |
| + |
| + public init() {} |
| + |
| + public mutating func apply(_ frame: UnsafeMutablePointer<Float>, count: Int, speech: Float) { |
| + var energy: Float = 0 |
| + var peak: Float = 0 |
| + for i in 0..<count { |
| + energy += frame[i] * frame[i] |
| + peak = max(peak, abs(frame[i])) |
| + } |
| + let rms = (energy / Float(max(count, 1))).squareRoot() / 32768 |
| + if speech >= SpeechGain.voiced, rms > 0.001 { // above -60 dBFS |
| + let levelDb = 20 * log10(rms) |
| + let wanted = min(max(SpeechGain.targetDb - levelDb, 0), SpeechGain.maxGainDb) |
| + // Up by 0.01 dB a frame (1 dB/s), down by up to 1 dB a frame. |
| + gainDb += min(max(wanted - gainDb, -1), 0.01) |
| + } |
| + guard gainDb > 0.01 else { return } |
| + var gain = pow(10, gainDb / 20) |
| + let limit: Float = 0.9 * 32768 |
| + if peak * gain > limit { gain = max(1, limit / peak) } |
| + for i in 0..<count { frame[i] *= gain } |
| + } |
| +} |
| + |
| +/// What the suppressor is doing, for the audio readout. |
| +public enum SuppressionState: Equatable, Sendable { |
| + /// Toggle off, or nothing has asked yet. |
| + case off |
| + /// Processing frames; `cpu` is the fraction of one core it is using. |
| + case running(engine: String, cpu: Double) |
| + /// Initialised and waiting — muted (WebRTC skips processing then), or |
| + /// no audio has flowed yet. |
| + case idle(engine: String) |
| + /// The capture runs at a rate the models do not (a Bluetooth headset's |
| + /// 16 kHz, say): frames pass through untouched. |
| + case unsupportedRate(Int) |
| + /// Apple's Voice Isolation is on; two suppressors in a row would only |
| + /// take more of the voice with them. |
| + case standingAside |
| +} |
| + |
| +/// The frame adapter between WebRTC's capture hook and a suppressor. |
| +/// |
| +/// WebRTC hands over 10 ms at a time, 480 samples at 48 kHz, which is one |
| +/// frame of either model; a buffer that is a whole number of frames is |
| +/// processed frame by frame, and anything else passes through untouched. |
| +/// So does every rate but 48 kHz, and every channel count but one. |
| +/// |
| +/// The suppressor is made once and kept across WebRTC's |
| +/// initialise/release cycles, which happen whenever the processing is |
| +/// reconfigured: a model can take a moment to load, and the audio thread |
| +/// is no place to do it. `release` stops processing; it does not free. |
| +public final class CaptureDenoiser: NSObject, RTCAudioCustomProcessingDelegate, @unchecked Sendable { |
| + public static let supportedRate = 48_000 |
| + |
| + private struct State { |
| + var enabled = true |
| + var standAside = false |
| + var rate = 0 |
| + var channels = 0 |
| + var initialised = false |
| + var frames: UInt64 = 0 |
| + var busySeconds: Double = 0 |
| + var lastFrame: Double = -.infinity |
| + var cpu: Double = 0 |
| + } |
| + private let lock = OSAllocatedUnfairLock(initialState: State()) |
| + private let suppressor: NoiseSuppressor |
| + private var gain = SpeechGain() |
| + private let clock: () -> Double |
| + /// Called (on the audio thread) when WebRTC reinitialises at a rate the |
| + /// suppressor can or cannot take, so the processing options can follow. |
| + public var rateChanged: (@Sendable (Int) -> Void)? |
| + |
| + public init(suppressor: NoiseSuppressor, clock: @escaping () -> Double = CaptureDenoiser.now) { |
| + self.suppressor = suppressor |
| + self.clock = clock |
| + } |
| + |
| + public static func now() -> Double { ProcessInfo.processInfo.systemUptime } |
| + |
| + public var engineName: String { suppressor.name } |
| + |
| + /// The toggle. Off passes frames through untouched. |
| + public var enabled: Bool { |
| + get { lock.withLock { $0.enabled } } |
| + set { lock.withLock { $0.enabled = newValue } } |
| + } |
| + /// Set while Apple's Voice Isolation is doing the job. |
| + public var standAside: Bool { |
| + get { lock.withLock { $0.standAside } } |
| + set { lock.withLock { $0.standAside = newValue } } |
| + } |
| + public var framesProcessed: UInt64 { lock.withLock { $0.frames } } |
| + |
| + public var state: SuppressionState { |
| + let now = clock() |
| + return lock.withLock { s in |
| + if !s.enabled { return .off } |
| + if s.standAside { return .standingAside } |
| + if s.initialised, s.rate != CaptureDenoiser.supportedRate { return .unsupportedRate(s.rate) } |
| + if s.initialised, now - s.lastFrame < 1 { return .running(engine: suppressor.name, cpu: s.cpu) } |
| + return .idle(engine: suppressor.name) |
| + } |
| + } |
| + |
| + // MARK: RTCAudioCustomProcessingDelegate |
| + |
| + public func audioProcessingInitialize(sampleRate sampleRateHz: Int, channels: Int) { |
| + let changed = lock.withLock { s -> Bool in |
| + let changed = !s.initialised || s.rate != sampleRateHz |
| + s.rate = sampleRateHz |
| + s.channels = channels |
| + s.initialised = true |
| + return changed |
| + } |
| + if changed { rateChanged?(sampleRateHz) } |
| + } |
| + |
| + public func audioProcessingProcess(audioBuffer: RTCAudioBuffer) { |
| + guard audioBuffer.channels == 1 else { return } |
| + process(audioBuffer.rawBuffer(forChannel: 0), frames: audioBuffer.frames) |
| + } |
| + |
| + public func audioProcessingRelease() { |
| + lock.withLock { $0.initialised = false } |
| + } |
| + |
| + // MARK: The adapter itself, apart from WebRTC's buffer type |
| + |
| + /// Processes one buffer in place. Public for the tests, which cannot |
| + /// make an RTCAudioBuffer. |
| + public func process(_ samples: UnsafeMutablePointer<Float>, frames: Int) { |
| + let run = lock.withLock { s in |
| + s.enabled && !s.standAside && s.initialised && s.channels == 1 |
| + && s.rate == CaptureDenoiser.supportedRate |
| + } |
| + let length = suppressor.frameLength |
| + guard run, frames > 0, frames % length == 0 else { return } |
| + let started = clock() |
| + var offset = 0 |
| + while offset < frames { |
| + let frame = samples + offset |
| + let speech = suppressor.process(frame) |
| + gain.apply(frame, count: length, speech: speech) |
| + offset += length |
| + } |
| + let finished = clock() |
| + let audio = Double(frames) / Double(CaptureDenoiser.supportedRate) |
| + lock.withLock { s in |
| + s.frames += UInt64(frames / length) |
| + s.lastFrame = finished |
| + // A smoothed share of one core: time spent against audio covered. |
| + let share = (finished - started) / audio |
| + s.cpu = s.frames <= UInt64(frames / length) ? share : s.cpu * 0.98 + share * 0.02 |
| + } |
| + } |
| +} |
diff --git a/native/apple/Tests/CallsKitTests/Fixtures/speech-48k.wav b/native/apple/Tests/CallsKitTests/Fixtures/speech-48k.wav| new file mode 100644 |
| index 0000000..962e67c |
| Binary files /dev/null and b/native/apple/Tests/CallsKitTests/Fixtures/speech-48k.wav differ |
diff --git a/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift b/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift| new file mode 100644 |
| index 0000000..ddaca5f |
| --- /dev/null |
| +++ b/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift |
| @@ -0,0 +1,362 @@ |
| +import Foundation |
| +import XCTest |
| + |
| +@testable import CallsKit |
| + |
| +/// Stands in for a model: records what it was handed and halves it, so a |
| +/// test can see both that a frame reached it and that the result was |
| +/// written back in place. |
| +private final class FakeSuppressor: NoiseSuppressor { |
| + let name = "Fake" |
| + let frameLength = 480 |
| + var calls = 0 |
| + var seen: [Float] = [] |
| + func process(_ frame: UnsafeMutablePointer<Float>) -> Float { |
| + calls += 1 |
| + if seen.isEmpty { seen = Array(UnsafeBufferPointer(start: frame, count: frameLength)) } |
| + for i in 0..<frameLength { frame[i] *= 0.5 } |
| + return 0 |
| + } |
| +} |
| + |
| +private final class Clock: @unchecked Sendable { |
| + var now = 100.0 |
| +} |
| + |
| +/// The frame adapter: what reaches the suppressor, when, and when nothing |
| +/// does. |
| +final class CaptureDenoiserTests: XCTestCase { |
| + private func adapter() -> (CaptureDenoiser, FakeSuppressor, Clock) { |
| + let fake = FakeSuppressor() |
| + let clock = Clock() |
| + return (CaptureDenoiser(suppressor: fake, clock: { clock.now }), fake, clock) |
| + } |
| + |
| + private func run(_ denoiser: CaptureDenoiser, count: Int, value: Float = 1000) -> [Float] { |
| + var samples = [Float](repeating: value, count: count) |
| + samples.withUnsafeMutableBufferPointer { denoiser.process($0.baseAddress!, frames: count) } |
| + return samples |
| + } |
| + |
| + func testTenMillisecondsAt48kIsOneFrameAndIsProcessedInPlace() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + let out = run(denoiser, count: 480) |
| + XCTAssertEqual(fake.calls, 1) |
| + XCTAssertEqual(out, [Float](repeating: 500, count: 480), "the result is written back in place") |
| + XCTAssertEqual(denoiser.framesProcessed, 1) |
| + } |
| + |
| + func testSamplesReachTheSuppressorInWebRTCScaleUnconverted() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + _ = run(denoiser, count: 480, value: 32_000) |
| + XCTAssertEqual(fake.seen.first, 32_000, "the adapter does not rescale; each suppressor owns its scale") |
| + } |
| + |
| + func testAWholeNumberOfFramesIsSplitIntoFrames() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + _ = run(denoiser, count: 960) |
| + XCTAssertEqual(fake.calls, 2) |
| + } |
| + |
| + func testAPartFrameIsPassedThroughUntouched() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + let out = run(denoiser, count: 441) |
| + XCTAssertEqual(fake.calls, 0) |
| + XCTAssertEqual(out, [Float](repeating: 1000, count: 441)) |
| + } |
| + |
| + func testOtherRatesPassThroughAndSaySo() { |
| + for rate in [8_000, 16_000, 32_000, 44_100] { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: rate, channels: 1) |
| + let out = run(denoiser, count: rate / 100) |
| + XCTAssertEqual(fake.calls, 0, "\(rate) Hz") |
| + XCTAssertEqual(out, [Float](repeating: 1000, count: rate / 100)) |
| + XCTAssertEqual(denoiser.state, .unsupportedRate(rate)) |
| + } |
| + } |
| + |
| + func testMoreThanOneChannelPassesThrough() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 2) |
| + _ = run(denoiser, count: 480) |
| + XCTAssertEqual(fake.calls, 0) |
| + } |
| + |
| + func testNothingIsProcessedBeforeInitialiseOrAfterRelease() { |
| + let (denoiser, fake, _) = adapter() |
| + _ = run(denoiser, count: 480) |
| + XCTAssertEqual(fake.calls, 0, "before initialise") |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + _ = run(denoiser, count: 480) |
| + denoiser.audioProcessingRelease() |
| + _ = run(denoiser, count: 480) |
| + XCTAssertEqual(fake.calls, 1, "after release") |
| + // Released is not freed: WebRTC reinitialises whenever processing is |
| + // reconfigured, and the same suppressor carries on. |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + _ = run(denoiser, count: 480) |
| + XCTAssertEqual(fake.calls, 2) |
| + } |
| + |
| + func testRateChangesAreReportedOnceEach() { |
| + let (denoiser, _, _) = adapter() |
| + let seen = LockedRates() |
| + denoiser.rateChanged = { seen.append($0) } |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + denoiser.audioProcessingInitialize(sampleRate: 16_000, channels: 1) |
| + XCTAssertEqual(seen.rates, [48_000, 16_000]) |
| + } |
| + |
| + func testTheToggleAndVoiceIsolationBothPassThrough() { |
| + let (denoiser, fake, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + denoiser.enabled = false |
| + XCTAssertEqual(run(denoiser, count: 480).first, 1000) |
| + XCTAssertEqual(denoiser.state, .off) |
| + denoiser.enabled = true |
| + denoiser.standAside = true |
| + XCTAssertEqual(run(denoiser, count: 480).first, 1000) |
| + XCTAssertEqual(denoiser.state, .standingAside) |
| + XCTAssertEqual(fake.calls, 0) |
| + } |
| + |
| + /// WebRTC stops calling the hook while the microphone is muted. The |
| + /// readout must then stop claiming the suppressor is running. |
| + func testMutedReadsAsIdleOnceFramesStop() { |
| + let (denoiser, _, clock) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + XCTAssertEqual(denoiser.state, .idle(engine: "Fake")) |
| + _ = run(denoiser, count: 480) |
| + guard case .running(let engine, _) = denoiser.state else { |
| + return XCTFail("expected running, got \(denoiser.state)") |
| + } |
| + XCTAssertEqual(engine, "Fake") |
| + clock.now += 2 |
| + XCTAssertEqual(denoiser.state, .idle(engine: "Fake")) |
| + } |
| +} |
| + |
| +private final class LockedRates: @unchecked Sendable { |
| + private let lock = NSLock() |
| + private var values: [Int] = [] |
| + func append(_ rate: Int) { lock.withLock { values.append(rate) } } |
| + var rates: [Int] { lock.withLock { values } } |
| +} |
| + |
| +final class SpeechGainTests: XCTestCase { |
| + private func frame(_ amplitude: Float) -> [Float] { |
| + (0..<480).map { amplitude * sin(Float($0) * 2 * .pi * 200 / 48_000) } |
| + } |
| + |
| + func testQuietSpeechIsLiftedSlowlyAndNoMoreThanTwelveDecibels() { |
| + var gain = SpeechGain() |
| + for _ in 0..<5_000 { |
| + var f = frame(100) // about -53 dBFS |
| + f.withUnsafeMutableBufferPointer { gain.apply($0.baseAddress!, count: 480, speech: 1) } |
| + } |
| + XCTAssertEqual(gain.gainDb, SpeechGain.maxGainDb, accuracy: 0.01) |
| + var one = SpeechGain() |
| + var f = frame(100) |
| + f.withUnsafeMutableBufferPointer { one.apply($0.baseAddress!, count: 480, speech: 1) } |
| + XCTAssertLessThanOrEqual(one.gainDb, 0.011, "one frame moves it by at most 0.01 dB") |
| + } |
| + |
| + func testNoiseIsNeverLiftedAndNothingIsAttenuated() { |
| + var gain = SpeechGain() |
| + for _ in 0..<1_000 { |
| + var f = frame(100) |
| + f.withUnsafeMutableBufferPointer { gain.apply($0.baseAddress!, count: 480, speech: 0.1) } |
| + } |
| + XCTAssertEqual(gain.gainDb, 0) |
| + var loud = SpeechGain() |
| + var f = frame(30_000) |
| + let before = f |
| + f.withUnsafeMutableBufferPointer { loud.apply($0.baseAddress!, count: 480, speech: 1) } |
| + XCTAssertEqual(f, before, "a loud voice is left alone") |
| + } |
| + |
| + func testTheLiftNeverClips() { |
| + var gain = SpeechGain() |
| + for _ in 0..<5_000 { |
| + var f = frame(100) |
| + f.withUnsafeMutableBufferPointer { gain.apply($0.baseAddress!, count: 480, speech: 1) } |
| + } |
| + var burst = frame(29_000) |
| + burst.withUnsafeMutableBufferPointer { gain.apply($0.baseAddress!, count: 480, speech: 1) } |
| + XCTAssertLessThanOrEqual(burst.map(abs).max()!, 0.9 * 32_768 + 1) |
| + } |
| +} |
| + |
| +final class ProcessingPlanTests: XCTestCase { |
| + func testMacRunsOursInPlaceOfWebRTCsSuppressorAndGain() { |
| + let plan = ProcessingPlan.plan(iPhone: false, reduceNoise: true, rate: 48_000, voiceIsolation: false) |
| + XCTAssertTrue(plan.suppressor) |
| + XCTAssertEqual(plan.noiseSuppression, .off) |
| + XCTAssertEqual(plan.gainControl, .off, "WebRTC's gain runs before ours and would pump the noise") |
| + XCTAssertEqual(plan.echoCancellation, .software) |
| + XCTAssertTrue(plan.highPass) |
| + } |
| + |
| + func testAnUnsupportedRateHandsBackToWebRTCsSuppressor() { |
| + let plan = ProcessingPlan.plan(iPhone: false, reduceNoise: true, rate: 16_000, voiceIsolation: false) |
| + XCTAssertFalse(plan.suppressor) |
| + XCTAssertEqual(plan.noiseSuppression, .software) |
| + XCTAssertEqual(plan.gainControl, .software) |
| + } |
| + |
| + func testOffMeansNoSuppressionButWebRTCsGain() { |
| + let plan = ProcessingPlan.plan(iPhone: false, reduceNoise: false, rate: 48_000, voiceIsolation: false) |
| + XCTAssertFalse(plan.suppressor) |
| + XCTAssertEqual(plan.noiseSuppression, .off) |
| + XCTAssertEqual(plan.gainControl, .software) |
| + } |
| + |
| + func testVoiceIsolationIsNeverStackedWith() { |
| + for iPhone in [false, true] { |
| + let plan = ProcessingPlan.plan(iPhone: iPhone, reduceNoise: true, rate: 48_000, voiceIsolation: true) |
| + XCTAssertFalse(plan.suppressor) |
| + } |
| + } |
| + |
| + func testIPhoneKeepsApplesCoupledEchoCancellerAndDropsItsGainForOurs() { |
| + let plan = ProcessingPlan.plan(iPhone: true, reduceNoise: true, rate: 48_000, voiceIsolation: false) |
| + XCTAssertTrue(plan.suppressor) |
| + XCTAssertEqual(plan.echoCancellation, .platform) |
| + XCTAssertEqual(plan.noiseSuppression, .platform) |
| + XCTAssertEqual(plan.gainControl, .off) |
| + } |
| + |
| + /// The rule, across every combination: ours never runs beside WebRTC's |
| + /// software suppressor, and never behind WebRTC's gain control. |
| + func testOursNeverStacksWithWebRTCsSuppressorOrFollowsItsGain() { |
| + for iPhone in [false, true] { |
| + for reduce in [false, true] { |
| + for rate in [nil, 8_000, 16_000, 48_000] as [Int?] { |
| + for isolation in [false, true] { |
| + let plan = ProcessingPlan.plan( |
| + iPhone: iPhone, reduceNoise: reduce, rate: rate, voiceIsolation: isolation) |
| + if plan.suppressor { |
| + XCTAssertNotEqual(plan.noiseSuppression, .software) |
| + XCTAssertEqual(plan.gainControl, .off) |
| + } |
| + } |
| + } |
| + } |
| + } |
| + } |
| +} |
| + |
| +/// Real speech (macOS's own synthesiser, normalised to -23 dBFS) with |
| +/// noise mixed in, through each real suppressor. |
| +/// |
| +/// Two claims, measured separately because a suppressor can satisfy |
| +/// either by failing the other: the noise-only lead-in is made much |
| +/// quieter, and the speech is not. |
| +final class SuppressorFixtureTests: XCTestCase { |
| + static let rate = 48_000 |
| + |
| + static func speech() throws -> [Float] { |
| + let url = try XCTUnwrap(Bundle.module.url(forResource: "Fixtures/speech-48k", withExtension: "wav")) |
| + let data = try Data(contentsOf: url) |
| + // A canonical 44-byte header, mono 16-bit at 48 kHz, as written. |
| + XCTAssertEqual(data.count > 44, true) |
| + return data.dropFirst(44).withUnsafeBytes { raw in |
| + raw.bindMemory(to: Int16.self).map { Float(Int16(littleEndian: $0)) } |
| + } |
| + } |
| + |
| + /// Broadband hiss (a fan, a room) at about -35 dBFS, plus a key click |
| + /// every 180 ms. Seeded, so every run hears the same room. |
| + static func noise(count: Int) -> [Float] { |
| + var seed: UInt32 = 0x2545_F491 |
| + var low: Float = 0 |
| + var out = [Float](repeating: 0, count: count) |
| + for i in 0..<count { |
| + seed = seed &* 1_664_525 &+ 1_013_904_223 |
| + let white = Float(Int32(bitPattern: seed)) / Float(Int32.max) |
| + low = 0.7 * low + 0.3 * white |
| + var sample = low * 900 |
| + let sinceClick = i % (rate * 18 / 100) |
| + if sinceClick < 240 { |
| + sample += 6_000 * white * exp(-Float(sinceClick) / 40) |
| + } |
| + out[i] = sample |
| + } |
| + return out |
| + } |
| + |
| + static func db(_ samples: ArraySlice<Float>) -> Double { |
| + let energy = samples.reduce(0.0) { $0 + Double($1) * Double($1) } / Double(max(samples.count, 1)) |
| + return 10 * log10(energy / (32_768.0 * 32_768.0) + 1e-12) |
| + } |
| + |
| + struct Result { |
| + let noiseBefore: Double |
| + let noiseAfter: Double |
| + let speechClean: Double |
| + let speechAfter: Double |
| + let voicedKept: Double |
| + let msPerFrame: Double |
| + } |
| + |
| + static func measure(_ suppressor: NoiseSuppressor) throws -> Result { |
| + let clean = try speech() |
| + let noise = noise(count: clean.count) |
| + var noisy = zip(clean, noise).map { $0 + $1 } |
| + let length = suppressor.frameLength |
| + let frames = noisy.count / length |
| + let started = ProcessInfo.processInfo.systemUptime |
| + noisy.withUnsafeMutableBufferPointer { buffer in |
| + for f in 0..<frames { _ = suppressor.process(buffer.baseAddress! + f * length) } |
| + } |
| + let elapsed = ProcessInfo.processInfo.systemUptime - started |
| + let out = noisy |
| + // The lead-in is noise alone; skip the first 300 ms while the model |
| + // settles. |
| + let lead = (rate * 3 / 10)..<rate |
| + let speechRange = rate..<(frames * length - rate / 2) |
| + // Per-frame: of the frames where the clean voice is clearly present, |
| + // how many came out within 10 dB of it. Compared a frame late too, |
| + // since a model may delay its output by a frame. |
| + var voiced = 0 |
| + var kept = 0 |
| + for f in stride(from: speechRange.lowerBound, to: speechRange.upperBound - 2 * length, by: length) { |
| + let reference = db(clean[f..<(f + length)]) |
| + guard reference > -40 else { continue } |
| + voiced += 1 |
| + let now = db(out[f..<(f + length)]) |
| + let late = db(out[(f + length)..<(f + 2 * length)]) |
| + if max(now, late) > reference - 10 { kept += 1 } |
| + } |
| + return Result( |
| + noiseBefore: db(noise[lead]), noiseAfter: db(out[lead]), |
| + speechClean: db(clean[speechRange]), speechAfter: db(out[speechRange]), |
| + voicedKept: Double(kept) / Double(max(voiced, 1)), |
| + msPerFrame: elapsed * 1000 / Double(frames)) |
| + } |
| + |
| + static func check(_ result: Result, name: String, minimumReduction: Double) { |
| + print( |
| + "\(name): noise \(String(format: "%.1f", result.noiseBefore)) -> \(String(format: "%.1f", result.noiseAfter)) dBFS;" |
| + + " speech \(String(format: "%.1f", result.speechClean)) -> \(String(format: "%.1f", result.speechAfter)) dBFS;" |
| + + " voiced frames kept \(Int(result.voicedKept * 100))%;" |
| + + " \(String(format: "%.3f", result.msPerFrame)) ms per 10 ms frame") |
| + XCTAssertLessThan( |
| + result.noiseAfter, result.noiseBefore - minimumReduction, |
| + "\(name) should take at least \(minimumReduction) dB off noise alone") |
| + XCTAssertGreaterThan( |
| + result.speechAfter, result.speechClean - 6, "\(name) must not take the speech with it") |
| + XCTAssertGreaterThan(result.voicedKept, 0.8, "\(name) must keep the voiced frames") |
| + } |
| + |
| + func testRNNoiseQuietsNoiseAndKeepsSpeech() throws { |
| + let result = try Self.measure(RNNoiseSuppressor()) |
| + Self.check(result, name: "RNNoise", minimumReduction: 10) |
| + } |
| +} |
diff --git a/native/ios/project.yml b/native/ios/project.yml| index 185319d..7b6766a 100644 |
| --- a/native/ios/project.yml |
| +++ b/native/ios/project.yml |
| @@ -21,6 +21,8 @@ targets: |
| - Assets.xcassets |
| - path: ../LICENSE.lucide |
| buildPhase: resources |
| + - path: ../LICENSE.rnnoise |
| + buildPhase: resources |
| info: |
| path: ../build/ios-info.plist |
| properties: |
diff --git a/native/scripts/build-iphone.sh b/native/scripts/build-iphone.sh| index 3de8608..b496d4f 100755 |
| --- a/native/scripts/build-iphone.sh |
| +++ b/native/scripts/build-iphone.sh |
| @@ -2,6 +2,7 @@ |
| set -eu |
| cd "$(dirname "$0")/.." |
| ./scripts/build-bridge.sh |
| +./scripts/build-rnnoise.sh |
| xcodegen generate --spec ios/project.yml |
| case "${1:-simulator}" in |
| simulator) |
diff --git a/native/scripts/build-mac.sh b/native/scripts/build-mac.sh| index 12a3634..b100234 100755 |
| --- a/native/scripts/build-mac.sh |
| +++ b/native/scripts/build-mac.sh |
| @@ -1,7 +1,7 @@ |
| #!/bin/sh |
| set -eu |
| cd "$(dirname "$0")/.." |
| -if [ "${OFICINA_SKIP_PREPARE:-0}" != 1 ]; then ./scripts/build-bridge.sh; fi |
| +if [ "${OFICINA_SKIP_PREPARE:-0}" != 1 ]; then ./scripts/build-bridge.sh; ./scripts/build-rnnoise.sh; fi |
| configuration="${OFICINA_BUILD_CONFIGURATION:-debug}" |
| swift build --package-path apple --configuration "$configuration" --product CallsMac |
| python3 scripts/bundle-mac.py "$configuration" |
diff --git a/native/scripts/build-rnnoise.sh b/native/scripts/build-rnnoise.sh| new file mode 100755 |
| index 0000000..91657f5 |
| --- /dev/null |
| +++ b/native/scripts/build-rnnoise.sh |
| @@ -0,0 +1,87 @@ |
| +#!/bin/sh |
| +# Builds native/build/RNNoise.xcframework: xiph's RNNoise (BSD-3) as a |
| +# static library for Mac, iPhone and the iPhone simulator. |
| +# |
| +# Built rather than committed, as CallsBridge is: the model's weights come |
| +# as a generated C file of 78 MB (rnnoise_data.c), which is no more source |
| +# than a framework binary is. Both the code and the model are pinned — the |
| +# code to a commit, the model to its SHA-256, which is also its name. |
| +set -eu |
| +cd "$(dirname "$0")/.." |
| + |
| +commit=70f1d256acd4b34a572f999a05c87bf00b67730d |
| +model=0a8755f8e2d834eff6a54714ecc7d75f9932e845df35f8b59bc52a7cfe6e8b37 |
| +out=build/RNNoise.xcframework |
| +stamp="$out/.built-$commit-$model" |
| +if [ -f "$stamp" ]; then exit 0; fi |
| + |
| +work=build/rnnoise |
| +mkdir -p "$work" |
| +if [ ! -d "$work/src/.git" ]; then |
| + rm -rf "$work/src" |
| + git clone --quiet https://gitlab.xiph.org/xiph/rnnoise.git "$work/src" |
| +fi |
| +git -C "$work/src" -c advice.detachedHead=false checkout --quiet "$commit" |
| + |
| +tarball="$work/rnnoise_data-$model.tar.gz" |
| +if [ ! -f "$tarball" ]; then |
| + curl --fail --silent --show-error --location -o "$tarball.part" \ |
| + "https://media.xiph.org/rnnoise/models/rnnoise_data-$model.tar.gz" |
| + mv "$tarball.part" "$tarball" |
| +fi |
| +echo "$model $tarball" | shasum -a 256 -c --status || { |
| + echo "build-rnnoise: model checksum mismatch; delete $tarball and retry" >&2 |
| + exit 1 |
| +} |
| +tar -xzf "$tarball" -C "$work/src" src/rnnoise_data.c src/rnnoise_data.h |
| + |
| +sources="denoise.c rnn.c pitch.c kiss_fft.c celt_lpc.c nnet.c nnet_default.c parse_lpcnet_weights.c rnnoise_tables.c rnnoise_data.c" |
| + |
| +# One static library per platform, fat across its architectures. |
| +build_slice() { # name sdk "triple triple..." |
| + name=$1 sdk=$2 triples=$3 |
| + root=$(xcrun --sdk "$sdk" --show-sdk-path) |
| + libs="" |
| + for triple in $triples; do |
| + objdir="$work/obj/$name/$triple" |
| + mkdir -p "$objdir" |
| + for f in $sources; do |
| + clang -c -O3 -target "$triple" -isysroot "$root" -fvisibility=hidden -DRNNOISE_BUILD \ |
| + -I"$work/src/include" -I"$work/src/src" "$work/src/src/$f" -o "$objdir/${f%.c}.o" & |
| + done |
| + wait |
| + libtool -static -o "$objdir/librnnoise.a" "$objdir"/*.o |
| + libs="$libs $objdir/librnnoise.a" |
| + done |
| + mkdir -p "$work/lib/$name" |
| + # shellcheck disable=SC2086 |
| + lipo -create $libs -output "$work/lib/$name/librnnoise.a" |
| +} |
| +build_slice macos macosx "arm64-apple-macos14.0 x86_64-apple-macos14.0" |
| +build_slice ios iphoneos "arm64-apple-ios17.0" |
| +build_slice simulator iphonesimulator "arm64-apple-ios17.0-simulator x86_64-apple-ios17.0-simulator" |
| + |
| +# Headers in a subdirectory of their own: Xcode copies every static |
| +# xcframework's headers into one include directory, and two module maps at |
| +# its root collide. |
| +headers="$work/headers" |
| +rm -rf "$headers" |
| +mkdir -p "$headers/RNNoise" |
| +cp "$work/src/include/rnnoise.h" "$headers/RNNoise/" |
| +cat >"$headers/RNNoise/module.modulemap" <<'EOF' |
| +module RNNoise { |
| + header "rnnoise.h" |
| + link "rnnoise" |
| + export * |
| +} |
| +EOF |
| + |
| +rm -rf "$out" |
| +xcodebuild -create-xcframework \ |
| + -library "$work/lib/macos/librnnoise.a" -headers "$headers" \ |
| + -library "$work/lib/ios/librnnoise.a" -headers "$headers" \ |
| + -library "$work/lib/simulator/librnnoise.a" -headers "$headers" \ |
| + -output "$out" >/dev/null |
| +cp "$work/src/COPYING" "$out/COPYING" |
| +touch "$stamp" |
| +printf 'Built %s/%s\n' "$PWD" "$out" |
diff --git a/native/scripts/bundle-mac.py b/native/scripts/bundle-mac.py| index e1bebec..9096c9a 100644 |
| --- a/native/scripts/bundle-mac.py |
| +++ b/native/scripts/bundle-mac.py |
| @@ -19,6 +19,7 @@ subprocess.run(['swift', str(root / 'scripts/generate-mac-icon.swift')], check=T |
| icon = root / 'apple/Sources/CallsMac/Resources/MacIcon.png' |
| shutil.copy2(icon, bundle / 'Resources/MacIcon.png') |
| shutil.copy2(root / 'LICENSE.lucide', bundle / 'Resources/LICENSE.lucide') |
| +shutil.copy2(root / 'LICENSE.rnnoise', bundle / 'Resources/LICENSE.rnnoise') |
| # Finder reads the ICNS before the app launches; the PNG also supplies the |
| # runtime Dock icon. Generate each required representation from the master. |
| iconset = root / 'build/AppIcon.iconset' |
diff --git a/test/browser/native.mjs b/test/browser/native.mjs| index e104eec..946437a 100644 |
| --- a/test/browser/native.mjs |
| +++ b/test/browser/native.mjs |
| @@ -73,6 +73,15 @@ try { |
| } |
| await wait(() => sample?.stats.framesDecoded > 15 && sample?.stats.audioBytesReceived > 0 && sample?.stats.videoBytesSent > 0 && sample?.stats.audioBytesSent > 0, 'native receives browser audio/video'); |
| console.log('PASS native decoded browser video and received audio'); |
| + // The synthetic microphone runs through WebRTC's capture processing, so |
| + // the noise suppressor in its post-processing hook must have seen frames. |
| + await wait(() => sample?.stats.noiseFramesProcessed > 100, 'native noise suppressor processes captured audio'); |
| + console.log('PASS native noise suppressor ran on captured audio'); |
| + // And it runs instead of WebRTC's own suppressor and gain, beside its echo canceller. |
| + assert.equal(sample.stats.webrtcNoiseSuppression, 0, "WebRTC's noise suppressor is off"); |
| + assert.equal(sample.stats.webrtcGainControl, 0, "WebRTC's gain control is off"); |
| + assert.equal(sample.stats.webrtcEchoCancellation, 1, "echo cancellation stays on"); |
| + console.log('PASS WebRTC noise suppression and gain off, echo cancellation on'); |
| // Each remote video has its own decoded frame count; local preview alone |
| // cannot satisfy this check. Read the actual room test observation contract. |
| // Remote videos exclude the muted local preview. getVideoPlaybackQuality |