amadan
Run DeepFilterNet 3 on Mac, RNNoise on iPhone
DeepFilterNet 3 (libDF's C API, MIT or Apache-2.0) is the Mac's noise suppressor: better than RNNoise on typing, wind and voices in the next room. 480-sample hops at 48 kHz, scaled by 1/32768 in and out, about 30 ms of added latency. build-deepfilter.sh builds a static xcframework for aarch64-apple-darwin, -ios and -ios-sim with a pinned rustup toolchain and cbindgen, and copies the standard model beside it; like CallsBridge and RNNoise it is not committed. It is linked on Mac only; the Mac bundle carries the model and the notice. The model takes about 0.3 s to load, so a call starts on RNNoise and swaps to DeepFilterNet when it has loaded off the main thread; a missing model leaves it on RNNoise. Both suppressors are limited to 30 dB of attenuation (RNNoise by mixing its input back in), so a room is not gated to the digital silence that sounds like a dropped line. Measured on an M1 Max: DeepFilterNet 3 1.16-1.23 ms per 10 ms frame in isolation, median 15% of the audio thread's time in the native/browser gate; RNNoise 0.69-0.81 ms and 12.9%. That holds real time with room to spare, so it is the Mac default. It has not been measured on an iPhone, which stays on RNNoise. If WebRTC refuses the platform-mode options (the iPhone's), the enabled stages fall back to automatic and the readout keeps the refusal. make native-browser points the harness at the model, and the gate now names the suppressor that ran. VERIFICATION.md records this branch.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
14 files changed,
+383
−39
Makefile+1 −1native/LICENSE.deepfilternet+22 −0native/README.md+42 −11native/VERIFICATION.md+40 −0native/apple/Package.swift+5 −1native/apple/Sources/CallsKit/Call.swift+4 −0native/apple/Sources/CallsKit/CallAudio.swift+28 −5native/apple/Sources/CallsKit/NoiseSuppression.swift+90 −9native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift+73 −10native/scripts/build-deepfilter.sh+71 −0native/scripts/build-iphone.sh+1 −0native/scripts/build-mac.sh+1 −1native/scripts/bundle-mac.py+4 −0test/browser/native.mjs+1 −1
diff --git a/Makefile b/Makefile| index 04938d4..dbeab73 100644 |
| --- a/Makefile |
| +++ b/Makefile |
| @@ -169,7 +169,7 @@ native-browser: native-mac |
| # The gate runs CallsInterop, which native-mac does not build; without |
| # this it runs whatever binary a previous build left behind. |
| swift build --package-path native/apple --product CallsInterop |
| - CGO_ENABLED=0 go test -tags browser -count=1 -timeout 3m -run 'TestNative(MacCallsBrowser|MacChat|BrowserAuthentication)' -v ./internal/callstest/ |
| + CALLS_DEEPFILTER_MODEL=$(CURDIR)/native/build/DeepFilterNet3_onnx.tar.gz CGO_ENABLED=0 go test -tags browser -count=1 -timeout 3m -run 'TestNative(MacCallsBrowser|MacChat|BrowserAuthentication)' -v ./internal/callstest/ |
| .PHONY: iphone iphone-device iphone-browser |
| iphone: |
diff --git a/native/LICENSE.deepfilternet b/native/LICENSE.deepfilternet| new file mode 100644 |
| index 0000000..0344137 |
| --- /dev/null |
| +++ b/native/LICENSE.deepfilternet |
| @@ -0,0 +1,22 @@ |
| +DeepFilterNet (https://github.com/Rikorose/DeepFilterNet), used under the MIT |
| +licence (it is dual-licensed MIT OR Apache-2.0). |
| + |
| +The MIT License (MIT) |
| +Copyright (c) 2021 Hendrik Schröter |
| + |
| +Permission is hereby granted, free of charge, to any person obtaining a copy of |
| +this software and associated documentation files (the "Software"), to deal in |
| +the Software without restriction, including without limitation the rights to |
| +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of |
| +the Software, and to permit persons to whom the Software is furnished to do so, |
| +subject to the following conditions: |
| + |
| +The above copyright notice and this permission notice shall be included in all |
| +copies or substantial portions of the Software. |
| + |
| +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR |
| +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS |
| +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR |
| +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER |
| +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN |
| +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
diff --git a/native/README.md b/native/README.md| index fb274a3..1a456ec 100644 |
| --- a/native/README.md |
| +++ b/native/README.md |
| @@ -89,14 +89,44 @@ core, and what WebRTC reports as running for echo cancellation, noise |
| suppression and gain — what it reports, not what was asked. WebRTC skips the |
| hook while the microphone is muted, and the readout says so. |
| -The suppressor is RNNoise (xiph, BSD-3; notice in `LICENSE.rnnoise`, bundled |
| -in both apps): 480-sample frames at 48 kHz in WebRTC's own ±32768 scale. |
| -Its model weights arrive as a generated C file of 78 MB, so it is not a |
| -source target: `scripts/build-rnnoise.sh` fetches the pinned commit and the |
| -checksummed model and builds `native/build/RNNoise.xcframework` (static, Mac, |
| -iPhone and simulator), which is not committed — the CallsBridge precedent. |
| -`build-mac.sh` and `build-iphone.sh` run it; run it yourself before |
| -`swift test`. It is cached against the pins. |
| +**On Mac the suppressor is DeepFilterNet 3**; on iPhone it is RNNoise. |
| + |
| +- **DeepFilterNet 3** (Rikorose/DeepFilterNet's libDF, MIT or Apache-2.0; |
| + notice in `LICENSE.deepfilternet`, bundled in the Mac app) is markedly |
| + better on non-stationary noise — typing, wind, voices in the next room. It |
| + takes 480-sample hops at 48 kHz in ±1 floats, so samples are scaled by |
| + 1/32768 on the way in and back on the way out, and adds about 30 ms of |
| + latency. `scripts/build-deepfilter.sh` builds `native/build/DeepFilter.xcframework` |
| + (static; `aarch64-apple-darwin`, `aarch64-apple-ios`, `aarch64-apple-ios-sim`) |
| + with a pinned rustup toolchain, cbindgen for the header, and the C API |
| + (`--features capi`), and copies the standard model |
| + (`DeepFilterNet3_onnx.tar.gz`, 8 MB) beside it; the Mac bundle carries the |
| + model. Needs rustup (https://rustup.rs). The library is linked on Mac only. |
| + The model takes about 0.3 s to load, so a call starts on RNNoise and moves |
| + to DeepFilterNet when it has loaded off the main thread; a missing model |
| + leaves it on RNNoise. `CALLS_DEEPFILTER_MODEL` points at a model file |
| + outside a bundle (the tests and the interop harness). |
| +- **RNNoise** (xiph, BSD-3; notice in `LICENSE.rnnoise`, bundled in both apps): |
| + 480-sample frames at 48 kHz in WebRTC's own ±32768 scale. Its model weights |
| + arrive as a generated C file of 78 MB, so it is not a source target: |
| + `scripts/build-rnnoise.sh` fetches the pinned commit and the checksummed |
| + model and builds `native/build/RNNoise.xcframework` (static, Mac, iPhone |
| + and simulator). |
| +- Both are limited to 30 dB of attenuation: unlimited, a room goes to digital |
| + silence between words, which is heard as the line dropping. DeepFilterNet |
| + has the limit built in; RNNoise gets it by mixing the untouched input back |
| + in at -30 dB. |
| +- Measured on an M1 Max (fixture test, isolated): RNNoise 0.69–0.76 ms and |
| + DeepFilterNet 3 1.16–1.23 ms per 10 ms frame, i.e. about 7% and 12% of one |
| + core. In the native/browser gate, as wall-clock share of the audio thread |
| + with Chromium busy on the same machine: RNNoise median 12.9%, DeepFilterNet |
| + median 15.1% (13–27%). DeepFilterNet has not been measured on an iPhone, |
| + which is why the phone stays on RNNoise; the xcframework already carries the |
| + iOS slices, so trying it is a Package.swift condition and a bundled model. |
| + |
| +None of these libraries is committed: `build-mac.sh` and `build-iphone.sh` |
| +run all three scripts (bridge, RNNoise, DeepFilter), each cached against its |
| +pins; run them yourself before `swift test`. |
| Choosing a Bluetooth microphone on iPhone puts the call on the hands-free |
| profile, which is narrowband in both directions. That used to happen |
| @@ -181,7 +211,7 @@ module on macOS (device lists and selection) and a capture post-processing hook |
| for noise suppression. Its Objective-C symbols are prefixed `LK`; CallsKit's |
| `WebRTCNames.swift` aliases the old `RTC…` names once, so code and open branches |
| keep using them and change only their import (`import LiveKitWebRTC`). |
| -Go Mobile is also pinned. The bridge includes Mac, iPhone and iOS Simulator |
| +Go Mobile is also pinned; so are RNNoise and DeepFilterNet (see above). The bridge includes Mac, iPhone and iOS Simulator |
| slices. Generated frameworks, Xcode projects and binaries are ignored. |
| ## Verification |
| @@ -195,8 +225,9 @@ make native-browser |
| CGO_ENABLED=0 go test -tags browser -run TestNativeBrowserAuthentication ./internal/callstest |
| ``` |
| -`swift test` needs `native/scripts/build-bridge.sh` and |
| -`native/scripts/build-rnnoise.sh` to have run. Its suppressor tests feed |
| +`swift test` needs `native/scripts/build-bridge.sh`, |
| +`native/scripts/build-rnnoise.sh` and `native/scripts/build-deepfilter.sh` to |
| +have run. Its suppressor tests feed |
| `Tests/CallsKitTests/Fixtures/speech-48k.wav` (macOS's own speech |
| synthesiser, normalised to -23 dBFS, a second of silence ahead) mixed with |
| seeded hiss and key clicks through each real suppressor, and require the |
diff --git a/native/VERIFICATION.md b/native/VERIFICATION.md| index b33c1bb..3ebb747 100644 |
| --- a/native/VERIFICATION.md |
| +++ b/native/VERIFICATION.md |
| @@ -1,3 +1,43 @@ |
| +# LiveKit WebRTC and neural noise reduction — 2026-09-23 |
| + |
| +Branch: `feat/livekit-webrtc-audio`, based on `origin/main` at `3846ca4`. Not |
| +merged, not deployed. Machine: M1 Max, macOS 27, Xcode 26.6. |
| + |
| +| Check | Result | |
| +| --- | --- | |
| +| `./native/scripts/build-bridge.sh` | Passed | |
| +| `native/scripts/build-rnnoise.sh`, `build-deepfilter.sh` | Built both xcframeworks from their pins (DeepFilter: Rust 1.92.0 via rustup, cbindgen 0.29.2) | |
| +| `swift test` (native/apple) | 48 tests, 0 failures, including the suppressor fixture tests | |
| +| `make native-test` (uncached, `GOFLAGS=-count=1`) | Passed | |
| +| `make native-mac` | Built; ad hoc signature verifies; bundle carries LiveKitWebRTC.framework, the DeepFilterNet model and the RNNoise/DeepFilterNet notices; app launches | |
| +| `make iphone` + `build-for-testing` (simulator) | Build and test build succeeded; RNNoise linked, DeepFilter not | |
| +| `make native-browser` | Passed (member, guest, chat); asserts DeepFilterNet 3 processed the synthetic microphone and WebRTC reports its suppressor and gain off, echo cancellation on | |
| +| `CGO_ENABLED=0 make browser` | 33 of 34 passed; `switching` (web camera switch) failed once under load from a concurrent simulator run and passed alone. This branch does not touch the web client | |
| +| `make iphone-browser` | Failed, and fails identically on a clean `origin/main` worktree (both peers stuck `reconnecting`, 0 frames); the Safari UI test flaked differently on each run. Environmental, not this branch; the first run's media test did pass on this branch | |
| +| `GOTOOLCHAIN=go1.26.5 make ci` | Passed | |
| + |
| +Fixture (synthesised speech at -23 dBFS with seeded hiss and key clicks, each |
| +suppressor alone): RNNoise takes the noise-only lead-in from -41 to -71 dBFS |
| +and DeepFilterNet 3 to -69 dBFS (both at their 30 dB limit); speech holds at |
| +-24.5 → -24.6 dBFS; 99% of voiced frames kept by each. Per 10 ms frame: |
| +RNNoise 0.69–0.81 ms, DeepFilterNet 1.16–1.23 ms (about 7–8% and 12% of one |
| +core). DeepFilterNet's model loads in 0.26–0.35 s. In the native/browser |
| +gate, as wall-clock share of the audio thread with Chromium on the same |
| +machine: DeepFilterNet median 15.1–15.3% (13–27%), RNNoise median 12.9%. |
| + |
| +Needs a real call or device: echo with speakers (software AEC3 on Mac, now on |
| +the AVAudioEngine module), typing, wind and room echo against both models; |
| +that SpeechGain lifts a quiet talker without audible pumping; switching Mac |
| +microphone and speaker mid-call (only one of each on the test machine); |
| +Bluetooth at 16 kHz handing back to WebRTC's suppressor; Voice Isolation on |
| +iPhone standing ours aside; whether iPhone accepts the platform-mode options |
| +(the readout says if not); DeepFilterNet's cost on an iPhone. The Mac audio |
| +bug (growling) is not addressed and the module change may move it either way. |
| +A test process cannot get microphone permission, so Apple voice processing's |
| +live state on Mac and the Mic Mode menu could not be observed. |
| + |
| +--- |
| + |
| # Native completion verification — 2026-09-12 |
| Branch: `codex/iphone-and-mirrored-preview`, based on `86ed9f3`. |
diff --git a/native/apple/Package.swift b/native/apple/Package.swift| index d4204b5..7f4f9ec 100644 |
| --- a/native/apple/Package.swift |
| +++ b/native/apple/Package.swift |
| @@ -21,10 +21,14 @@ let package = Package( |
| .binaryTarget(name: "CallsBridge", path: "../build/CallsBridge.xcframework"), |
| // Built by scripts/build-rnnoise.sh, as CallsBridge is by build-bridge.sh. |
| .binaryTarget(name: "RNNoise", path: "../build/RNNoise.xcframework"), |
| + // Built by scripts/build-deepfilter.sh. Linked on Mac only: the iPhone |
| + // runs RNNoise, and should not carry a model runtime it never uses. |
| + .binaryTarget(name: "DeepFilter", path: "../build/DeepFilter.xcframework"), |
| .target( |
| name: "CallsKit", |
| dependencies: [ |
| - "CallsBridge", "RNNoise", .product(name: "LiveKitWebRTC", package: "webrtc-xcframework"), |
| + "CallsBridge", "RNNoise", .target(name: "DeepFilter", condition: .when(platforms: [.macOS])), |
| + .product(name: "LiveKitWebRTC", package: "webrtc-xcframework"), |
| ]), |
| .executableTarget(name: "CallsMac", dependencies: [.product(name: "RastrilloMacUpdates", package: "native", condition: .when(platforms: [.macOS])), "CallsKit"], resources: [.copy("Resources/AppIcon.png"), .copy("Resources/MacIcon.png")]), |
| .executableTarget(name: "CallsInterop", dependencies: ["CallsKit"]), |
diff --git a/native/apple/Sources/CallsKit/Call.swift b/native/apple/Sources/CallsKit/Call.swift| index d49066f..2a010db 100644 |
| --- a/native/apple/Sources/CallsKit/Call.swift |
| +++ b/native/apple/Sources/CallsKit/Call.swift |
| @@ -357,6 +357,10 @@ private final class Events: NSObject, BridgeListenerProtocol { |
| values["invalidEvents"] = invalidEvents |
| values["rosterPeers"] = UInt64(state?.peers.count ?? 0) |
| values["noiseFramesProcessed"] = media.audio.denoiser.framesProcessed |
| + values["noiseSuppressor"] = media.audio.denoiser.engineName == "RNNoise" ? 1 : 2 |
| + if case .running(_, let cpu) = media.audio.denoiser.state { |
| + values["noiseCpuPermille"] = UInt64((cpu * 1000).rounded()) |
| + } |
| // What WebRTC reports running, not what was asked of it. |
| let running = media.audio.report() |
| values["webrtcEchoCancellation"] = running.echoCancellation == "off" ? 0 : 1 |
diff --git a/native/apple/Sources/CallsKit/CallAudio.swift b/native/apple/Sources/CallsKit/CallAudio.swift| index dd3cfe3..cf7ade5 100644 |
| --- a/native/apple/Sources/CallsKit/CallAudio.swift |
| +++ b/native/apple/Sources/CallsKit/CallAudio.swift |
| @@ -73,7 +73,7 @@ public struct ProcessingReport: Equatable, Sendable { |
| case .off: lines.append("Background noise reduction is off.") |
| case .running(let engine, let cpu): |
| lines.append( |
| - "Reducing background noise with \(engine) (\(String(format: "%.1f", cpu * 100))% of one core).") |
| + "Reducing background noise with \(engine) (\(String(format: "%.1f", cpu * 100))% of real time on the audio thread).") |
| case .idle(let engine): lines.append("Reducing background noise with \(engine) when you speak.") |
| case .unsupportedRate(let rate): |
| lines.append( |
| @@ -173,6 +173,7 @@ public struct ProcessingReport: Equatable, Sendable { |
| // delegate is never called (seen in the spike, M150). The property is |
| // weak; `denoiser` is what keeps it alive. |
| processing.capturePostProcessingDelegate = denoiser |
| + CallAudio.upgrade(denoiser) |
| denoiser.enabled = plan.suppressor |
| denoiser.rateChanged = { [weak self] rate in |
| DispatchQueue.main.async { |
| @@ -185,9 +186,21 @@ public struct ProcessingReport: Equatable, Sendable { |
| } |
| } |
| - /// The suppressor this platform runs. RNNoise on both, for now. |
| + /// The suppressor a call starts on. See `upgrade`. |
| static func makeSuppressor() -> NoiseSuppressor { RNNoiseSuppressor() } |
| + /// DeepFilterNet 3 on Mac, where the app carries its model: loaded off |
| + /// the main thread, then swapped in. RNNoise otherwise, and always on |
| + /// iPhone (DeepFilterNet's cost there has not been measured on a phone). |
| + private static func upgrade(_ denoiser: CaptureDenoiser) { |
| + #if canImport(DeepFilter) |
| + guard let model = DeepFilterSuppressor.model else { return } |
| + DispatchQueue.global(qos: .userInitiated).async { |
| + if let deep = DeepFilterSuppressor(model: model) { denoiser.replace(with: deep) } |
| + } |
| + #endif |
| + } |
| + |
| /// The microphone track, made here so its processing options are set |
| /// before it is ever sent. |
| func makeTrack() -> RTCAudioTrack { |
| @@ -220,14 +233,24 @@ public struct ProcessingReport: Equatable, Sendable { |
| guard let track, processing != nil else { return } |
| let result = track.setAudioProcessingOptions(CallAudio.options(plan)) |
| refused = result.isSuccess ? nil : result.message |
| + // Platform modes can be refused where Apple's unit cannot give them |
| + // (a simulator, a route without it). Then let WebRTC choose where each |
| + // enabled stage runs; what was off stays off, so the plan's rule — |
| + // never two suppressors, never gain before ours — still holds. |
| + if !result.isSuccess { |
| + let fallback = track.setAudioProcessingOptions(CallAudio.options(plan, automatic: true)) |
| + if !fallback.isSuccess { refused = fallback.message } |
| + } |
| } |
| - static func options(_ plan: ProcessingPlan) -> RTCAudioProcessingOptions { |
| + static func options(_ plan: ProcessingPlan, automatic: Bool = false) -> RTCAudioProcessingOptions { |
| func component(_ place: ProcessingPlan.Where) -> RTCAudioProcessingComponentOptions { |
| switch place { |
| case .off: return RTCAudioProcessingComponentOptions(enabled: false) |
| - case .software: return RTCAudioProcessingComponentOptions(enabled: true, mode: .software) |
| - case .platform: return RTCAudioProcessingComponentOptions(enabled: true, mode: .platform) |
| + case .software: |
| + return RTCAudioProcessingComponentOptions(enabled: true, mode: automatic ? .automatic : .software) |
| + case .platform: |
| + return RTCAudioProcessingComponentOptions(enabled: true, mode: automatic ? .automatic : .platform) |
| } |
| } |
| return RTCAudioProcessingOptions( |
diff --git a/native/apple/Sources/CallsKit/NoiseSuppression.swift b/native/apple/Sources/CallsKit/NoiseSuppression.swift| index 83935a1..8cdb00e 100644 |
| --- a/native/apple/Sources/CallsKit/NoiseSuppression.swift |
| +++ b/native/apple/Sources/CallsKit/NoiseSuppression.swift |
| @@ -3,6 +3,10 @@ import LiveKitWebRTC |
| import RNNoise |
| import os |
| +#if canImport(DeepFilter) |
| + import DeepFilter |
| +#endif |
| + |
| // Noise suppression for the microphone, run inside WebRTC's capture |
| // processing. |
| // |
| @@ -33,14 +37,25 @@ public protocol NoiseSuppressor: AnyObject { |
| func process(_ frame: UnsafeMutablePointer<Float>) -> Float |
| } |
| +/// How far either suppressor may take noise down, in dB. Unlimited, a |
| +/// room goes to digital silence between words, which a listener hears as |
| +/// the line dropping, and a model can hollow out a voice with the noise |
| +/// around it. 30 dB takes a room to near silence and leaves speech its own |
| +/// texture. |
| +public let noiseAttenuationLimitDb: Float = 30 |
| + |
| /// xiph's RNNoise: a small recurrent network, BSD-3, 480-sample frames at |
| /// 48 kHz, and — unusually — in WebRTC's own ±32768 scale already, so the |
| /// samples go in and come out untouched by any conversion. |
| +/// |
| +/// RNNoise has no attenuation limit of its own, so one is made the usual |
| +/// way: the untouched input is mixed back in at the limit. |
| public final class RNNoiseSuppressor: NoiseSuppressor { |
| public let name = "RNNoise" |
| public let frameLength: Int |
| private let state: OpaquePointer |
| private let scratch: UnsafeMutablePointer<Float> |
| + private let floor = pow(10, -noiseAttenuationLimitDb / 20) |
| public init() { |
| state = rnnoise_create(nil) |
| @@ -53,10 +68,62 @@ public final class RNNoiseSuppressor: NoiseSuppressor { |
| } |
| public func process(_ frame: UnsafeMutablePointer<Float>) -> Float { |
| scratch.update(from: frame, count: frameLength) |
| - return rnnoise_process_frame(state, frame, scratch) |
| + let speech = rnnoise_process_frame(state, frame, scratch) |
| + for i in 0..<frameLength { frame[i] += floor * scratch[i] } |
| + return speech |
| } |
| } |
| +#if canImport(DeepFilter) |
| + /// DeepFilterNet 3 (Rikorose/DeepFilterNet's libDF, MIT or Apache-2.0): |
| + /// a larger model than RNNoise, markedly better on non-stationary noise |
| + /// (typing, wind, voices in the next room). 480-sample hop at 48 kHz, |
| + /// mono, in ±1 floats — so samples are scaled by 1/32768 on the way in |
| + /// and back on the way out. It adds about 30 ms of latency (two frames |
| + /// of lookahead and the overlap of its transform). |
| + public final class DeepFilterSuppressor: NoiseSuppressor { |
| + public let name = "DeepFilterNet 3" |
| + public let frameLength: Int |
| + private let state: OpaquePointer |
| + private let input: UnsafeMutablePointer<Float> |
| + private let output: UnsafeMutablePointer<Float> |
| + |
| + /// The model, where the app carries one: `CALLS_DEEPFILTER_MODEL` |
| + /// first (tests and the interop harness), then the bundle. |
| + public static var model: URL? { |
| + if let path = ProcessInfo.processInfo.environment["CALLS_DEEPFILTER_MODEL"] { |
| + return URL(fileURLWithPath: path) |
| + } |
| + return Bundle.main.url(forResource: "DeepFilterNet3_onnx", withExtension: "tar.gz") |
| + } |
| + |
| + /// Nil when there is no model file. libDF aborts the process on a |
| + /// model it cannot read, so the file is checked here first. |
| + public init?(model: URL, attenuationLimit: Float = noiseAttenuationLimitDb) { |
| + guard FileManager.default.isReadableFile(atPath: model.path) else { return nil } |
| + guard let state = df_create(model.path, attenuationLimit, nil) else { return nil } |
| + self.state = state |
| + frameLength = Int(df_get_frame_length(state)) |
| + input = .allocate(capacity: frameLength) |
| + output = .allocate(capacity: frameLength) |
| + } |
| + deinit { |
| + df_free(state) |
| + input.deallocate() |
| + output.deallocate() |
| + } |
| + |
| + /// Returns the model's local SNR estimate mapped to 0...1: at or below |
| + /// 5 dB reads as noise, at or above 15 dB as speech. |
| + public func process(_ frame: UnsafeMutablePointer<Float>) -> Float { |
| + for i in 0..<frameLength { input[i] = frame[i] / 32768 } |
| + let snr = df_process_frame(state, input, output) |
| + for i in 0..<frameLength { frame[i] = output[i] * 32768 } |
| + return min(max((snr - 5) / 10, 0), 1) |
| + } |
| + } |
| +#endif |
| + |
| /// A gentle gain after the suppressor, instead of WebRTC's gain control. |
| /// |
| /// WebRTC's gain control runs *before* the capture post-processing hook, |
| @@ -106,7 +173,8 @@ public struct SpeechGain { |
| public enum SuppressionState: Equatable, Sendable { |
| /// Toggle off, or nothing has asked yet. |
| case off |
| - /// Processing frames; `cpu` is the fraction of one core it is using. |
| + /// Processing frames; `cpu` is the share of real time spent in it on |
| + /// the audio thread, 0 to 1 — wall clock, so preemption counts too. |
| case running(engine: String, cpu: Double) |
| /// Initialised and waiting — muted (WebRTC skips processing then), or |
| /// no audio has flowed yet. |
| @@ -145,7 +213,8 @@ public final class CaptureDenoiser: NSObject, RTCAudioCustomProcessingDelegate, |
| var cpu: Double = 0 |
| } |
| private let lock = OSAllocatedUnfairLock(initialState: State()) |
| - private let suppressor: NoiseSuppressor |
| + /// Read under the lock once per buffer, so it can be replaced mid-call. |
| + private var current: NoiseSuppressor |
| private var gain = SpeechGain() |
| private let clock: () -> Double |
| /// Called (on the audio thread) when WebRTC reinitialises at a rate the |
| @@ -153,10 +222,20 @@ public final class CaptureDenoiser: NSObject, RTCAudioCustomProcessingDelegate, |
| public var rateChanged: (@Sendable (Int) -> Void)? |
| public init(suppressor: NoiseSuppressor, clock: @escaping () -> Double = CaptureDenoiser.now) { |
| - self.suppressor = suppressor |
| + current = suppressor |
| self.clock = clock |
| } |
| + /// Swaps the model, between buffers. A call starts on RNNoise and moves |
| + /// to DeepFilterNet once its model has loaded off the main thread |
| + /// (a third of a second on an M1 Max; the audio thread is no place for |
| + /// it, and neither is joining). |
| + public func replace(with suppressor: NoiseSuppressor) { |
| + lock.withLock { _ in current = suppressor } |
| + } |
| + |
| + private var suppressor: NoiseSuppressor { lock.withLock { _ in current } } |
| + |
| public static func now() -> Double { ProcessInfo.processInfo.systemUptime } |
| public var engineName: String { suppressor.name } |
| @@ -179,8 +258,8 @@ public final class CaptureDenoiser: NSObject, RTCAudioCustomProcessingDelegate, |
| if !s.enabled { return .off } |
| if s.standAside { return .standingAside } |
| if s.initialised, s.rate != CaptureDenoiser.supportedRate { return .unsupportedRate(s.rate) } |
| - if s.initialised, now - s.lastFrame < 1 { return .running(engine: suppressor.name, cpu: s.cpu) } |
| - return .idle(engine: suppressor.name) |
| + if s.initialised, now - s.lastFrame < 1 { return .running(engine: current.name, cpu: s.cpu) } |
| + return .idle(engine: current.name) |
| } |
| } |
| @@ -211,9 +290,11 @@ public final class CaptureDenoiser: NSObject, RTCAudioCustomProcessingDelegate, |
| /// Processes one buffer in place. Public for the tests, which cannot |
| /// make an RTCAudioBuffer. |
| public func process(_ samples: UnsafeMutablePointer<Float>, frames: Int) { |
| - let run = lock.withLock { s in |
| - s.enabled && !s.standAside && s.initialised && s.channels == 1 |
| - && s.rate == CaptureDenoiser.supportedRate |
| + let (run, suppressor) = lock.withLock { s in |
| + ( |
| + s.enabled && !s.standAside && s.initialised && s.channels == 1 |
| + && s.rate == CaptureDenoiser.supportedRate, current |
| + ) |
| } |
| let length = suppressor.frameLength |
| guard run, frames > 0, frames % length == 0 else { return } |
diff --git a/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift b/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift| index ddaca5f..8389351 100644 |
| --- a/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift |
| +++ b/native/apple/Tests/CallsKitTests/NoiseSuppressionTests.swift |
| @@ -126,6 +126,20 @@ final class CaptureDenoiserTests: XCTestCase { |
| XCTAssertEqual(fake.calls, 0) |
| } |
| + /// A call starts on RNNoise and moves to DeepFilterNet once its model |
| + /// has loaded; the next buffer goes to the new model. |
| + func testTheModelCanBeReplacedBetweenBuffers() { |
| + let (denoiser, first, _) = adapter() |
| + denoiser.audioProcessingInitialize(sampleRate: 48_000, channels: 1) |
| + _ = run(denoiser, count: 480) |
| + let second = FakeSuppressor() |
| + denoiser.replace(with: second) |
| + _ = run(denoiser, count: 960) |
| + XCTAssertEqual(first.calls, 1) |
| + XCTAssertEqual(second.calls, 2) |
| + XCTAssertEqual(denoiser.framesProcessed, 3) |
| + } |
| + |
| /// WebRTC stops calling the hook while the microphone is muted. The |
| /// readout must then stop claiming the suppressor is running. |
| func testMutedReadsAsIdleOnceFramesStop() { |
| @@ -322,18 +336,25 @@ final class SuppressorFixtureTests: XCTestCase { |
| let lead = (rate * 3 / 10)..<rate |
| let speechRange = rate..<(frames * length - rate / 2) |
| // Per-frame: of the frames where the clean voice is clearly present, |
| - // how many came out within 10 dB of it. Compared a frame late too, |
| - // since a model may delay its output by a frame. |
| + // how many came out within 10 dB of it. A model may delay its output |
| + // (DeepFilterNet by three frames), so the best of 0 to 4 frames late |
| + // is taken. |
| var voiced = 0 |
| - var kept = 0 |
| - for f in stride(from: speechRange.lowerBound, to: speechRange.upperBound - 2 * length, by: length) { |
| - let reference = db(clean[f..<(f + length)]) |
| - guard reference > -40 else { continue } |
| - voiced += 1 |
| - let now = db(out[f..<(f + length)]) |
| - let late = db(out[(f + length)..<(f + 2 * length)]) |
| - if max(now, late) > reference - 10 { kept += 1 } |
| + var best = 0 |
| + for lag in 0...4 { |
| + var count = 0 |
| + var kept = 0 |
| + for f in stride(from: speechRange.lowerBound, to: speechRange.upperBound - 5 * length, by: length) { |
| + let reference = db(clean[f..<(f + length)]) |
| + guard reference > -40 else { continue } |
| + count += 1 |
| + let shifted = f + lag * length |
| + if db(out[shifted..<(shifted + length)]) > reference - 10 { kept += 1 } |
| + } |
| + voiced = count |
| + best = max(best, kept) |
| } |
| + let kept = best |
| return Result( |
| noiseBefore: db(noise[lead]), noiseAfter: db(out[lead]), |
| speechClean: db(clean[speechRange]), speechAfter: db(out[speechRange]), |
| @@ -358,5 +379,47 @@ final class SuppressorFixtureTests: XCTestCase { |
| func testRNNoiseQuietsNoiseAndKeepsSpeech() throws { |
| let result = try Self.measure(RNNoiseSuppressor()) |
| Self.check(result, name: "RNNoise", minimumReduction: 10) |
| + XCTAssertGreaterThan(result.noiseAfter, result.noiseBefore - 40, "the 30 dB limit holds") |
| } |
| + |
| + #if canImport(DeepFilter) |
| + /// The model is a build product (scripts/build-deepfilter.sh), found |
| + /// beside the xcframework. |
| + static var deepFilterModel: URL { |
| + URL(fileURLWithPath: #filePath) |
| + .deletingLastPathComponent().deletingLastPathComponent().deletingLastPathComponent() |
| + .deletingLastPathComponent().appendingPathComponent("build/DeepFilterNet3_onnx.tar.gz") |
| + } |
| + |
| + func testDeepFilterQuietsNoiseAndKeepsSpeech() throws { |
| + let started = ProcessInfo.processInfo.systemUptime |
| + let deep = try XCTUnwrap(DeepFilterSuppressor(model: Self.deepFilterModel), "run build-deepfilter.sh") |
| + print("DeepFilterNet 3 model loaded in \(Int((ProcessInfo.processInfo.systemUptime - started) * 1000)) ms") |
| + XCTAssertEqual(deep.frameLength, 480) |
| + let result = try Self.measure(deep) |
| + Self.check(result, name: "DeepFilterNet 3", minimumReduction: 10) |
| + XCTAssertGreaterThan(result.noiseAfter, result.noiseBefore - 40, "the 30 dB limit holds") |
| + } |
| + |
| + /// libDF wants ±1; WebRTC hands over ±32768. A full-scale voice must |
| + /// come back at full scale, not 32768 times louder or quieter. |
| + func testDeepFilterScalesInAndOut() throws { |
| + let deep = try XCTUnwrap(DeepFilterSuppressor(model: Self.deepFilterModel)) |
| + let clean = try Self.speech() |
| + var loudest: Float = 0 |
| + var inputLoudest: Float = 0 |
| + var samples = Array(clean.prefix(480 * 300)) |
| + inputLoudest = samples.map(abs).max()! |
| + samples.withUnsafeMutableBufferPointer { buffer in |
| + for f in 0..<300 { _ = deep.process(buffer.baseAddress! + f * 480) } |
| + } |
| + loudest = samples.map(abs).max()! |
| + XCTAssertGreaterThan(loudest, inputLoudest / 4, "output is in WebRTC's scale") |
| + XCTAssertLessThan(loudest, inputLoudest * 4, "output is in WebRTC's scale") |
| + } |
| + |
| + func testAMissingModelIsNoSuppressorRatherThanACrash() { |
| + XCTAssertNil(DeepFilterSuppressor(model: URL(fileURLWithPath: "/nonexistent/model.tar.gz"))) |
| + } |
| + #endif |
| } |
diff --git a/native/scripts/build-deepfilter.sh b/native/scripts/build-deepfilter.sh| new file mode 100755 |
| index 0000000..59e190e |
| --- /dev/null |
| +++ b/native/scripts/build-deepfilter.sh |
| @@ -0,0 +1,71 @@ |
| +#!/bin/sh |
| +# Builds native/build/DeepFilter.xcframework and copies the DeepFilterNet 3 |
| +# model beside it: Rikorose/DeepFilterNet's libDF (MIT or Apache-2.0) with |
| +# its C API, as a static library for Mac, iPhone and the iPhone simulator |
| +# (Apple silicon). |
| +# |
| +# Built rather than committed, as CallsBridge and RNNoise are. Needs rustup |
| +# (https://rustup.rs); the toolchain, its targets and cbindgen are pinned |
| +# and installed on first run. |
| +set -eu |
| +cd "$(dirname "$0")/.." |
| + |
| +commit=d375b2d8309e0935d165700c91da9de862a99c31 |
| +toolchain=1.92.0 |
| +cbindgen_version=0.29.2 |
| +out=build/DeepFilter.xcframework |
| +model=build/DeepFilterNet3_onnx.tar.gz |
| +stamp="$out/.built-$commit-$toolchain" |
| +if [ -f "$stamp" ] && [ -f "$model" ]; then exit 0; fi |
| + |
| +PATH="$HOME/.cargo/bin:$PATH" |
| +command -v rustup >/dev/null || { |
| + echo "build-deepfilter: rustup is required (https://rustup.rs)" >&2 |
| + exit 1 |
| +} |
| +targets="aarch64-apple-darwin aarch64-apple-ios aarch64-apple-ios-sim" |
| +# shellcheck disable=SC2086 |
| +rustup toolchain install "$toolchain" --profile minimal $(printf ' -t %s' $targets) >/dev/null |
| +if [ "$(cbindgen --version 2>/dev/null)" != "cbindgen $cbindgen_version" ]; then |
| + cargo "+$toolchain" install cbindgen --version "$cbindgen_version" --locked |
| +fi |
| + |
| +work=build/deepfilter |
| +mkdir -p "$work" |
| +if [ ! -d "$work/src/.git" ]; then |
| + rm -rf "$work/src" |
| + git clone --quiet https://github.com/Rikorose/DeepFilterNet.git "$work/src" |
| +fi |
| +git -C "$work/src" -c advice.detachedHead=false checkout --quiet --force "$commit" |
| + |
| +# Deployment targets matching the apps, so the linker does not warn that |
| +# the library was built for newer systems than they run on. |
| +export MACOSX_DEPLOYMENT_TARGET=14.0 IPHONEOS_DEPLOYMENT_TARGET=17.0 |
| +for target in $targets; do |
| + cargo "+$toolchain" build --manifest-path "$work/src/Cargo.toml" -p deep_filter --lib \ |
| + --release --features capi --target "$target" --target-dir "$work/target" |
| +done |
| + |
| +headers="$work/headers" |
| +rm -rf "$headers" |
| +mkdir -p "$headers/DeepFilter" |
| +cbindgen --config "$work/src/cbindgen.toml" --crate deep_filter \ |
| + --output "$headers/DeepFilter/deep_filter.h" "$work/src/libDF" |
| +cat >"$headers/DeepFilter/module.modulemap" <<'EOF' |
| +module DeepFilter { |
| + header "deep_filter.h" |
| + link "df" |
| + export * |
| +} |
| +EOF |
| + |
| +rm -rf "$out" |
| +xcodebuild -create-xcframework \ |
| + -library "$work/target/aarch64-apple-darwin/release/libdf.a" -headers "$headers" \ |
| + -library "$work/target/aarch64-apple-ios/release/libdf.a" -headers "$headers" \ |
| + -library "$work/target/aarch64-apple-ios-sim/release/libdf.a" -headers "$headers" \ |
| + -output "$out" >/dev/null |
| +cp "$work/src/models/DeepFilterNet3_onnx.tar.gz" "$model" |
| +cp "$work/src/LICENSE-MIT" "$out/LICENSE-MIT" |
| +touch "$stamp" |
| +printf 'Built %s/%s\n' "$PWD" "$out" |
diff --git a/native/scripts/build-iphone.sh b/native/scripts/build-iphone.sh| index b496d4f..3910137 100755 |
| --- a/native/scripts/build-iphone.sh |
| +++ b/native/scripts/build-iphone.sh |
| @@ -3,6 +3,7 @@ set -eu |
| cd "$(dirname "$0")/.." |
| ./scripts/build-bridge.sh |
| ./scripts/build-rnnoise.sh |
| +./scripts/build-deepfilter.sh |
| xcodegen generate --spec ios/project.yml |
| case "${1:-simulator}" in |
| simulator) |
diff --git a/native/scripts/build-mac.sh b/native/scripts/build-mac.sh| index b100234..1a8b436 100755 |
| --- a/native/scripts/build-mac.sh |
| +++ b/native/scripts/build-mac.sh |
| @@ -1,7 +1,7 @@ |
| #!/bin/sh |
| set -eu |
| cd "$(dirname "$0")/.." |
| -if [ "${OFICINA_SKIP_PREPARE:-0}" != 1 ]; then ./scripts/build-bridge.sh; ./scripts/build-rnnoise.sh; fi |
| +if [ "${OFICINA_SKIP_PREPARE:-0}" != 1 ]; then ./scripts/build-bridge.sh; ./scripts/build-rnnoise.sh; ./scripts/build-deepfilter.sh; fi |
| configuration="${OFICINA_BUILD_CONFIGURATION:-debug}" |
| swift build --package-path apple --configuration "$configuration" --product CallsMac |
| python3 scripts/bundle-mac.py "$configuration" |
diff --git a/native/scripts/bundle-mac.py b/native/scripts/bundle-mac.py| index 9096c9a..8b7a1c5 100644 |
| --- a/native/scripts/bundle-mac.py |
| +++ b/native/scripts/bundle-mac.py |
| @@ -20,6 +20,10 @@ icon = root / 'apple/Sources/CallsMac/Resources/MacIcon.png' |
| shutil.copy2(icon, bundle / 'Resources/MacIcon.png') |
| shutil.copy2(root / 'LICENSE.lucide', bundle / 'Resources/LICENSE.lucide') |
| shutil.copy2(root / 'LICENSE.rnnoise', bundle / 'Resources/LICENSE.rnnoise') |
| +shutil.copy2(root / 'LICENSE.deepfilternet', bundle / 'Resources/LICENSE.deepfilternet') |
| +# The Mac's noise suppressor model (scripts/build-deepfilter.sh). Without |
| +# it the app falls back to RNNoise. |
| +shutil.copy2(root / 'build/DeepFilterNet3_onnx.tar.gz', bundle / 'Resources/DeepFilterNet3_onnx.tar.gz') |
| # Finder reads the ICNS before the app launches; the PNG also supplies the |
| # runtime Dock icon. Generate each required representation from the master. |
| iconset = root / 'build/AppIcon.iconset' |
diff --git a/test/browser/native.mjs b/test/browser/native.mjs| index 946437a..cee1b62 100644 |
| --- a/test/browser/native.mjs |
| +++ b/test/browser/native.mjs |
| @@ -76,7 +76,7 @@ try { |
| // The synthetic microphone runs through WebRTC's capture processing, so |
| // the noise suppressor in its post-processing hook must have seen frames. |
| await wait(() => sample?.stats.noiseFramesProcessed > 100, 'native noise suppressor processes captured audio'); |
| - console.log('PASS native noise suppressor ran on captured audio'); |
| + console.log(`PASS native noise suppressor ran on captured audio (${sample.stats.noiseSuppressor === 2 ? 'DeepFilterNet 3' : 'RNNoise'})`); |
| // And it runs instead of WebRTC's own suppressor and gain, beside its echo canceller. |
| assert.equal(sample.stats.webrtcNoiseSuppression, 0, "WebRTC's noise suppressor is off"); |
| assert.equal(sample.stats.webrtcGainControl, 0, "WebRTC's gain control is off"); |