amadan

oficina / meet Public

Share the iPhone's screen through a broadcast extension

iOS lets an app share the whole screen and nothing smaller, and only
through ReplayKit's broadcast: the frames go to an upload extension, a
separate process capped at about fifty megabytes, which cannot hold the
call's peer connections and could not afford WebRTC if it did. So the
extension does the least it can. It scales each frame to 1280 on the
long side, at up to fifteen a second, straight from ReplayKit's buffer
into one reused buffer of its own, and hands it to the app over a Unix
socket in the App Group container. The app is already in the call and
sends it on the same sendonly screen transceiver the Mac and the
browser use, so nobody receiving it can tell which of the three sent it.

At most one frame is in flight. One that arrives while the last is still
being written is dropped, not queued, because a queue is how an
extension runs out of memory while the app is busy. The extension links
CallsBroadcastKit and nothing else of Calls, and comes to about 600 KB.

The order is the Mac's reversed. There the button asks and the picker
answers. Here the person starts a broadcast at the system's sheet, the
extension connects, and that connection is the share starting. The app
listens for as long as a call is on screen. A broadcast the call cannot
take (not admitted yet, or already sharing) is closed at once, and the
extension ends it with a sentence of its own, which is the only way an
extension can say anything. Stopping works from both ends. Stop in the
app closes the socket, and the extension's next write fails and ends the
broadcast. Stopping from Control Center closes it from the other side,
and the call stops the share.

The button is RPSystemBroadcastPickerView itself, because only a tap on
its own button starts a broadcast. It is dressed in the call's symbol
and tint and offers only Calls' extension.

A call that is sharing now survives the app going to the background.
Showing other apps is the point of sharing the whole screen, and that
takes the audio background mode. Every other call still ends there, as
before, and a broadcast stopped from outside while Calls is backgrounded
ends the call too.

The App Group is one build setting, CALLS_APP_GROUP
(group.net.amadan.oficina.calls). It feeds both targets' entitlements
and a CallsAppGroup key in both Info.plists, so a build under another
identifier changes it in one place. The extension is
net.amadan.oficina.calls.ios.broadcast, the app's identifier plus
.broadcast, which is how the app finds it.

`fitted`, which sizes frames on both platforms, moved into
CallsBroadcastKit so the extension can use it without CallsKit.

Tests: the wire header round-trips and refuses odd, oversized and
inconsistent frames. A real 1280x720 NV12 frame crosses a real socket
and arrives at 640x360 with its luma and interleaved chroma intact. The
extension going away ends the connection, the app closing its end
reaches the extension, a second broadcast is refused, and ReplayKit's
orientations map to upright rotations.

Gates: swift test (47), make ci (GOTOOLCHAIN=go1.26.5), make
native-test, make native-mac, make iphone (the simulator build, with the
extension embedded, 600 KB, no WebRTC linked), an unsigned generic iOS
device build, make native-browser, make browser. make native-pair passed
2 of 3; the third was the app-to-app connection failure described in the
previous commit, before any screen was shared.

Not done here: signing. The simulator build is unsigned, so it has no
App Group and cannot broadcast, and ReplayKit broadcasts need a device.
The portal steps are in native/ios/README.md. Nothing has run on a
device: the extension's memory under a real broadcast, a real socket
path in a real group container, rotation, and the background audio
session holding the call up are all unverified.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Paul Campbell pushed by paul@keymail.dev e1ff45e974b3cdbc3b91c9ea18308e30db782889 parent b2db264
16 files changed, +1122 −20
  • native/DESIGN.md +2 −2
  • native/README.md +9 −2
  • native/apple/Package.swift +9 −2
  • native/apple/Sources/CallsBroadcastKit/BroadcastInbox.swift +215 −0
  • native/apple/Sources/CallsBroadcastKit/BroadcastSender.swift +170 −0
  • native/apple/Sources/CallsBroadcastKit/BroadcastWire.swift +193 −0
  • native/apple/Sources/CallsKit/BroadcastScreenCapture.swift +70 −0
  • native/apple/Sources/CallsKit/MacScreenCapture.swift +1 −0
  • native/apple/Sources/CallsKit/ScreenShare.swift +0 −11
  • native/apple/Tests/CallsKitTests/BroadcastTests.swift +183 −0
  • native/apple/Tests/CallsKitTests/ScreenShareTests.swift +1 −0
  • native/ios/Broadcast/SampleHandler.swift +61 −0
  • native/ios/README.md +57 −2
  • native/ios/Sources/CallsPhone.swift +24 −1
  • native/ios/Sources/PhoneScreenShare.swift +88 −0
  • native/ios/project.yml +39 −0
diff --git a/native/DESIGN.md b/native/DESIGN.md
index 55ffc98..4b3ea5d 100644
--- a/native/DESIGN.md
+++ b/native/DESIGN.md
@@ -61,8 +61,8 @@ retention checks. Window closure never invokes the meeting leave action.
The iOS call stage now defaults to remote focus with a local thumbnail, offers
Focus/Grid and participant selection, and uses the same compact colored icon
control vocabulary. Chat is a native sheet with unread count and a retained
-draft. iOS continues to end calls when backgrounded; native screen sharing and
-background-call continuity remain follow-up media work.
+draft. iOS continues to end calls when backgrounded, except while sharing the screen;
+general background-call continuity remains follow-up media work.
Safari member/guest sign-in and the real iPhone simulator native-media tests
passed after this change. The signed development app was built and installed on
diff --git a/native/README.md b/native/README.md
index dbe913e..60b3969 100644
--- a/native/README.md
+++ b/native/README.md
@@ -94,6 +94,11 @@ bar's stop control and a shared window closing end it the same way as the
button. The source is a screencast, so WebRTC gives up frame rate before
resolution, and frames are capped at 1920 on the long side and 30 a second.
+On **iPhone** the whole screen is shared through a ReplayKit broadcast
+extension, which hands its frames to the app over a socket in an App Group
+container. See [ios/README.md](ios/README.md#sharing-the-screen) for how that
+works and what signing it needs.
+
**Knocks** are announced when someone asks to be let in while the app is
not frontmost. The admission queue is already on screen for a member who is
looking at the call; this is for the member who is in another Space,
@@ -200,8 +205,10 @@ and keeps its history visible during signaling reconnects.
There is no claim that a simulator or synthetic test proves physical camera,
microphone permissions, audio-device switching, sleep/wake or Home passkeys.
-Those need a real-device acceptance pass. Native screen sharing, background
-calling and long-call TURN renewal parity are follow-up work.
+Those need a real-device acceptance pass. So does screen sharing: the Mac's
+system picker and the iPhone's broadcast extension cannot be driven by any gate
+here. Background calling (other than while sharing the iPhone's screen) and
+long-call TURN renewal parity are follow-up work.
See [VERIFICATION.md](VERIFICATION.md) for the completed gate record and acceptance limits.
diff --git a/native/apple/Package.swift b/native/apple/Package.swift
index 522f2fd..174463b 100644
--- a/native/apple/Package.swift
+++ b/native/apple/Package.swift
@@ -10,6 +10,9 @@ let package = Package(
platforms: [.macOS(.v14), .iOS(.v17)],
products: [
.library(name: "CallsKit", targets: ["CallsKit"]),
+ // What the iPhone's broadcast extension links, and all it links: no
+ // WebRTC, so the extension stays far inside its memory ceiling.
+ .library(name: "CallsBroadcastKit", targets: ["CallsBroadcastKit"]),
.executable(name: "CallsMac", targets: ["CallsMac"]),
.executable(name: "CallsInterop", targets: ["CallsInterop"]),
],
@@ -19,10 +22,14 @@ let package = Package(
],
targets: [
.binaryTarget(name: "CallsBridge", path: "../build/CallsBridge.xcframework"),
+ .target(name: "CallsBroadcastKit"),
.target(
- name: "CallsKit", dependencies: ["CallsBridge", .product(name: "WebRTC", package: "WebRTC")]),
+ name: "CallsKit",
+ dependencies: [
+ "CallsBridge", "CallsBroadcastKit", .product(name: "WebRTC", package: "WebRTC"),
+ ]),
.executableTarget(name: "CallsMac", dependencies: [.product(name: "RastrilloMacUpdates", package: "native", condition: .when(platforms: [.macOS])), "CallsKit"], resources: [.copy("Resources/AppIcon.png"), .copy("Resources/MacIcon.png")]),
.executableTarget(name: "CallsInterop", dependencies: ["CallsKit"]),
- .testTarget(name: "CallsKitTests", dependencies: ["CallsKit"]),
+ .testTarget(name: "CallsKitTests", dependencies: ["CallsKit", "CallsBroadcastKit"]),
]
)
diff --git a/native/apple/Sources/CallsBroadcastKit/BroadcastInbox.swift b/native/apple/Sources/CallsBroadcastKit/BroadcastInbox.swift
new file mode 100644
index 0000000..76c4bc9
--- /dev/null
+++ b/native/apple/Sources/CallsBroadcastKit/BroadcastInbox.swift
@@ -0,0 +1,215 @@
+import CoreVideo
+import Foundation
+
+/// The app's half: listens for the extension for as long as a call is up,
+/// and hands each connection to the call.
+///
+/// One broadcast at a time is all iOS allows, so one connection at a time
+/// is all this takes; a second, which would be a bug, is closed at once.
+public final class BroadcastInbox: @unchecked Sendable {
+ private let path: String
+ private let queue = DispatchQueue(label: "calls.broadcast.inbox")
+ private var listener: DispatchSourceRead?
+ private var current: BroadcastConnection?
+
+ /// Called on the main queue for each extension that connects.
+ public var onConnection: ((BroadcastConnection) -> Void)?
+
+ public init(path: String) { self.path = path }
+
+ public func start() throws {
+ let fd = try BroadcastSocket.listen(path: path)
+ BroadcastSocket.nonBlocking(fd)
+ let source = DispatchSource.makeReadSource(fileDescriptor: fd, queue: queue)
+ source.setEventHandler { [weak self] in self?.accept(fd) }
+ source.setCancelHandler { Darwin.close(fd) }
+ listener = source
+ source.resume()
+ }
+
+ /// Stops listening. A connection already handed out is the call's to
+ /// close, not this.
+ public func stop() {
+ queue.sync {
+ listener?.cancel()
+ listener = nil
+ current = nil
+ }
+ unlink(path)
+ }
+
+ private func accept(_ fd: Int32) {
+ let client = Darwin.accept(fd, nil, nil)
+ guard client >= 0 else { return }
+ if let current, !current.isClosed {
+ Darwin.close(client)
+ return
+ }
+ var on: Int32 = 1
+ setsockopt(client, SOL_SOCKET, SO_NOSIGPIPE, &on, socklen_t(MemoryLayout<Int32>.size))
+ let connection = BroadcastConnection(fd: client)
+ current = connection
+ DispatchQueue.main.async { [weak self] in
+ guard let self, let onConnection = self.onConnection else {
+ connection.close()
+ return
+ }
+ onConnection(connection)
+ }
+ }
+}
+
+/// One broadcast: frames in, until either side closes.
+public final class BroadcastConnection: @unchecked Sendable {
+ private let fd: Int32
+ private let queue = DispatchQueue(label: "calls.broadcast.read", qos: .userInteractive)
+ private var source: DispatchSourceRead?
+ private let lock = NSLock()
+ private var closed = false
+ private var header = [UInt8](repeating: 0, count: BroadcastFrameHeader.size)
+ private var headerRead = 0
+ private var frame: BroadcastFrameHeader?
+ private var payload: UnsafeMutableRawPointer?
+ private var payloadCapacity = 0
+ private var payloadRead = 0
+ private var pool: CVPixelBufferPool?
+ private var poolShape: (Int, Int, Bool) = (0, 0, false)
+
+ /// Each complete frame, as an NV12 pixel buffer the receiver may keep,
+ /// with its CGImagePropertyOrientation. Called on the reading queue.
+ public var onFrame: ((CVPixelBuffer, UInt16) -> Void)?
+ /// Called once, on the reading queue, when the extension has gone — the
+ /// broadcast stopped from Control Center or the status bar, or the
+ /// extension ended. Not called for `close()`.
+ public var onClose: (() -> Void)?
+
+ init(fd: Int32) { self.fd = fd }
+
+ deinit { payload?.deallocate() }
+
+ var isClosed: Bool {
+ lock.lock()
+ defer { lock.unlock() }
+ return closed
+ }
+
+ /// Starts reading. Separate from creation so the reader is only live
+ /// once somebody is listening for its frames.
+ public func resume() {
+ BroadcastSocket.nonBlocking(fd)
+ let source = DispatchSource.makeReadSource(fileDescriptor: fd, queue: queue)
+ source.setEventHandler { [weak self] in self?.readable() }
+ source.setCancelHandler { [fd] in Darwin.close(fd) }
+ self.source = source
+ source.resume()
+ }
+
+ /// Closes from this side. The extension's next write fails, and it ends
+ /// the broadcast — which is how "stop sharing" in the app reaches the
+ /// system's recording indicator.
+ public func close() {
+ lock.lock()
+ let first = !closed
+ closed = true
+ lock.unlock()
+ guard first else { return }
+ if let source {
+ source.cancel()
+ } else {
+ Darwin.close(fd)
+ }
+ }
+
+ private func ended() {
+ lock.lock()
+ let first = !closed
+ closed = true
+ lock.unlock()
+ guard first else { return }
+ source?.cancel()
+ onClose?()
+ }
+
+ private func readable() {
+ while !isClosed {
+ if frame == nil {
+ let n = header.withUnsafeMutableBytes {
+ Darwin.read(fd, $0.baseAddress!.advanced(by: headerRead), BroadcastFrameHeader.size - headerRead)
+ }
+ guard let n = settle(n) else { return }
+ headerRead += n
+ guard headerRead == BroadcastFrameHeader.size else { continue }
+ guard let next = header.withUnsafeBytes({ BroadcastFrameHeader.read($0.baseAddress!) }) else {
+ ended()
+ return
+ }
+ frame = next
+ payloadRead = 0
+ if payloadCapacity < next.payloadLength {
+ payload?.deallocate()
+ payload = UnsafeMutableRawPointer.allocate(byteCount: next.payloadLength, alignment: 16)
+ payloadCapacity = next.payloadLength
+ }
+ }
+ guard let frame, let payload else { return }
+ let n = Darwin.read(fd, payload.advanced(by: payloadRead), frame.payloadLength - payloadRead)
+ guard let n = settle(n) else { return }
+ payloadRead += n
+ guard payloadRead == frame.payloadLength else { continue }
+ self.frame = nil
+ headerRead = 0
+ if let pixels = picture(frame, payload) { onFrame?(pixels, frame.orientation) }
+ }
+ }
+
+ /// A read's answer: bytes to count, nil to stop reading for now, and the
+ /// connection ended on end-of-file or a real error.
+ private func settle(_ n: Int) -> Int? {
+ if n > 0 { return n }
+ if n < 0 && (errno == EAGAIN || errno == EINTR) { return nil }
+ ended()
+ return nil
+ }
+
+ /// The packed frame, copied into a pixel buffer from a pool, row by row:
+ /// the pool's rows are padded and the wire's are not.
+ private func picture(_ frame: BroadcastFrameHeader, _ bytes: UnsafeMutableRawPointer)
+ -> CVPixelBuffer?
+ {
+ let shape = (frame.width, frame.height, frame.videoRange)
+ if pool == nil || poolShape != shape {
+ let format =
+ frame.videoRange
+ ? kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
+ : kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
+ let attributes: [CFString: Any] = [
+ kCVPixelBufferPixelFormatTypeKey: format,
+ kCVPixelBufferWidthKey: frame.width,
+ kCVPixelBufferHeightKey: frame.height,
+ kCVPixelBufferIOSurfacePropertiesKey: [:] as [CFString: Any],
+ ]
+ pool = nil
+ CVPixelBufferPoolCreate(nil, nil, attributes as CFDictionary, &pool)
+ poolShape = shape
+ }
+ guard let pool else { return nil }
+ var made: CVPixelBuffer?
+ CVPixelBufferPoolCreatePixelBuffer(nil, pool, &made)
+ guard let pixels = made else { return nil }
+ CVPixelBufferLockBaseAddress(pixels, [])
+ defer { CVPixelBufferUnlockBaseAddress(pixels, []) }
+ let planes = [
+ (bytes, frame.height),
+ (bytes.advanced(by: frame.width * frame.height), frame.height / 2),
+ ]
+ for (plane, (source, rows)) in planes.enumerated() {
+ guard let target = CVPixelBufferGetBaseAddressOfPlane(pixels, plane) else { return nil }
+ let stride = CVPixelBufferGetBytesPerRowOfPlane(pixels, plane)
+ for row in 0..<rows {
+ target.advanced(by: row * stride).copyMemory(
+ from: source.advanced(by: row * frame.width), byteCount: frame.width)
+ }
+ }
+ return pixels
+ }
+}
diff --git a/native/apple/Sources/CallsBroadcastKit/BroadcastSender.swift b/native/apple/Sources/CallsBroadcastKit/BroadcastSender.swift
new file mode 100644
index 0000000..9019005
--- /dev/null
+++ b/native/apple/Sources/CallsBroadcastKit/BroadcastSender.swift
@@ -0,0 +1,170 @@
+import Accelerate
+import CoreVideo
+import Foundation
+
+/// The extension's half: connects to the app, and sends it each frame it
+/// is given, smaller.
+///
+/// Everything here is shaped by the extension's memory ceiling. One frame
+/// buffer, allocated once and reused; the frame is scaled straight into
+/// it from ReplayKit's own pixel buffer, which is released as soon as
+/// `send` returns; and at most one frame is ever in flight. A frame that
+/// arrives while the last is still being written, or sooner than the rate
+/// allows, is dropped rather than queued — a queue is exactly how an
+/// extension runs out of memory while the app is busy.
+public final class BroadcastSender: @unchecked Sendable {
+ private let path: String
+ private let longest: CGFloat
+ private let interval: UInt64
+ private let queue = DispatchQueue(label: "calls.broadcast.send")
+ private let lock = NSLock()
+ // Guarded by lock.
+ private var fd: Int32 = -1
+ private var busy = false
+ private var last: UInt64 = 0
+ private var finished = false
+ // Only touched on queue, or before connect.
+ private var buffer: UnsafeMutableRawPointer?
+ private var capacity = 0
+
+ /// Called once, on the sending queue, when the app has gone: it closed
+ /// the socket because the person stopped sharing there, or the call
+ /// ended. Not called for `close()`.
+ public var onClosed: (@Sendable () -> Void)?
+
+ /// `longest` is the long side frames are sent at, at most; 1280 keeps a
+ /// phone's text legible on a laptop and the frame near a megabyte.
+ public init(path: String, longest: Int = 1280, framesPerSecond: Int = 15) {
+ self.path = path
+ self.longest = CGFloat(longest)
+ interval = 1_000_000_000 / UInt64(max(1, framesPerSecond))
+ }
+
+ deinit { buffer?.deallocate() }
+
+ /// Connects to the app, trying for up to `timeout`: the app may still be
+ /// settling into the call as the broadcast's countdown ends.
+ public func connect(timeout: TimeInterval) -> Bool {
+ let until = Date().addingTimeInterval(timeout)
+ repeat {
+ if let fd = try? BroadcastSocket.connect(path: path) {
+ lock.lock()
+ self.fd = fd
+ lock.unlock()
+ return true
+ }
+ Thread.sleep(forTimeInterval: 0.1)
+ } while Date() < until
+ return false
+ }
+
+ /// Offers one frame. Returns at once; the write happens on the sending
+ /// queue. `pixels` must be 4:2:0 bi-planar, which is what ReplayKit
+ /// delivers; anything else is dropped.
+ public func send(_ pixels: CVPixelBuffer, orientation: UInt16, timestampNs: UInt64) {
+ let format = CVPixelBufferGetPixelFormatType(pixels)
+ guard format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
+ || format == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange
+ else { return }
+ lock.lock()
+ let now = DispatchTime.now().uptimeNanoseconds
+ guard fd >= 0, !busy, !finished, now >= last + interval else {
+ lock.unlock()
+ return
+ }
+ busy = true
+ last = now
+ lock.unlock()
+ // Scaled here, on ReplayKit's thread, so the pixel buffer is not held
+ // past this call; only the copy crosses to the sending queue.
+ guard let length = pack(pixels, orientation: orientation, timestampNs: timestampNs) else {
+ lock.lock()
+ busy = false
+ lock.unlock()
+ return
+ }
+ queue.async { [self] in
+ let ok = writeAll(length)
+ lock.lock()
+ busy = false
+ let first = !ok && !finished
+ if !ok { finished = true }
+ lock.unlock()
+ if first { onClosed?() }
+ }
+ }
+
+ public func close() {
+ lock.lock()
+ finished = true
+ let fd = self.fd
+ self.fd = -1
+ lock.unlock()
+ if fd >= 0 { Darwin.close(fd) }
+ }
+
+ /// Scales `pixels` into the frame buffer behind a header, and answers the
+ /// number of bytes to write.
+ private func pack(_ pixels: CVPixelBuffer, orientation: UInt16, timestampNs: UInt64) -> Int? {
+ let size = fitted(
+ width: CGFloat(CVPixelBufferGetWidth(pixels)), height: CGFloat(CVPixelBufferGetHeight(pixels)),
+ longest: longest)
+ let header = BroadcastFrameHeader(
+ width: size.width, height: size.height, orientation: orientation, timestampNs: timestampNs,
+ videoRange: CVPixelBufferGetPixelFormatType(pixels)
+ == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange)
+ let length = BroadcastFrameHeader.size + header.payloadLength
+ if capacity < length {
+ buffer?.deallocate()
+ buffer = UnsafeMutableRawPointer.allocate(byteCount: length, alignment: 16)
+ capacity = length
+ }
+ guard let buffer else { return nil }
+ header.write(to: buffer)
+ CVPixelBufferLockBaseAddress(pixels, .readOnly)
+ defer { CVPixelBufferUnlockBaseAddress(pixels, .readOnly) }
+ guard let luma = CVPixelBufferGetBaseAddressOfPlane(pixels, 0),
+ let chroma = CVPixelBufferGetBaseAddressOfPlane(pixels, 1)
+ else { return nil }
+ var sourceLuma = vImage_Buffer(
+ data: luma, height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixels, 0)),
+ width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixels, 0)),
+ rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixels, 0))
+ var sourceChroma = vImage_Buffer(
+ data: chroma, height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixels, 1)),
+ width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixels, 1)),
+ rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixels, 1))
+ let picture = buffer.advanced(by: BroadcastFrameHeader.size)
+ var targetLuma = vImage_Buffer(
+ data: picture, height: vImagePixelCount(size.height), width: vImagePixelCount(size.width),
+ rowBytes: size.width)
+ var targetChroma = vImage_Buffer(
+ data: picture.advanced(by: size.width * size.height),
+ height: vImagePixelCount(size.height / 2), width: vImagePixelCount(size.width / 2),
+ rowBytes: size.width)
+ let flags = vImage_Flags(kvImageNoFlags)
+ guard vImageScale_Planar8(&sourceLuma, &targetLuma, nil, flags) == kvImageNoError,
+ vImageScale_CbCr8(&sourceChroma, &targetChroma, nil, flags) == kvImageNoError
+ else { return nil }
+ return length
+ }
+
+ private func writeAll(_ length: Int) -> Bool {
+ lock.lock()
+ let fd = self.fd
+ lock.unlock()
+ guard fd >= 0, let buffer else { return false }
+ var sent = 0
+ while sent < length {
+ let n = Darwin.write(fd, buffer.advanced(by: sent), length - sent)
+ if n > 0 {
+ sent += n
+ } else if n < 0 && errno == EINTR {
+ continue
+ } else {
+ return false
+ }
+ }
+ return true
+ }
+}
diff --git a/native/apple/Sources/CallsBroadcastKit/BroadcastWire.swift b/native/apple/Sources/CallsBroadcastKit/BroadcastWire.swift
new file mode 100644
index 0000000..4a4b837
--- /dev/null
+++ b/native/apple/Sources/CallsBroadcastKit/BroadcastWire.swift
@@ -0,0 +1,193 @@
+import CoreGraphics
+import Foundation
+
+// ---------------------------------------------------------------
+// The iPhone's screen, from the broadcast extension to the call
+// ---------------------------------------------------------------
+//
+// Sharing the whole iPhone screen is a ReplayKit broadcast, and ReplayKit
+// hands the frames to an extension — a separate process, capped at about
+// fifty megabytes, that cannot hold the call's peer connections and could
+// not afford WebRTC if it did. So the extension does the least it can:
+// shrinks each frame and passes it to the app, which is already in the
+// call, over a Unix socket in the App Group container both of them can
+// reach. The app listens for as long as a call is up; the extension
+// connecting *is* the broadcast starting, and either side closing is it
+// ending.
+//
+// This module is everything the two sides share, and nothing else: no
+// WebRTC, no UIKit, no ReplayKit. The extension links this and only this.
+//
+// One frame on the wire is a 32-byte header and then the picture, NV12
+// packed tight: the luma plane, `width` bytes a row, then the interleaved
+// chroma plane at half height, also `width` bytes a row. Little-endian.
+//
+// 0 magic u32 "CSF1"
+// 4 version u16 1
+// 6 orientation u16 CGImagePropertyOrientation, as ReplayKit says it
+// 8 width u32 even, 2...4096
+// 12 height u32 even, 2...4096
+// 16 timestamp u64 nanoseconds, the extension's clock
+// 24 length u32 width * height * 3 / 2
+// 28 flags u32 bit 0: video range (16-235) rather than full
+
+public struct BroadcastFrameHeader: Equatable, Sendable {
+ public static let size = 32
+ static let magic: UInt32 = 0x3146_5343 // "CSF1" read little-endian
+ static let version: UInt16 = 1
+ static let largest = 4096
+
+ public var width: Int
+ public var height: Int
+ public var orientation: UInt16
+ public var timestampNs: UInt64
+ /// Whether the luma runs 16-235 rather than 0-255. ReplayKit has sent
+ /// both over the years, and a picture decoded with the wrong one is
+ /// washed out or crushed.
+ public var videoRange: Bool
+
+ public init(
+ width: Int, height: Int, orientation: UInt16, timestampNs: UInt64, videoRange: Bool = false
+ ) {
+ self.width = width
+ self.height = height
+ self.orientation = orientation
+ self.timestampNs = timestampNs
+ self.videoRange = videoRange
+ }
+
+ public var payloadLength: Int { width * height * 3 / 2 }
+
+ public func write(to bytes: UnsafeMutableRawPointer) {
+ bytes.storeBytes(of: Self.magic.littleEndian, toByteOffset: 0, as: UInt32.self)
+ bytes.storeBytes(of: Self.version.littleEndian, toByteOffset: 4, as: UInt16.self)
+ bytes.storeBytes(of: orientation.littleEndian, toByteOffset: 6, as: UInt16.self)
+ bytes.storeBytes(of: UInt32(width).littleEndian, toByteOffset: 8, as: UInt32.self)
+ bytes.storeBytes(of: UInt32(height).littleEndian, toByteOffset: 12, as: UInt32.self)
+ bytes.storeBytes(of: timestampNs.littleEndian, toByteOffset: 16, as: UInt64.self)
+ bytes.storeBytes(of: UInt32(payloadLength).littleEndian, toByteOffset: 24, as: UInt32.self)
+ bytes.storeBytes(of: UInt32(videoRange ? 1 : 0).littleEndian, toByteOffset: 28, as: UInt32.self)
+ }
+
+ /// Reads a header, or nil for anything that is not one this side can
+ /// use. The socket is in a container only this app's own processes can
+ /// open, so a bad header is a bug or a version skew rather than an
+ /// attack — but it is still refused rather than trusted, because the
+ /// length in it is what the reader allocates.
+ public static func read(_ bytes: UnsafeRawPointer) -> BroadcastFrameHeader? {
+ func u32(_ at: Int) -> UInt32 { UInt32(littleEndian: bytes.loadUnaligned(fromByteOffset: at, as: UInt32.self)) }
+ func u16(_ at: Int) -> UInt16 { UInt16(littleEndian: bytes.loadUnaligned(fromByteOffset: at, as: UInt16.self)) }
+ guard u32(0) == magic, u16(4) == version else { return nil }
+ let width = Int(u32(8))
+ let height = Int(u32(12))
+ guard (2...largest).contains(width), (2...largest).contains(height),
+ width % 2 == 0, height % 2 == 0
+ else { return nil }
+ let header = BroadcastFrameHeader(
+ width: width, height: height, orientation: u16(6),
+ timestampNs: UInt64(littleEndian: bytes.loadUnaligned(fromByteOffset: 16, as: UInt64.self)),
+ videoRange: u32(28) & 1 == 1)
+ guard Int(u32(24)) == header.payloadLength else { return nil }
+ return header
+ }
+}
+
+/// Where the socket lives: one well-known name in the App Group container,
+/// short because a Unix socket's path has to fit in 104 bytes, and the
+/// container's own path on a device already takes about ninety.
+public enum BroadcastPath {
+ public static let name = "calls.sock"
+
+ /// The App Group this build shares with its extension, from the
+ /// `CallsAppGroup` key both Info.plists carry. One build setting feeds
+ /// the key and the entitlement, so a build with a different bundle
+ /// identifier changes the group in one place.
+ public static var appGroup: String? {
+ Bundle.main.object(forInfoDictionaryKey: "CallsAppGroup") as? String
+ }
+
+ public static func socket(appGroup: String) -> String? {
+ FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: appGroup)?
+ .appendingPathComponent(name).path
+ }
+}
+
+enum BroadcastSocket {
+ struct Failure: Error, CustomStringConvertible {
+ let description: String
+ }
+
+ /// A Unix-socket address for `path`, or nil if the path does not fit.
+ static func address(_ path: String) -> sockaddr_un? {
+ var address = sockaddr_un()
+ address.sun_family = sa_family_t(AF_UNIX)
+ let bytes = Array(path.utf8)
+ let capacity = MemoryLayout.size(ofValue: address.sun_path)
+ guard bytes.count < capacity else { return nil }
+ withUnsafeMutableBytes(of: &address.sun_path) { raw in
+ raw.copyBytes(from: bytes)
+ raw[bytes.count] = 0
+ }
+ address.sun_len = UInt8(MemoryLayout<sockaddr_un>.size)
+ return address
+ }
+
+ /// A socket that will not raise SIGPIPE when the far side has gone: a
+ /// write to a closed peer answers EPIPE instead, which is the ending
+ /// both sides are listening for, rather than killing the process.
+ static func make() throws -> Int32 {
+ let fd = Darwin.socket(AF_UNIX, SOCK_STREAM, 0)
+ guard fd >= 0 else { throw Failure(description: "socket: \(errno)") }
+ var on: Int32 = 1
+ setsockopt(fd, SOL_SOCKET, SO_NOSIGPIPE, &on, socklen_t(MemoryLayout<Int32>.size))
+ return fd
+ }
+
+ static func listen(path: String) throws -> Int32 {
+ guard var address = address(path) else { throw Failure(description: "socket path too long") }
+ unlink(path)
+ let fd = try make()
+ let bound = withUnsafePointer(to: &address) {
+ $0.withMemoryRebound(to: sockaddr.self, capacity: 1) {
+ bind(fd, $0, socklen_t(MemoryLayout<sockaddr_un>.size))
+ }
+ }
+ guard bound == 0, Darwin.listen(fd, 1) == 0 else {
+ let code = errno
+ Darwin.close(fd)
+ throw Failure(description: "listen: \(code)")
+ }
+ return fd
+ }
+
+ static func connect(path: String) throws -> Int32 {
+ guard var address = address(path) else { throw Failure(description: "socket path too long") }
+ let fd = try make()
+ let connected = withUnsafePointer(to: &address) {
+ $0.withMemoryRebound(to: sockaddr.self, capacity: 1) {
+ Darwin.connect(fd, $0, socklen_t(MemoryLayout<sockaddr_un>.size))
+ }
+ }
+ guard connected == 0 else {
+ let code = errno
+ Darwin.close(fd)
+ throw Failure(description: "connect: \(code)")
+ }
+ return fd
+ }
+
+ static func nonBlocking(_ fd: Int32) {
+ _ = fcntl(fd, F_SETFL, fcntl(fd, F_GETFL) | O_NONBLOCK)
+ }
+}
+
+/// A frame size that fits within `longest` on its long side, keeps its
+/// shape, and is even in both directions, which is what 4:2:0 chroma and
+/// the encoders need. The Mac's screen capture and the iPhone's broadcast
+/// both size their frames with it.
+public func fitted(width: CGFloat, height: CGFloat, longest: CGFloat) -> (width: Int, height: Int) {
+ guard width > 0, height > 0 else { return (2, 2) }
+ let scale = min(1, longest / max(width, height))
+ func even(_ value: CGFloat) -> Int { max(2, Int((value * scale).rounded(.down)) & ~1) }
+ return (even(width), even(height))
+}
diff --git a/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift
new file mode 100644
index 0000000..e15a8f0
--- /dev/null
+++ b/native/apple/Sources/CallsKit/BroadcastScreenCapture.swift
@@ -0,0 +1,70 @@
+import CallsBroadcastKit
+import CoreVideo
+import Foundation
+import WebRTC
+
+/// The iPhone's whole screen, arriving from the broadcast extension over
+/// its socket (see CallsBroadcastKit), delivered into the call's screen
+/// source.
+///
+/// There is no picker to wait on by the time one of these exists: the
+/// person chose to broadcast at the system's own sheet, and the extension
+/// connecting is the proof. So `start` returns at once, and `ended` is the
+/// extension going away — the broadcast stopped from Control Center or the
+/// red status bar. `stop` closes the connection, which is how stopping in
+/// the app reaches the system: the extension's next write fails and it
+/// ends the broadcast.
+@MainActor public final class BroadcastScreenCapture: ScreenCapture {
+ private let connection: BroadcastConnection
+ private var stopped = false
+ /// Called after the share has ended on its own, once the call has heard.
+ /// The app uses it to leave a call it is only still in for the screen.
+ public var afterEnded: (@MainActor () -> Void)?
+
+ public init(_ connection: BroadcastConnection) { self.connection = connection }
+
+ public func start(source: RTCVideoSource, ended: @escaping @MainActor () -> Void) async throws {
+ let capturer = RTCVideoCapturer(delegate: source)
+ connection.onFrame = { pixels, orientation in
+ // Stamped on arrival, on this device's clock, rather than with the
+ // extension's: the two are separate processes, and the encoder only
+ // needs the frames in order and spaced as they came.
+ source.capturer(
+ capturer,
+ didCapture: RTCVideoFrame(
+ buffer: RTCCVPixelBuffer(pixelBuffer: pixels), rotation: broadcastRotation(orientation),
+ timeStampNs: Int64(DispatchTime.now().uptimeNanoseconds)))
+ }
+ connection.onClose = { [weak self] in
+ Task { @MainActor [weak self] in
+ guard let self, !self.stopped else { return }
+ self.stopped = true
+ ended()
+ self.afterEnded?()
+ }
+ }
+ connection.resume()
+ }
+
+ public func stop() {
+ stopped = true
+ connection.onClose = nil
+ connection.onFrame = nil
+ connection.close()
+ }
+}
+
+/// ReplayKit's orientation for a frame, as the rotation WebRTC should
+/// apply to show it upright. ReplayKit says which way the *image* is
+/// turned, as a CGImagePropertyOrientation: `left` (8) is a phone held in
+/// landscape with its top to the left, whose frames need turning a quarter
+/// clockwise to stand up; `right` (6) the other way. Anything else — and a
+/// phone upright — is sent as it came.
+func broadcastRotation(_ orientation: UInt16) -> RTCVideoRotation {
+ switch orientation {
+ case 3, 4: return ._180
+ case 5, 8: return ._90
+ case 6, 7: return ._270
+ default: return ._0
+ }
+}
diff --git a/native/apple/Sources/CallsKit/MacScreenCapture.swift b/native/apple/Sources/CallsKit/MacScreenCapture.swift
index 2017e38..2e66cb1 100644
--- a/native/apple/Sources/CallsKit/MacScreenCapture.swift
+++ b/native/apple/Sources/CallsKit/MacScreenCapture.swift
@@ -1,4 +1,5 @@
#if os(macOS)
+ import CallsBroadcastKit
import CoreMedia
import Foundation
import ScreenCaptureKit
diff --git a/native/apple/Sources/CallsKit/ScreenShare.swift b/native/apple/Sources/CallsKit/ScreenShare.swift
index 9d293b9..c3661f1 100644
--- a/native/apple/Sources/CallsKit/ScreenShare.swift
+++ b/native/apple/Sources/CallsKit/ScreenShare.swift
@@ -1,4 +1,3 @@
-import CoreGraphics
import Foundation
// ---------------------------------------------------------------
@@ -119,13 +118,3 @@ public enum Stage {
return peers.first.map(StageItem.camera)
}
}
-
-/// A frame size that fits within `longest` on its long side, keeps its
-/// shape, and is even in both directions, which is what the encoders'
-/// 4:2:0 chroma needs.
-func fitted(width: CGFloat, height: CGFloat, longest: CGFloat) -> (width: Int, height: Int) {
- guard width > 0, height > 0 else { return (2, 2) }
- let scale = min(1, longest / max(width, height))
- func even(_ value: CGFloat) -> Int { max(2, Int((value * scale).rounded(.down)) & ~1) }
- return (even(width), even(height))
-}
diff --git a/native/apple/Tests/CallsKitTests/BroadcastTests.swift b/native/apple/Tests/CallsKitTests/BroadcastTests.swift
new file mode 100644
index 0000000..01a76e0
--- /dev/null
+++ b/native/apple/Tests/CallsKitTests/BroadcastTests.swift
@@ -0,0 +1,183 @@
+import CoreVideo
+import WebRTC
+import XCTest
+
+@testable import CallsBroadcastKit
+@testable import CallsKit
+
+/// The iPhone's screen travels from the broadcast extension to the app
+/// over a Unix socket, and nothing about that needs an iPhone: the same
+/// sender and inbox run here, over a real socket, with real pixel
+/// buffers. What cannot run here is ReplayKit itself, the App Group
+/// container and the extension's memory ceiling — those need a device.
+final class BroadcastTests: XCTestCase {
+ private var path = ""
+
+ override func setUp() {
+ // Short, because a socket path has to fit in 104 bytes.
+ path = "/tmp/calls-\(UUID().uuidString.prefix(8)).sock"
+ }
+ override func tearDown() { unlink(path) }
+
+ func testTheHeaderRoundTripsAndRefusesWhatItCannotUse() {
+ let header = BroadcastFrameHeader(
+ width: 640, height: 360, orientation: 8, timestampNs: 123_456_789, videoRange: true)
+ var bytes = [UInt8](repeating: 0, count: BroadcastFrameHeader.size)
+ bytes.withUnsafeMutableBytes { header.write(to: $0.baseAddress!) }
+ XCTAssertEqual(bytes.withUnsafeBytes { BroadcastFrameHeader.read($0.baseAddress!) }, header)
+ // Odd sizes, absurd sizes, a length that disagrees with them, and a
+ // stranger's magic are all refused: the length is what gets allocated.
+ for (offset, value) in [(8, UInt32(641)), (12, UInt32(100_000)), (24, UInt32(12)), (0, UInt32(7))] {
+ var bad = bytes
+ bad.withUnsafeMutableBytes { $0.storeBytes(of: value.littleEndian, toByteOffset: offset, as: UInt32.self) }
+ XCTAssertNil(bad.withUnsafeBytes { BroadcastFrameHeader.read($0.baseAddress!) }, "offset \(offset)")
+ }
+ }
+
+ func testAFrameCrossesTheSocketSmallerAndIntact() throws {
+ let inbox = BroadcastInbox(path: path)
+ let connected = expectation(description: "connected")
+ let arrived = expectation(description: "frame")
+ var received: (CVPixelBuffer, UInt16)?
+ inbox.onConnection = { connection in
+ connection.onFrame = { pixels, orientation in
+ guard received == nil else { return }
+ received = (pixels, orientation)
+ arrived.fulfill()
+ }
+ connection.resume()
+ connected.fulfill()
+ }
+ try inbox.start()
+ defer { inbox.stop() }
+
+ let sender = BroadcastSender(path: path, longest: 640, framesPerSecond: 100)
+ XCTAssertTrue(sender.connect(timeout: 2))
+ defer { sender.close() }
+ wait(for: [connected], timeout: 2)
+ // A 1280x720 frame, luma 100 and chroma (60, 200) throughout, as
+ // ReplayKit's full-range bi-planar buffers are laid out.
+ sender.send(try nv12(width: 1280, height: 720), orientation: 6, timestampNs: 1)
+ wait(for: [arrived], timeout: 2)
+
+ let (pixels, orientation) = try XCTUnwrap(received)
+ XCTAssertEqual(orientation, 6)
+ XCTAssertEqual(CVPixelBufferGetWidth(pixels), 640)
+ XCTAssertEqual(CVPixelBufferGetHeight(pixels), 360)
+ XCTAssertEqual(
+ CVPixelBufferGetPixelFormatType(pixels), kCVPixelFormatType_420YpCbCr8BiPlanarFullRange)
+ CVPixelBufferLockBaseAddress(pixels, .readOnly)
+ defer { CVPixelBufferUnlockBaseAddress(pixels, .readOnly) }
+ let luma = CVPixelBufferGetBaseAddressOfPlane(pixels, 0)!.assumingMemoryBound(to: UInt8.self)
+ let chroma = CVPixelBufferGetBaseAddressOfPlane(pixels, 1)!.assumingMemoryBound(to: UInt8.self)
+ let lumaStride = CVPixelBufferGetBytesPerRowOfPlane(pixels, 0)
+ let chromaStride = CVPixelBufferGetBytesPerRowOfPlane(pixels, 1)
+ XCTAssertEqual(luma[0], 100)
+ XCTAssertEqual(luma[359 * lumaStride + 639], 100)
+ XCTAssertEqual(chroma[0], 60)
+ XCTAssertEqual(chroma[1], 200)
+ XCTAssertEqual(chroma[179 * chromaStride + 638], 60)
+ XCTAssertEqual(chroma[179 * chromaStride + 639], 200)
+ }
+
+ func testTheExtensionGoingAwayEndsTheConnection() throws {
+ let inbox = BroadcastInbox(path: path)
+ let closed = expectation(description: "closed")
+ inbox.onConnection = { connection in
+ connection.onClose = { closed.fulfill() }
+ connection.resume()
+ }
+ try inbox.start()
+ defer { inbox.stop() }
+ let sender = BroadcastSender(path: path)
+ XCTAssertTrue(sender.connect(timeout: 2))
+ sender.close()
+ wait(for: [closed], timeout: 2)
+ }
+
+ func testStoppingInTheAppReachesTheExtension() throws {
+ // The app closing its end is the whole of "stop sharing": the
+ // extension learns it from its next write, and ends the broadcast.
+ let inbox = BroadcastInbox(path: path)
+ var held: BroadcastConnection?
+ let connected = expectation(description: "connected")
+ inbox.onConnection = { connection in
+ held = connection
+ connection.resume()
+ connected.fulfill()
+ }
+ try inbox.start()
+ defer { inbox.stop() }
+ let sender = BroadcastSender(path: path, longest: 64, framesPerSecond: 1000)
+ let gone = expectation(description: "extension told")
+ sender.onClosed = { gone.fulfill() }
+ XCTAssertTrue(sender.connect(timeout: 2))
+ wait(for: [connected], timeout: 2)
+ held?.close()
+ let frame = try nv12(width: 64, height: 64)
+ let until = Date().addingTimeInterval(2)
+ while Date() < until {
+ sender.send(frame, orientation: 1, timestampNs: 0)
+ if XCTWaiter.wait(for: [gone], timeout: 0.05) == .completed { return }
+ }
+ XCTFail("the extension never learned the app had stopped")
+ }
+
+ func testOnlyOneBroadcastAtATime() throws {
+ let inbox = BroadcastInbox(path: path)
+ var connections = 0
+ let first = expectation(description: "first")
+ inbox.onConnection = { connection in
+ connections += 1
+ connection.resume()
+ first.fulfill()
+ }
+ try inbox.start()
+ defer { inbox.stop() }
+ let one = BroadcastSender(path: path)
+ XCTAssertTrue(one.connect(timeout: 2))
+ defer { one.close() }
+ wait(for: [first], timeout: 2)
+ let two = BroadcastSender(path: path)
+ let refused = expectation(description: "second refused")
+ two.onClosed = { refused.fulfill() }
+ XCTAssertTrue(two.connect(timeout: 2))
+ defer { two.close() }
+ let frame = try nv12(width: 64, height: 64)
+ let until = Date().addingTimeInterval(2)
+ while Date() < until {
+ two.send(frame, orientation: 1, timestampNs: 0)
+ if XCTWaiter.wait(for: [refused], timeout: 0.05) == .completed { break }
+ }
+ XCTAssertEqual(connections, 1)
+ }
+
+ func testReplayKitOrientationBecomesAnUprightRotation() {
+ XCTAssertEqual(broadcastRotation(1), ._0)
+ XCTAssertEqual(broadcastRotation(3), ._180)
+ XCTAssertEqual(broadcastRotation(8), ._90)
+ XCTAssertEqual(broadcastRotation(6), ._270)
+ XCTAssertEqual(broadcastRotation(0), ._0)
+ }
+
+ private func nv12(width: Int, height: Int) throws -> CVPixelBuffer {
+ var made: CVPixelBuffer?
+ CVPixelBufferCreate(
+ nil, width, height, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
+ [kCVPixelBufferIOSurfacePropertiesKey: [:]] as CFDictionary, &made)
+ let pixels = try XCTUnwrap(made)
+ CVPixelBufferLockBaseAddress(pixels, [])
+ defer { CVPixelBufferUnlockBaseAddress(pixels, []) }
+ let luma = CVPixelBufferGetBaseAddressOfPlane(pixels, 0)!
+ memset(luma, 100, CVPixelBufferGetBytesPerRowOfPlane(pixels, 0) * height)
+ let chroma = CVPixelBufferGetBaseAddressOfPlane(pixels, 1)!.assumingMemoryBound(to: UInt8.self)
+ let stride = CVPixelBufferGetBytesPerRowOfPlane(pixels, 1)
+ for row in 0..<(height / 2) {
+ for pair in 0..<(width / 2) {
+ chroma[row * stride + pair * 2] = 60
+ chroma[row * stride + pair * 2 + 1] = 200
+ }
+ }
+ return pixels
+ }
+}
diff --git a/native/apple/Tests/CallsKitTests/ScreenShareTests.swift b/native/apple/Tests/CallsKitTests/ScreenShareTests.swift
index 9c5ebb6..78939ec 100644
--- a/native/apple/Tests/CallsKitTests/ScreenShareTests.swift
+++ b/native/apple/Tests/CallsKitTests/ScreenShareTests.swift
@@ -1,5 +1,6 @@
import XCTest
+@testable import CallsBroadcastKit
@testable import CallsKit
/// §5.4's receiving half. These are room-share.test.mjs's classifyTrack
diff --git a/native/ios/Broadcast/SampleHandler.swift b/native/ios/Broadcast/SampleHandler.swift
new file mode 100644
index 0000000..b20a947
--- /dev/null
+++ b/native/ios/Broadcast/SampleHandler.swift
@@ -0,0 +1,61 @@
+import CallsBroadcastKit
+import CoreMedia
+import ReplayKit
+
+/// The broadcast upload extension: ReplayKit's frames in, the app's socket
+/// out. It links CallsBroadcastKit and nothing else of Calls, and holds
+/// one frame buffer, because it lives under a memory ceiling of about
+/// fifty megabytes and is killed without a word past it.
+///
+/// It never talks to the meeting. The app is already in the call and owns
+/// every peer connection; this only hands it pictures.
+final class SampleHandler: RPBroadcastSampleHandler {
+ private var sender: BroadcastSender?
+
+ override func broadcastStarted(withSetupInfo setupInfo: [String: NSObject]?) {
+ guard let group = BroadcastPath.appGroup, let path = BroadcastPath.socket(appGroup: group)
+ else {
+ finish("Calls could not share your screen on this build.")
+ return
+ }
+ let sender = BroadcastSender(path: path)
+ // Two seconds: the countdown has just ended, and the app, which is in
+ // the call, is listening already. Longer would only delay the answer
+ // to someone who started a broadcast with no call to share it to.
+ guard sender.connect(timeout: 2) else {
+ finish("Join a call in Calls, then share your screen from there.")
+ return
+ }
+ sender.onClosed = { [weak self] in
+ self?.finish("You stopped sharing your screen in Calls.")
+ }
+ self.sender = sender
+ }
+
+ override func processSampleBuffer(
+ _ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType
+ ) {
+ guard sampleBufferType == .video, let sender, let pixels = sampleBuffer.imageBuffer else {
+ return
+ }
+ let orientation =
+ (CMGetAttachment(sampleBuffer, key: RPVideoSampleOrientationKey as CFString, attachmentModeOut: nil)
+ as? NSNumber)?.uint16Value ?? 1
+ let time = CMTimeGetSeconds(sampleBuffer.presentationTimeStamp)
+ sender.send(pixels, orientation: orientation, timestampNs: UInt64(max(0, time) * 1_000_000_000))
+ }
+
+ override func broadcastFinished() {
+ sender?.close()
+ sender = nil
+ }
+
+ /// Ends the broadcast with a sentence the system shows the person. The
+ /// only way an extension can say anything at all.
+ private func finish(_ sentence: String) {
+ sender?.close()
+ sender = nil
+ finishBroadcastWithError(
+ NSError(domain: "Calls", code: 1, userInfo: [NSLocalizedDescriptionKey: sentence]))
+ }
+}
diff --git a/native/ios/README.md b/native/ios/README.md
index 2cedca4..46e8a5c 100644
--- a/native/ios/README.md
+++ b/native/ios/README.md
@@ -47,12 +47,67 @@ available to Xcode, with Developer Mode enabled.
- Camera/microphone permission refusals appear in the app. Retry after changing
permission in Settings. The simulator has no physical camera.
-Calls are foreground-only in this milestone. Switching away ends the call;
-sign-in survives switching to the browser. Screen sharing, camera switching,
+Calls are foreground-only in this milestone. Switching away ends the call,
+except while you are sharing your screen; sign-in survives switching to the
+browser. Camera switching,
CallKit/background calling, stored credentials, and distribution are follow-up
work. Physical camera, audible audio, Bluetooth routing, rotation while capturing,
and interruption recovery need a real-device acceptance pass.
+## Sharing the screen
+
+**Share your screen** in the call is the system's broadcast button
+(`RPSystemBroadcastPickerView`, offering only Calls' own extension). The whole
+screen is shared, which is the only thing iOS lets any app share. The frames
+reach a **broadcast upload extension** (`Broadcast/SampleHandler.swift`, target
+`CallsBroadcast`), a separate process with a memory ceiling of about fifty
+megabytes. It links only `CallsBroadcastKit` and no WebRTC, which keeps it near
+600 KB. It shrinks each frame to 1280 on the long side, at up to fifteen a second,
+and passes it to the app over a Unix socket in the **App Group** container
+(`calls.sock`). The app is already in the call and sends the frames on the same
+`sendonly` screen transceiver the Mac and the browser use. Only one frame is in
+flight at a time; any frame that arrives while one is being written is dropped,
+never queued.
+
+Stopping works from either end. **Stop sharing** in the app closes the socket;
+the extension's next write fails and it ends the broadcast with "You stopped
+sharing your screen in Calls." Stopping from Control Center or the red status
+bar closes the socket from the other side, and the app stops the share. A
+broadcast started with no call to take it ends with "Join a call in Calls, then
+share your screen from there."
+
+A call that is sharing stays up when you leave the app, because showing other
+apps is the point. That needs the `audio` background mode. Every other call
+still ends when the app goes to the background. If the broadcast is stopped
+from outside while Calls is in the background, the call ends too.
+
+The App Group is one build setting, `CALLS_APP_GROUP` in `project.yml`
+(`group.net.amadan.oficina.calls`). It feeds both targets' entitlements and the
+`CallsAppGroup` key in both Info.plists. The extension's bundle identifier is the
+app's plus `.broadcast`. The app looks for its extension by that rule, so the
+two must stay in step.
+
+**Signing.** The simulator build is unsigned, so it has no App Group and cannot
+broadcast; ReplayKit broadcasts need a device anyway. Before the first signed
+device build, in the Apple Developer portal (Certificates, Identifiers &
+Profiles):
+
+1. Identifiers → App Groups → **+** → `group.net.amadan.oficina.calls`.
+2. Identifiers → `net.amadan.oficina.calls.ios` → enable **App Groups** →
+ choose that group. (Push stays as it is.)
+3. Identifiers → **+** → App IDs → App → `net.amadan.oficina.calls.ios.broadcast`,
+ description "Calls broadcast", with **App Groups** enabled and the same group.
+4. `CALLS_DEVELOPMENT_TEAM=<team> make iphone-device`. Automatic signing makes
+ both development profiles once the identifiers exist, and
+ `-allowProvisioningUpdates` can also create steps 1–3 for you if you would
+ rather let Xcode do it.
+
+A TestFlight or App Store build needs distribution profiles for **both** bundle
+identifiers, and the extension goes up inside the app. If the app ships under
+another identifier (the release branch uses `works.oficina.meet`), change
+`CALLS_APP_GROUP` to match, for example `group.works.oficina.meet`, and register
+`<app id>.broadcast`.
+
## Verification
```sh
diff --git a/native/ios/Sources/CallsPhone.swift b/native/ios/Sources/CallsPhone.swift
index 7d8cef6..68d975a 100644
--- a/native/ios/Sources/CallsPhone.swift
+++ b/native/ios/Sources/CallsPhone.swift
@@ -34,7 +34,9 @@ final class CallsPhoneDelegate: NSObject, UIApplicationDelegate {
.onChange(of: scenePhase) { _, phase in
// This milestone is foreground-only. Browser sign-in is kept alive;
// an active call leaves cleanly instead of silently sending audio.
- if phase == .background && model.joining { model.leave() }
+ // The one exception is a call sharing the whole screen, whose
+ // point is the other apps the person is about to go to.
+ if phase == .background && model.joining && !model.call.sharingScreen { model.leave() }
UIApplication.shared.isIdleTimerDisabled = phase == .active && model.joining
}
.onChange(of: model.joining) { _, joining in
@@ -68,6 +70,7 @@ struct PhoneCall: View {
@State private var grid = false
@State private var chosen: StageItem?
@StateObject private var draft = ChatDraft()
+ @StateObject private var screenShare = PhoneScreenShare()
private var peers: [RosterPeer] { call.state?.peers.filter { $0.peer != call.state?.peer } ?? [] }
private var focused: StageItem? {
@@ -80,6 +83,10 @@ struct PhoneCall: View {
VStack(alignment: .leading, spacing: 4) {
Text(call.title).font(.headline).lineLimit(2)
Text(call.status).font(.caption).foregroundStyle(.secondary)
+ if call.sharingScreen {
+ Label("You are sharing your screen", systemImage: "rectangle.on.rectangle")
+ .font(.caption.bold()).foregroundStyle(.orange)
+ }
}
Spacer()
Menu {
@@ -191,6 +198,17 @@ struct PhoneCall: View {
systemImage: call.cameraOff ? "video.slash.fill" : "video.fill"
) { call.toggleCamera() }
.buttonStyle(CallActionStyle(color: .blue, emphasized: call.cameraOff))
+ if call.sharingScreen {
+ Button("Stop sharing your screen", systemImage: "rectangle.on.rectangle.slash") {
+ call.stopSharingScreen()
+ }
+ .buttonStyle(CallActionStyle(color: .orange, emphasized: true))
+ .accessibilityIdentifier("shareScreen")
+ } else {
+ BroadcastButton().frame(width: 44, height: 44)
+ .disabled(call.state?.phase != "admitted")
+ .accessibilityIdentifier("shareScreen")
+ }
Button("Devices", systemImage: "slider.horizontal.3") { showingDevices = true }
.buttonStyle(CallActionStyle(color: .teal))
.accessibilityIdentifier("devices")
@@ -211,6 +229,11 @@ struct PhoneCall: View {
}.padding(16).background(.bar)
}
.task { await call.watchAudio() }
+ .onAppear { screenShare.listen(for: call, leave: leave) }
+ .onDisappear {
+ screenShare.stop()
+ call.stopSharingScreen()
+ }
.onChange(of: call.screens) { before, after in
// A new screen takes the stage, as it does on the Mac.
if after.contains(where: { !before.contains($0) }) { chosen = nil }
diff --git a/native/ios/Sources/PhoneScreenShare.swift b/native/ios/Sources/PhoneScreenShare.swift
new file mode 100644
index 0000000..30b460d
--- /dev/null
+++ b/native/ios/Sources/PhoneScreenShare.swift
@@ -0,0 +1,88 @@
+import CallsBroadcastKit
+import CallsKit
+import ReplayKit
+import SwiftUI
+
+/// Listens for the broadcast extension for as long as a call is on screen,
+/// and turns each broadcast into the call's screen share.
+///
+/// The order is the reverse of the Mac's. There, the button asks and the
+/// capture answers; here the person starts a broadcast at the system's
+/// sheet, the extension connects, and only then does the call learn there
+/// is a screen to send. A broadcast that arrives when the call cannot take
+/// it — not yet admitted, or already sharing — is closed at once, and the
+/// extension ends it with a sentence of its own.
+@MainActor final class PhoneScreenShare: ObservableObject {
+ private var inbox: BroadcastInbox?
+
+ func listen(for call: Call, leave: @escaping () -> Void) {
+ guard inbox == nil, let group = BroadcastPath.appGroup,
+ let path = BroadcastPath.socket(appGroup: group)
+ else { return }
+ let inbox = BroadcastInbox(path: path)
+ inbox.onConnection = { [weak call] connection in
+ MainActor.assumeIsolated {
+ guard let call, call.state?.phase == "admitted", !call.sharingScreen else {
+ connection.close()
+ return
+ }
+ let capture = BroadcastScreenCapture(connection)
+ // A broadcast stopped from Control Center while the app is in the
+ // background ends the only reason the call was still up there.
+ capture.afterEnded = {
+ if UIApplication.shared.applicationState == .background { leave() }
+ }
+ Task { await call.shareScreen(capture) }
+ }
+ }
+ do {
+ try inbox.start()
+ self.inbox = inbox
+ } catch {
+ // No App Group on this build (the simulator's unsigned one, say):
+ // the button still shows, and the extension says it could not
+ // reach a call. Nothing else in the call depends on this.
+ }
+ }
+
+ func stop() {
+ inbox?.stop()
+ inbox = nil
+ }
+}
+
+/// The system's broadcast button, dressed as the call's other controls.
+///
+/// RPSystemBroadcastPickerView is the only way an app may start a
+/// broadcast, and it only responds to a tap on its own button. So the
+/// picker is the button: its image is replaced with the call's symbol and
+/// tint, and it is sized like a CallActionStyle control beside the others.
+/// `preferredExtension` makes the sheet offer Calls' own extension rather
+/// than every broadcast service on the phone.
+struct BroadcastButton: UIViewRepresentable {
+ static let extensionIdentifier = Bundle.main.bundleIdentifier.map { $0 + ".broadcast" }
+
+ func makeUIView(context: Context) -> RPSystemBroadcastPickerView {
+ let picker = RPSystemBroadcastPickerView(frame: CGRect(x: 0, y: 0, width: 44, height: 44))
+ picker.preferredExtension = Self.extensionIdentifier
+ picker.showsMicrophoneButton = false
+ picker.accessibilityLabel = "Share your screen"
+ style(picker)
+ return picker
+ }
+ func updateUIView(_ picker: RPSystemBroadcastPickerView, context: Context) { style(picker) }
+
+ private func style(_ picker: RPSystemBroadcastPickerView) {
+ guard let button = picker.subviews.compactMap({ $0 as? UIButton }).first else { return }
+ let symbol = UIImage(
+ systemName: "rectangle.on.rectangle",
+ withConfiguration: UIImage.SymbolConfiguration(textStyle: .title3))
+ button.setImage(symbol, for: .normal)
+ button.tintColor = .systemOrange
+ button.backgroundColor = UIColor.systemOrange.withAlphaComponent(0.12)
+ button.layer.cornerRadius = 14
+ button.accessibilityLabel = "Share your screen"
+ button.frame = picker.bounds
+ button.autoresizingMask = [.flexibleWidth, .flexibleHeight]
+ }
+}
diff --git a/native/ios/project.yml b/native/ios/project.yml
index 185319d..b6f4c61 100644
--- a/native/ios/project.yml
+++ b/native/ios/project.yml
@@ -12,6 +12,11 @@ settings:
CODE_SIGN_STYLE: Automatic
MARKETING_VERSION: '0.2.0'
CURRENT_PROJECT_VERSION: '1'
+ # The App Group the app and its broadcast extension share: the socket
+ # the iPhone's screen crosses lives in its container. One setting
+ # feeds both entitlements and both Info.plists' CallsAppGroup key, so
+ # a build under another bundle identifier changes it here and only here.
+ CALLS_APP_GROUP: group.net.amadan.oficina.calls
targets:
CallsPhone:
type: application
@@ -27,9 +32,19 @@ targets:
CFBundleURLTypes:
- CFBundleURLName: net.amadan.oficina.calls.join
CFBundleURLSchemes: [oficina-calls]
+ CallsAppGroup: $(CALLS_APP_GROUP)
+ # Audio, so a call sharing the whole screen survives the person
+ # going to the app they meant to show. Only a call that is sharing
+ # stays up in the background; every other call still ends there.
+ UIBackgroundModes: [audio]
+ entitlements:
+ path: ../build/ios-app.entitlements
+ properties:
+ com.apple.security.application-groups: [$(CALLS_APP_GROUP)]
dependencies:
- package: CallsNative
product: CallsKit
+ - target: CallsBroadcast
settings:
base:
ASSETCATALOG_COMPILER_APPICON_NAME: AppIcon
@@ -41,6 +56,30 @@ targets:
INFOPLIST_KEY_NSMicrophoneUsageDescription: Calls uses your microphone so people in your meeting can hear you.
INFOPLIST_KEY_UILaunchScreen_Generation: YES
INFOPLIST_KEY_UISupportedInterfaceOrientations: UIInterfaceOrientationPortrait UIInterfaceOrientationLandscapeLeft UIInterfaceOrientationLandscapeRight UIInterfaceOrientationPortraitUpsideDown
+ CallsBroadcast:
+ type: app-extension
+ platform: iOS
+ sources: [Broadcast]
+ info:
+ path: ../build/ios-broadcast-info.plist
+ properties:
+ CFBundleDisplayName: Calls
+ CallsAppGroup: $(CALLS_APP_GROUP)
+ NSExtension:
+ NSExtensionPointIdentifier: com.apple.broadcast-services-upload
+ NSExtensionPrincipalClass: $(PRODUCT_MODULE_NAME).SampleHandler
+ RPBroadcastProcessMode: RPBroadcastProcessModeSampleBuffer
+ entitlements:
+ path: ../build/ios-broadcast.entitlements
+ properties:
+ com.apple.security.application-groups: [$(CALLS_APP_GROUP)]
+ dependencies:
+ - package: CallsNative
+ product: CallsBroadcastKit
+ settings:
+ base:
+ PRODUCT_BUNDLE_IDENTIFIER: net.amadan.oficina.calls.ios.broadcast
+ TARGETED_DEVICE_FAMILY: '1,2'
CallsPhoneTests:
type: bundle.unit-test
platform: iOS