ios 15 now support PIP, did u tried with this project?

[fukemy]

hi, i read the doc about PIP for video call, but i can not done it alone, i feel so hard to me, so I want to ask did you want to make PIP mode for videocall?
https://developer.apple.com/documentation/avkit/adopting_picture_in_picture_for_video_calls?changes=_1

[Meonardo]

https://developer.apple.com/documentation/avkit/adopting_picture_in_picture_for_video_calls?changes=_1
this is good news, I will try to do it this weekend.

[fukemy]

thanks bro, im stuck in converting buffer into AVSampleBufferDisplayLayer, waiting for your help now

[fukemy]

that's not good point: You need register your app with Apple to get Multitasking's certificate that using for PIP Call, that's so bad. You can see the doc here

https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_avfoundation_multitasking-camera-access
https://developer.apple.com/contact/request/multitasking-camera-access/

[Meonardo]

Yes, I saw those requirements, I did make a request earlier, wait for Apple to approve...

[fukemy]

ok, hope you get this PIP feature in demo <3

[Meonardo]

@fukemy Hi, recently I have a urgent task to finish, so I will try my best to do this after my task is done.

[fukemy]

dont worry, we have time, i can wait, if you have any ideal u can post here

[Derad6709]

Any updates?

[fukemy]

hi, i have new ideal based on PiPKit
He using view captured to render pip mode, can u check this support to us?
Im tried to using this but it's return black screen, but i think may it work?

[Meonardo]

sorry for no response for a long time @fukemy can you try the latest main brach?

[fukemy]

hi, welcome back @Meonardo, could u provide me what's new in lasted main branch?
Because your current version that i used is work normally, So I think i will check with Janus server

[Meonardo]

I add the PiP mode support, but it only work when the videoroom configures h.264 codec, I am not sure what problem is, still working on it...

[fukemy]

ok, i can convert codec to VP8 from IOS to test, i will try if i have free times, thanks
If you want to check VP8 with me, you can see this code below:

protocol ScreenSampleCapturerDelegate: AnyObject {
    func didCaptureVideo(sampleBuffer: CMSampleBuffer)
}

enum VideoRotation: Int {
    case _0 = 0
    case _90 = 90
    case _180 = 180
    case _270 = 270
}


open class ScreenSampleCapturer: RTCVideoCapturer, ScreenSampleCapturerDelegate {
    var kDownScaledFrameWidth = 360
    var kDownScaledFrameHeight = 960
    
    override init(delegate: RTCVideoCapturerDelegate) {
        super.init(delegate: delegate)
        let width = Int(UIScreen.main.bounds.width)
        let height = Int(UIScreen.main.bounds.height)
        kDownScaledFrameHeight = height * kDownScaledFrameWidth / width
        print("kDownScaledFrameWidth: \(kDownScaledFrameWidth) - kDownScaledFrameHeight: \(kDownScaledFrameHeight)")
    }
    
    func didCaptureVideo(sampleBuffer: CMSampleBuffer) {
        if sampleBuffer.numSamples != 1 || !sampleBuffer.isValid || !CMSampleBufferDataIsReady(sampleBuffer) {
            return
        }
        guard let pixelBuffer = sampleBuffer.imageBuffer else { return }
        var videoOrientation = VideoRotation._0
        guard let orientationAttachment = CMGetAttachment(sampleBuffer, key: RPVideoSampleOrientationKey as CFString, attachmentModeOut: nil) as? NSNumber else { return }
        let orientation: CGImagePropertyOrientation = CGImagePropertyOrientation(rawValue: orientationAttachment.uint32Value) ?? .up
        switch (orientation) {
        case .up, .upMirrored, .down, .downMirrored:
            videoOrientation = ._0
            break
        case .leftMirrored:
            videoOrientation = ._90
        case .left:
            videoOrientation = ._90
        case .rightMirrored:
            videoOrientation = ._270
        case .right:
            videoOrientation = ._270
        }
        let rotation = RTCVideoRotation(rawValue: videoOrientation.rawValue) ?? ._0


        var outPixelBuffer: CVPixelBuffer? = nil
        CVPixelBufferLockBaseAddress(pixelBuffer, [])

        let pixelFormat = CVPixelBufferGetPixelFormatType(pixelBuffer)

        if (pixelFormat != kCVPixelFormatType_420YpCbCr8BiPlanarFullRange) {
            print("Extension assumes the incoming frames are of type NV12")
            return
        }

        let status = CVPixelBufferCreate(kCFAllocatorDefault,
                                         Int(kDownScaledFrameWidth),
                                         Int(kDownScaledFrameHeight),
                                         pixelFormat,
                                         nil,
                                         &outPixelBuffer)
        if (status != kCVReturnSuccess) {
            print("Failed to create pixel buffer")
            return
        }

        CVPixelBufferLockBaseAddress(outPixelBuffer!, [])

        // Prepare source pointers.
        var sourceImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0),
                                         height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 0)),
                                         width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 0)),
                                         rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0))

        var sourceImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 1),
                                          height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(pixelBuffer, 1)),
                                          width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(pixelBuffer, 1)),
                                          rowBytes: CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 1))

        // Prepare out pointers.
        var outImageY = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 0),
                                      height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 0)),
                                      width: vImagePixelCount(CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 0)),
                                      rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 0))

        var outImageUV = vImage_Buffer(data: CVPixelBufferGetBaseAddressOfPlane(outPixelBuffer!, 1),
                                       height: vImagePixelCount(CVPixelBufferGetHeightOfPlane(outPixelBuffer!, 1)),
                                       width: vImagePixelCount( CVPixelBufferGetWidthOfPlane(outPixelBuffer!, 1)),
                                       rowBytes: CVPixelBufferGetBytesPerRowOfPlane(outPixelBuffer!, 1))


        var error = vImageScale_Planar8(&sourceImageY,
                                        &outImageY,
                                        nil,
                                        vImage_Flags(0))
        if (error != kvImageNoError) {
            print("Failed to down scale")
            return
        }

        error = vImageScale_CbCr8(&sourceImageUV,
                                  &outImageUV,
                                  nil,
                                  vImage_Flags(0));
        if (error != kvImageNoError) {
            print("Failed to down scale")
            return
        }
        CVPixelBufferUnlockBaseAddress(outPixelBuffer!, [])
        CVPixelBufferUnlockBaseAddress(pixelBuffer, [])
        if outPixelBuffer == nil { return }
        let timeStamp = Int64(CMTimeGetSeconds(CMSampleBufferGetPresentationTimeStamp(sampleBuffer)) * 1000000000)
        let rtcPixelBuffer = RTCCVPixelBuffer(pixelBuffer: outPixelBuffer!)
        let videoFrame = RTCVideoFrame(buffer: rtcPixelBuffer, rotation: rotation, timeStampNs: timeStamp)
        delegate?.capturer(self, didCapture: videoFrame)
    }
}

[fukemy]

kDownScaledFrameWidth and kDownScaledFrameHeight were calculated based on MEMORY problem( special to VP8)

[Meonardo]

I didnt mean the screen capturer, Its about render the pixel buff in AVSampleBufferDisplayLayer .
I can get RTCCVPixelBuffer directly from WebRTC when the codec is H.264, but when I change the codec to VP8 I get RTCI420Buffer, so there are some convert tasks to do.

[fukemy]

hi Meonardo, i just found big problem. The camera stopped right after app go background, Im so sad, but I think this current way can not work, because we can not do anything when camera stopped capture

About information: When app still active then PIP active, all work good(below picture is the "Hi" action for you)

But when app go to background the camera stopped, may be it's the WebRTC rule

[fukemy]

I think the problem is not have multitasking camera access profile from Apple.
https://developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_avfoundation_multitasking-camera-access

May I will tried to register again

[Meonardo]

Hi back!
Yes, you're right! I have not got the profile yet, so the camera capture session will pause when the app go to background.
Usually, we want the remote peer to show in the PiP window only just like making a FaceTime call.