swift – iOS display screen seize RPBroadcastSampleHandler callbacks some unhealthy video frames (yuv nv12) when the display screen fps is excessive?


I am engaged on display screen capturing and recording on iOS, it is based mostly on the ReplayKit and RPBroadcastSampleHandler, the video and audio body was callback in processSampleBuffer:withType:. it outcomes good if the display screen was nonetheless, however when it strikes, there might be some unhealthy frames callback in. it is just like the body was splitted.

I merely dump the body and return, and the dump is async to keep away from block.
the code was quite simple:



extension CVPixelBuffer{
    func asyncDump(_ f:UnsafeMutablePointer<FILE>){
        NSLog("dump a body.")
        let planeCount = CVPixelBufferGetPlaneCount(self)
        for airplane in 0 ..< planeCount {
            let supply      = CVPixelBufferGetBaseAddressOfPlane(self, airplane)
            let top      = CVPixelBufferGetHeightOfPlane(self, airplane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, airplane)
            let planeSize   = top * bytesPerRow

            let rawFrame = malloc(planeSize)
            memcpy(rawFrame, supply, planeSize)
            let f = f
            DispatchQueue.international(qos: .default).async{
                fwrite(rawFrame,1, planeSize,f)
                free(rawFrame);
                NSLog("dump a body performed.")
            }
        }
    }
}

    override func processSampleBuffer(_ sampleBuffer: CMSampleBuffer, with sampleBufferType: RPSampleBufferType) {
        if (DUMP_YUV){
            if RPSampleBufferType.video != sampleBufferType{
                return
            }
        }
        swap sampleBufferType {
        case RPSampleBufferType.video:


            // Deal with video pattern buffer
            videoFrames+=1
            if let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) {
                let planeCount = CVPixelBufferGetPlaneCount(imageBuffer)
                let top = UInt32(CVPixelBufferGetHeight(imageBuffer))
                let width =  UInt32(CVPixelBufferGetWidth(imageBuffer))
                printFPS("video-(width)x(top)")
                CVPixelBufferLockBaseAddress(imageBuffer, [.readOnly])
                defer { CVPixelBufferUnlockBaseAddress(imageBuffer, [.readOnly]) }
                
                if (DUMP_YUV){
                    if (videoFrames % 150 == 0){// dump 1 in 150 to keep away from an excessive amount of io.
                        if let f = f{
                            imageBuffer.asyncDump(f)
                        }
                    }
                    return
                }
...


one of many unhealthy frames is within the attachment.
I additionally discovered when the unhealthy body callback in, the callback framerate was very excessive to 90fps, and there was no methodology or parameter to configure this body fee.
it’s totally confused, and really much less informations about this on the web.

See also  ios - API to push information to HealthKit

Leave a Reply