iOS live audio video collection technology sharing

1. The process of iOS live Technology

The process of live broadcast technology can be roughly divided into several steps: data acquisition, image processing (real-time filter), video coding, packet, upload, cloud (transcoding, recording, distribution), live player.

  • Data acquisition: obtain real-time audio and video data through camera and microphone;
  • Image processing: filter the input stream of data acquisition in real time to get our beautified video frame;
  • Video coding: the coding is divided into soft coding and hard coding. At present, the general encoding method is H.264, the newer H.265 is said to have a higher compression rate, but the algorithm is also quite complex and not widely used. Soft coding uses CPU for coding. Hard coding uses GPU for coding. Soft coding supports all current system versions. Apple only opens hard coding API in IOS 8, so hard coding only supports systems above IOS 8;
  • Packet: now in live streaming, FLV is the general format;
  • Upload: RTMP is commonly used for streaming;
  • Cloud: transcode, distribute and record streams;
  • Live player: responsible for streaming, decoding and playing.

2. Get authorization for the system

The first step of live broadcast is to collect data, including video and audio data. Due to the requirements of iOS permission, you need to obtain the permission to access the camera and microphone first:

Request access to camera

__weak typeof(self) _self = self;
    AVAuthorizationStatus status = [AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeVideo];
    switch (status) {
        case AVAuthorizationStatusNotDetermined:{
            // License dialog does not appear, initiate license
            [AVCaptureDevice requestAccessForMediaType:AVMediaTypeVideo completionHandler:^(BOOL granted) {
                if (granted) {
                    dispatch_async(dispatch_get_main_queue(), ^{
                        [_self.session setRunning:YES];
                    });
                }
            }];
            break;
        }
        case AVAuthorizationStatusAuthorized:{
            // Authorization has been turned on, can continue
            [_self.session setRunning:YES];
            break;
        }
        case AVAuthorizationStatusDenied:
        case AVAuthorizationStatusRestricted:
            // The user explicitly denies authorization or the camera device cannot access it
            break;
        default:
            break;
    }

Request access to microphone

AVAuthorizationStatus status = [AVCaptureDevice authorizationStatusForMediaType:AVMediaTypeAudio];
    switch (status) {
        case AVAuthorizationStatusNotDetermined:{
            [AVCaptureDevice requestAccessForMediaType:AVMediaTypeAudio completionHandler:^(BOOL granted) {
            }];
            break;
        }
        case AVAuthorizationStatusAuthorized:{
            break;
        }
        case AVAuthorizationStatusDenied:
        case AVAuthorizationStatusRestricted:
            break;
        default:
            break;
    }

3. Configure sampling parameters

Audio: bit rate and sampling rate need to be configured;
Video: video resolution, frame rate and code rate need to be configured.

4. Recording of audio and video

Recording of audio

self.taskQueue = dispatch_queue_create("com.1905.live.audioCapture.Queue", NULL);
       
       AVAudioSession *session = [AVAudioSession sharedInstance];
       [session setActive:YES withOptions:kAudioSessionSetActiveFlag_NotifyOthersOnDeactivation error:nil];
       
       [[NSNotificationCenter defaultCenter] addObserver: self
                                                selector: @selector(handleRouteChange:)
                                                    name: AVAudioSessionRouteChangeNotification
                                                  object: session];
       [[NSNotificationCenter defaultCenter] addObserver: self
                                                selector: @selector(handleInterruption:)
                                                    name: AVAudioSessionInterruptionNotification
                                                  object: session];
       
       NSError *error = nil;
       
       [session setCategory:AVAudioSessionCategoryPlayAndRecord withOptions:AVAudioSessionCategoryOptionDefaultToSpeaker | AVAudioSessionCategoryOptionMixWithOthers error:nil];
       
       [session setMode:AVAudioSessionModeVideoRecording error:&error];
       
       if (![session setActive:YES error:&error]) {
           [self handleAudioComponentCreationFailure];
       }
       
       AudioComponentDescription acd;
       acd.componentType = kAudioUnitType_Output;
       acd.componentSubType = kAudioUnitSubType_RemoteIO;
       acd.componentManufacturer = kAudioUnitManufacturer_Apple;
       acd.componentFlags = 0;
       acd.componentFlagsMask = 0;
       
       self.component = AudioComponentFindNext(NULL, &acd);
       
       OSStatus status = noErr;
       status = AudioComponentInstanceNew(self.component, &_componetInstance);
       
       if (noErr != status) {
           [self handleAudioComponentCreationFailure];
       }
       
       UInt32 flagOne = 1;
       
       AudioUnitSetProperty(self.componetInstance, kAudioOutputUnitProperty_EnableIO, kAudioUnitScope_Input, 1, &flagOne, sizeof(flagOne));
       
       AudioStreamBasicDescription desc = {0};
       desc.mSampleRate = _configuration.audioSampleRate;
       desc.mFormatID = kAudioFormatLinearPCM;
       desc.mFormatFlags = kAudioFormatFlagIsSignedInteger | kAudioFormatFlagsNativeEndian | kAudioFormatFlagIsPacked;
       desc.mChannelsPerFrame = (UInt32)_configuration.numberOfChannels;
       desc.mFramesPerPacket = 1;
       desc.mBitsPerChannel = 16;
       desc.mBytesPerFrame = desc.mBitsPerChannel / 8 * desc.mChannelsPerFrame;
       desc.mBytesPerPacket = desc.mBytesPerFrame * desc.mFramesPerPacket;
       
       AURenderCallbackStruct cb;
       cb.inputProcRefCon = (__bridge void *)(self);
       cb.inputProc = handleInputBuffer;
       status = AudioUnitSetProperty(self.componetInstance, kAudioUnitProperty_StreamFormat, kAudioUnitScope_Output, 1, &desc, sizeof(desc));
       status = AudioUnitSetProperty(self.componetInstance, kAudioOutputUnitProperty_SetInputCallback, kAudioUnitScope_Global, 1, &cb, sizeof(cb));
       
       status = AudioUnitInitialize(self.componetInstance);
       
       if (noErr != status) {
           [self handleAudioComponentCreationFailure];
       }
       
       [session setPreferredSampleRate:_configuration.audioSampleRate error:nil];
       
       
       [session setActive:YES error:nil];

Recording of video: calling GPUImageVideoCamera in GPUImage

_videoCamera = [[GPUImageVideoCamera alloc] initWithSessionPreset:_configuration.avSessionPreset cameraPosition:AVCaptureDevicePositionFront];
_videoCamera.outputImageOrientation = _configuration.orientation;
_videoCamera.horizontallyMirrorFrontFacingCamera = NO;
_videoCamera.horizontallyMirrorRearFacingCamera = NO;
_videoCamera.frameRate = (int32_t)_configuration.videoFrameRate;
        
_gpuImageView = [[GPUImageView alloc] initWithFrame:[UIScreen mainScreen].bounds];
[_gpuImageView setFillMode:kGPUImageFillModePreserveAspectRatioAndFill];
[_gpuImageView setAutoresizingMask:UIViewAutoresizingFlexibleWidth | UIViewAutoresizingFlexibleHeight];
        [_gpuImageView setInputRotation:kGPUImageFlipHorizonal atIndex:0];
Figure play intelligence provides free live source code for enterprises to maximize the cost of development and meet the needs of various scenarios of enterprises. For details, please visit www.toivan.com for consultation.
 

Tags: Programming Session iOS encoding

Posted on Tue, 10 Mar 2020 19:20:45 -0700 by gilgimech