Custom video capture

Overview

The ARTC SDK supports custom video capture, letting you manage video capture devices to meet specific business needs.

Use the SDK's built-in capture by default. If it does not meet your requirements for video quality, device compatibility, or the capture workflow, custom video capture provides greater extensibility.

Sample code

Android custom video capture: Android/ARTCExample/AdvancedUsage/src/main/java/com/aliyun/artc/api/advancedusage/CustomVideoCaptureAndRender/CustomVideoCaptureActivity.java.

iOS custom video capture: iOS/ARTCExample/AdvancedUsage/CustomVideoCapture/CustomVideoCaptureVC.swift.

Prerequisites

Before you begin, make sure that:

You have an Alibaba Cloud account, created an ARTC application, and obtained an AppID and AppKey. Create an application. View your credentials in the ApsaraVideo Live console.
You have integrated the ARTC SDK and implemented basic audio and video calls. Integrate the ARTC SDK. Implement an audio and video call.

Implementation

1. Disable internal capture

Call enableLocalVideo to disable the SDK's built-in capture.

Android

/* Disable internal capture. */
mAliRtcEngine.enableLocalVideo(false);

/* Enable internal capture. This feature is enabled by default. */
mAliRtcEngine.enableLocalVideo(true);

iOS

/* Disable internal capture. */
[_engine enableLocalVideo:NO];

/* Enable internal capture. This feature is enabled by default. */
[_engine enableLocalVideo:YES];

Mac

/* Disable internal capture. */
[self.engine enableLocalVideo:NO];

/* Enable internal capture. This feature is enabled by default. */
[self.engine enableLocalVideo:YES];

Windows

/* Disable internal capture. */
mAliRtcEngine->EnableLocalVideo(false);

/* Enable internal capture. This feature is enabled by default. */
mAliRtcEngine->EnableLocalVideo(true);

2. Set an external video source

Call setExternalVideoSource to configure an external video source. Key parameters:

enable: Enables or disables the external video source.
useTexture: Uses texture input instead of raw data.
streamType: The stream to replace, such as a camera stream or screen-sharing stream.
renderMode: The scaling mode applied when the external video aspect ratio does not match the publishing profile.

Android

YUV input

/* YUV input */
/* Enable external capture. This example uses a camera stream. You can adjust the sourceType and renderMode as needed. */
mAliRtcEngine.setExternalVideoSource(true,false, AliRtcVideoTrackCamera,AliRtcRenderModeAuto );

/* Disable external capture. This example uses a camera stream. You can adjust the sourceType as needed. */
mAliRtcEngine.setExternalVideoSource(false,false, AliRtcVideoTrackCamera,AliRtcRenderModeAuto );

Texture input

/* Texture input */
/* Enable external capture. This example uses a camera stream. You can adjust the sourceType and renderMode as needed. */
mAliRtcEngine.setExternalVideoSource(true,true, AliRtcVideoTrackCamera,AliRtcRenderModeAuto );

/* Disable external capture. This example uses a camera stream. You can adjust the sourceType as needed. */
mAliRtcEngine.setExternalVideoSource(false,true, AliRtcVideoTrackCamera,AliRtcRenderModeAuto );

iOS

/* Enable external capture. This example uses a camera stream. You can adjust the sourceType and renderMode as needed. */
[_engine setExternalVideoSource:YES sourceType:AliRtcVideosourceCameraType renderMode:AliRtcRenderModeAuto]; 

/* Disable external capture. This example uses a camera stream. You can adjust the sourceType as needed. */
[_engine setExternalVideoSource:NO sourceType:AliRtcVideosourceCameraType renderMode:AliRtcRenderModeAuto];

Mac

/* Enable external capture. This example uses a camera stream. You can adjust the sourceType and renderMode as needed. */
[self.engine setExternalVideoSource:YES sourceType:AliRtcVideosourceCameraType renderMode:AliRtcRenderModeAuto];

/* Disable external capture. This example uses a camera stream. You can adjust the sourceType as needed. */
[self.engine setExternalVideoSource:NO sourceType:AliRtcVideosourceCameraType renderMode:AliRtcRenderModeAuto];

Windows

/* Texture input */
/* Enable external capture. This example uses a camera stream. You can adjust the sourceType and renderMode as needed. */
mAliRtcEngine->SetExternalVideoSource(true,true, AliEngineVideoTrackCamera,AliEngineRenderModeAuto );

/* Disable external capture. This example uses a camera stream. You can adjust the sourceType as needed. */
mAliRtcEngine->SetExternalVideoSource(false,true, AliEngineVideoTrackCamera,AliEngineRenderModeAuto );

3. Push video frames to the SDK

After capturing video frames, push them to the ARTC SDK by calling pushExternalVideoFrame.

Android

YUV input example:

/* This example uses the YUV (I420) format. */
int width = 720;
int height = 1280;
AliRtcEngine.AliRtcVideoFormat videoformat = AliRtcEngine.AliRtcVideoFormat.AliRtcVideoFormatI420;
int[] lineSize = {width, width / 2, width / 2, 0};
int frameLength = width * height * 3 / 2;
byte[] buffer = new byte[frameLength];

/* Construct the data object to pass to the SDK. */
AliRtcEngine.AliRtcRawDataFrame rawDataFrame
        = new AliRtcEngine.AliRtcRawDataFrame(buffer,
                                             videoformat,
                                             width, 
                                             height, 
                                             lineSize, 
                                             0, 
                                             buffer.length);

/* Call the API to push the data object. */
int ret = mAliRtcEngine.pushExternalVideoFrame(rawDataFrame, AliRtcVideoTrackCamera);
if (ret != 0) {
    /* Handle the error. */
}

Texture input example:

/* This example uses the texture format. */
/* Create the OpenGL environment. */
private static EglBase14 createEglBase14(EGLContext shareEglContext) {
    if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.JELLY_BEAN_MR1) {
        EglBase14.Context eglBase14Context = shareEglContext == null ? null : new EglBase14.Context(shareEglContext);
        EglBase14 eglBase14 = new EglBase14(eglBase14Context, EglBase.CONFIG_PIXEL_BUFFER);
        try {
            eglBase14.createDummyPbufferSurface();
            eglBase14.makeCurrent();
        } catch (RuntimeException e) {
            Log.e(TAG, "CreateEGLBase14Context, failed, " + e.getMessage());
        }
        return eglBase14;
    }
    return null;
}

/* Construct the context for the input data. */
float[] transformMatrix = {
  1, 0, 0, 0,
  0, 1, 0, 0,
  0, 0, 1, 0,
  0, 0, 0, 1
};
int frameWidth = 720; /* Image width. */
int frameHeight = 1280; /* Image height. */
int textureID = xxx; /* Your texture ID. */

AliRtcEngine.AliRtcRawDataFrame aliRawDataFrame 
  = new AliRtcEngine.AliRtcRawDataFrame(textureID,
                                        AliRtcVideoFormatTexture2D,
                                        frameWidth, 
                                        frameHeight, 
                                        transformMatrix, 
                                        0, 
                                        0, 
                                        frameWidth, 
                                        frameHeight, 
                                        mEglBase14.getEglContext());

/* Call the API to push the data object. */
int ret = mAliRtcEngine.pushExternalVideoFrame(aliRawDataFrame, AliRtcVideoTrackCamera);
if (ret != 0) {
    /* Handle the error. */
}

iOS

/* Specify the following parameters based on your data format. This example uses the I420 format. */
AliRtcVideoDataSample *dataSample = [[AliRtcVideoDataSample alloc] init];
dataSample.dataPtr = (long)yuv_read_data;
dataSample.format = AliRtcVideoFormat_I420;
dataSample.type = AliRtcBufferType_Raw_Data;
dataSample.width = width;
dataSample.height = height;
dataSample.strideY = width;
dataSample.strideU = width/2;
dataSample.strideV = width/2;
dataSample.dataLength = dataSample.strideY * dataSample.height * 3/2;

/* Call the API to push the data object. */
int ret = [self.engine pushExternalVideoFrame:dataSample sourceType:AliRtcVideosourceCameraType];
if (ret != 0) {
    /* Handle the error. */
}

Mac

/* Specify the following parameters based on your data format. This example uses the I420 format. */
AliRtcVideoDataSample *dataSample = [[AliRtcVideoDataSample alloc] init];
dataSample.dataPtr = (long)yuv_read_data;
dataSample.format = AliRtcVideoFormat_I420;
dataSample.type = AliRtcBufferType_Raw_Data;
dataSample.width = width;
dataSample.height = height;
dataSample.strideY = width;
dataSample.strideU = width/2;
dataSample.strideV = width/2;
dataSample.dataLength = dataSample.strideY * dataSample.height * 3/2;

/* Call the API to push the data object. */
int ret = [self.engine pushExternalVideoFrame:dataSample sourceType:AliRtcVideosourceCameraType];
if (ret != 0) {
    /* Handle the error. */
    if ( ret == AliRtcErrAudioBufferFull ) {
       // The buffer is full. Wait 20 ms and retry.
        [NSThread sleepForTimeInterval:0.02] ;
        continue ;
    } 
    break ;
}

Windows

AliEngineVideoRawData sample;
sample.dataPtr = (unsigned char*)cache_buf;
sample.format = AliEngineVideoFormatI420;
sample.width = mIYuvWidth;
sample.height = mIYuvHeight;
sample.strideY = mIYuvWidth;
sample.strideU = mIYuvWidth / 2;
sample.strideV = mIYuvWidth / 2;
sample.dataLength = frame_length;
sample.rotation = 0;


    /* Call the API to push the data object. */
    int ret = mAliRtcEngine->PushExternalVideoFrame(sample, AliEngineVideoTrackScreen);
    if ( ret != 0 ) {
        if ( ret == AliEngineErrorAudioBufferFull ) {
            Sleep(20);
            continue;
        }else {
            // Handle the error.
            break ;
        }
    } else {
        
    }