Access and process raw video data on Android and iOS - ApsaraVideo Live

During a real-time communication (RTC) session, you can process raw video data from the SDK or access it at different stages of the processing pipeline. This topic explains how to use video observers to access and process this data.

Use cases

Common use cases for custom video data processing include:

Custom retouching: Pull the local video stream and process it using a third-party retouching SDK before sending it to other users in the channel.
Custom video editing: Edit the video stream at different stages of the processing pipeline, such as adding special effects, watermarks, cropping, or clipping. The modified stream can then be broadcast or saved.
Stream preview or monitoring: Pull the video stream at different stages of the processing pipeline for local or cloud-based monitoring and preview. You can monitor the content, quality, and status of the video stream in real time to ensure proper transmission and playback.

Before you begin

Ensure you have completed the following preparations:

Create an ARTC application and get the AppID and AppKey from ApsaraVideo Live console.
Integrate the ARTC SDK into your project and implement basic audio and video call features.

How it works

Video capture:

Decoding and rendering:

The ARTC SDK provides observation points at different stages of the video processing pipeline.

The observation points correspond to the following callbacks:

Observation point 1: Get the original, unscaled video frame data through onCaptureVideoSample (iOS) or onLocalVideoSample (Android) right after capture.
Observation point 2: Get pre-encoding data through onPreEncodeVideoSample.
Observation point 3: Get decoded data through onRemoteVideoSample.
Observation point 4: Get the OpenGL context (glcontext) through onTextureCreate and bind it to your retouching module.
Observation point 5: Receive the updated texture data through onTextureUpdate to apply your effects.
Observation point 6: Receive a notification to release glcontext through onTextureDestroy.

Note

For observation points 1, 2, and 3, returning true indicates that you have modified the data. Returning false leaves the original data unchanged.
For observation point 5, the method must return a valid textureId. Returning a new textureId tells the SDK to use your modified texture for encoding and processing. If you do not perform any custom processing, return the original textureId.

Implementation

Android

Video frame callbacks

1. Register callbacks

To receive raw video data, implement the AliRtcEngine.AliRtcVideoObserver interface and register your instance by calling the registerVideoSampleObserver() method.

public abstract void registerVideoSampleObserver(AliRtcVideoObserver observer);

2. Specify observation points

The SDK determines which callbacks to trigger based on the bitmask returned by this method.

 public enum AliRtcVideoObserPosition{
        /*! Locally captured video data, corresponds to the onLocalVideoSample callback */
        AliRtcPositionPostCapture(1),
        /*! Decoded remote video data, corresponds to the onRemoteVideoSample callback */
        AliRtcPositionPreRender(2),
        /*! Pre-encoding video data, corresponds to the onPreEncodeVideoSample callback */
        AliRtcPositionPreEncoder(4);
    }

public int onGetObservedFramePosition(){
    return AliRtcVideoObserPosition.AliRtcPositionPostCapture.getValue() | AliRtcVideoObserPosition.AliRtcPositionPreRender.getValue();
}

3. Specify the output format

/**
 * @brief Video data format
 */
public enum AliRtcVideoFormat{
        AliRtcVideoFormatUNKNOW(-1),
        AliRtcVideoFormatBGRA(0),
        AliRtcVideoFormatI420(1),
        AliRtcVideoFormatNV21(2),
        AliRtcVideoFormatNV12 (3),
        AliRtcVideoFormatRGBA(4),
        AliRtcVideoFormatI422 (5),
        AliRtcVideoFormatARGB(6),
        AliRtcVideoFormatABGR (7),
        AliRtcVideoFormatRGB24(8),
        AliRtcVideoFormatBGR24(9),
        AliRtcVideoFormatRGB565(10),
        AliRtcVideoFormatTextureOES(11),
        AliRtcVideoFormatTexture2D(12),
        AliRtcVideoFormatH264(13),
        AliRtcVideoFormatH265(14),
        AliRtcVideoFormatFile(15);
};

/*
* The SDK calls this method after you call AliRtcEngine::registerVideoSampleObserver.
*/

/**
* @brief Video data output format
* @return The desired video output format. See {@link AliRtcVideoFormat}.
*/
public AliRtcVideoFormat onGetVideoFormatPreference(){
    return AliRtcVideoFormat.AliRtcVideoFormatI420;
}

4. Specify the memory alignment

public enum AliRtcVideoObserAlignment{
        /*! Keep the original video width (default) */
        AliRtcAlignmentDefault(0),
        /*! Align the stride to a multiple of 2 bytes */
        AliRtcAlignmentEven(1),
        /*! Align the stride to a multiple of 4 bytes */
        AliRtcAlignment4(2),
        /*! Align the stride to a multiple of 8 bytes */
        AliRtcAlignment8(3),
        /*! Align the stride to a multiple of 16 bytes */
        AliRtcAlignment16(4);
};

/*
* The SDK calls this method after you call AliRtcEngine::registerVideoSampleObserver.
*/

public int onGetVideoAlignment(){
  return AliRtcVideoObserAlignment.AliRtcAlignmentDefault.getValue();
}

5. Specify whether to apply a mirror effect

public boolean onGetObserverDataMirrorApplied(){
  return false;
}

6. Implement the callbacks

/*
* Return true to write your modifications back to the SDK's processing pipeline. 
* This is required if you modify AliRtcVideoSample.data.
*/

@Override
public boolean onLocalVideoSample(AliRtcEngine.AliRtcVideoSourceType sourceType, AliRtcEngine.AliRtcVideoSample videoSample) {
  boolean ret = false;
  /*
  * Process locally captured data.
  */
  return ret;
}

@Override
public boolean onRemoteVideoSample(String userId, AliRtcEngine.AliRtcVideoSourceType sourceType, AliRtcEngine.AliRtcVideoSample videoSample) {
  /*
  * Process the decoded data from a remote user.
  */
  return false;
}

@Override
public boolean onPreEncodeVideoSample(AliRtcEngine.AliRtcVideoSourceType aliVideoSourceType, AliRtcEngine.AliRtcVideoSample videoSample) {
  boolean ret = false;
  /*
  * Process pre-encoding data.
  */
  return false ;
}

7. Unregister the callback

When you no longer need to observe video frames, unregister the observer to avoid unnecessary processing.

unregisterVideoSampleObserver

Texture processing

All texture-related callbacks are invoked on the same dedicated OpenGL thread. To use them, implement the AliRtcTextureObserver interface.

Prerequisites

Enable texture capture and texture encoding by setting the following options when you initialize the engine.

String extras = "{\"user_specified_camera_texture_capture\":\"TRUE\",\"user_specified_texture_encode\":\"TRUE\" }";
_engine = AliRtcEngine.getInstance(getApplicationContext(), extras);

1. Register the texture callback

public abstract void registerLocalVideoTextureObserver(AliRtcTextureObserver observer);

2. Implement the callback

Callback after OpenGL context creation

@Override
public void onTextureCreate(long context) {
      context_ = context ;
      Log.d(TAG, "texture context: "+context_+" create!") ;
}

OpenGL texture update callback

@Override
public int onTextureUpdate(int textureId, int width, int height, AliRtcEngine.AliRtcVideoSample videoSample) {
    /*
    *  Process the textureid.
    */
     ++log_ref ;
     return textureId;
}

OpenGL context destruction callback

@Override
public void onTextureDestroy() {
     Log.d(TAG, "texture context: "+context_+" destory!") ;
}

3. Unregister the texture callback

public abstract void  unRegisterLocalVideoTextureObserver();

iOS