The short video SDK provides the AliyunIMixRecorder class for you to record a duet that consists of an existing sample video and a video that is being taken by the camera. The two videos are arranged in the specified layout, such as left-right split-screen, up-down split-screen, or picture-in-picture (PiP). The duet recording feature is an upgraded version of the video recording feature, which adds a new local video track.

Supported editions

Edition Supported
Professional Yes
Standard Yes
Basic No

Terms

This section describes the terms that help you better understand the duet recording feature. For more information, see Duet recording, Track, and Track layout.

Related classes

Class Description
AliyunIMixRecorder A core class that defines recording features, including recording, preview settings, effect settings, and callback settings.
AliyunMixRecorderCreator A factory class that is used to create a duet recording instance.
AliyunMixMediaInfoParam A class that defines the parameters of duet recording, including the track arrangement parameter and the output path parameter.
AliyunMixRecorderDisplayParam A class that defines the tracks for duet recording, including the layout information and layout hierarchy of tracks.
MediaInfo A class that defines recording parameters, including the width and height of the video, encoder type, and frame rate.
RecordCallback A class that defines recording callbacks, including the recording completion callback, recording progress callback, and recording error callback.
OnFrameCallBack A class that defines callbacks for data capture, including the callback for the selected preview resolution, the callback for the frame data captured by the camera, and the callback for an error that occurs when the camera fails to be enabled.
OnAudioCallBack A class that defines callbacks for audio data in PCM format.
AliyunIClipManager A class that manages video clips, for example, deletes video clips and configures the recording duration.

Duet recording process

Note The camera and microphone permissions are required for duet recording. Otherwise, the recording fails.

The procedure to configure duet recording is similar to that to configure video recording. The main difference lies in the input and output parameters and preview settings that you need to configure.

Configuration Step Description Sample code
Basic configurations 1 Create and destroy recording instances and configure recording parameters. Initialize the AliyunIMixRecorder class and configure parameters
2 Configure callbacks. Configure callbacks
3 Configure a preview view and enable the preview. Enable the preview
4 Start, cancel, or stop recording a video clip. Start recording
5 Stop recording and generate configuration information. Stop recording
Advanced configurations 6 Configure parameters for camera control and video clip management. Parameters for camera control include the camera type and flash mode. The parameters for video clip management include the maximum or minimum recording duration, how to delete a video clip, and how to query the number of video clips. Configure these parameters based on your business requirements. Configure cameras and manage video clips
7 Configure recording effects such as retouching and filters. Configure effects
8 Configure photo taking or facial recognition. Configure other features

Initialize the AliyunIMixRecorder class and configure parameters

Initialize the AliyunIMixRecorder class, create a recording instance, and configure recording parameters. For more information about the parameters that are used in the code, see Related classes.

Initialize the class
// Create a recording instance.
AliyunIMixRecorder recorder = AliyunMixRecorderCreator.createAlivcMixRecorderInstance(context);

// Destroy the recording instance.
// Destroy the recording instance if you no longer need to use the SDK, or before you exit the program. Do not destroy the recording instance during usage.
AliyunIMixRecorder.release();
Configure recording parameters
// Configure the quality of a recorded video.
AliyunIMixRecorder.setVideoQuality(quality);

// Configure the bitrate of the recorded video.
AliyunIMixRecorder.setVideoBitrate(int bitrate);// Unit: Kbit/s.

// Configure output parameters for the recorded video.
// inputMediaInfo indicates the parameters of the produced video, including track arrangement and output video. outputInfo indicates the information about the output video.
AliyunIMixRecorder.setMixMediaInfo(AliyunMixMediaInfoParam inputMediaInfo, MediaInfo outputInfo);// For more information, see AliyunMixMediaInfoParam and MediaInfo in the SDK references.

// Configure the output path of the recorded video.
AliyunIMixRecorder.setOutputPath(String path);

// Set the output group of pictures (GOP) of the recorded video.
AliyunIMixRecorder.setGop(int gop);// Unit: frames.

Configure callbacks

You can configure callbacks to obtain the processing progress and status of audio and video in a timely manner. For more information about the parameters that are used in the code, see Related classes.

// Configure the recording callback.
AliyunIMixRecorder.setRecordCallBack(RecordCallback callBack);

// Configure the callback for video frame collection.
AliyunIMixRecorder.setOnFrameCallback(OnFrameCallBack callback);

// Configure the audio collection callback.
AliyunIMixRecorder.setOnAudioCallback(OnAudioCallBack callback);

Enable the preview

During preview, you must configure SurfaceView. At the same time, startPreview is performed in onResume() of Activity/Fragment, and stopPreview is performed in onPause(). For more information about the parameters that are used in the code, see Related classes.

// Configure the preview View.
// The cameraView parameter is the view of the camera, and the videoView parameter is the view of the local video.
AliyunIMixRecorder.setDisplayView(SurfaceView cameraView, Surface videoView);

// Start the preview.
AliyunIMixRecorder.startPreview();

// Stop the preview.
// Perform stopPreview in onPause() of Activity/Fragment.
AliyunIMixRecorder.stopPreview();

Start recording

In the actual recording process, you may need to stop, cancel, and re-record the video until you record the complete video that you want. If you stop recording, a video clip is generated. However, if you cancel recording, the current video clip is not retained. For more information about the parameters that are used in the code, see Related classes.

Start recording
// Start recording.
AliyunIMixRecorder.startRecording();
Record a video clip
// Start recording.
AliyunIMixRecorder.startRecording();

// Stop recording. A video clip is generated.
AliyunIMixRecorder.stopRecording();
AliyunIMixRecorder.startRecording();
// Cancel recording. The current video clip is not saved.
AliyunIMixRecorder.cancelRecording();

// Continue to record the next video clip.
AliyunIMixRecorder.startRecording();
AliyunIMixRecorder.stopRecording();

Stop recording

If you stop recording, the recorded video clips are merged into a video or the configuration information about the recorded video clips is generated. For more information about the parameters that are used in the code, see Related classes.

// Stop recording and merge the recorded video clips into one video.
AliyunIMixRecorder.finishRecording();

// Stop recording and generate the configuration information about the recorded video clips without merging the video clips.
AliyunIMixRecorder.finishRecordingForEdit();

Configure cameras and manage video clips

This section describes how to configure parameters for camera control and video clip management. Parameters for camera control include the camera type and flash mode. The parameters for video clip management include the maximum or minimum recording duration, how to delete a video clip, and how to query the number of video clips. Configure these parameters based on your business requirements. For more information about the parameters that are used in the code, see Related classes.

Configure cameras
// Obtain the number of cameras.
AliyunIMixRecorder.getCameraCount();

// Configure the camera type.
AliyunIMixRecorder.setCamera(cameraType);

// Specify whether to mute the audio during recording.
AliyunIMixRecorder.setMute(boolean isMute);

// Configure the angle of the sensor.
// This configuration is very important. We recommend that you read the SDK references. 
AliyunIMixRecorder.setRotation(int rotation);

// Configure the angle at which a video is recorded.
// This configuration is very important. We recommend that you read the SDK references. 
AliyunIMixRecorder.setRecordRotation(int rotation);

// Configure the camera parameters during preview, including the flash mode, focus mode, zoom factor, and exposure level. You can also configure the preview parameters separately by using the following methods:
AliyunIMixRecorder.setCameraParam(CameraParam cameraParam);

// Switch between the front and rear cameras.
AliyunIMixRecorder.switchCamera();

// Configure the flash mode.
AliyunIMixRecorder.setLight(FlashType flashType);

// Configure the zoom factor. 
AliyunIMixRecorder.setZoom(float rate);

// Configure the exposure level. 
AliyunIMixRecorder.setExposureCompensationRatio(float value);

// Configure the focus mode. 
AliyunIMixRecorder.setFocusMode(int mode);

// Configure manual focus. 
AliyunIMixRecorder.setFocus(float xRatio, float yRatio);
Manage video clips
// Obtain the clip manager.
AliyunIClipManager manager = AliyunIRecorder.getClipManager();

// Configure the maximum recording duration, which is the total duration of all video clips instead of a single video clip.
manager.setMaxDuration(int maxDurationMs);

// Configure the minimum recording duration, which is the total duration of all video clips instead of a single video clip.
manager.setMinDuration(int minDurationMs);

// Delete the last video clip.
manager.deletePart();

// Delete the specified video clip.
manager.deletePart(int index);

// Delete all video clips.
manager.deleteAllPart();

// Obtain the total duration of video clips.
manager.getDuration();

// Obtain the total number of video clips.
manager.getPartCount();

// Obtain the paths of video clips.
manager.getVideoPathList();

Configure effects

This section describes how to configure recording effects such as retouching and filters. For more information about the parameters that are used in the code, see Related classes.

Filters

You can create custom filters. For more information, see Filters and transitions.

// Apply a filter.
AliyunIMixRecorder.applyFilter(effectFilter);

// Remove a filter.
// If the setting is the same as the following, the filter effect is removed.
AliyunIMixRecorder.applyFilter(new EffectFilter(null));
Animated filters
// Apply an animated filter.
AliyunIMixRecorder.applyAnimationFilter(effectFilter);

// Remove an animated filter.
AliyunIMixRecorder.removeAnimationFilter(effctFilter);
Recording speed
// Configure the recording speed.
AliyunIMixRecorder.setRate(float rate);
Static stickers and static watermarks
// Add a static watermark or a static sticker.
AliyunIMixRecorder.addImage(effctImage);

// Remove a static watermark or a static sticker.
AliyunIMixRecorder.removeImage(effctImage);

// Update the position of a static watermark or a static sticker.
AliyunIMixRecorder.setEffectView(float xRatio,float yRatio,float widthRatio,float heightRatio,EffectBase effectBase);

Animated stickers

You can create custom animated stickers. For more information, see Animation.
// Add an animated sticker.
AliyunIMixRecorder.addPaster(effectPaster,float sx,float sy,float sw,float sh,float rotation,boolean flip);

// Remove an animated sticker.
AliyunIMixRecorder.removePaster(effectPaster);

// Update the position of an animated sticker.
AliyunIMixRecorder.setEffectView(float xRatio,float yRatio,float widthRatio,float heightRatio,effectBase);

Advanced retouching

The video recording module not only provides basic built-in retouching features, but also supports the use of retouching SDKs, such as Alibaba Cloud Queen SDK and FaceUnity. You can use built-in retouching features to configure only the retouching level. The retouching SDKs provide abundant retouching effects, such as face retouching, face shaping, makeup, filters, and stickers.
  • Built-in retouching
    // Enable or disable retouching.
    AliyunIMixRecorder.setBeautyStatus(boolean on);
    
    // Configure the retouching level.
    AliyunIMixRecorder.setBeautyLevel(int level);
  • Retouching SDKs
    To use effects provided by a retouching SDK in the short video SDK, you must first obtain the permissions on the retouching SDK and integrate the retouching SDK into the short video SDK.
    • For more information about how to integrate Alibaba Cloud Queen SDK, see Queen SDK. For more information about the sample code for configuring effects, see Sample code.
    • For more information about how to purchase, integrate, and use FaceUnity, see FaceUnity.
    The camera texture ID and the camera raw frame are required for retouching SDKs to implement retouching effects. The following sample code provides an example on how to obtain the camera texture ID and the camera raw frame.
    • Obtain the camera texture ID
      AliyunIMixRecorder.setOnTextureIdCallback(new OnTextureIdCallBack() {
                  @Override
                  public int onTextureIdBack(int textureId, int textureWidth, int textureHeight, float[] matrix) {
                      if (mBeautyInterface != null) {
                          return mBeautyInterface.onTextureIdBack(textureId, textureWidth, textureHeight, matrix, mControlView.getCameraType().getType());
                      }
                      return textureId;
                  }
      
                  @Override
                  public int onScaledIdBack(int scaledId, int textureWidth, int textureHeight, float[] matrix) {
      
                      return scaledId;
                  }
      
                  @Override
                  public void onTextureDestroyed() {
                      // For a version of the short video SDK that is earlier than V3.7.8, you can destroy gl resources for third-party custom rendering by using GLSurfaceView.queueEvent in GLSurfaceView. For a version of the short video SDK that is V3.7.8 or later than V3.7.8, we recommend that you destroy gl resources in this callback.
                      if (mBeautyInterface != null) {
                          mBeautyInterface.release();
                          mBeautyInterface = null;
                      }
                  }
              });
    • Obtain the camera raw frame
      AliyunIMixRecorder.setOnFrameCallback(new OnFrameCallBack() {
                  @Override
                  public void onFrameBack(byte[] bytes, int width, int height, Camera.CameraInfo info) {
                      // The data type of the callback raw data is NV21. The obtained raw data is used by FaceUnity.
                      if (mBeautyInterface != null) {
                          mBeautyInterface.onFrameBack(bytes, width, height, info);
                      }
                  }
      
                  @Override
                  public Camera.Size onChoosePreviewSize(List<Camera.Size> supportedPreviewSizes,
                                                         Camera.Size preferredPreviewSizeForVideo) {
                      return null;
                  }
      
                  @Override
                  public void openFailed() {
                      
                  }
              });

Configure other features

Photo taking and facial recognition are supported during video recording. For more information about the parameters that are used in the code, see Related classes.

Photo taking

You can take a photo with effects enabled or by using the default camera settings. The default camera settings do not support effects. After a photo is taken, the data is returned by using RecordCallback.onPictureBack(Bitmap) or RecordCallback.onPictureDataBack(byte[]).
// Take a photo with effects enabled.
AliyunIMixRecorder.takePhoto(boolean needBitmap);

// Take a photo by using the default camera settings. The default camera settings do not support effects.
AliyunIMixRecorder.takePicture(boolean needBitmap);

// Configure the size of a photo. This method is supported only if you take a photo by using the default camera settings.
AliyunIMixRecorder.setPictureSize(Camera.Size size);

Facial recognition

If you want to use facial recognition, you must access the built-in facial recognition model of your application. You can download the Facial recognition model demo for reference.
// Specify whether to enable facial recognition.
AliyunIMixRecorder.needFaceTrackInternal(boolean need);

// Configure the path of the facial recognition model file.
AliyunIMixRecorder.setFaceTrackInternalModelPath(String path);

// Configure the angle at which facial recognition is performed.
// This configuration is very important. We recommend that you read the SDK references. 
AliyunIMixRecorder.setFaceDetectRotation(int rotation);

// Configure the maximum number of faces that can be recognized at the same time.
// Configure the maximum number of faces that can be recognized at the same time. Maximum value: 3.
AliyunIMixRecorder.setFaceTrackInternalMaxFaceCount(int maxFaceCount);

// Add an animated face sticker.
AliyunIMixRecorder.addPaster(EffectPaster effectPaster);

Sample code for duet recording

import android.graphics.Bitmap
import android.os.Bundle
import android.util.Log
import android.view.SurfaceView
import android.widget.ImageView
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import com.aliyun.svideosdk.common.struct.common.VideoDisplayMode
import com.aliyun.svideosdk.common.struct.encoder.VideoCodecs
import com.aliyun.svideosdk.common.struct.recorder.MediaInfo
import com.aliyun.svideosdk.mixrecorder.AliyunIMixRecorder
import com.aliyun.svideosdk.mixrecorder.AliyunMixMediaInfoParam
import com.aliyun.svideosdk.mixrecorder.AliyunMixRecorderDisplayParam
import com.aliyun.svideosdk.mixrecorder.AliyunMixTrackLayoutParam
import com.aliyun.svideosdk.mixrecorder.impl.AliyunMixRecorderCreator
import com.aliyun.svideosdk.recorder.RecordCallback


/**
 * Sample code for video recording
 */
class MixRecordActivity : AppCompatActivity() {

    enum class RecordStatus {
        Idle,
        Recording
    }

    private lateinit var mAliyunRecord : AliyunIMixRecorder
    private lateinit var mVideoPreviewView : SurfaceView
    private lateinit var mCameraPreviewView : SurfaceView
    private lateinit var mRecordBtn : ImageView
    private var mRecordStatus = RecordStatus.Idle

    companion object {
        const val TAG = "MixRecordActivity"
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContentView(R.layout.activity_mixrecord)

        mVideoPreviewView = findViewById(R.id.videoPreviewView)
        mCameraPreviewView = findViewById(R.id.cameraPreviewView)
        mRecordBtn = findViewById(R.id.btnRecordControl)
        // Start recording.
        mRecordBtn.setOnClickListener {
            if(mRecordStatus == RecordStatus.Recording) {
                mAliyunRecord.finishRecording()
                Toast.makeText(this@MixRecordActivity, "Stop recording", Toast.LENGTH_SHORT).show()
                updateRecordStatus(RecordStatus.Idle)
            } else {
                val curTime = System.currentTimeMillis()
                mAliyunRecord.setOutputPath("/storage/emulated/0/DCIM/Camera/svideo_mixrecord_video_$curTime.mp4")
                mAliyunRecord.startRecording()
                Toast.makeText(this@MixRecordActivity, "Start recording", Toast.LENGTH_SHORT).show()
                updateRecordStatus(RecordStatus.Recording)
            }

        }

        mAliyunRecord = AliyunMixRecorderCreator.createAlivcMixRecorderInstance(this)


        val videoDisplayParam = AliyunMixRecorderDisplayParam.Builder()
                .displayMode(VideoDisplayMode.FILL)
                .layoutParam(
                        AliyunMixTrackLayoutParam.Builder()
                                .centerX(0.25f)
                                .centerY(0.5f)
                                .widthRatio(0.5f)
                                .heightRatio(1.0f)
                                .build()
                )
                .build()

        val cameraDisplayParam = AliyunMixRecorderDisplayParam.Builder()
                .displayMode(VideoDisplayMode.FILL)
                .layoutParam(AliyunMixTrackLayoutParam.Builder()
                        .centerX(0.75f)
                        .centerY(0.5f)
                        .widthRatio(0.5f)
                        .heightRatio(1.0f)
                        .build())
                .build()

        val mixMediaParam = AliyunMixMediaInfoParam.Builder()
                .streamStartTimeMills(0L)
                .streamEndTimeMills(0L) // If you set streamStartTimeMills and streamEndTimeMills to 0L, the duration of the video is automatically used.
                .mixVideoFilePath("/storage/emulated/0/DCIM/Camera/VID_20210317_174802.mp4")
                .mixDisplayParam(videoDisplayParam)
                .recordDisplayParam(cameraDisplayParam)
                .build()

        val mediaInfo = MediaInfo()
        mediaInfo.fps = 30
        mediaInfo.crf = 6
        mediaInfo.videoWidth = 720
        mediaInfo.videoHeight = 1080
        mediaInfo.videoCodec = VideoCodecs.H264_SOFT_OPENH264

        mAliyunRecord.setMixMediaInfo(mixMediaParam, mediaInfo)
        mAliyunRecord.setDisplayView(mCameraPreviewView, mVideoPreviewView)

        mAliyunRecord.setRecordCallback(object : RecordCallback {
            override fun onComplete(validClip: Boolean, clipDuration: Long) {
                Log.i(TAG, "onComplete")
            }

            override fun onFinish(outputPath: String?) {
                Log.i(TAG, "onFinish path : $outputPath")
                mAliyunRecord.clipManager.deleteAllPart()
            }

            override fun onProgress(progress: Long) {
                Log.i(TAG, "onProgress  : $progress")
            }

            override fun onMaxDuration() {
                Log.i(TAG, "onMaxDuration")
            }

            override fun onError(errorCode: Int) {
                Log.i(TAG, "onError : $errorCode")
            }

            override fun onInitReady() {
                Log.i(TAG, "onInitReady")
            }

            override fun onDrawReady() {
                Log.i(TAG, "onDrawReady")
            }

            override fun onPictureBack(bitmap: Bitmap?) {
                Log.i(TAG, "onPictureBack")
            }

            override fun onPictureDataBack(p0: ByteArray?) {
                Log.i(TAG, "onPictureDataBack")
            }

        })

    }

    private fun updateRecordStatus(recordStatus: RecordStatus)
    {
        mRecordStatus = recordStatus
        when(recordStatus) {
            RecordStatus.Idle -> {
                mRecordBtn.setImageResource(R.mipmap.alivc_svideo_bg_record_start)
            }
            RecordStatus.Recording -> {
                mRecordBtn.setImageResource(R.mipmap.alivc_svideo_bg_record_storp)
            }
        }
    }

    override fun onResume() {
        super.onResume()
        mAliyunRecord.startPreview()
    }

    override fun onPause() {
        super.onPause()
        mAliyunRecord.stopPreview()
    }

    override fun onDestroy() {
        super.onDestroy()
        mAliyunRecord.release()
    }
}

Configuration example in XML

<?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <androidx.constraintlayout.widget.Guideline
        android:id="@+id/guideline3"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:orientation="vertical"
        app:layout_constraintGuide_percent="0.5" />


    <SurfaceView
        android:id="@+id/videoPreviewView"
        android:layout_width="0dp"
        android:layout_height="400dp"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toStartOf="@+id/guideline3"
        app:layout_constraintHorizontal_bias="0.0"
        app:layout_constraintStart_toStartOf="parent"
        tools:layout_editor_absoluteY="0dp" />

    <SurfaceView
        android:id="@+id/cameraPreviewView"
        android:layout_width="0dp"
        android:layout_height="400dp"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintStart_toStartOf="@+id/guideline3"
        app:layout_constraintEnd_toEndOf="parent" />

    <ImageView
        android:id="@+id/btnRecordControl"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_centerInParent="true"
        android:layout_marginBottom="50dp"
        android:src="@mipmap/alivc_svideo_bg_record_start"
        android:visibility="visible"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintEnd_toEndOf="parent"
        app:layout_constraintStart_toStartOf="parent">

    </ImageView>

</androidx.constraintlayout.widget.ConstraintLayout>