ApsaraVideo Live:Send and receive SEI messages

Feature introduction

SEI is part of video encoding standards such as H.264/AVC and H.265/HEVC. It is used to carry additional information related to the video content. This information provides supplementary functionality for the receiver, such as synchronization or error recovery, and can be used for scenarios such as precise video layout, lyric synchronization, and interactive live quizzes.

SEI is the recommended method for sending custom messages. The message is embedded into the video stream and transmitted along with the video data to other users in the channel.

• Pros:

  • High real-time performance: Text messages are synchronized with the media stream.

  • Unlimited receivers: Anyone who subscribes to the stream can receive the message.

• Cons:

  • Size limit: To avoid impacting video stream quality, each SEI message is limited to a maximum of 4 KB.

Sample code

ARTC provides an open-source demo project for your reference. You can download them or view the code online.

Android: Android/ARTCExample/BasicUsage/src/main/java/com/aliyun/artc/api/basicusage/SEIUsage/SEIActivity.java

iOS: iOS/ARTCExample/BasicUsage/SEIUsage/SEIUsageVC.swift

Before you begin

Before you implement the feature, make sure you meet the following requirements:

• Create an ARTC application and obtain the AppID and AppKey from the ApsaraVideo Live console.

• Integrate the ARTC SDK into your project and implement basic audio and video call features.

Implementation

// Prerequisite: You have joined a channel and started stream ingest (automatic ingest by default).
mSendSEIButton.setOnClickListener(v -> {
    if(mAliRtcEngine == null) {
        return;
    }
    String seiMessage = mEditText.getText().toString();
    if (TextUtils.isEmpty(seiMessage)) {
        return;
    }
    byte[] seiData = seiMessage.getBytes();
    mAliRtcEngine.sendMediaExtensionMsg(seiData,1,0,false);
    // mAliRtcEngine.sendMediaExtensionMsgEx(seiData,1,0,false, 5);
});

// Send an SEI message
func sendSEI(seiMessage: String) -> Bool {
    guard let data = seiMessage.data(using: .utf8) else {
        return false
    }
    let repeatCount: Int32 = 1
    let delay: Int32 = 0
    let payloadType: Int32 = 5
    let isKeyFrameOnly = false
    let ret = self.rtcEngine?.sendMediaExtensionMsg(data, repeatCount: repeatCount, delay: delay, isKeyFrame: isKeyFrameOnly)
    // let ret = self.rtcEngine?.sendMediaExtensionMsgEx(data, repeatCount: repeatCount, delay: delay, isKeyFrame: isKeyFrameOnly, payloadType: payloadType)
    debugPrint("sendSEI: \(ret ?? -1)")
    return ret == 0
}

/* Specify the following parameters as needed. */
char * data = "xxxxx";
int length = strlen(data);
mAliRtcEngine->SendMediaExtensionMsg(data, length, 1,0,true);

@Override
public void onMediaExtensionMsgReceived(String uid, int payloadType, byte[]message) {
    super.onMediaExtensionMsgReceived(uid,payloadType, message);

    handler.post(new Runnable() {
        @Override
        public void run() {
            // Process the message.
            String receivedMsg = new String(message);
            ToastHelper.showToast(SEIActivity.this, receivedMsg, Toast.LENGTH_SHORT);
            
        }
    });
}

extension SEIUsageMainVC: AliRtcEngineDelegate {
    
    // Callback for receiving an SEI message
    // receive SEI message
    func onMediaExtensionMsgReceived(_ uid: String, payloadType: Int32, message data: Data) {
        // Process the message.
        guard let message = String(data: data, encoding: .utf8) else {
            print("Failed to parse the message")
            return
        }
        self.showToast(message: "Received SEI: \(message), from uid: \(uid), payloadType: \(payloadType)")
    }
    // Other callbacks...

}

public:
virtual void OnMediaExtensionMsgReceived(const char* uid, const int8_t * message, uint32_t size) override {
    /* TODO: Write the processing logic as needed. */
}