How to configure audio and video SEI - - Alibaba Cloud Documentation Center

In live streaming scenarios, viewers usually want to know the co-streaming information of the streamer in real time, such as the battle state, volume, and network state. You can use supplemental enhancement information (SEI) to allow frames to carry synchronized key information. This topic describes how to use SEI to carry synchronized key information in frames in ApsaraVideo Real-time Communication (ARTC).

Client SEI

You can use the client SDK to insert SEI that carries custom information into video frames. The co-streamer who is co-streaming with the streamer and the viewers can both receive the custom information.

The following sample code provides an example on how the streamer and co-streamer call Alibaba Cloud Push SDK to send and receive SEI.

Sample code for Android:

// Send SEI: AlivcLivePusher#sendMessage. The stream ingest client sends a custom message.
public void sendSEI(String text) {
    mAlivcLivePusher.sendMessage(text, 0, 0, false);
}

// Receive SEI: AlivcLivePlayInfoListener#onReceiveSEIMessage. The callback for media extension information is received.
@Override
public void onReceiveSEIMessage(int payload, byte[] data) {
    //Log.d(TAG, "onReceiveSEIMessage: " + payload + ", " + new String(data, StandardCharsets.UTF_8));
}

Sample code for iOS:

// Send the SEI by calling the sendMessage method in AlivcLivePusher.
[self.rtcPusher sendMessage:data repeatCount:1 delayTime:0 KeyFrameOnly:false];

// The SEI is received by the streamer and co-streamer from the onReceiveSeiMessage callback in AliLivePlayerDelegate.
- (void)onReceiveSeiMessage:(AlivcLivePlayer *)player payloadType:(int)payloadType data:(NSData *)data
{
    if (data.bytes){
        NSString *str = [NSString stringWithUTF8String:data.bytes];
    } 
}

The following sample code provides an example on how the viewers use ApsaraVideo Player to open and parse SEI frames.

Sample code for Android:

mAliPlayer = AliPlayerFactory.createAliPlayer(mContext);

PlayerConfig playerConfig = mAliPlayer.getConfig();
// Configure SEI frame parsing for audio-only or video-only streams in the FLV format to reduce the startup time.
// The playback start buffer. A larger value indicates more stable playback start. However, a too large value may affect the time it takes to start playing the stream. Specify this parameter as appropriate.
playerConfig.mStartBufferDuration = 1000;
// The cache required for stuttering to recover. Specify a larger value in poor network conditions. We recommend that you set the value to 500 for an audio-only stream or 3000 for a video stream.
playerConfig.mHighBufferDuration = 500;
// Enable SEI listening.
playerConfig.mEnableSEI = true;
mAliPlayer.setConfig(playerConfig);

mAliPlayer.setAutoPlay(true);

mAliPlayer.setOnErrorListener(errorInfo -> {
    mAliPlayer.prepare();
});

mAliPlayer.setOnSeiDataListener(new IPlayer.OnSeiDataListener() {
    @Override
    public void onSeiData(int i, byte[] bytes) {

    }
});

Sample code for iOS:

self.cdnPlayer = [[AliPlayer alloc] init];
self.cdnPlayer.delegate = self;
AVPConfig *config = [self.cdnPlayer getConfig];
config.enableSEI = YES;
[self.cdnPlayer setConfig:config];

// Listen to SEI-related callbacks.
- (void)onSEIData:(AliPlayer*)player type:(int)type data:(NSData *)data {
    if (data.bytes){
        NSString *str = [NSString stringWithUTF8String:data.bytes];
        // Process the SEI message.
    }
}

Server SEI

Server SEI includes system SEI and custom SEI.

System SEI: automatically generated by the ARTC server. It is used to describe the layout and the microphone status, volume, camera status, and network status of the streamer.
Custom SEI: You call an API operation on the server side to send custom business information.

System SEI

When the streamer enters the co-streaming state or switches from the co-streaming state back to the single-streamer state, the layout of the live stream changes. In this case, the stream mixing server in the cloud automatically inserts SEI into the video frames so that the player can perceive the changes in the layout. During live streaming, system SEI can be sent periodically or sent with keyframes.

Before you use the client SDK to initiate a stream mixing and relay task, you can configure the system SEI sending policy in the ApsaraVideo Live console. For more information, see the description about SEI insertion in Configure stream relay.
When you call an API operation on the server side to initiate a stream mixing and relay task, you need to configure the SeiParams.LayoutVolume parameter. For more information, see StartLiveMPUTask (new) or UpdateLiveMPUTask (new).
- FollowIdr specifies whether to send the SEI with keyframes.
- Interval specifies the interval at which the SEI is sent.
If the value of the FollowIdr parameter is set to 0 and the Interval parameter is not specified, the SEI is sent only when the layout changes, the microphone status changes, or the camera status changes. The SEI is not sent periodically.

Custom SEI

Custom SEI is used to send custom messages. You can only call an API operation on the server side to send custom SEI. In your call of the API operation, you need to configure the SeiParams.PassThrough parameter. For more information, see StartLiveMPUTask (new) or UpdateLiveMPUTask (new).

PayloadContent specifies the custom message.
FollowIdr specifies whether to send the SEI with keyframes.
Interval specifies the interval at which the SEI is sent.

If the value of the FollowIdr parameter is set to 0 and the Interval parameter is not specified, the SEI is sent once only when the API operation is called. The SEI is not sent periodically.

SEI data format

The following figure shows the data format of SEI. The current SEI payload type is 5, which is UserDataUnregistered.

For more information about the SEI payload content, see StartLiveMPUTask (new).