Intelligent Media Services:SEI

Usage notes

Common scenarios for SEI:

• Transmitting timestamps to calculate network latency or synchronize data.

• Transmitting descriptive information (up to 4 KB). We recommend using JSON or a plain string.

• SEI is implemented internally using the SEI extension protocol.

Send SEI messages

Checking the environment

Before enabling SEI, verify that the current environment supports sending and receiving SEI messages.

// Static method
const result = AliRtcEngine.isSupported();
if (!result.detail.isSendMediaExtensionMsgSupported) {
  console.log('Your browser does not support sei.');
}

Send an SEI message

Enabling this feature increases client performance overhead. To send SEI messages, call setEnableMediaExtensionMsg before joining a channel.

// Prerequisite: Create an engine instance first.
// Enable media extension. This must be called before joining a channel.
aliRtcEngine.setEnableMediaExtensionMsg(true);

// Assume that an SEI message is used to send a DataView object.
const buffer = new ArrayBuffer(4);
const view = new DataView(buffer);

// Use DataView to set the content of ArrayBuffer to 7f ff 7f ff.
view.setInt8(0, 127); // Set the first byte to 127.
view.setUint8(1, 255); // Set the second byte to 255.
view.setInt16(2, 32767); // Set the third and fourth bytes to 32767.

aliRtcEngine
  .sendMediaExtensionMsg(buffer, 2)
  .then(() => {
    console.log(
      'Successfully sent the SEI message. Content: `7fff7fff`. Repetitions: 2. payloadType: 5.',
    );
  })
  .catch((err: Error) => {
    message.error(err.message);
  });

Media extension messages reuse the audio and video data tunnel. Keep the following limits in mind:

• The maximum send rate is profile messages per second. SEI messages are transmitted in H.264/H.265 streams, so extension information can be attached only when video frames are encoded.

• The custom message body is limited to 4 KB to avoid affecting media data quality.

• The repeatCount parameter in sendMediaExtensionMsg specifies message redundancy. Setting it to a value greater than 1 sends the message multiple times, which helps prevent loss from network packet drops. Other users in the channel receive the same message multiple times and must deduplicate accordingly.

• During a bypass live stream, subscribers in the channel also receive custom messages.

• Only one media extension message can be transmitted at a time. Calling sendMediaExtensionMsg multiple times overwrites previous data with the new data.

Receive SEI messages

Check the environment

The receiving client must also verify that the environment supports SEI before enabling this feature.

// Static method
const result = AliRtcEngine.isSupported();
if (!result.detail.isSendMediaExtensionMsgSupported) {
  console.log('Your browser does not support sei.');
}

Receive an SEI message

Receiving SEI messages increases client performance overhead. Call setEnableMediaExtensionMsg before joining a channel to enable this feature.

// Prerequisite: Create an engine instance first.
// Listen for the mediaExtensionMsgReceived event.
aliRtcEngine.on('mediaExtensionMsgReceived', (remoteUserId, data, payloadType) => {
  console.log(`User ${remoteUserId} sent an SEI message with payloadType ${payloadType}. Content:`, data);
});