Play raw audio data or local files with the SDK - Intelligent Media Services

This topic explains how to use the AICallKit SDK to achieve low-latency, echo-free playback of raw audio data (such as PCM) and local audio files (such as WAV/MP3) by leveraging the underlying AliRtcEngine.

Feature introduction

When you need to play sound effects or background music during a call, the AICallKit SDK provides access to the underlying AliRtcEngine object. You can obtain this object and use its APIs to play audio. This allows you to take advantage of the RTC engine's built-in audio processing and acoustic echo cancellation (AEC) capabilities, which effectively prevent played audio from being re-captured by the microphone, thus avoiding echo issues.

Prerequisites

You have integrated the call agent and implemented basic call features. For more information, see Audio and video calls.

Implementation

The AICallKit SDK does not directly provide audio playback interfaces. Instead, it relies on the APIs provided by AliVCSDK_ARTC for pushing external raw audio data or playing audio accompaniments. You can obtain the AliRtcEngine object from AICallKit and call its interfaces to play audio.

The interaction flow is as follows:

Cache the AliRtcEngine and pre-start the player

You can get the ARTC SDK engine object, AliRtcEngine, in the onAliRtcEngineCreated callback of the AICallKit SDK. If you need to play audio while the call is connecting, you must start the audio player in advance.

Android

@Override
public void onAliRtcEngineCreated(AliRtcEngine engine) {
    if(engine != null) {
        // Get the AliRtcEngine object
        mRtcEngine = engine;

        // If you need to play audio during the connection process, start the player beforehand.
        mRtcEngine.startAudioPlayer();
    }
}

iOS

public func onAICallRTCEngineCreated() {
    guard let engine = self.engine.getRTCInstance() as? AliRtcEngine else {
        return
    }
    // Save the AliRtcEngine object
    self.rtcEngine = engine

    // If you need to play audio during the connection process, start the player beforehand.
    self.rtcEngine?.startAudioPlayer()
}

Scenario 1: Play PCM Data

This method is suitable for scenarios where your app receives raw audio data from a server, for example, over a persistent connection.

Add an external audio stream to the AliRtcEngine object.

Android

AliRtcEngine.AliRtcExternalAudioStreamConfig config = new AliRtcEngine.AliRtcExternalAudioStreamConfig();
config.channels = mPcmChannels;    // Number of PCM channels
config.sampleRate = mPcmSampleRate;// PCM sample rate
config.playoutVolume = 100;
config.publishVolume = 0;  // Play locally. Do not publish the stream.
mStreamId = mAliRtcEngine.addExternalAudioStream(config);
if mStreamId <= 0 {
    debugPrint("addExternalAudioStream failed: \(mStreamId)")
    return
}

iOS

guard let rtc = self.rtcEngine else {
    return
}
let config = AliRtcExternalAudioStreamConfig()
config.sampleRate = self.pcmSampleRate // PCM sample rate
config.channels = self.pcmChannels     // Number of PCM channels
config.playoutVolume = 100
config.publishVolume = 0  // Play locally. Do not publish the stream.
self.streamId = rtc.addExternalAudioStream(config) 
if self.streamId <= 0 {
    debugPrint("addExternalAudioStream failed: \(self.streamId)")
    return
}

Push PCM audio data to the SDK.

Android

// Recommended duration to push at one time: 20-50 ms.
int duration = 30;
// Get the next chunk of PCM data from your buffer (duration cannot exceed 'duration').
byte[] dataToSend = ...
// Check dataToSend: if playback is active but data is insufficient, sleep and wait for more data.

// Calculate the number of samples in dataToSend.
int numOfSamples = dataToSend_byte_length / (2 * frameInfo.audio_channels);


AliRtcEngine.AliRtcAudioFrame rawData = new AliRtcEngine.AliRtcAudioFrame();
rawData.data = dataToSend;
rawData.numSamples = numOfSamples;
rawData.bytesPerSample = 2;
rawData.numChannels = mPcmChannels;
rawData.samplesPerSec = mPcmSampleRate;
int ret = mAliRtcEngine.pushExternalAudioStreamRawData(mStreamId, rawData);
if(ret == 0x01070101) {
    // The RTC buffer is full. dataToSend must be sent to the RTC next time.
} 
else if(ret < 0) {
    // RTC playback failed. Do not continue sending data to the RTC. Check the parameters and publishing status.
    return;
}
else {
    // The data was sent to the RTC successfully. Remove the current dataToSend from the cache. Do not send it again.
}
sleep(duration - 10);

iOS

guard let rtc = self.rtcEngine, self.streamId > 0 else {
    return
}

// Recommended duration to push at one time: 20-50 ms.
let duration = 30
// Get the next chunk of PCM data from your buffer (duration cannot exceed 'duration').
let dataToSend = ...
// Check dataToSend: if playback is active but data is insufficient, sleep and wait for more data.

// Calculate the number of samples in dataToSend.
let numOfSamples = dataToSend_byte_length / (2 * frameInfo.audio_channels)

let pushFrame = AliRtcAudioFrame()
pushFrame.dataPtr = dataToSend
pushFrame.numOfSamples = numOfSamples  // The number of samples
pushFrame.bytesPerSample = 2
pushFrame.samplesPerSec = self.pcmSampleRate
pushFrame.numOfChannels = self.pcmChannels
let result = rtc.pushExternalAudioStream(self.pushStreamId, rawData: pushFrame)
if result == 0x01070101 {
    // The RTC buffer is full. dataToSend must be sent to the RTC next time.
}
else if result < 0 {
    // RTC playback failed. Do not continue sending data to the RTC. Check the parameters and publishing status.
    return
}
else {
    // The data was sent to the RTC successfully. Remove the current dataToSend from the cache. Do not send it again.
}
Thread.sleep(forTimeInterval: Double(duration - 10) / 1000.0)

Remove the external audio stream.

Android

mRtcEngine.removeExternalAudioStream(mStreamId);

mRtcEngine = null;
mStreamId = -1;

iOS

self.rtcEngine?.removeExternalAudioStream(self.streamId)

self.rtcEngine = nil
self.streamId = -1

Scenario 2: Play a local audio file

This method is suitable for playing audio from a local file.

Start playback.

Android

// Path to the audio file
private String mFilepath = "/assets/music.wav";

// Playback configuration
AliRtcEngine.AliRtcAudioAccompanyConfig config = new AliRtcEngine.AliRtcAudioAccompanyConfig();
config.loopCycles = 1;     // The number of loops
config.publishVolume = 0;   // Play locally. Do not publish the stream.
config.playoutVolume = 100;
config.startPosMs = 0;      // The start position for playback
// Start playback
mRtcEngine.startAudioAccompany(mFilepath, config);

iOS

// Path to the audio file
let filePath = Bundle.main.path(forResource: "music", ofType: "wav")

// Playback configuration
let config = AliRtcAudioAccompanyConfig()
config.loopCycles = 1   // The number of loops. -1 indicates an infinite loop.
config.publishVolume = 0   // Play locally. Do not publish the stream.
config.playoutVolume = 100
config.startPosMs = 0
// Start playback
let result = rtcEngine.startAudioAccompany(withFile: filePath, config: config)

Note

After starting playback, you can use other APIs for control:

pauseAudioAccompany: Pauses playback.
resumeAudioAccompany: Resumes playback.
stopAudioAccompany: Stops playback.

Listen for the playback completion callback.

Android

@Override
public void onAudioAccompanyStateChanged(ARTCAICallEngine.ARTCAICallAudioAccompanyStateCode
                                                 playState, ARTCAICallEngine.ARTCAICallAudioAccompanyErrorCode
                                                 errorCode) {
    if (playState == ARTCAICallEngine.ARTCAICallAudioAccompanyStateCode.ARTCAICallAudioAccompanyEnded) {
        // Handle the end of playback
    }
}

iOS

func onAudioAccompanyStateChanged(state: ARTCAICallAudioAccompanyState, errorCode: ARTCAICallAudioAccompanyErrorCode) {
    if state == .ARTCAICallAudioAccompanyEnded {
        // Handle the end of playback
    }
}