ApsaraVideo Live:Play and publish external audio

How it works

This feature is designed for playing or publishing audio files during a real-time session. It provides two sets of APIs for different use cases:

• Accompaniment APIs: Intended for playing longer audio files, such as background music or musical accompaniments. You can play only one audio file at a time.

• Sound effect APIs: Optimized for playing short, transient audio clips, such as applause or laughter. You can play multiple sound effects simultaneously.

The related features are as follows:

Play an accompaniment

You can play only one accompaniment file at a time.

1. Join a channel and publish the audio stream. This is enabled by default in the SDK.

   Android

   // To play an external audio source, you must first publish the primary audio stream. This is enabled by default.
   mAliRtcEngine.publishLocalAudioStream(true);

   iOS

   // To play an external audio source, you must first publish the primary audio stream. This is enabled by default. 
   engine.publishLocalAudioStream(true)

2. Control playback

After joining a channel and publishing the audio stream, call startAudioAccompany to play the accompaniment. A successful call triggers the onAudioAccompanyStateChanged callback on the local client and the onRemoteAudioAccompanyStarted callback on remote clients.

Use the following APIs to control playback:

• stopAudioAccompany: Stops playback.

• pauseAudioAccompany: Pauses playback.

• resumeAudioAccompany: Resumes playback.

• getAudioAccompanyDuration, getAudioAccompanyCurrentPosition, and setAudioAccompanyPosition: Gets the file duration and controls playback progress.

• setAudioAccompanyVolume: Sets both the local playout and remote publishing volume of the accompaniment.

• getAudioAccompanyPublishVolume and setAudioAccompanyPublishVolume: Gets or adjusts the remote playback volume of the accompaniment.

• getAudioAccompanyPlayoutVolume and setAudioAccompanyPlayoutVolume: Gets or adjusts the local playback volume of the accompaniment.

// Path to the audio accompaniment file.
private String mMixingMusicFilepath = "/assets/music.wav";

// Configure audio accompaniment playback.
AliRtcEngine.AliRtcAudioAccompanyConfig config = new AliRtcEngine.AliRtcAudioAccompanyConfig();
config.loopCycles = -1; // Number of loops. -1 indicates infinite looping.
config.publishVolume = publishVolume; // Publishing volume. Range: [0-100].
config.playoutVolume = playbackVolume; // Local playback volume. Range: [0-100].
config.startPosMs = 0; // Playback starting position in milliseconds.
// Start playback
mAliRtcEngine.startAudioAccompany(mMixingMusicFilepath, config);

// Pause/Resume accompaniment
mAliRtcEngine.pauseAudioAccompany();
mAliRtcEngine.resumeAudioAccompany();

// Stop playback
mAliRtcEngine.stopAudioAccompany();

// Get the duration of the accompaniment file in ms. Call this method after startAudioAccompany, otherwise it returns -1.
int duration = mAliRtcEngine.getAudioAccompanyDuration();
// To get the duration of a specific file, call getAudioFileInfo. You can call this after creating the engine. The result is returned in the onAudioFileInfo callback.
mAliRtcEngine.getAudioFileInfo(filePath);
@Override
public void onAudioFileInfo(AliRtcEngine.AliRtcAudioFileInfo info, AliRtcEngine.AliRtcAudioAccompanyErrorCode errorCode) {

    handler.post(() -> {
        String msg = "onAudioFileInfo.file:" + info.filePath + "duration:" + info.durationMs + "audioPlayingErrorCode=" + errorCode;
        ToastHelper.showToast(PlayAudioFilesActivity.this, msg, Toast.LENGTH_SHORT);
    });
}

// Get/Set the volume.
mAliRtcEngine.setAudioAccompanyVolume(50); // [0, 100]
int publishVolume = mAliRtcEngine.getAudioAccompanyPublishVolume();
mAliRtcEngine.setAudioAccompanyPublishVolume(50);
int playoutVolume = mAliRtcEngine.getAudioAccompanyPlayoutVolume();
mAliRtcEngine.setAudioAccompanyPlayoutVolume(50);

// Get/Set playback progress.
int currPosition = mAliRtcEngine.getAudioAccompanyCurrentPosition(); // Current position in ms.
int targetPosition = 1000; // Example: start playing from the 1000 ms mark.
mAliRtcEngine.setAudioAccompanyPosition(targetPosition);

// Path to the accompaniment file. 
let filePath = Bundle.main.path(forResource: "music", ofType: "wav")

// Configure accompaniment playback.
let config = AliRtcAudioAccompanyConfig()
config.loopCycles = loopCount
config.publishVolume = publishVolume
config.playoutVolume = playoutVolume
config.startPosMs = 0
// Start playback.
let result = rtcEngine.startAudioAccompany(withFile: filePath, config: config)

// Pause/Resume playback.
rtcEngine.pauseAudioAccompany()
rtcEngine.resumeAudioAccompany()

// Stop playback.
rtcEngine.stopAudioAccompany()

// Get the duration of the audio accompaniment file in ms. Call this method after startAudioAccompany, otherwise it returns -1.
let duration = rtcEngine.getAudioAccompanyDuration()
// To get the duration of a specific file, call getAudioFileInfo. You can call this after creating the engine. The result is returned in the onAudioFileInfo callback.
rtcEngine.getAudioFileInfo(filePath)
func onAudioFileInfo(_ info: AliRtcAudioFileInfo, errorCode: AliRtcAudioAccompanyErrorCode) {
    "onAudioFileInfo, filePath: \(info.filePath), durationMs: \(info.durationMs), errorCode: \(errorCode)".printLog()
}

// Get/Set the volume.
rtcEngine.setAudioAccompanyVolume(50) // Set both publishing and playout volume. Range: [0-100].
let audioPublishVolume = rtcEngine.getAudioAccompanyPublishVolume() // Get the publishing volume.
rtcEngine.setAudioAccompanyPublishVolume(50) // Set the publishing volume. Range: [0-100].
let audioPlayoutVolume = rtcEngine.getAudioAccompanyPlayoutVolume() // Get the local playback volume.
rtcEngine.setAudioAccompanyPlayoutVolume(50) // Set the local playback volume. Range: [0-100].

// Get/Set playback progress.
let currentPosition = rtcEngine.getAudioAccompanyCurrentPosition() // Current position in ms.
let targetPosition:Int32 = 1000; // Example: start playing from the 1000 ms mark.
rtcEngine.setAudioAccompanyPosition(targetPosition)

Play sound effects

To play multiple sound effects at once (for example, applause and laughter), use APIs such as preloadAudioEffect and playAudioEffect.

1. Join a channel and publish the audio stream. This is enabled by default in the SDK.

   Android

   // To play an external audio source, you must first publish the primary audio stream. This is enabled by default.
   mAliRtcEngine.publishLocalAudioStream(true);

   iOS

   // To play an external audio source, you must first publish the primary audio stream. This is enabled by default. 
   engine.publishLocalAudioStream(true)

2. (Optional) Preload resources

To improve performance when playing an effect repeatedly, preload the file into memory by calling preloadAudioEffect.

• preloadAudioEffect: Loads the specified sound effect file. Supports both local files and online file URLs. Local files are recommended.

• unloadAudioEffect: When a sound effect file is no longer needed, call this API to unload it and release resources.

// Preload the sound effect file with a specified ID and file path. You must define the ID.
mAliRtcEngine.preloadAudioEffect(mCurrSoundID, filePath);
// Unload the sound effect file with the specified ID.
mAliRtcEngine.unloadAudioEffect(mCurrSoundID);

// Preload the sound effect file with a specified ID and file path. You must define the ID.
rtcEngine.preloadAudioEffect(withSoundId: self.soundId, filePath: filePath)
// Unload the sound effect file with the specified ID.
rtcEngine.unloadAudioEffect(withSoundId: self.currentEffectIndex)

3. Control playback

After joining a channel and publishing the audio stream, call playAudioEffect to play the sound effect file. When playback completes, the onAudioEffectFinished callback is triggered on the local client.

Use the following APIs to control playback:

• stopAudioEffect, stopAllAudioEffects: Stops playback.

• pauseAudioEffect, pauseAllAudioEffects: Pauses playback.

• resumeAudioEffect, resumeAllAudioEffects: Resumes playback.

• getAudioEffectPublishVolume, setAudioEffectPublishVolume, setAllAudioEffectsPublishVolume: Gets or sets the publishing volume.

• getAudioEffectPlayoutVolume, setAudioEffectPlayoutVolume, setAllAudioEffectsPlayoutVolume: Gets or sets the local playback volume.

// Play a sound effect.
AliRtcEngine.AliRtcAudioEffectConfig config = new AliRtcEngine.AliRtcAudioEffectConfig();
config.loopCycles = -1;			// Number of playback loops. -1 indicates infinite looping.
config.startPosMs = 0;			// Playback starting position in milliseconds.
config.publishVolume = 50;		// Publishing volume. Range: [0-100].
config.playoutVolume = 50;		// Local playback volume. Range: [0, 100].
mAliRtcEngine.playAudioEffect(mCurrSoundID, filePath, config);

// Stop playback.
mAliRtcEngine.stopAudioEffect(soundId);
mAliRtcEngine.stopAllAudioEffects();

// Pause playback.
mAliRtcEngine.pauseAudioEffect(soundId);
mAliRtcEngine.pauseAllAudioEffects();

// Resume playback.
mAliRtcEngine.resumeAudioEffect(soundId);
mAliRtcEngine.resumeAllAudioEffects();

// Get or set the volume.
// Publishing volume.
mAliRtcEngine.getAudioEffectPublishVolume(soundId);
mAliRtcEngine.setAudioEffectPublishVolume(soundId, 50);
mAliRtcEngine.setAllAudioEffectsPublishVolume(50);
// Playback volume.
mAliRtcEngine.getAudioEffectPlayoutVolume(soundId);
mAliRtcEngine.setAudioEffectPlayoutVolume(soundId, 50);
mAliRtcEngine.setAllAudioEffectsPlayoutVolume(50);

// Play
let config = AliRtcAudioEffectConfig()
config.loopCycles = -1
config.startPosMs = 0
config.publishVolume = 50
config.playoutVolume = 50
let result = rtcEngine.playAudioEffect(withSoundId: self.soundId, filePath: filePath, config: config)
if result != 0 {
    UIAlertController.showAlertWithMainThread(msg: "Failed to play the sound effect. Error code: \(result)", vc: self)
}

// Stop playback.
rtcEngine.stopAudioEffect(withSoundId: soundId)
rtcEngine.stopAllAudioEffects()

// Pause playback.
rtcEngine.pauseAudioEffect(withSoundId: soundId)
rtcEngine.pauseAllAudioEffects()

// Resume playback.
rtcEngine.resumeAudioEffect(withSoundId: soundId)
rtcEngine.resumeAllAudioEffects()

// Get or set the volume.
rtcEngine.getAudioEffectPublishVolume(withSoundId: soundId)
rtcEngine.setAudioEffectPublishVolumeWithSoundId(soundId, volume: 50)
rtcEngine.setAllAudioEffectsPublishVolume(50)
rtcEngine.resumeAudioEffect(withSoundId: soundId)    
rtcEngine.getAudioEffectPlayoutVolume(withSoundId: soundId)
rtcEngine.setAudioEffectPlayoutVolumeWithSoundId(soundId, volume: 50)

How it works

Sample code

/* Set the parameters as needed. */
AliRtcEngine.AliRtcExternalAudioStreamConfig config = new AliRtcEngine.AliRtcExternalAudioStreamConfig();
/* To disable local playback, set the local playout volume to 0. */
config.playoutVolume = currentAudioPlayoutVolume;
/* To disable publishing, set the publishing volume to 0. */
config.publishVolume = currentAudioPublishVolume;
config.channels = 1;
config.sampleRate = 48000;
// The method returns an external input stream ID. Use this ID to push data into the SDK.
audioStreamID = mAliRtcEngine.addExternalAudioStream(config);

/* Set the parameters as needed. */
AliRtcExternalAudioStreamConfig *config = [AliRtcExternalAudioStreamConfig new];
config.channels = _pcmChannels;
config.sampleRate = _pcmSampleRate;
/* To disable local playback, set playout volume to 0. */
config.playoutVolume = 0;
/* To disable publishing, set publish volume to 100. */
config.publishVolume = 100;
// The method returns an external input stream ID. Use this ID to push data into the SDK.
_externalPlayoutStreamId = [self.engine addExternalAudioStream:config];

AliRtcEngine.AliRtcAudioFrame rawData = new AliRtcEngine.AliRtcAudioFrame();
rawData.data = frameInfo.audio_data[0];
rawData.numSamples = (int) (frameInfo.audio_data[0].length / (2 * frameInfo.audio_channels));
rawData.bytesPerSample = 2;
rawData.numChannels = frameInfo.audio_channels;
rawData.samplesPerSec = frameInfo.audio_sample_rate;

int ret = mAliRtcEngine.pushExternalAudioStreamRawData(audioStreamID, rawData);
if(ret == 0x01070101) {
    // The buffer is full.
    sleep(20);
} else if(ret < 0) {
    /* An exception occurred. Check parameters and publishing state. */
}

AliRtcAudioFrame *sample = [AliRtcAudioFrame new];
sample.dataPtr = _pcmLocalData;
sample.samplesPerSec = _pcmLocalSampleRate;
sample.bytesPerSample = sizeof(int16_t);
sample.numOfChannels = _pcmLocalChannels;
sample.numOfSamples = numOfSamples;
int rc = [self.engine pushExternalAudioStream:_externalPlayoutStreamId rawData:sample];

if(rc == 0x01070101) {
    // The buffer is full.
    sleep(20);
} else if(ret < 0) {
    /* An exception occurred. Check parameters and publishing state. */
}

mAliRtcEngine.removeExternalAudioStream(audioStreamID);

[self.engine removeExternalAudioStream:_externalPublishStreamId];