ApsaraVideo Live:Custom audio capture

1. Enable or disable internal capture

To use the custom audio capture feature, you typically need to disable the SDK's internal audio capture module. To do this, you can pass the extras parameter when you call getInstance to create an engine. The relevant parameter is:

user_specified_use_external_audio_record: Specifies whether to use external audio capture (which disables the SDK's internal capture).

• TRUE: Use external audio capture (disables internal SDK capture).

• FALSE: Do not use external audio capture (enables internal SDK capture).

String extras = "{\"user_specified_use_external_audio_record\":\"TRUE\"}";
mAliRtcEngine = AliRtcEngine.getInstance(this, extras);

// Create and initialize the engine.
var customAudioCaptureConfig: [String: String] = [:]
// Use external capture.
customAudioCaptureConfig["user_specified_use_external_audio_record"] = "TRUE"
// Serialize to JSON.
guard let jsonData = try? JSONSerialization.data(withJSONObject: customAudioCaptureConfig, options: []),
let extras = String(data: jsonData, encoding: .utf8) else {
     print("JSON serialization failed")
     return
 }
let engine = AliRtcEngine.sharedInstance(self, extras:extras)

/* Windows supports enabling or disabling audio capture during creation. */
/* Disable internal capture module. */
char* extra = "{\"user_specified_enable_use_virtual_audio_device\":\"TRUE\", \"user_specified_use_external_audio_record\":\"TRUE\"}";
mAliRtcEngine = AliRtcEngine.Create(extra);

/* Enable internal capture module. */
char* extra = "{\"user_specified_enable_use_virtual_audio_device\":\"FALSE\", \"user_specified_use_external_audio_record\":\"FALSE\"}";
mAliRtcEngine = AliRtcEngine.Create(extra);

2. Add an external audio stream

Call the addExternalAudioStream method to add an external audio stream and obtain its stream ID. If you need the SDK's 3A audio processing features (AEC, AGC, ANS), set the enable3A parameter in the AliRtcExternalAudioStreamConfig object.

AliRtcEngine.AliRtcExternalAudioStreamConfig config = new AliRtcEngine.AliRtcExternalAudioStreamConfig();
config.sampleRate = SAMPLE_RATE; // Sample rate
config.channels = CHANNEL; // Number of channels
// Publishing volume
config.publishVolume = 100;
// Local playback volume
config.playoutVolume = isLocalPlayout ? 100 : 0;
config.enable3A = true;

int result = mAliRtcEngine.addExternalAudioStream(config);
if (result <= 0) {
    return;
}
// The return value is the stream ID. You will need it to push data to the SDK later.
mExternalAudioStreamId = result;

/* Set the parameters as needed for your business. */
AliRtcExternalAudioStreamConfig *config = [AliRtcExternalAudioStreamConfig new];
// This must be the same as the number of channels of the external PCM audio stream. Set it to 1 for mono or 2 for stereo.
config.channels = _pcmChannels;
// This must be the same as the sample rate of the external PCM audio stream.
config.sampleRate = _pcmSampleRate;
config.playoutVolume = 0;
config.publishVolume = 100;
_externalPlayoutStreamId = [self.engine addExternalAudioStream:config];

/* Get the media engine. */
IAliEngineMediaEngine* mAliRtcMediaEngine = nullptr;
    
mAliRtcEngine->QueryInterface(AliEngineInterfaceMediaEngine, (void **)&mAliRtcMediaEngine);
/* Set the parameters as needed for your business. */
AliEngineExternalAudioStreamConfig config;
config.playoutVolume = currentAudioPlayoutVolume;
config.publishVolume = currentAudioPublishVolume;
config.channels = 1;
config.sampleRate = 48000;
config.publishStream = 0;
audioStreamID = mAliRtcMediaEngine->AddExternalAudioStream(config);

mAliRtcMediaEngine->Release();

3. Implement a custom audio capture module

You must implement your own logic to capture and process audio data based on your business scenario, and then pass the data to the SDK for transmission.

Alibaba Cloud provides sample code that demonstrates how to read PCM data from a local file or a microphone. For implementation details, see the Custom capture sample.

4. Push audio data to the SDK based on stream ID

After the audio stream is published (when the onAudioPublishStateChanged callback reports a status of AliRtcStatsPublished), call the pushExternalAudioStreamRawData method. Set the mExternalAudioStreamId parameter to the stream ID obtained in Step 2, and pass the captured audio data to the SDK. The captured audio data must be converted to an AliRtcAudioFrame object with the following properties:

• data: The audio data.

• numSamples: The number of sample points per channel in the provided data.

• bytesPerSample: The number of bytes per sample point, which is related to the bit depth. For 16-bit PCM, this value is 2.

• numChannels: The number of audio channels.

• samplesPerSec: The sample rate, such as 16000 or 48000.

// Assume that the captured audio data is in audioData, the data size is bytesRead bytes, and it is 10 ms of data.
if (mAliRtcEngine != null && bytesRead > 0) {
    // Construct an AliRtcAudioFrame object. bitsPerSample is the bit depth, which is usually 16.
    AliRtcEngine.AliRtcAudioFrame sample = new AliRtcEngine.AliRtcAudioFrame();
    sample.data = audioData;
    sample.numSamples = bytesRead / (channels * (bitsPerSample / 8)); // Calculate the number of samples based on the actual number of bytes read.
    sample.numChannels = channels;
    sample.samplesPerSec = sampleRate;
    sample.bytesPerSample = bitsPerSample / 8;

    int ret = 0;
    // Retry when the push fails because the buffer is full.
    int retryCount = 0;
    final int MAX_RETRY_COUNT = 20;
    final int BUFFER_WAIT_MS = 10;
    do {
        // Push the obtained data to the SDK.
        ret = mAliRtcEngine.pushExternalAudioStreamRawData(mExternalAudioStreamId, sample);
        if(ret == ErrorCodeEnum.ERR_SDK_AUDIO_INPUT_BUFFER_FULL) {
            // Handle the buffer full case. Wait for a period and retry. The maximum retry duration is several hundred ms.
            retryCount++;
            if(mExternalAudioStreamId <= 0 || retryCount >= MAX_RETRY_COUNT) {
                // The stream has stopped or the maximum number of retries has been reached. Exit the loop.
                break;
            }

            try {
                // Pause for a period.
                Thread.sleep(BUFFER_WAIT_MS);
            } catch (InterruptedException e) {
                e.printStackTrace();
                break;
            }
        } else {
            // The push is successful or another error occurred. Exit the loop directly.
            break;
        }
    } while (retryCount < MAX_RETRY_COUNT);
}

// Construct an AliRtcAudioFrame object from the captured audio data.
let sample = AliRtcAudioFrame()
sample.dataPtr = UnsafeMutableRawPointer(mutating: pcmData)
sample.samplesPerSec = pcmSampleRate
sample.bytesPerSample = Int32(MemoryLayout<Int16>.size)
sample.numOfChannels = pcmChannels
sample.numOfSamples = numOfSamples

var retryCount = 0

while retryCount < 20 {
    if !(pcmInputThread?.isExecuting ?? false) {
        break
    }
    // Push the audio data to the SDK.
    let rc = rtcEngine?.pushExternalAudioStream(externalPublishStreamId, rawData: sample) ?? 0

    // Handle buffer full.
    // 0x01070101 SDK_AUDIO_INPUT_BUFFER_FULL: The buffer is full. Retransmission is required.
    if rc == 0x01070101 && !(pcmInputThread?.isCancelled ?? true) {
        Thread.sleep(forTimeInterval: 0.03) // 30ms
        retryCount += 1;
    } else {
        if rc < 0 {
            "pushExternalAudioStream error, ret: \(rc)".printLog()
        }
        break
    }
}

/* Get the media engine. */
IAliEngineMediaEngine* mAliRtcMediaEngine = nullptr;   
mAliRtcEngine->QueryInterface(AliEngineInterfaceMediaEngine, (void **)&mAliRtcMediaEngine);

// Construct an audio frame from the data.
AliEngineAudioRawData rawData;
rawData.dataPtr = frameInfo.audio_data[0];
rawData.numOfSamples = (int) (frameInfo.audio_data[0].length / (2 * frameInfo.audio_channels));
rawData.bytesPerSample = 2;
rawData.numOfChannels = frameInfo.audio_channels;
rawData.samplesPerSec = frameInfo.audio_sample_rate;
// Push the data to the SDK.
int ret = mAliRtcMediaEngine->PushExternalAudioStreamRawData(audioStreamID, rawData);
// Handle buffer full and other errors.

// Release the media engine.
mAliRtcMediaEngine->Release();

5. Remove the external audio stream

To stop publishing audio from a custom audio source, call the removeExternalAudioStream method to remove the external audio stream.

mAliRtcEngine.removeExternalAudioStream(mExternalAudioStreamId);

[self.engine removeExternalAudioStream:_externalPublishStreamId];

/* Get the media engine. */
IAliEngineMediaEngine* mAliRtcMediaEngine = nullptr;
    
mAliRtcEngine->QueryInterface(AliEngineInterfaceMediaEngine, (void **)&mAliRtcMediaEngine);

mAliRtcMediaEngine->RemoveExternalAudioStream(audioStreamID);
mAliRtcMediaEngine->Release();

6. (Optional) Dynamically enable or disable internal capture

If your business scenario requires dynamically enabling or disabling the SDK's internal capture during a call, call the setParameter method.

/* Dynamically disable internal capture module. */
String parameter = "{\"audio\":{\"enable_system_audio_device_record\":\"FALSE\"}}";
mAliRtcEngine.setParameter(parameter);

/* Dynamically enable internal capture module. */
String parameter = "{\"audio\":{\"enable_system_audio_device_record\":\"TRUE\"}}"; 
mAliRtcEngine.setParameter(parameter);

// Dynamically disable internal capture module.
engine.setParameter("{\"audio\":{\"enable_system_audio_device_record\":\"FALSE\"}}")
// Dynamically enable internal capture module.
engine.setParameter("{\"audio\":{\"enable_system_audio_device_record\":\"TRUE\"}}")

/* Windows supports enabling or disabling audio capture during creation. */
/* Disable internal internal capture module. */
char* extra = "{\"user_specified_enable_use_virtual_audio_device\":\"TRUE\", \"user_specified_use_external_audio_record\":\"TRUE\"}";
mAliRtcEngine = AliRtcEngine.Create(extra);

/* Enable internal capture module. */
char* extra = "{\"user_specified_enable_use_virtual_audio_device\":\"FALSE\", \"user_specified_use_external_audio_record\":\"FALSE\"}";
mAliRtcEngine = AliRtcEngine.Create(extra);