Use the HarmonyOS SDK for Qwen-Audio-3.0-Realtime to build real-time voice conversations with audio input and audio or text output.
User guide: For model introductions and selection advice, see Real-time voice conversation.
Quick start
-
Download the SDK and run the sample code:
- Download the latest SDK package.
- Extract the
.tar.gzSDK package. Obtain the HAR SDK fromentry/libsand add it to your project dependencies. For C++ integration, usenative/libsandnative/includein the package to obtain the dynamic libraries and header files. - Open the project in DevEco Studio. The sample code is in
DashQwenAudioChatPage.ets. Replace the API key to try the feature.
Call procedure
- Initialize the SDK.
- Set parameters for your use case. Use the
parametersargument of initialize to set the connection and control parameters, and use setParams to set the voice conversation parameters. - Call startDialog to start the conversation.
- In onNuiAudioStateChanged, start the recording device based on the audio state.
- Continuously supply recording data in onNuiNeedAudioData, or call updateAudio to actively push recording data.
- Continuously receive the audio returned by the model in onNuiAssistEventCallback.
- Listen for events and obtain event information in onNuiEventCallback.
- Call stopDialog to stop the conversation, and listen for
EVENT_TRANSCRIBER_COMPLETEto confirm that it has ended. - When the conversation feature is no longer needed, call release to release the SDK resources.
Audio device management
Unlike Android, which uses AudioRecord and AudioTrack, HarmonyOS provides audio capture and playback through @kit.AudioKit, using AudioCapturer for recording and AudioRenderer for playback. The sample project provides the AudioRecorder.ets and AudioPlayer.ets utility classes for reuse.
Recording (AudioCapturer)
- Create: Call
audio.createAudioCapturer(capturerOptions)asynchronously. The sample uses 16 kHz, 16-bit, mono audio (SAMPLE_RATE_16000,CHANNEL_1,SAMPLE_FORMAT_S16LE, andENCODING_TYPE_RAW). -
Audio source (
audio.SourceType):SOURCE_TYPE_MIC: Raw microphone audio. Use this source when SDK-internal AEC is enabled, and send data to the SDK by callingupdateAudio.SOURCE_TYPE_VOICE_COMMUNICATION: Call audio on which the system has already performed echo cancellation. Use this source when SDK-internal AEC is disabled, and provide audio throughonNuiNeedAudioData.
- Data event: Call
capturer.on('readData', (buffer: ArrayBuffer) => void)to continuously obtain recorded audio. - State event: Call
capturer.on('stateChange', (state: audio.AudioState) => void).STATE_RUNNINGindicates that recording has started, andSTATE_STOPPEDindicates that it has stopped. - Control: Call
start()to start,stop()to stop, andrelease()to release the recorder.
import { audio } from '@kit.AudioKit';
const audioStreamInfo: audio.AudioStreamInfo = {
samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_16000,
channels: audio.AudioChannel.CHANNEL_1,
sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioCapturerInfo: audio.AudioCapturerInfo = {
source: audio.SourceType.SOURCE_TYPE_MIC,
capturerFlags: 0
};
const options: audio.AudioCapturerOptions = { streamInfo: audioStreamInfo, capturerInfo: audioCapturerInfo };
audio.createAudioCapturer(options).then((capturer) => {
capturer.on('readData', (buffer: ArrayBuffer) => {
// Send the recorded audio to the SDK.
nuiInstance.updateAudio(buffer, false);
});
capturer.start();
});
Note: HarmonyOS creates
AudioCapturerasynchronously. Callstart()only after creation is complete. Do not create and immediately start a recorder inSTATE_OPEN. Create it first, then callstart()from theSTATE_OPENcallback. The sample creates the recorder duringdoInitand starts it duringonNuiAudioStateChanged.
Playback (AudioRenderer)
- Create: Call
audio.createAudioRenderer(rendererOptions)asynchronously. - Sample rate: The synthesized response audio returned by DashScope Realtime is 24 kHz. Pass this rate to the
AudioPlayerconstructor. - Data event: Call
renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => ...)to provide audio for playback. ReturnAudioDataCallbackResult.VALIDafter filling the buffer orINVALIDwhen no data is available. - State event: Call
renderer.on('stateChange', (state: audio.AudioState) => void)to listen for playback start and end events. - Control: Call
start()to start,stop()to stop, andpause()to pause playback. Pausing retains buffered data.
import { audio } from '@kit.AudioKit';
const audioStreamInfo: audio.AudioStreamInfo = {
samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_24000,
channels: audio.AudioChannel.CHANNEL_1,
sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioRendererInfo: audio.AudioRendererInfo = {
usage: audio.StreamUsage.STREAM_USAGE_VOICE_ASSISTANT,
rendererFlags: 0
};
const options: audio.AudioRendererOptions = { streamInfo: audioStreamInfo, rendererInfo: audioRendererInfo };
audio.createAudioRenderer(options, (err, renderer) => {
renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => {
// Fill data with model response audio from the queue and return VALID or INVALID.
return audio.AudioDataCallbackResult.VALID;
});
renderer.start();
});
AEC reference signal: When SDK-internal AEC is enabled, send the player's output audio to the SDK as the reference signal by calling
nuiInstance.pushReferenceData(data, false), which corresponds toupdateRefAudioon Android.
Permission declaration
Declare the microphone permission in module.json5 before recording audio:
{
"requestPermissions": [
{ "name": "ohos.permission.MICROPHONE" }
]
}
Request parameters
Connection and control parameters
Pass a JSON string in the parameters argument of initialize.
Example: The following JSON string does not list every parameter. Add parameters as needed for your use case.
{
"url": "wss://dashscope.aliyuncs.com/api-ws/v1/inference",
"apikey": "st-****",
"device_id": "my_device_id",
"service_mode": "1"
}
Parameters
| Parameter | Type | Required | Description |
|---|---|---|---|
|
| Yes | Service endpoint:
{WorkspaceId} with your actual Workspace ID. |
|
| Yes | API key. |
|
| Yes | Runtime mode. Set this parameter to |
|
| Yes | A unique string that identifies the end user. You can use an in-app user ID or a client-generated device identifier. This ID is mainly used for log tracing and troubleshooting. |
|
| No | Whether to actively push audio data. Default: |
|
| No | The storage path for on-device resource files. This parameter is required when |
|
| No | The log file storage path. This parameter takes effect only when |
|
| No | Whether to save debug audio under |
|
| No | The maximum log file size in bytes. This parameter takes effect only when |
|
| No | The filter level for logs sent through |
|
| No | Advanced on-device AEC configuration. This object takes effect only when |
|
| No | Whether to enable on-device AEC. If |
|
| No | Whether to save audio processed by the on-device AEC module. If |
|
| No | Whether to return AEC-processed audio through |
|
| No | Advanced on-device VAD configuration. This object takes effect only when |
|
| No | Whether to enable on-device VAD. If |
|
| No | Whether to save audio processed by the on-device VAD module. If |
Voice conversation parameters
Pass a JSON string in the params argument of setParams.
Example: The following JSON string does not list every parameter. Add parameters as needed for your use case.
{
"service_type": 4,
"nls_config": {
"model": "qwen-audio-3.0-realtime-plus",
"sr_format": "pcm"
}
}
Parameters
| Top-level parameter | Type | Required | Description |
|---|---|---|---|
|
| Yes | Voice service type. Set this parameter to |
|
| Yes | Core voice conversation configuration, including model selection and conversation behavior. |
|
| Yes | Model name. Supports the |
|
| Yes | Input audio format. Only |
|
| No | A string containing an array of output modalities. Valid values:
|
|
| No | TTS voice. Default: |
|
| No | Whether to enable enhanced emotional expression. When enabled, the response voice has more noticeable emotional variation. Default: |
|
| No | System instructions that define the model's role, response style, and behavioral preferences for the entire session. |
|
| No | Maximum number of historical question-answer turns allowed in a request. Valid values: 1 to 50. Default: |
|
| No | A string containing an array of Function Calling tool definitions. After you configure this parameter, the model decides whether to call a tool based on the user input. Each definition uses |
|
| No | A string containing a JSON object for turn detection. If omitted, the session uses push-to-talk mode, in which audio is committed and inference is triggered manually. If set, duplex conversation mode is enabled. |
|
| No | VAD type. |
|
| No | VAD sensitivity. This parameter applies only to |
|
| No | Minimum silence duration after speech, in milliseconds, before the model response is triggered. This parameter applies only to |
|
| No | A string containing an array of publicly accessible prerecorded audio URLs for the target speaker. This parameter applies only to |
Key APIs
NativeNui
initialize
Initializes a voice conversation SDK instance. The SDK uses the singleton pattern. Do not initialize it again before you call release.
This method can block. Call it on a non-UI thread.
Method signaturepublic initialize(
callback: INativeNuiCallback,
parameters: string,
level: number,
save_log: boolean = false
): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| An implementation of the event and data callback interface. |
|
| A JSON string that contains authentication, connection, and debugging parameters. See Connection and control parameters. |
|
| Controls the print level of the SDK's own logs. |
|
| Whether to save logs locally. If set to |
An error code. See Error code reference.
setParams
Sets the voice conversation parameters in JSON format. Call this method before startDialog.
Method signaturepublic setParams(params: string): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
|
An error code. See Error code reference.
startDialog
Starts the conversation.
Method signaturepublic startDialog(vad_mode: Constants.VadMode, dialog_params: string): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| VAD mode. Fixed to |
|
| If |
An error code. See Error code reference.
stopDialog
Ends the conversation. After you call this method, the server returns the final conversation result and ends the task.
Method signaturepublic stopDialog(): number
Return value
An error code. See Error code reference.
cancelDialog
Ends the conversation immediately. After you call this method, the task ends without waiting for the server to return the final conversation result.
Method signaturepublic cancelDialog(): number
Return value
An error code. See Error code reference.
dialogAction
Sends a conversation action command during an interaction to update runtime behavior such as the conversation context.
Method signaturepublic dialogAction(params: string): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| A JSON string used to update runtime behavior such as the conversation context. |
|
| Set to |
|
| Runtime command. Valid values:
|
|
| Function call request update. Used when |
|
| Event type. This parameter is required when |
|
| Required when |
|
| Optional when |
context.item parameters:
| Parameter | Type | Description |
|---|---|---|
|
| Optional unique conversation item ID. If omitted, the server generates one. An error is returned if the specified ID already exists. |
|
| Required item type. Valid values:
|
|
| Required for |
|
| Required for |
|
| Required for |
|
| Required for |
|
| Required for |
|
| Required for |
context.response parameters:
| Parameter | Type | Description |
|---|---|---|
|
| Output modalities. |
|
| Overrides the TTS voice for this inference. |
Example:
{
"type": "action",
"command": "function_call",
"context": {
"item": {
"call_id": "call_xxxx",
"output": "{\"city\":\"Hangzhou\",\"condition\":\"sunny\",\"temperature\":18}",
"type": "function_call_output"
},
"type": "conversation.item.create"
}
}
{
"type": "action",
"command": "function_call",
"context": {
"response": {
"modalities": ["text", "audio"]
},
"type": "response.create"
}
}
Return value
An error code. See Error code reference.
updateAudio
When audio_update_manually is set to "true", recording data is no longer supplied through onNuiNeedAudioData. Use this method to actively push the data instead.
public updateAudio(data: ArrayBuffer, first_pack: boolean): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| Audio data to push (PCM). |
|
| Whether this is the first audio packet. The SDK calculates the byte count from |
An error code. See Error code reference.
pushReferenceData (corresponds to Android updateRefAudio)
When audio_update_manually is set to "true" and on-device AEC is enabled, use this method to push audio played by the player as the reference signal.
public pushReferenceData(data: ArrayBuffer, first_pack: boolean): number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| Audio data to push (PCM). |
|
| Whether this is the first audio packet. The SDK calculates the byte count from |
An error code. See Error code reference.
release
Releases all internal SDK resources. The SDK instance becomes unavailable after this call. To use the SDK again, call initialize to reinitialize it.
Method signaturepublic release(): number
Return value
An error code. See Error code reference.
GetVersion
Returns information about the current SDK version.
Method signaturepublic GetVersion(): string
Return value
Information about the current SDK version.
INativeNuiCallback: listener callbacks
onNuiEventCallback: listen for event information
Method signatureonNuiEventCallback: (
event: Constants.NuiEvent,
resultCode: number,
arg2: number,
kwsResult: KwsResult,
asrResult: AsrResult
) => void;
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| Callback event. |
|
| Error code. This parameter is valid when |
|
| Reserved parameter. |
|
| Speech recognition result. |
|
| Voice wake-up result. You do not need to use this parameter. |
onNuiAudioStateChanged: listen for the audio state
The SDK uses this callback to notify the application when to start or stop recording.
Method signatureonNuiAudioStateChanged: (state: Constants.AudioState) => void
AudioState values
| State | Description |
|---|---|
| The interaction has started. You can open the recording device and start recording. |
| The interaction has stopped. You can stop recording. |
| The SDK instance has been released. You can completely close the recording device. |
onNuiAudioRMSChanged: listen for the recording volume
Listens for the recording volume, which can be displayed in the UI.
Method signatureonNuiAudioRMSChanged: (val: number) => number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| Recording volume. |
onNuiNeedAudioData: supply audio data
After the conversation starts, this callback is triggered continuously. Supply the audio data to process in this callback. You do not need to use this callback when audio_update_manually is set to "true".
onNuiNeedAudioData: (buffer: ArrayBuffer) => number
Parameters
| Parameter | Type | Description |
|---|---|---|
|
| Audio data to supply. The SDK uses |
The actual number of bytes supplied.
onNuiAssistEventCallback: receive auxiliary data and information
Receives internal SDK auxiliary events and related data.
Method signatureonNuiAssistEventCallback?: (
event: Constants.NuiEvent,
info: string,
infoLen: number,
data: ArrayBuffer
) => void;
Parameters
| Parameter | Type | Description |
|---|---|---|
|
|
|
|
| Additional information, usually a JSON string. |
|
| Length of the additional information. |
|
| Additional binary data, such as TTS audio returned by the model. |
Note: This callback is optional (
?) on HarmonyOS. Omit it if you do not need the information.
onNuiLogTrackCallback: listen for trace logs
Receives detailed internal SDK logs for troubleshooting and debugging.
onNuiLogTrackCallback: (level: Constants.LogLevel, log: string) => void
NuiEvent: event types
In the HarmonyOS SDK, event types are defined by the Constants.NuiEvent enumeration. The following table lists the events related to real-time voice conversation:
| Event | Description |
|---|---|
| The task started successfully. |
| Triggered immediately after the task starts. This does not mean that the start of speech has been detected. |
| The end of speech was detected. |
| An intermediate speech recognition result. |
| A complete speech recognition result. |
| An error occurred during the voice conversation. |
| Triggered when no audio data is received for two consecutive seconds. |
| The start of a sentence was detected. |
| The end of a sentence was detected and a complete recognition result was returned. |
| The voice conversation ended. |
| An incremental text transcript event for audio output. Transcript segments are returned in streaming mode. |
| The transcript for audio output is complete. |
| Other event information, such as a Function Calling result. |
| The model started returning TTS audio. |
| TTS audio returned by the model. |
| The model finished returning TTS audio. |
| An intermediate translation result. |
| Translation result output is complete. |
| Audio data processed by AEC. |