All Products
Search
Document Center

Alibaba Cloud Model Studio:Qwen-Audio-3.0-Realtime real-time voice conversation HarmonyOS SDK

Last Updated:Sep 04, 2026

Use the HarmonyOS SDK for Qwen-Audio-3.0-Realtime to build real-time voice conversations with audio input and audio or text output.

User guide: For model introductions and selection advice, see Real-time voice conversation.

Quick start

  1. Obtain and configure an API key.

  2. Download the SDK and run the sample code:
    • Download the latest SDK package.
    • Extract the .tar.gz SDK package. Obtain the HAR SDK from entry/libs and add it to your project dependencies. For C++ integration, use native/libs and native/include in the package to obtain the dynamic libraries and header files.
    • Open the project in DevEco Studio. The sample code is in DashQwenAudioChatPage.ets. Replace the API key to try the feature.

Call procedure

  1. Initialize the SDK.
  2. Set parameters for your use case. Use the parameters argument of initialize to set the connection and control parameters, and use setParams to set the voice conversation parameters.
  3. Call startDialog to start the conversation.
  4. In onNuiAudioStateChanged, start the recording device based on the audio state.
  5. Continuously supply recording data in onNuiNeedAudioData, or call updateAudio to actively push recording data.
  6. Continuously receive the audio returned by the model in onNuiAssistEventCallback.
  7. Listen for events and obtain event information in onNuiEventCallback.
  8. Call stopDialog to stop the conversation, and listen for EVENT_TRANSCRIBER_COMPLETE to confirm that it has ended.
  9. When the conversation feature is no longer needed, call release to release the SDK resources.

Audio device management

Unlike Android, which uses AudioRecord and AudioTrack, HarmonyOS provides audio capture and playback through @kit.AudioKit, using AudioCapturer for recording and AudioRenderer for playback. The sample project provides the AudioRecorder.ets and AudioPlayer.ets utility classes for reuse.

Recording (AudioCapturer)

  • Create: Call audio.createAudioCapturer(capturerOptions) asynchronously. The sample uses 16 kHz, 16-bit, mono audio (SAMPLE_RATE_16000, CHANNEL_1, SAMPLE_FORMAT_S16LE, and ENCODING_TYPE_RAW).
  • Audio source (audio.SourceType):
    • SOURCE_TYPE_MIC: Raw microphone audio. Use this source when SDK-internal AEC is enabled, and send data to the SDK by calling updateAudio.
    • SOURCE_TYPE_VOICE_COMMUNICATION: Call audio on which the system has already performed echo cancellation. Use this source when SDK-internal AEC is disabled, and provide audio through onNuiNeedAudioData.
  • Data event: Call capturer.on('readData', (buffer: ArrayBuffer) => void) to continuously obtain recorded audio.
  • State event: Call capturer.on('stateChange', (state: audio.AudioState) => void). STATE_RUNNING indicates that recording has started, and STATE_STOPPED indicates that it has stopped.
  • Control: Call start() to start, stop() to stop, and release() to release the recorder.
import { audio } from '@kit.AudioKit';

const audioStreamInfo: audio.AudioStreamInfo = {
  samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_16000,
  channels: audio.AudioChannel.CHANNEL_1,
  sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
  encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioCapturerInfo: audio.AudioCapturerInfo = {
  source: audio.SourceType.SOURCE_TYPE_MIC,
  capturerFlags: 0
};
const options: audio.AudioCapturerOptions = { streamInfo: audioStreamInfo, capturerInfo: audioCapturerInfo };

audio.createAudioCapturer(options).then((capturer) => {
  capturer.on('readData', (buffer: ArrayBuffer) => {
    // Send the recorded audio to the SDK.
    nuiInstance.updateAudio(buffer, false);
  });
  capturer.start();
});

Note: HarmonyOS creates AudioCapturer asynchronously. Call start() only after creation is complete. Do not create and immediately start a recorder in STATE_OPEN. Create it first, then call start() from the STATE_OPEN callback. The sample creates the recorder during doInit and starts it during onNuiAudioStateChanged.

Playback (AudioRenderer)

  • Create: Call audio.createAudioRenderer(rendererOptions) asynchronously.
  • Sample rate: The synthesized response audio returned by DashScope Realtime is 24 kHz. Pass this rate to the AudioPlayer constructor.
  • Data event: Call renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => ...) to provide audio for playback. Return AudioDataCallbackResult.VALID after filling the buffer or INVALID when no data is available.
  • State event: Call renderer.on('stateChange', (state: audio.AudioState) => void) to listen for playback start and end events.
  • Control: Call start() to start, stop() to stop, and pause() to pause playback. Pausing retains buffered data.
import { audio } from '@kit.AudioKit';

const audioStreamInfo: audio.AudioStreamInfo = {
  samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_24000,
  channels: audio.AudioChannel.CHANNEL_1,
  sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
  encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
};
const audioRendererInfo: audio.AudioRendererInfo = {
  usage: audio.StreamUsage.STREAM_USAGE_VOICE_ASSISTANT,
  rendererFlags: 0
};
const options: audio.AudioRendererOptions = { streamInfo: audioStreamInfo, rendererInfo: audioRendererInfo };

audio.createAudioRenderer(options, (err, renderer) => {
  renderer.on('writeData', (data: ArrayBuffer): audio.AudioDataCallbackResult => {
    // Fill data with model response audio from the queue and return VALID or INVALID.
    return audio.AudioDataCallbackResult.VALID;
  });
  renderer.start();
});

AEC reference signal: When SDK-internal AEC is enabled, send the player's output audio to the SDK as the reference signal by calling nuiInstance.pushReferenceData(data, false), which corresponds to updateRefAudio on Android.

Permission declaration

Declare the microphone permission in module.json5 before recording audio:

{
  "requestPermissions": [
    { "name": "ohos.permission.MICROPHONE" }
  ]
}

Request parameters

Connection and control parameters

Pass a JSON string in the parameters argument of initialize.

Example: The following JSON string does not list every parameter. Add parameters as needed for your use case.

{
  "url": "wss://dashscope.aliyuncs.com/api-ws/v1/inference",
  "apikey": "st-****",
  "device_id": "my_device_id",
  "service_mode": "1"
}
Parameters
ParameterTypeRequiredDescription

url

String

Yes

Service endpoint:

  • wss://dashscope.aliyuncs.com/api-ws/v1/realtime?model=<model_name>
  • China (Beijing): wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/realtime?model=<model_name>
  • Singapore: wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/realtime?model=<model_name>
Replace {WorkspaceId} with your actual Workspace ID.

apikey

String

Yes

API key.

service_mode

String

Yes

Runtime mode. Set this parameter to "1" for real-time voice conversation.

device_id

String

Yes

A unique string that identifies the end user. You can use an in-app user ID or a client-generated device identifier. This ID is mainly used for log tracing and troubleshooting.

audio_update_manually

String

No

Whether to actively push audio data. Default: "false". If set to "true" and the SDK supports on-device audio capabilities such as AEC and VAD, those capabilities are enabled by default.

workspace

String

No

The storage path for on-device resource files. This parameter is required when audio_update_manually is "true" and an on-device audio capability such as AEC or VAD is enabled.

debug_path

String

No

The log file storage path. This parameter takes effect only when save_log is true in initialize. In this case, the path is required. The SDK retains at most two log files locally.

save_wav

String

No

Whether to save debug audio under debug_path. Default: "false". Valid values are "true" and "false". This parameter takes effect only when save_log is true, and debug_path must also be set.

max_log_file_size

int

No

The maximum log file size in bytes. This parameter takes effect only when save_log is true. Default: 104857600 (100 × 1024 × 1024 bytes, or 100 MiB).

log_track_level

int

No

The filter level for logs sent through onNuiLogTrackCallback. Default: 2. Valid values: 0 (VERBOSE), 1 (DEBUG), 2 (INFO), 3 (WARNING), 4 (ERROR), and 5 (NONE). A log is returned only when its level is greater than or equal to both log_track_level and the level passed to initialize. For example, if log_track_level is 2 (INFO) and level is 3 (WARNING), only WARNING and higher-level logs (values greater than or equal to 3) are returned.

aec_params

object

No

Advanced on-device AEC configuration. This object takes effect only when audio_update_manually is "true".

aec_params.enable_aec

boolean

No

Whether to enable on-device AEC. If audio_update_manually is "true" and the SDK supports on-device AEC, AEC is enabled by default.

aec_params.save_audio

boolean

No

Whether to save audio processed by the on-device AEC module. If save_wav is "true" and debug_path is set, this feature is enabled by default and the audio is saved under debug_path.

aec_params.enable_aec_data_callback

boolean

No

Whether to return AEC-processed audio through EVENT_AEC_DATA in onNuiAssistEventCallback. Default: false.

vad_params

object

No

Advanced on-device VAD configuration. This object takes effect only when audio_update_manually is "true".

vad_params.enable_vad

boolean

No

Whether to enable on-device VAD. If audio_update_manually is "true" and the SDK supports on-device VAD, VAD is enabled by default.

vad_params.save_audio

boolean

No

Whether to save audio processed by the on-device VAD module. If save_wav is "true" and debug_path is set, this feature is enabled by default and the audio is saved under debug_path.

Voice conversation parameters

Pass a JSON string in the params argument of setParams.

Example: The following JSON string does not list every parameter. Add parameters as needed for your use case.

{
  "service_type": 4,
  "nls_config": {
    "model": "qwen-audio-3.0-realtime-plus",
    "sr_format": "pcm"
  }
}
Parameters
Top-level parameterTypeRequiredDescription

service_type

int

Yes

Voice service type. Set this parameter to 4 for real-time voice conversation.

nls_config

object

Yes

Core voice conversation configuration, including model selection and conversation behavior.

nls_config.model

string

Yes

Model name. Supports the qwen-audio-3.0-realtime-plus and qwen-audio-3.0-realtime-flash model series.

nls_config.sr_format

string

Yes

Input audio format. Only pcm is supported. The default format is 16 kHz, 16-bit, mono PCM.

nls_config.modalities

string

No

A string containing an array of output modalities. Valid values:

  • ["text"]: returns text only.
  • ["audio", "text"] (default): returns both audio and text.

nls_config.voice

string

No

TTS voice. Default: longanqian. This parameter can be set only in the first session.update; later values are ignored. System voices: longanqian, longanlingxin, longanlingxi, longanxiaoxin, and longanlufeng. You can also specify a cloned voice_id created through the voice cloning API. See Voice configuration.

nls_config.enable_speech_emotion

boolean

No

Whether to enable enhanced emotional expression. When enabled, the response voice has more noticeable emotional variation. Default: true. Valid values: true and false.

nls_config.instructions

string

No

System instructions that define the model's role, response style, and behavioral preferences for the entire session.

nls_config.max_history_turns

int

No

Maximum number of historical question-answer turns allowed in a request. Valid values: 1 to 50. Default: 20.

nls_config.tools

string

No

A string containing an array of Function Calling tool definitions. After you configure this parameter, the model decides whether to call a tool based on the user input. Each definition uses type (required and fixed to function) and a function object containing name (required), description (optional), and parameters (optional). If a function has no arguments, omit parameters.
Example:

[
  {
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Queries weather information for a specified city.",
      "parameters": {
        "type": "object",
        "properties": {
          "city": {
            "type": "string",
            "description": "City"
          }
        },
        "required": [
          "city"
        ]
      }
    }
  }
]

nls_config.turn_detection

string

No

A string containing a JSON object for turn detection. If omitted, the session uses push-to-talk mode, in which audio is committed and inference is triggered manually. If set, duplex conversation mode is enabled.

nls_config.turn_detection.type

string

No

VAD type. server_vad (default) detects speech boundaries from acoustic features and automatically triggers inference. smart_turn combines acoustic and semantic signals; sounds without semantic content, such as filler sounds, do not start a turn or interrupt model playback.

nls_config.turn_detection.threshold

float

No

VAD sensitivity. This parameter applies only to server_vad and has no effect on smart_turn. Lower values make VAD more sensitive to quiet sounds and background noise; higher values require clearer and louder speech. Valid range: [-1.0, 1.0]. Default: 0.5.

nls_config.turn_detection.silence_duration_ms

int

No

Minimum silence duration after speech, in milliseconds, before the model response is triggered. This parameter applies only to server_vad and has no effect on smart_turn. Lower values reduce latency but can trigger on brief pauses. Valid range: [200, 6000]. Default: 800. Recommended for conversation: 400 to 800.

nls_config.turn_detection.voiceprint_audio_urls

string

No

A string containing an array of publicly accessible prerecorded audio URLs for the target speaker. This parameter applies only to smart_turn. In duplex conversations, it helps the model focus on the target speaker and ignore other speakers and background noise. Up to five URLs are supported. Audio must be 16 kHz PCM or WAV.

Key APIs

NativeNui

initialize

Initializes a voice conversation SDK instance. The SDK uses the singleton pattern. Do not initialize it again before you call release.

This method can block. Call it on a non-UI thread.

Method signature
public initialize(
  callback: INativeNuiCallback,
  parameters: string,
  level: number,
  save_log: boolean = false
): number
Parameters
ParameterTypeDescription

callback

INativeNuiCallback

An implementation of the event and data callback interface.

parameters

string

A JSON string that contains authentication, connection, and debugging parameters. See Connection and control parameters.

level

number

Controls the print level of the SDK's own logs.

save_log

boolean

Whether to save logs locally. If set to true, specify the path by using debug_path in the connection and control parameters. You can also set the file size by using max_log_file_size.

Return value

An error code. See Error code reference.

setParams

Sets the voice conversation parameters in JSON format. Call this method before startDialog.

Method signature
public setParams(params: string): number
Parameters
ParameterTypeDescription

params

string

Voice conversation parameters.

Return value

An error code. See Error code reference.

startDialog

Starts the conversation.

Method signature
public startDialog(vad_mode: Constants.VadMode, dialog_params: string): number
Parameters
ParameterTypeDescription

vad_mode

Constants.VadMode

VAD mode. Fixed to Constants.VadMode.TYPE_P2T.

dialog_params

string

If apikey in the connection and control parameters is a temporary API key, update it here after it expires.
JSON format:

{
  "apikey": "st-****"
}
Return value

An error code. See Error code reference.

stopDialog

Ends the conversation. After you call this method, the server returns the final conversation result and ends the task.

Method signature
public stopDialog(): number
Return value

An error code. See Error code reference.

cancelDialog

Ends the conversation immediately. After you call this method, the task ends without waiting for the server to return the final conversation result.

Method signature
public cancelDialog(): number
Return value

An error code. See Error code reference.

dialogAction

Sends a conversation action command during an interaction to update runtime behavior such as the conversation context.

Method signature
public dialogAction(params: string): number
Parameters
ParameterTypeDescription

params

string

A JSON string used to update runtime behavior such as the conversation context.

params.type

String

Set to "action".

params.command

String

Runtime command. Valid values:

  • function_call: updates a function call request.
  • play_start: when on-device AEC is used, notifies the SDK that audio playback has started.
  • play_over: when on-device AEC is used, notifies the SDK that audio playback has ended.

params.context

String

Function call request update. Used when command is "function_call".

params.context.type

String

Event type. This parameter is required when command is "function_call". conversation.item.create inserts a conversation item for historical context, supplemental text, or a tool result. After sending conversation.item.create, use response.create to trigger another inference.

params.context.item

object

Required when params.context.type is conversation.item.create. The conversation item to create.

params.context.response

object

Optional when params.context.type is response.create. Overrides the session defaults for this inference. If omitted, the current session configuration is used.

context.item parameters:

ParameterTypeDescription

id

String

Optional unique conversation item ID. If omitted, the server generates one. An error is returned if the specified ID already exists.

type

String

Required item type. Valid values:

  • message: a regular message.
  • function_call: a function call request. This type is typically generated by the server. The client can also use it to add historical context.
  • function_call_output: a tool execution result. After receiving a function_call, the client runs the tool and uses this type to write back the result.

role

String

Required for message. Valid values: system, user, and assistant.

content

array

Required for message. Each item contains a type and the associated data field. system supports input_text with text; user supports input_text with text and input_audio with Base64-encoded audio; assistant supports output_text with text.

call_id

String

Required for function_call and function_call_output. The unique ID that associates a function call request with its result.

name

String

Required for function_call. The name of the function to call.

arguments

String

Required for function_call. Function arguments as a JSON string.

output

String

Required for function_call_output. Tool execution result as a JSON string.

context.response parameters:

ParameterTypeDescription

modalities

array

Output modalities. ["text"] returns text only. ["audio", "text"] (default) returns both audio and text.

voice

string

Overrides the TTS voice for this inference.

Example:

{
  "type": "action",
  "command": "function_call",
  "context": {
    "item": {
      "call_id": "call_xxxx",
      "output": "{\"city\":\"Hangzhou\",\"condition\":\"sunny\",\"temperature\":18}",
      "type": "function_call_output"
    },
    "type": "conversation.item.create"
  }
}

{
  "type": "action",
  "command": "function_call",
  "context": {
    "response": {
      "modalities": ["text", "audio"]
    },
    "type": "response.create"
  }
}
Return value

An error code. See Error code reference.

updateAudio

When audio_update_manually is set to "true", recording data is no longer supplied through onNuiNeedAudioData. Use this method to actively push the data instead.

Method signature
public updateAudio(data: ArrayBuffer, first_pack: boolean): number
Parameters
ParameterTypeDescription

data

ArrayBuffer

Audio data to push (PCM).

first_pack

boolean

Whether this is the first audio packet. The SDK calculates the byte count from data.byteLength.

Return value

An error code. See Error code reference.

pushReferenceData (corresponds to Android updateRefAudio)

When audio_update_manually is set to "true" and on-device AEC is enabled, use this method to push audio played by the player as the reference signal.

Method signature
public pushReferenceData(data: ArrayBuffer, first_pack: boolean): number
Parameters
ParameterTypeDescription

data

ArrayBuffer

Audio data to push (PCM).

first_pack

boolean

Whether this is the first audio packet. The SDK calculates the byte count from data.byteLength.

Return value

An error code. See Error code reference.

release

Releases all internal SDK resources. The SDK instance becomes unavailable after this call. To use the SDK again, call initialize to reinitialize it.

Method signature
public release(): number
Return value

An error code. See Error code reference.

GetVersion

Returns information about the current SDK version.

Method signature
public GetVersion(): string
Return value

Information about the current SDK version.

INativeNuiCallback: listener callbacks

onNuiEventCallback: listen for event information

Method signature
onNuiEventCallback: (
  event: Constants.NuiEvent,
  resultCode: number,
  arg2: number,
  kwsResult: KwsResult,
  asrResult: AsrResult
) => void;
Parameters
ParameterTypeDescription

event

Constants.NuiEvent

Callback event.

resultCode

number

Error code. This parameter is valid when EVENT_ASR_ERROR occurs.

arg2

number

Reserved parameter.

asrResult

AsrResult

Speech recognition result.

kwsResult

KwsResult

Voice wake-up result. You do not need to use this parameter.

onNuiAudioStateChanged: listen for the audio state

The SDK uses this callback to notify the application when to start or stop recording.

Method signature
onNuiAudioStateChanged: (state: Constants.AudioState) => void
AudioState values
StateDescription

STATE_OPEN

The interaction has started. You can open the recording device and start recording.

STATE_PAUSE

The interaction has stopped. You can stop recording.

STATE_CLOSE

The SDK instance has been released. You can completely close the recording device.

onNuiAudioRMSChanged: listen for the recording volume

Listens for the recording volume, which can be displayed in the UI.

Method signature
onNuiAudioRMSChanged: (val: number) => number
Parameters
ParameterTypeDescription

val

number

Recording volume.

onNuiNeedAudioData: supply audio data

After the conversation starts, this callback is triggered continuously. Supply the audio data to process in this callback. You do not need to use this callback when audio_update_manually is set to "true".

Method signature
onNuiNeedAudioData: (buffer: ArrayBuffer) => number
Parameters
ParameterTypeDescription

buffer

ArrayBuffer

Audio data to supply. The SDK uses buffer.byteLength as the requested byte count.

Return value

The actual number of bytes supplied.

onNuiAssistEventCallback: receive auxiliary data and information

Receives internal SDK auxiliary events and related data.

Method signature
onNuiAssistEventCallback?: (
  event: Constants.NuiEvent,
  info: string,
  infoLen: number,
  data: ArrayBuffer
) => void;
Parameters
ParameterTypeDescription

event

Constants.NuiEvent

NuiEvent event.

info

string

Additional information, usually a JSON string.

infoLen

number

Length of the additional information.

data

ArrayBuffer

Additional binary data, such as TTS audio returned by the model.

Note: This callback is optional (?) on HarmonyOS. Omit it if you do not need the information.

onNuiLogTrackCallback: listen for trace logs

Receives detailed internal SDK logs for troubleshooting and debugging.

onNuiLogTrackCallback: (level: Constants.LogLevel, log: string) => void

NuiEvent: event types

In the HarmonyOS SDK, event types are defined by the Constants.NuiEvent enumeration. The following table lists the events related to real-time voice conversation:

EventDescription

EVENT_TRANSCRIBER_STARTED

The task started successfully.

EVENT_VAD_START

Triggered immediately after the task starts. This does not mean that the start of speech has been detected.

EVENT_VAD_END

The end of speech was detected.

EVENT_ASR_PARTIAL_RESULT

An intermediate speech recognition result.

EVENT_ASR_RESULT

A complete speech recognition result.

EVENT_ASR_ERROR

An error occurred during the voice conversation.

EVENT_MIC_ERROR

Triggered when no audio data is received for two consecutive seconds.

EVENT_SENTENCE_START

The start of a sentence was detected.

EVENT_SENTENCE_END

The end of a sentence was detected and a complete recognition result was returned.

EVENT_TRANSCRIBER_COMPLETE

The voice conversation ended.

EVENT_AUDIO_TRANSCRIPTION

An incremental text transcript event for audio output. Transcript segments are returned in streaming mode.

EVENT_AUDIO_TRANSCRIPTION_COMPLETED

The transcript for audio output is complete.

EVENT_OTHER_RESULT

Other event information, such as a Function Calling result.

EVENT_ASR_TTS_START

The model started returning TTS audio.

EVENT_ASR_TTS_DATA

TTS audio returned by the model.

EVENT_ASR_TTS_COMPLETE

The model finished returning TTS audio.

EVENT_RESULT_TRANSLATED

An intermediate translation result.

EVENT_RESULT_TRANSLATED_END

Translation result output is complete.

EVENT_AEC_DATA

Audio data processed by AEC.