Alibaba Cloud Model Studio:Qwen-Audio-3.0-ASR-Flash-Streaming/Fun-ASR-Realtime Speech Recognition HarmonyOS SDK

Learn about HarmonyOS SDK integration, request parameters, APIs, callbacks, and sample code for real-time speech recognition.

For model descriptions and selection guidance, see Speech recognition.

Quick start

1. Obtain an API key. Do not hard-code a long-lived API key in a client application. We recommend that your application server obtain a temporary API key and send it to the client.
2. Download the latest SDK package. Extract the package, copy entry/libs/neonui.har to the entry/libs directory of your application, and add the dependency to entry/oh-package.json5:

{
  "dependencies": {
    "neonui": "file:libs/neonui.har"
  }
}

For a HarmonyOS C++ integration, use the shared libraries in native/libs and the header files in native/include.

3. Declare network and microphone permissions in the application module.json5 file, and request microphone permission at runtime. reason_internet and reason_microphone are example resource names. Define the corresponding descriptions in your application resources.

"requestPermissions": [
  {
    "name": "ohos.permission.INTERNET",
    "reason": "$string:reason_internet",
    "usedScene": { "abilities": ["EntryAbility"], "when": "always" }
  },
  {
    "name": "ohos.permission.MICROPHONE",
    "reason": "$string:reason_microphone",
    "usedScene": { "abilities": ["EntryAbility"], "when": "always" }
  }
]

4. Open the sample project from the SDK package in DevEco Studio. The sample page is entry/src/main/ets/pages/dashscope/DashFunAsrSpeechTranscriberPage.ets. Configure an API key and run the project.

Call sequence

1. Create a NativeNui(Constants.ModeType.MODE_DIALOG) instance.
2. Call initialize to initialize the SDK and configure connection and control parameters.
3. Call setParams to configure the model and recognition parameters.
4. Call startDialog to start recognition.
5. In onNuiAudioStateChanged, start, pause, or close the recording device based on the audio state.
6. Continuously supply recording data in onNuiNeedAudioData. If active audio pushing is enabled, call updateAudio instead.
7. Retrieve recognition results and task states in onNuiEventCallback.
8. Call stopDialog, and wait for EVENT_TRANSCRIBER_COMPLETE.
9. When recognition is no longer needed, call release to release resources.

Request parameters

Connection and control parameters

Pass a JSON string in the parameters argument of initialize.

{
  "url": "wss://dashscope.aliyuncs.com/api-ws/v1/inference",
  "device_id": "my_device_id",
  "service_mode": "1",
  "audio_update_manually": "false"
}

Parameter	Type	Required	Description
url	string	Yes	Service endpoint. Use the public endpoint wss://dashscope.aliyuncs.com/api-ws/v1/inference, or a workspace-specific endpoint: wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference for Beijing or wss://{WorkspaceId}.ap-southeast-1.maas.aliyuncs.com/api-ws/v1/inference for Singapore. Replace {WorkspaceId} with your workspace ID.
service_mode	string	Yes	Run mode. Set this to "1", which is Constants.ModeFullCloud, for real-time speech recognition.
device_id	string	Yes	A unique end-user identifier, such as an in-app user ID or a client-generated device ID. It is used mainly for log tracing and troubleshooting.
apikey	string	No	API key. You can pass it during initialization. We recommend that you pass a temporary API key in the dialog_params argument of startDialog.
audio_update_manually	string	No	Whether to enable active audio push mode. Default: "false". If set to "true", call updateAudio. If set to "false", the SDK pulls audio through onNuiNeedAudioData. When this field is "true" and the SDK supports on-device audio processing such as AEC or VAD, the supported capabilities are enabled by default.
workspace	string	No	Directory that stores on-device resource files. This parameter is required when audio_update_manually is "true" and on-device audio processing such as AEC or VAD is enabled.
debug_path	string	No	Log directory. This parameter is required when save_log is true. The SDK keeps at most two log files.
save_wav	string	No	Whether to save debugging audio. Default: "false". Audio files are saved in debug_path. If set to "true", also set debug_path and pass true for save_log to initialize.
save_wav_by_id	string	No	Whether to name saved audio files by task_id for easier retrieval when save_wav is enabled. Default: "false".
max_log_file_size	number	No	Maximum size of a single log file in bytes. Default: 104857600 (100 MiB). This parameter takes effect only when save_log is true.
log_track_level	number	No	Filter level for logs returned through onNuiLogTrackCallback. Default: 2. Valid values: 0 (VERBOSE), 1 (DEBUG), 2 (INFO), 3 (WARNING), 4 (ERROR), and 5 (NONE). A log is returned only if its level is greater than or equal to both log_track_level and the level argument of initialize. For example, if the values are 2 and 3, only WARNING and higher-level logs are returned.
enable_reconnection	string	No	Whether to enable reconnection and transmission resumption after a network interruption. Default: "false".
aec_params	object	No	On-device AEC settings. Used only when audio_update_manually is "true".
aec_params.enable_aec	boolean	No	Whether to enable on-device AEC. It is enabled by default when the SDK build supports it.
aec_params.save_audio	boolean	No	Whether to save audio processed by AEC. It is enabled by default when save_wav is enabled and debug_path is set.
aec_params.enable_aec_data_callback	boolean	No	Whether to return AEC-processed data through the EVENT_AEC_DATA event of onNuiAssistEventCallback. Default: false.
vad_params	object	No	On-device VAD settings. Used only when audio_update_manually is "true".
vad_params.enable_vad	boolean	No	Whether to enable on-device VAD. It is enabled by default when the SDK build supports it.
vad_params.save_audio	boolean	No	Whether to save audio processed by VAD. It is enabled by default when save_wav is enabled and debug_path is set.
audio_config	object	No	Audio capture settings for SDK pull mode. Used only when audio_update_manually is "false".
audio_config.mic.enable_volume_calculation	boolean	No	Whether to calculate and report volume. Default: true. Disable this option if volume callbacks are not needed.
audio_config.mic.volume_mode	string	No	Volume calculation mode. Set this to "dbfs" to calculate standard dBFS as 20*log10(rms/32768), where full scale is 0 dB.

Recognition parameters

Pass a JSON string in the params argument of setParams.

{
  "service_type": 4,
  "nls_config": {
    "model": "qwen-audio-3.0-asr-flash-streaming",
    "sr_format": "opus",
    "sample_rate": 16000
  }
}

Parameter	Type	Required	Description
service_type	number	Yes	Speech service type. Set this to 4, which is Constants.kServiceTypeSpeechTranscriber.
nls_config	object	Yes	Recognition configuration object.
nls_config.model	string	Yes	Model name. Qwen-Audio-3.0-ASR-Flash-Streaming and Fun-ASR-Realtime model families are supported. See Supported models and regions.
nls_config.sr_format	string	Yes	Audio format: pcm or opus. When set to opus, your application still supplies PCM data and the SDK encodes it as Opus.
nls_config.sample_rate	number	Yes	Sample rate in Hz. 8 kHz models require 8000; other models accept any sample rate. A sample rate of 8000 Hz is not supported when on-device AEC or VAD is enabled.
nls_config.semantic_punctuation_enabled	boolean	No	Whether to enable semantic sentence segmentation. Default: false. true enables semantic segmentation and disables VAD segmentation, which is suitable for meeting transcription scenarios that prioritize segmentation accuracy. false enables VAD segmentation and disables semantic segmentation, which is suitable for interactive scenarios that prioritize low latency.
nls_config.max_sentence_silence	number	No	VAD silence threshold in milliseconds. The system determines that an utterance has ended when the silence after speech exceeds this threshold. Default: 1300. Valid range: [200, 6000]. When semantic segmentation is enabled, this parameter is not used to determine sentence_end, but an excessively small value may still affect recognition.
nls_config.multi_threshold_mode_enabled	boolean	No	Whether to enable multi-threshold mode. Default: false. This mode helps prevent excessively long VAD segments. It takes effect only when semantic_punctuation_enabled is false.
nls_config.heartbeat	boolean	No	Whether to enable heartbeat packets. Default: false. When enabled, continuously sending silent audio can keep the connection active. Otherwise, the connection times out after a period of time. Silent audio contains no audible signal in the audio file or stream.
nls_config.vocabulary_id	string	No	ID of a precompiled hotword list. Create the list in advance. Use this field when the vocabulary is known and relatively stable and the same list must be reused across requests. See Precompiled hotwords.
nls_config.instant_vocabulary	object	No	Request-level hotwords. Keys are hotword strings, and values are integer weights. No list must be created in advance, making this field suitable for temporary session-level optimization. Valid weights are [1, 5] or 50. For [1, 5], a larger value makes the model more likely to output the word. At most 50 super hotwords with a weight of 50 are allowed. When precompiled and request-level hotwords are both configured, the sets are merged. If the merged set exceeds 2000 entries, 2000 are selected at random. Only qwen-audio-3.0-asr-flash-streaming supports request-level hotwords. See Request-level hotwords.
nls_config.language_hints	string[]	No	Languages in the audio. No default. If this field is omitted, the model detects the language. Qwen-Audio-3.0-ASR-Flash-Streaming uses at most the first four values. Fun-ASR-Realtime uses only the first value. For supported language codes, see the real-time speech recognition guide.
nls_config.speech_noise_threshold	number	No	VAD speech/noise threshold. Valid range: [-1.0, 1.0]. Values closer to -1 classify more noise as speech and may transcribe more noise. Values closer to 1 classify more speech as noise and may filter some speech. This advanced setting can significantly affect recognition. Test it thoroughly and adjust it in small increments of 0.1.
nls_config.special_word_filter	object	No	Sensitive-word filtering settings. See Sensitive-word filtering.
nls_config.enable_connection_fast_check	boolean	No	Whether to enable fast network interruption detection. Default: false.

APIs

NativeNui

Import the SDK:

import { AsrResult, Constants, INativeNuiCallback, KwsResult, NativeNui } from 'neonui';

Create an instance

constructor(mode_type: Constants.ModeType, flag?: string)

Parameter	Type	Description
mode_type	Constants.ModeType	SDK mode: MODE_DIALOG for dialog or recognition, MODE_TTS for speech synthesis, or MODE_STREAM_INPUT_TTS for streaming-input text-to-speech. Set this to MODE_DIALOG for real-time speech recognition.
flag	string	Optional instance identifier used to distinguish logs.

initialize

initialize(
  callback: INativeNuiCallback,
  parameters: string,
  level: number,
  save_log: boolean = false
): number

Initializes the SDK. This method can block. Do not call it on the UI thread.

Parameter	Type	Description
callback	INativeNuiCallback	Event and data callback.
parameters	string	JSON string of connection and control parameters.
level	number	SDK log level: LOG_LEVEL_VERBOSE (0), LOG_LEVEL_DEBUG (1), LOG_LEVEL_INFO (2), LOG_LEVEL_WARNING (3), LOG_LEVEL_ERROR (4), or LOG_LEVEL_NONE (5).
save_log	boolean	Whether to save local logs. Default: false. If set to true, set debug_path in parameters. You can also set max_log_file_size.

The method returns an error code.

setParams

setParams(params: string): number

Sets recognition parameters before startDialog. params is a JSON string that contains the recognition parameters. The method returns an error code.

startDialog

startDialog(vad_mode: Constants.VadMode, dialog_params: string): number

Starts recognition.

Parameter	Type	Description
vad_mode	Constants.VadMode	VAD mode. Set this to Constants.VadMode.TYPE_P2T for real-time speech recognition.
dialog_params	string	JSON string. Use it to refresh an expired temporary API key or pass input context in input_context.

Example:

{
  "apikey": "st-****",
  "input_context": [
    { "role": "user", "content": [{ "type": "input_text", "text": "Example context" }] }
  ]
}

The method returns an error code.

stopDialog

stopDialog(): number

Notifies the server to finish recognition and return the final result. The task ends when EVENT_TRANSCRIBER_COMPLETE is received. The method returns an error code.

cancelDialog

cancelDialog(): number

Immediately terminates recognition without waiting for a final server result. The method returns an error code.

dialogAction

dialogAction(action_params: string): number

Sends a runtime action to update recognition context or notify AEC of playback state changes.

Parameter	Type	Description
action_params	string	JSON string that contains the action.
action_params.type	string	Set this to "action".
action_params.command	string	Action command: "context" updates input context, "play_start" notifies AEC that reference-audio playback has started, and "play_over" notifies AEC that playback has ended.
action_params.context	object[]	Input context to use when command is "context".

Example for updating context:

{
  "type": "action",
  "command": "context",
  "context": [
    {
      "role": "user",
      "content": [
        { "text": "Example context", "type": "input_text" }
      ]
    }
  ]
}

The method returns an error code.

updateAudio

updateAudio(data: ArrayBuffer, first_pack: boolean): number

When audio_update_manually is "true", actively pushes recording data through this method instead of filling onNuiNeedAudioData.

Parameter	Type	Description
data	ArrayBuffer	Audio data to recognize.
first_pack	boolean	Whether this is the first audio packet. Set this to true for the first packet and false for subsequent packets.

The method returns an error code.

pushReferenceData

pushReferenceData(data: ArrayBuffer, first_pack: boolean): number

When audio_update_manually is "true" and on-device AEC is enabled, pushes audio played by the player as the AEC reference signal.

Parameter	Type	Description
data	ArrayBuffer	Reference audio data.
first_pack	boolean	Whether this is the first audio packet. Set this to true for the first packet and false for subsequent packets.

The method returns an error code.

release

release(): number

Releases all internal SDK resources. The instance cannot be used after this call. To use it again, call initialize first. The method returns an error code.

GetVersion

GetVersion(): string

Returns the current SDK version.

refreshApikey

refreshApikey(apikey: string, url: string = ''): string

Refreshes an API key and returns a temporary authentication token. This method performs a synchronous network call. Do not call it on the UI thread.

Parameter	Type	Description
apikey	string	Existing API key.
url	string	Optional authentication endpoint. Default: an empty string, which uses the default endpoint.

INativeNuiCallback

onNuiEventCallback

onNuiEventCallback: (
  event: Constants.NuiEvent,
  resultCode: number,
  arg2: number,
  kwsResult: KwsResult,
  asrResult: AsrResult
) => void;

Receives recognition events and results.

Parameter	Type	Description
event	Constants.NuiEvent	Callback event.
resultCode	number	Error code. This field is valid for EVENT_ASR_ERROR.
arg2	number	Reserved.
kwsResult	KwsResult	Wake-word result. Ignore this field for real-time speech recognition.
asrResult	AsrResult	Recognition result. allResponse contains the complete server JSON response. Retrieve the task ID from header.task_id and utterance text from payload.output.sentence.text.

Events:

Event	Description
EVENT_TRANSCRIBER_STARTED	The task has started. header.task_id in asrResult.allResponse contains the task ID. Record this ID for troubleshooting.
EVENT_VAD_START	Triggered after the task starts. This event does not indicate that the start of speech was detected.
EVENT_VAD_END	The end of speech was detected.
EVENT_SENTENCE_START	The start of an utterance was detected.
EVENT_ASR_PARTIAL_RESULT	An intermediate recognition result is available.
EVENT_SENTENCE_END	The end of an utterance was detected, and the complete result for the utterance is available.
EVENT_ASR_WARN	A non-fatal warning occurred during recognition, such as a network interruption when reconnection is enabled.
EVENT_ASR_ERROR	An error occurred during recognition. resultCode contains the error code.
EVENT_MIC_ERROR	No audio data was received for two consecutive seconds. Check the recording code, permissions, or whether another application is using the recorder.
EVENT_TRANSCRIBER_COMPLETE	Recognition has ended.
EVENT_AEC_DATA	AEC-processed audio data returned through onNuiAssistEventCallback.

onNuiAudioStateChanged

onNuiAudioStateChanged: (state: Constants.AudioState) => void;

The SDK uses this callback to notify the application when to start or stop recording.

State	Description
STATE_OPEN	The interaction has started. The recording device can be opened.
STATE_PAUSE	The interaction has stopped. Recording can be stopped.
STATE_CLOSE	The SDK instance has been released. The recording device can be closed completely.

On HarmonyOS, AudioCapturer is created asynchronously. Create the recorder instance during initialization. When STATE_CLOSE is received, stop recording but retain the instance for reuse. Release it in the unified release flow. This prevents a recreated recorder from ignoring an immediate start call when the next STATE_OPEN event arrives.

onNuiNeedAudioData

onNuiNeedAudioData: (buffer: ArrayBuffer) => number;

Called continuously when the SDK pulls audio. Fill buffer.byteLength bytes, typically 20 ms of mono 16-bit PCM, and return the number of bytes written. A value less than or equal to 0 indicates an error or no available data.

onNuiAudioRMSChanged

onNuiAudioRMSChanged: (val: number) => number;

Reports the current audio volume for UI updates. When audio_config.mic.volume_mode is "dbfs", val ranges from -160 to 0. The callback implementation can return 0.

onNuiAssistEventCallback

onNuiAssistEventCallback?: (
  event: Constants.NuiEvent,
  info: string,
  infoLen: number,
  data: ArrayBuffer
) => void;

Optional callback for internal SDK auxiliary events and data. Omit this callback if the information is not needed.

Parameter	Type	Description
event	Constants.NuiEvent	Auxiliary event.
info	string	Additional information, usually a JSON string.
infoLen	number	Length of the additional information.
data	ArrayBuffer	Auxiliary data, such as AEC-processed audio.

onNuiLogTrackCallback

onNuiLogTrackCallback: (level: Constants.LogLevel, log: string) => void;

Receives SDK trace logs. The returned levels are determined by both log_track_level and the level argument of initialize.

Result objects

AsrResult

Property	Type	Description
finish	boolean	Whether the current result is final.
resultCode	number	Result status code.
asrResult	string	Recognition text. For EVENT_ASR_ERROR, this field contains the error message.
allResponse	string	Complete server response as a JSON string, including the task ID and utterance text.

KwsResult

Property	Type	Description
type	Constants.WuwType	Wake-word type. Ignore this field for real-time speech recognition.
kws	string	Wake word. Ignore this field for real-time speech recognition.

Constants and enumerations

Name	Description
Constants.ModeType	SDK modes: MODE_DIALOG, MODE_TTS, and MODE_STREAM_INPUT_TTS.
Constants.VadMode	VAD mode. Real-time speech recognition uses TYPE_P2T, where the user calls stopDialog to end recognition.
Constants.AudioState	Audio states: STATE_OPEN, STATE_PAUSE, and STATE_CLOSE.
Constants.LogLevel	Log levels from LOG_LEVEL_VERBOSE (0) to LOG_LEVEL_NONE (5).
Constants.NuiResultCode	SDK error codes, such as SUCCESS (0), ILLEGAL_PARAM (240002), NECESSARY_PARAM_LACK (240004), and SDK_NOT_INIT (240011).
Constants.kServiceTypeSpeechTranscriber	service_type value for real-time speech recognition. The value is 4.
Constants.ModeFullCloud	Full-cloud mode. The service_mode value is "1".

Sample code

The following code shows the core SDK flow. For complete permission handling, recording queue, and response parsing, see DashFunAsrSpeechTranscriberPage.ets in the SDK package.

import { AsrResult, Constants, INativeNuiCallback, KwsResult, NativeNui } from 'neonui';

const callback: INativeNuiCallback = {
  onNuiEventCallback: (event: Constants.NuiEvent, resultCode: number, arg2: number,
    kwsResult: KwsResult, asrResult: AsrResult): void => {
    if (event == Constants.NuiEvent.EVENT_ASR_PARTIAL_RESULT
      || event == Constants.NuiEvent.EVENT_SENTENCE_END) {
      // Parse payload.output.sentence.text from asrResult.allResponse.
    } else if (event == Constants.NuiEvent.EVENT_TRANSCRIBER_COMPLETE) {
      // Recognition is complete.
    } else if (event == Constants.NuiEvent.EVENT_ASR_ERROR) {
      // resultCode contains the error code.
    }
  },
  onNuiAudioStateChanged: (state: Constants.AudioState): void => {
    // Control AudioCapturer for STATE_OPEN, STATE_PAUSE, and STATE_CLOSE.
  },
  onNuiNeedAudioData: (buffer: ArrayBuffer): number => {
    // Copy recording data to buffer and return the number of bytes written.
    return 0;
  },
  onNuiAudioRMSChanged: (val: number): number => 0,
  onNuiLogTrackCallback: (level: Constants.LogLevel, log: string): void => {}
};

const nuiInstance = new NativeNui(Constants.ModeType.MODE_DIALOG);

const initParams: Record<string, Object> = {};
initParams['url'] = 'wss://dashscope.aliyuncs.com/api-ws/v1/inference';
initParams['device_id'] = 'my_device_id';
initParams['service_mode'] = Constants.ModeFullCloud;
initParams['audio_update_manually'] = 'false';

const initResult = nuiInstance.initialize(
  callback,
  JSON.stringify(initParams),
  Constants.LogLevel.LOG_LEVEL_DEBUG,
  false
);

if (initResult == Constants.NuiResultCode.SUCCESS) {
  const nlsConfig: Record<string, Object> = {
    'model': 'qwen-audio-3.0-asr-flash-streaming',
    'sr_format': 'opus',
    'sample_rate': 16000
  };
  const params: Record<string, Object> = {
    'service_type': Constants.kServiceTypeSpeechTranscriber,
    'nls_config': nlsConfig
  };
  nuiInstance.setParams(JSON.stringify(params));

  const dialogParams: Record<string, Object> = { 'apikey': 'st-****' };
  nuiInstance.startDialog(Constants.VadMode.TYPE_P2T, JSON.stringify(dialogParams));
}

// Stop recognition when the user finishes recording, and wait for EVENT_TRANSCRIBER_COMPLETE.
function stopRecognition(): void {
  nuiInstance.stopDialog();
}
// Call nuiInstance.release() from the EVENT_TRANSCRIBER_COMPLETE handler.

Use AudioCapturer from @kit.AudioKit for recording. Audio must be mono 16-bit PCM with a sample rate supported by the selected model.

import { audio } from '@kit.AudioKit';

const options: audio.AudioCapturerOptions = {
  streamInfo: {
    samplingRate: audio.AudioSamplingRate.SAMPLE_RATE_16000,
    channels: audio.AudioChannel.CHANNEL_1,
    sampleFormat: audio.AudioSampleFormat.SAMPLE_FORMAT_S16LE,
    encodingType: audio.AudioEncodingType.ENCODING_TYPE_RAW
  },
  capturerInfo: {
    source: audio.SourceType.SOURCE_TYPE_MIC,
    capturerFlags: 0
  }
};

const capturer = await audio.createAudioCapturer(options);
capturer.on('readData', (buffer: ArrayBuffer): void => {
  // Pull mode: enqueue data for onNuiNeedAudioData.
  // Push mode: call nuiInstance.updateAudio(buffer, firstPack).
});
await capturer.start();