Paraformer real-time speech recognition HarmonyOS SDK_Alibaba Cloud Model Studio(Model Studio) - Alibaba Cloud Model Studio

This guide explains how to use the Paraformer real-time speech recognition HarmonyOS SDK to convert speech to text.

ImportantAlibaba Cloud Model Studio has introduced a workspace-specific domain for the China (Beijing) region. The domain provides superior performance and higher stability for inference requests. We recommend migrating from dashscope.aliyuncs.com to {WorkspaceId}.cn-beijing.maas.aliyuncs.com.

Replace {WorkspaceId} with your actual workspace ID. The existing domain remains available.

User guide: For model introductions and selection recommendations, see Real-time speech recognition - Fun-ASR and Paraformer.

Online experience: Only paraformer-realtime-v2, paraformer-realtime-8k-v2, and paraformer-realtime-v1 support the online experience.

Quick start

Obtain an API key: Obtain and configure an API key. For security, we recommend configuring the API key as an environment variable.

NoteTo grant temporary access to third-party applications or users, or to strictly control high-risk operations such as accessing or deleting sensitive data, use a temporary API key. A temporary API key is valid for 60 seconds by default. Obtain a new one after it expires.
Download the SDK and run the sample code:
- Download the latest SDK package.
- Extract the TAR package. Obtain the HAR-format SDK from the neonui directory and add it to your project dependencies. For C++ integration, obtain the dynamic libraries and header files from native/libs and native/include in the TAR package.
- Open the project in DevEco Studio. The sample code is located in DashParaformerSpeechTranscriberPage.ets. Replace the API key to try the feature.

Invocation steps

Initialize the SDK.
Configure parameters based on your business requirements: Use the parameters parameter of initialize to set connection and control parameters, and use setParams to set speech recognition effect parameters.
Call startDialog to start recognition.
In the onNuiAudioStateChanged callback, start the recording device based on the audio state.
In the onNuiNeedAudioData callback, continuously provide recorded audio data.
In the onNuiEventCallback callback, listen for events and obtain speech recognition results.
Call stopDialog to stop recognition, and listen for the EVENT_TRANSCRIBER_COMPLETE event to confirm that recognition has ended.
When speech recognition is no longer needed, call release to release SDK resources.

Request parameters

Connection and control parameters

Pass a JSON string to the parameters parameter of initialize to configure the following parameters. Example: The following JSON string is an example and does not include all parameters. Add parameters as needed:

{
    "url": "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference",
    "apikey": "st-****",
    "device_id": "my_device_id",
    "service_mode": "1"
}

Parameter descriptions

Parameter	Type	Required	Description
`url`	`string`	Yes	The endpoint. This is fixed at `wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference`. Replace `{WorkspaceId}` with your actual workspace ID.
`apikey`	`string`	Yes	The API key. We recommend using a more secure temporary API key with a short validity period to reduce the risk of leaking a long-term key.
`service_mode`	`string`	Yes	The operating mode. For real-time speech recognition, this is fixed at `"1"`.
`device_id`	`string`	Yes	A unique string that identifies the end user. You can set it to an in-app user ID or a unique device identifier generated by the client. This ID is mainly used for log tracking and troubleshooting.
`debug_path`	`string`	No	The storage path for log files. This parameter takes effect only if you set `save_log` to true when you call initialize. In this case, you must specify a log file path. Otherwise, an error occurs. A maximum of two log files are retained locally.
`save_wav`	`string`	No	Specifies whether to save audio files for debugging. Audio files are saved under `debug_path`. Default value: `"false"`. Valid values: - `"true"`: yes - `"false"`: no. This parameter takes effect only if you set `save_log` to true when you call initialize. You must also set `debug_path`.
`max_log_file_size`	`number`	No	Sets the maximum size of a log file in bytes. This parameter takes effect only if you set `save_log` to true when you call initialize. Default value: 104857600 (100 * 1024 * 1024 bytes, or 100 MiB).
`log_track_level`	`number`	No	Controls the filtering level of log content sent externally through the `onNuiLogTrackCallback` callback. Default value: 2. Valid values: - 0: LOG_LEVEL_VERBOSE - 1: LOG_LEVEL_DEBUG - 2: LOG_LEVEL_INFO - 3: LOG_LEVEL_WARNING - 4: LOG_LEVEL_ERROR - 5: LOG_LEVEL_NONE (disables this feature). Note: `log_track_level` and `level`, which is set through initialize, together determine which logs are sent to the callback. A log level must be greater than or equal to both `log_track_level` and `level` to trigger the callback. For example, if `log_track_level` is 2 (INFO) and `level` is 3 (WARNING), only logs at WARNING or a higher level (value >=3) trigger the callback.

Speech recognition effect parameters

Pass a JSON string to the params parameter of setParams to configure the following parameters. Example: The following JSON string is an example and does not include all parameters. Add parameters as needed:

{
    "service_type": 4,
    "nls_config": {
        "model": "paraformer-realtime-v2",
        "sr_format": "pcm",
        "sample_rate": "16000"
    }
}

Parameter descriptions

Level-1 parameter	Type	Required	Description
`service_type`	`int`	Yes	The speech service type. For real-time speech recognition, this is fixed at `4`.
`nls_config`	`object`	Yes	The core speech recognition configuration object, which contains key parameters for model selection and recognition effect control.
`nls_config.model`	`string`	Yes	The speech recognition model.
`nls_config.sr_format`	`string`	Yes	The format of the audio to recognize. Supported formats: pcm, wav, and opus. Important opus: The source audio must be PCM-encoded. The SDK encodes it into OPUS. wav/pcm: The audio must be PCM-encoded.
`nls_config.sample_rate`	`int`	Yes	The sample rate of the audio to recognize, in Hz. This varies by model: - paraformer-realtime-v2 supports any sample rate. - paraformer-realtime-v1 supports only 16000 Hz. - paraformer-realtime-8k-v2 supports only 8000 Hz. - paraformer-realtime-8k-v1 supports only 8000 Hz.
`nls_config.disfluency_removal_enabled`	`boolean`	No	Specifies whether to remove disfluencies such as filler words. Default value: false.
`nls_config.language_hints`	`array[string]`	No	Specifies language codes for the audio to recognize. If the language cannot be determined in advance, omit this parameter and the model automatically detects the language. Supported language codes: - zh: Chinese - en: English - ja: Japanese - yue: Cantonese - ko: Korean - de: German - fr: French - ru: Russian. This parameter takes effect only for models that support multiple languages.
`nls_config.semantic_punctuation_enabled`	`boolean`	No	Specifies the sentence segmentation mode. Default value: false. Valid values: - true: Enables semantic segmentation and disables VAD segmentation. - false: Enables VAD segmentation and disables semantic segmentation. Semantic segmentation is more accurate and is suitable for meeting transcription. VAD (Voice Activity Detection) segmentation has lower latency and is suitable for real-time interaction. This parameter takes effect only for v2 and later models.
`nls_config.max_sentence_silence`	`int`	No	The silence threshold for VAD (Voice Activity Detection) sentence segmentation, in milliseconds. Default value: 800. Valid range: `[200, 6000]`. When the silence after a segment exceeds this threshold, the system determines that the sentence has ended. This parameter takes effect only when `semantic_punctuation_enabled` is false and the model is v2 or later.
`nls_config.multi_threshold_mode_enabled`	`boolean`	No	Specifies whether to enable the overlong-segment prevention mode. This mode prevents VAD segments from becoming too long. Default value: false (disabled). Valid values: - true: enables the mode - false: disables the mode. This parameter takes effect only when `semantic_punctuation_enabled` is false and the model is v2 or later.
`nls_config.punctuation_prediction_enabled`	`boolean`	No	Specifies whether to automatically add punctuation to recognition results. Default value: true. Valid values: - true: yes - false: no. This parameter takes effect only for v2 and later models.
`nls_config.heartbeat`	`boolean`	No	Specifies whether to maintain a persistent connection to the server. Default value: false. Valid values: - true: The connection remains active while silent audio is continuously sent. - false: Even if silent audio is continuously sent, the connection times out after a period of time. The timeout is a server-side default and cannot be configured on the client. This parameter takes effect only for v2 and later models.
`nls_config.inverse_text_normalization_enabled`	`boolean`	No	Specifies whether to enable inverse text normalization (ITN). When enabled, Chinese numerals are converted to Arabic numerals. Default value: true (enabled). Valid values: - true: enabled - false: disabled. This parameter takes effect only for v2 and later models.
`nls_config.vocabulary_id`	`string`	No	The hotword vocabulary ID, which improves the recognition accuracy of specific words. This parameter applies to v2 and later models. For more information, see Custom hotwords.
`nls_config.resources`	`array[object]`	No	The hotword resource configuration for v1 models. It provides the same function as `vocabulary_id` but uses a different configuration method. `resources` is an array of objects. Each object contains `resource_id` and `resource_type`: - `resource_id`: A `string` that specifies the hotword ID. - `resource_type`: A `string` fixed at `asr_phrase`. Example: `{ "nls_config": { "resources": [ { "resource_id": "xxxxxxxxxxxx", "resource_type": "asr_phrase" } ] } }`. For more information, see Create and manage hotwords for Paraformer speech recognition.

Key interfaces

NativeNui

initialize

Initializes a speech recognition SDK instance. Do not initialize the instance again before you call release.

This interface blocks the calling thread. Call it from a non-UI thread.

Method signature

public initialize(callback: INativeNuiCallback,
                  parameters: string,
                  level: number,
                  save_log: boolean = false): number

Parameter descriptions

Parameter	Type	Description
`callback`	`INativeNuiCallback`	An implementation of the event and data callback interface.
`parameters`	`string`	A JSON string that contains authentication, connection, and debugging parameters. See Connection and control parameters.
`level`	`number`	Controls the SDK log level. Valid values are defined by the Constants.LogLevel enumeration.
`save_log`	`boolean`	Specifies whether to save local logs. If this parameter is `true`, use `debug_path` in Connection and control parameters to specify a path. You can also use `max_log_file_size` to set the file size.

Return value

An error code. See Error code reference.

setParams

Sets the speech recognition effect parameters in JSON format. Call this interface before startDialog.

Method signature

public setParams(params: string): number

Parameter descriptions

Parameter	Type	Description
`params`	`string`	The speech recognition effect parameters.

Return value

An error code. See Error code reference.

startDialog

Starts recognition.

Method signature

public startDialog(vad_mode: Constants.VadMode, dialog_params: string): number

Parameter descriptions

Parameter	Type	Description
`vad_mode`	`Constants.VadMode`	The VAD mode. This is fixed at `Constants.VadMode.TYPE_P2T`.
`dialog_params`	`string`	If the temporary API key specified by `apikey` in Connection and control parameters has expired, update it here. The value is in JSON format: `typescript { "apikey": "st-****" }`

Return value

An error code. See Error code reference.

stopDialog

Stops recognition. After you call this interface, the server returns the final recognition result and ends the task.

Method signature

public stopDialog(): number

Return value

An error code. See Error code reference.

cancelDialog

Immediately stops recognition without waiting for the server to return the final recognition result.

Method signature

public cancelDialog(): number

Return value

An error code. See Error code reference.

release

Releases all internal SDK resources. After this method is called, the SDK instance becomes unavailable. To use it again, call initialize to reinitialize it.

Method signature

public release(): number

Return value

An error code. See Error code reference.

GetVersion

Obtains the current SDK version information.

Method signature

public GetVersion(): string

Return value

The current SDK version information.

INativeNuiCallback

Listens for events, audio states, volume changes, and logs during real-time speech recognition.

onNuiEventCallback

Listens for recognition events and obtains speech recognition results.

Method signature

onNuiEventCallback: (event: Constants.NuiEvent, resultCode: number, arg2: number,
                    kwsResult: KwsResult, asrResult: AsrResult) => void;

Parameter descriptions

Parameter	Type	Description
`event`	`Constants.NuiEvent`	The callback event.
`resultCode`	`number`	The error code. This parameter is valid when `EVENT_ASR_ERROR` occurs.
`asrResult`	`AsrResult`	The speech recognition result.
`kwsResult`	`KwsResult`	The voice wake-up feature. You do not need to use this parameter.
`arg2`	`number`	A reserved parameter.

onNuiAudioStateChanged

The SDK uses this callback to indicate when recording should start or stop.

Method signature

onNuiAudioStateChanged: (state: Constants.AudioState) => void

AudioState descriptions

State	Description
`STATE_OPEN`	The interaction starts. You can start the recording device.
`STATE_PAUSE`	The interaction is paused. You can pause recording.
`STATE_CLOSE`	The interaction stops. You can completely stop the recording device.

onNuiAudioRMSChanged

Monitors the volume of recorded audio data for UI display.

Method signature

onNuiAudioRMSChanged: (val: number) => number

Parameter descriptions

Parameter	Type	Description
`val`	`number`	The volume of the recorded audio data. The output range is generally `[-160, 0]`.

onNuiNeedAudioData

After recognition starts, this callback is continuously triggered. Supply the audio data to recognize in the callback.

Method signature

onNuiNeedAudioData: (buffer: ArrayBuffer) => number

Parameter descriptions

Parameter	Type	Description
`buffer`	`ArrayBuffer`	The audio data to supply. The SDK uses `buffer.byteLength` as the requested number of bytes.

Return value

The actual number of bytes supplied. A return value of <=0 indicates an error or no data.

onNuiLogTrackCallback

This callback receives detailed internal SDK logs for troubleshooting and debugging.

onNuiLogTrackCallback: (level: Constants.LogLevel, log: string) => void

Constants.NuiEvent

HarmonyOS SDK event types are defined by the Constants.NuiEvent enumeration. The following events apply to real-time speech recognition:

Event	Description
`EVENT_TRANSCRIBER_STARTED`	The task starts successfully.
`EVENT_VAD_START`	This event is triggered immediately after the task starts. It does not indicate that the start of speech has been detected.
`EVENT_VAD_END`	The end of speech is detected.
`EVENT_ASR_PARTIAL_RESULT`	An intermediate speech recognition result.
`EVENT_ASR_RESULT`	A complete speech recognition result.
`EVENT_ASR_ERROR`	An error occurs during speech recognition.
`EVENT_MIC_ERROR`	This event is triggered when no audio data is received for 2 consecutive seconds.
`EVENT_SENTENCE_START`	The start of a sentence is detected.
`EVENT_SENTENCE_END`	The end of a sentence is detected, and a complete recognition result is returned.
`EVENT_TRANSCRIBER_COMPLETE`	Speech recognition ends.

Auxiliary types

Constants.LogLevel

The enumeration values for the level parameter are as follows:

Value	Description
`LOG_LEVEL_VERBOSE`	The most detailed logs.
`LOG_LEVEL_DEBUG`	Debug logs.
`LOG_LEVEL_INFO`	Informational logs (default).
`LOG_LEVEL_WARNING`	Warning logs.
`LOG_LEVEL_ERROR`	Error logs.
`LOG_LEVEL_NONE`	Disables logging.

Audio device management

Unlike Android, which uses AudioRecord, HarmonyOS uses AudioCapturer from @kit.AudioKit to capture audio. The product sample encapsulates this logic in the AudioRecorder.ets utility class, which you can reuse directly.

Creation: audio.createAudioCapturer(capturerOptions) asynchronously creates the capturer with a fixed sample rate of 16 kHz, 16-bit depth, and one channel (SAMPLE_RATE_16000/CHANNEL_1/SAMPLE_FORMAT_S16LE/ENCODING_TYPE_RAW).
Data event: capturer.on('readData', (buffer: ArrayBuffer) => void) continuously obtains recorded audio data. Buffer the data in a queue so that the onNuiNeedAudioData callback can retrieve it as needed.
State event: capturer.on('stateChange', (state: audio.AudioState) => void). STATE_RUNNING indicates that recording has started, and STATE_STOPPED indicates that recording has stopped.
Control: start() starts, stop() stops, and release() releases the capturer.

NoteHarmonyOS creates AudioCapturer asynchronously. Call start() only after creation is complete. Therefore, do not create and immediately start a recorder when STATE_OPEN occurs. Create it first, and then call start() in the STATE_OPEN callback. The sample creates the recorder during doInit and starts it during onNuiAudioStateChanged. When STATE_CLOSE occurs, stop the recorder but retain the instance for reuse. Release it in release.

Permission declaration

Declare microphone permission in module.json5 to use recording:

{
  "requestPermissions": [
    { "name": "ohos.permission.MICROPHONE" }
  ]
}