Alibaba Cloud Model Studio:Sambert speech synthesis HarmonyOS SDK

This guide explains how to use the Sambert speech synthesis HarmonyOS SDK to convert text into high-quality, expressive speech.

User guide: For model introductions and selection recommendations, see Speech synthesis - Sambert.

Online experience: Not supported.

Replace {WorkspaceId} with your actual workspace ID. The existing domain remains available.

NativeNui

This SDK is based on the NativeNui architecture and uses callbacks to process speech synthesis events.

Architecture characteristics:

• Instance mode: Create a speech synthesis instance by using new NativeNui(Constants.ModeType.MODE_TTS).
• Callback-driven: Receive events and data through the NuiTtsSdkListener interface.
• Event types:
  • NuiSdkTtsEvent: The synthesis task starts.
  • onTtsDataCallback: Audio data is returned.
  • NuiSdkTtsEvent: The synthesis task ends.
  • NuiSdkTtsEvent: A synthesis error occurs.

Procedure

1. tts_initialize - Initialize the SDK and configure the callback interface and connection parameters.
2. setParamTts - Configure speech synthesis effect parameters such as the model, voice, and volume.
3. startTts - Start a speech synthesis task.
4. onTtsDataCallback - Receive audio data.
5. tts_release - Release SDK resources.

Sambert methods

tts_initialize

Initializes a speech synthesis SDK instance. Create an instance by using new NativeNui(Constants.ModeType.MODE_TTS). Each instance corresponds to one speech synthesis channel. Do not initialize the same instance again before you call tts_release. To process multiple tasks concurrently, create multiple instances.

This interface blocks the calling thread. Call it from a non-UI thread.

Method signature:

public tts_initialize(callback: NuiTtsSdkListener,
                      ticket: string,
                      level: number,
                      save_log: boolean): number

Parameter descriptions:

Parameter	Type	Description
callback	NuiTtsSdkListener	An implementation of the event and data callback interface.
ticket	string	A JSON string that contains authentication, connection, and debugging parameters. See the ticket parameter descriptions below.
level	number	Controls the SDK log level. Valid values are defined by the Constants.LogLevel enumeration.
save_log	boolean	Specifies whether to save local logs. If this parameter is true, use debug_path in the ticket parameters to specify a path. You can also use max_log_file_size to set the file size.

Return value:

An error code.

ticket JSON example:

{
    "url": "wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference",
    "apikey": "sk-****",
    "device_id": "my_device_id",
    "mode_type": "2"
}

ticket parameter descriptions:

Parameter	Type	Required	Description
url	string	Yes	The endpoint. This is fixed at wss://{WorkspaceId}.cn-beijing.maas.aliyuncs.com/api-ws/v1/inference. Replace {WorkspaceId} with your actual workspace ID.
apikey	string	Yes	The API key. We recommend using a more secure temporary API key with a short validity period to reduce the risk of leaking a long-term key.
mode_type	string	Yes	The mode type. This must be set to the string "2", which indicates online speech synthesis mode and corresponds to Constants.TtsModeTypeCloud.
device_id	string	Yes	A unique string that identifies the end user. You can set it to an in-app user ID or a unique device identifier generated by the client. This ID is mainly used for log tracking and troubleshooting.
debug_path	string	No	The storage path for log files. This parameter takes effect only if you set save_log to true when you call tts_initialize. In this case, you must specify a log file path. Otherwise, an error occurs. A maximum of two log files are retained locally.
max_log_file_size	number	No	Sets the maximum size of a log file in bytes. This parameter takes effect only if you set save_log to true when you call tts_initialize. Default value: 104857600 (100 * 1024 * 1024 bytes, or 100 MiB).

setParamTts

Sets speech synthesis effect parameters as key-value pairs. Call this method before startTts.

Method signature:

public setParamTts(param: string, value: string): number

Parameter descriptions:

Parameter	Type	Description
param	string	The parameter name.
value	string	The parameter value.

Return value:

An error code.

Available parameters:

Parameter	Type	Required	Description
model	string	Yes	The model name, such as sambert-zhinan-v1.
format	string	No	The audio encoding format. Valid values: - pcm - wav - mp3 (default).
volume	string	No	The volume. Default value: 50. Valid range: [0, 100].
sample_rate	string	No	The audio sample rate in Hz. Valid values: 8000, 16000, 22050, 24000, and 48000. The default sample rate for most Sambert voice models is 48000. Configure the player to use the corresponding sample rate.
rate	string	No	The speech rate. Default value: 1.0. Valid range: [0.5, 2.0].
pitch	string	No	The pitch. Default value: 1.0. Valid range: [0.5, 2.0].
word_timestamp_enabled	string	No	Specifies whether to enable word-level timestamps. Default value: false. This parameter applies to all Sambert models.
phoneme_timestamp_enabled	string	No	Specifies whether to enable phoneme-level timestamps. Default value: false. Enable word_timestamp_enabled first.
enable_audio_decoder	string	No	Specifies whether to enable the built-in audio decoder. Default value: 0. Valid values: - 1: enabled. When format is mp3, set this parameter to "1" to enable the built-in SDK decoder. onTtsDataCallback then returns decoded PCM data. - 0: disabled.
enable_callback_vol	string	No	Specifies whether to enable the volume callback. Set it to "1" to enable onTtsVolCallback.
apikey	string	No	Refreshes a temporary API key during runtime. Before a synthesis task starts, inject the latest temporary key by using setParamTts('apikey', ...).

getparamTts

Obtains a parameter value. This method is mainly used for troubleshooting.

Method signature:

public getparamTts(param: string): string

Parameter descriptions:

Parameter	Type	Description
param	string	The parameter name. Currently, only "error_msg" is supported.

Return value:

The parameter value.

startTts

Starts a speech synthesis task. The synthesis result is returned through callbacks.

Method signature:

public startTts(priority: string, taskid: string, text: string): number

Parameter descriptions:

Parameter	Type	Description
priority	string	The task priority. Set this parameter to 1.
taskid	string	The task ID. If an empty string '' is passed, the SDK automatically generates an ID.
text	string	The text to synthesize.

Return value:

An error code.

pauseTts

Pauses the current speech synthesis task. After the task is paused, call resumeTts to resume it or cancelTts to cancel it. The SDK cannot start a new synthesis task while a task is paused.

Note: This operation only pauses data retrieval from the server. Audio data already buffered in the player continues to play.

Method signature:

public pauseTts(): number

Return value:

An error code.

resumeTts

Resumes a paused speech synthesis task.

Method signature:

public resumeTts(): number

Return value:

An error code.

cancelTts

Cancels a synthesis task.

Note: This operation only cancels data retrieval from the server. Audio data already buffered in the player continues to play.

Method signature:

public cancelTts(taskid: string): number

Parameter descriptions:

Parameter	Type	Description
taskid	string	The ID of the task to cancel. If an empty string '' is passed, all paused or active synthesis tasks are canceled.

Return value:

An error code.

tts_release

Releases all internal SDK resources and forcibly terminates all active synthesis tasks. After this method is called, the SDK instance becomes unavailable. To use it again, call tts_initialize to reinitialize it.

Method signature:

public tts_release(): number

Return value:

An error code.

NuiTtsSdkListener

The Sambert speech synthesis callback interface receives synthesis events and audio data. In the HarmonyOS SDK, callbacks are defined as ArkTS arrow functions.

onTtsEventCallback

Listens for speech synthesis task start, end, cancel, pause, resume, and error events.

Method signature:

onTtsEventCallback: (event: NuiSdkTtsEvent, taskid: string, ret_code: number) => void;

Parameter descriptions:

Parameter	Type	Description
event	NuiSdkTtsEvent	The callback event.
taskid	string	The speech synthesis task ID.
ret_code	number	The error code. This parameter is valid only for the TTS_EVENT_ERROR event.

onTtsDataCallback

During synthesis, the SDK continuously triggers this callback. Obtain audio data from the callback.

Method signature:

onTtsDataCallback: (info: string, info_len: number, buffer: ArrayBuffer | null) => void;

Parameter descriptions:

Parameter	Type	Description
info	string	A JSON-formatted timestamp result. This parameter takes effect when word_timestamp_enabled is set to "1".
info_len	number	The data length of the info field. You can ignore this parameter.
buffer	ArrayBuffer | null	The audio data for the current segment. This parameter may be null. Check for null in the callback.

onTtsVolCallback

After enable_callback_vol is enabled, this callback returns the volume of the synthesis data just received by the SDK. This is not the volume currently being played.

Method signature:

onTtsVolCallback: (vol: number) => void;

Parameter descriptions:

Parameter	Type	Description
vol	number	The volume of the synthesis data.

NuiSdkTtsEvent

The Sambert speech synthesis event type enumeration.

Event	Description
TTS_EVENT_START	The synthesis task starts. Audio data is about to be returned.
TTS_EVENT_END	The synthesis task ends normally. All audio data has been returned through the callback.
TTS_EVENT_CANCEL	The synthesis task is canceled.
TTS_EVENT_PAUSE	The synthesis task is paused.
TTS_EVENT_RESUME	The synthesis task is resumed.
TTS_EVENT_ERROR	An error occurs during synthesis. Call getparamTts("error_msg") to obtain details. { "header": { "task_id": "xxxxxxxxx", "event": "task-failed", "error_code": "InvalidParameter", "error_message": "Please ensure input text is valid.", "attributes": {} }, "payload": {} }

Auxiliary types

Constants.LogLevel

The enumeration values for the level parameter are as follows:

Value	Description
LOG_LEVEL_VERBOSE	The most detailed logs.
LOG_LEVEL_DEBUG	Debug logs.
LOG_LEVEL_INFO	Informational logs (default).
LOG_LEVEL_WARNING	Warning logs.
LOG_LEVEL_ERROR	Error logs.
LOG_LEVEL_NONE	Disables logging.

Sample code

1. Obtain an API key: Obtain and configure an API key. For security, we recommend configuring the API key as an environment variable.

   NoteTo grant temporary access to third-party applications or users, or to strictly control high-risk operations such as accessing or deleting sensitive data, use a temporary API key. A temporary API key is valid for 60 seconds by default. Obtain a new one after it expires.

2. Download the SDK and run the sample code:

   • Download the latest SDK package.
   • Extract the TAR package. Obtain the HAR-format SDK from the neonui directory and add it to your project dependencies. For C++ integration, obtain the dynamic libraries and header files from native/libs and native/include in the TAR package.
   • Open the project in DevEco Studio. The sample code is located in DashSambertTtsPage.ets. Replace the API key to try the feature.

Invocation steps

1. Initialize the SDK: Call tts_initialize and pass the NuiTtsSdkListener callback and ticket parameters.
2. Configure parameters based on your business requirements: Use setParamTts to configure speech synthesis effect parameters such as the model, format, sample rate, voice, and volume. We recommend configuring them immediately after initialization succeeds.
3. Call startTts to start speech synthesis.
4. Obtain audio data from the onTtsDataCallback callback. We recommend streaming playback as described in Audio playback below. To save the audio locally, append the audio data to the same file until synthesis is complete.
5. After the task ends, call tts_release to release SDK resources.

Audio playback

HarmonyOS uses AudioRenderer from @kit.AudioKit to play synthesized audio. The default sample rate of Sambert synthesized audio is 48 kHz, so configure the player sample rate to 48000.

The product sample encapsulates the logic in the AudioPlayer.ets utility class. Specify the sample rate in the constructor:

// Play Sambert audio at the default sample rate of 48000. The default for AudioPlayer is 16000.
this.mAudioPlayer = new AudioPlayer(this, 48000);

The player uses the writeData callback to retrieve audio data from a queue and returns AudioDataCallbackResult.VALID or AudioDataCallbackResult.INVALID. When onTtsEventCallback receives TTS_EVENT_END, the SDK has completed synthesis and returned all data through the callbacks. Mark the playback queue as complete so that the player automatically stops after playing the remaining data.