Real-time speech recognition API reference - Intelligent Speech Interaction

Billing and concurrency limits

Real-time speech recognition is available as a trial edition and a commercial edition. For billing details, see Billable items.

Usage notes

Before calling the API, make sure that the audio format, sample rate, and project model meet the following requirements.

Supported input: mono audio with a bit depth of 16 bits in PCM, PCM-encoded WAV, or Ogg Opus format.
Supported sample rates: 8,000 Hz and 16,000 Hz.
Configurable output features include intermediate recognition results, punctuation added during post-processing, and conversion of Chinese numerals to Arabic numerals.
Speaker diarization and speaker role analysis are not supported.
The project model determines the recognition language or dialect. It cannot be specified in a request parameter. For model configuration instructions, see Manage projects.

Endpoints

Access type	Description	URL
Internet access	Use the endpoint in the Singapore region.	wss://nls-gateway-ap-southeast-1.aliyuncs.com/ws/v1

Interaction workflow

Note

The header.task_id field in a server response identifies the recognition task. Record this value for troubleshooting.

1. Authenticate

Use an NLS token to authenticate when establishing a WebSocket connection to the server.

For instructions, see Obtain a token by using an SDK.

2. Start recognition

Send a StartTranscription command with the recognition parameters. Start streaming audio after the server returns TranscriptionStarted. When using an SDK, configure the parameters through its corresponding methods. The following table describes the parameters.

Parameter	Type	Required	Description
appkey	String	Yes	The Appkey of a project created in the Intelligent Speech Interaction console.
format	String	No	The audio format: pcm, wav, or opus.
sample_rate	Integer	No	The audio sample rate. Default: 16,000 Hz. In the console, configure the project with a model that supports the audio sample rate and use case.
enable_intermediate_result	Boolean	No	Specifies whether to return intermediate recognition results. Default: false.
enable_punctuation_prediction	Boolean	No	Specifies whether to add punctuation during post-processing. Default: false.
enable_inverse_text_normalization	Boolean	No	Specifies whether to enable inverse text normalization (ITN) to convert Chinese numerals to Arabic numerals. Default: false.
customization_id	String	No	The ID of the custom linguistic model.
vocabulary_id	String	No	The ID of the custom hotword vocabulary.
max_sentence_silence	Integer	No	The silence threshold for sentence segmentation, in milliseconds. After speech is detected, silence longer than this threshold ends the sentence. Valid values: 200 to 2000. Default: 800. When enable_semantic_sentence_detection is enabled, this threshold is not used for silence-based sentence segmentation. Its value must still be within the allowed range. Silence duration is calculated from the audio data, not from the time spent waiting after data transmission stops. To use silence-based segmentation, continue sending audio, including silent segments. In PCM audio, zero-valued samples can represent silence.
enable_words	Boolean	No	Specifies whether to return word information. Default: false.
disfluency	Boolean	No	Specifies whether to remove filler words. Default: false (disabled).
speech_noise_threshold	Float	No	The noise threshold. Valid values: [-1, 1]. The value affects classification as follows: The closer the value is to -1, the more likely noise is to be classified as speech. The closer the value is to +1, the more likely speech is to be classified as noise. Important This is an advanced parameter. Adjust it with caution and test the effect thoroughly.
enable_semantic_sentence_detection	Boolean	No	Specifies whether to enable semantic sentence segmentation. Default: false. When enabling this feature, also enable intermediate results by setting enable_intermediate_result to true.

3. Receive recognition results

Continue streaming audio and receiving recognition events. The following JSON snippets illustrate the event message structures.

Fields in the header object:

Parameter	Type	Description
namespace	String	The namespace of the message.
name	String	The event name.
status	Integer	The status code, which indicates whether the request succeeded. See the Status codes section.
status_text	String	The status message.
task_id	String	The globally unique task ID. Record this value for troubleshooting.
message_id	String	The message ID.

SentenceBegin

A SentenceBegin event indicates that the server has detected the start of a sentence. The service automatically detects sentence boundaries. Example:

{
        "header": {
                "namespace": "SpeechTranscriber",
                "name": "SentenceBegin",
                "status": 20000000,
                "message_id": "a426f3d4618447519c9d85d1a0d1****",
                "task_id": "5ec521b5aa104e3abccf3d361822****",
                "status_text": "Gateway:SUCCESS:Success."
        },
        "payload": {
                "index": 1,
                "time": 0
        }
}

Fields in the payload object:

Parameter	Type	Description
index	Integer	The sentence index, which starts at 1 and increments for each sentence.
time	Integer	The duration of audio processed so far, in milliseconds.

TranscriptionResultChanged

A TranscriptionResultChanged event contains an updated intermediate result for the current sentence. These events are returned only when enable_intermediate_result is true. Example:

{
        "header": {
                "namespace": "SpeechTranscriber",
                "name": "TranscriptionResultChanged",
                "status": 20000000,
                "message_id": "dc21193fada84380a3b6137875ab****",
                "task_id": "5ec521b5aa104e3abccf3d361822****",
                "status_text": "Gateway:SUCCESS:Success."
        },
        "payload": {
                "index": 1,
                "time": 1835,
                "result": "The weather in",
                "confidence": 1.0,
                "words": [{
                        "text": "The",
                        "startTime": 630,
                        "endTime": 930
                }, {
                        "text": "weather",
                        "startTime": 930,
                        "endTime": 1110
                }, {
                        "text": "in",
                        "startTime": 1110,
                        "endTime": 1140
                }]
        }
}

For this event, header.name is TranscriptionResultChanged, indicating an intermediate recognition result.

Fields in the payload object:

Parameter	Type	Description
index	Integer	The sentence index, which starts at 1 and increments for each sentence.
time	Integer	The duration of audio processed so far, in milliseconds.
result	String	The recognition result for the current sentence.
words	List< Word >	Word information for the current sentence. Requires enable_words to be true.
confidence	Double	The confidence score for the recognition result, in the range [0.0, 1.0]. A higher value indicates higher confidence.

SentenceEnd

A SentenceEnd event indicates that the server has detected the end of a sentence and returns the recognition result for that sentence. Example:

{
        "header": {
                "namespace": "SpeechTranscriber",
                "name": "SentenceEnd",
                "status": 20000000,
                "message_id": "c3a9ae4b231649d5ae05d4af36fd****",
                "task_id": "5ec521b5aa104e3abccf3d361822****",
                "status_text": "Gateway:SUCCESS:Success."
        },
        "payload": {
                "index": 1,
                "time": 1820,
                "begin_time": 0,
                "result": "The weather in Beijing.",
                "confidence": 1.0,
                "words": [{
                        "text": "The",
                        "startTime": 630,
                        "endTime": 930
                }, {
                        "text": "weather",
                        "startTime": 930,
                        "endTime": 1110
                }, {
                        "text": "in Beijing",
                        "startTime": 1110,
                        "endTime": 1380
                }]
        }
}

For this event, header.name is SentenceEnd, indicating the end of a sentence.

Fields in the payload object:

Parameter	Type	Description
index	Integer	The sentence index, which starts at 1 and increments for each sentence.
time	Integer	The duration of audio processed so far, in milliseconds.
begin_time	Integer	The time of the corresponding SentenceBegin event, in milliseconds.
result	String	The recognition result.
words	List< Word >	Word information for the current sentence. Requires enable_words to be true.
confidence	Double	The confidence score for the recognition result, in the range [0.0, 1.0]. A higher value indicates higher confidence.

Fields in each words entry:

Parameter	Type	Description
text	String	The word text.
startTime	Integer	The word start time, in milliseconds.
endTime	Integer	The word end time, in milliseconds.

4. End recognition

After sending all audio, send StopTranscription to end the recognition task. The server processes the remaining audio and returns TranscriptionCompleted when the task ends. Close the connection only after receiving this event.

StopTranscription does not force a sentence boundary while keeping the task running. If the remaining audio contains valid speech, the server may return SentenceEnd before completing the task. A task containing only silence does not necessarily return SentenceEnd.

Status codes

Use header.status and header.status_text in the response to check the request status. The following tables list common errors and resolutions.

Common error codes

Status code	Status message	Cause	Solution
40000000	The default client error code. Multiple error messages can use this code.	Invalid parameters or request logic.	Compare the request with the documented sample code and test the corrected request.
40000001	The token 'xxx' has expired; The token 'xxx' is invalid	A general client error, usually caused by an expired or invalid token.	Compare the request with the documented sample code and test the corrected request.
40000002	Gateway:MESSAGE_INVALID:Can't process message in state'FAILED'!	An invalid or malformed message.	Compare the request with the documented sample code and test the corrected request.
40000003	PARAMETER_INVALID; Failed to decode url params	Invalid request parameters. This error commonly occurs in RESTful requests.	Compare the request with the documented sample code and test the corrected request.
40000005	Gateway:TOO_MANY_REQUESTS:Too many requests!	Too many concurrent requests.	Reduce simultaneous recognition requests to stay within the available concurrency quota.
40000009	Invalid wav header!	An invalid header.	If sending a WAV file with `format` set to `wav`, check that the WAV header is valid. Otherwise, the server may reject the audio.
40000009	Too large wav header!	An invalid WAV header in the transmitted audio.	Use a format such as PCM or Opus. For WAV audio, check that the header contains the correct data length.
40000010	Gateway:FREE_TRIAL_EXPIRED:The free trial has expired!	The trial has expired and the commercial edition is not activated, or the account has overdue payments.	Check service activation and the account balance.
40010001	Gateway:NAMESPACE_NOT_FOUND:RESTful url path illegal	An unsupported API or parameter.	Check the request parameters against the API documentation and correct them based on the error message. For example, check the URL when making a RESTful request with curl.
40010003	Gateway:DIRECTIVE_INVALID:[xxx]	A general client error.	The client sent an invalid parameter or command. Use the API-specific error details and documentation to correct the request.
40010004	Gateway:CLIENT_DISCONNECT:Client disconnected before task finished!	The client closed the connection before the request completed.	Close the connection after receiving TranscriptionCompleted.
40010005	Gateway:TASK_STATE_ERROR:Got stop directive while task is stopping!	The client sent a command that is not supported in the current state.	Check the command sequence. Do not send StopTranscription again while the task is stopping.
40020105	Meta:APPKEY_NOT_EXIST:Appkey not exist!	The specified Appkey does not exist.	Check the Appkey in the project configuration in the console.
40020106	Meta:APPKEY_UID_MISMATCH:Appkey and user mismatch!	The Appkey and token belong to different account UIDs.	Make sure that the Appkey and token belong to the same account. Do not use the Appkey from one account with a token from another.
403	Forbidden	The token is invalid, for example, because it does not exist or has expired.	Specify a valid token. Obtain a new token before the current token expires.
41000003	MetaInfo doesn't have end point info	Routing information could not be obtained for the Appkey.	Make sure that the Appkey and token belong to the same account. Do not use the Appkey from one account with a token from another.
41010101	UNSUPPORTED_SAMPLE_RATE	An unsupported sample rate.	Real-time speech recognition supports sample rates of 8,000 Hz and 16,000 Hz.
41040201	Realtime:GET_CLIENT_DATA_TIMEOUT:Client data does not send continuously!	The server timed out waiting for data from the client.	Send audio continuously at a real-time rate. After all audio is sent, send StopTranscription and wait for TranscriptionCompleted before closing the connection.
50000000	GRPC_ERROR:Grpc error!	An intermittent error caused by factors such as server load or network conditions.	Retry the request.
50000001	GRPC_ERROR:Grpc error!	An intermittent error caused by factors such as server load or network conditions.	Retry the request.
52010001	GRPC_ERROR:Grpc error!	An intermittent error caused by factors such as server load or network conditions.	Retry the request.

Real-time speech recognition error codes

Status code	Status message	Cause	Solution
40000004	Gateway:IDLE_TIMEOUT:Websocket session is idle for too long time	The client sent no data for more than 10 seconds after establishing the connection.	After establishing the connection, stream audio continuously as it is captured. After all audio is sent, send StopTranscription and wait for TranscriptionCompleted before closing the connection.
40270002	NO_VALID_AUDIO_ERROR	Invalid audio.	No valid text was recognized from the audio.
40270003	DECODE_ERROR	Audio decoding failed.	Set format to match the actual audio format.
41000002	APPKEY_KEY_IS_NULL	The appkey parameter is not set correctly.	Refer to the API documentation and sample code.