Intelligent Speech Interaction:Overview

Description

Compared with common SDKs, the NUI SDK is smaller in size and supports more comprehensive status management. The NUI SDK provides comprehensive speech processing capabilities and can also serve as an atomic SDK, meeting diverse user requirements. In addition, the NUI SDK uses a unified API.

Features

• Supports pulse-code modulation (PCM) encoded 16-bit mono audio files.

• Supports the audio sampling rates of 8,000 Hz and 16,000 Hz.

• Allows you to specify whether to return intermediate results, whether to add punctuation marks during post-processing, and whether to convert Chinese numerals to Arabic numerals.

• Allows you to select linguistic models to recognize speeches in different languages when you manage projects in the Intelligent Speech Interaction console. For more information, see Manage projects.

Endpoints

Interaction process

1. Authenticate the client and initialize the SDK

To establish a WebSocket connection with the server, the client must use a token for authentication. For more information about how to obtain the token, see Obtain an access token.

The following table describes the parameters used for authentication and initialization.

2. Send a request to use the real-time speech recognition service

You must set the request parameters for the client to send a service request. You can set the request parameters in the JSON format by using the setParams method in the SDK. The parameter configuration applies to all service requests. The following table describes the request parameters.

The following table describes the parameters in the nls_config parameter.

3. Send audio data from the client

The client cyclically sends audio data to the server and continuously receives recognition results from the server.

• If an EVENT_SENTENCE_START event is reported, the server detects the beginning of a sentence. Real-time speech recognition uses VAD to determine the beginning and end of a sentence. For example, the server returns the following response:

  {
      "header": {
          "namespace": "SpeechTranscriber",
          "name": "SentenceBegin",
          "status": 20000000,
          "message_id": "a426f3d4618447519c9d85d1a0d1****",
          "task_id": "5ec521b5aa104e3abccf3d361822****",
          "status_text": "Gateway:SUCCESS:Success."
      },
      "payload": {
          "index": 1,
          "time": 0
      }
  }

  The following table describes the parameters in the header object.

  Parameter	Type	Description
  namespace	String	The namespace of the message.
  name	String	The name of the message. The SentenceBegin message indicates that the server detects the beginning of a sentence.
  status	Integer	The HTTP status code. It indicates whether the request is successful. For more information, see the "Error codes" section of this topic.
  message_id	String	The ID of the message, which is automatically generated by the SDK.
  task_id	String	The GUID of the task. Record the value of this parameter to facilitate troubleshooting.
  status_text	String	The status message.

  The following table describes the parameters in the payload object.

  Parameter	Type	Description
  index	Integer	The sequence number of the sentence, which starts from 1.
  time	Integer	The duration of the processed audio stream. Unit: milliseconds.

• If the enable_intermediate_result parameter is set to true, the SDK reports multiple EVENT_ASR_PARTIAL_RESULT events by calling the onNuiEventCallback method to return intermediate results of a sentence. For example, the server returns the following response:

  {
      "header": {
          "namespace": "SpeechTranscriber",
          "name": "TranscriptionResultChanged",
          "status": 20000000,
          "message_id": "dc21193fada84380a3b6137875ab****",
          "task_id": "5ec521b5aa104e3abccf3d361822****",
          "status_text": "Gateway:SUCCESS:Success."
      },
      "payload": {
          "index": 1,
          "time": 1835,
          "result": "Sky in Beijing",
          "confidence": 1.0,
          "words": [{
              "text": "Sky",
              "startTime": 630,
              "endTime": 930
          }, {
              "text": "in",
              "startTime": 930,
              "endTime": 1110
          }, {
              "text": "Beijing",
              "startTime": 1110,
              "endTime": 1140
          }]
      }
  }

  Note

  As shown in the header object, the value of the name parameter is TranscriptionResultChanged, which indicates that an intermediate result is obtained. For more information about other parameters in the header object, see the preceding table.

  The following table describes the parameters in the payload object.

  Parameter	Type	Description
  index	Integer	The sequence number of the sentence, which starts from 1.
  time	Integer	The duration of the processed audio stream. Unit: milliseconds.
  result	String	The recognition result of the sentence.
  words	List< Word >	The word information of the sentence. The word information is returned only when the enable_words parameter is set to true.
  confidence	Double	The confidence level of the recognition result of the sentence. Valid values: 0.0 to 1.0. A larger value indicates a higher confidence level.

• If an EVENT_SENTENCE_END event is reported, the server detects the end of a sentence and returns the recognition result of the sentence. For example, the server returns the following response:

  {
      "header": {
          "namespace": "SpeechTranscriber",
          "name": "SentenceEnd",
          "status": 20000000,
          "message_id": "c3a9ae4b231649d5ae05d4af36fd****",
          "task_id": "5ec521b5aa104e3abccf3d361822****",
          "status_text": "Gateway:SUCCESS:Success."
      },
      "payload": {
          "index": 1,
          "time": 1820,
          "begin_time": 0,
          "result": "Weather in Beijing.",
          "confidence": 1.0,
          "words": [{
              "text": "Weather",
              "startTime": 630,
              "endTime": 930
          }, {
              "text": "in",
              "startTime": 930,
              "endTime": 1110
          }, {
              "text": "Beijing",
              "startTime": 1110,
              "endTime": 1380
          }]
      }
  }

  Note

  As shown in the header object, the value of the name parameter is SentenceEnd, which indicates that the server detects the end of the sentence. For more information about other parameters in the header object, see the preceding table.

  The following table describes the parameters in the payload object.

  Parameter	Type	Description
  index	Integer	The sequence number of the sentence, which starts from 1.
  time	Integer	The duration of the processed audio stream. Unit: milliseconds.
  begin_time	Integer	The time when the server returns the SentenceBegin message for the sentence. Unit: milliseconds.
  result	String	The recognition result of the sentence.
  words	List< Word >	The word information of the sentence. The word information is returned only when the enable_words parameter is set to true.
  confidence	Double	The confidence level of the recognition result of the sentence. Valid values: 0.0 to 1.0. A larger value indicates a higher confidence level.

  The following table describes the parameters in the words object.

  Parameter	Type	Description
  text	String	The text of the word.
  startTime	Integer	The time when the word appears in the sentence. Unit: milliseconds.
  endTime	Integer	The time when the word ends in the sentence. Unit: milliseconds.

4. Complete the recognition task

The client notifies the server that all audio data is sent. The server completes the recognition task and notifies the client that the task is completed.

Error codes

For more information about the error codes that the real-time speech recognition service may return, see error code.