AlCallKit Web SDK Capabilities & API Overview - Intelligent Media Services - Alibaba Cloud - Intelligent Media Services

Lists the API operations of the Web SDK, including engine methods and callback events.

API overview

Note

Earlier SDK versions contain deprecated parameters and methods. We recommend that you upgrade to the latest version. For more information, see Web SDK usage guide.

Class/Protocol	API	Description
ARTCAICallEngine Defines the engine API operations.	call	Starts a call.
	handup	Hangs up. Note Call handup to hang up the call before the page exits, for example, by listening for the beforeunload event. If you do not, the agent waits about 90 seconds to exit. This may cause issues such as exceeding the concurrent limit for digital humans.
	setAgentView	Sets the agent rendering view.
	setLocalView	Sets the local preview view.
	interruptSpeaking	Interrupts the agent's speech.
	enableVoiceInterrupt	Enables or disables smart interruption.
	muteLocalCamera	Enables or disables camera push.
	switchCamera	Switches the camera.
	switchVoiceId	Switches the voice.
	mute	Mutes or unmutes the microphone.
	muteAgentAudioPlaying	Stops or resumes the playback of the agent's audio stream.
	startPushToTalk	Starts speaking in push-to-talk mode.
	finishPushToTalk	Finishes speaking in push-to-talk mode.
	cancelPushToTalk	Cancels the current speech in push-to-talk mode.
	enablePushToTalk	Enables or disables push-to-talk mode. In push-to-talk mode, the agent reports the result only after finishPushToTalk is called.
	getRTCInstance	Gets the Alibaba Real-Time Communication (ARTC) engine.
	sendTextToAgent	Sends a text message to the agent.
	sendCustomMessageToServer	Sends a custom message to the server.
	updateLlmSystemPrompt	Updates the system prompt of the Large Language Model (LLM).
	updateVcrConfig	Updates the VCR configuration.
	updateBailianAppParams	Updates the parameters of the Model Studio application.
	startVisionCustomCapture	For a visual understanding agent, starts custom frame capture. After this is started, you cannot talk to the agent by voice.
	stopVisionCustomCapture	For a visual understanding agent, stops custom frame capture.
	destroy	Releases resources.
ARTCAICallEngine Engine callback events	errorOccurred	An error occurred.
	callBegin	The call started.
	callEnd	The call ended.
	agentStateChange	The agent status changed.
	speakingVolumeChanged	The volume changed.
	userSubtitleNotify	Reports the agent's recognition result for the user's question.
	agentSubtitleNotify	Reports the agent's response.
	voiceIdChanged	The voice of the current call changed.
	pushToTalkChanged	The push-to-talk mode changed.
	agentWillLeave	The current agent is about to leave (end the current call).
	receivedAgentCustomMessage	A custom message was received from the current agent.
	voiceInterruptChanged	The voice interruption for the current call was enabled or disabled.
	humanTakeoverWillStart	A live agent is about to take over from the current agent.
	humanTakeoverConnected	The live agent takeover is connected.
	agentDataChannelAvailable	Fires when the agent message channel becomes available.
	receivedAgentVcrResult	A VCR result was received from the current agent.

API details

ARTCAICallEngine details

call

Starts a call.

async call(userId: string, agentInfo: AICallAgentInfo, config?: AICallEngineConfig): Promise<void>>

Parameters:

Parameter	Type	Description
userId	String	The UID of the current user.
agentInfo	AICallAgentInfo	The agent information.
config	AICallEngineConfig	The initialization configuration. The type is defined as follows: `{ // Specifies whether to mute the microphone. muteMicrophone?: boolean; // Specifies whether to turn off the camera in visual understanding mode. muteCamera?: boolean; // Specifies whether to enable push-to-talk. enablePushToTalk?: boolean; // The preview video element in visual understanding mode. previewElement?: string \| HTMLVideoElement; // The camera settings in visual understanding mode. cameraConfig?: AICallCameraConfig; }`

handup

Ends the call.

async handup(): Promise<void>

setAgentView

Sets the agent rendering view.

setAgentView(view: HTMLVideoElement | string): void

Parameters:

Parameter	Type	Description
view	HTMLVideoElement \| string	The video tag or its ID.

setLocalView

Sets the local preview view.

setLocalView(view?: HTMLVideoElement | string): void

Parameters:

Parameter	Type	Description
view	HTMLVideoElement \| string	The video tag or its ID. If you do not pass this parameter, the preview stops.

interruptSpeaking

Interrupts the agent's speech.

async interruptSpeaking(): Promise<void>

enableVoiceInterrupt

Enables or disables smart interruption.

async enableVoiceInterrupt(enable: boolean): Promise<void>

Parameters:

Parameter	Type	Description
enable	boolean	Enables or disables the feature.

muteLocalCamera

Enables or disables sending the local camera stream.

async muteLocalCamera(mute: boolean)

Parameters:

Parameter	Type	Description
mute	boolean	Specifies whether to mute the camera.

switchCamera

Switches the camera.

async switchCamera(deviceId?: string)

Parameters:

Parameter	Type	Description
deviceId	string	The device ID. You can get the ID by calling ARTCAICallEngine.getCameraList(). If you do not pass this parameter on a mobile device, the system switches between the front and rear cameras.

switchVoiceId

Switches the voice.

async switchVoiceId(voiceId: string): Promise<void>

Parameters:

Parameter	Type	Description
voiceId	string	The voice ID.

mute

Mutes or unmutes the microphone.

async mute(mute: boolean): Promise<void>

Parameters:

Parameter	Type	Description
mute	boolean	Mutes or unmutes the microphone.

muteAgentAudioPlaying

Stops or resumes the playback of the agent's audio stream.

muteAgentAudioPlaying(mute: boolean)

Parameters:

Parameter	Type	Description
mute	boolean	Specifies whether to mute the audio.

startPushToTalk

In push-to-talk mode, starts speaking.

startPushToTalk() :boolean;

finishPushToTalk

In push-to-talk mode, finishes speaking.

finishPushToTalk() :boolean;

cancelPushToTalk

In push-to-talk mode, cancels the current speech.

cancelPushToTalk() :boolean;

enablePushToTalk

Enables or disables push-to-talk mode. When this mode is enabled, the agent sends the recognition result only after finishPushToTalk is called.

enablePushToTalk(enable: boolean):boolean;

Parameters:

Parameter	Type	Description
enable	Bool	Disables or enables the mode.

getRTCInstance

Retrieves the ARTC engine instance.

getRTCInstance(): AliRtcEngine | undefined

sendTextToAgent

Sends a text message to the agent.

sendTextToAgent(req: AICallSendTextToAgentRequest);

Parameters:

Parameter

Type

Description

req

AICallSendTextToAgentRequest

The message struct to send.

new AICallSendTextToAgentRequest(text)

sendCustomMessageToServer

Sends a custom message to the server. Must be called after the call is connected.

sendCustomMessageToServer(msg: string)

Parameters:

Parameter	Type	Description
msg	string	The content to send.

updateLlmSystemPrompt

Updates the system prompt of the Large Language Model (LLM). Must be called after the call is connected.

updateLlmSystemPrompt(prompt: string)

Parameters:

Parameter	Type	Description
prompt	string	The prompt.

updateVcrConfig

Updates the VCR configuration.

 public updateVcrConfig(vcrConfig: AICallAgentVcrConfig)

Parameters:

Parameter	Type	Description
vcrConfig	AICallAgentVcrConfig	The VCR configuration.

updateBailianAppParams

Updates the parameters of the Model Studio application.

public updateBailianAppParams(params: JSONObject)

Parameters:

Parameter	Type	Description
params	object	The parameters of the Model Studio application.

startVisionCustomCapture

For a visual understanding agent, starts custom frame capture. After calling this method, voice interaction with the agent is disabled. Must be called after the call is connected.

startVisionCustomCapture(req: AICallVisionCustomCaptureRequest)

Parameters:

Parameter

Type

Description

req

AICallVisionCustomCaptureRequest

The configuration information.

{
  /**
   * The text parameter for the request to the multimodal large model.
   */
  text?: string;

  /**
   * If set to true, a single frame is captured, and the system exits the custom frame capture state immediately.
   * If set to false, frames are captured continuously, and the system exits the custom frame capture state after the specified duration.
   */
  isSingle?: boolean;

  /**
   * The interval for frame capture.
   * Default: 5 seconds.
   */
  eachDuration?: number;

  /**
   * The number of images captured each time.
   * Default: 2.
   */
  num?: number;

  /**
   * The duration of continuous frame capture in seconds. This parameter is ignored if isSingle is true. Otherwise, the custom frame capture state ends when the specified duration is reached.
   */
  duration?: number;

  /**
   * A JSON string that contains custom business information. This information is sent to the large model along with the text and image frames for business processing.
   */
  userData?: string;
}

stopVisionCustomCapture

For a visual understanding agent, stops custom frame capture. Must be called after the call is connected.

stopVisionCustomCapture()

destroy

Releases resources.

async destroy()

ARTCAICallEngine event details

errorOccurred

Indicates that an error occurred during the call.

Parameters:

Parameter	Type	Description
code	AICallErrorCode	The error type.

callBegin

Indicates that the call has started.

callEnd

Indicates that the call has ended.

agentStateChange

Indicates that the agent status has changed.

Parameters:

Parameter	Type	Description
state	AICallAgentState	The current status of the agent: Listening, Thinking, or Speaking.

speakingVolumeChanged

Indicates a change in the speaking volume.

Parameters:

Parameter	Type	Description
uid	string	The UID of the speaker. An empty string indicates the current user.
volume	Int32	The volume. Range: 0 to 100.

userSubtitleNotify

Provides the agent's speech recognition result for the user's input.

Parameters:

Parameter	Type	Description
subtitle	AICallSubtitleData	The caption information.

agentSubtitleNotify

Provides the text of the agent's response.

Parameters:

Parameter	Type	Description
subtitle	AICallSubtitleData	The caption information.

voiceIdChanged

Indicates that the voice of the current call has changed.

Parameters:

Parameter	Type	Description
voiceId	string	The current voice ID.

pushToTalkChanged

Indicates that the push-to-talk mode has changed.

Parameters:

Parameter	Type	Description
enable	boolean	Specifies whether the mode is enabled.

agentWillLeave

Indicates that the agent is about to end the call.

Parameters:

Parameter	Type	Description
reason	number	The reason for leaving: 2001 (idle timeout) or 0 (other).
message	string	A description of the reason for leaving.

receivedAgentCustomMessage

Indicates that a custom message was received from the agent.

Parameters:

Parameter	Type	Description
data	Object	The message content.

voiceInterruptChanged

Indicates that voice interruption for the current call was enabled or disabled.

Parameters:

Parameter	Type	Description
enable	boolean	Specifies whether the feature is enabled.

humanTakeoverWillStart

Indicates that a live agent is about to take over the call.

Parameters:

Parameter	Type	Description
takeoverUid	string	The UID of the live agent.
takeoverMode	number	1: The output uses the live agent's voice. 0: The output uses the agent's voice.

humanTakeoverConnected

Indicates that the live agent has connected and taken over the call.

Parameters:

Parameter	Type	Description
takeoverUid	string	The UID of the live agent.

agentEmotionNotify

Provides the agent's emotion analysis result.

Parameters:

Parameter	Type	Description
emotion	string	The emotion label, such as neutral, happy, angry, or sad.
userAsrSentenceId	number	The sentence ID of the answer to the user's question.

agentDataChannelAvailable

Fires when the agent message channel becomes available. After this callback fires, you can start sending messages to the agent.

receivedAgentVcrResult

Indicates that a VCR result was received from the agent. For more information, see AICallAgentVcrResult.

receivedAgentVcrResult: (result: AICallAgentVcrResult) => void;

Parameters:

Parameter	Type	Description
result	AICallAgentVcrResult	The VCR result.