Intelligent Media Services:API reference

Initialization

/**
 * Initializes the engine.
 * @param config The initialization configuration.
 */
public abstract void init(ARTCAICallConfig config);

Parameters:

Parameter	Type	Description
config	ARTCAICallConfig	The initialization configuration.

Sets the agent type.

/**
 * Sets the agent type.
 * @param aiAgentType
 */
public abstract void setAICallAgentType(ARTCAICallAgentType aiAgentType);

Parameters:

Parameter	Type	Description
aiAgentType		You can set the agent type.

Creates and starts a call.

/**
 * Creates and starts a call.
 */
public abstract void call(String rtcToken, String aiAgentInstanceId, String aiAgentUserId, String channelId);

Parameters:

Parameter	Type	Description
rtcToken	String	The token for joining the channel. This parameter cannot be empty.
aiAgentInstanceId	String	The instance ID. This parameter cannot be empty.
aiAgentUserId	String	The user ID of the agent. This parameter cannot be empty.
channelId	String	The channel ID. This parameter cannot be empty.

Initiates a call to an agent using the SDK. You must pass the ARTC token to join the channel and set the agent ID, agent region, and startup parameters in `mAiCallAgentTemplateConfig` of `ARTCAICallConfig`.

/**
   * Initiates a call to an agent.
   * @param rtcToken The authentication token for the local user to join the ARTC channel.
   * @detail This operation uses the SDK to call the agent. You must pass the ARTC token for joining the channel
   * and set the agent ID, agent region, and startup parameters in `mAiCallAgentTemplateConfig` of `ARTCAICallConfig`.
   */
public abstract void call(String rtcToken);

Parameters:

Parameter	Type	Description
rtcToken	String	The token for joining the channel.

Hangs up the call.

/**
 * Hangs up the call.
 */
public abstract void handup();

Toggles the microphone on or off.

/**
 * Switches the microphone status.
 * @param on
 */
public abstract void switchMicrophone(boolean on);

Parameters:

Parameter	Type	Description
on	boolean	Indicates whether the sound is muted.

Interrupts the agent's speech.

/**
 * Interrupts the agent's speech.
 */
public abstract boolean interruptSpeaking();

Enables or disables smart interruption.

/**
 * Enables or disables smart interruption.
 */
public abstract boolean enableVoiceInterrupt(boolean enable);

Parameters:

Parameter	Type	Description
enable	boolean	Specifies whether to enable smart interruption.

Switches between the speaker and the earpiece. `true` enables the speaker. `false` enables the earpiece.

/**
 * Switches between the speaker and the earpiece.
 */
public abstract boolean enableSpeaker(boolean enable);

Parameters:

Parameter	Type	Description
enable	boolean	Specifieswhether to enable the speaker or the earpiece. `true` enables the speaker. `false` enables the earpiece.

Stops or resumes playback of the agent's audio stream.

public abstract boolean muteAgentAudioPlaying(boolean mute);

Parameters:

Parameter	Type	Description
mute	boolean	Specifies whether to mute the audio.

Enables or disables the push-to-talk mode.

/**
 * Enables or disables the push-to-talk mode. In push-to-talk mode, the agent reports the result only after finishPushToTalk is called.
 * @param enable
 * @return
 */
public abstract boolean enablePushToTalk(boolean enable);

Parameters:

Parameter	Type	Description
enable	boolean	Specifies whether to enable the push-to-talk mode. In push-to-talk mode, the agent reports the result only after finishPushToTalk is called.

Checks whether the push-to-talk mode is enabled.

/**
 * Checks whether the push-to-talk mode is enabled.
 * @return
 */
public abstract boolean isPushToTalkEnable();

Push-to-talk mode: Starts speaking.

/**
 * Push-to-talk mode: Starts speaking.
 * @return
 */
public abstract boolean startPushToTalk();

Push-to-talk mode: Stops speaking.

/**
 * Push-to-talk mode: Stops speaking.
 * @return
 */
public abstract boolean finishPushToTalk();

Push-to-talk mode: Cancels the current speech.

/**
 * Push-to-talk mode: Cancels the current speech.
 * @return
 */
public abstract boolean cancelPushToTalk();

Switches the voice.

/**
 * Switches the voice.
 */
public abstract boolean switchRobotVoice(String voiceId);

Parameters:

Parameter	Type	Description
voiceId	String	Change voice

Retrieves the current voice.

/**
 * Gets the current voice.
 * @return
 */
public abstract String getRobotVoiceId();

Registers a callback.

/**
 * Registers a callback.
 * @param engineCallback
 */
public abstract void setEngineCallback(IARTCAICallEngineCallback engineCallback);

Parameters:

Parameter	Type	Description
engineCallback	IARTCAICallEngineCallback	The callback to register.

Do you want to turn off the microphone?

/**
 * Checks whether the microphone is on.
 * @return
 */
public abstract boolean isMicrophoneOn();

Checks whether the speaker is used for playback. `true` indicates speaker playback. `false` indicates earpiece playback.

/**
 * Checks whether the speaker is used for playback.
 * @return
 */
public abstract boolean isSpeakerOn();

Checks whether smart interruption is enabled.

/**
 * Checks whether smart interruption is enabled.
 * @return
 */
public abstract boolean isVoiceInterruptEnable();

Sets the view container for the digital human and specifies the rendering mode.

public abstract void setAvatarAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams avatarLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Parameter	Type	Description
viewGroup	ViewGroup	The rendering view for the digital human.
avatarLayoutParams	ViewGroup.LayoutParams	The layout parameters.
canvas	ARTCAICallVideoCanvas	The rendering view configuration, including the rendering view, rendering mode, mirror mode, and rotation mode.

Sets the local view container for visual understanding and specifies the rendering mode.

public abstract void setVisionPreviewView(ViewGroup viewGroup, ViewGroup.LayoutParams visionLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Parameter	Type	Description
viewGroup	ViewGroup	The local rendering view.
avatarLayoutParams	ViewGroup.LayoutParams	The layout parameters.
canvas	ARTCAICallVideoCanvas	The rendering view configuration. This includes settings such as the rendering view, rendering mode, mirror mode, and rotation mode.

Enable/Disable Camera

/**
 * Mutes or unmutes the camera.
 */
public abstract boolean muteLocalCamera(boolean mute);

Parameters:

Parameter	Type	Description
mute	boolean	Turn the camera on or off

Is the camera off?

/**
 * Checks whether the camera is muted.
 * @return
 */
public abstract boolean isLocalCameraMute();

Switches between the front and rear cameras.

/**
 * Switches between the front and rear cameras.
 */
public abstract boolean switchCamera();

You can obtain the RTC engine instance.

/**
 * Gets the ARTC engine instance.
 * @return
 */
public abstract AliRtcEngine getRtcEngine();

Retrieves the official protocol implementation.

/**
 * Gets the official protocol implementation.
 * @return
 */
public abstract IARTCAICallService getIARTCAICallService();

Sends a custom message to the Large Language Model (LLM).

/**
 * Sends a custom message to the LLM.
 * @param req
*/
public  boolean sendTextToAgent(ARTCAICallSendTextToAgentRequest req);

Parameters:

Parameter	Type	Description
req	ARTCAICallSendTextToAgentRequest	The message struct to send.

Sends a custom message to the business server. You must call this method after the call is connected.

/**
 * Sends a custom message to the AppServer.
 * Reuses the agent's message channel to send a message to the AppServer. The AppServer must listen for the callback to receive the custom message.
 * @param msg
 * @return true: The message was sent successfully. false: The message failed to be sent.
*/
public boolean sendCustomMessageToServer(String msg);

Parameters:

Parameter	Type	Description
msg	String	The message content.

Updates the system prompt for the LLM. You must call this method after the call is connected.

/**
 * Updates the system prompt for the LLM. For example: "You are a friendly and helpful assistant focused on providing users with accurate information and advice."
 * @param prompt
 * @return true: The update was successful. false: The update failed.
*/
public boolean updateLlmSystemPrompt(String prompt)

Parameters:

Parameter	Type	Description
prompt	String	The prompt.

Updates the parameters for the Model Studio application center.

public abstract boolean updateBailianAppParams(String params);

Parameters:

Parameter	Type	Description
params	String	The parameters for the Model Studio application center.

Updates the VCR configuration.

 public abstract boolean updateVcrConfig(ARTCAICallAgentVcrConfig config);

Parameters:

Parameter	Type	Description
config	ARTCAICallAgentVcrConfig	The VCR configuration.

For a visual understanding agent, starts custom frame capture. After this mode is started, voice communication with the agent is disabled. You must call this method after the call is connected.

/**
 * For a Vision agent, starts custom frame capture. After this is started, you cannot talk to the agent by voice.
 * @param request The request parameters.
 * @return true: The request was sent successfully. false: The request failed to be sent.
*/
public  boolean startVisionCustomCapture(ARTCAICallVisionCustomCaptureRequest request);

Parameters:

Parameter	Type	Description
request	ARTCAICallVisionCustomCaptureRequest	The configuration information.

For a visual understanding agent, stops custom frame capture. You must call this method after the call is connected.

public  boolean stopVisionCustomCapture();

Sets the video view container for the agent.

    public abstract void setAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams agentLayoutParams);

Parameters:

Parameter	Type	Description
viewGroup	ViewGroup	The digital human rendering view.
avatarLayoutParams	ViewGroup.LayoutParams	The layout parameter.

Sets the agent video view and specifies the rendering mode.

    public abstract void setAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams agentLayoutParams, ARTCAICallVideoCanvas canvas);

Parameter details:

Parameter	Type	Description
viewGroup	ViewGroup	The rendering view for the digital human.
agentLayoutParams	ViewGroup.LayoutParams	Layout parameters.
canvas	ARTCAICallVideoCanvas	The rendering view configuration. This includes the rendering view and rendering mode.

Sets the local preview view.

public abstract void setLocalView(ViewGroup viewGroup, ViewGroup.LayoutParams localLayoutParams);

Parameters:

Parameter	Type	Description
viewGroup	ViewGroup	The local capture view.
localLayoutParams	ViewGroup.LayoutParams	The layout parameters.

Sets the local preview view and specifies the rendering mode.

public abstract void setLocalView(ViewGroup viewGroup, ViewGroup.LayoutParams localLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Parameter	Type	Description
viewGroup	ViewGroup	The local capture view.
localLayoutParams	ViewGroup.LayoutParams	The layout parameters.
canvas	ARTCAICallVideoCanvas	The rendering view configuration, including the rendering view and rendering mode.

Updates the Voice Activity Detection (VAD) threshold. A period of silence that exceeds this threshold is considered a sentence break.

public abstract boolean updateAsrMaxSilence(int duration);

Parameters:

Parameter	Type	Description
duration	int	The duration in milliseconds. The value ranges from 200 ms to 1200 ms.

Updates the Text-to-Speech (TTS) playback speed. This applies to all TTS types. The value can range from 0.5 to 2.0.

public abstract boolean updateTtsSpeechRate(double rate);

Parameters:

Parameter	Type	Description
rate	double	The speech rate. The value ranges from 0.5 to 2.0.

Retrieves the runtime information of the current agent.

public abstract ARTCAICallAgentInfo getAgentInfo();

Return value:

This operation returns an ARTCAICallAgentInfo object that contains the following parameters:

Parameter	Type	Description
agentId	String	The current agent ID.
agentType	ARTCAICallAgentType	The agent type.
agentUserId	String	The user ID of the agent in the RTC channel.
channelId	String	The RTC channel ID where the agent is located.
instanceId	String	The instance ID of the currently running agent. When an agent starts, the system assigns a unique instance ID to identify and track its entire lifecycle and running state.
requestId	String	The request ID of the currently running agent.

A fault occurred.

/**
 * An error occurred.
 * @param errorCode The error code.
 */
void onErrorOccurs(AICallErrorCode errorCode);

Parameters:

Parameter	Type	Description
errorCode	AICallErrorCode	The error that occurred.

This callback is triggered when the agent starts successfully. If you use the client-side call[2/2] operation to call the agent, this callback is triggered when the call is successful.

/**
 * The agent started successfully.
 */     
public void onAgentStarted() {}

Start a call (Join Meeting)

/**
 * The call started (joined the channel).
 */
void onCallBegin();

Leave meeting

/**
 * The call ended (left the channel).
 */
void onCallEnd();

The underlying AliRtcEngine has been created.

 public void onAliRtcEngineCreated(AliRtcEngine engine)

Parameters:

Parameter	Type	Description
engine	AliRtcEngine	The underlying AliRtcEngine engine object. This is useful when the AICallKit SDK operations do not meet your needs and you need to use the atomic operations of the ARTC SDK.

Robot State Synchronization

/**
 * The agent status is synchronized.
 * @param oldRobotState
 * @param newRobotState
 */
void onAICallEngineRobotStateChanged(ARTCAICallRobotState oldRobotState, ARTCAICallRobotState newRobotState);

Parameters:

Parameter	Type	Description
oldRobotState	ARTCAICallRobotState	The previous state.
newRobotState	ARTCAICallRobotState	The new state.

This callback is triggered when the user is speaking.

/**
 * Callback for when the user is speaking.
 * @param isSpeaking Whether the user is speaking.
 */
void onUserSpeaking(boolean isSpeaking);

Parameters:

Parameter	Type	Description
isSpeaking	boolean	User speech callback

This callback synchronizes the text recognized from the user's speech by Automatic Speech Recognition (ASR).

/**
 * Synchronizes the text recognized from the user's speech by ASR.
 * @param text The specific text recognized by ASR.
 * @param isSentenceEnd Whether the current text is the final result for this sentence.
 * @param sentenceId The ID of the sentence to which the current text belongs.
 * @param voicePrintStatusCode The voiceprint recognition status.
 */
void onUserAsrSubtitleNotify(String text, boolean isSentenceEnd, int sentenceId, VoicePrintStatusCode voicePrintStatusCode);

Parameters:

Parameter	Type	Description
text	String	The specific text recognized by ASR.
isSentenceEnd	boolean	Whether the current text is the final result for this sentence.
sentenceId	int	The sentence ID.
voicePrintStatusCode	VoicePrintStatusCode	The voiceprint recognition status.

Synchronous agent response

/**
 * Synchronizes the agent's response text.
 * @param text The agent's text.
 * @param end Whether the current reply is finished.
 * @param userAsrSentenceId The LLM content that corresponds to the speech input with the specified sentenceId.
 */
void onAIAgentSubtitleNotify(String text, boolean end, int userAsrSentenceId);

Parameters:

Parameter	Type	Description
text	String	The agent's reply.
end	boolean	Whether the current reply is finished.
userAsrSentenceId	int	The LLM content that corresponds to the speech input with the specified sentenceId.

The LLM has finished responding in the current call.

/**
 * The LLM has finished replying in the current call.
 * @param text The text output by the LLM.
 * @param userAsrSentenceId The ID of the sentence that answers the user's question.
 * @note This callback provides the complete LLM response to a question and is not synchronized with the agent's voice playback. For real-time captions, use onAIAgentSubtitleNotify.
 */
public void onLLMReplyCompleted(String text, int userAsrSentenceId) {}

Parameters:

Parameter	Type	Description
text	String	The text output by the LLM.
userAsrSentenceId	int	The ID of the sentence that answers the user's question.

Network Status Callback

/**
 * Callback for network status changes.
 * @param uid The user ID.
 * @param quality The network status.
 */
void onNetworkStatusChanged(String uid, ARTCAICallNetworkQuality quality);

Parameters:

Parameter	Type	Description
uid	String	User ID
quality	ARTCAICallNetworkQuality	The network status.

Volume modification

/**
 * The volume changed.
 * @param uid The user ID.
 * @param volume The volume, from 0 to 255.
 */
void onVoiceVolumeChanged(String uid, int volume);

Parameters:

Parameter	Type	Description
uid	String	User ID
volume	int	The volume, from 0 to 255.

The current call's timbre has changed.

/**
 * The voice for the current call changed.
 */
void onVoiceIdChanged(String voiceId);

Parameters:

Parameter	Type	Description
voiceId	String	The call tone has changed.

The barge-in settings for the current call have been changed.

/**
 * The voice interruption setting for the current call changed.
 */
void onVoiceInterrupted(boolean enable);

Parameters:

Parameter	Type	Description
enable	boolean	The barge-in settings for the current call have been changed.

Is stream ingest active for the agent's video?

/**
 * Indicates whether the agent's video stream is available.
 */
void onAgentVideoAvailable(boolean available);

Parameters:

Parameter	Type	Description
Enable	boolean	Indicates whether the agent's video stream is available.

Is the agent's audio stream ingest active?

/**
 * Indicates whether the agent's audio stream is available.
 */
void onAgentAudioAvailable(boolean available);

Parameters:

Parameter	Type	Description
Enable	boolean	Indicates whether the agent's audio stream is available.

Audio loopback latency

/**
         * Audio loopback latency.
         *  @param id The statement ID.
         *  @param delay_ms The latency.
         */
public void onAudioDelayInfo(int id, int delay_ms) {}

Parameters:

Parameter	Type	Description
id	int	The statement ID.
delay_ms	int	The latency.

Digital human initial frame rendering

/**
*  The first video frame of the digital human is rendered.
*/
void onAgentAvatarFirstFrameDrawn();

User Online Callback

/**
*  Callback for when a user comes online.
*/
void onUserOnLine(String uid);

Parameters:

Parameter	Type	Description
uid	String	User joined callback

The agent's speech was interrupted.

public void onSpeakingInterrupted(ARTCAICallSpeakingInterruptedReason reason);

Metric description:

Parameter	Type	Description
reason	ARTCAICallSpeakingInterruptedReason	The reason for the interruption, such as a specific keyword.

This callback indicates whether custom frame capture mode is enabled for the current visual understanding call.

public void onVisionCustomCapture(boolean enable)

Metric description:

Parameter	Type	Description
enable	boolean	Indicates whether the mode is enabled.

The call will end because the current agent is about to disconnect.

/**
 * The current agent is about to leave the call.
 * @param reason The reason: 2001 (idle exit), 2002 (human takeover ended), 0 (other).
 * @param message A description of the reason.
 */
void onAgentWillLeave(int reason, String message);

Parameters:

Parameter	Type	Description
reason	int	The reason: 2001 (idle exit), 2002 (human takeover ended), 0 (other).
message	String	A description of the reason.

When a human agent is about to take over from the current agent

public void onHumanTakeoverWillStart(String takeoverUid, int takeoverMode);

Parameters:

Parameter	Type	Description
takeoverUid	String	The human agent's UID.
takeoverMode	int	1: Use the human agent's voice for output. 0: Use the agent's voice for output.

This callback is triggered when the human agent takeover is connected.

public void onHumanTakeoverConnected(String takeoverUid);

The human agent takeover is connected.

Parameter	Type	Description
takeoverUid	String	Real-person UID

Custom Agent Messages

/**
 * A custom message from the agent.
 * @param data The custom message body in a JSON string.
 */
public void onReceivedAgentCustomMessage(String data);

Parameters:

Parameter	Type	Description
data	String	The custom message body in a JSON string.

Agent Emotion Notifications

public void onAgentEmotionNotify(String emotion,int userAsrSentenceId);

Parameters:

Parameter	Type	Description
emotion	String	The emotion label, such as neutral, happy, angry, or sad.
userAsrSentenceId	int	The ID of the sentence that answers the user's question.

This callback indicates the availability of the agent's message channel. You can send messages to the agent only after this callback is triggered.

public void onAgentDataChannelAvailable() {}

This callback is triggered when a VCR result is received from the current agent. For more information, see ARTCAICallAgentVcrResult.

 public void onReceivedAgentVcrResult(ARTCAICallAgentVcrResult result) {}

The connection status can change during the call.

public void onConnectionStatusChange(ARTCAICallConnectionStatus status, ARTCAICallConnectionStatusChangeReason reason) {}

Parameters:

Parameter	Type	Description
status	ARTCAICallConnectionStatus	The status code: Init: Initialization is complete. Disconnected: The network connection is disconnected. Connecting: A network connection is being established. Connected: The network is connected. Reconnecting: The network connection is being re-established. Failed: The network connection failed.
reason	int	Cause