Intelligent Media Services:API operation details

Initialization

/**
 * Initializes the engine.
 * @param config The initialization configuration.
 */
public abstract void init(ARTCAICallConfig config);

Parameters:

Sets the agent type.

/**
 * Sets the agent type.
 * @param aiAgentType
 */
public abstract void setAICallAgentType(ARTCAICallAgentType aiAgentType);

Parameters:

Creates and starts a call.

/**
 * Creates and starts a call.
 */
public abstract void call(String rtcToken, String aiAgentInstanceId, String aiAgentUserId, String channelId);

Parameters:

Starts an agent call. This API operation is used to call an agent from the SDK. You must pass the ARTC channel token as the authentication token and set the agent ID, agent region, and related startup parameters in `mAiCallAgentTemplateConfig` of `ARTCAICallConfig`.

/**
   * Starts an agent call.
   * @param rtcToken The authentication token for the local user to join the ARTC channel.
   * @detail This API operation uses the SDK to call an agent. You must pass the ARTC channel token and set the agent ID, agent region, and related startup parameters in `mAiCallAgentTemplateConfig` of `ARTCAICallConfig`.
   */
public abstract void call(String rtcToken);

Parameters:

Hangs up the call.

/**
 * Hangs up the call.
 */
public abstract void handup();

Switches the microphone status.

/**
 * Switches the microphone status.
 * @param on
 */
public abstract void switchMicrophone(boolean on);

Parameters:

Interrupts the agent's speech.

/**
 * Interrupts the agent's speech.
 */
public abstract boolean interruptSpeaking();

Enables or disables smart interruption.

/**
 * Enables or disables smart interruption.
 */
public abstract boolean enableVoiceInterrupt(boolean enable);

Parameters:

Switches between the speaker and the earpiece. `true` uses the speaker for playback. `false` uses the earpiece for playback.

/**
 * Switches between the speaker and the earpiece.
 */
public abstract boolean enableSpeaker(boolean enable);

Parameters:

Stops or resumes the playback of the agent's audio stream.

public abstract boolean muteAgentAudioPlaying(boolean mute);

Parameters:

Enables or disables the push-to-talk mode.

/**
 * Enables or disables the push-to-talk mode. In push-to-talk mode, the agent reports the result only after `finishPushToTalk` is called.
 * @param enable
 * @return
 */
public abstract boolean enablePushToTalk(boolean enable);

Parameters:

Checks whether the push-to-talk mode is enabled.

/**
 * Checks whether the push-to-talk mode is enabled.
 * @return
 */
public abstract boolean isPushToTalkEnable();

In push-to-talk mode, this operation starts speaking.

/**
 * Push-to-talk mode: Starts speaking.
 * @return
 */
public abstract boolean startPushToTalk();

In push-to-talk mode, this operation finishes speaking.

/**
 * Push-to-talk mode: Finishes speaking.
 * @return
 */
public abstract boolean finishPushToTalk();

In push-to-talk mode, this operation cancels the current speech.

/**
 * Push-to-talk mode: Cancels the current speech.
 * @return
 */
public abstract boolean cancelPushToTalk();

Switches the agent's voice.

/**
 * Switches the voice.
 */
public abstract boolean switchRobotVoice(String voiceId);

Parameters:

You can obtain the current voice.

/**
 * Gets the current voice.
 * @return
 */
public abstract String getRobotVoiceId();

Registers a callback.

/**
 * Registers a callback.
 * @param engineCallback
 */
public abstract void setEngineCallback(IARTCAICallEngineCallback engineCallback);

Parameters:

Do you want to turn off the microphone?

/**
 * Checks whether the microphone is muted.
 * @return
 */
public abstract boolean isMicrophoneOn();

Checks whether the speaker is used for playback. `true` indicates that the speaker is used. `false` indicates that the earpiece is used.

/**
 * Checks whether the speaker is used for playback.
 * @return
 */
public abstract boolean isSpeakerOn();

Checks whether smart interruption is enabled.

/**
 * Checks whether smart interruption is enabled.
 * @return
 */
public abstract boolean isVoiceInterruptEnable();

Sets the view container for the digital human and specifies the rendering mode.

public abstract void setAvatarAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams avatarLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Sets the local view container for visual understanding and specifies the rendering mode.

public abstract void setVisionPreviewView(ViewGroup viewGroup, ViewGroup.LayoutParams visionLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Turn the camera on or off

/**
 * Mutes or unmutes the camera.
 */
public abstract boolean muteLocalCamera(boolean mute);

Parameters:

Is the camera off?

/**
 * Checks whether the camera is muted.
 * @return
 */
public abstract boolean isLocalCameraMute();

Switches between the front and rear cameras.

/**
 * Switches between the front and rear cameras.
 */
public abstract boolean switchCamera();

Obtain the RTC engine instance

/**
 * Gets the ARTC engine instance.
 * @return
 */
public abstract AliRtcEngine getRtcEngine();

Retrieves the official protocol implementation.

/**
 * Gets the official protocol implementation.
 * @return
 */
public abstract IARTCAICallService getIARTCAICallService();

Sends a custom message to the Large Language Model (LLM).

/**
 * Sends a custom message to the LLM.
 * @param req
*/
public  boolean sendTextToAgent(ARTCAICallSendTextToAgentRequest req);

Parameters:

Sends a custom message to the business server. This operation must be called after the call is connected.

/**
 * Sends a custom message to the AppServer.
 * Reuses the agent's message channel to send a message to the AppServer. The AppServer must listen for a callback to get the custom message.
 * @param msg
 * @return true: The message was sent successfully. false: The message failed to be sent.
*/
public boolean sendCustomMessageToServer(String msg);

Parameters:

Updates the system prompt for the LLM. This operation must be called after the call is connected.

/**
 * Updates the system prompt for the LLM. For example: "You are a friendly and helpful assistant focused on providing users with accurate information and advice."
 * @param prompt
 * @return true: The update was successful. false: The update failed.
*/
public boolean updateLlmSystemPrompt(String prompt)

Parameters:

Updates the parameters for the Alibaba Cloud Model Studio application center.

public abstract boolean updateBailianAppParams(String params);

Parameters:

Updates the Video Content Recognition (VCR) configuration.

 public abstract boolean updateVcrConfig(ARTCAICallAgentVcrConfig config);

Parameters:

For a visual understanding agent, this operation starts custom frame capture. After this operation is called, you cannot talk to the agent by voice. This operation must be called after the call is connected.

/**
 * For a Vision agent, starts custom frame capture. After this is started, you cannot talk to the agent through voice.
 * @param request The request parameters.
 * @return true: The request was sent successfully. false: The request failed to be sent.
*/
public  boolean startVisionCustomCapture(ARTCAICallVisionCustomCaptureRequest request);

Parameters:

For a visual understanding agent, this operation stops custom frame capture. This operation must be called after the call is connected.

public  boolean stopVisionCustomCapture();

Sets the video view container for the agent.

    public abstract void setAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams agentLayoutParams);

Parameters:

Sets the video view container for the agent and specifies the rendering mode.

    public abstract void setAgentView(ViewGroup viewGroup, ViewGroup.LayoutParams agentLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Sets the local preview view control.

public abstract void setLocalView(ViewGroup viewGroup, ViewGroup.LayoutParams localLayoutParams);

Parameters:

Sets the local preview view control and specifies the rendering mode.

public abstract void setLocalView(ViewGroup viewGroup, ViewGroup.LayoutParams localLayoutParams, ARTCAICallVideoCanvas canvas);

Parameters:

Retrieves the runtime information of the current agent.

public abstract ARTCAICallAgentInfo getAgentInfo();

Return value:

This API operation returns an ARTCAICallAgentInfo object that contains the following parameters:

An error occurred.

/**
 * An error occurred.
 * @param errorCode The error code.
 */
void onErrorOccurs(AICallErrorCode errorCode);

Parameters:

This callback is triggered when the agent starts successfully. If you use the client-side call[2/2] API operation to call the agent, this callback is received when the call is successful.

/**
 * The agent started successfully.
 */     
public void onAgentStarted() {}

Joining the call

/**
 * The call started (the user joined the channel).
 */
void onCallBegin();

You have left the meeting.

/**
 * The call ended (the user left the channel).
 */
void onCallEnd();

An instance of the underlying AliRtcEngine is created.

 public void onAliRtcEngineCreated(AliRtcEngine engine)

Parameters

Robot Status Synchronization

/**
 * The agent status is synchronized.
 * @param oldRobotState
 * @param newRobotState
 */
void onAICallEngineRobotStateChanged(ARTCAICallRobotState oldRobotState, ARTCAICallRobotState newRobotState);

Parameters:

This callback is triggered when the user is speaking.

/**
 * Callback for when the user is speaking.
 * @param isSpeaking Specifies whether the user is speaking.
 */
void onUserSpeaking(boolean isSpeaking);

Parameters:

Synchronous user speech recognition

/**
 * Synchronizes the text recognized by ASR from the user's speech.
 * @param text The specific text recognized by ASR.
 * @param isSentenceEnd Specifies whether the current text is the final result for this sentence.
 * @param sentenceId The ID of the sentence to which the current text belongs.
 * @param voicePrintStatusCode The voiceprint recognition status.
 */
void  void onUserAsrSubtitleNotify(String text, boolean isSentenceEnd, int sentenceId, VoicePrintStatusCode voicePrintStatusCode);

Parameters:

Synchronous response from the agent

/**
 * Synchronizes the agent's response text.
 * @param text The agent's speech.
 * @param end Specifies whether the current response is complete.
 * @param userAsrSentenceId Indicates the LLM content that responds to the speech input with the corresponding sentenceId.
 */
void onAIAgentSubtitleNotify(String text, boolean end, int userAsrSentenceId);

Parameters:

The LLM has finished responding to the current call.

/**
 * The LLM has finished answering in the current call.
 * @param text The text output by the LLM.
 * @param userAsrSentenceId The ID of the sentence that answers the user's question.
 * @note This callback provides the complete answer from the LLM to a specific question and is not synchronized with the agent's voice playback. For real-time captions, use onAIAgentSubtitleNotify.
 */
public void onLLMReplyCompleted(String text, int userAsrSentenceId) {}

Parameters:

Network Status Callback

/**
 * Callback for network status changes.
 * @param uid The ID of the user who joined the channel.
 * @param quality The network quality.
 */
void onNetworkStatusChanged(String uid, ARTCAICallNetworkQuality quality);

Parameters:

Volume Changes

/**
 * The volume changed.
 * @param uid The user ID.
 * @param volume The volume, from 0 to 255.
 */
void onVoiceVolumeChanged(String uid, int volume);

Parameters:

The current call's timbre has changed.

/**
 * The voice for the current call changed.
 */
void onVoiceIdChanged(String voiceId);

Parameters:

The barge-in settings for the current call have changed.

/**
 * The voice interruption setting for the current call changed.
 */
void onVoiceInterrupted(boolean enable);

Parameters:

Is there an active video stream from the agent (stream ingest)?

/**
 * The agent's video stream is available.
 */
void onAgentVideoAvailable(boolean available);

Parameters:

Agent audio is active (stream ingest).

/**
 * The agent's audio stream is available.
 */
void onAgentAudioAvailable(boolean available);

Parameters:

Audio loopback latency

/**
         * Audio loopback latency.
         *  @param id The statement ID.
         *  @param delay_ms The latency.
         */
public void onAudioDelayInfo(int id, int delay_ms) {}

Parameters:

First video frame rendering for a digital human

/**
*  The first video frame of the digital human is rendered.
*/
void onAgentAvatarFirstFrameDrawn();

User Online Status Callback

/**
*  Callback for when a user comes online.
*/
void onUserOnLine(String uid);

Parameters:

The agent was interrupted while speaking.

public void onSpeakingInterrupted(ARTCAICallSpeakingInterruptedReason reason);

Parameters:

Is the custom frame capture pattern enabled for the current visual understanding call?

public void onVisionCustomCapture(boolean enable)

Parameters:

The agent is about to disconnect from the current call.

/**
 * The current agent is about to leave the call.
 * @param reason The reason: 2001 (idle exit), 2002 (human takeover ended), 0 (other).
 * @param message A description of the reason.
 */
void onAgentWillLeave(int reason, String message);

Parameters:

When a live agent is about to take over from the current agent

public void onHumanTakeoverWillStart(String takeoverUid, int takeoverMode);

Parameters:

This callback is triggered when the human agent takeover is connected.

public void onHumanTakeoverConnected(String takeoverUid);

When a human agent is connected

Custom Agent Messages

/**
 * A custom message from the agent is received.
 * @param data The custom message body in a JSON string.
 */
public void onReceivedAgentCustomMessage(String data);

Parameters:

Agent Emotion Notifications

public void onAgentEmotionNotify(String emotion,int userAsrSentenceId);

Parameters:

This callback is triggered when the agent's message channel becomes available. You can send messages to the agent only after this callback is triggered.

public void onAgentDataChannelAvailable() {}

This callback is triggered when a VCR result is received from the current agent. For more information, see ARTCAICallAgentVcrResult.

 public void onReceivedAgentVcrResult(ARTCAICallAgentVcrResult result) {}