Intelligent Media Services:Hand off a conversation to a human agent

Detailed process

Join the channel

1. The human agent-side calls TakeoverAIAgentCall.

2. The RTC token and other information are returned for joining the channel.

3. The human agent joins the channel based on the returned information and starts to push and pull the streams. They receive transcripts of the previous communication from the AI agent.

   For more information, see Get started with ARTC SDK for Web

Sample code for receiving transcripts from the AI agent:

1. Listen for the event.

   aliRtcEngine.on('dataChannelMsg', (uid, msg) => {
     try {
       const dataChannelMsg = JSON.parse(new TextDecoder().decode(msg.data));
       if (dataChannelMsg.type === 1002) {
         // Receive transcripts of the AI agent. Data structure: 
         // text: The transcribed text.
         // sentenceId: The sentence ID. During streaming, the words are sent to the system piece by piece. This ID is used to identify which sentence different pieces belong to. 
         // end: A tag that indicates whether the current piece is the final segment of the sentence.
         console.log('agentSubtitleNotify', dataChannelMsg.data);
       } else if (dataChannelMsg.type === 1003) {
         // Receive transcripts of the user. Data structure: 
         // text: The transcribed text.
         // sentenceId: The sentence ID. During streaming, the words are sent to the system piece by piece. This ID is used to identify which sentence different pieces belong to. 
         // end: A tag that indicates whether the current piece is the final segment of the sentence.
         console.log('userSubtitleNotify', dataChannelMsg.data);
       } else {
         // Ignore other types of messages
       }
     } catch(error) {
       console.error(error)
     }
   });

2. Before joining the channel, call the aliRtcEngine.setParameter method to enable the capability for receiving the transcripts.

   aliRtcEngine.setParameter(
     JSON.stringify({
       data: {
         enableSubDataChannel: true,
       },
     }),
   );

Subscribe to content on-demand

By default, a human agent subscribes to content of all users in the channel, including the end user and AI agent. So, they will receive the audio and video streams from the end user and the speech-to-text recognition results from the AI agent.

If content from the AI agent is not required, subscribe to only the audio and video streams from the end user by performing the following steps:

1. Before joining the channel, call setDefaultSubscribeAllRemoteAudioStreams to disable automatic subscription.

   // Disable automatic subscription to audio streams.
   aliRtcEngine.setDefaultSubscribeAllRemoteAudioStreams(false);
   // Disable automatic subscription to video and screen-sharing streams. 
   aliRtcEngine.setDefaultSubscribeAllRemoteVideoStreams(false);

2. Call subscribeRemoteMediaStream to subscribe to a specific user.

   // Import AliRtcVideoTrack enum via npm.
   import { AliRtcVideoTrack } from 'aliyun-rtc-sdk';
   // You can also import it from the window object.
   // const AliRtcVideoTrack = window.AliRtcEngine.AliRtcVideoTrack;
   
   // Specify the ID of the user you want to subscribe to.
   const subscribeUserId = 'xxxx';
   
   // Set up an event listener for when a user comes online (remoteUserOnLineNotify) and subscribe to the joined user.
   // Decide whether to subscribe to a user based on your business logic.
   aliRtcEngine.on('remoteUserOnLineNotify', (userId) => {
     if (userId === subscribeUserId) {
       // Subscribe to all streams of the user (audio, video, and screen-sharing). The system does not prompt an error if the user does not ingest certain streams. 
       // Future streams added by the user will be automatically subscribed to. 
       aliRtcEngine.subscribeRemoteMediaStream(
         userId,
         AliRtcVideoTrack.AliRtcVideoTrackBoth,
         true,
         true
       );
     }
   });
   
   // Subscribe to a user already in the channel.
   if (aliRtcEngine.getOnlineRemoteUsers()?.includes(subscribeUserId)) {
     // Subscribe to all streams of the user (audio, video, and screen-sharing). The system does not prompt an error if the user does not ingest certain streams. 
     // Future streams added by the user will be automatically subscribed to. 
     aliRtcEngine.subscribeRemoteMediaStream(
       userId,
       AliRtcVideoTrack.AliRtcVideoTrackBoth,
       true,
       true
     );
   }
   

Inform users about human takeover

You can obtain the status of human agents during the takeover process through callbacks and provide real-time feedback to users.

To notify the user that the conversation is handed off to a human agent, implement the callback interfaces onHumanTakeoverWillStart and onHumanTakeoverConnected. Sample code:

// Callbacks (for reference only)
ARTCAICallEngine.IARTCAICallEngineCallback mCallEngineCallbackWrapper = new ARTCAICallEngine.IARTCAICallEngineCallback() {
    
	@Override
    public void onHumanTakeoverWillStart(String takeoverUid, int takeoverMode) {
        // When the transition is about to start.
    }

    @Override
    public void onHumanTakeoverConnected(String takeoverUid) {
        // When the human agent has successfully taken over.
    }

    // Other callbacks.
    ...
};

extension AUIAICallStandardController: ARTCAICallEngineDelegate {
    
	public func onHumanTakeoverWillStart(takeoverUid: String, takeoverMode: Int) {
        // When the transition is about to start.
        debugPrint("AUIAICallStandardController onHumanTakeoverWillStart:\(takeoverUid) , takeoverMode:\(takeoverMode)")
    }
    
    public func onHumanTakeoverConnected(takeoverUid: String) {
        // When the human agent has successfully taken over.
        debugPrint("AUIAICallStandardController onHumanTakeoverConnected:\(takeoverUid)")
    }

    // Other callbacks.
    ...
}

engine.on('humanTakeoverWillStart', (uid: string, mode: number) => {
  // When the transition is about to start.
  console.log('AICallHumanTakeoverWillStart', uid, mode);
});
engine.on('humanTakeoverConnected', (uid: string) => {
  // When the human agent has successfully taken over.
  console.log('AICallHumanTakeoverConnected', uid);
});