Intelligent Media Services:Overview of Real-time Conversational AI

SessionId

A developer-defined ID. We recommend that you use it as the unique identifier for chat records. Examples:

• User association: When users chat with the AI agent on a mobile phone or PC, the sessionId can link conversations across different times.

• Session association: When a user initiates multiple sessions, the sessionId can be used to isolate them.

Messaging

Users interact with the agent through voice or text in a chat dialog box. This allows both parties to quickly share ideas, ask questions, or get information.

Audio call

Users interact with the intelligent assistant through voice to get timely information and support.

3D digital human call

Uses 3D technology to simulate a virtual character for interaction. A 3D digital human can perform voice interactions and use rich body movements and facial expressions to enhance the realism and engagement of the user experience.

Visual understanding call

A new interactive method that combines video and audio. It analyzes images captured by the camera in real time and combines them with user voice commands to provide accurate feedback through multi-modal interaction. This gives users a more intuitive, efficient, and personalized intelligent interaction experience during calls, breaking the limits of traditional voice or text communication.

Video call

Combines the benefits of digital humans and visual understanding. During a video call, both the digital human's and the user's video are displayed. The digital human can understand the user's video feed and provide feedback, which enhances the sense of interaction and realism.

Interactive messages

A service that enhances message communication between users and improves the interactive experience.

ApsaraVideo Real-time Communication (ARTC)

Real-time audio and video calls between users and AI agents require Web Real-Time Communication (WebRTC) technology. Backed by over 3,200 points of presence (POPs) worldwide and years of audio and video technology experience, Alibaba Cloud ApsaraVideo Real-time Communication (ARTC) provides high availability (HA), high-quality, and ultra-low-latency audio and video communication services. For more information, see Introduction to ApsaraVideo Real-time Communication.

Real-time workflow

A real-time workflow is a key part of an AI agent. It lets you flexibly orchestrate AI components, such as speech-to-text, large language models, speech synthesis, and self-developed vector databases, using plug-ins and drag-and-drop actions. The AI agent operates according to this predefined workflow.

AI agent

A highly realistic cloud-based user defined in the Real-time Conversational AI solution. An AI agent can be preset by the system or created by a user. It can directly interact with end users through audio and video.

Feature

Description

Real-time workflow

Flexibly orchestrate the agent's AI workflow in a visual editor.

• Speech-to-text:

  • Integrates Alibaba Cloud Qwen's capabilities.

• Speech synthesis (text-to-speech):

  • Integrates Alibaba Cloud Qwen's capabilities.

  • Connects to your self-developed speech synthesis module using standard protocols.

  • Supports integrating MiniMax's voice capabilities as a third-party plugin.

• Text-to-text large language model:

  • Integrates Alibaba Cloud Qwen's capabilities.

  • Select AI models from the Model Hub or Application Center on the Alibaba Cloud Model Studio platform.

  • You can integrate your custom large language model by following the OpenAI specifications.

• Digital human

  • Supports integrating Faceunity's digital human capabilities as a third-party plugin.

• Video frame extraction

• Multi-modal large language model

  • Integrates Alibaba Cloud Qwen's capabilities.

  • You can integrate your custom multimodal large language model by following OpenAI specifications.

Custom agent image

Upload an image for the agent you create to display its avatar in voice call scenarios.

Agent emotion recognition

The agent can recognize the user's current emotion and provide an emotional response.

Welcome message

Configure a welcome message in the console. When a user starts a conversation with the AI agent, the agent plays the welcome message.

Proactive broadcast

The business server can use an OpenAPI operation to have the agent proactively output audio and video content to the user.

Real-time captions

The conversation between the user and the agent can be rendered in real time on the end-user interface.

Intelligent noise reduction

The AI agent automatically filters noise from the user's side during the conversation. When multiple people on the user's side are speaking at the same time, the voice with the highest volume is prioritized.

Intelligent interruption

When talking with the AI agent, the agent can effectively recognize the user's intent to interrupt the conversation.

Intelligent sentence segmentation

The agent can automatically recognize and segment long or complex sentences to improve text readability and user experience.

Sentence-by-sentence audio callback

Configure a callback in the console to store real-time audio data in OSS.

Walkie-talkie mode

Users can set the call mode to walkie-talkie mode at the start of or during a call and interact with the agent by pressing a button.

ASR hotwords

Define business-related hotwords to improve the AI agent's accuracy in speech recognition.

Voiceprint denoising

In a multi-person conversation scenario, the agent identifies the voiceprint features of the main speaker to more accurately capture and retain their speech while reducing interference from irrelevant noise.

Human takeover

When a user interacts with the agent, if a situation cannot be handled or a key decision needs to be made, a human can take over to make the decision.

Graceful shutdown

When the business server needs to stop the agent, the agent is allowed to finish its current utterance before stopping. This avoids abrupt interruptions to the conversation.

Data archiving

Convert the conversation between the user and the AI agent into text and store it. Businesses can call an API operation to consume this data. Businesses can also store the audio and video data from calls between users and AI agents on the Object Storage Service (OSS) or ApsaraVideo VOD (VOD) platform.