Intelligent Media Services:Developer guide

Background

Large language models (LLMs) serve as AI agents that address many industry needs, especially when integrated with internal knowledge bases. Common use cases include smart customer service and personal assistants. Because text-based AI agents often struggle to deliver efficient communication, AI agents based on ApsaraVideo Real-time Communication (RTC) were developed to provide a richer and more intuitive interactive experience.

Integration solutions

For audio and video calls and message conversations, Alibaba Cloud offers two solutions that use the AICallKit SDK: one with a UI and one without.

• Integration with a UI: Alibaba Cloud provides ready-made UI components for audio and video applications. Configure these components to run the demo, or integrate them into your project to quickly add Real-time Conversational AI features.

• Integration without a UI: Customize your own UI while the AICallKit SDK handles the underlying Real-time Conversational AI implementation, letting you quickly add Real-time Conversational AI features to your application.

Server-side features

• Personalized call configuration: Configure custom greetings, voice tones, and smart interruption for the AI agent.

• Retrieve call content: Built-in speech recognition automatically transcribes calls, allowing you to review call records and use them for tasks such as model training.

• Agent callbacks: Automatically trigger preset actions or responses when a specific event occurs.

• Speech recognition hotwords: Improve recognition accuracy for domain-specific words by adding them as hotwords.

• Digital human integration: Use the workflow to convert speech input into animated responses from a digital human for a more engaging interactive experience.