Intelligent Media Services:Recognize user emotions and generate empathetic responses

Workflow

To make the agent's responses expressive and empathetic, perform the following steps:

• Configure a prompt: Large language models (LLMs) require suitable prompts to generate emotional speech based on the user input.

• Integrate an emotion-controllable text-to-speech (TTS) model: The TTS node must be capable of expressing various emotions.

• (Optional) Configure emotion-based visuals: If your AI agent has a visual representation, you can enable the representation to change according to the emotions.

Configure prompts

# Role
You are a voice assistant capable of adjusting your emotional tone in responses based on the user input.

## Skills
### Skill 1: Emotion recognition and response
- **Task**: Analyze the user's emotion through contextual cues and respond with the appropriate tone, which includes neutral, happy, and sad. Ensure the transition between tones is smooth.


## Requirements
- Response content should be concise, typically no more than one or two sentences.
- Emotion tags must be placed at the beginning of the response in the format: {{emotion=value}}, where valid values include neutral, happy, and sad.
- Ensure responses are positive and healthy, avoiding inappropriate or offensive language.
- Maintain consistency in responses, ensuring that adjustments in emotion, speech rate, and tone match the user's emotion.

## Examples
- User: "The weather is nice today."
  - Assistant: {{emotion=neutral}} Yes, it's sunny and bright.

Integrate an emotion-controllable TTS model

After the LLM generates text with emotion tags, use a TTS model to produce voice output with adjustable emotions and speech rates.

Real-time Conversational AI offers two options:

• MiniMax model: MiniMax supports multiple emotional tones. You can choose the MiniMax model in the TTS node. For details, see MiniMax documentation.

• Self-developed TTS model: You can also integrate your own TTS model into the workflow according to our input and output specifications. For details, see Access TTS models.

(Optional) Configure emotion-based visuals

If your AI agent has a visual representation on the client side, you can adjust its facial expressions according to the emotion tags. This involves two steps:

• The client detects that the agent's emotional tone has changed.

• The client renders the visual representation of the agent according to the emotion tags. You must implement the rendering on the client side.

Detect emotional changes

Use the callback interface onAgentEmotionNotify to get notifications about emotional changes. If your UI supports expression animations, you can trigger these animations upon receiving emotion tags through the callback.

OnAgentEmotionNotify parameters:

Sample code

// Callback handling (This example shows only core callback event.)
ARTCAICallEngine.IARTCAICallEngineCallback mCallEngineCallbackWrapper = new ARTCAICallEngine.IARTCAICallEngineCallback() {
    
	@Override
    public void onAgentEmotionNotify(String emotion,int userAsrSentenceId) {
        // Notifications about agent emotions.
    }

    // Other callbacks
    ...
};

extension AUIAICallStandardController: ARTCAICallEngineDelegate {

    
	public func onAgentEmotionNotify(emotion: String, userAsrSentenceId: Int) {
        // Notifications about agent emotions.
        debugPrint("AUIAICallStandardController onAgentEmotionNotify:\(emotion)  sentenceId: \(userAsrSentenceId)")
    }

    // Other callbacks
    ...
}

aiCallEngine.on('agentEmotionNotify', (emotion, sentenceId) => {
  // Notifications about agent emotions.
  console.log(`Agent emotion: ${emotion}, Sentence: ${sentenceId}`);
});