Intelligent Media Services:Implement agent animations

How it works

The official AI Real-Time Interaction demo uses After Effects to design the agent avatar animation, which is then exported as a Lottie resource file. The client integrates a Lottie component library to play and control the animation. For more information about Lottie, see LottieFiles.

In the official demo, animations designed in After Effects are split into modular components: head, hand, eyes, and covering eyes. The animation resources for each module are exported separately. The complete Lottie resource structure is as follows:

├── Avatar                       // Root directory for avatar animation resources
│   ├── bg.png                   // Background image
│   ├── Enter                    // Intro animation (while the call is connecting)
│   ├── Head                     // Head animation
│   ├── Hand                     // Hand animation
│   ├── CoveringEyes             // Hand animation for covering eyes (when the agent is interrupted)
│   ├── EyeEmotions              // Eye animations
│       ├── Interrupting         // Eye animation - Interrupted
│       ├── Listening            // Eye animation - Listening
│       ├── Thinking             // Eye animation - Thinking
│       ├── Speaking             // Eye animation - Speaking
│       ├── Happy                // Eye animation - Speaking happily
│       ├── Sad                  // Eye animation - Speaking sadly

Reference

The source code for the animations in the official demo can be found here:

Implementation details

In the official demo, the agent's head (nodding motion) and hand animations play in a loop. When the agent's state or emotion changes, the corresponding eye animation plays in a loop. When the agent is interrupted, the "covering eyes" hand animation plays once and then returns to the previous state. Additionally, the following logic requires special handling:

1. As the head animation loops up and down, the eyes follow this movement. You can achieve this by listening to the head animation's playback progress to estimate the Y-axis offset and then updating the eye animation's position in real-time.

   // Listen for update events on the head animation.
   headAnimator.addAnimatorUpdateListener(new ValueAnimator.AnimatorUpdateListener() {
       @Override
       public void onAnimationUpdate(ValueAnimator animation) {
           float animatedFraction = animation.getAnimatedFraction();
           float speed = headAnimator.getSpeed();
           float progress;
           if (speed < 0) {
               progress = 1 - animatedFraction;
           } else {
               progress = animatedFraction;
           }
           // Estimate the offset based on the playback progress.
           float y = eyeOffset * (progress - 1.0f);
           // Update the eye animation's Y translation.
           eyeAnimatorContainerView.setTranslationY(y);
       }
   });

2. Define the eye animation types and handle the transition logic:

   • The Error state is final and cannot be transitioned from.

   • If the next state is Interrupting, it cannot be overridden.

   • While in the Interrupting state, the animation cannot be switched. The next state is cached and will play after the interruption animation finishes.

   • When the agent is speaking with an emotion (HappySpeaking or SadSpeaking), it cannot be switched to the neutral Speaking state.

   • Transitions to Interrupting or Error are queued and execute after the current animation cycle finishes.

   public enum EyeAnimator {
       None,           // No animation
       Start,          // Agent starting up
       Error,          // Agent error state
       Interrupting,   // Agent is interrupted
       Thinking,       // Agent is thinking or loading
       Listening,      // Agent is listening
       Speaking,       // Agent is speaking (neutral)
       HappySpeaking,  // Agent is speaking (happy)
       SadSpeaking     // Agent is speaking (sad)
   }
   
   public void setEyeAnimatorType(EyeAnimator type) {
       Log.d("Animator", "Will Set Animator Type Curr: " + currEyeAnimatorType + " To: " + type);
       if (type == EyeAnimator.None) {
           return;
       }
   
       if (currEyeAnimatorType == type) {
           return;
       }
   
       if (currEyeAnimatorType == EyeAnimator.Error) {
           // The Error state is final and cannot be transitioned from.
           return;
       }
   
       if (nextEyeAnimatorType == EyeAnimator.Interrupting || nextEyeAnimatorType == EyeAnimator.Error) {
           // If the next state is Interrupting, it cannot be overridden.
           return;
       }
       if (currEyeAnimatorType == EyeAnimator.Interrupting) {
           // While in the Interrupting state, the animation cannot be switched. The next state is cached and will play after the interruption animation finishes.
           if (type == EyeAnimator.Speaking && nextEyeAnimatorType.ordinal() > type.ordinal()) {
               // If the next state is an emotional speaking state, it cannot be switched to the neutral Speaking state.
               return;
           }
           nextEyeAnimatorType = type;
           return;
       }
   
       if (type == EyeAnimator.Speaking && currEyeAnimatorType.ordinal() > type.ordinal()) {
           // If the agent is speaking with an emotion, it cannot be switched to the neutral Speaking state.
           return;
       }
   
       if (type == EyeAnimator.Interrupting || type == EyeAnimator.Error) {
           // If the next state is Interrupting or Error, wait for the current animation to complete.
           nextEyeAnimatorType = type;
           return;
       }
       Log.d("Animator", "Set Animator Type Next: " + type + " Curr: " + currEyeAnimatorType);
       startNextEyeAnimator(type);
   }

3. When an interruption event is triggered, the "covering eyes" animation must be synchronized with the head animation. It must start simultaneously with the first frame (progress=0) of the head animation's cycle. During this time, the eye animation cannot be switched to another state.

   // Listen for the end of the head animation.
   headAnimator.addAnimatorListener(new AnimatorListenerAdapter() {
       @Override
       public void onAnimationEnd(Animator animation) {
           headAnimator.reverseAnimationSpeed();
           headAnimator.playAnimation();  // Loop the head animation by playing it again.
           if (nextEyeAnimatorType != EyeAnimator.None) { // Check for a queued animation type.
               startNextEyeAnimator(nextEyeAnimatorType); // Synchronously start the next animation (e.g., covering eyes).
               if (currEyeAnimatorType == EyeAnimator.Interrupting) {
                   nextEyeAnimatorType = EyeAnimator.Listening;
               } else {
                   nextEyeAnimatorType = EyeAnimator.None;
               }
           }
       }
   });

4. Avatar animations can be difficult to design purely with vector graphics and may rely on many rasterized assets (bitmaps), which can increase memory consumption. To reduce memory usage, you can export only a one-way animation (e.g., nodding down) from After Effects and use the AutoReverse mode during playback to create a loop.

Implement custom agent animations