Emotion level dubbing - Intelligent Media Services - Alibaba Cloud Documentation Center

Emotion level dubbing (EmotionLevel) controls the emotional intensity of video translation dubbing. It weakens the intonation patterns carried over from the source language, so the translated audio sounds more natural to native speakers of the target language. This feature is available only through the API, supports only speech translation between Chinese and English, and takes effect only in supported regions.

Quick example

When you call the SubmitVideoTranslationJob operation, set EditingConfig.SpeechTranslate.EmotionLevel to specify the emotion level. The following example translates a Chinese video into English with the SlightEmotion level:

{
  "InputConfig": {
    "Type": "Video",
    "Video": "1628ae20c36******8f6f7c77a6302"
  },
  "EditingConfig": {
    "SourceLanguage": "zh",
    "TargetLanguage": "en",
    "NeedSpeechTranslate": true,
    "SpeechTranslate": {
      "EmotionLevel": "SlightEmotion"
    }
  },
  "Title": "Emotion level dubbing example",
  "OutputConfig": {
    "MediaURL": "https://****.oss-cn-shanghai.aliyuncs.com/output.mp4"
  }
}

After you set "EmotionLevel": "SlightEmotion", the dubbing weakens the intonation patterns of the source language and keeps slight emotional inflections. This level suits corporate promotions, product introductions, and similar content.

How it works

Emotion level dubbing is an enhancement configuration for speech translation, not an independent translation level. By default, speech translation uses voice cloning to preserve the timbre, emotions, and prosody of the original speaker in the target language dubbing. Because these characteristics carry over from the source language, the dubbed audio can keep a noticeable accent and sound unnatural to native speakers. Emotion level dubbing weakens the carried-over intonation patterns and offers the following two emotion control modes, so you can choose how much emotional expression the dubbed audio keeps:

Emotion level	Value	Description
Slight emotion	`SlightEmotion`	Retains the basic intonation of the original audio. The dubbing sounds natural with slight emotional inflections.
No emotion	`NoEmotion`	Uses a completely flat and stable tone for dubbing with no emotional inflections.

If you do not set the EmotionLevel parameter, the system uses the default voice cloning mode for dubbing.

Scenarios

During cross-language dubbing, the default voice cloning mode carries the intonation habits of the source language into the target language:

Chinese to English: Chinese intonation patterns carry into the English dubbing. The English then sounds accented, with abnormal emotional inflections that do not match native English speech patterns.
English to Chinese: English intonation patterns carry into the Chinese dubbing. The Chinese then sounds like a non-native speaker, with unnatural rhythm and tones.

The following table compares the three dubbing modes:

Dubbing mode	Effect	Advantages	Disadvantages	Scenarios
Default voice cloning mode	Fully preserves the intonation and emotions of the original speaker	Rich emotional expression with distinctive original voice characteristics	Noticeable source language accent in cross-language dubbing	Content that requires high emotional fidelity, such as film commentary and emotional short videos
Slight emotion (`SlightEmotion`)	Weakens source language intonation while retaining slight emotional inflections	More stable intonation and a reduced source language accent, while preserving some emotional coloring	Weakened emotional expression, not suitable for highly emotional content	Corporate promotions, product introductions, online courses, and other content that requires stable intonation
No emotion (`NoEmotion`)	Completely flat and stable tone with no emotional inflections	Maximally removes the source language accent, producing the most neutral and stable dubbing	Complete absence of emotional expression, not suitable for content that requires emotional impact	News broadcasts, technical explanations, operation demonstrations, and other extremely neutral content

If your content requires high emotional fidelity, do not set EmotionLevel so that the default voice cloning mode applies. To remove the source language accent as much as possible, set EmotionLevel to NoEmotion, which weakens emotional and intonation inflections the most and produces completely stable dubbing. To reduce the accent while keeping a small amount of the original intonation, set EmotionLevel to SlightEmotion; the result sits between the default voice cloning mode and NoEmotion.

Supported scope

Supported languages

Emotion level dubbing supports only the following language pairs:

Source language	Target language	Supported
Chinese (zh)	English (en)	Yes
English (en)	Chinese (zh)	Yes
Other language combinations	—	Not supported

If you do not set EmotionLevel, speech translation in all supported target languages is unaffected and continues to use the default voice cloning mode.

Supported regions

Region	Supported
China (Shanghai)	Yes
China (Beijing)	Yes
China (Shenzhen)	Yes
China (Hangzhou)	Yes
Singapore	Yes

API usage

Emotion level dubbing is available only through the API. The console does not support this configuration. Call the SubmitVideoTranslationJob operation to set the emotion level.

Parameter reference

Use the EditingConfig.SpeechTranslate.EmotionLevel field to set the emotion level:

Parameter	Type	Required	Values	Description
`EmotionLevel`	String	No	`SlightEmotion`	Slight emotion dubbing. For a comparison of the three dubbing modes, see Scenarios.
`EmotionLevel`	String	No	`NoEmotion`	No emotion dubbing. For a comparison of the three dubbing modes, see Scenarios.

If you do not pass this parameter, the system uses the default voice cloning mode.
EmotionLevel takes effect only when speech translation is enabled (NeedSpeechTranslate set to true) and translation runs between Chinese (zh) and English (en) in either direction. For more information, see Supported scope.
If you also set the VoiceCloneMode parameter, EmotionLevel takes priority.

Examples

The following example translates an English video into Chinese and sets EmotionLevel to NoEmotion:

{
  "InputConfig": {
    "Type": "Video",
    "Video": "a3f9b21d457******c8e2a1b3d4e5f6a7b"
  },
  "EditingConfig": {
    "SourceLanguage": "en",
    "TargetLanguage": "zh",
    "NeedSpeechTranslate": true,
    "SpeechTranslate": {
      "EmotionLevel": "NoEmotion"
    }
  },
  "Title": "No emotion dubbing example",
  "OutputConfig": {
    "MediaURL": "https://****.oss-cn-shanghai.aliyuncs.com/output_noemotion.mp4"
  }
}

Limits

Translation method — EmotionLevel applies only to speech translation (NeedSpeechTranslate set to true). Subtitle translation does not involve dubbing, so this parameter does not apply.
Languages and regions — The parameter takes effect only for supported language pairs and regions. For more information, see Supported scope.
Access method — Only API calls are supported. The console does not provide a configuration entry.
Parameter priority — If you set both EmotionLevel and VoiceCloneMode, EmotionLevel takes priority.

FAQ

What is the difference between video translation in Intelligent Media Services and Machine Translation?

Intelligent Media Services and Machine Translation use similar underlying logic for video translation, but they open the capability in different ways. Intelligent Media Services provides a video translation API and SDKs, so you can build your own integration. The video translation feature of Machine Translation does not provide an API; its APIs cover only text, image, and document translation.

To develop video translation through an API, use Intelligent Media Services.

References

SubmitVideoTranslationJob: Set the EmotionLevel parameter in this operation to submit a video translation job.
Video translation: Learn about the full video translation capability, including subtitle translation, speech translation, and lip-sync translation.