Movie highlights scenario - parameters and SDK examples - Intelligent Media Services

Usage notes

To produce videos in batches from multiple media assets, call SubmitBatchMediaProducingJob - Produce multiple smart videos at a time. Key parameters (InputConfig, EditingConfig, and OutputConfig) are described below.
To query a batch job, call GetBatchMediaProducingJob - Get information about a batch smart video generation job.

InputConfig parameters

InputConfig specifies basic asset parameters: video clips, narration, background music, and stickers.

Parameter	Type	Description	Example	Required	Supported modes
MediaArray	List<String>	List of media asset IDs or OSS URLs. Total video duration cannot exceed two hours. Supported formats: Video formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	Yes	Fully supported
TitleArray	List<String>	Array of titles. A random title is selected per video. Maximum: 50 titles, 50 characters each.	["Title 1","Title 2"]	No	Fully supported
SubHeadingArray	List<SubHeading>	Subtitle settings.	[{"Level":1,"TitleArray":["Level-1 Subtitle 1","Level-1 Subtitle 2"]},{"Level":3,"TitleArray":["Level-3 Subtitle"]}]	No	All modes
SpeechTextArray	List<String>	Array of narration scripts. A random script is selected per video. Maximum: 50 scripts, 1,000 characters each. Control speech synthesis with SSML. Important Currently, only <break>, <s>, and <phoneme> are supported. For CosyVoice timbres, only <break>, <s>, and <sub> are supported.	["Narration content 1","Narration content 2"]	No	Global narration mode
SceneInfo	SceneInfo	Scenario-related parameters.	Parameter example: Global narration mode and Parameter example: Storyboard mode.	Yes	Global narration mode Storyboard mode
StickerArray	List<Sticker>	Array of stickers. A random sticker is selected per video. Maximum: 50. Use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 stickers and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, stickers are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. Supported formats: Image formats.	[{"MediaId":"**9d46c8b4548681030f6e**","X":10,"Y":100,"Width":300,"Height":300,"Opacity":0.6}]	No	All modes
BackgroundMusicArray	List<String>	Array of background music tracks. A random track is selected per video. Maximum: 50. Use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 background music tracks and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, tracks are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. Supported formats: Audio formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	No	Fully supported
BackgroundImageArray	List<String>	Array of background images. A random image is selected per video. Maximum: 50. Use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 background images and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, images are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. Supported formats: Image formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	No	All modes

SceneInfo parameters

Parameter	Type	Description	Required	Supported modes
Scene	String	Matching scenario type. Set to the static field "MovieHighlights" for this scenario.	Yes	Global narration mode Storyboard mode
ShotInfo	ShotInfo	Configure the storyboard.	Yes	Storyboard mode
FaceInfo	FaceInfo	Configure face information.	No	Fully supported

ShotInfo parameters

Note

Applies only to storyboard mode.

Parameter	Type	Description	Required
ShotScripts	List<ShotScript>	An array of shot scripts.	Yes

ShotScript parameters

Note

Applies only to storyboard mode.
Two shot modes are available: Text description mode and Manual parsing mode. Select one when setting parameters.

Parameter	Type	Description	Example	Required	Shot mode
ScriptText	String	Script text describing the shot content. Summarize in one or two sentences.	In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.	No	Text description mode
SpeechText	String	Narration script for a single shot. Maximum: 100 characters. Control speech synthesis with SSML. Important Currently, only <break>, <s>, and <phoneme> are supported.	The late-night car is quiet, but some people light up the whole car with their songs.	No	All modes
Duration	Float	The duration of the shot in seconds. The value must be 1 or greater. Takes effect only without narration. With narration, duration is calculated from the narration length.	5	No	All modes
Descriptions	List<String>	Shot descriptions. Summarize in one or two sentences.	["Lin Xia sits in a corner of the subway car, quietly recording with her phone", "An old woman holds her grandson's hand, humming a song softly"]	No	Manual parsing mode
Characters	List<String>	Character (face) names for a single shot. Must match ImageInfo.Name in FaceInfo.ImageInfoList.	["Lin Xia", "Old woman", "Grandson"]	No
Settings	List<String>	The scene description for a single shot.	["Late-night subway car"]	No
Volume	Float	Input video volume for this shot. Overrides EditingConfig.MediaConfig.Volume when set. Range: [0, 10.0]. Up to two decimal places.	0.5	No	Fully supported

FaceInfo parameters

Parameter	Type	Description	Required
ImageInfoList	List<ImageInfo>	List of character (face) photos. Maximum: 200.	No

ImageInfo parameters

Parameter	Type	Description	Example	Required
Name	String	Character (face) name.	Daniel	Yes
ImageURL	String	URL of the character (face) photo. Must be publicly accessible. The image must contain only one face, clearly visible without obstructions or missing parts.	http://[your-cdn-domain]/[your-file-path]/face1.png	Yes. You must specify one of the two parameters.
ImageId	String	Image media asset ID.	**9d46c886b45481030f6e**	Yes. You must specify one of the two parameters.

Parameter example: Global narration mode

{
  "MediaArray": [
    "****9d46c886b45481030f6e****",
    "****c886810b4549d4630f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.mp4",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test2.png"
  ],
    "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character B",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/622762d0f703918ffbedc1125b3d269759eec42e?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character C",
          "ImageId": "****b681034549d46c880f6e****"
        }
      ]
    }
  },
  "TitleArray": [
    "Hema Fresh in Huilongguan is now open!",
    "Hema Fresh is now open!"
  ],
  "SubHeadingArray": [
    {
      "Level": 1,
      "TitleArray": ["Subtitle 1", "Subtitle 2"]
    },
    {
      "Level": 3,
      "TitleArray": ["Level-3 Subtitle"]
    }
  ],
  "SpeechTextArray": [
    "A new Hema Fresh just opened in the nearby mall. It's the grand opening today, so I rushed over to join the fun. This Hema isn't very big, but the mall is packed. Snacks and drinks are quite cheap, and the lines are incredibly long. Come and check it out!",
    "A new Hema Fresh just opened in the nearby mall. It's the grand opening today, so I rushed over to join the fun."
  ],
  "Sticker": {
    "MediaId": "****b681034549d46c880f6e****",
    "X": 10,
    "Y": 100,
    "Width": 300,
    "Height": 300
  },
  "StickerArray": [
    {
      "MediaId": "****9d46c8b4548681030f6e****",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300,
      "Opacity": 0.6
    },
    {
      "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test3.png",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300
    }
  ],
  "BackgroundMusicArray": [
    "****b4549d46c88681030f6e****",
    "****549d46c88b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test4.mp3"
  ],
  "BackgroundImageArray": [
    "****6c886b4549d481030f6e****",
    "****9d46c8548b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.png"
  ]
}

Parameter example: Storyboard mode

{
  "MediaArray": [
    "****9d46c886b45481030f6e****",
    "****c886810b4549d4630f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.mp4",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test2.png"
  ],
   "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "ShotInfo": {
      "ShotScripts": [
        // Start of text description mode example. For each shot, choose either manual parsing mode or text description mode.
        {
          "ScriptText": "In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.",
          "SpeechText": "The late-night car is quiet, but some people light up the whole car with their songs."
        },
        {
           "ScriptText": "The old woman's white hair glows under the car lights. She strokes her grandson's head and whispers, 'Do you remember this song?'",
           "SpeechText": "Her world may be blurry, but in front of him, she is still the most gentle mother."
        },
        {
           "ScriptText": "Lin Xia stops filming and looks at the billboards flashing by outside the window, her expression softening.",
           "SpeechText": "I thought I was recording someone else's story, but I was quietly healed."
        },
        {
           "ScriptText": "At home, Lin Xia takes out her camera and writes her first diary entry: 'Today, I saw what happiness looks like.'",
           "SpeechText": "Sometimes, happiness isn't about big things, but the small moments we overlook."
        },
        {
           "ScriptText": "A few days later, a series of heartwarming photos on Lin Xia's social media resonates with netizens.",
           "SpeechText": "She starts telling stories with her lens, and every story holds a piece of love."
        },
        {
           "ScriptText": "At night, Lin Xia enters the subway station again, this time with a smile and a new goal.",
           "SpeechText": "This city is big, but there's always someone waiting for you, and someone worth discovering."
        }
        // End of text description mode example
        
        // Start of manual parsing mode example. For each shot, choose either manual parsing mode or text description mode.
        {
            "Descriptions": ["Lin Xia sits in a corner of the subway car, quietly recording with her phone", "An old woman holds her grandson's hand, humming a song softly"],
            "Characters": ["Lin Xia", "Old woman", "Grandson"],
            "Settings": ["Late-night subway car"],
            "SpeechText": "The late-night car is quiet, but some people light up the whole car with their songs."
        },
        {
            "Descriptions": ["The old woman's white hair glows under the car lights", "She strokes her grandson's head and speaks softly"],
            "Characters": ["Old woman", "Grandson"],
            "Settings": ["Subway car"],
            "SpeechText": "Her world may be blurry, but in front of him, she is still the most gentle mother."
        },
        {
            "Descriptions": ["Lin Xia stops filming", "Looks at the billboards flashing by outside the window, her expression softening"],
            "Characters": ["Lin Xia"],
            "Settings": ["Moving subway car"],
            "SpeechText": "I thought I was recording someone else's story, but I was quietly healed."
        },
        {
            "Descriptions": ["Lin Xia takes out her camera at home", "Writes her first diary entry: 'Today, I saw what happiness looks like'"],
            "Characters": ["Lin Xia"],
            "Settings": ["Modern apartment / Room"],
            "SpeechText": "Sometimes, happiness isn't about big things, but the small moments we overlook."
        },
        {
            "Descriptions": ["Heartwarming photos are posted on Lin Xia's social media page", "Netizens like and comment one after another"],
            "Characters": ["Lin Xia"],
            "Settings": ["Phone screen / Social media interface"],
            "SpeechText": "She starts telling stories with her lens, and every story holds a piece of love."
        },
        {
            "Descriptions": ["At night, Lin Xia enters the subway station", "She has a smile and a new goal"],
            "Characters": ["Lin Xia"],
            "Settings": ["City subway station"],
            "SpeechText": "This city is big, but there's always someone waiting for you, and someone worth discovering."
        }
        // End of manual parsing mode example

        // You can choose not to configure narration. Configure only the shot script and duration to play the original video sound.
       {
            "ScripText": "In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.",
            "Duration": 8.0, // Can be set when there is no narration script 
            "Volume": 1.0 // Set the volume of the original video asset
       }
      ]
    },
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character B",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/622762d0f703918ffbedc1125b3d269759eec42e?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character C",
          "ImageId": "****b681034549d46c880f6e****"
        }
      ]
    }
  },
  "TitleArray": [
    "Hema Fresh in Huilongguan is now open!",
    "Hema Fresh is now open!"
  ],
  "StickerArray": [
    {
      "MediaId": "****9d46c8b4548681030f6e****",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300
    },
    {
      "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test3.png",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300,
      "Opacity": 0.6
    }
  ],
  "BackgroundMusicArray": [
    "****b4549d46c88681030f6e****",
    "****549d46c88b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test4.mp3"
  ],
  "BackgroundImageArray": [
    "****6c886b4549d481030f6e****",
    "****9d46c8548b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.png"
  ]
}

EditingConfig parameters

Note

EditingConfig specifies volume, position, and other composition parameters for output video assets. Leave empty to use defaults.
Parameters apply to both global narration and storyboard modes.

Parameter	Type	Description	Example	Required
MediaConfig	JSON	Configurations for input video assets.	Parameter example.	No
TitleConfig	JSON	Title configurations. Caption parameters are described in Banner text.	Parameter example.	No
SubHeadingConfig	JSON	Configurations for multi-level subtitles. You can set caption parameters. JSON field description: key: Level value: Banner text	Parameter example.	No
SpeechConfig	JSON	Configurations for the narration script.	Parameter example.	No
BackgroundMusicConfig	JSON	Configurations for background music.	{"Volume":0.2}	No
BackgroundImageConfig	JSON	Configurations for the background image.	{"SubType":"Blur","Radius":0.5}	No
ProcessConfig		Remix processing configurations.	Parameter example.	No
FECanvas	JSON	Canvas configurations for frontend page previews.	{"Width": 1080,"Height": 1920}	No
ProduceConfig	JSON	Standard video editing and composition configurations. Field details: EditingProduceConfig.	{"AutoRegisterInputVodMedia":true,"OutputWebmTransparentChannel":true,"CoverConfig":{"StartTime":3.3},"AudioChannelCopy":"left","PipelineId":"*d54a97cff4108b555b01166d4b*","MaxBitrate":5000,"KeepOriginMaxBitrate":false,"KeepOriginVideoMaxFps":false}	No

ProcessConfig parameters

Parameter	Type	Description	Example	Required
AllowVfxEffect	Boolean	Whether to add special effects.	true	No. Default value: false.
VfxEffectProbability	Float	Probability of applying special effects per clip. Range: 0.0 to 1.0. Up to two decimal places.	0.6	No. Default value: 0.5.
VfxFirstClipEffectList	List<String>	If VfxFirstClipEffectList is not empty, the effect for the first clip of the video is selected from this list. If VfxFirstClipEffectList is empty, the effect for the first clip is randomly selected from the following: "slightshow", "starfieldshinee", "starfieldshinee2", "starsparkle", "colorfulripples", and "starfield". For effect examples, see Special effect examples.	["slightshow","starfieldshinee"]	No
VfxNotFirstClipEffectList	List<String>	If VfxNotFirstClipEffectList is not empty, the effects for clips other than the first one are selected from this list. If VfxNotFirstClipEffectList is empty, the effects for clips other than the first one are selected from the following: "zoomslight", "zoom", "zoominout", and "slightshake". For effect examples, see Special effect examples.	["zoomslight","zoom"]	No
AllowTransition	Boolean	Whether to add transitions.	true	No. Default value: false.
TransitionDuration	Float	Transition duration in seconds. If greater than clip duration minus 1, the transition is skipped for that clip.	0.5	No. Default value: 0.5s.
TransitionList	List<String>	Custom transition list. When AllowTransition is true, a random transition from this list is used. Available transitions: Transition library. If empty, defaults to: "linearblur", "colordistance", "crosshatch", "dreamyzoom", and "doomscreentransition_up".	["directional", "linearblur"]	No
UseUniformTransition	Boolean	Whether to use the same transition throughout a single video.	true	No. Default value: true.
AllowFilter	Boolean	Whether to add custom filters.	false	No. Default value: false.
FilterList	List<String>	Custom filter list. When AllowFilter is true, a random filter from this list is used. Available filters: Filter effect examples. If empty, no filter is added.	["m1", "m2"]	No
AllowDuplicateMatch	Boolean	Whether matched clips can be reused.	false	No. Default value: false.
EnableClipDetection:	Boolean	Whether to perform shot detection on the source material. This can only be configured for the movie highlights scenario. It supports automatic shot division and identifies shot types. Opening/closing credits, advertisements, and black screens are not included in the final video.	true	No. Default value: true.
EnableTemporalOpt	Boolean	Whether to perform temporal optimization on the matching results. This can only be configured for the movie highlights scenario. We recommend setting this to true when the input shot information or narration script is mostly consistent with the content order of the source video.	true	No. Default value: false.
EnableSubtitleMatch	Boolean	Optimizes the match between the narration script and the video frames based on Optical Character Recognition (OCR) and Automatic Speech Recognition (ASR) results. Currently, this parameter is supported only in global narration mode. We recommend enabling it when the script is similar to the original subtitles. This parameter takes effect only when EnableTemporalOpt is set to true.	true	No. Default value: false.

EditingConfig parameter example

All parameters in EditingConfig are optional. The following code shows the default configurations.

{
  "MediaConfig": {
    "Volume": 0 // Mute the source video by default
  },
  "TitleConfig": {
    "Alignment": "TopCenter",
    "AdaptMode": "AutoWrap",
    "Font": "Alibaba PuHuiTi 2.0 95 ExtraBold",
    "SizeRequestType": "Nominal",
    "Y": 0.1, // Y-coordinate of the title for a vertical video
    "Y": 0.05, // Y-coordinate of the title for a horizontal video
    "Y": 0.08 // Y-coordinate of the title for a square video
  },
  "SpeechConfig": {
    "Volume": 1,  // Use the original volume for the narration audio by default
    "SpeechRate": 0,
    "Voice": null,
    "Style": null,
    "CustomizedVoice": null, // Voice clone voiceId. If this field is specified, Voice and Style become invalid.
    "AsrConfig": {
      "Alignment": "TopCenter",
      "AdaptMode": "AutoWrap",
      "Font": "Alibaba PuHuiTi 2.0 65 Medium",
      "SizeRequestType": "Nominal",
      "Spacing": -1,
      "Y": 0.8, // Y-coordinate of the captions for a vertical video
      "Y": 0.9, // Y-coordinate of the captions for a horizontal video
      "Y": 0.85 // Y-coordinate of the captions for a square video
    }
  },
  "SubHeadingConfig": {
    "1": {
      "Y": 0.3,
      "FontSize": 40
    },
    "3": {
      "Y": 0.5,
      "FontSize": 30
    }
  },
  "BackgroundMusicConfig": {
    "Volume": 0.2,   // Use 20% volume for the background music by default
    "Style": null
  },
  "ProcessConfig": {
    "AllowVfxEffect": false,	  // Whether to add special effects
    "AllowTransition": false,	  // Whether to add transitions
    "AllowDuplicateMatch": false, // In smart content matching mode, specifies whether matched clips can be reused
    "EnableClipDetection": true, // Whether to perform shot detection
    "EnableTemporalOpt": true // Whether to perform temporal optimization
  }
}

TemplateConfig parameters

TemplateConfig sets a template for one-click video generation. Parameters and examples: TemplateConfig parameters.

OutputConfig parameters

Note

OutputConfig specifies the output address, naming rules, dimensions, and number of videos to produce.

Parameter	Type	Description	Example	Required
MediaURL	String	The output video address. It must contain the {index} placeholder.	Rule: http://[your-bucket].oss-[your-region-id].aliyuncs.com/[your-file-path]/[your-file-name]_{index}.mp4 Example: http://example.oss-[your-region-id].aliyuncs.com/example/example_{index}.mp4	Required when GeneratePreviewOnly is false and the output video is sent to OSS.
StorageLocation	String	The storage address for the media asset file to be output to VOD.	Rule: [your-vod-bucket].oss-[your-region-id].aliyuncs.com Example: outin-**6c886b4549d481030f6e**.oss-[your-region-id].aliyuncs.com	Required when GeneratePreviewOnly is false and the output video is sent to VOD.
FileName	String	The output file name. It must contain the {index} placeholder.	Rule: [your-file-name]__{index}.mp4 Example: example_{index}.mp4	Required when GeneratePreviewOnly is false and the output video is sent to VOD.
GeneratePreviewOnly	Boolean	If GeneratePreviewOnly is set to true, the current task only generates a timeline for preview and does not perform actual composition. You do not need to specify the output video address. After the one-click video generation task is complete, call GetBatchMediaProducingJob to query the task result. The returned sub-task list will contain the editing project ID. Then, call GetEditingProject to get the preview timeline.	false	No. Default value: false.
Count	Integer	The number of output videos. Global narration mode: The maximum number cannot exceed the length of InputConfig.SpeechTextArray. Storyboard mode: Currently, only Count=1 is supported. Support for multiple output videos may be added later.	1	No. Default value: 1.
Width	Integer	The width of the output video in pixels.	1080	Yes
Height	Integer	The height of the output video in pixels.	1920	Yes
Video	JSONObject	Configurations for the output video stream, such as Crf and Codec.	{"Crf": 27}	No

Parameter example

{
 	"MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/[your-file-path]/[your-file-name]_{index}.mp4",
 	"Count": 1,
 	"Width": 1080,
 	"Height": 1920,
 	"Video": {"Crf": 27},
        "GeneratePreviewOnly":false
}

SDK call example

Prerequisites

IMS server-side SDK installed. Preparations.

Code example

The following example shows how to use the global narration mode.

Expand to view the code example

package com.example;

import com.alibaba.fastjson.JSONObject;
import com.aliyun.ice20201109.Client;
import com.aliyun.ice20201109.models.*;
import com.aliyun.teaopenapi.models.Config;

import java.util.*;

/**
 *  You need to add the following Maven dependencies:
 *   <dependency>
 *      <groupId>com.aliyun</groupId>
 *      <artifactId>ice20201109</artifactId>
 *      <version>2.3.0</version>
 *  </dependency>
 *  <dependency>
 *      <groupId>com.alibaba</groupId>
 *      <artifactId>fastjson</artifactId>
 *      <version>1.2.9</version>
 *  </dependency>
 */

public class SmartMixBatchEditingService {

    static final String regionId = "[your-region-id]"; // Smart content matching is supported in cn-shanghai, cn-beijing, and cn-hangzhou.
    static final String bucket = "[your-bucket]";
    private Client iceClient;

    public static void main(String[] args) throws Exception {
        SmartMixBatchEditingService smartMixBatchEditingService = new SmartMixBatchEditingService();
        smartMixBatchEditingService.initClient();
        smartMixBatchEditingService.runExample();
    }

    public void initClient() throws Exception {
        // An Alibaba Cloud account AccessKey has full access to all APIs. We recommend that you use a RAM user for API calls and routine O&M.
        // This example shows how to store the AccessKey ID and AccessKey secret in environment variables. For configuration methods, see https://www.alibabacloud.com/help/en/sdk/developer-reference/v2-manage-access-credentials
        com.aliyun.credentials.Client credentialClient = new com.aliyun.credentials.Client();

        Config config = new Config();
        config.setCredential(credentialClient);

        // To hard-code the AccessKey ID and AccessKey secret, use the following code. However, we strongly recommend that you do not store your AccessKey ID and AccessKey secret in your project code. This can lead to an AccessKey leak and compromise the security of all resources in your account.
        // config.accessKeyId = <The AccessKey ID you created in Step 2>;
        // config.accessKeySecret = <The AccessKey secret you created in Step 2>;
        config.endpoint = "ice." + regionId + ".aliyuncs.com";
        config.regionId = regionId;
        iceClient = new Client(config);
    }

    public void runExample() throws Exception {

        // Video assets
        List<String> mediaArray = Arrays.asList(
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-1.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-2.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-3.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-4.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-5.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-6.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-7.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-8.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-9.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-10.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-11.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-12.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-13.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-14.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-15.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-16.mp4"
        );

        // Narration script
        String speechText = "In the vast, boundless blue depths, a vibrant picture is slowly unfolding. In the clear, azure water, coral reefs are like an underwater forest, colorful and magnificent. They are an important cornerstone of marine biodiversity, providing shelter for countless small fish, shellfish, and algae. Lazy and adorable sea lions bask in the warm sunlight on the rocks, enjoying the sun in harmony with nature. Dolphins swim together in the blue water, chasing and playing with each other, showing the agility and wisdom endowed by nature. However, the beautiful marine environment is facing unprecedented challenges. Various types of garbage pollution are like scars on the ocean's skin, ruthlessly eroding the once pristine ecosystem. Many species face food shortages, increased diseases, and even population decline. We must act to protect this beautiful and fragile blue home, so that the future ocean can once again restore its purity and vitality.";

        // Video title
        String title = "Protecting Our Blue Home";
        // Scenario information
        JSONObject sceneInfo = new JSONObject();
        sceneInfo.put("Scene","MovieHighlights");
        // Face information
        JSONObject faceInfo = new JSONObject();
        sceneInfo.put("FaceInfo",faceInfo);
        JSONArray imageInfoList =new JSONArray();
        JSONObject imageInfo1=new JSONObject();
        imageInfo1.put("Name","Character A");
        imageInfo1.put("ImageURL", "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080");
        imageInfoList.add(imageInfo1);
        faceInfo.put("ImageInfoList",imageInfoList);

        JSONObject inputConfig = new JSONObject();
        inputConfig.put("MediaArray", mediaArray);
        inputConfig.put("SpeechText", speechText);
        inputConfig.put("Title", title);
        inputConfig.put("SceneInfo", sceneInfo);

        // Number of videos to produce
        int produceCount = 4;

        // Width and height of the output video. A vertical video is generated.
        int outputWidth = 1080;
        int outputHeight = 1920;

        //// Width and height of the output video. A horizontal video is generated.
        //int outputWidth = 1920;
        //int outputHeight = 1080;

        // OSS address of the output video. It must contain the {index} placeholder.
        String mediaUrl = "http://" + bucket + ".oss-" + regionId + ".aliyuncs.com/smart_mix/output_{index}.mp4";

        JSONObject outputConfig = new JSONObject();
        outputConfig.put("MediaURL", mediaUrl);
        outputConfig.put("Count", produceCount);
        outputConfig.put("Width", outputWidth);
        outputConfig.put("Height", outputHeight);

        // Submit the one-click video generation task
        SubmitBatchMediaProducingJobRequest request = new SubmitBatchMediaProducingJobRequest();
        request.setInputConfig(inputConfig.toJSONString());
        request.setOutputConfig(outputConfig.toJSONString());

        SubmitBatchMediaProducingJobResponse response = iceClient.submitBatchMediaProducingJob(request);
        String jobId = response.getBody().getJobId();
        System.out.println("Start smart mix batch job, batchJobId: " + jobId);

        // Poll the task status until it is complete
        System.out.println("Waiting for job to finish...");
        int maxTry = 3000;
        int i = 0;
        while (i < maxTry) {
            Thread.sleep(3000);
            i++;
            GetBatchMediaProducingJobRequest getRequest = new GetBatchMediaProducingJobRequest();
            getRequest.setJobId(jobId);
            GetBatchMediaProducingJobResponse getResponse = iceClient.getBatchMediaProducingJob(getRequest);
            String status = getResponse.getBody().getEditingBatchJob().getStatus();
            System.out.println("BatchJobId: " + jobId + ", status:" + status);

            if ("Failed".equals(status)) {
                System.out.println("Batch job failed. JobInfo: " + JSONObject.toJSONString(getResponse.getBody().getEditingBatchJob()));
                throw new Exception("Produce failed. BatchJobId: " + jobId);
            }

            if ("Finished".equals(status)) {
                System.out.println("Batch job finished. JobInfo: " + JSONObject.toJSONString(getResponse.getBody().getEditingBatchJob()));
                break;
            }
        }
    }
}

Details of API request parameters

Expand to view InputConfig

{
  "MediaArray": [
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-1.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-2.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-3.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-4.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-5.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-6.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-7.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-8.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-9.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-10.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-11.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-12.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-13.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-14.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-15.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-16.mp4"
  ],
  "SpeechText": "In the vast, boundless blue depths, a vibrant picture is slowly unfolding. In the clear, azure water, coral reefs are like an underwater forest, colorful and magnificent. They are an important cornerstone of marine biodiversity, providing shelter for countless small fish, shellfish, and algae. Lazy and adorable sea lions bask in the warm sunlight on the rocks, enjoying the sun in harmony with nature. Dolphins swim together in the blue water, chasing and playing with each other, showing the agility and wisdom endowed by nature. However, the beautiful marine environment is facing unprecedented challenges. Various types of garbage pollution are like scars on the ocean's skin, ruthlessly eroding the once pristine ecosystem. Many species face food shortages, increased diseases, and even population decline. We must act to protect this beautiful and fragile blue home, so that the future ocean can once again restore its purity and vitality.",
  "Title":"Protecting Our Blue Home",
  "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://****.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        }
      ]
    }
  }
}

Expand to view OutputConfig

{
  "Count": 1,
  "Height": 1080,
  "Width": 1920,
  "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/script/output_{index}_w.mp4"
}

Advanced configurations

Advanced configurations: Remix logic and advanced configurations for batch one-click video generation.

FAQ

For answers to frequently asked questions about automated script-based video generation, see Movie highlights FAQ:

Intelligent Media Services:Movie highlights scenario

Usage notes

InputConfig parameters

SceneInfo parameters

ShotInfo parameters

ShotScript parameters

FaceInfo parameters

ImageInfo parameters

Parameter example: Global narration mode

Parameter example: Storyboard mode

EditingConfig parameters

ProcessConfig parameters

EditingConfig parameter example

TemplateConfig parameters

OutputConfig parameters

Parameter example

SDK call example

Prerequisites

Code example

Advanced configurations

FAQ

References