Smart content matching - Movie highlights scenario parameter description - Intelligent Media Services

This topic describes the composition parameters, advanced configurations, and software development kit (SDK) call examples for the movie highlights scenario.

Important

Automated script-based video generation and smart content matching video generation use the same API to submit tasks. For more information about how to use parameters to distinguish between them, see Parameter differences.
Note: For this API, the region specified in the Object Storage Service (OSS) URL of all media assets must match the region of the OpenAPI endpoint.
Supported regions: China (Shanghai), China (Beijing), China (Hangzhou), China (Shenzhen), US (West), and Singapore.
In the examples, replace placeholders such as [your-bucket], [your-region-id], [your-file-name], [your-file-path], and media asset IDs (for example, "****9d46c8b4548681030f6e****") with your actual values.

Note

To better understand this topic, read User guide for one-click smart video generation to learn about the concepts and procedures for the movie highlights scenario.
The movie highlights scenario supports two video generation modes. This topic describes the following modes in detail:
- Global narration mode
- Storyboard mode

Usage notes

For information about the API used to produce videos in batches from multiple video, audio, and image assets, see SubmitBatchMediaProducingJob - Produce multiple smart videos at a time. For more information about the key API parameters, see the descriptions of the InputConfig, EditingConfig, and OutputConfig parameters later in this topic.
To retrieve detailed information about a batch video generation job, see GetBatchMediaProducingJob - Get information about a batch smart video generation job.

InputConfig parameters

Use InputConfig to specify parameters for basic assets such as video clips, narration, background music, and stickers.

Parameter	Type	Description	Example	Required	Supported modes
MediaArray	List<String>	Smart content matching mode. You can enter a list of media asset IDs or OSS URLs. The total duration of the videos cannot exceed two hours. For more information about supported video formats, see Video formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	Yes	Fully supported
TitleArray	List<String>	An array of titles. A random title is selected for each video generation. You can specify up to 50 titles. Each title can contain up to 50 characters.	["Title 1","Title 2"]	No	Fully supported
SubHeadingArray	List<SubHeading>	Subtitle settings.	[{"Level":1,"TitleArray":["Level-1 Subtitle 1","Level-1 Subtitle 2"]},{"Level":3,"TitleArray":["Level-3 Subtitle"]}]	No	All modes
SpeechTextArray	List<String>	An array of narration scripts. A random script is selected for each video generation. You can specify up to 50 scripts. Each script can contain up to 1,000 characters. You can use Speech Synthesis Markup Language (SSML) to control speech synthesis. For more information, see SSML. Important Currently, only <break>, <s>, and <phoneme> are supported. For CosyVoice timbres, only <break>, <s>, and <sub> are supported.	["Narration content 1","Narration content 2"]	No	Global narration mode
SceneInfo	SceneInfo	Scenario information. Used for scenario-related parameters.	For more information, see Parameter example: Global narration mode and Parameter example: Storyboard mode.	Yes	Global narration mode Storyboard mode
StickerArray	List<Sticker>	An array of stickers. A random sticker is selected for each video generation. You can specify up to 50 stickers. You can use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 stickers and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, stickers are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. For more information about supported image formats, see Image formats.	[{"MediaId":"**9d46c8b4548681030f6e**","X":10,"Y":100,"Width":300,"Height":300,"Opacity":0.6}]	No	All modes
BackgroundMusicArray	List<String>	An array of background music tracks. A random track is selected for each video generation. You can specify up to 50 tracks. You can use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 background music tracks and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, tracks are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. For more information about supported audio formats, see Audio formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	No	Fully supported
BackgroundImageArray	List<String>	An array of background images. A random image is selected for each video generation. You can specify up to 50 images. You can use media asset IDs or OSS URLs. Randomization rule: Assume you provide 10 background images and set the number of videos to produce to 20. First, a random number from 1 to 10 is generated, for example, 3. Then, images are selected in the order of 3, 4, 5, 6, 7, 8, 9, 10, 1, 2, 3, 4, 5, 6, 7, and so on. For more information about supported image formats, see Image formats.	["**b4549d46c88681030f6e","549d46c88b4681030f6e**"]	No	All modes

SceneInfo parameters

Parameter	Type	Description	Required	Supported modes
Scene	String	The matching scenario type. For the movie highlights scenario, set this parameter to the static field "MovieHighlights".	Yes	Global narration mode Storyboard mode
ShotInfo	ShotInfo	Configure the storyboard.	Yes	Storyboard mode
FaceInfo	FaceInfo	Configure face information.	No	Fully supported

ShotInfo parameters

Note

This parameter applies only to the storyboard mode. You do not need to set this parameter for the global narration mode.

Parameter	Type	Description	Required
ShotScripts	List<ShotScript>	An array of shot scripts.	Yes

ShotScript parameters

Note

This parameter applies only to the storyboard mode. You do not need to set this parameter for the global narration mode.
In the storyboard mode of the movie highlights scenario, two shot modes are available: Text description mode and Manual parsing mode. You must select one of these modes when you set the parameters.

Parameter	Type	Description	Example	Required	Shot mode
ScriptText	String	The script text for a single shot. It describes the content of the shot. Summarize the shot content in one or two sentences.	In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.	No	Text description mode
SpeechText	String	The narration script for a single shot. The script can contain up to 100 characters. You can use SSML to control speech synthesis. For more information, see SSML. Important Currently, only <break>, <s>, and <phoneme> are supported.	The late-night car is quiet, but some people light up the whole car with their songs.	No	All modes
Duration	Float	The duration of the shot in seconds. The value must be 1 or greater. This parameter takes effect only when there is no narration. If narration exists, the shot duration is automatically calculated based on the narration duration.	5	No	All modes
Descriptions	List<String>	Detailed descriptions for a single shot. Summarize the shot content in one or two sentences.	["Lin Xia sits in a corner of the subway car, quietly recording with her phone", "An old woman holds her grandson's hand, humming a song softly"]	No	Manual parsing mode
Characters	List<String>	The names of the characters (faces) in a single shot. Note: The character names must be the same as the ImageInfo.Name in FaceInfo.ImageInfoList.	["Lin Xia", "Old woman", "Grandson"]	No
Settings	List<String>	The scene description for a single shot.	["Late-night subway car"]	No
Volume	Float	The volume of the input video. If you set the Volume here, the volume of the video in the current shot will align with this value. The EditingConfig.MediaConfig.Volume parameter will no longer apply to this group. Value range: [0, 10.0]. Up to two decimal places are supported.	0.5	No	Fully supported

FaceInfo parameters

Parameter	Type	Description	Required
ImageInfoList	List<ImageInfo>	A list of character (face) photos. The list can contain up to 200 photos.	No

ImageInfo parameters

Parameter	Type	Description	Example	Required
Name	String	Character (face) name.	Daniel	Yes
ImageURL	String	The storage address of the character (face) photo. It must be a URL accessible over the Internet. Ensure that the face image contains only one individual and the face is clear, without significant obstructions or missing parts.	http://[your-cdn-domain]/[your-file-path]/face1.png	Yes. You must specify one of the two parameters.
ImageId	String	Image media asset ID.	**9d46c886b45481030f6e**	Yes. You must specify one of the two parameters.

Parameter example: Global narration mode

{
  "MediaArray": [
    "****9d46c886b45481030f6e****",
    "****c886810b4549d4630f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.mp4",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test2.png"
  ],
    "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character B",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/622762d0f703918ffbedc1125b3d269759eec42e?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character C",
          "ImageId": "****b681034549d46c880f6e****"
        }
      ]
    }
  },
  "TitleArray": [
    "Hema Fresh in Huilongguan is now open!",
    "Hema Fresh is now open!"
  ],
  "SubHeadingArray": [
    {
      "Level": 1,
      "TitleArray": ["Subtitle 1", "Subtitle 2"]
    },
    {
      "Level": 3,
      "TitleArray": ["Level-3 Subtitle"]
    }
  ],
  "SpeechTextArray": [
    "A new Hema Fresh just opened in the nearby mall. It's the grand opening today, so I rushed over to join the fun. This Hema isn't very big, but the mall is packed. Snacks and drinks are quite cheap, and the lines are incredibly long. Come and check it out!",
    "A new Hema Fresh just opened in the nearby mall. It's the grand opening today, so I rushed over to join the fun."
  ],
  "Sticker": {
    "MediaId": "****b681034549d46c880f6e****",
    "X": 10,
    "Y": 100,
    "Width": 300,
    "Height": 300
  },
  "StickerArray": [
    {
      "MediaId": "****9d46c8b4548681030f6e****",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300,
      "Opacity": 0.6
    },
    {
      "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test3.png",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300
    }
  ],
  "BackgroundMusicArray": [
    "****b4549d46c88681030f6e****",
    "****549d46c88b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test4.mp3"
  ],
  "BackgroundImageArray": [
    "****6c886b4549d481030f6e****",
    "****9d46c8548b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.png"
  ]
}

Parameter example: Storyboard mode

{
  "MediaArray": [
    "****9d46c886b45481030f6e****",
    "****c886810b4549d4630f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.mp4",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test2.png"
  ],
   "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "ShotInfo": {
      "ShotScripts": [
        // Start of text description mode example. For each shot, choose either manual parsing mode or text description mode.
        {
          "ScriptText": "In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.",
          "SpeechText": "The late-night car is quiet, but some people light up the whole car with their songs."
        },
        {
           "ScriptText": "The old woman's white hair glows under the car lights. She strokes her grandson's head and whispers, 'Do you remember this song?'",
           "SpeechText": "Her world may be blurry, but in front of him, she is still the most gentle mother."
        },
        {
           "ScriptText": "Lin Xia stops filming and looks at the billboards flashing by outside the window, her expression softening.",
           "SpeechText": "I thought I was recording someone else's story, but I was quietly healed."
        },
        {
           "ScriptText": "At home, Lin Xia takes out her camera and writes her first diary entry: 'Today, I saw what happiness looks like.'",
           "SpeechText": "Sometimes, happiness isn't about big things, but the small moments we overlook."
        },
        {
           "ScriptText": "A few days later, a series of heartwarming photos on Lin Xia's social media resonates with netizens.",
           "SpeechText": "She starts telling stories with her lens, and every story holds a piece of love."
        },
        {
           "ScriptText": "At night, Lin Xia enters the subway station again, this time with a smile and a new goal.",
           "SpeechText": "This city is big, but there's always someone waiting for you, and someone worth discovering."
        }
        // End of text description mode example
        
        // Start of manual parsing mode example. For each shot, choose either manual parsing mode or text description mode.
        {
            "Descriptions": ["Lin Xia sits in a corner of the subway car, quietly recording with her phone", "An old woman holds her grandson's hand, humming a song softly"],
            "Characters": ["Lin Xia", "Old woman", "Grandson"],
            "Settings": ["Late-night subway car"],
            "SpeechText": "The late-night car is quiet, but some people light up the whole car with their songs."
        },
        {
            "Descriptions": ["The old woman's white hair glows under the car lights", "She strokes her grandson's head and speaks softly"],
            "Characters": ["Old woman", "Grandson"],
            "Settings": ["Subway car"],
            "SpeechText": "Her world may be blurry, but in front of him, she is still the most gentle mother."
        },
        {
            "Descriptions": ["Lin Xia stops filming", "Looks at the billboards flashing by outside the window, her expression softening"],
            "Characters": ["Lin Xia"],
            "Settings": ["Moving subway car"],
            "SpeechText": "I thought I was recording someone else's story, but I was quietly healed."
        },
        {
            "Descriptions": ["Lin Xia takes out her camera at home", "Writes her first diary entry: 'Today, I saw what happiness looks like'"],
            "Characters": ["Lin Xia"],
            "Settings": ["Modern apartment / Room"],
            "SpeechText": "Sometimes, happiness isn't about big things, but the small moments we overlook."
        },
        {
            "Descriptions": ["Heartwarming photos are posted on Lin Xia's social media page", "Netizens like and comment one after another"],
            "Characters": ["Lin Xia"],
            "Settings": ["Phone screen / Social media interface"],
            "SpeechText": "She starts telling stories with her lens, and every story holds a piece of love."
        },
        {
            "Descriptions": ["At night, Lin Xia enters the subway station", "She has a smile and a new goal"],
            "Characters": ["Lin Xia"],
            "Settings": ["City subway station"],
            "SpeechText": "This city is big, but there's always someone waiting for you, and someone worth discovering."
        }
        // End of manual parsing mode example

        // You can choose not to configure narration. Configure only the shot script and duration to play the original video sound.
       {
            "ScripText": "In the subway car, Lin Xia sits in a corner, quietly recording with her phone. An old woman is holding her grandson's hand, humming a song softly.",
            "Duration": 8.0, // Can be set when there is no narration script 
            "Volume": 1.0 // Set the volume of the original video asset
       }
      ]
    },
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character B",
          "ImageURL": "https://bkimg.cdn.bcebos.com/pic/622762d0f703918ffbedc1125b3d269759eec42e?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        },
        {
          "Name": "Character C",
          "ImageId": "****b681034549d46c880f6e****"
        }
      ]
    }
  },
  "TitleArray": [
    "Hema Fresh in Huilongguan is now open!",
    "Hema Fresh is now open!"
  ],
  "StickerArray": [
    {
      "MediaId": "****9d46c8b4548681030f6e****",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300
    },
    {
      "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test3.png",
      "X": 10,
      "Y": 100,
      "Width": 300,
      "Height": 300,
      "Opacity": 0.6
    }
  ],
  "BackgroundMusicArray": [
    "****b4549d46c88681030f6e****",
    "****549d46c88b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test4.mp3"
  ],
  "BackgroundImageArray": [
    "****6c886b4549d481030f6e****",
    "****9d46c8548b4681030f6e****",
    "http://[your-bucket].oss-[your-region-id].aliyuncs.com/test1.png"
  ]
}

EditingConfig parameters

Note

Use EditingConfig to specify the volume, position, and other composition parameters for the output video assets. If you do not have special requirements, you can use the default configurations and leave this field empty.
The parameter descriptions are the same for the global narration and storyboard modes in the movie highlights scenario.

Parameter	Type	Description	Example	Required
MediaConfig	JSON	Configurations for input video assets.	For more information, see Parameter example.	No
TitleConfig	JSON	Configurations for the title. You can configure caption parameters. For more information about the fields, see Banner text.	For more information, see Parameter example.	No
SubHeadingConfig	JSON	Configurations for multi-level subtitles. You can set caption parameters. JSON field description: key: Level value: Banner text	For more information, see Parameter example.	No
SpeechConfig	JSON	Configurations for the narration script.	For more information, see Parameter example.	No
BackgroundMusicConfig	JSON	Configurations for background music.	{"Volume":0.2}	No
BackgroundImageConfig	JSON	Configurations for the background image.	{"SubType":"Blur","Radius":0.5}	No
ProcessConfig		Remix processing configurations.	For more information, see Parameter example.	No
FECanvas	JSON	Canvas configurations for frontend page previews.	{"Width": 1080,"Height": 1920}	No
ProduceConfig	JSON	Standard video editing and composition configurations. For more information about the fields, see EditingProduceConfig.	{"AutoRegisterInputVodMedia":true,"OutputWebmTransparentChannel":true,"CoverConfig":{"StartTime":3.3},"AudioChannelCopy":"left","PipelineId":"*d54a97cff4108b555b01166d4b*","MaxBitrate":5000,"KeepOriginMaxBitrate":false,"KeepOriginVideoMaxFps":false}	No

ProcessConfig parameters

Parameter	Type	Description	Example	Required
AllowVfxEffect	Boolean	Specifies whether to add special effects.	true	No. Default value: false.
VfxEffectProbability	Float	The probability of applying special effects to each video clip. Value range: 0.0 to 1.0. Up to two decimal places are supported.	0.6	No. Default value: 0.5.
VfxFirstClipEffectList	List<String>	If VfxFirstClipEffectList is not empty, the effect for the first clip of the video is selected from this list. If VfxFirstClipEffectList is empty, the effect for the first clip is randomly selected from the following: "slightshow", "starfieldshinee", "starfieldshinee2", "starsparkle", "colorfulripples", and "starfield". For effect examples, see Special effect examples.	["slightshow","starfieldshinee"]	No
VfxNotFirstClipEffectList	List<String>	If VfxNotFirstClipEffectList is not empty, the effects for clips other than the first one are selected from this list. If VfxNotFirstClipEffectList is empty, the effects for clips other than the first one are selected from the following: "zoomslight", "zoom", "zoominout", and "slightshake". For effect examples, see Special effect examples.	["zoomslight","zoom"]	No
AllowTransition	Boolean	Specifies whether to add transitions.	true	No. Default value: false.
TransitionDuration	Float	The transition duration in seconds. If the transition duration is greater than the clip duration minus 1, the transition for that clip does not take effect.	0.5	No. Default value: 0.5s.
TransitionList	List<String>	A list of custom transitions. If AllowTransition is set to true, a random transition from the list is selected for composition. For the available transitions, see Transition library. If this parameter is empty, a random effect is selected from the following: "linearblur", "colordistance", "crosshatch", "dreamyzoom", and "doomscreentransition_up".	["directional", "linearblur"]	No
UseUniformTransition	Boolean	Specifies whether to use the same transition throughout a single video.	true	No. Default value: true.
AllowFilter	Boolean	Specifies whether to add custom filters.	false	No. Default value: false.
FilterList	List<String>	A list of custom filters. If AllowFilter is set to true, a random filter from the list is selected for composition. For available filters, see Filter effect examples. If this parameter is empty, no filter is added.	["m1", "m2"]	No
AllowDuplicateMatch	Boolean	Specifies whether matched clips can be reused.	false	No. Default value: false.
EnableClipDetection:	Boolean	Specifies whether to perform shot detection on the source material. This can only be configured for the movie highlights scenario. It supports automatic shot division and identifies shot types. Opening/closing credits, advertisements, and black screens are not included in the final video.	true	No. Default value: true.
EnableTemporalOpt	Boolean	Specifies whether to perform temporal optimization on the matching results. This can only be configured for the movie highlights scenario. We recommend setting this to true when the input shot information or narration script is mostly consistent with the content order of the source video.	true	No. Default value: false.
EnableSubtitleMatch	Boolean	Optimizes the match between the narration script and the video frames based on Optical Character Recognition (OCR) and Automatic Speech Recognition (ASR) results. Currently, this parameter is supported only in global narration mode. We recommend enabling it when the script is similar to the original subtitles. This parameter takes effect only when EnableTemporalOpt is set to true.	true	No. Default value: false.

EditingConfig parameter example

All parameters in EditingConfig are optional. The following code shows the default configurations.

{
  "MediaConfig": {
    "Volume": 0 // Mute the source video by default
  },
  "TitleConfig": {
    "Alignment": "TopCenter",
    "AdaptMode": "AutoWrap",
    "Font": "Alibaba PuHuiTi 2.0 95 ExtraBold",
    "SizeRequestType": "Nominal",
    "Y": 0.1, // Y-coordinate of the title for a vertical video
    "Y": 0.05, // Y-coordinate of the title for a horizontal video
    "Y": 0.08 // Y-coordinate of the title for a square video
  },
  "SpeechConfig": {
    "Volume": 1,  // Use the original volume for the narration audio by default
    "SpeechRate": 0,
    "Voice": null,
    "Style": null,
    "CustomizedVoice": null, // Voice clone voiceId. If this field is specified, Voice and Style become invalid.
    "AsrConfig": {
      "Alignment": "TopCenter",
      "AdaptMode": "AutoWrap",
      "Font": "Alibaba PuHuiTi 2.0 65 Medium",
      "SizeRequestType": "Nominal",
      "Spacing": -1,
      "Y": 0.8, // Y-coordinate of the captions for a vertical video
      "Y": 0.9, // Y-coordinate of the captions for a horizontal video
      "Y": 0.85 // Y-coordinate of the captions for a square video
    }
  },
  "SubHeadingConfig": {
    "1": {
      "Y": 0.3,
      "FontSize": 40
    },
    "3": {
      "Y": 0.5,
      "FontSize": 30
    }
  },
  "BackgroundMusicConfig": {
    "Volume": 0.2,   // Use 20% volume for the background music by default
    "Style": null
  },
  "ProcessConfig": {
    "AllowVfxEffect": false,	  // Specifies whether to add special effects
    "AllowTransition": false,	  // Specifies whether to add transitions
    "AllowDuplicateMatch": false, // In smart content matching mode, specifies whether matched clips can be reused
    "EnableClipDetection": true, // Specifies whether to perform shot detection
    "EnableTemporalOpt": true // Specifies whether to perform temporal optimization
  }
}

TemplateConfig parameters

TemplateConfig is a common parameter used to set a template for one-click video generation. For detailed parameter descriptions and examples, see TemplateConfig parameters.

OutputConfig parameters

Note

Use OutputConfig to specify parameters such as the output address, naming rules, width, height, and the number of videos to produce.

Parameter	Type	Description	Example	Required
MediaURL	String	The output video address. It must contain the {index} placeholder.	Rule: http://[your-bucket].oss-[your-region-id].aliyuncs.com/[your-file-path]/[your-file-name]_{index}.mp4 Example: http://example.oss-[your-region-id].aliyuncs.com/example/example_{index}.mp4	Required when GeneratePreviewOnly is false and the output video is sent to OSS.
StorageLocation	String	The storage address for the media asset file to be output to VOD.	Rule: [your-vod-bucket].oss-[your-region-id].aliyuncs.com Example: outin-**6c886b4549d481030f6e**.oss-[your-region-id].aliyuncs.com	Required when GeneratePreviewOnly is false and the output video is sent to VOD.
FileName	String	The output file name. It must contain the {index} placeholder.	Rule: [your-file-name]__{index}.mp4 Example: example_{index}.mp4	Required when GeneratePreviewOnly is false and the output video is sent to VOD.
GeneratePreviewOnly	Boolean	If GeneratePreviewOnly is set to true, the current task only generates a timeline for preview and does not perform actual composition. You do not need to specify the output video address. After the one-click video generation task is complete, call GetBatchMediaProducingJob to query the task result. The returned sub-task list will contain the editing project ID. Then, call GetEditingProject to get the preview timeline.	false	No. Default value: false.
Count	Integer	The number of output videos. Global narration mode: The maximum number cannot exceed the length of InputConfig.SpeechTextArray. Storyboard mode: Currently, only Count=1 is supported. Support for multiple output videos may be added later.	1	No. Default value: 1.
Width	Integer	The width of the output video in pixels.	1080	Yes
Height	Integer	The height of the output video in pixels.	1920	Yes
Video	JSONObject	Configurations for the output video stream, such as Crf and Codec.	{"Crf": 27}	No

Parameter example

{
 	"MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/[your-file-path]/[your-file-name]_{index}.mp4",
 	"Count": 1,
 	"Width": 1080,
 	"Height": 1920,
 	"Video": {"Crf": 27},
        "GeneratePreviewOnly":false
}

SDK call example

Prerequisites

You have installed the Intelligent Media Service (IMS) server-side SDK. For more information, see Preparations.

Code example

The following example shows how to use the global narration mode.

Expand to view the code example

package com.example;

import com.alibaba.fastjson.JSONObject;
import com.aliyun.ice20201109.Client;
import com.aliyun.ice20201109.models.*;
import com.aliyun.teaopenapi.models.Config;

import java.util.*;

/**
 *  You need to add the following Maven dependencies:
 *   <dependency>
 *      <groupId>com.aliyun</groupId>
 *      <artifactId>ice20201109</artifactId>
 *      <version>2.3.0</version>
 *  </dependency>
 *  <dependency>
 *      <groupId>com.alibaba</groupId>
 *      <artifactId>fastjson</artifactId>
 *      <version>1.2.9</version>
 *  </dependency>
 */

public class SmartMixBatchEditingService {

    static final String regionId = "[your-region-id]"; // Smart content matching is supported in cn-shanghai, cn-beijing, and cn-hangzhou.
    static final String bucket = "[your-bucket]";
    private Client iceClient;

    public static void main(String[] args) throws Exception {
        SmartMixBatchEditingService smartMixBatchEditingService = new SmartMixBatchEditingService();
        smartMixBatchEditingService.initClient();
        smartMixBatchEditingService.runExample();
    }

    public void initClient() throws Exception {
        // An Alibaba Cloud account AccessKey has full access to all APIs. We recommend that you use a RAM user for API calls and routine O&M.
        // This example shows how to store the AccessKey ID and AccessKey secret in environment variables. For configuration methods, see https://www.alibabacloud.com/help/en/sdk/developer-reference/v2-manage-access-credentials
        com.aliyun.credentials.Client credentialClient = new com.aliyun.credentials.Client();

        Config config = new Config();
        config.setCredential(credentialClient);

        // To hard-code the AccessKey ID and AccessKey secret, use the following code. However, we strongly recommend that you do not store your AccessKey ID and AccessKey secret in your project code. This can lead to an AccessKey leak and compromise the security of all resources in your account.
        // config.accessKeyId = <The AccessKey ID you created in Step 2>;
        // config.accessKeySecret = <The AccessKey secret you created in Step 2>;
        config.endpoint = "ice." + regionId + ".aliyuncs.com";
        config.regionId = regionId;
        iceClient = new Client(config);
    }

    public void runExample() throws Exception {

        // Video assets
        List<String> mediaArray = Arrays.asList(
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-1.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-2.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-3.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-4.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-5.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-6.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-7.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-8.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-9.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-10.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-11.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-12.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-13.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-14.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-15.mp4",
            "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-16.mp4"
        );

        // Narration script
        String speechText = "In the vast, boundless blue depths, a vibrant picture is slowly unfolding. In the clear, azure water, coral reefs are like an underwater forest, colorful and magnificent. They are an important cornerstone of marine biodiversity, providing shelter for countless small fish, shellfish, and algae. Lazy and adorable sea lions bask in the warm sunlight on the rocks, enjoying the sun in harmony with nature. Dolphins swim together in the blue water, chasing and playing with each other, showing the agility and wisdom endowed by nature. However, the beautiful marine environment is facing unprecedented challenges. Various types of garbage pollution are like scars on the ocean's skin, ruthlessly eroding the once pristine ecosystem. Many species face food shortages, increased diseases, and even population decline. We must act to protect this beautiful and fragile blue home, so that the future ocean can once again restore its purity and vitality.";

        // Video title
        String title = "Protecting Our Blue Home";
        // Scenario information
        JSONObject sceneInfo = new JSONObject();
        sceneInfo.put("Scene","MovieHighlights");
        // Face information
        JSONObject faceInfo = new JSONObject();
        sceneInfo.put("FaceInfo",faceInfo);
        JSONArray imageInfoList =new JSONArray();
        JSONObject imageInfo1=new JSONObject();
        imageInfo1.put("Name","Character A");
        imageInfo1.put("ImageURL", "https://bkimg.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080");
        imageInfoList.add(imageInfo1);
        faceInfo.put("ImageInfoList",imageInfoList);

        JSONObject inputConfig = new JSONObject();
        inputConfig.put("MediaArray", mediaArray);
        inputConfig.put("SpeechText", speechText);
        inputConfig.put("Title", title);
        inputConfig.put("SceneInfo", sceneInfo);

        // Number of videos to produce
        int produceCount = 4;

        // Width and height of the output video. A vertical video is generated.
        int outputWidth = 1080;
        int outputHeight = 1920;

        //// Width and height of the output video. A horizontal video is generated.
        //int outputWidth = 1920;
        //int outputHeight = 1080;

        // OSS address of the output video. It must contain the {index} placeholder.
        String mediaUrl = "http://" + bucket + ".oss-" + regionId + ".aliyuncs.com/smart_mix/output_{index}.mp4";

        JSONObject outputConfig = new JSONObject();
        outputConfig.put("MediaURL", mediaUrl);
        outputConfig.put("Count", produceCount);
        outputConfig.put("Width", outputWidth);
        outputConfig.put("Height", outputHeight);

        // Submit the one-click video generation task
        SubmitBatchMediaProducingJobRequest request = new SubmitBatchMediaProducingJobRequest();
        request.setInputConfig(inputConfig.toJSONString());
        request.setOutputConfig(outputConfig.toJSONString());

        SubmitBatchMediaProducingJobResponse response = iceClient.submitBatchMediaProducingJob(request);
        String jobId = response.getBody().getJobId();
        System.out.println("Start smart mix batch job, batchJobId: " + jobId);

        // Poll the task status until it is complete
        System.out.println("Waiting for job to finish...");
        int maxTry = 3000;
        int i = 0;
        while (i < maxTry) {
            Thread.sleep(3000);
            i++;
            GetBatchMediaProducingJobRequest getRequest = new GetBatchMediaProducingJobRequest();
            getRequest.setJobId(jobId);
            GetBatchMediaProducingJobResponse getResponse = iceClient.getBatchMediaProducingJob(getRequest);
            String status = getResponse.getBody().getEditingBatchJob().getStatus();
            System.out.println("BatchJobId: " + jobId + ", status:" + status);

            if ("Failed".equals(status)) {
                System.out.println("Batch job failed. JobInfo: " + JSONObject.toJSONString(getResponse.getBody().getEditingBatchJob()));
                throw new Exception("Produce failed. BatchJobId: " + jobId);
            }

            if ("Finished".equals(status)) {
                System.out.println("Batch job finished. JobInfo: " + JSONObject.toJSONString(getResponse.getBody().getEditingBatchJob()));
                break;
            }
        }
    }
}

Details of API request parameters

Expand to view InputConfig

{
  "MediaArray": [
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-1.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-2.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-3.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-4.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-5.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-6.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-7.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-8.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-9.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-10.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-11.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-12.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-13.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-14.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-15.mp4",
    "http://ice-document-materials.oss-cn-shanghai.aliyuncs.com/test_media/sea/sea-16.mp4"
  ],
  "SpeechText": "In the vast, boundless blue depths, a vibrant picture is slowly unfolding. In the clear, azure water, coral reefs are like an underwater forest, colorful and magnificent. They are an important cornerstone of marine biodiversity, providing shelter for countless small fish, shellfish, and algae. Lazy and adorable sea lions bask in the warm sunlight on the rocks, enjoying the sun in harmony with nature. Dolphins swim together in the blue water, chasing and playing with each other, showing the agility and wisdom endowed by nature. However, the beautiful marine environment is facing unprecedented challenges. Various types of garbage pollution are like scars on the ocean's skin, ruthlessly eroding the once pristine ecosystem. Many species face food shortages, increased diseases, and even population decline. We must act to protect this beautiful and fragile blue home, so that the future ocean can once again restore its purity and vitality.",
  "Title":"Protecting Our Blue Home",
  "SceneInfo": {
    "Scene": "MovieHighlights", // MovieHighlights for movie highlights matching
    "FaceInfo": {
      "ImageInfoList": [
        {
          "Name": "Character A",
          "ImageURL": "https://****.cdn.bcebos.com/pic/3853ad1bdd9f70558718bf38?x-bce-process=image/format,f_auto/watermark,image_d2F0ZXIvYmFpa2UyNzI,g_7,xp_5,yp_5,P_20/resize,m_lfit,limit_1,h_1080"
        }
      ]
    }
  }
}

Expand to view OutputConfig

{
  "Count": 1,
  "Height": 1080,
  "Width": 1920,
  "MediaURL": "http://[your-bucket].oss-[your-region-id].aliyuncs.com/script/output_{index}_w.mp4"
}

Advanced configurations

For more information about advanced configurations, see Remix logic and advanced configurations for batch one-click video generation.

FAQ

For answers to frequently asked questions about automated script-based video generation, see Movie highlights FAQ:

What is the difference between global narration mode and storyboard mode?
How do I correctly set face information?

Intelligent Media Services:Movie highlights scenario

Usage notes

InputConfig parameters

SceneInfo parameters

ShotInfo parameters

ShotScript parameters

FaceInfo parameters

ImageInfo parameters

Parameter example: Global narration mode

Parameter example: Storyboard mode

EditingConfig parameters

ProcessConfig parameters

EditingConfig parameter example

TemplateConfig parameters

OutputConfig parameters

Parameter example

SDK call example

Prerequisites

Code example

Advanced configurations

FAQ

References