How IMS Smart Highlight Extraction Works & Core Concepts - Intelligent Media Services

This topic describes the data structures of the request parameters of SubmitHighlightExtractionJob and the response parameters of GetSmartHandleJob.

Important

In this API, the region specified in the OSS URL of all media assets must be the same as the OpenAPI service endpoint.
Supported regions: China (Shanghai), China (Beijing), China (Hangzhou), China (Shenzhen), US (Silicon Valley), Singapore. The action recognition capability (corresponding to Strategy.EnableActionRecog and Strategy.CustomActions) is only available in the China (Shanghai) region.
Videos must contain either subtitles or human voices. Materials lacking both are unsupported.

Relevant APIs

To submit a highlight extraction job, see SubmitHighlightExtractionJob. The data structures of key parameters (InputConfig and OutputConfig) are described in the following sections.
To obtain details about a highlight extraction job, see GetSmartHandleJob. The result parameters (AiResult) are described in the following sections.

InputConfig

Configure InputConfig to specify video materials and highlight extraction strategy.

Parameter	Type	Description	Required
MediaArray	List<Media>	The video materials. You can specify a list of media asset IDs or OSS URLs. The total video duration can be up to two hours, and the maximum number of videos is 30. For supported formats, see Video formats.	Yes
Strategy	Strategy	The highlight extraction strategy.	No

Strategy

Parameter	Type	Description	Required
Count	Integer	The number of highlight clips to extract from a single asset. Valid value: [1, 10]. Default value: 5.	No
ClipDuration	Float	Expected duration of each highlight clip in seconds. Valid values: [3,60]. Default value: 15. The actual duration of each highlight may vary slightly from this value.	No
EnableActionRecog	Boolean	Specifies whether to enable action recognition. Default value: false. Note Action recognition is supported only in the China (Shanghai) region.	No
CustomActions	List<String>	Custom action tags, which are mapped according to the input tag names. For example: ["Fight","Cry"]. The maximum number of tags is 50, with each no longer than 5 characters. Note Action recognition is supported only in the China (Shanghai) region.	No
HighlightDescription	String	A description of the highlight extraction strategy. This parameter takes effect only when ThemeConfig.ThemeType is set to SmoothHighlight. Example: Prioritize scenes with strong emotional expression, high contrast, concentrated plot conflict, and dramatic tension, such as the male lead expressing anger through actions, creating tension through identity/behavior contrast, focusing on core conflicts, and including bizarre dialogue or plot twists to enhance engagement and create buzz.	No
FaceInfo	FaceInfo	Set face information to help identify characters. Configure this parameter if you want to feature specific characters more prominently in the highlights.	No

FaceInfo

Parameter	Type	Description	Required
ImageInfoList	List<ImageInfo>	A list of character (face) photos. Max 200 images.	No

ImageInfo

Parameter	Type	Description	Example	Required
Name	String	The name of the character (face).	Daniel	Yes
ImageURL	String	The URL of the character's photo, which must be publicly accessible. Ensure that each image contains only one individual and that the face is clear, without significant occlusion or missing parts.	http://[your-cdn-domain]/[your-file-path]/face1.png	Yes (choose one: ImageURL or ImageId).
ImageId	String	The ID of the image asset.	**9d46c886b45481030f6e**	Yes (choose one: ImageURL or ImageId).

Media

Parameter	Type	Description	Required
MediaId	String	The ID of the media asset.	Yes (choose one: MediaId or MediaURL). MediaId takes priority if both are provided.
MediaURL	String	The OSS URL to the media file.

Parameter example

{
  "MediaArray": [
    {
      "MediaId": "1cb94770da*******75e6e6c5486302"
    }
  ],
  "Strategy": {
    "Count": 5,
    "ClipDuration": 15,
    "EnableActionRecog": true,
    "CustomActions":  ["Fight","Cry"],
    "HighlightDescription":"Prioritize scenes with strong emotional expression, high contrast, concentrated plot conflict, and dramatic tension, such as the male lead expressing anger through actions, creating tension through identity/behavior contrast, focusing on core conflicts, and including bizarre dialogue or plot twists to enhance engagement and create buzz.",
    "FaceInfo":{"ImageInfoList":[{"Name":"Daniel","ImageURL":"http://[your-cdn-domain]/[your-file-path]/face1.png"}]}
  }
}

OutputConfig

Configure OutputConfig to specify output settings, such as the storage location and naming of output videos.

Parameter	Type	Description	Required	Example
NeedExport	Boolean	Specifies whether to directly export clips. Valid values: true: returns the extracted highlight clips. false (default): only the time ranges of highlight clips are returned.	No	false
OutputMediaTarget	String	Required when NeedExport is set to true. Valid value: oss-object (default): saves the outputs as objects in OSS buckets.	No.	oss-object
Endpoint	String	The S3 protocol-compatible endpoint. For OSS, the region must match the service region. Defaults to the OSS endpoint in the same region.	No	https://oss-cn-shanghai.aliyuncs.com
Bucket	String	Required when NeedExport is set to true. Specify your OSS bucket, which is S3 protocol-compatible.	No	your bucket
ObjectKey	String	Required when NeedExport is set to true. The naming for the OSS objects. Supported placeholder: {index}: must be included in the object path.	No	dir/to/testOutput_{index}.mp4
ExportAsNewMedia	Boolean	Optional when NeedExport is set to true. Specifies whether to output as new media assets. Only supported when OutputMediaTarget is set to oss-object.	No. Default value: false.	false
Width	Integer	Optional when NeedExport is set to true. The output video width in pixels. If not specified, it will be the same as the source video	No	1280
Height	Integer	Optional when NeedExport is set to true. The output video height in pixels. If not specified, it will be the same as the source video	No	720
Video	JSONObject	Optional when NeedExport is set to true. The stream configuration of the output video, such as CRF and codec.	No	{ "Bitrate": 3000 }

Parameter example

 {
    "NeedExport": true,
    "OutputMediaTarget": "oss-object",
    "Endpoint": "https://oss-cn-shanghai.aliyuncs.com"
    "Bucket": "your-bucket",
    "ObjectKey": "dir/to/testOutput_{index}.mp4",
    "ExportAsNewMedia": false,
    "Width": 1280,
    "Height": 720,
    "Video": {
      "Bitrate": 3000
    }
  }

GetSmartHandleJob

Retrieve the results of a highlight extraction task using GetSmartHandleJob. The parameters of the AiResult are described below.

AiResult

{
  "HighlightResults": [
    {
      "Media": "MediaId1", //If URL was specified in InputConfig, then URL will be returned here.
      "TimeRanges": [
        {
          "In": 20,
          "Out": 30,
          "Tags": ["Fight","Cry"], // Detected action tags.
          "OutputURL": "http://your bucket.oss-cn-shanghai.aliyuncs.com/output_0.mp4", // Only returned when needExport is set to true.
          "MediaId": "MediaId11", // Only returned when ExportAsNewMedia is set to true.
        }
      ]
    },
    {
      "Media": "MediaId2", //If URL was specified in InputConfig, then URL will be returned here.
      "TimeRanges": [
        {
          "In": 2,
          "Out": 10,
          "Tags": ["Run","Shout"],
          "OutputURL": "http://your bucket.oss-cn-******.aliyuncs.com/output_1.mp4" // Only returned when needExport is set to true.
        },
        {
          "In": 40,
          "Out": 50,
          "OutputURL": "http://your bucket.oss-cn-******.aliyuncs.com/output_2.mp4" // Only returned when needExport is set to true.
        }
      ]
    }
  ]
}