Model Hub of Machine Learning Platform for AI (PAI) provides a wide range of intelligent video processing models. This topic describes the features, input formats, and output formats of these models and provides examples.

Background information

The following table describes the intelligent video processing models that are provided by Model Hub of PAI.
Model Description
General video classification model Uses the UCF101 dataset for model training based on the ResNet3D framework.
Video highlight generation model Generates a five-second clip of video highlights from a video.
Super-resolution model Uses super-resolution technologies to improve video resolution.

Go to Model Hub

To go to Model Hub, perform the following steps:
  1. Log on to the PAI console.
  2. In the left-side navigation pane, choose AI Computing Asset Management > Model Management.
  3. On the Model Management page, click the Model Hub tab.

General video classification model

  • Overview

    The general video classification model uses the UCF101 dataset for model training based on the ResNet3D framework. For more information, see Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?

  • Input format
    The input data must be in the JSON format. It contains the url and play_duration fields. The value of the url field is the URL of the input video. The value of the play_duration field is the first several microseconds of the input video to be processed for classification. If the play_duration field is not specified, the total length of the input video is processed. The following code provides an example of the input data:
    {
      "input": {
        "url": "URL of the input video",
        "play_duration": "Length of the input video clip to be processed"
      }
    }
  • Output format
    The output data is key-value pairs (KVPs) in the JSON format. The following table describes the fields in the output data.
    Field Description Dimension Type
    class The name of the category. [] STRING
  • Example
    The following code provides an example of the input data of the model:
    {"input" : {"url": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com/tmp/50005632219.mp4"}}
    PAI displays information that is similar to the following output:
    null

Video highlight generation model

  • Overview

    This model generates a five-second clip of video highlights from a video.

  • Input format
    The input data must be in the JSON format. It contains the url and play_duration fields. The value of the url field is the URL of the input video. The value of the play_duration field is the first several microseconds of the input video to be processed to generate the clip. If the play_duration field is not specified, the total length of the input video is processed. The following code provides an example of the input data:
    {
      "input": {
        "url": "URL of the input video",
        "play_duration": "Length of the input video clip to be processed"
      }
    }
  • Output format
    The output data is KVPs in the JSON format. The following table describes the fields in the output data.
    Field Description Dimension Type
    oss_path The Object Storage Service (OSS) path of the generated clip. [] STRING
    result The result field. [] STRING
    success Indicates whether the clip is generated. Valid values:
    • true: The clip is generated.
    • false: The clip failed to be generated.
    [] BOOL
  • Example
    The following code provides an example of the input data of the model:
    {"input" : {"url": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com/tmp/50005632219.mp4"}}
    PAI displays information that is similar to the following output:
    {
       "oss_path": "oss://experience-ai/video_5s/50005632219.mp4",
       "result": "",
       "success": true
    }

Super-resolution model

  • Overview

    The super-resolution model uses super-resolution technologies to improve video resolution.

  • Input format
    The input data is must be in the JSON format. The following table describes the fields in the input data.
    Field Required Description Type
    url Yes The URL of the input video. STRING
    play_duration No The first several microseconds of the input video from which a clip with higher resolution to be generated. If the play_duration field is not specified, the entire input video is processed. STRING
    The following code provides an example of the input data:
    {
      "input": {
        "url": "URL of the input video",
        "play_duration": "Length of the input video clip to be processed"
      }
    }
  • Output format
    Field Description Dimension Type
    oss_path The OSS path of the generated clip. [] STRING
    success Indicates whether a clip with higher resolution is obtained after super resolution. Valid values:
    • true: A clip with higher resolution is obtained after super resolution.
    • false: A clip with higher resolution failed to be obtained after super resolution.
    [] BOOL
    result The result field. [] STRING
  • Example
    The following code provides an example of the input data of the model:
    {
        "input" : {
            "url": "http://pai-vision-data-sh.oss-cn-shanghai-internal.aliyuncs.com/tmp/5000563****.mp4"
        }
    }
    PAI displays information that is similar to the following output:
    {
        "oss_path": "oss://experience-ai/video_sr/5000563****.mp4", 
        "result": "", 
        "success": true
    }