Train a video classification model

Platform for AI (PAI) provides the video classification algorithm. You can use the algorithm to train video classification models based on tens of millions of videos. This topic describes how to generate a video classification model based on short video data.

Data description

The video classification algorithm supports raw video data in common formats, such as .avi and .mp4. In this topic, video files named eyemakeup and lipsmakeup are used to train a video classification model. For more information, see Download resources for video classification.

Data format conversion

The data format conversion module can convert raw video files into TFRecord files. You can use TFRecord files to accelerate model training. You can run the following command to convert raw video files into TFRecord files. You can run PAI commands by using the SQL Script component, the MaxCompute client, or an ODPS SQL node of DataWorks. For more information, see MaxCompute client (odpscmd) or Develop a MaxCompute SQL task.

pai -name easy_vision_ext
-project algo_public
-Dbuckets='oss://{bucket_name}.{oss_host}/{path}/'
-Darn='acs:ram::*******:role/aliyunodpspaidefaultrole'
-DossHost='{oss_host}'
-Dcmd convert
-Dconvert_config='{bucket_name}.{oss_host}/{path}/{config_file}'
-Dlabel_file='{bucket_name}.{oss_host}/{path}/{config_file}/{label_file}'
-Doutput_tfrecord='{bucket_name}.{oss_host}/{path}/'

Example:

pai -name easy_vision_ext
-project algo_public
-Dbuckets='oss://examplebucket.oss-cn-beijing-internal.aliyuncs.com/vip/'
-Darn='acs:ram::******:role/aliyunodpspaidefaultrole'
-DossHost='oss-cn-beijing-internal.aliyuncs.com'
-Dcmd convert
-Dconvert_config='oss://examplebucket.oss-cn-beijing-internal.aliyuncs.com/vip/ucf101_qince.config'
-Dlabel_file='oss://examplebucket.oss-cn-beijing-internal.aliyuncs.com/vip/vip.csv'
-Doutput_tfrecord='oss://examplebucket.oss-cn-beijing-internal.aliyuncs.com/vip/'

Parameters in the preceding code:

buckets: the root directory of the Object Storage Service (OSS) endpoint. The directory must end with a forward slash (/).
arn: the Alibaba Cloud Resource Name (ARN) of the RAM role that has the permissions to access OSS resources. To obtain the value of arn, perform the following steps: Go to the PAI console. In the Designer section of the Dependent Services page, click View Authorization in the Actions column. For more information, see Grant the permissions that are required to use Machine Learning Designer.
ossHost: the endpoint of the OSS bucket. For more information, see Regions and endpoints.
Dconvert_config: the configuration file, which specifies the categories of the raw video files. The data description section of this topic provides the download address of the configuration file. The following example shows the content of the configuration file:
```
class_map {
  label_name: "ApplyEyeMakeup"
}
class_map {
  label_name: "ApplyLipstick"
}
model_type: VIDEO_CLASSIFICATION
converter_class: "QinceConverter"
write_thread_num: 8
part_record_num: 64
test_ratio: 0.0
```

Dlabel_file: the OSS path in which the video files that are used for the training are stored. You must upload the video files to the OSS endpoint and specify the path in the configuration file. Sample code:

# The entry ID, raw data, and labeling result.
1,"{""tfspath"": ""oss://examplebucket/data/eye/public_v_ApplyEyeMakeup_g01_c01.avi""}","{""option"": ""ApplyEyeMakeup""}"
2,"{""tfspath"": ""oss://examplebucket/data/eye/public_v_ApplyEyeMakeup_g02_c03.avi""}","{""option"": ""ApplyEyeMakeup""}"
3,"{""tfspath"": ""oss://examplebucket/data/eye/public_v_ApplyEyeMakeup_g02_c04.avi""}","{""option"": ""ApplyEyeMakeup""}"
4,"{""tfspath"": ""oss://examplebucket/data/eye/public_v_ApplyEyeMakeup_g03_c01.avi""}","{""option"": ""ApplyEyeMakeup""}"
5,"{""tfspath"": ""oss://examplebucket/data/eye/public_v_ApplyEyeMakeup_g04_c01.avi""}","{""option"": ""ApplyEyeMakeup""}"
6,"{""tfspath"": ""oss://examplebucket/data/lips/public_v_ApplyLipstick_g04_c02.avi""}","{""option"": ""ApplyEyeMakeup""}"
7,"{""tfspath"": ""oss://examplebucket/data/lips/public_v_ApplyLipstick_g05_c01.avi""}","{""option"": ""ApplyLipstick""}"
8,"{""tfspath"": ""oss://examplebucket/data/lips/public_v_ApplyLipstick_g07_c04.avi""}","{""option"": ""ApplyLipstick""}"
9,"{""tfspath"": ""oss://examplebucket/data/lips/public_v_ApplyLipstick_g01_c02.avi""}","{""option"": ""ApplyLipstick""}"

Replace the paths in the configuration file with actual OSS endpoints.

Doutput_tfrecord: the output path of the TFRecord files.

You can run the following command to train a video classification model based on the converted data: You can run PAI commands by using the SQL Script component, the MaxCompute client, or an ODPS SQL node of DataWorks. For more information, see MaxCompute client (odpscmd) or Develop a MaxCompute SQL task.

pai -name ev_video_classification_ext
-project algo_public
-Dbackbone='resnet_3d_50'
-Dnum_epochs=50
-Ddecay_epochs=5
-Dsave_checkpoints_epochs=1
-Dmodel_dir='{bucket_name}.{oss_host}/{output_model_path}/'
-Duse_pretrained_model=true
-Dpretrained_model='{bucket_name}.{oss_host}/{model_path}/resent_3d_50_model.ckpt'
-Dtrain_data='{bucket_name}.{oss_host}/{path}/data_train_0_0.tfrecord'
-Dtest_data='{bucket_name}.{oss_host}/{path}/data_train_0_0.tfrecord'
-Dlabel_map_path='{bucket_name}.{oss_host}/{path}/data_label_map.pbtxt'
-Dnum_test_example=10
-Dtrain_batch_size=2
-Dtest_batch_size=2
-Dbuckets='{bucket_name}.{oss_host}/{path}'
-Darn='acs:ram::*********:role/aliyunodpspaidefaultrole'
-DossHost='{oss_host}'
-Dinitial_learning_rate=0.0001
-Dstaircase=false
-DgpuRequired=100
-Dnum_classes=2

The following sample code provides an example on how to train a video classification model:

pai -name ev_video_classification_ext
-project algo_public
-Dbackbone='resnet_3d_50'
-Dnum_epochs=50
-Ddecay_epochs=5
-Dsave_checkpoints_epochs=1
-Dmodel_dir='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/model/'
-Duse_pretrained_model=true
-Dpretrained_model='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/model/resent_3d_50_model.ckpt'
-Dtrain_data='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/vip/data_train_0_0.tfrecord'
-Dtest_data='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/vip/data_train_0_0.tfrecord'
-Dlabel_map_path='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/vip/data_label_map.pbtxt'
-Dnum_test_example=10
-Dtrain_batch_size=2
-Dtest_batch_size=2
-Dbuckets='oss://demo-yuze.oss-cn-beijing-internal.aliyuncs.com/vip/'
-Darn='acs:ram::********:role/aliyunodpspaidefaultrole'
-DossHost='oss-cn-beijing-internal.aliyuncs.com'
-Dinitial_learning_rate=0.0001
-Dstaircase=false
-DgpuRequired=100
-Dnum_classes=2

Parameters in the preceding code:

backbone: the type of the backbone network.
model_dir: the directory of the output model.
pretrained_model: the uploaded address of the pretrained model.
train_data: the TFRecord file that is generated after the training data is converted.
test_data: the TFRecord file that is generated after the test data is converted.
label_map_path: a .pbtxt file that is generated after data conversion.
num_test_example: the number of samples used in the test.
train_batch_size: the number of samples that are trained in a batch.
buckets: the root directory.
arn: the ARN of the RAM role that has the permissions to access OSS resources. To obtain the value of arn, perform the following steps: Go to the PAI console. In the Designer section of the Dependent Services page, click View Authorization in the Actions column. For more information, see Grant the permissions that are required to use Machine Learning Designer.
num_classes: the number of categories.

The final video classification model is generated in the TensorFlow SaveModel format and stored in the model_dir directory. You can deploy the model to EAS. For more information, see Model service deployment by using the PAI console.