CloudFlow:Build an elastic and highly available audio and video processing system based on CloudFlow

Conventional self-managed architecture

As the IT technology develops, services of cloud vendors are constantly improving. You can use Alibaba Cloud services, such as ECS, OSS, and CDN, to build an audio and video processing system to store audio and video files and accelerate playbacks of the files.

Serverless architecture

Video processing workflow system

If you need to accelerate the transcoding of large videos or perform complex operations on videos, use CloudFlow orchestration functions to build a powerful video processing system. The following figure shows the architecture of the solution.

When a user uploads an .mp4 video to OSS, OSS automatically triggers the execution of flows in CloudFlow to transcode the format of the video. You can use this solution to meet the following business requirements:

• A video can be transcoded to various formats at the same time and processed based on custom requirements, such as watermarking or synchronizing the updated information about the processed video to a database.

• When multiple files are uploaded to OSS at the same time, Function Compute performs auto scaling to process the files in parallel and transcodes the files into multiple formats at the same time.

• You can transcode oversized videos by using NAS and video segmentation. Each oversized video is first segmented, transcoded in parallel, and then merged. Proper segmentation intervals improve the transcoding speed of larger videos.

  Note

  When you segment a video, you split the video stream into multiple segments at specified time intervals and record the information about the segments in an index file.

This solution uses CloudFlow to orchestrate functions and implement a video processing system. Code and workflows of multiple functions need to be configured. In this example, Serverless Devs is used to deploy the system.

1. Access the multimedia-process-flow template.

2. Click Deploy in the Development & Experience section to go to the Function Compute application center to deploy the application.

3. On the Create Application page, configure parameters and then click Create and Deploy Default Environment.

   The following table describes the major parameters that you need to configure. Use the default values for other parameters.

   Parameter	Description
   Basic Configurations
   Deployment Type	Select Directly Deploy.
   Role Name	If you use an Alibaba Cloud account to create an application in the application center for the first time, click Authorize Now to go to the Role Templates page, create the AliyunFCServerlessDevsRole service-linked role, and then click Confirm Authorization Policy. If you are using a RAM user, follow the instructions on the page to copy the authorization link to your Alibaba Cloud account for authorization. After you complete the authorization, click Authorized. Note If the Failed to obtain the role message appears, contact the corresponding Alibaba Cloud account to attach the AliRAMReadOnlyAccess and AliyunFCFullAccess policies to the current RAM user. For more information, see Grant permissions to a RAM user by using an Alibaba Cloud account.
   Advanced Settings
   Region	Select the region in which you want to deploy the application.
   Workflow RAM Role ARN	The service-linked role that is used to execute the workflow. Create a service role in advance and attach the AliyunFCInvocationAccess policy to the service-linked role.
   Role Service Function Compute	The service-linked role that is used by Function Compute to access other cloud services. If you do not have special requirements, we recommend that you use the default service role AliyunFCDefaultRole provided by Function Compute.
   Object Storage Bucket Name	Enter the name of a bucket in the same region as the workflow and function.
   Prefix	The prefix of the directory in which the original video is stored. In this example, the prefix is src.
   The directory in which the transcoded video is saved	The directory in which the transcoded video is stored. In this example, the directory is dst.
   OSS trigger RAM role ARN	We recommend that you use the AliyunOSSEventNotificationRole role. If you create the application for the first time, click Create Role and complete authorization. Then, the AliyunOSSEventNotificationRole role is created.

   After 1 to 2 minutes, the application is deployed. The system automatically creates five functions and the multimedia-process-flow-3oih workflow. You can log on to the Function Compute console and CloudFlow console to view the creation results. The functions have the following features:

   • oss-trigger-workflow: uses a corresponding trigger to trigger the execution of the workflow when the system perceives that a new video is uploaded to the specified directory.

   • split: segments a video in the working directory based on the segment length.

   • transcode: transcodes a segmented video based on the specified video format.

   • merge: splices the video after the segments are transcoded.

   • after-process: clears the working directory.

4. Check whether the audio and video processing system is built.

   1. Log on to the CloudFlow console. In the top navigation bar, select a region.

   1. On the workflows page, click the multimedia-process-flow-3oih workflow. On the Execution Records tab of the details page of the workflow, click Started Execution to execute the workflow. In the Execute Workflow panel, specify Execution Name and Input of Execution and then click OK. Example of the input of execution:

      {
        "oss_bucket_name": "buckettestfnf",
        "video_key": "source/test.mov",
        "output_prefix": "outputs",
        "segment_time_seconds": 15,
        "dst_formats": [
          "mp4",
          "flv"
        ]
      }

      If Execution Succeeded is displayed next to Execution Status on the Basic Information tab of the Execution Details page, the execution is complete.

   2. Log on to the OSS console and go to the bucket and the /outputs directory in which the transcoded video is stored. Then, view the transcoded video.

      If the transcoded video file is stored in the /outputs directory, the audio and video processing system works as expected. The following figure shows that the video has been transcoded and merged.