Analysis of cloud native event-driven elastic transcoding scheme

Background

Before the popularization of container technology, event-driven technology was widely used in the database field. The conceptual model is simple: every time data is added, changed or deleted, an event is triggered to perform various operations. The event-driven method can complete the execution of subsequent actions in a very short time delay. The core of event-driven architecture is to react to various events on the system and perform corresponding actions. Elastic scaling has become an indispensable part of almost all cloud platforms. In Kubernetes, HPA (Horizontal Pod Autoscaler) is the most commonly used application elasticity solution. The core of container horizontal scaling is to confirm the scaling plan based on the relationship between resource utilization and the preset threshold water level. The container horizontal scaling method has the characteristics of simple use and rich resource indicators, but it is insufficient for scenarios requiring instant flexibility, especially for offline operations based on event sources. ACK provides ack-keda to provide event-driven flexibility, which is applicable to audio and video offline transcoding, event-driven jobs, streaming data processing and other scenarios.

Event-driven elasticity principle

ACK provides event-driven resilience through the enhanced version of ack-keda. The following figure shows the basic principle of ack-keda.

Ack-keda will periodically consume data from the event source. When messages accumulate, a batch of offline task scaling can be triggered in seconds. When the next cycle comes, the job scaling of the next batch will be performed asynchronously. Ack-keda has the following features:

Rich event sources

Ack-keda built-in supports Kafka, MySQL, PostgreSQL, Rabbitmq, Redis and other built-in data sources. At the same time, it supports obtaining events from customer-defined event sources and performing flexible scaling of job or pod dimensions.

Concurrency control of offline tasks

For large-scale offline operations, the stability of the underlying control will face greater challenges. It is necessary to provide overall control of resources, quotas, and API requests. Ack-keda provides task concurrency control for single batch and total batch to ensure system stability.

Automatically clean up metadata after task completion&&support task backtracking

After the execution of large-scale offline jobs, a large amount of metadata information will be retained. The accumulation of metadata information will cause the stability of APIServer to decline, the performance and stability of the cluster to decline, and may even affect other businesses. Ack-keda will automatically clean up metadata after the task execution, reducing the magnitude of metadata. At the same time, ack-keda also supports the retention of some failed jobs, which is convenient for backtracking and locating the reasons.

Event-driven elastic transcoding case

In this case, we prepare a simple transcoding job. When a new task arrives, we will insert a piece of data similar to the following {"type": "mp4", "state": "waiting", "createTimeStamp": "1610332940", "fileName": "World and peace", "endTimeStamp": "", "uuid": "1fae72ff-3239-42f5-af97-04711d8007e8"} into MongoDB. At this time, The event-driven controller of the container service will query the job with the status of "state": "waiting" from the database, pop up the Job Pod matching the number of tasks to carry the transcoding job, complete the transcoding business, and modify the state field in the data from the previous waiting to finished. At the same time, after the job is completed, it will automatically clean up, reduce the pressure of metadata on APIServer, and greatly reduce the burden of developers.

1. Install the event-driven elastic controller - ack-keda

Log in to the Alibaba Cloud container service kubernetes console, click the application market in the left sidebar, and search for ack-keda

Select the corresponding cluster and click Deploy to deploy to the cluster

Select the workload in the left sidebar, select stateless service, select the kube-system namespace, and confirm the successful deployment of ack-keda

2. Deployment based on mongoDB event source-driven elasticity example

1. Deploy mongoDB

2. MongoDB creates a new user

3. Deploy TriggerAuthentication and ScaledJob

4. Insert the business data to be transcoded

5. View job status

Write at the end

The transcoding business introduced in this article is also a requirement we often encounter in daily scenarios. It can be seen that the use of ack-keda is relatively easy, and the actual effect can also meet our daily needs. We have recently added support for mongoDB event sources on the basis of the keda community, and have PR it to the community. So far, the built-in event sources have been able to meet most of our event-driven scenarios. If you want to learn more, please click to view the keda community.