By Qianxi
There are a large number of timed and periodic triggers to run specified business tasks in all kinds of business system scenarios. The significance of the existence of distributed task scheduling middleware platform is to support these scenarios for management. Crontab in Linux and Timer in Java are involved in business scenarios where tasks are scheduled to complete a series of automated processing.
Periodic running is the most basic feature of task scheduling. However, with the development of business, more requirements will be put forward for the capabilities of the task scheduling middleware platform. In the traditional business system, Quartz and Spring scheduling packages provide a lot of convenience for business development timing tasks in a framework integration manner. However, with the distributed and micro-service deployment of various business applications, a large number of scattered timing tasks are used in various business application systems. It is difficult to monitor and manage all tasks globally in a unified visual way. The distributed task scheduling middleware platform will visualize each scheduled task effectively and uniformly.
The distributed task scheduling platform takes highly reliable scheduled task scheduling as the core foundation, uses visual control as the core value embodiment, and expands the platform capability around these two basic elements in combination with the business development trend.
From the global perspective of unified operation and maintenance control for all business applications, the concept of space is abstracted for the team and deployment environment to isolate the resource management. A layer of abstraction for business applications and machine clusters is called application grouping. Each business application can create a corresponding application group on the scheduling platform and connect with its actual business application, thus realizing the tasks of each business team to control each business team separately without interference on a unified platform. In addition, based on RAM policies on Alibaba Cloud, reasonable resource permission control and isolation can be performed in a unified manner.
Business platform architecture managers can make reasonable space and application planning and policy configuration according to the organizational characteristics of their teams and realize RAM isolation and global control of task scheduling resources through the preceding resource model.
The task is the basic unit of the task scheduling platform to control scheduling operations. Task operation visualization enables tasks that were originally hidden in various business applications to come out so the running status and execution results of each task can be displayed and reminded. The task running states and running results will be unknown or difficult to detect in the absence of visualization. Even after a large number of business iterations, how many scheduled tasks exist in the application system cannot realize standardized management.
With the development of business volume, distributed batch processing of large data volume requirements will be accompanied by some scheduled task scenarios. Coordinating multiple machines to complete high-volume data processing regularly has become an important feature of the task scheduling platform. Users can simply improve the efficiency of high-volume data processing through the distributed batch processing model.
Based on the task scheduling platform, the following lists several simple business scenarios to get a preliminary understanding of what can be done on the task scheduling platform.
In broadcast mode, the created scheduling task will send the task running instructions to the business application cluster or the ECS cluster where the Agent is installed at a given periodic frequency. When the newly expanded resources are added, they will be dynamically broadcast to the ECS cluster. Based on this feature, users can construct many custom scenarios:
In business scenarios in various industries, scheduled task processing on the task scheduling platform is the most extensive business form:
When the volume of data to be processed in the preceding business expands, the processing efficiency bottleneck will appear in the original stand-alone processing mode. As such, the distributed cluster batch processing capability of task scheduling can be used to parallel process large quantities of data.
Based on the business batch processing capability, when the real-time business volume is large, users can cooperate with the service degradation strategy to build regular business batch compensation tasks to improve the peak business processing capability and ensure the throughput of the core business during the peak period. As the architectural designer of the business system platform, if the timing distributed batch processing capability can be effectively combined with specific services, the overall business processing capability and stability of the system will be fully utilized. The following is an abstract summary of relevant business scenarios for reference.
In normal scenarios, the core transaction service may complete the call processing of downstream services through RPC/MQ. However, some problems occur in this mode when the business volume increases. The downstream service capability of RPC affects the core service throughput capability. When relying on MQ, you need to ensure that message delivery is normal and that the consumer is processed correctly. These will become bottlenecks in the improvement of core service processing capability. Therefore, within the scope of business acceptance, the periodic batch processing compensation method that decouples business services will improve the processing efficiency, and the idempotence and repeatable execution will improve the reliability of downstream services. At the same time, the distributed batch processing model can be combined to load and process data in batches to reduce DB interaction pressure. At the same time, the compensation mode can be integrated with microservice service degradation in architecture design.
After entering the task scheduling console, select the public network region. First, create an application in application management. This application information will become the core element for business applications, Agents, and the task scheduling platform to establish a docking relationship in the following steps.
Add the following dependencies to the locally built SpringBoot project and add the configuration information of the application group created in the corresponding console to Properties. Start the application to establish the relationship between the business application and the task scheduling platform.
<dependencies>
<dependency>
<groupId>com.aliyun.schedulerx</groupId>
<artifactId>schedulerx2-spring-boot-starter</artifactId>
<version>1.3.2</version>
</dependency>
</dependencies># Public cloud network environment
spring.schedulerx2.namespace=aad167f6-8bee-41a7-ba41-*********
spring.schedulerx2.endpoint=acm.aliyun.com
spring.schedulerx2.groupId=qianxi.text
spring.schedulerx2.appKey=lYgR6qq **********After completing the preceding steps, you can create and develop specific business scheduled tasks under subsequent applications.
Please visit this GitHub page for subsequent use.
Official Documentation: https://help.aliyun.com/document_detail/148185.html (Currently only available in Chinese)
This article describes the resource definition, visualized control capability, and distributed batch processing capability of the task scheduling platform and provides some basic reference cases based on the ability of SchedulerX combined with the actual business scenarios. Hopefully, this explanation makes you familiar with the access and use of the task scheduling platform. For existing users, the platform resource control and isolation can be carried out according to the characteristics of their teams, and the processing efficiency can be improved through distributed batch processing capability after the product business volume increases.
In the future, SchedulerX will continue to improve visual control capabilities to serve enterprise-level scheduled task management needs, and platform control and scheduled scheduling service capabilities will gradually be compatible with common open-source component clients in the market. We will continue to analyze in combination with the actual scene.
OPLG: Best Observability Practices of New Generation Cloud-Native
Hybrid Deployment of SLO-Aware Workload Scheduling for Alibaba Cloud Container Service
481 posts | 48 followers
FollowAlibaba Cloud Native - November 23, 2022
Alibaba Cloud Native Community - September 19, 2023
Alibaba Cloud MaxCompute - May 5, 2019
Alibaba Cloud Native Community - April 23, 2023
Alibaba EMR - September 30, 2019
Alibaba Cloud Data Intelligence - September 6, 2023
481 posts | 48 followers
Follow
Function Compute
Alibaba Cloud Function Compute is a fully-managed event-driven compute service. It allows you to focus on writing and uploading code without the need to manage infrastructure such as servers.
Learn More
Elastic High Performance Computing Solution
High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.
Learn More
Quick Starts
Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.
Learn More
ECS(Elastic Compute Service)
Elastic and secure virtual cloud servers to cater all your cloud hosting needs.
Learn MoreMore Posts by Alibaba Cloud Native Community