×
Community Blog How Does SchedulerX Help Users Solve Distributed Task Scheduling Problems?

How Does SchedulerX Help Users Solve Distributed Task Scheduling Problems?

This article describes the resource definition, visualized control capability, and distributed batch processing capability of the task scheduling platform.

By Qianxi

Preface

There are a large number of timed and periodic triggers to run specified business tasks in all kinds of business system scenarios. The significance of the existence of distributed task scheduling middleware platform is to support these scenarios for management. Crontab in Linux and Timer in Java are involved in business scenarios where tasks are scheduled to complete a series of automated processing.

Periodic running is the most basic feature of task scheduling. However, with the development of business, more requirements will be put forward for the capabilities of the task scheduling middleware platform. In the traditional business system, Quartz and Spring scheduling packages provide a lot of convenience for business development timing tasks in a framework integration manner. However, with the distributed and micro-service deployment of various business applications, a large number of scattered timing tasks are used in various business application systems. It is difficult to monitor and manage all tasks globally in a unified visual way. The distributed task scheduling middleware platform will visualize each scheduled task effectively and uniformly.

1

The distributed task scheduling platform takes highly reliable scheduled task scheduling as the core foundation, uses visual control as the core value embodiment, and expands the platform capability around these two basic elements in combination with the business development trend.

2

SchedulerX Overview

Platform Resource Management

From the global perspective of unified operation and maintenance control for all business applications, the concept of space is abstracted for the team and deployment environment to isolate the resource management. A layer of abstraction for business applications and machine clusters is called application grouping. Each business application can create a corresponding application group on the scheduling platform and connect with its actual business application, thus realizing the tasks of each business team to control each business team separately without interference on a unified platform. In addition, based on RAM policies on Alibaba Cloud, reasonable resource permission control and isolation can be performed in a unified manner.

3

Business platform architecture managers can make reasonable space and application planning and policy configuration according to the organizational characteristics of their teams and realize RAM isolation and global control of task scheduling resources through the preceding resource model.

Platform Visualized Control

The task is the basic unit of the task scheduling platform to control scheduling operations. Task operation visualization enables tasks that were originally hidden in various business applications to come out so the running status and execution results of each task can be displayed and reminded. The task running states and running results will be unknown or difficult to detect in the absence of visualization. Even after a large number of business iterations, how many scheduled tasks exist in the application system cannot realize standardized management.

4

Distributed Shard Batch Processing

With the development of business volume, distributed batch processing of large data volume requirements will be accompanied by some scheduled task scenarios. Coordinating multiple machines to complete high-volume data processing regularly has become an important feature of the task scheduling platform. Users can simply improve the efficiency of high-volume data processing through the distributed batch processing model.

5

Application Scenarios

Based on the task scheduling platform, the following lists several simple business scenarios to get a preliminary understanding of what can be done on the task scheduling platform.

Broadcast-Based Cluster O&M Scenarios

In broadcast mode, the created scheduling task will send the task running instructions to the business application cluster or the ECS cluster where the Agent is installed at a given periodic frequency. When the newly expanded resources are added, they will be dynamically broadcast to the ECS cluster. Based on this feature, users can construct many custom scenarios:

  • Periodic Cleaning of Logs/Temporary Data: Periodic cleaning of logs or temporary files generated by the service
  • Server Detection: Users can use broadcast shell scripts to build simple server disk/memory/CPU health detection scenarios quickly.
  • Business Cache Refresh: For business scenarios with local caches, you can periodically perform cache refresh and prefetching for specified business applications.
  • Business Service Detection: Users can customize the collection and judgment of various business types of metrics and flexibly build lightweight business monitoring based on the scheduling platform.

6

Scheduled Business Scenarios

In business scenarios in various industries, scheduled task processing on the task scheduling platform is the most extensive business form:

  • Send Message Reminders Regularly: Payment reminders, credit reminders, birthday reminders for customers and employees, and transaction order processing notices
  • Periodic Data Synchronization Processing: Employee information synchronization, basic business information synchronization, daily business data clearing, and transaction order timeout processing
  • Regular Data Push: Monthly/quarterly/annual report push, regular activity announcement push, and regular consumption bill push

7

When the volume of data to be processed in the preceding business expands, the processing efficiency bottleneck will appear in the original stand-alone processing mode. As such, the distributed cluster batch processing capability of task scheduling can be used to parallel process large quantities of data.

Distributed Batch Processing Scenarios

Based on the business batch processing capability, when the real-time business volume is large, users can cooperate with the service degradation strategy to build regular business batch compensation tasks to improve the peak business processing capability and ensure the throughput of the core business during the peak period. As the architectural designer of the business system platform, if the timing distributed batch processing capability can be effectively combined with specific services, the overall business processing capability and stability of the system will be fully utilized. The following is an abstract summary of relevant business scenarios for reference.

In normal scenarios, the core transaction service may complete the call processing of downstream services through RPC/MQ. However, some problems occur in this mode when the business volume increases. The downstream service capability of RPC affects the core service throughput capability. When relying on MQ, you need to ensure that message delivery is normal and that the consumer is processed correctly. These will become bottlenecks in the improvement of core service processing capability. Therefore, within the scope of business acceptance, the periodic batch processing compensation method that decouples business services will improve the processing efficiency, and the idempotence and repeatable execution will improve the reliability of downstream services. At the same time, the distributed batch processing model can be combined to load and process data in batches to reduce DB interaction pressure. At the same time, the compensation mode can be integrated with microservice service degradation in architecture design.

8

Quick Access to SchedulerX

Create an Application Group

After entering the task scheduling console, select the public network region. First, create an application in application management. This application information will become the core element for business applications, Agents, and the task scheduling platform to establish a docking relationship in the following steps.

9

SpringBoot Project Introduces SchedulerX

Add the following dependencies to the locally built SpringBoot project and add the configuration information of the application group created in the corresponding console to Properties. Start the application to establish the relationship between the business application and the task scheduling platform.

<dependencies>
<dependency>
<groupId>com.aliyun.schedulerx</groupId>
<artifactId>schedulerx2-spring-boot-starter</artifactId>
<version>1.3.2</version>
</dependency>
</dependencies>
# Public cloud network environment
spring.schedulerx2.namespace=aad167f6-8bee-41a7-ba41-*********
spring.schedulerx2.endpoint=acm.aliyun.com
spring.schedulerx2.groupId=qianxi.text
spring.schedulerx2.appKey=lYgR6qq **********

Note

After completing the preceding steps, you can create and develop specific business scheduled tasks under subsequent applications.

Please visit this GitHub page for subsequent use.

Official Documentation: https://help.aliyun.com/document_detail/148185.html (Currently only available in Chinese)

Summary

This article describes the resource definition, visualized control capability, and distributed batch processing capability of the task scheduling platform and provides some basic reference cases based on the ability of SchedulerX combined with the actual business scenarios. Hopefully, this explanation makes you familiar with the access and use of the task scheduling platform. For existing users, the platform resource control and isolation can be carried out according to the characteristics of their teams, and the processing efficiency can be improved through distributed batch processing capability after the product business volume increases.

In the future, SchedulerX will continue to improve visual control capabilities to serve enterprise-level scheduled task management needs, and platform control and scheduled scheduling service capabilities will gradually be compatible with common open-source component clients in the market. We will continue to analyze in combination with the actual scene.

0 0 0
Share on

You may also like

Comments

Related Products

  • Link IoT Edge

    Link IoT Edge allows for the management of millions of edge nodes by extending the capabilities of the cloud, thus providing users with services at the nearest location.

    Learn More
  • Elastic High Performance Computing Solution

    High Performance Computing (HPC) and AI technology helps scientific research institutions to perform viral gene sequencing, conduct new drug research and development, and shorten the research and development cycle.

    Learn More
  • Starter Package

    High-performance virtual machines with data transfer plan, starting from $2.50 per month

    Learn More
  • Quick Starts

    Deploy custom Alibaba Cloud solutions for business-critical scenarios with Quick Start templates.

    Learn More