Serverless JOB | New transformation of traditional tasks

Job, as a type of load that stops after running, carries a wealth of usage scenarios in enterprise level development. SAE Job extends the inclusive benefits brought by Serverless technology from the application field to the task field. By combining the use scenarios and best practices of longrun+shortrun, SAE Job has built a Serverless integrated enterprise level development and operation and maintenance platform to meet the diverse demands of different industries, providing users with more complete and diverse capability support and stable and reliable technical support.

Currently, the mainstream way to run jobs is to use distributed task frameworks, such as Quartz, XXLJob, ElasticJob, and so on. As an open source project that has been around for a long time, such frameworks are widely used by many enterprises and have mature functions. In the cloud native era, K8S Jobs and CronJobs are also gradually being considered for adoption. However, the following pain points commonly exist in the above schemes:

1. Low resource utilization. Adopting an open source distributed framework requires program residency and a 7 * 24 hour fee in the virtual machine. The K8s scheme also requires users to maintain cluster nodes, resulting in cost waste.

2. Poor observability. Users need to completely build a set of log collection, cluster, and business monitoring indicator collection and alarm systems to meet the needs of the production environment.

3. Operation and maintenance are complex. Both open source frameworks and K8s need to pay attention to the high availability, high concurrent task capacity, and flexibility of underlying resources, and their operation and maintenance operations have high technical complexity.

SAE Job, as a task oriented Serverless PaaS platform, perfectly solves the above pain points. SAE Job focuses on solving user efficiency and cost issues. It combines traditional task usage experience and functions while using on-demand and pay-as-you-go, achieving low threshold tasks on the cloud, and saving idle resource costs. At the same time, the experience adopts event driven plus intrusion free task scheduling and control, allowing users to have out of the box functions such as task lifecycle management and observable capabilities through zero modification.

SAE Jobs support multiple invocation methods, including the Alibaba Cloud standard API/SDK, which can visually configure Cron expressions to achieve scheduled tasks, and pull SAE Jobs through multiple triggers such as HTTP/MQ/OSS. At the same time, it supports many core features of tasks, including task lifecycle management, execution logging, event notification, log monitoring alerts, timeout retries, blocking policies, task fragmentation, and multiple concurrent tasks.

SAE Jobs provide three core values:

1. Complete full hosting: Provides a one-stop full hosting management interface, with task lifecycle management and scalability out of the box. Users can learn to use SAE with low mental burden and zero cost.

2. Simple operation and maintenance free: The underlying resources are shielded, and users only need to focus on their core business logic development without worrying about cluster availability, capacity, performance, and other issues.

3. Ultra high cost performance ratio: Using the on-demand and pay-as-you-go model, fees are only raised when tasks execute business logic, and no fees are charged for the rest of the time, greatly saving the cost of resources.

The following is a demonstration of the overall use process of SAE Jobs:

Click to view: https://developer.aliyun.com/live/249288

SAE Jobs are task centric and provide a traditional user experience. Currently, the focus is on tasks that support stand-alone broadcast and parallel fragmentation models, while supporting many features such as event driven, concurrency strategies, and timeout retries. It provides low-cost, multi specification, and highly resilient resource instances to meet the execution of short-term tasks.

Compared to the open source distributed framework, its advantages lie in the user experience of fully managed operation and maintenance, complete functionality available out of the box, and white screen control. Resources are released immediately after tasks are completed, without wasting idle resource costs.

Compared to the K8s job, in addition to being fully hosted and free of charge, users do not need to understand the K8s related concepts and technical details, and do not need to maintain its complexity.

SAE supports transformation and migration of XXL job 0. Users can deploy XXL JOB applications to SAE jobs without any code and configuration changes. Users only need to pay for the actual execution logic of the task. In this process, SAE Job acts as the scheduling center and executor of XXL Job. Users only need to focus on task codes and simple configurations, such as task templates, concurrent retries, and SAE is responsible for non-invasive task scheduling and control.

The core value of deploying XXL Jobs to SAE is to reduce costs and improve efficiency:

Cost reduction is reflected in: If the original XXL job is used, in order to maintain its high availability, it requires at least a resident fee of MySQL+2ECS+SLB+N * ECS, while deploying to SAE requires only a payment for the CPU memory consumed by its tasks to execute specific business logic, which is the actual resource consumption of SAE.

The efficiency improvement is reflected in the SAE full custody management operation and maintenance experience, which reduces the overall operation and maintenance complexity and improves application availability.

The following is a demonstration of the XXL Job 0 transformation migration process:

Click to view: https://developer.aliyun.com/live/249289

SAE Job currently focuses on industries such as pan internet, new retail, e-commerce, cultural media, manufacturing, IoT, logistics, financial securities, healthcare, and insurance. The main scenarios include the following six:

1. Scheduled tasks: regularly pulling data and crawlers.

2. Batch processing: data cleaning, conversion, and analysis.

3. Asynchronous execution: Asynchronous status refresh and offline query.

4. Traditional framework migration: XXL Job 0 transformation migration, etc.

5. Microservice architecture: call communication and process decoupling with the original microservice architecture.

6. CI/CD: Implement GitOps using SAE Jobs as a carrier for constructing images, thereby improving the CI/CD process.

Finally, share a customer case using SAE job combined with microservices. The user's business appeal is that they need to regularly push data on hotel product changes to third-party platforms, such as Feizhu, through scheduled tasks. Its task has two characteristics: first, the initialization of the task takes a long time and requires minutes; Secondly, the task execution time is very long, requiring 5-6 hours. In addition to processing business logic, other microservices need to be called to obtain metadata and other information.

The solution we provide for this is to directly deploy microservices such as order centers, product centers, and user centers to SAE applications, and deploy scheduled tasks to SAE jobs. Users can communicate through the built-in registration center of SAE without modification.

This solution provides users with many values:

• Unified entrance operation for two types of loads, and inter-application call 0 transformation.

• Stop the task immediately after it is completed, and immediately release idle resources, greatly saving resource costs.

• Automatically retry when timeout fails, without manual intervention to achieve self-healing.

• Provide a complete task runtime monitoring and alarm mechanism.

With this set of solutions, the user 0 code transformation completes the entire architecture Serverless, while saving resource costs and operation and maintenance costs. SAE will continue to protect the availability of its applications and tasks.

Finally, welcome to use SAE Job, a task oriented Serverless PaaS platform, to experience the new changes it has brought to traditional tasks.