When applications are connected to a self-managed job scheduling system based on open source XXL-JOB, issues such as complex configurations, low execution efficiency, and difficulties in job monitoring and management may occur. Our solution offers time-based job scheduling and job sharding to help you execute self-managed scheduled jobs.
Prerequisites
The Resource Access Management (RAM) user to be used is granted the required permissions. For more information, see XXL-JOB on SchedulerX authorization.
An XXL-JOB on SchedulerX instance is created. For more information, see Create an instance.
Solution overview
This solution describes how to create an XXL-JOB task in Java, build a Docker image and upload it to an Alibaba Cloud image repository, deploy the application, connect the containerized application to XXL-JOB on SchedulerX, and test the connection by using standalone and sharding broadcast tasks. It is intended to help you understand how to use and configure XXL-JOB on SchedulerX. The process consists of three steps:
Create an application: You can create an application to manage scheduled jobs in a centralized manner. This allows you to view, configure, and schedule jobs with ease and improves job management efficiency.
Develop and deploy the application: You can write code for scheduled jobs, build a Docker image, and then upload the image to an Alibaba Cloud image repository. This way, the application can be deployed and managed in a container.
Test and verify the application: This makes sure that jobs on the connected application can be automatically scheduled and managed in XXL-JOB on SchedulerX, and can be executed at the specified point in time.
Step 1: Create an application
Log on to the MSE console, go to the XXL-JOBVersion page, and then select a region in the top navigation bar.
On the XXL-JOB Version page, click the ID of your instance to go to the instance details page. On the instance details page, click in the left-side navigation pane. On the page that appears, click CreateApplication. In the CreateApplication panel, configure the AppName and Name parameters, set the AccessToken parameter to a system-generated token, and then click OK.
Step 2: Develop and deploy the application
1. Develop an XXL-JOB job
XXL-JOB on SchedulerX supports Java, Go, and Python applications. For more information, see the following demo projects of open source XXL-JOB:
Python: xxl-job-executor-python
Java
Configure the environment: Add the
xxl-job-coreMaven dependency to the pom.xml file. For more information about the version, see xxl-job-executor-sample-springboot.<!-- xxl-job-core --> <dependency> <groupId>com.xuxueli</groupId> <artifactId>xxl-job-core</artifactId> <version>2.2.x</version> </dependency>Initialize the executor.
@Configuration public class XxlJobConfig { private Logger logger = LoggerFactory.getLogger(XxlJobConfig.class); @Value("${xxl.job.admin.addresses}") private String adminAddresses; @Value("${xxl.job.accessToken}") private String accessToken; @Value("${xxl.job.executor.appname}") private String appname; @Value("${xxl.job.executor.address}") private String address; @Value("${xxl.job.executor.ip}") private String ip; @Value("${xxl.job.executor.port}") private int port; @Value("${xxl.job.executor.logpath}") private String logPath; @Value("${xxl.job.executor.logretentiondays}") private int logRetentionDays; @Bean public XxlJobSpringExecutor xxlJobExecutor() { logger.info(">>>>>>>>>>> xxl-job config init."); XxlJobSpringExecutor xxlJobSpringExecutor = new XxlJobSpringExecutor(); xxlJobSpringExecutor.setAdminAddresses(adminAddresses); xxlJobSpringExecutor.setAppname(appname); xxlJobSpringExecutor.setAddress(address); xxlJobSpringExecutor.setIp(ip); xxlJobSpringExecutor.setPort(port); xxlJobSpringExecutor.setAccessToken(accessToken); xxlJobSpringExecutor.setLogPath(logPath); xxlJobSpringExecutor.setLogRetentionDays(logRetentionDays); return xxlJobSpringExecutor; } }Write the job execution code. In this example, XXL-JOB V2.2.x is used.
NoteXXL-JOB on SchedulerX of different engine versions provides different API operations. For more information, see the demo project of open source XXL-JOB.
@Component public class SampleXxlJob { private static Logger logger = LoggerFactory.getLogger(SampleXxlJob.class); @XxlJob("helloworld") public ReturnT<String> helloworld(String param) throws Exception { XxlJobLogger.log("XXL-JOB, Hello World, start..."); for (int i = 0; i < 5; i++) { XxlJobLogger.log("beat at:" + i); TimeUnit.SECONDS.sleep(2); } System.out.println("XXL-JOB, Hello World, finished"); return ReturnT.SUCCESS; } }
Golang
Configure the environment: Run the following command to obtain XXL-JOB on SchedulerX SDK for Go by using the latest tag: For more information about specific versions, see xxl-job-executor-go.
go get github.com/xxl-job/xxl-job-executor-go@{Latest tag}Write an initialization executor.
package main import ( "context" "fmt" xxl "github.com/xxl-job/xxl-job-executor-go" "github.com/xxl-job/xxl-job-executor-go/example/task" "log" ) func main() { exec := xxl.NewExecutor( xxl.ServerAddr("xxxxxx"), // The endpoint. To obtain the endpoint, find your application on the Application Management page of the XXL-JOB on SchedulerX console and click Access Configuration in the Operation column. xxl.AccessToken("xxxxxxx"), // The access token. To obtain the access token, find your application on the Application Management page of the XXL-JOB on SchedulerX console and click Access Configuration in the Operation column. xxl.ExecutorPort("9999"), // Optional. The port number of the executor. Default value: 9999. xxl.RegistryKey("golang-jobs"), // The name of the executor. xxl.SetLogger(&logger{}), // The custom logger. ) exec.Init() exec.Use(customMiddleware) // Configure a log handler. exec.LogHandler(customLogHandle) // Register a job handler. exec.RegTask("task.test", task.Test) exec.RegTask("task.shardingTest", task.ShardingTest) log.Fatal(exec.Run()) } // Specify a custom log handler. func customLogHandle(req *xxl.LogReq) *xxl.LogRes { return &xxl.LogRes{Code: xxl.SuccessCode, Msg: "", Content: xxl.LogResContent{ FromLineNum: req.FromLineNum, ToLineNum: 2, LogContent: "This is a custom log handler.", IsEnd: true, }} } // Implement the xxl.Logger interface. type logger struct{} func (l *logger) Info(format string, a ...interface{}) { fmt.Println(fmt.Sprintf("Custom log - "+format, a...)) } func (l *logger) Error(format string, a ...interface{}) { log.Println(fmt.Sprintf("Custom log - "+format, a...)) } // Specify custom middleware. func customMiddleware(tf xxl.TaskFunc) xxl.TaskFunc { return func(cxt context.Context, param *xxl.RunReq) string { log.Println("I am a middleware start") res := tf(cxt, param) log.Println("I am a middleware end") return res } }Write the job execution code.
package task import ( "context" "fmt" xxl "github.com/xxl-job/xxl-job-executor-go" ) func Test(cxt context.Context, param *xxl.RunReq) (msg string) { fmt.Println("test one task" + param.ExecutorHandler + " param:" + param.ExecutorParams + " log_id:" + xxl.Int64ToStr(param.LogID)) return "test done" } func ShardingTest(cxt context.Context, param *xxl.RunReq) (msg string) { fmt.Println("shardingId:" + xxl.Int64ToStr(param.BroadcastIndex) + ", shardingTotal:" + xxl.Int64ToStr(param.BroadcastTotal)) return "ShardingTest done" }
Python
Pull dependencies. For more information about specific versions, see xxl-job-executor-python.
pip install pyxxl # If logs need to be written to redis pip install "pyxxl[redis]" # If you need to load configurations from. env pip install "pyxxl[dotenv]" # Install all features pip install "pyxxl[all]"Write the job execution code.
import asyncio import time from pyxxl import ExecutorConfig, PyxxlRunner from pyxxl.ctx import g config = ExecutorConfig( xxl_admin_baseurl="http://xxljob-1b3fd81****.schedulerx.mse.aliyuncs.com/api/", executor_app_name="xueren-test", access_token="default_token", # executor_listen_host="0.0.0.0", # If the xxl-admin can directly connect to the IP address of the executor, do not specify the executor_listen_host ) app = PyxxlRunner(config) @app.register(name="demoJobHandler") async def test_task(): # you can get task params with "g" g.logger.info("get executor params: %s" % g.xxl_run_data.executorParams) for i in range(10): g.logger.warning("test logger %s" % i) await asyncio.sleep(5) return "Success..." @app.register(name="sync_func") def test_task4(): # if you want to see the execution log on the xxl-admin, you must use g.logger to print the log. By default, only info and above logs are printed. n = 1 g.logger.info("Job %s get executor params: %s" % (g.xxl_run_data.jobId, g.xxl_run_data.executorParams)) # If the synchronization task contains loops, you must check the value of g.cancel_event every time to support the cancel operation. while n <= 10 and not g.cancel_event.is_set(): # If you do not need to view the log from the xxl-admin, you can use your own logger. g.logger.info( "log to {} logger test_task4.{},params:{}".format( g.xxl_run_data.jobId, n, g.xxl_run_data.executorParams, ) ) time.sleep(2) n += 1 return "Success 3" if __name__ == "__main__": app.run_executor()
2. Deploy the application to Alibaba Cloud
XXL-JOB on SchedulerX supports only the internal networks of Alibaba Cloud. In this case, you must deploy your application to Alibaba Cloud. In this example, a Java application is deployed to a Container Service for Kubernetes (ACK) cluster.
The ACK cluster must reside in the same virtual private cloud (VPC) as the XXL-JOB on SchedulerX instance in which the application is created.
Edit the Dockerfile file in the root directory of the Spring Boot application.
# Replace the image in the following code with your base image: FROM reg.docker.alibaba-inc.com/xxx/xxxx-java:1.0-beta MAINTAINER xueren ENV JAVA_OPTS="" ADD target/xxl-job-executor-sample-springboot-*.jar /app.jar ENTRYPOINT ["sh","-c","java -jar $JAVA_OPTS /app.jar]Use Docker to create a Docker image and upload the image to an Alibaba Cloud image repository.
docker login --username=xxx@aliyun.com registry.cn-hangzhou.aliyuncs.com --password=xxxxxx docker buildx build --platform linux/amd64 -t registry.cn-hangzhou.aliyuncs.com/schedulerx/xxljob-demo:2.2.0 . docker push registry.cn-hangzhou.aliyuncs.com/schedulerx/xxljob-demo:2.2.0Log on to the XXL-JOB on SchedulerX console and go to the details page of your instance. On the page, find the application that you want to deploy and click Access Configuration in the Operation column.
Log on to the Container Service for Kubernetes (Container Service) console. In the left-side navigation pane, choose . On the page that appears, click Create from YAML in the upper-right corner to create a Deployment. In this example, Access Method 2 is used and the application is restarted by using the -D parameter. Copy the code provided by the access method and replace the value of the JAVA_OPTS environment variable with the copied code. This injects Java virtual machine (JVM) parameters into the environment variable.

apiVersion: apps/v1 kind: Deployment metadata: name: xxljob-xueren-test labels: app: xxljob-xueren-test spec: replicas: 2 selector: matchLabels: app: xxljob-xueren-test template: metadata: labels: app: xxljob-xueren-test spec: containers: - name: xxljob-executor image: registry.cn-hangzhou.aliyuncs.com/schedulerx/xxljob-demo:2.2.0 ports: - containerPort: 9999 env: - name: JAVA_OPTS value: >- -Dxxl.job.admin.addresses=http://xxljob-xxxxx.schedulerx.mse.aliyuncs.com -Dxxl.job.executor.appname=xueren_test -Dxxl.job.accessToken=xxxxxxx
Step 3: Test and verify the application
1. Check executors
On the instance details page of the XXL-JOB on SchedulerX console, click Application Management in the left-side navigation pane. On the Application Management page, find the deployed application and click the value in the Number of actuators column to view the IP address and online status of each executor.

2. Test jobs
Standalone job
A standalone job is executed in an idempotent manner in an executor selected based on the routing policy.
On the instance details page, click Task Management in the left-side navigation pane. On the page that appears, click Create Task. In the Basic configuration step of the Create Task panel, configure the Task Name and jobhandler Name parameters, select the deployed application from the Associated Applications drop-down list, set the Routing Policy parameter to Polling, and then click Next step.

In the Timing Configuration step, set the Time Type parameter to cron and click Using the Build Tool to generate a cron expression. In this example, the job is executed once every day at 12:00. Then, click Next step.

In the Notification Configuration step, you can configure alerts for execution timeout, execution success, and execution failure, and specify notification methods and alert contacts. In this example, the default configurations in the console are used.

After the job is created, find the job on the Task Management page and click Run once in the Operation column. In the Perform tasks manually dialog box, select a machine on which you want to execute the job, enter instance parameters, and then click OK.

Find the job on the Task Management page and choose in the Operation column to view the execution records of the job.
On the instance details page, click Execution List in the left-side navigation pane. On the page that appears, find the execution record that you want to view and click Log in the Operation column to view the logs of the job execution.

Sharding broadcast job
A sharding broadcast job is executed in all executors of the application, and each executor is assigned a unique shard number. This type of job is applicable to distributed batch processing. Unlike open source XXL-JOB that does not support the aggregation feature, XXL-JOB on SchedulerX aggregates and displays the execution of a sharding broadcast job on each shard.
On the instance details page, click Task Management in the left-side navigation pane. On the page that appears, click Create Task. In the Basic configuration step of the Create Task panel, configure the Task Name and jobhandler Name parameters, select the deployed application from the Associated Applications drop-down list, set the Routing Policy parameter to Multipart Broadcast, and then click Next step.

In the Timing Configuration step, set the Time Type parameter to cron and click Using the Build Tool to generate a cron expression. In this example, the job is executed at the 10th minute of every hour. Then, click Next step.

In the Notification Configuration step, you can configure alerts for execution timeout, execution success, and execution failure, and specify notification methods and alert contacts. In this example, the default configurations in the console are used.

After the job is created, find the job on the Task Management page and click Run once in the Operation column. In the Perform tasks manually dialog box, select one or more machines on which you want to execute the job, enter instance parameters, and then click OK.

Find the job on the Task Management page and choose in the Operation column to view the execution records of the job.
On the instance details page, click Execution List in the left-side navigation pane. On the page that appears, find the execution record that you want to view and click Details in the Operation column. In the Scheduling Details panel, view the execution details on each machine on the Slice Details tab.

Find a shard and click Log in the Operation column to view the logs of the job execution of the shard.
