Microservices Engine:Implement graceful shutdown using MSE XXL-JOB

How open-source XXL-JOB processes jobs internally

The job distribution and execution process in open-source XXL-JOB involves two modules: XXL-JOB Admin and XXL-Job Executor.

Executor registration:

1. After the XXL-JOB SDK starts, the business application initializes the ExecutorRegistryThread thread, which continuously sends heartbeat messages to XXL-JOB Admin.

2. Upon receipt of heartbeat messages, XXL-JOB Admin writes executor information to the xxl_job_registry database table through JobRegistryHelper.

3. A thread in JobRegistryHelper periodically queries and updates the address_list field in the xxl_job_group table, which provides a list of registered executors.

Online executor selection:

1. After a scheduling thread triggers a job, XxlJobTrigger runs the job.

2. Before running the job, XxlJobTrigger reads a list of executors from the address_list field in the xxl_job_group table.

3. ExecutorRouter selects an executor from the list based on the specified routing policy.

4. XxlJobTrigger sends an RPC request to distribute the job to the selected node. If the selected node is offline, the job fails to be triggered.

Job execution and result feedback:

1. After the executor receives a job request, it creates a JobThread thread for each job based on the job ID.

2. When a job request is triggered, it is added to the queue for the current JobThread thread to process. Different jobs have different blocking policies.

3. The JobThread thread continuously reads triggering results in the queue and executes the corresponding JobHandler to complete business logic processing.

4. After a job ends, the JobThread thread submits the execution information to the execution response queue of TriggerCallbackThread and proceeds to the next job.

5. When the executor stops, it executes the XxlJobExecutor.destroy method to stop the JobThread thread and clears the queue of scheduling requests.

The TriggerCallbackThread thread continuously runs, loads the current queue of execution results, and distributes them to XXL-JOB Admin in batches. If it fails to send results, it stores them to local disks and retries later.

Implement graceful shutdown with open-source XXL-JOB

If you use open-source XXL-JOB without the MSE SchedulerX plug-in, you can implement graceful shutdown manually by following three steps: remove traffic, wait for jobs in the queue to complete, and then shut down the application.

The com.xxl.job.core.executor.XxlJobExecutor#destroy method automatically performs a callback when an application process exits in Spring Boot mode. However, the default logic does not fully implement graceful shutdown. The following modifications are required.

Step 1: Remove traffic from application nodes

The stopEmbedServer() method in XxlJobExecutor#destroy stops the heartbeat registration mechanism and sends the registryRemove request to XXL-JOB Admin to remove the current executor. However, the address_list field in the xxl_job_group table is not synchronized in real time, so traffic is not effectively removed.

To fix this, modify the XXL-JOB Admin server using one of the following methods:

• Add processing logic to the JobRegistryHelper.registryRemove method to update the address_list field in the xxl_job_group table. You can also implement the update logic in the freshGroupRegistryInfo method.

• Modify the XxlJobTrigger#trigger() method to read address_list directly from the xxl_job_registry table during the automatic registration process.

Step 2: Wait for jobs in the queue to complete

Modify the XxlJobExecutor#destroy method to wait for all jobs in the queue to complete:

public void destroy(){

    // destroy executor-server
    stopEmbedServer();

    // destroy jobThreadRepository
    if (jobThreadRepository.size() > 0) {
        List keyList = new ArrayList(jobThreadRepository.keySet());
        for (int i=0; i < keyList.size(); i++) {
            JobThread jobThread = jobThreadRepository.get(keyList.get(i));
            // Wait for all jobs in the queue to complete.
            while (jobThread != null && jobThread.isRunningOrHasQueue()) {
                try {
                    TimeUnit.SECONDS.sleep(1L);
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }
            }
        }
    }
    jobHandlerRepository.clear();

    // destroy JobLogFileCleanThread
    JobLogFileCleanThread.getInstance().toStop();

    // destroy TriggerCallbackThread
    TriggerCallbackThread.getInstance().toStop();

}

The TriggerCallbackThread.getInstance().toStop() method synchronizes the execution results after the TriggerCallbackThread thread is stopped, so no additional processing is required for result feedback.

Step 3: Stop application processes

To stop application processes, use kill -15 in the application deployment script to trigger a JVM shutdown hook. You can also forcefully stop application processes upon timeouts based on your business requirements. Alternatively, you can integrate the graceful shutdown feature by using the Spring Boot Actuator /actuator/shutdown endpoint.