Microservices Engine:Implementing graceful shutdown based on MSE XXL-JOB

Issue 1: Delayed traffic removal from offline nodes

When an executor goes offline, it cannot update the scheduling and execution machine list in time, resulting in scheduling failures when tasks are scheduled to offline nodes.

Summary: Due to the asynchronous periodic updates between the executor registration logic and the trigger task execution list data retrieval, there is a delay in refreshing the available online machine list.

Issue 2: Forced interruption of task execution

Currently, when XXL-Job Executor exits, it directly interrupts the task running thread and treats it as a failure. Additionally, all execution requests waiting in the queue are discarded and treated as failures.

Summary: In the above shutdown process, removeJobThread directly interrupts running task threads, and tasks waiting in the thread queue are directly ignored and treated as failures.

Step 1: Remove traffic from application nodes

• First, in the XxlJobExecutor#destroy method, there is a stopEmbedServer() method that stops heartbeat registration and sends a registryRemove request to the scheduling center to remove the current execution node.

• After receiving the request, the scheduling server removes the current node from the xxl_job_registry table in the database. However, as explained in the principle analysis, the actual list used is the address_list in the xxl_job_group table (which is not synchronized and updated), so the traffic removal action is not truly completed at this point.

• The scheduling server needs to be modified to implement traffic removal. Modification points to choose from (choose one):

  • Add subsequent processing to the JobRegistryHelper.registryRemove method to directly refresh the address_list in the xxl_job_group table, or implement refresh logic in freshGroupRegistryInfo.

  • Modify the XxlJobTrigger#trigger() method to adjust how addressList is read from group. For automatic registration, read the address list directly from the xxl_job_registry table.

After completing the above modifications, the first step of traffic removal is completed.

Step 2: Wait for running business to complete

• Modify the next step in the XxlJobExecutor#destroy method to wait for all pending tasks to complete. Refer to the following code:

  public void destroy(){
  
      // destroy executor-server
      stopEmbedServer();
  
      // destroy jobThreadRepository
      if (jobThreadRepository.size() > 0) {
          List keyList = new ArrayList(jobThreadRepository.keySet());
          for (int i=0; i < keyList.size(); i++) {
              JobThread jobThread = jobThreadRepository.get(keyList.get(i));
              // Wait for all task queues to complete execution
              while (jobThread != null && jobThread.isRunningOrHasQueue()) {
                  try {
                      TimeUnit.SECONDS.sleep(1L);
                  } catch (InterruptedException e) {
                      e.printStackTrace();
                  }
              }
          }
      }
      jobHandlerRepository.clear();
  
      // destroy JobLogFileCleanThread
      JobLogFileCleanThread.getInstance().toStop();
  
      // destroy TriggerCallbackThread
      TriggerCallbackThread.getInstance().toStop();
  
  }

• Wait for all execution results in the response queue to be sent. The current open-source implementation of the TriggerCallbackThread.getInstance().toStop() method processes the results one last time after interrupting the response result thread, so no additional processing is needed.

At this point, waiting for running tasks to complete is finished. Additionally, you can customize the shutdown process for different task types as needed.

Step 3: Application process shutdown

• It is recommended to trigger the above JVM Hook callback in the deployment script by using kill -15 to stop the application, with timeout forced shutdown control as needed for business requirements.

• You can integrate task scheduling graceful shutdown with SpringBoot Actuator functionality and shut down the application through the /actuator/shutdown interface.