Microservices Engine:Implement graceful shutdown based on MSE XXL-JOB

Issue 1: Jobs are scheduled to offline nodes

After executors become offline, the system does not complete job scheduling and update the executor list in a timely manner. In this case, jobs may be scheduled to offline nodes, which results in scheduling failures.

Summary: The executor list obtained during executor registration is not synchronized with the executor list obtained for running the job in real time. As a result, the online executor list is not updated in a timely manner.

Issue 2: Jobs are forcefully stopped

When an XXL-JOB executor is disabled, the JobThread thread of the job is stopped and the job is marked as failed. In addition, the requests of all jobs in the queue are discarded and the jobs are marked as failed.

Summary: In the preceding shutdown process, the underlying layer of the removeJobThread thread directly stops the JobThread thread, ignores the job requests in the thread queue, and marks the jobs as failed.

Step 1: Remove traffic from application nodes

• The stopEmbedServer() method in XxlJobExecutor#destroy stops the heartbeat registration mechanism and sends the registryRemove request to XXL-JOB Admin to remove the current executor.

• After XXL-JOB Admin receives a request, it removes the current executor from the xxl_job_registry table in the database. However, based on the preceding analysis, the address_list field in the xxl_job_group table is not synchronized and updated in real time. Therefore, traffic is not removed.

• The features provided by the XXL-JOB Admin server need to be modified to remove traffic. You can use one of the following methods to modify the features.

  • Add the subsequent processing logic to the JobRegistryHelper.registryRemove method to update the address_list field in the xxl_job_group table. You can also implement the update logic in the freshGroupRegistryInfo method.

  • Modify the XxlJobTrigger#trigger() method to adjust the method for reading the address_list field in the xxl_job_group table. This allows the XXL-JOB Admin server to read address_list directly from the xxl_job_registry table during the automatic registration process.

After you perform the preceding operations, traffic is removed.

Step 2: Wait for jobs in the queue to complete

• Modify the subsequent processing logic of the XxlJobExecutor#destroy method to wait for all jobs in the queue to complete. The following sample code shows an example.

  public void destroy(){
  
      // destroy executor-server
      stopEmbedServer();
  
      // destroy jobThreadRepository
      if (jobThreadRepository.size() > 0) {
          List keyList = new ArrayList(jobThreadRepository.keySet());
          for (int i=0; i < keyList.size(); i++) {
              JobThread jobThread = jobThreadRepository.get(keyList.get(i));
              // Wait for all jobs in the queue to complete.
              while (jobThread != null && jobThread.isRunningOrHasQueue()) {
                  try {
                      TimeUnit.SECONDS.sleep(1L);
                  } catch (InterruptedException e) {
                      e.printStackTrace();
                  }
              }
          }
      }
      jobHandlerRepository.clear();
  
      // destroy JobLogFileCleanThread
      JobLogFileCleanThread.getInstance().toStop();
  
      // destroy TriggerCallbackThread
      TriggerCallbackThread.getInstance().toStop();
  
  }

• Wait for the queue of execution results to feed back all execution results. In open source XXL-JOB, the TriggerCallbackThread.getInstance().toStop() method finally synchronizes the execution results at a time after the TriggerCallbackThread thread is stopped. Therefore, additional processing is not required.

After you perform this step, you can wait for running jobs to complete. You can also perform custom shutdown operations based on job types.

Step 3: Stop application processes

• To stop application processes, we recommend that you add kill -15 to the application deployment script to trigger a JVM shutdown hook. You can also forcefully stop application processes upon timeouts based on your business requirements.

• You can integrate the graceful shutdown feature into XXL-JOB by using the Spring Boot Actuator feature and then use the /actuator/shutdown interface to shut down the application.