SchedulerX:Job FAQ

Spring application cannot find a bean

1. On the Application Management page, check the startup mode of the connected machine. Ensure that the mode is Spring or Spring Boot.

2. Inject the JobProcessor as a bean, for example, by adding the @Component annotation.

3. Check your POM dependencies. If the project depends on spring-boot-devtools, you must exclude it.

4. If the JobProcessor class or the process method uses Aspect-Oriented Programming (AOP) annotations, upgrade the SchedulerX agent to the latest version. Earlier versions do not support AOP.

5. An extra proxy layer can cause a bean type mismatch. To troubleshoot, set a breakpoint in the DefaultListableBeanFactory class. The beanDefinitionNames member variable lists all beans registered with Spring. Check this list to determine if a bean is proxied by an unexpected aspect, which can happen if a faulty third-party library is indirectly imported. Exclude the library to resolve the issue.

In the ProxyFactory log, you can see that TrackerInterceptorFilter acts as an additional AOP proxy, causing a bean type mismatch:

org.springframework.aop.framework.ProxyFactory: 1 interfaces [com.alibaba.schedulerx.worker.processor.JobProcessor]; 1 advisors [org.springframework.aop.support.DefaultPointcutAdvisor: pointcut [Pointcut.TRUE]; advice [com.alibaba.onetouch.tracker.filter.TrackerInterceptorFilter@7dc779bb]]; targetSource [SingletonTargetSource for target object [com.alibaba.onetouch.lc.dts.presentation.LcForexMessageRedoJob@11087483]]; proxyTargetClass=false; optimize=false; opaque=false; exposeProxy=false; frozen=false

If the preceding solutions do not resolve the issue, debug the ThreadContainer.start method. If a class.forName error occurs for a class that exists, a class loader inconsistency is the likely cause. This can happen if your application uses certain frameworks. To resolve this, specify the correct class loader by calling SchedulerxWorker.setClassLoader.

Job fails with "Unable to make field private" error

MapReduce jobs use serialization and deserialization frameworks. Starting from Java 9, reflective access to private fields must be explicitly enabled. Add the following parameter to your JVM arguments:

--add-opens java.base/java.lang=ALL-UNNAMED

Job fails with "submit jobInstanceId to worker timeout" error

If this error occurs occasionally or during an application deployment, you can ignore it.

If the error persists and the workerAddr in the error message always points to the same machine, the persistent connection between the server and the agent is disconnected. To resolve this issue, restart the affected worker node or upgrade the SchedulerX agent to the latest version. After an upgrade, a disconnected persistent connection can automatically recover.

Job fails with "used space beyond 90.0%!" error

The disk is full. Free up disk space on your ECS instance or container.

Job fails with "ClassNotFoundException" error

This error indicates that the class is not found on the worker node that runs the job. Use the fully qualified class name for the Java job's Processor, not the simple name.

If the configured jobProcessor class name is correct, the class is not present on the worker node. Deploying an incorrect package or connecting the application to another user's machine typically causes this issue. You can log on to the worker node and use a decompiler to investigate.

Job fails with "jobInstance=xxx don't update progress more than 60s" error

If a worker node that runs a job stops working or is undergoing deployment and does not report its progress for more than 60 seconds, the server forcibly terminates the job instance. If you can confirm that the worker node causes the issue or that the worker node no longer exists, you can ignore this error.

Job fails without an error message

Symptom:

The job fails, but no error message is displayed.

Possible causes:

A machine failure or a business logic failure can cause this.

Solution:

• Go to the Execution List page and select the Task Instance List tab. Find the failed job instance and click Details in the Operation column to go to the Task Instance Details page and identify the machine where the failure occurred.

  If no task details are available, the job is a standalone job. In this case, find the workerAddr in the basic information section to identify the machine that ran the job.

• Log on to the machine and open the ~/logs/schedulerx/worker.log file.

  Run the grep <Instance ID> worker.log command to view the logs related to the job instance. If an ERROR-level exception exists, check the stack trace for the specific cause.

• If the error description is empty, the failure likely originates in your business logic, which did not report an error. In this case, investigate your business logic first.

• If the error description indicates a framework exception, contact SchedulerX technical support by joining the DingTalk group (ID: 23103656).

Investigate a job failure

• For a standalone job that throws an exception, go to the Execution List page and click the Task Instance List tab. In the Operation column for the failed job instance, click Details to view the error message.

• If the job does not throw an exception or if it is a distributed job, you can use Log Service to troubleshoot the issue if you use the Professional Edition.

• If you use the Basic Edition, you can log on to the worker node and check the SchedulerX logs and your application logs to troubleshoot the issue.

Job stuck in Running state

Symptom:

A scheduled job remains in the Running state and does not finish.

Possible causes:

• An issue in your business logic.

• An issue in SchedulerX.

Solution:

If the issue is related to your business logic, use the following methods to troubleshoot. For other issues, contact SchedulerX technical support by joining the DingTalk group (ID: 23103656).

• Professional Edition: Use the View Stack Trace feature in the console to examine the stack trace of the job exception. This feature is available for agent versions 1.4.2 and later.

• Basic Edition: Log on to the stuck worker node and run the jstack command to view the stack trace.

  jstack <pid> | grep <job instance id> -A 20

  The following example output shows the thread stack trace:

  $jstack 29191 |grep 58903617 -A 200
  
  "Schedulerx-Container-Thread-58903617-0" #4093 prio=5 os_prio=0 tid=xxx               nid=xxx    waiting on condition [0x00002ad443101000]
     java.lang.Thread.State: WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x000000072b837ae0> (a java.util.concurrent.CountDownLatch$Sync)
  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
  at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
  at com.aliyun.ticket.importworker.WOImportJob.startImport(WOImportJob.java:353)
  at com.aliyun.ticket.importworker.WOImportJob.doImportForAll(WOImportJob.java:242)
  at com.aliyun.ticket.importworker.WOImportJob.process(WOImportJob.java:163)
  at com.alibaba.schedulerx.worker.container.ThreadContainer.start(ThreadContainer.java:90)
  at com.alibaba.schedulerx.worker.container.ThreadContainer.run(ThreadContainer.java:60)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:756)
  
  "Container-Batch-Statues-Retrieve-Thread-58903617" #4092 prio=5 os_prio=0 tid=xxx               nid=xxx    waiting on condition [0x00002ad44fe0e000]
     java.lang.Thread.State: TIMED_WAITING (sleeping)
  at java.lang.Thread.sleep(Native Method)
  at com.alibaba.schedulerx.worker.batch.BaseReqHandler$2.run(BaseReqHandler.java:74)
  at java.lang.Thread.run(Thread.java:756)
  
  "TDDL-Druid-ConnectionPool-DestroyScheduler--2-thread-237" #4052 daemon prio=5 os_prio=0 tid=xxx               nid=xxx    waiting on condition [0x00002ad462e00000]
     java.lang.Thread.State: TIMED_WAITING (parking)
  at sun.misc.Unsafe.park(Native Method)
  - parking to wait for  <0x00000007436c7190> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:1129)
  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:809)
  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

Troubleshoot slow job execution

Enable the Professional Edition and use distributed tracing. For more information, see How to enable distributed tracing.

Job instance limit reached

Symptom:

On the Task Management page, you click Run Once and receive the following error message: The number of running job instances has reached the job concurrency limit. Please try again later.

Possible causes:

• An instance of the job is already running.

• The number of running job instances has reached the maximum concurrency configured for the job.

Solution:

If the configured concurrency is appropriate, no action is required. You can view running job instances by going to the Task Management page and clicking More > Execution History.

If the concurrency is not appropriate, click Edit in the Actions column for the target job. In the Advanced Configuration section, set the instance concurrency.

Job not finished before next run

By default, the instance concurrency is 1, which means jobs run serially. If a long-running job has not finished by its next scheduled time, SchedulerX discards the new job instance instead of queuing it.

If you set the instance concurrency to 2, a new job instance can start even if the previous one has not finished. A maximum of two job instances can run concurrently.

Create a one-time job

SchedulerX 2.0 allows you to create one-time jobs. To do this, set the time type to one_time. One-time jobs do not retain any execution history.

View execution history of one-time jobs

To prevent data accumulation, SchedulerX automatically destroys a one-time job after it runs and retains no execution history. If you need to keep a history, enable Log Service to retain execution logs for all jobs for the last two weeks. This can help with troubleshooting. For information about how to enable Log Service, see Application Management.

Configure second-level scheduling

SchedulerX supports second-level scheduling. The cron and fix_rate time types do not support second-level scheduling. You can select the second_delay time type, which runs the job a specified number of seconds after the previous run is complete.

Job not scheduled at a specific time

If a standalone job is not scheduled at a specific time, check whether any machines are in the machine list and whether all machines are busy. If no machines are available or all machines are busy, troubleshoot the issue accordingly. For more information, see No worker available and What to do if all workers are busy.

Configure an alert for when no machines are available for a job. For more information, see Task Management.

Configure a timeout for a job

SchedulerX supports setting a timeout for an entire job but not for individual tasks within the job. You can dynamically modify the timeout period in the console. For more information, see Task Management.

Job instance continues running after being stopped

Symptom:

A job instance continues to run after it has been stopped.

Possible cause:

When you stop a job instance, SchedulerX sends a kill signal to the agent. After receiving the signal, the agent stops dispatching new tasks, stops executing pending tasks, and destroys the instance context and all its thread pools. However, the agent does not immediately stop tasks that are already in progress. The agent only interrupts their threads, so the tasks continue to run until they are complete.

Solution:

• Typically, no action is required. Wait for the running tasks to finish.

• If you need to immediately terminate all running tasks when an instance is stopped, you must modify your task processing logic to handle the thread's interrupt state.

Using advanced job configurations

For more information, see Advanced configuration parameters for Task Management.

All workers are busy

You can view instances on the Application Management page to locate busy worker nodes. Then, click Busy to view the metrics that have exceeded their thresholds.

You can configure busy thresholds on the Application Management page by clicking Edit Application Group.

The Instance Busy Configuration section contains the following thresholds:

• load5: Default value is 0

• Memory usage: Default value is 90%

• Disk usage: Default value is 95%

• Trigger busy status: Enabled by default

If a worker node is busy due to a high load and you are using a Kubernetes deployment, you may need to configure the following two parameters. Otherwise, the collected CPU utilization may be inaccurate. For more information, see Connect a Spring Boot application to SchedulerX.

Enable distributed tracing

Job scheduling supports end-to-end distributed tracing. For more information, see How to enable distributed tracing.

Job stuck or slow during deployment

Symptom:

During an application deployment, a job gets stuck or slows down.

Possible cause:

For a distributed job, if a machine that is processing tasks goes offline, the system redistributes its tasks. The system then polls to check if machines are online, which can slow down the entire process.

Solution:

Upgrade the agent to the latest version. Agent versions 1.7.9 and later include optimizations for this behavior.

Instance parameter prompt for "Run Once"

On the Task Management page, you can click Run Once in the Operation column to run a scheduled job one time. The instance parameters field in the dialog box is optional and is mainly used for testing.

The dialog box also contains the Specify Machine field. You can use the drop-down list to select the instance to run. Note: Instance parameters do not override job parameters.

Instance parameters vs. job parameters

Instance parameters and job parameters are different concepts. The parameters that your code retrieves depend on your business logic.

Retrieve job or instance parameters

For details, see the following code:

@Component
public class JavaDemoProcessor extends JavaProcessor {


    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");


    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {

        LOGGER.info(JSON.toJSONString(jobContext));
        // Retrieve the job parameters.
        String jobParameters = jobContext.getJobParameters();
        // Retrieve the instance parameters.
        String instanceParameters = jobContext.getInstanceParameters();
        LOGGER.info("Job parameters: " + jobParameters);
        LOGGER.info("Instance parameters: " + instanceParameters);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}

Fallback from instance to job parameters

For details, see the following code:

@Component
public class JavaDemoProcessor extends JavaProcessor {

    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");

    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {
        String params;
        if (StringUtils.isNotBlank(jobContext.getInstanceParameters())) {
            params = jobContext.getInstanceParameters();
        } else {
            params = jobContext.getJobParameters();
        }
        LOGGER.info("JavaDemoProcessor params:{}", params);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}