Microservices Engine:Job management FAQ

My Spring application cannot find the beans

Work through these checks in order:

1. On the Application Management page in the SchedulerX console, find your application and click the number in the Total number of instances column. Confirm that the boot mode is Spring or Spring Boot.

2. Verify that a bean is injected into the JobProcessor in your application code -- for example, by adding the @Component annotation.

3. Remove the spring-boot-devtools dependency from your pom.xml file if it is present.

4. If the JobProcessor or the process method uses any aspect-oriented programming (AOP) annotation, upgrade the SchedulerX agent to the latest version. Earlier versions do not support AOP annotations.

5. Determine whether the beans are proxied by other aspects. Add breakpoints to the DefaultListableBeanFactory class and inspect the beanDefinitionNames member variable, which lists all beans registered in Spring. If the beans are proxied (for example, because invalid second-party libraries are indirectly imported), remove those aspects.

If none of these steps resolve the issue, debug ThreadContainer.start. A class.forName error where the class clearly exists usually means the Spring application framework and the SchedulerXWorker agent are using different class loaders. Call SchedulerxWorker.setClassLoader to align them.

My job fails with "Unable to make field private"

MapReduce uses serialization and deserialization frameworks. Java 9 and later versions restrict reflective access to private fields by default, which causes this error. Add the following JVM parameter to re-enable access:

--add-opens java.base/java.lang=ALL-UNNAMED

My job fails with "submit jobInstanceId to worker timeout"

You can safely ignore this error if it occurs during an application release or only occasionally.

If the error persists and the workAddr value is always the same, the persistent connection between the server and the agent is broken. Restart the worker node or upgrade the SchedulerX agent to the latest version. The connection recovers automatically after either action.

My job fails with "used space beyond 90.0%!"

The disk is full. Clear disk space on the Elastic Compute Service (ECS) instance or container where the worker runs.

My job fails with "ClassNotFoundException"

The worker running the job does not have the required class. Verify that the Class full path configured for the Java job is a fully qualified class name.

If the path is correct but the class is still missing, the likely cause is an incorrect package or a misconfigured application connecting to the wrong worker. Log on to the worker and use Java decompilation to inspect the deployed classes.

My job fails with "jobInstance=xxx don't update progress more than 60s"

When a running worker fails to update its progress for 60 seconds, the server forcibly terminates it. If the worker crashed or no longer exists, you can safely ignore this error.

My job fails but no error message appears

Locate the failing worker and check its logs:

1. Go to the Instances page, click the Task instance List tab, find the job, and click Details in the Operation column. The Task instance details panel shows which worker ran the failed job. For simple jobs without task details, check the WorkAddr parameter on the Basic information tab instead.

2. Log on to the worker and open ~/logs/schedulerx/worker.log. Filter for the relevant instance:

      grep <Instance ID> worker.log

3. Examine the stack trace if an error is present.

4. If no error appears in the log, the failure is in your business logic (which did not throw an exception). Troubleshoot your business code directly.

5. If the log shows a framework exception, join DingTalk group 23103656 for technical support.

How do I find the root cause of a job failure?

The approach depends on your SchedulerX edition and job type:

My job is stuck and will not complete

First, determine whether the problem is in your business logic or in SchedulerX itself. Capture a thread stack to investigate:

• Professional Edition: Use the ThreadDump feature in the console (requires agent version 1.4.2 or later).

• Basic Edition: Log on to the stuck worker node and run:

    jstack <pid> | grep <Instance ID> -A 20

If the thread stack points to a SchedulerX issue rather than business logic, join DingTalk group 23103656 for technical support.

How do I troubleshoot slow jobs?

Enable the Tracing Analysis feature in Professional Edition. For step-by-step instructions, see Integrate tracing analysis.

My job gets stuck or slows down during a release

For distributed jobs, when a worker goes offline during a release, SchedulerX redistributes its tasks and polls for online workers. This redistribution process slows down the overall job execution.

Update the SchedulerX agent to version 1.7.9 or later to resolve this.

My job was not triggered at the expected time

For standalone jobs, check two things:

1. Whether workers appear in the worker list.

2. Whether all workers are marked as busy.

If no workers are available, see What do I do if the system displays the message "no workers available"?. If all workers are busy, see What do I do if all workers are busy?.

To catch this proactively, enable the No machine alarm available switch. For details, see Job management.

What happens if a job does not finish before its next scheduled run?

The behavior depends on the Instance concurrency setting:

How do I set up second-delay scheduling?

Set the Time type parameter to second_delay. The job runs repeatedly, with each run starting after the specified delay from the end of the previous run.

Jobs with Time type set to cron or fix_rate do not support second-delay scheduling.

How do I create a one-time job?

SchedulerX 2.0 allows you to create one-time jobs. Set the Time type parameter to one_time. One-time jobs are automatically destroyed after execution and do not retain execution records.

How do I view historical records for a one-time job?

One-time jobs are automatically destroyed after execution, so no historical records are kept. To retain execution logs for troubleshooting, enable Log Service, which stores logs for the last two weeks. For more information, see Application management.

How do I configure a job timeout?

SchedulerX supports timeout settings for jobs but not for individual tasks. Set the timeout in the console. For details, see Job management.

Where do I find advanced job management settings?

See Advanced parameters for job management.

"The number of concurrent job instances reaches the upper limit" when I click Run once

The running instances have reached the Instance concurrency limit for this job.

• If the limit is correct, no action is needed. To view running instances, go to the Tasks page, find the job, and choose More > Record in the Operation column.

• To raise the limit, go to the Task Management page, find the job, click Edit in the Operation column, and update Instance concurrency in Advanced Configuration.

My job keeps running after I stop the instance

When you stop a job instance, SchedulerX sends a Kill message to the agent. The agent stops delivering new tasks, destroys the instance context, and tears down all thread pools. However, tasks already in progress continue until they finish -- SchedulerX only interrupts their threads.

• In most cases, wait for the in-progress tasks to complete.

• To force immediate termination, add interrupt-state checks to your task processing logic so tasks exit when their thread is interrupted.

All my workers are busy

1. On the Applications page, find the application and click View instances in the Operation column. Workers in the Busy state are highlighted -- hover over Busy to see which metrics exceeded the threshold.

2. To adjust the threshold, click Edit on the same application and update the parameters in the Edit application grouping panel.

3. If the application runs in a Kubernetes container, the CPU utilization readings may be inaccurate. Enable cgroup-based metrics collection with the following parameters. For setup details, see Connect a Spring Boot application to SchedulerX.

How do I set up Tracing Analysis?

SchedulerX supports end-to-end tracing. For setup instructions, see Integrate tracing analysis.

What are the TargetMachine and Instance Parameters fields?

When you click Run once on the Tasks page, a dialog asks for TargetMachine and Instance Parameters. Instance Parameters is optional and typically used for testing.

What is the difference between job parameters and instance parameters?

Instance parameters are different from job parameters. Which parameters your code obtains depends on your business logic.

How do I retrieve job parameters or instance parameters in code?

Use getJobParameters() and getInstanceParameters() from JobContext:

@Component
public class JavaDemoProcessor extends JavaProcessor {

    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");

    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {
        LOGGER.info(JSON.toJSONString(jobContext));
        // Retrieve job parameters
        String jobParameters = jobContext.getJobParameters();
        // Retrieve instance parameters
        String instanceParameters = jobContext.getInstanceParameters();
        LOGGER.info("Job parameters:" + jobParameters);
        LOGGER.info("Instance parameters:" + instanceParameters);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}

How do I fall back to job parameters when no instance parameters are provided?

Check whether instance parameters are present. If they are, use them; otherwise, fall back to job parameters:

@Component
public class JavaDemoProcessor extends JavaProcessor {

    private static final Logger LOGGER = LoggerFactory.getLogger("schedulerxLog");

    @Override
    public ProcessResult process(JobContext jobContext) throws InterruptedException {
        String params = null;
        if (StringUtils.isNotBlank(jobContext.getInstanceParameters())) {
            params = jobContext.getInstanceParameters();
        } else {
            params = jobContext.getJobParameters();
        }
        LOGGER.info("JavaDemoProcessor params:{}", params);
        return new ProcessResult(InstanceStatus.SUCCESS);
    }

}