SchedulerX:SchedulerX SDK for Java

Feature

Type

Description

References

Workflow

New feature

Workflow IDs can be obtained by using JobContext.

JobContext parameters

Optimized SDK

Optimized feature

• Unnecessary dependencies are removed from the SchedulerX SDK.

• Log4j2.xml is removed from the SchedulerX SDK, which prevents log configuration conflicts.

None

Feature

Type

Description

References

Kubernetes job

New feature

Job parameters can be read from environment variables.

None

XXL-JOB log

New feature

Log isolation by application is supported.

None

Security and log display

Fixed issue

• Security vulnerabilities in Netty are fixed.

• Security vulnerabilities in Logback are fixed.

• The following issue is fixed: Logs of the SchedulerX framework cannot be displayed due to log conflicts.

None

Feature

Type

Description

References

Graceful shutdown

New feature

The graceful shutdown feature is supported. An application can be shut down only after all the ongoing jobs of the application are complete. This ensures service availability during the release or restart process.

Gracefully shut down jobs and tasks executed in an application

Push model job

New feature

Push model jobs are optimized to support the configuration of job distribution rate.

None

H2 database

New feature

H2 databases are optimized to support the configurations of username and password.

• spring.schedulerx2.h2DatabaseUser: the username of the H2 built-in database.

• spring.schedulerx2.h2DatabasePassword: the password of the H2 built-in database.

Connect a Spring Boot application to SchedulerX

Business log

Optimized feature

Business logs can be isolated by instance group.

None

Broadcast job

Optimized feature

Broadcast jobs are optimized to support sorting and distribution by worker IP address.

None

Spring job

Optimized feature

• Spring jobs are optimized to support Spring 4.

• You can specify scheduling methods for Spring jobs to return any processing results.

Spring jobs

Optimized SDK

Optimized feature

• ScheudlerX SDK for Java is optimized to prevent heartbeat threads from being repeatedly created when Spring containers are repeatedly initialized.

• Akka Actor supports automatic restart upon unexpected termination.

None

Job execution

Fixed issue

• The following issue is fixed: An exception is returned after Spring fails to read a processor bean.

• The following issue is fixed: Map jobs are stuck in V 1.10.x.

• The following issue is fixed: The instance of a broadcast job cannot be terminated when an exception occurs during the preprocessing of the broadcast job.

• The following issue is fixed: An alert is triggered due to a job execution timeout error caused by residual threads when the job instance is stopped unexpectedly.

• The following issue is fixed: The status of an application is unknown or stuck upon application startup in the shared thread pool mode.

• The following issue is fixed: The access configurations cannot be loaded as expected when multiple instance groups or AppKeys contain special characters.

• The following issue is fixed: You cannot use the features of the Professional Edition of a visual MapReduce job.

None

Feature

Type

Description

References

Template for Kubernetes script jobs

New feature

The following types of templates are supported for Kubernetes script jobs:

• Standard templates: Images and image pull policies can be modified in standard templates.

• Custom templates: Custom templates are compatible with native Kubernetes. To use a custom template, you must update schedulerx2-plugin-kubernetes to V1.0.5 or later.

None

Distribution method for Map jobs

New feature

The distribution of Map jobs can be started from a random position. To use this feature, you must set the spring.schedulerx2.mapMasterDispatchRandom parameter to true. The default value of this parameter is false.

None

Built-in parameter for Shell jobs

New feature

The following regular built-in parameters are supported for Shell jobs:

• schedule.timestamp: the time when a job is scheduled.

• data.timestamp: the time offset of a job when the job is scheduled.

None

Feature

Type

Description

References

Adjustment for execution thread models

New feature

Adjustment for execution thread models is supported.

• Execution thread models can be switched to the unshared thread pool mode.

• Thread pool parameters can be shared by models in shared thread pool mode.

  • spring.schedulerx2.shareContainerPool=true: enables the shared thread pool.

  • spring.schedulerx2.sharePoolSize=32: the global size of the shared thread pool. Default value: 64.

  • spring.schedulerx2.sharePoolQueueSize=20480: the queue size of the shared thread pool. Default value: Integer.MAX_VALUE.

None

Idle master node and retry on failure for broadcast jobs

New feature

The following parameters can be specified to support the idle master node and retry on failures:

• spring.schedulerx2.broadcastMasterExecEnable=false: specifies whether the master node is required to execute a broadcast job. Default value: true.

• spring.schedulerx2.broadcastDispatchRetryTimes=5: the number of retries that are allowed after a broadcast job fails to be distributed.

None

Map job

Optimized feature

The performance of Map jobs during large-scale application release and scale-in is optimized.

None

Heartbeat checkpoint for application startup

Optimized feature

The heartbeat checkpoint for application startup is optimized to prevent time-consuming execution of jobs that is caused by long startup time of containers.

None

Job execution

Fixed issue

• The following issue is fixed: A job is rejected after it runs for a long time in shared thread pool mode.

• The following issue is fixed: Empty pointer logs are occasionally generated when job status is being reported.

• The following issue is fixed: The number of broadcast job threads continues to grow after the release of V1.7.X.

• The following issue is fixed: Memory leaks occur in shared thread pool mode.

• The following issue is fixed: A dynamic proxy cannot implement the Reduce method.

• The following issue is fixed: A second-level job is stuck because specific responses are missing after the execution responses are combined.

• The following issue is fixed: An error occurs when Diamond reads environment addresses.

• The following issue is fixed: The heartbeat timeout period is excessively long when you connect to Alibaba Cloud.

• The following issue is fixed: Alerts are triggered in the log4j log format.

• The following issue is fixed: If a standalone job fails to be distributed, the job is stuck.

• The following issue is fixed: The new version is incompatible with the previous ClassName:BeanId configuration mode.

• The following issue is fixed: The maximum file size of a Logback standalone log cannot be increased.

None

Type

Description

References

New feature

• Native and scheduled Spring jobs can be integrated.

• End-to-end tracing is supported.

• Spring jobs

• Integrate tracing analysis

Optimized feature

The response is accelerated when an agent is stopped or disconnected.

None

• The following issue is fixed: The Out of Memory (OOM) error occurs. The OOM error occurs when a large number of Simple Log Service logs are generated for a job that is scheduled frequently.

• The following issue is fixed: Large latency occurs when many second-delay jobs are run together.

None

Type

Description

References

Optimized feature

The following issue is fixed: High CPU usage is caused by second-delay broadcast detection threads.

None

Type

Description

References

New feature

Kubernetes job scheduling capabilities can be integrated.

None

Optimized feature

• The processor class can be used to configure the Spring Aspect Orient Programming (AOP) proxy.

• The content in the error log that is generated occasionally for an available worker monitor timer is optimized.

• Logs of second_delay jobs can be collected.

None

Type

Description

References

New feature

You can dynamically enable or disable the logging feature.

None

Optimized feature

• The following issue is fixed: V1.5.0.x is incompatible with Spring Boot 2.0.3.

• The following issue is fixed: An error may be reported during JobSyncService Bean initialization in V1.5.0.x.

None

Type

Description

References

New feature

Spring Boot declarative jobs are supported. You can declare namespaces, applications, jobs, and alerts by using configuration files. This facilitates job management. You can modify the configuration files based on your business requirements and launch your application with a few clicks in all environments.

Periodic jobs for Spring Cloud applications

Visualized MapReduce jobs, formerly known as parallel computing jobs, are supported. You can view the tasks of a specific visualized MapReduce job, view the logs of visualized MapReduce jobs, and rerun visualized MapReduce jobs. You can also add custom tags to tasks and filter tasks by tag.

Enterprise-class distributed batch processing solution

Type

Description

References

New feature

The logging feature is supported by SchedulerX V2.0. To enable the logging feature, you need to only add the Log4j or Logback configuration to the corresponding job. Then, you can view the log that is generated each time the job is triggered in the SchedulerX console. Distributed jobs also support the logging feature.

Integrate Simple Log Service with SchedulerX 2.0

You can view the stacks based on which jobs are run in the SchedulerX console. This helps troubleshoot jobs that are stuck. You must update your agent to 1.4.0 or a later version before you can view stacks.

None

SchedulerX can be interfaced with the executor that is developed for elastic jobs.

None

Type

Description

References

New feature

• DataWorks jobs are supported.

• schedulerx2-plugin-xxljob is added to support the XXL-JOB interface.

• DataWorks jobs

• XXL-JOB jobs

Type

Description

References

Optimized feature

The default dependency Log4j 2 is updated to V2.15.0.

None

Type

Description

References

New feature

Second-delay jobs enhanced in standalone mode: Second-delay jobs can be distributed to different idle workers in standalone mode. To distribute second-delay jobs to different idle workers in standalone mode, set the spring.schedulerx2.enableSecondDelayStandaloneDispatch parameter to true. Default value of the parameter: false.

None

Optimized feature

• The response is accelerated when an agent is stopped or disconnected.

• The following issue is fixed: The running workers send heartbeats only to the specified scheduled service if an agent belongs to multiple instance groups.

• The following issue is fixed: Threads are occasionally interrupted or the null pointer exception occasionally occurs when SchedulerX schedules second-delay jobs.

• The following issue is fixed: Jobs become stuck and time out if the SchedulerX agent is integrated with a Spring Cloud application.

• The following issue is fixed: Jobs are interrupted if SchedulerX runs the jobs in parallel in broadcast mode.

• The following issue is fixed: SchedulerX fails to update the worker list when SchedulerX runs second-delay jobs.

• The following issue is fixed: The scan thread is not terminated after a second-delay job that is scheduled in broadcast mode executes the terminate instruction.

• The following issue is fixed: A second-delay job that is scheduled in broadcast mode may become stuck when a large number of workers publish the job in batches.

None

Type

Description

References

Optimized feature

The following issue is fixed: Jobs become stuck and time out if the SchedulerX agent is integrated with a Spring Cloud application.

None

Type

Description

References

New feature

Second-delay jobs enhanced in standalone mode: Second-delay jobs can be distributed to different idle workers in standalone mode. To distribute second-delay jobs to different idle workers in standalone mode, set the spring.schedulerx2.enableSecondDelayStandaloneDispatc parameter to true. Default value of the parameter: false.

None

Optimized feature

The following issue is fixed: Jobs are interrupted if SchedulerX runs the jobs in parallel in broadcast mode and the list of available workers is not updated.

None

Type

Description

References

Optimized feature

The following issue is fixed: The scan thread is not terminated after a second-delay job that is scheduled in broadcast mode executes the terminate instruction.

None

Type

Description

References

Optimized feature

The following issue is fixed: A second-delay job that is scheduled in broadcast mode may become stuck when a large number of workers publish the job in batches.

None

Type

Description

References

New feature

• Enhanced broadcast mode: postProcess can be used to retrieve the status (JobContext.getTaskStatuses()) and execution results (JobContext.getTaskResults()) of all workers.

• Enhanced sharding broadcast mode: JobContext can be used to retrieve the number (shardingNum) and IDs (sharingId) of shards. You can use the index of your worker to process data in a distributed manner.

• Enhanced sharding mode: JobContext can be used to retrieve the number (shardingNum) of all shards.

None

Type

Description

References

New feature

A failover switch is added to allow you to enable or disable the failover feature.

None

Optimized feature

• The following issue is fixed: The socket may be disclosed if the heartbeat probe fails.

• The following issue is fixed: MapTaskMaster frequently performs failovers when MapTaskMaster needs to handle a large number of tasks.

None

Type

Description

References

New feature

Second-delay jobs can be triggered at a millisecond granularity.

None

Optimized feature

• The MapReduce model is optimized. Root tasks are scheduled only to the master worker. This can help you identify the cause when exceptions occur.

• The SchedulerX agent is no longer strongly reliant on Log4j 2. If you use Logback, you can remove the Log4j and Log4j 2 dependencies.

None

Type

Description

References

New feature

• The progress of jobs that are scheduled in broadcast mode can be reported.

• Heartbeat logging can be enabled or disabled.

None

Optimized feature

• The interval of heartbeat probes is changed to 10 seconds.

• The following issue is fixed: Broadcast jobs may be stuck.

None

Type

Description

References

Optimized feature

The following issue is fixed: Jobs cannot continue to be scheduled due to traffic surges during the persistent connection caused by heavy workloads on the SchedulerX client.

None

Type

Description

References

New feature

• The actual CPU utilization of a container in a Container Service for Kubernetes (ACK) cluster can be collected. However, you must add the corresponding configurations. For example, if you use starter, you must perform the following operations: Set the spring.schedulerx2.enableCgroupMetrics parameter to true and the spring.schedulerx2.cgroupPathPrefix parameter to /sys/fs/cgroup/cpu/. If the /sys/fs/cgroup/cpu/ path exists on your container, you do not need to specify the spring.schedulerx2.cgroupPathPrefix parameter. Otherwise, you must specify the actual path of cgroup.

• Tags are supported. You can add custom tags for client startup. For example, if you use starter, you can set the spring.schedulerx2.labels parameter to xxx. You can specify a worker that has specific tags to run a job in the Task Management module. This feature is suitable for canary release and cell-based management scenarios.

None

Optimized feature

• The heartbeat thread and Akka thread of the SchedulerX agent can run independently. This ensures that heartbeat probes are not affected when the SchedulerX agent is busy.

• h2 is removed from shade. shade h2 may cause the failure to load the MySQL driver.

None

Type

Description

References

New feature

• The shared container pool feature is supported. All jobs that are scheduled by the SchedulerX agent can share the same thread pool. This increases the performance and stability of the SchedulerX agent when the agent needs to schedule a large number of jobs in parallel.

• Shade Scala is supported to eliminate scala conflicts.

• Authentication based on AppKeys is supported in the public cloud.

• The MapReduce model is enhanced and allows you to specify whether to perform the Reduce method only after all tasks are complete.

• The AppKey can be specified in the -Dschedulerx.appKey parameter on the SchedulerX agent.

None

Optimized feature

• The Netty thread pool of the SchedulerX agent is optimized.

• Jobs that are scheduled in broadcast mode may become stuck.

• The following issue is fixed: The AccessKey ID or AccessKey secret of the Simple Log Service feature is leaked.

None

Type

Description

References

New feature

• More than 100,000 jobs are supported by an application that is deployed in the public cloud.

• A logging switch is added to allow you to enable or disable logging for the SchedulerX agent.

• The status of a job can be configured if you use the SchedulerX API to create the job.

• The diamond-client, logger.API, and Log4j dependencies are removed.

None

Optimized feature

• The following issue is fixed: The SchedulerX agent disconnects from the server if you disable Internet access for the agent.

• The following issue is fixed: EDAS applications cannot read the AccessKey secret.

None

Type

Description

References

New feature

• Namespaces can be created.

• Multiple SchedulerX workers can be initialized at the same time.

• The MapReduce model is enhanced.

• The Reduce method can be performed even if a task fails.

• JobContext.getTaskStatuses can be used to obtain the status of a task. The key of the Map<Long,TaskStatus> structure indicates the ID of the task and the value indicates the status of the task.

None

Optimized feature

• The following issue is fixed: A null pointer error occurs if ProcessResult returns an empty result.

• The following issue is fixed: A job becomes stuck if thread-dispatcher-delivery becomes hung.

None

Type

Description

References

New feature

shade protobuf and netty from AccessKey IDka are used to eliminate more than 90% of JAR package conflicts when you connect an agent to SchedulerX.

None

Optimized feature

AppKeys are not supported when you create multiple groups.

None

Type

Description

References

New feature

• Support for sharding models in multiple languages (similar to elastic-job): ShedulerX2.0 supports sharding models in multiple languages.

• The AppKey of a group can be returned when you use the SchedulerX API to create the group.

• Job instances that are in the Successful state can be rerun. Job instances in a workflow can rerun their jobs and their downstream jobs.

None

Optimized feature

• The performance of the server is optimized. The method that the server uses to communicate with the SchedulerX agent is changed from synchronous to asynchronous. The configurations of the default dispatcher of Akka are optimized.

• The performance of heartbeat probes provided by the SchedulerX agent V1.1.0 is increased by three times.

• The Task Management page in the SchedulerX console is redesigned to display more information.

• Global tasks may not take effect if the pull model is used to distribute jobs.

• The isolated cell environment is improved. The SchedulerX agent may fail to be launched in an isolated cell environment if no domain is specified.

None

Type

Description

References

New feature

• The BlockAppStart setting is added. The setting specifies whether to block the startup of the application when SchedulerX fails to start up. The default value is true.

• An API operation is added to allow you to query the status of a workflow. The API operation is GetWorkflowInstanceRequest.

• The jobName field is added to jobContext. This allows you to obtain the name of a job during the job run.

None

Optimized feature

• The following issue is fixed: The result of Hessian deserialization on BigDecimal is 0.

• The following issue is fixed: A LocalDateTime error occurs after Hessian deserialization.

• The following issue is fixed: Tasks of a job fail to be scheduled to specified workers. The following issue is fixed: If a job runs for a specific period of time, the tasks of the job are scheduled to workers that you did not specify.

• The following issue is fixed: The springContext.getBean of the SchedulerX agent throws an AnnotationConfigApplicationContext has not been refreshed yet exception.

• The following issue is fixed: The ServeletWebServer of Spring Boot is terminated if the configuration of the implementation class of the job is invalid. As a result, the web service is shut down but the application process continues to run.

• The following issue is fixed: A job becomes stuck if the system enables the user.dir='/' variable.

• The following issue is fixed: The springContext.getBean of the SchedulerX agent throws an AnnotationConfigApplicationContext has been closed already exception.

• The following issue is fixed: The values of workerId that are generated by the SchedulerX agent may be duplicate. As a result, the job may be scheduled to workers that do not belong to the application to which the job belongs.

• The following issue is fixed: Custom class loaders are not supported by Spring applications.

• The following issue is fixed: The counter that counts the number of times a second-delay job in broadcast mode is triggered does not display the correct value.

• The following issue is fixed: jobContext.getScheduleTime is not updated when a second-delay job is periodically scheduled.

None

Type

Description

References

New feature

• Important changes: JobProcessor.postProcess is reconfigured. ProcessResult is added to return values. You need to modify the code of the job if you use the original postProcess interface.

• The broadcast mode is enhanced. BroadcastJobProcessor supports preProcess and postProcess. All workers must perform preProcess once before they perform process. All workers must perform postProcess once after they perform process.

• JobContext.getTaskAttempt can be used to retrieve the number of times that SchedulerX attempts to run the current task.

• Custom listening ports can be specified for the SchedulerX agent. Example: SchedulerxWorker.setPort.

• JobProcessor can be implemented by Java jobs. Java jobs no longer need to inherit JavaProcessor.

None

Optimized feature

• The following issue is fixed: Tasks whose taskId=1 cannot be automatically retried.

• The following issue is fixed: The cause of failure is not displayed if the root task of a distributed job fails.

• The following issue is fixed: Tasks in the task list of a job that is scheduled in parallel computing mode cannot be retried.

None

Type

Description

References

Optimized feature

This SchedulerX agent version is compatible with the SchedulerX 1.0 (DTS) interface. The agent version cannot rely on both the schedulerx-client and schedulerx-worker packages. The agent version can rely on only the schedulerx-worker package. Therefore, you must migrate all Data Transmission Service (DTS) jobs to SchedulerX 2.0.

None

Type

Description

References

New feature

• shade: aliyun-log, commons-validator, gson, fastjson, guava, and commons-collections are added.

• errorMsg can be displayed in logs in the SchedulerX console if you use ProcessResult(false,errorMsg) to return execution results.

• The performance of at-least-once-delivery is optimized.

• The following issue is fixed: Duplicate messages are no longer submitted in heavy load scenarios. Duplicate messages cause second-delay jobs to become stuck or interrupt application threads.

None

Optimized feature

• The following issue is fixed: Jobs that are scheduled in broadcast mode become stuck.

• The following issue is fixed: Second-delay jobs become stuck.

• The following issue is fixed: at-least-once-delivery may cause SchedulerX to keep retrying specific tasks.

• The following issue is fixed: The SchedulerX agent fails to start up if logcollector fails to initialize.

• The following issue is fixed: A workflow cannot be recovered because the database cleans the job instances in the workflow.

• The following issue is fixed: kill is not supported if the application is launched by using Spring.

None

Type

Description

References

New feature

• The execution results of all tasks can be returned by the MapReduce model and processed by using the Reduce method.

• The pull model is supported for jobs that use distributed programming models. This model can eliminate the barrel effect that occurs due to the limited performance of standalone computing. This model allows workers to dynamically scale out to pull tasks.

• The global degree of concurrency can be configured for tasks if you use the pull mode. This helps you limit the number of concurrent tasks.

• wfInstanceId is added to the JobContext.

• An exception is thrown when the SchedulerX agent fails to start up. This helps you identify the error at the earliest opportunity. If the SchedulerX agent fails to start up, the startup of the Java virtual machine (JVM) is blocked.

• The SchedulerX agent prints the JAR version and path on which Maven relies when the SchedulerX agent starts up. This helps you troubleshoot JAR package conflicts.

• Distributed programming models can return details about tasks. You can view the queue of tasks that are cached on each worker.

None

Optimized feature

• Worker can be automatically recovered because the server load is high and the client is restarted differently.

• Distributed programming models can return details about tasks. You can view the number tasks that run on each worker.

• The server automatically cleans the resources on slave nodes to prevent memory leaks if the master node fails.

None

Type

Description

References

New feature

• The following expressions are supported when you create a periodic scheduling job: crontab and fixed_rate.

• Workflows are supported. You can create workflows to orchestrate jobs.

• Second-delay expressions are supported. You can use second-delay expressions to create second-delay jobs.

• Jobs that use the following programming languages are supported: Java, Shell, Python, and Go.

• The following execution modes are supported: standalone, broadcast, parallel computing, memory grid, and grid computing.

• The following distributed programming models are supported: Map and MapReduce.

• SchedulerX can be configured to automatically retry failed job instances or tasks. By default, this feature is disabled.

• Time-series data is supported and the outputs of jobs can be updated.

None