During peak scheduling periods, a burst of jobs starting simultaneously can overwhelm backend resources and cause failures. SchedulerX provides two mechanisms to prevent this:
Concurrency limiting -- Cap the number of jobs that run at the same time within an application. Excess jobs queue automatically instead of being dropped.
Priority-based preemption -- Assign priorities to jobs so that higher-priority jobs run ahead of lower-priority ones when a queue slot opens.
Use these mechanisms independently or combine them to build a priority queue that keeps the system stable while ensuring mission-critical jobs run first.
When to use each mechanism
| Scenario | Recommended approach | Why |
|---|---|---|
| Hundreds of daily batch jobs start at 00:00 and overload the system | Concurrency limiting only | Smooths out execution by queuing excess jobs |
| Real-time data sync must run before nightly reports | Concurrency limiting + priority-based preemption | High-priority jobs preempt lower-priority ones in the queue |
| Burst scheduling during peak hours, but individual jobs complete quickly | Concurrency limiting only | Prevents resource exhaustion without adding priority complexity |
| Mixed workloads where some jobs are time-sensitive and others are not | Concurrency limiting + priority-based preemption | Guarantees time-sensitive jobs get the next available slot |
Prerequisites
Before you begin, make sure you have:
A SchedulerX application group. See Create an application group
At least one job created in the application group
Set a concurrency limit for an application
Limit how many jobs run concurrently within an application. When the limit is reached, new jobs wait in the queue and are not discarded.
Open your application group and go to the Advanced Configuration section.
Turn on Flow Control. Flow Control is turned off by default.
Set the Number of concurrent task instances parameter to the maximum number of jobs that can run at the same time.
To verify the configuration, create three jobs in the application group, then click Run once in the Operation column for each job.
In the left-side navigation pane, click Instances, then click the Task instance List tab. The first triggered job (hello_jobA) is running. The remaining jobs (hello_jobB and hello_jobC) wait in the queue.
After hello_jobA completes, hello_jobB moves from the queue into the execution slot automatically.
Set up priority-based preemption
Combine a concurrency limit with job priorities so that higher-priority jobs preempt lower-priority ones when a queue slot becomes available.
How priority preemption works
This mechanism follows the same principle as Yet Another Resource Negotiator (YARN) priority queues, where resources are allocated based on job priority levels.
When multiple jobs are queued and a slot opens, SchedulerX selects the job with the highest priority -- not the one that was submitted first.
Assign priorities and test preemption
Assign a priority to each job. Jobs with higher priorities are scheduled first within the same application.
Turn on Flow Control for the application (for example,
dts-all.hxm) and set Number of concurrent task instances to1so that only one job runs at a time. Then create three jobs with high, medium, and low priorities.
Trigger the jobs in this order: medium priority, low priority, high priority.
Click Instances in the left-side navigation pane, then click the Task instance List tab. Because no job was in the queue when the medium-priority job was triggered, it starts running immediately.
After the medium-priority job completes and a slot opens, the high-priority job preempts the low-priority job and runs next -- even though the low-priority job was submitted earlier.
Usage notes
Queue-based throttling works best for burst scheduling scenarios where many jobs are triggered in a short window. If your application continuously schedules jobs every minute with heavy workloads, the queue can build up. For high-frequency scheduling, use client-side throttling or scale your processing capacity instead.
Priority settings are scoped to a single application. A high-priority job in one application does not affect job scheduling in other applications.
FAQ
Do priority settings in my application affect other applications?
No. Job priorities take effect only within the application where they are configured. They have no impact on scheduling in other applications.
Should I use queue-based throttling for jobs that run every minute?
Not if the per-minute workload is consistently heavy. Queue-based throttling is designed for burst scenarios where a spike of jobs hits the system at once. For minute-level jobs with sustained high volume, use client-side throttling or scale your processing infrastructure to match the load.