SchedulerX:Advanced parameters for job management

Parameter

Applicable execution mode

Description

Default value

Task failure retry count

All modes

The number of automatic retries if a job fails.

0

Task failure retry interval

All modes

The interval between two consecutive retries. Unit: seconds.

30

Task concurrency

All modes

The number of instances that run the same job at the same time. A value of 1 specifies that concurrent execution is not allowed.

1

Cleaning strategy

All modes

The cleanup policy for job execution history.

Keep last N entries

Retained Number

All modes

The number of retained job execution records.

300

Number of single-machine concurrent subtasks

• Visual MapReduce

• MapReduce

• Shard run

The number of tasks that concurrently run on a single worker in a distributed model. To speed up the execution, you can specify a larger value. If the downstream or the databases cannot withstand the value that you specified, you can specify a smaller value.

5

Number of failed retries of subtasks

• Visual MapReduce

• MapReduce

• Shard run

The number of automatic retries if a task fails in a distributed model.

0

Sub-task failure retry interval

• Visual MapReduce

• MapReduce

• Shard run

The interval between two consecutive retries if a task fails in a distributed model. Unit: seconds.

0

Subtask Failover Strategy

• Visual MapReduce

• MapReduce

• Shard run

Specifies whether to distribute a task to a new worker after the worker fails to execute the task and is stopped. If you turn on the switch, the system may execute a task more than once when a failover is triggered. You must implement the idempotence of tasks.

The master node participates in the execution

• Visual MapReduce

• MapReduce

• Shard run

Specifies whether the master node participates in the execution of tasks. At least two workers must be available to run tasks. If an extremely large number of tasks exist, we recommend that you turn off the switch.

Subtask distribution method

• Visual MapReduce

• MapReduce

• Shard run

• Push model: Tasks are evenly distributed to workers.

• Pull model: Every worker pulls tasks. The Wooden Bucket Theory is not applicable to this model. During the pull process, all tasks are cached on the master node. This puts pressure on the memory. We recommend that you do not distribute more than 10,000 tasks at a time.

Push model

distribution policy

• Visual MapReduce

• MapReduce

The task distribution policy. Default value: Polling Scheme.

• Polling Scheme: The system evenly distributes the same number of tasks to each worker. This policy is suitable for scenarios in which each worker requires almost the same amount of time to process a task.

• WorkerLoad optimal strategy: The master node automatically detects the loads of workers. This policy is suitable for scenarios in which a large difference exists between the amount of time each worker requires to process a task.

Distribution rate

• Visual MapReduce

• MapReduce

The task distribution rate. You can specify the number of tasks to be distributed per second or per minute.

Number of subtasks pulled per time

• Visual MapReduce

• MapReduce

• Shard run

The number of tasks that a slave node pulls from the master node at a time. This parameter is displayed only if the Subtask distribution method parameter is set to Pull model.

5

Subtask queue capacity

• Visual MapReduce

• MapReduce

• Shard run

The size of the queue that caches tasks on a slave node. This parameter is displayed only if the Subtask distribution method parameter is set to Pull model.

10

Global concurrency of subtasks

• Visual MapReduce

• MapReduce

• Shard run

The total number of concurrent tasks on all workers in the pull model. This parameter helps you limit the number of concurrent tasks. This parameter is displayed only if the Subtask distribution method parameter is set to Pull model.

1000