SchedulerX:Sharding models for different programming languages

Features

The sharding models have the following features:

• The static sharding model is compatible with elastic-job.

• The sharding models support Java, Python, Shell, and Go.

• High availability: The sharding models are developed based on the MapReduce model and can also ensure high availability. If a worker encounters an error, the master worker automatically fails over the shards to another worker.

• Traffic throttling: The sharding models are developed based on the MapReduce model and also support traffic throttling. This feature allows you to control the concurrency of tasks on a single worker. For example, you have 1,000 shards and 10 workers. You can configure each worker to run at most five shards in parallel. The remaining shards must wait in the queue.

• Automatic resharding: The sharding models are developed based on the MapReduce model and also support automatic resharding. This feature is used to automatically rerun failed tasks.

You can configure the high availability and traffic throttling features in the advanced settings of a job when you create the job. For more information, see the Create a job section of the "Job management" topic and Advanced parameters for job management.

Create a Java sharding job

1. Log on to the SchedulerX console.

2. In the top navigation bar, select a region.

3. In the left-side navigation pane, click Task Management.

4. On the Jobs page, select a namespace in which you want to create a job from the Namespace to which the namespace belongs drop-down list and click Create task.

5. In the Basic configuration step of the Create task wizard, set the Execution mode parameter to Shard run and specify Sharding parameters. Then, click Next Step.

   Separate multiple sharding parameters with commas (,) or specify only one sharding parameter in each line. Example: Shard index 1=Shard parameter 1,Shard index 2=Shard parameter 2,....

   

6. Inherit the JavaProcessor in the application code, use JobContext.getShardingId() to obtain a shard index, and use JobContext.getShardingParameter() to obtain the corresponding sharding parameter.

   Sample code:

   @Component
   public class HelloWorldProcessor extends JavaProcessor {
       @Override
       public ProcessResult process(JobContext context) throws Exception {
           System.out.println("Shard index=" + context.getShardingId() + ", Shard parameter=" + context.getShardingParameter());
           return new ProcessResult(true);
       }
   }

7. On the Instances page, find the job that you want to view and click Details in the Operation column.

Create a Python sharding job

If you want to configure a Python application to perform distributed batch processing, you need to only install the SchedulerX agent. You can use SchedulerX to maintain scripts.

1. Download the SchedulerX agent and use the agent to deploy a script job.

2. Create a Python sharding job in SchedulerX. For more information, see the Create a job section of the "Job management" topic.

   sys.argv[1] indicates a shard index and sys.argv[2] indicates a sharding parameter.

   Separate multiple sharding parameters with commas (,) or specify only one sharding parameter in each line. Example: Shard index 1=Shard parameter 1,Shard index 2=Shard parameter 2,....

3. On the Instances page, find the job that you want to view and click Details in the Operation column.

Create a Shell sharding job or a Go sharding job

You can create Shell sharding jobs or Go sharding jobs in a similar way as you create Python sharding jobs. For more information, see the Create a Python sharding job section of this topic.