Multilingual sharding model

Features

The multilingual sharding model also provides the following features:

• Compatibility with the static sharding model of elastic-job.

• Support for Java, Python, Shell, and Go.

• High availability: The sharding model is based on MapReduce, which provides high availability. If a worker fails during execution, a master worker initiates a failover and reassigns the shard to another slave node.

• Throttling: The sharding model inherits the throttling feature from the MapReduce model, which allows you to control the concurrency of subtasks on a single machine. For example, if you have 1,000 shards and 10 machines, you can limit each machine to running a maximum of five shards concurrently, with the rest waiting in a queue.

• Automatic shard retry on failure: Based on the MapReduce model, the sharding model inherits the automatic retry of failed subtasks.

You can configure high availability and throttling in the Advanced Settings when you create a job. For more information, see Create a scheduled job and Parameters for advanced settings of a job.

Java sharding job

1. Log on to the MSE SchedulerX console.

2. In the top navigation bar, select a region.

3. If you are using the MSE SchedulerX console, choose Task Management in the left-side navigation pane.

4. On the Task Management page, select a namespace and click Add Task in the upper-left corner.

5. In the Add Task panel, on the Basic configuration tab, set Execution mode to Shard run, configure Sharding parameters, and then click Next Step.

   Separate sharding parameters with commas (,) or line breaks. For example: shard_id_1=parameter_1,shard_id_2=parameter_2,....

   On the Basic Information tab of the Create Job page, set Job Name (for example, sharding_job), set Job Type to Java, and enter the Full Class Name. Set Execution Mode to Sharded and Priority to Medium. In the Sharding Parameters field, enter the shard information in the specified format, and then click Next.

6. In your application code, extend JavaProcessor. Use JobContext.getShardingId() to get the shard ID and JobContext.getShardingParameter() to get the sharding parameter.

   Example:

   @Component
   public class HelloWorldProcessor extends JavaProcessor {
       @Override
       public ProcessResult process(JobContext context) throws Exception {
           System.out.println("Shard ID=" + context.getShardingId() + ", Shard Parameter=" + context.getShardingParameter());
           return new ProcessResult(true);
       }
   }

7. On the Execution List page, click Details in the Actions column of the target job to view shard details.

Python sharding job

To run distributed batch jobs for a Python application, install the Agent. You can then manage the script in SchedulerX.

1. Download the SchedulerX Agent and use it to deploy the script job.

2. Create a Python sharding job in SchedulerX. For more information, see Create a scheduled job.

   The script receives the shard ID as sys.argv[1] and the sharding parameter as sys.argv[2].

   Separate sharding parameters with commas (,) or line breaks. For example: shard_id_1=parameter_1,shard_id_2=parameter_2,....

   In the Job Configuration dialog box, set Job Type to Python. In the code editor, enter the shard processing script. Set Execution Mode to Sharded. In the Sharding Parameters field, enter the shard configuration, such as 0=hello,1=world,2=my,3=name,4=is,5=huang. Then, click Next.

3. On the Execution List page, click Details in the Actions column of the target job to view shard details.

Shell and Go sharding jobs

The process for creating Shell and Go sharding jobs is similar to that for Python sharding jobs. For more information, see Python Sharding Job.