Multi-language sharding model - Microservices Engine - Alibaba Cloud Documentation Center

SchedulerX lets you schedule a variety of tasks, including timed jobs, orchestrations, and historical data reprocessing. To help you process large datasets, SchedulerX provides a multi-language Sharding Model that supports Java, Python, Shell, and Go.

Background information

The Sharding Model uses static sharding, where the number of shards is predefined and distributed evenly among workers. For example:

Multiple machines process a fixed set of 128 sharded database tables in parallel.
One hundred shards, based on user ID prefixes from 00 to 99, are processed in parallel across multiple machines.

Features

The multi-language sharding model also has the following features.

Compatible with the static sharding model of Elastic-Job.
Supports four languages: Java, Python, Shell, and Go.
High Availability (HA): The Sharding Model is built on the MapReduce model and inherits its HA features. If a worker fails during execution, SchedulerX automatically reassigns the shard to a healthy worker node.
Throttling: This feature lets you control the concurrency of subtasks on a single machine. For example, if you have 1,000 shards distributed across 10 machines, you can limit the concurrency to a maximum of 5 shards running concurrently per machine, while others wait in a queue.
Automatic shard retries: The model automatically retries failed subtasks.

You can configure HA and Throttling in the Advanced Settings when you create a task. For more information, see Create a scheduling task and Advanced parameters for job management.

Note

The multi-language Sharding Model requires client version 1.1.0 or later.

Java sharding task

Log on to the MSE SchedulerX console.
In the top menu bar, select a Region.
In the left-side navigation pane, select Task Management.
On the Task Management page, select the target Namespace and click Create Task in the upper-left corner.
In the Create Task panel, on the Basic Information tab, set Execution Mode to Sharding, configure the Sharding Parameters, and then click Next.
Separate sharding parameters with commas (,) or line breaks. For example: shard_id1=parameter1,shard_id2=parameter2,....

In your application code, extend the JavaProcessor class. Call JobContext.getShardingId() to get the shard ID and JobContext.getShardingParameter() to get the shard parameter.

Example:

@Component
public class HelloWorldProcessor extends JavaProcessor {
    @Override
    public ProcessResult process(JobContext context) throws Exception {
        System.out.println("shard id=" + context.getShardingId() + ", shard parameter=" + context.getShardingParameter());
        return new ProcessResult(true);
    }
}

On the Execution List page, find the task and click Details in the Actions column to view the shard details.

Python sharding task

To run distributed Python batch jobs, install the agent to allow SchedulerX to manage your scripts.

Download the SchedulerX agent and use it to deploy script tasks.
Create a Python sharding task in SchedulerX. For more information, see Create a scheduling task.
sys.argv[1] is the shard ID, and sys.argv[2] is the shard parameter.
Separate sharding parameters with commas (,) or line breaks. For example: shard_id1=parameter1,shard_id2=parameter2,....
On the Execution List page, find the task and click Details in the Actions column to view the shard details.

Shell and Go sharding tasks

The process for creating Shell and Go sharding tasks is similar to creating a Python task.Python sharding task