Batch Compute:Use cluster

1. Differences

2. How to use a cluster

2.1. Use Python SDK

# AutoCluster is not required if you specify a cluster ID.
job_desc['DAG']['Tasks']['my-task']['ClusterId'] = "cls-xxxxxxx"

2.2. Use Java SDK

The following is a code snippet:

// AutoCluster is not required if you specify a cluster ID.
desc.setClusterId("cls-xxxxxxx");

2.3. Use command line tool

# Specify a cluster ID
bcs sub "python main.py" -c cls-0101010299123

3. How to use AutoCluster

3.1. Use Python SDK

...
autoCluster = {
  'ImageId': 'img-ubuntu',
  'InstanceType': 'ecs.sn1.medium'
}
...
job_desc['DAG']['Tasks']['my-task']['AutoCluster'] = autoCluster
...

3.2. Use Java SDK

The following is a code snippet:

AutoCluster autoCluster = new AutoCluster();
autoCluster.setImageId("img-ubuntu");
autoCluster.setInstanceType("ecs.sn1.medium");

TaskDescription desc = new TaskDescription();

desc.setAutoCluster(autoCluster);

3.3. Use command line tool

# Use AutoCluster
bcs sub "python main.py" -c img=img-ubuntu:type=ecs.sn1.medium

FAQ

Should AutoCluster or Cluster be used?

If you specify a cluster ID when submitting a job, the job is dispatched to the specified cluster to run.

If you do not specify a cluster, you can use the AutoCluster configuration. In this case, specify an image and an instance type. The corresponding cluster is automatically created when tasks start to run and is automatically released after tasks are completed.

When do I use a cluster?

Use a cluster if you have a large number of jobs to run.

For example, if you have 100 jobs, create a cluster with 10 VMs and submit all these jobs to the cluster. The system automatically runs the tasks one by one, and you only need to wait until all the tasks are completed. After all the tasks are completed, manually release (delete) the cluster. This method saves time and cost.

When do I use an AutoCluster?

To use AutoCluster, specify the number of instances and instance capacity when submitting jobs. The system automatically creates a cluster when running tasks of the job and releases the cluster after completing all tasks.

You can use AutoCluster if you can accept a long waiting time or have only a few jobs.