All Products
Search
Document Center

Submit job

Last Updated: May 11, 2018

This article describes the commands used to submit various types of jobs.

1. Submit a simplest job

If you have configured the default image, instance type, and network type according to Preparations, use the following simple command to submit a job.

  1. bcs sub "echo 123" # Submit and run a single-task job "echo 123".

If default settings are unavailable for the BatchCompute-cli tool, you must specify more parameters when submitting a job.

  1. bcs sub "echo 123" -t ecs.sn1ne.large -i img-ubuntu-vpc --vpc_cidr_block 192.168.0.0/16

For the remaining examples in this article, it is assumed that you have configured the default settings.

2. Submit a Python script job

  1. bcs sub "python test.py" -p ./test.py # "-p" indicates that test.py is packed into worker.tar.gz and then uploaded to OSS before the job is submitted.

3. Submit a directory (multi-file) job

Generally, the src directory contains multiple files, for example:

  1. src
  2. |-- test.py
  3. |-- dep.py

Test.py must depend on dep.py. Therefore, the entire directory can be packed.

  1. bcs sub "python test.py" -p ./src/ # Pack all files under the src directory into worker.tar.gz, upload it to the OSS, and then submit the job.

Specify multiple files at a time and separate these files by commas (,):

  1. cd src # Switch to the src directory.
  2. bcs sub "python test.py" -p test.py,dep.py # Pack the two files into worker.tar.gz.

If you do not switch to the src directory, submit the job as follows:

  1. bcs sub "python test.py" -p src/test.py,src/dep.py # Pack the two files into worker.tar.gz.

Then, use the following command to view the content of worker.tar.gz:

  1. tar -tvf worker.tar.gz

The content of worker.tar.gz is as follows:

  1. test.py
  2. dep.py

4. Submit a job by mounting a task program

Assume that you have uploaded test.py to oss://mybucket/test/test.py.You can mount the OSS directory “oss://mybucket/test/“ to a VM local directory, for example, “/home/admin/test/“. Then use the command “python /home/admin/test/test.py” to run the program.

Submit a job as follows:

  1. bcs sub "python /home/admin/test/test.py" -r oss://mybucket/test/:/home/admin/test/

The parameter -r oss://mybucket/test/:/home/admin/test/ indicates read-only mounting and “oss://mybucket/test/“ is mounted onto “/home/admin/test/“.

In this way, you do not need to pack test.py and dep.py into worker.tar.gz.

For more information about how to submit a job, run “bcs sub -h” to view the help information.

Other tasks

1. Mount the input data

Assume that your data has been uploaded to the directory “oss://my-bucket/inputs/“.

  1. bcs sub "python test.py" -p ./src/ -r oss://my-bucket/inputs/:/home/admin/inputs/
  • “-r” indicates read-only mounting and “oss://my-bucket/inputs/“ is mounted to “/home/admin/inputs/“. In this way, your program can read files under the directory “/home/admin/inputs/“ same as reading local files.
  • All mounted directory cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.
  • To mount multiple directories, separate them by using commas (,). For example, -r oss://my-bucket/inputs/:/home/admin/inputs/,oss://my-bucket/inputs2/:/home/admin/inputs2/.

2. Use the mounting method to automatically upload the program running results

Your program will write the running results to the directory “/home/admin/outputs/“. To upload all data under this directory to OSS, run the following command:

  1. bcs sub "python test.py" -p ./src/ -r oss://my-bucket/inputs/:/home/admin/inputs/ -w oss://my-bucket/outputs/:/home/admin/outputs/
  • “-w” indicates writable mounting and all data written to this directory will be automatically uploaded to the corresponding OSS directory after the program finishes running.
  • All mounted directory cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.
  • To mount multiple directories, separate them by using commas (,). For example, -w oss://my-bucket/outputs/:/home/admin/outputs/,oss://my-bucket/outputs2/:/home/admin/outputs2/.

3. Use the custom image and instance type

  1. bcs sub "python test.py" -p ./src/ -c img=img-ubuntu-vpc:type=ecs.sn1ne.large
  • “-c” indicates that a cluster is used. Next to “-c”, you can specify the cluster ID or the AutoCluster configuration. The AutoCluster configuration format is as follows: img=${ImageId}:type=${InstanceType}.

  • “img=img-ubuntu-vpc” indicates that an image is used. You can specify a custom image.

  • “type=ecs.sn1ne.large” indicates that an instance type is used. You can run bcs it to view the list of the supported instance types.

  • Specify only -c img=${ImageId} or only -c type=${InstanceType}. If none of them are specified, the Default image and default instance types are used.

4. Use a cluster

  1. bcs sub "python test.py" -p ./src/ -c cls-xxxxxxx
  • “-c cls-xxxxxxx” indicates that the cluster with the ID set to cls-xxxxxxx is used.

For more information about the cluster, see Use a cluster.

5. Use Docker

  1. bcs sub "python test.py" -p ./src/ --docker myubuntu@oss://my-bucket/dockers/
  • “myubuntu” is short for “localhost:5000/myubuntu”. “oss://my-bucket/dockers/“ indicates the path of the OSS Docker registry.

  • For more information, see Use the Docker.

6. Custom disks

  1. bcs sub "python test.py" --disk system:cloud_efficiency:50,data:cloud_efficiency:200
  • The command is valid only when AutoCluster is used. System disks and one data disk (optional) can be configured. The format is similar to —disk system:cloud_efficiency:40,data:cloud_efficiency:50:/home/disk1. The system disk configuration is separated from the data disk configuration by a comma (,). You can specify only the system disk or only the data disk, for example:—disk system:cloud_efficiency:40.

  • By default, only one system disk is mounted, and the size is 40 GB.

  • The format of the system disk configuration is as follows: system:< cloud_efficiency|cloud_ssd>:<40-500>. For example, “system:cloud_efficiency:40” indicates that the system disk is mounted to a 40 GB ultra cloud disk.

  • The format of the data disk configuration is as follows: data:< cloud_efficiency|cloud_ssd>:<5-2000>:For example, “data:cloud_ssd:50:/home/disk1” indicates that a 50 GB SSD cloud disk is mounted as a data disk. In the Windows operating system, the data disk can be mounted only to the drive. For example, “data:cloud_ssd:50:E” indicates that the data disk is mounted to the E drive.

Note: For an old and dedicated instance that starts with “bcs”, set the disk type to ephemeral, and set the size of the data disk to a value ranging from 5 GB to 1024 GB.

View job information

For more information, see How to view the job information.