This article describes the commands used to submit various types of jobs.
1. Submit a simplest job
If you have configured the default image, instance type, and network type according to Preparations, use the following simple command to submit a job.
bcs sub "echo 123" # Submit and run a single-task job "echo 123".
If default settings are unavailable for the BatchCompute-cli tool, you must specify more parameters when submitting a job.
bcs sub "echo 123" -t ecs.sn1ne.large -i img-ubuntu-vpc --vpc_cidr_block 192.168.0.0/16
For the remaining examples in this article, it is assumed that you have configured the default settings.
2. Submit a Python script job
bcs sub "python test.py" -p ./test.py # "-p" indicates that test.py is packed into worker.tar.gz and then uploaded to OSS before the job is submitted.
3. Submit a directory (multi-file) job
Generally, the src directory contains multiple files, for example:
src
|-- test.py
|-- dep.py
Test.py must depend on dep.py. Therefore, the entire directory can be packed.
bcs sub "python test.py" -p ./src/ # Pack all files under the src directory into worker.tar.gz, upload it to the OSS, and then submit the job.
Specify multiple files at a time and separate these files by commas (,):
cd src # Switch to the src directory.
bcs sub "python test.py" -p test.py,dep.py # Pack the two files into worker.tar.gz.
If you do not switch to the src directory, submit the job as follows:
bcs sub "python test.py" -p src/test.py,src/dep.py # Pack the two files into worker.tar.gz.
Then, use the following command to view the content of worker.tar.gz:
tar -tvf worker.tar.gz
The content of worker.tar.gz is as follows:
test.py
dep.py
4. Submit a job by mounting a task program
Assume that you have uploaded test.py to oss://mybucket/test/test.py.You can mount the OSS directory “oss://mybucket/test/“ to a VM local directory, for example, “/home/admin/test/“. Then use the command “python /home/admin/test/test.py” to run the program.
Submit a job as follows:
bcs sub "python /home/admin/test/test.py" -r oss://mybucket/test/:/home/admin/test/
The parameter -r oss://mybucket/test/:/home/admin/test/
indicates read-only mounting and “oss://mybucket/test/“ is mounted onto “/home/admin/test/“.
In this way, you do not need to pack test.py and dep.py into worker.tar.gz.
For more information about how to submit a job, run “bcs sub -h” to view the help information.
Other tasks
1. Mount the input data
Assume that your data has been uploaded to the directory “oss://my-bucket/inputs/“.
bcs sub "python test.py" -p ./src/ -r oss://my-bucket/inputs/:/home/admin/inputs/
- “-r” indicates read-only mounting and “oss://my-bucket/inputs/“ is mounted to “/home/admin/inputs/“. In this way, your program can read files under the directory “/home/admin/inputs/“ same as reading local files.
- All mounted directory cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.
- To mount multiple directories, separate them by using commas (,). For example,
-r oss://my-bucket/inputs/:/home/admin/inputs/,oss://my-bucket/inputs2/:/home/admin/inputs2/
.
2. Use the mounting method to automatically upload the program running results
Your program will write the running results to the directory “/home/admin/outputs/“. To upload all data under this directory to OSS, run the following command:
bcs sub "python test.py" -p ./src/ -r oss://my-bucket/inputs/:/home/admin/inputs/ -w oss://my-bucket/outputs/:/home/admin/outputs/
- “-w” indicates writable mounting and all data written to this directory will be automatically uploaded to the corresponding OSS directory after the program finishes running.
- All mounted directory cannot be system directories, such as “/bin” and “/usr”. The directory “/home/“ is recommended.
- To mount multiple directories, separate them by using commas (,). For example,
-w oss://my-bucket/outputs/:/home/admin/outputs/,oss://my-bucket/outputs2/:/home/admin/outputs2/
.
3. Use the custom image and instance type
bcs sub "python test.py" -p ./src/ -c img=img-ubuntu-vpc:type=ecs.sn1ne.large
“-c” indicates that a cluster is used. Next to “-c”, you can specify the cluster ID or the AutoCluster configuration. The AutoCluster configuration format is as follows:
img=${ImageId}:type=${InstanceType}
.“img=img-ubuntu-vpc” indicates that an image is used. You can specify a custom image.
“type=ecs.sn1ne.large” indicates that an instance type is used. You can run
bcs it
to view the list of the supported instance types.Specify only
-c img=${ImageId}
or only-c type=${InstanceType}
. If none of them are specified, the Default image and default instance types are used.
4. Use a cluster
bcs sub "python test.py" -p ./src/ -c cls-xxxxxxx
- “-c cls-xxxxxxx” indicates that the cluster with the ID set to cls-xxxxxxx is used.
For more information about the cluster, see Use a cluster.
5. Use Docker
bcs sub "python test.py" -p ./src/ --docker myubuntu@oss://my-bucket/dockers/
“myubuntu” is short for “localhost:5000/myubuntu”. “oss://my-bucket/dockers/“ indicates the path of the OSS Docker registry.
For more information, see Use the Docker.
6. Custom disks
bcs sub "python test.py" --disk system:cloud_efficiency:50,data:cloud_efficiency:200
The command is valid only when AutoCluster is used. System disks and one data disk (optional) can be configured. The format is similar to
—disk system:cloud_efficiency:40,data:cloud_efficiency:50:/home/disk1
. The system disk configuration is separated from the data disk configuration by a comma (,). You can specify only the system disk or only the data disk, for example:—disk system:cloud_efficiency:40
.By default, only one system disk is mounted, and the size is 40 GB.
The format of the system disk configuration is as follows: system:< cloud_efficiency|cloud_ssd>:<40-500>. For example, “system:cloud_efficiency:40” indicates that the system disk is mounted to a 40 GB ultra cloud disk.
The format of the data disk configuration is as follows: data:< cloud_efficiency|cloud_ssd>:<5-2000>:
For example, “data:cloud_ssd:50:/home/disk1” indicates that a 50 GB SSD cloud disk is mounted as a data disk. In the Windows operating system, the data disk can be mounted only to the drive. For example, “data:cloud_ssd:50:E” indicates that the data disk is mounted to the E drive.
Note: For an old and dedicated instance that starts with “bcs”, set the disk type to ephemeral, and set the size of the data disk to a value ranging from 5 GB to 1024 GB.
View job information
For more information, see How to view the job information.