Batch Compute not only allows you to install software on an image directly but also supports application deployment on a Docker image. You can make a Docker image and use the registry tool to upload it to Alibaba Cloud OSS. Then, you can run tasks of your jobs on this image.
The AutoCluster model is used as an example to explain the difference in support for common VMs and Docker containers:
Using common VMs: Each job you submit can have multiple tasks, each having an image (Linux or Windows) specified.
When running a task, the system starts the corresponding VM based on the specified image.
The task then runs on this VM. After the task is finished, the system uploads the task execution result to the specified permanent storage, destroys the VM, and then runs the next task.
Using Docker: You can specify a Docker image for each task.
When running a task, the system starts a VM to run the system image (such as Ubuntu) that supports Docker.
Then, the system downloads the Docker image you specified and starts it in the VM. Your task runs in the Docker container.
After the task is finished, the system uploads the task execution result to the specified permanent storage, destroys the VM, and then runs the next task.
Currently, one VM can run only one Docker image.
# Use VMs:
|-- VM (specified VM that supports Windows and Linux images)
|-- program (your program)
# Use Docker containers:
|-- VM (Ubuntu operating system supporting Docker)
|-- Docker-Container (image specified for the Docker container)
|-- program (your program)
|-||Docker not used||Docker used|
|Use of images||Specify an ECS image ID||Specify the ID of the ECS image that supports a Docker container (Ubuntu released on the official website), and specify a custom Docker image.|
|Program running platform||Windows and Linux||Linux|
|Local debugging||Not supported||Supported, using a Docker image made locally.|
- For a Windows or Mac operating system, download the toolbox at https://www.docker.com/docker-toolbox.
After you install the toolbox, two shortcut icons appear on the desktop:
Kitematic: used to manage the Docker container GUI.
Docker Quickstart Terminal: used to start the Docker command line window quickly.
- For a Linux operating system, download Docker for Ubuntu at https://www.docker.com/docker-ubuntu.
Install Docker 1.10 or a later version. Earlier Docker versions may cause compatibility issues.
The booster can help you obtain an official Docker image more quickly in China:Alibaba Cloud Container Service Development Platform
This example describes how to make an Ubuntu image containing Python. The image name is myubuntu.
Create a directory dockerUbuntu with the following structure:
The content of the Dockerfile is as follows:
# Replace your_name with your own name and your_email with your email address.
MAINTAINER your_name <your_email>
# Update the source.
RUN apt-get update
# Clear the cache.
RUN apt-get autoclean
# Install Python.
RUN apt-get install -y python
# Run the following command after the system starts:
Run the following command to build an image:
cd dockerUbuntu # Enter the dockerUbuntu directory.
docker build -t myubuntu ./ # Build the image named myubuntu.
- NOTE: In Ubuntu, docker commands can only be executed by a sudo user by default.In a Mac or Windows operating system, start the command line window by using Docker Quickstart Terminal and run docker commands in the window.
After building the image, run the following command to view the image:
The output information is similar to the following:
In addition to using the Dockerfile, you can also make Docker images in a more intuitive way.
To use the Docker service of Batch Compute, upload the Docker image myubuntu you have made to OSS.The system then downloads the image from OSS to run your task programs.
Assume that you want to save Docker in the following directory of OSS:
Use the official image of Docker Registry 2 to create a private image repository, and set OSS information including the AccessKey ID, AccessKey Secret, region, and bucket.
The installation procedure is as follows:
Create the config.yml file in the current directory.
Replace the following variables:
|your_access_key_id||Your AccessKey ID|
|your_access_key_secret||Your AccessKey secret|
|oss-cn-shenzhen||Region where your bucket belongs|
For more information about OSS configuration, see official Docker documentation.
Run the following command to install OSS Docker Registry 2.
docker pull registry:2
docker run -v `pwd`/config.yml:/etc/docker/registry/config.yml -p 5000:5000 --name registry -d registry:2
In this example, oss-cn-shenzhen represents OSS in the China South 1 (Shenzhen) region. Jobs submitted subsequently can only work normally in this region.
After the installation is finished, run the following command to check the result:
docker ps # Check the running container.
If the installation is successful, registry:2 is displayed.
docker tag myubuntu localhost:5000/myubuntu
docker push localhost:5000/myubuntu
- The prefix of the image name must be localhost:5000/. The image cannot be uploaded if any other prefix is used.Port 5000 in the prefix is specified by -p 5000:5000 (the number before the colon) in step (1).
- The actual name of the image you made is localhost:5000/myubuntu, but not myubuntu.
To verify whether the image is uploaded successfully, log on to the OSS console and check for the following directory: oss://your-bucket/dockers/docker/registry/v2/repositories/myubuntu/.
In a running Docker container, the user name is root, and the default value of the path environment variable is /sbin:/usr/sbin:/bin:/usr/bin, not /usr/local/bin.
If you do not set the PWD environment variable, its value is ‘/batchcompute/workdir’. All your program packages are decompressed into /batchcompute/workdir.
The Docker registry never stops after it starts. Therefore, when you submit jobs by specifying cluster IDs, make sure that all jobs submitted to the same cluster have the same BATCH_COMPUTE_DOCKER_REGISTRY_OSS_PATH value.
InputMapping and OutputMapping cannot be attached to the same directory.
Batch Compute uses the —privileged=false mode to start Docker containers, and therefore it does not allow a Docker container to start in another Docker container.