All Products
Search
Document Center

Introduction

Last Updated: May 11, 2018

Batch Compute not only allows you to install software on an image directly but also supports application deployment on a Docker image. You can make a Docker image and use the registry tool to upload it to Alibaba Cloud OSS. Then, you can run tasks of your jobs on this image.

1. Support of Docker

The AutoCluster model is used as an example to explain the difference in support for common VMs and Docker containers:

  • Using common VMs: Each job you submit can have multiple tasks, each having an image (Linux or Windows) specified.

    When running a task, the system starts the corresponding VM based on the specified image.

    The task then runs on this VM. After the task is finished, the system uploads the task execution result to the specified permanent storage, destroys the VM, and then runs the next task.

  • Using Docker: You can specify a Docker image for each task.

    When running a task, the system starts a VM to run the system image (such as Ubuntu) that supports Docker.

    Then, the system downloads the Docker image you specified and starts it in the VM. Your task runs in the Docker container.

    After the task is finished, the system uploads the task execution result to the specified permanent storage, destroys the VM, and then runs the next task.

Currently, one VM can run only one Docker image.

  1. # Use VMs:
  2. ---
  3. |-- job
  4. |-- task
  5. |-- VM (specified VM that supports Windows and Linux images)
  6. |-- program (your program)
  7. # Use Docker containers:
  8. ---
  9. |-- job
  10. |-- task
  11. |-- VM (Ubuntu operating system supporting Docker)
  12. |-- Docker-Container (image specified for the Docker container)
  13. |-- program (your program)

2. Scenarios with and without Docker

- Docker not used Docker used
Use of images Specify an ECS image ID Specify the ID of the ECS image that supports a Docker container (Ubuntu released on the official website), and specify a custom Docker image.
Program running platform Windows and Linux Linux
Local debugging Not supported Supported, using a Docker image made locally.

3. Install Docker

3.1. Download from the Docker official

After you install the toolbox, two shortcut icons appear on the desktop:

Kitematic: used to manage the Docker container GUI.

Docker Quickstart Terminal: used to start the Docker command line window quickly.

NOTE: Install Docker 1.10 or a later version. Earlier Docker versions may cause compatibility issues.

3.2. Configure booster

The booster can help you obtain an official Docker image more quickly in China:Alibaba Cloud Container Service Development Platform

4. Make a Docker image

This example describes how to make an Ubuntu image containing Python. The image name is myubuntu.

Create a directory dockerUbuntu with the following structure:

  1. dockerUbuntu
  2. |-- Dockerfile

The content of the Dockerfile is as follows:

  1. FROM ubuntu:14.04
  2. # Replace your_name with your own name and your_email with your email address.
  3. MAINTAINER your_name <your_email>
  4. # Update the source.
  5. RUN apt-get update
  6. # Clear the cache.
  7. RUN apt-get autoclean
  8. # Install Python.
  9. RUN apt-get install -y python
  10. # Run the following command after the system starts:
  11. CMD ["/bin/bash"]

Run the following command to build an image:

  1. cd dockerUbuntu # Enter the dockerUbuntu directory.
  2. docker build -t myubuntu ./ # Build the image named myubuntu.
  • NOTE: In Ubuntu, docker commands can only be executed by a sudo user by default.In a Mac or Windows operating system, start the command line window by using Docker Quickstart Terminal and run docker commands in the window.

After building the image, run the following command to view the image:

  1. docker images

The output information is similar to the following:

In addition to using the Dockerfile, you can also make Docker images in a more intuitive way.

5. Upload the Docker image to OSS

To use the Docker service of Batch Compute, upload the Docker image myubuntu you have made to OSS.The system then downloads the image from OSS to run your task programs.

5.4. Install OSS Docker Registry 2

Assume that you want to save Docker in the following directory of OSS:

  1. oss://your-bucket/dockers/

Use the official image of Docker Registry 2 to create a private image repository, and set OSS information including the AccessKey ID, AccessKey Secret, region, and bucket.

The installation procedure is as follows:

Create the config.yml file in the current directory.

  1. version: 0.1
  2. log:
  3. level: debug
  4. storage:
  5. oss:
  6. accesskeyid: your_access_key_id
  7. accesskeysecret: your_access_key_secret
  8. region: oss-cn-shenzhen
  9. bucket: your-bucket
  10. rootdirectory: dockers
  11. secure: false
  12. internal: false
  13. http:
  14. addr: 0.0.0.0:5000

Replace the following variables:

Parameter Description
your_access_key_id Your AccessKey ID
your_access_key_secret Your AccessKey secret
your-bucket Your bucket
oss-cn-shenzhen Region where your bucket belongs

For more information about OSS configuration, see official Docker documentation.

Run the following command to install OSS Docker Registry 2.

  1. docker pull registry:2
  2. docker run -v `pwd`/config.yml:/etc/docker/registry/config.yml -p 5000:5000 --name registry -d registry:2
  • NOTE: In this example, oss-cn-shenzhen represents OSS in the China South 1 (Shenzhen) region. Jobs submitted subsequently can only work normally in this region.

After the installation is finished, run the following command to check the result:

  1. docker ps # Check the running container.

If the installation is successful, registry:2 is displayed.

5.2. Run the following command to upload the image to OSS:

  1. docker tag myubuntu localhost:5000/myubuntu
  2. docker push localhost:5000/myubuntu

NOTE:

  1. The prefix of the image name must be localhost:5000/. The image cannot be uploaded if any other prefix is used.Port 5000 in the prefix is specified by -p 5000:5000 (the number before the colon) in step (1).
  2. The actual name of the image you made is localhost:5000/myubuntu, but not myubuntu.

To verify whether the image is uploaded successfully, log on to the OSS console and check for the following directory: oss://your-bucket/dockers/docker/registry/v2/repositories/myubuntu/.

6. Notes

  • In a running Docker container, the user name is root, and the default value of the path environment variable is /sbin:/usr/sbin:/bin:/usr/bin, not /usr/local/bin.

  • If you do not set the PWD environment variable, its value is ‘/batchcompute/workdir’. All your program packages are decompressed into /batchcompute/workdir.

  • The Docker registry never stops after it starts. Therefore, when you submit jobs by specifying cluster IDs, make sure that all jobs submitted to the same cluster have the same BATCH_COMPUTE_DOCKER_REGISTRY_OSS_PATH value.

  • InputMapping and OutputMapping cannot be attached to the same directory.

  • Batch Compute uses the —privileged=false mode to start Docker containers, and therefore it does not allow a Docker container to start in another Docker container.