MaxCompute:Custom images

Base images

MaxCompute provides two base images:

Create and build the image

1. Create a Dockerfile.

      vim DockerFile

2. Add the following content to the Dockerfile:

      # Use the MaxCompute CentOS base image
      FROM registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/base_image:latest
   
      # If you use the Ubuntu image, replace the image address with registry.cn-zhangjiakou.aliyuncs.com/maxcompute_image/ubuntu_20.04:latest
   
      # Install system dependencies
      RUN yum install vi -y
   
      # Install third-party libraries
      RUN /usr/ali/python3.7/bin/python3 -m pip install --no-cache-dir pandas

3. Build the image.

   • image_name: The name of the custom image.

   • tag: The version of the custom image.

      sudo docker build -f DockerFile -t <image_name>:<tag> .

Create an ACR Enterprise Edition instance

1. Log on to the Container Registry console and select a region in the upper-left corner.

2. In the navigation pane on the left, click Instances.

3. On the Instances page, click Create ACR EE. If you have already created an instance, skip this step.

   Important

   Upload custom images only to Standard Edition or Premium Edition instances of ACR Enterprise Edition.

Create a repository

1. On the Instances page, find the target Enterprise instance and click Manage to open its overview page.

2. In the navigation pane on the left, choose Repository > Repositories.

3. On the Repository > Repositories page, click Create Repository. In the Create Repository dialog box, enter the following information and click Next.

   Parameter	Required	Description
   Region	Yes	The region where the current instance resides is automatically selected.
   Namespace	Yes	The namespace of the image repository. This parameter cannot be modified after it is set. Create a namespace that corresponds to a company, an organization, or an individual user, such as Aliyun. Do not create a namespace that corresponds to a module or system, such as Tomcat, CentOS, an application, or a module.
   Repository Name	Yes	The name must be 2 to 120 characters in length and can contain lowercase letters, digits, and separators. The separators can be underscores (_), hyphens (-), periods (.), and forward slashes (/). A separator cannot be the first or last character.
   Repository Type	No	To pull a public image, you must log on to the Enterprise Edition instance. To allow anonymous pulls, enable anonymous pulls for the instance. To pull a private image from an Enterprise Edition instance, you must log on to the instance and have the pull permission. The default value is Private.
   Tags	No	After selected, all tags other than the latest in the repository cannot be overwritten, ensuring the consistency of the container images.
   Accelerated Image	No	After selected, all images in the repository automatically generate accelerated images with an "_accelerated" suffix. Full Mode: Provides a significant acceleration effect. The size of an accelerated image is approximately 130% the size of the original image. The system requires approximately 25 seconds to generate a 1 GB-sized accelerated image. If an accelerated image layer has been generated for an image layer, the system does not generate an accelerated image layer again for the image layer. Index-only Mode (Public Preview): Provides about 70% of the acceleration effect of the full mode. The size of the accelerated image is about 3% of the original image size. It takes about 3 seconds to generate a 1 GB accelerated image. Image layers for which an index is already generated are not generated again.
   Summary	Yes	Max. 100 characters.
   Description	No	Supports Markdown Format.

4. Configure the code source:

   Set Code Source to Local Repository and click Create Repository.

Upload the image

1. On the Repository > Repositories page, find the desired repository and click Manage in the Actions column.

2. In the navigation pane on the left, click Details.

3. Follow the instructions in the Instructions on Images section on the Details page to upload a custom image from your Docker environment to the ACR image repository.

4. (Optional) If your machine is in a VPC, perform the following steps:

   1. Configure VPC access for the Enterprise Edition instance. For more information, see Configure access over a VPC.

   2. When you perform operations on the ACR Enterprise Edition instance in your Docker environment, add vpc to the domain name. For example, change acr-test-registry.cn-wulanchabu.cr.aliyuncs.com to acr-test-registry-vpc.cn-wulanchabu.cr.aliyuncs.com in the following command:

      $ docker login --username=***@test.aliyunid.com acr-test-registry.cn-wulanchabu.cr.aliyuncs.com

   Note

   If an error occurs when you log on to the instance, check whether public network access is enabled for the repository.

UDF development

Specify the image and Python version at the SQL session level using flags:

set odps.sql.python.version=cp37;
set odps.session.image = <image_name>;

PyODPS development

Specify the image using the image parameter of the execute or persist method:

image='<image_name>'

MaxFrame development

Specify the image for the current job through the configuration settings:

config.options.sql.settings = {
    "odps.session.image": "<image_name>"
}