DataWorks:Custom images

Usage notes

• Version limits:

  • All editions support creating and using custom images.

  • Only Professional Edition and higher support image building.

• Resource group limits: This feature supports only serverless resource groups.

  For old resource groups, please use O&M Assistant to install external dependencies.

• Permission limits: You need the AliyunDataWorksFullAccess or ModifyResourceGroup policy.

  For authorization details, see Manage product-level and console access with RAM policies.

Quotas and limits

• Image quantity: Custom image limits vary by DataWorks edition.

  • Basic and Standard Editions: 10.

  • Professional Edition: 50.

  • Enterprise Edition: 100.

• Build concurrency: You can build up to two images simultaneously per region.

• ACR image requirements:

  • Instance edition: Only Enterprise Edition Alibaba Cloud ACR instances are supported.

  • Instance architecture: Only AMD64 architecture is supported.

  • Image size: A single image cannot exceed 5 GB.

  • Timezone configuration: Install the tzdata package to prevent container exceptions due to timezone inconsistencies.

• Image build: Only custom images based on DataWorks official images support persistence builds. Images referencing Alibaba Cloud ACR images do not; they are pulled and deployed for every task run.

• Supported node types and methods:

  Node type	Official image build	ACR image build
  PyODPS2	Supported	Unsupported
  PyODPS3	Supported	Unsupported
  EMR Spark	Supported	Unsupported
  EMR Spark SQL	Supported	Unsupported
  EMR SHELL	Supported	Unsupported
  Shell	Supported	Supported
  Python	Supported	Supported
  Notebook	Unsupported	Supported
  CDH	Supported	Unsupported
  Assignment Node	Supported	Unsupported

Procedure

Billing

Building an image incurs computing fees based on CU quantity × Build duration. The system allocates 0.5 CUs by default. For billing details, see Serverless resource group billing standards.

Best practices for production

Follow these recommendations for stable and efficient custom images in production:

• Persistent image: We recommend building persistent images for published and stable images. This avoids re-installing dependencies every time a task runs, shortening startup time, reducing computing costs, and improving stability.

• Environment consistency: Ensure consistency in VPC binding and network configuration for the Serverless resource groups used for testing, building, and production scheduling, especially when accessing private ACR repositories or the Internet.

• Version locking: When installing dependencies via Script, we strongly recommend explicitly specifying version numbers (e.g., pip install pandas==1.5.3) to avoid unexpected behavior changes caused by upstream library updates.

• Rollback plan: If a production task fails after an image update, you can roll back to the previous version via the task publication history or repoint the image to an older, stable version in the scheduling configuration.

Use cases

This example shows how to use a custom image for word segmentation in a PyODPS node. You will process data in a MaxCompute table and store the results for downstream nodes. You can pre-install the jieba segmentation tool package in a custom image, then use this image in a PyODPS task to process the data and store the results in a new table, seamlessly integrating into the downstream scheduling flow.

1. Create test data.

   1. Create a DataWorks workspace and bind MaxCompute computing resources. For details, see Create a workspace and Computing resource management.

   2. In DataStudio, create an ODPS node (legacy DataStudio) or MaxCompute SQL node (new Data Studio), create a test table, and add test data.

      Note

      The following example uses scheduling parameters. Set the parameter name to bday and the parameter value to $[yyyymmdd] in the Scheduling panel on the right.

      Create test table

      -- Create test table
         CREATE TABLE IF NOT EXISTS custom_img_test_tb
         (
             c_customer_id BIGINT NOT NULL,
             c_customer_text STRING NOT NULL,
             PRIMARY KEY (c_customer_id)
         )
         COMMENT 'Test table for custom image demo'
         PARTITIONED BY (ds STRING COMMENT 'Partition')
         LIFECYCLE 90;
      
         -- Insert test data
         INSERT INTO custom_img_test_tb PARTITION (ds='${bday}') (c_customer_id, c_customer_text) VALUES
         (1, 'The sky is getting dark and it looks like it will snow. Would you like a cup of wine?'),
         (2, 'The moon sets, crows caw, and frost fills the sky. I lie awake, facing the river maples and fishing lights.'),
         (3, 'Mountains and rivers seem to block the way. But among shady willows and bright flowers, another village appears.'),
         (4, 'I sleep in spring, unaware of the dawn. Everywhere I hear the birds sing.'),
         (5, 'Thoughts on a quiet night. Moonlight shines before my bed. I mistake it for frost on the ground.'),
         (6, 'The bright moon rises over the sea. We share this moment, though we are far apart.'),
         (7, 'The swallows that once graced the halls of nobles now fly into the homes of common people.'),
         (8, 'A line of egrets flies up to the blue sky. My window frames the ancient snow on the Western Hills.'),
         (9, 'When life is good, enjoy it to the fullest. Do not let the golden goblet face the moon empty.'),
         (10, 'Heaven gave me talent, so it must be used. A thousand pieces of gold, once spent, can be earned again.');

   3. Save and deploy.

2. Create a custom image.

   See 1. Create a custom image. Key parameters are as follows:

   • Image Name/ID: Select dataworks_pyodps_task_pod, the DataWorks PyODPS node official image.

   • Supported Task Type: Support PyODPS2 and PyODPS 3.

   • Packages: Select Python3 and jieba.

3. Publish the custom image and modify the owner workspace. For details, see Publish a custom image and Modify owner workspace.

4. Use the custom image in a scheduling task.

   1. In DataStudio, create a PyODPS3 node and configure the following content:

      Use custom image

      import jieba
      from odps import ODPS
      from odps.models import TableSchema as Schema, Column, Partition
      
      # Read table data
      table = o.get_table('custom_img_test_tb')
      partition_spec = f"ds={args['bday']}"
      with table.open_reader(partition=partition_spec) as reader:
          records = [record for record in reader]
      
      # Perform word segmentation on the extracted text
      participles = [' | '.join(jieba.cut(record['c_customer_text'])) for record in records]
      
      # Create target table
      if not o.exist_table("participle_tb"):
          schema = Schema(columns=[Column(name='word_segment', type='string', comment='Segmentation Result')], partitions=[Column(name='ds', type='string', comment='Partition Field')])
          o.create_table("participle_tb", schema)
      
      # Write segmentation results to the target table
      # Define output partition and table
      output_partition = f"ds={args['bday']}"
      output_table = o.get_table("participle_tb")
      
      # Create partition if it does not exist
      if not output_table.exist_partition(output_partition):
          output_table.create_partition(output_partition)
      
      # Write segmentation results to table
      record = output_table.new_record()
      with output_table.open_writer(partition=output_partition, create_partition=True) as writer:
          for participle in participles:
              record['word_segment'] = participle
              writer.write(record)

   2. Set the following key parameters in the scheduling configuration on the right:

      • Scheduling Parameter: Parameter name bday, parameter value $[yyyymmdd].

      • Resource Group: Select the Serverless resource group, which must be the same as the test resource group selected when publishing the image.

      • Image: Select the published custom image bound to the current workspace.

   3. Node debugging.

      • If using old DataStudio, click Run with Parameters () in the top toolbar, configure Resource Group Name, CUs for Running, Image, and Custom Parameters, and then click Run.

      • If using new DataStudio, configure Compute Resource, Resource Group, Compute CUs, Image, and Script Parameters in the Debugging Configuration panel on the right, and then click Run in the top toolbar.

   4. (Optional) Create a temporary query (legacy DataStudio) or create an SQL file in your personal directory (new Data Studio), and use the following SQL to query whether data is generated in the output table.

      -- Replace <Partition Date> with the specific date.
      SELECT * FROM participle_tb WHERE ds=<Partition Date>;

   5. Deploy the PyODPS node to the production environment.

      Note

      Image modifications in DataStudio are not synchronized to the production environment. You must publish the task for the changes to take effect in production. For details, see Publish tasks (Old DataStudio) or Node/Workflow publication (New DataStudio).

5. Build the custom image into a persistent image. For details, see 5. Build a persistent image.

FAQ

Q: Python task error "urllib3 v2.0 only supports OpenSSL 1.1.1+".

A: urllib3 v2.0 only supports OpenSSL 1.1.1+. You can downgrade urllib3 to be compatible with OpenSSL. For example, force the urllib3 version when installing third-party packages: /home/tops/bin/pip3 install urllib3==1.26.16.

References

• Official images in DataWorks

• Use serverless resource groups

• Develop tasks based on a self-managed Hadoop cluster

• MaxCompute PyODPS Development

Installation commands

If you use the Script method to configure installation commands for custom images, refer to the following commands:

• If depending on a PyODPS 2 node, execute the following command.

  pip install <package_name> -i https://pypi.tuna.tsinghua.edu.cn/simple

  pip install <package_name>

  Note

  After executing the command, if prompted to upgrade the PIP version, execute pip install --upgrade pip.

• If depending on a PyODPS 3 node, execute the following command.

  /home/tops/bin/pip3 install <package_name> -i https://pypi.tuna.tsinghua.edu.cn/simple

  /home/tops/bin/pip3 install <package_name>

  Note

  • After executing the command, if prompted to upgrade the PIP version, execute /home/tops/bin/pip3 install --upgrade pip.

  • If the error /home/admin/usertools/tools/cmd-0.sh: line 3: /home/tops/bin/python3: No such file or directory occurs, please submit a ticket to request permission activation.

  Refer to the following Python public mirror sources and switch as needed.

  Organization	Mirror address
  Alibaba Cloud (Aliyun)	https://mirrors.aliyun.com/pypi/simple/ Important Obtaining Python packages from Alibaba Cloud does not require Internet access capability.
  Tsinghua University (Tsinghua)	https://pypi.tuna.tsinghua.edu.cn/simple
  University of Science and Technology of China (USTC)	https://pypi.mirrors.ustc.edu.cn/simple/