DataWorks:Custom images

Usage notes

• Edition requirements:

  • All editions: Create and use custom images.

  • Professional edition or higher: Build persistent images.

• Resource group support: Custom images work only with serverless resource groups.

  Legacy resource groups: Use O&M Assistant to install external dependencies.

• Permissions required: You need one of the following policies: AliyunDataWorksFullAccess or ModifyResourceGroup.

  For more information, see Product and console access control: RAM Policy.

Quotas and limits

• Image count limits:

  • Basic and standard editions: 10.

  • Professional Edition: 50.

  • Enterprise Edition: 100.

• Build concurrency: Maximum 2 concurrent builds per region.

• ACR image requirements:

  • Instance edition: Enterprise edition.

  • Architecture: AMD64.

  • Image size: Maximum 5 GB per image.

• Persistent builds: Only supported for images built from DataWorks official images (not ACR images).

• Supported node types:

  Node type	Build from official images	Build from ACR images
  PyODPS2	Supported	Not supported
  PyODPS3	Supported	Not supported
  EMR Spark	Supported	Not supported
  EMR Spark SQL	Supported	Not supported
  EMR SHELL	Supported	Not supported
  Shell	Supported	Supported
  Python	Supported	Supported
  Notebook	Not supported	Supported
  CDH	Supported	Not supported
  Assignment node	Supported	Not supported

Procedure

Billing

Image builds are charged as: CU count × Build duration. The system allocates 0.5 CUs by default. For more information about billing, see Billing for Serverless resource groups.

Production best practices

Follow these recommendations for stable, efficient, cost-effective use in production:

• Persistent image: Build persistent images from published configurations with stable dependencies. This eliminates reinstallation on every run, reducing startup time, compute costs, and improving stability.

• Environment consistency: Ensure VPCs and network configurations match across test, build, and production serverless resource groups, especially when accessing private ACR repositories or the internet.

• Version locking: When installing dependencies via Script mode, always specify versions such as pip install pandas==1.5.3. This prevents unexpected behavior from upstream library updates.

• Rollback plan: If production tasks fail after updating an image:

  • Roll back via task publishing history.

  • Revert to a previous stable version in scheduling configuration.

Example

This example demonstrates using a custom image with PyODPS to perform word segmentation on a MaxCompute table. Segment text in a MaxCompute table column and store results in another table for downstream scheduling.

1. Create test data.

   1. Create a DataWorks workspace with attached MaxCompute resources. For more information, see Create a workspace, Add a data source or register a cluster to a workspace, and Associate a computing resource.

   2. In Data Studio, create an ODPS node (legacy) or MaxCompute SQL node (new version):

      Note

      This example uses scheduling parameters. Set parameter name to bday with value to $[yyyymmdd] in the Scheduling pane.

      Create a test table.

      -- Create test table
         CREATE TABLE IF NOT EXISTS custom_img_test_tb
         (
             c_customer_id BIGINT NOT NULL,
             c_customer_text STRING NOT NULL,
             PRIMARY KEY (c_customer_id)
         )
         COMMENT 'Test table for custom image demo'
         PARTITIONED BY (ds STRING COMMENT 'Partition')
         LIFECYCLE 90;
      
         -- Insert test data
         INSERT INTO custom_img_test_tb PARTITION (ds='${bday}') (c_customer_id, c_customer_text) VALUES
         (1, 'The sky is getting dark and it looks like it will snow. Would you like a cup of wine?'),
         (2, 'The moon sets, crows caw, and frost fills the sky. I lie awake, facing the river maples and fishing lights.'),
         (3, 'Mountains and rivers seem to block the way. But among shady willows and bright flowers, another village appears.'),
         (4, 'I sleep in spring, unaware of the dawn. Everywhere I hear the birds sing.'),
         (5, 'Thoughts on a quiet night. Moonlight shines before my bed. I mistake it for frost on the ground.'),
         (6, 'The bright moon rises over the sea. We share this moment, though we are far apart.'),
         (7, 'The swallows that once graced the halls of nobles now fly into the homes of common people.'),
         (8, 'A line of egrets flies up to the blue sky. My window frames the ancient snow on the Western Hills.'),
         (9, 'When life is good, enjoy it to the fullest. Do not let the golden goblet face the moon empty.'),
         (10, 'Heaven gave me talent, so it must be used. A thousand pieces of gold, once spent, can be earned again.');

   3. Save and publish the node.

2. Create a custom image.

   Create a custom image with these key parameters:

   • Image Name/ID: Select dataworks_pyodps_task_pod (DataWorks official PyODPS image)

   • Supported Task Type: PyODPS2 and PyODPS 3.

   • Installation Package: Select Python 3 and add jieba.

3. Publish and assign images.

   Publish the image and assign it to your workspace. For more information, see Test and publish the image and Assign the image to workspaces.

4. Create PyODPS Tasks.

5. In Data Studio, create a PyODPS3 node with this code

   Use the custom image.

   import jieba
      from odps import ODPS
      from odps.models import TableSchema as Schema, Column, Partition
   
      # Read data from the source table
      table = o.get_table('custom_img_test_tb')
      partition_spec = f"ds={args['bday']}"
      with table.open_reader(partition=partition_spec) as reader:
          records = [record for record in reader]
   
      # Perform word segmentation
      participles = [' | '.join(jieba.cut(record['c_customer_text'])) for record in records]
   
      # Create destination table
      if not o.exist_table("participle_tb"):
          schema = Schema(
              columns=[Column(name='word_segment', type='string', comment='Segmentation result')], 
              partitions=[Column(name='ds', type='string', comment='Partition field')]
          )
          o.create_table("participle_tb", schema)
   
      # Write results to destination table
      output_partition = f"ds={args['bday']}"
      output_table = o.get_table("participle_tb")
   
      # Create partition if it doesn't exist
      if not output_table.exist_partition(output_partition):
          output_table.create_partition(output_partition)
   
      # Write segmentation results
      record = output_table.new_record()
      with output_table.open_writer(partition=output_partition, create_partition=True) as writer:
          for participle in participles:
              record['word_segment'] = participle
              writer.write(record)

6. Configure scheduling parameters in the right pane:

   • Scheduling Parameters: bday = $[yyyymmdd].

   • Resource Group: Same Serverless group used for image testing.

   • Image: Your published custom image.

7. Debug the node.

   • Legacy version: Click Run with Parameters (), configure settings, and click Run.

   • New version: Configure in Debugging Configurations pane and click Running Duration in toolbar

8. (Optional) Verify results with a SQL query:

   -- Replace <partition_date> with actual date
      SELECT * FROM participle_tb WHERE ds=<partition_date>;

9. Publish the PyODPS node to production.

   Note

   Image changes in Data Studio don't sync to production automatically. You need to publish the task for changes to take effect. For more information, see Deploy nodes or Node or workflow deployment.

10. Build persistent Images.

    Build your custom image as a persistent image. For more information, see Build a persistent image.

References

• Official images in DataWorks

• Use serverless resource groups

• Develop tasks based on a self-managed Hadoop cluster

• PyODPS

Appendix: Installation command reference

When using Script mode to configure installation commands:

• For PyODPS 2 Dependencies:

  pip install <package_name>

  Note

  If prompted to upgrade pip, run: pip install --upgrade pip.

• For PyODPS 3 dependencies:

  /home/tops/bin/pip3 install <package_name>

  Note

  • If prompted to upgrade pip, run: /home/tops/bin/pip3 install --upgrade pip.

  • If you encounter error /home/admin/usertools/tools/cmd-0.sh: line 3: /home/tops/bin/python3: No such file or directory, submit a ticket to request permissions.

• Python mirror sources

  Switch to these public mirrors as needed:

  Organization	Mirror URL
  Alibaba Cloud (Aliyun)	https://mirrors.aliyun.com/pypi/simple/ Important No internet access required for Alibaba Cloud mirrors.
  Tsinghua University	https://pypi.tuna.tsinghua.edu.cn/simple
  USTC	https://pypi.mirrors.ustc.edu.cn/simple/