All Products
Search

This Product

    DataWorks:Use third-party packages in a PyODPS node

    Document Center

    DataWorks:Use third-party packages in a PyODPS node

    Last Updated:Jun 20, 2026

    When built-in PyODPS features do not meet complex business requirements, you may need to reuse Python code or use open-source libraries. DataWorks offers two solutions: loading custom scripts through a resource reference, or integrating third-party packages using custom images or O&M Assistant. This topic describes how to call custom Python scripts or install and use open-source third-party packages in DataWorks PyODPS nodes to extend your data processing capabilities.

    Quick selection

    Select an approach based on your environment and requirements.

    Scenario

    Resource group type

    Solution

    Using open-source third-party packages

    serverless resource group

    Install open-source packages using a custom image

    exclusive resource group for scheduling

    Install open-source packages using O&M Assistant

    Using only custom .py script files

    serverless resource group / exclusive resource group for scheduling

    Reference a custom Python resource

    The following figure outlines the main workflows for each solution.

    Prerequisites

    Before you begin, understand the following two key concepts to choose the correct configuration method.

    Key concepts

    1. Node type: PyODPS 2 vs. PyODPS 3

      • PyODPS 2: Based on a Python 2.7 environment.

      • PyODPS 3: Based on a Python 3.7+ environment.

      • Recommendation: We strongly recommend that you use PyODPS 3 because official support for Python 2 has ended. This topic uses PyODPS 3 as the primary example.

    2. Resource group type: serverless resource group vs. exclusive resource group for scheduling

      • Serverless resource group (Recommended): This resource group is elastic and maintenance-free. It lets you use custom images to manage third-party packages.

      • Exclusive resource group for scheduling (Not recommended): This is a legacy solution. You must purchase and maintain ECS servers, and its resources cannot scale elastically. Using O&M Assistant to install dependencies has many limitations and may corrupt the environment.

    Determine the resource group type

    In the DataWorks console, go to the Workspace Details page for your workspace. On the Resource Group page, check the resource group type associated with the workspace.

    • If the type is General-purpose, you are using a serverless resource group.

    • If the type is Data Scheduling, you are using an exclusive resource group for scheduling.

    Install open-source packages using a custom image

    Important

    This method applies to serverless resource groups.

    This section provides an end-to-end example of creating a custom environment that includes the pendulum package. You will then use a PyODPS 3 node to call this package to retrieve and format the current time for a specific time zone.

    Step 1: Create a custom image that contains the pendulum library

    Custom images define the runtime environment for a serverless resource group.

    1. Log on to the DataWorks console and click Image Management in the left-side navigation pane.

    2. Go to the Custom Image tab.

    3. Click Create Image in the upper-left corner. On the Create page, configure the following key parameters:

      Parameter

      Description

      Image Name

      Enter a name for the custom image, for example, pyodps3_with_pendulum.

      Reference Type

      Select DataWorks Official Image.

      Image Name/ID

      From the drop-down list, select the official DataWorks image dataworks_pyodps_task_pod.

      Supported Task Types

      Select the PyODPS 3 task type.

      Installation Package

      Select Python3, and then select the system-provided pendulum package from the drop-down list.

      To install an open-source package that is not built-in, you can use Script mode for manual installation. For more information about configuration methods, see Parameters for creating a custom image.
      Important

      To install open-source packages from the public internet, the VPC associated with the serverless resource group must have public network access.

    4. Click OK to create the custom image.

    5. On the Custom Images page, test and then deploy the image. An image must pass the test before it can be deployed.

    6. In the Operation column of the target image, click image > Modify Owning Workspace to associate an owning workspace with the custom image.

      After the image is successfully deployed, its Deployment Status changes to Deployed, and a Disable button appears in the Actions column. The Actions drop-down menu also includes options such as Modify, Build, View Versions, and Delete Image.

    Step 2: Create and configure a PyODPS 3 node

    1. In the left navigation bar, click Data Development and O&M > Data Studio, select the target workspace from the drop-down list, and click Go to DataStudio.

    2. In an existing business flow, create a PyODPS 3 node named pyodps3_pendulum_test.

    3. In the code editor for the pyodps_pendulum_test node, enter the following Python 3 code:

      # The pendulum package is installed in the custom image, so you can import it directly.
import pendulum
print("Start testing the third-party package pendulum...")
try:
    # Use pendulum to get the current time in the "Asia/Shanghai" time zone.
    shanghai_time = pendulum.now('Asia/Shanghai')
    # Print the formatted time and time zone information.
    print(f"Successfully imported the 'pendulum' package.")
    print(f"The current time in Shanghai is: {shanghai_time.to_datetime_string()}")
    print(f"The corresponding time zone is: {shanghai_time.timezone_name}")
    print("\nTest passed! The PyODPS node successfully called the third-party package.")
except Exception as e:
    print(f"Test failed. An error occurred: {e}")

    Step 3: Test and verify the result

    1. Click the ** icon in the toolbar to run the code, and in the Parameter dialog box, select the image pyodps3_with_pendulum that you created.

      Important

      If you cannot find the target image, ensure that the image is associated with the current workspace. For more information, see Step 6 in "Step 1: Create a custom image".

    2. View the run log at the bottom of the page. The following output indicates that the pendulum package was successfully called.

      Start testing the third-party package pendulum...
Successfully imported the 'pendulum' package.
The current time in Shanghai is: 2025-09-27 15:45:00
The corresponding time zone is: Asia/Shanghai
Test passed! The PyODPS node successfully called the third-party package.

    Step 4: Deploy the PyODPS 3 node

    After the test is complete, go to Scheduling > Resource Group in the right-side pane. Select your serverless resource group and set the image to the pyodps3_with_pendulum custom image you created. You can then deploy the node to Operation Center.

    Install open-source packages using O&M Assistant

    Important

    This method applies to exclusive resource groups for scheduling, which are a legacy solution and no longer recommended. We recommend that you migrate to serverless resource groups, which are more powerful and flexible.

    1. Log on to the DataWorks workspace list, switch the region at the top of the page, and then click Details in the Operation column for the target workspace to go to the workspace details page.

    2. In the left navigation bar, click Resource Group, find the associated exclusive resource group for scheduling, and in the Actions column, click image > O&M Assistant.

    3. On the O&M Assistant page, click Create Command in the upper-left corner.

    4. Enter the command that corresponds to your Python version:

      • Python 3 (PyODPS 3): Keep the default options. From the drop-down list for the Python3 package type, select the pendulum installation package.

      • Python 2 (PyODPS 2): Select Manual Input mode and enter the command content.

        pip install --upgrade pip
pip install "pendulum<2.0"

    5. On the O&M Assistant page, click Run Command in the Actions column. After the command runs successfully, you can directly use import pendulum in the corresponding PyODPS node.

    Reference a custom Python resource

    If you only need to call a function from another .py file, follow these steps:

    1. Create a Python resource

      1. On the Data Development page, right-click the target business flow and choose Create Resource > MaxCompute > Python.

      2. In the Create Resource dialog box, enter a Name for the resource, for example, my_utils.py, and click Create.

      3. Enter the following code in the Python resource editor.

        # my_utils.py (Python 3 syntax)
def say_hello(name):
    print(f"Hello, {name}! This is from the my_utils module.")

      4. Save and submit the resource.

    2. Create a node and reference a resource

      • In the target business flow, right-click MaxCompute, select Create Node > PyODPS 3, and follow the on-screen instructions to create the node.

      • In the node, reference the resource using the ##@resource_reference{"my_utils.py"} directive. The full code is as follows:

        ##@resource_reference{"my_utils.py"}
import sys
import os
# Add the resource's current directory to the Python interpreter's search path.
sys.path.append(os.path.dirname(os.path.abspath('my_utils.py')))
# You can now import and use the resource like a regular module.
import my_utils
my_utils.say_hello("DataWorks")

    3. Run the node. The output "Hello, DataWorks! This is from the my_utils module." appears in the log.

    FAQ

    • Q: When I manually install a package by using a command, why does the custom image test not make progress for a long time?

      • If the task's runtime environment depends on a third-party package from the public internet, the VPC associated with the serverless resource group must have public network access. For more information, see Enable public network access for a resource group.

      • Try switching to a different Python package source, such as https://mirrors.aliyun.com/pypi/simple/. Alibaba Cloud mirrors do not require public network access.

    • Q: What can I do if importing a third-party package fails?

      1. Ensure the custom image has been successfully deployed.

      2. Ensure the task type supported by the image (PyODPS 2 or PyODPS 3) matches your node's type.

      3. Ensure the correct custom image is selected in the node's Properties.

        You cannot select a public resource group.

      4. Check whether the installed package version is compatible with your Python version. For example, pendulum 2.0 and later do not support Python 2.

    Related documents