Documentation
Home Page DataWorks User Guide Data Studio (new version) Best practices for Data Studio Use third-party packages in PyODPS nodes

Use third-party packages in PyODPS nodes

Updated at:
Copy as MD
Product

When built-in PyODPS functions are insufficient for complex business requirements, you can reuse existing Python code or use open-source libraries. DataWorks offers two solutions: referencing custom Python resources to load scripts, or configuring the runtime environment with a custom image or O&M Assistant to install third-party packages. This document explains how to use these methods in PyODPS nodes to extend your data processing capabilities.

Choose your approach

Select an approach based on your environment and requirements:

Scenario

Resource group type

Solution

Using open-source third-party packages

serverless resource group

Install packages using a custom image

exclusive resource group for scheduling

Install packages using O&M Assistant

Using only custom.py script files

serverless resource group/exclusive resource group for scheduling

Reference a custom Python resource

Workflow for each solution:

Prerequisites

Before you start, understand the following key concepts to determine your configuration method.

Key concepts

  1. Node type: PyODPS 2 vs. PyODPS 3

    • PyODPS 2: Based on a Python 2.7 environment.

    • PyODPS 3: Based on a Python 3.7+ environment.

    • Recommendation: We strongly recommend using PyODPS 3 because Python 2 has reached its end of life and is no longer officially supported. This document uses PyODPS 3 as the main example.

  2. Resource group type: serverless resource group vs. exclusive resource group for scheduling

    • serverless resource group: Recommended. This type is elastic and maintenance-free. You can manage third-party dependencies using a custom image, which is powerful and flexible.

    • exclusive resource group for scheduling: This is a legacy solution and is no longer recommended. It requires you to purchase and maintain ECS servers in advance. Resources cannot be elastically scaled, and installing dependencies using O&M Assistant is restrictive and risks polluting the environment.

Check your resource group type

In the DataWorks console, navigate to the workspace details page and open the Resource Group tab to view the type of resource group bound to your workspace.

  • If the type is General-purpose, you are using a serverless resource group.

  • If the type is Data Scheduling, you are using an exclusive resource group for scheduling.

Install open-source packages using a custom image

Important

This method applies to serverless resource groups.

This section walks you through a complete end-to-end example: creating a custom environment that includes the pendulum package and calling it in a PyODPS 3 node to get and format the current time for a specific time zone.

Step 1: Create a custom image with pendulum

A custom image defines the runtime environment for a serverless resource group.

  1. Log on to the DataWorks console and click Image Management in the left-side navigation pane.

  2. Switch to the Custom Image tab.

  3. Click Create Image in the upper-left corner and configure the following parameters:

    Parameter

    Description

    Image Name

    The name of the custom image, for example pyodps3_with_pendulum.

    Reference Type

    Select DataWorks Official Images.

    Image Name/ID

    From the drop-down list, select the official DataWorks image dataworks_pyodps_task_pod .

    Supported Task Type

    Select the PyODPS 3 task type.

    Installation Package

    Select Python3 and then select the system-provided pendulum package from the drop-down list.

    To install an open-source package that is not built-in, you can use Script mode to install it manually. For more information about configuration options, see Parameters for creating a custom image.
    Important

    To install open-source packages from the public internet, the VPC bound to the serverless resource group must have public network access .

  4. Click Confirm to create the custom image.

  5. In the list of custom images, Test and then Publish the target image. You can publish an image only after it passes the test.

  6. In the Operation column for the target image, click image > Change Workspace to bind the custom image to the appropriate workspace.

Step 2: Configure a PyODPS 3 node

  1. In the left-side navigation pane, choose Data Development and O&M > DataStudio, select the target workspace from the drop-down list, and click Enter Data Studio.

  2. Click the image icon to the right of the project directory to create a new PyODPS 3 node. For example, name it pyodps3_pendulum_test.

  3. In the code editor of the pyodps_pendulum_test node, write Python 3 code:

    # Because pendulum is installed in the custom image, you can import it directly.
import pendulum
print("Start testing the third-party package pendulum...")
try:
    # Get the current time in the 'Asia/Shanghai' time zone
    shanghai_time = pendulum.now('Asia/Shanghai')
    # Print the formatted time and time zone information
    print(f"Successfully imported the 'pendulum' package.")
    print(f"The current time in Shanghai is: {shanghai_time.to_datetime_string()}")
    print(f"The corresponding time zone is: {shanghai_time.timezone_name}")
    print("\nTest passed! The PyODPS node successfully called the third-party package.")
except Exception as e:
    print(f"Test failed. An error occurred: {e}")

Step 3: Run and verify the result

  1. On the right side of the node edit page, under Run Configuration > Resource Group, select the prepared Serverless Resource Group and change the image to the created image pyodps3_with_pendulum.

    Important

    If you cannot find the target image, bind the image to the current workspace. For details, see step 6 in Step 1.

  2. Click the Run button in the toolbar to run the code and wait for it to complete.

  3. View the run logs below, and you will see output similar to the following, which confirms that the pendulum package has been successfully called.

    Start testing the third-party package pendulum...
Successfully imported the 'pendulum' package.
The current time in Shanghai is: 2025-09-27 15:45:00
The corresponding time zone is: Asia/Shanghai
Test passed! The PyODPS node successfully called the third-party package.

Step 4: Publish the PyODPS 3 node

After you complete the test, go to Scheduling Settings > Scheduling Policy on the right side of the node editing page, select the serverless resource group, and change the image to the image pyodps3_with_pendulum. Then, publish the node to the Operation Center by following the standard procedure.

Install packages using O&M Assistant

Important

This method applies to legacy exclusive resource groups for scheduling, which are not recommended. We suggest migrating to the more powerful and flexible serverless resource groups.

  1. On the DataWorks Workspaces page, switch the region at the top, find the target workspace, and click Details in the Operation column.

  2. In the left-side navigation pane, click Resource Group. Find the bound exclusive resource group for scheduling and click image > O&M Assistant in the Actions column.

  3. On the O&M Assistant page, click Create Command in the upper-left corner.

  4. Enter the command for your Python version:

    • Python 3 (PyODPS 3): Keep the default options and select the pendulum package from the drop-down list under the Python3 package type.

    • Python 2 (PyODPS 2): Select the Manual Input mode and enter the command content.

      pip install --upgrade pip
pip install "pendulum<2.0"

  5. On the O&M Assistant page, click Run command in the Actions column. After successful execution, you can directly use import pendulum in the corresponding PyODPS node.

Reference a custom Python resource

If you just want to call a function from another .py file that you wrote, do the following:

  1. Create a Python resource:

    1. On the Resource Management page, click the image icon and select New Resource > MaxCompute Python.

    2. In the Create Resource and Function dialog box, enter a resource Name (for example, my_utils.py), and click Confirm.

    3. Upload the local file my_utils.py to the Document Content section, and for Data Sources, select the compute resource that you bound in the prerequisites.

    4. In the toolbar, Save and then Publish the resource.

      After publishing, the deployment pipeline proceeds through the Build Package, Production Checker, and Deploy to Production stages. When the status of all three stages shows Completed, the resource has been successfully deployed to the production environment.

  2. Create a PyODPS 3 node and reference the resource:

    • In the left-side navigation pane, click the image icon to go to the Project Directory on the DataStudio page.

    • Click the image icon next to the project directory, select New Node > MaxCompute > PyODPS 3, and follow the on-screen instructions to create the node.

    • On the node editing page, you can reference the resource mentioned above by using ##@resource_reference{"my_utils.py"}. The code is as follows: 

      ##@resource_reference{"my_utils.py"}
import sys
import os
# Add the current directory containing the resource to the Python interpreter's search path
sys.path.append(os.path.dirname(os.path.abspath('my_utils.py')))
# You can now import and use the resource like a regular module
import my_utils
my_utils.say_hello("DataWorks")

  3. Run the node. The logs will show the output "Hello, DataWorks! This is from my_utils module.".

FAQ

  • Q: When I install a package manually, the custom image test takes a long time or gets stuck.

    • If the task environment needs to access third-party packages from the public internet, the VPC bound to the serverless resource group must have public network access. For more information, see Enable public network access for a resource group.

    • Try switching to a different Python package source, such as https://mirrors.aliyun.com/pypi/simple/.

  • Q: What should I do if importing a third-party package fails?

    1. Confirm that the custom image has been published successfully.

    2. Confirm that the task type supported by the image (PyODPS 2 or PyODPS 3) matches the type of node you created.

    3. In the Scheduling settings of the PyODPS node, confirm that the custom image is correctly selected.

      The resource group cannot be a shared resource group.

    4. Check if the installed package version is compatible with your Python version (for example, pendulum 2.0+ does not support Python 2).

Related documents

Previous：Find IP geolocation by using MaxCompute UDFsNext：DataStudio (legacy)