Python node - DataWorks - Alibaba Cloud Documentation Center

A Python node supports Python 3 syntax but not Python 2 syntax. You can use a Python node to run Python code and periodically schedule jobs. This topic describes how to configure and schedule Python tasks in DataWorks.

Prerequisites

The RAM user that you want to use is added to your workspace.
If you want to use a RAM user to develop tasks, you must add the RAM user to your workspace as a member and assign the Develop or Workspace Administrator role to the RAM user. The Workspace Administrator role has more permissions than necessary. Exercise caution when you assign the Workspace Administrator role. For more information about how to add a member and assign roles to the member, see Add members to a workspace.
A serverless resource group is associated with your workspace. For more information, see the topics in the Use serverless resource groups directory.
Create a Python node before you start development. For more information, see Create a node for a scheduling workflow.

Notes

When you run a task using a serverless resource group, a single task can use a maximum of 64 CU. However, do not exceed 16 CU to prevent resource shortages that can affect task startup.
A Python node provides only a basic Python runtime environment. If your Python code requires third-party packages, you must create a custom image, install the dependencies in the image, and then run the Python node using the image.

Step 1: Develop the Python node

Develop the Python node.

Edit the Python code.

The following code provides a simple example of a bubble sort algorithm.

def bubble_sort(arr):
    n = len(arr)
    # Outer loop controls each traversal
    for i in range(n):
        # Inner loop compares and swaps adjacent elements
        for j in range(0, n-i-1):
            # If the current element is greater than the next, swap them
            if arr[j] > arr[j+1]:
                arr[j], arr[j+1] = arr[j+1], arr[j]
    return arr

# Test code
if __name__ == "__main__":
    example_list = [64, 34, 25, 12, 22, 11, 90]
    sorted_list = bubble_sort(example_list)
    print("Sorted list:", sorted_list)

After you develop the code, click Run Configuration in the right-side panel. Configure settings for the test run, such as the resource group, and then click the Run button to test your code.
Note
Python nodes can be debugged and scheduled only using a serverless resource group. Make sure that the current workspace has a serverless resource group associated. For more information, see Use serverless resource groups.
After you develop and test the Python node script, configure the scheduling properties for the node to run the node periodically .
(Optional) Configure scheduling parameters.
A Python node receives scheduling parameters as command-line arguments. You can add parameter values in order in the scheduling parameter configuration section. Separate multiple values with spaces. In your code, use sys.argv[1], sys.argv[2], and so on to access the parameter values by their index.
Note
- Unlike a PyODPS node, a Python node does not support referencing parameters directly by name, such as $var1. You must explicitly read the parameters using sys.argv.
- Scheduling parameters are passed as command-line arguments only after the node is submitted and deployed and runs as a scheduled instance (such as periodic scheduling or backfill data). You can read them using sys.argv. When you run the node directly in Data Studio (debug run), scheduling parameters are not injected. In this case, sys.argv contains only the script path, and you cannot retrieve the parameter values.
- Example of defining parameters
```
$[yyyymmdd] abc
```
- Example of using parameters
```
import sys

# Print all received arguments
print("All received arguments:", sys.argv)

# Get the first and second arguments
param1 = sys.argv[1]
param2 = sys.argv[2]
print("Parameter 1:", param1)
print("Parameter 2:", param2)
```
For more information about scheduling parameter configuration, see Configure scheduling parameters.
After you configure the schedule settings, save the node before you proceed to the next step.

Step 2: Deploy and manage the node

After you configure the schedule settings, you can submit and deploy the Python node to the production environment .
After deployment, the task runs periodically based on the schedule settings you configured. You can go to Operation and Maintenance Center > Node O&M > Auto Triggered Task O&M > Auto Triggered Task to view the deployed scheduled tasks and perform operations on them. For more information, see Manage scheduled tasks.

Develop tasks in a personal development environment

A personal development environment supports Python programming. To use a personal development environment to edit Python node tasks, see Use a personal development environment.

Run a node by using an associated role

You can associate a RAM role to run a node, which allows you to run node tasks with a specific RAM role for fine-grained permission control and security management.