When tasks running on an exclusive resource group for scheduling depend on third-party packages, you must use the resource group's O&M Assistant to install the required packages. This ensures that the tasks can run properly. DataWorks supports one-click installation for various built-in third-party packages. If the built-in packages do not meet your business needs, you can also install other third-party packages by using shell commands.
Applicability
Resource group limits: This feature applies only to an exclusive resource group for scheduling. Other types of resource groups, such as exclusive resource groups for data integration and serverless resource groups, are not supported.
Permissions: You must have the AliyunDataWorksFullAccess or ModifyResourceGroup RAM policy.
For more information about how to grant permissions, see Product and console access control: RAM policies.
Considerations
After a third-party package is installed on an exclusive resource group for scheduling, it applies to all tasks that run on that Resource Group. Version conflicts or compatibility issues can affect all tasks. We recommend that you carefully select and test packages before installing them.
Quotas and limits
Command management: Created installation commands cannot be modified. You can only create new commands or delete existing ones.
Applicable task types: Python third-party packages installed by this method apply only to Python task nodes, such as PyODPS 2 and PyODPS 3 nodes, that run on this Resource Group.
To use a third-party package in a MaxCompute Python user-defined function (UDF), see UDF example: Use third-party packages in a Python user-defined function (UDF).
Command support: The manual input method does not support the
pipcommand.
Access O&M Assistant
Log on to the DataWorks console. In the target region, click Resource Group in the left-side navigation pane to open the resource group list.
On the Exclusive Resource Group tab, find the resource group that is used for Data Scheduling, click the
icon next to the Resource Group, and select O&M Assistant to go to the corresponding page.
Install a third-party package
Create an installation command
On the O&M Assistant page, click Create Command and select an installation method.
Quick installation
Use this method to install common third-party packages that are built into DataWorks.
The following table describes the key parameters.
Parameter
Description
Command Name
A custom name for the command.
Command Type
Select Quick Installation.
DataWorks automatically generates a shell installation command based on the package you select.
Built-in third-party package
Select the package and version to install.
DataWorks supports a variety of common Python 2, Python 3, and Yum packages. Examples:
Aliyun-python-sdk-core: The core library of the Alibaba Cloud Python SDK. It provides basic functionality for interacting with Alibaba Cloud services, such as API calls and authentication.
NumPy: A foundational library for scientific computing and data analysis. It provides high-performance multidimensional arrays and numerical computing features.
Pandas: Provides high-performance, easy-to-use data structures and data analysis tools for processing and analyzing structured data.
For a complete list, see the user interface.
Generated Shell Script
DataWorks automatically generates the installation command based on your selection.
For example, if you select aliyun-python-sdk-core, the system generates the following command:
pip install aliyun-python-sdk-coreTimeout
The maximum run time of the command, in seconds. If this period is exceeded, the command is terminated.
Manual input
If the built-in packages do not meet your needs, you can install third-party packages by using commands.
Parameter
Description
Command Name
A custom name for the command.
Command Type
Select Manually enter.
Command Content
Enter the command that you want to run. Example:
yum install -y git.ImportantThe manual input method does not support using pip commands to install third-party packages.
The success of manually entered commands is not guaranteed. We recommend that you validate the script in advance.
After a third-party package is installed, you must use an absolute path to reference the package in a Data Studio node.
Installation Directories
Specify the directory paths that the command can access. Separate multiple paths with semicolons (;). DataWorks adds the specified directories to a whitelist to ensure their accessibility.
You can install packages to a /home/ directory or a non-/home/ directory. If you install packages to a /home/ directory, they must be stored in the /home/admin/usertools/tools/ path.
If you do not specify a directory, the default path /home/admin/usertools/tools/ is used.
Timeout
The maximum execution time for the command, in seconds. If the timeout period is exceeded, the command is terminated.
Click Confirm to complete the command configuration.
Run the installation command
On the O&M Assistant page, find the command that you created and click Run command in the Operation column. The system automatically generates a run record for the command.
View the run result
In the command run record, view basic information such as the Run Status and Command Content. Click View Result in the Operation column of the corresponding record to analyze the logs for the cause of failure and troubleshoot the issue. After the package is successfully installed, you can use it in scheduling tasks on the current Resource Group.
View the environment configuration
On the O&M Assistant page, click View Detailed Environment Configuration in the upper-left corner to view the Resource Group's overall environment configuration, such as installed third-party packages, their versions, and statuses.
Use the third-party package in a task
Data Studio (new version)
Go to Data Studio: Go to the DataWorks workspaces page. In the top navigation bar, select the target region. Find the target workspace and choose in the Operation column.
Configure the resource group: In Data Studio, find the task node for which you want to test the third-party package. In the right-side pane, click Scheduling Settings.
Resource Group: Select an exclusive resource group for scheduling.
If the target Resource Group is not displayed, check whether the Resource Group is bound to the current workspace. You can go to the Resource Group page, find the target Resource Group, and click Associate Workspace in the Operation column.
Debug the node: In the Run Configuration pane on the right, configure Computing Resources, Resource Group, and Script Parameters. Then, click Run in the top toolbar.
Publish the node: In the top toolbar, click Publish to deploy the node to the production environment.
Data Development (old version)
Go to Data Development: Log on to the DataWorks console. After you switch to the target region, choose in the navigation pane on the left. From the drop-down list, select the desired workspace and click Data Analytics.
Configure the resource group: In Data Development, find the task node for which you want to test the third-party package. In the right-side pane, click Scheduling Settings.
Resource Group for Scheduling: Select an exclusive resource group for scheduling.
If the target Resource Group is not displayed, check whether the Resource Group is bound to the current workspace. You can go to the Resource Group page, find the target Resource Group, and click Associate Workspace in the Operation column.
Debug the node: In the top toolbar, click Run with Parameters (
). Configure Resource Group Name and then click Run.Publish the node: In the top toolbar, click Save and Submit to deploy the node to the production environment.
Related documents
After a third-party package is installed, tasks that run on the exclusive resource group for scheduling can use the package. For more information, see the following topics: