All Products
Search
Document Center

DataWorks:Use the O&M Assistant feature

Last Updated:Jan 12, 2024

If a task that is run on an exclusive resource group for scheduling depends on a third-party package, you must use the O&M Assistant feature to install the third-party package on the exclusive resource group for scheduling first. This ensures that the task can be run as expected. DataWorks O&M Assistant provides multiple built-in third-party packages. You can directly install and use a built-in third-party package. If the built-in third-party packages cannot meet your business requirements, you can use a Shell command to upload the required package or resource file for installation.

Prerequisites

Limits

You must take note of the following limits when you use the O&M Assistant feature:

  • You can use the O&M Assistant feature only on an exclusive resource group for scheduling. You cannot use the feature on an exclusive resource group for Data Integration or a resource group of the new version. .

  • You can only create a command that is used to install a third-party package. You cannot modify the command.

  • You can upload only a resource file whose size is no more than 50 MB to an exclusive resource group for scheduling.

    Note

    If you want to upload a MaxCompute resource file whose size is greater than 50 MB, you can upload the resource file in a visualized manner in the DataWorks console. For more information, see Create and use MaxCompute resources.

  • A third-party Python package that is installed on an exclusive resource group for scheduling by using the O&M Assistant feature can be referenced only when the exclusive resource group for scheduling is used to run PyODPS tasks.

    Note

    For information about how to reference a third-party Python package in a Python user-defined function (UDF) of MaxCompute, see Example: Reference third-party packages in Python UDFs.

Go to the O&M Assistant page

  1. Log on to the DataWorks console. In the left-side navigation pane, click Resource Groups.

  2. On the Exclusive Resource Groups tab of the Resource Groups page, find the desired exclusive resource group for scheduling, click the image.png icon in the Actions column of the resource group, and then select O&M Assistant.

    You can create a command based on your business requirements and run the command to install the third-party package that is required to run tasks.

    Note

    You can use the O&M Assistant feature only on an exclusive resource group for scheduling. You cannot use the feature on an exclusive resource group for Data Integration or a resource group of the new version.

Install a third-party package

  1. Create a command that can be used to install a third-party package.

    The installed third-party package can be referenced when the exclusive resource group for scheduling is used to run scheduling tasks. On the O&M Assistant page, click Create Command. In the Create Command panel, you can set the Command Type parameter to Quick Installation or Manual Installation.

    Method 1: Quick Installation

    This method is used to install a built-in third-party package.

    image.png

    The following table describes the key parameters that you must configure when you install a built-in third-party package by using this method.

    Parameter

    Description

    Command Name

    The name of the command. You can specify a name based on your business requirements.

    Command Type

    The method used to create the command. Select Quick Installation.

    If you select this method, DataWorks automatically generates a Shell command that can be used to install the third-party package you select.

    Package to Install

    The third-party package that you want to install and the version of the package.

    DataWorks provides multiple built-in third-party packages of the Python2, Python3, and Yum type. You can select a built-in third-party package based on your business requirements. Example for commonly used built-in third-party packages:

    • Aliyun-python-sdk-core: the core library of Alibaba Cloud SDK for Python. This library provides the basic API operation call and authentication features and is the basic library used for interactions with Alibaba Cloud services.

    • NumPy: the basic library used for scientific computing and data analysis. This library provides the high-performance computing feature for multi-dimensional arrays and numeric values.

    • Pandas: provides high-performance, easy-to-use data structure and data analysis tools and is used to process and analyze structured data.

    You can view all built-in third-party packages that are supported in the DataWorks console.

    Generated Shell Script

    The Shell command that is generated by DataWorks based on the built-in third-party package you select.

    You can run the Shell command to install the built-in third-party package.

    For example, after you select the aliyun-python-sdk-core package, DataWorks automatically generates the pip install aliyun-python-sdk-core command. You can run the command to install the package.

    Timeout

    The timeout period for running the command. Unit: seconds. If the running duration of the command exceeds the timeout period that you specify, the command is forcefully stopped.

    Method 2: Manual Installation

    If the built-in third-party packages cannot meet your business requirements, you can manually specify a Shell command and use the command to upload the required package or resource file. image

    Note

    If you use this method, you cannot run pip commands to install third-party packages.

    Parameter

    Description

    Command Name

    The name of the command. You can specify a name based on your business requirements.

    Command Type and Command Content

    Command Type: Select Manual Installation.

    Command Content: Manually enter the Shell command that you want to run to upload a package or resource file from your on-premises machine.

    Note
    • The success rate of running manually entered Shell commands cannot be ensured.

    • After a resource file is uploaded, you must use the absolute path of the resource when you reference the resource in nodes and tasks in DataStudio.

    Installation Directories

    The directory to which you want to install the third-party package by using the command. DataWorks automatically adds the directory to the related directory whitelist to ensure that the directory is accessible. If you specify multiple directories, separate them with semicolons (;).

    Note
    • You can install the third-party package in a /home/ directory or a directory other than /home/. If you want to install the third-party package in a /home/ directory, you can use only the /home/admin/usertools/tools/ path of the exclusive resource group for scheduling.

    • If no directory is specified, the third-party package is automatically installed in the /home/admin/usertools/tools/ path.

    Timeout

    The timeout period for running the command. Unit: seconds. If the running duration of the command exceeds the timeout period that you specify, the command is forcefully stopped.

  2. After you complete the configuration, click Create.

  3. Run the command.

    After the command is created, you can find the command on the O&M Assistant page and click Run command in the Actions column of the command to install the third-party package. After the third-party package is installed, you can use the package to run scheduling tasks on the exclusive resource group for scheduling.

Manage a command

After a command is created, you can find the command on the O&M Assistant page and perform the following operations that are related to the command:

Note

You can click View Detailed Environment Configuration to view the overall environment configuration of the related resource group. For example, you can view the third-party package that is installed, and the version and status of the package.

  • View information of the command: You can view information such as the running status, execution ID, and content of the command.

  • View the running result: You can view whether the running of the command is successful or fails. If the running fails, you can analyze the cause for the failure based on logs and resolve the issue.

  • Contact technical support: If you cannot resolve the issue that you encounter, you can follow the instructions that are displayed to join the DataWorks user DingTalk group and contact technical support.

What to do next

After a third-party package is installed, you can reference the package when you use the related exclusive resource group for scheduling to run scheduling tasks. References: