Resource management - DataWorks - Alibaba Cloud Documentation Center

The resource management feature in Data Studio allows you to create and manage resources and functions for MaxCompute, EMR (including EMR Jar, EMR File, and Ray File), CDH, and Flink (including Flink Jar, Flink File, and Flink Python) for use in data development or SQL queries.

Overview

The resource management feature in Data Studio lets you centrally manage your resources and functions. You can upload files from OSS or local storage to create resources and then register these resources as functions for use in data development nodes. You can only manage resources and functions created using the resource management module.

Prerequisites

To create a resource by uploading a file from OSS:

You have activated OSS and created a bucket, and stored the files to be uploaded in the OSS bucket. To upload from OSS, you need to select a file from a specified bucket. Therefore, before using this method to create a resource, you must first create a bucket and store the relevant files in it. For more information, see Simple upload.
Your Alibaba Cloud account must have the required permissions to access the destination bucket. Grant the required permissions before uploading to prevent errors.

Limitations

Resource size: The maximum size of a resource that you can upload is 500 MB.
Resource deployment: If you use a workspace in standard mode, you must deploy a resource to the production environment to make it available in the production project.
Note
The data source information differs between the development environment and the production environment. Before you query tables or resources in an environment, verify the data source information for the target environment.
Resource management: DataWorks allows you to view and manage only the resources uploaded through DataWorks.

Supported resources and functions

Resources

The resource management feature allows you to upload and create resources through a visual interface. You can use these resources in data development tasks or register them as UDFs.

Data source	Resource	Creation methods
		Local	OSS
MaxCompute	MaxCompute Python Python code for registering a Python UDF. The file must have a `.py` extension.
	MaxCompute Jar A compiled Java JAR package for running a Java program. The file must have a `.jar` extension.
	MaxCompute Archive Only compressed files with the `.zip`, `.tgz`, `.tar.gz`, or `.tar` extension can be uploaded as resources. The compression type is identified by the file name extension.
	MaxCompute File You can upload files of any type as File resources. The actual usage depends on the capabilities of each engine.
EMR	EMR Jar A compiled Java JAR package for running a Java program. The file must have a `.jar` extension.
	EMR File You can upload files of any type as File resources. The actual usage depends on the capabilities of each engine.
	Ray File A dependency file for Serverless Ray nodes, such as multi-file Python jobs, configuration files, or data files. Reference it in a node by using `##@resource_reference`. For more information, see Create and manage EMR resources and functions.
CDH	CDH Jar A compiled Java JAR package for running a Java program. The file must have a `.jar` extension.
	CDH File You can upload files of any type as File resources. The actual usage depends on the capabilities of each engine.
Flink	Flink Jar A compiled Java JAR package for running a Java program. The file must have a `.jar` extension.
	Flink File You can upload files of any type to Flink. These files can be used as additional dependency files or Python Libraries for Flink JAR and Flink Python nodes.
	Flink Python You can upload Python files with the `.py` extension to Flink for use in Flink Python node development. To upload files with the `.zip` extension, use the Flink File resource type.

Functions

You can use resources to create MaxCompute, EMR, CDH, and Flink functions. These functions can be used in Data Studio and SQL query.

Entry point

Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane, click to go to the Resource Management page.
On the Resource Management page, click the icon to create a resource or function. You can also first click Create Directory to organize the resource management directory, and then right-click the desired directory and select Create to create a resource or function.

Create and manage resources and functions

Other operations

Clone resources and functions

You can quickly create resources and functions by cloning existing ones.

In the left-side Resource Management panel, right-click the resource or function that you want to clone, and select Cloning from the context menu.
In the dialog, modify the Name and Path (or keep the default values), and click Confirm to start cloning.
After the cloning is complete, you can view the cloned resource or function in Resource Management and edit or modify other parameter information.

Version management

You can use the view and compare features in version management to quickly analyze version differences of resources or functions and make adjustments.

In the left-side Resource Management panel, double-click the name of the resource or function that you want to view and manage versions for, to open the resource or function editing page.
On the right side of the resource or function editing page, click Version. On the Version page, view and manage Developer Record and Publish Record information.
- View a version:
  1. In the Developer Record or Publish Record tab, find the version of the resource or function that you want to view.
  2. Click View in the Operation column to go to the details page and view the resource or function code information.
- Compare versions:
  In the Developer Record or Publish Record tab, you can compare different versions of a resource or function. The following example uses developer records to demonstrate the comparison operation.
  - Compare within the development or production environment: On the Developer Record tab, select two versions and click Select Comparison at the top to compare the code content and schedule settings of the resource or function across different versions.
  - Compare between development and production or build environments:
    1. On the Developer Record tab, locate the desired version of the resource or function.
    2. Click Compare in the Operation column. On the details page, select a version from Publish Record or Build Records for comparison.
- Download a version:
  You can only download file versions from the Developer Record of deployed Resource files. On the Developer Record tab, find the target version, and click Download in the Operation column to download the resource file to your local machine.