Create and use MaxCompute resources in DataWorks - DataWorks

If your code or function needs to use a MaxCompute resource, you must first create or upload the resource. You can create or upload resources through MaxCompute SQL commands or the DataWorks visual interface, then reference them in nodes and register functions.

Overview

MaxCompute resources enable user-defined functions (UDFs) or MapReduce jobs. DataWorks lets you upload local or OSS resource packages, or create resources directly. The following table lists the supported resource types.

Resource type	Description	Creation methods
Python	Python code for registering Python UDFs. Extension: .py.	Create in the online editor
JAR	Compiled Java Archive package for running Java programs. Extension: .jar.	Upload a local resource Upload an OSS resource
Archive	Compressed files: .zip, .tgz, .tar.gz, and .tar formats. The extension identifies the compression type.	Upload a local resource Upload an OSS resource
File	Supports .zip, .so, and .jar files.	Upload a local resource Upload an OSS resource Create in the online editor

Resource management: Manage resources, Manage resources by using commands, and Add external resources to DataWorks for management.

Limits

Resource size
- Online editing: up to 10 MB for Python resources, 500 KB for file resources.
- Local upload: up to 500 MB.
- OSS upload: up to 500 MB.
Resource deployment
In standard mode workspaces, deploy the resource to production for it to take effect.
Note
The development and production environments use different compute engines. Verify the compute engine for each environment before querying tables or resources. Data Studio (legacy version): Associate a MaxCompute compute engine.
Resource management
DataWorks only displays resources uploaded through its visual interface. Resources added via other tools (such as MaxCompute Studio) must be manually loaded using the MaxCompute Resources feature. After loading, you can view and manage the resource in DataWorks. Manage MaxCompute resources.

Billing

DataWorks does not charge for resource creation or upload. MaxCompute charges for resource storage. Storage pricing.

Prerequisites

You have associated a MaxCompute compute engine for development tasks.
(Optional) To upload a resource from OSS:
- OSS is activated, a bucket is created, and the file is uploaded. When you upload from OSS, you must select a file from a specific bucket. Create a bucket and Upload objects.
- The Alibaba Cloud account has the required permissions to access the target bucket. To avoid permission errors, grant the permissions to the account in advance. Access control overview.

Go to the resource creation page

Log on to the DataWorks console. In the target region, click Data Development and O&M > Data Development in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Data Development.
On the Data Studio page, right-click the target workflow, select Create Resource, and then select a resource type from the MaxCompute directory.
Note
If no workflow is available, Create a workflow.

Step 1: Create or upload a resource

Upload local or OSS resource packages to DataWorks. For example, package and upload a locally developed UDF before registering it as a function. Python resources and file resources (up to 500 KB) can also be created directly.

Note

When creating or uploading a resource:

If the resource has not been uploaded to MaxCompute, select Upload to MaxCompute. If the resource already exists in MaxCompute, deselect Upload to MaxCompute. Otherwise, the upload will fail.
Selecting Upload to MaxCompute stores the resource in both DataWorks and MaxCompute. Deleting it from MaxCompute by command does not remove it from DataWorks.
The resource name does not need to match the uploaded file name.

Online editor

Configure the resource as needed.

Note

For Python resources larger than 10 MB or file resources larger than 500 KB, use Method 2: Upload a local resource or Method 3: Upload an OSS resource.
For a tutorial, see Analyze IP address origins by using a MaxCompute UDF.

When creating a Python resource or a File resource, configure the following fields:

Engine type: Select MaxCompute.
Engine instance: Select the target engine instance.
Path: Select the business flow path for storing the resource.
Upload to ODPS: When selected, the resource is synchronously uploaded to MaxCompute.
Name: Enter the resource name. For Python resources, the file name must include the .py extension.

For File resources, you also need to configure File source, which can be set to Local, OSS (one-click authorization), or Online editor.

Local resource

Configure the resource as needed.

In the Create resource dialog box, configure the following fields based on the resource type (JAR, Archive, or File):

Engine type: Select MaxCompute.
Engine instance: Select the target engine instance.
Resource type: Select JAR, Archive, or File.
Path: Select the storage path for the resource.
Upload to ODPS: When selected, the resource is synchronously uploaded to MaxCompute.
File source: Select Local or OSS (one-click authorization). File type also supports Online editor.
Upload file: Click Click to upload to upload a local file.
Name: Enter the resource name. JAR type requires the .jar extension. Archive type requires the .zip, .tgz, .tar.gz, or .tar extension.

OSS resource

Configure the resource as needed.

Note

You can use this method to upload a resource of up to 500 MB.
The Alibaba Cloud account that you use for the upload must be granted the AliyunDataWorksAccessingOSSRole policy. Follow the on-screen instructions to grant the permission with a single click.

Step 2: Submit and deploy

After creating the resource, click the Submit icon in the toolbar to submit it to the scheduling server in the development environment.

Note

To use this resource in production, deploy it to the production environment. Deploy tasks.

Step 3: Use the resource

Use in a node

Reference the resource in the relevant node. A successful reference adds a line like @resource_reference{"Resource Name"} to the node configuration. The format varies by node type. For example, PyODPS 2 nodes use ##@resource_reference{"Resource Name"}.

Note

If no node exists, Create a compute node.
If your PyODPS code depends on third-party packages, use a custom image to install them in the runtime environment. Custom images.

Reference a resource as shown below.

In the DataWorks Data Studio interface, follow these steps to reference a resource:

Double-click the target PyODPS node to open it.
In the Resource directory on the left, right-click the target resource file (for example, pyodps_packagetest.py), and select Reference resource from the context menu.
The code editor automatically generates ##@resource_reference{"pyodps_packagetest.py"} at the top of the file, indicating a successful reference. You can then call methods from the resource using import.

Register a function

To register a function, first Create and use a custom function. On the function configuration page, enter the resource name.

Important

The resource must be submitted before you register a function. Step 2: Submit and deploy.

In the Register function form, set the Resource list field to the name of the uploaded resource (for example, xc_ip2region.jar), and configure the Class name (for example, org.alidata.odps.udf.Ip2Region) and Command format (for example, getregion('ip')) fields.

MaxCompute built-in functions: Use built-in functions.

To view functions in a compute engine, manage function history, or perform other management operations: Manage MaxCompute functions.

Manage resources

In the resource directory of a workflow, right-click the target resource to perform management operations:

View History: View and compare saved or submitted versions to track changes.
Note
To compare versions, you must select at least two versions.
Delete: Deletes the resource from the development environment only. Deploy this change to remove it from production; the resource is removed from production after a successful deployment. Deploy tasks.

Appendix 1: Manage resources with commands

The following table describes common commands for resource operations.

Operation	Description	Role	Platform
Add a resource	Adds a resource to a MaxCompute project.	Users with the `Write` permission on the resource.	You can run these commands on the following platforms: Connect to MaxCompute with odpscmd Analyze connections using SQL Connect using odpscmd in Cloud Shell Connect by using DataWorks Develop and submit an SQL script
View resource information	Views the details of a resource.	Users with the `Read` permission on the resource.
List resources	Lists all resources in the current project.	Users with the `List` permission on the project.
Create an alias for a resource	Creates an alias for a resource.	Users with the `Write` permission on the resource.
Download a resource	Downloads a resource from a MaxCompute project to your local machine.	Users with the `Write` permission on the resource.
Delete a resource	Deletes an existing resource from a MaxCompute project.	Users with the `Delete` permission on the resource.

If no project name is specified, commands operate on the current project. Examples:

View all resources in the current project. In Data Studio, this queries the development environment's compute engine by default.
```
list resources;
```

View resources in a specific project:

use <MaxCompute_project_name>;
list resources;

Full command reference: Resource operations.

Appendix 2: Add external resources to DataWorks

Use the MaxCompute Resource feature to load MaxCompute resources (up to 500 MB) into DataWorks for visual management. Manage MaxCompute resources.