Create and use MaxCompute resources and functions - DataWorks

Data Studio allows you to manage resources in MaxCompute projects. You can create resources from local files or files stored in Object Storage Service (OSS), and register these resources as functions for use in data development nodes. This topic describes how to create different types of MaxCompute resources and functions in Resource Management.

Prerequisites

You have associated a MaxCompute compute resource.
You have developed the resource files. You can upload the files from your local computer or obtain them from OSS. To create resources from files stored in OSS, make sure that the following requirements are met:
- You have activated OSS, created a bucket, and stored the resource files to upload in the bucket. For more information, see Create a bucket and Simple upload.
  Note
  For information about supported resource files, see Resource overview.
- The Alibaba Cloud account used to upload files has permissions to access the destination bucket. To prevent permission issues, grant the required permissions to the account before you upload files.

Go to Resource Management

Go to the Workspaces page in the DataWorks console. In the top navigation bar, select a desired region. Find the desired workspace and choose Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane, click the Resource Management icon to open the Resource Management page.
On the Resource Management page, click the icon to create a resource or function. You can also click Create Directory first to organize your resource directories. Then, right-click a directory, choose Create, and select the type of resource or function to create.

Create and manage resources

Resource overview

Resources are the basis for implementing UDF or MapReduce in MaxCompute. In Data Studio, you can upload resources from your local computer or OSS by using the graphical interface. These resources can be read and used when user-defined functions (UDFs) and MapReduce jobs run. The following table describes the supported MaxCompute resource types.

Important

Uploading resources to MaxCompute through DataWorks incurs MaxCompute Storage fees.

Resource type	Description
Resource type	Description
Python	A file that contains Python code and is used to register Python UDFs. The file extension is `.py`.
JAR	A compiled Java JAR package used to run Java programs. The file extension is `.jar`.
Archive	Only compressed files such as `.zip`, `.tgz`, `.tar.gz`, `.tar`, and `.jar` can be uploaded as archive resources. The compression format is determined by the file extension in the resource name.
File	For resources of the `File` type, you can upload any file. Whether a file can be used depends on the engine.

Limits

The following limits apply when you upload resources:

Resource size:
- Online editing: The maximum size is 10MB for Python resources and 500KB for File resources.
- Local file upload: The maximum resource size is 500MB.
- OSS file upload: The maximum resource size is 500MB.
Resource deployment: If you use a workspace in standard mode, you must deploy resources to the production environment for them to take effect.
Note
Data source information may differ between the development and production environments. Before you query tables or resources in an environment, verify the data source information for that environment.
Resource management: DataWorks allows you to view and manage only resources uploaded through DataWorks.

Create a resource

You can upload MaxCompute resources from your local computer or OSS. After you create a resource, you can reference it directly in Data Studio or register it as a function.

On the Resource Management page, create a resource. In the Create Resource and Function dialog box, specify the resource Type, storage Path, and resource Name.

Upload a file from OSS or your local computer as the source. The following table describes the key parameters for uploading resources.

Parameter	Description
File source	The source of the file. Valid values are Local and OSS.
File content	If you select Local, in the Document Content section, click Click Upload to upload a local file. If you select OSS, select the OSS file from the Document Content drop-down list.
Data source	Select the data source to which the uploaded MaxCompute resource belongs.

In the top toolbar, click Save and then Publish for the resource. Only deployed resources can be used in Data Studio.

Use a resource

After you create a resource, open a data development node for editing and click Resource Management in the left-side navigation pane. Find the resource or function, right-click it, and choose Insert Resource Path. After the resource is referenced, code in the following format appears: ##@resource_reference{"resource_name"}.

Note

For example, a PyODPS 3 node displays ##@resource_reference{"example.py"}. The format varies by node type. Refer to the actual interface.

In addition to using a resource directly, you can register it as a function and then use the function in a data development node.

Manage resources

DataWorks allows you to view and manage only resources uploaded through its graphical interface. On the Resource Management page, click a resource to perform the relevant management operations.

View historical versions: View and compare deployed resource versions to identify changes between versions.
Note
Select at least two versions to compare.
Delete a resource: Deleting a resource removes it only from the project in the development environment. To delete it from the production environment, deploy the deletion operation. The resource is removed from the production environment only after the deployment succeeds. For more information, see Deploy a node.
View other resources.
A MaxCompute project may also contain resources uploaded through methods other than DataWorks. You can view these resources in the following ways:
- Use Data Catalog Management to view all resources in a MaxCompute project.
  After you add a MaxCompute project to Data Catalog, open the corresponding MaxCompute catalog and view all resources in the project under the resource directory.
- Use MaxCompute SQL node to view other resources in a MaxCompute project.
  - View all resources in the current project. If you create a MaxCompute SQL script in Data Studio and run this command, the MaxCompute compute resource associated with the development environment is accessed by default.
```
list resources;
```
  - View all resources in a specified project.
```
use maxcompute_project_name;
list resources;
```
  For more information about resource commands, see Manage resources.

Create and manage functions

Before you create a function, complete the steps in Create a resource.

Note

When you create MaxCompute resources, refer to Developing Java UDFs and UDF development (Python 3) to prepare the resource files.

Function overview

In Resource Management in Data Studio, you can register resources as functions for use in data development nodes or SQL queries. Follow the procedure in Create a function to register uploaded and deployed resources as functions. You can also create embedded functions by using JAVA, PYTHON2, or PYTHON3, or directly use MaxCompute built-in functions.

Create a function

On the Resource Management page, create a function. In the Create Resource and Function dialog box, specify the function Type, storage Path, and function Name.

Create a function resource and configure its settings.

Before you configure a MaxCompute function, make sure that you have registered the MaxCompute project as a compute resource in DataWorks and uploaded a MaxCompute resource. The following table describes the key parameters for a MaxCompute function.

Parameter	Description
Function type	Select the function type. Valid values are MATH (mathematical functions), AGGREGATE (aggregate functions), STRING (string functions), DATE (date functions), ANALYTIC (window functions), and OTHER (other functions).
Class Name	The class name of the UDF, in the format `resource_name.class_name`. The resource name can be a Java package name or a Python resource name. DataWorks supports JAR and Python MaxCompute resources for creating UDFs. Configure the class name based on the resource type: For a JAR resource, set Class Name to `java_package_name.class_name`. In `IntelliJ IDEA`, use `Copy Reference` to obtain the value. For example, if `com.aliyun.odps.examples.udf` is the Java package name and `UDAFExample` is the class name, set Class Name to `com.aliyun.odps.examples.udf.UDAFExample`. For a Python resource, set Class Name to `python_resource_name.class_name`. For example, if `LcLognormDist_sh` is the Python resource name and `LcLognormDist_sh` is the class name, set Class Name to `LcLognormDist_sh.LcLognormDist_sh`. Note Do not include the `.jar` or `.py` extension in the resource name. You must submit and deploy a resource before you can use it.
Type	Select Resource function or Embedded function: If you select Resource function, configure Resource List. If you select Embedded function, in addition to selecting Resource List, configure Language (`JAVA`, `PYTHON2`, and `PYTHON3` are supported) and Code.
Resource List	Select the resources required to register the function. Wizard mode: You can select only resources uploaded or added to DataWorks. Script mode: You can enter any resources in the corresponding data source. If a UDF uses multiple resources, separate the resource names with commas (,). Note You do not need to specify the paths of resources that have already been added. In script mode, you can manually specify resources that DataWorks does not support uploading through the graphical interface, such as table resources, and resources uploaded to MaxCompute through other methods that are not managed through the DataWorks graphical interface.
Command Format	An example that shows how to use the UDF.

In the top toolbar, click Save and then Publish for the function. Only deployed functions can be used in Data Studio.

Use functions

Use UDFs

After you create and deploy a function, you can reference it directly in data development nodes or SQL queries.

When you edit a data development node, click Resource Management in the left-side navigation pane. Find the resource or function, right-click it, and choose Insert Function.
After you reference the function, its name is inserted in the current node editor, for example, example_function().
When you edit an SQL query, you can use the function directly.

SELECT example_function(column_name) FROM table;

Use built-in functions

DataWorks supports UDFs and MaxCompute built-in functions. You can view built-in functions by type or in alphabetical order.

Usage notes: For considerations when using built-in functions, see Usage notes.
Limits: For limits on built-in functions, see Overview of built-in functions and Overview of built-in functions.
You can view built-in functions in the following three ways:
- View built-in functions by type.
- View built-in functions in alphabetical order.
- Run the following command in a MaxCompute SQL node to view built-in functions.
```
show builtin functions [<function_name>]; -- <function_name> specifies the name of a built-in function.
```
  Note
  - <function_name> is a placeholder. Replace it with the name of a built-in function.
  - If you run the show builtin functions; command by using the MaxCompute client (odpscmd), use odpscmd V0.43.0 or later.
For typical examples of built-in functions, see:
- Example of precision issues with the ROUND function.
- Implement capabilities provided by the GROUP_CONCAT function.
To quickly troubleshoot issues with built-in functions, see:
- Common error codes for built-in functions
- FAQ about built-in functions

Manage functions

On the Resource Management page, click a function to perform the relevant management operations.

View historical versions: Click the Versions icon on the right side of the function editor to view and compare saved or submitted function versions and identify changes between versions.
Note
Select at least two versions to compare.
Delete a resource: Right-click the function and choose Delete to delete it.
To delete a function from the production environment, deploy the deletion operation. The function is removed from the production environment only after the deployment succeeds. For more information, see Deploy a node.

View the UDF list

// View functions in the MaxCompute project associated with the current DataWorks compute resource.
SHOW FUNCTIONS;

View UDF details

Run the DESCRIBE command or its abbreviated form, DESC, followed by the function name to view details about a UDF.
```
// Use the abbreviated command to view details about a UDF.
DESC FUNCTION <function_name>;
```
If existing functions cannot implement the processing logic required by your workflow in DataWorks, you can develop MaxCompute UDFs and upload and associate the corresponding resources, such as JAR packages and Python files, to manage and extend your data processing capabilities.

FAQ

Q: After I upload a resource through DataWorks and register it as a UDF, can I use the UDF in SQL queries in DataAnalysis as well as in ODPS SQL nodes in Data Studio?

A: Yes. UDFs registered through DataWorks are stored in MaxCompute projects. You can use them in both MaxCompute SQL node and SQL query (legacy) in DataAnalysis.