Create and use Flink resources and functions - DataWorks

Data Studio allows you to visually create and manage Flink Jar, Flink File, and Flink Python resources. You can use these resources to create custom functions or for data development. This topic describes how to use Resource Management to create different types of Flink resources and functions.

Prerequisites

You have associated a fully managed Flink compute resource with your workspace, which you will use to create resources and functions.
You have your resource files ready for upload from your local machine.

Create and use resources

Resources

On the Resource Management page in Data Studio, you can upload local resources to Flink through DataWorks. You can then use these resources to develop Flink jobs in DataWorks or to create custom functions.

Limitations

Uploaded resources must comply with the following limits:

Resource size:
- Online editing: The maximum size is 10 MB for a Python resource and 500 KB for a File resource.
- Upload from a local file: The maximum size for a single resource is 500 MB.
- Upload from an OSS file: The maximum size for a single resource is 500 MB.
Resource publishing: If you use a standard mode workspace, you must publish the resource to the production environment before you can use it.
Note
Data source information may differ between development and production environments. Before you query tables or resources, confirm the data source information for the current environment.
Resource management: In DataWorks, you can view and manage only the resources that are uploaded through the DataWorks UI.

Create a resource

You can upload Flink resources from a local file. You can reference a created resource directly for data development or register it as a function. Flink resources include the following types:

Flink Jar: A compiled Java JAR package with a .jar extension, used to run Java programs or create custom functions.
Flink File: For uploading files of any type to Flink. These can be used as additional dependency files for Flink JAR and Flink Python nodes, such as Python Libraries and Python Archives.
Flink Python: For uploading Python files with a .py extension to Flink for development in Flink Python nodes. To upload a .zip file, use the Flink File resource type. The uploaded file can also serve as the main entry point for a Python job.

On the Resource Management page, create a resource, which opens the Create Resource and Function dialog box. Configure the resource Type, storage Path, and resource Name.
After you create the resource, upload a local file to it as the file source.
In the top toolbar, Save and Publish the resource, because only deployed resources can be used in data development.

Use a resource

After you create a resource, when you edit a data development node, click Resource Management in the left navigation bar, find the target resource or function, and then right-click and select Insert Resource Path. After the resource is successfully referenced, code in the format --@resource_reference{"Resource Name"} is displayed.

Note

For example, a Flink SQL Streaming node displays the format --@resource_reference{"example.jar"}. The display format may vary for different node types. Refer to the actual interface for details.

In addition to using resources directly, you can also register a resource as a function and then use it in data development nodes.

Create and use functions

Functions

In Data Studio Resource Management, you can register resources as functions. For data development, you can use built-in functions provided by Flink and Flink custom functions.

Create a function

On the Resource Management page, create a function, which opens the Create Resource and Function dialog box. Configure the function Type, storage Path, and function Name.

Click Confirm to create a function resource, and then configure the function information based on the function type.

Before you configure a Flink function, make sure that you have created the Flink cluster as a compute resource in DataWorks and have uploaded the required Flink resource. The following table describes the parameters of a Flink function.

Parameter	Description
Function type	Select a function type: MATH (mathematical functions), AGGREGATE (aggregate functions), STRING (string processing functions), DATE (date functions), ANALYTIC (window functions), or OTHER (other functions).
Flink cluster	Select a registered Flink cluster from the drop-down list.
Resource List	When you configure a Flink function, you can only select a Flink Jar resource from the drop-down list in the visual interface.
Command Format	An example of how to use this UDF.

In the top toolbar, Save and Publish the function, because only deployed functions can be used in data development.

Use a function

After a function is created and deployed, when you edit a data development node, click Resource Management in the left navigation bar, find the target resource or function, and then right-click and select Insert Function. The custom function name is automatically generated on the current node editing page, for example: example_function().

Manage resources and functions

After you upload resources or create functions through the Data Studio visual interface, you can manage them on the resource management page by clicking the target resource or function.

View version history: Click the version button on the right side of the resource or function editor page to view and compare saved or submitted function versions and see changes between different versions.

Note
For version comparison, you must select at least two versions.
Delete a resource or function: Right-click the target resource or function and click the Delete button to delete it.

To delete the resource or function in the production environment, you must deploy the task and deploy the deletion to the production environment. After the deployment succeeds, the resource or function is synchronously deleted from the production environment.