Create and use Flink resources and functions - DataWorks

Prerequisites

You have associated a fully managed Flink compute resource with your workspace. Resources and functions are created based on this compute resource.
You have your resource files ready for upload from your local machine.

Create and use resources

Resources

In Data Studio, you can upload local resources to Flink through DataWorks. These resources can be used to develop Flink jobs or to create custom functions.

Limitations

Uploaded resources must comply with the following limits:

Resource size:
- Online editing: The maximum size is 10MB for Python resources and 500KB for File resources.
- Local file upload: The maximum resource size is 500MB.
- OSS file upload: The maximum resource size is 500MB.
Resource deployment: If you use a workspace in standard mode, you must deploy resources to the production environment for them to take effect.

Note
Data source information may differ between development and production environments. Before you query tables or resources, confirm the data source information for the current environment.
Resource management: DataWorks allows you to view and manage only resources uploaded through DataWorks.

Create a resource

Upload Flink resources from a local file. A created resource can be referenced directly in data development or registered as a function. Flink resources include the following types:

Flink Jar: A compiled Java JAR package with a .jar extension, used to run Java programs or create custom functions.
Flink File: Uploads files of any type to Flink. These can serve as additional dependency files for Flink JAR and Flink Python nodes, such as Python Libraries and Python Archives.
Flink Python: Uploads Python files with a .py extension to Flink for development in Flink Python nodes. To upload a .zip file, use the Flink File resource type. The uploaded file can also serve as the main entry point for a Python job.

On the Resource Management page, create a resource. In the Create Resource and Function dialog box, specify the resource Type, storage Path, and resource Name.
After you create the resource, upload a local file to it as the file source.
In the top toolbar, Save and Publish the resource, because only deployed resources can be used in data development.

Use a resource

After you create a resource, open a data development node for editing. Click Resource Management in the left navigation bar, find the target resource or function, right-click it, and select Insert Resource Path. After the resource is referenced, code in the format --@resource_reference{"Resource Name"} is displayed.

Note

For example, a Flink SQL Streaming node displays the format --@resource_reference{"example.jar"}. The display format may vary by node type. Refer to the actual interface for details.

You can also register a resource as a function and use it in data development nodes.

Create and use functions

Functions

In Data Studio Resource Management, you can register resources as functions. For data development, you can use built-in Flink functions and Flink custom functions.

Create a function

On the Resource Management page, create a function. In the Create Resource and Function dialog box, specify the function Type, storage Path, and function Name.

Click Confirm to create a function resource, and then configure the function information based on the function type.

Before you configure a Flink function, make sure that you have created the Flink cluster as a compute resource in DataWorks and have uploaded the required Flink resource. The following table describes the parameters of a Flink function.

Parameter	Description
Function type	Select a function type: MATH (mathematical functions), AGGREGATE (aggregate functions), STRING (string processing functions), DATE (date functions), ANALYTIC (window functions), or OTHER (other functions).
Flink cluster	Select a registered Flink cluster from the drop-down list.
Resource List	Only Flink Jar resources can be selected from the drop-down list in the visual interface.
Command Format	An example of how to use the UDF.

In the top toolbar, Save and Publish the function, because only deployed functions can be used in data development.

Use a function

After a function is created and deployed, open a data development node for editing. Click Resource Management in the left navigation bar, find the target resource or function, right-click it, and select Insert Function. The custom function name is automatically inserted on the current node editing page, for example: example_function().

Manage resources and functions

After you upload resources or create functions through the Data Studio visual interface, you can manage them on the resource management page by clicking the target resource or function.

View version history: Click the version button on the right side of the resource or function editor page to view and compare saved or submitted function versions and see changes between different versions.

Note
For version comparison, you must select at least two versions.
Delete a resource or function: Right-click the target resource or function and click the Delete button to delete it.

To delete the resource or function in the production environment, you must deploy the task and deploy the deletion to the production environment. After the deployment succeeds, the resource or function is synchronously deleted from the production environment.