DataWorks:CDH resources and functions

Resource types

DataWorks supports uploading the following resource types to a CDH cluster. Use them to develop Flink jobs or create custom functions.

Limits

• Size: Maximum resource size is 500 MB.

• Publishing: In a workspace in standard mode, publish the resource to make it available in the production environment. Data source configurations may differ between development and production environments — confirm the correct data source before operating in each environment.

• Management scope: DataWorks only supports viewing and managing resources that were uploaded through DataWorks.

Create a resource

1. On the Resource Management page, click the create button to open the Create Resource And Function dialog box. Configure the Type, Path, and Name.

2. Upload a local file as the resource source. Configure the following parameters:

   Parameter	Description
   Storage Path	The HDFS path where the file is stored on the CDH cluster. Default: /user/admin/lib. If Kerberos authentication is enabled, grant the current user write permission to this directory before uploading.
   Data Source	Select the CDH data source to upload to.
   Resource Group	Select a Serverless resource group with connectivity to the CDH cluster.

3. Click Save and then Publish in the toolbar. Only published resources are available for use in data development.

Use a resource

After creating a resource, reference it in a data development node:

1. In the node editor, click Resource Management in the left-side navigation pane.

2. Find the target resource, right-click it, and select Reference Resource.

After referencing, the resource appears as a code snippet in the following format:

##@resource_reference{"resource name"}

For example, in a CDH Hive node:

##@resource_reference{"example"}

Note: The display format varies by node type. Check the DataWorks console to confirm the format for your node type.

Alternatively, register resources as functions and use them in development nodes or SQL queries. See Create and use functions.

When to use custom functions

Use a custom function (User-Defined Function, or UDF) when Apache Hive built-in functions cannot express the logic you need — for example, custom data encryption, hashing, JSON parsing, or domain-specific string transformations. For standard aggregations, filters, and string operations, use built-in Hive functions instead.

Function types

CDH functions support the following types. Select the type that matches the return behavior of your UDF:

Create a function

Before creating a function, make sure:

• The CDH engine is registered as a computing resource in DataWorks.

• The CDH Jar or CDH File resource to be used is already uploaded.

To create a function:

1. On the Resource Management page, click the create button to open the Create Resource And Function dialog box. Select the function Type, configure the Path, and enter the function Name.

2. Click OK. Configure the function parameters:

   Parameter	Description
   Function Type	Select the type that matches your UDF: MATH, AGGREGATE, STRING, DATE, ANALYTIC, or OTHER.
   Data Source	Select the CDH data source from the dropdown.
   Class Name	The class name of the UDF, in the format resource_name.class_name. See Determine the class name below.
   Resource List	Select the CDH Jar or CDH File resource to use. CDH functions only support visual mode.
   Command Format	An example showing how to call the UDF.

3. Click Save and then Publish in the toolbar. Only published functions are available in data development.

Use a function

After publishing a function, reference it in a data development node or SQL query:

• In a development node: In the node editor, click Resource Management in the left-side navigation pane. Find the function, right-click it, and select Reference Function. The function name is inserted into the editor automatically, for example: example_function().

• In a SQL query: Call the function directly by name:

  SELECT example_function(column_name) FROM table;