Dataphin allows you to create offline computing templates. You can then reference these templates to develop tasks more efficiently. This topic describes how to create, configure, and submit an offline computing template.
Background information
If multiple tasks have similar code logic and differ only in a few configuration items or input parameters, you can encapsulate the code in an offline computing template and define the differing items as template variable parameters. When a task references this template, you can assign values to these template variable parameters. This approach allows for single-point maintenance and quick reuse of common code logic, significantly improving the efficiency of task development.
Typically, each task uses dedicated running resources. When a large number of tasks run simultaneously, high resource utilization can affect concurrency. Dataphin supports sharing running resources among multiple tasks that reference the same offline computing template to ensure other tasks can run normally. You can enable shared running resources for an offline computing template. This allows multiple tasks that reference the template to run on the same shared resources, freeing up Dataphin resources for other tasks.
Prerequisites
If you plan to enable shared running resources for an offline computing template, ensure the global shared resource feature is enabled first. For information about how to enable global shared resources, see Runtime settings.
Limitations
Only super administrators, project administrators, and project developers can create, configure, and submit offline computing templates.
For information about how to obtain the project administrator and developer roles, see Add project members.
Only super administrators and system administrators can enable shared running resources for an offline computing template.
Procedure
On the Dataphin homepage, in the top navigation bar, select Development > Data Development.
On the Development page, select a project. In Dev-Prod mode, you must also select an environment.
In the left-side navigation pane, choose Data Processing > Computing Templates. In the Computing Templates list, click the
icon and select Offline Computing Template.In the Create Offline Computing Template dialog box, configure the following parameters.
Parameter
Description
Template Name
The name must follow these rules:
Can contain letters (case-sensitive), digits, underscores (_), and hyphens (-).
Must be globally unique.
Cannot exceed 64 characters.
Node type
Supported node types include Shell, Python, and database SQL. Different compute engines support different template types.
If you select Shell or Python as the node type, you can optionally configure Python third-party packages.
After you add a third-party module to a Python third-party package, you must reference it in the task before you can import the module in your code. You can manage the referenced modules in the task's properties under the Python third-party packages section.
If you select database SQL as the node type, you must also select a Database/Schema. If the data source type is Presto, you must also configure a Catalog.
Directory
The default directory is Offline Computing Template. You can also create a target folder on the Computing Templates page and select it as the template's directory. To create a new folder:
Above the computing template list, click the
icon to open the Create Folder dialog box.In the Create Folder dialog box, enter a folder Name, select the Offline Type, and then select a Directory as needed.
Click OK.
Description
Enter a brief description of the offline computing template, up to 1,000 characters.
Click OK to create the offline computing template.
In the template's code editor, write the template code.
Define template variable parameters in the format
@@{parameter_name}. The parameter name can contain only letters, digits, and underscores (_) and must start with a letter. For example,@@{variable}.
After you finish editing the code, click Debug in the upper-left corner of the page. In the Fill in Parameters dialog box, enter the runtime values and click OK.
If the parameter type is a template parameter, the parameter value and runtime value display the default value. If no default value exists, a hyphen (-) is displayed. If parameter encryption is enabled for the property, the value is masked. If the parameter type is a local variable, the parameter value and runtime value are populated based on the task parameter configuration in the properties. You can modify the runtime value, click Restore Default Values, or click Set to Null.
On the code development page for the offline computing template, click Properties on the right.
In the Properties panel, configure the parameters.
Parameter
Description
Basic Information
Description
Enter a brief description of the offline computing template.
Python third-party packages
Select the Python third-party packages that you want to import. For more information, see Install and manage Python third-party packages.
NoteAfter you add a third-party module, you must declare a reference to it in the task before you can import the module in your code.
Runtime configuration
Shared running resources
After you enable shared running resources, task instances that reference this template can share resources to reduce resource consumption. Only a super administrator can perform this action.
ImportantTo configure shared running resources for a template, you must first enable the global shared resource feature in the Operations (Metastore) tenant. Otherwise, this configuration is not supported, and any computing task created from the template will use dedicated resources. For more information, see Runtime settings.
Template parameters
Description
Enter a description for the parameters in the code to help other developers understand them.
Default Value
The default value for a parameter. You can override this value in any task that references the template. The new value takes effect when the task runs.
Parameter encryption
If you enable parameter encryption, the default value is stored as ciphertext to protect sensitive data. The plaintext value is not visible in tasks that reference this template. Dataphin automatically decrypts the default value when the task runs.
If you disable parameter encryption, any configured default value is automatically cleared.
Task parameters
Parameter Name
The name of the parameter defined in the code editor. The name can contain only digits, letters, and underscores (_) and cannot exceed 32 characters.
Parameter Type
Types include local variable and non-variable. For more information, see Variable categories.
You can select multiple parameters or click Select All, and then click Modify Parameter Type at the bottom of the parameter list to modify the parameter types in bulk.
Parameter Value
Local variable: The value can be up to 258 characters or null. If you need to enter a date and time, you can select an expression from the list. For more information, see Variable assignment.
Non-variable: A value cannot be defined.
Resource configuration
Scheduling resource group
Specify a scheduling resource group for instances generated from this template. When an instance runs, it consumes resources from the specified group's quota. If the group has insufficient resources, the instance enters the waiting for scheduling resources state. Resource isolation between different groups ensures scheduling stability.
If you enable shared running resources, you cannot specify a custom resource group. Tasks created from the template use a default shared scheduling resource pool. To modify the maximum concurrency limit for this shared pool, contact a metastore system administrator.
If you disable shared running resources, you can specify a custom resource group. You can only select a scheduling resource group that is configured for the task daily scheduling use case and is associated with the current project. For more information, see Resource group configuration.
ImportantThe scheduling resource group for tasks created from a template can be configured only in the template.
If you select the default project resource group, the configuration is automatically updated if the project's default resource group changes.
Click OK to save the configuration for the offline computing template.
On the code development page for the offline computing template, click Submit at the top of the page. In the dialog box that appears, enter a comment.
Click OK and Submit.
When you submit an edited database SQL computing template:
The system automatically checks whether the template code introduces new input tables. If no new input tables are introduced, the template is submitted successfully.
If the template code introduces new input tables, the system then checks for downstream tasks that have Automatically Reference the Latest Version enabled. If such tasks exist, the submission fails. Otherwise, the submission succeeds.
If the SQL code fails to parse, you can still submit the template. However, the system displays a prompt: The system cannot parse the code because it contains template variables. Modifying the SQL template code may introduce new input tables, which can cause task instances that reference this template to lack upstream dependencies. Proceed with caution.
Next steps
Create tasks based on the offline computing template you created. For more information, see the following topics: