Dataphin:Create offline computing template

Background information

When multiple tasks share similar code logic but differ only in configuration items or input parameters, you can encapsulate the common logic into an offline computing template with variable parameters. Subsequent tasks that reference this template inherit the shared logic, enabling single-point maintenance and faster development.

By default, each task uses dedicated runtime resources, which increases resource consumption and can limit concurrency when many tasks run simultaneously. Dataphin allows tasks that reference the same template to share runtime resources, improving utilization without disrupting other tasks. To enable this, activate the shared runtime resource switch on the template.

Prerequisites

Before you enable shared runtime resources for a template, make sure the global shared resource switch is turned on. For more information, see Runtime Settings.

Limits

• Only users with the superuser, project administrator, or project developer role can create, configure, and submit offline computing templates.

  For more information about how to obtain the project administrator and developer roles, see Add project members.

• The shared runtime resource switch for offline computing templates can only be enabled by Super Administrator and System Administrator.

Procedure

1. Navigate to the Dataphin home page, select Development > Data Development from the top menu bar.

2. On the Development page, select Project from the top menu bar (select the environment in Dev-Prod mode).

3. In the navigation pane on the left, click on Data Processing, then Template. From the Template list, select the icon and choose Offline Computing Template.

4. In the Create Offline Computing Template dialog box, you can configure the following parameters.

   Parameter	Description
   Template Name	The naming convention is as follows: Supports uppercase and lowercase English letters, numbers, underscores (_), and hyphens (-). Globally unique. Cannot exceed 64 characters.
   Node Type	Supports Shell, Python, and Database Sql. Select a node type based on your offline compute engine. Offline computing templates supported by different compute engines MaxCompute compute engine MaxCompute SQL Spark on MaxCompute MapReduce on MaxCompute Hadoop (Hive) compute engine Hive SQL Impala SQL Note Impala must be configured and enabled. Spark on Yarn MapReduce on Yarn TDH Inceptor compute engine Inceptor SQL Spark on Yarn MapReduce on Yarn StarRocks compute engine: StarRocks SQL ArgoDB compute engine: ArgoDB SQL GaussDB (DWS) compute engine: GaussDB SQL Databricks compute engine: Databricks SQL SelectDB compute engine: SelectDB SQL Doris compute engine: Doris SQL AnalyticDB for PostgreSQL compute engine: AnalyticDB for PostgreSQL SQL Data source types supported by Database SQL MySQL Microsoft SQL Server PostgreSQL AnalyticDB for MySQL 2.0 AnalyticDB for MySQL 3.0 AnalyticDB for PostgreSQL OceanBase Oracle ClickHouse DM openGauss GaussDB (DWS) Hologres StarRocks Doris SelectDB Presto Trino If Node Type is Shell or Python, you can configure Python Third-party Packages. After adding a third-party module, declare a reference to it in the node before importing it in code. You can configure the referenced module in the Python Module setting of the compute node properties. When Node Type is set to Database SQL, you must also select a Database/Schema. For Presto data sources, you must also configure the Catalog.
   Select Directory	Defaults to the offline computing template directory. You can also create a folder on the Template page and select it as the target directory. To create a folder: Click the icon above the compute template list to open the Create Folder dialog box. In the Create Folder dialog box, enter the folder Name, select Offline Type, and then Select Directory location as needed. Click Confirm.
   Description	Enter a brief description of the offline computing template. Maximum 1,000 characters.

5. Click Confirm.

6. Develop the offline computing template code on the code development page.

   You can define template variable parameters in the format @@{variable_parameter_name}. The parameter name must start with a letter and can contain only letters, digits, and underscores (_). For example, @@{variable}.

   

7. After you finish developing the code for the offline computing template, click the Test button in the upper-left corner of the page. In the Enter Parameters dialog box, enter values for the parameters.

8. Click Confirm.

9. On the code development page, click Attribute on the right side.

10. In the Attribute panel, set the parameters.

    Parameter		Description
    Basic Information	Description	Enter a brief description of the template.
    Python Third-Party Packages		Select the required Python third-party package. For more information, see Install and manage Python third-party packages. Note After adding a third-party module in the Python third-party package, you need to declare a reference in the task before you can import the module in code.
    Runtime Configuration	Shared Runtime Resources	When enabled, task instances that reference this template share runtime resources, reducing overall consumption. Only superusers can enable this setting. Important The Shared Runtime Resources configuration of the template requires the global shared resources switch to be enabled in the operations (metadata warehouse) tenant. Otherwise, configuration is not supported, and tasks created from this template use exclusive resources. For details, see Runtime settings.
    Parameter Settings	Parameter Description	Add descriptions for parameters defined in the code to help other developers understand their purpose.
    	Default Value	Set default values for parameters in the code. Tasks that reference this template can override these values, which take effect at runtime.
    	Parameter Encryption	When enabled, default parameter values are stored as ciphertext to protect sensitive data. Tasks that reference this template cannot view the default values in plaintext. Dataphin automatically decrypts the values at runtime. Disabling encryption automatically clears all configured default parameter values.
    Resource Configuration	Resource Group	Task scheduling consumes scheduling resources. You can specify which resource group each task instance uses. During scheduling, resources are allocated from the specified group's quota. If the group has no available resources, the instance enters the Waiting for Scheduling Resources status. Resources are isolated between groups, ensuring scheduling stability. When Shared Runtime Resources is enabled, custom resource groups cannot be specified. Tasks that reference this template use a shared scheduling resource pool by default. To modify the maximum concurrency of the shared resource pool, contact the metadata warehouse system administrator. When Shared runtime resources is disabled, you can specify a custom resource group. Only resource groups used for Daily node scheduling and associated with the project to which the current node belongs are available. For more information, see Resource group configuration. Important Tasks that reference this template can only configure the scheduling resource group at the template level. If Project Default Resource Group is selected, the resource group automatically updates when the project default changes.

11. Click Confirm to finalize the offline computing template configuration.

12. On the code development page for the offline computing template, click the Submit button at the top of the page. In the submission remarks dialog box, enter your Remarks.

13. Click Confirm And Submit.

    After editing a submitted Database SQL template:

    • The system checks whether the template code introduces a new input table. If no new table is found, submission proceeds normally.

    • If a new input table is found, the system checks whether any downstream nodes are set to automatically reference the latest version. If such nodes exist, the submission fails. Otherwise, submission proceeds normally.

    • If the SQL code cannot be parsed, you can still submit the template. However, the system displays the following prompt: The system cannot parse the code because template variables exist. Modifying the SQL template code may introduce new input tables, which can cause node instances that reference this template to lack upstream dependencies. Proceed with caution.