Configure offline pipeline operation parameters - Dataphin

Operation parameter configuration assigns values to variables in offline pipelines so that parameter variables can be automatically replaced with corresponding variable values during node scheduling. This topic describes how to configure operation parameters for offline pipelines.

Procedure

In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.
In the top navigation bar, select a project (for Dev-Prod projects, you also need to select an environment).
In the left-side navigation pane, choose Integration > Batch Pipeline. In the Batch Pipeline list, click the name of the target task.
On the task tab, click Property on the right to open the Property panel.

Click the Scan Configuration tab, and set the relevant parameters in the Runtime Parameter section.

Parameters configured in the offline integration pipeline through the expression ${variable name} will be displayed in the Runtime Parameter section. You can modify the parameter type and assign values to parameters in the runtime parameter list.

Parameter	Description
Parameter Name	The parameter name defined in the offline pipeline. We recommend that variable names contain only digits, letters, and underscores (_) and do not exceed 32 characters in length.
Parameter Type	Including Global Variable, Namespace Variable, Local Variable, Cross-Node Variable, and Fixed Value types. For more information, see Variable classification. After selecting multiple parameters or clicking Select All, click Modify Parameter Type at the bottom of the parameter list to batch modify parameter types.
Parameter Value	Global Variable: Defaults to the variable value defined by the global variable and cannot be modified. For more information, see Create a global variable. Space Name Variable: The parameter value is the space name. Local Variable: Supports up to 258 characters or can be set to an empty value. To input date and time, you can select or refer to expressions in the list. For details, see Variable assignment. Cross-Node Variable: After setting upstream dependencies, you can select cross-node output parameters from upstream as parameter values. Non-Variable: Not supported for definition.

Note

If you reference a global variable with permission control enabled, the system will automatically verify whether the current user has permission. Global variables without permission will be marked with an icon. You can click Request Permission to go to the Permission Management > Variable Permission Management page to request permission.

Variable classification

The following types of variables exist in Dataphin:

Variable type	Description
Global Variable	Variables that are predefined in Dataphin Planning > Common Business Logic > Global Variables and can be referenced in any node or code within the tenant. The system initially has built-in global variables such as `${bizdate}`. Note Global variables do not currently support real-time computing and integration sync tasks. For specific operations on setting global variables, see Create a global variable.
Space Name Variable	Refers to `${Production/Development Project Name}` and `${Production/Development Business Unit Name}`. Note Space name variables cannot be converted to other types of variables.
Local Variable	No need to declare or define in advance. You can directly use `${variable_name}` in the offline pipeline to define a local variable named `variable_name`. It is recommended that variable names only use digits, letters, and underscores (_), and do not start with glb, which is reserved for global variables. The length should not exceed 32 characters. Important If `variable_name` is identical to a global variable name, the system will prioritize it as a reference to the global variable. You can convert its type to local variable in the parameter configuration. If `variable_name` is identical to a production project or production business unit name, the system will forcibly treat it as a space name variable. If `${variable_name}` is part of the code content rather than a variable, you can set this variable as Non-Variable in the Parameter Type configuration. If `${variable_name}` is a cross-node variable, you can set this variable as Cross-Node Variable in the Parameter Type configuration. To set the parameter value of a local variable to an empty value, you can click Set To Empty Value in the Parameter Value field.
Cross-Node Variable	Refers to variables that are passed from upstream nodes to the current object node. It can be used in the following scenarios: Upstream node A passes a variable `${var}` to downstream node B, and `${var}` is referenced by B's computation logic. Upstream node A passes a variable `${var}` to downstream node B, and B determines its running strategy based on the value of `${var}`. Important After introducing cross-node parameter variables, when performing data backfill for the current node, it is recommended to backfill both the upstream node that outputs the cross-node parameters and the current node simultaneously.

Variable assignment

Date and time assignment

When assigning values to local variables, if you need to input date and time, you can select or refer to the following formats.

${yyyymmdd - 1}: Previous business date in yyyyMMdd format.
${yyyy-mm-dd - 1}: Previous business date in yyyy-MM-dd format.
$[yyyymmdd - 1]: Previous scheduling date.
${yyyy - 1}: Previous business year.
${yyyymm - 1}: Previous business month.
$[yyyymmdd]: Current scheduling date.
$[HH - 1/24]: Previous scheduling hour.
$[HH:mm:ss - 1/24/60]: Previous scheduling minute.
$[yyyymmdd - 1/24]: Previous scheduling hour, taking the date part.

After configuring the parameter values, you can click Preview to view the parameter values after selecting the Business Date in the parameter value preview dialog box. The business date defaults to the current business date. See below for date and time calculation logic and examples.

Date and time calculation expressions

When assigning values to local variables, if you need to perform calculations on date and time, Dataphin supports the following shortcut date calculation operators:

Dataphin provides calculations based on two date-time formats
- Based on the business date ${bizdate}: ${(yyyyMMdd|yyyy/MM/dd|yyyy-MM-dd) (+|-) n}.
- Based on the expected execution time (the scheduled start time of the node task): ${(yyyyMMddHHmmss|yyyy/MM/dd HH:mm:ss|yyyy-MM-dd HH:mm:ss) (+|-) n}.
Identifiers for each part of date and time
- Year: yyyy.
- Month: mm (can be used when date and time is only to day granularity) or MM (must use uppercase when date and time is to minute granularity, to distinguish from minute mm).
- Day: dd.
- Hour: HH.
- Minute: mm.
- Second: ss.
The unit (interval) for date and time calculations is day
- 1 hour is represented as 1/24, 5 hours as 5/24.
- 1 minute is represented as 1/24/60, 30 minutes as 30/24/60.

Example references

Current date	Configuration based on business date (T-1)	Actual value	Configuration based on expected execution time (T)	Actual value
2018-01-10	${yyyymmdd}	20180109	$[yyyymmdd]	20180110
2018-01-10	${yyyyMMdd}	20180109	$[yyyyMMdd]	20180110
2018-01-10	${yyyymmdd -1}	20180108	$[yyyymmdd -1]	20180109
2018-01-10	${yyyymmdd -7}	20180102	$[yyyymmdd -7]	20180103
2018-01-10	${yyyy-mm-dd}	2018-01-09	$[yyyy-mm-dd]	2018-01-10
2018-01-10	${yyyy-mm}	2018-01	$[yyyy-mm]	2018-01
2018-01-10	${yyyy-mm -1}	2017-12	$[yyyy-mm -1]	2017-12
2018-01-10	${yyyy}	2018	$[yyyy]	2018
2018-01-10	${yyyy+1}	2019	$[yyyy + 1]	2019
2018-01-10	${yyyy/MM/dd HH:mm:ss}	2018/01/09 00:00:00	$[yyyy/MM/dd HH:mm:ss]	2018/01/10 00:00:00
2018-01-10	${yyyy/MM/dd HH:mm:ss-1/24}	2018/01/08 23:00:00	$[HH:mm:ss-5/24/60]	23:55:00