Assign values to variables used in offline pipelines so that parameter variables are automatically replaced with the corresponding values during node scheduling.
Procedure
-
In the top navigation bar of the Dataphin homepage, choose Development > Data Integration.
-
In the top navigation bar, select a project. For Dev-Prod projects, also select an environment.
-
In the left-side navigation pane, choose Integration > Batch Pipeline. In the Batch Pipeline list, click the target task name.
-
On the task tab, click Property on the right to open the Property panel.
-
Click the Scan Configuration tab and configure the parameters in the Runtime Parameter section.
Variables defined in the offline pipeline using the expression
${variable name}appear in the Runtime Parameter section. You can modify the parameter type and assign values in the parameter list.Parameter
Description
Parameter Name
The variable name defined in the offline pipeline. Variable names should contain only digits, letters, and underscores (_) and not exceed 32 characters.
Parameter Type
Supported types: Global Variable, Namespace Variable, Local Variable, Cross-Node Variable, and Fixed Value. Variable classification.
To batch modify parameter types, select multiple parameters or click Select All, and then click Modify Parameter Type at the bottom of the list.
Parameter Value
-
Global Variable: Uses the value defined by the global variable. This value cannot be modified. Create a global variable.
-
Space Name Variable: The value is the space name.
-
Local Variable: Supports up to 258 characters or an empty value. For date and time input, select or refer to expressions in the list. Variable assignment.
-
Cross-Node Variable: After configuring upstream dependencies, select cross-node output parameters from upstream as values.
-
Non-Variable: Cannot be defined.
NoteIf you reference a global variable with permission control enabled, the system verifies whether you have permission. Unauthorized global variables are marked with an
icon. Click Request Permission to go to Permission Management > Variable Permission Management and request access. -
Variable classification
Dataphin supports the following variable types:
|
Variable type |
Description |
|
Global Variable |
Predefined in Planning > Common Business Logic > Global Variables. Global variables can be referenced in any node or code within the tenant. The system includes built-in global variables such as Note
|
|
Space Name Variable |
Refers to Note
Space name variables cannot be converted to other variable types. |
|
Local Variable |
Important
|
|
Cross-Node Variable |
Variables passed from upstream nodes to the current node. Use cases:
Important
When using cross-node parameter variables, backfill both the upstream node that outputs the cross-node parameters and the current node simultaneously. |
Variable assignment
Date and time assignment
When assigning date and time values to local variables, use the following formats.
-
${yyyymmdd - 1}: Previous business date in yyyyMMdd format. -
${yyyy-mm-dd - 1}: Previous business date in yyyy-MM-dd format. -
$[yyyymmdd - 1]: Previous scheduling date. -
${yyyy - 1}: Previous business year. -
${yyyymm - 1}: Previous business month. -
$[yyyymmdd]: Current scheduling date. -
$[HH - 1/24]: Previous scheduling hour. -
$[HH:mm:ss - 1/24/60]: Previous scheduling minute. -
$[yyyymmdd - 1/24]: Previous scheduling hour, taking the date part.
After configuring parameter values, click Preview and select a Business Date in the preview dialog box to view the resolved values. The business date defaults to the current date. The following section describes the date and time calculation logic with examples.
Date and time calculation expressions
Dataphin supports the following date calculation operators for local variable assignment:
-
Two date-time base formats
-
Business date
${bizdate}:${(yyyyMMdd|yyyy/MM/dd|yyyy-MM-dd) (+|-) n}. -
Expected execution time (scheduled start time):
${(yyyyMMddHHmmss|yyyy/MM/dd HH:mm:ss|yyyy-MM-dd HH:mm:ss) (+|-) n}.
-
-
Date-time part identifiers
-
Year: yyyy.
-
Month: mm (day-granularity only) or MM (required when minute-granularity is included, to distinguish from minute mm).
-
Day: dd.
-
Hour: HH.
-
Minute: mm.
-
Second: ss.
-
-
Calculation unit is day
-
1 hour = 1/24, 5 hours = 5/24.
-
1 minute = 1/24/60, 30 minutes = 30/24/60.
-
Example references
|
Current date |
Configuration based on business date (T-1) |
Actual value |
Configuration based on expected execution time (T) |
Actual value |
|
2018-01-10 |
${yyyymmdd} |
20180109 |
$[yyyymmdd] |
20180110 |
|
2018-01-10 |
${yyyyMMdd} |
20180109 |
$[yyyyMMdd] |
20180110 |
|
2018-01-10 |
${yyyymmdd -1} |
20180108 |
$[yyyymmdd -1] |
20180109 |
|
2018-01-10 |
${yyyymmdd -7} |
20180102 |
$[yyyymmdd -7] |
20180103 |
|
2018-01-10 |
${yyyy-mm-dd} |
2018-01-09 |
$[yyyy-mm-dd] |
2018-01-10 |
|
2018-01-10 |
${yyyy-mm} |
2018-01 |
$[yyyy-mm] |
2018-01 |
|
2018-01-10 |
${yyyy-mm -1} |
2017-12 |
$[yyyy-mm -1] |
2017-12 |
|
2018-01-10 |
${yyyy} |
2018 |
$[yyyy] |
2018 |
|
2018-01-10 |
${yyyy+1} |
2019 |
$[yyyy + 1] |
2019 |
|
2018-01-10 |
${yyyy/MM/dd HH:mm:ss} |
2018/01/09 00:00:00 |
$[yyyy/MM/dd HH:mm:ss] |
2018/01/10 00:00:00 |
|
2018-01-10 |
${yyyy/MM/dd HH:mm:ss-1/24} |
2018/01/08 23:00:00 |
$[HH:mm:ss-5/24/60] |
23:55:00 |