Variables help you reduce the workload of repeatedly writing the same values and improve the efficiency of configuration management. You can reference variables in various scenarios, such as the development of SQL jobs, batch jobs, and workflows, to simplify code maintenance and adjustment, and improve development efficiency. This topic describes how to add and use variables in different scenarios.
Background information
A variable consists of a name and a value. The variable name reflects the meaning of the value and can be customized. The variable value is the actual data that corresponds to the name. You can reference variables in an object in the ${Variable name} format. When the object runs, the system automatically replaces the variable with the actual value. You can use variables based on your business requirements in the following scenarios:
Data development: You can use variables when you develop an SQL or a batch job. For more information, see Data development variables.
Workflow development: You can use workflow-level and node-level variables when you develop a workflow. For more information, see Workflow-level and node-level variables.
Data development variables
Add a variable
Add a variable in a Spark SQL job
For information about how to develop a Spark SQL job, see Get started with the development of Spark SQL jobs.
On the configuration tab of a Spark SQL job, copy the following code to the code editor:
CREATE TABLE IF NOT EXISTS students_info ( name VARCHAR(64), address VARCHAR(64) ) USING PARQUET PARTITIONED BY (data_date STRING); INSERT OVERWRITE TABLE students_info PARTITION (data_date = '${my_date}') VALUES ('Ashua Hill', '456 Erica Ct, Cupertino'), ('Brian Reed', '723 Kern Ave, Palo Alto');In the right-side navigation pane, click Variables Management. In the Variables Management panel, enter a value in the Variable Value column. In this example, the variable value is 2025-04-15.
In the code editor of the job, enter and execute the following statement to query table data:
SELECT * FROM students_info;On the Execution Results tab in the lower part, check whether the variable is correctly parsed.
Add a variable in a batch job
For information about how to develop a batch or streaming job, see Develop a batch or streaming job.
On the configuration tab of a batch job, enter a variable name for Execution Parameter. In this example, enter
${osspath}.In the right-side navigation pane, click Variables Management. In the Variables Management panel, enter a value in the Variable Value column.
Use a built-in variable
For information about how to create a Spark SQL job, see Get started with the development of Spark SQL jobs.
Copy the following code to the code editor of a Spark SQL job and run the code:
CREATE TABLE IF NOT EXISTS students_address (
name VARCHAR(64),
address VARCHAR(64)
)
USING PARQUET
PARTITIONED BY (data_date STRING);
INSERT OVERWRITE TABLE students_address PARTITION (data_date = '${ts}') VALUES
('Ashua Hill', '456 Erica Ct, Cupertino'),
('Brian Reed', '723 Kern Ave, Palo Alto');
SELECT * FROM students_address;The following figure shows the output.
When you use a built-in variable, the default value is the day before current day based on the time zone UTC+8 and cannot be modified. The following table describes supported built-in date and time variables.
Variable | Data type | Description |
{data_date} | str | A variable that indicates the date. Format: Example: 2023-09-18. |
{ds} | str | |
{dt} | str | |
{data_date_nodash} | str | A variable that indicates the date. Format: Example: 20230918. |
{ds_nodash} | str | |
{dt_nodash} | str | |
{ts} | str | A variable that indicates the timestamp. Format: Example: 2023-09-18T16:07:43. |
{ts_nodash} | str | A variable that indicates the timestamp. Format: Example: 20230918160743. |
View or modify a variable value
In the right-side navigation pane of the configuration tab of a Spark SQL job, click Variables Management.
In the Variables Management panel, view the variables referenced in the Spark job.
View or modify the value of a custom variable in the Variable Value column.
Workflow variables
You can use workflow-level and node-level variables when you develop a workflow. If you configure the same variable for a workflow and a node in the workflow, the node-level variable overwrites the workflow-level variable.
You cannot modify the names of built-in variables.
Add a workflow-level variable
In the Create Workflow panel, click the right arrow to the right of Variable Settings. In the Variable Settings section, click Add to add a variable. When the workflow is scheduled, the variable is automatically replaced with the specified value.
You can select a variable value from the Variable Value drop-down list and modify the value based on your business requirements.
Add a node-level variable
In the Add Node panel, configure the Source File Path parameter. Then, click Add to the right of Node-level Variables.
You can select a variable value from the Variable Value drop-down list and modify the value based on your business requirements.
