Variables help you avoid hardcoding repeated values across SQL jobs, batch jobs, and workflows. Instead of updating the same date string in multiple places, you define it once as a variable and reference it with ${variable_name}. This topic explains how to add and use variables in each supported scenario.
Variable types and scope
A variable consists of a name and a value. Reference a variable using the ${Variable name} format. When a job or workflow runs, the system replaces each variable reference with the configured value.
EMR Serverless Spark supports two variable types:
| Type | Description | Can modify name | Can modify value |
|---|---|---|---|
| Custom variables | Defined by you. Set any name and value to match your business logic. | Yes | Yes |
| Built-in variables | Provided by the system. All are date and time variables. Default value is the day before the current day (UTC+8). | No | No |
Variables are scoped to the context where they are defined:
Job-level variables: Defined and used within a single Spark SQL or batch job.
Workflow-level variables: Applied to all nodes in a workflow.
Node-level variables: Applied to a single node within a workflow.
Priority rule: When the same variable name is configured at both the workflow level and a node level, the node-level value takes precedence.
Data development variables
Add a variable in a Spark SQL job
For information about how to develop a Spark SQL job, see Get started with the development of Spark SQL jobs.
On the configuration tab of a Spark SQL job, copy the following code to the code editor:
CREATE TABLE IF NOT EXISTS students_info ( name VARCHAR(64), address VARCHAR(64) ) USING PARQUET PARTITIONED BY (data_date STRING); INSERT OVERWRITE TABLE students_info PARTITION (data_date = '${my_date}') VALUES ('Ashua Hill', '456 Erica Ct, Cupertino'), ('Brian Reed', '723 Kern Ave, Palo Alto');In the right-side navigation pane, click Variables Management. In the Variables Management panel, enter a value in the Variable Value column. In this example, the variable value is
2025-04-15.
In the code editor, enter and run the following statement to query table data:
SELECT * FROM students_info;On the Execution Results tab in the lower part, verify that the variable is correctly parsed.
Add a variable in a batch job
For information about how to develop a batch or streaming job, see Develop a batch or streaming job.
On the configuration tab of a batch job, enter a variable name in the Execution Parameter field. In this example, enter
${osspath}.In the right-side navigation pane, click Variables Management. In the Variables Management panel, enter a value in the Variable Value column.
Use a built-in variable
Built-in variables are system-provided date and time variables. Their default value is the day before the current day (UTC+8). You cannot modify built-in variable names or values.
The following table lists all supported built-in variables.
| Variable | Data type | Format | Example |
|---|---|---|---|
{data_date} | str | YYYY-MM-DD | 2023-09-18 |
{ds} | str | — | — |
{dt} | str | — | — |
{data_date_nodash} | str | YYYYMMDD | 20230918 |
{ds_nodash} | str | — | — |
{dt_nodash} | str | — | — |
{ts} | str | YYYY-MM-DDTHH:MM:SS | 2023-09-18T16:07:43 |
{ts_nodash} | str | YYYYMMDDHHMMSS | 20230918160743 |
To use a built-in variable, copy the following code to the code editor of a Spark SQL job and run it:
CREATE TABLE IF NOT EXISTS students_address (
name VARCHAR(64),
address VARCHAR(64)
)
USING PARQUET
PARTITIONED BY (data_date STRING);
INSERT OVERWRITE TABLE students_address PARTITION (data_date = '${ts}') VALUES
('Ashua Hill', '456 Erica Ct, Cupertino'),
('Brian Reed', '723 Kern Ave, Palo Alto');
SELECT * FROM students_address;The following figure shows the output.
View or modify a variable value
In the right-side navigation pane of the configuration tab, click Variables Management.
In the Variables Management panel, view the variables referenced in the job.
In the Variable Value column, view or modify the value of a custom variable.
Workflow variables
Workflows support two variable scopes: workflow-level and node-level. When the same variable is configured for both a workflow and one of its nodes, the node-level variable takes precedence.
You cannot modify the names of built-in variables.
Add a workflow-level variable
In the Create Workflow panel, click the right arrow next to Variable Settings. In the Variable Settings section, click Add to add a variable. When the workflow is scheduled, the variable is automatically replaced with the specified value.
Select a variable value from the Variable Value drop-down list and modify the value based on your requirements.
Add a node-level variable
In the Add Node panel, configure the Source File Path parameter. Then, click Add next to Node-level Variables.
Select a variable value from the Variable Value drop-down list and modify the value based on your requirements.
