DataWorks:Common use cases of scheduling parameters

Before you begin

Before clicking Run or Run with Parameters in the top toolbar of the configuration tab of a batch synchronization node, manually assign constants to the variables referenced in the node code. Data Integration does not support smoke testing on batch synchronization nodes in the development environment.

To verify that scheduling parameters are replaced as expected, use one of the following methods:

• Run smoke testing on an SQL node that uses the same scheduling parameters. If the SQL node passes, the parameters are configured correctly. For details, see Configure and use scheduling parameters.

• Commit the node to Operation Center in the development environment.

Scenario overview

Scenario 1: Synchronize incremental data

Scheduling parameters in a filter condition act as dynamic time boundaries: at runtime, each expression is replaced with the actual date derived from the node's data timestamp. This ensures each run reads only the data generated within the target time window, avoiding full reloads. Make sure the filter condition syntax is supported by the source.

For more information, see Scenario: Configure an incremental offline data synchronization task.

Example 1: Sync LogHub data every 10 minutes to a T-1 MaxCompute partition

The node generates a scheduling instance every 10 minutes. $bizdate specifies the data timestamp of the data synchronization node, and the partition filter expressions resolve to the corresponding 10-minute window boundaries at runtime.

Task configuration key points: Set Data Source to LogHub, set Log Start Time to ${startTime}, and set Log End Time to ${endTime}. Set Data Destination to MaxCompute (ODPS), set Partition to pt = ${bizdate}, and set Write Mode to Insert Overwrite. In the scheduling configuration, define the following parameters:

• bizdate=$bizdate

• startTime=${yyyymmddh24miss-10/24/60}

• endTime=${yyyymmddh24miss}

Set Scheduling Cycle to Day and Scheduled Time to 00:05.

Example 2: Sync the previous day's LogHub data to a T-1 MaxCompute partition at 00:00 daily

The filter condition uses $bizdate to capture all data generated on the previous day. The time range is a left-closed, right-open interval.

In the offline batch synchronization task, complete the following configuration to associate scheduling parameters with data sources:

• Add three scheduling parameters at the top of the right-side scheduling configuration panel: startTime=${yyyymmdd}, endTime=${yyyymmdd}, bizdate=${yyyymmdd}.

• Data Source (Logstore): Set Log Start Time to ${startTime}000000, Log End Time to ${endTime}000000, and Batch Size to 256.

• Data Destination (MaxCompute): Set Partition to pt=${bizdate} and Cleaning Rule to Insert Overwrite.

• Scheduling configuration: Set Scheduling Cycle to Day, Scheduled Time to 00:00 (cron expression: 00 00 00 * * ?), and Instance Generation to T+1 (generate on the next day).

Scenario 2: Synchronize data from tables or files with dynamic names

Embed scheduling parameters directly in the source table name or file path so the node automatically reads from the correct table or file on each run.

Example 1: Sync OSS files with date-based names to MaxCompute

Configure the source object path with a scheduling parameter so the node reads from the correct dated folder each day and writes data to the corresponding partition in MaxCompute.

In the Select Data Source step of the data synchronization task, configure as follows:

• Data Source: Select the OSS data source. Set the text type to CSV, the file path to /xx/user_log_${var1}.txt, the column delimiter to comma, and the encoding to UTF-8. ${var1} is a scheduling parameter used to dynamically match the date in the file name.

• Data Destination: Select the MaxCompute (ODPS) data source. Set the target table to ods_raw_log_d, the partition to dt=${var1}, the write mode to Insert Overwrite, and Convert empty strings to null values to No.

In the Scheduling Parameters configuration panel, set the following parameters:

• Parameter name bizdate, parameter value $bizdate, source: manually added.

• Parameter name var1, parameter value ${yyyymmdd-1}, source: manually added.

Example 2: Sync MySQL tables with date-based names to MaxCompute

Configure the source table name with a scheduling parameter so the node reads from the correct dated table each day. The code editor configuration and the scheduling parameter settings are shown below.

"version": "2.0",
"steps": [
    {
        "stepType": "mysql",
        "parameter": {
            "envType": 0,
            "useSpecialSecret": false,
            "column": [
                "id"
            ],
            "tableComment": "",
            "connection": [
                {
                    "datasource": "xc_workshop_public",
                    "table": [
                        "person_${var1}"
                    ]
                }
            ],
            "where": "",
            "splitPk": "id",
            "encoding": "UTF-8"
        },
        "name": "Reader",
        "category": "reader"
    },
    {
        "stepType": "odps",
        "parameter": {
            "partition": "ds=${var1}",
            "truncate": true,
            "datasource": "odps_first",
            "isSupportThreeModel": false,
            "column": [
                "id"

In the Parameters configuration area, confirm the scheduling parameter settings: parameter bizdate has value $bizdate (manually added), and parameter var1 has value ${yyyymmdd-1} (auto-generated by code parsing). After completing the configuration, click Preview Parameters to preview the parameter substitution results.

Scenario 3: Define destination fields

When configuring field mappings, assign a constant derived from the node's data timestamp to a destination field. The system updates that field automatically on every run based on the scheduling parameter value. Support varies by data source type.

For example, the following configuration assigns the data timestamp to a ds field in the destination table, which is updated daily.

In the Field Mapping area, add a row with type Constant for the destination field, and set the value to ${bizdate}. In the right-side Scheduling panel's parameters area, add the parameter bizdate = $bizdate so that this scheduling parameter takes effect as a constant value in the field mapping.

Scenario 4: Synchronize historical data

Scheduling parameters are automatically replaced with specific values based on the data timestamps of nodes and value formats of scheduling parameters. This enables dynamic parameter configuration for node scheduling. In a data backfill run, each parameter is replaced using the data timestamp specified by the backfill operation.

Use the data backfill feature in Operation Center to generate data for a specified historical time range. For details, see Data backfill instance O&M.

Example: Backfill incremental MySQL data into time-partitioned MaxCompute tables

The following example demonstrates how to write incremental data from a MySQL source to a specified time partition in MaxCompute using data backfill.

In an offline batch synchronization task, the bizdate scheduling parameter must be configured in the following three locations:

• In the Data Filter condition of the data source, reference ${bizdate}. Example: STR_TO_DATE('${bizdate}','%Y%m%d') <= gmt_modify_time AND gmt_modify_time < DATE_ADD(STR_TO_DATE('${bizdate}','%Y%m%d'), interval 1 day)

• In the Partition field of the data destination, set the value to pt=${bizdate}, and set the cleaning rule to Insert Overwrite.

• In the Parameters area of the right-side Scheduling panel, configure bizdate=$bizdate, and set Instance Generation to T+1 (generate on the next day).