Backfill data for recurring tasks - Dataphin - Alibaba Cloud Documentation Center

Data backfill refreshes historical data for recurring tasks within specified time ranges. After a recurring task is published, it runs on its configured schedule. Use data backfill to run a task for a specific period or refresh historical data. Scheduling parameters are automatically replaced with values matching the selected business time.

Scenarios

Common scenarios:

New recurring tasks start scheduling from the next day. To view historical partition data immediately, run a data backfill.
Upstream tasks are rerun or backfilled, and downstream historical partition data must be refreshed accordingly.
Historical business data is incomplete and must be refreshed periodically.

Data backfill modes

Operation Center supports two data backfill modes:

Backfill Current Task: Backfills only the current task. Suitable scenarios:
- Refresh the current node without updating descendant nodes.
- Verify updated computation logic on the current task before refreshing downstream tasks.
Backfill Current And Upstream/Downstream Tasks: Refreshes the current task along with its upstream and downstream tasks. Use this when the entire data lineage must be refreshed.

Data backfill operation entry

In the top menu bar of the Dataphin homepage, choose Development > Task O&M.
In the navigation pane on the left, choose Task O&M > Recurring Task.
In the top navigation bar, select the production or development environment.
On the Integration and Computing Tasks or Modeling Tasks tab, click the icon in the Actions column of the target task, select Data Backfill, and then select Backfill Current Task or Backfill Current and Downstream Tasks.

Note
You can also backfill data from the DAG graph of recurring tasks.

Backfill data for the current task

In the Data Backfill - Backfill Current Task dialog box, configure the data backfill task.

Step 1: Data backfill configuration

Parameter	Description
Basic information
Data Backfill Instance Name	Auto-generated in the format Node Name_Run Date_Instantiation Number. You can modify it manually.
Data Backfill Runtime	When the backfill instance starts scheduling. Select Run Now or Custom. Run Immediately: The backfill instance runs as soon as configuration is complete. Custom: Specify when the backfill instance starts scheduling. If the system time zone (User Center) differs from the scheduling time zone, both are displayed. The system converts the selected run time to the scheduling time zone automatically. Note The custom runtime must be later than the current time. After you configure a custom runtime, you can set Data Timestamp to a date up to the custom date. Scheduled backfill tasks generate pending instances at 23:00 on the day before the run time.
Data Timestamp	Select the data timestamp range for backfill, calculated in the configured scheduling time zone. Options vary by scheduling cycle: For tasks with daily, weekly, or monthly scheduling cycles, you can select By Range, By Cycle, and Custom data timestamps. The application scenarios for each option are as follows: By Range: Refresh data for consecutive timestamps. Select a start and end time (max one year). Note For a single day, select the same start and end date. By Cycle: Refresh data for specific weekdays or monthly dates within a time range (max one year). Weekly: Refreshes the selected days of the week within the continuous time period. Monthly: Refreshes the selected dates of each month within the continuous time period. Note Month-end refers to the last day of each month. Custom: Enter non-consecutive timestamps manually (from `1900-01-01` to today, format: `YYYY-MM-DD`). Separate multiple timestamps with line breaks. For tasks with hourly or minute scheduling cycles, you need to first select the data timestamp, and then select the data backfill time range accurate to the minute, which defines the data timestamp and time range for the data backfill.
Select Field	For modeling task backfills, select the fields to include: If the primary key or source table has changed, to ensure data consistency and correctness, only the full table data backfill mode is supported. If the primary key or source table has not changed, you can choose full table data backfill mode or specified field data backfill mode: Full Table: Backfills all fields in the table. Note This does not include registered fields. Specified Fields: Select specific fields for backfill. The system automatically includes co-located fields in the same materialization node and system-required fields: Fields in the same materialization node as the selected fields. Fields that are required by the system implementation, such as when the scheduling cycle of a modified metric is refreshed but the materialization remains unchanged.
Task Runtime	Controls when each node instance runs. Select Ignore Scheduled Instance Runtime or Wait For Scheduled Instance Runtime. Ignore Scheduled Instance Runtime: (Default) Runs the instance without waiting for its scheduled time. Wait For Scheduled Instance Runtime: Runs the instance only after its scheduled time arrives.
Other Configurations
Single Instance Data Backfill	Only logical fact tables can be selected. Updates data for all selected dates through a single instance, saving computing resources and reducing backfill time compared to multi-instance concurrent backfill.
Number Of Concurrent Groups	Controls how many backfill processes run simultaneously (1 to 12 groups). If the timestamp span is shorter than the group count, actual parallelism equals the number of days. If the timestamp span exceeds the group count, instances within a group run sequentially while groups run in parallel. Example: timestamps Jan 11–13 with 2 groups → Group 1: Jan 11, Jan 12 (sequential); Group 2: Jan 13. Jan 11 and Jan 13 start simultaneously; Jan 12 starts after Jan 11 completes. Note Concurrent execution is not supported when selected nodes have cross-epoch dependencies.
Data Backfill Order	Run backfill in ascending or descending order of business time. Note Descending order is not supported when selected nodes have cross-cycle or self-dependencies.
Instances Of Paused Scheduled Tasks	Configure how backfill instances of paused tasks behave: Pause Running (May Block Data Backfill Process): Backfill instances from paused tasks remain paused, blocking downstream instances. Note Use when neither the current task nor its downstream tasks need to run. Dry-Run: Backfill instances from paused tasks succeed directly without execution. Note Use when the current task does not need to run but downstream tasks must proceed normally. Run Normally: All data backfill instances generated by paused tasks run normally. Note Use when a paused node must run normally for the selected backfill timestamps.
Instances Of Dry-Run Scheduled Tasks	Configure how backfill instances of dry-run tasks behave: Dry-Run: If dry-run is selected, the data backfill instances generated by the selected dry-run scheduled tasks will directly succeed in dry-run mode. Run Normally: All data backfill instances generated by dry-run tasks run normally.
Specify Temporary Schedule Resource Group	If you have enabled the custom resource group feature, you can specify a temporary resource group for this backfill to handle temporary resource needs. Resource group overview. If not specified, each task uses its configured schedule resource group. Note Only resource groups with batch operations in their application scenarios are supported. Backfill Current Task (modeling tasks) does not support this configuration.

Step 2: Select recommended fields

When Specified Fields for a modeling task have associated fields, you can select optional associated fields in the Select Recommended Fields step. Association reasons include calculation logic has changed, primary key of the primary table with changed calculation logic, and primary key of the child table where the primary table's primary key has changed.
- Fields With Changed Calculation Logic: The field's current calculation logic differs from historical partitions within the selected timestamps. Include these fields in the backfill.
- Primary Key Of The Primary Table With Changed Calculation Logic: The primary key logic in the primary table of a parent-child dimensional table has changed within the selected timestamps. Include these primary key fields in the backfill.
- Primary Key Of The Child Table Where The Primary Table's Primary Key Has Changed: The primary key logic in the primary table has changed (affecting child table output). Include the child table's primary key fields in the backfill.

Click OK to complete the data backfill operation for the current task.

Backfill data for the current and upstream/downstream tasks

In the Data Backfill - Backfill Current And Downstream Tasks dialog box, configure the data backfill task.

Step 1: Basic information configuration

Parameter	Description
Data Backfill Instance Name	Auto-generated in the format Node Name_Run Date_Instantiation Number. You can modify it manually.
Data Backfill Runtime	When the backfill instance starts scheduling. Select Run Now or Custom. Run Immediately: The backfill instance runs as soon as configuration is complete. Custom: Specify when the backfill instance starts scheduling. If the system time zone (User Center) differs from the scheduling time zone, both are displayed. The system converts the selected run time to the scheduling time zone automatically. Note The custom runtime must be later than the current time. After you configure a custom runtime, you can set Data Timestamp to a date up to the custom date. Scheduled backfill tasks generate pending instances at 23:00 on the day before the run time.
Data Timestamp	Select the data timestamp range for backfill, calculated in the configured scheduling time zone. Options vary by scheduling cycle: For tasks with daily, weekly, monthly scheduling cycles, you can select By Range, By Cycle, and Custom data timestamps. The application scenarios for each option are as follows: By Range: Refresh data for consecutive timestamps. Note For a single day, select the same start and end date. By Cycle: Refresh data for specific weekdays or monthly dates within a time range. Weekly: Refreshes the selected days of the week within the continuous time period. Monthly: Refreshes the selected dates of each month within the continuous time period. Note Month-end refers to the last day of each month. Custom: Enter non-consecutive timestamps manually (format: `YYYY-MM-DD`). Separate multiple timestamps with line breaks. For tasks with hourly or minute scheduling cycles, you need to first select the data timestamp, and then select the data backfill time range accurate to the minute, which defines the data timestamp and time range for the data backfill.
Select Field	For modeling task backfills, select the fields to include: If the primary key or source table has changed, to ensure data consistency and correctness, only the full table data backfill mode is supported. If the primary key or source table has not changed, you can choose full table data backfill mode or specified field data backfill mode: Full Table: Backfills all fields in the table. Specified Fields: Select specific fields for backfill. The system automatically includes co-located fields in the same materialization node and system-required fields: Fields in the same materialization node as the selected fields. Fields that are required by the system implementation, such as when the scheduling cycle of a modified metric is refreshed but the materialization remains unchanged.
Task Runtime	Controls when each node instance runs. Select Ignore scheduled instance runtime or Wait for scheduled instance runtime. Ignore scheduled instance runtime: (Default) Runs the instance without waiting for its scheduled time. Wait for scheduled instance runtime: Runs the instance only after its scheduled time arrives.

Step 2: Select recommended fields

When Specified Fields for a modeling task have associated fields, you can select optional associated fields in the Select Recommended Fields step. Association reasons include calculation logic has changed, primary key of the primary table with changed calculation logic, and primary key of the child table where the primary table's primary key has changed.
- Fields With Changed Calculation Logic: The field's current calculation logic differs from historical partitions within the selected timestamps. Include these fields in the backfill.
- Primary Key Of The Primary Table With Changed Calculation Logic: The primary key logic in the primary table of a parent-child dimensional table has changed within the selected timestamps. Include these primary key fields in the backfill.
- Primary Key Of The Child Table Where The Primary Table's Primary Key Has Changed: The primary key logic in the primary table has changed (affecting child table output). Include the child table's primary key fields in the backfill.
Data backfill configuration

Data Backfill Range List Mode Mass Mode: Select upstream and downstream tasks for backfill using List Mode or Mass Mode.

Important
Cross-node parameter related instructions: When selecting nodes, select all upstream nodes that the node references cross-node parameters from. When a downstream (Down) node references cross-node output parameters from an upstream (Up) node, if you backfill the downstream node without selecting the upstream node in the same backfill instance, the cross-node input parameters use values from the upstream node's most recent N days of running records. If no records exist or they exceed N days, the default value is used. N defaults to 15 days, which may be subject to change. Select both upstream and downstream nodes. Parameter configuration and using node parameters.
List mode

Select upstream and downstream tasks at all levels (1 to 10 or all). The list displays up to 2,000 nodes — use Mass Mode if this limit is exceeded. Click the icon to filter by Node Type, Project, or O&M Owner.
Note

If the starting task is a logical table, the display range of downstream tasks depends on the logical table fields selected for data backfill.

The display range of upstream and downstream tasks includes all upstream and downstream tasks of the selected fields of the current table, including associated required fields, but not including associated recommended fields.
Filter Paused Tasks And Their Downstream:
- Selected by default. When selected, the list does not display nodes with paused scheduling and all their downstream nodes at the specified level and filtering conditions, and also cancels any already selected paused tasks.
- For logical tables, if they contain paused fields, they are filtered out. All downstream tasks of fields in logical tables marked as paused in the dependency downstream list are also filtered out.
  
  Note
  Downstream logical table fields can only be selected for data backfill as a whole, and paused fields cannot be filtered out separately.
Mass mode

Use Mass Mode when List Mode cannot meet your node selection requirements (too many nodes or batch selection needed). Mass Mode searches downstream from the current node within the selected scope, filtering and orchestrating nodes by their dependencies. It supports these filter parameters:
- Coverage: Specify the scope using one of the following options: Specify projects, Specify node output names, All descendant nodes of the current node, Specify level-1 child nodes and all their descendants, Specify endpoints, Specify node names, Specify node IDs, Specify start points, All ancestor and descendant nodes of the current node, or All ancestor nodes of the current node.
  - Specify Project: Specify the data backfill range by specifying projects.
  - Specify Node Output Name: Specify the data backfill range by entering node output names. When entering multiple nodes, use line breaks to separate them. You can enter up to 1000 nodes.
  - All Downstream/Upstream of the current node: Backfill data for all upstream/downstream nodes of the current node.
  - Specify Level-1 Child Nodes And All Their Downstream: Backfill data for several level-1 child nodes of the current node and all their downstream nodes.
  - Specify End Point: Backfill data for all nodes on the chain from the starting point to the end point. The starting point defaults to the current node and cannot be modified. You can select multiple end point nodes.
  - Specify Starting Point: Backfill data for all nodes on the chain from the starting point to the end point. The end point defaults to the current node and cannot be modified. You can select multiple starting point nodes.
  - Specify node names/Specify node IDs: Backfills data for descendant nodes of the current node that have the specified names or IDs. Separate multiple entries with line breaks. You can enter a maximum of 5,000 entries. If a node name corresponds to multiple tasks, click Select Data Backfill Nodes in the prompt. In the Nodes With Duplicate Node Names dialog box, select the correct nodes to confirm which nodes require a data backfill.
    
    Note
    
    If the selected end point node is not a downstream node of the starting point, data backfill will only be performed on the two isolated nodes: the starting point and the end point.
    
    End points can be searched by ID/Node Name, with the search scope covering all nodes within the current tenant.
    
    Logical table task end points only support selection of the full table (all fields).
  - All Upstream And Downstream Of Current Node: Backfill data for all upstream and downstream nodes of the current node.
- Exclude Within Selected Range: Specify Node Output Names or Node Names to be excluded from the coverage range. Exclude Paused Nodes And Their Downstream is selected by default, similar to Filter Paused Nodes And Their Downstream in list mode.
  Note
  
  After excluding certain tasks within the selected range, isolated task nodes may appear in the DAG graph of the data backfill instance.
  
  This is suitable for scenarios where data backfill is only needed for one downstream task node.
- Selected node list: In massive mode, you can click View Selected Node List to confirm the nodes for data backfilling or click Export Selected Node List to export the list as a local csv file.

Other Configurations

Parameter	Description
Number Of Concurrent Groups	Controls how many backfill processes run simultaneously (1 to 12 groups). If the timestamp span is shorter than the group count, actual parallelism equals the number of days. If the timestamp span exceeds the group count, instances within a group run sequentially while groups run in parallel. Example: timestamps Jan 11–13 with 2 groups → Group 1: Jan 11, Jan 12 (sequential); Group 2: Jan 13. Jan 11 and Jan 13 start simultaneously; Jan 12 starts after Jan 11 completes. Note Concurrent execution is not supported when selected nodes have cross-cycle dependencies.
Data Backfill Order	Run backfill in ascending or descending order of business time. Note Descending order is not supported when selected nodes have cross-cycle dependencies.
Dry-Run This Node	Select whether this task runs in dry-run mode: Yes: The backfill instance for this task succeeds directly without execution. Note Use when the current node does not need backfill but downstream nodes do. No: This node runs normally.
Suspending Scheduled Task Instances	Configure how backfill instances of paused tasks behave: Pause Running (May Block Data Backfill Process): Backfill instances from paused tasks remain paused, blocking downstream instances. Note Use when neither the current task nor its downstream tasks need to run. Dry-Run: Backfill instances from paused tasks succeed directly without execution. Note Use when the current task does not need to run but downstream tasks must proceed normally. Run Normally: All data backfill instances generated by paused tasks run normally. Note Use when a paused node must run normally for the selected backfill timestamps.
Instances Of Dry-Run Scheduled Tasks	Configure how backfill instances of dry-run tasks behave: Dry-Run: Backfill instances from dry-run tasks succeed directly without execution. Run Normally: All data backfill instances generated by dry-run tasks run normally.
Hourly Range Impact Scope	For hourly or minute tasks, configure the effective range: Do Not Affect Daily/Weekly/Monthly Scheduled Tasks (Run When Selected): Downstream tasks are unaffected by the hourly range and all run. Daily/Weekly/Monthly Scheduled Tasks Only Run If Their Scheduled Run Time Is Within The Selected Hourly Range: Downstream tasks only run if their scheduled time falls within the selected hourly range.
Specify Temporary Schedule Resource Group	If you have enabled the custom resource group feature, you can specify a temporary resource group for this backfill to handle temporary resource needs. Resource group overview. If not specified, each task uses its configured schedule resource group. Note Only resource groups with batch operations in their application scenarios are supported.

Click OK to complete the data backfill operation for the current and downstream tasks.

What to do next

After submitting the backfill, you can manage backfill instances — view running logs, node code, or stop execution. Data backfill instance O&M overview.