DataWorks:Backfill data instance operations

1. Instance lifecycle and log retention

• Instance cleanup: Data backfill instances cannot be deleted manually. They are automatically cleaned up by the platform about 30 days after creation. If a task no longer needs to run, you can freeze its instances to stop scheduling them.

• Retention policy: Instance and log retention periods vary by resource group type.

  Resource group type	Instance retention policy	Log retention policy
  Shared resource group for scheduling	30 days	7 days
  Exclusive resource group for scheduling	30 days	30 days
  Serverless resource group	30 days	30 days

• Large log cleanup: For completed instances, when their runtime logs exceed 3 MB, the platform cleans them up on a daily schedule.

2. Instance runtime rules

• Strict day-by-day dependency: Data backfill runs serially by business date. Instances for the next day begin running only after all instances for the previous day succeed. If any instance fails, all tasks for subsequent dates are blocked.

• Cross-day dependencies do not take effect: Cross-day dependencies between tasks do not take effect during data backfill. If you backfill tasks for a period spanning multiple days, by default the instances for the next day run only after all instances for the previous day finish, rather than being triggered based on the original cross-day dependencies.

• Cross-day handling for specified range/set dependencies: For dependencies of the specified range or set type, if the dependency spans days, only same-day dependencies are added when data backfill instances are generated. Cross-day dependencies are automatically ignored.

• Concurrency behavior for hourly/minute tasks: When you backfill all instances for a given day, how they run is determined by the task's self-dependency attribute:

  • Self-dependency not set: All instances within the day (for example, 00:00, 01:00, and so on) can run in groups, as long as their respective upstream dependencies are met.

  • Self-dependency set: All instances within the day run strictly in serial (for example, the 01:00 instance runs only after the 00:00 instance succeeds).

• Conflict with scheduled instances: To ensure regular scheduling, scheduled instances have higher priority than data backfill instances. If both run at the same time, manually terminate the data backfill instance.

• Handling of blocklisted tasks: If a blocklisted task is an intermediate node in the data backfill chain, it also runs as a dry run, which may affect the data output of downstream tasks.

3. Scheduling resources and priority

• Configure resources reasonably: Too many data backfill instances or too high a degree of parallelism consumes large amounts of scheduling resources, which may affect the normal operation of scheduled tasks. Configure them reasonably as needed.

• Priority downgrade policy: To ensure core business, the platform dynamically adjusts task priority based on the business date of the data backfill:

  • Backfilling yesterday's (T-1) data: Task priority remains unchanged and is determined by the priority of the baseline it belongs to.

  • Backfilling historical (T-2 and earlier) data: Tasks are automatically downgraded according to the following rules:

    • Original level 7 or 8 → downgraded to level 3

    • Original level 5 or 3 → downgraded to level 2

    • Original level 1 → remains unchanged

Step 1: Go to the data backfill page

1. Log on to the DataWorks console. In the target region, click Data Development and O&M > Operation Center in the left-side navigation pane. Select a workspace from the drop-down list and click Go to Operation Center.

2. In the left-side navigation pane, click O&M Assistant > Run to go to the data backfill page.

Step 2: Create a data backfill task

On the data backfill page, click Create Data Backfill Task and configure the data backfill task based on your business requirements.

1. Configure Basic information.

   The platform automatically generates a data backfill name in the default format, which you can modify as needed.

2. Select the tasks to backfill.

   You can use methods such as Manually Select, Select by Link, Select by Workspace, and Specify Task and All Descendant Tasks to initiate data backfill for tasks that you have permission to operate, and select other tasks to backfill based on that task. The configuration parameters differ across methods.

   Manual selection

   Select one or more tasks as root tasks, and then select downstream tasks of the root tasks as the task scope for this data backfill. This method is compatible with the previous Current Node, Current Node and Descendant Nodes, and Advanced Mode data backfill solutions.

   The detailed parameters are described below.

   Parameter	Description
   Method	Select Manually Select.
   Add Root Task	Search for and add root tasks by name or ID. You can also click Batch Add to search by conditions such as resource group, schedule, and workspace, and add multiple root tasks in batches. Note You can select only tasks in workspaces that you have joined (that is, workspaces of which you are a member).
   Selected Data Backfill List	The tasks to be backfilled. The list shows the added root tasks. Based on the root tasks, you can select the downstream tasks to backfill. Note You can filter downstream tasks by dependency level. The direct downstream tasks of a root task are on level 1 by default, and so on. You can backfill up to 500 root tasks at a time, with a maximum of 2000 total tasks (root tasks and their downstream tasks) (3000 in the China (Beijing) and China (Hangzhou) regions). When a task is configured with a maximum number of parallel instances, data backfill instances are also subject to this limit and share the concurrency quota with scheduled instances.
   Task Blacklist	If a task does not need to be backfilled, you can add it to the blocklist. Tasks in the blocklist do not participate in this data backfill. Note Only root tasks can be added to the blocklist. If a child task of a root task does not need to be backfilled, remove it from the Selected Data Backfill List. If a task in the blocklist is an intermediate task in this data backfill (that is, its upstream and downstream dependencies are within this data backfill scope), the task runs as a dry run to ensure that downstream tasks run (it does not actually run and immediately returns a success status after starting), but this may cause abnormal data output for the downstream tasks of this task.

   Selection by chain

   Select a start task and one or more end tasks. Through automatic analysis, all tasks between the root task and the end tasks are used as the task scope for this data backfill (including the start and end tasks).

   The detailed parameters are described below.

   Parameter	Description
   Method	Select Select by Link.
   Search Tasks	Search by name or ID to add a start task and one or more end tasks. The platform analyzes the intermediate tasks based on the start and end tasks (that is, intermediate tasks are direct or indirect downstream tasks of the start task and direct or indirect upstream tasks of the end tasks).
   Intermediate Task List	The list of intermediate tasks automatically analyzed by the platform based on the start and end tasks. Note The list shows only 2000 tasks. Tasks beyond 2000 are not displayed but run normally.
   Task Blacklist	If a task does not need to be backfilled, you can add it to the blocklist. Tasks in the blocklist do not participate in this data backfill. Note If a task in the blocklist is an intermediate task in this data backfill (that is, its upstream and downstream dependencies are within this data backfill scope), the task runs as a dry run to ensure that downstream tasks run (it does not actually run and immediately returns a success status after starting), but this may cause abnormal data output for the downstream tasks of this task.

   Selection by workspace

   Select a task as the root task, and determine the task scope for this data backfill based on the workspaces of the downstream nodes. This method is compatible with the previous Massive Node Mode data backfill solution.

   Note

   • Compatible with the previous Massive Node Mode data backfill solution.

   • Configuring a task blocklist is not supported.

   The detailed parameters are described below.

   Parameter	Description
   Method	Select Select by Workspace.
   Add Root Task	Search for and add root tasks by name or ID. The platform backfills tasks in the workspaces of the downstream tasks of the root task. Note You can select only tasks in workspaces that you have joined (that is, workspaces of which you are a member).
   Include Root Node	Define whether this data backfill includes the root task.
   Data Backfill Workspaces	Based on the workspaces of the downstream tasks of the root task, select the workspaces whose nodes you want to backfill. Note You can select only DataWorks workspaces in the current region for data backfill operations. After you select a workspace, all nodes in the workspace are backfilled by default. You can customize the data backfill allowlist and blocklist as needed.
   Add to Whitelist	Nodes that still need to be backfilled in addition to the nodes contained in the selected workspaces.
   Task Blacklist	Nodes in the selected workspaces that do not need to be backfilled.

   Specify a task and all its downstream tasks

   After you select a root task, the platform automatically analyzes and uses the task and all its downstream tasks as the task scope for this data backfill.

   Important

   You can see the triggered tasks only while the data backfill task is running. Use with caution.

   The detailed parameters are described below.

   Parameter	Description
   Method	Select Specify Task and All Descendant Tasks.
   Add Root Task	Search for and add root tasks by name or ID. The platform backfills the root task and all its downstream tasks. Note You can select only tasks in workspaces that you have joined (that is, workspaces of which you are a member). If the selected root task has no downstream tasks, only the data of the current root task is backfilled after the data backfill task is submitted.
   Task Blacklist	If a task does not need to be backfilled, you can add it to the blocklist. Tasks in the blocklist do not participate in this data backfill. Note If a task in the blocklist is an intermediate task in this data backfill (that is, its upstream and downstream dependencies are within this data backfill scope), the task runs as a dry run to ensure that downstream tasks run (it does not actually run and immediately returns a success status after starting), but this may cause abnormal data output for the downstream tasks of this task.

   1. Configure the data backfill runtime policy.

      Configure information such as the runtime of the data backfill task, whether to run in groups, whether to trigger alerts, and the resource group to use, based on your business requirements.

      The parameters are described below.

      Parameter	Description
      Data Timestamp	You can specify the business dates to backfill for the task. Manual entry, AI-based generation, and batch entry are supported. Based on the dates and options you select, the system uses different execution policies: Scenario 1: Backfill historical data (business date < current date) This is the most common data backfill scenario. When the business date you select is earlier than today, the system immediately creates and runs the task instance for that historical date to retrace and recompute past data. Purpose: Fix historical data errors and backfill missing data. Execution mode: Run immediately. Scenario 2: Schedule a future task (business date > current date) If you select a future business date and do not select any special options, this is equivalent to creating a future "one-time" scheduled task. Purpose: Schedule a one-time task run in advance for a specific known date in the future. Execution mode: Run when scheduled. The instance is created and enters the pending state. It runs automatically based on the task's own schedule settings only when its corresponding business date arrives. Scenario 3: Immediately run instances whose scheduled time is later than the current time (select Run immediately) Run Retroactive Instances Scheduled to Run after the Current Time is an advanced option. This option is displayed when the business date of the data backfill is later than the current date, or when the business date is T-1 and the task contains instances whose scheduled time is later than the current time. After you select it, instances whose scheduled time is later than the current time run immediately instead of waiting until their corresponding scheduled time. Purpose: Run task instances for future dates in advance, prepare data for specific partitions in advance for data migration or testing, or immediately run hourly or minute instances whose scheduled time has not yet arrived when backfilling T-1 data. Execution mode: Run immediately after selection. Example 1 (backfilling future data): The current date is 2024-03-12, and you choose to backfill the data for 2024-03-17 and select Run Retroactive Instances Scheduled to Run after the Current Time. In this case, the task instance starts immediately on 2024-03-12, but the business date parameter it uses when running (which affects the data partition, for example) is 2024-03-17. Example 2 (backfilling T-1 data): The current time is 2024-03-12 14:30, and you choose to backfill the data for 2024-03-11 (T-1). The task is scheduled hourly (runs once every hour). If you do not select this option, instances scheduled for 15:00, 16:00, and other times later than 14:30 must wait until their respective scheduled times to run. After you select Run Retroactive Instances Scheduled to Run after the Current Time, all instances run immediately. Note The concept of business date: In batch computing, a task typically processes yesterday's (T-1) data today (T). The data backfill feature generates instances for a specified "business date" so that you can precisely control which day's data the task processes. Multiple time periods: To backfill data for multiple non-consecutive dates, click Add to configure multiple time periods. Resource planning: We recommend that you do not set too long a time span for a single data backfill. A large number of data backfill instances consume scheduling resources, which may affect the operation of regular scheduled tasks.
      Period	Specify the period during which the selected tasks run. A task generates and runs instances only when its scheduled time falls within this period. You can use this feature to run only instances of specified periods for hourly or minute scheduled tasks (that is, to backfill only data within the specified periods). The default is 00:00-23:59. Note If a task's scheduled time is not within this period, the task does not generate instances. If a case exists where a larger-cycle task depends on a smaller-cycle task (for example, a daily task depends on an hourly task), isolated instances may be generated, which block task operation. We recommend that you modify this parameter only when an hourly or minute scheduled task needs to backfill data for specified periods.
      Run by Group	When you backfill data for multiple business dates, you can specify a number of groups to run this data backfill task concurrently. The values are as follows: Yes: The platform splits the business dates by the specified number of groups and generates multiple data backfill batch groups that run concurrently based on the grouping result. No: Runs serially in business date order. The next data backfill instance runs only after the previous one finishes. Note Whether the instances of a given day for an hourly or minute task run concurrently in groups depends on whether the hourly or minute task itself is configured with a self-dependency. The number of groups ranges from 2 to 10. Multiple instance groups run concurrently as follows: If the time span of the business dates is less than the number of groups, the tasks run concurrently in groups. For example, if the business dates are January 11 to January 13 and the number of groups is 4, only three data backfill instances are generated (each data backfill instance corresponds to one business date), and the three instances run concurrently in groups. If the time span of the business dates is greater than the number of groups, the platform runs tasks both serially and in parallel based on the business date Order. For example, if the business dates are January 11 to January 13 and the number of groups is 2, two data backfill instances are generated (one of the data backfill instances contains two business dates, and the tasks for these two business dates run serially), and the two data backfill instances run in parallel.
      Alert for Data Backfill	Set whether this data backfill operation triggers alerts. Yes: An alert is generated when the trigger condition is met. No: This data backfill does not trigger alerts.
      Trigger Condition	Set the alert trigger condition only when Alert for Data Backfill is set to Yes: Alert on Failure or Success: An alert is generated whether the data backfill succeeds or fails. Alert on Success: An alert is generated only when the data backfill succeeds. Alert on Failure: An alert is generated only when the data backfill fails.
      Alert Notification Method	Only when Alert for Data Backfill is set to Yes can you choose to receive alerts by Text Message and Email, SMS, or Email. The alert recipient is the initiator of the data backfill. Note Click Check Contact Information to verify whether the phone number or email address of the alert recipient is registered. If not registered, see Configure alert contacts to configure it.
      Order	Choose to run the data backfill in Ascending by Business Date or Descending by Business Date.
      Scheduling Resource Group	Specify the resource group used to run the data backfill instances. Inherit Task Configuration: Uses the resource group originally configured for the scheduled task to run the data backfill instances. Specify Resource Group for Scheduling: Uses a specified resource group to run the data backfill instances, avoiding resource contention between data backfill instances and scheduled instances. Note Make sure that the network connectivity of the resource group is configured; otherwise, the task may fail. If the specified resource group is not associated with the relevant workspace, the resource group of the original scheduled task is still used. We recommend that you use a serverless resource group or an exclusive resource group for scheduling, which can provide a dedicated compute resource group to ensure fast and stable data transfer when tasks run with high concurrency and cannot be run in staggered time periods.
      Execution Period	Specify when the data backfill task generated this time runs. Inherit Task Configuration: Runs based on the scheduled time of the data backfill instances. Specify Time Period: Set the data backfill task to be triggered and run only within a specified time period. Set the runtime period reasonably based on the number of tasks to be backfilled. Note Tasks that are in the not-run state after this period are not run, while tasks that are in the running state after this period continue to run.
      Computing Resources	Currently, only EMR and Serverless Spark compute resources can be set as the compute resource for data backfill. Make sure that the mapped compute resource exists and is available; otherwise, task scheduling may be affected.

3. Configure the data backfill task check policy.

   Used to configure whether to terminate task execution when the data backfill check fails. The platform checks the basic status and potential risks of this data backfill task, as follows:

   • Basic status: The number of tasks involved in this data backfill, the number of instances generated, and whether node cycles, isolated nodes, or instances without permission exist.

   • Risk detection: Detects whether node cycles and isolated nodes exist. Both situations cause abnormal task operation. You can set the data backfill task to terminate when the check fails.

4. Click Submit to complete the creation of the data backfill task.

Step 3: Run the data backfill task

When the data backfill task reaches its configured runtime and no anomalies exist, the data backfill task is automatically triggered to run.

The data backfill task cannot run if the following conditions are met:

• The data backfill task has checks enabled, and a check failure blocks task operation. For more information, see Create a data backfill task: Step 4.

• The data backfill operation has extension checks enabled, and a check failure blocks task operation. For more information, see Manage extensions.

Query data backfill instances

In the left-side navigation pane, click O&M Assistant > Run to go to the data backfill page.

On the right side of the data backfill page, click Show Search Options to filter the instances to query by conditions such as Data Backfill Instance Name, Running status, and Task type. You can also quickly terminate running data backfill tasks in batches.

View data backfill instance status

In this area, you can view information about data backfill instances, including:

• Node Name: Shows the name of the data backfill instance. Click the icon before the instance name to show the run date and run status of the instance, as well as the nodes it contains and their run details.

• Check Status: The check status of the current data backfill instance.

• Running status: Includes statuses such as succeeded, failed, waiting for resources, and waiting to be triggered. You can troubleshoot related issues based on abnormal statuses.

• Nodes: The number of nodes contained in the data backfill instance.

• Data Timestamp: The date on which the data backfill instance runs.

• Max Parallel Instances: Shows the maximum number of parallel instances configured for the task, either Unlimited or a specific value (1 to 10000). This setting limits the maximum number of instances of the same task that can run at the same time. This value is shared among scheduled instances, data backfill instances, and test instances.

• View View Task Analysis Results: You can view the instances the task is expected to generate, the run dates, and the risk check results, to promptly handle blocked tasks. The Data backfill task analysis result page displays the following information: Data backfill name, Task analysis (which lists each workspace name and the expected number of instances generated in a table), the Business date range, the Specified period time range, and the Risk check result (including whether the cycle detection and isolated task detection passed).

• Actions: You can perform operations such as Batch Terminate, Batch Rerun, and Reuse on data backfill instances.

  Operation	Description
  Batch Terminate	You can select and batch terminate instances in the Running state within the data backfill instances. After this operation, the corresponding instances are set to the Run failed state. Note Terminating instances in the Not Running, Succeeded, or Run failed state is not supported.
  Batch Rerun	Rerun data backfill instances in batches. Note Only instances in the Succeeded or Run failed state can be rerun. A batch rerun immediately reruns the selected instances and does not run them according to the dependencies among the instances. To run them in order, use Rerun Descendent Nodes or perform the Run operation again.
  Reuse	You can reuse the node set of a previous data backfill operation to quickly select the nodes to backfill.

Manage data backfill tasks

In this area, you can view information about the nodes contained in the data backfill instance, including:

• Name: Click a node name to go to the node details page and view more node information.

• Timing Time: The scheduled runtime set for the node task.

• Start run time: The start time of the node task run.

• End Time: The end time of the node task run.

• Run when Long: The duration of the node task run.

• Actions: You can manage a data backfill node task with operations such as viewing the DAG, Terminate, and Rerun.

  Operation		Description
  DAG		View the DAG of the node to analyze the upstream and downstream tasks of the node. For more information, see Introduction to the DAG feature.
  Terminate		You can terminate a node in the Running state. After this operation, the node is set to the Run failed state. Note Terminating nodes in the Not Running, Succeeded, or Run failed state is not supported. This operation causes the instance to fail and blocks the operation of the instance's downstream nodes. Be aware of the risks and proceed with caution.
  Rerun		Rerun the target node task. Note Only nodes in the succeeded or failed state can be rerun.
  More	Rerun Descendent Nodes	Rerun the downstream nodes of the target node task.
  	Set to Successful	Set the status of the node task to succeeded.
  	Pause (Freeze)	Set the current node to the paused (frozen) state and stop scheduling. Note The Waiting for resources, Waiting time, and Running (node code running, data quality check in progress) states do not support the freeze operation.
  	Unfreeze	Resume scheduling for a paused (frozen) node.
  	View Lineage	View the lineage graph of the node.

For the selected task nodes, click Terminate or Rerun to batch terminate or rerun the selected node tasks.