All Products
Search

This Product

    DataWorks:Data backfilling FAQ

    Document Center

    DataWorks:Data backfilling FAQ

    Last Updated:Jun 20, 2026

    This topic answers frequently asked questions about data backfilling.

    Note

    We recommend reading the user guide Manage data backfilling instances first.

    How data backfilling works

    You can use the data backfilling feature to process data for a historical or future time range. The scheduling parameters used by a node are automatically replaced with the corresponding values based on the selected data timestamp. The following example shows how to write incremental data from a MySQL database to a time-based partition in MaxCompute. For a batch synchronization node, the scheduling parameter bizdate=$bizdate must be configured in three places: 1) In the data source settings, use the filter condition STR_TO_DATE('${bizdate}','%Y%m%d') <= gmt_modify_time AND gmt_modify_time < DATE_ADD(STR_TO_DATE('${bizdate}','%Y%m%d'), interval 1 day) to filter for incremental data. 2) In the data destination settings, set the partition information to pt=${bizdate} and select Insert Overwrite as the cleanup rule. 3) On the scheduling configuration panel, enter bizdate=$bizdate in the Parameters field and select Generate instance for the next day (T+1). The ${bizdate} value must be consistent across all three settings to ensure that each scheduled run synchronizes only the incremental data for the corresponding data timestamp.

    Why parallelism fails for hourly and minute nodes

    • Symptom

      For a node scheduled by hour or minute, instances do not run in parallel even when you enable parallelism for data backfilling.

    • Cause

      The parallelism setting for data backfilling determines whether instances for a range of data timestamps, at a daily granularity, run concurrently. It does not control the concurrency of instances within the same day for hourly or minute-level nodes. The intra-day concurrency for these nodes depends on whether a self-dependency is configured. For more information, see Scenario 2: Configure a dependency on the previous cycle.

    • Resolution

      • If you disable parallelism, instances for different data timestamps run serially. The instance for the next data timestamp starts only after the previous one is complete.

      • If you enable parallelism, you can set the number of concurrent instances (for example, 2, 3, 4, or 5) to run data backfilling instances for multiple data timestamps in parallel.

      Example scenario: Suppose you need to backfill one week of data for an hourly or minute-level node.

      • If the node has a self-dependency, the instances for that week run sequentially in chronological order.

      • If the node does not have a self-dependency, all instances for each day run in parallel.

    Why instances enter the Pending (Schedule) state

    • Symptom

      After you specify a data timestamp to backfill data, the instance does not run. It remains in the Pending (Schedule) state and is highlighted in yellow.

    • Cause

      This occurs if the scheduled run time for the data timestamp that you selected is in the future.

    • Resolution

      To run these instances immediately, select the Run Retroactive Instances Scheduled to Run after the Current Time option in the Backfill Data dialog box.

      Note

      • If you select a data timestamp that corresponds to a future run time and you do not select this checkbox, the instance enters the Pending (Schedule) state.

      • If you select a data timestamp that corresponds to a future run time and you select this checkbox, the instance runs immediately.

    Why are recent backfills in a pending state?

    • Symptom

      When you backfill data for yesterday's or today's data timestamp, the instance enters the Pending (Schedule) state.

    • Cause

      The platform's scheduling is typically based on a T+1 model, which means that data with yesterday's data timestamp is processed by a run scheduled for today. When you backfill data for a specific data timestamp, you are re-running the cycle instance for that date.

      To find the cycle instance scheduled to run today, you need to filter by yesterday's data timestamp. On the Cycle Instance page in Operation Center > Cycle Task O&M, set the Data Timestamp filter to Yesterday to view the corresponding cycle instances and their statuses.

    Why a backfill creates multiple instances

    • Symptom

      Backfilling data for a time range such as 00:00 to 01:00 generates multiple instances.

    • Cause

      The number of instances generated depends on the node's scheduling configuration.

      • For example, if an hourly node is scheduled to run every hour from 00:00 to 23:59, backfilling data for the 00:00 to 01:00 time range generates two instances with scheduled run times at 00:00 and 01:00.

      • Similarly, if a node is scheduled to run every 30 minutes, backfilling data for the 00:00 to 01:00 time range generates three instances with scheduled run times at 00:00, 00:30, and 01:00.

    Why large backfills are pending for resources

    • Symptom

      During a large-scale data backfilling operation, some instances enter the Pending (Resources) state and are highlighted in yellow.

    • Cause

      Each resource group has a maximum concurrency limit for running nodes. If the number of concurrently running nodes exceeds this limit, new instances are queued in the Pending (Resources) state.

      Note

      To troubleshoot this issue, see Waiting for resources.

    Why the "run time out of range" error?

    • Symptom

      An error might occur during data backfilling, stating that the triggered node's run time is outside the selected time range.

    • Cause

      This error occurs because the node's actual scheduled run time falls outside the time range you specified. For hourly and minute-level nodes, you must specify a run time window for data backfilling. For example, in the Backfill Data dialog box, if you set the Select a run time window to 00:00 ~ 01:00 but the triggered node is scheduled to run at a different time, the backfill operation fails. To resolve this, adjust the run time window to include the node's scheduled run time.

    Why are no backfill instances generated?

    • Symptom

      You initiated a data backfilling operation for a node, but no backfill instances were generated.

    • Cause

      Instances are not generated if the data timestamp is outside the node's effective date range. Ensure the data timestamps for the backfill fall within the node's effective date range. On the node's scheduling configuration panel, you can see settings like these: Time attribute is set to Normal scheduling, Rerun attribute is not selected, Auto rerun on error is not selected, the effective date range is 1999-09-09 to 1999-10-22 (scheduling takes effect only within this range), Suspend scheduling is not selected, the scheduling cycle is Daily, and the scheduled time is 00:19. If the data timestamp for the data backfilling is outside the effective date range, no backfill instances are generated.

    Backfilling weekly and monthly nodes

    • Description: When backfilling data for weekly or monthly nodes, you must set the data timestamp to the day before the scheduled run time. A node scheduled to run on a specific day of the week or month executes only on that designated day. On other days, dry-run instances are generated, but the node does not actually run. The status of such an instance indicates a weekly or monthly dry-run cycle. For more information, see Scenario 1: Dry run for weekly/monthly cycle.

      Note

      • The data timestamp for the backfill is calculated as: data timestamp = Scheduled run date - 1 day.

      • For more information about the relationship between scheduling parameters, data timestamps, scheduled times, and actual run times, see Supported formats for scheduling parameters.

    • Example: Backfill data for a monthly node

      For a node scheduled to run at 00:00 on the first day of every month, you must set the data timestamp to the last day of the previous month. For a monthly node where the scheduling cycle is set to Monthly and its run time is 00 00 00 1 * ?, set both the start and end dates for the Data Timestamp in the Backfill Data dialog box to 2022-11-30 to process data for November 2022, and then click OK.

    How the parallel instance limit affects backfills

    • When the maximum parallel instances setting is enabled for a node, this limit applies to both data backfilling instances and regular cycle instances. They share the same concurrency quota.

    • When instance concurrency exceeds the limit, the system primarily throttles historical data backfilling instances, which are instances for data timestamps before today.

    • The maximum parallel instances limit applies only to instances generated after the setting is enabled. Instances that are already running or queued are not affected.

    • The value for maximum parallel instances can range from 1 to 10,000, and the default is 1.