Instance generation mode

How it works

In DataWorks, after you modify and Submit a task, the new configuration affects when your changes apply: either on the same day or on the next day.

Next day (default)

This default, recommended option maximizes the stability of your production environment by isolating changes from the current day's running instances.

Operating principle:
A deployment on Day T only changes the task definition and does not interfere with any instance runs on that day.
Impact on Day T (Deployment Day):
- Behavior: The system only updates the task's code and property definitions. This does not affect any instances already generated or scheduled to run on Day T. All instances on Day T run based on the configuration that was in place before the deployment.
- Recommendation: If you want the new logic to apply to the current day's data, manually perform a backfill data operation for the Day T instances after deployment.
Impact on Day T+1 (Next Day):
All changes take effect starting with the first auto-triggered instance on the day after deployment (Day T+1). These instances are generated and run based on the new configuration.

Immediately after deployment

This mode applies task changes on the day of deployment. It compares the task's deployment time with the scheduling time of its instances to determine how they should run on the current day (Day T).

Operating principle:
The system compares the scheduling time of each auto-triggered instance on Day T with the task's deployment time plus a 10-minute system buffer.
Scheduling time < Deployment time + 10 minutes
- Result: New tasks perform a dry run; modified tasks do not generate an instance.
- Behavior: For new tasks that have never been deployed, the instance expires, performs a dry run, and does not execute its business logic. This prevents new code from running on pre-deployment instances. For modified and re-deployed tasks, expired instances are not generated.
Scheduling time > Deployment time + 10 minutes
- Result: Normal execution.
- Behavior: The system immediately generates and runs the instance based on the new configuration.
Impact on Day T+1 (Next Day):
All auto-triggered instances on the next day (Day T+1) are generated based on the new configuration.

Important

To ensure predictable scheduling across days, a special cross-day window exists from 23:30 to 24:00. For any deployment submitted within this window, changes take effect on Day T+2, regardless of the selected generation mode.

Limitations

Effective Time of Change: The system generates instances in batches from 23:30 to 24:00 every day. For deployment operations submitted during this period, the changes take effect in the auto triggered instances that are generated on T+2.
Limitations on data source changes: If you only modify the data source associated with a node, auto triggered instances that have already been generated for the day are not updated, even if you select Immediately After Deployment. They will continue to run by using the data source that was configured before the change. To make the change take effect immediately, use the backfill data feature.

Immediate deployment scenarios

The Immediately After Deployment mode carries a higher risk. Improper use can lead to disordered scheduling dependencies, unexpected instance deletion or replacement, and compromised stability of same-day tasks.

Recommended use cases

Use this mode with caution and only in the following scenarios:

New tasks that must run on the same day: Use for new tasks without complex ancestor or descendant dependencies that need to run on the day of deployment.
Replacing existing instances: Use this mode to replace a pending auto-triggered instance that was generated for the current day but has not yet run.

High-risk scenarios (not recommended)

Avoid using this mode in the following scenarios, as it can complicate same-day scheduling dependencies and cause scheduling failures:

Modifying the schedule of a deployed task: This is especially risky for tasks with complex ancestor and descendant dependencies. Changing the scheduling cycle (for example, from daily to hourly) and deploying immediately can result in a mix of old instances being retained and new ones being created, leading to disordered dependencies.
Inconsistent Instance Generation Modes for Ancestor and Descendant Tasks: For example, an ancestor task uses the T+1 mode, while a descendant task uses Immediately After Deployment. This configuration causes instances of the descendant task generated on the same day to be unable to find their ancestor dependencies, which turns them into isolated tasks that cannot run automatically.

Alternative solution

For scenarios involving modifications to a deployed task, a safer approach is to:

Use the default T+1 mode to deploy tasks.
After the deployment succeeds, perform a backfill data operation on the task to manually trigger an instance for the current day.

Scenarios

Scenario 1: New task

After a new task is deployed, its instance execution depends on whether its scheduling time is before or after the deployment time, including the 10-minute buffer.

Scheduling time

Behavior

Later than (deployment time + 10 minutes)

The system generates a normal auto-triggered instance that runs at its scheduled time.

Earlier than or equal to (deployment time + 10 minutes)

The system generates an expired instance that is generated in real time. This instance is in a dry-run state and does not actually execute.

If you need to process data for the current day, you can backfill data for that day's data timestamp. This operation also has a 10-minute delay before the instance is generated. For more information, see How it works.

For example: If a task is deployed to the production environment at 12:00, the real-time instance becomes effective at 12:10.

If the task's scheduling time is after 12:10, the task will be scheduled for execution.
If the scheduling time of a task is before 12:10, the task performs a dry-run and its instance status is expired instance that is generated in real time.

Scenario 2: Update scheduling cycle

If you update the scheduling properties (such as frequency and time) of a production task and deploy the changes, instances from before and after the change may coexist on the same day. This can lead to complex scheduling dependencies for that day.

Note

This scenario only occurs on the day the task is deployed with immediate generation. On the following day, the task generates auto-triggered instances normally based on the new configuration.

已发布的任务更新调度频率 The specific behavior is as follows:

If the new scheduling time is in the future:

DataWorks replaces the already generated instances for future time slots with new instances based on the latest scheduling configuration.
If the new scheduling time is in the past:

DataWorks retains instances scheduled before the new time and replaces or deletes instances scheduled after it.

Scenario 3: Inconsistent modes

If an ancestor task and its descendant task are both new but have different instance generation modes, this can create an isolated task. For example, if the ancestor task is set to Next Day and the descendant task is set to Immediately After Deployment, the descendant task may become an isolated task. An isolated task does not run automatically. If this task has many descendant dependencies, it can cause severe disruptions to downstream business processes. 实例生成方式不一致

Scenario 4: Change ancestor schedule

If you modify the schedule of an ancestor task that has descendant tasks with different scheduling frequencies, the dependencies for the descendant tasks are adjusted based on the ancestor's latest scheduling configuration (for example, daily, monthly, or hourly).

Note

When you change a production task's schedule, its descendant instances have their dependencies re-established based on the new schedule. This affects both newly generated instances and older, un-replaced instances. For details on hourly and minute-level dependency scenarios, see Principles and samples of scheduling configurations in complex dependency scenarios. This scenario occurs only when a task version that is ready to be deployed has its Instance Generation Mode set to Immediately After Deployment and also includes a change in its scheduling time.

The following are example scenarios:

Scenario 1: The schedule of an ancestor task is changed from every 6 hours to every 8 hours (00:00, 08:00, 16:00), and the Immediately After Deployment mode is selected.
Scenario 2: The schedule of an ancestor task is changed from every 6 hours to 16:00 daily, and the Immediately After Deployment mode is selected.