After you commit and deploy an auto triggered task for which scheduling properties are configured to the scheduling system, DataWorks automatically generates instances for the auto triggered task based on the scheduling properties. When you configure scheduling properties for an auto triggered task, you can set the Instance Generation Mode parameter to Next Day or Immediately After Deployment. If you set the parameter to Next Day, DataWorks schedules the auto triggered task the next day after the task is deployed to the production environment. If you set the parameter to Immediately After Deployment, DataWorks immediately schedules the auto triggered task after the task is deployed to the production environment. This topic describes the rules for immediate instance generation and how to configure the immediate instance generation feature.
Precautions
After you deploy your auto triggered task for which the Instance Generation Mode parameter is set to Immediately After Deployment to the production environment, we recommend that you go to Operation Center to check whether the configuration meets your business requirements.
Time when auto triggered instances take effect
After an auto triggered task is deployed to the production environment, you can view the latest scheduling dependencies of the auto triggered task on the Auto Triggered Nodes page in Operation Center. The time when the auto triggered instances generated for the auto triggered task take effect and the time when the scheduling dependencies of the auto triggered instances are updated vary based on the instance generation mode that you select for the auto triggered task.
Time when a change operation takes effect
If you perform a change operation on an auto triggered task during the period from
23:30 to 24:00, the change takes effect on the instances that are generated for the auto triggered task on the day after the next day.Scenarios in which the immediate instance generation feature can be used
We recommend that you configure the immediate instance generation feature for an auto triggered task only when you want to replace the existing instances generated for the auto triggered task based on the new scheduling properties of the auto triggered task. In other cases, we recommend that you do not configure the immediate instance generation feature for an auto triggered task unless unnecessary.
NoteWhen you change scheduling properties for an auto triggered task that is deployed to the production environment, we recommend that you do not configure the immediate instance generation feature for the task. If you configure the feature for the auto triggered task, the scheduling dependencies of the instances generated for the auto triggered task on the day may be changed or incorrect, or some instances generated for the auto triggered instance on the day are replaced or deleted. As a result, the scheduling dependencies of the instances on the day become complicated.
If you modify the configuration of a deployed auto triggered task and you want the task to be run based on the latest configurations, you can set the Instance Generation Mode parameter to Next Day for the task and backfill data for the task.
If you use the immediate instance generation feature for an auto triggered task, the scheduling dependencies of the instances generated for the task on the next day become normal.
Difference between the time when an auto triggered task is deployed and the time when the immediate instance generation feature takes effect
To prevent instance changes from blocking the running of an auto triggered task for which the immediate instance generation feature is configured, a time difference of 10 minutes exists between the time when the auto triggered task is deployed and the time when the immediate instance generation feature takes effect. DataWorks runs the auto triggered task, generates instances for the auto triggered task based on the latest scheduling properties of the auto triggered task, and schedules the instances 10 minutes after the auto triggered task is deployed.
The time difference between the time when a task is deployed and the time when instances are generated for the task is less than 10 minutes.
Scenario 1: Create an auto triggered task, configure immediate instance generation for the task, and then commit and deploy the task
DataWorks generates a dry-run instance for the auto triggered task. The dry-run instance is an expired instance that is generated in real time.
Scenario 2: Change the scheduling time of a deployed auto triggered task for which the immediate instance generation feature is configured
DataWorks runs the auto triggered task and generates instances for the task based on the original scheduling time of the task.
The time difference between the time when a task is deployed and the time when instances are generated for the task is greater than or equal to 10 minutes.
Scenario 3: Create an auto triggered task, configure immediate instance generation for the task, and then commit and deploy the task
DataWorks runs the auto triggered task and generates instances for the task based on the configurations of the task.
Scenario 4: Change the scheduling time of a deployed auto triggered task for which the immediate instance generation feature is configured
DataWorks generates instances for the auto triggered task based on the new scheduling time of the task and runs the instances.
Effective scope of the immediate instance generation feature
Some changes to the configurations of the auto triggered task may not immediately take effect. For example, if you change the data source that is used by an auto triggered task, configure the immediate instance generation feature for the task, and then deploy the task, the instances generated for the task on the current day are not affected and are still run based on the original data source.
If you want the change to immediately take effect for the auto triggered task, you can backfill data for the task. The data backfill operation is performed based on the latest configurations of the auto triggered task.
Scenarios in which the immediate instance generation feature can be used
The following table describes the use scenarios of the immediate instance generation feature. This feature has different impacts on the running of instances generated for an auto triggered task and on the scheduling dependencies of the instances in different use scenarios.
Scenario | References |
If you configure the immediate instance generation feature for an auto triggered task that is deployed and deploy the auto triggered task again after the configuration, DataWorks generates instances for the auto triggered task on the current day. Whether the generated instances are run depends on the scheduling time of the auto triggered task. | |
If you change the scheduling time and scheduling frequency of an auto triggered task that is deployed, the scheduling dependencies of the instances generated for the auto triggered task on the current day may be complex. We recommend that you do not configure the immediate instance generation feature for a deployed auto triggered task unless unnecessary. | |
If an auto triggered task and its ancestor task are both newly created tasks and have different instance generation modes, the auto triggered task may become an isolated node. | |
If you change the scheduling time of an auto triggered task and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically adjusts scheduling dependencies for the instances generated for the descendant tasks of the auto triggered task based on the latest scheduling properties of the auto triggered task. The scheduling frequency of an auto triggered task can be daily, monthly, or hourly. |
Scenario 1: Configure the immediate instance generation feature for an auto triggered task that is deployed
Scenario 2: Change the scheduling properties of a deployed auto triggered task for which the immediate instance generation feature is configured
If you change the scheduling time and scheduling frequency of an auto triggered task that is deployed to the production environment and configured with the immediate instance generation feature, and deploy the change operation to the production environment, the instances generated for the auto triggered task on the current day before the change and the instances generated for the auto triggered task on the current day after the change may exist at the same time. In this case, the scheduling dependencies of the instances generated for the auto triggered task on the current day may be complex.
The co-existence of the instances generated before and after the change occurs only on the day on which the auto triggered task is deployed. DataWorks generates instances for the auto triggered task the next day based on the latest scheduling properties of the auto triggered task.
Example 1: The scheduling frequency of Auto Triggered Task A is changed from every 6 hours to every day, and the scheduling time of Task A is changed to a point in time that is earlier than the time when Task A is deployed.
The scheduling frequency of Auto Triggered Task A is changed at 09:00 from every 6 hours to 08:00 every day, which is a point in time earlier than the time when Task A is deployed.
Example 2: The scheduling frequency of Auto Triggered Task A is changed from every 6 hours to every day, and the scheduling time of Task A is changed to a point in time that is later than the time when Task A is deployed.
The scheduling frequency of Auto Triggered Task A is changed at 09:00 from every 6 hours to 18:00 every day, which is a point in time later than the time when Task A is deployed.
Generate instances: DataWorks generates the daily-scheduled instance A3 for Auto Triggered Task A after 09:00 based on the latest scheduling configurations of Task A.
Replace the existing instances: DataWorks replaces the hourly-scheduled instances A3 and A4 with the daily-scheduled instance A3.
Retain instances: DataWorks retains hourly-scheduled instances that are generated before 09:10.
If you change the scheduling frequency and scheduling time of an auto triggered task and the new scheduling time of the task is later than the time when the task is deployed, DataWorks generates instances based on the latest scheduling configurations of the task and uses the new instances to replace the existing instances that have not been run.
If you change the scheduling frequency and scheduling time of an auto triggered task and the new scheduling time of the task is earlier than the time when the task is deployed, DataWorks retains the instances generated for the task before the time when the immediate instance generation feature takes effect and replaces or deletes the instances generated for the task after the time when the immediate instance generation feature takes effect.
After you change the scheduling time of an auto triggered task and deploy the change operation to the production environment, whether DataWorks generates instances and runs the instances based on the latest scheduling properties of the auto triggered task depends on the new scheduling time of the auto triggered task and the time when you deploy the change operation. For more information, see the Precautions section in this topic.
Scenario 3: Configure different instance generation modes for an auto triggered task and its ancestor task
If an auto triggered task and its ancestor task are both newly created tasks and have different instance generation modes, the auto triggered task may become an isolated task. For example, if you set the Instance Generation Mode parameter to Next Day for the ancestor task and the Instance Generation Mode parameter to Immediately After Deployment for the auto-triggered task, the auto triggered task may become an isolated task. For more information, see Scenario 2: Node isolation. An isolated task is not automatically scheduled. If multiple tasks depend on the isolated task, your business may be severely affected.
Scenario 4: Change the scheduling time of an ancestor auto triggered task, which affects the scheduling dependencies of descendant instances
If you change the scheduling time of an auto triggered task and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically adjusts scheduling dependencies for the instances that are generated for the descendant task of the auto triggered task based on the latest scheduling properties of the auto triggered task. The scheduling frequency of an auto triggered task can be daily, monthly, or hourly.
If you change the scheduling time of an auto triggered task in the production environment and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically establishes scheduling dependencies for the instances that are generated for the descendant task of the auto triggered task based on the latest scheduling properties of the auto triggered task. The instances include the instances that are generated after the change of the scheduling time and the instances that are generated before the change of the scheduling time but are not replaced. For more information, see Principles and samples of scheduling configurations in complex dependency scenarios. If you change the scheduling time of an auto triggered task that waits to be deployed and is configured with the immediate instance generation feature, the scheduling dependencies of the instances generated for the descendant task of the auto triggered task after the auto triggered task is deployed are affected.
Examples:
Example 1: The scheduling frequency of an ancestor auto triggered task is changed from every 6 hours to every 8 hours, and the immediate instance generation feature is configured for the auto triggered task.
Example 2: The scheduling frequency of an ancestor auto triggered task is changed from every 6 hours to 16:00 every day, and the immediate instance generation feature is configured for the auto triggered task.
