All Products
Search
Document Center

DataWorks:Configure immediate instance generation for a task

Last Updated:Aug 30, 2023

After you commit and deploy an auto triggered task for which scheduling properties are configured to the scheduling system, DataWorks automatically generates instances for the auto triggered task based on the scheduling properties. When you configure scheduling properties for an auto triggered task, you can set the Instance Generation Mode parameter to Next Day or Immediately After Deployment. If you set the parameter to Next Day, DataWorks schedules the auto triggered task the next day after the task is deployed to the production environment. If you set the parameter to Immediately After Deployment, DataWorks immediately schedules the auto triggered task after the task is deployed to the production environment. This topic describes the rules for immediate instance generation and how to configure the immediate instance generation feature.

Precautions

  • Time when auto triggered instances take effect

    After an auto triggered task is deployed to the production environment, you can view the latest scheduling dependencies of the auto triggered task on the Cycle Task page in Operation Center. The time when the auto triggered instances generated for the auto triggered task take effect and the time when the scheduling dependencies of the auto triggered instances are updated vary based on the instance generation mode that you select for the auto triggered task.

  • Time when a change operation takes effect

    If you perform a change operation on an auto triggered task during the period from 23:30 to 24:00, the change takes effect on the instances that are generated for the auto triggered task on the day after the next day.

  • Scenarios in which the immediate instance generation feature can be used

    If you configure the immediate instance generation feature for an auto triggered task and deploy the task for multiple times on a single day, the scheduling dependencies of the instances generated for the auto triggered task on the day may be complex. This is because specific instances generated for the auto triggered task on the day need to be replaced. DataWorks establishes scheduling dependencies for the instances based on the latest scheduling properties of the task the next day. We recommend that you configure the immediate instance generation feature for an auto triggered task only when you want to replace the existing instances generated for the auto triggered task based on the new scheduling properties of the auto triggered task.

    If you cannot determine whether the scheduling dependencies that will be established for the instances of an auto triggered task after the configuration meet your business requirements, you can refer to Principles and samples of scheduling configurations in complex dependency scenarios to learn the principles of configuring scheduling dependencies between tasks whose scheduling cycles are different in advance. After you configure the immediate instance generation feature for the auto triggered task, you can preview the scheduling dependencies that will be established for the instances of the task. If the scheduling dependencies meet your business requirements, you can commit and deploy the auto triggered task.

    If you set the Instance Generation Mode parameter to Next Day for an auto triggered task but you need to immediately run the auto triggered task based on the new scheduling properties of the auto triggered task, you can backfill data whose data timestamp is the previous day for the auto triggered task. The effect of performing the data backfill operation is equivalent to running the instances generated for the auto triggered task on the current day.

  • Difference between the time when an auto triggered task is deployed and the time when the immediate instance generation feature takes effect

    To prevent instance changes from blocking the running of an auto triggered task for which the immediate instance generation feature is configured, a time difference of 10 minutes exists between the time when the auto triggered task is deployed and the time when the immediate instance generation feature takes effect. DataWorks runs the auto triggered task, generates instances for the auto triggered task based on the latest scheduling properties of the auto triggered task, and schedules the instances 10 minutes after the auto triggered task is deployed.时间差

Overview

The following table describes the use scenarios of the immediate instance generation feature. This feature has different impacts on the running of instances generated for an auto triggered task and on the scheduling dependencies of the instances in different use scenarios.

Scenario

References

If you configure the immediate instance generation feature for an auto triggered task that is deployed and deploy the auto triggered task again after the configuration, DataWorks generates instances for the auto triggered task on the current day. Whether the generated instances are run depends on the scheduling time of the auto triggered task.

Scenario 1: Configure the immediate instance generation feature for an auto triggered task that is deployed

If you change the scheduling time and scheduling frequency of an auto triggered task that is deployed, the scheduling dependencies of the instances generated for the auto triggered task on the current day may be complex. We recommend that you do not configure the immediate instance generation feature for a deployed auto triggered task unless unnecessary.

Scenario 2: Change the scheduling properties of a deployed auto triggered task for which the immediate instance generation feature is configured

If an auto triggered task and its ancestor task are both newly created tasks and have different instance generation modes, the auto triggered task may become an isolated task.

Scenario 3: Configure different instance generation modes for an auto triggered task and its ancestor task

If you change the scheduling time of an auto triggered task and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically adjusts scheduling dependencies for the instances generated for the descendant tasks of the auto triggered task based on the latest scheduling properties of the auto triggered task. The scheduling frequency of an auto triggered task can be daily, monthly, or hourly.

Scenario 4: Change the scheduling time of an ancestor auto triggered task, which affects the scheduling dependencies of descendant instances

Scenario 1: Configure the immediate instance generation feature for an auto triggered task that is deployed

If you configure the immediate instance generation feature for an auto triggered task that is deployed and deploy the auto triggered task to the production environment again after the configuration, the running of the instances generated for the auto triggered task depends on the scheduling time of the auto triggered task.

Scheduling time

Description

The scheduling time of the auto triggered task is later than the time when the auto triggered task is deployed.

DataWorks generates instances for the auto triggered task and schedules the instances based on the scheduling time of the auto triggered task.

If the scheduling time of the auto triggered task is later than the time when the auto triggered task is deployed, only instances generated for the auto triggered task on the current day are affected. Whether the existing instances of the auto triggered task are replaced varies based on whether the scheduling time of the task is 10 minutes later than the time when the task is deployed. If the scheduling time of the task is 10 minutes later than the time when the task is deployed, the existing instances are replaced. For more information, see the "Precautions" section in this topic.

The scheduling time of the auto triggered task is earlier than the time when the auto triggered task is deployed.

DataWorks generates a dry-run instance for the auto triggered task. The dry-run instance is an expired instance that is generated in real time.

If you want the auto triggered task to generate data on the current day, you can backfill data whose data timestamp is the previous day for the auto triggered task. A time difference of 10 minutes exists between the time when the data backfill operation is performed and the time when a data backfill instance is generated for the auto triggered task. For more information, see the "Precautions" section in this topic.

Example: If you configure the immediate instance generation feature for an auto triggered task and deploy the auto triggered task to the production environment at 12:00, the immediate instance generation feature takes effect at 12:10 for the auto triggered task.

  • If the scheduling time of the auto triggered task is later than 12:10, DataWorks runs the auto triggered task and generates instances for the auto triggered task as scheduled.

  • If the scheduling time of the auto triggered task is earlier than 12:10, DataWorks generates a dry-run instance for the auto triggered task. The dry-run instance is an expired instance that is generated in real time.

Scenario 2: Change the scheduling properties of a deployed auto triggered task for which the immediate instance generation feature is configured

If you change the scheduling time and scheduling frequency of an auto triggered task that is deployed to the production environment and configured with the immediate instance generation feature, and deploy the change operation to the production environment, the instances generated for the auto triggered task on the current day before the change and the instances generated for the auto triggered task on the current day after the change may exist at the same time. In this case, the scheduling dependencies of the instances generated for the auto triggered task on the current day may be complex.

Note

The co-existence of the instances generated before and after the change occurs only on the day on which the auto triggered task is deployed. DataWorks generates instances for the auto triggered task the next day based on the latest scheduling properties of the auto triggered task.

已发布的任务更新调度频率Impacts on the instances generated for the auto triggered task vary based on the new scheduling time of the auto triggered task:

  • The new scheduling time of the auto triggered task is later than the time when the auto triggered task is deployed.

    DataWorks generates instances based on the latest scheduling properties of the auto triggered task and use the new instances to replace the existing instances.

  • The new scheduling time of the auto triggered task is earlier than the time when the auto triggered task is deployed.

    DataWorks retains the instances that are generated before the scheduling time and replaces or deletes the instances that are generated after the scheduling time.

Note

After you change the scheduling time of an auto triggered task and deploy the change operation to the production environment, whether DataWorks generates instances and runs the instances based on the latest scheduling properties of the auto triggered task depends on the new scheduling time of the auto triggered task and the time when you deploy the change operation. For more information, see the "Precautions" section in this topic.

Scenario 3: Configure different instance generation modes for an auto triggered task and its ancestor task

If an auto triggered task and its ancestor task are both newly created tasks and have different instance generation modes, the auto triggered task may become an isolated task. For example, if you set the Instance Generation Mode parameter to Next Day for the ancestor task and the Instance Generation Mode parameter to Immediately After Deployment for the auto triggered task, the auto triggered task may become an isolated task. An isolated task is not automatically scheduled. If multiple tasks depend on the isolated task, your business may be severely affected.实例生成方式不一致

Scenario 4: Change the scheduling time of an ancestor auto triggered task, which affects the scheduling dependencies of descendant instances

If you change the scheduling time of an auto triggered task and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically adjusts scheduling dependencies for the instances that are generated for the descendant task of the auto triggered task based on the latest scheduling properties of the auto triggered task. The scheduling frequency of an auto triggered task can be daily, monthly, or hourly.

Note

If you change the scheduling time of an auto triggered task in the production environment and configure the immediate instance generation feature for the auto triggered task, DataWorks automatically establishes scheduling dependencies for the instances that are generated for the descendant task of the auto triggered task based on the latest scheduling properties of the auto triggered task. The instances include the instances that are generated after the change of the scheduling time and the instances that are generated before the change of the scheduling time but are not replaced. For more information, see Principles and samples of scheduling configurations in complex dependency scenarios. If you change the scheduling time of an auto triggered task that waits to be deployed and is configured with the immediate instance generation feature, the scheduling dependencies of the instances generated for the descendant task of the auto triggered task after the auto triggered task is deployed are affected.

Examples:

  • Example 1: The scheduling frequency of an ancestor auto triggered task is changed from every 6 hours to every 8 hours, and the immediate instance generation feature is configured for the auto triggered task.场景一

  • Example 2: The scheduling frequency of an ancestor auto triggered task is changed from every 6 hours to 16:00 every day, and the immediate instance generation feature is configured for the auto triggered task.场景二