DataWorks provides the immediate instance generation feature to generate instances 10 minutes after the code is deployed. You can view the node dependencies in Operation Center in real time.

Immediate instance generation rules

  • Node groups do not support immediate instance generation.
  • You can commit and deploy nodes at any time. However, during the time period from 23:30 to 24:00, the immediate instance generation feature does not take effect.
  • If one or more instances have been generated for a node and its instance generation method is changed from Next Day to Immediately After Deployment, instance generation for the node is affected on the current day. Instances that have been run are retained. Instances that have not been run are deleted and replaced by new instances generated immediately after the modified node is deployed.
  • If the instance generation method of the ancestor node is Next Day, descendant nodes whose instance generation method is Immediately After Deployment become isolated and cannot be run as scheduled.
  • If the instance generation method of both the ancestor and descendant nodes is Immediately After Deployment, instances are generated normally for nodes whose scheduled time to run is 10 minutes later than the time when the nodes are deployed. If the scheduled time to run a node is within 10 minutes after the node is deployed, an instance in the Expired state is generated and descendant nodes can be run as scheduled.
For example, a node is scheduled to run at 18:00. You change its scheduled time to 17:00 and the instance generation method to Immediately After Deployment. If you deploy the node before 16:50, an instance is generated for the node. If you deploy the node after 16:50, an instance in the Expired state is generated.View the node information

Instance generation methods

Currently, two instance generation methods are available for a node:
  • Next Day

    If the node is committed and deployed before 23:30, instances are generated from the next day. If the node is committed and deployed between 23:30 and 00:00, instances are generated from the day after tomorrow.

  • Immediately After Deployment
    Instances are generated 10 minutes after the code is deployed.
    Note If the scheduled time to run the node is 10 minutes later than the time when the code is deployed, the node can be run properly.

Create a node that uses the immediate instance generation method

  1. Create a workflow.
    1. Log on to the DataWorks console. In the left-side navigation pane, click Workspaces. On the Workspaces page, find the target workspace and click Data Analytics in the Actions column.
    2. On the Data Analytics tab, move the pointer over the Create icon icon and select Workflow.
    3. In the Create Workflow dialog box that appears, set Workflow Name and Description.
    4. Click Create.
  2. In the left-side navigation pane, click the created workflow, right-click MaxCompute, and then choose Create > ODPS SQL.
  3. In the Create Node dialog box that appears, set Node Name and Location, and then click Commit.
  4. On the node configuration tab that appears, edit the code. Then, click the Properties tab in the right-side navigation pane, and set Start Instantiation to Immediately After Deployment.

Scenarios

The immediate instance generation feature is typically used in the following scenario: The instance generation method of the ancestor node is Next Day, whereas that of descendant nodes is Immediately After Deployment. The following figure shows the dependencies between these nodes.Immediate instance generation
This scenario includes the following situations:
  • When both the ancestor and descendant nodes are newly created on the current day:
    • Descendant node scheduled by day: An instance is generated immediately after the node is deployed. This instance is independent of the ancestor node. If this node is configured to depend on its last-cycle instance, instance generation is not affected on the current day. This node can also depend on the last-cycle existing instance of a custom node.
    • Descendant node scheduled by hour or minute: If this node is configured to depend on its last-cycle instance, all the generated instances except the first one have a dependent last-cycle instance.
    • Descendant node scheduled by week or month: An instance is generated immediately after the node is deployed. This instance is independent of the ancestor node.
      Note In this scenario, instances are generated for the ancestor node from the next day. Instances immediately generated for descendant nodes become isolated. These instances cannot be run on the current day.

      If descendant nodes depend on their own last-cycle instances, instances generated for them on the next day depend on the last-cycle instances of the previous day, which are isolated and not run. Even if instances are generated for the ancestor node on the next day, descendant nodes are still not run.

    Conclusion: If a new node uses the immediate instance generation method and depends on its last-cycle instance, all the generated instances except the first one have a dependent last-cycle instance. However, the node cannot be run as scheduled because all of its instances cannot be run.

  • When an instance has been generated for the ancestor node scheduled by day and descendant nodes that use the immediate instance generation method are created on the current day:
    • Descendant node scheduled by day: An instance is generated and depends on the existing instance of the ancestor node. If this node is configured to depend on its last-cycle instance, instance generation is not affected on the current day.
    • Descendant node scheduled by hour or minute: Instances are generated and depend on the existing instance of the ancestor node. If this node is configured to depend on its last-cycle instance, all the generated instances except the first one have a dependent last-cycle instance.
    • Descendant node scheduled by week or month: An instance is generated and depends on the existing instance of the ancestor node. If this node is configured to depend on its last-cycle instance, instance generation is not affected on the current day. null-null-null

    Conclusion: To run a node that is configured to depend on its last-cycle instance, you must make sure that the instances of the node are run properly on the previous day.

  • When an instance has been generated for the ancestor node scheduled by day and two descendant nodes scheduled by hour are changed to a descendant node scheduled by day that uses the immediate instance generation method:
    • Before the change: All the descendant nodes of the ancestor node scheduled by day are scheduled by hour and use the next-day instance generation method.Next Day
    • Update: Change two descendant nodes scheduled by hour to a descendant node scheduled by day. Change the instance generation method from Next Day to Immediately After Deployment.
    • Instance generation and node dependencies after the change: Assume that the new descendant node scheduled by day is committed and deployed at the time indicated by the dashed line. If the original two descendant nodes scheduled by hour are scheduled to run within 10 minutes after this time, the instances generated for them are deleted. Instead, an instance is generated for the new descendant node scheduled by day. The two descendant nodes of the original two nodes scheduled by hour depend on the new node scheduled by day after the change. If the new node is configured to depend on its last-cycle instance, the instance generated for it depends on the last-cycle instances of the original two nodes scheduled by hour before the time indicated by the dashed line.After the change
    • Current-day instances after the change: Before the new node is deployed, instances are generated for the original two nodes scheduled by hour. After the new node is deployed, an instance is generated for the new node scheduled by day.

    Conclusion: The new node inherits the dependencies of the original nodes. However, instance generation on the current day is affected.