causes and solutions for mismatches between input/output and data lineage - DataWorks

Symptom

When you commit a node, the system warns that the configured input and output do not match the data lineage. In the Commit New Version dialog box that appears, a pink warning area displays a more specific message, such as "Input and data lineage do not match." This indicates that the input you submitted is workshop0423.ods_log_info_d, but the input derived from data lineage is workshop0423.ods_log_info_d;workshop0423.ods_user_info_d_1. You must select the I confirm that I want to continue with the commit checkbox to enable the Confirm button. In this scenario, on the DataWorks development page, the ODPS SQL code on the left uses INSERT OVERWRITE TABLE dw_user_info_all_d to join the ods_log_info_d and ods_user_info_d_1 tables and write the result to the destination table. On the right, the scheduling configuration panel has two sections: Parent Nodes and Outputs. Under Parent Nodes, the system automatically parses the input table workshop0423.ods_log_info_d. The Outputs section includes the system-default output workshop0423.500889455_out and the manually added output table workshop0423.dw_user_info_all_d. This mismatch occurs because the tables parsed from the data lineage conflict with the dependencies you declared in the scheduling configuration panel.

Possible causes

This warning appears when the tables specified in your code's SELECT statements do not match the Parent Nodes configuration, or when the tables in your INSERT or CREATE statements do not match the Outputs configuration.

For example:

The node code you submitted selects data from table2, but table2 is not configured as an ancestor node for this node.
You have configured doc_test in the node's Outputs, but the code does not write to it.

Solution

For tables that are not generated periodically, you can ignore the warning and commit the node.
DataWorks scheduling dependencies are primarily designed to ensure that data in periodically updated tables is available for scheduled nodes. The platform cannot monitor tables that are not updated by its own scheduling service. If your node's code selects data from a table that is not generated periodically, you should remove the automatically parsed upstream dependency from the Parent Nodes configuration. Tables that are not generated periodically include:
- Tables uploaded to DataWorks from a local machine
- Dimension tables
- Tables not generated by a DataWorks auto triggered node
- Tables generated by a manually triggered node
For tables that are generated periodically, you must carefully check that the data lineage and scheduling dependencies are consistent.
If you force a commit without this check, you may encounter the following issues:
- A descendant node might fail to get the latest data. For example, your code selects from Table A, which is generated daily by a scheduled auto triggered node. If you do not add the node that generates Table A to your current node's Parent Nodes to establish a scheduling dependency, and the node that generates Table A fails, your descendant node uses stale data from the previous successful run, which can cause inconsistencies.
- You might encounter a "Dependent ancestor output name not found" error. For example, your code creates or inserts data into Table B, but you do not configure Table B as an output in the Outputs section. If another node selects from Table B, the system automatically parses this relationship and adds Table B as an input for that node, creating a dependency. However, the system cannot trace this dependency back to the node that produces Table B. When you try to commit the node, it fails with the error "Node commit error: Dependent ancestor output name not found". For more information, see Node commit error: Dependent ancestor output name not found.