Dataphin:Configure custom lineage

Limitations

• You can configure custom lineage only for non-SQL compute tasks.

• A maximum of 20 lineage groups are supported. Each group allows up to 5 inputs and 5 outputs.

• Dataset lineage for Python or Shell tasks requires the unstructured data feature to be enabled.

Lineage configuration

• Lineage group and lineage: A task can have multiple independent lineage groups. Within a group, every input maps to every output, generating both table-level and field-level lineage.

  For example, if you select field a from table A for Input 1, field b from table B for Input 2, and fields c and d from table C for the output, this generates the following lineage:

  • Table-level lineage: table A-->table C; table B-->table C.

  • Field-level lineage: table A.a-->table C.c; table B.b-->table C.c; table A.a-->table C.d; table B.b-->table C.d.

  You can configure lineage between tables and datasets.

  • Lineage between a table (full table) and a file dataset or a hybrid dataset.

  • Lineage between a table (specified fields) and a hybrid dataset (specified fields).

  • Lineage between a table (full table or specified fields) and another table (full table or specified fields).

• You can specify the environment for input and output tables. When set to Auto, Dataphin replaces environment variables during submission and publishing to generate lineage for both development and production environments.

• Dataphin processes custom lineage when you submit or publish a task. Resubmitting or republishing updates the lineage.

Configure lineage

1. In the top navigation bar of the Dataphin homepage, choose Develop > Data Development.

2. On the Develop page, select a project from the top menu bar. In Dev-Prod mode, also select an environment.

3. In the left-side navigation pane, choose Data Processing > Compute Task.

4. Click the target compute task to open it.

5. In the right pane, click Lineage to open the Lineage Configuration panel.

6. In the Lineage Configuration panel, click + Add Lineage Group to configure input and output objects.

   • Configure input objects

     1. In the lineage group area, click Configure Input Object. In the Configure Input Object dialog box, configure the parameters. Available parameters vary by object type.

        Table

        Parameter	Description
        Environment	Valid values: Auto, Development, and Production. Auto: Tables are selected from the development environment via a project variable. On submission, the variable resolves to the development table; on publishing, it resolves to the production table. If no production table exists, submission and publishing still succeed, but production lineage cannot be parsed. Development or Production: No variable replacement. The configuration is used as-is.
        Object type	Select Table.
        Input table	Supported table types: physical tables, physical views, dimension logical tables, fact logical tables, summary logical tables, tag logical tables, and logical views.
        Selected scope	Entire table: Generates lineage based on all fields at the time of submission and publishing. Note Lineage updates only on successful submission and publishing. Schema changes to input or output tables do not trigger automatic updates. Specified fields: Select fields from the table as needed.

        Dataset

        Parameter	Description
        Environment	Valid values: Auto, Development, and Production. Auto: Datasets are selected from the development environment via a project variable. On submission, the variable resolves to the development dataset; on publishing, it resolves to the production dataset. If no production dataset exists, submission and publishing still succeed, but production lineage cannot be parsed. Development or Production: No variable replacement. The configuration is used as-is.
        Object type	Select Dataset.
        Input dataset	Only datasets added to the current compute task properties are available. Search by keyword to find a dataset.
        Selected scope	Entire dataset: For a file dataset, only dataset-level lineage is generated. For a hybrid dataset, lineage is generated based on all fields in the metadata table at submission and publishing. Specify Fields: You can select all fields in the selected dataset. Field-level lineage is supported only for hybrid datasets. Note When a file dataset is an input or output, only overall lineage between the file dataset and the associated object is created, regardless of field selections. Field-level lineage is not supported between a file dataset and another dataset.

     2. Click OK to save the input object configuration.

   • Configure output objects

     1. In the lineage group area, click Configure Output Object. In the Configure Output Object dialog box, configure the parameters. The output object parameters are the same as the input object parameters described in the tables above.

     2. Click OK to save the output object configuration.

7. After you configure all lineage groups, click OK in the Lineage Configuration panel to save.

Submit and publish the lineage

1. Click Submit above the code editor of the compute task.

2. In the Submission Details dialog box, click Confirm and Submit.

3. The Object Check step in the Submission Details dialog box validates that all objects and fields referenced in the lineage configuration exist.

   Note

   • Object checks run only during submission, not during publication.

   • Submission parses lineage for the development environment; publishing parses lineage for the production environment. The maximum is 100,000 lineage relationships per submission or publication. Exceeding this limit prevents lineage from being recorded in the asset catalog.

View the lineage

After the task is submitted and published, view the custom table-level and field-level lineage on the Physical Table Details page.