Dataphin:Import deployment package

Prerequisites

• The current publisher has the role of cross-tenant publisher and has permission for the object type to be published. For more information, see Publishing object permission description.

• Before importing the deployment package file, you must enable cross-tenant publishing mode. For more information, see cross-tenant publishing settings.

• The cross-tenant publisher in the target environment has obtained the deployment package file exported from the source environment. For more information, see Export deployment package.

• Before importing the deployment package file, you must fill in the cross-tenant publishing credential information in the target environment. For more information, see cross-tenant publishing settings.

Permission description

Supports cross-tenant publishers to import deployment package files.

Limits

If the deployment package is exported in Dataphin 3.11 or earlier versions and imported into the target environment, the manually configured data lineage in the task cannot be recognized. To export the data lineage information configured for the task, upgrade to Dataphin 3.12 or later versions, re-export the deployment package, and then import it into the target environment to recognize the manually configured data lineage information.

Import validation items

When importing a deployment package file, the system will validate the deployment package file to be imported. The validation priorities and descriptions are shown in the following table.

The successfully imported deployment package file will automatically enter the list of objects to be published, and the system will automatically recognize and mark its change type. For more information, see Object to be published.

Procedure

1. Log on to Dataphin using the cross-tenant publisher account.

2. In the top menu bar on the Dataphin home page, select Management Center > Migration.

3. In the left-side navigation pane, select Migration > Import Deployment Package.

4. On the Import Deployment Package page, click Import Deployment Package.

5. On the Import Deployment Package page, configure the following parameters.

   Parameter	Description
   Source Of Deployment File	The system supports selecting Local File and OSS storage files. The OSS name is consistent with the display name set in the cross-tenant publishing settings. Note If you need to import the deployment file from OSS, you need to enable OSS storage in the cross-tenant publishing settings. For more information, see cross-tenant publishing settings.
   Deployment File	When Source Of Deployment File is selected as Local File, click icon to select the downloaded deployment package file. You can also drag the downloaded deployment package file to the deployment file display box. When Source Of Deployment File is selected as OSS, click icon, and select the file in the OSS storage directory in the Select Deployment File Dialog Box. Important The file name of the uploaded deployment package only supports 0-9, a-z, A-Z, Chinese, and special characters - _ . *.
   Import File Description	Fill in the import file description.
   Import Validation	No configuration is required. The system displays the file name, generation time, exception check items, and check results. For more information about the check items, see import validation items.

6. Click Upload Deployment Package to upload the deployment package file.

7. On the deployment package import settings tab, configure Import Policy and Replacement Rules.

   • Import policy

     Parameter	Description
     Global general
     New Object Owner	Do not change: When publishing objects, if the owner of the object exported from the source environment exists in the current target environment, the owner will be used. If not, you can choose to use the operator of this publication or specify a user as the new object owner. This is applicable to scenarios where the members of the source environment and the target environment are basically the same. Important For objects in the project, if the owner is not in the project to which the object belongs, the publication may fail. Unified modification to: When the deployment package is imported into the target environment, regardless of whether the object exported from the source environment has an owner, it will be uniformly modified to the operator of this publication or the specified user according to the configuration.
     Development
     Development Object	Supports publishing to the development environment or production environment. Publish to development (submit): Required. Objects in Dev-Prod mode projects are submitted to the development environment, and objects in Basic mode are submitted to the production environment (published within the same tenant). Publish to production: Optional. Objects in Dev-Prod mode projects are published to the production environment (published within the same tenant). If submission to the development environment fails, no publication will be made.
     Tag
     Tag Object	By default, Publish To Project is not supported for modification.
     Standard The data standard feature must be enabled.
     New And Changed Standards	For new or changed data standard objects, you can choose Publish To Draft Or In Revision and Submit For Online Application. Publish To Draft Or In Revision: Required. If the change type of the data standard object is new, a draft standard will be generated. If the change type of the data standard object is update, different operations will be performed based on the status of the standard in the target environment. If the object in the target environment is in draft or in revision status, the existing standard will be updated. If the object in the target environment is in published status (effective/pending/expired), a standard in revision status will be created based on the content imported this time. Submit For Online Application: Optional. For data standards that are successfully published to draft or in revision status, the submit online operation is automatically performed, and the corresponding approval task is generated based on the standard approval configuration in the cross-tenant publishing settings. Note If an approval task needs to be generated, the failure to generate the task will also cause the publication to fail. Only standards in production status support the configuration of mapping relationships. It is recommended that you first publish the imported data standards in the target environment and then import the mapping relationships.
     Mapping Rule	For the effective status of the mapping rules imported into the target environment, you can choose Retain The Current Environment Effective Status And Only Update The Rule Configuration (based on the effective status of the target environment) or Overwrite And Update The Current Environment Rule Configuration And Effective Status (based on the effective status of the source environment).
     Mapping	For the imported mapping relationships in the current target environment, you can choose the handling strategy of overwrite or append. Overwrite: Clears the current target environment mapping relationships first and adds the source environment mapping relationships; applicable to scenarios where both environments are completely consistent. If the mapping rules are executed, the mapping results may be updated. Append: Retains the existing mapping relationships in the target environment and adds new mapping relationships, which may cause the mapping relationships deleted in the source environment to not be deleted in the target environment, leading to inconsistencies between the two environments. Applicable to incremental update scenarios.
     Effective Mapping Conflict Handling	When the mapping relationship is append, configuration is supported. For the scenario where the imported effective mapping relationship is an invalid mapping in the current target environment, you can choose to Set "invalid Mapping" To "effective Mapping" or Retain "invalid Mapping" And Skip Without Updating. Set "invalid Mapping" To "effective Mapping": Sets the invalid mapping relationship in the target environment to an effective mapping. If the invalid mapping relationship in the target environment fails to be deleted, it cannot be published; applicable to scenarios where the source environment configuration is the standard. Retain "invalid Mapping" And Skip Without Updating: Retains the invalid mapping in the target environment, applicable to scenarios where the target environment configuration is the standard.
     Invalid Mapping Conflict Handling	When the mapping relationship is append, configuration is supported. For the scenario where the imported invalid mapping relationship is an effective mapping in the current target environment, you can choose to Set "effective Mapping" To "invalid Mapping" or Retain "effective Mapping" And Skip Without Updating. Set "effective Mapping" To "invalid Mapping": Sets the effective mapping relationship in the target environment to an invalid mapping. If the effective mapping relationship in the target environment fails to be deleted, it cannot be published; applicable to scenarios where the source environment configuration is the standard. Retain "effective Mapping" And Skip Without Updating: Retains the effective mapping in the target environment, applicable to scenarios where the target environment configuration is the standard.
     Root Name Conflict Handling	The root name is used as the unique identifier of the object. For the handling strategy of the imported root name conflict in the current target environment, you can choose Overwrite If Duplicated or Skip If Duplicated. Overwrite If Duplicated: Overwrites the root name inthe target environment with the root name from the source environment. Skip If Duplicated: Retains the root name in the target environment and skips without updating.
     Quality The data quality feature must be enabled.
     Monitored Object Content Import	Supports append and overwrite strategies. Append: Only publishes new and changed rules and schedules (automatically renames in case of name conflicts), but does not publish deletion types of rules and schedules; and does not overwrite archived tables (if configured) and view permission settings; applicable to incremental update scenarios. Overwrite: Clears the existing rules and schedules of the monitored objects in the current target environment first and adds the rules and schedules from the source environment; also overwrites archived tables and view permission settings; applicable to scenarios where both environments need to be completely consistent.
     Effective Status Settings	Supports retain the current environment and overwrite the current environment. Retain the current environment: Retains the effective status of the monitored objects and quality rules in the current target environment. Overwrite the current environment: Uses the effective status of the monitored objects and quality rules in the deployment package to overwrite the current target environment.
     Alert Import	Supports retain the current environment and overwrite the current environment. Retain the current environment: Retains the alert recipients and shift schedules in the current target environment, applicable to scenarios where the current target environment is configured separately. Overwrite the current environment: Overwrites the alert recipients and shift schedules in the current target environment, applicable to scenarios where both environments need to be completely consistent.
     Dependent Data Does Not Exist	When the data that the imported object depends on does not exist, you can choose to force publish or publish with errors. Force publish: Ignores dependencies and directly publishes to the current target environment. Subsequent editing or resolving dependency issues is required for normal validation. Publish with errors: When the dependent object does not exist, the monitored object publication will fail, and the dependency issue needs to be resolved before it can be published.
     Security The data security feature must be enabled.
     Add New Key	For keys with the change type of new, the key value supports selecting Import Original Value, System Automatically Generates, or System-generated Keys Automatically Refresh, Manually Generated Keys Need To Be Manually Updated After Import. Import Original Value: Imports the key value from the source environment to the current target environment for direct use; applicable to scenarios where the source and target environments have consistent users. System Automatically Generates: Automatically regenerates the key values for all new keys. System-generated Keys Automatically Refresh, Manually Generated Keys Need To Be Manually Updated After Import: Automatically refreshes the key values generated by the system, manually generated keys need to be manually set after import.
     Existing Key	For keys that already exist in the target environment, the key value in the current target environment is used, and it is skipped without updating.
     Classification Result	For the classification results imported into the target environment, you can filter them by selecting Import Only Manually Specified Results or Import All Classification Results. Import Only Manually Specified Results: Imports the manually specified classification results from the source environment, including classification results uploaded in bulk via Excel and manually added classification results. Import All Classification Results: Supports importing all classification results exported from the source environment.
     Manual Classification Result Conflict	For scenarios where both the source environment and the target environment have effective manual classification results, and the manual classification results of the source environment conflict with those of the current target environment, you can choose Retain The Current Environment or Overwrite The Current Environment. Retain The Current Environment: Ignores the manual classification results in the source environment and skips without updating. Overwrite The Current Environment: Deletes the conflicting manual classification results in the target environment first, and then adds a manual classification result that is the same as the manual classification result in the source environment. Note For scenarios where the source environment has effective manual classification results and the target environment has effective automatic classification results, the classification results from the source environment will be imported into the target environment and unified arbitration will be performed to determine the final effective result.
     Automatic Classification Result	For scenarios where both the source environment and the target environment have effective automatic classification records, and the automatic classification results of the source environment conflict with those of the target environment, you can choose Append, Overwrite, or Overwrite And Lock. Append: Adds the automatic classification results from the source environment to the target environment as a classification record. Unified arbitration will be performed during import to generate the final classification result. In this mode, the final classification results of the source environment and the target environment may be inconsistent. Overwrite: Deletes all classification results in the target environment first, and then adds an automatic classification result that is the same as the automatic classification result in the source environment, but does not lock the result; only ensures that the classification results of the source environment and the target environment are consistent during import. The result may be updated during the next automatic classification. Overwrite And Lock: Deletes all classification results in the target environment first, and then adds an automatic classification result that is the same as the automatic classification result in the source environment, and locks the result; the locked result cannot be changed by subsequent automatic classification results generated in the target environment. Note For scenarios where the source environment has effective automatic classification results and the target environment has effective manual classification results, the classification results from the source environment will be imported into the target environment and unified arbitration will be performed to determine the final effective result. The automatic detection results apply to the source environment, while the manual detection results apply to the target environment. After importing the results from the source environment to the target environment, the final effective results are determined through unified arbitration.

   • Replacement rules

     • Data source

       The attribute configurations of the source environment and the target environment are generally different. Therefore, before publishing, you can use the data source replacement rules function to set replacement rules and batch replace the data source configurations with the target environment configurations. When the data source object is published, the first matching rule will be used in sequence to replace the attributes.

       1. Click +new Replacement Rule to add a new replacement rule.

          Note

          • Only up to 100 replacement rules can be set.

          • The scope of the replacement rules: only for New and Changed types of data sources.

       2. In the Set Replacement Rules dialog box, configure the replacement rules.

          Configuration steps	Description
          1. Select the scope of the replacement data source	Set the data source type. Select the data source type whose object attributes you need to replace. For example, MaxCompute. Add selection rules. Click +add Rule and configure your replacement rules in the newly added rule item. Replacement rules support AND operations. Configure the selection scope. Set rule configuration items. Configuration items vary based on the data source type and support different configuration item types. Please refer to the actual operation. For example, MaxCompute supports Datasource Name, Owner, Endpoint, Project Name, and Access ID for range selection. Set matching conditions. The matching conditions supported by each configuration item include: Exact Match, Not Match, Contains, Does Not Contain, Starts With, Ends With, Is Empty, Is Not Empty, Blank Text, Non-blank Text. Note Owner only supports Belongs To.
          2. Set the replacement configuration items and replacement values	Configure the replacement configuration items for the production environment and the development environment. Click New Configuration Item Replacement Rule. Configure the replacement rules in the newly added configuration item. Replacement configuration items vary depending on the data source type and support different types of replacement configuration items. Please refer to the actual operation for details. For example, MaxCompute supports Endpoint, Project Name, Access ID, Access key. Set matching rules. The matching methods support full text, text matching, and regular expressions. Case sensitive. Note Regular expressions use Java language specifications. For more information, see Java regular expression tutorial. Set replacement text. Enter the text to be used for replacement in the input box.

       3. Click Save to complete the data source replacement rule configuration.

          After completing the data source replacement rule configuration, when publishing the data source object, you can view the details of the data source matched by the rule and perform mapping operations for data sources with the same name and type. For more information, see mapping existing data sources.

          Important

          The set rules will be replaced in top-down order.

     • Schedule resource group

       The schedule resource group list will display the projects included in the deployment package (excluding projects with a change type of delete). You can set the resource group after the project is imported in the schedule resource group list.

       Note

       • Only Dataphin instances deployed based on the latest architecture support schedule resource group replacement.

       • If the project where the task is located in the deployment package does not exist in the deployment package, it will also be displayed. Usually, the schedule resource group set for the task is not included in the target project, and you need to specify the replacement rules.

       • If the project in the deployment package is the tenant's default resource group, it cannot be changed. After import, the resource group defaults to the tenant's default resource group.

     • Offline physical table

       When publishing an offline physical table, you can replace the table DDL, such as the location URL of the foreign table. The system automatically copies all replacement rules from the most recent deployment package import (regardless of the effective status). The copied rules are set to disabled by default.

       1. Click +new Replacement Rule to open the New Replacement Rule dialog box.

       2. In the New Replacement Rule dialog box, Select The Scope Of The Replacement Offline Physical Table and Set The Replacement Configuration Items And Replacement Values.

          • Select the scope of the replacement offline physical table (optional)

            1. Click +add Rule and select Configuration Item. You can choose Project Name or Table Name.

            2. Select operator. If the configuration item is project name, the operator only supports Belongs To; if the configuration item is table name, the operator can be Starts With, Ends With, or Regex Match.

            3. Enter or select a value. If the configuration item is project name, you can select one or more values; if the configuration item is table name, you need to manually enter the value.

            Note

            • If no scope rule is set, it will default to all offline physical tables.

            • Multiple rules are in an AND relationship and cannot be modified.

            • If the offline physical table in the imported deployment package matches the combined rules, it will be replaced according to the replacement rules.

          • Set the replacement configuration items and replacement values

            1. Click New Configuration Item Replacement Rule to create a blank rule.

            2. Configure the rule's Replacement Configuration Item. You can choose Entire DDL Statement or Location URL.

               • Entire DDL statement: The replacement source is the entire change DDL of the offline physical table.

               • Location URL: The replacement source is the location clause in the offline physical table DDL.

                 Note

                 • In create external table table_name (...) location '{hdfs://...}' , {hdfs://...} is the location URL.

                 • In alter table table_name set location '{hdfs://...}' , {hdfs://...} is the location URL.

            3. Configure Matching Rules. Choose Text Matching or Regular Expression and enter the corresponding text or regular expression.

            4. Enter New Configuration Item. Enter the corresponding replacement value based on the matching rules. The maximum length of the input content is 512 characters.

               When the matching rule is regular expression, the replacement value supports regular replacement expressions (back-references are supported).

       3. Click Preview Replacement Effect. In the Preview Replacement Effect dialog box, enter the original DDL, and then click Replace. The system will automatically execute the replacement rules and generate the replaced DDL.

       Note

       Each imported deployment package will import all offline physical table replacement rules from the most recent deployment package (regardless of the effective status). After import, the effective status defaults to disabled.

8. Single click Save or Save And Publish to start publishing the object.

What to do next

After importing the deployment package into the target environment, you can view the objects to be published and the publishing status on the deployment package overview page, and manage the objects to be published. For more information, see Object to be published.