Git code synchronization and merge

Step 1: Prepare resources and network

Code synchronization tasks run on a serverless resource group and require network access to your Git repository and OSS.

1. Prepare a serverless resource group and configure the network.

   • Prepare a DataWorks serverless resource group and bind it to the target workspace.

   • Ensure that the resource group can access the SSH port of the Git server. Only port 22 is supported.

     • Public Git repository (for example, Alibaba Cloud DevOps CodeUp or GitHub.com): Configure a NAT Gateway and an EIP for the resource group's VPC to enable public network access.

     • Private Git repository: Ensure that the resource group and the Git server can communicate over the private network (VPC). For more information, see VPC peering connection.

2. Prepare an OSS data source. Code Sync uses OSS for temporary storage.

   • In the same region as your DataWorks workspace, create an OSS bucket.

   • On the Data Source Management page of your DataWorks workspace, create an OSS data source that points to the bucket.

     Note

     Ensure that the access credentials used for the OSS data source have the following permissions: oss:GetObject, oss:ListObjects, oss:PutObject, and oss:DeleteObject.

Step 2: Prepare the Git environment

Create a dedicated account, configure an SSH key, and set protected branch rules in your Git platform.

1. Create a dedicated Git account and generate an SSH key.

   1. In your Git platform, such as Alibaba Cloud DevOps or GitLab, create a dedicated account for automatic code commits, such as dataworks_pusher, and configure a valid email address for it.

   2. Generate an SSH key pair for this account using the RSA algorithm. Do not set a passphrase when prompted.

      • Purpose: Generate an SSH public key and private key for authentication.

      • Command:

        # Replace "your_user_email@example.com" with the email address of the account that you configured in the previous step.
        ssh-keygen -t rsa -C "your_user_email@example.com"
        
        # When prompted for a passphrase, press Enter to skip.
        # Enter passphrase (empty for no passphrase): [Enter]
        # Enter same passphrase again: [Enter]

      • Result: The command creates the id_rsa (private key) and id_rsa.pub (public key) files in the ~/.ssh/ directory. You will use the content of these files in subsequent steps.

        # Copy and save the content of the public and private keys for use in subsequent steps.
        cat ~/.ssh/id_rsa.pub
        cat ~/.ssh/id_rsa

2. Add the public key to the Git platform.

   Log on to the Git platform. On the SSH key management page for the dataworks_pusher account, add the content of the public key (id_rsa.pub file) that you generated in the previous step.

3. Create a repository and configure protected branch rules.

   1. Create a Git repository dedicated to DataWorks code management, for example, DataWorks_code.

   2. In the repository, set up protected branch rules to prevent direct modification of the branches automatically managed by DataWorks. A repository administrator must perform this operation.

      Important

      If protected branch rules are misconfigured, the branches automatically managed by DataWorks can be modified unexpectedly, which disrupts synchronization.

      • Branch name pattern: dataworks_*

      • Allowed to push: Select specific members and add the dataworks_pusher account.

      • Allowed to merge: Select No one.

Step 3: Configure and initialize Code Sync

Step 4: Verify code synchronization

1. Create a Shell node in DataStudio and name it shell_test.

2. In the code editor, enter the following code and click the Save button in the toolbar.

   echo 'Code push test.'

3. Log on to Git, go to the target repository, and select the save branch. Find the shell_test folder. You should see three files in the folder: shell_test.sh (code file), shell_test.spec.json (scheduling configuration file), and dataworks.properties (variable file). Click shell_test.sh to view the shell_test node and its code content. This indicates that the synchronization is successful.

4. To verify synchronization for the other two branches, publish the node to the corresponding environments. For more information about the publish operation, see Publish nodes within the same workspace.

Reverse merge

DataWorks supports synchronizing code to Git and provides a powerful reverse merge feature.

This capability is compatible with both the main synchronization branch and any feature branch created from it. Developers can follow standard development practices on independent branches for development, testing, and code review, and then safely integrate the mature code back into DataWorks. This process enables efficient team collaboration.

1. Initiate a merge

   On the DataStudio > Code Management page, expand the Code Merge section.

2. Preview the merge

   In the Branch field, enter the name of the branch to be merged, for example, feature_cn-shanghai_branch, and click Merge Preview. The system compares the source branch with the default DataWorks save branch to identify added, changed, and deleted content.

   • No conflicts: The interface displays a diff of the changes, clearly listing the nodes and code modifications to be added, modified, or deleted.

   • Conflicts: The interface directly reports the conflicts and displays their specific details. In this case, you must return to your local environment, manually resolve the conflicts in Git, and then retry the merge.

     

3. Confirm the merge

   1. After you confirm that the preview is correct, click Confirm merge.

   2. The system starts the merge task. You can view the merge progress in real time.

   3. All historical merge records are displayed in the Code Merge History area below. You can view the owner, merge status, and branch details at any time.

      

Cross-workspace merge

The Git synchronization feature supports cloning DataWorks projects across workspaces and even across regions. This design allows you to reuse a standardized set of code as a template and distribute it to multiple business scenarios. For example, you can deploy a universal user analysis model to separate workspaces for different business lines and run it on their respective dedicated computing resources and data sources.

1. Initial setup

   1. Create two projects: git_cross_project_1 and git_cross_project_2. They can be in different regions.

   2. Follow the instructions in Step 1 to configure a resource group and an OSS data source for each workspace. You can use the same data source for both. Then, configure the network connections.

   3. Follow the instructions in Step 2 to prepare a single Git repository that will serve as the shared repository for both workspaces.

   4. Follow the instructions in Step 3 to configure and initialize Git synchronization for both workspaces. Use the same SSH address and private key for both configurations. Ensure that initialization is successful. After initialization, branches for both workspaces (for example, 270256 for project1 and 270257 for project2) appear in the Git repository.

2. Cross-project merge strategy

   1. When you merge code branches across projects, only the node code, basic attributes, and scheduling configurations are merged. Runtime configurations are not merged into the target workspace.

   2. Because the two workspaces may use different computing resources, resource groups, and data sources, you must configure a merge_mapping resource mapping file.

      1. On your local machine, open a terminal, clone the save branch of the source workspace to your local machine, and then switch to the save branch of project1 in the source workspace, for example, dataworks_cn_shenzhen_270256_save.

         # Clone the remote Git repository to your local machine.
         git clone git@your_git_server_domain:64dc86a16800a4a57137536/cross_project_shenzhen.git
         
         # Switch to the save branch of project1.
         git checkout <your branch name>

      2. Create the merge_mapping mapping file.

         # cross_project_shenzhen is the repository name.
         cd cross_project_shenzhen 
         
         # Create the directory.
         mkdir -p DATAWORKS_SYSTEM_CONFIG/merge_mapping
         
         # Create the mapping file. The file name should be in the format, for example, cn_shenzhen_270256_to_cn_shenzhen_270257.properties
         vi DATAWORKS_SYSTEM_CONFIG/merge_mapping/<region>_<projectId>_to_<region>_<projectId>.properties

         Configure the file content based on your environment and modify the parameter values on both sides of the = sign. You can specify multiple parameters for each type.

         # Data source
         # spec.datasource.name.<data_source_name_of_project1>=<data_source_name_of_project2>
         spec.datasource.name.mysql_01=mysql_02
         
         # Resource group
         # spec.runtimeResource.resourceGroup.<resource_group_ID_of_project1>=<resource_group_ID_of_project2>
         spec.runtimeResource.resourceGroup.group_524257424564736=Serverless_res_group_524257424564736_764027070300961
         
         # Node output name prefix
         # spec.output-prefix.<name_of_project1>=<name_of_project2>
         spec.output-prefix.git_cross_project_1=git_cross_project_2
         
         # Project prefix for tables in MaxCompute SQL
         # script.project-identifier.<name_of_project1>=<name_of_project2>
         script.project-identifier.git_cross_project_1=git_cross_project_2
         
         # Image
         # spec.script.runtime.container.imageId.<image_ID_used_in_project1>=<image_ID_used_in_project2>
         spec.script.runtime.container.imageId.Default=System_python311_ubuntu2204_20251201
         
         # RAM role
         # spec.script.runtime.linkedRoleArn.<RAM_role_ARN_used_in_project1>=<RAM_role_ARN_used_in_project2>
         spec.script.runtime.linkedRoleArn.acs:ram::1107550004253538:role/aliyundataworksaccessingenirole=acs:ram::1107550004253538:role/aliyundataworksaccessingossrole

         Push the code to the Git repository.

         # Add the changes in the current directory to the staging area.
         git add .
         # Commit the mapping file.
         git commit -m "add mapping files"
         # Push the branch content to the remote Git repository.
         git push

   3. Merge code across projects

      1. Go to DataStudio in the target workspace git_cross_project_2, click Code Management in the left-side navigation pane, and then go to the Code Merge tab.

      2. In the branch input field, enter the name of the save branch of the source workspace, for example, dataworks_cn_shenzhen_270256_save.

         

      3. Click Merge Preview. After you confirm that the preview is correct, merge the code from workspace git_cross_project_1 into workspace git_cross_project_2. For more information about the merge operation, see Reverse merge.