Data Management:Tutorial: Implement automated CI/CD for DMS Airflow tasks based on third-party DevOps pipelines

Step 1: Create a pipeline

1. Log on to the Flow console.

2. On the My Pipelines page, click Create Pipeline in the upper-right corner. For more information, see Create your first pipeline.

3. On the Visualization Orchestration tab, click Empty Template in the navigation pane on the left, select **Empty Template**, and then click Create.

   After the pipeline is created, you are redirected to the pipeline configuration page.

Step 2: Configure a trigger source to trigger pipeline runs

A pipeline can be triggered by sources such as webhooks or GitHub repositories.

Step 3: Configure global environment variables

Global environment variables can be used in all task nodes of the pipeline.

1. On the Flow Edit page, click Variables And Cache.

2. In the String Variable section, click New Variable.

3. Add the following variables and their default values, and then click Add.

   The following table describes the variables.

   Variable Name	Required	Description
   ALIBABA_CLOUD_ACCESS_KEY_ID	Yes	The AccessKey of your Alibaba Cloud account
   ALIBABA_CLOUD_ACCESS_KEY_SECRET	Yes	The AccessKey secret of your Alibaba Cloud account
   DMS_CICD_PROJECT_ID	Yes	The REPOS (code repository) project ID of the workspace
   DMS_CICD_BRANCH_NAME	Yes	Deployment branch
   DMS_CICD_WORKSPACE_ID	Yes	Workspace ID

Step 4: Configure the task flow

1. Delete the empty task node in Stage 1 of the flow.

   On the flow configuration page, select and delete the Empty Task node.

2. Create a new node.

   In the new stage, click New Task and find the Execute Command node.

3. Edit the task execution node.

   Go to the node configuration page to configure the parameters for the task node. You only need to configure the following parameters. You can use the default values for other parameters.

   Parameter	Description
   Task Name	Each task node corresponds to a deployment environment. We recommend that the name of each task node reflect the environment context of the node. For example, dev environment.
   Build Cluster	Keep the default selection (Singapore Build Cluster).
   Download Pipeline	Select Do Not Download Pipeline Sources.
   Task Step	Expand the text box for executing commands and enter the following script. wget 'https://dms-cicd-tool-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/dms_cicd_tool' chmod +x ./dms_cicd_tool ./dms_cicd_tool

4. Configure environment variables for the task node.

   In the Environment Variables section of the node configuration page, select Add > Custom Parameter.

   Variable Name	Meaning	Required	Example
   DMS_CICD_DEPLOY_COMMON	Remarks for the deployment task	No	DEV environment deployment test
   DMS_CICD_DEPLOY_PARAM	Additional parameters for the deployment task	No	{"param1":"value1"}

   Note

   This is a local variable, which means it is effective only in this task node.

5. You can repeat substeps 2, 3, and 4 to create and configure multiple task nodes.

6. In the upper-right corner of the page, click Save and Run.

Step 5: Run the pipeline and view run records

After you configure the trigger source, the task is automatically triggered by the webhook. You can also run the task manually.

• Run the task manually

  In the upper-right corner of the pipeline configuration page, click Save and Run.

• View run records

  On the run history page, click View to see the details of all pipeline runs, including the run duration and logs of task nodes. Click Log in the task node section to view information, such as the run result and the cause of a deployment failure for the destination node.

  Note

  • A success flag for a task node run: XXX success appears in the **Execute Command** section.

  • A failure flag for a task node run: XXX failed appears in the **Execute Command** section.

Step 1: Configure the pipeline script

1. Log on to Jenkins.

   If you have not downloaded Jenkins, go to the official Jenkins website to download and install it.

2. In the navigation pane on the left, click New Item.

3. Enter a task name and click the Pipeline tab.

   Modify the information in the following script, such as `ALIBABA_CLOUD_ACCESS_KEY_ID`, `ALIBABA_CLOUD_ACCESS_KEY_SECRET`, and `Webhook Token`, as needed. Paste the content into the text box and then click Save.

   pipeline { 
       agent any 
       environment { 
           DMS_CICD_PROJECT_ID=xxxx
           DMS_CICD_BRANCH_NAME="master"
           ALIBABA_CLOUD_ACCESS_KEY_ID="xxxxx"
           ALIBABA_CLOUD_ACCESS_KEY_SECRET="xxxxx"
           DMS_CICD_WORKSPACE_ID=xxxx
       }
       stages {
           stage('Build') {        
               steps { 
                   sh 'wget "https://dms-cicd-tool-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/dms_cicd_tool"'
                   sh 'chmod +x ./dms_cicd_tool'
                   echo 'Build step finished' 
               }
           }
           stage('Deploy') {
               steps {
                   sh './dms_cicd_tool'
                   echo 'Deploy finished'
               }
           }
       }
   }

Step 2: Configure a trigger

Subsequent Airflow tasks are automatically triggered. You can also manually run tasks in the Airflow workspace.

1. Make sure that the Generic Webhook Trigger plugin is installed in Jenkins.

   Go to the pipeline project, click Manage Jenkins > Manage Plugins, select Generic Webhook Trigger, and specify the GitLab Token.

   You can obtain the GitLab token from the procedure described in the Associate a GitLab repository section.

2. Configure a GitHub trigger.

   1. In the top menu bar of the destination GitHub repository, click Settings, and then click Webhooks in the navigation pane on the left.

   2. Click Add webhook.

   3. Paste the generic webhook URL of the pipeline into the Payload URL field, enter the token in the Secret field, and select a Trigger Type.

   4. Click Add webhook.

3. Configure a GitLab trigger.

   1. Open the destination GitLab repository and choose Settings > Webhooks.

   2. Paste the generic webhook URL of the pipeline into the Payload URL field, enter the token in the Secret field, and select a Trigger Source.

   3. Click Add Webhook.

View the REPO project ID

Go to the workspace and click the icon in the left-side menu bar. In the REPOS section, click the icon next to the destination repository to view the REPO Project ID.