Data Management:Tutorial: Implement automated CI/CD for DMS Airflow tasks based on third-party DevOps pipelines

Step 1: Create a pipeline

1. Log on to the Flow console.Flow

2. On the My Pipelines page, click Create Pipeline in the upper-right corner. For more information, see Create your first pipeline.

3. On the Visualization Orchestration tab, select Empty Template in the left navigation pane and click Create.

After the pipeline is created, you are redirected to the pipeline configuration page.

Step 2: Configure a trigger source

Choose one of the following methods to trigger pipeline runs automatically when code is pushed.

Step 3: Configure global environment variables

Global environment variables are available across all task nodes in the pipeline.

1. On the Flow Edit page, click Variables And Cache.

2. In the String Variable section, click New Variable.

3. Add the following variables with their values and click Add.

Step 4: Configure the task flow

Each task node in the flow corresponds to a deployment environment (for example, dev or staging).

1. On the flow configuration page, select and delete the Empty Task node in Stage 1.

2. In the new stage, click New Task and select the Execute Command node.

3. On the node configuration page, set the following parameters. Leave others at their default values. In the task step text box, enter the following script. This script downloads and runs dms_cicd_tool, which calls the DMS API to deploy the configured branch to your workspace.

   Parameter	Value
   Task name	A name reflecting the environment context, for example, dev environment
   Build cluster	Keep the default: Singapore Build Cluster
   Download pipeline	Select Do Not Download Pipeline Sources
   Task step	Enter the script below

   wget 'https://dms-cicd-tool-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/dms_cicd_tool'
   chmod +x ./dms_cicd_tool
   ./dms_cicd_tool

4. In the Environment Variables section, click Add > Custom Parameter to configure the following optional node-level variables.

   These are local variables — they apply only to this task node.

   Variable name	Description	Required	Example
   DMS_CICD_DEPLOY_COMMON	Remarks for the deployment task	No	DEV environment deployment test
   DMS_CICD_DEPLOY_PARAM	Additional parameters for the deployment task	No	{"param1":"value1"}

5. To deploy to multiple environments, repeat steps 2–4 to create additional task nodes.

6. Click Save and Run in the upper-right corner.

Step 5: Run the pipeline and view run history

Once the trigger source is configured, the pipeline runs automatically on each code push. To run it manually, click Save and Run on the pipeline configuration page.

To view run history, go to the run history page and click View to see details for each run, including duration and task node logs. In the task node section, click Log to see the run result and any deployment failure details.

In the Execute Command section of a task node log:

• A successful run shows: XXX success

• A failed run shows: XXX failed

Step 1: Configure the pipeline script

1. Log on to Jenkins. If Jenkins is not installed, download it from the official Jenkins website.

2. In the left navigation pane, click New Item.Save and Run

3. Enter a task name and select the Pipeline tab.

4. Replace the placeholder values in the script below with your actual credentials and IDs. Paste the script into the text box and click Save.

   Important

   Store sensitive credentials such as ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET as Jenkins credentials rather than entering them as plaintext in the pipeline script.

pipeline {
    agent any
    environment {
        DMS_CICD_PROJECT_ID=<your-project-id>
        DMS_CICD_BRANCH_NAME="master"
        ALIBABA_CLOUD_ACCESS_KEY_ID="<your-access-key-id>"
        ALIBABA_CLOUD_ACCESS_KEY_SECRET="<your-access-key-secret>"
        DMS_CICD_WORKSPACE_ID=<your-workspace-id>
    }
    stages {
        stage('Build') {
            steps {
                sh 'wget "https://dms-cicd-tool-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/dms_cicd_tool"'
                sh 'chmod +x ./dms_cicd_tool'
                echo 'Build step finished'
            }
        }
        stage('Deploy') {
            steps {
                sh './dms_cicd_tool'
                echo 'Deploy finished'
            }
        }
    }
}

Replace the following placeholders:

Step 2: Configure a trigger

After the trigger is configured, Airflow tasks run automatically on each code push. You can also run tasks manually in the Airflow workspace.

Install the required plugin

In the pipeline project, go to Manage Jenkins > Manage Plugins, install the Generic Webhook Trigger plugin, and set the GitLab token. Obtain the token from the steps in Associate a GitLab repository.

Configure a GitHub trigger

1. In your GitHub repository, go to Settings > Webhooks and click Add webhook.

2. Paste the generic webhook URL into the Payload URL field, enter the token in the Secret field, and select a Trigger Type.

3. Click Add webhook.

Configure a GitLab trigger

1. In your GitLab repository, go to Settings > Webhooks.

2. Paste the generic webhook URL into the Payload URL field, enter the token in the Secret field, and select a Trigger Source.

3. Click Add Webhook.

View the REPO project ID

Go to the workspace and click the icon in the left-side menu bar. In the REPOS section, click the icon next to the destination repository to view the REPO Project ID.