Migrate task orchestration to DMS Airflow - Data Management

The legacy task orchestration feature in DMS is being deprecated. You can use Data Agent to migrate your existing task orchestrations to DMS Airflow. This topic describes the prerequisites, billing, and step-by-step migration procedure.

Prerequisites

If you have not purchased Data Agent, you receive a free daily quota of 30 minutes with a single seat. If this quota is insufficient, submit a ticket to add your account to the whitelist for free access. The whitelisted seat is automatically removed after one month. If the migration is not complete within this period, submit another ticket to extend the access.
In the region where you plan to create the DMS Airflow instance, create a VPC, a vSwitch, and an OSS bucket. The OSS bucket stores Airflow DAG code and log data.
Note
The Airflow instance and the scheduled tasks to migrate can be in different regions. For example, you can migrate scheduled tasks from the China (Beijing) region to an Airflow instance in the China (Hangzhou) region.

Billing

Billable item	Description
Data Agent	Data Agent is a paid product. Free-tier users receive a daily quota of 30 minutes. If this quota is insufficient, submit a ticket to request free access until the migration is complete.
DMS Airflow	The cost of an Airflow instance depends on its workflow specifications, measured in compute units (CUs). For pricing details, check the resource configuration page of the workspace that hosts the instance.
OSS bucket	Storing DAG code and log data in an OSS bucket incurs storage fees. For more information, see Billing overview.
AnalyticDB (optional)	If your tasks include offline data integration or cross-database Spark tasks, AnalyticDB for MySQL is required. For AnalyticDB pricing, see AnalyticDB for MySQL billable items.

Step 1: Create an Airflow instance

(Recommended) Create using Data Agent

Open the Data Agent page and click Datapaw in the left navigation pane to switch to development mode.
Note
Free-tier users receive a daily quota of 30 minutes. If this quota is insufficient, request additional access as described in Prerequisites.

Send the following message to Data Agent:

Use skills airflow-operation to create an Airflow instance. Instance name: xxx, region: xxx, vpc: xxx, vSwitch: xxx, security group: xxx, oss bucket: xxx, logs directory: airflow-logs, instance specification: SMALL

Note

Replace xxx with your actual VPC, vSwitch, security group, and OSS bucket information.

Wait for the Airflow instance to be created.

Create manually

Automatic creation through Data Agent is recommended. To create an Airflow instance manually, perform the following steps in order:

(Optional) Create a workspace

Note

If you already have a workspace in DMS, skip this step and proceed to Configure workspace resources.

Log on to Data Management DMS 5.0.
Click the icon in the upper-left corner of the console, and then click Workspace in the Data+AI section.
Note
If prompted for RAM authorization, follow the on-screen instructions. If no prompt appears, authorization has already been granted.
On the page that appears, click New Workspace, configure the following parameters, and then click Confirm.
1. Workspace Name: Enter a name for the workspace.
2. Workspace Description (Optional): Add a description for the workspace.
3. Region: Select the same region as the VPC, vSwitch, and OSS bucket that you prepared. This region is where the Airflow instance is hosted.
4. Owner: Select an administrator for the workspace. You can also change the roles of other members.
5. VPC ID: Select the VPC that you prepared.

Configure workspace resources

Log on to Data Management DMS 5.0.
Click the icon in the upper-left corner of the console, and then click Workspace in the Data+AI section.
Note
If prompted for RAM authorization, follow the on-screen instructions. If no prompt appears, authorization has already been granted.
In the workspace list, click the name of an existing workspace or the one you just created.
In the lower-left corner of the workspace page, choose > Workspace Management.
On the page that opens, click the Resource Configuration tab.
In the Workspace Resource Configuration section, click Resource Configuration.
In the Resource Configuration panel, specify the CU Configuration. Set the value to 200 or higher to prevent resource shortages when creating the Airflow instance.
Click Create. On the payment page, click Pay to complete the purchase. The initial payment for resource configuration is zero. Subsequent charges are based on actual CU usage.

Create an Airflow instance

In the left-side menu of the configured workspace page, choose > Airflow Instances, and then click Create Instance.

Configure the following instance parameters. Keep the default values for any parameters not listed.

Parameter	Description
Instance Name	Specify a name for the Airflow instance.
Workflow Specifications	Select the specifications based on your workflow scale and complexity. For general workflows, the default specification is sufficient. For more information, see Airflow specifications.
Worker Node Scaling	Airflow automatically adjusts the number of worker nodes based on task load. For general workflows, the default setting is sufficient. Worker nodes can range from 1 to 10.
VPC ID	No adjustment is needed. This value defaults to the VPC of the workspace.
vSwitch	Select the vSwitch that you created earlier.
Security Group	Select a basic security group. For more information, see Basic and advanced security groups.
Log Storage	Select the OSS bucket that you created earlier and specify a folder for storing Airflow logs.
DAG Directory	Keep the default directory.

Click Create. The initial creation takes about 20 minutes.
After the instance is created, find it on the Airflow Instances page and click Open in the Actions column.
Note
If the browser fails to open the page, click the icon to open it in a new tab.

Step 2: Convert task flow code

Use Data Agent to convert task orchestration flows into Airflow DAG code. Airflow DAGs support both Python and YAML formats, but the conversion tool currently generates Python code only.

Note

A Data Agent session expires after 6 hours. If your session has expired, click New Task in the left navigation pane to start a new session.

Initiate the conversion task

Open the Data Agent page and click Datapaw in the left navigation pane to switch to development mode.
Send a message to Data Agent to start the conversion. The following are examples. You can adjust the message to suit your needs, such as specifying a few task flows for initial testing.
- To migrate task orchestration flows:
```
Use skills airflow-migration-from-dms to migrate all task flows. Region: cn-hangzhou
```
- To migrate data warehouse development flows:
```
Use skills airflow-migration-from-dms to migrate all task flows in the data warehouse development project <project_name>. Region: cn-hangzhou
```
  Note
  Replace <project_name> with the name of your data warehouse development project.
Note
Specify the region where your DMS is deployed. For example, if DMS is deployed in the Singapore region, specify ap-southeast-1.
Wait for Data Agent to complete the migration.

Check the conversion results

After the migration is complete, Data Agent generates a migration report named migration_report.md. Review the report to confirm the results.
In Data Agent, enter post-migration considerations or What do I need to do after migration?. Data Agent analyzes your specific situation and provides guidance.

Upload the Airflow task flow

In Data Agent, send the following message to synchronize DAG files to OSS:
```
Use skills airflow-operation to upload and publish the dags directory to workspace {workspace_id/name}
```
Note
Replace {workspace_id/name} with your actual workspace ID or name.
After the upload is complete, Airflow loads the new DAGs within approximately 5 minutes.

Step 3: Configure and run the Airflow task flow

After the migration, review the notes in the migration_report.md file. You can also send What else do I need to do to run the DAGs? in Data Agent to view your to-do items.

Configure Airflow variables

After the migration, configure dependencies for offline data integration and cross-database Spark SQL tasks, such as AnalyticDB cluster credentials and database account credentials, by using Airflow variables.

Open the airflow_variables.json file generated by the migration and fill in the required variable values. The following parameters must be specified:
- spark_adb_cluster_id and spark_adb_resource_group: The ID and resource group of the AnalyticDB cluster. If all database instances involved in offline data integration and cross-database Spark SQL nodes are in the same region, they can share a single AnalyticDB cluster. The migration script assumes a single cluster. Adjust the configuration manually if different clusters are used.
- spark_eni_vswitch_id and spark_eni_security_group_id: The vSwitch ID and security group ID within the VPC where the database instances are located. All database instances in a single AnalyticDB Spark task must be in the same VPC and region. If they are in different VPCs, use VPC Peering Connection or a similar method to establish network connectivity first. The migration script assumes all instances are in the same VPC. Adjust the configuration manually if they are not.
- db_xxx_user and db_xxx_password: The username and password of the database used in AnalyticDB Spark tasks. For security reasons, AnalyticDB Spark tasks cannot use the credentials managed by DMS. Configure these credentials in Airflow instead. The migration script automatically populates the instances that require configuration into the airflow_variables.json file.
airflow_variables.json example:
In the left-side menu of the Airflow page, choose Admin > Variables, and then click Import Variables.
On the Import Variables page, upload the airflow_variables.json file, select Overwrite in the Select Variable Conflict Resolution section, and click Import.

Note

After import, variables of the password type are automatically encrypted.

(Optional) Configure an ECS connection

If your task flow includes ECS-related nodes, configure an SSH connection.

Upload the SSH private key of the ECS instance to the OSS mount directory. If SSH access is not configured on the ECS instance, configure it first.
In the left-side menu of the Airflow page, choose Admin > Connections, and then click Add a new record.
On the Add Connection page, configure the following parameters, and then click Save.
- In the Standard Fields section, configure the Host field with the IP address of the ECS instance. If the ECS instance and Airflow are in the same VPC, use the internal IP address. Otherwise, use a public IP address or establish VPC connectivity first.
- In the Additional Fields JSON section, specify the key_file path, which is the OSS mount directory path within Airflow.

Run the Airflow task flow

In the left-side menu of the Airflow page, click Dags. Select the configured task flow and turn on the scheduling toggle to enable scheduled runs, or click the run button to trigger a manual test run.

Additional notes

Supported node types: In Data Agent, send skills airflow-migration-from-dms which DMS task orchestration nodes are supported for conversion to Airflow to view the supported node types.
Unsupported scenarios: The conversion tool does not cover all scenarios because some task types are deprecated or differ significantly in implementation. Use Data Agent to resolve any issues, or report them to the DMS team for improvement.
Instance login expired: If the error "SQL execution failed, reason: instance login status expired" occurs during a task run, the database instance credentials are not managed by DMS. Re-edit the instance and configure managed credentials.
Variable feature conversion: Issues may arise with the variable feature during task orchestration migration. Follow the prompts from Data Agent to make adjustments.
Task flow status requirement: Only published task flows can be migrated.
Chinese language support: Airflow automatically translates Chinese task names to English during migration. You can adjust the translation logic as needed.