Data Management:Set up and manage Airflow environments

Prerequisites

Prepare the required instance resources as described in Preparations.

Billing

Airflow instance fees are based on the Workflow Specifications (number of CUs). Unit prices are shown on the Configure Resources page.

Step 1: Create an Airflow instance

1. Log in to DMS 5.0.

2. Go to the workspace page.

   DMS provides two methods to access the workspace page. You can choose either method.

   Method 1

   In the upper-left corner of the console, click the icon and choose All Features > Data+AI > workspace.

   Note

   If you are not using the simplified console, choose Data+AI > workspace from the top menu bar.

   Method 2

   Important

   This method is in beta testing and is available only to some users.

   In the navigation pane on the left, click the Data Factory icon, and then click workspace.

   Note

   If you are not using the simplified console, choose Data Factory > workspace from the top menu bar.

3. Click the name of your target workspace or create a new workspace to open the workspace page.

4. In the navigation pane on the left of the workspace, choose > Airflow instance, and then click Create instance.

5. Configure the instance parameters.

   The following table describes the key parameters.

   Parameter	Description
   Workflow specification	Select a specification based on the scale and complexity of your workflows. For more information, see Appendix: Airflow specifications.
   Worker node extension	Airflow automatically scales the number of worker nodes based on the task load. The number of worker nodes can range from 1 to 10.
   VPC ID	No changes are required. By default, this is the same as the VPC of the workspace.
   VSwitch	Select the target VSwitch.
   Security group	Select the security group that controls the workflow.
   OSS Bucket	Select an OSS Bucket in the same region as the workspace.
   OSS Path	Enter the data storage path that you created during Preparations.

6. Click OK.

   The instance is successfully deployed when its status changes to Running.

Step 2: Add a linked account

1. In the upper-right corner of the workspace, click your profile picture, and then click the icon to add a new linked account.

   This icon is a + symbol located next to linked account in the drop-down panel.

2. In the New Service Provider Account dialog box, select an account Type.

   DMS currently supports three account types: GitHub, Alibaba Cloud Codeup, and Private GitLab.

3. Select a Creation method and enter the Username, Password, or access token.

   The Creation method supports importing the account by using a username and password or an access token.

   .

4. Click OK.

Step 3: Create a code repository

1. In the navigation pane on the left of the workspace, click the icon to go to the Explorer page.

2. In the CODE (code repository) section, click the icon and select Add existing git repository.

3. Enter a Project Name, select the corresponding Git provider and Git repository URL, and then click OK.

   If you use Alibaba Cloud services, we recommend that you select Codeup for Git provider. DMS then uses the Codeup-linked account by default.

   The repository is successfully created when its name appears in the repository list.

Step 4: Develop code

1. To the right of the target code repository name, click the current branch name, such as master. You can then switch branches, create a new branch, edit code, or save your changes.

   Note

   The save action is equivalent to running the git push command.

2. Confirm the environment settings.

   Hover over the repository name, click the icon, and then confirm the environment and parameter settings.

3. Hover over the repository name and click the button.

4. In the dialog box that appears, click OK to deploy the code.

   The dialog box states that deployment can take up to 10 minutes and that any uncommitted changes in the current branch are automatically committed and pushed.

Step 5: View published tasks in Airflow

1. In the navigation pane on the left of the workspace, click the icon.

2. Click the Airflow instance under the target repository to view the published tasks.

   You can click a DAG name to view its execution results.

   The Airflow DAGs page lists the published DAGs, such as hello_world_dag2. The list includes fields such as Owner, Runs (successes and failures), Schedule (for example, @daily), Last Run, and Next Run.

Appendix: Airflow specifications