Airflow is an open-source tool for orchestrating and scheduling workflows in big data development scenarios. It supports job development, directed acyclic graph (DAG) scheduling, and monitoring workflow batch processing. Airflow lets you define workflows using Python code. Its Python plugins enable flexible integration with most external technologies or systems.
Scenarios
Orchestrating data development tasks in DMS, such as executing SQL scripts and data cleansing.
Scheduling Spark in AnalyticDB for MySQL to execute batch processing tasks.
Scheduling data integration tasks in DTS.
Supported regions and zones
Region | Zone | ||
Name | ID | Name | ID |
China (Hangzhou) | cn-hangzhou | Zone B | cn-hangzhou-b |
Zone E | cn-hangzhou-e | ||
Zone F | cn-hangzhou-f | ||
Zone G | cn-hangzhou-g | ||
Zone H (Recommended) | cn-hangzhou-h | ||
Zone I (Recommended) | cn-hangzhou-i | ||
Zone J (Recommended) | cn-hangzhou-j | ||
Zone K | cn-hangzhou-k | ||
China (Shanghai) | cn-shanghai | Zone A | cn-shanghai-a |
Zone B | cn-shanghai-b | ||
Zone C | cn-shanghai-c | ||
Zone D | cn-shanghai-d | ||
Zone E | cn-shanghai-e | ||
Zone F | cn-shanghai-f | ||
Zone G | cn-shanghai-g | ||
Zone K | cn-shanghai-k | ||
Zone L (Recommended) | cn-shanghai-l | ||
Zone M (Recommended) | cn-shanghai-m | ||
Zone N (Recommended) | cn-shanghai-n | ||
China (Beijing) | cn-beijing | Zone C | cn-beijing-c |
Zone D | cn-beijing-d | ||
Zone E | cn-beijing-e | ||
Zone F | cn-beijing-f | ||
Zone G | cn-beijing-g | ||
Zone H (Recommended) | cn-beijing-h | ||
Zone I (Recommended) | cn-beijing-i | ||
Zone J (Recommended) | cn-beijing-j | ||
Zone K | cn-beijing-k | ||
Zone L | cn-beijing-l | ||
China (Shenzhen) | cn-shenzhen | Zone A | cn-shenzhen-a |
Zone B | cn-shenzhen-b | ||
Zone C | cn-shenzhen-c | ||
Zone D (Recommended) | cn-shenzhen-d | ||
Zone E (Recommended) | cn-shenzhen-e | ||
Zone F (Recommended) | cn-shenzhen-f | ||
Singapore | ap-southeast-1 | Zone A | ap-southeast-1a |
Zone B | ap-southeast-1b | ||
Zone C | ap-southeast-1c | ||
Indonesia (Jakarta) | ap-southeast-5 | Zone A | ap-southeast-5a |
Zone B | ap-southeast-5b | ||
Zone C | ap-southeast-5c | ||
Billing
The fees for an Airflow instance are based on its Workflow Specifications (the number of CUs used). The unit price is displayed on the Configure Resources page.
Solution overview
Prepare resources for Airflow.
Prepare OSS resources and enable Internet access for Airflow.
A linked Git account is a dedicated account for performing operations in the code repository. These operations are not visible to other users in the same workspace, such as the Alibaba Cloud account or RAM users.
Create a code repository in Git or another versioning system to store and manage Airflow DAG files.
Edit and publish code in the code repository.
View Airflow execution status.
View DAG scheduling and monitor workflow batch processing in the Airflow workspace.