Alibaba Cloud provides a component compatible with Argo Workflows, a cloud-native workflow engine, to simplify batch task orchestration for batch data processing, ML pipelines, infrastructure automation, and CI/CD workloads.
Introduction to Argo Workflows
Argo Workflows is a powerful, cloud-native workflow engine designed to define, manage, and schedule complex workflows in Kubernetes. A workflow consists of multiple tasks with dependencies, and this flexibility simplifies task configuration.
Scenarios
Argo Workflows supports scenarios such as batch data processing, machine learning pipelines, infrastructure automation, and CI/CD. It is widely used in industries such as autonomous driving, scientific computing, quantitative finance, and digital media.
Batch data processing: Common scenarios include large-scale high-precision map processing, quantitative finance backtesting and simulation, parallel audio and video processing, and animation rendering.
Scientific computing: Common scenarios include complex scientific computing simulation, drug discovery and training, gene sequencing, mutation comparison and detection, and energy exploration.
Simulation: Common scenarios include autonomous driving algorithm simulation, molecular dynamics simulation, astronomical data simulation, and financial modeling.
Machine learning pipeline: Common scenarios include machine learning data pre-processing, distributed training, Large Language Model (LLM) parameter tuning, and model evaluation and deployment.
Infrastructure automation: Common scenarios include automated cloud resource management, resource backup and recovery, node pool migration, and cluster migration and upgrades.
CI/CD: Common scenarios include parallel CI pipelines, multi-stage build and testing, cross-cloud application deployment, and approval flow integration.
Advantages of Argo Workflows
Cloud-native: Designed specifically for Kubernetes. Each task is a pod, which takes full advantage of the lightweight and flexible nature of containers.
Lightweight and scalable: Argo Workflows is lightweight and has no extra overhead or limitations compared to traditional virtual machines (VMs). Using the scheduling capabilities of Kubernetes, you can run thousands of tasks in parallel to improve processing efficiency.
Flexible orchestration: The flexible combination of Directed Acyclic Graphs (DAGs) and steps supports custom workflow logic of any complexity. Powerful retry and caching mechanisms improve the success rate of workflow execution.
Rich ecosystem: Supports the orchestration of various task types, such as Spark, Ray, and TensorFlow Job. Combined with event-driven features, you can build a fully automated task processing platform.
Use Argo Workflows
ACK Argo Workflows is compatible with and enhances the open source version. You can migrate existing Argo workflows without modification. Benefits over the community version:
Elastic auto-scaling to optimize computing costs.
Multi-zone load balancing and reliable scheduling for high availability.
Enhanced control plane with improved scalability, performance, efficiency, stability, and observability.
Enhanced OSS storage management that supports large file uploads, artifact garbage collection (GC), and streaming.
Expert support from container service specialists to optimize workflows, improve performance, and reduce costs.
ACK Argo Workflows supports two deployment methods:
Serverless Argo Workflows: For fully managed, high-performance workflows, create a dedicated workflow cluster. Serverless Argo Workflows.
Argo Workflows component on ACK: If you have an existing ACK cluster, install the Argo Workflows component to orchestrate workflows with your cluster resources. This topic covers this deployment method.
After you install the component, use the Alibaba Cloud Argo CLI or the Argo console to submit and manage workflows.
The following table outlines the workflow by role.
Process | Description |
1. Preparations |
|
2. Set up the environment |
|
3. Manage workflows | (Data engineer) Use the Argo CLI or Argo console to submit and manage orchestrated tasks.
|
(Cluster administrator)
|
Billing
Batch task orchestration is free. You pay only for underlying resources such as ACK computing power and other cloud products. Argo Server automatically creates a pay-as-you-go SLB instance, which incurs fees. Billing overview of SLB.
Contact us
For questions or suggestions, join the DingTalk group (ID: 35688562).