Container Service for Kubernetes:Migrate from Batch Compute to Argo Workflow Cluster

Jobs

A job is a unit of work, such as a shell script, a Linux executable, or a Docker container image, that you submit to the Batch Compute system. The system then allocates computing resources from a compute environment and runs the job.

Array jobs

An array job is a collection of similar jobs submitted and run as a group. All jobs within the same array job share the same job definition but are distinguished by an index. Each job instance processes a different dataset or performs a slightly different task.

Job definitions

A job definition specifies how a job runs. You must create a job definition before you can run a job.

A job definition typically includes the image used to run the job, specific commands and parameters, required CPU and memory resources, environment variables, and disk storage.

Job queues

When you submit a job to the Batch Compute system, it is placed in a specified job queue until it is scheduled to run. You can set priorities for job queues and associate them with specific compute environments.

Compute environment

A compute environment is a set of computing resources that can run jobs. A compute environment requires you to specify the instance type, the minimum and maximum number of vCPUs for the environment, and the price for spot instances.

Templates

A template defines a task (job) and is a building block of a workflow. A workflow must contain at least one template. A template specifies the Kubernetes container to run and its corresponding input and output parameters.

Workflows

A workflow contains one or more tasks and orchestrates them into complex flows, such as running tasks in sequence or in parallel, and executing specific tasks only when certain conditions are met. After you create a workflow, its tasks run as Pods in a Kubernetes cluster.

Workflow templates

A workflow template is a reusable, static definition of a workflow, similar to a function. It can be referenced and run in multiple workflows. You can reuse existing workflow templates when defining complex workflows to reduce repetitive definitions.

Container Argo Workflow Cluster

A Container Argo Workflow Cluster includes a serverless compute environment, which you do not need to create or manage. After you submit a workflow, the system uses Alibaba Cloud Container Compute Service (ACS) to run the workflow tasks in a serverless manner, without requiring you to maintain Kubernetes nodes. By leveraging the elasticity of Alibaba Cloud, you can run large-scale workflows with tens of thousands of task Pods and use hundreds of thousands of vCPUs. Resources are automatically released after the workflow completes. This speeds up workflow execution and reduces computing costs.

Batch Compute

• Users must learn the specifications for Batch Compute job definitions, which can lead to vendor lock-in.

• Users need to manage the compute environment by setting instance types and scale. This is not a serverless approach and results in higher operational costs.

• Due to the scale limitations of the compute environment, you need to manage job queues to set job priorities, which adds complexity.

Argo Workflows

• Built on Kubernetes and open-source Argo Workflows, it orchestrates and runs workflows in a cloud-native way, eliminating the risk of vendor lock-in.

• It supports the orchestration of complex workflow tasks, making it suitable for scenarios like data processing, simulation, and scientific computing.

• The compute environment uses ACS, so you do not need to maintain nodes.

• It provides large-scale computing power on demand with a pay-as-you-go model. This avoids job queuing, improves efficiency, and reduces computing costs.

Feature mapping

Simple workflow

The following example starts a task Pod that uses the alpine image to run the shell command echo helloworld.

You can extend this workflow by specifying and executing multiple shell commands in args, or by using a custom image to run commands defined within it.

cat > helloworld.yaml << EOF
apiVersion: argoproj.io/v1alpha1
kind: Workflow                  # new type of k8s spec
metadata:
  generateName: hello-world-    # name of the workflow spec
spec:
  entrypoint: main         # invoke the main template
  templates:
    - name: main              # name of the template
      container:
        image: mirrors-ssl.aliyuncs.com/alpine:3.18
        command: [ "sh", "-c" ]
        args: [ "echo helloworld" ]
EOF
argo submit helloworld.yaml

Loops in a workflow

In the following example, the pets.input text file and the print-pet.sh script file are contained in the print-pet image. The print-pet.sh script takes job-index as an input parameter and prints the pet from the line in pets.input that corresponds to the job-index value. For the specific file content, visit the GitHub repository.

The workflow starts five Pods simultaneously, passing each a job-index value from 1 to 5. Each Pod prints the corresponding pet based on its job-index value.

You can use loops in a workflow for data sharding and parallel processing, which accelerates the processing of massive datasets. For more Loops examples, see Workflows Loops.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: loops-
spec:
  entrypoint: loop-example
  templates:
  - name: loop-example
    steps:
    - - name: print-pet
        template: print-pet
        arguments:
          parameters:
          - name: job-index
            value: "{{item}}"
        withSequence:  # loop to run print-pet template with parameter job-index 1 ~ 5 respectively.
          start: "1"
          end: "5"
  - name: print-pet
    inputs:
      parameters:
      - name: job-index
    container:
      image: acr-multiple-clusters-registry.cn-hangzhou.cr.aliyuncs.com/ack-multiple-clusters/print-pet
      command: [/tmp/print-pet.sh]
      args: ["{{inputs.parameters.job-index}}"] # input parameter job-index as args of container

DAG workflow (MapReduce)

In real-world batch processing scenarios, multiple jobs often need to work together. This requires defining dependencies between them, and a Directed Acyclic Graph (DAG) is the best way to do this.

However, mainstream Batch Compute systems require you to specify job dependencies by using job IDs. Because a job ID is available only after a job is submitted, you need to write a script to manage dependencies between jobs (as shown in the pseudocode below). When there are many jobs, the dependency relationships are not intuitive and are costly to maintain.

// In a Batch Compute system, Job B depends on Job A and runs after Job A completes.
batch submit JobA | get job-id
batch submit JobB --dependency job-id (JobA)

Argo Workflows allows you to define dependencies between subtasks by using a DAG, as shown in the following example:

• Task B and Task C depend on Task A.

• Task D depends on Task B and Task C.

# The following workflow executes a diamond workflow
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-diamond-
spec:
  entrypoint: diamond
  templates:
  - name: diamond
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        depends: "A"
        template: echo
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        depends: "A"
        template: echo
        arguments:
          parameters: [{name: message, value: C}]
      - name: D
        depends: "B && C"
        template: echo
        arguments:
          parameters: [{name: message, value: D}]
  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: mirrors-ssl.aliyuncs.com/alpine:3.7
      command: [echo, "{{inputs.parameters.message}}"]

A MapReduce workflow example is also provided in the Git repository. It supports sharding data for processing and aggregating the results. For the specific example, see map-reduce.