Container Service for Kubernetes:Use Argo Workflows to orchestrate genomics computing workflows

Background Information

Genomics computing workflows

Genomics computing workflows are used for data analytics in genomics research. A genomic computing workflow consists of a collection of interrelated computing tasks and data processing steps that are executed in a specific order. Genomics computing workflows involve complex steps such as data processing, sequence alignment, mutation detection, gene expression analysis, and phylogenetic tree building.

Argo Workflows for genomics computing workflow orchestration

Argo Workflows is an open source Kubernetes-native workflow engine that facilitates flexible and efficient orchestration of workflows in containerized environments. Argo Workflows is particularly suitable for genomics computing due to the following advantages:

• Containerization and environment consistency: Genomic analysis needs to use various software tools and dependency libraries. Argo Workflows can encapsulate all analysis steps in Docker containers, which can be deployed across platforms and environments. This ensures cross-platform consistency and reusability of genomic analysis tasks.

• Flexible orchestration: Most genomics workflows involve multiple steps, conditional branches, and parallel jobs. Argo Workflows supports complex execution logic and conditions, allowing you to customize workflows in a simplified and explicit manner.

However, Argo Workflows also faces the following challenges:

• Large-scale O&M: When genomics computing workflows consist of a large number of tasks, optimization and maintenance policies of the cluster cannot be efficiently implemented, especially when users have limited experience in cluster maintenance.

• Complex workflow orchestration: Scientific experiments involve a large parameter space and a large number of steps. A scientific experiment may need to run tens of thousands of jobs. In this case, open source workflow engines are incapable of orchestrating workflows for large-scale scientific experiments.

• Cost optimization and resource elasticity: Genomic analysis consumes a large amount of compute resources. Users want to enable intelligent resource scheduling for workloads to maximize resource utilization. In addition, users want to automate resource scaling based on the resource demand of workloads. Open source solutions may fail to meet the preceding requirements.

To tackle the preceding challenges in large-scale O&M, complex workflow orchestration, cost optimization, and resource elasticity, Distributed Cloud Container Platform for Kubernetes (ACK One) provides Kubernetes clusters for distributed Argo workflows.

Kubernetes clusters for distributed Argo workflows

Kubernetes clusters for distributed Argo workflows (workflow clusters) are deployed on top of a serverless architecture. This type of cluster runs Argo workflows on elastic container instances and optimizes cluster parameters to schedule large-scale workflows with efficiency, elasticity, and cost-effectiveness. Workflow clusters can run workflows concurrrently, loop workflows, or retry workflows. These workflow execution policies are suitable for genomics computing and complex workflow orchestration.

Use Argo Workflows to orchestrate genomics computing workflows

This section provides an example on how to use Argo Workflows to configure and run a sequence alignment workflow with the Burrows-Wheeler Alignment algorithm (BWA).

1. Create a Kubernetes cluster for distributed Argo workflows.

2. Enable Argo Server for a workflow cluster.

3. Mount an Object Storage Service (OSS) volume to the workflow. This way, the workflow can access files in the OSS volume in the same way the workflow accesses local files. For more information, see Use volumes.

4. Create a workflow based on the following YAML content. For more information, see Create a workflow.

   The workflow consists of three steps:

   • bwaprepare: the data preparation step. This step downloads and decompresses FASTQ files and a reference file, and then creates an index of the reference genome.

   • bwamap: the sequence alignment step. This step aligns the sequence data to the reference genome. This step processes multiple files in parallel.

   • bwaindex: the result generation step. This step generates a binary alignment and map (BAM) file to record the alignments, sorts the alignments, and then creates an index of the BAM file. You can also view the alignments.

   Show YAML content

   apiVersion: argoproj.io/v1alpha1
   kind: Workflow
   metadata:
     generateName: bwa-oss-
   spec:
     entrypoint: bwa-oss
     arguments: 
       parameters:
       - name: fastqFolder     # The path to which the file is downloaded. 
         value: /gene
       - name: reference       # The reference genome file. 
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/subset_assembly.fa.gz
       - name: fastq1          # The sequencing data. 
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/SRR1976948_1.fastq.gz
       - name: fastq2
         value: https://ags-public.oss-cn-beijing.aliyuncs.com/alignment/SRR1976948_2.fastq.gz
   
   
     volumes:                  # Mount remote storage. 
     - name: ossdir
       persistentVolumeClaim:
         claimName: pvc-oss
   
     templates:
     - name: bwaprepare      # The data preparation step downloads and decompresses FASTQ files and a reference file, and then creates an index of the reference genome. 
       container:
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         command: [sh,-c]
         args:
         - mkdir -p /bwa{{workflow.parameters.fastqFolder}}; cd /bwa{{workflow.parameters.fastqFolder}}; rm -rf SRR1976948*;
           wget {{workflow.parameters.reference}};
           wget {{workflow.parameters.fastq1}};
           wget {{workflow.parameters.fastq2}};
           gzip -d subset_assembly.fa.gz;
           gunzip -c SRR1976948_1.fastq.gz | head -800000 > SRR1976948.1;
           gunzip -c SRR1976948_2.fastq.gz | head -800000 > SRR1976948.2;
           bwa index subset_assembly.fa;
   
         volumeMounts:
         - name: ossdir
           mountPath: /bwa
       retryStrategy:         # Configure the retry mechanism. 
         limit: 3
   
     - name: bwamap           # Preprocess the alignment between the sequencing data and the reference genome and then generate alignments. 
       inputs:
         parameters:
         - name: object
       container:
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         command:
         - sh
         - -c
         args:
         - cd /bwa{{workflow.parameters.fastqFolder}};
           bwa aln subset_assembly.fa {{inputs.parameters.object}} > {{inputs.parameters.object}}.untrimmed.sai;
         volumeMounts:
         - name: ossdir
           mountPath: /bwa
       retryStrategy:
         limit: 3
   
     - name: bwaindex       # Generate a BAM file to record the alignments, sort the alignments, and then create an index of the BAM file. You can also view the alignments. 
       container:
         args:
         - cd /bwa{{workflow.parameters.fastqFolder}};
           bwa sampe subset_assembly.fa SRR1976948.1.untrimmed.sai SRR1976948.2.untrimmed.sai SRR1976948.1 SRR1976948.2 > SRR1976948.untrimmed.sam;
           samtools import subset_assembly.fa SRR1976948.untrimmed.sam SRR1976948.untrimmed.sam.bam;
           samtools sort SRR1976948.untrimmed.sam.bam -o SRR1976948.untrimmed.sam.bam.sorted.bam;
           samtools index SRR1976948.untrimmed.sam.bam.sorted.bam;
           samtools tview SRR1976948.untrimmed.sam.bam.sorted.bam subset_assembly.fa -p k99_13588:1000 -d T;
         command:
         - sh
         - -c
         image: registry.cn-beijing.aliyuncs.com/geno/alltools:v0.2
         imagePullPolicy: Always
         volumeMounts:
         - mountPath: /bwa/
           name: ossdir
       retryStrategy:
         limit: 3
   
     - name: bwa-oss           # Orchestration of different steps. 
       dag:
         tasks:
         - name: bwaprepare          # Prepare data. 
           template: bwaprepare
   
         - name: bwamap         # Preprocess the alignment and generate alignments. 
           template: bwamap
           dependencies: [bwaprepare]        # Prepare dependency data. 
           arguments:
             parameters:
             - name: object
               value: "{{item}}"
           withItems: ["SRR1976948.1","SRR1976948.2"]      # Process multiple files in parallel. 
   
         - name: bwaindex               # Align the sequencing data to the reference genome and view the result. 
           template: bwaindex
           dependencies: [bwamap]           # Prepare dependencies.

5. After the workflow starts running, you can go to the Workflow Console (Argo) to view the directed acyclic graph (DAG) process and the result.

   

   You can also log on to the OSS console to check whether a file that stores the alignments is generated.

References

• For more information about Kubernetes clusters for distributed Argo workflows, see Overview of Kubernetes clusters for distributed Argo workflows.

• For more information about Argo Workflows, see Open source Argo Workflows.