This topic provides examples about how to use Alibaba Cloud Genomics Service (AGS).
Prerequisites
- A Container Service for Kubernetes (ACK) cluster is created. For more information, see Create a managed kubernetes cluster.
- You are connected to the cluster. For more information, see Use kubectl to connect to an ACK cluster.
Log
List
You can set the --limit parameter to specify the number of workflow entries that you want to query.
[root@iZwz92q9h36kv8posr0i6uZ ~]# ags remote list --limit 8
+-----------------------+-------------------------------+------------+
| JOB NAME | CREATE TIME | JOB STATUS |
+-----------------------+-------------------------------+------------+
| merge-6qk46 | 2020-09-02 16:52:34 +0000 UTC | Pending |
| rna-mapping-gpu-ck4cl | 2020-09-02 14:47:57 +0000 UTC | Succeeded |
| wgs-gpu-n5f5s | 2020-09-02 13:14:14 +0000 UTC | Running |
| merge-5zjhv | 2020-09-02 12:03:11 +0000 UTC | Succeeded |
| merge-jjcw4 | 2020-09-02 10:44:51 +0000 UTC | Succeeded |
| wgs-gpu-nvxr2 | 2020-09-01 22:18:44 +0000 UTC | Succeeded |
| merge-4vg42 | 2020-09-01 20:52:13 +0000 UTC | Succeeded |
| rna-mapping-gpu-2ss6n | 2020-09-01 20:34:45 +0000 UTC | Succeeded |
Run kubectl commands
[root@iZwz92q9h36kv8posr0i6uZ ~]# ags get test-v2
Name: test-v2
Namespace: default
ServiceAccount: default
Status: Running
Created: Thu Nov 22 11:06:52 +0800 (2 minutes ago)
Started: Thu Nov 22 11:06:52 +0800 (2 minutes ago)
Duration: 2 minutes 46 seconds
STEP PODNAME DURATION MESSAGE
● test-v2
└---● bcl2fq test-v2-2716811808 2m
[root@iZwz92q9h36kv8posr0i6uZ ~]# ags kubectl describe pod test-v2-2716811808
Name: test-v2-2716811808
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: cn-shenzhen.i-wz9gwobtqrbjgfnqxl1k/192.168.0.94
Start Time: Thu, 22 Nov 2018 11:06:52 +0800
Labels: workflows.argoproj.io/completed=false
workflows.argoproj.io/workflow=test-v2
Annotations: workflows.argoproj.io/node-name=test-v2[0].bcl2fq
workflows.argoproj.io/template={"name":"bcl2fq","inputs":{},"outputs":{},"metadata":{},"container":{"name":"main","image":"registry.cn-hangzhou.aliyuncs.com/dahu/curl-jp:1.2","command":["sh","-c"],"ar...
Status: Running
IP: 172.16. *. ***
Controlled By: Workflow/test-v2
After you run the ags kubectl command, the state of describe pod is returned. AGS supports all native kubectl commands.
Run ossutil commands
After AGS is initialized, you can run the following commands to upload and query files:
[root@iZwz92q9h36kv8posr0i6uZ ~]# ags oss cp test.fq.gz oss://my-test-shenzhen/fasq/
Succeed: Total num: 1, size: 690. OK num: 1(upload 1 files).
average speed 3000(byte/s)
0.210685(s) elapsed
[root@iZwz92q9h36kv8posr0i6uZ ~]# ags oss ls oss://my-test-shenzhen/fasq/
LastModifiedTime Size(B) StorageClass ETAG ObjectName
2020-09-02 17:20:34 +0800 CST 690 Standard 9FDB86F70C6211B2EAF95A9B06B14F7E oss://my-test-shenzhen/fasq/test.fq.gz
Object Number is: 1
0.117591(s) elapsed
You can run the ags oss command to upload and download files. AGS supports all native ossutil commands.
View the resource usage of a workflow
Configure securityContext
ags submit arguments-security-context.yaml
command to use Pod Security Policy (PSP) for permission control.apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
name: test
spec:
arguments: {}
entrypoint: test-security-
templates:
- inputs: {}
metadata: {}
name: test-security-
outputs: {}
parallelism: 1
steps:
- - arguments: {}
name: bcl2fq
template: bcl2fq
- container:
args:
- id > /tmp/yyy;echo `date` > /tmp/aaa;ps -e -o comm,euid,fuid,ruid,suid,egid,fgid,gid,rgid,sgid,supgid
> /tmp/ppp;ls -l /tmp/aaa;sleep 100;pwd
command:
- sh
- -c
image: registry.cn-hangzhou.aliyuncs.com/dahu/curl-jp:1.2
name: main
resources: #don't use too much resources
requests:
memory: 320Mi
cpu: 1000m
inputs: {}
metadata: {}
name: bcl2fq
outputs: {}
securityContext:
runAsUser: 800
Configure automatic retry with YAML
Unexpected errors may occur for some bash commands. You can execute the command again to fix this issue. AGS provides an automatic retry mechanism based on YAML. When the system fails to run a command in a pod, AGS automatically executes the command again. You can set the maximum number of retry attempts.
ags submit arguments-auto-retry.yaml
command to configure the automatic retry mechanism for a workflow.# This example demonstrates the use of retries for a single container.
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: retry-container-
spec:
entrypoint: retry-container
templates:
- name: retry-container
retryStrategy:
limit: 10
container:
image: python:alpine3.6
command: ["python", -c]
# fail with a 66% probability
args: ["import random; import sys; exit_code = random.choice([0, 1, 1]); sys.exit(exit_code)"]
Retry a workflow from a failed step
A workflow consists of a number of steps. A step may fail when you run a workflow. AGS allows you to retry the workflow from a failed step.
Run a workflow by using ECIs
For more information about Elastic Container Instance (ECI), see Elastic Container Instance.
Install AGS before you use ECI to run a workflow. For more information, see Introduction to AGS CLI.
Query the actual and peak resource usage of a workflow
The AGS workflow controller automatically queries the actual resource usage of pods per minute through metrics-server. It also calculates the total and peak resource usage of each pod.
ags get steps-jr6tw --metrics
command to query the actual and peak resource usage of a workflow.➜ ags get steps-jr6tw --metrics
Name: steps-jr6tw
Namespace: default
ServiceAccount: default
Status: Succeeded
Created: Tue Apr 16 16:52:36 +0800 (21 hours ago)
Started: Tue Apr 16 16:52:36 +0800 (21 hours ago)
Finished: Tue Apr 16 19:39:18 +0800 (18 hours ago)
Duration: 2 hours 46 minutes
Total CPU: 0.00275 (core*hour)
Total Memory: 0.04528 (GB*hour)
STEP PODNAME DURATION MESSAGE CPU(core*hour) MEMORY(GB*hour) MaxCpu(core) MaxMemory(GB)
✔ steps-jr6tw 0 0 0 0
└---✔ hello1 steps-jr6tw-2987978173 2h 0.00275 0.04528 0.000005 0.00028
Set workflow priorities
You can set the priority of a workflow to high, medium, or low to prioritize urgent tasks. A workflow with a higher priority can preempt the resources of a workflow with a lower priority.
- You can use the following method to set a high priority for a pod:
Create the arguments-high-priority-taskA.yaml file and copy the following content to the file. Run the
ags submit arguments-high-priority-taskA.yaml
command to set a high priority for Task A.apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: high-priority value: 1000000 globalDefault: false description: "This priority class should be used for XYZ service pods only."
- You can use the following method to set a medium priority for a pod:
Create the arguments-high-priority-taskB.yaml file and copy the following content to the file. Run the
ags submit arguments-high-priority-taskB.yaml
command to set a medium priority for Task B.apiVersion: scheduling.k8s.io/v1beta1 kind: PriorityClass metadata: name: medium-priority value: 100 globalDefault: false description: "This priority class should be used for XYZ service pods only."
- You can use the following method to set a high priority for a workflow:
Create the arguments-high-priority-Workflow.yaml file and copy the following content to the file. Run the
ags submit arguments-high-priority-Workflow.yaml
command to set a high priority for all pods of the workflow.apiVersion: argoproj.io/v1alpha1 kind: Workflow # new type of k8s spec metadata: generateName: high-proty- # name of the workflow spec spec: entrypoint: whalesay # invoke the whalesay template podPriorityClassName: high-priority # workflow level priority templates: - name: whalesay # name of the template container: image: ubuntu command: ["/bin/bash", "-c", "sleep 1000"] resources: requests: cpu: 3
The following example describes how to set priorities for pods in a workflow. Assume that the workflow has two pods and you set a high priority for one pod and a medium priority for the other pod. Then, the pod with the high priority can preempt the resources of the pod with the medium priority.
Workflow Filter
When a workflow contains a large number of pods, you can use the workflow filter in the ags get workflow command to query pods that are in a specified state.
Use Autoscaler in the agility edition
- A virtual private cloud (VPC).
- A VSwitch.
- A security group.
- The internal endpoint of APIServer of the agility edition.
- The specification for node scaling.
- An Elastic Compute Service (ECS) instance that can access the Internet.
$ags config autoscaler Enter the required information based on the tip: Please input VSwitches separated with commas (,).
vsw-hp3cq3fnv47bpz7x58wfe
Please input security group id
sg-hp30vp05x6tlx13my0qu
Please input the instanceTypes with comma separated
ecs.c5.xlarge
Please input the new ecs ssh password
xxxxxxxx
Please input k8s cluster APIServer address like(192.168.1.100)
172.24.61.156
Please input the autoscaling mode (current: release. Type enter to skip.)
Please input the min size of group (current: 0. Type enter to skip.)
Please input the max size of group (current: 1000. Type enter to skip.)
Create scaling group successfully.
Create scaling group config successfully.
Enable scaling group successfully.
Succeed
After you complete the preceding operations, log on to the Auto Scaling console to check the scaling group that you have created.
Configure and use a ConfigMap
By default, hostNetwork is used in this example.