Istio Ecosystem on ASM (2): Integrate Argo Rollouts into Alibaba Cloud Service Mesh for Canary Rollout

Background

ArgoCD is responsible for monitoring the changes in application orchestration in the Git repository. It compares the actual running status of applications in the cluster and automatically/manually synchronizes the changes in application orchestration to the deployment cluster. Argo Rollouts provides powerful blue-green and canary deployment capabilities. The two can be combined in practice to provide progressive delivery capabilities based on GitOps.

Prerequisites

Create an Alibaba Cloud Service Mesh (ASM) enterprise edition instance whose Istio version is 1.12.4.50 or later.
Create a Container Service for Kubernetes (ACK) cluster
Add the ACK cluster to the ASM instance
Argo Rollouts is installed (v1.2.1).

Install Argo Rollout

Use the kubeconfig of the data plane cluster and run the following command to install the Argo Rollout server:

Please see the Arog Rollout official website for more information.

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Run the following command to install the Kubectl Argo Rollout plug-in for easy management using kubectl:

brew install argoproj/tap/kubectl-argo-rollouts

Enable the Data Plane KubeAPI Access Capability of ASM

Canary Rollout

Create Rollout

Save the following content as rollout.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: istio-rollout
spec:
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: istio-rollout
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: istio-rollout
    spec:
      containers:
      - name: istio-rollout
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
  strategy:
    canary:
      canaryService: istio-rollout-canary
      stableService: istio-rollout-stable
      trafficRouting:
        istio:
          virtualService:
            name: istio-rollout-vsvc
            routes:
            - primary
      steps:
      - setWeight: 10
      - pause: {}         # manual pause
      - setWeight: 20
      - pause: {duration: 20s}
      - setWeight: 30
      - pause: {duration: 20s}
      - setWeight: 40
      - pause: {duration: 20s}
      - setWeight: 50
      - pause: {duration: 20s}
      - setWeight: 60
      - pause: {duration: 20s}
      - setWeight: 70
      - pause: {duration: 20s}
      - setWeight: 80
      - pause: {duration: 20s}
      - setWeight: 90
      - pause: {duration: 20s}

Run the following command to use kubeconfig of the data plane cluster:

kubectl apply -f rollout.yaml

Create Service

Save the following content as service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: istio-rollout-canary
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: istio-rollout

---
apiVersion: v1
kind: Service
metadata:
  name: istio-rollout-stable
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: istio-rollout

Run the following command to create:

kubectl apply -f service.yaml

Create Istio-Related Resources

1. Create VirtualService

Since the KubeAPI access capability of the data plane of ASM is enabled, you can use kubeconfig of the data plane to access Istio resources (such as VirtualService, Gateway, and DestinationRule) in ASM. You can also use the ASM console or ASM kubeconfig to create Istio resources.

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: istio-rollout-vsvc
spec:
  gateways:
    - istio-rollout-gateway
  hosts:
    - '*'
  http:
    - match:
        - uri:
            prefix: /
      name: primary
      route:
        - destination:
            host: istio-rollout-stable
          weight: 100
        - destination:
            host: istio-rollout-canary

2. Create a Gateway Rule

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-rollout-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - '*'
      port:
        name: http
        number: 80
        protocol: HTTP

Create an ASM Gateway

Select the ASM Gateway by Alibaba Cloud as the entry point for the test access and set the port to 80.

View the Creation Status

Run the following commands:

kubectl argo rollouts get rollout istio-rollout
kubectl argo rollouts get rollout istio-rollout
Name:            istio-rollout
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          18/18
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                       KIND        STATUS     AGE  INFO
⟳ istio-rollout                            Rollout     ✔ Healthy  52s  
└──# revision:1                                                        
   └──⧉ istio-rollout-7f96d86486           ReplicaSet  ✔ Healthy  52s  stable
      └──□ istio-rollout-7f96d86486-vpqvb  Pod         ✔ Running  52s  ready:2/2

Test Initial Status

In Alibaba Cloud Service Mesh → ASM Gateway, you can obtain the IP address of the ASM gateway just created and enter http://{ASM Gateway IP}/ in the browser to access it:

The following is the access effect. This interface concurrently calls http://{ASM gateway IP}/color to fill the obtained color information in the grid. Since the color we specified is blue, and canary publishing has not been performed in the Rollout istio-rollout, only blue is displayed.

Rolling Deployment

1. Update the Image Version

Next, we hope to gradually change the color in the grid to yellow. Run the following command to set the image version:

kubectl argo rollouts set image istio-rollout "*=argoproj/istio-rollout:yellow"

The image in the rollout is updated to the yellow version.

The yellow version (canary) of the pod is created, but the blue version (stable) is still saved.

2. Query Effect

View http://{ASM Gateway IP}/ again:

You can see the 10% grids are set to yellow.

3. Reasons

The reason is that in ASM, the weight of the previously configured VirtualService has changed, and the weight of stable (blue) has been changed from 100 to 90. The weight of the canary (yellow) changes from 0 to 10.

This change is controlled by Rollout. The weight of the canary version is set to 10 in the setWeight of the first step of Rollout we started to create. When rolling, the controller of Argo Rollout will modify the weight of VirtualService configured in Rollout. The pause parameter is set to empty, which indicates a manual pause is required before proceeding to the next stage.

4. Continue Publishing

Run the following command to continue publishing. The weights in VirtualService continue to be adjusted based on the configuration in Rollout. Since the time is set in the later steps, it will automatically adjust after waiting a while.

kubectl argo rollouts promote istio-rollout

5. Update Completed

After a while, all the color grids turn yellow:

Check the Rollout status, and the image of stable is updated to yellow:

kubectl argo rollouts get rollout istio-rollout --watch
Name:            istio-rollout
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          18/18
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                       KIND        STATUS        AGE  INFO
⟳ istio-rollout                            Rollout     ✔ Healthy     48m  
├──# revision:4                                                           
│  └──⧉ istio-rollout-5fcf5864c4           ReplicaSet  ✔ Healthy     27m  stable
│     └──□ istio-rollout-5fcf5864c4-vw6kh  Pod         ✔ Running     26m  ready:2/2
├──# revision:3                                                           
│  └──⧉ istio-rollout-897cb5b6d            ReplicaSet  • ScaledDown  27m  
└──# revision:1                                                           
   └──⧉ istio-rollout-7f96d86486           ReplicaSet  • ScaledDown  48m

Combine with Prometheus to Rollback Automatically

You can manually run the following command to rollback to the stable version during the canary process. A more elegant way is to combine with the Prometheus monitoring system. If the monitoring metric is abnormal, it will be automatically rolled back to the stable version and marked as degraded.

kubectl argo rollouts abort istio-rollout

Enable Prometheus in ASM

Please see Integrate ARMS Prometheus for Mesh Monitoring or Integrate Self-built Prometheus for Mesh Monitoring for more information.

Configure the Argo AnalysisTemplate

Configure the Prometheus address of ASM at the AnalysisTemplate address:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: istio-success-rate
spec:
  args:
  - name: service
  - name: namespace
  metrics:
  - name: success-rate
    initialDelay: 60s
    interval: 20s
    successCondition: result[0] > 0.90
    provider:
      prometheus:
        address: http://xxx.aliyuncs.com:9090/api/v1/prometheus/
        query: >+
          sum(irate(istio_requests_total{
            reporter="source",
            destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local",
            response_code!~"5.*"}[40s])
          )
          /
          sum(irate(istio_requests_total{
            reporter="source",
            destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local"}[40s])
          )

Associate Analysis for Rollout

Configure analysis in strategy and enable monitoring with analysis from the second step to rollback automatically. The initial image is yellow. Save the content as rollout.yaml. Run the kubectl apply -f rollout.yaml command to update the istio-rollout.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: istio-rollout
spec:
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: istio-rollout
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app: istio-rollout
    spec:
      containers:
      - name: istio-rollout
        image: argoproj/rollouts-demo:yellow
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
  strategy:
    canary:
      canaryService: istio-rollout-canary
      stableService: istio-rollout-stable
      analysis:
        startingStep: 1   # index of step list, of when to start this analysis
        templates:
        - templateName: istio-success-rate
        args:
        - name: service 
          value: canary
        - name: namespace
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
      trafficRouting:
        istio:
          virtualService:
            name: istio-rollout-vsvc
            routes:
            - primary
      steps:
      - setWeight: 10
      - pause: {}         # manual pause
      - setWeight: 20
      - pause: {duration: 20s}
      - setWeight: 30
      - pause: {duration: 20s}
      - setWeight: 40
      - pause: {duration: 20s}
      - setWeight: 50
      - pause: {duration: 20s}
      - setWeight: 60
      - pause: {duration: 20s}
      - setWeight: 70
      - pause: {duration: 20s}
      - setWeight: 80
      - pause: {duration: 20s}
      - setWeight: 90
      - pause: {duration: 20s}

Change the Color to Orange

kubectl argo rollouts set image istio-rollout "*=argoproj/rollouts-demo:orange"

Manual Pause Confirmation

Run the following command to enter the subsequent automatic canary state. Prometheus Monitoring is combined from the second step. If the error rate of the canary version is higher than 90%, a rollback is triggered.

kubectl argo rollouts promote istio-rollout

After the manual pause is passed, the subsequent progressive release will begin automatically, which can be viewed with the following command:

kubectl argo rollouts get rollout istio-rollout –watch

Set Error

We can manually set Error in subsequent progressive releases. After moving Error to 100%, all canary versions (orange) have a red box to indicate an error. After waiting for a while, it will automatically switch back to the yellow only (stable version).

(In the Canary)

(Automatic Rollback to Stable Version)

Summary

The Sidecar mode used in Alibaba Cloud Service Mesh (ASM) is helpful to unify the metrics used in services, facilitate monitoring of the health status of applications in canaries, and automatically detect problems timely for rollback.

Community