Run a Canary Upgrade for ASM Gateway - Alibaba Cloud Service Mesh

A canary upgrade lets you validate a new ASM gateway version with live traffic before completing a full upgrade. A single canary pod runs alongside the existing gateway Deployment, receives a share of production traffic, and lets you confirm correct behavior -- or roll back instantly if something goes wrong. In this topic, an in-place upgrade is used as an example.

How it works

Control plane and data plane upgrades in ASM are independent. After you upgrade the control plane, the gateway (data plane) stays at its original version until you explicitly upgrade it.

An ASM gateway manages the ingress or egress of service traffic. A canary upgrade adds a temporary Deployment -- called the Canary Deployment -- to the data plane. This Canary Deployment:

Provisions a single pod running the new gateway version.
Belongs to the same Service as the original gateway Deployment, so it automatically receives a share of live traffic.
Uses the naming convention istio-<gateway-name>-canary (for example, istio-ingressgateway-canary).

After you confirm that traffic flows correctly through the canary pod, upgrade the gateway to the new version and remove the Canary Deployment.

Note

A canary upgrade of an ASM gateway is separate from a canary upgrade of an ASM instance. Gateway canary upgrades work with both in-place upgrades and canary upgrades of the ASM instance.

In-place upgrade workflow

Upgrade the control plane. The gateway stays at the original version.
Deploy a Canary Deployment. Some traffic routes to the canary pod.
Verify that traffic forwards correctly through the canary pod.
Upgrade the gateway to the new version.
Remove the Canary Deployment.

Canary upgrade of an ASM instance workflow

Start the canary upgrade of the ASM instance. After the new control plane becomes the default, the gateway enters a to-be-upgraded state.
Deploy a Canary Deployment for the gateway.
Verify traffic forwarding through the canary pod.
Upgrade the gateway.
Remove the Canary Deployment.

Prerequisites

Before you begin, make sure that you have:

An ASM instance running version 1.20 or later with a Container Service for Kubernetes (ACK) cluster added. For more information, see Add a cluster to an ASM instance. To update an existing ASM instance, see Update an ASM instance
An ingress gateway deployed in the ASM instance. For more information, see Create an ingress gateway
No existing Deployment with the Canary Deployment name in the gateway namespace. For example, if the gateway name is ingressgateway, make sure no Deployment named istio-ingressgateway-canary exists in the istio-system namespace

Step 1: Upgrade the control plane

Before upgrading, enable graceful shutdown for the ASM gateway to minimize traffic loss. For more information, see Enable graceful shutdown to prevent traffic loss.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the In-place Upgrades tab of the Upgrade Management page, click Perform Upgrade Precheck. In the Note message, click OK.
Note
If the message "Currently, it is the latest version and does not need to be upgraded" appears, the ASM instance is already at the latest version.
After the precheck passes, click Upgrade. In the Note message, click OK.

Step 2: Deploy the Canary Deployment

Adding the asm.alibabacloud.com/enable-canary-deployment: "true" annotation to the ASM gateway resource triggers ASM to create the Canary Deployment automatically.

Choose one of the following methods:

Use the ASM console

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
On the Ingress Gateway page, find the target ingress gateway and click YAML in the Actions column.
In the Edit dialog box, add the following annotation to the annotations field in the metadata section, then click OK:
```
   asm.alibabacloud.com/enable-canary-deployment: "true"
```

Run a kubectl command

Open the ASM gateway YAML for editing:

   kubectl edit IstioGateway <your-gateway-name> -n istio-system

Add the canary annotation to the metadata.annotations field, then save the file.

Sample YAML with the canary annotation

apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  labels:
    asm-gateway-type: ingress
  annotations:
    asm.alibabacloud.com/enable-canary-deployment: "true" # Add this annotation.
  name: ingressgateway
  namespace: istio-system
spec:
  clusterIds:
    - xxxxxxxxxxxxxx
  disableContainerPortExposed: true
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  gatewayType: ingress
  ports:
    - name: http-0
      port: 80
      protocol: HTTP
      targetPort: 80
  replicaCount: 1
  resources:
    limits:
      cpu: '2'
      memory: 4G
    requests:
      cpu: 200m
      memory: 256Mi
  rollingMaxSurge: 25%
  rollingMaxUnavailable: 25%
  serviceAnnotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: 'on'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: '10'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: PayByCLCU
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
  serviceType: LoadBalancer

Verify the Canary Deployment

After adding the annotation, connect to the data plane cluster and verify that the Canary Deployment is running:

kubectl -n istio-system get deployment | grep ingressgateway

Expected output:

istio-ingressgateway                1/1     1            1           10m
istio-ingressgateway-canary         1/1     1            1           55s

The Deployment with the -canary suffix is the Canary Deployment.

Roll back if something goes wrong

If traffic does not flow correctly through the canary pod:

Save the ASM gateway logs for troubleshooting.
Remove the asm.alibabacloud.com/enable-canary-deployment: "true" annotation. This deletes the Canary Deployment and routes all traffic back to the original gateway.
Resolve the issue, then re-add the annotation to redeploy the Canary Deployment and repeat verification.

Step 3: Upgrade the ASM gateway

After you confirm that traffic forwards correctly through the canary pod, upgrade the gateway.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
In the Data Plane section, select the ASM gateway to upgrade in the Upgrade column and click Upgrade Gateway. In the Note message, click OK.

Step 4: Remove the Canary Deployment

After the gateway upgrade completes, remove the asm.alibabacloud.com/enable-canary-deployment: "true" annotation added in Step 2. ASM deletes the Canary Deployment automatically.

Important

The Canary Deployment version always matches the control plane version. If you do not remove the Canary Deployment, it upgrades automatically the next time you upgrade the control plane.

Alibaba Cloud Service Mesh:Canary upgrade an ASM gateway