Canary upgrade for an ASM gateway - Alibaba Cloud Service Mesh

ASM gateways support canary upgrades for a smooth upgrade experience. This topic uses an in-place upgrade of an ASM instance as an example to show how one pod is started on the new version so that you can verify traffic on it before all gateway pods are upgraded.

Prerequisites

ASM instance and version — Make sure that you have added a cluster to an ASM instance, and that the ASM version is 1.20 or later. For instructions on upgrading an ASM instance, see Upgrade an ASM instance.
Ingress gateway — Make sure that you have created an ingress gateway.
Graceful shutdown — Enable graceful shutdown for the gateway before you perform the upgrade. This minimizes the risk of traffic loss during the upgrade. For instructions, see Enable graceful shutdown to prevent traffic loss.
No name conflict with the Canary Deployment — Make sure that no Deployment exists in the namespace of the gateway whose name is the name of the gateway Deployment plus a -canary suffix. For example, if the gateway is named ingressgateway, make sure that no Deployment named istio-ingressgateway-canary exists in the istio-system namespace.
Data plane access — Obtain the KubeConfig file of the data plane so that you can run kubectl commands against the data plane cluster.

How a gateway canary upgrade works

As the single entry and exit point for service traffic, a gateway must meet stringent availability requirements. A canary upgrade helps you maintain business continuity. You change the gateway configuration to first start a gateway pod that runs the new version and verify traffic on that pod. The gateway is fully upgraded only after the verification passes. If any issue occurs during the verification, you can delete this new pod at any time, and resume the upgrade after you resolve the issue.

Note

A canary upgrade of an ASM gateway is not the same as a canary upgrade of an ASM instance. The canary upgrade of an ASM gateway is available regardless of whether you perform an in-place upgrade or a canary upgrade of the ASM instance.

In ASM, the control plane upgrade and the data plane upgrade are separate. The data plane can be upgraded after the control plane upgrade is complete.

A gateway canary upgrade creates a single-replica Deployment that matches the current control plane version. This Deployment is called the Canary Deployment, and the single pod that it manages is the canary pod. The Canary Deployment belongs to the same Service as the original Deployment of the gateway. The difference is that its name is the name of the original Deployment plus a -canary suffix.

Important

If a Deployment with the same name and a -canary suffix already exists in the namespace of the gateway, that Deployment is not overwritten.

Both of the following ASM instance upgrade paths support a gateway canary upgrade. Identify your path before you start:

In-place upgrade of an ASM instance — After the control plane upgrade is complete, the gateway still runs the earlier version. Perform Step 1 to Step 4 in this topic.
Canary upgrade of an ASM instance — Step 1 in this topic does not apply. During the canary upgrade of an ASM instance, after you switch the new version to the default control plane, the gateway is pending upgrade. Continue from Step 2 to deploy a canary pod for the gateway, verify the traffic, and upgrade the gateway after the verification passes.

Step 1: Upgrade the control plane

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Instance Information > Upgrade Management.
On the Upgrade Management page, on the In-place Upgrades tab, click Perform Upgrade Precheck. In the Note dialog box, click OK.

Note
If the page displays Currently, it is the latest version and does not need to be upgraded., the service mesh is already the latest version and cannot be upgraded.
After the upgrade precheck passes, click Upgrade. In the Note dialog box, click OK.

Step 2: Deploy a canary pod for the ASM gateway

To deploy a canary pod for an ASM gateway, add the asm.alibabacloud.com/enable-canary-deployment: "true" annotation to annotations under the metadata field in the YAML file of the ASM gateway. Use either the ASM console or the kubectl command line.

Method 1: Use the ASM console

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
On the Ingress Gateway page, click View YAML to the right of the target gateway. In the Edit dialog box, add the annotation, and then click OK.

Method 2: Use the kubectl command

Run the following command to open the YAML file of the ASM gateway.

kubectl edit IstioGateway <YOUR-INGRESSGATEWAY-NAME> -n istio-system

Edit the YAML file to add the annotation, and then save the file.

The following sample YAML file shows an ASM gateway configuration with the annotation added. The result is the same for both methods.

apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  labels:
    asm-gateway-type: ingress
  annotations:
    asm.alibabacloud.com/enable-canary-deployment: "true"  # Add this annotation.
  name: ingressgateway
  namespace: istio-system
spec:
  clusterIds:
    - xxxxxxxxxxxxxx
  disableContainerPortExposed: true
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  gatewayType: ingress
  ports:
    - name: http-0
      port: 80
      protocol: HTTP
      targetPort: 80
  replicaCount: 1
  resources:
    limits:
      cpu: '2'
      memory: 4G
    requests:
      cpu: 200m
      memory: 256Mi
  rollingMaxSurge: 25%
  rollingMaxUnavailable: 25%
  serviceAnnotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: 'on'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: '10'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: PayByCLCU
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
  serviceType: LoadBalancer

Verify the canary pod and its traffic

Perform the following steps regardless of the method that you used to add the annotation.

Use the KubeConfig file of the data plane to run the following command to view the deployed Canary Deployment. In this example, the gateway name is ingressgateway. Replace it based on your actual scenario.
```
kubectl -n istio-system get deployment | grep ingressgateway
```
Expected output:
```
istio-ingressgateway                1/1     1            1           10m
istio-ingressgateway-canary         1/1     1            1           55s
```
The Deployment with the -canary suffix is the Canary Deployment. Wait until its ready replicas reach 1/1, as shown in the preceding output.
After the canary pod is ready, part of the traffic is routed to the canary pod. Check whether the traffic that reaches the gateway is normal.

If the traffic is normal, continue with Step 3 to upgrade the gateway. If the traffic is abnormal, roll back the canary pod as described in the following section.

Roll back if the verification fails

If the traffic that reaches the gateway is abnormal, perform the following steps:

Save the gateway logs.
Delete the asm.alibabacloud.com/enable-canary-deployment: "true" annotation that you configured. The canary pod is deleted, and all traffic is routed to the gateway pods that run the earlier version.
After you resolve the issue, add the annotation again to redeploy the canary pod, and then verify the traffic again.

Step 3: Upgrade the gateway after the verification passes

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Instance Information > Upgrade Management.
In the Data Plane section, in the Upgrade column, select the target gateway and click Upgrade Gateway. In the Note dialog box, click OK and wait until the gateway upgrade is complete.

Step 4: Delete the canary pod

After the gateway upgrade is complete, delete the annotation that you added in Step 2. The canary pod is automatically deleted. To confirm the result, view the Deployments in the namespace of the gateway again and check that the Deployment with the -canary suffix no longer exists.

Important

The canary pod always follows the version of the control plane. If you do not delete it, the canary pod is upgraded together with the control plane the next time you upgrade the control plane.