Canary Upgrade ASM Ingress Gateways with Zero Downtime - Alibaba Cloud Service Mesh

A canary upgrade deploys a single new-version gateway pod alongside your existing pods, routes a portion of live traffic to it, and gives you a rollback path if anything goes wrong. This lets you verify that the new version handles traffic correctly before you commit to a full upgrade.

This topic walks through the canary upgrade process using an in-place upgrade scenario. The same canary verification steps apply when you perform a canary upgrade of the ASM instance.

How it works

An ASM gateway manages the ingress or egress of service traffic and is required to be highly available. To ensure business continuity during an upgrade, ASM creates a second Deployment -- named <gateway-name>-canary -- in the same namespace and behind the same Kubernetes Service as the original gateway Deployment. Because both Deployments share the Service, traffic is automatically distributed across original and canary pods. The Canary Deployment provisions only one pod, so the blast radius is small. After you confirm that the canary pod forwards traffic without errors, you upgrade all gateway pods and delete the Canary Deployment.

Note

A canary upgrade of an ASM gateway is independent of the ASM instance upgrade method. You can use gateway canary upgrades with both in-place and canary instance upgrades.

In-place upgrade vs. canary upgrade of an ASM instance

Upgrades of the control plane and data plane of an ASM instance are independent from each other. You can upgrade the data plane after the control plane upgrade is complete.

In-place upgrade -- After the control plane upgrade completes, the gateway remains on the old version. Deploy the Canary Deployment, verify traffic, then upgrade the gateway.
Canary upgrade of the ASM instance -- The gateway enters a to-be-upgraded state after the new control plane becomes the default. Deploy the Canary Deployment, verify traffic, then upgrade the gateway.

In both cases, the gateway canary verification workflow is the same.

Upgrade workflow

A canary upgrade of an ASM gateway follows four steps:

Step	Action	Purpose
1	Upgrade the control plane	Bring the control plane to the target version
2	Deploy a canary gateway pod	Start one new-version pod behind the same Service
3	Upgrade the gateway	Apply the new version to all gateway pods
4	Clean up the Canary Deployment	Remove the temporary canary resources

Prerequisites

Before you begin, make sure that you have:

A Container Service for Kubernetes (ACK) cluster added to an ASM instance of version 1.20 or later. For more information, see Add a cluster to an ASM instance. To update an existing ASM instance, see Update an ASM instance
An ingress gateway deployed in the ASM instance. For more information, see Create an ingress gateway
No existing Deployment whose name matches the Canary Deployment name in the gateway namespace. For example, if the gateway is named ingressgateway, confirm that no Deployment named istio-ingressgateway-canary exists in the istio-system namespace

Important

If a Deployment with the same name as the Canary Deployment already exists, ASM does not overwrite it. Remove the conflicting Deployment before you proceed.

Step 1: Upgrade the control plane

To minimize the risk of traffic loss during the upgrade, enable graceful shutdown for the ASM gateway before you start. For more information, see Enable graceful shutdown to prevent traffic loss.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the In-place Upgrades tab, click Perform Upgrade Precheck. In the Note message, click OK.
Note
If the message "Currently, it is the latest version and does not need to be upgraded" appears, the ASM instance is already at the latest version.
After the precheck passes, click Upgrade. In the Note message, click OK.

Step 2: Deploy a canary gateway pod

Add the following annotation to the ASM gateway resource to create a Canary Deployment:

asm.alibabacloud.com/enable-canary-deployment: "true"

Use either the ASM console or kubectl.

Use the ASM console

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
On the Ingress Gateway page, find the target gateway and click YAML in the Actions column.
In the Edit dialog box, add the annotation to the annotations field under metadata, then click OK.

Use kubectl

Open the gateway YAML for editing:

   kubectl edit IstioGateway <YOUR-INGRESSGATEWAY-NAME> -n istio-system

Add the annotation under metadata.annotations, then save and close the file.

Sample YAML with the annotation

apiVersion: istio.alibabacloud.com/v1beta1
kind: IstioGateway
metadata:
  labels:
    asm-gateway-type: ingress
  annotations:
    asm.alibabacloud.com/enable-canary-deployment: "true" # Add this annotation.
  name: ingressgateway
  namespace: istio-system
spec:
  clusterIds:
    - xxxxxxxxxxxxxx
  disableContainerPortExposed: true
  dnsPolicy: ClusterFirst
  externalTrafficPolicy: Local
  gatewayType: ingress
  ports:
    - name: http-0
      port: 80
      protocol: HTTP
      targetPort: 80
  replicaCount: 1
  resources:
    limits:
      cpu: '2'
      memory: 4G
    requests:
      cpu: 200m
      memory: 256Mi
  rollingMaxSurge: 25%
  rollingMaxUnavailable: 25%
  serviceAnnotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: 'on'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: '10'
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: PayByCLCU
    service.beta.kubernetes.io/alicloud-loadbalancer-address-type: internet
  serviceType: LoadBalancer

Verify the Canary Deployment

After the annotation is applied, ASM creates a Canary Deployment in the same namespace. Confirm that both the original and canary Deployments are running:

kubectl -n istio-system get deployment | grep ingressgateway

Expected output:

istio-ingressgateway                1/1     1            1           10m
istio-ingressgateway-canary         1/1     1            1           55s

The Deployment suffixed with -canary is the Canary Deployment.

Verify traffic forwarding

After the Canary Deployment is ready, observe whether some traffic flows to the pod created by the Canary Deployment. You can check the canary pod logs for incoming requests:

kubectl -n istio-system logs deployment/istio-ingressgateway-canary --tail=20

If the canary pod forwards traffic correctly, proceed to Step 3.

Roll back if needed

If the canary pod shows errors:

Retain the ASM gateway logs for troubleshooting.
Remove the asm.alibabacloud.com/enable-canary-deployment: "true" annotation from the ASM gateway resource (reverse the process in Use the ASM console or Use kubectl).
ASM automatically deletes the Canary Deployment, and all traffic returns to the original gateway pods.
After the issue is resolved, re-add the annotation to redeploy the Canary Deployment and retry verification.

Step 3: Upgrade the gateway

After you confirm that the canary pod forwards traffic without errors, upgrade all gateway pods to the new version:

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of your ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
In the Data Plane section, locate the Upgrade column, select the target ASM gateway, and click Upgrade Gateway. In the Note message, click OK.

Step 4: Clean up the Canary Deployment

After the gateway upgrade completes, remove the asm.alibabacloud.com/enable-canary-deployment: "true" annotation that you added in Step 2. ASM automatically deletes the Canary Deployment.

Important

The Canary Deployment version always matches the control plane version. If you leave the Canary Deployment in place, it is upgraded automatically the next time you upgrade the control plane.