How to use a canary upgrade to improve stability - Alibaba Cloud Service Mesh

A canary release allows you to test a new version of an application by deploying it alongside the stable version. This method helps you find and fix issues early without compromising the stability of your system. This topic describes how to use a canary upgrade to improve the stability of your ASM instance.

Prerequisites

You have an ASM instance of Enterprise Edition or Ultimate Edition that runs version 1.22.6.114 or later. For more information, see Create an ASM instance.
You have created an ACK cluster and added it to the ASM instance. For more information, see Add a cluster to an ASM instance.
You have deployed the Bookinfo application. For more information, see Deploy an application in an ACK cluster that is associated with an ASM instance.

How it works

Alibaba Cloud Service Mesh (ASM) supports a revision-based upgrade model that uses labels. This allows you to perform a canary upgrade of the control plane for greater stability and security. In this model, each sidecar proxy on the data plane is associated with a specific control plane version, or revision. A new revision can be deployed with minimal risk because no proxies will connect to it until you explicitly migrate workloads. Each independent control plane is called a revision and is identified by the istio.io/rev label.

To support revision-based upgrades, Istio uses an istio.io/rev label for namespaces. This label tells the control plane which revision to use for injecting sidecar proxies into workloads within that namespace. For example, the label istio.io/rev=1-23-6 instructs the control plane to inject the sidecar proxy of version 1.23.6 for workloads in the namespace.

During a canary upgrade, you can verify the target version by upgrading a small subset of services first. If the verification fails, you can quickly roll back to ensure service stability. After the new version is verified, you can promote the canary version to be the new stable version, upgrade all workloads by using a rolling update, and then unpublish the old version to complete the upgrade process.

Preparations

A canary upgrade requires you to explicitly specify the sidecar proxy version to be injected by using namespace labels. Therefore, you must ensure that your injection policy is correctly configured. Follow these steps to verify your injection policy configuration.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose Data Plane Component Management > Sidecar Proxy injection.
On the Sidecar Proxy injection page, under Injection strategy configuration management, confirm that The labels of the pod's namespace must meet the conditions. is set to Include istio-injection: enabled. In the Select the pods that require sidecar injection section of the Injection policy configuration page, the label condition for the pod's namespace is Contains istio-injection: enabled by default.

Note
The istio-injection: enabled label is semantically equivalent to istio.io/rev: stable. During a canary upgrade, the istio.io/rev: stable label injects the stable revision of the sidecar proxy for pods in the corresponding namespace, while the istio.io/rev: canary label injects the canary revision.

After the upgrade, you do not need to replace istio.io/rev:stable with istio-injection: enabled because they are semantically identical.

Step 1: Upgrade the control plane

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, click the Canary Upgrade tab. On the Control Plane tab, select a Canary version: and Create a new Server Load Balancer (CLB) instance, and then click Confirm. In the Confirm upgrade? dialog box, click OK.

A canary upgrade can skip at most one minor version. In this example, the ASM instance is version 1.22, so you can upgrade it up to version 1.23. This topic uses v1.23.6 as an example. When you deploy the target version of a canary upgrade, ASM creates and associates a Server Load Balancer (CLB) instance with the target version. If you have no special requirements, you can use the default specifications for the CLB instance. For more information about CLB billing, see Billing overview of CLB.

Wait for the components to deploy. After the deployment is complete, the page updates automatically.

The progress bar shows that the process is in the second step, Deploy new version. The status of both the current version and the canary version is Canary upgrading. The page indicates that the new control plane is deployed and prompts you to verify the new version. You can click Version switch to proceed with the upgrade or click Revoke upgrade to roll back.

Step 2: Upgrade the reviews-v2 sidecar proxy

In Step 1, you deployed an Istio control plane of version v1.23.6 by using a canary upgrade. The following steps describe how to verify this version by updating the injected sidecar proxy for the reviews-v2 service of the Bookinfo application to v1.23.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Global Namespace.
On the Global Namespace page, in the Automatic Sidecar Injection column, check whether the label for the default namespace is istio-injection: enabled.

If the label is istio-injection: enabled, it indicates that the sidecar proxy of version 1.22 is injected.
On the Global Namespace page, find the default namespace. In the Automatic Sidecar Injection column, click Switch to inject 1-23-6 version. In the Confirm dialog box, click OK.

The label of the default global namespace is changed to istio.io/rev: canary, and the label of the default namespace on the data plane is also synchronized to istio.io/rev: canary. The Global Namespace page shows that the sidecar proxy of version 1.23 is injected for the default namespace. New pods created in the default namespace will have the version 1.23 sidecar proxy injected. The labels of other namespaces are not changed, so they continue to use the sidecar proxy of version 1.22.

Note
The preceding steps describe how to switch the sidecar proxy version for a namespace that had automatic injection enabled before the upgrade. For a namespace where automatic injection was not enabled, you can enable it and select the desired sidecar proxy version. ASM adds the istio.io/rev:stable or istio.io/rev:canary label to the namespace based on your selection.
Perform a rolling update of the reviews-v2 workload.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.
3. On the Deployments page, find reviews-v2. In the Actions column, choose More > Redeploy. In the Redeploy dialog box, click OK.
On the Deployments page, click reviews-v2. On the Pods tab, check whether the pod for reviews-v2 started successfully after the rolling update and whether the new pod has the sidecar proxy of version 1.23 injected.

The pod for reviews-v2 starts successfully after the rolling update, and the sidecar proxy of v1.23 is injected into the new pod.
Open a browser and visit the Bookinfo page to check if traffic is routed as expected.

As shown in the following figure, you can access the reviews-v2 service as expected. The Bookinfo page loads correctly. The Book Reviews section displays book reviews with black star ratings, and the lower-right corner of the page shows Reviews served by: reviews-v2-5b5fdb5d7c-xxx. This indicates that the reviews-v2 pod is serving the reviews and traffic is routed as expected.

Step 3: Roll back reviews-v2

If the verification in Step 2 fails, or if you need to roll back for other reasons, follow these steps.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Global Namespace.
On the Global Namespace page, find the default namespace. In the Automatic Sidecar Injection column, click Switch to inject 1-22-6 version. In the Confirm dialog box, click OK.

Before the rollback, the namespace label is istio.io/rev: canary, and the sidecar proxy of version 1.23 is injected for the default namespace.

After the rollback, the label is switched to istio.io/rev: stable, and the label of the default namespace on the data plane is also updated to istio.io/rev: stable. As a result, the version 1.22 mesh proxy is injected into the default namespace.
Redeploy the reviews-v2 workload in the ACK console.
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Workloads > Deployments.
3. On the Deployments page, select default from the Namespace drop-down list. Find reviews-v2. In the Actions column, choose More > Redeploy. In the Redeploy dialog box, click OK.
4. On the Deployments page, click reviews-v2. On the Pods tab, check whether the pod for reviews-v2 started successfully after the rolling update and whether the new pod has the sidecar proxy of version 1.22 injected.
  
  The pod for reviews-v2 starts successfully after the rolling update, and the sidecar proxy of version 1.22 is injected into the new pod.

Step 4: Revoke the upgrade

After a successful rollback, you can revoke the canary upgrade to restore the ASM instance to the initial version 1.22.

Important

Before you revoke the upgrade, make sure that the stable sidecar proxy version is injected into all namespaces. Otherwise, the revocation will fail.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, on the Canary Upgrade tab, click Undo upgrade. In the Confirm the cancellation of the upgrade? dialog box, click OK.

After you click Revoke upgrade, the components of the canary control plane (version 1.23 in this example) are deleted. Only the components of the stable control plane (version 1.22 in this example) are retained.

Step 5: Retry upgrade and verification

At this point, the ASM instance is in the same state as it was before Step 1. This example only verifies the reviews-v1, reviews-v2, and reviews-v3 workloads. In a real-world scenario, you should verify all necessary workloads until the verification is successful.

Redo Step 1 to deploy the canary version of the control plane for the ASM cluster.
Redo Step 2 to verify that the injected sidecar proxy of the new version works as expected.

For example, redeploy the reviews-v1, reviews-v2, and reviews-v3 deployments from the ACK console. You can see that the sidecar proxy of version 1.23 is injected into the pods of reviews-v1, reviews-v2, and reviews-v3.

Step 6: Promote the new version

After verifying that reviews-v1, reviews-v2, and reviews-v3 work as expected with the new sidecar proxy of version 1.23, you can promote version 1.23 to be the stable version.

Important

Once you promote the version, the upgrade process proceeds to the unpublishing stage for the old version. All workloads must be switched to the new sidecar proxy of version 1.23, and you can no longer revoke the upgrade. Make sure to complete all verifications during the New version deployment stage.
After you promote the version, ASM will inject the new version 1.23 sidecar proxy into namespaces with the istio.io/rev: stable or istio-injection: enabled label. The istio.io/rev: canary label will no longer be effective. Therefore, when you promote the version, ASM automatically changes all istio.io/rev: canary labels to istio.io/rev: stable. Confirm this change when you click Version switching on the Canary Upgrade tab of the Upgrade Management page.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, on the Canary Upgrade tab, click Update Version. In the Confirm version switching? dialog box, carefully read the prompt, and then click OK.

After the promotion, existing workloads with the sidecar proxy of version 1.22 are not affected. However, all redeployed pods will have the sidecar proxy of version 1.23 injected. The page updates after the promotion is complete.

The progress bar shows that the process is in the third step, Switch version. The status of both the current version and the canary version is Canary upgrading. The page indicates that the version switch is successful. You can now upgrade the data plane. At this point, you can click Upgrade data plane to proceed or Unpublish old version to finalize the upgrade.

Step 7: Upgrade the data plane

At this point, the ASM version is 1.23.6. Namespaces with the istio.io/rev=stable, istio.io/rev=1-23-6, or istio-injection=enabled label will have the sidecar proxy of version 1.23 injected. You can complete the data plane upgrade by performing a rolling update of your workloads to upgrade their injected sidecar proxies to the new version 1.23.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, on the Canary Upgrade tab, click the Data plane tab to upgrade ASM gateways or workloads as needed.
- Upgrade an ASM gateway: In the ASM Gateways section, find the target gateway. In the Actions column, click Rolling Upgrade. In the Confirm to perform rolling upgrade? dialog box, click OK to upgrade the ASM gateway to the new version 1.23.
- Upgrade a workload: In the Workload to be upgraded section, select a Namespace. Find the target workload. In the Actions column, click Rolling Upgrade. In the Confirm to perform rolling upgrade? dialog box, click OK to upgrade the workload to the new version 1.23.
  
  Note
  ASM gateways or workloads that have been upgraded are not shown in the list.
  
  You can also click Instances Status in the navigation pane on the left to view all workloads or gateway instances that have not been upgraded.

Step 8: Unpublish the old version

After upgrading all data plane workloads, you can unpublish the old version 1.22.

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Upgrade Management.
On the Upgrade Management page, on the Canary Upgrade tab, click Offline old version. In the Are you sure to offline the previous version of the Control Plane? dialog box, click OK.