Alibaba Cloud Service Mesh:Integrate ASM with Argo Rollouts to implement canary releases

Preparations

Install Argo Rollouts

To install Argo Rollouts, perform the following steps. For more information, see Installation.

1. Run the following commands to install the Argo Rollouts server:

   kubectl create namespace argo-rollouts
   kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

2. Run the following command to install the Argo Rollouts kubectl plug-in.
   The Argo Rollouts kubectl plug-in can provide kubectl-based management.

   brew install argoproj/tap/kubectl-argo-rollouts

Enable access to the Istio resources of the ASM instance on the data plane by using the Kubernetes API

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Base Information.
3. Click Enable to the right of Enable Data-plane KubeAPI access.
4. In the message that appears, click OK.

Implement a canary release

The following example shows you how to create a stable version and a canary release version and how to progressively switch traffic from the stable version to the canary release version to implement a canary release based on the traffic ratio. For more information about a canary release, see Configure a canary release.

Step 1: Create the Rollout and Service applications

1. Create a Rollout.
   1. Create the rollout.yaml file that contains the following content:

      apiVersion: argoproj.io/v1alpha1
      kind: Rollout
      metadata:
        name: istio-rollout
      spec:
        revisionHistoryLimit: 2
        selector:
          matchLabels:
            app: istio-rollout
        template:
          metadata:
            annotations:
              sidecar.istio.io/inject: "true"
            labels:
              app: istio-rollout
          spec:
            containers:
            - name: istio-rollout
              image: argoproj/rollouts-demo:blue
              ports:
              - name: http
                containerPort: 8080
                protocol: TCP
              resources:
                requests:
                  memory: 32Mi
                  cpu: 5m
        strategy:
          canary:
            canaryService: istio-rollout-canary
            stableService: istio-rollout-stable
            trafficRouting:
              istio:
                virtualService:
                  name: istio-rollout-vsvc
                  routes:
                  - primary
            steps:
            - setWeight: 10
            - pause: {}         # Manually update the Rollout. 
            - setWeight: 20
            - pause: {duration: 20s}
            - setWeight: 30
            - pause: {duration: 20s}
            - setWeight: 40
            - pause: {duration: 20s}
            - setWeight: 50
            - pause: {duration: 20s}
            - setWeight: 60
            - pause: {duration: 20s}
            - setWeight: 70
            - pause: {duration: 20s}
            - setWeight: 80
            - pause: {duration: 20s}
            - setWeight: 90
            - pause: {duration: 20s}

      The strategy field defines the canary release policy. The following content describes the setWeight and pause parameters under the strategy field:

      • setWeight: the weight of traffic to be routed to the canary release version.
      • pause: pauses the Rollout. If no duration is specified for a pause step, the Rollout needs to be manually updated. If the duration is specified for a pause step, the system automatically updates the Rollout after the specified duration ends.
   2. Run the following command to deploy the Rollout to the cluster that is added to your ASM instance:

      kubectl apply -f rollout.yaml

2. Create a service.
   1. Create the service.yaml file that contains the following content:

      apiVersion: v1
      kind: Service
      metadata:
        name: istio-rollout-canary
      spec:
        ports:
        - port: 80
          targetPort: http
          protocol: TCP
          name: http
        selector:
          app: istio-rollout
      
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: istio-rollout-stable
      spec:
        ports:
        - port: 80
          targetPort: http
          protocol: TCP
          name: http
        selector:
          app: istio-rollout

   2. Run the following command to deploy the service to the cluster that is added to your ASM instance:

      kubectl apply -f service.yaml

Step 2: Create related Istio resources

1. Create a virtual service named istio-rollout-vsvc.
   You have enabled access to the Istio resources of the ASM instance on the data plane by using the Kubernetes API. Therefore, you can use the kubeconfig file on the data plane to access Istio resources such as virtual services, Istio gateways, and destination rules. Alternatively, you can use the ASM console or a kubeconfig file to create Istio resources.
   1. Create the istio-rollout-vsvc.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1alpha3
      kind: VirtualService
      metadata:
        name: istio-rollout-vsvc
      spec:
        gateways:
          - istio-rollout-gateway
        hosts:
          - '*'
        http:
          - match:
              - uri:
                  prefix: /
            name: primary
            route:
              - destination:
                  host: istio-rollout-stable
                weight: 100
              - destination:
                  host: istio-rollout-canary

   2. Run the following command to deploy istio-rollout-vsvc:

      kubectl apply -f istio-rollout-vsvc.yaml

2. Create an Istio gateway named istio-rollout-gateway.
   1. Create the istio-rollout-gateway.yaml file that contains the following content:

      apiVersion: networking.istio.io/v1beta1
      kind: Gateway
      metadata:
        name: istio-rollout-gateway
      spec:
        selector:
          istio: ingressgateway
        servers:
          - hosts:
              - '*'
            port:
              name: http
              number: 80
              protocol: HTTP
                                      

   2. Run the following command to deploy istio-rollout-gateway:

      kubectl apply -f istio-rollout-gateway.yaml

Step 3: Create an ingress gateway

Create an ingress gateway for which port 80 is enabled for service access test.

1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
3. On the Ingress Gateway page, click Create. Set the parameters of the ingress gateway and click Create.
   The following table describes only several parameters of the ingress gateway. For more information about the parameters of an ingress gateway, see Create an ingress gateway service.

   Parameter	Description
   Name	The name of the ingress gateway. For this example, set the parameter to ingressgateway.
   Gateway types	The type of the ingress gateway. For this example, set the parameter to North-South IngressGateway.
   Port Mapping	The port mappings of the ingress gateway. Click Add Port. In the row that appears, set the Protocol parameter to HTTP and the Service Port parameter to 80.

Step 4: View the status of the Rollout

Run the following command to view the status of the Rollout:

kubectl argo rollouts get rollout istio-rollout

kubectl argo rollouts get rollout istio-rollout
Name:            istio-rollout
Namespace:       default
Status:           Healthy
Strategy:        Canary
  Step:          18/18
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       1
  Current:       1
  Updated:       1
  Ready:         1
  Available:     1

NAME                                       KIND        STATUS     AGE  INFO
⟳ istio-rollout                            Rollout      Healthy  52s
└──# revision:1
   └──⧉ istio-rollout-7f96d86486           ReplicaSet   Healthy  52s  stable
      └──□ istio-rollout-7f96d86486-vpqvb  Pod          Running  52s  ready:2/2

Step 5: Test the initial status of the Rollout

1. Obtain the IP address of the ingress gateway.
   1. Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
   2. On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Gateways > Ingress Gateway.
   3. Obtain Service address of the ingress gateway.
2. Use a browser to access http://{IP address of the ingress gateway}/.
   The following figure shows the access result. Concurrent calls to http://{IP address of the ingress gateway}/color are invoked on this page to fill the grids with the obtained color. The color specified in Rollout istio-rollout is blue, and the canary release is not started. Therefore, the blue color is displayed.

Step 6: Update the Rollout

In this example, yellow is used to indicate a canary release version. When the color of the grids on the http://{IP address of the ingress gateway}/ page changes from blue to yellow, a canary release is implemented.

1. Update the image version of the container for the Rollout.
   1. Run the following command to update the image version:

      kubectl argo rollouts set image istio-rollout "*=argoproj/istio-rollout:yellow"

   2. View the image versions of corresponding containers.
      1. Log on to the ACK console and click Clusters in the left-side navigation pane.
      2. On the Clusters page, click the name of a cluster and choose Workloads > Pods in the left-side navigation pane.
      3. In the Name column, view the image versions of corresponding containers.

         The pod in which the yellow version (canary release version) runs is created, but the pod in which the blue version (stable version) runs also exists, as shown in the following figure.

         
2. Use a browser to access http://{IP address of the ingress gateway}/.
   The result shows that 10% of the blue grids are changed to yellow, as shown in the following figure. This is because the traffic weight of the configured virtual service changes. Specifically, the traffic weight of the stable version (blue version) changes from 100 to 90, whereas the traffic weight of the canary release version (yellow version) changes from 0 to 10. The Rollout controls the traffic weight of the virtual service. The traffic weight that you specified for the canary release version in the first step is set to 10 when the Rollout is created. Therefore, when the Rollout starts, the Argo Rollouts controller changes the traffic weight configured for the virtual service in the Rollout accordingly. In addition, the pause parameter is left empty. This means that the Rollout needs to be manually confirmed to proceed to the next stage.
3. Continue the canary release.
   1. Run the following command to continue the canary release:

      kubectl argo rollouts promote istio-rollout

   2. Use a browser to access http://{IP address of the ingress gateway}/.
      The following figure shows the result. The traffic weight of the virtual service continues to be adjusted based on the configuration in the Rollout. In this stage, the pause duration is specified. For more information, see Step 1. Therefore, the system automatically adjusts the traffic weight of the virtual service after the specified pause duration ends.
4. Check whether the canary release is successful.
   1. Wait for a while. Then, use a browser to access http://{IP address of the ingress gateway}/.
      All blue grids are changed to yellow, as shown in the following figure.
   2. Run the following command to view the status of the Rollout:

      kubectl argo rollouts get rollout istio-rollout --watch

      Expected output:

      Name:            istio-rollout
      Namespace:       default
      Status:           Healthy
      Strategy:        Canary
        Step:          18/18
        SetWeight:     100
        ActualWeight:  100
      Images:          argoproj/rollouts-demo:yellow (stable)
      Replicas:
        Desired:       1
        Current:       1
        Updated:       1
        Ready:         1
        Available:     1
      
      NAME                                       KIND        STATUS        AGE  INFO
      ⟳ istio-rollout                            Rollout      Healthy     48m
      ├──# revision:4
      │  └──⧉ istio-rollout-5fcf5864c4           ReplicaSet   Healthy     27m  stable
      │     └──□ istio-rollout-5fcf5864c4-vw6kh  Pod          Running     26m  ready:2/2
      ├──# revision:3
      │  └──⧉ istio-rollout-897cb5b6d            ReplicaSet  • ScaledDown  27m
      └──# revision:1
         └──⧉ istio-rollout-7f96d86486           ReplicaSet  • ScaledDown  48m

      The output shows that the image version of the container for the Rollout is updated to yellow.

Use Prometheus to implement automatic rollback

During the canary release, you can run the kubectl argo rollouts abort istio-rollout command to roll back to the stable version. You can also use a Prometheus system to monitor the health status of an application involved in the canary release. If a metric anomaly occurs, the canary release version is automatically rolled back to the stable version and is marked as Degraded.

1. Enable Prometheus in ASM. For more information, see Monitor service meshes based on ARMS Prometheus or Monitor ASM instances by using a self-managed Prometheus instance.
2. Configure an Argo AnalysisTemplate.
   1. Create the istio-success-rate.yaml file that contains the following content.
      Set the address parameter of the AnalysisTemplate to the endpoint of the Prometheus instance that is connected to ASM.

      apiVersion: argoproj.io/v1alpha1
      kind: AnalysisTemplate
      metadata:
        name: istio-success-rate
      spec:
        args:
        - name: service
        - name: namespace
        metrics:
        - name: success-rate
          initialDelay: 60s
          interval: 20s
          successCondition: result[0] > 0.90
          provider:
            prometheus:
              address: http://xxx.aliyuncs.com:9090/api/v1/prometheus/
              query: >+
                sum(irate(istio_requests_total{
                  reporter="source",
                  destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local",
                  response_code!~"5.*"}[40s])
                )
                /
                sum(irate(istio_requests_total{
                  reporter="source",
                  destination_service=~"{{args.service}}.{{args.namespace}}.svc.cluster.local"}[40s])
                )

   2. Run the following command to deploy the Argo AnalysisTemplate:

      kubectl apply -f istio-success-rate.yaml

3. Associate an Analysis with the Rollout.
   1. Create the rollout.yaml file that contains the following content.
      Set the analysis parameter under strategy so that the Analysis can be used for monitoring and automatic rollback from the second step. The initial image version of the container for the Rollout is yellow.

      apiVersion: argoproj.io/v1alpha1
      kind: Rollout
      metadata:
        name: istio-rollout
      spec:
        revisionHistoryLimit: 2
        selector:
          matchLabels:
            app: istio-rollout
        template:
          metadata:
            annotations:
              sidecar.istio.io/inject: "true"
            labels:
              app: istio-rollout
          spec:
            containers:
            - name: istio-rollout
              image: argoproj/rollouts-demo:yellow
              ports:
              - name: http
                containerPort: 8080
                protocol: TCP
              resources:
                requests:
                  memory: 32Mi
                  cpu: 5m
        strategy:
          canary:
            canaryService: istio-rollout-canary
            stableService: istio-rollout-stable
            analysis:
              startingStep: 1
              templates:
              - templateName: istio-success-rate
              args:
              - name: service
                value: canary
              - name: namespace
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
            trafficRouting:
              istio:
                virtualService:
                  name: istio-rollout-vsvc
                  routes:
                  - primary
            steps:
            - setWeight: 10
            - pause: {}         # Manually update the Rollout. 
            - setWeight: 20
            - pause: {duration: 20s}
            - setWeight: 30
            - pause: {duration: 20s}
            - setWeight: 40
            - pause: {duration: 20s}
            - setWeight: 50
            - pause: {duration: 20s}
            - setWeight: 60
            - pause: {duration: 20s}
            - setWeight: 70
            - pause: {duration: 20s}
            - setWeight: 80
            - pause: {duration: 20s}
            - setWeight: 90
            - pause: {duration: 20s}

   2. Run the following command to update the Rollout:

      kubectl apply -f rollout.yaml

4. Run the following command to update the image version:

   kubectl argo rollouts set image istio-rollout "*=argoproj/rollouts-demo:orange"

   Use a browser to access http://{IP address of the ingress gateway}/. The following figure shows the result.
5. Manually confirm the Rollout update.
   1. Run the following command to continue the canary release.
      After you run the command, the automatic canary release starts. In addition, Prometheus monitoring is enabled from the second step. In this case, if the canary release version has an error rate higher than 90%, rollback is triggered.

      kubectl argo rollouts promote istio-rollout

   2. Run the following command to view the status of the Rollout:

      kubectl argo rollouts get rollout istio-rollout --watch

      Expected output:
6. Configure the error rate.
   In subsequent progressive releases, you can manually adjust the error rate for canary release versions. If the error progress bar reaches 100%, all orange grids (canary release versions) are framed in red to indicate an error. After a while, all canary release versions are automatically rolled back to their stable versions (yellow grids).