All Products
Search
Document Center

Container Service for Kubernetes:Use Kruise Rollout to implement phased releases (canary & A/B testing)

Last Updated:Dec 04, 2025

Kruise Rollout is a standard extension for Kubernetes. It works with native workloads, such as deployments and StatefulSets, and OpenKruise workloads, such as CloneSets and Advanced StatefulSets. Kruise Rollout provides features such as canary releases, A/B testing, and phased releases. This topic uses examples to describe how to use Kruise Rollout to perform phased releases for cloud-native applications.

Prerequisites

  • A Kubernetes cluster has been created. For more information, see Create an ACK managed cluster.

    • To use the A/B testing or canary release feature, the cluster version must be 1.19 or later.

    • To use the phased release feature, the cluster version must be 1.16 or later.

  • kubectl-kruise has been installed. For more information about how to install kubectl-kruise, see kubectl-kruise.

Introduction to Kruise Rollout

Kruise Rollout is an open source progressive delivery framework from the OpenKruise community. Kruise Rollout supports phased releases, blue-green deployments, and A/B testing that coordinate with traffic and instance scaling. Based on Prometheus metrics, Kruise Rollout can also automate batching and pausing during the release process. It provides seamless bypass integration and is compatible with various existing workloads, such as deployments, CloneSets, and StatefulSets. For more information, see Kruise Rollout.

To use Kruise Rollout, you must configure a Rollout resource and apply it to your Kubernetes cluster. No extra operations are required for subsequent application releases and upgrades. Kruise Rollout can be seamlessly and cost-effectively integrated with platforms such as Helm and Platform as a Service (PaaS). The following figure shows the architecture for performing a phased release using Kruise Rollout.gray

Preparations

  1. Install the Kruise Rollout add-on.

    1. Log on to the ACK console. In the navigation pane on the left, choose Clusters.

    2. On the Clusters page, click the name of the target cluster. In the navigation pane on the left, click Add-ons.

    3. On the Add-ons page, click the Manage Applications tab, then in the lower-right corner of the ack-kruise card, click Install.

    4. In the dialog box, confirm the information and click OK.

    Note

    The ack-kruise add-on version 1.8 or later supports the v1beta1 API. For more information, see API Specifications.

  2. Deploy the business application (deployment and Service).

    Note

    The business application configuration deploys an echoserver service based on a deployment and exposes the service through an Nginx Ingress.

    1. Create an echoserver.yaml file.

      Expand to view the YAML content

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: echoserver
        labels:
          app: echoserver
      spec:
        replicas: 6
        selector:
          matchLabels:
            app: echoserver
        template:
          metadata:
            labels:
              app: echoserver
          spec:
            containers:
            - name: echoserver
              image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.2
              imagePullPolicy: IfNotPresent
              ports:
              - containerPort: 8080
              env:
              - name: NODE_NAME
                value: version1
              - name: PORT
                value: '8080'
              - name: POD_NAME
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.name
              - name: POD_NAMESPACE
                valueFrom:
                  fieldRef:
                    fieldPath: metadata.namespace
              - name: POD_IP
                valueFrom:
                  fieldRef:
                    fieldPath: status.podIP
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: echoserver
        labels:
          app: echoserver
      spec:
        ports:
        - port: 80
          targetPort: 8080
          protocol: TCP
          name: http
        selector:
          app: echoserver

The following sections provide examples of how to perform canary releases, A/B testing, and phased releases.

Use case 1: Implement canary or A/B testing releases based on Ingress

Nginx Ingress and MSE Ingress are commonly used to expose services. This example shows how to use Kruise Rollout with Nginx Ingress or MSE Ingress to implement a canary release or A/B testing.

  1. Install the Ingress controller and create an Ingress for the business application.

    Nginx Ingress Controller

    1. Install the Nginx Ingress Controller.

    2. Create an echoserver-ingress.yaml file.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: echoserver
      spec:
        ingressClassName: nginx
        rules:
        - http:
            paths:
            - backend:
                service:
                  name: echoserver
                  port:
                    number: 80
              path: /apis/echo
              pathType: Exact
    3. Deploy the Ingress for the business application.

      kubectl apply -f echoserver-ingress.yaml

    MSE Ingress Controller

    1. Install the MSE Ingress Controller and create an MseIngressConfig and an IngressClass. For more information, see Access container services through an MSE Ingress.

    2. Create an echoserver-ingress.yaml file.

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: echoserver
      spec:
        # You must set ingressClassName to mse.
        ingressClassName: mse
        rules:
          - http:
              paths:
                - backend:
                    service:
                      name: echoserver
                      port:
                        number: 80
                  path: /apis/echo
                  pathType: Exact
    3. Deploy the Ingress for the business application.

      kubectl apply -f echoserver-ingress.yaml
  2. Verify access.

    1. Retrieve the external IP address.

      Nginx Ingress

      export EXTERNAL_IP=$(kubectl get ingress echoserver -o jsonpath="{.status.loadBalancer.ingress[0].ip}" )

      MSE Ingress

      export EXTERNAL_IP=$(kubectl get ingress echoserver -o jsonpath="{.status.loadBalancer.ingress[0].hostname}" )
    2. Test access.

      curl http://${EXTERNAL_IP}/apis/echo

      Expected output:

      Hostname: echoserver-75d49c475c-ls2bs
      
      Pod Information:
          node name:    version1
          pod name:    echoserver-75d49c475c-ls2bs
          pod namespace:    default
      
      Server values:
          server_version=nginx: 1.13.3 - lua: 10008
      ...
  3. Define the Kruise Rollout phased release rule.

    The following Rollout resource defines a phased release rule. The release is divided into three batches:

    Canary

    • First batch: A canary release where 20% of the traffic is routed to the new version and the rest is routed to the old version.

    • Second batch: A phased release based on traffic percentage. In this batch, 50% of the instances and traffic are released.

    • Third batch: A full release of all instances.

    A/B test

    • First batch: An A/B testing release where traffic matching header[User-Agent]=Android is routed to the new version, and all other traffic is routed to the old version.

    • Second batch: A phased release based on the pod percentage. In this batch, 50% of the instances are released.

    • Third batch: A full release of all instances.

    1. Create a rollout.yaml file with the following content.

      Canary

      apiVersion: rollouts.kruise.io/v1alpha1
      kind: Rollout
      metadata:
        name: rollouts-demo
      spec:
        objectRef:
          workloadRef:
            apiVersion: apps/v1
            kind: Deployment
            name: echoserver
        strategy:
          canary:
            steps:
            - weight: 20
              replicas: 1
              pause: {}
            - weight: 50
              replicas: 50%
              pause: {duration: 60}
            - weight: 100
              replicas: 100%
              pause: {duration: 60}
            trafficRoutings:
            - service: echoserver
              ingress:
                name: echoserver
      

      A/B test

      apiVersion: rollouts.kruise.io/v1alpha1
      kind: Rollout
      metadata:
        name: rollouts-demo
      spec:
        objectRef:
          workloadRef:
            apiVersion: apps/v1
            kind: Deployment
            name: echoserver
        strategy:
          canary:
            steps:
              # Stage 1: 1 pod, matches Android Traffic
              - matches:
                  - headers:
                      - type: Exact
                        name: User-Agent
                        value: Android
                pause: {}
                replicas: 1
              # Stage 2: 50% pods, automatically pauses for 60 seconds
              - matches:
                  - headers:
                      - type: Exact
                        name: User-Agent
                        value: Android
                pause: {duration: 60}
                replicas: 50%
              # Stage 3: 100% pods that match the Traffic
              - matches:
                  - headers:
                      - type: Exact
                        name: User-Agent
                        value: Android
                pause: {duration: 60}
                replicas: 100%
            trafficRoutings:
              - service: echoserver
                ingress:
                  name: echoserver
      
    2. Run the following command to apply the Rollout resource to the cluster.

      kubectl apply -f rollout.yaml
    3. Run the following command to check the status of the Rollout resource.

      kubectl get rollout

      Expected output:

      NAME            STATUS    CANARY_STEP   CANARY_STATE   MESSAGE                            AGE
      rollouts-demo   Healthy   3             Completed      workload deployment is completed   7s                              rollout is healthy   32s

      The expected output STATUS=Healthy indicates that the Rollout resource is healthy.

  4. Upgrade the application version.

    Kruise Rollout is a persistent configuration. After you apply it to the cluster, you only need to adjust the deployment configuration for subsequent application releases. No other operations are required for Kruise Rollout. For example, to upgrade the image version of the echoserver service to 1.10.3, run the kubectl apply -f echoserver.yaml command to update the deployment in the cluster. In addition to kubectl, you can also use tools such as Helm and Vela to apply the deployment configuration to the Kubernetes cluster.

    1. Modify the echoserver.yaml file to upgrade the image version of the echoserver service to 1.10.3.

      # echoserver.yaml
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: echoserver
      ...
      spec:
        ...
        containers:
        - name: echoserver
          # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm
          image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.3
          imagePullPolicy: IfNotPresent
              env:
          - name: NODE_NAME
            # Optional. To clearly show the phased release effect, change the value to version2.
            value: version2
                                      
    2. Run the following command to check the status of the Rollout resource.

      kubectl get rollouts rollouts-demo -n default

      Expected output:

      NAME            STATUS        CANARY_STEP   CANARY_STATE   MESSAGE                                                                         AGE
      rollouts-demo   Progressing   1             StepPaused     Rollout is in step(1/3), and you need manually confirm to enter the next step   41m

      The STATUS and CANARY fields in the expected output show the status and progress of the rollout.

      • An output of STATUS=Progressing indicates that the canary release is in progress.

      • An output of CANARY_STEP=1 means the release is in the first batch.

      • If the output is CANARY_STATE=StepPaused, the current batch is complete. You must manually confirm to proceed.

    3. Verify the traffic distribution between the old and new versions.

      Canary

      1. Access the service 10 consecutive times and check the returned node name value.

        for i in {1..10}; do curl -s http://${EXTERNAL_IP}/apis/echo | grep 'node name'; sleep 1; done  

        Expected output:

                node name:      version1
                node name:      version1
                node name:      version2
                node name:      version1
                node name:      version2
                node name:      version1
                node name:      version1
                node name:      version1
                node name:      version1
                node name:      version1

        The output shows that the traffic ratio of version 1 to version 2 is approximately 8:2. This meets the weight expectation of the first batch.

      2. Manually proceed to the next batch.

        kubectl-kruise rollout approve rollouts/rollouts-demo -n default
      3. Continuously check the rollout status.

        kubectl get rollouts rollouts-demo -n default -w

        Expected output:

        NAME            STATUS        CANARY_STEP   CANARY_STATE         MESSAGE                                                        AGE
        rollouts-demo   Progressing   2             StepTrafficRouting   Rollout is in step(2/3), and upgrade workload to new version   31m
        rollouts-demo   Progressing   2             StepMetricsAnalysis   Rollout is in step(2/3), and upgrade workload to new version   31m
        rollouts-demo   Progressing   2             StepPaused            Rollout is in step(2/3), and upgrade workload to new version   31m
        rollouts-demo   Progressing   2             StepPaused            Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   31m
        rollouts-demo   Progressing   2             StepReady             Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   32m
        rollouts-demo   Progressing   3             BeforeStepUpgrade     Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepTrafficRouting    Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepMetricsAnalysis   Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepPaused            Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             StepReady             Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             Completed             Rollout is in step(3/3), and upgrade workload to new version                    32m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  32m
        rollouts-demo   Progressing   3             Completed             Rollout progressing has been completed                                          33m
        rollouts-demo   Healthy       3             Completed             Rollout progressing has been completed                                          33m

        The output shows that after you run the approve command, the Rollout resource proceeds to the second batch, and then automatically proceeds to the third batch after the waiting period. Finally, the status changes to STATUS=Healthy and CANARY_STATE=Completed, which indicates that the rollout is complete.

      A/B test

      1. You can access the application with and without the header[User-Agent]=Android request header.

        curl -s http://${EXTERNAL_IP}/apis/echo |grep 'Pod Information:' -A 3
        curl -sH "User-Agent: Android" http://${EXTERNAL_IP}/apis/echo | grep 'Pod Information:' -A 3

        Expected output:

        Pod Information:
                node name:      version1
                pod name:       echoserver-69598f9458-7c66v
                pod namespace:  default
        Pod Information:
                node name:      version2
                pod name:       echoserver-fvhzg-687b4b56-qbhc8
                pod namespace:  default

        The output shows that the two requests return version1 and version2. This indicates that the request header-based routing rule has taken effect.

      2. Manually proceed to the next batch.

        kubectl-kruise rollout approve rollouts/rollouts-demo -n default
      3. Continuously check the rollout status.

        kubectl get rollouts rollouts-demo -n default -w

        Expected output:

        NAME            STATUS        CANARY_STEP   CANARY_STATE         MESSAGE                                                        AGE
        rollouts-demo   Progressing   2             StepTrafficRouting   Rollout is in step(2/3), and upgrade workload to new version   26m
        rollouts-demo   Progressing   2             StepMetricsAnalysis   Rollout is in step(2/3), and upgrade workload to new version   26m
        rollouts-demo   Progressing   2             StepPaused            Rollout is in step(2/3), and upgrade workload to new version   26m
        rollouts-demo   Progressing   2             StepPaused            Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   26m
        rollouts-demo   Progressing   2             StepReady             Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   27m
        rollouts-demo   Progressing   3             BeforeStepUpgrade     Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepUpgrade           Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepTrafficRouting    Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepMetricsAnalysis   Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepPaused            Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             StepReady             Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             Completed             Rollout is in step(3/3), and upgrade workload to new version                    27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  27m
        rollouts-demo   Progressing   3             Completed             Rollout progressing has been completed                                          27m
        rollouts-demo   Healthy       3             Completed             Rollout progressing has been completed                                          27m

        The output shows that after you run the approve command, the Rollout resource proceeds to the second batch, and then automatically proceeds to the third batch after the waiting period. Finally, the status changes to STATUS=Healthy and CANARY_STATE=Completed, which indicates that the rollout is complete.

  5. (Optional) If the new version is abnormal, roll back the application.

    If the new version is abnormal during the rollout, you can restore the previous version using the deployment configuration. Then, run the kubectl apply -f echoserver.yaml command to redeploy it. You do not need to modify the Rollout resource.

    # echoserver.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echoserver
    ...
    spec:
      ...
      containers:
      - name: echoserver
        # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm.
        image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.2
        imagePullPolicy: IfNotPresent
            env:
        - name: NODE_NAME
          value: version1

Use case 2: Phased release based on the number of pod instances (for applications that use microservice frameworks such as Nacos)

Most applications that use a microservice framework, such as Nacos, do not require a corresponding Service or Ingress configuration. The traffic rerouting feature is already integrated into the microservice framework. For this type of application, the phased release feature of Kruise Rollout is more suitable.

Note

Because the microservice framework handles traffic shifting for the phased release, this scenario does not include the verification of traffic distribution between the old and new versions. It only demonstrates how to switch between rollout batches.

  1. Define and deploy the Kruise Rollout phased release rule.

    The following Rollout resource defines a phased release rule that does not require you to configure the trafficRoutings field. The release is divided into three batches:

    • First batch: Release one pod.

    • Second batch: Release 50% of the pods.

    • Third batch: Release all remaining instances.

    # Save the following content to a file named rollout.yaml.
    apiVersion: rollouts.kruise.io/v1alpha1
    kind: Rollout
    metadata:
      name: rollouts-demo
      annotations:
        rollouts.kruise.io/rolling-style: partition
    spec:
      objectRef:
        workloadRef:
          apiVersion: apps/v1
          kind: Deployment
          # Deployment Name
          name: echoserver
      strategy:
        canary:
          steps:
          # Step 1: Update one pod, then pause and wait for manual confirmation.
          - replicas: 1
            pause: {} # Manually decide whether to proceed to the next batch.
          # Step 2: Update 50% of the pod instances.
          - replicas: 50%
            # Automatically proceed to the next batch after a 60-second pause.
            pause: {duration: 60}
          # Step 3: Full release. Update all pods to the new version.
          - replicas: 100%
            pause: {duration: 60}
  2. Modify the echoserver.yaml file to upgrade the image version of the echoserver service to 1.10.3.

    # echoserver.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: echoserver
    ...
    spec:
      ...
      containers:
      - name: echoserver
        # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm
        image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.3
        imagePullPolicy: IfNotPresent
            env:
        - name: NODE_NAME
          # Optional. To clearly show the phased release effect, change the value to version2.
          value: version2
                                    
  3. Check the status of the Rollout resource.

    kubectl get rollouts rollouts-demo -n default

    Expected output:

    NAME            STATUS        CANARY_STEP   CANARY_STATE   MESSAGE                                                                         AGE
    rollouts-demo   Progressing   1             StepPaused     Rollout is in step(1/3), and you need manually confirm to enter the next step   41m

    The STATUS and CANARY fields in the expected output show the progress of the rollout.

    • An output of STATUS=Progressing indicates that the canary release is in progress.

    • An output of CANARY_STEP=1 indicates that the release is in the first batch.

    • An output of CANARY_STATE=StepPaused indicates that the current batch is complete and you must manually confirm whether to proceed.

  4. Manually proceed to the next batch.

    kubectl-kruise rollout approve rollouts/rollouts-demo -n default
  5. Continuously check the rollout status.

    kubectl get rollouts rollouts-demo -n default -w

    Expected output:

    NAME            STATUS        CANARY_STEP   CANARY_STATE   MESSAGE                                                                         AGE
    rollouts-demo   Progressing   2             StepPaused     Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   45m
    rollouts-demo   Progressing   2             StepReady      Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   45m
    rollouts-demo   Progressing   3             BeforeStepUpgrade   Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step   45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    45m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepUpgrade         Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepMetricsAnalysis   Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepPaused            Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             StepReady             Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             Completed             Rollout is in step(3/3), and upgrade workload to new version                    46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout has been completed and some closing work is being done                  46m
    rollouts-demo   Progressing   3             Completed             Rollout progressing has been completed                                          46m
    rollouts-demo   Healthy       3             Completed             Rollout progressing has been completed                                          46m