Kruise Rollout is a standard extension for Kubernetes. It works with native workloads, such as deployments and StatefulSets, and OpenKruise workloads, such as CloneSets and Advanced StatefulSets. Kruise Rollout provides features such as canary releases, A/B testing, and phased releases. This topic uses examples to describe how to use Kruise Rollout to perform phased releases for cloud-native applications.
Prerequisites
A Kubernetes cluster has been created. For more information, see Create an ACK managed cluster.
To use the A/B testing or canary release feature, the cluster version must be 1.19 or later.
To use the phased release feature, the cluster version must be 1.16 or later.
kubectl-kruise has been installed. For more information about how to install kubectl-kruise, see kubectl-kruise.
Introduction to Kruise Rollout
Kruise Rollout is an open source progressive delivery framework from the OpenKruise community. Kruise Rollout supports phased releases, blue-green deployments, and A/B testing that coordinate with traffic and instance scaling. Based on Prometheus metrics, Kruise Rollout can also automate batching and pausing during the release process. It provides seamless bypass integration and is compatible with various existing workloads, such as deployments, CloneSets, and StatefulSets. For more information, see Kruise Rollout.
To use Kruise Rollout, you must configure a Rollout resource and apply it to your Kubernetes cluster. No extra operations are required for subsequent application releases and upgrades. Kruise Rollout can be seamlessly and cost-effectively integrated with platforms such as Helm and Platform as a Service (PaaS). The following figure shows the architecture for performing a phased release using Kruise Rollout.
Preparations
Install the Kruise Rollout add-on.
Log on to the ACK console. In the navigation pane on the left, choose Clusters.
On the Clusters page, click the name of the target cluster. In the navigation pane on the left, click Add-ons.
On the Add-ons page, click the Manage Applications tab, then in the lower-right corner of the ack-kruise card, click Install.
In the dialog box, confirm the information and click OK.
NoteThe ack-kruise add-on version 1.8 or later supports the v1beta1 API. For more information, see API Specifications.
Deploy the business application (deployment and Service).
NoteThe business application configuration deploys an
echoserverservice based on a deployment and exposes the service through an Nginx Ingress.Create an echoserver.yaml file.
The following sections provide examples of how to perform canary releases, A/B testing, and phased releases.
Use case 1: Implement canary or A/B testing releases based on Ingress
Nginx Ingress and MSE Ingress are commonly used to expose services. This example shows how to use Kruise Rollout with Nginx Ingress or MSE Ingress to implement a canary release or A/B testing.
Install the Ingress controller and create an Ingress for the business application.
Nginx Ingress Controller
Install the Nginx Ingress Controller.
New cluster: During cluster creation, select Nginx Ingress in the Ingress configuration section. For more information, see Create an ACK managed cluster.
For an existing cluster: To install the Nginx Ingress Controller, see Create and use an Nginx Ingress to expose services.
Create an echoserver-ingress.yaml file.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echoserver spec: ingressClassName: nginx rules: - http: paths: - backend: service: name: echoserver port: number: 80 path: /apis/echo pathType: ExactDeploy the Ingress for the business application.
kubectl apply -f echoserver-ingress.yaml
MSE Ingress Controller
Install the MSE Ingress Controller and create an MseIngressConfig and an IngressClass. For more information, see Access container services through an MSE Ingress.
Create an echoserver-ingress.yaml file.
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echoserver spec: # You must set ingressClassName to mse. ingressClassName: mse rules: - http: paths: - backend: service: name: echoserver port: number: 80 path: /apis/echo pathType: ExactDeploy the Ingress for the business application.
kubectl apply -f echoserver-ingress.yaml
Verify access.
Retrieve the external IP address.
Nginx Ingress
export EXTERNAL_IP=$(kubectl get ingress echoserver -o jsonpath="{.status.loadBalancer.ingress[0].ip}" )MSE Ingress
export EXTERNAL_IP=$(kubectl get ingress echoserver -o jsonpath="{.status.loadBalancer.ingress[0].hostname}" )Test access.
curl http://${EXTERNAL_IP}/apis/echoExpected output:
Hostname: echoserver-75d49c475c-ls2bs Pod Information: node name: version1 pod name: echoserver-75d49c475c-ls2bs pod namespace: default Server values: server_version=nginx: 1.13.3 - lua: 10008 ...
Define the Kruise Rollout phased release rule.
The following Rollout resource defines a phased release rule. The release is divided into three batches:
Canary
First batch: A canary release where 20% of the traffic is routed to the new version and the rest is routed to the old version.
Second batch: A phased release based on traffic percentage. In this batch, 50% of the instances and traffic are released.
Third batch: A full release of all instances.
A/B test
First batch: An A/B testing release where traffic matching
header[User-Agent]=Androidis routed to the new version, and all other traffic is routed to the old version.Second batch: A phased release based on the pod percentage. In this batch, 50% of the instances are released.
Third batch: A full release of all instances.
Create a rollout.yaml file with the following content.
Canary
apiVersion: rollouts.kruise.io/v1alpha1 kind: Rollout metadata: name: rollouts-demo spec: objectRef: workloadRef: apiVersion: apps/v1 kind: Deployment name: echoserver strategy: canary: steps: - weight: 20 replicas: 1 pause: {} - weight: 50 replicas: 50% pause: {duration: 60} - weight: 100 replicas: 100% pause: {duration: 60} trafficRoutings: - service: echoserver ingress: name: echoserverA/B test
apiVersion: rollouts.kruise.io/v1alpha1 kind: Rollout metadata: name: rollouts-demo spec: objectRef: workloadRef: apiVersion: apps/v1 kind: Deployment name: echoserver strategy: canary: steps: # Stage 1: 1 pod, matches Android Traffic - matches: - headers: - type: Exact name: User-Agent value: Android pause: {} replicas: 1 # Stage 2: 50% pods, automatically pauses for 60 seconds - matches: - headers: - type: Exact name: User-Agent value: Android pause: {duration: 60} replicas: 50% # Stage 3: 100% pods that match the Traffic - matches: - headers: - type: Exact name: User-Agent value: Android pause: {duration: 60} replicas: 100% trafficRoutings: - service: echoserver ingress: name: echoserverRun the following command to apply the Rollout resource to the cluster.
kubectl apply -f rollout.yamlRun the following command to check the status of the Rollout resource.
kubectl get rolloutExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Healthy 3 Completed workload deployment is completed 7s rollout is healthy 32sThe expected output
STATUS=Healthyindicates that the Rollout resource is healthy.
Upgrade the application version.
Kruise Rollout is a persistent configuration. After you apply it to the cluster, you only need to adjust the deployment configuration for subsequent application releases. No other operations are required for Kruise Rollout. For example, to upgrade the image version of the echoserver service to 1.10.3, run the
kubectl apply -f echoserver.yamlcommand to update the deployment in the cluster. In addition to kubectl, you can also use tools such as Helm and Vela to apply the deployment configuration to the Kubernetes cluster.Modify the echoserver.yaml file to upgrade the image version of the echoserver service to 1.10.3.
# echoserver.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echoserver ... spec: ... containers: - name: echoserver # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.3 imagePullPolicy: IfNotPresent env: - name: NODE_NAME # Optional. To clearly show the phased release effect, change the value to version2. value: version2Run the following command to check the status of the Rollout resource.
kubectl get rollouts rollouts-demo -n defaultExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Progressing 1 StepPaused Rollout is in step(1/3), and you need manually confirm to enter the next step 41mThe
STATUSandCANARYfields in the expected output show the status and progress of the rollout.An output of
STATUS=Progressingindicates that the canary release is in progress.An output of
CANARY_STEP=1means the release is in the first batch.If the output is
CANARY_STATE=StepPaused, the current batch is complete. You must manually confirm to proceed.
Verify the traffic distribution between the old and new versions.
Canary
Access the service 10 consecutive times and check the returned
node namevalue.for i in {1..10}; do curl -s http://${EXTERNAL_IP}/apis/echo | grep 'node name'; sleep 1; doneExpected output:
node name: version1 node name: version1 node name: version2 node name: version1 node name: version2 node name: version1 node name: version1 node name: version1 node name: version1 node name: version1The output shows that the traffic ratio of version 1 to version 2 is approximately 8:2. This meets the weight expectation of the first batch.
Manually proceed to the next batch.
kubectl-kruise rollout approve rollouts/rollouts-demo -n defaultContinuously check the rollout status.
kubectl get rollouts rollouts-demo -n default -wExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Progressing 2 StepTrafficRouting Rollout is in step(2/3), and upgrade workload to new version 31m rollouts-demo Progressing 2 StepMetricsAnalysis Rollout is in step(2/3), and upgrade workload to new version 31m rollouts-demo Progressing 2 StepPaused Rollout is in step(2/3), and upgrade workload to new version 31m rollouts-demo Progressing 2 StepPaused Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 31m rollouts-demo Progressing 2 StepReady Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 32m rollouts-demo Progressing 3 BeforeStepUpgrade Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepTrafficRouting Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepMetricsAnalysis Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepPaused Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 StepReady Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 Completed Rollout is in step(3/3), and upgrade workload to new version 32m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 32m rollouts-demo Progressing 3 Completed Rollout progressing has been completed 33m rollouts-demo Healthy 3 Completed Rollout progressing has been completed 33mThe output shows that after you run the
approvecommand, the Rollout resource proceeds to the second batch, and then automatically proceeds to the third batch after the waiting period. Finally, the status changes toSTATUS=HealthyandCANARY_STATE=Completed, which indicates that the rollout is complete.
A/B test
You can access the application with and without the
header[User-Agent]=Androidrequest header.curl -s http://${EXTERNAL_IP}/apis/echo |grep 'Pod Information:' -A 3 curl -sH "User-Agent: Android" http://${EXTERNAL_IP}/apis/echo | grep 'Pod Information:' -A 3Expected output:
Pod Information: node name: version1 pod name: echoserver-69598f9458-7c66v pod namespace: default Pod Information: node name: version2 pod name: echoserver-fvhzg-687b4b56-qbhc8 pod namespace: defaultThe output shows that the two requests return
version1andversion2. This indicates that the request header-based routing rule has taken effect.Manually proceed to the next batch.
kubectl-kruise rollout approve rollouts/rollouts-demo -n defaultContinuously check the rollout status.
kubectl get rollouts rollouts-demo -n default -wExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Progressing 2 StepTrafficRouting Rollout is in step(2/3), and upgrade workload to new version 26m rollouts-demo Progressing 2 StepMetricsAnalysis Rollout is in step(2/3), and upgrade workload to new version 26m rollouts-demo Progressing 2 StepPaused Rollout is in step(2/3), and upgrade workload to new version 26m rollouts-demo Progressing 2 StepPaused Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 26m rollouts-demo Progressing 2 StepReady Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 27m rollouts-demo Progressing 3 BeforeStepUpgrade Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepTrafficRouting Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepMetricsAnalysis Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepPaused Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 StepReady Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 Completed Rollout is in step(3/3), and upgrade workload to new version 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 27m rollouts-demo Progressing 3 Completed Rollout progressing has been completed 27m rollouts-demo Healthy 3 Completed Rollout progressing has been completed 27mThe output shows that after you run the
approvecommand, the Rollout resource proceeds to the second batch, and then automatically proceeds to the third batch after the waiting period. Finally, the status changes toSTATUS=HealthyandCANARY_STATE=Completed, which indicates that the rollout is complete.
(Optional) If the new version is abnormal, roll back the application.
If the new version is abnormal during the rollout, you can restore the previous version using the deployment configuration. Then, run the
kubectl apply -f echoserver.yamlcommand to redeploy it. You do not need to modify the Rollout resource.# echoserver.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echoserver ... spec: ... containers: - name: echoserver # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm. image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.2 imagePullPolicy: IfNotPresent env: - name: NODE_NAME value: version1
Use case 2: Phased release based on the number of pod instances (for applications that use microservice frameworks such as Nacos)
Most applications that use a microservice framework, such as Nacos, do not require a corresponding Service or Ingress configuration. The traffic rerouting feature is already integrated into the microservice framework. For this type of application, the phased release feature of Kruise Rollout is more suitable.
Because the microservice framework handles traffic shifting for the phased release, this scenario does not include the verification of traffic distribution between the old and new versions. It only demonstrates how to switch between rollout batches.
Define and deploy the Kruise Rollout phased release rule.
The following Rollout resource defines a phased release rule that does not require you to configure the
trafficRoutingsfield. The release is divided into three batches:First batch: Release one pod.
Second batch: Release 50% of the pods.
Third batch: Release all remaining instances.
# Save the following content to a file named rollout.yaml. apiVersion: rollouts.kruise.io/v1alpha1 kind: Rollout metadata: name: rollouts-demo annotations: rollouts.kruise.io/rolling-style: partition spec: objectRef: workloadRef: apiVersion: apps/v1 kind: Deployment # Deployment Name name: echoserver strategy: canary: steps: # Step 1: Update one pod, then pause and wait for manual confirmation. - replicas: 1 pause: {} # Manually decide whether to proceed to the next batch. # Step 2: Update 50% of the pod instances. - replicas: 50% # Automatically proceed to the next batch after a 60-second pause. pause: {duration: 60} # Step 3: Full release. Update all pods to the new version. - replicas: 100% pause: {duration: 60}Modify the echoserver.yaml file to upgrade the image version of the echoserver service to 1.10.3.
# echoserver.yaml apiVersion: apps/v1 kind: Deployment metadata: name: echoserver ... spec: ... containers: - name: echoserver # mac m1 can choice image e2eteam/echoserver:2.2-linux-arm image: openkruise-registry.cn-shanghai.cr.aliyuncs.com/openkruise/demo:1.10.3 imagePullPolicy: IfNotPresent env: - name: NODE_NAME # Optional. To clearly show the phased release effect, change the value to version2. value: version2Check the status of the Rollout resource.
kubectl get rollouts rollouts-demo -n defaultExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Progressing 1 StepPaused Rollout is in step(1/3), and you need manually confirm to enter the next step 41mThe
STATUSandCANARYfields in the expected output show the progress of the rollout.An output of
STATUS=Progressingindicates that the canary release is in progress.An output of
CANARY_STEP=1indicates that the release is in the first batch.An output of
CANARY_STATE=StepPausedindicates that the current batch is complete and you must manually confirm whether to proceed.
Manually proceed to the next batch.
kubectl-kruise rollout approve rollouts/rollouts-demo -n defaultContinuously check the rollout status.
kubectl get rollouts rollouts-demo -n default -wExpected output:
NAME STATUS CANARY_STEP CANARY_STATE MESSAGE AGE rollouts-demo Progressing 2 StepPaused Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 45m rollouts-demo Progressing 2 StepReady Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 45m rollouts-demo Progressing 3 BeforeStepUpgrade Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(2/3), and wait duration(60 seconds) to enter the next step 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 45m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepUpgrade Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepMetricsAnalysis Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepPaused Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 StepReady Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 Completed Rollout is in step(3/3), and upgrade workload to new version 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout has been completed and some closing work is being done 46m rollouts-demo Progressing 3 Completed Rollout progressing has been completed 46m rollouts-demo Healthy 3 Completed Rollout progressing has been completed 46m