Understanding OpenKruise Kubernetes Resource Update Mechanisms

By Jiuzhu, Technical Expert of Alibaba Cloud

Currently, in Alibaba's internal cloud-native environment, OpenKruise is used for pod deployment and release management for most applications. Many companies in the industry and users of Alibaba Cloud also use OpenKruise for application deployment because Kubernetes native workload controllers, such as deployments cannot fully meet the requirements. This article begins with an Alibaba Cloud user's question about connecting to OpenKruise.

Background

OpenKruise is an open-source automated management engine for large-scale applications, which is provided by Alibaba Cloud. It provides features similar to Kubernetes' native controllers, such as Deployment and StatefulSet. In addition, OpenKruise provides more enhancements, such as graceful in-place update, release priority and scatter update, abstract management of multi-zone workload, and centralized management of sidecar container injection. These features have been tested in ultra-large-scale application scenarios at Alibaba. Moreover, these features help us cope with more diverse deployment environments and requirements, and bring more flexible deployment and release combination strategies for cluster maintainers and application developers.

Currently, in Alibaba's internal cloud-native environment, OpenKruise is used for pod deployment and release management for most applications. Many companies in the industry and users of Alibaba Cloud also use OpenKruise to deploy applications because Kubernetes native workload controllers, such as deployments cannot fully meet the requirements.

This article starts with a question from an Alibaba Cloud user about connecting to OpenKruise. Here is what the customer did. The following YAML data is a demo only.

1. Prepare a YAML file of Advanced StatefulSet, and submit for creation. The following is an example:

apiVersion: apps.kruise.io/v1alpha1
kind: StatefulSet
metadata:
  name: sample
spec:
  # ...
  template:
    # ...
    spec:
      containers:
        - name: main
          image: nginx:alpine
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      podUpdatePolicy: InPlaceIfPossible

2. Modify the image version in the YAML file and call the Kubernetes API to update the image. An error is returned and it looks like this:

metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

3. If you run the kubectl apply command to update the image, a success message is returned.

statefulset.apps.kruise.io/sample configured

Why did the update fail when you used the API call, but succeed when you used the kubectl apply command for the same modified YAML file? This is not due to any special validation of OpenKruise, but rather to the update mechanism of Kubernetes.

Based on the communication with users, we find that most of them have already used kubectl commands or SDKs to update Kubernetes resources. However, only a few of them understand the mechanism behind these update operations. This article focuses on the resource update mechanism of Kubernetes and how some common update methods are implemented.

Mechanisms for an Update

Have you ever wondered, for a Kubernetes resource object, such as a deployment, when you try to modify the image in it, if other users are also modifying the deployment at the same time, what will happen?

Two questions can be derived here:

If both sides modify the same field, such as the image field, what will happen?
If both sides modify different fields, for example, one modifies an image and the other modifies replicas, what will happen?

You update a Kubernetes resource object by notifying the kube-apiserver component of how you want to modify the object. Kubernetes defines two "notification" methods for this type of requirement: update and patch. In the update request, you need to submit the modified object to Kubernetes. In the patch request, you only need to submit the modifications of fields in the object to Kubernetes.

Now, let's get back to the former question: Why did the update fail after the user submitted a modified YAML file? This is because the update is limited by the version control of Kubernetes for update requests.

The Update Mechanism

All resource objects in Kubernetes have a globally unique version number (metadata.resourceVersion). Each resource object has a version number from the time it is created. The version number changes every time it is modified (whether by update or patch).

According to the official documentation, the version number is an internal Kubernetes mechanism. You cannot assume that it is a number or compare two version numbers to determine whether resources are new or old. Instead, you can determine whether objects are from the same version and whether it has changed by comparing the version numbers. The resourceVersion is also used to control the versions of update requests.

Kubernetes requires that objects submitted in user update requests must contain resourceVersion. This means that data submitted for an update must first come from the existing objects in Kubernetes. Therefore, a complete update process is:

Obtain an existing object from Kubernetes. You can choose to query the object directly from Kubernetes. If list-watch is performed on the client, we recommend that you obtain an object from the local informer.
Modify the object. For example, add or subtract replicas in the deployment or modify the value of the image field to a new version of the image.
Submit the modified object to Kubernetes by using the update request.
The kube-apiserver verifies the resourceVersion of the object submitted in the update request, which must be consistent with the latest resourceVersion of the object in the current Kubernetes before the update is accepted. Otherwise, Kubernetes rejects the request and notifies that a version conflict has occurred.

The preceding figure shows what happens when multiple users update a resource object at the same time. If a conflict occurs, User A needs to retry and obtain the object of the latest version again, and submit the update request.

Therefore, both the preceding questions have been answered:

The update fails after you modify the YAML file because the YAML file does not contain the resourceVersion field. You must obtain the object in the current Kubernetes for modification and then submit the update request.
If two users update a resource object at the same time, whether the operation is performed on the same field or different fields in the object, the version control mechanism ensures that the update request of the two users does not cause overlap.

The Patch Mechanism

Compared with the version control of the update mechanism, the patch mechanism of Kubernetes is simpler.

When you submit a patch request for a resource object, the kube-apiserver does not consider the version issue but accepts the request as long as the patch request content is valid. The object is patched and the version number is updated at the same time.

However, the complexity of the patch mechanism lies in the fact that Kubernetes provides four patch policies: JSON patch, merge patch, strategic merge patch, and apply patch (server-side apply is supported since Kubernetes Version 1.14.) You can also see this policy option by running the kubectl patch –h command. By default, strategic is used.

$ kubectl patch -h
# ...
  --type='strategic': The type of patch being provided; one of [json merge strategic]

Due to space limitations, we will not describe each policy in detail here. Let's use a simple example to see their differences. For an existing deployment object, assume that a container named app already exists in the template:

If you want to add a container named nginx, how do you use a patch for the update?
If you want to modify the image of the app container, how do you use a patch for the update?

JSON patch (RFC 6902)

Add a container named nginx:

kubectl patch deployment/foo --type='json' -p \
  '[{"op":"add","path":"/spec/template/spec/containers/1","value":{"name":"nginx","image":"nginx:alpine"}}]'

Modify the image of the app container:

kubectl patch deployment/foo --type='json' -p \
  '[{"op":"replace","path":"/spec/template/spec/containers/0/image","value":"app-image:v2"}]'

In the JSON patch, you need to specify the operation type, such as add or replace. In addition, you need to specify the container by using the index when you modify the container list.

In this way, if the object has been modified by other users before you use a patch, your patch may have unexpected results. For example, when you update the image of the app container, you can specify container 0. However, another container is inserted into the first position of the container list. The updated image is incorrectly inserted into this unexpected container.

merge patch (RFC 7386)

You cannot use the merge patch alone to update an element in the list. Therefore, whether you want to add a container or modify the image, env, and other fields of an existing container, you need to specify the entire container list to submit the patch:

kubectl patch deployment/foo --type='merge' -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"app","image":"app-image:v2"},{"name":"nginx","image":"nginx:alpline"}]}}}}'

Obviously, this policy is not suitable for updating the fields deeply embedded in a list. It is more suitable for updating large segments.

However, for updates of map-type elements, such as labels and annotations, the merge patch allows you to specify the key-value operation separately. Compared with JSON patches, the merge patch is more convenient and intuitive:

kubectl patch deployment/foo --type='merge' -p '{"metadata":{"labels":{"test-key":"foo"}}}'

strategic merge patch

This patch policy does not have a general RFC standard. It is unique to Kubernetes but more powerful than the first two policies.

In the source code of Kubernetes, some additional policy annotations are defined in the data structure definitions of native Kubernetes resources. The following example shows the definition of the container list in the podSpec. For more information, see GitHub:

// ...
// +patchMergeKey=name
// +patchStrategy=merge
Containers []Container `json:"containers" patchStrategy:"merge" patchMergeKey:"name" protobuf:"bytes,2,rep,name=containers"`

You can see two key parts: patchStrategy: "merge" and patchMergeKey: "name".

This indicates that when the container list is updated based on the strategic merge patch policy, the name field in each element is considered as the key.

In short, when you use a strategic merge patch to update the containers, you no longer need to specify the index. Instead, you need to modify a container by specifying the name. Kubernetes calculates the merge by using the name as the key. Take the following patch operation as an example:

kubectl patch deployment/foo -p \
  '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","image":"nginx:mainline"}]}}}}'

If Kubernetes finds that a container named nginx already exists in the current containers, it only updates the image. If the container named nginx does not exist, Kubernetes inserts this container into the container list.

Note: The strategic merge patch policy can only be used for native Kubernetes resources and custom resources based on Aggregated API. They cannot be used for resource objects defined by CRD. This is easy to understand because kube-apiserver does not know the CRD resource structure and the merge policy. If you run the kubectl patch command to update a CR, the merge patch policy is used by default.

Kubectl Encapsulation

Now that we've learned about the basic update mechanism of Kubernetes, let's return to the original issue. After you modify a YAML file, you cannot directly call the update operation to update the YAML file. Why can you use the kubectl apply command to update it instead?

To give CLI users a good experience of interaction, kubectl provides a complex internal execution logic. Common operations, such as apply and edit, do not correspond to a simple update request. Update operations are limited by version control, and an update conflict is unfriendly to common users. The following content briefly introduces the logic of several kubectl update operations. If you are interested, see kubectl encapsulated source code.

apply

When you run the kubectl apply command with default parameters, client-side apply is triggered. This is the kubectl logic:

First, the data submitted by the user (YAML or JSON) is parsed into object A and then calls the Get operation to query the corresponding resource object from Kubernetes:

If the resource object does not exist, kubectl logs the data submitted by the current user to the annotation of object A (the key is kubectl.kubernetes.io/last-applied-configuration) and submits object A to Kubernetes for creation.
If the returned results show that Kubernetes already contains this resource object, which corresponds to object B, kubectl attempts to retrieve the value of kubectl.kubernetes.io/last-applied-configuration from the annotation of object B, which corresponds to the content submitted by the apply command in the previous operation. Then, kubectl calculates the diff based on the content of the previous apply and the content of the current apply. The default format is strategic merge patch. If the resource object is not a native resource, merge patch is used. Finally, add the kubectl.kubernetes.io/last-applied-configuration annotation to the diff, and submit a patch request to Kubernetes for updates.

This is just a general process. The real logic is more complex. The server-side apply has been supported by Kubernetes since Version 1.14. If you are interested, see the source code implementation.

edit

The kubectl edit command is simpler logically. After you run the command, kubectl finds the current resource object in Kubernetes and opens a command line editor (vi is used by default) to provide an editing interface to you.

When you complete the modification, save the object and exit. kubectl does not directly submit the modified object for updates. This is to avoid conflict if the resource object is updated during your modification process. Instead, the diff is calculated between the modified object and the object obtained initially. Finally, the diff content is submitted to Kubernetes in a patch request.

Summary

Now, you have a preliminary understanding of the Kubernetes update mechanism after the preceding introduction. Let's think about this question: Kubernetes provides two update methods, but how can you choose between update or patch in different scenarios? Below are the recommendations:

If you only modify the fields to be updated (for example, you have some custom labels and use operator to manage them), patching is the simplest way.
If the field to be updated may be modified by other users (for example, the modified replicas field may also be modified by other components, such as HPA), we recommend using the update policy to update the field and avoid overlap.

Finally, our customer modified the object obtained and submitted it for updates, which ultimately triggered the in-place upgrade of Advanced StatefulSet. We also welcome and encourage more people to participate in the OpenKruise community. Let's work together to create a high-performance application delivery solution for large-scale scenarios.

Community

Understanding OpenKruise Kubernetes Resource Update Mechanisms

Background

Mechanisms for an Update

The Update Mechanism

The Patch Mechanism

JSON patch (RFC 6902)

merge patch (RFC 7386)

strategic merge patch

Kubectl Encapsulation

apply

edit

Summary

Read previous post:

Read next post:

Alibaba Cloud Native Community

You may also like

Comments

Alibaba Cloud Native Community

Related Products

Cloud-Native Applications Management Solution

Container Service for Kubernetes

ACK One

Managed Service for Prometheus