Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Getting Started with Kubernetes | Operator and Operator Framework

Getting Started with Kubernetes | Operator and Operator Framework

Suxing, a senior engineer, discusses Kubebuilder and shows how to extend Kubernetes API through the Operator Framework in three main parts.

By Suxing, Senior Engineer at Alibaba

This article shows how to extend the Kubernetes API through the Operator Framework based on current practices and use cases. This article consists of three parts: (1) basics of Operators, (2) basics of the Operator Framework and development process case study, and (3) Operators case study.

1. Operator Overview

Basic Concepts

To start, let's take a look at the basic concepts involved in this article.

  • CustomResourceDefinition (CRD): a type of custom Kubernetes resource.
  • Custom Resource (CR): a specific CRD instance.
  • Webhook: a type of HTTP callback registered to the Kubernetes API server. When a specific event occurs, the Kubernetes API server queries registered webhooks and forwards a message.

Webhooks are divided into mutating webhooks and validating webhooks. Mutating webhooks modify input objects, whereas validating webhooks only read input objects.

  • Work queue: a key component of controllers. Work queues monitor resource changes in clusters and store related objects as events, including their actions and keys, such as the Create action of a pod.
  • Controller: Controllers cyclically process work queues and push the cluster status to the expected status according to their respective logic. Different types of controllers execute different processing actions. For example, ReplicaSet controllers monitor the number of replicas and process pod-related events.
  • Operator: a mechanism used to describe, deploy, and manage Kubernetes applications. From the implementation perspective, an Operator is a combination of a CRD, webhook, and controller used to implement user business logic.

Common Operator Modes

1

Workflow:

  1. You create a CRD.
  2. The Kubernetes API server forwards the CRD request to a webhook based on its registered pass list.
  3. The webhook completes default parameter settings and parameter check for the CRD. Then, the related CR is written to a database and returned to you.
  4. A controller monitors the CR in the background and processes the CR-related special operations based on the business logic.
  5. The preceding processing results in cluster status changes, which are monitored and recorded by the controller as the status data of the CRD.

The Operator workflow is described at the high level and will be detailed through a case study later in this article.

2. Operator Framework Practices

Operator Framework Overview

To start, let's take a look at the basics of the Operator Framework. The Operator Framework provides webhooks and controllers and shields general underlying details from developers, allowing them to focus on implementing the O&M logic of managed applications without having to implement message notification triggers and re-queue upon failure.

Mainstream Operator frameworks include Kubebuilder and Operator SDK, both of which use controller-tools and controller-runtime. Kubebuilder provides comprehensive testing, deployment, and code scaffolding, whereas the Operator SDK provides better support for Ansible Operators and upper-layer operations.

Kubebuilder Practices

This section shows how to use Kubebuilder through SidercarSets in Alibaba Cloud's open source project Kruise.

SidercarSets insert Sidecar containers (also called secondary containers) to pods. For example, you can insert containers used for monitoring and log collection to enrich the functions of the pods. SidercarSets update themselves based on the insertion status and pod status to record the secondary container status.

Step 1: Initialize

Procedure: Create a GitLab project and run the kubebuilder init --domain=kruise.io command.

Parameter description: The domain parameter indicates the group domain name used to register the CRD object.

Result: The dependency code library is pulled, and a code framework together with tool files, such as a Makefile and Dockerfile, is created.

The following is a simplified version of the created content:

2

You can confirm the details in practice.

Step 2: Create an API

Procedure: Run the kubebuilder create api --group apps --version v1alpha1 --kind SidecarSet --namespace=false command.

The command creates a controller framework along with an API (or a CRD).

Parameter description:

  • Group: apps.kruise.io of the CRD consists of the group and domain parameters.

  • The version parameter has three values specified by the community standard:

    • v1alpha1: This API is unstable. The CRD may be discarded, fields may be adjusted, and dependencies are not required.
    • v1beta1: The API is stable and backward compatible, and features may be adjusted.
    • v1: The API and features are stable.
  • The kind parameter indicates the CRD type, similar to the native service type of the community.
  • The namespaced parameter indicates whether the CRD is globally unique or is unique within the namespace, which is similar to the node and pod concepts.

The parameters can be roughly divided into two types. The group, version, and kind parameters map to three important elements of the CRD metadata. You can refer to the general standards here when setting parameters. The namespaced parameter indicates whether the CRD is globally unique (similar to a node) or unique within the namespace (similar to a pod). Here, namespaced is set to false, indicating that the SidecarSet is globally unique.

Result:

Fill in the code after a CRD and a controller framework are created.

The result is as follows.

3

Pay attention to the content in blue. The CRD is defined by sidecarset_types.go, which must be filled in manually. The controller is defined by sidecarset_controller.go, which must also be filled in.

Step 3: Fill In the CRD-related Code

1. The created CRD is located in pkg/apis/apps/v1alpha1/sidecarset_types.go. Perform the following two operations:

(1) Adjust comments

The code generator depends on comments to generate code, so sometimes you need to adjust comments. In this example, adjust the following comments of the SidecarSet:

+genclient:nonNamespaced: Create a non-namespace object.

+kubebuilder:subresource:status: Create a status subresource.

+kubebuilder:printcolumn:name=MATCHED,type=integer,JSONPath=.status.matchedPods,description=xxx: kubectl get sidecarset: will be described later.

(2) Set fields

Set the following fields to make the CRD take effect:

  • SidecarSetSpec: Enter descriptive information about the CRD.
  • SidecarSetStatus: Enter the CRD status.

2. Then, run the make command to generate code.

You do not need to complete underlying controller implementation for the CRD, such as gRPC interface configuration and codec.

The final fields are as follows.

4

SidecarSets are used to inject Sidecars into pods. Selector and Containers are defined in SidecarSetSpec, as shown in the figure on the left. Selector indicates the pods into which Sidecars are injected, and Containers defines a Sidecar container.

As shown in the figure on the right, SidecarSetStatus defines the status information. MatchedPods indicates the number of pods that match the SidecarSet. UpdatedPods indicates the number of pods injected with Sidecars. ReadyPods indicates the number of pods running normally.

For more information, see this document.

Step 4: Create a Webhook Framework

1. Create two mutating webhooks by running the following commands:

kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=mutating --operations=create
kubebuilder alpha webhook --group core --version v1 --kind Pod --type=mutating --operations=create

2. Create a validating webhook by running the following command:

kubebuilder alpha webhook --group apps --version v1alpha1 --kind SidecarSet --type=validating --operations=create,update

Parameter description:

  • The group and kind parameters indicate the resource objects to be processed.
  • The type parameter indicates the type of the framework to be created.
  • The operations parameter indicates the key operations on the resource objects.

Result:

  • A webhook framework is created. The code must be manually filled in.

The result is as follows.

5

The content in blue indicates the three handlers that are created by running the preceding three commands. The following shows how to configure the handlers.

Step 5: Configure the Webhooks

The created webhook handlers are located in:

  • pkg/webhook/default_server/sidecarset/mutating/xxx_handler.go
  • pkg/webhook/default_server/sidecarset/validating/xxx_handler.go
  • pkg/webhook/default_server/pod/mutating/xxx_handler.go

Rewrite or fill in the code as follows.

  • Inject the Kubernetes client if resources other than the input CRD are required. In this practice, the Kubernetes client is injected because Sidecars are injected into pods after a SidecarSet is created.
  • Configure the webhooks through mutatingSidecarSetFn or validatingSidecarSetFn. After the resource object pointer is passed in, adjust the object attribute for webhook configuration.

The final code is as follows.

6

In Step 4, three webhooks are created, including the SidecarSet mutating webhook, SidecarSet validating webhook, and pod mutating webhook.

The preceding figure on the left shows the SidecarSet mutating webhook. The default settings are completed through setDefaultSidecarSet.

The preceding figure on the right shows the validating webhook used to check the fields of the SidecarSet.

The pod mutating webhook is not shown here. mutatingSidecarSetFn does not set the default value, but fetches the value of setDefaultSidecarSet and injects the value into pods.

Step 6: Configure the Controller

The created controller framework is located in pkg/controller/sidecarset/sidecarset_controller.go. Make the following modifications:

  • Modify the permission comment. The controller framework automatically creates a comment, such as //+kuberbuilder:rbac;groups=apps,resources=deployments/status,verbs=get;update;path. Modify the comment as needed to create role-based access control (RBAC) rules.
  • Add the queuing logic. The default code framework fills in the queuing logic of the CRD. For example, the actions of adding, deleting, and modifying SidecarSet objects are added to a work queue. If you need a trigger mechanism for associated resource objects, for example, requiring the SidecarSet to monitor pod changes, then manually add the queuing logic.
  • Enter the business logic. Modify the Reconcile function to process the work queue cyclically. The Reconcile function implements the business logic based on spec and returns the business logic implementation result to status. By default, an error message returned by the Reconcile function triggers re-queuing.

The following figure shows the final code of the SidecarSet controller.

7

addPod is used to fetch the pod-matched SidecarSet and queues it up for processing by the Reconcile function.

After retrieving the SidecarSet, the Reconcile function selects the matched pod based on the Selector setting, calculates the cluster status based on the current pod status, and fills in the CRD status accordingly.

3. SidecarSet Workflow

This section summarizes the SidecarSet workflow to help you understand how Operators work.

8

  1. You create a SidecarSet.
  2. After receiving the SidecarSet creation request, a webhook completes default settings and checks the settings. Then, the SidecarSet is stored in a database and returned to you.
  3. You create a pod.
  4. A webhook fetches the corresponding SidecarSet, retrieves the container, and injects it into the pod. Therefore, the pod includes the Sidecar when stored in the database.
  5. The controller constantly polls in the background to monitor cluster status changes. The injection in Step 4 triggers SidecarSet queuing. The controller adds 1 to UpdatedPods of the SidecarSet.

The preceding process is a simple implementation of a SidecarSet.

Additional information: Generally, webhooks implement business logic and update statuses through controllers. However, controllers are optional in some cases. In the preceding example, webhooks implement the main business logic without controller intervention.

4. Summary

Let's summarize what we have learned in this article:

  • Operators, in combination with CRDs, webhooks, and controllers, are used to extend user business logic in Kubernetes.
  • Kubebuilder is an official and standardized Operator Framework highly recognized by the community.
  • You can implement an Operator through custom code based on the procedure provided in this article.
Container Kubernetes Kubernetes Operator Operator Framework Kubebuilder Controllers SidecarSet
0 0 0
Share on

Read previous post:

Getting Started with Kubernetes | API Programming Paradigm

Read next post:

Getting Started with Kubernetes | Kubernetes Network Model

Alibaba Cloud Native Community

480 posts | 48 followers

You may also like

Comments

Alibaba Cloud Native Community

480 posts | 48 followers

Related Products

More Posts by Alibaba Cloud Native Community

See All