Container Service for Kubernetes provides a rich set of components to extend cluster features. This topic describes how to use Terraform to configure components and manage your business in various scenarios.
Component types
ACK manages two types of cluster components: system components and optional components. For more information about components, see Components.
System components
System components are the basic components required to run an ACK cluster. They are installed by default when you create a cluster. Examples include the following:
kube-apiserver
kube-controller-manager
cloud-controller-manager
kube-proxy
CoreDNS
Optional components
Optional components are not required for deployment. You can install them to extend the features of your cluster. Optional components are categorized into application management, log and monitoring, storage, network, and security components.
Generate Terraform call parameters for component management in the console
You can generate Terraform call parameter configurations for components in the console to improve the efficiency and usability of component management.
Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, find the one you want to manage and click its name. In the left navigation pane, click Add-ons.
On the Component Management page, search for the component. On the component card, click Install or Configure as needed.
In the dialog box that appears, click Equivalent Code. In the panel that appears on the right, click the Terraform tab. The tab displays the parameters for installing or configuring the component. You can copy and use these parameters.
Component management practices
You can use Terraform to specify which components to install when you create a cluster. After the cluster is created, you can manage the full lifecycle of the components. This section describes how to manage the lifecycle of cluster components and provides best practices for different scenarios.
Specify components to install during cluster creation
You can specify which components to install when you create a cluster. The following resources are involved when you create a cluster:
ACK Managed cluster: alicloud_cs_managed_kubernetes
ACK Dedicated cluster: alicloud_cs_kubernetes
ACK Edge cluster: alicloud_cs_edge_kubernetes
ACK serverless cluster: alicloud_cs_serverless_kubernetes
You can use the addons attribute in these resources to define the components to install during cluster creation. The addons attribute is defined as follows:
# Take a managed cluster as an example.
resource "alicloud_cs_managed_kubernetes" "default" {
# Other parameters.
# ...
# The addons attribute is a list. Define the addons attribute in the resource to install the component during cluster creation.
addons {
# The name of the component. You can query it using alicloud_cs_kubernetes_addons in the data source.
# The components that are installed and can be installed in the current cluster, and their version information.
name = "XXX"
# The custom parameters of the component. Some cluster components support custom parameter configuration. You can use this field to specify custom parameters for the component. For more information about how to specify the parameters, see the "Modify custom configuration parameters of a cluster component" section.
config = jsonencode(
{
....
}
)
# The default value of this parameter is false (Boolean). ACK installs some components by default to facilitate cluster management. If you do not want to install these components when you create a cluster, set disabled to true.
disabled = XXX
}
}You can install components by specifying addons in the cluster resource only when you create a cluster. After a cluster is created, you cannot manage the component lifecycle by modifying the attributes in the addons code block. For example, you cannot upgrade, uninstall, or update the configuration of components. To manage the component lifecycle after a cluster is created, see Manage the lifecycle of components after cluster creation.
The following table describes how to configure components in ACK.
Component name | Component type | Component description | Terraform configuration |
appcenter | Application Management | An application center component that provides unified management of multi-cluster application deployment and application lifecycle. |
|
progressive-delivery-tool | Application Management | A component that provides phased release for applications. |
|
alicloud-monitor-controller | Log and monitoring | A system component provided by ACK to connect to CloudMonitor. |
|
metrics-server | Log and monitoring | A monitoring and offline component that is modified and enhanced by ACK based on the open source monitoring component. It provides a Metrics API for data consumption and supports Horizontal Pod Autoscaler (HPA). |
|
ack-node-problem-detector | Log and monitoring | A cluster node anomaly monitoring component that is modified and enhanced by ACK based on an open source project. It can be connected to third-party monitoring platforms. |
|
ags-metrics-collector | Log and monitoring | A monitoring service component for gene computing customers. You can use this component to monitor the resource usage details of each node in a gene workflow. |
|
ack-arms-prometheus | Log and monitoring | Use Alibaba Cloud Prometheus to monitor ACK clusters. |
|
loongcollector | Log and monitoring | Use Simple Log Service to collect Kubernetes container logs. | |
csi-plugin | Storage component | Supports attaching and detaching volumes. If you select the CSI plug-in to connect to Alibaba Cloud storage when you create a cluster, this component is installed by default. |
|
csi-provisioner | Storage component | Supports automatic creation of volumes. If you select the CSI plug-in to connect to Alibaba Cloud storage when you create a cluster, this component is installed by default. |
|
storage-operator | Storage component | Manages the lifecycle of storage components. |
|
alicloud-disk-controller | Storage component | Supports automatic creation of disk volumes. |
|
flexvolume | Storage component | An early storage volume extension mechanism implemented by the Kubernetes community. FlexVolume supports attaching and detaching volumes. If you select the FlexVolume plug-in to connect to Alibaba Cloud storage when you create a cluster, this component is installed by default. |
|
nginx-ingress-controller | Network component | The Nginx Ingress controller parses the forwarding rules of an Ingress. After the Ingress controller receives a request, it forwards the request to a backend service based on the matched Ingress forwarding rule. | |
terway-eniip | Network component | An open source Container Network Interface (CNI) plug-in developed by Alibaba Cloud based on virtual private cloud (VPC). It supports Kubernetes-native network policies to define access policies between containers. You can use the Terway network component to enable network communication within a Kubernetes cluster. If you select the Terway network plug-in to enable network communication within the cluster when you create a cluster, this component is installed by default. |
|
ack-node-local-dns | Network component | A DNS local cache solution based on the open source project NodeLocal DNSCache. | |
aliyun-acr-credential-helper | Security component | A component that lets you pull private images from ACR Personal Edition or Enterprise Edition in an ACK cluster without a password. |
|
gatekeeper | Security component | Helps manage and apply Open Policy Agent (OPA) policies in a cluster to implement features such as namespace label management. |
|
kritis-validation-hook | Security component | A key component for container image signature verification when you deploy trusted containers. |
|
security-inspector | Security component | A key component that implements security inspection. |
|
ack-kubernetes-webhook-injector | Security component | A Kubernetes component that can dynamically add pod IP addresses to or remove them from the whitelists of various Alibaba Cloud products. This eliminates the need to manually configure pod IP addresses in the whitelists of cloud products. |
|
ack-arena | Other | Further simplifies the installation of open source Arena, allowing you to install Arena with a single click in the console. |
|
ack-cost-exporter | Other | A plug-in for data processing in the ACK cost analysis feature. |
|
ack-kubernetes-cronhpa-controller | Other | Use ack-kubernetes-cronhpa-controller to implement scheduled scaling of application workloads. |
|
ack-virtual-node | Other | Based on the open source project Virtual Kubelet, this component extends support for Aliyun Provider and is highly optimized to seamlessly connect Kubernetes with Elastic Container Instance (ECI). |
|
aesm | Other | Intel® SGX Architectural Enclave Service Manager (Intel® SGX AESM) is a system component of Intel® SGX. It provides services such as SGX Enclave startup support, key configuration, and remote authentication. |
|
aliyun-acr-acceleration-suite | Other | A client plug-in that provides image acceleration based on on-demand loading. It is deployed as a DaemonSet on worker nodes. |
|
migrate-controller | Other | A Kubernetes application migration component developed based on the open source project Velero. |
|
resource-controller | Other | A key component that dynamically controls pod resources. This component must be installed to use CPU topology-aware scheduling in ACK Pro clusters. |
|
sandboxed-container-controller | Other | A dedicated controller component provided by the sandboxed container runtime to enhance and extend the basic features of sandboxed containers. |
|
sandboxed-container-helper | Other | A component that provides diagnostics and O&M for sandboxed containers. |
|
sgx-device-plugin | Other | A Kubernetes device plug-in for Intel SGX jointly developed by the Alibaba Cloud Container Service team and the Ant Financial secure computing team. It lets you use SGX in containers more conveniently. |
|
Manage the lifecycle of components after cluster creation
Before you manage the component lifecycle, you must have a Kubernetes cluster. If you do not have a cluster, create one first.
For components in a cluster, you can use the alicloud_cs_kubernetes_addon resource to manage their lifecycle. This includes installation, upgrades, uninstallation, and custom configuration modifications. The attributes and definition of alicloud_cs_kubernetes_addon are as follows:
resource "alicloud_cs_kubernetes_addon" "addon-example" {
# The cluster ID.
cluster_id = "XXXX"
# The name of the component. You can use alicloud_cs_kubernetes_addons in the data source to query all installed and installable components in the current cluster and their version information.
name = "XXXX"
# The version of the component.
version = "XXXX"
# The custom parameters of the component in the format of a JSON string. You can use the built-in jsonencode method of Terraform or a JSON string to configure the parameters. If you use a JSON string, pay attention to escape characters. Some cluster components support custom parameter configuration. You can use this field to specify custom parameters for the component. For more information about how to specify the parameters, see the "Modify custom configuration parameters of a cluster component" section.
config = jsonencode(
{
....
}
)
}You can configure custom parameters for a component by writing a JSON string. Pay attention to escape characters. For example, you can use one of the following methods to configure the nginx-ingress-controller component:
Configure parameters using
jsonencode:config = jsonencode( { IngressSlbNetworkType="internet" IngressSlbSpec="slb.s2.small" } )Configure parameters using a string:
config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
Import components installed in a cluster to Terraform for management
For components that are already installed in a cluster, you can use terraform import to import them into Terraform for management. The following example shows how to import an installed nginx-ingress-controller component into Terraform for management.
Create a file with the .tf extension or use an existing .tf file, and then define a resource.
The
alicloud_cs_kubernetes_addonin the resource is used to manage the add-on of the cluster. You do not need to enter any content at this step.resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" { }Run the following command to import the installed nginx-ingress-controller component of the cluster.
Terraform pulls the configuration of the nginx-ingress-controller component in the cluster and writes it to a file with the .state extension.
terraform import alicloud_cs_kubernetes_addon.nginx-ingress-controller <cluster_id>:nginx-ingress-controllerRun the
terraform plancommand. Based on the output, you can view the differences between the configuration of the nginx-ingress-controller component in the cluster and the defined resource.Based on the differences and the content of the file with the .state extension, update the resource that you defined in Step 1. Repeat this process until the
terraform plancommand shows no difference between the local configuration and the component configuration in the cluster. This indicates that the component is imported.resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" { cluster_id = "XXXXX" name = "nginx-ingress-controller" version = "v1.2.1-aliyun.1" config = jsonencode( { IngressSlbNetworkType = "internet" IngressSlbSpec = "slb.s2.small" } ) }
Install a cluster component
You can use the alicloud_cs_kubernetes_addon resource to install a component in an existing cluster. The following example shows how to install the gatekeeper component.
Define the information about the component to be installed in the .tf file. You must specify the following information.
The cluster ID.
The component name and version:
You can use
alicloud_cs_kubernetes_addonsin the data source to query the names and versions of installable components in the cluster. The query result returns only the latest installable version of each component. If you want to install a historical version of a component, view the release log of the component and specify the corresponding version number.(Optional) The custom configuration of the component:
Modify the
configfield to customize the component configuration. You can use the built-injsonencodemethod of Terraform to build the required configuration. You can usealicloud_cs_kubernetes_addon_metadatain the data source to query the configurable parameters of the component. For more information, see Modify custom configuration parameters of a cluster component.
Run the following command to install the component in the cluster.
terraform applyExpected output:
Plan: 1 to add, 0 to change, 0 to destroy. Do you want to perform these actions? Terraform will perform the actions described above. Only 'yes' will be accepted to approve. Enter a value: yes alicloud_cs_kubernetes_addon.gatekeeper: Creating... alicloud_cs_kubernetes_addon.gatekeeper: Still creating... [10s elapsed] alicloud_cs_kubernetes_addon.gatekeeper: Creation complete after 16s [id=XXXXX:gatekeeper] Apply complete! Resources: 1 added, 0 changed, 0 destroyed.When
Apply complete!is displayed, the component is installed.
Upgrade a cluster component
You can use alicloud_cs_kubernetes_addons in the data source to query the upgradeable versions of a component. If a new version is available, you can upgrade the component by modifying the version number. The following example shows how to upgrade the gatekeeper component.
Run the terraform apply command to upgrade the component. If the command runs successfully, the component is upgraded.
Modify custom configuration parameters of a cluster component
Some ACK components support custom parameter configuration. You can use the alicloud_cs_kubernetes_addon resource to update the configuration of your component. For example, you can modify the config field to modify the configuration of the gatekeeper component.
To view all configurable parameters that are supported by a component, you can use alicloud_cs_kubernetes_addon_metadata in the data source to query them. The returned result is in the JSON Schema format. For example, to query the parameters for the gatekeeper component, add the following content to the .tf file.
# Define a data source to obtain the configurable parameter schema of the gatekeeper component.
data "alicloud_cs_kubernetes_addon_metadata" "default" {
cluster_id = "ce36b7c61e126430b8b245730ca6d****"
name = "gatekeeper"
version = "v3.8.1.113-geb7947ef-aliyun"
}
# Use Output to generate the output.
output "addon_config_schema" {
value = data.alicloud_cs_kubernetes_addon_metadata.default.config_schema
}Run the terraform apply command. The returned result is in the JSON Schema format. The properties attribute defines all supported configuration parameters. Based on the returned schema, you can specify any supported configuration parameters. The configurable parameters are described as follows:
default: the default value.
description: the description of the parameter.
pattern: the regular expression for the value format.
type: the field type.
Configure network components
In ACK, you can use the Terway network mode to implement container network features. For more information, see Network.
The following example shows how to configure network components using Terraform.
Configure storage components
ACK provides two types of storage components: FlexVolume and Container Storage Interface (CSI). FlexVolume is no longer maintained. ACK maintains the CSI storage component. If you do not specify a storage component when you create a cluster using Terraform, CSI is installed by default. You can define storage components in Terraform in the following way.
Configure log components
The log collection component provided by ACK stores collected logs in Simple Log Service (SLS). The log component supports the following two log storage methods:
Store logs in an existing SLS project.
Automatically create a new SLS project to store logs when a cluster is created.
The configurations for the two scenarios are different. The following Terraform example provides more details.
Configure monitoring components
The monitoring components provided by ACK include the CloudMonitor agent, which is installed on ECS nodes, and Prometheus Service. In Terraform, the installation of the CloudMonitor agent on ECS nodes is controlled by the install_cloud_monitor parameter.
Configure Ingress routing components
ACK provides two traffic ingress solutions: Nginx Ingress and ALB Ingress.
Nginx Ingress component: This component is optimized based on the community edition of ingress-nginx to provide flexible and reliable routing services for your Kubernetes cluster. For more information, see Nginx Ingress overview.
ALB Ingress component: This is a fully managed and highly reliable ALB Ingress component that provides flexible and reliable routing services for your Kubernetes cluster. For more information, see Use an ALB Ingress to access a service.
The following example shows how to configure routing components using Terraform.
Disable default components
ACK installs some components by default to facilitate cluster management. If you do not want to install these components when you create a cluster, you can set the disabled = true field to disable them. The following example shows how to disable nginx-ingress-controller.
# Disable the installation of the nginx-ingress-controller component.
resource "alicloud_cs_managed_kubernetes" "default" {
# Other parameters.
# ...
addons {
name = "nginx-ingress-controller"
disabled = true
}
}Components installed by default when no addons are specified
If you do not specify any addons when you create a cluster, the following components are installed by default.
Cluster type | Component type | Default component name | Component description | |
ACK cluster | System component | kube-scheduler | Uses Kube Scheduler to schedule cluster resources. | |
cloud-controller-manager | Uses cloud-controller-manager to create Server Load Balancer (SLB) instances for Kubernetes applications and manage node route entries. | |||
kube-apiserver | kube-apiserver is the bus and ingress gateway of a Kubernetes cluster. | |||
kube-controller-manager | kube-controller-manager is the resource manager inside a Kubernetes cluster. | |||
Log and monitoring | alicloud-monitor-controller | Monitors the lifecycle and status changes of application containers. | ||
metrics-server | Metrics Server provides resource monitoring metrics of application containers for the auto scaling mechanism of the cluster. | |||
Storage | csi-plugin | Uses the csi-plugin plug-in to manage the lifecycle of storage volumes (recommended). | ||
csi-provisioner | Uses the csi-provisioner plug-in to create and delete storage volumes (recommended). | |||
storage-operator | Uses the storage-operator plug-in to manage storage O&M (recommended). | |||
Network | CoreDNS | The domain name resolution server of a Kubernetes cluster. | ||
Gateway API | The Gateway API gateway resource model. | |||
terway-eniip | The Terway network plug-in. | |||
nginx-ingress-controller (installed by default for Pro Edition) | An Ingress controller based on Nginx traffic forwarding. | |||
ACK serverless cluster | System component | kube-scheduler | Uses Kube Scheduler to schedule cluster resources. | |
ack-virtual-node | Uses virtual nodes and the elastic capabilities of ECI. | |||
cloud-controller-manager | Uses cloud-controller-manager to create SLB instances for Kubernetes applications and manage node route entries. | |||
kube-apiserver | kube-apiserver is the bus and ingress gateway of a Kubernetes cluster. | |||
kube-controller-manager | kube-controller-manager is the resource manager inside a Kubernetes cluster. | |||
Network | CoreDNS | The domain name resolution server of a Kubernetes cluster. | ||
ACK Edge cluster | System component | kube-scheduler | Uses Kube Scheduler to schedule cluster resources | |
cloud-controller-manager | Uses cloud-controller-manager to create SLB instances for Kubernetes applications and manage node route entries. | |||
kube-apiserver | kube-apiserver is the bus and ingress gateway of a Kubernetes cluster. | |||
kube-controller-manager | kube-controller-manager is the resource manager inside a Kubernetes cluster. | |||
Log and monitoring | alicloud-monitor-controller | Monitors the lifecycle and status changes of application containers. | ||
metrics-server | Metrics Server provides resource monitoring metrics of application containers for the auto scaling mechanism of the cluster | |||
Network | CoreDNS | The domain name resolution server of a Kubernetes cluster. | ||
terway-eniip | The Terway network plug-in. | |||
Other | edge-controller-manager | - | ||
edge-tunnel-agent | edge-tunnel uses a client-server architecture to build a reverse O&M channel between the cloud and the edge. | |||
edge-tunnel-server | edge-tunnel uses a client-server architecture to build a reverse O&M channel between the cloud and the edge. | |||
yurt-app-manager | Uses yurt-app-manager to provide node pool and unit-based deployment features for ACK@Edge. | |||
Common configuration examples
The following section provides common examples for your reference.
Select Terway as the network component.
Select either CSI or FlexVolume as the storage component. Because FlexVolume is no longer maintained, we recommend that you use CSI.
Select either Nginx Ingress or ALB Ingress as the routing component based on your requirements.
You can install and combine other components based on your requirements.