Container Service for Kubernetes:Use Terraform to manage plug-ins

Types of plug-ins

ACK manages the following types of cluster plug-ins: system plug-ins and optional plug-ins. For more information, see Component overview.

System plug-ins

Optional plug-ins

You can deploy optional plug-ins to extend the capabilities of your clusters on demand. Optional plug-ins are classified into application management plug-ins, logging and monitoring plug-ins, volume plug-ins, network plug-ins, and security plug-ins.

Best practices for plug-in management

You can use Terraform to specify the plug-ins to be installed when you create a cluster. After the cluster is created, you can manage the lifecycle of the plug-ins in the cluster. The following sections describe how to manage the lifecycle of plug-ins installed in a cluster and provide best practices that apply to different scenarios.

Specify the plug-ins to be installed when you create a cluster

You can specify the plug-ins to be installed when you create a cluster. The following Resources are involved when you create different types of clusters:

• ACK managed cluster: alicloud_cs_managed_kubernetes

• ACK dedicated cluster: alicloud_cs_kubernetes

• ACK edge cluster: alicloud_cs_edge_kubernetes

• ACK Serverless cluster: alicloud_cs_serverless_kubernetes

You can set the addons object to specify the plug-ins to be installed when you create one of the preceding Resources. The following code block shows the attributes of the addons object:

# This example uses an ACK managed cluster. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  # The addons object is a list. You can set the addons object in a Resource to specify the plug-ins to be installed when the system creates the cluster. 
  addons {
    # The name of the plug-in. You can query the name of a plug-in by using alicloud_cs_kubernetes_addons of Data Source. 
    # The plug-ins that are already installed and can be installed in the cluster and the versions of the plug-ins. 
    name = "XXX"

    # Custom plug-in parameters. You can set this attribute for cluster plug-ins that support custom parameters. For more information, see the Modify the custom parameters of a cluster plug-in section. 
    config = jsonencode(
      {
        ....
      }
    )

    # The value is of Boolean type and the default is false. By default, ACK automatically installs specific plug-ins for you to manage the cluster. If you do not want ACK to install plug-ins when ACK creates the cluster, set disabled=true. 
    disabled = XXX
  }
}

The following table describes how to configure plug-ins by using Terraform in ACK clusters.

Manage the lifecycle of plug-ins after the cluster is created

To manage the lifecycle of plug-ins, make sure that you have an ACK cluster. If you do not have an ACK cluster, create one first.

You can set the alicloud_cs_kubernetes_addon object in the cluster Resource to manage the lifecycle of the plug-ins installed in the cluster. You can install, update, and uninstall plug-ins and customize the configuration of the plug-ins. The following code block shows the attributes of the alicloud_cs_kubernetes_addon object:

resource "alicloud_cs_kubernetes_addon" "addon-example" {
  # The ID of the cluster. 
  cluster_id = "XXXX"

  # The name of the plug-in. You can query the plug-ins that are already installed and can be installed and their versions by using alicloud_cs_kubernetes_addons of Data Source. 
  name = "XXXX"

  # The version of the plug-in. 
  version = "XXXX"

  # Custom plug-in parameters in a JSON string. You can use the jsonencode method of Terraform to specify the parameters or directly specify the parameters in a JSON string. Pay attention to character escaping if you directly specify the parameters in a JSON string. You can set this attribute for cluster plug-ins that support custom parameters. For more information, see the Modify the custom parameters of a cluster plug-in section. 
  config = jsonencode(
    {
      ....
    }
  )
}

You can directly specify custom parameters in a JSON string. Pay attention to character escaping when you use this method. For example, you can use one of the following methods to configure nginx-ingress-controller:

• Use jsonencode to configure custom parameters:

  config = jsonencode(   
    {       
      IngressSlbNetworkType="internet"       
      IngressSlbSpec="slb.s2.small"     
    }  
  )

• Directly specify custom parameters in a JSON string:

  config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"

Import the installed plug-ins to Terraform for management

You can use the terraform import method to import the plug-ins that are installed in the cluster to Terraform and then manage these plug-ins by using Terraform. This section uses nginx-ingress-controller as an example to demonstrate how to import the installed plug-ins to Terraform.

1. Create a file whose suffix is .tf and define a Resource. If you already have a file whose suffix is .tf, define a Resource in the file.

   The alicloud_cs_kubernetes_addon object in the Resource is used to manage the plug-ins in the cluster. You do not need to add content to the object.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
   }

2. Run the following command to import nginx-ingress-controller:

   Terraform automatically pulls the configuration of nginx-ingress-controller from the cluster and adds the configuration to the file whose suffix is .state.

   terraform import alicloud_cs_kubernetes_addon.nginx-ingress-controller <cluster_id>:nginx-ingress-controller

3. Run the terraform plan command. The command output shows the difference between the configurations of nginx-ingress-controller and Resource.

   Modify the Resource defined in Step 1 based on the configuration difference and the content of the file whose suffix is .state. If the output of the terraform plan command shows no difference between the configurations of nginx-ingress-controller and Resource, the plug-in is imported to Terraform.

   resource "alicloud_cs_kubernetes_addon" "nginx-ingress-controller" {
     cluster_id = "XXXXX"
     name = "nginx-ingress-controller"
     version = "v1.2.1-aliyun.1"
     config = jsonencode(
       {
         IngressSlbNetworkType = "internet"
         IngressSlbSpec        = "slb.s2.small"
       }
     )
   }

Install cluster plug-ins

You can use the alicloud_cs_kubernetes_addon object in the Resource to install plug-ins in the cluster. This section uses the Gatekeeper plug-in as an example.

1. Specify the following information about the plug-in to be installed in the file whose suffix is .tf:

   • The ID of the cluster.

   • The name and version of the plug-in:

     You can query the names and versions of the plug-ins that can be installed by using alicloud_cs_kubernetes_addons of Data Source. The result displays only the latest version of each plug-in available for installation. If you want to install an earlier version, check the release notes of the plug-in and specify the corresponding version number.

   • (Optional) Custom plug-in configuration:

     You can modify the config field to customize the plug-in configuration by using the jsonencode method of Terraform. You can query the custom parameters of a plug-in by using alicloud_cs_kubernetes_addon_metadata of Data Source. For more information, see Modify the custom parameters of a cluster plug-in.

     Click to view details

     resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
       cluster_id = "ce36b7c61e126430b8b245730ca6d****"
       name = "gatekeeper"
       version = "v3.8.1.113-geb7947ef-aliyun"
       config = jsonencode(
         {
           AdmissionPodCpuLimit      = "1000m"
           AdmissionPodCpuRequest    = "100m"
           AdmissionPodMemoryLimit   = "512Mi"
           AdmissionPodMemoryRequest = "256Mi"
           AdmissionPodNumber        = 3
           AuditInterval             = 1800
           AuditPodCpuLimit          = "1000m"
           AuditPodCpuRequest        = "100m"
           AuditPodMemoryLimit       = "512Mi"
           AuditPodMemoryRequest     = "256Mi"
           EnableAuditPod            = false
           EnableMutatingWebhook     = false
         }
       )
     }

2. Run the following command to install the plug-in in the cluster:

   terraform apply

   Expected output:

   Plan: 1 to add, 0 to change, 0 to destroy.
   
   Do you want to perform these actions?
     Terraform will perform the actions described above.
     Only 'yes' will be accepted to approve.
   
     Enter a value: yes
   
   alicloud_cs_kubernetes_addon.gatekeeper: Creating...
   alicloud_cs_kubernetes_addon.gatekeeper: Still creating... [10s elapsed]
   alicloud_cs_kubernetes_addon.gatekeeper: Creation complete after 16s [id=XXXXX:gatekeeper]
   
   Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

   If Apply complete! is displayed, the plug-in is installed.

Update cluster plug-ins

You can query the versions of a plug-in that are available for updating by using alicloud_cs_kubernetes_addons of Data Source. If a new version is available, you can change the version number to update the plug-in. This section uses the Gatekeeper plug-in as an example.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"

  # Change version to the version number that you want to use. 
  version = "XXXXXXXXX"
  config = jsonencode(
    {
      AdmissionPodCpuLimit      = "1000m"
      AdmissionPodCpuRequest    = "100m"
      AdmissionPodMemoryLimit   = "512Mi"
      AdmissionPodMemoryRequest = "256Mi"
      AdmissionPodNumber        = 3
      AuditInterval             = 1800
      AuditPodCpuLimit          = "1000m"
      AuditPodCpuRequest        = "100m"
      AuditPodMemoryLimit       = "512Mi"
      AuditPodMemoryRequest     = "256Mi"
      EnableAuditPod            = false
      EnableMutatingWebhook     = false
    }
  )
}

Run the terraform apply command to update the plug-in. If Apply complete! is displayed, the plug-in is updated.

Modify the custom parameters of a cluster plug-in

You can use alicloud_cs_kubernetes_addons in the Resource to modify the configuration of a plug-in if the plug-in supports custom parameters. This section uses the Gatekeeper plug-in as an example to demonstrate how to modify the config field to customize the plug-in configuration.

resource "alicloud_cs_kubernetes_addon" "gatekeeper" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name = "gatekeeper"
  version = "v3.8.1.113-geb7947ef-aliyun"

  # You can modify and apply the attributes in Config to customize the configuration of a cluster plug-in. 
  config = jsonencode(
    {
      AdmissionPodCpuLimit      = "1000m"
      AdmissionPodCpuRequest    = "100m"
      AdmissionPodMemoryLimit   = "512Mi"
      AdmissionPodMemoryRequest = "256Mi"
      AdmissionPodNumber        = 3
      AuditInterval             = 1800
      AuditPodCpuLimit          = "1000m"
      AuditPodCpuRequest        = "100m"
      AuditPodMemoryLimit       = "512Mi"
      AuditPodMemoryRequest     = "256Mi"
      EnableAuditPod            = false
      EnableMutatingWebhook     = false
    }
  )
}

To query the custom parameters supported by a plug-in, use alicloud_cs_kubernetes_addon_metadata of Data Source. The result is returned in a JSON schema. For example, to customize the configuration of the Gatekeeper plug-in, add the following content to the file whose suffix is .tf.

# Define Data Source to obtain the schema that includes the custom parameters supported by the Gatekeeper plug-in. 
data "alicloud_cs_kubernetes_addon_metadata" "default" {
  cluster_id = "ce36b7c61e126430b8b245730ca6d****"
  name       = "gatekeeper"
  version    = "v3.8.1.113-geb7947ef-aliyun"
}

# Output the schema. 
output "addon_config_schema" {
  value = data.alicloud_cs_kubernetes_addons.default.config_schema
}

Configure network plug-ins

ACK provides two network plug-ins to help you manage container networks: Flannel and Terway. The two network plug-ins use different networking models. For more information, see Overview.

# Configure the Flannel network plug-in. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "flannel"
  }
}

# Configure the Terway network plug-in and enable the Assign One ENI to Each Pod mode. This is the default mode. 
# In this mode, the number of pods on a node is limited by the elastic network interface (ENI) quota of Elastic Compute Service (ECS) instances. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "terway-eni"
  }
}

# Configure the Terway network plug-in and enable the IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
}

# Configure the Terway network plug-in and enable Kubernetes network policies in IPVLAN mode. 
# You can use only the Alibaba Cloud Linux 2 operating system because the One ENI for Multi-Pod mode uses the IPVLAN + eBPF virtualization technology. 
# The IPVLAN mode provides network access control based on Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

Configure volume plug-ins

# Configure FlexVolume. If you do not specify a volume plug-in, FlexVolume is installed. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "flexvolume"
  }
}

# Configure the CSI plug-in. The CIS plug-in consists of csi-plugin and csi-provisioner. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
}

# If you use the CSI plug-in and want ACK to create a default NAS file system and CNFS file system for dynamically provisioned volumes, you must also install the storage-operator plug-in. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name   = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
}

Configure logging plug-ins

# Use the automatically created Log Service project. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
  }
}

# Use the automatically created Log Service project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\"}
  }
}


# Use an existing Log Service project. Enable the automatically created Ingress dashboard. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\",\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use the automatically created Log Service project to store events. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
}

# Install and configure node-problem-detector. 
# Configure node-problem-detector to use an existing Log Service project to store events. node-problem-detector can share the Logstore that is used by logtail-ds. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"k8s-log-c55c35ff493df47b88783bea48827****\"}"
  }
}

Configure monitoring plug-ins

# Install the CloudMonitor agent on ECS nodes. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...
  install_cloud_monitor = true
}

# Install Prometheus plug-in. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name = "arms-prometheus"
  }
}

# Install the CloudMonitor agent and Prometheus plug-in at the same time. 

# Install the Prometheus plug-in. We recommend that you install the plug-in and then enable Prometheus Monitoring. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...
  install_cloud_monitor = true
  addons {
    name = "arms-prometheus"
  }
}

Configure Ingress plug-ins

# Configure nginx-ingress-controller. 
# If you use an Internet-facing Server Load Balancer (SLB) instance, you must set the IngressSlbNetworkType parameter to internet in the Config. 
# If you use an internal-facing SLB instance, you must set the IngressSlbNetworkType parameter to intranet in the Config. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   =  "nginx-ingress-controller",
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
}

# Configure alb-ingress-controller. 
resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name   = "alb-ingress-controller",
  }
}

Disable the system to install default plug-ins

# Disable the system to install the nginx-ingress-controller plug-in. 

resource "alicloud_cs_managed_kubernetes" "default" {
  # Other parameters. 
  # ...

  addons {
    name     =  "nginx-ingress-controller",
    disabled = true
  }
}

Plug-ins that are automatically installed if no plug-in is specified in the addons object

The system automatically installs the following plug-ins if you do not specify a plug-in in the addons object:

• kube-flannel-ds

• flexvolume

• alicloud-disk-controller

• alicloud-monitor-controller

• CoreDNS

• cloud-controller-manager

• nginx-ingress-controller

• managed-kube-proxy

• storage-operator

• managed-kube-proxy-windows (for Windows)

• metrics-server

• ack-scheduler

• aliyun-acr-credential-helper

Examples of commonly used configurations

# Do not install any plug-ins when you create the cluster. In this scenario, only the default plug-ins are installed. 
# A simple template. Replace the variables with the desired values. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  worker_number                = 2
  password                     = var.password
  pod_cidr                     = "172.20.0.0/16"
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = [var.vswitch_id]
  worker_instance_types        = [var.instance_types]
}

# Create a cluster that has Terway installed. 
# Enable the Assign One ENI to Each Pod mode. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  worker_number                = 2
  password                     = var.password
  pod_vswitch_ids              = [var.vswitch_id]
  service_cidr                 = "172.21.0.0/20"
  worker_vswitch_ids           = [var.vswitch_id]
  worker_instance_types        = [var.instance_types]

  addons {
    name = "terway-eni"
  }
}

# Create a cluster that has Terway installed. 
# Enable the IPVLAN mode and enable Kubernetes network policies. 
resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  worker_number                = 2
  password                     = var.password
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = [var.vswitch_id]
  worker_vswitch_ids           = [var.vswitch_id]
  worker_instance_types        = [var.instance_types]

  addons {
    name   =  "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"true\"}"
  }
}

# A template for installing the Flannel, CSI, and nginx-ingress-controller plug-ins. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  worker_number                = 2
  password                     = var.password
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = [var.pod_vswitch_id]
  worker_vswitch_ids           = [var.vswitch_id]
  worker_instance_types        = [var.instance_types]

  addons {
    name = "flannel"
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\"}"
  }
  addons {
    name = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
  addons {
    name = "nginx-ingress-controller"
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
  addons {
    name = "ack-node-local-dns"
  }
  addons {
    name = "arms-prometheus"
  }
  addons {
    name = "alicloud-monitor-controller"
    config = "{\"group_contact_ids\":\"[10619]\"}"
  }
}

# A template for installing the Terway, CSI, and nginx-ingress-controller plug-ins. 

resource "alicloud_cs_managed_kubernetes" "default" {
  name                         = var.name
  cluster_spec                 = "ack.pro.small"
  is_enterprise_security_group = true
  worker_number                = 2
  password                     = var.password
  service_cidr                 = "172.21.0.0/20"
  pod_vswitch_ids              = [var.vswitch_id]
  worker_vswitch_ids           = [var.vswitch_id]
  worker_instance_types        = [var.instance_types]

  addons {
    name = "terway-eniip",
    config = "{\"IPVlan\":\"true\",\"NetworkPolicy\":\"false\"}"
  }
  addons {
    name = "csi-plugin"
  }
  addons {
    name = "csi-provisioner"
  }
  addons {
    name = "storage-operator"
    config = "{\"CnfsOssEnable\":\"false\",\"CnfsNasEnable\":\"true\"}"
  }
  addons {
    name = "logtail-ds"
    config = "{\"IngressDashboardEnabled\":\"true\"}"
  }
  addons {
    name = "ack-node-problem-detector"
    config = "{\"sls_project_name\":\"\"}"
  }
  addons {
    name = "nginx-ingress-controller"
    config = "{\"IngressSlbNetworkType\":\"internet\",\"IngressSlbSpec\":\"slb.s2.small\"}"
  }
  addons {
    name = "ack-node-local-dns"
  }
  addons {
    name = "arms-prometheus"
  }
  addons {
    name = "alicloud-monitor-controller"
    config = "{\"group_contact_ids\":\"[10619]\"}"
  }
}