This topic introduces the cloud controller manager (CCM) and provides usage notes and release notes for the component.

Introduction

The CCM allows you to integrate Kubernetes with Alibaba Cloud services, such as Classic Load Balancer (CLB) and Virtual Private Cloud (VPC). CLB is formerly known as Server Load Balancer (SLB). The CCM provides the following features:
  • Manage CLB instances

    If you set Type=LoadBalancer for a Service, the CCM automatically creates a CLB instance for the Service, and configures listeners and backend server groups. When the endpoint of an Elastic Compute Service (ECS) instance in a vServer group for a Service is changed or the cluster nodes are changed, the CCM automatically updates the vServer groups of the CLB instance.

  • Enable cross-node communication

    If Flannel is used as the network plug-in of a Kubernetes cluster, the CCM can enable network connections between containers and nodes. This allows you to implement cross-node communication. The CCM adds the pod CIDR block to the route table of the VPC where the cluster is deployed. This enables cross-node communication. This feature is ready for use after the CCM is installed.

Usage notes

Release notes

March 2023

VersionImage addressRelease dateDescriptionImpact
v2.6.0registry-cn-hangzhou.ack.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.6.02023-03-02
  • New features:
    • The label alpha.service-controller.kubernetes.io/exclude-balancer, which is used to remove backend servers from CLB and Network Load Balancer (NLB) instances, is deprecated. The label node.kubernetes.io/exclude-from-external-load-balancers is now used to remove backend servers from CLB and NLB instances.
    • A listener can be configured to use both TCP and UDP for a CLB instance or NLB instance.
    • The label service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch can be used to disable TCP health checks and UDP health checks for CLB instances.
    • The label service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol can be used to enable Proxy Protocol for TCP listeners and UDP listeners of CLB instances.
      Important Enabling this feature results in service interruptions. You must stop your applications before you enable Proxy Protocol. Proceed with caution.
    • The validity period of certificates can be verified during HTTPS listener updates. If a certificate has expired, the CLB instance update fails.
    • The label service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids can be used to configure security groups for NLB instances.
  • Improvements:
    • Leases are used to lock resources for CCM leader election instead of endpointsleases. This reduces the frequency of leader switches.
    • The update logic of CLB and NLB instances is optimized. When attributes of a CLB or NLB instance, such as the name and resource group, fail to be updated, the vServer groups of the CLB or NLB instance are still updated.
    • The conditions for identifying node changes are narrowed to reduce the frequency of Service updates.
  • Fixed issues:

    The issue that ready nodes are occasionally recognized as NotReady is fixed.

No impact on workloads

October 2022

VersionImage addressRelease dateDescriptionImpact
v2.5.1registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.5.12022-10-12
  • New features:
    • NLB instances can be created for LoadBalancer Services whose loadBalancerClass is set to alibabacloud.com/nlb. Only Kubernetes 1.24 and later support this feature. For more information, see What is NLB?.
    • Network resources can be created for Services based on the spec.loadBalancerClass of the Services. If the spec.loadBalancerClass is left empty, a CLB instance is created. If the spec.loadBalancerClass is set to alibabacloud.com/nlb, an NLB instance is created. Only Kubernetes 1.24 and later support this feature.
  • Improvements:
    • The issue that reused IPv6 SLB instances cannot be deleted is fixed.
    • The occasionally occurring issue that nodes cannot be deleted is fixed.
    • HTTPS is specified as the default protocol for API calls.
No impact on workloads
v2.4.3registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.4.32023-03-02

The issue that ready nodes are occasionally recognized as NotReady is fixed.

No impact on workloads
v2.4.2registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.4.22022-10-12Improvements:
  • The issue that reused IPv6 SLB instances cannot be deleted is fixed.
  • The occasionally occurring issue that nodes cannot be deleted is fixed.
No impact on workloads

June 2022

VersionImage addressRelease dateDescriptionImpact
v2.4.0registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.4.02022-06-20
  • New features:
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type can be used to specify the billing method of CLB instances.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy can be used to configure TLS security policies for CLB instances. Only HTTPS is supported.
    • The CCM automatically assigns a value to the node.spec.providerID field if the field is empty when you add a node.
    • The label service.k8s.alibaba/loadbalancer-id can be added to LoadBalancer Services to indicate the IDs of the CLB instances that are associated with the Services.
  • Improvements:
    • A node is not added as a backend server to a CLB instance if the node has the ToBeDeletedByClusterAutoscaler taint.
    • The following issue is fixed: Conflicted routes cannot be deleted if the destination CIDR blocks of the routes are the same.
    • The logic of concurrent route synchronization is optimized to reduce false positives.
No impact on workloads

March 2022

VersionImage addressRelease dateDescriptionImpact
v2.3.0registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.3.02022-03-21
  • New features:
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname can be used to specify the hostname of a Service.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout can be used to specify the connection timeout period for TCP listeners of CLB instances.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout can be used to specify the request timeout period for HTTP and HTTPS listeners of CLB instances.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method can be used to specify the health check method for HTTP health checks of CLB instances.
  • Improvements:
    • The format of vServer groups is verified when you reuse existing vServer groups.
    • The logic of vSwitch selection is optimized to resolve the issue that the default vSwitch is not specified.
    • The synchronization logic of vServer groups is optimized to reduce the number of API calls.
No impact on workloads

November 2021

VersionImage addressRelease dateDescriptionImpact
v2.1.0registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.1.02021-11-22
  • New features:
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto can be used to obtain the listener protocol of a CLB instance from the X-Forwarded-Proto header field.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout can be used to specify the timeout period of idle connections.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled can be used to enable HTTP2.
  • Improvements:

    The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight can be set to 0 to stop distributing traffic to specific backend servers.

  • Fixed issues:
    • The issue that listeners cannot be created for a CLB instance when a large number of backend pods are added to the CLB instance.
    • The issue that the CLB instance used by a Service is not updated after the targetPort parameter of the Service is updated.
No impact on workloads

September 2021

VersionImage addressRelease dateDescriptionImpact
v2.0.1registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v2.0.12021-09-02
  • New features:
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port can be used to reuse an existing vServer group that is added to a CLB instance. This annotation takes effect only when the CLB instance is reused. For more information, see Use the CCM to deploy services across clusters.
    • When a reused CLB instance is shared among multiple Services, the annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight can be used to set the weight of each Service to enable weighted round robin. This annotation takes effect only when the existing vServer group is reused. For more information, see Use the CCM to deploy services across clusters.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain can be used to configure connection draining for a CLB instance. Only TCP and UDP are supported.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout can be used to set the timeout period of connection draining for a CLB instance. Only TCP and UDP are supported.
    • The targetPort field can be set to a String value.
    • Finalizers can be specified for LoadBalancer Services.
  • Improvements:
    • Alpine Linux is updated to V3.13 for base images.
    • The port used by Prometheus metrics is changed from 10258 to 8080.
    • The node labels are synchronized by schedule.
No impact on workloads

April 2021

VersionImage addressRelease dateDescriptionImpact
v1.9.3.380-gd6d0962-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.380-gd6d0962-aliyun2021-04-20
  • The issue that the default server group cannot be updated is fixed.
  • Events are generated and alerts are triggered when a CLB instance is not associated with backend servers.
No impact on workloads

March 2021

VersionImage addressRelease dateDescriptionImpact
v1.9.3.378-g42eac35-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.378-g42eac35-aliyun2021-03-08
New features:
  • ECS instances other than those in the Container Service for Kubernetes (ACK) cluster can be added to a vServer group.
  • The label kubernetes.reused.by.user is automatically added to a reused CLB instance.
Improvements:
  • The number of concurrent threads for processing Services is increased to improve processing speed.
  • The processing logic of virtual-node is optimized to ignore Service updates caused by the status changes of virtual-node.
  • The node label service.beta.kubernetes.io/exclude-node is deprecated. To exclude a node from the management of the CCM, use the label service.alibabacloud.com/exclude-node instead.
  • Resource groups are verified when a CLB instance is reused. The resource group ID specified in annotations must be the ID of the resource group to which the CLB instance belongs. Otherwise, the CLB instance cannot be used to expose more than one Service.
  • The readability of event content is improved.
  • The version priority setting of annotations is optimized. If two versions of an annotation are added to the Service configurations, the later version prevails over the earlier version.
Fixed issues:
  • The issue that route entries failed to be deleted due to incomplete node configurations.
  • The logic of node initialization is optimized to fix the issue of taint missing. This prevents pods from being scheduled to a node for which route entries are not created during the initialization process.
No impact on workloads

December 2020

VersionImage addressRelease dateDescriptionImpact
v1.9.3.339-g9830b58-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.339-g9830b58-aliyun2020-12-18
  • Hash values are supported in the configurations of LoadBalancer Services. This way, when the CCM is restarted, only the vServer groups of the related CLB instances are updated if the Service configuration is not changed. The configurations of the related CLB instances and listeners are not updated.
  • CLB API calls are optimized to reduce the chances of throttling.
No impact on workloads

September 2020

VersionImage addressRelease dateDescriptionImpact
v1.9.3.316-g8daf1a9-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.316-g8daf1a9-aliyun2020-09-29
  • The occasional failure to update the vServer groups of CLB instances is fixed.
  • The health check port is changed from 10252 to 10258.
No impact on workloads

August 2020

VersionImage addressRelease dateDescriptionImpact
v1.9.3.313-g748f81e-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.313-g748f81e-aliyun2020-08-10
  • New features:
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection can be used to set deletion protection for CLB instances. By default, deletion protection is enabled for newly created CLB instances.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection can be used to set the configuration read-only mode for CLB instances. By default, the configuration read-only mode is enabled for newly created CLB instances.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id can be used to specify the resource group to which a CLB instance belongs. This setting applies only when you create a CLB instance and cannot be modified after the instance is created.
    • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name can be used to specify the name of a CLB instance.
    • The API operations of Alibaba Cloud services can be called over internal networks instead of the Internet. To call the CCM operations, Internet access is no longer required in all regions.
    • Tags are added to a CLB instance that is created for a LoadBalancer Service. The tags are in the ack.aliyun.com: {your-cluster-id} format. This feature applies to only newly created clusters.
    • The cloud provider ID can be specified in the <cloudProvider>://<optional>/<segments>/<provider id> format, which is compatible with open source Kubernetes.
    • When a LoadBalancer Service is created in a cluster that uses Terway, the backed pods are automatically added to the CLB instance that is associated with the Service. The IP addresses of elastic network interfaces (ENIs) that are allocated to the pods are added as the backend servers of the CLB instance. This improves network performance. For LoadBalancer Services, the targetPort field cannot be set to a string value.
  • Improvements:
    • Alpine Linux is updated to V3.11.6 for base images.
    • Listener updates are automatically synchronized to vServer groups.
    • CLB API operations are optimized. You can call the CLB API to create CLB instances with improved speed.
No impact on workloads

June 2020

VersionImage addressRelease dateDescriptionImpact
v1.9.3.276-g372aa98-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64: v1.9.3.276-g372aa98-aliyun2020-06-11
  • New features:
    • The CLB instance attached to the cluster API server cannot be reused by LoadBalancer Services.
    • Prometheus metrics (ccm_node_latencies_duration_milliseconds, ccm_route_latencies_duration_milliseconds, and ccm_slb_latencies_duration_milliseconds) are added to monitor the synchronization latency of the CCM.
    • Events are collected to monitor the synchronization process between a Service and the related CLB instance.
  • Improvements:
    • Weight calculation is optimized for Services in Local mode. To enable the Local mode, set externalTrafficPolicy=Local in Service configurations. This improves load balancing among pods. For more information, see How does the CCM calculate node weights in Local mode?.

    • API calls of cloud services are optimized to improve efficiency and reduce the chances of throttling.
    • When you delete a node with the label service.beta.kubernetes.io/exclude-node, the related route entries are no longer deleted.
  • Fixed issues:
    • The issue that persistence timeout cannot be set to 0 by adding annotations during Service updates.
    • The issue that bandwidth cannot be set to 100 by adding annotations during Service updates.
No impact on workloads

March 2020

VersionImage addressRelease dateDescriptionImpact
v1.9.3.239-g40d97e1-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64: v1.9.3.239-g40d97e1-aliyun2020-03-05
  • New features:

    For LoadBalancer Services, the CCM allows you to specify both ECS nodes and ENIs as the backend servers of the related CLB instances.

  • Improvements:
    • The API operations of Alibaba Cloud services can be called over internal networks instead of the Internet. To call the CCM operations, Internet access is no longer required in regions other than China (Beijing), China (Shanghai), and UAE (Dubai).

    • The API operation that is used to query VPC route entries is changed to DescribeRouteEntryList. This provides higher performance when hundreds of queries are received within a short period of time.

No impact on workloads

December 2019

VersionImage addressRelease dateDescriptionImpact
v1.9.3.220-g24b1885-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64: v1.9.3.220-g24b1885-aliyun2019-12-31
  • vSwitch IDs are supported. You can set vSwitch IDs in CloudConfig by using the following format: :vswithid1,:vswitchid2.
  • Backoff is supported when throttling is enabled. Backoff allows failed requests to rejoin the reconcile queue every 30 to 180 seconds.
  • The number of worker threads to be reconciled is adjusted to 2. This allows you to fully utilize the queries per second (QPS) quota on API calls to speed up the reconcile process.
  • The issue that the CCM quits unexpectedly due to concurrent Map reads and writes based on the aliyungo SDK is fixed.
  • When a node is removed from an ACK cluster, the related route entries are automatically deleted from the VPC route table by the CCM.
  • The issue that port configurations cannot be changed due to port dependencies for HTTP port forwarding is fixed.
  • If the backend server of a CLB instance is an ECS instance, the serverip field is no longer required when you change the backend server. This prevents errors caused by the changes of default serverip values in API requests when you add backend servers.
  • The route entries of a node are added to the VPC route table only if the status of the node is known.
  • NAT IP addresses are no longer added to node metadata by the CCM. This fixes the issue that the API server occasionally fails to connect to kubelet.
  • When you modify the configurations of a listener, the start listener operation is called only if the listener is in the inactive state. This prevents throttling on API requests.
No impact on workloads

November 2019

VersionImage addressRelease dateDescriptionImpact
v1.9.3.193-g6cddde4-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.193-g6cddde4-aliyun2019-11-19
  • The label service.beta.kubernetes.io/exclude-node can be added to a node. After the label is added, the node is no longer managed by the CCM.
  • Multiple backend pods can be added to a CLB instance at a time. The network type of the pods must be Terway.
  • The node weight cannot be less than 1 for Services in Local mode (when externalTrafficPolicy=Local is set for the Services).
  • The issue that vServer groups are repeatedly created when concurrent requests are processed is fixed.
  • The issue that stale data is generated due to caching when you set node weights is fixed.
No impact on workloads

September 2019

VersionImage addressRelease dateDescriptionImpact
v1.9.3.164-g2105d2e-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3-164-g2105d2e-aliyun2019-09-11
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id can be used to renew a certificate.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port can be used to enable port forwarding from an HTTP port to an HTTPS port.
  • The following annotations can be used to create CLB instances with access control list (ACL) settings: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status, service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id, and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend can be used to remove unschedulable nodes.
  • When the Terway network plug-in is used, you can use the annotation service.beta.kubernetes.io/backend-type: "eni" to add pods that are assigned ENIs as the backend servers of a CLB instance. This improves network forwarding performance.
  • Services in Local mode (when externalTrafficPolicy=Local is set for the Services) can automatically set node weights based on the number of pods on each node.
No impact on workloads

April 2019

VersionImage addressRelease dateDescriptionImpact
v1.9.3.105-gfd4e547-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.105-gfd4e547-aliyun2019-04-15
  • Multiple route tables can be created for a VPC. Configuration files can be used to set multiple route tables for a cluster.
  • The issue that updated HTTP configurations do not take effect is fixed.
No impact on workloads

March 2019

VersionImage addressRelease dateDescriptionImpact
v1.9.3.81-gca19cd4-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.81-gca19cd4-aliyun2019-03-20
  • Existing CLB instances that are not created by ACK can be reused by ACK managed clusters and ACK dedicated clusters.
  • Custom node names are supported. Node naming is no longer reliant on the nodeName field in Kubernetes.
  • The compatibility issue between CCM 1.8.4 and Kubernetes 1.11.5 is fixed. We recommend that you update the CCM to the latest version.
No impact on workloads

December 2018

VersionImage addressRelease dateDescriptionImpact
v1.9.3.59-ge3bc999-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.59-ge3bc999-aliyun2018-12-26
  • A CLB instance can be shared by multiple Kubernetes Services.
    • If a CLB instance is created along with a Service, you cannot reuse this CLB instance when you create other Services. Otherwise, the CLB instance may be deleted. Only CLB instances that are manually created in the console or by calling the API can be used to expose multiple Services.
    • Kubernetes services that share the same CLB instance must use different frontend listening ports. Otherwise, port conflicts may occur.
    • When you reuse a CLB instance, you must use the listener name and vServer group name as identifiers. Do not modify the names of listeners or VServer groups.
    • You can modify the CLB instance name.
    • You cannot share CLB instances across clusters.
  • VPC route tables are managed in sequence instead of in parallel. This prevents throttling.
No impact on workloads

August 2018

VersionImage addressRelease dateDescriptionImpact
v1.9.3.10-gfb99107-aliyunregistry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.3.10-gfb99107-aliyun2018-08-15
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid can be used to specify the primary zone for an automatically created CLB instance.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid can be used to specify secondary zones for an automatically created CLB instance.
    Note This parameter does not take effect in regions that do not support CLB instances that are deployed across the primary zone and secondary zones.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners can be used to specify whether to overwrite the existing listeners when you reuse an existing CLB instance. A value of true overwrites the existing listeners when you reuse an existing CLB instance.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth can be used to specify the bandwidth when you create a pay-by-bandwidth CLB instance. The bandwidth is shared among listeners of the CLB instance.
No impact on workloads

June 2018

VersionImage addressRelease dateDescriptionImpact
v1.9.3registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:v1.9.32018-06-25
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label can be used to add worker nodes with specific labels as the backend servers of a CLB instance.
  • The annotation service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec can be used to specify the CLB instance type, such as shared-resource or high-performance.
  • The externalTraffic: Local mode for Services is supported. If this mode is enabled, only nodes that host the pods are added as the backend servers of the related CLB instance.
  • If a node is added to or removed from a cluster, the node is automatically added to or removed from the backend servers of the related CLB instances.
  • When the labels of a node are changed, the node is automatically added to or removed from the backend servers of the related CLB instances.
  • Sticky sessions are supported.
  • Listeners are no longer managed by the system when you create a Service by using an existing CLB instance. You must manually add listeners.
No impact on workloads