Configure Classic Load Balancer (CLB) with annotations

For advanced load balancing, you can configure Classic Load Balancer (CLB), listeners, and backend server groups by adding annotations to a Service YAML file.

Index

Category	Feature category	Configuration link
Annotation usage
Typical CLB operations	Create a Classic Load Balancer	Create a public CLB Create a private CLB Create a pay-by-bandwidth CLB Specify primary and secondary AZs Create a pay-as-you-go CLB Create an IPv6 CLB
	Specify an existing instance	Use an existing CLB Use an existing CLB and force-overwrite listeners
	Configure a CLB	Set CLB specification Specify a VSwitch for a CLB Set an IP address for a private CLB Add tags to a CLB Set a name for a CLB Set the resource group for a CLB Set a hostname for a Service
	Enable instance protection	Enable deletion protection for a CLB Enable configuration protection for a CLB
Typical listener operations	Session persistence settings	Set session persistence timeout for TCP listeners Configure session persistence (insert cookie) for HTTP/HTTPS listeners
	Port and protocol configuration	Set the frontend port for a listener Create a listener with a health check Create a UDP listener Create an HTTP listener Create an HTTPS listener Add certificates to an HTTPS listener Configure a listener for TCP and UDP Configure Proxy Protocol for TCP/UDP listeners
	Advanced settings	Set the scheduling algorithm for a listener Set an access control group for a listener Set a security policy for a listener Enable graceful close for a listener Add request headers to a listener Set the idle timeout for a listener Set the request timeout for a listener Set the connection timeout for a listener Disable HTTP/2 for a listener
Typical backend server group operations	Configuration management	Use labeled worker nodes as backends Use Pod host nodes as backends Remove unschedulable nodes from CLB backends Attach Pod ENIs to the CLB backend Reuse an existing VServer Group Set the traffic weight for a backend Service Ignore updates to backend server weights Set the default weight for backend servers
Related documents

Annotation usage

Annotations are case-sensitive.
Before using an annotation, verify the required CCM component version. To upgrade the CCM component, see component management. For the changelog of the CCM component, see Cloud Controller Manager.
As of September 11, 2019, alicloud in the annotations field has been updated to alibaba-cloud.
For example:
Before: service.beta.kubernetes.io/alicloud-loadbalancer-id
After: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id
The system still supports older alicloud annotations, so you do not need to modify your existing configurations.

Typical CLB operations

Create a public load balancer

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a private load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Supported CCM version

Specifies the address type of the load balancer. Valid values:

internet: The service is accessed over the public network. The Address Type of the corresponding CLB must be Public.
intranet: The service is accessed over a private network. The Address Type of the corresponding CLB must be Private.

Default value: internet

We recommend that you manually specify a VSwitch for the CLB. For more information, see Specify a VSwitch for the load balancer. If you do not specify a VSwitch, the system automatically selects one based on the CCM version:

For versions earlier than v2.13.0, the system uses the first VSwitch configured in the cluster's control plane by default.
For versions v2.13.0 and later, the system automatically selects a VSwitch from all available VSwitches.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Load balancer specification

Annotation: Multiple annotations are used, as described in the following table.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

The instance charge type. Valid values:

PayBySpec: pay-by-specification.
PayByCLCU: pay-by-usage.

Default value: PayBySpec

Important

For Cloud Controller Manager (CCM) versions v2.5.0 and later, the default value changes to PayByCLCU.
You cannot set a specification for a pay-by-usage load balancer. Therefore, you cannot set the value to PayByCLCU and use the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation at the same time.

v2.4.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec

The specification of the load balancer instance. Use this annotation to create a CLB instance with a specific specification or update the specification of an existing instance. Valid values:

slb.s1.small
slb.s2.small
slb.s2.medium
slb.s3.small
slb.s3.medium
slb.s3.large

Default value: slb.s1.small

For more information about the valid values for this option, see CreateLoadBalancer.

Important

If you modify the specification of a pay-by-specification CLB instance in the CLB console, the CCM might revert the change. Proceed with caution.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Description

Supported CCM version

Important

To prevent unexpected behavior such as cluster unavailability or traffic interruptions, do not reuse the API Server CLB or a CLB that is created by the CCM. Instead, manually create a new instance in the Classic Load Balancer (CLB) console.
Do not add or modify the reuse annotation on an existing LoadBalancer Service. Otherwise, the reuse operation might fail, or CCM might not release the load balancer correctly.

The ID of the load balancer instance. Use this annotation to specify an existing CLB.

By default, reusing an existing load balancer instance does not overwrite its listeners. To force an overwrite of the existing listeners, set the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners annotation to "true". This setting affects only the listeners of the current service and has no impact on other listeners.
Note
By default, reusing an existing load balancer does not overwrite existing listeners for the following reasons:
- If the listeners on the existing load balancer are already processing traffic for other services, a forced overwrite may cause service interruptions.
- The CCM provides limited support for backend configurations and may not correctly handle complex setups. If you have complex backend requirements, you can configure listeners in the console without overwriting them.
For these reasons, avoid forcing an overwrite of listeners. You can perform a forced overwrite only if the listener ports on the existing load balancer are no longer in use.
(For versions earlier than v2.10.0) When you reuse an existing load balancer, you cannot add extra tags by using the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags annotation.

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing load balancer and force-overwrite listeners

Annotation: Multiple annotations are used, as described in the following table. Force-overwriting existing listeners deletes any existing listeners if port conflicts occur.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

Important

To prevent unexpected behavior such as cluster unavailability or traffic interruptions, do not reuse the API Server CLB or a CLB that is created by the CCM. Instead, manually create a new instance in the Classic Load Balancer (CLB) console.

The ID of the load balancer instance. Use this annotation to specify an existing CLB.

By default, reusing an existing load balancer instance does not overwrite its listeners. To force an overwrite of the existing listeners, set the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners annotation to "true". This setting affects only the listeners of the current service and has no impact on other listeners.
Note
By default, reusing an existing load balancer does not overwrite existing listeners for the following reasons:
- If the listeners on the existing load balancer are already processing traffic for other services, a forced overwrite may cause service interruptions.
- The CCM provides limited support for backend configurations and may not correctly handle complex setups. If you have complex backend requirements, you can configure listeners in the console without overwriting them.
For these reasons, we do not recommend that you force an overwrite of listeners. You can perform a forced overwrite only if the listener ports on the existing load balancer are no longer in use.
(For versions earlier than v2.10.0) When you reuse an existing load balancer, you cannot add extra tags by using the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags annotation.

v1.9.3.81-gca19cd4-aliyun and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

When reusing an existing load balancer, this annotation determines whether to synchronize the service's listener configuration with the CLB.

"true": The CCM manages the CLB listeners associated with the service based on the service configuration (create, update, and delete). It does not affect listeners that it does not manage.
"false": The CCM does not make any changes to the CLB listeners.

Default value: "false"

Important

To avoid listener configuration conflicts, do not configure multiple services to reuse the same listener on a CLB instance when force-override is set to "true".

v1.9.3.81-gca19cd4-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${YOUR_LOADBALANCER_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify primary and standby availability zones

Annotation: Multiple annotations are used, as described in the following table. The primary and standby availability zones cannot be modified after creation.

Load balancers in some regions do not support primary and standby availability zones. For definitive information, refer to the instance creation page in the CLB console.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid	The availability zone ID for the primary backend servers.	v1.9.3.10-gfb99107-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid	The availability zone ID for the standby backend servers.	v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid: "cn-hangzhou-k"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid: "cn-hangzhou-j"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a pay-by-bandwidth load balancer

Annotation: Multiple annotations are used. The following two annotations are required.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type

The charge type of the load balancer instance. Valid values:

paybytraffic
paybybandwidth

Default value: paybytraffic

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth

The peak bandwidth of the load balancer. Default value: 50. This annotation applies only to public load balancers. For other limitations, see Change the billing method of a public load balancer instance.

v1.9.3.10-gfb99107-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type: "paybybandwidth"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth: "2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a VSwitch for the load balancer

Annotation: Multiple annotations are used. The following two annotations are required.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Specifies the address type of the load balancer. Valid values:

internet: The service is accessed over the public network. The Address Type of the corresponding CLB must be Public.
intranet: The service is accessed through a private network. The Address Type of the corresponding CLB must be Private.

Default value: internet

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id

The VSwitch ID for the load balancer instance. The VSwitch must be in the same Virtual Private Cloud (VPC) as the Kubernetes cluster.

When you use this annotation, you must also set service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type to "intranet".

To find the VSwitch ID, go to the Virtual Private Cloud console.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
   service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify an IP address for a private load balancer

Annotation: Multiple annotations are used. The following three annotations are required.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type	Specifies the address type of the load balancer. Valid values: `internet`: The service is accessed over the public network. This is the default value. The Address Type of the corresponding CLB must be Public. `intranet`: The service is accessed via the private network. The Address Type of the corresponding CLB must be Private. Default value: `internet`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id	The VSwitch ID for the load balancer instance. The VSwitch must be in the same Virtual Private Cloud (VPC) as the Kubernetes cluster. When you use this annotation, you must also set `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type` to "intranet". To find the VSwitch ID, you can log on to the Virtual Private Cloud console.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip	The IP address of the private load balancer instance. The IP address must be within the CIDR block of the specified VSwitch. Only IPv4 addresses are supported. This annotation must be used with `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id`. The IP address cannot be changed after the load balancer instance is created.	v2.7.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "192.168.x.x"
  name: nginx
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      name: http
  selector:
    app: nginx

Add additional tags to a load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Supported CCM version
A comma-separated list of tags to add, such as `"k1=v1,k2=v2"`. In CCM versions v2.10.0 and later, you can modify the tags of existing and reused instances. Important After you add this annotation to a service, any subsequent modifications to the tags of the corresponding load balancer instance in the console might be overwritten.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2" 
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an IPv6 load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Supported CCM version

The IP version of the load balancer instance. The IP version cannot be changed after the instance is created. To use this feature, the kube-proxy mode of your cluster must be IPVS. Valid values:

ipv4: IPv4.
ipv6: IPv6. The generated IPv6 address can be accessed only from IPv6-enabled environments.

Default value: ipv4

v1.9.3.220-g24b1885-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "ipv6"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable deletion protection for a load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection

Description

Supported CCM version

The deletion protection setting for the load balancer. Valid values:

on
off

Default value: on

Important

For load balancers created by a LoadBalancer service, if you enable deletion protection in the CLB console, you can still delete the load balancer associated with the service by running kubectl delete svc {your-svc-name}.

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable modification protection for a load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection

Description

Supported CCM version

The modification protection setting for the load balancer. Valid values:

ConsoleProtection
NonProtection

Default value: ConsoleProtection

v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify load balancer name

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Supported CCM version
The name of the load balancer instance. The name must be 2 to 128 characters in length. It must start with a letter (case-insensitive) or a Chinese character and can contain digits, periods (.), underscores (_), and hyphens (-).	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "your-svc-name"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Specify the resource group for the load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description	Supported CCM version
The resource group ID of the load balancer. The resource group ID cannot be changed after the load balancer is created. To find the resource group ID, go to the Resource Management console.	v1.9.3.313-g748f81e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Set a hostname for a service

Annotation: Multiple annotations are used, as described in the following table.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with commas (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname

Sets a hostname for the service. The hostname must follow DNS naming conventions.

Notes:

After you add this annotation, the EXTERNAL-IP of the service changes from the default CLB IP address to your_service_name. When you access the CLB IP address from within the cluster, traffic hairpins through the CLB and then back to the cluster.
After you add this annotation, if the listener protocol is TCP or UDP, accessing the CLB IP address from within the cluster can cause a hairpinning issue. For more information, see Clients cannot access a load balancer.
This annotation does not automatically bind the CLB instance to a domain name. You must purchase a domain name and then manually bind it to the instance. For information on purchasing a domain name, see What is Alibaba Cloud DNS?.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "${your_service_hostname}"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Expected output:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP            PORT(S)                      AGE
nginx-svc    loadBalancer   47.100.XX.XX     www.example.com        80:30248/TCP,443:32670/TCP   10s

Create a pay-by-usage load balancer

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type

Description

Supported CCM version

The instance charge type. Valid values:

PayBySpec: pay-by-specification.
PayByCLCU: pay-by-usage.

Default value: PayBySpec

Important

For Cloud Controller Manager (CCM) versions v2.5.0 and later, the default value changes to PayByCLCU.
You cannot set a specification for a pay-by-usage load balancer. Therefore, you cannot set the value to PayByCLCU and use the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec annotation at the same time.

v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayByCLCU"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Listener operations

Configure session persistence timeout for TCP listeners

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout

Description

Supported CCM versions

Sets the session persistence timeout in seconds for TCP listeners. By default, this setting applies to all TCP listeners on a load balancer.

The value must be an integer from 0 to 3600. A value of 0 disables session persistence. For more information, see CreateLoadBalancerTCPListener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout: "1800"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure sticky sessions (insert cookie)

Annotations: The available annotations are described in the following table. To use the insert cookie method, four of the annotations described below are required.

This feature supports only load balancer instances that use the HTTP or HTTPS protocol.
If you configure multiple HTTP or HTTPS listener ports, the sticky session setting applies to all of them by default.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session	Specifies whether to enable sticky sessions. This annotation applies only to HTTP and HTTPS listeners. Valid values: `on` `off` Default value: `off` For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type	The sticky session type. This annotation applies only to HTTP and HTTPS listeners. This annotation is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on`. Valid values: `insert`: Inserts a cookie. `server`: Rewrites a cookie. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout	Cookie timeout. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `insert`. The unit is seconds. The value range is [1, 86400]. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie	The name of the cookie that is configured on the server. The name must be 1 to 200 characters long and contain only ASCII letters and digits. The name cannot contain commas (,), semicolons (;), or spaces. The name cannot start with a dollar sign ($). This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session` is set to `on` and `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type` is set to `server`. For more information, see CreateLoadBalancerHTTPListener and CreateLoadBalancerHTTPSListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with a comma (,), for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure an access control list

Annotations: The following three annotations are required.

Before using these annotations to create a load balancer with access control, create an access control list in the Classic Load Balancer (CLB) console and record its ID (acl-id).

Annotation	Description	CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status	Specifies whether to enable access control. Valid values: `on` `off` Default: `off`	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id	The ID of the access control list for the listener. This parameter is required when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status` is set to `"on"`.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type	The type of access control. Valid values: `white`: Forwards requests only from the IP addresses or address blocks specified in the access control list. If you enable a whitelist but do not add any IP addresses to it, the listener forwards all requests. `black`: Blocks requests from the IP addresses or address blocks specified in the access control list. If you enable a blacklist but do not add any IP addresses to it, the listener forwards all requests.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: "${YOUR_ACL_ID}" # Multiple access control lists are not supported.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure port forwarding for the load balancer

Port forwarding redirects requests from an HTTP port to an HTTPS port.

Annotation: The following table describes the three required annotations.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Use a comma (,) to separate multiple values. For example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the certificate on Alibaba Cloud. Log on to the CLB console and view the certificate ID on the certificate management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port	Forwards HTTP requests to the specified HTTPS port. For example, `80:443`.	v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port: "80:443"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure scheduling algorithm

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Supported CCM version

Specifies the scheduling algorithm. Valid values:

wrr: weighted round robin. Backend servers with higher weights receive more requests.
rr: round robin. Requests are distributed to backend servers sequentially.
sch: source IP hash (consistent hashing). Routes requests from the same source IP address to the same backend server.
tch: consistent hashing based on a 4-tuple (source IP address, destination IP address, source port, and destination port). Routes requests from the same flow to the same backend server.

Default value: rr.

For more information, see the Scheduler parameter in the API reference for creating CLB listeners, such as CreateTCPListener.

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

HTTP listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Health check: By default, health checks are disabled for HTTP listeners. To ensure your backend service is available, configure an HTTP or TCP health check for the listener port.
When you access a Service by using the Classic Load Balancer (CLB) IP address from within the cluster, request traffic may be intercepted, causing access issues. To force traffic to hairpin through the CLB, add the required annotation as described in Set a Hostname for a Service. For more information, see Notes on accessing the external IP address of a LoadBalancer Service from within a cluster.

Description	Supported CCM versions
Use a comma (,) to separate multiple values, for example, `https:443,http:80`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTPS listener

annotation: The following table lists the available annotations.

CLB decrypts HTTPS requests and sends them to the backend pods as HTTP requests.

Health check: Health checks for HTTPS listeners are disabled by default. To ensure your backend service is available, configure an HTTP or TCP health check for the listener port.
When you access a Service by using the Classic Load Balancer (CLB) IP address from within the cluster, request traffic may be intercepted, causing access issues. To force traffic to hairpin through the CLB, add the required annotation as described in Set a Hostname for a Service. For more information, see Notes on accessing the external IP address of a LoadBalancer Service from within a cluster.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Use a comma (,) to separate multiple values, for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id

The certificate ID from Alibaba Cloud.

Log on to the CLB console and view the certificate ID on the certificate management page.

Note

To create a certificate, see Select an Alibaba Cloud-signed certificate.

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

SSL protocol error for intra-cluster CLB access

Symptoms

After you create an HTTPS listener for a service, you can access the service from outside the cluster. However, when you use curl to access the service's CLB HTTPS port from a node or pod within the cluster, the following error is returned:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Cause

This issue is caused by the IPVS rules on the node:

Service configuration: Because CLB HTTPS listeners support only the HTTP protocol for backend services, the service can only forward traffic from port: 443 to targetPort: 80. As a result, ACK creates an IPVS rule on the node to directly forward traffic destined for port 443 to the backend's port 80.
Layer 4 forwarding: IPVS operates at Layer 4 of the TCP/IP stack. It only forwards TCP packets and does not parse application-layer protocols such as TLS/HTTPS.
Protocol mismatch: IPVS directly forwards the HTTPS request (TLS handshake data) from the client (curl) to the backend service's HTTP port 80. Because this port is not configured for TLS, it cannot parse the TLS request. As a result, the port returns an HTTP 400 error, and the client reports an SSL protocol error.

Solution

Add the service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname annotation to the service. This annotation prevents IPVS rules from being generated on the node, forcing intra-cluster traffic to be routed through the CLB and ensuring the CLB can correctly terminate the TLS connection. For detailed steps, see Set a Hostname for a Service.

Configure extended domain certificates

annotation: The following table lists the available annotations.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Use a comma (,) to separate multiple values, for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The certificate ID from Alibaba Cloud. Log on to the CLB console and view the certificate ID on the certificate management page. Note To create a certificate, see Select an Alibaba Cloud-signed certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions	Specify the additional domain names and certificate IDs in the `{domain_name}:{certificate_id}` format. Use a comma (,) to separate multiple entries. Log on to the CLB console and view the certificate ID on the certificate management page.	v2.13.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions: "${DOMAIN_1}:${CERT_ID_1},${DOMAIN_2}:${CERT_ID_2}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Listener with a health check

TCP health check

Annotation: All the following annotations are required.

Health checks are enabled by default for TCP ports.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Enables or disables health checks for TCP and UDP listeners. Valid values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	Specifies the health check type. Valid values: `tcp` `http` Default value: `tcp`. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	Specifies the response timeout for a TCP health check. If a backend ECS instance does not respond within this time, the health check fails. Unit: seconds. Value range: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is less than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, this timeout is ignored, and the interval value is used instead. Default value: 5. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	Specifies the number of consecutive successful health checks required to change the health status of a backend server from `fail` to `success`. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	Specifies the number of consecutive failed health checks required to change the health status of a backend server from success to fail. Value range: [2, 10]. Default value: 3. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	Specifies the interval between health checks, in seconds. Value range: [1, 50]. Default value: 2. For more information, see CreateLoadBalancerTCPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

UDP health checks

Annotation: The annotations in the table below are required.

Health checks are enabled by default for UDP ports.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch	Specifies whether to enable health checks for TCP and UDP listeners. Valid values: `on` `off` Default value: `on`	v2.6.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The timeout period for a health check response. If a backend server does not respond within this period, the health check fails. Unit: seconds. The value must be an integer from 1 to 300. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` is less than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout` setting is invalid, and the timeout is set to the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`. The default value is 5. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to mark a backend server as healthy. The value must be an integer from 2 to 10. The default value is 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to mark a backend server as unhealthy. The value must be an integer from 2 to 10. The default value is 3. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between health checks, in seconds. The value must be an integer from 1 to 50. The default value is 2. For more information, see CreateLoadBalancerUDPListener.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable TCP and UDP health checks

annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch

Description

Supported CCM versions

Specifies whether to enable the health check for TCP and UDP listeners. Valid values:

on
off

Default value: on

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off" # Disables the health check.
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

HTTP health check

Annotations: The following table describes the available annotations.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Valid values: `on`: The TCP listener is `on` by default and cannot be changed. You do not need to change this annotation parameter for TCP listeners. `off`: An HTTP listener is `off` by default. Default value: `off`	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	Specifies the health check type. Valid values: `tcp` `http` Default value: `tcp` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The URI for the health check. This annotation applies only to HTTP health checks.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode	The HTTP status codes that indicate a successful health check. Separate multiple codes with a comma (`,`). Valid values: `http_2xx` `http_3xx` `http_4xx` `http_5xx` Default value: `http_2xx` For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name used for the health check. Valid values: `$_ip`: Uses the private IP of the backend server for the health check. This is the default if you omit the annotation. `domain`: A domain name, 1 to 80 characters long, containing only letters, digits, periods (.), and hyphens (-). For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout	The maximum time to wait for a health check response. If a backend ECS instance fails to respond within this period, the health check fails. Unit: seconds. Value range: [1, 300]. If the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` is less than the value of `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval`, the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout` setting is ignored, and the timeout defaults to the value of the `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval` annotation. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required to mark a failed backend server as healthy. The value must be an integer from 2 to 10. The default value is 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required to mark a healthy backend server as failed. The value must be an integer from 2 to 10. The default value is 3. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between health checks, in seconds. The value must be an integer from 1 to 50. The default value is 2. For more information, see CreateLoadBalancerHTTPListener.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with a comma (`,`), for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The HTTP method for the health check. Valid values: `head` `get`	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    # Optional. Configure the HTTP status code for health checks.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode: "http_4xx"
    # Optional. Configure the domain name for health checks.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.aliyun.com"
    # Optional. Configure the health check method.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Enable connection draining

Annotation: To enable this feature, you must specify both annotations described in the following table.

This feature supports only TCP and UDP listeners.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Enables or disables connection draining.

on
off

v2.0.1 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

The timeout for connection draining, in seconds. Value range: 10-900.

v2.0.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure listener request headers

annotation: The available annotations are described in the following table.

Only HTTP and HTTPS are supported.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Use a comma (,) to separate multiple values, for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto	Specifies whether to use the 'X-Forwarded-Proto' request header to retrieve the listener protocol. Valid values: `on` `off` Default value: `off`	v2.1.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport	Specifies whether to use the 'XForwardedFor_SLBPORT' request header to retrieve the listener port of the load balancer instance. Valid values: `on` `off` Default value: `off`	v2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport	Specifies whether to use the 'XForwardedFor_ClientSrcPort' request header to retrieve the client source port. Valid values: `on` `off` Default value: `off`	v2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Connection idle timeout

annotation: Multiple. See the following table.

Only HTTP and HTTPS are supported.

Annotation

Description

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Separate multiple values with a comma (,), for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

Specifies the connection idle timeout for the listener, in seconds. The value must be from 1 to 60.

Default: 15

v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable HTTP/2 for a listener

Annotations: This feature uses the annotations listed in the following table.

This feature applies only to HTTPS listeners.

Annotation	Description	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Use a comma (,) to separate multiple values, for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The certificate ID on Alibaba Cloud. Log on to the CLB console and find the certificate ID on the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-signed certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled	Enables or disables HTTP/2. Valid values: `on` `off` Default value: `on`	v2.1.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set listener request timeout

Annotation: The following table describes the available annotations.

Only HTTP and HTTPS listeners are supported.

Annotation

Description

Supported CCM version

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port

Use a comma to separate multiple values, for example, https:443,http:80.

v1.9.3 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout

The request timeout, in seconds. The value must be an integer from 1 to 180. Default value: 60.

If a backend server fails to respond within the specified timeout, the load balancer returns an HTTP 504 error to the client.

v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the established timeout

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout

Only the TCP protocol is supported.

Description	Supported CCM versions
The established timeout in seconds. Valid values: [10, 900]. For details, see CreateLoadBalancerTCPListener.	v2.3.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure listener security policy

Annotation: The following table describes the available annotations.

Only the HTTPS protocol is supported.

Annotation	Description	Supported CCM version
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Separate multiple values with a comma, for example, `https:443,http:80`.	v1.9.3 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	Specifies the ID of the certificate on Alibaba Cloud. To find the ID, log in to the CLB console and navigate to the Certificate Management page. Note To create a certificate, see Select an Alibaba Cloud-issued certificate.	v1.9.3.164-g2105d2e-aliyun and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	A security policy defines the supported TLS protocol versions and cipher suites for an HTTPS listener. For more information, see CreateLoadBalancerHTTPSListener. Valid values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3` Default value: `tls_cipher_policy_1_0`	v2.4.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${YOUR_CERT_ID}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

TCP and UDP on a listener

Note

This feature requires Kubernetes cluster v1.24 or later. To upgrade the K8s version of an ACK cluster, see Upgrade the K8s version of an ACK cluster.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Configure proxy protocol

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

When you access a service from within the cluster by using its Cloud Load Balancer (CLB) IP address, traffic may be interrupted and cause access issues. To force traffic to hairpin through the CLB, add the required annotation as described in Set a Hostname for a Service. For more information, see Notes on Accessing the External IP Address of a LoadBalancer Service Within a Cluster.

Description

Supported CCM version

Configures Proxy Protocol for TCP and UDP listeners. When enabled, the listener uses Proxy Protocol to pass the client source address to backend servers. Valid values:

on
off

Default: off

Important

This feature does not support seamless online changes. Toggling Proxy Protocol on or off requires a service restart. Therefore, configure this setting with caution.

v2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Typical operations for backend server groups

Use worker nodes with specific labels

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label

Description	Supported CCM versions
Specifies the worker nodes to add as backend servers for a Classic Load Balancer (CLB) instance based on their labels. Separate multiple labels with a comma (,), for example, `"k1=v1,k2=v2"`. Multiple labels use AND logic. This annotation applies only when the backend server type is set to `ecs`.	v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label: "failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use nodes that host pods

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

By default, externalTrafficPolicy is set to Cluster, which adds all nodes in the cluster to the backend server group. In Local mode, only nodes that run the service's Pods become backend servers.

Description

Supported CCM versions

The scheduling algorithm. Valid values:

wrr: weighted round robin. A backend server with a higher weight receives a higher proportion of requests.
For services in Local mode, you must set the scheduling algorithm to weighted round robin (wrr).
Note
For CCM versions v1.9.3.164-g2105d2e-aliyun and later, if a service has its externalTrafficPolicy set to Local, the CCM automatically sets the weight for each node based on the number of Pods running on it. For more information about how weights are calculated, see How are node weights automatically set in Local mode?
rr: round robin. Distributes requests to backend servers sequentially.

Default value: rr

v1.9.3 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Remove unschedulable nodes

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend

Description

Supported CCM versions

Specifies whether to remove nodes with a SchedulingDisabled status from the backend server group of a Classic Load Balancer (CLB) instance. Valid values:

on: Removes nodes in the unschedulable state from the CLB backend server group.
off: The kubectl cordon and kubectl drain commands mark a node as unschedulable. This value keeps nodes in the unschedulable state within the CLB backend server group.

Default value: off

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Attach Pod ENIs to the backend server group

Annotation: service.beta.kubernetes.io/backend-type

Description

Default

Supported CCM versions

The type of backend server for a Classic Load Balancer (CLB) instance. Valid values:

eni: Attaches Pods to the CLB backend server group, improving network forwarding performance. This value applies only to the Terway network mode.
By changing the annotation service.beta.kubernetes.io/backend-type: "eni", you can also change the value from eni to ecs to attach Elastic Compute Service (ECS) instances to the CLB backend server group.
ecs: Attaches Elastic Compute Service (ECS) instances to the CLB backend server group.

Flannel network mode: ecs
Terway network mode:
- For Terway clusters created before August 10, 2020: ecs
- For Terway clusters created on or after August 10, 2020: eni

v1.9.3.164-g2105d2e-aliyun and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/backend-type: "eni"
  name: nginx
spec:
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Reuse an existing VServer Group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

Reuses an existing VServer Group. This annotation applies only when you reuse an existing Classic Load Balancer (CLB) instance. For a usage example, see Deploy services across clusters by reusing an existing load balancer.

Set the traffic weight for a Service

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple Services reuse the same Classic Load Balancer (CLB) instance, you can use this annotation to set the traffic weight for the current Service. This annotation applies only when you reuse an existing VServer Group. For a usage example, see Deploy services across clusters by reusing an existing load balancer.

Ignore backend server weight updates

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update

Description

Supported CCM versions

Specifies whether to skip updating the weights of backend servers in the VServer Group during Service reconciliation. This is useful when backend server weights are managed by another mechanism. Valid values:

on
off

Default value: off

v2.11.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Set the default weight for backend servers

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-default-weight

Description

Supported CCM versions

Sets the default weight for servers in the backend server group. A backend server with a higher weight receives more traffic.

The value must be an integer from 0 to 100. If this annotation is omitted, the system uses a default weight based on the network type.

This setting does not apply when the backend instances are Elastic Compute Service (ECS) instances and the Service's externalTrafficPolicy is set to Local.

v2.14.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    # Sets the default weight for servers in the backend server group to 35.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-default-weight: "35"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer