All Products
Search
Document Center

Container Service for Kubernetes:Configure Classic Load Balancer (CLB) Using Annotations

Last Updated:Mar 26, 2026

Configure Classic Load Balancer (CLB) instances, listeners, and backend server groups by adding annotations to Service YAML files. The Cloud Controller Manager (CCM) reads these annotations and applies your settings to the corresponding CLB resources.

All annotation values are case-sensitive. Before using an annotation, verify that your CCM version meets the minimum version requirement listed in this topic. To upgrade CCM, see Manage components. For CCM release notes, see Cloud Controller Manager.

All annotations use the prefix service.beta.kubernetes.io/. The tables in this topic show only the suffix after this prefix. For example, alibaba-cloud-loadbalancer-id refers to the full annotation key service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id.

Annotations in this topic use the alibaba-cloud prefix. The legacy alicloud prefix is still supported for backward compatibility. For example, service.beta.kubernetes.io/alicloud-loadbalancer-id and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id are equivalent.

CLB instance configuration

Create an Internet-facing CLB instance

By default, ACK creates an Internet-facing CLB instance when you create a Service of type LoadBalancer.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an internal CLB instance

Annotation: alibaba-cloud-loadbalancer-address-type

Value

Description

Min CCM

internet

Creates an Internet-facing CLB instance. This is the default value.

v1.9.3

intranet

Creates an internal-facing CLB instance accessible only within the VPC.

v1.9.3

For internal CLB instances, specify a vSwitch explicitly using alibaba-cloud-loadbalancer-vswitch-id. If you don't, CCM selects a vSwitch automatically based on its version:

  • CCM earlier than v2.13.0: uses the first vSwitch configured for the cluster control plane

  • CCM v2.13.0 or later: automatically selects an available vSwitch

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "${YOUR_VSWITCH_ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify the CLB instance type

Annotations: Multiple annotations required.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-instance-charge-type

PayBySpec: pay by specification. PayByCLCU: pay by usage.

PayBySpec (changes to PayByCLCU for CCM v2.5.0+)

v2.4.0

alibaba-cloud-loadbalancer-spec

slb.s1.small, slb.s2.small, slb.s2.medium, slb.s3.small, slb.s3.medium, slb.s3.large. For more values, see CreateLoadBalancer.

slb.s1.small

v1.9.3

Important

PayByCLCU and alibaba-cloud-loadbalancer-spec cannot be set at the same time. If you modify the CLB instance specification in the CLB console (supported only for PayBySpec instances), CCM may revert it to the value set in the annotation.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-instance-charge-type: "PayBySpec"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-spec: "slb.s1.small"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Create a pay-by-bandwidth CLB instance

Annotations: Both annotations are required.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-charge-type

paybytraffic / paybybandwidth

paybytraffic

v1.9.3

alibaba-cloud-loadbalancer-bandwidth

Bandwidth cap in Mbps. Applies only to Internet-facing CLB instances.

50

v1.9.3.10-gfb99107-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-charge-type: "paybybandwidth"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth: "2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing CLB instance

Annotation: alibaba-cloud-loadbalancer-id

Value

Example

Min CCM

Existing CLB instance ID

lb-bp1ids9hmq5924m6***

v1.9.3.81-gca19cd4-aliyun

Important
  • Do not reuse the API server CLB instance or any CLB instance created by CCM. Create a new instance in the CLB console before referencing it here.

  • Do not add or modify reuse annotations on existing LoadBalancer Services. This can cause reuse to fail or prevent CCM-created load balancers from being released properly.

  • By default, reusing a CLB instance does not override existing listeners. To force override, set alibaba-cloud-loadbalancer-force-override-listeners to "true". This overrides only listeners for the current Service.

  • (CCM earlier than v2.10.0) Adding extra tags via alibaba-cloud-loadbalancer-additional-resource-tags is not supported when reusing an existing CLB instance.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "<your-clb-instance-id>"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use an existing CLB instance and force override listeners

Annotations: Both annotations are required. Force override deletes existing listeners when port conflicts occur.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-id

Existing CLB instance ID

v1.9.3.81-gca19cd4-aliyun

alibaba-cloud-loadbalancer-force-override-listeners

"true": CCM manages listeners for this Service (create, update, delete). Other listeners are unaffected. "false": CCM does not manage listeners.

"false"

v1.9.3.81-gca19cd4-aliyun

Important

When force-override-listeners is "true", do not share the same listener across multiple Services — this causes listener configuration conflicts. Only force override if the listener port is no longer in use.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "<your-clb-instance-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify primary and secondary zones

Annotations: Both annotations are required. Zones cannot be changed after creation. Not all regions support primary and secondary zones — check the CLB console.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-master-zoneid

Zone ID of the primary backend server

cn-hangzhou-k

v1.9.3.10-gfb99107-aliyun

alibaba-cloud-loadbalancer-slave-zoneid

Zone ID of the secondary backend server

cn-hangzhou-j

v1.9.3.10-gfb99107-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-master-zoneid: "cn-hangzhou-k"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-slave-zoneid: "cn-hangzhou-j"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Specify a vSwitch for an internal CLB instance

Annotations: Both annotations are required.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-address-type

"intranet"

intranet

v1.9.3

alibaba-cloud-loadbalancer-vswitch-id

vSwitch ID. The vSwitch must belong to the same VPC as the ACK cluster. Find the ID in the VPC console.

vsw-bp1s5fnvk4gn2tws0****

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "<your-vswitch-id>"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Specify a static IP for an internal CLB instance

Annotations: All three annotations are required.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-address-type

"intranet"

intranet

v1.9.3

alibaba-cloud-loadbalancer-vswitch-id

vSwitch ID

vsw-bp1s5fnvk4gn2tws0****

v1.9.3

alibaba-cloud-loadbalancer-ip

IPv4 address within the vSwitch CIDR block. Cannot be changed after creation.

192.168.x.x

v2.7.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vswitch-id: "<your-vswitch-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip: "192.168.x.x"
  name: nginx
  namespace: default
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 80
      name: http
  selector:
    app: nginx

Create an IPv6 CLB instance

Annotation: alibaba-cloud-loadbalancer-ip-version

Value

Description

Default

Min CCM

ipv4

IPv4 address

Yes

v1.9.3.220-g24b1885-aliyun

ipv6

IPv6 address. Accessible only in IPv6-enabled environments.

v1.9.3.220-g24b1885-aliyun

The IP version cannot be changed after creation. The cluster's kube-proxy mode must be IPVS.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "ipv6"
  name: nginx
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Add tags to the CLB instance

Annotation: alibaba-cloud-loadbalancer-additional-resource-tags

Value

Example

Min CCM

Comma-separated key=value pairs

"Key1=Value1,Key2=Value2"

v1.9.3

Important

This annotation overwrites any tags you manually set on the CLB instance in the console. Starting with CCM v2.10.0, tags can be modified for both new and reused CLB instances.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the CLB instance name

Annotation: alibaba-cloud-loadbalancer-name

Value

Example

Min CCM

2–128 characters. Must start with a letter or Chinese character. Can contain letters, digits, periods (.), underscores (_), and hyphens (-).

your-svc-name

v1.9.3.313-g748f81e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "your-svc-name"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Assign the CLB instance to a resource group

Annotation: alibaba-cloud-loadbalancer-resource-group-id

Value

Example

Min CCM

Resource group ID. Cannot be changed after assignment. Find the ID in the Alibaba Cloud Resource Management consoleVirtual Private Cloud (VPC) consoleVirtual Private Cloud (VPC) console.

rg-xxxx

v1.9.3.313-g748f81e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "rg-xxxx"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable deletion protection

Annotation: alibaba-cloud-loadbalancer-delete-protection

Value

Default

Min CCM

on / off

on

v1.9.3.313-g748f81e-aliyun

Important

Even with deletion protection enabled in the CLB console, you can still delete the CLB instance by running kubectl delete svc <svc-name>.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-delete-protection: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Enable configuration modification protection

Annotation: alibaba-cloud-loadbalancer-modification-protection

Value

Description

Default

Min CCM

ConsoleProtection

Prevents modification from the CLB console

ConsoleProtection

v1.9.3.313-g748f81e-aliyun

NonProtection

No protection

v1.9.3.313-g748f81e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-modification-protection: "ConsoleProtection"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Preserve the CLB instance when deleting a Service

Annotation: alibaba-cloud-loadbalancer-preserve-lb-on-delete

Value

Description

Min CCM

Non-empty string (e.g., "true")

Preserve the CLB instance. CCM removes the kubernetes.do.not.delete and ack.aliyun.com tags from the instance. Existing servers in the vServer group remain unchanged.

v2.10.0

Empty or unset

Delete the CLB instance with the Service

v2.10.0

When enabled, a Warning event of type PreservedOnDelete appears during Service synchronization — check for this event to confirm the annotation is active.

Important

To trigger preservation, delete the Service using kubectl delete svc. Do not change the Service type instead, as this may cause the Service to incorrectly reattach to the preserved CLB instance.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set a hostname for the Service

Annotations: Both annotations are required.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

https:443,http:80

v1.9.3

alibaba-cloud-loadbalancer-hostname

DNS-compliant hostname

www.example.com

v2.3.0

After adding this annotation:

  • The Service's EXTERNAL-IP changes from the CLB IP to the hostname you specified.

  • Traffic from inside the cluster routes through the CLB before forwarding back to the cluster. For TCP or UDP listeners, this may cause loopback issues. See Clients cannot access load balancer.

  • The annotation does not automatically bind a domain name to the CLB. To bind a domain, purchase one on the Domain Names page. See Purchase a domain name.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-hostname: "<your-service-hostname>"
  name: nginx-svc
  namespace: default
spec:
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Expected output:

NAME         TYPE           CLUSTER-IP       EXTERNAL-IP            PORT(S)                      AGE
nginx-svc    loadBalancer   47.100.XX.XX     www.example.com        80:30248/TCP,443:32670/TCP   10s

Listener configuration

Set the scheduling algorithm

Annotation: alibaba-cloud-loadbalancer-scheduler

Value

Description

Default

Min CCM

rr

Round-robin: distribute requests sequentially to backend servers

rr

v1.9.3

wrr

Weighted round-robin: higher-weight servers receive more requests

v1.9.3

sch

Source IP hash: requests from the same source IP always go to the same backend server

v1.9.3

tch

Four-tuple hash (source IP + destination IP + source port + destination port): requests in the same flow go to the same backend server

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure an access control list

Annotations: All three annotations are required. Before applying, create an access control policy group in the CLB console and record its ACL ID.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-acl-status

on / off

off

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-acl-id

ACL ID. Required when acl-status is "on". Multiple policy groups are not supported.

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-acl-type

white: allow only IPs in the ACL. black: block IPs in the ACL. For both types, if the ACL is empty, all requests are forwarded.

v1.9.3.164-g2105d2e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-status: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-id: "<your-acl-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-acl-type: "white"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure session persistence for TCP listeners

Annotation: alibaba-cloud-loadbalancer-persistence-timeout

Value

Example

Default

Min CCM

Session persistence timeout in seconds. Range: [0, 3600]. 0 disables session persistence. If the CLB instance has multiple TCP listeners, the setting applies to all of them.

1800

0

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-persistence-timeout: "1800"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure session persistence for HTTP and HTTPS listeners

Annotations: All four annotations are required when inserting cookies. Applies only to HTTP and HTTPS listeners. If multiple HTTP or HTTPS ports are configured, the setting applies to all of them.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-sticky-session

on / off

on

off

v1.9.3

alibaba-cloud-loadbalancer-sticky-session-type

insert: insert cookie. server: rewrite cookie. Required when sticky-session is on.

insert

v1.9.3

alibaba-cloud-loadbalancer-cookie-timeout

Cookie timeout in seconds. Range: [1, 86400]. Required when sticky-session-type is insert.

1800

v1.9.3

alibaba-cloud-loadbalancer-cookie

Cookie name set on the server. 1–200 ASCII alphanumeric characters. Cannot contain ,, ;, or spaces, and cannot start with $. Required when sticky-session-type is server.

my-cookie

v1.9.3

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

https:443,http:80

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-sticky-session-type: "insert"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cookie-timeout: "1800"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Redirect HTTP to HTTPS

Port forwarding redirects HTTP requests to an HTTPS port. All three annotations are required.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

https:443,http:80

v1.9.3

alibaba-cloud-loadbalancer-cert-id

Certificate ID. Find it on the Certificate Management page in the CLB console. To create a certificate, see Select an Alibaba Cloud-issued certificate.

<your-cert-id>

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-forward-port

HTTP-to-HTTPS port mapping

80:443

v1.9.3.164-g2105d2e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "<your-cert-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-forward-port: "80:443"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTP listener

Annotation: alibaba-cloud-loadbalancer-protocol-port

Value

Example

Min CCM

Protocol and port pairs

http:80

v1.9.3

Health checks for HTTP listeners are disabled by default. Configure HTTP or TCP health checks to ensure backend service availability.

When accessing the Service via the CLB IP from inside the cluster, traffic may be intercepted internally. To force traffic through the CLB, add the alibaba-cloud-loadbalancer-hostname annotation. See Notes on accessing the External IP of a LoadBalancer Service from inside the cluster.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Create an HTTPS listener

Annotations: Both annotations are required. HTTPS requests are decrypted at the CLB layer and forwarded to backend Pods as HTTP.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

https:443

v1.9.3

alibaba-cloud-loadbalancer-cert-id

Certificate ID. Find it on the Certificate Management page in the CLB console.

<your-cert-id>

v1.9.3.164-g2105d2e-aliyun

Health checks for HTTPS listeners are disabled by default. Configure HTTP or TCP health checks to ensure backend service availability.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "<your-cert-id>"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

SSL protocol error when accessing CLB from inside the cluster

Symptom: After creating an HTTPS listener, the Service is accessible from outside the cluster. But when you run curl against the CLB's HTTPS port from a node or Pod inside the cluster, this error appears:

curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Cause: IPVS rules on the node forward traffic from port 443 directly to port 80 on the backend. Because IPVS operates at Layer 4, it doesn't parse TLS. The backend's HTTP port receives a TLS handshake it can't process and returns an HTTP 400 error.

Solution: Add the alibaba-cloud-loadbalancer-hostname annotation to the Service. This prevents IPVS rules from being created on the node and forces inside-cluster traffic to route through the CLB, where TLS termination happens correctly.

Configure extended domain name certificates for HTTPS listeners

Annotations: Multiple annotations required.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

https:443

v1.9.3

alibaba-cloud-loadbalancer-cert-id

Default certificate ID

<your-cert-id>

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-domain-extensions

Extended domain-to-certificate mappings in {domain}:{cert-id} format, comma-separated

example.com:cert-id-1,api.example.com:cert-id-2

v2.13.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "<your-cert-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-domain-extensions: "<domain-1>:<cert-id-1>,<domain-2>:<cert-id-2>"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure a TLS security policy

Annotations: All three annotations are required. Applies only to HTTPS listeners.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

v1.9.3

alibaba-cloud-loadbalancer-cert-id

Certificate ID

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-tls-cipher-policy

tls_cipher_policy_1_0, tls_cipher_policy_1_1, tls_cipher_policy_1_2, tls_cipher_policy_1_2_strict, tls_cipher_policy_1_2_strict_with_1_3

tls_cipher_policy_1_0

v2.4.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443,http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "<your-cert-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: http
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure both TCP and UDP for a single listener

Requires Kubernetes v1.24 or later. To upgrade, see Upgrade an ACK cluster.
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  type: LoadBalancer

Configure Proxy Protocol

Annotation: alibaba-cloud-loadbalancer-proxy-protocol

Applies to TCP and UDP listeners. Proxy Protocol passes the client's source IP to the backend server.

Value

Default

Min CCM

on / off

off

v2.6.0

Important

Switching Proxy Protocol does not support live migration — enabling or disabling it requires a service outage. Plan accordingly.

When accessing a Service through the CLB IP from inside the cluster, traffic may be intercepted. See Notes on accessing the External IP of a LoadBalancer Service from inside the cluster.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure connection draining

Annotations: Both annotations are required. Applies only to TCP and UDP listeners.

Annotation suffix

Value

Example

Min CCM

alibaba-cloud-loadbalancer-connection-drain

on / off

on

v2.0.1

alibaba-cloud-loadbalancer-connection-drain-timeout

Draining timeout in seconds. Range: [10, 900].

30

v2.0.1

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Add forwarded request headers

Annotations: Applies only to HTTP and HTTPS listeners.

Annotation suffix

Value

Header passed

Default

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

v1.9.3

alibaba-cloud-loadbalancer-xforwardedfor-proto

on / off

X-Forwarded-Proto (CLB listener protocol)

off

v2.1.0

alibaba-cloud-loadbalancer-xforwardedfor-slbport

on / off

XForwardedFor_SLBPORT (CLB listener port)

off

v2.9.1

alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport

on / off

XForwardedFor_ClientSrcPort (client source port)

off

v2.9.1

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-proto: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-slbport: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-xforwardedfor-clientsrcport: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the idle connection timeout

Annotations: Applies only to HTTP and HTTPS listeners.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

http:80

v1.9.3

alibaba-cloud-loadbalancer-idle-timeout

Idle timeout in seconds. Range: [1, 60].

30

15

v2.1.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the request timeout

Annotations: Applies only to HTTP and HTTPS listeners.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

http:80

v1.9.3

alibaba-cloud-loadbalancer-request-timeout

Request timeout in seconds. Range: [1, 180]. If the backend doesn't respond within this period, CLB returns HTTP 504 to the client.

60

60

v2.3.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-request-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Set the connection timeout

Annotation: alibaba-cloud-loadbalancer-established-timeout

Applies only to TCP listeners.

Value

Example

Min CCM

Connection timeout in seconds. Range: [10, 900].

60

v2.3.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-established-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable HTTP/2

Annotations: Applies only to HTTPS listeners.

Annotation suffix

Value

Default

Min CCM

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

v1.9.3

alibaba-cloud-loadbalancer-cert-id

Certificate ID

v1.9.3.164-g2105d2e-aliyun

alibaba-cloud-loadbalancer-http2-enabled

on / off

on

v2.1.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "https:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "<your-cert-id>"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-http2-enabled: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure health checks for TCP listeners

All annotations are required. TCP port health checks are enabled by default.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-health-check-switch

on / off

on

on

v2.6.0

alibaba-cloud-loadbalancer-health-check-type

tcp / http

tcp

tcp

v1.9.3

alibaba-cloud-loadbalancer-health-check-connect-timeout

TCP response timeout in seconds. Range: [1, 300]. If this value is less than the check interval, the interval takes effect.

8

5

v1.9.3

alibaba-cloud-loadbalancer-healthy-threshold

Consecutive successes to mark a backend healthy. Range: [2, 10].

4

3

v1.9.3

alibaba-cloud-loadbalancer-unhealthy-threshold

Consecutive failures to mark a backend unhealthy. Range: [2, 10].

4

3

v1.9.3

alibaba-cloud-loadbalancer-health-check-interval

Check interval in seconds. Range: [1, 50].

3

2

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Configure health checks for UDP listeners

All annotations are required. UDP port health checks are enabled by default.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-health-check-switch

on / off

on

on

v2.6.0

alibaba-cloud-loadbalancer-health-check-connect-timeout

TCP response timeout in seconds. Range: [1, 300]. If this value is less than the check interval, the interval takes effect.

10

5

v1.9.3

alibaba-cloud-loadbalancer-healthy-threshold

Consecutive successes to mark a backend healthy. Range: [2, 10].

3

3

v1.9.3

alibaba-cloud-loadbalancer-unhealthy-threshold

Consecutive failures to mark a backend unhealthy. Range: [2, 10].

3

3

v1.9.3

alibaba-cloud-loadbalancer-health-check-interval

Check interval in seconds. Range: [1, 50].

5

2

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "3"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: UDP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Disable health checks for TCP and UDP listeners

Annotation: alibaba-cloud-loadbalancer-health-check-switch

Value

Default

Min CCM

on / off

on

v2.6.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-switch: "off"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Configure HTTP health checks

Applies to HTTP and TCP listeners. All annotations are required.

Annotation suffix

Value

Example

Default

Min CCM

alibaba-cloud-loadbalancer-health-check-flag

on: enable (default for TCP, cannot be changed). off: disable (default for HTTP).

on

off

v1.9.3

alibaba-cloud-loadbalancer-health-check-type

tcp / http

http

tcp

v1.9.3

alibaba-cloud-loadbalancer-health-check-uri

URI for HTTP health check. Not required for TCP health checks.

/test/index.html

v1.9.3

alibaba-cloud-loadbalancer-health-check-httpcode

HTTP status codes that indicate a healthy backend. Comma-separated. Valid values: http_2xx, http_3xx, http_4xx, http_5xx.

http_4xx

http_2xx

v1.9.3

alibaba-cloud-loadbalancer-health-check-domain

Domain name for the health check. $_ip uses the backend's private IP. Domain: 1–80 characters, letters/digits/periods/hyphens only.

www.example.com

v1.9.3

alibaba-cloud-loadbalancer-health-check-timeout

HTTP response timeout in seconds. Range: [1, 300]. If less than the interval, the interval takes effect.

10

v1.9.3

alibaba-cloud-loadbalancer-healthy-threshold

Consecutive successes to mark a backend healthy. Range: [2, 10].

4

3

v1.9.3

alibaba-cloud-loadbalancer-unhealthy-threshold

Consecutive failures to mark a backend unhealthy. Range: [2, 10].

4

3

v1.9.3

alibaba-cloud-loadbalancer-health-check-interval

Check interval in seconds. Range: [1, 50].

3

2

v1.9.3

alibaba-cloud-loadbalancer-protocol-port

Protocol and port pairs

http:80

v1.9.3

alibaba-cloud-loadbalancer-health-check-method

HTTP health check method: head / get

head

v2.3.0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "3"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "http:80"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-httpcode: "http_4xx"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.aliyun.com"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Backend server group configuration

Filter backend servers by node label

Annotation: alibaba-cloud-loadbalancer-backend-label

Value

Example

Min CCM

Comma-separated key=value node labels. Multiple labels have an AND relationship. Effective only when the backend type is ecs.

"failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"

v1.9.3

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-backend-label: "failure-domain.beta.kubernetes.io/zone=ap-southeast-5a"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

Use only nodes running Pods as backend servers

Set externalTrafficPolicy to Local to attach only the nodes where Pods are scheduled, instead of all cluster nodes.

Local mode requires the scheduling algorithm to be wrr. For CCM v1.9.3.164-g2105d2e-aliyun and later, node weights are automatically calculated based on Pod count. See How are node weights automatically set in Local mode?

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "wrr"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    run: nginx
  type: LoadBalancer

Remove unschedulable nodes from the backend server group

Annotation: alibaba-cloud-loadbalancer-remove-unscheduled-backend

Value

Description

Default

Min CCM

on

Remove nodes marked as unschedulable (e.g., after kubectl cordon or kubectl drain) from the CLB backend server group.

v1.9.3.164-g2105d2e-aliyun

off

Keep unschedulable nodes in the CLB backend server group.

off

v1.9.3.164-g2105d2e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-remove-unscheduled-backend: "on"
  name: nginx
spec:
  externalTrafficPolicy: Local
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Use Pod ENIs as backend servers

Annotation: backend-type

Value

Description

Default

Min CCM

eni

Attach Pod Elastic Network Interfaces (ENIs) directly to the CLB backend. Improves network forwarding performance. Effective only in Terway network mode. To switch to ECS-based backend, change the value from eni to ecs in the service.beta.kubernetes.io/backend-type annotation.

Terway clusters created after August 10, 2020

v1.9.3.164-g2105d2e-aliyun

ecs

Attach ECS instances to the CLB backend.

Flannel clusters; Terway clusters created before August 10, 2020

v1.9.3.164-g2105d2e-aliyun

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/backend-type: "eni"
  name: nginx
spec:
  ports:
  - name: http
    port: 30080
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer

Reuse an existing vServer group

Annotation: alibaba-cloud-loadbalancer-vgroup-port

Reuse an existing vServer group. Effective only when reusing an existing CLB instance. For a usage example, see Deploy services across clusters by reusing an existing CLB instance.

Set the traffic weight for the Service

Annotation: alibaba-cloud-loadbalancer-weight

When multiple Services share the same CLB instance, set the traffic weight for the current Service. Effective only when reusing an existing vServer group. For a usage example, see Deploy services across clusters by reusing an existing CLB instance.

Ignore backend server weight updates

Annotation: alibaba-cloud-loadbalancer-ignore-weight-update

Value

Description

Default

Min CCM

on

During Service synchronization, skip updating backend server weights in the vServer group. Use this when managing weights manually outside of CCM.

v2.11.1

off

CCM updates backend server weights during synchronization.

off

v2.11.1

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - port: 443
    protocol: TCP
    targetPort: 443
  selector:
    run: nginx
  type: LoadBalancer

What's next