What annotations are supported for configuring an NLB instance in an ACK cluster - Container Compute Service

Network Load Balancer (NLB) is a Layer 4 load balancing service from Alibaba Cloud that offers high performance and auto-scaling. You can use annotations in your YAML files to configure a wide range of load balancing features. This topic describes how to use annotations to configure NLB instances, listeners, and server groups.

Precautions

The version of the cloud controller manager (CCM) component in the ACK cluster must be 2.5.0 or later. To upgrade the component, see Manage components.
In the service, set spec.loadBalancerClass to alibabacloud.com/nlb. If you do not specify this field, a Classic Load Balancer (CLB) instance is created by default.
After a service is created, you cannot change the value of spec.loadBalancerClass. You cannot convert a CLB instance to an NLB instance or vice versa.

Typical NLB operations

Create a public-facing NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps

Description

Default value

Supported CCM versions

Specifies the zones for the NLB instance.

To view the regions and zones that NLB supports, log on to the NLB console. You must specify at least two different zones. Separate multiple zones with commas (,), for example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.

Note

NLB supports multi-zone deployment. To ensure high availability, select vSwitches in at least two different zones if the current region has two or more zones.

None

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-***** and cn-hangzhou-j:vsw-*****. To ensure high availability, select vSwitches in at least two different zones.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a private NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Description

Default value

Supported CCM versions

Specifies the network type of the NLB instance. You can change the value of this annotation to switch between a public-facing and a private NLB instance. Valid values:

internet: a public-facing NLB instance.
intranet: a private NLB instance.

Note

NLB supports multi-zone deployment. To ensure high availability, select vSwitches in at least two different zones if the current region has two or more zones.

internet

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. To ensure high availability, select vSwitches in at least two different zones.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify a name for the NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

Description	Default value	Supported CCM versions
Specifies the name of the NLB instance. The name must be 2 to 128 characters in length, start with a letter or a Chinese character, and can contain digits, periods (.), underscores (_), and hyphens (-).	None	2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "${your-nlb-name}" # The name of the NLB instance.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the resource group for the NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Description

Default value

Supported CCM versions

Specifies the resource group to which the NLB instance belongs. After you specify the resource group ID, you cannot change it.

You can log on to the Alibaba Cloud Resource Management platform to query the resource group ID.

None

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id:  "${your-resource-group-id}" # The ID of the resource group.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a dual-stack NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Description

Default value

Supported CCM versions

Specifies the IP version of the NLB instance. After the instance is created, you cannot change the IP version. Valid values:

ipv4: IPv4.
DualStack: dual-stack. The instance supports both IPv4 and IPv6.
- The two vSwitches specified in service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps must have IPv6 enabled.
- The generated IPv6 address can be accessed only in an IPv6-enabled environment.

ipv4

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Add extra tags to the NLB instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Description	Default value	Supported CCM versions
Adds extra tags to the instance. Separate multiple tags with commas (,), for example, `k1=v1,k2=v2`.	None	2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Use an existing NLB instance

Annotation: Multiple, as shown in the following table.

Annotation

Description

Default value

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id

The ID of an existing NLB instance.

None

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

Specifies whether to synchronize the NLB listener configurations based on the service. Valid values:

true: CCM creates, updates, and deletes NLB listeners based on the service configuration.
false: CCM does not perform any operations on the NLB listeners.

false

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${your-nlb-id}" # The ID of the NLB instance.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Attach an Internet Shared Bandwidth instance

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id

Description

Default value

Supported CCM versions

The ID of the Internet Shared Bandwidth instance to attach.

You can log on to the VPC console to query the ID of the Internet Shared Bandwidth instance.

None

2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id: "cbwp-xxxxxxxxxx" 
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the network type for IPv6

Annotation: Multiple, as shown in the following table.

Annotation

Description

Default value

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Specifies the IP version of the NLB instance. After the instance is created, you cannot change the IP version. When you use this annotation, the kube-proxy mode of the cluster must be IPVS. Valid values:

ipv4: IPv4.
DualStack: dual-stack. The instance supports both IPv4 and IPv6.
- The two vSwitches specified in service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps must have IPv6 enabled.
- The generated IPv6 address can be accessed only in an IPv6-enabled environment.

ipv4

2.5.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type

Specifies the IPv6 network type of the NLB instance. Valid values:

intranet: private IPv6
internet: public IPv6

Note

When you enable a public IPv6 address, an IPv6 gateway must be available in the VPC where the NLB instance resides. For more information, see Create and manage an IPv6 gateway.

intranet

2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type: internet # Set the IPv6 network type to public.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Retain the auto-created NLB instance when deleting the service

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete

Description

Supported CCM versions

Retains the NLB instance created by the service when you delete the LoadBalancer service. The kubernetes.do.not.delete and ack.aliyun.com tags on the NLB instance and server group are removed. The existing servers in the server group are retained.

When this feature is enabled, a PreservedOnDelete warning event is generated during service synchronization. After you configure this annotation, we recommend that you check whether this event exists to confirm that the feature is enabled.

Valid values:

Not empty: enables the retention feature.
Empty or not set: disables the retention feature.

Important

Perform this operation by deleting the service instead of changing the service type. Otherwise, the service may be incorrectly re-associated with the previously retained NLB instance.

2.10.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Typical listener operations

Set security groups for a listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids

Description	Default value	Supported CCM versions
Sets the security groups for the listener. Separate multiple security group IDs with commas (,), for example, `sg-aaaaa,sg-bbbbb`.	None	2.6.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids: "sg-aaaaa,sg-bbbbb" # Separate multiple security groups with commas.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure TCP and UDP protocols for a listener

Description	Default value	Supported CCM versions
This feature requires the Kubernetes cluster to be version 1.24 or later. For more information about how to upgrade a cluster, see Upgrade an ACK cluster.	None	Not applicable

apiVersion: v1
kind: Service
metadata:
  annotations:
      service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a TCP listener

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a UDP listener

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
  name: nginx
  namespace: default
spec:
  ports:
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a TCP-over-SSL listener

Annotation: Multiple, as shown in the following table.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Sets the protocol for the listener. Separate multiple values with a comma (,), for example, `TCP:80,TCPSSL:443`.	None	2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the server certificate. You can log on to the Certificate Management Service console to create and view the certificate.	None	2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    # If the cluster is in a region in the Chinese mainland, the combined certificate ID is ${your-cert-id}-cn-hangzhou.
    # If the cluster is in a region outside the Chinese mainland, the combined certificate ID is ${your-cert-id}-ap-southeast-1.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The combined certificate ID}"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable mutual authentication

Annotation: Multiple, as shown in the following table.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Sets the protocol for the listener. Separate multiple values with a comma (,), for example, `TCP:80,TCPSSL:443`.	None	2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the server certificate. You can log on to the Certificate Management Service console to view the certificate.	None
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert-id	The ID of the CA certificate. You can log on to the Certificate Management Service console to view the certificate.	None
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert	Specifies whether to enable mutual authentication. Valid values: `on`: Enabled. `off`: Shutdown.	None

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"   
    # If the cluster is in a region in the Chinese mainland, the combined certificate ID is ${your-cert-id}-cn-hangzhou.
    # If the cluster is in a region outside the Chinese mainland, the combined certificate ID is ${your-cert-id}-ap-southeast-1.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The combined certificate ID}" 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert-id: "${your-cacert-id}"  # The ID of the CA certificate.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Set a TLS security policy

Annotation: Multiple, as shown in the following table.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port	Sets the protocol for the listener. Separate multiple values with a comma (,), for example, `TCP:80,TCPSSL:443`.	None	2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id	The ID of the server certificate. You can log on to the Certificate Management Service console to view the certificate.	None
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy	The ID of the security policy. System security policies and custom security policies are supported. Valid values: `tls_cipher_policy_1_0` `tls_cipher_policy_1_1` `tls_cipher_policy_1_2` `tls_cipher_policy_1_2_strict` `tls_cipher_policy_1_2_strict_with_1_3`	tls_cipher_policy_1_0

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    # If the cluster is in a region in the Chinese mainland, the combined certificate ID is ${your-cert-id}-cn-hangzhou.
    # If the cluster is in a region outside the Chinese mainland, the combined certificate ID is ${your-cert-id}-ap-southeast-1.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${The combined certificate ID}" 
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_0"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Set Proxy Protocol

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

Description

Default value

Supported CCM versions

Specifies whether to use the Proxy Protocol to pass client IP addresses to backend servers. Valid values:

on: enables the feature.
off: disables the feature.

Important

Before you enable Proxy Protocol, make sure that Proxy Protocol v2 is enabled for the backend service. Otherwise, access may fail. Configure this parameter with caution.

off

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Carry additional information over Proxy Protocol

Annotation: Multiple, as shown in the following table.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol	Specifies whether to use the Proxy Protocol to pass client IP addresses to backend servers. Valid values: `on`: enables the feature. `off`: disables the feature. Important Before you enable Proxy Protocol, make sure that Proxy Protocol v2 is enabled for the backend service. Otherwise, access may fail. Configure this parameter with caution.	off	2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-vpc-id-enabled	Specifies whether to carry the VpcId to the backend server over Proxy Protocol. Valid values: `on`: enables the feature. `off`: disables the feature.	off	2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-ep-id-enabled	Specifies whether to carry the PrivateLinkEpId to the backend server over Proxy Protocol. Valid values: `on`: enables the feature. `off`: disables the feature.	off	2.9.1 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-eps-id-enabled	Specifies whether to carry the PrivateLinkEpsId to the backend server over Proxy Protocol. Valid values: `on`: enables the feature. `off`: disables the feature.	off	2.9.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-ep-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-eps-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-vpc-id-enabled: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Set the maximum number of new connections per second for a listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps

Description	Default value	Supported CCM versions
The maximum number of new connections that can be created per second on the NLB instance. Valid values: 0 to 1000000. A value of 0 indicates that the number of new connections is unlimited.	None	2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps: "100"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Set the idle connection timeout period for a listener

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

Description	Default value	Supported CCM versions
The idle connection timeout period. Unit: seconds. Valid values: 10 to 900.	900	2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "60"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure a listener port range

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-listener-port-range

Description

Supported CCM versions

By configuring a listener port range for all-port forwarding on the NLB instance, you can listen on a specified port range and forward traffic to the corresponding ports of the backend servers.

The format is Port range:Service port. Separate multiple values with commas (,), for example, 80-100:80,400-500:443. The ports and port ranges for the same protocol cannot overlap.

targetPort is the health check port of the backend server. The value must be an integer from 1 to 65535.

2.11.4 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    # Configure listeners for the 80-100 and 400-500 port ranges.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-listener-port-range: "80-100:80,400-500:443"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    # targetPort is the health check port of the backend server. The value must be an integer.
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Typical server group operations

Set a scheduling algorithm

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

Description

Default value

Supported CCM versions

The scheduling algorithm. Valid values:

wrr (default): weighted round-robin. Backend servers with higher weights receive more requests than those with lower weights.
rr: round-robin. Requests are distributed to backend servers in sequence.
sch: source IP hash. Requests from the same source IP address are forwarded to the same backend server.
tch: four-tuple hash. A consistent hash based on a four-tuple (source IP, destination IP, source port, and destination port). The same stream is scheduled to the same server.
wlc: weighted least connections. In addition to the weight of each backend server, the number of connections to each backend server is also considered. When backend servers have the same weight, the backend server with the fewest active connections receives more requests.

For more information about the possible values of this option, see the Scheduler field in CreateServerGroup.

wrr

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "sch"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure connection draining

Annotation: Multiple, as shown in the following table.

Annotation

Description

Default value

Supported CCM versions

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain

Specifies whether to enable connection draining. Valid values:

on: Enabled.
off: The shutdown state.

off

2.5.0 and later

service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

The timeout period for connection draining. Unit: seconds. Valid values: 10 to 900.

None

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable client IP preservation

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip

Description

Default value

Supported CCM versions

Specifies whether to enable client IP preservation. Valid values:

on: enables the feature.
off: disables the feature.

2.5.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip: "on"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure TCP health checks

Annotation: Multiple, as shown in the following table. These annotations configure a TCP health check. All the following annotations are required. Health checks are enabled for TCP ports by default.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Specifies whether to enable health checks. Valid values: `on`: Enabled. `off`: Shutdown.	on
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check protocol. Valid values: `tcp` `http`	tcp
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-port	The port on the backend server that is used for health checks. Valid values: 0 to 65535. The default value 0 indicates that the port of the backend server is used.	0
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The maximum timeout period for a health check response. Unit: seconds. Valid values: 1 to 300.	5
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required before a backend server is declared healthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required before a backend server is declared unhealthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Valid values: 1 to 50.	10

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
  name: nginx
  namespace: default
spec:  
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure HTTP health checks

Annotation: Multiple, as shown in the following table. These annotations configure an HTTP health check. All the following annotations are required. Health checks are enabled for TCP ports by default.

Annotation	Description	Default value	Supported CCM versions
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag	Specifies whether to enable health checks. Valid values: `on`: Enabled. `off`: Specifies a shutdown.	on	2.5.0 and later
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type	The health check protocol. Valid values: `tcp` `http`	http
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri	The path for health checks. The path must be 1 to 80 characters in length and can contain letters, digits, and special characters. It must start with a forward slash (/). For more information, see CreateServerGroup. Note This parameter takes effect only when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` is set to `HTTP`.	None
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain	The domain name for health checks. Valid values: $_ip: Use the private IP address of the server. domain: A specific domain name. The domain name must be 1 to 80 characters in length and can contain lowercase letters, digits, hyphens (-), and periods (.). Note This parameter takes effect only when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` is set to `HTTP`.	None
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-port	The port on the backend server that is used for health checks. Valid values: 0 to 65535. The default value 0 indicates that the port of the backend server is used.	0
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout	The maximum timeout period for a health check response. Unit: seconds. Valid values: 1 to 300.	5
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold	The number of consecutive successful health checks required before a backend server is declared healthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold	The number of consecutive failed health checks required before a backend server is declared unhealthy. Valid values: 2 to 10.	2
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval	The interval between two consecutive health checks. Unit: seconds. Valid values: 1 to 50.	10
service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method	The health check method. Valid values: `GET` `HEAD` Note This parameter takes effect only when `service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type` is set to `HTTP`.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.test.com"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    # Set the health check method. This annotation is optional.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Set the server group type

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type

Description

Default value

Supported CCM versions

Sets the server group type. Valid values:

Ip: IP address type. You can directly add backend servers of the IP address type.
Instance (default): Instance type. You can add instances of the ECS and ENI types.

For more information about the classification and description of NLB server groups, see NLB server groups.

Instance

2.8.0 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type: "Ip"
  name: nginx
  namespace: default
spec:
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Reuse an existing server group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

This annotation lets you reuse an existing vServer group. This annotation takes effect only when you reuse an existing NLB instance. For a usage example, see Deploy services across clusters by reusing an existing SLB instance.

Set the weight of a service to receive traffic

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple services reuse the same NLB instance, you can use this annotation to set the traffic weight for the current service. This annotation takes effect only when you reuse an existing server group. For a usage example, see Deploy services across clusters by reusing an existing SLB instance.

Ignore backend server weight updates

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update

Description

Supported CCM versions

Skips updating the weights of backend servers in the server group during service synchronization. This configuration is suitable for scenarios where you need to manually manage the weights of backend servers. Valid values:

on
off

Default value: off

2.11.1 and later

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer