Container Service for Kubernetes:Configure Network Load Balancer (NLB) Using Annotations

NLB instance annotations

Listener annotations

Server group annotations

All annotations use the prefix service.beta.kubernetes.io/. For example, the full annotation key for alibaba-cloud-loadbalancer-zone-maps is service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps.

Create a public NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps

Specify zone-to-vSwitch mappings for the NLB instance. The format is {zone}:{vSwitchID}:{private IP address}:{EIP instance ID}. Separate multiple entries with commas.

• Zone and vSwitchID: Required. Specify at least two zones, for example cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321. Check the NLB console for supported regions and zones.

• Private IP address: Optional (v2.12.1 and later).

• EIP instance ID: Optional (v2.12.1 and later).

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a private NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type

Set to intranet to create a private NLB. Change this value at any time to switch the NLB between public and private networks.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify a private IP address for the NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps

Include a private IP address in the zone-maps format: {zone}:{vSwitchID}:{private IP address}.

apiVersion: v1
kind: Service
metadata:
  annotations:
    # For example, cn-hangzhou-k:vsw-i123456:10.1.0.1,cn-hangzhou-j:vsw-j654321:10.2.0.1
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A}:${private-ipv4-A},${zone-B}:${vsw-B}:${private-ipv4-B}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify an EIP for the NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps

Include an Elastic IP Address (EIP) instance ID in the zone-maps format: {zone}:{vSwitchID}:{private IP address}:{EIP instance ID}. Leave the private IP address field blank if not needed.

apiVersion: v1
kind: Service
metadata:
  annotations:
    # Leave the private IP address field blank if not needed.
    # For example, cn-hangzhou-k:vsw-i123456::eip-12345,cn-hangzhou-j:vsw-j654321::eip-54321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A}::${eip-A},${zone-B}:${vsw-B}::${eip-B}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the NLB name

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name

The name must be 2–128 characters, start with a letter or Chinese character, and may contain digits, periods (.), underscores (_), and hyphens (-).

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-name: "${your-nlb-name}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify the NLB resource group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id

Find the resource group ID in the Resource Management console.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-resource-group-id: "${your-resource-group-id}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a dual-stack NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version

Set to DualStack to support both IPv4 and IPv6. Requirements:

• Both vSwitches in alibaba-cloud-loadbalancer-zone-maps must have IPv6 enabled.

• The cluster's kube-proxy mode must be IPVS.

• The generated IPv6 address is only accessible from IPv6-enabled environments.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Specify IPv6 network type

Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type

When creating a dual-stack NLB, use alibaba-cloud-loadbalancer-ipv6-address-type to set the IPv6 network type.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ip-version: "DualStack"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ipv6-address-type: "internet"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Add tags to the NLB

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags

Specify key-value tags separated by commas, for example k1=v1,k2=v2. Modifying tags on existing and reused instances requires v2.10.0 or later.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-resource-tags: "Key1=Value1,Key2=Value2"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Reuse an existing load balancer

Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-id: "${your-nlb-id}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-force-override-listeners: "true"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Bind an Internet Shared Bandwidth package

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id

Find the bandwidth package ID in the VPC console.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-bandwidth-package-id: "cbwp-xxxxxxxxxx"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure cross-AZ forwarding

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cross-zone-enabled

Enable or disable forwarding across availability zones. Default: on.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cross-zone-enabled: "off"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Retain the NLB instance when deleting a Service

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete

When a LoadBalancer Service is deleted, retain the NLB instance instead of deleting it. CCM removes the kubernetes.do.not.delete and ack.aliyun.com tags from the NLB instance and server group, and retains existing servers.

• Non-empty value: Enable retention.

• Empty or unset: Disable retention (default).

After setting this annotation, verify that the feature is active by checking for a Warning event of type PreservedOnDelete during Service synchronization.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-lb-on-delete: "true"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure listener security groups

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids

Specify one or more security group IDs separated by commas, for example sg-aaaaa,sg-bbbbb.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-security-group-ids: "sg-aaaaa,sg-bbbbb"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a TCP listener

Define TCP ports in spec.ports with protocol: TCP. CCM creates one TCP listener per port.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a UDP listener

When creating a UDP listener without explicitly configuring health checks, TCP health checks are enabled for the server group by default. Use the health check annotations to configure UDP health checks, or disable health checks explicitly.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 80
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure both TCP and UDP listeners on the same port

Requires Kubernetes cluster version v1.24 or later. To upgrade, see Upgrade the Kubernetes version of an ACK cluster.

Define both TCP and UDP entries for the same port number in spec.ports. CCM creates separate listeners for each protocol.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: udp
    port: 80
    protocol: UDP
    targetPort: 81
  selector:
    app: nginx
  sessionAffinity: None
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Create a TCP/SSL listener

Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${CertIdentifier}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure a listener port range

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-listener-port-range

Only supported with the Terway network plug-in.

Configure full-port forwarding to listen on a port range and forward traffic to the corresponding ports on backend servers.

The format is startPort-endPort:servicePort. Separate multiple ranges with commas. For example: 80-100:80,400-500:443. Port ranges under the same protocol cannot overlap. Valid range: 1–65535.

targetPort specifies the health check port on backend servers and must be an integer.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    # Configure listeners for port ranges 80-100 and 400-500.
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-listener-port-range: "80-100:80,400-500:443"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    # targetPort is the health check port of the backend server and must be an integer.
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure the connection rate limit

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps

Set the maximum new connections per second (CPS) for the NLB instance. Range: 0–1,000,000. Set to 0 to disable rate limiting.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cps: "100"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure the connection idle timeout

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout

Set the connection idle timeout in seconds. Range: 10–900. Default: 900.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-idle-timeout: "60"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure Proxy Protocol

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol

Enable to pass client source IP addresses to backend servers via Proxy Protocol. Default: off.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Pass additional information via Proxy Protocol

When Proxy Protocol is enabled, use the following annotations to pass additional metadata to backend servers.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-proxy-protocol: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-ep-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-eps-id-enabled: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ppv2-pvl-vpc-id-enabled: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Enable mutual authentication

Annotations: alibaba-cloud-loadbalancer-protocol-port, alibaba-cloud-loadbalancer-cert-id, alibaba-cloud-loadbalancer-cacert-id, and alibaba-cloud-loadbalancer-cacert

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${CertIdentifier}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert-id: "${your-cacert-id}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cacert: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure a TLS security policy

Annotations: alibaba-cloud-loadbalancer-protocol-port, alibaba-cloud-loadbalancer-cert-id, and alibaba-cloud-loadbalancer-tls-cipher-policy

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${CertIdentifier}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-tls-cipher-policy: "tls_cipher_policy_1_0"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure an ALPN policy

Annotations: alibaba-cloud-loadbalancer-protocol-port, alibaba-cloud-loadbalancer-cert-id, alibaba-cloud-loadbalancer-alpn, and alibaba-cloud-loadbalancer-alpn-policy

For additional values, see the AlpnPolicy field in the CreateListener API.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${CertIdentifier}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-alpn: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-alpn-policy: "HTTP1Only"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure extended certificates

Annotations: alibaba-cloud-loadbalancer-protocol-port, alibaba-cloud-loadbalancer-cert-id, and alibaba-cloud-loadbalancer-additional-cert-ids

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-protocol-port: "tcpssl:443"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-cert-id: "${CertIdentifier}"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-additional-cert-ids: "${CertIdentifier-1},${CertIdentifier-2}"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure the scheduling algorithm

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler

For additional values, see the Scheduler field in the CreateServerGroup API.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-scheduler: "sch"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure connection draining

Annotations: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain and service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout

When connection draining is enabled and a backend server is removed or fails a health check, existing connections continue to be served for the draining timeout period. After the timeout, connections are actively closed.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-connection-drain-timeout: "30"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure client IP preservation

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip

Enable to pass the original client IP address to backend servers. Default: on.

TCP/SSL listeners do not support server groups with client IP preservation enabled. If you use a TCP/SSL listener, set this annotation to off or omit the alibaba-cloud-loadbalancer-preserve-client-ip annotation. See Obtain client originating IP addresses through NLB for alternatives.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-preserve-client-ip: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure TCP health checks

Health checks are enabled by default for TCP ports. The following annotations configure TCP health check behavior.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "tcp"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "8"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure HTTP health checks

To use HTTP health checks, set alibaba-cloud-loadbalancer-health-check-type to http and configure the following annotations.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-flag: "on"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-type: "http"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-uri: "/test/index.html"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-domain: "www.test.com"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-healthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-unhealthy-threshold: "4"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-connect-timeout: "10"
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-interval: "5"
    # Optional: configure the health check method
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-health-check-method: "head"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Configure the server group type

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type

For details on server group types, see NLB server groups.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-server-group-type: "Ip"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer

Reuse an existing server group

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-vgroup-port

Reuse an existing server group. This only takes effect when reusing an existing NLB. For a full example, see Deploy services across clusters by reusing an existing load balancer.

Configure service traffic weight

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-weight

When multiple Services share the same NLB, use this annotation to set the traffic weight for the current Service. This only takes effect when reusing an existing server group. For a full example, see Deploy services across clusters by reusing an existing load balancer.

Ignore backend server weight updates

Annotation: service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update

When set to on, CCM skips updating backend server weights in the server group during Service synchronization. Use this when managing backend server weights through a mechanism other than CCM. Default: off.

apiVersion: v1
kind: Service
metadata:
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-zone-maps: "${zone-A}:${vsw-A},${zone-B}:${vsw-B}" # For example, cn-hangzhou-k:vsw-i123456,cn-hangzhou-j:vsw-j654321
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-ignore-weight-update: "on"
  name: nginx
  namespace: default
spec:
  externalTrafficPolicy: Local
  ports:
  - name: tcp
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: nginx
  loadBalancerClass: "alibabacloud.com/nlb"
  type: LoadBalancer