Advanced features of APIG Ingress - API Gateway - Alibaba Cloud Documentation Center

In a Kubernetes cluster, the APIG Ingress is an API object that provides Layer 7 load balancing to manage external access to services. This topic describes the advanced features of the APIG Ingress that you can use to manage the ingress traffic of your cluster.

Canary release

The APIG Ingress provides routing capabilities, including support for canary releases based on headers, query parameters, cookies, and weights. You can configure this feature by adding annotations to your Ingress resources. To enable canary releases, add the nginx.ingress.kubernetes.io/canary: "true" annotation. You can then use other specific annotations to implement different types of canary releases.

Note

When multiple methods are configured, the priority order for canary release selection is: Based on header | Based on query parameter > Based on cookie > Based on weight (from high to low).

Header-based canary release

If you configure only nginx.ingress.kubernetes.io/canary-by-header, traffic is routed based on the request header. If the configured header value is always, traffic is routed to the canary service endpoint. Otherwise, traffic is not routed to the canary service.
If you configure both nginx.ingress.kubernetes.io/canary-by-header-value and nginx.ingress.kubernetes.io/canary-by-header, traffic is routed to the canary service only if the header and header value in the request match the configured values. Otherwise, traffic is not routed to the canary service.

Note

The NGINX Ingress and ALB Ingress support a maximum of two service versions during a canary release. The APIG Ingress supports an unlimited number of service versions during a canary release.

Examples:

If the request header is apig: always, requests are routed to the canary service demo-service-canary. Otherwise, requests are routed to the production service demo-service. The following sample code shows the configuration:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

If the request header is apig: v1, requests are routed to the canary service demo-service-canary-v1. If the request header is apig: v2, requests are routed to the canary service demo-service-canary-v2. Otherwise, requests are routed to the production service demo-service. The following sample code shows the configuration:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v1
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
  name: demo-canary-v2
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v2
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v1
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
  name: demo-canary-v2
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v2
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Query parameter-based canary release

Configure only higress.ingress.kubernetes.io/canary-by-query
Requests are sent based on the query parameter in URLs. In URLs of requests, if the query parameter key is the same as that specified by the annotation and the query parameter value is always, requests are sent to the canary version. In other cases, requests are not sent to the canary version.
Configure both higress.ingress.kubernetes.io/canary-by-query-value and higress.ingress.kubernetes.io/canary-by-query
If the query parameter key and query parameter value in the request match the configured values, traffic is routed to the canary service. Otherwise, traffic is not routed to the canary service.
Note
Header-based canary release can be used together with query parameter-based canary release. Requests are sent to the canary version only if the match conditions for both the canary release methods are met.

Examples:

If the query parameter in the request URL is canary: gray, requests are routed to the canary service demo-service-canary. Otherwise, requests are routed to the production service demo-service. The following sample code shows the configuration:

Clusters that run Kubernetes 1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    higress.ingress.kubernetes.io/canary-by-query: "canary"
    higress.ingress.kubernetes.io/canary-by-query-value: "gray"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    higress.ingress.kubernetes.io/canary-by-query: "canary"
    higress.ingress.kubernetes.io/canary-by-query-value: "gray"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

If the query parameter in the request URL is canary: gray and the request header contains x-user-id: test, requests are routed to the canary service demo-service-canary. Otherwise, requests are routed to the production service demo-service. The following sample code shows the configuration:

Clusters that run Kubernetes 1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    higress.ingress.kubernetes.io/canary-by-query: "canary"
    higress.ingress.kubernetes.io/canary-by-query-value: "gray"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    higress.ingress.kubernetes.io/canary-by-query: "canary"
    higress.ingress.kubernetes.io/canary-by-query-value: "gray"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Cookie-based canary release

nginx.ingress.kubernetes.io/canary-by-cookie: This annotation splits traffic based on a cookie. If the configured cookie value is always, traffic is routed to the canary service. Otherwise, traffic is not routed to the canary service.

Note

Cookie-based canary release does not support custom values. The configured cookie value must be always.

For example, if the request cookie is demo=always, requests are routed to the canary service demo-service-canary. Otherwise, requests are routed to the production service demo-service. The following sample code shows the configuration:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Weight-based canary release

Annotation	Description
nginx.ingress.kubernetes.io/canary-weight	Specifies the percentage of requests that are sent to the canary version. The value is an integer that ranges from 0 to 100.
nginx.ingress.kubernetes.io/canary-weight-total	Specifies the total weight. Default value: 100.

For example, you can set the weight of the canary version demo-service-canary-v1 to 30%, set the weight of the canary version demo-service-canary-v2 to 20%, and set the weight of the base version demo-service to 50%. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v1
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v2
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v1
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v2
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Service subset

A service subset is suitable for scenarios in which a Service is associated with multiple Deployments. You can use an Ingress to forward requests to a subset of pods of a Service. In most cases, requests are forwarded to Service pods that have a specific label. You can configure a Service subset using one of the following methods:

Use the pod label convention for APIG Ingress

You can use the annotation higress.ingress.kubernetes.io/service-subset to set the service version. By default, the APIG Ingress maps the configured service version to the pod label that is prefixed with `opensergo.io/canary`. The annotation works as follows:

If the value is "" or base, requests are forwarded to pods that have the label opensergo.io/canary: "" or do not have a label key with the opensergo.io/canary prefix. These pods are considered to have an empty label or no label.
If the value is set to another string, requests are forwarded to pods that have the label `opensergo.io/canary-{value}: {value}`. For example, if the value is gray, requests are forwarded to pods with the label opensergo.io/canary-gray: gray.

For example, a Kubernetes Service named go-httpbin is associated with two Deployments. The pods managed by one Deployment do not have label keys that are prefixed with opensergo.io/canary. The pods managed by the other Deployment have the canary label opensergo.io/canary-gray: gray. Sample configurations:

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/apig/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin gray deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        opensergo.io/canary-gray: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/apig/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

If the header of the example.com/test request contains x-user-id: test, the request is forwarded to go-httpbin-gray. Otherwise, the request is forwarded to httpbin-base. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label opensergo.io/canary-gray: gray.
    higress.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label opensergo.io/canary-gray: gray.
    higress.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080

Use custom labels

You can configure a custom label to define the pod collection for a subset by setting both the higress.ingress.kubernetes.io/service-subset and higress.ingress.kubernetes.io/subset-labels annotations.

Note

In this case, the subset is no longer mapped to the label whose prefix is opensergo.io/canary.

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/apig/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin base gray
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        version: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/apig/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

If the header of the example.com/test request contains x-user-id: test, the request is forwarded to go-httpbin-gray. Otherwise, the request is forwarded to httpbin-base.

Clusters that run Kubernetes 1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label version: gray.
    higress.ingress.kubernetes.io/service-subset: gray
    higress.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label version: gray.
    higress.ingress.kubernetes.io/service-subset: gray
    higress.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    higress.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080

CORS

Cross-origin resource sharing (CORS) allows web application servers to access cross-origin resources and secures cross-origin data transmission. For more information about CORS, see Cross-Origin Resource Sharing (CORS).

Annotation	Description
nginx.ingress.kubernetes.io/enable-cors	Specifies whether to enable CORS.
nginx.ingress.kubernetes.io/cors-allow-origin	Specifies the allowed third-party sites. Separate third-party sites with commas (,). You can use asterisks () as wildcard characters. Default value: . This value indicates that all third-party sites are allowed for CORS.
nginx.ingress.kubernetes.io/cors-allow-methods	Specifies the allowed request methods such as GET, POST, and PUT. Separate request methods with commas (,). You can use asterisks (*) as wildcard characters. Default value: GET,PUT,POST,DELETE,PATCH,OPTIONS.
nginx.ingress.kubernetes.io/cors-allow-headers	Specifies the allowed request headers. Separate request headers with commas (,). You can use asterisks (*) as wildcard characters. Default value: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization.
nginx.ingress.kubernetes.io/cors-expose-headers	Specifies the response headers that can be exposed to a browser. Separate response headers with commas (,).
nginx.ingress.kubernetes.io/cors-allow-credentials	Specifies whether to allow credentials to be carried in CORS requests. By default, credentials can be passed during CORS operations.
nginx.ingress.kubernetes.io/cors-max-age	Specifies the maximum period of time in which precheck results can be cached. Unit: seconds. Default value: 1728000.

For example, configure example.com as the allowed third-party website, GET and POST as the allowed request methods, and X-Foo-Bar as the allowed request header, and disallow credential passing during CORS operations. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Regular expression match

Standard Kubernetes Ingresses support only exact and prefix matching, but the APIG Ingress also supports regular expression matching. You can use the annotation nginx.ingress.kubernetes.io/use-regex: true to enable regular expression matching for the path defined in the Ingress spec.

If the expected domain name is example.com, the request whose request path starts with /app or /test is forwarded to the service demo. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo
                port: 
                  number: 8080
            path: /(app|test)/(.*)
            pathType: Prefix

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /(app|test)/(.*)
            backend:
              serviceName: demo
              servicePort: 8080

Path and host rewrite

You can perform a rewrite operation to modify the original paths and hosts in requests before the requests are forwarded to a destination backend service.

Annotation	Description
nginx.ingress.kubernetes.io/rewrite-target	Specifies the destination path for a rewrite operation. Capture groups are supported.
nginx.ingress.kubernetes.io/upstream-vhost	Host Rewrite

Path rewrite

Before the example.com/test request is forwarded to the backend service, rewrite example.com/test as example.com/dev. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/dev"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/dev"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /test
            pathType: Exact
            backend:
              serviceName: demo-service
              servicePort: 80

Before the example.com/v1/xxx request is forwarded to the backend service, remove the prefix /v1 and rewrite the request as example.com/xxx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/$1"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /v1/(.*)
            pathType: Prefix

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/$1"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /v1/(.*) 
          	pathType: Prefix
            backend:
              serviceName: demo-service
              servicePort: 80

Before the example.com/v1/xxx request is forwarded to the backend service, change the prefix /v1 to /v2 and rewrite the request as example.com/v2/xxx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /v1/(.*)
            pathType: Prefix

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /v1/(.*)
            pathType: Prefix
            backend:
              serviceName: demo-service
              servicePort: 80

Host rewrite

For example, before the example.com/test request is forwarded to a backend service, rewrite example.com/test as test.com/test. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirect

Redirect is used to change an original client request to a destination request.

HTTP-to-HTTPS redirect

Annotation	Description
nginx.ingress.kubernetes.io/ssl-redirect	Specifies whether to redirect HTTP requests to HTTPS requests.
nginx.ingress.kubernetes.io/force-ssl-redirect	Specifies whether to redirect HTTP requests to HTTPS requests.

Note

The APIG Ingress treats both annotations the same and forcefully redirects HTTP requests to HTTPS.

For example, redirect the request http://example.com/test to https://example.com/test. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Permanent redirect

Annotation	Description
nginx.ingress.kubernetes.io/permanent-redirect	Specifies the destination URL for a permanent redirect. The destination URL must contain the scheme, which is HTTP or HTTPS.
nginx.ingress.kubernetes.io/permanent-redirect-code	Specifies the HTTP status code for a permanent redirect. Default value: 301.

For example, perform a permanent redirect from http://example.com/test to http://example.com/app. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Temporary redirect

nginx.ingress.kubernetes.io/temporal-redirect: specifies the destination URL for a temporal redirect. The destination URL must contain the scheme, which is HTTP or HTTPS.

For example, perform a temporal redirect from http://example.com/test to http://example.com/app. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Header control

Header control lets you add, delete, or modify request headers before the Higress Ingress forwards the requests to a backend service. Header control also lets you add, delete, or modify response headers before the Higress Ingress forwards the received responses to the client.

Request header control

Annotation	Description
higress.ingress.kubernetes.io/request-header-control-add	Specifies the header that is added to a request when the request is forwarded to the backend service. If the header exists, its value is concatenated after the original value. Syntax: Single header: A key-value pair is used. Multiple headers: Use the YAML special character `\|`, with each key-value pair on a separate line.
higress.ingress.kubernetes.io/request-header-control-update	Specifies the header that is modified in a request when the request is forwarded to the backend service. If the header exists, its value overwrites the original value. Syntax: Single header: A key-value pair is used. Multiple headers: Use the YAML special character `\|`, with each key-value pair on a separate line.
higress.ingress.kubernetes.io/request-header-control-remove	Specifies the header that is deleted from a request when the request is forwarded to the backend service. Syntax: Single header: A key is used. Multiple headers: Separate multiple headers with commas (,).

Examples:

Add the headers foo: bar and test: true to the request example.com/test. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/request-header-control-add: |
      foo bar
      test true
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/request-header-control-add: |
      foo bar
      test true
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Header control can be used with canary releases to identify the requests that are sent to the canary version. If the request header is apig: v1, the request is sent to the canary version demo-service-canary-v1 and the header stage: gray is added to the request. In other cases, the request is sent to the base version demo-service and the header stage: production is added to the request. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
    higress.ingress.kubernetes.io/request-header-control-add: "stage gray"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v1
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/request-header-control-add: "stage production"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "apig"
    nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
    higress.ingress.kubernetes.io/request-header-control-add: "stage gray"
  name: demo-canary-v1
spec:
  ingressClassName: apig
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v1
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/request-header-control-add: |
      foo bar
      test true
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Response header control

Annotation	Description
higress.ingress.kubernetes.io/response-header-control-add	Specifies the header that is added to a response received from a backend service before the response is forwarded to the client. If the header exists, its value is concatenated after the original value. Syntax: Single header: A key-value pair is used. Multiple headers: Use the YAML special character `\|`, with each key-value pair on a separate line.
higress.ingress.kubernetes.io/response-header-control-update	Specifies the header that is modified in a response received from a backend service before the response is forwarded to the client. If the header exists, its value overwrites the original value. Syntax: Single header: A key-value pair is used. Multiple headers: Use the YAML special character `\|`, with each key-value pair on a separate line.
higress.ingress.kubernetes.io/response-header-control-remove	Specifies the header that is deleted from a response received from a backend service before the response is forwarded to the client. Syntax: Single header: A key is used. Multiple headers: Separate multiple headers with commas (,).

For example, delete the header req-cost-time from the response to the request example.com/test. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Retry

The APIG Ingress provides route-level retry settings to automatically retry failed requests. You can set retry conditions as needed. For example, you can retry requests when a connection fails, a backend service is unavailable, or a specific HTTP status code is returned.

Annotation	Description
nginx.ingress.kubernetes.io/proxy-next-upstream-tries	Specifies the maximum number of request retries. Default value: 3.
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout	Specifies the timeout period for request retries. Unit: seconds. By default, no timeout periods are configured.
nginx.ingress.kubernetes.io/proxy-next-upstream	The conditions for retrying a request. Separate multiple conditions with a comma (,). The default value is `error,timeout`. Valid values are: error: The connection establishment fails and the status code 5xx is returned. timeout: The connection establishment times out and the status code 5xx is returned. invalid_header: A request error occurs and the status code 5xx is returned. http_xxx: Retries are triggered based on the status code returned. Examples: http_502 and http_403. non_idempotent: Retries are triggered for failed non-idempotent requests. By default, the APIG Ingress does not retry failed non-idempotent requests, such as POST and PATCH requests. If you configure `non_idempotent`, you can enable retries for these requests. off: Retries are disabled.

For example, the example/test request is available. For the request, set the maximum number of request retries to 2, set the retry timeout period to 5 seconds, trigger retries only when the status code 502 is returned, and enable retries for non-idempotent requests. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Access control based on IP address whitelists and blacklists

The APIG Ingress provides IP blacklist and whitelist access control at the domain and route levels. Route-level rules have a higher priority than domain-level rules.

IP address whitelists and blacklists at the route level

Annotation	Description
nginx.ingress.kubernetes.io/whitelist-source-range	Specifies the IP address whitelist of a specific route. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).
higress.ingress.kubernetes.io/blacklist-source-range	Specifies the IP address blacklist of a specific route. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

Examples:

Allow access to example.com/test from the client IP address 1.1.xx.xx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Deny access to example.com/test from the client IP address 2.2.xx.xx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

IP address whitelists and blacklists at the domain name level

Annotation	Description
higress.ingress.kubernetes.io/domain-whitelist-source-range	Specifies the IP address whitelist of a specific domain name. IP address whitelists at the route level take precedence over IP address whitelists at the domain name level. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).
higress.ingress.kubernetes.io/domain-blacklist-source-range	Specifies the IP address blacklist of a specific domain name. IP address blacklists at the route level take precedence over IP address blacklists at the domain name level. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

Examples:

Allow access to all routes of the domain name example.com from the client IP addresses 1.1.xx.xx and 2.2.xx.xx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact
         - backend:
              service:
                name: app-service
                port: 
                  number: 80
            path: /app
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80
          - path: /app
            backend:
              serviceName: app-service
              servicePort: 80

Use IP address whitelists and blacklists at the domain name level with IP address whitelists and blacklists at the route level. Allow access to all routes of the example.com domain name from the client IP addresses 1.1.xx.xx and 2.2.xx.xx, and allow access to the example.com/order route only from the client IP address 3.3.xx.xx. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
  name: demo-domain
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact
         - backend:
              service:
                name: app-service
                port: 
                  number: 80
            path: /app
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
  name: demo-route
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /order
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
  name: demo-domain
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80
          - path: /app
            backend:
              serviceName: app-service
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
  name: demo-route
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /order
            backend:
              serviceName: demo-service
              servicePort: 80

Single-gateway throttling

The APIG Ingress supports route-level throttling policies for a single gateway instance. These policies limit the number of requests for a specific route on each gateway replica to a defined threshold within a set period.

Note

The system performs throttling for a single Cloud-native API Gateway instance based on the configured threshold. If you want to throttle traffic of a route for a Cloud-native API Gateway cluster, you can use global throttling control policies instead.

Annotation	Description
higress.ingress.kubernetes.io/route-limit-rpm	Specifies the maximum number of requests per minute (RPM) that are routed on a Cloud-native API Gateway instance. The burst limit for the maximum number of RPM is equal to the specified value multiplied by higress.ingress.kubernetes.io/route-limit-burst-multiplier. When throttling is triggered, the response body is `local_rate_limited`. The response status code is as follows: If the Cloud-native API Gateway version is earlier than 1.2.23, the status code 503 is returned. If the Cloud-native API Gateway version is 1.2.23 or later, the status code 429 is returned.
higress.ingress.kubernetes.io/route-limit-rps	Specifies the maximum number of requests per second (RPS) that are routed on a Cloud-native API Gateway instance. The burst limit for the maximum number of RPS is equal to the specified value multiplied by higress.ingress.kubernetes.io/route-limit-burst-multiplier. When throttling is triggered, the response body is `local_rate_limited`. The response status code is as follows: If the Cloud-native API Gateway version is earlier than 1.2.23, the status code 503 is returned. If the Cloud-native API Gateway version is 1.2.23 or later, the status code 429 is returned.
higress.ingress.kubernetes.io/route-limit-burst-multiplier	Specifies the multiplier of the burst limits. Default value: 5.

Examples:

For the example.com/test request, set the maximum number of RPM to 100 and the burst limit for the maximum number of RPM to 200. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/route-limit-rpm: "100"
    higress.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/route-limit-rpm: "100"
    higress.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

For the example.com/test request, set the maximum number of RPS to 10 and the burst limit for the maximum number of RPS to 50. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/route-limit-rps: "10"
    # The default value is 5.
    # higress.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/route-limit-rps: "10"
    # The default value is 5.
    # higress.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Global throttling control

The APIG Ingress integrates with Sentinel to provide global, route-level throttling for the gateway cluster. This limits the maximum number of requests per second (RPS) for a specific route across the entire cluster.

Note

This feature requires APIG Ingress gateway version 1.2.25 or later.

Use the higress.ingress.kubernetes.io/rate-limit annotation to set the maximum RPS for a route across the gateway cluster. When throttling is triggered, the default behavior is to return a 429 status code with the response body `sentinel rate limited`. The APIG Ingress provides two ways to customize this behavior: a custom response or a redirection. You can use only one of these methods.

Custom response

higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: The response status code when throttling is triggered. The default is 429.
higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body-type: The format of the response body when throttling is triggered. The default is text.
- If set to text, the Content-Type of the response is text/plain; charset=UTF-8.
- If set to json, the Content-Type of the response is application/json; charset=UTF-8.
higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: The response body when throttling is triggered. The default is sentinel rate limited.

Example 1: Set the maximum number of RPS of the example.com/test request in a Cloud-native API Gateway cluster to 100 and retain the default throttling behavior. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Example 2: Set the maximum number of RPS of the example.com/test request in a Cloud-native API Gateway cluster to 100 and set the response code to 503 and response body to server is overload when throttling is triggered.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirection

higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url: The URL to redirect to when throttling is triggered.

Example 1: Set the maximum number of RPS of the example.com/test request in a Cloud-native API Gateway cluster to 100 and the redirect URL to example.com/fallback when throttling is triggered.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/rate-limit: "100"
    higress.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Global concurrency control

The APIG Ingress integrates with Sentinel to provide global, route-level concurrency control for the gateway cluster. This limits the maximum number of concurrent requests being processed for a specific route across the entire cluster.

Note

This feature requires APIG Ingress gateway version 1.2.25 or later.

Use the higress.ingress.kubernetes.io/concurrency-limit annotation to set the maximum number of concurrent requests for a route across the gateway cluster. When concurrency control is triggered, the default response has a status code of 429 and a body of sentinel rate limited. The APIG Ingress provides two ways to customize this behavior: a custom response or a redirection. You can use only one of these methods.

Custom response

higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: The response status code when concurrency control is triggered. The default is 429.
higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body-type: The format of the response body when concurrency control is triggered. The default is text.
- If set to text, the Content-Type of the response is text/plain; charset=UTF-8.
- If set to json, the Content-Type of the response is application/json; charset=UTF-8.
higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: The response body when concurrency control is triggered. The default is sentinel rate limited.

Example 1: Set the maximum number of example.com/test requests processed on a gateway cluster to 1000 and retain the default concurrency control behavior. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Example 2: Limit the maximum number of concurrent requests for `example.com/test` on the gateway cluster to 1,000. When concurrency control is triggered, return a status code of 503 and a response body of server is overloaded.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    higress.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirection

higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: The URL to redirect to when concurrency control is triggered.

Set the maximum number of example.com/test requests processed by a Cloud-native API Gateway cluster to 1,000 and the redirect URL to example.com/fallback when concurrency control is triggered.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/concurrency-limit: "1000"
    higress.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Traffic mirroring

You can configure traffic mirroring to copy traffic to a specified service. Traffic mirroring is suitable for scenarios such as operational auditing and traffic testing.

higress.ingress.kubernetes.io/mirror-target-service: the destination service to which copied traffic is forwarded. The service format is namespace/name:port.
- namespace: the namespace where the Kubernetes Service resides. This parameter is optional. The default namespace is the namespace where the Ingress gateway resides.
- name: the name of the Kubernetes Service. This parameter is required.
- port: the Kubernetes Service port to which mirrored traffic is forwarded. This parameter is optional. By default, the first port is used.
higress.ingress.kubernetes.io/mirror-percentage: the percentage of mirrored traffic. Valid values: 0 to 100. Default value: 100.

Note

When the copied traffic is forwarded to the destination service, the -shadow suffix is automatically added to the Host header in the original request.

For example, the requests from example.com/test are copied and forwarded to the destination service test/app:8080.

Note

In this example, the value of the Host header is automatically changed to example.com-shadow when the copied traffic is forwarded to the destination service.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

For example, the requests from example.com/test are copied and forwarded to the destination service test/app:8080 and the percentage of mirrored traffic is 10%.

Note

In this example, the value of the Host header is automatically changed to example.com-shadow when the copied traffic is forwarded to the destination service.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
    higress.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/mirror-target-service: test/app:8080
    higress.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Backend service protocols: HTTPS and gRPC

By default, the APIG Ingress uses HTTP to forward requests to backend application containers. If your application container uses HTTPS, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" to forward requests over HTTPS. If your application container is a gRPC service, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC" to forward requests over gRPC.

Note

Unlike NGINX Ingress, if the port name in the Kubernetes service resource for your backend service is defined as `gRPC` or `HTTP2`, you do not need to configure the nginx.ingress.kubernetes.io/backend-protocol: "GRPC" annotation. The APIG Ingress automatically uses gRPC or HTTP/2.

Examples:

Use HTTPS to forward the example/test request to a backend service. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Use gRPC to forward the example/test request to a backend service. The following methods can be used:

Method 1: Use annotations. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Method 2: Use spec > ports > name in Ingress resource configurations. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /order
            pathType: Exact
---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
spec:
  ports:
    - name: grpc
      port: 80
      protocol: TCP
  selector:
    app: demo-service

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: demo-service
spec:
  ports:
    - name: grpc
      port: 80
      protocol: TCP
  selector:
    app: demo-service

Load balancing algorithms for backend services

Load balancing algorithms determine how nodes are selected when a Cloud-native API Gateway instance forwards requests to backend services.

Common load balancing algorithms

nginx.ingress.kubernetes.io/load-balance: specifies the common load balancing algorithm that is used by a backend service. Default value: round_robin. Valid values:

round_robin: load balancing based on round robin.
least_conn: load balancing based on least connections.
random: randomized load balancing.

Important

Cloud-native API Gateway does not support the exponentially weighted moving average (EWMA) algorithm. If you configure the EWMA algorithm, the algorithm is rolled back to the round robin load balancing algorithm.

For example, configure the least connection load balancing algorithm for the demo-service backend service. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /order
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Load balancing algorithms based on consistent hashing

A consistent hash-based load balancing algorithm provides request affinity, which ensures that requests with the same characteristics are always routed to the same node. The APIG Ingress supports using NGINX variables, request headers, and request path parameters as the hash key.

nginx.ingress.kubernetes.io/upstream-hash-by: specifies the load balancing algorithm based on consistent hashing. Cloud-native gateways support the following types of consistent hashing:

The cloud-native gateway supports the configuration of specific Nginx variables:
- $request_uri: the request path, which is used as the hash key. Path parameters are included.
- $host: the request host, which is used as the hash key.
- $remote_addr: the client IP address, which is used as the hash key.
Consistent hashing based on request headers. You need only to configure $http_headerName.
Consistent hashing based on request path parameters. You need only to configure $arg_varName.

Examples:

Use a client IP address for requests as the hash key. Requests from the client IP address are always distributed to the same node. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Use the request header X-Stage as the hash key, and distribute requests with the same value of the request header X-Stage to the same node. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Use the request path parameter X-Stage as the hash key, and distribute requests with the same value of the request path parameter X-Stage to the same node. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Service prefetching (graceful start)

The service prefetching feature allows traffic to gradually increase in a specified service prefetching window when a new node is released. This ensures that the node is completely prefetched.

higress.ingress.kubernetes.io/warmup: specifies the period of time in which services are prefetched. Unit: seconds. By default, the service prefetching feature is not enabled.

Note

Service prefetching depends on the selected load balancing algorithm. Only the load balancing algorithms based on round robin and least connections are supported.

For example, enable the service prefetching feature for the backend service demo-service and set the prefetching time window to 30 seconds. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Cookie affinity (session persistence)

The gateway always routes requests with the same cookie to the same node. If a client's first request does not have the affinity cookie, the APIG Ingress generates one in the response. This ensures that subsequent requests from that client are always routed to the same node.

Annotation	Description
nginx.ingress.kubernetes.io/affinity	Specifies the affinity type. The default and only valid value is cookie.
nginx.ingress.kubernetes.io/affinity-mode	Specifies the affinity mode. The default and only valid value is balanced.
nginx.ingress.kubernetes.io/session-cookie-name	Specifies the name of the cookie that is used as the hash key. Default value: INGRESSCOOKIE.
nginx.ingress.kubernetes.io/session-cookie-path	Specifies the path of the cookie that is generated if the specified cookie does not exist. Default value: /.
nginx.ingress.kubernetes.io/session-cookie-max-age	Specifies the expiration time of the cookie that is generated when the specified cookie does not exist. Unit: seconds. By default, this annotation is specified at the session level.
nginx.ingress.kubernetes.io/session-cookie-expires	Specifies the expiration time of the cookie that is generated when the specified cookie does not exist. Unit: seconds. By default, this annotation is specified at the session level.

Examples:

To enable cookie affinity with the default APIG Ingress configuration, use the following sample code. The default cookie name is `INGRESSCOOKIE`, the path is `/`, and the cookie lifecycle is at the session level.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Enable cookie affinity and set the cookie name to test, the path to /, and the expiration time of the cookie to 10s. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "test"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/affinity: "cookie"
    nginx.ingress.kubernetes.io/session-cookie-name: "test"
    nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Configuration of a connection pool between a Cloud-native API Gateway instance and a backend service

After you configure a connection pool for a specified backend service on the Cloud-native API Gateway instance, you can control the number of connections between the Cloud-native API Gateway instance and backend service. This helps prevent the backend service from being overloaded and ensures the stability and high availability of the backend service.

higress.ingress.kubernetes.io/connection-policy-tcp-max-connection: the maximum number of connections that can be established between a Cloud-native API Gateway instance and a backend service.
higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: the maximum number of connections that can be established between a Cloud-native API Gateway instance and a single node of a backend service.
higress.ingress.kubernetes.io/connection-policy-http-max-request-per-connection: the maximum number of requests on a single connection between a Cloud-native API Gateway instance and a backend service.

For example, for the backend service demo-service, you can set the maximum number of connections that can be established between a Cloud-native API Gateway instance and the backend service to 10, and set the maximum number of connections that can be established between a Cloud-native API Gateway instance and a single node of the backend service to 2.

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
  	higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
  	higress.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

TLS versions and cipher suites for communications between a client and a Cloud-native API Gateway instance

By default, the APIG Ingress supports TLS versions from 1.0 to 1.3. The default cipher suites are:

ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA
ECDHE-RSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA

The following table describes the annotations that can be used to specify the earliest or latest TLS version and cipher suites for a specific domain name.

Annotation	Description
higress.ingress.kubernetes.io/tls-min-protocol-version	Specifies the minimum version of TLS. Default value: TLSv1.0. Valid values: TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
higress.ingress.kubernetes.io/tls-max-protocol-version	Specifies the latest version of TLS. Default value: TLSv1.3.
nginx.ingress.kubernetes.io/ssl-cipher	Specifies the TLS cipher suites. You can specify multiple TLS cipher suites, which are separated by commas (,). This annotation takes effect only if a TLS version that ranges from v1.0 to v1.2 is used for TLS handshakes.

For example, the domain name example.com is available. For this domain name, set the earliest TLS version to TLSv1.2 and the latest version to TLSv1.2. Sample configurations:

Clusters that run Kubernetes 1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    higress.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    higress.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    higress.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

mTLS authentication between a Cloud-native API Gateway instance and a backend service

By default, the APIG Ingress uses HTTP to forward requests to backend application containers. You can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" to configure the APIG Ingress to access the backend service over HTTPS. This is one-way TLS, where only the APIG Ingress verifies the certificate provided by the backend service. The backend service certificate must typically be issued by a trusted Certificate Authority (CA). For a more secure zero-trust model, you can use mutual TLS (mTLS). With mTLS, the gateway and the backend service perform mutual authentication. The gateway verifies the backend service's certificate, and the backend service verifies the gateway's certificate.

Annotation	Description
nginx.ingress.kubernetes.io/proxy-ssl-secret	Specifies the client certificate used by the Cloud-native API Gateway instance. The client certificate is used for a backend service to authenticate the Cloud-native API Gateway instance. The format is secretNamespace/secretName.
nginx.ingress.kubernetes.io/proxy-ssl-name	Specifies the Server Name Indication (SNI) that is used during a TLS handshake.
nginx.ingress.kubernetes.io/proxy-ssl-server-name	Specifies whether to enable or disable the SNI that is used during a TLS handshake.

For example, perform an mTLS between a Cloud-native API Gateway instance and a backend service, set secretName to gateway-cert, and set secretNamespace to default. Sample configurations:

Clusters that run Kubernetes 1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than 1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: apig
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80