All Products
Search
Document Center

Microservices Engine:Advanced usage of MSE Ingress

Last Updated:Sep 27, 2024

Microservices Engine (MSE) Ingress is an API object that provides Layer-7 load balancing to manage external access to services in a Kubernetes cluster. This topic describes the advanced usage of MSE Ingress. This helps you govern ingress traffic of clusters.

Canary release

MSE Ingress allows you to configure canary releases that are based on headers, query parameters, cookies, or weights to handle complex traffic routing. You can configure the canary release feature by adding annotations to Ingress resource configurations. To enable the canary release feature, you must add the annotation nginx.ingress.kubernetes.io/canary: "true". This section describes how to use different annotations to configure different types of canary release methods.

Note

If four types of canary release methods are all configured, MSE Ingress selects the canary release methods in the following order of precedence: Header-based canary release or query parameter-based canary release > Cookie-based canary release > Weight-based canary release.

Header-based canary release

  • When you configure only the nginx.ingress.kubernetes.io/canary-by-header annotation, requests are sent based on their headers. If the header value is always, requests are sent to the canary version. In other cases, requests are not sent to the canary version.

  • When you configure both the nginx.ingress.kubernetes.io/canary-by-header-value and nginx.ingress.kubernetes.io/canary-by-header annotations, requests are sent to the canary version only if the key and value of the request header are the same as those specified in the annotations. In other cases, requests are not sent to the canary version.

Note

NGINX Ingress and ALB Ingress support up to two versions of a service during a canary release. MSE Ingress supports more than two versions of a service during a canary release. The number of versions is not limited.

Examples:

  • If the request header is mse:always, requests are sent to the canary version demo-service-canary. In other cases, requests are sent to the base version demo-service. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact          

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • If the request header is mse:v1, requests are sent to the canary version demo-service-canary-v1. If the request header is mse:v2, requests are sent to the canary version demo-service-canary-v2. In other cases, requests are sent to the base version demo-service. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v1
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
      name: demo-canary-v2
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v2
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v1
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
      name: demo-canary-v2
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v2
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

Query parameter-based canary release

  • Configure mse.ingress.kubernetes.io/canary-by-query

    Requests are sent based on the query parameter in URLs. In URLs of requests, if the query parameter key is the same as that specified by the annotation and the query parameter value is always, requests are sent to the canary version. In other cases, requests are not sent to the canary version.

  • Configure both mse.ingress.kubernetes.io/canary-by-query-value and mse.ingress.kubernetes.io/canary-by-query

    In URLs of requests, if the query parameter key and the query parameter value are the same as those specified by the annotations, requests are sent to the canary version. In other cases, requests are not sent to the canary version.

    Note

    Header-based canary release can be used together with query parameter-based canary release. Requests are sent to the canary version only if the match conditions for both the canary release methods are met.

Examples:

  • If the query parameter in request URLs is set to canary:gray, requests are sent to the canary version demo-service-canary. In other cases, requests are sent to the base version demo-service. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion:networking.k8s.io/v1 
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        mse.ingress.kubernetes.io/canary-by-query: "canary"
        mse.ingress.kubernetes.io/canary-by-query-value: "gray"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact 

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        mse.ingress.kubernetes.io/canary-by-query: "canary"
        mse.ingress.kubernetes.io/canary-by-query-value: "gray"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • If the query parameter in request URLs is set to canary:gray and x-user-id: test is included in request headers, requests are sent to the canary version demo-service-canary. In other cases, requests are sent to the base version demo-service. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion:networking.k8s.io/v1 
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        mse.ingress.kubernetes.io/canary-by-query: "canary"
        mse.ingress.kubernetes.io/canary-by-query-value: "gray"
        nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
        nginx.ingress.kubernetes.io/canary-by-header-value: "test"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact 

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        mse.ingress.kubernetes.io/canary-by-query: "canary"
        mse.ingress.kubernetes.io/canary-by-query-value: "gray"
        nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
        nginx.ingress.kubernetes.io/canary-by-header-value: "test"
      name: demo-canary
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

Cookie-based canary release

If you configure the nginx.ingress.kubernetes.io/canary-by-cookie annotation, requests are sent based on cookies. If cookie is set to always, requests are sent to the canary version. In other cases, requests are not sent to the canary version.

Note

If you configure a cookie-based canary release, custom cookie values are not supported. The configured cookie value must be always.

For example, if the request cookie is demo=always, requests are sent to the canary version demo-service-canary. In other cases, requests are sent to the base version demo-service. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
  name: demo-canary
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Weight-based canary release

Annotation

Description

nginx.ingress.kubernetes.io/canary-weight

Specifies the percentage of requests that are sent to the canary version. The value is an integer that ranges from 0 to 100.

nginx.ingress.kubernetes.io/canary-weight-total

Specifies the total weight. Default value: 100.

For example, you can set the weight of the canary version demo-service-canary-v1 to 30%, set the weight of the canary version demo-service-canary-v2 to 20%, and set the weight of the base version demo-service to 50%. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v1
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service-canary-v2
                port: 
                  number: 80
            path: /hello
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "30"
  name: demo-canary-v1
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v1
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "20"
  name: demo-canary-v2
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service-canary-v2
              servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Service subset

A service subset is suitable for scenarios in which a service is associated with multiple deployments. You can use an Ingress to forward requests to a subset of pods of a service. In most cases, requests are forwarded to service pods that have a specific label. You can configure a service subset by using one of the following methods:

Use the pod label in MseIngressConfig

You use the annotation mse.ingress.kubernetes.io/service-subset to configure a service subset. By default, the service subset configured in MseIngressConfig is mapped to the pod label that is prefixed with opensergo.io/canary. This annotation has the following meanings:

  • If this annotation is set to "" or base, requests are forwarded to the pods whose labels contain opensergo.io/canary: "" or the pods whose label keys are not prefixed with opensergo.io/canary. This way, requests are forwarded to the pods that have an empty label or do not have a label.

  • If this annotation is not set to "" or base, requests are forwarded to the pods whose labels contain opensergo.io/canary-{Specified value}: {Specified value}. For example, if you set this annotation to gray, requests are forwarded to the pods whose labels contain opensergo.io/canary-gray: gray.

For example, a Kubernetes service named go-httpbin is associated with two deployments. The pods managed by one deployment do not have label keys that are prefixed with opensergo.io/canary. The pods managed by the other deployment have the canary label opensergo.io/canary-gray: gray. Sample configurations:

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin gray deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        opensergo.io/canary-gray: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

If the header of the example.com/test request contains x-user-id: test, the request is forwarded to go-httpbin-gray. Otherwise, the request is forwarded to httpbin-base. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label opensergo.io/canary-gray: gray.
    mse.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    mse.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact 

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label opensergo.io/canary-gray: gray.
    mse.ingress.kubernetes.io/service-subset: gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    mse.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080

Use custom labels

You can use both the mse.ingress.kubernetes.io/service-subset and mse.ingress.kubernetes.io/subset-labels annotations to configure a custom label that is used to define the pods to which the subset belongs.

Note

In this case, the subset is no longer mapped to the label whose prefix is opensergo.io/canary.

For example, a Kubernetes service named go-httpbin is associated with two deployments. The pods managed by one deployment do not have label keys that are prefixed with opensergo.io/canary. The pods managed by the other deployment have the canary label opensergo.io/canary-gray: gray. Sample configurations:

# go-httpbin k8s service
apiVersion: v1
kind: Service
metadata:
  name: go-httpbin
  namespace: default
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: go-httpbin
---
# go-httpbin base deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-base
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=base"
          imagePullPolicy: Always
          name: go-httpbin
---
# go-httpbin base gray
apiVersion: apps/v1
kind: Deployment
metadata:
  name: go-httpbin-gray
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: go-httpbin
  template:
    metadata:
      labels:
        app: go-httpbin
        version: gray
    spec:
      containers:
        - image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
          args:
            - "--version=gray"
          imagePullPolicy: Always
          name: go-httpbin

If the header of the example.com/test request contains x-user-id: test, the request is forwarded to go-httpbin-gray. Otherwise, the request is forwarded to httpbin-base.

Clusters that run Kubernetes V1.19 or later

apiVersion:networking.k8s.io/v1 
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label version: gray.
    mse.ingress.kubernetes.io/service-subset: gray
    mse.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    mse.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: go-httpbin
                port: 
                  number: 8080
            path: /test
            pathType: Exact 

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
    nginx.ingress.kubernetes.io/canary-by-header-value: "test"
    # Forward requests to the pods that have the canary label version: gray.
    mse.ingress.kubernetes.io/service-subset: gray
    mse.ingress.kubernetes.io/subset-labels: version gray
  name: demo-canary
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    # Forward requests to the pods whose labels are not prefixed with opensergo.io/canary.
    mse.ingress.kubernetes.io/service-subset: ""
  name: demo
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /test
            backend:
              # Configure the go-httpbin service and specify the version in the annotation.
              serviceName: go-httpbin
              servicePort: 8080

CORS

Cross-origin resource sharing (CORS) allows web application servers to access cross-origin resources and secures cross-origin data transmission. For more information about CORS, see Cross-Origin Resource Sharing (CORS).

Annotation

Description

nginx.ingress.kubernetes.io/enable-cors

Specifies whether to enable CORS.

nginx.ingress.kubernetes.io/cors-allow-origin

Specifies the allowed third-party sites. Separate third-party sites with commas (,). Wildcards (*) are supported. Default value: *. This value indicates that all third-party sites are allowed for CORS.

nginx.ingress.kubernetes.io/cors-allow-methods

Specifies the allowed request methods such as GET, POST, and PUT. Separate request methods with commas (,). Wildcards (*) are supported. Default value: GET,PUT,POST,DELETE,PATCH,OPTIONS.

nginx.ingress.kubernetes.io/cors-allow-headers

Specifies the allowed request headers. Separate request headers with commas (,). Wildcards (*) are supported. Default value: DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Authorization.

nginx.ingress.kubernetes.io/cors-expose-headers

Specifies the response headers that can be exposed to a browser. Separate response headers with commas (,).

nginx.ingress.kubernetes.io/cors-allow-credentials

Specifies whether credentials can be included in CORS requests. By default, credentials can be passed during CORS operations.

nginx.ingress.kubernetes.io/cors-max-age

Specifies the maximum period of time in which precheck results can be cached. Unit: seconds. Default value: 1728000.

For example, configure example.com as the allowed third-party website, GET and POST as the allowed request methods, and X-Foo-Bar as the allowed request header, and disallow credential passing during CORS operations. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /hello
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
    nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /hello
            backend:
              serviceName: demo-service
              servicePort: 80

Regular expression match

MSE Ingress supports regular expression match in addition to exact match and prefix match supported by standard Kubernetes Ingress. You can use the annotation nginx.ingress.kubernetes.io/use-regex: true to change the path match defined in the Ingress Spec to the regular expression match.

If the expected domain name is example.com, the request whose request path starts with /app or /test is forwarded to the service demo. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - backend:
              service:
                name: demo
                port: 
                  number: 8080
            path: /(app|test)/(.*)
            pathType: Prefix

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/use-regex: 'true'
  name: regex-match
  namespace: default
spec:
  ingressClassName: mse
  rules:
    - http:
        paths:
          - path: /(app|test)/(.*)
            backend:
              serviceName: demo
              servicePort: 8080

Path and host rewrite

You can perform a rewrite operation to modify the original paths and hosts in requests before the requests are forwarded to a destination backend service.

Annotation

Description

nginx.ingress.kubernetes.io/rewrite-target

Specifies the destination path for a rewrite operation. Capture groups are supported.

nginx.ingress.kubernetes.io/upstream-vhost

Specifies the destination host for a rewrite operation.

Path rewrite

  1. Before the example.com/test request is forwarded to the backend service, rewrite example.com/test as example.com/dev. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/dev"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/dev"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /test
                pathType: Exact
                backend:
                  serviceName: demo-service
                  servicePort: 80
  2. Before the example.com/v1/xxx request is forwarded to the backend service, remove the prefix /v1 and rewrite the request as example.com/xxx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /v1/(.*)
                pathType: Prefix

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /v1/(.*) 
              	pathType: Prefix
                backend:
                  serviceName: demo-service
                  servicePort: 80
  3. Before the example.com/v1/xxx request is forwarded to the backend service, change the prefix /v1 to /v2 and rewrite the request as example.com/v2/xxx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /v1/(.*)
                pathType: Prefix

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /v1/(.*)
                pathType: Prefix
                backend:
                  serviceName: demo-service
                  servicePort: 80

Host rewrite

For example, before the example.com/test request is forwarded to a backend service, rewrite example.com/test as test.com/test. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirect

Redirect is used to change an original client request to a destination request.

HTTP-to-HTTPS redirect

Annotation

Description

nginx.ingress.kubernetes.io/ssl-redirect

Specifies whether to redirect HTTP requests to HTTPS requests.

nginx.ingress.kubernetes.io/force-ssl-redirect

Specifies whether to redirect HTTP requests to HTTPS requests.

Note

MSE Ingress does not distinguish the two annotations. Both of the two annotations are used to forcefully redirect HTTP requests to HTTPS requests.

For example, redirect the request http://example.com/test to https://example.com/test. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Permanent redirect

Annotation

Description

nginx.ingress.kubernetes.io/permanent-redirect

Specifies the destination URL for a permanent redirect. The destination URL must contain the scheme, which is HTTP or HTTPS.

nginx.ingress.kubernetes.io/permanent-redirect-code

Specifies the HTTP status code for a permanent redirect. Default value: 301.

For example, perform a permanent redirect from http://example.com/test to http://example.com/app. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Temporal redirect

nginx.ingress.kubernetes.io/temporal-redirect: specifies the destination URL for a temporal redirect. The destination URL must contain the scheme, which is HTTP or HTTPS.

For example, perform a temporal redirect from http://example.com/test to http://example.com/app. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Header control

Header control allows you to add, delete, or modify request headers before MSE Ingress forwards the requests to a backend service. Header control also allows you to add, delete, or modify response headers before MSE Ingress forwards the received responses to the client.

Request header control

Annotation

Description

mse.ingress.kubernetes.io/request-header-control-add

Specifies the header that is added to a request when the request is forwarded to a backend service. If the header exists, its value is concatenated after the original value. The following syntax is used:

  • Single header: A key-value pair is used.

  • Multiple headers: A vertical bar (|) in the YAML file is used. Each key-value pair occupies a separate line.

mse.ingress.kubernetes.io/request-header-control-update

Specifies the header that is modified in a request when the request is forwarded to a backend service. If the header exists, its value overwrites the original value. The following syntax is used:

  • Single header: A key-value pair is used.

  • Multiple headers: A vertical bar (|) in the YAML file is used. Each key-value pair occupies a separate line.

mse.ingress.kubernetes.io/request-header-control-remove

Specifies the header that is deleted from a request when the request is forwarded to a backend service. The following syntax is used:

  • Single header: A key is used.

  • Multiple headers: Separate multiple headers with commas (,).

Examples:

  • Add the headers foo: bar and test: true to the request example.com/test. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Header control can be used with canary releases to identify the requests that are sent to the canary version. If the request header is mse: v1, the request is sent to the canary version demo-service-canary-v1 and the header stage: gray is added to the request. In other cases, the request is sent to the base version demo-service and the header stage: production is added to the request. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
        mse.ingress.kubernetes.io/request-header-control-add: "stage gray"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service-canary-v1
                    port: 
                      number: 80
                path: /hello
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/request-header-control-add: "stage production"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /hello
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/canary: "true"
        nginx.ingress.kubernetes.io/canary-by-header: "mse"
        nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
        mse.ingress.kubernetes.io/request-header-control-add: "stage gray"
      name: demo-canary-v1
    spec:
      ingressClassName: mse
      rules:
        - http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service-canary-v1
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/request-header-control-add: |
          foo bar
          test true
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /hello
                backend:
                  serviceName: demo-service
                  servicePort: 80

Response header control

Annotation

Description

mse.ingress.kubernetes.io/response-header-control-add

Specifies the header that is added to a response received from a backend service before the response is forwarded to the client. If the header exists, its value is concatenated after the original value. The following syntax is used:

  • Single header: A key-value pair is used.

  • Multiple headers: A vertical bar (|) in the YAML file is used. Each key-value pair occupies a separate line.

mse.ingress.kubernetes.io/response-header-control-update

Specifies the header that is modified in a response received from a backend service before the response is forwarded to the client. If the header exists, its value overwrites the original value. The following syntax is used:

  • Single header: A key-value pair is used.

  • Multiple headers: A vertical bar (|) in the YAML file is used. Each key-value pair occupies a separate line.

mse.ingress.kubernetes.io/response-header-control-remove

Specifies the header that is deleted from a response received from a backend service before the response is forwarded to the client. The following syntax is used:

  • Single header: A key is used.

  • Multiple headers: Separate multiple headers with commas (,).

For example, delete the header req-cost-time from the response to the request example.com/test. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Retry

MSE Ingress provides route-level retry settings that can be used to automatically retry error requests. You can specify the retry conditions based on your business requirements. For example, you can configure settings to retry requests when connection establishment fails, a backend service is unavailable, or a specified HTTP status code is returned.

Annotation

Description

nginx.ingress.kubernetes.io/proxy-next-upstream-tries

Specifies the maximum number of request retries. Default value: 3.

nginx.ingress.kubernetes.io/proxy-next-upstream-timeout

Specifies the timeout period for request retries. Unit: seconds. By default, no timeout periods are configured.

nginx.ingress.kubernetes.io/proxy-next-upstream

Specifies retry conditions. Separate multiple retry conditions with commas (,). Default value: error,timeout. Valid values:

  • error: The connection establishment fails and the status code 5xx is returned.

  • timeout: The connection establishment times out and the status code 5xx is returned.

  • invalid_header: A request error occurs and the status code 5xx is returned.

  • http_xxx: Retries are triggered based on the status code returned. Examples: http_502 and http_403.

  • non_idempotent: Retries are triggered when an error is returned for requests such as non-idempotent requests. By default, MSE Ingress does not retry non-idempotent requests such as POST and PATCH requests if errors are returned for the requests. If you set nginx.ingress.kubernetes.io/proxy-next-upstream to non_idempotent, retries can be enabled for non-idempotent requests.

  • off: Retries are disabled.

For example, the example/test request is available. For the request, set the maximum number of request retries to 2, set the retry timeout period to 5s, trigger retries only when the status code 502 is returned, and enable retries for non-idempotent requests. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
    nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
    nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com 
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Access control based on IP address whitelists and blacklists

MSE Ingress provides IP address whitelists and blacklists at the domain name and route levels. IP address whitelists and blacklists at the route level take precedence over IP address whitelists and blacklists at the domain name level.

IP address whitelists and blacklists at the route level

Annotation

Description

nginx.ingress.kubernetes.io/whitelist-source-range

Specifies the IP address whitelist of a specific route. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

mse.ingress.kubernetes.io/blacklist-source-range

Specifies the IP address blacklist of a specific route. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

Examples:

  • Allow access to example.com/test from the client IP address 1.1.xx.xx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com 
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Deny access to example.com/test from the client IP address 2.2.xx.xx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

IP address whitelists and blacklists at the domain name level

Annotation

Description

mse.ingress.kubernetes.io/domain-whitelist-source-range

Specifies the IP address whitelist of a specific domain name. IP address whitelists at the route level take precedence over IP address whitelists at the domain name level. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

mse.ingress.kubernetes.io/domain-blacklist-source-range

Specifies the IP address blacklist of a specific domain name. IP address blacklists at the route level take precedence over IP address blacklists at the domain name level. IP addresses and CIDR blocks are supported. Separate IP addresses or CIDR blocks with commas (,).

Examples:

  • Allow access to all routes of the domain name example.com from the client IP addresses 1.1.xx.xx and 2.2.xx.xx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact
             - backend:
                  service:
                    name: app-service
                    port: 
                      number: 80
                path: /app
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
              - path: /app
                backend:
                  serviceName: app-service
                  servicePort: 80
  • Use IP address whitelists and blacklists at the domain name level with IP address whitelists and blacklists at the route level. Allow access to all routes of the example.com domain name from the client IP addresses 1.1.xx.xx and 2.2.xx.xx, and allow access to the example.com/order route only from the client IP address 3.3.xx.xx. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo-domain
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact
             - backend:
                  service:
                    name: app-service
                    port: 
                      number: 80
                path: /app
                pathType: Exact
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
      name: demo-route
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /order
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
      name: demo-domain
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
              - path: /app
                backend:
                  serviceName: app-service
                  servicePort: 80
    ---
    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
      name: demo-route
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /order
                backend:
                  serviceName: demo-service
                  servicePort: 80

Single-gateway throttling

MSE Ingress allows you to configure throttling policies for a single gateway instance at the route level. The throttling policies ensure that the number of requests matched on a route for a gateway replica does not exceed the configured threshold in a specified period of time.

Note

The system performs throttling for a single gateway instance based on the configured threshold. If you want to throttle traffic of a route for a gateway cluster, you can use global throttling control policies instead.

Annotation

Description

mse.ingress.kubernetes.io/route-limit-rpm

Specifies the maximum number of requests per minute (RPM) that are routed on a gateway. The burst limit for the maximum number of RPM is equal to the specified value multiplied by mse.ingress.kubernetes.io/route-limit-burst-multiplier.

When throttling is triggered, the response body content is local_rate_limited. Response status codes:

  • If the gateway version is earlier than V1.2.23, the status code 503 is returned.

  • If the gateway version is V1.2.23 or later, the status code 429 is returned.

mse.ingress.kubernetes.io/route-limit-rps

Specifies the maximum number of requests per second (RPS) that are routed on a gateway. The burst limit for the maximum number of RPS is equal to the specified value multiplied by mse.ingress.kubernetes.io/route-limit-burst-multiplier.

When throttling is triggered, the response body content is local_rate_limited. Response status codes:

  • If the gateway version is earlier than V1.2.23, the status code 503 is returned.

  • If the gateway version is V1.2.23 or later, the status code 429 is returned.

mse.ingress.kubernetes.io/route-limit-burst-multiplier

Specifies the multiplier of the burst limits. Default value: 5.

Examples:

  • For the example.com/test request, set the maximum number of RPM to 100 and the burst limit for the maximum number of RPM to 200. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/route-limit-rpm: "100"
        mse.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/route-limit-rpm: "100"
        mse.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • For the example.com/test request, set the maximum number of RPS to 10 and the burst limit for the maximum number of RPS to 50. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/route-limit-rps: "10"
        # The default value is 5.
        # mse.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        mse.ingress.kubernetes.io/route-limit-rps: "10"
        # The default value is 5.
        # mse.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

Global throttling control

MSE Ingress is integrated with Sentinel to provide global throttling control for a gateway cluster at the route level. To implement global throttling control, you must specify the maximum number of RPS on a route in a gateway cluster.

Note

To use this feature, you must make sure that the version of the MSE Ingress gateway is 1.2.25 or later.

You can use the mse.ingress.kubernetes.io/rate-limit annotation to specify the maximum number of RPS on a route in a gateway cluster. By default, the response code is 429, and the response body is sentinel rate limited when throttling is triggered. MSE Ingress provides two methods to configure the custom throttling behavior: custom response and redirect. You can select only one of the two methods.

Custom response

  • mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: the response code when throttling is triggered. The default response code is 429.

  • mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body-type: the response body type when throttling is triggered. The default response body type is text.

    • If you set this annotation to text, the Content-Type value in the response is text/plain; charset=UTF-8.

    • If you set this annotation to json, the Content-Type value in the response is application/json; charset=UTF-8.

  • mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: the response body when throttling is triggered. The default value is sentinel rate limited.

Example 1: Set the maximum number of RPS of the example.com/test request in a gateway cluster to 100 and retain the default throttling behavior. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Example 2: Set the maximum number of RPS of the example.com/test request in a gateway cluster to 100 and set the response code to 503 and response body to server is overload when throttling is triggered.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
    mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
    mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
    mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirect

  • mse.ingress.kubernetes.io/rate-limit-fallback-redirect-url: the redirect URL when throttling is triggered.

Example 1: Set the maximum number of RPS of the example.com/test request in a gateway cluster to 100 and the redirect URL to example.com/fallback when throttling is triggered.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
    mse.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/rate-limit: "100"
    mse.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Global concurrency control

MSE Ingress is integrated with Sentinel to provide global concurrency control for a gateway cluster at the route level. To implement global concurrency control, you must specify the maximum number of requests that are being processed on a route in a gateway cluster.

Note

To use this feature, you must make sure that the version of the MSE Ingress gateway is 1.2.25 or later.

You can use the mse.ingress.kubernetes.io/concurrency-limit annotation to specify the maximum number of requests that can be processed on a route in a gateway cluster. When global concurrency control is triggered, the response code is 429 and the response body is sentinel rate limited. MSE Ingress provides two methods to configure the custom concurrency behavior: custom response and redirect. You can select only one of the two methods.

Custom response

  • mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: the response code when concurrency control is triggered. The default response code is 429.

  • mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body-type: the response body type when concurrency control is triggered. The default response body type is text.

    • If you set this annotation to text, the Content-Type value in the response is text/plain; charset=UTF-8.

    • If you set this annotation to json, the Content-Type value in the response is application/json; charset=UTF-8.

  • mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: the response body when concurrency control is triggered. The default value is sentinel rate limited.

Example 1: Set the maximum number of example.com/test requests processed on a gateway cluster to 1000 and retain the default concurrency control behavior. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Example 2: Set the maximum number of requests from example.com/test that are processed on a gateway cluster to 1,000 and set the response code to 503 and response body to server is overloaded when concurrency control is triggered.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
    mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
    mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
    mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overload"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Redirect

  • mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: the redirect URL when concurrency control is triggered.

Set the maximum number of example.com/test requests processed on a gateway cluster to 1000 and the redirect URL to example.com/fallback when concurrency control is triggered.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
    mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/concurrency-limit: "1000"
    mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Traffic mirroring

You can configure traffic mirroring to copy traffic to a specified service. Traffic mirroring is suitable for scenarios such as operational auditing and traffic testing.

  • mse.ingress.kubernetes.io/mirror-target-service: the destination service to which copied traffic is forwarded. The service format is namespace/name:port.

    • namespace: the namespace where the Kubernetes service resides. This parameter is optional. The default namespace is the namespace where the Ingress gateway resides.

    • name: the name of the Kubernetes service. This parameter is required.

    • port: the Kubernetes service port to which mirrored traffic is forwarded. This parameter is optional. By default, the first port is used.

  • mse.ingress.kubernetes.io/mirror-percentage: the mirrored traffic percentage. Valid values: 0-100. Default value: 100.

Note

When the copied traffic is forwarded to the destination service, the -shadow suffix is automatically added to the Host header in the original request.

For example, the requests from example.com/test are copied and forwarded to the destination service test/app:8080.

Note

In this example, the value of the Host header is automatically changed to example.com-shadow when the copied traffic is forwarded to the destination service.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

For example, the requests from example.com/test are copied and forwarded to the destination service test/app:8080 and the mirrored traffic percentage is 10%.

Note

In this example, the value of the Host header is automatically changed to example.com-shadow when the copied traffic is forwarded to the destination service.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
    mse.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
    mse.ingress.kubernetes.io/mirror-percentage: 10
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Backend service protocols: HTTPS and gRPC

By default, MSE Ingress uses HTTP to forward requests to backend service containers. If your backend service container uses HTTPS, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" to forward requests to your backend service container. If your backend service container uses gRPC, you can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC" to forward requests to your backend service container.

Note

If gRPC or HTTP/2 is specified for name in ports of the resource configuration for the Kubernetes service to which your backend service belongs, MSE Ingress automatically uses the gRPC or HTTP/2 protocol to forward requests to your backend service container. You do not need to configure the annotation nginx.ingress.kubernetes.io/backend-protocol: "GRPC". This implementation is different from that in NGINX Ingress.

Examples:

  • Use HTTPS to forward the example/test request to a backend service. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Use gRPC to forward the example/test request to a backend service. The following methods can be used:

    • Method 1: Use annotations. Sample configurations:

      Clusters that run Kubernetes V1.19 or later

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - backend:
                    service:
                      name: demo-service
                      port: 
                        number: 80
                  path: /test
                  pathType: Exact

      Clusters that run Kubernetes versions earlier than V1.19

      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress
      metadata:
        annotations:
          nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - path: /test
                  backend:
                    serviceName: demo-service
                    servicePort: 80
    • Method 2: Use spec > ports > name in Ingress resource configurations. Sample configurations:

      Clusters that run Kubernetes V1.19 or later

      apiVersion: networking.k8s.io/v1
      kind: Ingress
      metadata:
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - backend:
                    service:
                      name: demo-service
                      port: 
                        number: 80
                  path: /order
                  pathType: Exact
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: demo-service
      spec:
        ports:
          - name: grpc
            port: 80
            protocol: TCP
        selector:
          app: demo-service

      Clusters that run Kubernetes versions earlier than V1.19

      apiVersion: networking.k8s.io/v1beta1
      kind: Ingress
      metadata:
        name: demo
      spec:
        ingressClassName: mse
        rules:
          - host: example.com
            http:
              paths:
                - path: /test
                  backend:
                    serviceName: demo-service
                    servicePort: 80
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: demo-service
      spec:
        ports:
          - name: grpc
            port: 80
            protocol: TCP
        selector:
          app: demo-service

Load balancing algorithms for backend services

Load balancing algorithms determine how to select nodes when a gateway forwards requests to backend services.

Common load balancing algorithms

nginx.ingress.kubernetes.io/load-balance: specifies the common load balancing algorithm that is used by a backend service. Default value: round_robin. Valid values:

  • round_robin: load balancing based on round robin.

  • least_conn: load balancing based on least connections.

  • random: randomized load balancing.

Important

Cloud-native gateways do not support the exponentially weighted moving average (EWMA) algorithm. If you configure the EWMA algorithm, the algorithm is rolled back to the round robin load balancing algorithm.

For example, configure the least connection load balancing algorithm for the demo-service backend service. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /order
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/load-balance: "least_conn"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Load balancing algorithms based on consistent hashing

The load balancing algorithm based on consistent hashing has request affinity. Requests that have the same characteristics are always distributed to the same node. MSE Ingress can use NGINX variables, request headers, and request path parameters as hash keys.

nginx.ingress.kubernetes.io/upstream-hash-by: specifies the load balancing algorithm based on consistent hashing. Cloud-native gateways support the following types of consistent hashing:

  • Consistent hashing based on NGINX variables:

    • $request_uri: the request path, which is used as the hash key. Path parameters are included.

    • $host: the request host, which is used as the hash key.

    • $remote_addr: the client IP address, which is used as the hash key.

  • Consistent hashing based on request headers. You need only to configure $http_headerName.

  • Consistent hashing based on request path parameters. You need only to configure $arg_varName.

Examples:

  • Use a client IP address for requests as the hash key. Requests from the client IP address are always distributed to the same node. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Use the request header X-Stage as the hash key, and distribute requests with the same value of the request header X-Stage to the same node. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Use the request path parameter X-Stage as the hash key, and distribute requests with the same value of the request path parameter X-Stage to the same node. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

Service prefetching (graceful start)

The service prefetching feature allows traffic to gradually increase in a specified service prefetching window when a new node is released. This ensures that the node is completely prefetched.

mse.ingress.kubernetes.io/warmup: specifies the period of time in which services are prefetched. Unit: seconds. By default, the service prefetching feature is not enabled.

Note

Service prefetching depends on the selected load balancing algorithm. Only the load balancing algorithms based on round robin and least connections are supported.

For example, enable the service prefetching feature for the backend service demo-service and set the prefetching time window to 30 seconds. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/warmup: "30"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

Cookie affinity (session persistence)

Requests with the same cookie are always distributed to the same node on a cloud-native gateway. If the client sends the first request with a cookie to the gateway, MSE Ingress returns a response that contains a cookie generated for the client. The generated cookie is used to ensure that subsequent requests are always distributed to the same node of the gateway.

Annotation

Description

nginx.ingress.kubernetes.io/affinity

Specifies the affinity type. The default and only valid value is cookie.

nginx.ingress.kubernetes.io/affinity-mode

Specifies the affinity mode. The default and only valid value is balanced.

nginx.ingress.kubernetes.io/session-cookie-name

Specifies the name of the cookie that is used as the hash key. Default value: INGRESSCOOKIE.

nginx.ingress.kubernetes.io/session-cookie-path

Specifies the path of the cookie that is generated if the specified cookie does not exist. Default value: /.

nginx.ingress.kubernetes.io/session-cookie-max-age

Specifies the expiration time of the cookie that is generated when the specified cookie does not exist. Unit: seconds. By default, this annotation is specified at the session level.

nginx.ingress.kubernetes.io/session-cookie-expires

Specifies the expiration time of the cookie that is generated when the specified cookie does not exist. Unit: seconds. By default, this annotation is specified at the session level.

Examples:

  • Enable cookie affinity and use the default configuration of MSE Ingress. Set the cookie name to INGRESSCOOKIE, set the path to /, and configure the lifecycle of the cookie at the session level. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80
  • Enable cookie affinity and set the cookie name to test, the path to /, and the expiration time of the cookie to 10s. Sample configurations:

    Clusters that run Kubernetes V1.19 or later

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
        nginx.ingress.kubernetes.io/session-cookie-name: "test"
        nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - backend:
                  service:
                    name: demo-service
                    port: 
                      number: 80
                path: /test
                pathType: Exact

    Clusters that run Kubernetes versions earlier than V1.19

    apiVersion: networking.k8s.io/v1beta1
    kind: Ingress
    metadata:
      annotations:
        nginx.ingress.kubernetes.io/affinity: "cookie"
        nginx.ingress.kubernetes.io/session-cookie-name: "test"
        nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
      name: demo
    spec:
      ingressClassName: mse
      rules:
        - host: example.com
          http:
            paths:
              - path: /test
                backend:
                  serviceName: demo-service
                  servicePort: 80

Configuration of a connection pool between a gateway and a backend service

After you configure a connection pool for a specified backend service on the gateway, you can control the number of connections between the gateway and backend service. This helps prevent the backend service from being overloaded and ensures the stability and high availability of the backend service.

  • mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: the maximum number of connections that can be established between a gateway and a backend service.

  • mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: the maximum number of connections that can be established between a gateway and a single node of a backend service.

  • mse.ingress.kubernetes.io/connection-policy-http-max-request-per-connection: the maximum number of requests on a single connection between a gateway and a backend service.

For example, for the backend service demo-service, you can set the maximum number of connections that can be established between a gateway and the backend service to 10, and set the maximum number of connections that can be established between a gateway and a single node of the backend service to 2.

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
  	mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
  	mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

TLS versions and cipher suites for communications between a client and a cloud-native gateway

The default earliest TLS version that is supported by MSE Ingress is TLSv1.0. The default latest TLS version is TLSv1.3. By default, the following cipher suites are used:

  • ECDHE-ECDSA-AES128-GCM-SHA256

  • ECDHE-RSA-AES128-GCM-SHA256

  • ECDHE-ECDSA-AES128-SHA

  • ECDHE-RSA-AES128-SHA

  • AES128-GCM-SHA256

  • AES128-SHA

  • ECDHE-ECDSA-AES256-GCM-SHA384

  • ECDHE-RSA-AES256-GCM-SHA384

  • ECDHE-ECDSA-AES256-SHA

  • ECDHE-RSA-AES256-SHA

  • AES256-GCM-SHA384

  • AES256-SHA

The following table describes the annotations that can be used to specify the earliest or latest TLS version and cipher suites for a specific domain name.

Annotation

Description

mse.ingress.kubernetes.io/tls-min-protocol-version

Specifies the earliest version of TLS. Default value: TLSv1.0. Valid values:

  • TLSv1.0

  • TLSv1.1

  • TLSv1.2

  • TLSv1.3

mse.ingress.kubernetes.io/tls-max-protocol-version

Specifies the latest version of TLS. Default value: TLSv1.3.

nginx.ingress.kubernetes.io/ssl-cipher

Specifies the TLS cipher suites. You can specify multiple TLS cipher suites, which are separated by commas (,). This annotation takes effect only if a TLS version ranging from v1.0 to v1.2 is used for a TLS handshake.

For example, the domain name example.com is available. For this domain name, set the earliest TLS version to TLSv1.2 and the latest version to TLSv1.2. Sample configurations:

Clusters that run Kubernetes V1.19 or later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    mse.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    mse.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
    mse.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80

mTLS between gateways and backend services

By default, MSE Ingress uses HTTP to forward requests to backend service containers. You can use the annotation nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" to allow MSE Ingress to access the backend service over HTTPS. In this case, one-way TLS is used. MSE Ingress authenticates the certificate provided by the backend service and the certificate must be issued by a well-known certificate authority (CA). Mutual TLS (mTLS) is also allowed. If mTLS is used, the gateway and backend service perform two-way authentication. The gateway verifies whether the certificate provided by the backend service is valid and the backend service verifies whether the certificate provided by the gateway is valid.

Annotation

Description

nginx.ingress.kubernetes.io/proxy-ssl-secret

Specifies the client certificate used by the gateway. The client certificate is used for a backend service to authenticate the gateway. The format is secretNamespace/secretName.

nginx.ingress.kubernetes.io/proxy-ssl-name

Specifies the Server Name Indication (SNI) that is used during a TLS handshake.

nginx.ingress.kubernetes.io/proxy-ssl-server-name

Specifies whether to enable or disable the SNI that is used during a TLS handshake.

For example, perform an mTLS between a cloud-native gateway and a backend service, set secretName to gateway-cert, and set secretNamespace to default. Sample configurations:

Clusters that run Kubernetes 1.19 and later

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - backend:
              service:
                name: demo-service
                port: 
                  number: 80
            path: /test
            pathType: Exact

Clusters that run Kubernetes versions earlier than V1.19

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/ateway-cert"
  name: demo
spec:
  ingressClassName: mse
  rules:
    - host: example.com
      http:
        paths:
          - path: /test
            backend:
              serviceName: demo-service
              servicePort: 80