MSE Ingress provides Layer-7 load balancing for managing external access to services in a Kubernetes cluster. Configure annotation-based features on Ingress resources — from canary releases and rate limiting to TLS and traffic mirroring.
All examples include configurations for both Kubernetes v1.19+ (networking.k8s.io/v1) and earlier versions (networking.k8s.io/v1beta1).
Annotation quick reference
All annotations in this document use either the nginx.ingress.kubernetes.io/ or mse.ingress.kubernetes.io/ prefix. The table below lists every annotation with its accepted values, default, and scope.
Annotation | Accepted values | Default | Scope |
|
| — | Route |
| string | — | Route |
| string | — | Route |
| string | — | Route |
| integer (0–100 or 0–total) | — | Route |
| integer |
| Route |
| string | — | Route |
| string | — | Route |
| string | — | Route |
|
| — | Route |
|
|
| Route |
| comma-separated origins or |
| Route |
| comma-separated HTTP methods or |
| Route |
| comma-separated headers or |
| Route |
| comma-separated headers | — | Route |
|
|
| Route |
| integer (seconds) |
| Route |
|
| — | Route |
| path string | — | Route |
| hostname | — | Route |
|
|
| Route |
|
|
| Route |
| URL (with scheme) | — | Route |
| HTTP status code |
| Route |
| URL (with scheme) | — | Route |
|
| — | Route |
|
| — | Route |
| comma-separated header names | — | Route |
|
| — | Route |
|
| — | Route |
| comma-separated header names | — | Route |
| integer |
| Route |
| integer (seconds) | no timeout | Route |
| comma-separated conditions |
| Route |
| comma-separated IPs or CIDRs | — | Route |
| comma-separated IPs or CIDRs | — | Route |
| comma-separated IPs or CIDRs | — | Domain |
| comma-separated IPs or CIDRs | — | Domain |
| integer | — | Route |
| integer | — | Route |
| integer |
| Route |
| integer (RPS) | — | Route |
| HTTP status code |
| Route |
|
|
| Route |
| string |
| Route |
| URL | — | Route |
| integer | — | Route |
| HTTP status code |
| Route |
|
|
| Route |
| string |
| Route |
| URL | — | Route |
|
| — | Route |
| integer (0–100) |
| Route |
|
| — | Route |
|
|
| Route |
| NGINX variable or expression | — | Route |
| integer (seconds) | disabled | Route |
|
| — | Route |
|
|
| Route |
| string |
| Route |
| path string |
| Route |
| integer (seconds) | session-scoped | Route |
| integer (seconds) | session-scoped | Route |
| integer | — | Route |
| integer | — | Route |
| integer | — | Route |
|
|
| Domain |
|
|
| Domain |
| comma-separated cipher suites | see defaults | Domain |
|
| — | Route |
| hostname | — | Route |
|
| — | Route |
Canary release
MSE Ingress supports four canary release strategies. Enable canary on any Ingress resource by adding the annotation nginx.ingress.kubernetes.io/canary: "true". Then apply one or more of the strategy-specific annotations below.
MSE Ingress supports more than two service versions in a single canary release, with no upper limit. NGINX Ingress and ALB Ingress support at most two versions.
When multiple strategies are configured simultaneously, MSE Ingress applies them in this precedence order:
Header-based or query parameter-based (highest priority)
Cookie-based
Weight-based (lowest priority)
Header-based canary release
Annotation | Description | Default | Required |
| Routes requests to the canary version when the specified header is present and set to | — | Yes (for header-based canary) |
| Routes requests to the canary version only when the header matches both the key (from | — | No |
Example 1: Route by header presence
When the request header mse is set to always, requests go to demo-service-canary. All other requests go to demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Example 2: Route multiple canary versions by exact header value
When the header mse: v1 is present, requests go to demo-service-canary-v1. When mse: v2 is present, they go to demo-service-canary-v2. All other requests go to demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary-v1
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
name: demo-canary-v2
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary-v2
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary-v1
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v2"
name: demo-canary-v2
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary-v2
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Query parameter-based canary release
Annotation | Description | Default | Required |
| Routes requests to the canary version when the specified query parameter is present and set to | — | Yes (for query-based canary) |
| Routes requests to the canary version only when the query parameter matches both the key and this value. | — | No |
Header-based and query parameter-based conditions can be combined. Requests are sent to the canary version only when both conditions match simultaneously.
Example 1: Route by query parameter value
When the URL contains canary=gray, requests go to demo-service-canary. All other requests go to demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
mse.ingress.kubernetes.io/canary-by-query: "canary"
mse.ingress.kubernetes.io/canary-by-query-value: "gray"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
mse.ingress.kubernetes.io/canary-by-query: "canary"
mse.ingress.kubernetes.io/canary-by-query-value: "gray"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Example 2: Combine query parameter with header conditions
When the URL contains canary=gray AND the request header contains x-user-id: test, requests go to demo-service-canary. Both conditions must match.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
mse.ingress.kubernetes.io/canary-by-query: "canary"
mse.ingress.kubernetes.io/canary-by-query-value: "gray"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
mse.ingress.kubernetes.io/canary-by-query: "canary"
mse.ingress.kubernetes.io/canary-by-query-value: "gray"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Cookie-based canary release
Annotation | Description | Default | Required |
| Routes requests to the canary version when the specified cookie is set to | — | Yes (for cookie-based canary) |
Example: When the request cookie demo=always is present, requests go to demo-service-canary. All other requests go to demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-cookie: "demo"
name: demo-canary
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Weight-based canary release
Annotation | Description | Default | Required |
| Percentage of requests sent to the canary version. Integer in the range 0–100 (or 0– | — | Yes (for weight-based canary) |
| Total weight used as the denominator when calculating request distribution. |
| No |
Example: Distribute traffic across three versions — 30% to demo-service-canary-v1, 20% to demo-service-canary-v2, and 50% to demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "30"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary-v1
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "20"
name: demo-canary-v2
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary-v2
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "30"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary-v1
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "20"
name: demo-canary-v2
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary-v2
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Service subset
A service subset lets you route requests to a specific group of Pods within a single Kubernetes Service — useful when one Service is backed by multiple Deployments running different versions. MSE Ingress provides two methods for defining subsets.
Use pod labels in MseIngressConfig
Use the mse.ingress.kubernetes.io/service-subset annotation to select Pods by their OpenSergo canary labels:
""orbase: routes to Pods without anopensergo.io/canarylabel prefix, or withopensergo.io/canary: "".Any other value (for example,
gray): routes to Pods labeledopensergo.io/canary-{value}: {value}— for example,opensergo.io/canary-gray: gray.
Example: A Service named go-httpbin is backed by two Deployments. Base Pods have no OpenSergo canary label. Gray Pods have the label opensergo.io/canary-gray: gray.
Service and Deployment manifests:
# go-httpbin Service
apiVersion: v1
kind: Service
metadata:
name: go-httpbin
namespace: default
spec:
ports:
- port: 8080
protocol: TCP
selector:
app: go-httpbin
---
# Base Deployment — no OpenSergo canary label
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-httpbin-base
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: go-httpbin
template:
metadata:
labels:
app: go-httpbin
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
args:
- "--version=base"
imagePullPolicy: Always
name: go-httpbin
---
# Gray Deployment — has the OpenSergo canary label
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-httpbin-gray
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: go-httpbin
template:
metadata:
labels:
app: go-httpbin
opensergo.io/canary-gray: gray
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
args:
- "--version=gray"
imagePullPolicy: Always
name: go-httpbinIngress configuration — requests with x-user-id: test go to go-httpbin-gray; all others go to go-httpbin-base:
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
# Forward to Pods labeled opensergo.io/canary-gray: gray
mse.ingress.kubernetes.io/service-subset: gray
name: demo-canary
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: go-httpbin
port:
number: 8080
path: /test
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
# Forward to Pods without the opensergo.io/canary label prefix
mse.ingress.kubernetes.io/service-subset: ""
name: demo
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: go-httpbin
port:
number: 8080
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
# Forward to Pods labeled opensergo.io/canary-gray: gray
mse.ingress.kubernetes.io/service-subset: gray
name: demo-canary
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /test
backend:
serviceName: go-httpbin
servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
# Forward to Pods without the opensergo.io/canary label prefix
mse.ingress.kubernetes.io/service-subset: ""
name: demo
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /test
backend:
serviceName: go-httpbin
servicePort: 8080Use custom labels
Combine mse.ingress.kubernetes.io/service-subset with mse.ingress.kubernetes.io/subset-labels to define the subset by any Pod label instead of the opensergo.io/canary prefix.
Whensubset-labelsis set, the subset is no longer mapped toopensergo.io/canary-prefixed labels.
Example: The gray Deployment has version: gray instead of an OpenSergo label.
Service and Deployment manifests:
# go-httpbin Service
apiVersion: v1
kind: Service
metadata:
name: go-httpbin
namespace: default
spec:
ports:
- port: 8080
protocol: TCP
selector:
app: go-httpbin
---
# Base Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-httpbin-base
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: go-httpbin
template:
metadata:
labels:
app: go-httpbin
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
args:
- "--version=base"
imagePullPolicy: Always
name: go-httpbin
---
# Gray Deployment — uses a custom label instead of opensergo.io/canary
apiVersion: apps/v1
kind: Deployment
metadata:
name: go-httpbin-gray
namespace: default
spec:
replicas: 1
selector:
matchLabels:
app: go-httpbin
template:
metadata:
labels:
app: go-httpbin
version: gray
spec:
containers:
- image: registry.cn-hangzhou.aliyuncs.com/mse/go-httpbin
args:
- "--version=gray"
imagePullPolicy: Always
name: go-httpbinIngress configuration — requests with x-user-id: test go to go-httpbin-gray; all others go to go-httpbin-base:
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
# Forward to Pods labeled version: gray
mse.ingress.kubernetes.io/service-subset: gray
mse.ingress.kubernetes.io/subset-labels: version gray
name: demo-canary
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: go-httpbin
port:
number: 8080
path: /test
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/service-subset: ""
name: demo
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: go-httpbin
port:
number: 8080
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "x-user-id"
nginx.ingress.kubernetes.io/canary-by-header-value: "test"
# Forward to Pods labeled version: gray
mse.ingress.kubernetes.io/service-subset: gray
mse.ingress.kubernetes.io/subset-labels: version gray
name: demo-canary
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /test
backend:
serviceName: go-httpbin
servicePort: 8080
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/service-subset: ""
name: demo
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /test
backend:
serviceName: go-httpbin
servicePort: 8080CORS
Cross-Origin Resource Sharing (CORS) lets web applications access resources on a different origin. For the CORS specification, see Cross-Origin Resource Sharing (CORS).
Annotation | Description | Default | Required |
| Enables CORS on this Ingress. |
| Yes (to enable CORS) |
| Allowed third-party origins. Accepts comma-separated values and wildcards ( |
| No |
| Allowed HTTP methods. Accepts comma-separated values and wildcards ( |
| No |
| Allowed request headers. Accepts comma-separated values and wildcards ( |
| No |
| Response headers exposed to the browser. Accepts comma-separated values. | — | No |
| Whether credentials can be included in CORS requests. |
| No |
| Cache duration for preflight results, in seconds. |
| No |
Example: Allow example.com as the origin, restrict methods to GET and POST, allow the X-Foo-Bar request header, and disable credential passing.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/enable-cors: "true"
nginx.ingress.kubernetes.io/cors-allow-origin: "example.com"
nginx.ingress.kubernetes.io/cors-allow-methods: "GET,POST"
nginx.ingress.kubernetes.io/cors-allow-headers: "X-Foo-Bar"
nginx.ingress.kubernetes.io/cors-allow-credentials: "false"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Regular expression match
In addition to exact and prefix match, MSE Ingress supports regular expression matching. Set nginx.ingress.kubernetes.io/use-regex: "true" to treat the path defined in the Ingress spec as a regular expression.
Example: Forward requests to example.com whose paths start with /app or /test to the demo service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/use-regex: 'true'
name: regex-match
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo
port:
number: 8080
path: /(app|test)/(.*)
pathType: PrefixKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/use-regex: 'true'
name: regex-match
namespace: default
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /(app|test)/(.*)
backend:
serviceName: demo
servicePort: 8080Path and host rewrite
Rewrite operations modify the path or host in a request before it reaches the backend service.
Annotation | Description | Default | Required |
| Destination path. Supports capture groups from the Ingress path pattern. | — | Yes (for path rewrite) |
| Destination host. | — | Yes (for host rewrite) |
Path rewrite
Example 1: Rewrite example.com/test to example.com/dev before forwarding to the backend.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/dev"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/dev"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /test
pathType: Exact
backend:
serviceName: demo-service
servicePort: 80Example 2: Strip the /v1 prefix — rewrite example.com/v1/xxx to example.com/xxx.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /v1/(.*)
pathType: PrefixKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/$1"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /v1/(.*)
pathType: Prefix
backend:
serviceName: demo-service
servicePort: 80Example 3: Replace the /v1 prefix with /v2 — rewrite example.com/v1/xxx to example.com/v2/xxx.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /v1/(.*)
pathType: PrefixKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/v2/$1"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /v1/(.*)
pathType: Prefix
backend:
serviceName: demo-service
servicePort: 80Host rewrite
Example: Rewrite the host from example.com to test.com before forwarding example.com/test to the backend.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-vhost: "test.com"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Redirect
HTTP-to-HTTPS redirect
Annotation | Description | Default | Required |
| Forces HTTP requests to redirect to HTTPS. |
| Yes (to enable redirect) |
| Forces HTTP requests to redirect to HTTPS. |
| Yes (alternative to |
MSE Ingress treats bothssl-redirectandforce-ssl-redirectidentically — both force HTTP-to-HTTPS redirection.
Example: Redirect http://example.com/test to https://example.com/test.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: "true"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Permanent redirect
Annotation | Description | Default | Required |
| Destination URL for the permanent redirect. Must include the scheme ( | — | Yes (for permanent redirect) |
| HTTP status code for the redirect. |
| No |
Example: Permanently redirect http://example.com/test to http://example.com/app.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/permanent-redirect: "http://example.com/app"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Temporary redirect
Annotation | Description | Default | Required |
| Destination URL for the temporary redirect. Must include the scheme ( | — | Yes (for temporary redirect) |
Example: Temporarily redirect http://example.com/test to http://example.com/app.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/temporal-redirect: "http://example.com/app"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Header control
Header control lets you add, modify, or remove request headers before MSE Ingress forwards a request to a backend service, and response headers before the response is returned to the client.
Request header control
Annotation | Description | Default | Required |
| Adds a header to the request. If the header already exists, the value is appended to the original value. For a single header, use a key-value pair. For multiple headers, use YAML block scalar ( | — | No |
| Modifies an existing header in the request. If the header already exists, the value overwrites the original. Same syntax as | — | No |
| Removes a header from the request. For multiple headers, separate with commas ( | — | No |
Example 1: Add foo: bar and test: true to all requests to example.com/test.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/request-header-control-add: |
foo bar
test true
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/request-header-control-add: |
foo bar
test true
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Combine header control with canary releases. When the request header is mse: v1, the request goes to demo-service-canary-v1 with stage: gray added. All other requests go to demo-service with stage: production added.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
mse.ingress.kubernetes.io/request-header-control-add: "stage gray"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service-canary-v1
port:
number: 80
path: /hello
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/request-header-control-add: "stage production"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /hello
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-by-header: "mse"
nginx.ingress.kubernetes.io/canary-by-header-value: "v1"
mse.ingress.kubernetes.io/request-header-control-add: "stage gray"
name: demo-canary-v1
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service-canary-v1
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/request-header-control-add: "stage production"
name: demo
spec:
ingressClassName: mse
rules:
- http:
paths:
- path: /hello
backend:
serviceName: demo-service
servicePort: 80Response header control
Annotation | Description | Default | Required |
| Adds a header to the response from the backend before forwarding to the client. If the header already exists, the value is appended. Same syntax as request header control. | — | No |
| Modifies an existing response header. If the header already exists, the value overwrites the original. Same syntax as request header control. | — | No |
| Removes a response header before forwarding to the client. For multiple headers, separate with commas ( | — | No |
Example: Remove the req-cost-time header from responses to example.com/test.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/response-header-control-remove: "req-cost-time"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Retry
MSE Ingress provides route-level automatic retry for failed requests. Configure retry conditions, limits, and timeouts with the following annotations.
Annotation | Description | Default | Required |
| Maximum number of retries. |
| No |
| Timeout for all retry attempts combined, in seconds. | No timeout | No |
| Retry conditions, comma-separated. Valid values: |
| No |
By default, MSE Ingress does not retry non-idempotent requests (POST, PATCH) even when other retry conditions are met. Addnon_idempotenttoproxy-next-upstreamto enable retries for these methods.
Example: For example.com/test, set a maximum of 2 retries with a 5-second timeout, trigger retries only on HTTP 502, and enable retries for non-idempotent requests.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-next-upstream-tries: "2"
nginx.ingress.kubernetes.io/proxy-next-upstream-timeout: "5"
nginx.ingress.kubernetes.io/proxy-next-upstream: "http_502,non_idempotent"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80IP address allowlists (whitelists) and blocklists (blacklists)
MSE Ingress supports IP address allowlists (whitelists) and blocklists (blacklists) at both the route level and the domain level. Route-level rules take precedence over domain-level rules.
Route-level IP control
Annotation | Description | Default | Required |
| IP addresses or CIDR blocks allowed for this route. Comma-separated. | — | No |
| IP addresses or CIDR blocks blocked for this route. Comma-separated. | — | No |
Example 1: Allow access to example.com/test only from 1.1.X.X.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 1.1.X.X
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Block access to example.com/test from 2.2.2.2.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/blacklist-source-range: 2.2.2.2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Domain-level IP control
Annotation | Description | Default | Required |
| IP addresses or CIDR blocks allowed for all routes of a domain. Route-level allowlists take precedence. Comma-separated. | — | No |
| IP addresses or CIDR blocks blocked for all routes of a domain. Route-level blocklists take precedence. Comma-separated. | — | No |
Example 1: Allow access to all routes of example.com from 1.1.X.X and 2.2.2.2.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: Exact
- backend:
service:
name: app-service
port:
number: 80
path: /app
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80
- path: /app
backend:
serviceName: app-service
servicePort: 80Example 2: Allow all routes of example.com from 1.1.X.X and 2.2.2.2, but restrict /order to only 3.3.X.X. The route-level allowlist on /order overrides the domain-level allowlist for that route.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
name: demo-domain
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: Exact
- backend:
service:
name: app-service
port:
number: 80
path: /app
pathType: Exact
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
name: demo-route
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /order
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/domain-whitelist-source-range: 1.1.X.X,2.2.2.2
name: demo-domain
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80
- path: /app
backend:
serviceName: app-service
servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 3.3.X.X
name: demo-route
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /order
backend:
serviceName: demo-service
servicePort: 80Single-gateway throttling
Single-gateway throttling limits the request rate on a route per gateway replica. Use this for per-instance rate control. For cluster-wide rate control, see Global throttling control.
MSE Ingress uses a token bucket model for burst control. The route-limit-rpm or route-limit-rps annotation sets the sustained rate — how fast the token bucket refills. The route-limit-burst-multiplier annotation sets the burst capacity as a multiple of that rate: burst limit = rate × multiplier. Requests that exceed the burst capacity are rejected with local_rate_limited in the response body.
Annotation | Description | Default | Required |
| Maximum requests per minute (RPM) per gateway replica. Burst limit = RPM × | — | No |
| Maximum requests per second (RPS) per gateway replica. Burst limit = RPS × | — | No |
| Burst capacity multiplier applied to the configured RPM or RPS limit. |
| No |
Example 1: Limit example.com/test to 100 RPM with a burst limit of 200 (multiplier = 2).
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/route-limit-rpm: "100"
mse.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/route-limit-rpm: "100"
mse.ingress.kubernetes.io/route-limit-burst-multiplier: "2"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Limit example.com/test to 10 RPS with the default burst limit of 50 (multiplier = 5).
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/route-limit-rps: "10"
# Default burst multiplier is 5, giving a burst limit of 50 RPS
# mse.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/route-limit-rps: "10"
# Default burst multiplier is 5, giving a burst limit of 50 RPS
# mse.ingress.kubernetes.io/route-limit-burst-multiplier: "5"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Global throttling control
MSE Ingress integrates with Sentinel to enforce a global RPS limit across all replicas of a gateway cluster at the route level. When throttling is triggered, the default response code is 429 and the response body is sentinel rate limited.
Global throttling control requires MSE Ingress gateway version 1.2.25 or later.
Configure the cluster-wide rate limit with mse.ingress.kubernetes.io/rate-limit. To customize the throttled response, choose one of two approaches: custom response or redirect. Only one approach can be active at a time.
Custom response
Annotation | Description | Default | Required |
| Maximum RPS for this route across the entire gateway cluster. | — | Yes |
| HTTP response code returned when throttling is triggered. |
| No |
| Response body content type. |
| No |
| Response body content when throttling is triggered. |
| No |
Example 1: Limit example.com/test to 100 RPS cluster-wide with the default response (HTTP 429, sentinel rate limited).
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Limit example.com/test to 100 RPS cluster-wide. When throttled, return HTTP 503 with the body server is overloaded.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overloaded"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: 503
mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: "server is overloaded"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Redirect
Annotation | Description | Default | Required |
| URL to redirect the client to when throttling is triggered. | — | No |
Example: Limit example.com/test to 100 RPS cluster-wide. When throttled, redirect to example.com/fallback.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
mse.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/rate-limit: "100"
mse.ingress.kubernetes.io/rate-limit-fallback-redirect-url: "example.com/fallback"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Global concurrency control
MSE Ingress integrates with Sentinel to limit the number of in-flight requests on a route across the entire gateway cluster. When the limit is exceeded, the default response code is 429 and the response body is sentinel rate limited.
Global concurrency control requires MSE Ingress gateway version 1.2.25 or later.
Configure the cluster-wide concurrency limit with mse.ingress.kubernetes.io/concurrency-limit. To customize the response when the limit is exceeded, use either custom response or redirect. Only one approach can be active at a time.
Custom response
Annotation | Description | Default | Required |
| Maximum number of in-flight requests for this route across the gateway cluster. | — | Yes |
| HTTP response code when concurrency control is triggered. |
| No |
| Response body content type. |
| No |
| Response body content when concurrency control is triggered. |
| No |
Example 1: Limit concurrent in-flight requests to example.com/test to 1,000 cluster-wide, with the default response.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Limit concurrent in-flight requests to example.com/test to 1,000 cluster-wide. When the limit is exceeded, return HTTP 503 with the body server is overloaded.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overloaded"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-code: 503
mse.ingress.kubernetes.io/concurrency-limit-fallback-custom-response-body: "server is overloaded"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Redirect
Annotation | Description | Default | Required |
| URL to redirect the client to when concurrency control is triggered. | — | No |
Example: Limit concurrent in-flight requests to example.com/test to 1,000 cluster-wide. When the limit is exceeded, redirect to example.com/fallback.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/concurrency-limit: "1000"
mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "example.com/fallback"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Traffic mirroring
Traffic mirroring copies live requests to a secondary service without affecting the original response. Use it for auditing, testing, or debugging traffic against a shadow environment.
Annotation | Description | Default | Required |
| Destination service for mirrored traffic. Format: | — | Yes (to enable mirroring) |
| Percentage of traffic to mirror. Valid range: 0–100. |
| No |
When mirrored traffic is forwarded to the destination service, the-shadowsuffix is automatically appended to theHostheader. For example,example.combecomesexample.com-shadow.
Example 1: Mirror 100% of requests to example.com/test to test/app:8080.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Mirror 10% of requests to example.com/test to test/app:8080.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
mse.ingress.kubernetes.io/mirror-percentage: 10
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/mirror-target-service: test/app:8080
mse.ingress.kubernetes.io/mirror-percentage: 10
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Backend service protocols: HTTPS and gRPC
By default, MSE Ingress uses HTTP to forward requests to backend containers. Use nginx.ingress.kubernetes.io/backend-protocol to switch to HTTPS or gRPC.
If the Kubernetes Service resource for your backend has a port namedgrpcorhttp2, MSE Ingress automatically uses the corresponding protocol without requiring thebackend-protocolannotation. This differs from standard NGINX Ingress behavior.
Example 1: Forward requests to a backend over HTTPS.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Forward requests to a gRPC backend. Two methods are available.
Method 1 — use the backend-protocol annotation:
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Method 2 — name the Service port grpc so MSE Ingress auto-detects the protocol:
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /order
pathType: Exact
---
apiVersion: v1
kind: Service
metadata:
name: demo-service
spec:
ports:
- name: grpc
port: 80
protocol: TCP
selector:
app: demo-serviceKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
name: demo-service
spec:
ports:
- name: grpc
port: 80
protocol: TCP
selector:
app: demo-serviceLoad balancing algorithms
MSE Ingress supports several load balancing algorithms for distributing requests to backend Pods. Choose based on your traffic distribution and session requirements:
Algorithm | Annotation value | Description | Use when |
Round robin |
| Distributes requests evenly across Pods in rotation. | Default; suitable for stateless, evenly performing backends. |
Least connections |
| Routes each request to the Pod with the fewest active connections. | Backends with variable request durations. |
Random |
| Routes requests to a randomly selected Pod. | Simple even distribution without state. |
Consistent hashing | See below | Routes requests with the same hash key to the same Pod. | Session affinity by IP, header, or query parameter. |
The exponentially weighted moving average (EWMA) algorithm is not supported. If configured, it falls back to round robin.
Common load balancing algorithms
Annotation | Description | Default | Required |
| Load balancing algorithm. Valid values: |
| No |
Example: Use least connections for demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/load-balance: "least_conn"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /order
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/load-balance: "least_conn"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Consistent hashing
Consistent hashing routes requests with the same hash key to the same backend Pod. MSE Ingress supports hashing by NGINX variables, request headers, or query parameters.
Annotation | Description | Default | Required |
| Hash key expression. Supported values: NGINX variables (e.g., | — | Yes (for consistent hashing) |
Example 1: Hash by client IP — requests from the same IP always reach the same Pod.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$remote_addr"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Hash by the X-Stage request header — requests with the same X-Stage value always reach the same Pod.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$http_x-stage"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 3: Hash by the x-stage query parameter — requests with the same x-stage value always reach the same Pod.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/upstream-hash-by: "$arg_x-stage"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Warm-up (graceful start)
The warm-up feature gradually ramps up traffic to a newly started Pod over a specified time window, preventing a cold Pod from receiving full load immediately. Only the round robin and least connections algorithms support warm-up.
Annotation | Description | Default | Required |
| Warm-up window duration in seconds. When a new Pod starts, traffic increases gradually over this period. | Warm-up disabled | No |
Example: Enable a 30-second warm-up window for demo-service.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/warmup: "30"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/warmup: "30"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Cookie affinity (session persistence)
Cookie affinity pins a client to the same backend Pod across requests by using a gateway-generated cookie. On the first request, MSE Ingress sets a cookie in the response. Subsequent requests from the same client include that cookie and are routed to the same Pod.
Annotation | Description | Default | Required |
| Enables cookie affinity. The only valid value is | — | Yes (to enable cookie affinity) |
| Affinity mode. The only valid value is |
| No |
| Name of the cookie used as the hash key. |
| No |
| Path attribute of the generated cookie. |
| No |
| Expiration time of the generated cookie, in seconds. | Session-scoped | No |
| Expiration time of the generated cookie, in seconds. | Session-scoped | No |
Example 1: Enable cookie affinity with defaults — cookie name INGRESSCOOKIE, path /, session-scoped lifetime.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Example 2: Enable cookie affinity with a custom cookie named test, path /, and a 10-second expiration.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "test"
nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/affinity: "cookie"
nginx.ingress.kubernetes.io/session-cookie-name: "test"
nginx.ingress.kubernetes.io/session-cookie-max-age: "10"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Connection pool between a gateway and a backend service
Configure a connection pool to control the number of connections between the gateway and a backend service. This prevents the backend from being overloaded during traffic spikes.
Annotation | Description | Default | Required |
| Maximum total TCP connections between the gateway and the backend service. | — | No |
| Maximum TCP connections between the gateway and a single Pod of the backend service. | — | No |
| Maximum HTTP requests per connection between the gateway and the backend service. | — | No |
Example: For demo-service, set a maximum of 10 total connections and 2 connections per Pod.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: 10
mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: 2
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80TLS versions and cipher suites
The default TLS version range for MSE Ingress is TLSv1.0 (earliest) to TLSv1.3 (latest). The default cipher suites are:
ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES128-GCM-SHA256
ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA
AES128-GCM-SHA256
AES128-SHA
ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384
ECDHE-ECDSA-AES256-SHA
ECDHE-RSA-AES256-SHA
AES256-GCM-SHA384
AES256-SHA
Use the following annotations to configure TLS settings per domain name.
Annotation | Description | Default | Required |
| Earliest TLS version accepted. Valid values: |
| No |
| Latest TLS version accepted. |
| No |
| TLS cipher suites, comma-separated. Takes effect only for TLS versions 1.0–1.2. | See default list above | No |
Example: Restrict example.com to TLSv1.2 only.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
mse.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
mse.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
mse.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80Mutual TLS (mTLS) between a gateway and a backend service
By default, MSE Ingress uses HTTP to forward requests to backend containers. When nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" is set, MSE Ingress uses one-way TLS: it authenticates the backend's certificate, which must be issued by a well-known certificate authority (CA).
For mutual TLS (mTLS), both the gateway and the backend service authenticate each other's certificates.
Annotation | Description | Default | Required |
| Client certificate used by the gateway to authenticate to the backend. Format: | — | Yes (for mTLS) |
| Server Name Indication (SNI) used during the TLS handshake. | — | No |
| Enables or disables SNI during the TLS handshake. | — | No |
Example: Configure mTLS between the gateway and demo-service using gateway-cert in the default namespace.
Kubernetes v1.19+:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/gateway-cert"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- backend:
service:
name: demo-service
port:
number: 80
path: /test
pathType: ExactKubernetes earlier than v1.19:
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-ssl-secret: "default/gateway-cert"
name: demo
spec:
ingressClassName: mse
rules:
- host: example.com
http:
paths:
- path: /test
backend:
serviceName: demo-service
servicePort: 80