Microservices Engine:Annotations supported by MSE Ingress gateways

Scope

Each annotation applies at one of three levels:

Prefixes

MSE Ingress gateways accept both nginx.ingress.kubernetes.io/ and mse.ingress.kubernetes.io/ prefixes for compatible annotations. Use either prefix based on your preference.

MSE-exclusive annotations support only the mse.ingress.kubernetes.io/ prefix.

Annotation value formats

Kubernetes annotations are always strings. The following encoding conventions apply:

Precedence and interaction rules

Compatibility notes

• NGINX variables: MSE implements features differently from NGINX. NGINX variables configured in annotations or code snippets are incompatible with MSE.

• proxy-body-size behavior: MSE cloud-native gateways use chunked transfer encoding, which automatically breaks large request bodies into chunks. To handle large file transfers, adjust the DownstreamConnectionBufferLimits parameter in Parameter Settings for your MSE gateway.

Canary release

# Canary release: route traffic based on a header value
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-by-header: "x-canary"
    nginx.ingress.kubernetes.io/canary-by-header-value: "true"
spec:
  rules:
    - host: example.com
      http:
        paths:
          - path: /api
            pathType: Prefix
            backend:
              service:
                name: my-service-canary
                port:
                  number: 80

Multi-service routing

Route traffic to multiple backend services with weight-based distribution.

Syntax: {weight}% {serviceName}.{serviceNamespace}.svc.cluster.local:{port}

annotations:
  # Route 60% of traffic to foo and 40% to bar
  mse.ingress.kubernetes.io/destination: |
    60% foo.default.svc.cluster.local:8080
    40% bar.default.svc.cluster.local:9090

After you configure this annotation, all routing rules on the Ingress are directed to the specified services. If the syntax is invalid, the annotation is ignored and the original routing rules remain unchanged.

Service subset

Route traffic to a subset of pods within a service, useful when one service manages multiple deployments.

How service-subset routes traffic:

• Without subset-labels configured:

  • Set to "" or base: Routes to pods labeled opensergo.io/canary: "" or pods without the opensergo.io/canary label key.

  • Set to any other value (e.g., gray): Routes to pods labeled opensergo.io/canary-gray: gray.

• With subset-labels configured: Routes only to pods whose labels match the Key:Value pairs defined in subset-labels.

If no pods match the specified label, traffic is automatically routed to all pods in the service.

annotations:
  # Route traffic to pods labeled as "gray" subset
  mse.ingress.kubernetes.io/service-subset: "gray"

Fallback

Regex match

Rewrite

Redirect

Some NGINX Ingress versions support NGINX variables for redirection, but this is not documented in the official Annotations reference and may cause incompatibility issues. Avoid using NGINX variables for redirection.

CORS

Header control

Manipulate request and response headers at the gateway level. Header control annotations for base routes and canary routes are independent -- configure different header policies for each.

Request headers:

Response headers:

Syntax for add and update annotations:

• Single header: key value

• Multiple headers: Use a YAML pipe (|), one key-value pair per line.

Syntax for remove annotations:

• Single header: key

• Multiple headers: Separate with commas (,).

annotations:
  # Add a custom header to requests
  mse.ingress.kubernetes.io/request-header-control-add: "x-request-id 12345"
  # Add multiple headers to responses
  mse.ingress.kubernetes.io/response-header-control-add: |
    x-custom-header value1
    x-another-header value2
  # Remove headers from responses
  mse.ingress.kubernetes.io/response-header-control-remove: "x-powered-by,server"

Timeout

This timeout applies at the application layer, not at the TCP transport layer.

annotations:
  mse.ingress.kubernetes.io/timeout: "30"

Retry

Traffic mirroring

mirror-target-service parameters:

annotations:
  # Mirror 50% of traffic to a shadow service
  mse.ingress.kubernetes.io/mirror-target-service: "default/shadow-service:8080"
  mse.ingress.kubernetes.io/mirror-percentage: "50"

Domain alias

Rate limiting

annotations:
  # Limit to 100 RPS with a custom JSON response
  mse.ingress.kubernetes.io/rate-limit: "100"
  mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-code: "429"
  mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body-type: "JSON"
  mse.ingress.kubernetes.io/rate-limit-fallback-custom-response-body: '{"error":"rate limit exceeded"}'

Concurrency control

Global concurrency control limits the number of requests processed simultaneously on a route.

annotations:
  # Limit to 50 concurrent requests with a redirect on overflow
  mse.ingress.kubernetes.io/concurrency-limit: "50"
  mse.ingress.kubernetes.io/concurrency-limit-fallback-redirect-url: "https://example.com/busy"

Backend protocol

Load balancing

Supported consistent hashing methods:

• NGINX variables: $request_uri (request path including parameters), $host (request host), $remote_addr (client IP address)

• Request headers: $http_headerName

• Request path parameters: $arg_varName

Graceful start (service prefetching)

annotations:
  # Warm up new pods over 60 seconds
  mse.ingress.kubernetes.io/warmup: "60"

Cookie affinity

IP access control

Precedence rules:

• Route-level IP lists take precedence over domain-level IP lists.

• nginx.ingress.kubernetes.io/denylist-source-range takes precedence over mse.ingress.kubernetes.io/blacklist-source-range.

annotations:
  # Allow only specific IPs at the route level
  nginx.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8,192.168.1.100"

Gateway-to-backend connection pool

annotations:
  # Limit connection pool to backend
  mse.ingress.kubernetes.io/connection-policy-tcp-max-connection: "1024"
  mse.ingress.kubernetes.io/connection-policy-tcp-max-connection-per-endpoint: "100"
  mse.ingress.kubernetes.io/connection-policy-http-max-request-per-connection: "500"

Client-to-gateway TLS

Default cipher suites:

ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-SHA, ECDHE-RSA-AES128-SHA, AES128-GCM-SHA256, AES128-SHA, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-AES256-SHA, ECDHE-RSA-AES256-SHA, AES256-GCM-SHA384, AES256-SHA

annotations:
  # Enforce TLS 1.2 minimum
  mse.ingress.kubernetes.io/tls-min-protocol-version: "TLSv1.2"
  mse.ingress.kubernetes.io/tls-max-protocol-version: "TLSv1.2"

Gateway-to-backend TLS

Authentication

annotations:
  # Basic authentication with a Kubernetes secret
  nginx.ingress.kubernetes.io/auth-type: "basic"
  nginx.ingress.kubernetes.io/auth-secret: "default/my-auth-secret"
  nginx.ingress.kubernetes.io/auth-secret-type: "auth-file"
  nginx.ingress.kubernetes.io/auth-realm: "Protected Area"