What are the best practices for the NGINX Ingress controller? - Container Service for Kubernetes

This topic describes best practices for configuring the NGINX Ingress controller for optimal performance in various scenarios.

In this topic

Improve the performance and stability of the NGINX Ingress controller
Improve the observability of the NGINX Ingress controller
Advanced features of the NGINX Ingress controller
References

NGINX Ingress controller performance and stability

Appropriate replicas and resource limits

By default, an NGINX Ingress controller installed during cluster creation or from the Add-ons page has two replicas. You can adjust this number based on your business requirements. Ensure that the NGINX Ingress controller pods are distributed across different nodes to prevent resource contention and single points of failure. You can also assign exclusive nodes to guarantee performance and stability. For more information, see Use exclusive nodes to improve NGINX Ingress performance and stability. To avoid traffic interruptions from out-of-memory (OOM) events, do not set resource limits for the NGINX Ingress controller. If you must set limits, set the CPU limit to at least 1,000 millicores and the memory limit to at least 2 GiB.

Use exclusive nodes

For high stability requirements, assign exclusive nodes to the Ingress controller to prevent resource contention. For more information, see Deploy a highly available NGINX Ingress controller.
For high-load scenarios, you can also configure the Ingress controller to support high-traffic applications. For more information, see Configure the NGINX Ingress controller for high-load scenarios.

Optimize NGINX Ingress performance

Performance tuning for the NGINX Ingress controller involves two main areas: system parameter tuning and NGINX parameter tuning.

System parameter tuning: Alibaba Cloud operating systems include default optimizations for common parameters. Other system parameters you may need to tune include the maximum system backlog and the range of available ports. Tuning system parameters helps ensure that NGINX can handle high concurrency and prevents connection failures to backends due to port exhaustion.
NGINX parameter tuning:
- Adjusting the maximum number of connections per worker process ensures the NGINX Ingress controller can handle high concurrency.
- Increase the connection timeout: Unlike the default NGINX behavior, the NGINX Ingress controller uses keep-alive connections to send requests to backend pods. Therefore, increase the connection timeout to allow a single connection to handle more requests, reducing the overhead of creating new connections.
- Set the keep-alive timeout: Set the keep-alive timeout for backend services to be equal to or greater than the connection timeout of the NGINX Ingress controller, which defaults to 900s in ACK clusters.

The NGINX Ingress component includes default tuning parameters that deliver optimal performance in most scenarios. If you have specific requirements, you can further optimize system and NGINX parameters by using the relevant fields in the ConfigMap. For more information about ConfigMaps, see ConfigMaps.

Auto scaling with HPA

In most cases, the NGINX Ingress controller can handle sudden traffic spikes. If the default capacity is insufficient for your high-load scenarios, configure a Horizontal Pod Autoscaler (HPA) to automatically scale the NGINX Ingress controller. For more information, see Use Horizontal Pod Autoscaler (HPA).

Important

Pod scaling events may temporarily interrupt existing connections. Configure this feature with caution.

The following is a sample YAML configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-ingress-controller-hpa
  namespace: kube-system
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-ingress-controller
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

Configure a preStop hook

During a rolling update of a backend service, the NGINX Ingress controller removes the endpoint of a terminating pod while maintaining connections for in-flight requests. If a backend service pod exits immediately after receiving a termination signal, in-flight requests may fail. Due to timing issues, some new traffic might still be routed to the terminated pod, resulting in traffic loss.

To prevent traffic loss during rolling updates, configure a preStop hook in your backend service pods. This hook allows the pod to continue processing in-flight requests after its endpoint is removed, preventing service interruptions.

Add the following code to the container configuration in the pod template:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
      - name: app
        lifecycle:
          # Configure a preStop hook to wait for 30 seconds before exiting.
          # The sleep command must be available in the container.
          preStop:
            exec:
              command:
              - sleep
              - "30"

NGINX Ingress controller observability

Log Service and Managed Service for Prometheus

The NGINX Ingress controller provides monitoring dashboards based on Log Service (SLS) logs and Prometheus to help you better understand your service traffic.

Log Service (SLS) logs:
- If you select Enable Log Service and Create Ingress Dashboard when you create a cluster, you can go to the ACK console, choose Network > Ingresses, and then view the Log Service-based dashboard on the NGINX Ingress Overview page. You can also go to Operations > Log Center to view the logs generated by the NGINX Ingress controller. For more information, see Collect and analyze NGINX Ingress access logs.
- If you did not enable these options during cluster creation, you can manually configure the log collection component and rules. For more information, see Collect and analyze NGINX Ingress access logs. For more information about monitoring, see Ingress dashboard monitoring.
Managed Service for Prometheus monitoring: You can choose to install Managed Service for Prometheus monitoring when you create a cluster. Alternatively, after creating the cluster, you can install or view it by choosing Operations > Prometheus Monitoring. For more information, see Connect and configure Managed Service for Prometheus.

Note
When using Managed Service for Prometheus, add the host field to the Ingress resources in the cluster. Otherwise, some metrics for the Ingress will be missing by default. Alternatively, add the --metrics-per-host=false startup argument to the controller in the NGINX Ingress controller Deployment.

Advanced features of the NGINX Ingress controller

Use multiple NGINX Ingress controllers

You may need to deploy multiple NGINX Ingress controllers in a cluster, for example, to isolate internal and external traffic. For more information, see Deploy multiple Ingress controllers for traffic isolation.

Intra-cluster access

In a cluster, traffic to the external endpoint of a LoadBalancer Service (the public IP address of the NGINX Ingress controller) is typically intercepted and forwarded by iptables or IPVS. When externalTrafficPolicy is set to Local and no NGINX Ingress pod is running on a node, network connectivity issues occur. By default, the NGINX Ingress controller in an ACK cluster uses a LoadBalancer Service in Local mode. Consequently, you may experience connectivity issues when accessing the associated Classic Load Balancer (CLB) address from within the cluster. Therefore, for intra-cluster access, use the Service's ClusterIP or the internal domain name (nginx-ingress-lb.kube-system). Also, avoid scenarios where the NGINX Ingress controller attempts to access itself, which can lead to network failures due to hairpinning issues. For more information about how to resolve this issue, see Cannot access the SLB instance of a LoadBalancer Service from within a Kubernetes cluster.

Use WAF

To block malicious requests, you can enable Web Application Firewall (WAF) for the Classic Load Balancer (CLB) used by your cluster's NGINX Ingress controller. When you enable WAF on an HTTPS port, you must also configure the required certificate in the WAF console. This can lead to the following issues:

TLS requests are terminated at the WAF layer. Therefore, certificates configured in a Kubernetes Secret will not be used for the public endpoint.
When you access port 443 from within the cluster by using the CLB IP address or the Service's ClusterIP, the traffic may not pass through WAF, which causes certificate errors.
With WAF enabled, the NGINX Ingress controller cannot obtain the real client IP address by default. Add the following content to the ConfigMap (by default, nginx-configuration in the kube-system namespace for NGINX Ingress controllers installed through component management) to enable NGINX's Realip module and use the X-Forwarded-For header to obtain the real client IP address.
```
use-forwarded-headers: "true" # Use this option for NGINX Ingress controller version 0.30.0 and earlier.
enable-real-ip: "true" # Use this option for NGINX Ingress controller version 0.44.0 and later.
proxy-real-ip-cidr: <The back-to-origin CIDR block that you obtain from WAF>
```

Blue-green deployments or canary releases

You can implement a canary release with the NGINX Ingress controller by using the feature in the ACK console or by manually adding an annotation. For more information, see Use NGINX Ingress to implement canary releases and blue-green deployments.

Important

The original and canary Services should only be referenced by the canary Ingress. Otherwise, canary rule conflicts may occur and lead to traffic routing errors.

Proxy non-HTTP requests

By default, the NGINX Ingress controller connects to backend Services by using the HTTP protocol. However, it also supports multiple backend protocols, with WebSocket, HTTPS, and gRPC being the most common. For a complete list of supported backend protocols, see Backend Protocol.

WebSocket: The NGINX Ingress controller provides native support for WebSocket. No additional configuration is required to forward WebSocket connections. If you have long-lived WebSocket connections, you can use an annotation to adjust the backend connection timeout to prevent timeout-related disconnections. For more information about how to make adjustments, see Custom timeouts.
HTTPS: For backend Services that use HTTPS, you can add the nginx.ingress.kubernetes.io/backend-protocol:"HTTPS" annotation to the Ingress to switch to HTTPS connections.
gRPC: gRPC can be accessed only over TLS ports. Therefore, ensure that you use an encrypted TLS port when your application accesses gRPC services through the NGINX Ingress controller. For more information about how to configure gRPC, see Configure a gRPC service for NGINX Ingress.