Container Service for Kubernetes:Troubleshoot Nginx Ingress issues

Contents

Background information

Container Service for Kubernetes (ACK) provides the NGINX Ingress controller optimized based on the open-source version, with full compatibility and support for all community annotations.

Ingresses can work as normal only if you deploy an NGINX Ingress controller in the cluster to parse the routing rules of the Ingresses. After the NGINX Ingress controller receives a request that matches a routing rule, the NGINX Ingress controller routes the request to a corresponding backend Service. The backend Service then forwards the request to pods. In a Kubernetes cluster, Services, Ingresses, and the NGINX Ingress controller work in the following process:

• A Service is an abstraction of a backend application that runs on a set of replicated pods.

• An Ingress contains reverse proxy rules. It controls to which Service pods HTTP or HTTPS requests are routed. For example, requests are routed to different Service pods based on the hosts and URL paths in the requests.

• The NGINX Ingress controller is a reverse proxy program that parses Ingress rules. If changes are made to the Ingress rules, the NGINX Ingress controller updates the Ingress rules accordingly. After the NGINX Ingress controller receives a request, it redirects the request to Service pods based on the Ingress rules.

The NGINX Ingress controller acquires Ingress rule changes from the API server and dynamically generates configuration files, such as nginx.conf. These configuration files are required by a load balancer, such as NGINX. Then, the NGINX Ingress controller reloads the load balancer. For example, the NGINX Ingress controller runs the nginx -s reload command to reload NGINX and generates new Ingress rules.

Diagnostic process

1. Follow these steps to check whether the issue is caused by the Ingress and to ensure that the Ingress Controller is configured correctly.

   1. In the Controller pod, verify that access works as expected. For more information, see Manually access the Ingress and backend pods from the Controller pod.

   2. Confirm that the Nginx Ingress Controller is used correctly. For more information, see the Nginx Ingress community documentation.

2. Use the Ingress diagnostics feature to check the Ingress and component configurations, and apply the recommended changes. For more information about the Ingress diagnostics feature, see Use the Ingress diagnostics feature.

3. Follow the Troubleshooting ideas to identify and resolve related issues.

4. If the issue persists, perform the following checks:

   • For HTTPS certificate issues:

     1. Check whether WAF in transparent proxy mode is enabled for the domain name.

        • If it is enabled, confirm that no TLS certificate is set for WAF or WAF in transparent proxy mode.

        • If it is not enabled, proceed to the next step.

     2. Check whether the SLB instance uses a Layer 7 listener.

        • If it does, confirm that no TLS certificate is set for the Layer 7 listener of the SLB instance.

        • If it does not, proceed to the next step.

   • For non-HTTPS certificate issues, check for error logs in the Controller pod. For more information, see Check for error logs in the Controller pod.

5. If the issue persists, capture packets in the Controller pod and the corresponding backend application pods to identify the problem. For more information, see Capture packets.

Troubleshooting ideas

Common inspection methods

$remote_addr - [$remote_addr] - $remote_user [$time_local]
    "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
    $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]

kubectl logs <controller pod name> -n <namespace> | less

42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:30 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 46b79dkahflhakjhdhfkah**** 47.11.**.**[]
42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:31 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 fadgrerthflhakjhdhfkah**** 47.11.**.**[]

Capture packets

When you cannot locate the problem, you need to capture packets for further diagnosis.

1. Based on the initial problem location, determine whether the network issue is in the Ingress pod or the application pod. If you do not have enough information, you can capture packets from both.

2. Log on to the node where the problematic application pod or Ingress pod is located.

3. On the ECS instance (not inside a container), run the following command to capture Ingress traffic information to a file.

   tcpdump -i any host <application pod IP or Ingress pod IP> -C 20 -W 200 -w /tmp/ingress.pcap

4. Observe the logs. When the expected error occurs, stop capturing packets.

5. Use the error messages in the application logs to locate the corresponding messages at the exact time of the error.

   Note

   • Under normal circumstances, packet capturing does not affect applications. It only slightly increases CPU load and disk writes.

   • The preceding command rotates the captured packets. It can write up to 200 .pcap files, each 20 MB in size.

Cannot access the external address of a cluster LoadBalancer from within the cluster

Symptom

In a cluster, some pods on certain nodes cannot access backend pods through the external address (SLB instance IP address) of the Nginx Ingress Controller, while others can.

Cause

This issue is caused by the externalTrafficPolicy configuration of the Controller's Service. This configuration determines how external traffic is handled. When set to local, only backend pods on the same node as the Controller's pod can receive requests. When set to cluster, access is successful. Requests from resources in the cluster that use the external LoadBalancer address to access the Service are also treated as external traffic.

Solution

• (Recommended) Access the service within the Kubernetes cluster using its ClusterIP or the Ingress service name. The Ingress service name is nginx-ingress-lb.kube-system.

• Run the kubectl edit svc nginx-ingress-lb -n kube-system command to modify the Ingress service. Set externalTrafficPolicy to Cluster in the LoadBalancer service. If the cluster's container network plugin is Flannel, the client's source IP address is lost. If you use Terway, the source IP address can be preserved.

• Example:

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni   # Direct ENI access.
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

  For more information about Service annotations, see Configure a Classic Load Balancer (CLB) instance using annotations.

Cannot access the Ingress Controller itself

Symptom

For a Flannel cluster, when you access the Ingress from within its own pod using a domain name, SLB IP address, or ClusterIP, some or all requests fail.

Cause

The default configuration of Flannel does not allow loopback access.

Solution

• (Recommended) If possible, recreate the cluster and use the Terway network plug-in. Migrate the existing cluster's applications to the Terway-mode cluster.

• If you cannot recreate the cluster, you can resolve the issue by modifying the Flannel configuration to enable hairpinMode. After modifying the configuration, recreate the Flannel pod for the changes to take effect.

  1. Run the following command to edit Flannel.

     kubectl edit cm kube-flannel-cfg -n kube-system

  2. In the returned cni-conf.json, add "hairpinMode": true to the delegate section.

     Example:

     cni-conf.json: |
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "type": "flannel",
           "delegate": {
             "isDefaultGateway": true,
             "hairpinMode": true
           }
         }

  3. Run the following command to delete and recreate Flannel.

     kubectl delete pod -n kube-system -l app=flannel   

A TLS certificate is added or modified in the cluster, but the default or old certificate is still used for access

Symptom

After you add or modify a Secret in the cluster and specify the secretName in the Ingress, accessing the Ingress still uses the default certificate (Kubernetes Ingress Controller Fake Certificate) or the old certificate.

Cause

• The certificate is not returned by the Ingress Controller within the cluster.

• The certificate is invalid and was not loaded correctly by the Controller.

• The Ingress Controller returns the corresponding certificate based on Server Name Indication (SNI), but the SNI may not have been included during the TLS handshake.

Solution

• Use one of the following methods to confirm whether the SNI field was included when establishing the TLS connection:

  • Use a newer version of a browser that supports SNI.

  • When using the openssl s_client command to test the certificate, include the -servername parameter.

  • When using the curl command, add a hosts entry or use the --resolve parameter to map the domain name, instead of using the IP address and Host request header method.

• Confirm that no TLS certificate is set for WAF, WAF in transparent proxy mode, or the SLB Layer 7 listener. The TLS certificate should be returned by the Ingress Controller within the cluster.

• Perform an Ingress diagnosis in the ACK console to check for configuration errors and error logs. For more information, see Use the Ingress diagnostics feature.

• Run the following command to manually view the Ingress pod error logs and apply fixes based on the logs.

  kubectl logs <ingress pod name> -n <pod namespace> | grep -E ^[EW]

Cannot connect to a gRPC service exposed by an Ingress

Symptom

Cannot access a gRPC service behind an Ingress.

Cause

• The annotation specifying the backend protocol type is not set in the Ingress resource.

• The gRPC service can only be accessed through a TLS port.

Solution

• Set the annotation in the corresponding Ingress resource: nginx.ingress.kubernetes.io/backend-protocol:"GRPC".

• Confirm that the client is using the TLS port and encrypting the traffic when sending requests.

Cannot connect to a backend HTTPS service

Symptom

• Cannot access a backend HTTPS service through an Ingress.

• The response may be a 400 error with the message The plain HTTP request was sent to HTTPS port.

Cause

The request from the Ingress Controller to the backend pod uses the default HTTP protocol.

Solution

Set the annotation in the Ingress resource: nginx.ingress.kubernetes.io/backend-protocol:"HTTPS".

Cannot preserve the originating IP address in an Ingress pod

Symptom

The client's real IP address cannot be preserved in the Ingress pod. It appears as the node IP address, an address in the 100.XX.XX.XX range, or another address.

Cause

• externalTrafficPolicy in the Service used by the Ingress is set to Cluster.

• A Layer 7 proxy is used on the SLB instance.

• WAF in transparent proxy mode is used.

Solution

• For cases where externalTrafficPolicy is set to Cluster and a Layer 4 SLB instance is used at the frontend.

  You can change externalTrafficPolicy to Local. However, this may cause attempts to access the Ingress using the SLB IP address from within the cluster to fail. For a solution, see Cannot access the external address of a cluster LoadBalancer from within the cluster.

• For cases where a Layer 7 proxy (Layer 7 SLB, WAF, or WAF in transparent proxy mode) is used, follow these steps:

  1. Ensure that a Layer 7 proxy is used and the X-Forwarded-For request header is enabled.

  2. In the Ingress Controller's ConfigMap (default is nginx-configuration in the kube-system namespace), add enable-real-ip: "true".

  3. Observe the logs to verify if the source IP address can be obtained.

• For long link paths with multiple forwards (for example, an additional reverse proxy service is configured in front of the Ingress Controller), you can observe the value of remote_addr in the logs after enabling enable-real-ip to determine if the real IP address is passed to the Ingress container through the X-Forwarded-For request header. If not, ensure that the client's real IP address is carried using methods such as X-Forwarded-For before the request reaches the Ingress Controller.

Phased release rules do not take effect

Symptom

Phased release is set up in the cluster, but the phased release rules do not take effect.

Cause

There are two possible reasons:

• When using canary-* related annotations, nginx.ingress.kubernetes.io/canary: "true" is not set.

• When using canary-* related annotations with Nginx Ingress Controller versions before 0.47.0, you must specify your application domain name in the Host field of the Ingress rule. It cannot be empty.

Solution

• Based on the causes, modify nginx.ingress.kubernetes.io/canary: "true" or the Host field in the Ingress rule. For more information, see Routing rules.

• If the preceding situations do not apply, see Traffic is not distributed based on phased release rules or other traffic is routed to the phased release service.

Traffic is not distributed based on phased release rules or other traffic is routed to the phased release service

Symptom

Phased release rules are set, but traffic is not distributed according to the rules, or traffic from other normal Ingresses is routed to the phased release service.

Cause

In the Nginx Ingress Controller, phased release rules are not applied to a single Ingress. Instead, they are applied to all Ingresses that use the same Service.

For more information about this issue, see Ingress with canary annotation will affect all ingresses with same service.

Solution

For Ingresses that require phased release (including those using service-match and canary-* related annotations), create separate services (both a production and a phased release service) that point to the original pods. Then, enable phased release for that Ingress. For more information, see Implement phased releases and blue-green deployments using Nginx Ingress.

The "failed calling webhook" error is reported when you create an Ingress resource

Symptom

When adding an Ingress resource, the message "Internal error occurred: failed calling webhook..." is displayed, as shown in the following figure.

Cause

When an Ingress resource is added, its validity must be verified by a service (by default, ingress-nginx-controller-admission). If there is a problem with the link (for example, the service is deleted or the Ingress controller is deleted), the verification fails, and the Ingress resource cannot be added.

Solution

• Check the webhook link to ensure that all resources exist and are working correctly. The link is ValidatingWebhookConfiguration -> Service -> Pod.

• Confirm that the admission feature of the Ingress Controller pod is enabled and that the pod can be accessed from the outside.

• If the Ingress Controller has been deleted or the webhook feature is not needed, you can directly delete the ValidatingWebhookConfiguration resource.

The SSL_ERROR_RX_RECORD_TOO_LONG error is reported for HTTPS access

Symptom

When accessing via HTTPS, the error SSL_ERROR_RX_RECORD_TOO_LONG or routines:CONNECT_CR_SRVR_HELLO:wrong version number is reported.

Cause

An HTTPS request is accessing a non-HTTPS port, such as an HTTP port.

Solution

Modify the SLB settings or Service settings as needed to ensure that HTTPS can access the correct port.

Common HTTP error codes appear

Symptom

The request returns a non-2xx or non-3xx status code, such as 502, 503, 413, or 499.

Cause and solution

Check the logs to determine if the error was returned by the Ingress Controller. For more information, see View access logs in the Controller pod using SLS. If so, refer to the following solutions:

• 413 Error

  • Cause: The size of the request data exceeds the configured limit.

  • Solution: Run kubectl edit cm -n kube-system nginx-configuration to modify the Controller configuration. Adjust the values of nginx.ingress.kubernetes.io/client-max-body-size and nginx.ingress.kubernetes.io/proxy-body-size as needed. The default value is 20m.

• 499 Error

  • Cause: The client disconnected before the server could send a response. This is not necessarily an issue with the component or the backend application.

  • Solution:

    • A small number of 499 errors may be normal depending on the application and can be ignored.

    • If many 499 errors occur, check whether the backend application processing time and the frontend request timeout are as expected.

• 502 error

  • Cause: The Nginx Ingress is functioning correctly, but the Nginx Ingress pod cannot connect to the target backend pod.

  • Solution:

    • Trigger conditions:

      • It may be due to incorrect configuration in the backend Service and pod. Check the port configuration of the backend Service and the application code in the container.

    • Intermittent issues

      • It may be due to high load on the Nginx Ingress Controller's pod. You can assess the load by checking the number of requests and connections on the Controller's SLB instance and add more resources to the Controller if necessary. For more information, see "Deploy a highly available Nginx Ingress Controller".

      • It may be because the backend pod actively closes the session. The Nginx Ingress Controller enables persistent connections by default. Ensure that the idle timeout for persistent connections on the backend is greater than the Controller's idle timeout (default is 900 seconds).

    • If none of these methods identify the problem, analyze the captured packets.

• 503 Error

  • Cause: The Ingress Controller cannot find an available backend pod to route the request to.

  • Solution:

    • Uncommon scenarios

      • See the solution for the 502 error.

      • Check the readiness status of the backend application and configure a reasonable health check.

    • If the error is persistent:

      Check whether the backend Service is configured correctly and whether an Endpoint exists.

The net::ERR_HTTP2_SERVER_REFUSED_STREAM error occurs

Symptom

When accessing a webpage, some resources fail to load correctly, and the console shows a net::ERR_HTTP2_SERVER_REFUSED_STREAM or net::ERR_FAILED error.

Cause

The number of parallel resource requests is high, reaching the HTTP/2 maximum concurrent streams limit.

Solution

• (Recommended) In the ConfigMap, increase the value of http2-max-concurrent-streams as needed. The default is 128. For more information, see http2-max-concurrent-streams.

• In the ConfigMap, disable HTTP/2 support by setting use-http2 to false. For more information, see use-http2.

The "The param of ServerGroupName is illegal" error occurs

Cause

The format for generating ServerGroupName is namespace+svcName+port. The server group name must be 2 to 128 characters in length, and must start with a letter or a Chinese character, and can contain digits, periods (.), underscores (_), and hyphens (-).

Solution

Modify the name to comply with the server group naming requirements.

The "certificate signed by unknown authority" error is reported when you create an Ingress

Cause

When you create an Ingress, the error shown in the preceding figure occurs. This is because multiple Ingress controllers are deployed, and they use the same resources (which may include Secrets, services, webhook configurations, etc.). This causes the SSL certificate used during webhook execution to be inconsistent with the backend service, resulting in an error.

Solution

Redeploy the two Ingress controllers and ensure that their resources do not overlap. For information about the resources included in an Ingress, see What updates are made to the system when I upgrade the Nginx Ingress Controller component in ACK component management?.

An Ingress pod restarts because its health check fails

Symptom

The Controller pod fails its health check and restarts.

Cause

• The Ingress pod or its node has a high load, causing the health check to fail.

• The tcp_tw_reuse or tcp_timestamps kernel parameter is set on the cluster node, which may cause the health check to fail.

Solution

• Scale out the Ingress pod and observe if the issue persists. For more information, see Highly available deployment of Nginx Ingress Controller.

• Disable tcp_tw_reuse or set it to 2, and also disable tcp_timestamps. Then, observe if the issue persists.

Add TCP and UDP services

1. In the corresponding ConfigMap (by default, tcp-services and udp-services in the kube-system namespace), add the corresponding entry.

   For example, to map port 8080 of example-go in the default namespace to port 9000, see the following example.

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tcp-services
     namespace: ingress-nginx
   data:
     9000: "default/example-go:8080"  # Map port 8080 to port 9000.

2. In the Ingress deployment (by default, nginx-ingress-controller in the kube-system namespace), add the mapped port.

3. In the Service corresponding to the Ingress, add the mapped port.

   Click to view sample code

   apiVersion: v1
   kind: Service
   metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   spec:
     type: LoadBalancer
     ports:
       - name: http
         port: 80
         targetPort: 80
         protocol: TCP
       - name: https
         port: 443
         targetPort: 443
         protocol: TCP
       - name: proxied-tcp-9000
         port: 9000
         targetPort: 9000
         protocol: TCP
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx

   For more information about adding TCP and UDP services, see Exposing TCP and UDP services.

Ingress rules do not take effect

Symptom

Ingress rules are added or modified but do not take effect.

Cause

• An error in the Ingress configuration prevents new Ingress rules from being loaded correctly.

• The Ingress resource is configured incorrectly and does not match the expected configuration.

• There is a permission issue with the Ingress Controller, preventing it from monitoring Ingress resource changes correctly.

• An old Ingress uses a server-alias configuration for a domain name that conflicts with the new Ingress, causing the rule to be ignored.

Some resources fail to load or a blank screen appears after rewriting to the root directory

Symptom

After rewriting access using the Ingress rewrite-target annotation, some page resources fail to load, or a blank screen appears.

Cause

• rewrite-target may not be configured with a regular expression.

• The resource request path in the application is hardcoded to the root directory.

Solution

• Check if rewrite-target is used with a regular expression and a capturing group. For more information, see Rewrite.

• Check if the frontend request is accessing the correct path.

How to fix abnormal log parsing in SLS after an upgrade

Symptom

The ingress-nginx-controller component currently has two main versions: 0.20 and 0.30. After upgrading from version 0.20 to 0.30 through the Components page in the console, the Ingress Dashboard may not correctly display the actual backend service access status when using the Ingress phased release or blue-green deployment features.

Cause

Because the default output formats of versions 0.20 and 0.30 are different, the Ingress Dashboard may not correctly display the actual backend service access status when using the Ingress phased release or blue-green deployment features.

Solution

Perform the following steps to fix the issue by updating the nginx-configuration configmap and k8s-nginx-ingress configurations.

1. Update the nginx-configuration configmap.

   • If you have not modified the nginx-configuration configmap, save the following content as nginx-configuration.yaml and then run the kubectl apply -f nginx-configuration.yaml command to deploy it.

     apiVersion: v1
     kind: ConfigMap
     data:
       allow-backend-server-header: "true"
       enable-underscores-in-headers: "true"
       generate-request-id: "true"
       ignore-invalid-headers: "true"
       log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]
       max-worker-connections: "65536"
       proxy-body-size: 20m
       proxy-connect-timeout: "10"
       reuse-port: "true"
       server-tokens: "false"
       ssl-redirect: "false"
       upstream-keepalive-timeout: "900"
       worker-cpu-affinity: auto
     metadata:
       labels:
         app: ingress-nginx
       name: nginx-configuration
       namespace: kube-system

   • If you have modified the nginx-configuration configmap, run the following command to fix it to avoid overwriting your configuration:

     kubectl edit configmap nginx-configuration -n kube-system

   At the end of the log-format-upstream field, add [$proxy_alternative_upstream_name], then save and exit.

2. Update the k8s-nginx-ingress configuration.

   Save the following content as a k8s-nginx-ingress.yaml file, and then run the kubectl apply -f k8s-nginx-ingress.yaml command to deploy it.

   Click to view YAML content

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     namespace: kube-system
     # your config name, must be unique in you k8s cluster
     name: k8s-nginx-ingress
   spec:
     # logstore name to upload log
     logstore: nginx-ingress
     # product code, only for k8s nginx ingress
     productCode: k8s-nginx-ingress
     # logtail config detail
     logtailConfig:
       inputType: plugin
       # logtail config name, should be same with [metadata.name]
       configName: k8s-nginx-ingress
       inputDetail:
         plugin:
           inputs:
           - type: service_docker_stdout
             detail:
               IncludeLabel:
                 io.kubernetes.container.name: nginx-ingress-controller
               Stderr: false
               Stdout: true
           processors:
           - type: processor_regex
             detail:
               KeepSource: false
               Keys:
               - client_ip
               - x_forward_for
               - remote_user
               - time
               - method
               - url
               - version
               - status
               - body_bytes_sent
               - http_referer
               - http_user_agent
               - request_length
               - request_time
               - proxy_upstream_name
               - upstream_addr
               - upstream_response_length
               - upstream_response_time
               - upstream_status
               - req_id
               - host
               - proxy_alternative_upstream_name
               NoKeyError: true
               NoMatchError: true
               Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)\s(\d+)\s"([^"]*)"\s"([^"]*)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S*)\s*\[*([^]]*)\]*.*
               SourceKey: content

The "cannot list/get/update resource" error occurs

Symptom

You can find Controller error logs in the pod using the method described in Check for error logs in the Controller pod. The logs are similar to the following:

User "system:serviceaccount:kube-system:ingress-nginx" cannot list/get/update resource "xxx" in API group "xxx" at the cluster scope/ in the namespace "kube-system"

Cause

The Nginx Ingress Controller lacks the necessary permissions to update the relevant resources.

Solution

• Based on the log, determine whether the issue is caused by a ClusterRole or a Role.

  • If the log contains at the cluster scope, the issue is with the ClusterRole (ingress-nginx).

  • If the log contains in the namespace "kube-system", the issue is with the Role (kube-system/ingress-nginx).

• Confirm that the corresponding permissions and permission bindings are complete.

  • For a ClusterRole:

    • Ensure that the ClusterRole ingress-nginx and ClusterRoleBinding ingress-nginx exist. If they do not exist, you can create them, restore them, or uninstall and reinstall the component.

    • Ensure that the ClusterRole ingress-nginx includes the permission corresponding to the log (in the example, the List permission for networking.k8s.io/ingresses). If the permission does not exist, you can manually add it to the ClusterRole.

  • For a Role:

    • Confirm that the Role kube-system/ingress-nginx and RoleBinding kube-system/ingress-nginx exist. If they do not exist, you can create them, restore them, or uninstall and reinstall the component.

    • Confirm that the Role ingress-nginx includes the permission corresponding to the log (in the example, the Update permission for the ConfigMap ingress-controller-leader-nginx). If the permission does not exist, you can manually add it to the Role.

The "configuration file failed" error occurs

Symptom

You can find Controller error logs in the pod using the method described in Check for error logs in the Controller pod. The logs are similar to the following:

requeuing……nginx: configuration file xxx test failed (multiple lines)

Cause

A configuration error caused the Nginx configuration reload to fail. This is usually caused by a syntax error in a snippet inserted into an Ingress rule or ConfigMap.

Solution

• Check the error message in the logs (warning level messages can be ignored) to roughly locate the problem. If the error message is not clear, you can use the file line number from the error message to view the corresponding file in the pod. In the following example, the file is /tmp/nginx/nginx-cfg2825306115 and the line is 449.

  Run the following command to check for errors in the configuration near the corresponding line.

  # Run the command in the pod.
  kubectl exec -n <namespace> <controller pod name> -it -- bash
  # View the error file with line numbers and check for errors in the configuration near the corresponding line.
  cat -n /tmp/nginx/nginx-cfg2825306115

• Based on the error message and the configuration file, locate the cause of the error and fix it according to your actual configuration.

The "Unexpected error validating SSL certificate" error occurs

Symptom

You can find Controller error logs in the pod using the method described in Check for error logs in the Controller pod. The logs are similar to the following:

Unexpected error validating SSL certificate "xxx" for server "xxx"

Cause

The certificate is configured incorrectly. A common reason is that the domain name included in the certificate does not match the domain name configured in the Ingress. Some Warning-level logs, such as a certificate not having a SAN extension, do not affect the normal use of the certificate. Determine if there is a problem based on the actual situation.

Solution

Check the certificate issue in the cluster based on the error message.

• Are the format and content of the certificate's cert and key correct?

• Does the domain name included in the certificate match the domain name configured in the Ingress?

• Has the certificate expired?

Many configuration files are not cleared from the Controller

Troubleshoot the issue where a pod remains in the Pending state after a Controller upgrade

kubectl -n kube-system describe pod <pending-pod-name>

kubectl -n kube-system get events

TCP stream confusion occurs when multiple CLB instances are used for Nginx Ingress in a high-concurrency Flannel CNI and IPVS cluster