NGINX Ingress controller troubleshooting

Table of contents

Background information

Container Service for Kubernetes (ACK) provides the NGINX Ingress controller that is optimized based on the open source version. You can install the NGINX Ingress controller when you create an ACK cluster.

The NGINX Ingress controller acquires Ingress rule changes from the API server and dynamically generates configuration files, such as nginx.conf. These configuration files are required by a load balancer, such as NGINX. Then, the NGINX Ingress controller reloads the load balancer. For example, the NGINX Ingress controller runs the nginx -s load command to reload NGINX and generates new Ingress rules.

Diagnostic procedure

1. You can perform the following steps to check whether an issue is caused by the Ingress. Make sure that the configuration of the Ingress controller is valid.
   1. Check whether you can access a specific pod from the controller pod. For more information, see Manually access the Ingress and backend pod by using the Ingress controller pod.
   2. Check whether the NGINX Ingress controller is properly configured. For more information, see Documentation for the NGINX Ingress controller.
2. Use the Ingress diagnostics feature to check the configurations of the Ingress and components. Then, modify the configurations based on the prompts. For more information, see Use the Ingress diagnostics feature.
3. Locate the cause of the issue and refer to the relevant solution based on Troubleshooting.
4. If the issue persists, perform the following steps:
   • Issues that are related to TLS certificates:
     1. Check whether the domain name is added to Web Application Firewall (WAF) in CNAME record mode or transparent proxy mode.
        • If the domain name is added to WAF, check whether the domain name has a TLS certificate.
        • If the domain name is not added to WAF, proceed to the next step.
     2. Check whether a Layer 7 listener is added for the Server Load Balancer (SLB) instance.
        • If a Layer 7 listener is added for the SLB instance, check whether a TLS certificate is associated with the listener.
        • If no Layer 7 listener is added for the SLB instance, proceed to the next step.
   • If the issue is not related to TLS certificates, diagnose the error log of the controller pod. For more information, see Diagnose the error log of the controller pod.
5. If the issue persists, locate the cause of the issue by capturing packets in the controller pod and the backend pod. For more information, see Capture packets.
6. If the issue persists, Submit a ticket for troubleshooting.

Troubleshooting

Commonly used diagnostic methods

1. Log on to the ACK console.
2. In the left-side navigation pane of the ACK console, click Clusters.
3. On the Clusters page, find the cluster on which you want to perform a check and choose More > Cluster Check in the Actions column.
4. In the left-side navigation pane of the Container Intelligence Service page, choose Cluster Check > Diagnosis.
5. On the Diagnosis page, click Ingress Diagnosis.
6. In the Ingress Diagnosis panel, enter the URL that cannot be accessed, such as https://www.example.com. Select I know and agree and then click Create diagnosis.
   After the diagnostic is completed, you can view the diagnostic result and try to fix the issue.

You can check the access log format of the NGINX Ingress controller in the nginx-configuration ConfigMap in the kube-system namespace.

$remote_addr - [$remote_addr] - $remote_user [$time_local]
    "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length
    $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length
    $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]

The following figure shows the page on which the access log of the NGINX Ingress controller is displayed in the Log Service console. For more information, see Step 4: View log data.

kubectl logs <controller pod name> -n <namespace> | less

42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:30 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 46b79dkahflhakjhdhfkah**** 47.11.**.**[]
42.11.**.** - [42.11.**.**]--[25/Nov/2021:11:40:31 +0800]"GET / HTTP/1.1" 200 615 "_" "curl/7.64.1" 76 0.001 [default-nginx-svc-80] 172.16.254.208:80 615 0.000 200 fadgrerthflhakjhdhfkah**** 47.11.**.**[]

You can diagnose the error log of the NGINX Ingress controller pod to narrow down the scope of troubleshooting. The error log of the Ingress controller pod includes the following types:

• The log that records errors of the Ingress controller. Typically, this type of error log is generated due to invalid Ingress configurations. You can run the following command to query this type of error log:

  kubectl logs <controller pod name> -n <namespace> | grep -E ^[WE]

  Note During the initialization of an Ingress controller, a few warning events may be generated. For example, if you do not specify the kubeconfig file or IngressClass resource, warning events are generated. These events do not affect the Ingress controller and can be ignored.
• The log that records errors of the NGINX application. Typically, this type of error log is generated due to request processing failures. You can run the following command to query this type of error log:

  kubectl logs <controller pod name> -n <namespace> | grep error

1. Run the following command to log on to the Ingress controller pod:

   kubectl exec <controller pod name> -n <namespace> -it -- bash

2. The Ingress controller pod is preinstalled with curl and OpenSSL, which allow you to test network connectivity and verify certificates.
   • Run the following command to test the network connectivity between the Ingress and the backend pod:

     # Replace your.domain.com with the actual domain name of the Ingress. 
     curl -H "Host: your.domain.com" http://127.0.**.**/ # for http
     curl --resolve your.domain.com:443:127.0.0.1 https://127.0.0.1/ # for https

   • Run the following command to verify the certificate:

     openssl s_client -servername your.domain.com -connect 127.0.0.1:443

   • Test access to the backend pod.
     Note An Ingress controller directly connects to the IP address of the backend pod instead of using a ClusterIP Service.
     1. Run the following kubectl command to query the IP address of the backend pod:

        kubectl get pod -n <namespace> <pod name> -o wide

        Expected output:

        NAME                      READY    STATUS    RESTARTS   AGE    IP            NODE                        NOMINATED NODE    READINESS GATES
        nginx-dp-7f5fcc7f-****    1/1      Running   0          23h    10.71.0.146   cn-beijing.192.168.**.**    <none>            <none>

        The output shows that the IP address of the backend pod is 10.71.0.146.

     2. To test the network connectivity between the Ingress controller pod and the backend pod, run the following command to connect to the IP address by using the Ingress controller pod:

        curl http://<your pod ip>:<port>/path

If you cannot identify the issue, capture and diagnose packets.

1. Check whether the issue is related to the Ingress controller pod or the application pod. If this cannot be done, capture packets for both the Ingress controller pod and the application pod.
2. Log on to the nodes on which the application pod and Ingress controller pod run.
3. Run the following command on the Elastic Compute Service (ECS) instances to capture all recent packets that are received by the Ingress:

   tcpdump -i any host <Application pod IP or Ingress controller pod IP> -C 20 -W 200 -w /tmp/ingress.pcap

4. If an error is identified in the log data, stop capturing packets.
5. Diagnose the packets that are transferred during the time period in which the error occurred.
   Note

   • Packet capture does not affect your service and only causes a slight increase in the CPU utilization and disk I/O.
   • The preceding command rotates the captured packets and can generate at most 200 .pcap files, each of which is 20 MB in size.

Why do I fail to access the IP address of the LoadBalancer from within the Kubernetes cluster?

Symptom

In a Kubernetes cluster, only specific nodes can access the IP address of the LoadBalancer if externalTrafficPolicy is set to Local for the LoadBalancer.

Causes

externalTrafficPolicy: Local is set for the LoadBalancer. In Local mode, the IP address of the LoadBalancer is accessible only from pods that are provisioned on the local node (the node that runs the LoadBalancer). The IP address of the LoadBalancer is inaccessible from pods on other nodes in the cluster. The IP address of the LoadBalancer is external to the Kubernetes cluster. If nodes or pods in the ACK cluster cannot access the IP address without using a second hop, requests do not pass through the LoadBalancer. As a result, the IP address of the LoadBalancer is considered an extended IP address of the Service that uses the LoadBalancer. Requests are forwarded by kube-proxy based on iptables or IP Virtual Server (IPVS).

In this scenario, if the requested pod is not provisioned on the local node, a connectivity issue occurs. The IP address of the LoadBalancer is accessible only if the requested pod is provisioned on the local node. For more information about external-lb, see kube-proxy adds the IP address of external-lb to the node local iptables rules.

Solutions

• We recommend that you access the IP address of the LoadBalancer from within the Kubernetes cluster by using the ClusterIP Service or the Ingress name.

  The Ingress name is nginx-ingress-lb in the kube-system namespace.

• Run the kubectl edit svc nginx-ingress-lb -n kube-system command to modify the Ingress. Set externalTrafficPolicy to Cluster for the LoadBalancer. After you change the setting, client IP addresses cannot be preserved.
• If the network plug-in of the cluster is set to Terway and the exclusive or inclusive ENI mode is selected, you can set externalTrafficPolicy to Cluster for the LoadBalancer and add the ENI annotation. The annotation adds pods that are signed elastic network interfaces (ENIs) as the backend servers of the LoadBalancer. This way, client IP addresses can be preserved and the IP address of the LoadBalancer can be accessed from within the cluster.
  Example:

  apiVersion: v1
  kind: Service
  metadata:
    annotations:
      service.beta.kubernetes.io/backend-type: eni   # Add pods that are signed ENIs as the backend servers of the LoadBalancer. 
    labels:
      app: nginx-ingress-lb
    name: nginx-ingress-lb
    namespace: kube-system
  spec:
    externalTrafficPolicy: Cluster

  For more information about the annotations of Services, see Use annotations to configure load balancing.

Why does the Ingress controller pod fail to access the Ingress controller?

Symptom

In a cluster for which Flannel is used, when the Ingress controller pod accesses the Ingress controller through a domain name, an SLB IP address, or cluster IP address, some or all of the requests sent from the Ingress controller pod are dropped.

Causes

By default, Flannel does not allow loopback requests.

Solutions

• We recommend that you create a new cluster that uses the Terway network plug-in and migrate your workloads to the new cluster.
• If you do not want to create a new cluster, you can enable hairpinMode in the configurations of Flannel. After you modify the configurations, recreate the Flannel pod for the modification to take effect.
  1. Run the following command to modify the configurations of Flannel:

     kubectl edit cm kube-flannel-cfg -n kube-system

  2. In the cni-config.json file that is returned, add "hairpinMode": true in the delegate field.
     Example:

     cni-conf.json: |
         {
           "name": "cb0",
           "cniVersion":"0.3.1",
           "type": "flannel",
           "delegate": {
             "isDefaultGateway": true,
             "hairpinMode": true
           }
         }

  3. Run the following command to restart Flannel:

     kubectl delete pod -n kube-system -l app=flannel   

  4. Delete and recreate the pod.

Why is the default TLS certificate or previous TLS certificate used after I add a TLS certificate to the cluster or change the TLS certificate?

Symptom

You added a Secret to the cluster or modified a Secret in the cluster, and updated the Secret name in the secretName field in the Ingress. When you access the cluster, the default certificate (Kubernetes Ingress Controller Fake Certificate) or the previous certificate is used.

Causes

• The certificate is not returned by the Ingress controller in the cluster.
• The certificate is invalid and the Ingress controller cannot load the certificate.
• The certificate is returned by the Ingress controller based on the Server Name Indication (SNI) field. The SNI field may not be sent as a part of the TLS handshake.

Solutions

• Use one of the following methods to check whether the SNI field is sent as a part of the TLS handshake:
  • Upgrade your browser to a version that supports SNI.
  • When you run the openssl s_client command to check whether the certificate is in use, specify the -servername parameter.
  • When you run curl commands, add hosts or use the --resolve parameter to map the domain name, other than using the host request header.
• Make sure that no TLS certificate is specified when you connect the website to WAF in CNAME record mode or transparent proxy mode, or no TLS certificate is associated with the Layer 7 listener of the SLB instance. The TLS certificate must be returned by the Ingress controller in the cluster.
• Navigate to the Container Intelligence Service console and diagnose the Ingress. In the diagnostic report, check whether the configurations of the Ingress are valid and whether the log data shows errors. For more information, see Use the Ingress diagnostics feature.
• Run the following command to view the error log of the Ingress controller pod and troubleshoot the issue based on the log data.

  kubectl logs <ingress pod name> -n <pod namespace> | grep -E ^[EW]

Why do I fail to access the Ingress by using the test domain name provided in the ACK console?

Symptom

You cannot access the Ingress by using the test domain name (*.cxxxxxxxxxxx.xxx.alicontainer.com) provided in the ACK console.

Causes

The test domain name is not resolved to the IP address of the Service that is associated with the Ingress. Possible causes:

• The Service is not nginx-ingress-lb in the kube-system namespace.
• The external IP address of the Service is changed after the Ingress is created.

Solutions

• Associate the desired IP address with the nginx-ingress-lb Service in the kube-system namespace.
• On the cluster details page of the Container Service for Kubernetes (ACK) console, click the Basic Information tab and then click Rebind Domain Name.

Why do I fail to access gRPC Services that are exposed by an Ingress?

Symptom

You cannot access gRPC Services that are exposed by an Ingress.

Causes

• You do not set annotations in the Ingress to specify the backend protocol.
• gRPC Services can be accessed only by using Transport Layer Security (TLS).

Solutions

• Set the following annotation in the Ingress: nginx.ingress.kubernetes.io/backend-protocol:"GRPC".
• Make sure that clients use HTTPS ports to send requests and the traffic is encrypted by using TLS.

Why do I fail to access backend HTTPS services?

Symptom

• You fail to access backend HTTPS services through the Ingress.
• A 400 error code may be returned and the following error message may be prompted: The plain HTTP request was sent to HTTPS port.

Causes

The Ingress controller sends HTTP requests to the backend pods. This is the default setting.

Solutions

Set the following annotation in the Ingress: nginx.ingress.kubernetes.io/backend-protocol:"HTTPS".

Why does the Ingress controller pod fail to preserve client IP addresses?

Symptom

The Ingress controller pod cannot preserve client IP addresses. Only the node IP, the CIDR blocks 100.XX.XX.XX, or other IP addresses are preserved.

Causes

• externalTrafficPolicy is set to Cluster for the Service that is associated with the Ingress.
• A Layer 7 proxy is used by the SLB instance.
• Your website is connected to WAF in CNAME record mode or transparent proxy mode.

Solutions

• If externalTrafficPolicy is set to Cluster for the Service and a Layer 4 SLB instance is used, perform the following steps:

  Set externalTrafficPolicy to Local. However, you may fail to access the Ingress by using the IP address of the SLB instance from within the cluster. For more information, see Why do I fail to access the IP address of the LoadBalancer from within the Kubernetes cluster?.

• Perform the following steps if a Layer 7 proxy is used, for example, a Layer 7 SLB instance is used or your website is connected to WAF in CNAME record mode or transparent proxy mode:
  1. Make sure that the X-Forwarded-For header is enabled for the Layer 7 proxy.
  2. Add enable-real-ip: "true" to the ConfigMap of the Ingress controller. By default, the ConfigMap is named nginx-configuration and belongs to the kube-system namespace.
  3. Analyze the log data to check whether client IP addresses can be preserved.
• If a client request traverses multiple hops before it reaches the Ingress controller pod, for example, the request must pass through a reverse proxy before it reaches the Ingress controller pod, you can check the value of remote_addr after you set enable-real-ip to true. If the value is a client IP address, this indicates that the X-Forwarded-For header is enabled to pass client IP addresses to the Ingress controller pod. If the X-Forwarded-For header is disabled, enable the X-Forwarded-For header or use other methods to add client IP addresses to requests before the requests reach the Ingress controller pod.

Why do canary release rules fail to take effect?

Symptom

You set canary release rules in a cluster but the rules do not take effect.

Causes

Possible causes:

• When you add canary-* annotations, you do not set nginx.ingress.kubernetes.io/canary: "true".
• The version of the NGINX Ingress controller is earlier than 0.47.0. When you add canary-* annotations, you do not specify the domain name of your application in the host field.

Solutions

• Set nginx.ingress.kubernetes.io/canary: "true" or specify the domain name of your application in the host field.
• If the issue persists, see Why are requests not distributed based on the specified canary release rules or why do the canary release rules affect other Ingresses that are associated with the same Service?.

Why are requests not distributed based on the specified canary release rules or why do the canary release rules affect other Ingresses that are associated with the same Service?

Symptom

Requests are not distributed based on the canary release rules that you set, or the canary release rules affect other Ingresses that are associated with the same Service.

Causes

Canary release rules in an NGINX Ingress controller take effect on all Ingresses that are associated with the Service for which the canary release rules are created.

For more information about this issue, see An Ingress with canary annotations affects other Ingresses that are associated with the same Service.

Solutions

Canary Ingresses include Ingresses that are assigned the service-match or canary-* annotations. Before you create a canary Ingress, create two same Services that are used for canary releases, and then map the Services to the backend pods that you want to access.

Why does the following error occur when you create an Ingress: "failed calling webhook"?

Symptom

The following error occurs when you create an Ingress: "Internal error occurred: failed calling webhook...".

Causes

When you create an Ingress resource, a Service is used to check whether the Ingress is valid. By default, the Service named ingress-nginx-controller-admission is used. If Webhook link errors occur, for example, the Service or the Ingress controller is deleted, the Ingress cannot be created.

Solutions

• Check whether the resource exists and works as expected based on the following Webhook link: ValidatingWebhookConfiguration > Service > Pod.
• Make sure that the admission feature is enabled for the Ingress controller pod and the pod can be accessed from outside the cluster.
• If the Ingress controller is deleted or you do not want to use the Webhook feature, you can delete the ValidatingWebhookConfiguration resource.

Why is the following error returned for HTTPS requests: SSL_ERROR_RX_RECORD_TOO_LONG?

Symptom

One of the following errors is returned for HTTPS requests: SSL_ERROR_RX_RECORD_TOO_LONG or routines:CONNECT_CR_SRVR_HELLO:wrong version number.

Causes

HTTPS requests are distributed to a non-HTTPS port, such as an HTTP port.

Solutions

Modify the configurations of the SLB instance or Service to ensure that HTTPS requests can be distributed to the proper port.

Common HTTP status codes

Symptom

HTTP status codes other than 2xx and 3xx are returned, such as 502, 503, 413, and 499.

Causes and solutions

View the log and check whether the error is returned by the Ingress controller. For more information, see Diagnose the access log of the NGINX Ingress controller pod in Log Service. If the error is returned by the Ingress controller, use the following solutions:

• 413 error
  • Cause: The request size exceeds the upper limit.
  • Solution: Increase the value of proxy-body-size in the ConfigMap of the Ingress controller. The default value of proxy-body-size is 1 MB for the NGINX Ingress controller of open source Kubernetes and the default value of proxy-body-size is 20 MB for the NGINX Ingress controller of ACK.
• 499 error
  • Cause: The client terminates the connection in advance. The error may not be caused by the Ingress controller or backend services.
  • Solution:
    • If the 499 error does not occur frequently and your workloads are not affected, you can ignore the error.
    • If the 499 error occurs frequently, you must check whether the amount of time that the backend pods cost to process requests exceeds the request timeout period that is set on the client.
• 502 error
  • Cause: The Ingress controller cannot connect to backend pods.
  • Solution:
    • The issue occurs occasionally:
      • Check whether the backend pods work as expected. If the backend pods are overloaded, add more backend pods.
      • By default, the Ingress controller sends HTTP 1.1 requests to backend services and HTTP keep-alive is enabled. Make sure that the keep-alive timeout period configured for the backend pods is greater than that configured for the Ingress controller. By default, the timeout period is set to 60 seconds.
    • The issue occurs every time:
      • Check whether the Service port is valid and whether the Service can be accessed from the Ingress controller pod.
      • In the Container Intelligence Service console, choose Cluster Check > Diagnosis. On the Diagnosis page, click Network diagnosis to check the network connectivity.
    • If the issue persists, capture and analyze packets, and then Submit a ticket.
• 503 error
  • Symptom: The Ingress controller cannot discover the backend pods, or the Ingress controller fails to access all backend pods.
  • Solution:
    • The issue occurs occasionally:
      • Refer to the solution for resolving the 502 error.
      • Check the status of the backend pods and configure health checks.
    • The issue occurs every time:

      Check whether the Service configuration is valid and whether the endpoint exists.

    • If you cannot locate the cause by using the preceding methods, Submit a ticket.

The following error occurs: net::ERR_HTTP2_SERVER_REFUSED_STREAM.

Symptom

When you access the website, some resources cannot be loaded and one of the following errors is prompted in the console: net::ERR_HTTP2_SERVER_REFUSED_STREAM or net::ERR_FAILED.

Causes

The number of concurrent HTTP/2 streams to the resource has reached the upper limit.

Solutions

• We recommend that you change http2-max-concurrent-streams in the ConfigMap to a greater value. The default value is 128. For more information, see http2-max-concurrent-streams.
• Disable HTTP/2 by setting use-http2 to false in the ConfigMap. For more information, see use-http2.

Why does the following error occur: The param of ServerGroupName is illegal?

Causes

ServerGroupName is generated in the following format: namespace+svcName+port. The server group name must be 2 to 128 characters in length and can contain letters, digits, periods (.), underscores (_), and hyphens (-). The name must start with a letter.

Solutions

Modify the server group name based on the required format.

Why does the "certificate signed by unknown authority" error occur when I create an Ingress?

Causes

If multiple Ingresses are deployed in the cluster and the Ingresses use the same resources, such as Secrets, Services, or webhook configurations, the preceding error occurs because different SSL certificates are used to communicate with backend servers when webhooks are triggered.

Solutions

Redeploy two Ingresses and make sure that the Ingresses use different resources. For more information about the resources used by Ingresses, see What are the system updates after I update the NGINX Ingress controller on the Add-ons page of the ACK console?.

Why does the Ingress controller pod restart after it fails the health check?

Symptom

The Ingress controller pod restarts after it fails the health check.

Causes

• The Ingress controller pod or the node where the pod is deployed is overloaded. As a result, the pod failed to pass the health check.
• Kernel parameters such as tcp_tw_reuse or tcp_tw_timestamps may be configured for the cluster node where the Ingress controller pod is deployed. This may cause health check failures.

Solutions

• Add more Ingress controller pods and check whether the issue persists. For more information, see Deploy an Ingress access layer with high reliability.
• Disable tcp_tw_reuse or set the value of the parameter to 2, disable tcp_tw_timestamps, and then check whether the issue persists.

How do I add Services that use TCP or UDP?

1. Add specific entries to the tcp-services and udp-services ConfigMaps. By default, the ConfigMaps belong to the kube-system namespace.

   The following code block shows an example on how to map port 8080 of example-go in the default namespace to port 9000:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: tcp-services
     namespace: ingress-nginx
   data:
     9000: "default/example-go:8080"  # Map port 8080 to port 9000. 

2. Add port 9000 to the Deployment of the Ingress. By default, the Deployment is named nginx-ingress-controller and belongs to the kube-system namespace.
3. Add the port 9000 to the Service that is associated with the Ingress.
   Example:

   apiVersion: v1
   kind: Service
   metadata:
     name: ingress-nginx
     namespace: ingress-nginx
     labels:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx
   spec:
     type: LoadBalancer
     ports:
       - name: http
         port: 80
         targetPort: 80
         protocol: TCP
       - name: https
         port: 443
         targetPort: 443
         protocol: TCP
       - name: proxied-tcp-9000
         port: 9000
         targetPort: 9000
         protocol: TCP
     selector:
       app.kubernetes.io/name: ingress-nginx
       app.kubernetes.io/part-of: ingress-nginx

   For more information about how to add Services that use TCP or UDP, see Expose Services that use TCP or UDP.

Why do Ingress rules fail to take effect?

Symptom

After you add or modify Ingress rules, the rules do not take effect.

Causes

• The configuration of the Ingress contains errors. As a result, the Ingress failed to load the Ingress rules.
• The configurations of Ingress resources contain errors.
• The Ingress controller does not have the required permissions. As a result, the Ingress controller cannot monitor the changes made to Ingress resources.
• The previous Ingress uses a domain name specified in the server-alias field. The domain name is in conflict with that of the new Ingress. As a result, the Ingress rules are ignored.

Why does the system fail to load some web page resources or return a blank white screen when requests are redirected to the root directory?

Symptom

After you set the rewrite-target annotation in the Ingress to rewrite requests, some web page resources cannot be loaded or a blank white screen is displayed when you access the backend service.

Causes

• You do not set rewrite-target in regular expressions.
• The path of the requested resource is set to the root directory.

Solutions

• Check whether rewrite-target is set in regular expressions and whether capture groups are used. For more information, see Rewrite.
• Check whether requests are redirected to the expected path.

How do I fix the issue that Log Service cannot parse logs as expected after ingress-nginx-controller is upgraded?

Symptom

The ingress-nginx-controller component has two commonly used versions: ingress-nginx-controller 0.20 and 0.30. After you upgrade ingress-nginx-controller from 0.20 to 0.30 on the Add-ons page in the console, the Ingress dashboard may not show the actual statistics of requests to the backend servers when you perform canary releases or blue-green releases with an Ingress.

Causes

The default log format of ingress-nginx-controller 0.20 is different from that of ingress-nginx-controller 0.30. Therefore, the Ingress dashboard may not show the actual statistics of requests to the backend servers when you perform canary releases or blue-green releases with an Ingress.

Solutions

To fix the issue, perform the following steps to update the nginx-configuration ConfigMap and the configuration of k8s-nginx-ingress.

1. Update the nginx-configuration ConfigMap.
   • If you have not modified the nginx-configuration ConfigMap, copy the following content to a file named nginx-configuration.yaml and run the kubectl apply -f nginx-configuration.yaml command to deploy the file.

     apiVersion: v1
     kind: ConfigMap
     data:
       allow-backend-server-header: "true"
       enable-underscores-in-headers: "true"
       generate-request-id: "true"
       ignore-invalid-headers: "true"
       log-format-upstream: $remote_addr - [$remote_addr] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id $host [$proxy_alternative_upstream_name]
       max-worker-connections: "65536"
       proxy-body-size: 20m
       proxy-connect-timeout: "10"
       reuse-port: "true"
       server-tokens: "false"
       ssl-redirect: "false"
       upstream-keepalive-timeout: "900"
       worker-cpu-affinity: auto
     metadata:
       labels:
         app: ingress-nginx
       name: nginx-configuration
       namespace: kube-system

   • If you have modified the nginx-configuration ConfigMap, run the following command to update the configuration. This ensures that your previous modifications are not overwritten.

     kubectl edit configmap nginx-configuration -n kube-system

   Append [$proxy_alternative_upstream_name] to the log-format-upstream field, save the changes, and then exit.

2. Update the configuration of k8s-nginx-ingress.
   Copy the following content to a file named k8s-nginx-ingress.yaml and run the kubectl apply -f k8s-nginx-ingress.yaml command to start the deployment.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
       namespace: kube-system
     # your config name, must be unique in you k8s cluster
     name: k8s-nginx-ingress
   spec:
     # logstore name to upload log
     logstore: nginx-ingress
     # product code, only for k8s nginx ingress
     productCode: k8s-nginx-ingress
     # logtail config detail
     logtailConfig:
       inputType: plugin
       # logtail config name, should be same with [metadata.name]
       configName: k8s-nginx-ingress
       inputDetail:
         plugin:
           inputs:
           - type: service_docker_stdout
             detail:
               IncludeLabel:
                 io.kubernetes.container.name: nginx-ingress-controller
               Stderr: false
               Stdout: true
           processors:
           - type: processor_regex
             detail:
               KeepSource: false
               Keys:
               - client_ip
               - x_forward_for
               - remote_user
               - time
               - method
               - url
               - version
               - status
               - body_bytes_sent
               - http_referer
               - http_user_agent
               - request_length
               - request_time
               - proxy_upstream_name
               - upstream_addr
               - upstream_response_length
               - upstream_response_time
               - upstream_status
               - req_id
               - host
               - proxy_alternative_upstream_name
               NoKeyError: true
               NoMatchError: true
               Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)\s(\d+)\s"([^"]*)"\s"([^"]*)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S*)\s*\[*([^]]*)\]*.*
               SourceKey: content