All Products
Search
Document Center

Container Service for Kubernetes:Analyze and monitor the access log of nginx-ingress-controller

Last Updated:Aug 02, 2023

Container Service for Kubernetes (ACK) allows you to configure the nginx-ingress-controller component for an ACK cluster. This component provides URLs that can be visited by servers outside the cluster, and supports server load balancing, SSL termination, and name-based virtual hosting. You can also use nginx-ingress-controller to write the log data of HTTP requests to stdout. You can enable Log Service for an ACK cluster when you create the cluster. After Log Service is enabled, you can monitor nginx-ingress-controller in real time and view dashboards in Log Service. The dashboards show statistics that are collected from the access log of nginx-ingress-controller. This topic describes how to configure log collection for nginx-ingress-controller and how to view the collected log.

Table of contents

Prerequisites

  1. The logtail-ds component is installed in your cluster.

    By default, logtail-ds is installed when you create a cluster. If this component is not installed, you can manually install it. For more information, see Collect log data from containers by using Log Service.

  2. The alibaba-log-controller component is updated.

    Run the kubectl edit deployment alibaba-log-controller -n kube-system command to update the component.

Configure log collection for nginx-ingress-controller

ACK defines a type of CustomResourceDefinition (CRD) named AliyunLogConfig. You can create an AliyunLogConfig to configure log collection. alibaba-log-controller automatically generates configurations for Log Service to collect log data and update data in relevant dashboards.

Important

Take note of the following items when you create an AliyunLogConfig:

  • If nginx-ingress-controller already exists, you must recreate the pods of nginx-ingress-controller to enable log collection after you create an AliyunLogConfig.

  • Make sure that the version of alibaba-log-controller is 0.2.0.0-76648ee-aliyun or later. After you update alibaba-log-controller, if you find that an AliyunLogConfig is already created, delete the AliyunLogConfig and recreate it.

  • The AliyunLogConfig applies only to the default log format that ACK defines for the Ingress controller. If you have changed the log format, you must modify the processor_regex settings in the AliyunLogConfig configuration. For more information, see Use CRDs to collect container logs in DaemonSet mode.

  • Set IncludeLabel to the label information that is retrieved by using the docker inspect command.

  • A namespace and a container name in a Kubernetes cluster are separately mapped to the following Docker labels: io.kubernetes.pod.namespace and io.kubernetes.container.name. For example, the pod that you create belongs to the backend-prod namespace and the container name is worker-server.

    • If you specify io.kubernetes.pod.namespace : backend-prod, the logs of all containers in the pods in the specified namespace are collected.

    • If you specify io.kubernetes.container.name : worker-server, the log of the specified container is collected.

  • We recommend that you specify only the io.kubernetes.pod.namespace and io.kubernetes.container.name labels. If you want to specify other labels, you can configure the IncludeEnv or ExcludeEnv parameter. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

To disable log collection for nginx-ingress-controller, delete the AliyunLogConfig.

Run the following command to create an AliyunLogConfig:

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the AliyunLogConfig. The name must be unique in the cluster.
  name: k8s-nginx-ingress
  # The namespace of the AliyunLogConfig. Set the value to the namespace of the Ingress controller from which you want to collect logs.
  namespace: kube-system
spec:
  # The name of the Logstore that you want to use.
  logstore: nginx-ingress
  productCode: k8s-nginx-ingress
  # The Logtail configuration.
  logtailConfig:
    inputType: plugin
    # The name of the Logtail configuration. The name must be the same as the value of the metadata.name parameter.
    configName: k8s-nginx-ingress
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel: # The labels of the Docker containers from which you want to collect logs.
              # The name of the container from which you want to collect logs is nginx-ingress-controller.
              # If you collect logs from multiple Ingress controllers, duplicate logs may be collected. We recommend that you read and understand the description of the IncludeLabel parameter.
              io.kubernetes.container.name: nginx-ingress-controller
            Stderr: true
            Stdout: true
        processors:
        - type: processor_regex # Use a regular expression to extract fields.
          detail:
            KeepSource: false
            Keys: # The extracted fields are sorted in the following order:
            - client_ip
            - x_forward_for
            - remote_user
            - time
            - method
            - url
            - status
            - body_bytes_sent
            - http_referer
            - http_user_agent
            - request_length
            - request_time
            - proxy_upstream_name
            - upstream_addr
            - upstream_response_length
            - upstream_response_time
            - upstream_status
            - req_id
            - host
            - proxy_alternative_upstream_name
            NoKeyError: true
            NoMatchError: true
            # The regular expression that is used to extract fields. If a log matches the regular expression, the values of the preceding fields are automatically specified based on the content in each capturing group.
            Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S*)\s*\[([^]]+)]\s"(\w+)\s([^"]+)"\s(\d+)\s(\d+)[^"]+"([^"]*)"\s"([^"]+)"\s(\d+)\s(\S+)\s\[([^]]+)]\s(\S+)\s(\d+)\s(\S+)\s(\d+)\s(\w+)\s(\S+)\s*\[*([^]]*)\]*.*
            SourceKey: content

View the access log and dashboards of nginx-ingress-controller

  1. Log on to the Log Service console.

  2. In the Projects section, click the name of the project that you specified when you created the ACK cluster. The details page of the project appears. By default, a project that is named in the format of k8s-log-{cluster-id} is created for the ACK cluster.

  3. In the left-side navigation pane, click the Nginx Ingress.png icon, and then click a dashboard in the dashboard list to view data on the dashboard.

Ingress overview

The Ingress overview dashboard displays information about network traffic that flows through nginx-ingress-controller for a website. You can view the following information:

  • Website data of the last 24 hours, including the number of page views (PVs), the number of unique visitors (UVs), inbound and outbound traffic, the average latency, the proportion of mobile users, and the proportions of 5xx errors and 404 errors.

  • Website data of the last 1 minute, including the number of PVs, the number of UVs, the success rate of requests, the average latency, the P95 latency, and the P99 latency.

  • Detailed information about requests within the last 24 hours, including the PV trend of the last 24 hours (based on the PVs within the last 24 hours), the PV trend of the last 7 days (based on the PVs within the last 7 days), regional distribution of request sources, the top N source areas and cities, the proportion of mobile users, and the proportions of Android users and iOS users.

  • Top N URLs within the last 1 hour, including the 10 URLs with the highest PVs, the 10 URLs with the highest latencies, the 10 URLs that return the most 5xx errors, and the 10 URLs that return the most 404 errors.

Ingress access center

The Ingress access center dashboard displays up-to-date information about requests. You can obtain and analyze the following data to help make business decisions: the numbers of UVs and PVs within the last 24 hours, the regional distribution of PVs and UVs, the top N areas by request, the top N cities by request, the top N browsers with the highest PVs, the top N source IP addresses with the highest PVs, the proportion of mobile users, and the proportions of Android users and iOS users.

ingress访问中心

Ingress monitoring center

The Ingress monitoring center dashboard provides real-time monitoring and alerting data of the website. You can view the following data within the last 1 hour: the success rate of requests, the proportion of 5xx errors, the proportion of 404 errors, the proportion of requests that are not forwarded, the average latency, the P95 latency, the P99 latency, the P999 latency, the request distribution by status code, the proportion of PVs on each Ingress, the top 10 Services with the highest PVs, the top 10 Services with the highest request failure rates, the top 10 Services with the highest average latencies, and the top 10 Services with the most requests.

Ingress监控中心

Ingress monitoring center for blue-green deployments

The Ingress monitoring center for blue-green deployments dashboard displays the real-time monitoring data of the Services that are used to expose the old and new application versions. The dashboard also compares the monitoring data of the Services. This allows you to identify exceptions and roll back the application version at the earliest opportunity. You must specify ServiceA and ServiceB for monitoring data comparison. The dashboard displays the following dynamic monitoring data of each Service: the number of PVs, the proportion of 5xx errors, the success rate of requests, the average latency, the P95 latency, the P99 latency, the P999 latency, and the number of requests.

Ingress蓝绿发布监控中心

Ingress exceptions center

The Ingress exceptions center automatically detects anomalies in the log of nginx-ingress-controller. This service uses the machine learning algorithms provided by Log Service and the time series analysis algorithms to improve detection efficiency.

Ingress异常检测中心

Configure alerting

Log Service enables interactive log analytics and provides visualized dashboards. You can also use Log Service to configure alerting based on data on the dashboards. You can select one or more alert notification methods, such as emails, DingTalk chatbot webhooks, custom webhooks, and (Short Message Service) messages.

For more information about how to configure alerting, see Configure an alert rule.

The following example describes how to configure alerting based on the proportion of 5xx errors. After you configure alerting, the system checks whether the alert conditions are met every 5 minutes. If the proportion of 5xx errors within the specified time range reaches 1%, an alert is triggered.

  1. In the Dashboard section, click the Ingress Monitoring Center dashboard. On the Ingress Monitoring Center page, move the pointer over the 提示 icon in the upper-right corner of the 5XX Proportion chart and click Save as Alert (Old Version).

    配置告警
  2. In the Create Alert wizard, set Alert Name, Associated Chart, Frequency, and Trigger Condition. The total field in the query statement indicates the proportion of 5XX errors. Specify total > 1 in the Trigger Condition field.

  3. On the Notifications wizard page, select one or more alerting methods based on your requirements and set the parameters. Then, click Submit to create the alert rule.

Subscribe to a dashboard

Log Service allows you to subscribe to dashboards. This feature takes snapshots of a dashboard and sends the snapshots by email or DingTalk group message at a specified interval.

For more information about how to subscribe to a dashboard, see Subscribe to a dashboard.

The following example describes how to subscribe to the Ingress overview dashboard. After you subscribe to the dashboard, a message is sent at 10:00 every day to the specified DingTalk group.

  1. In the Dashboard section, click the Ingress Overview V1.2 dashboard. On the Ingress Overview V1.2 page, choose Subscribe > Create.

  2. In the Create Subscription wizard, select Daily and 10:00 in the Frequency field. Turn off Add Watermark. Then, click Next.

  3. On the Notifications wizard page, select WebHook-DingTalk Bot from the Notifications drop-down list and set Request URL to the webhook URL of your DingTalk chatbot. Then, click Submit to complete the configuration.