Container Service for Kubernetes (ACK) allows you to configure the nginx-ingress component for an ACK cluster. This component provides URLs that can be visited by servers outside the cluster, and supports server load balancing, SSL termination, and name-based virtual hosting. You can also use nginx-ingress to write the log data of HTTP requests to stdout. You can enable Log Service for an ACK cluster when you create the cluster. After Log Service is enabled, you can monitor nginx-ingress in real time and view dashboards in Log Service. The dashboards show statistics that are collected from the access log of nginx-ingress. This topic describes how to configure log collection for nginx-ingress and how to view the collected log.

Prerequisites

  1. The logtail-ds component is installed.

    By default, logtail-ds is installed when you create a cluster. If this component is not installed, you can manually install it. For more information, see Collect log data from containers by using Log Service.

  2. The alibaba-log-controller component is updated.

    Run the kubectl edit deployment alibaba-log-controller -n kube-system command to update the component.

Configure log collection for nginx-ingress

ACK defines a type of custom object named AliyunLogConfig by using the CustomResourceDefinition (CRD) resource. You can create an AliyunLogConfig to configure log collection. alibaba-log-controller automatically generates configurations for Log Service to collect log data and update data in relevant dashboards.

Notice
Take note of the following limits when you create an AliyunLogConfig:
  • Make sure that the version of alibaba-log-controller is 0.2.0.0-76648ee-aliyun or later. After you update alibaba-log-controller, if you find that an AliyunLogConfig is already created, delete the AliyunLogConfig and recreate it.
  • The AliyunLogConfig applies only to the default log format that ACK defines for the Ingress controller. If you have changed the log format, you must modify the processor_regex settings in the AliyunLogConfig configuration. For more information, see Use CRDs to collect container logs in DaemonSet mode.
  • Set IncludeLabel to the label information that is retrieved by using the docker inspect command.
  • A namespace and a container name in a Kubernetes cluster are separately mapped to the following Docker labels: io.kubernetes.pod.namespace and io.kubernetes.container.name. For example, the pod that you create belongs to the backend-prod namespace and the container name is worker-server.
    • If you specify io.kubernetes.pod.namespace : backend-prod, the logs of all containers in the pods in the specified namespace are collected.
    • If you specify io.kubernetes.container.name : worker-server, the log of the specified container is collected.
  • We recommend that you specify only the io.kubernetes.pod.namespace and io.kubernetes.container.name labels. If you want to specify other labels, you can configure the IncludeEnv or ExcludeEnv parameter. For more information, see Use the Log Service console to collect container stdout and stderr in DaemonSet mode.

To disable log collection for nginx-ingress, delete the AliyunLogConfig.

Run the following command to create an AliyunLogConfig:
cat <<-EOF | kubectl apply -n kube-system -f -
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # your config name, must be unique in you k8s cluster
  name: k8s-nginx-ingress
spec:
  # logstore name to upload log
  logstore: nginx-ingress
  # product code, only for k8s nginx ingress
  productCode: k8s-nginx-ingress
  # logtail config detail
  logtailConfig:
    inputType: plugin
    # logtail config name, should be same with [metadata.name]
    configName: k8s-nginx-ingress
    inputDetail:
      plugin:
        inputs:
        - type: service_docker_stdout
          detail:
            IncludeLabel:
              io.kubernetes.container.name: nginx-ingress-controller  # If you want to collect logs from multiple Ingress controllers, duplicate logs may be collected. We recommend that you read and understand the description of the IncludeLabel parameter.
            Stderr: false
            Stdout: true
        processors:
        - type: processor_regex
          detail:
            KeepSource: false
            Keys:
            - client_ip
            - x_forward_for
            - remote_user
            - time
            - method
            - url
            - version
            - status
            - body_bytes_sent
            - http_referer
            - http_user_agent
            - request_length
            - request_time
            - proxy_upstream_name
            - upstream_addr
            - upstream_response_length
            - upstream_response_time
            - upstream_status
            - req_id
            - host
            - proxy_alternative_upstream_name
            NoKeyError: true
            NoMatchError: true
            Regex: ^(\S+)\s-\s\[([^]]+)]\s-\s(\S+)\s\[(\S+)\s\S+\s"(\w+)\s(\S+)\s([^"]+)"\s(\d+)\s(\d+)\s"([^"]*)"\s"([^"]*)"\s(\S+)\s(\S+)+\s\[([^]]*)]\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S*)\s*\[*([^]]*)\]*.*
            SourceKey: content
EOF

View the access log and dashboards of nginx-ingress

  1. Log on to the Log Service console.
  2. In the Projects section, click the name of the project that you specified when you created the ACK cluster. The details page of the project appears. By default, a project that is named in the format of k8s-log-{cluster-id} is created for the ACK cluster.
  3. In the left-side navigation pane, click the Nginx Ingress.png icon, and then click a dashboard in the dashboard list to view data on the dashboard.

Ingress overview

The Ingress overview dashboard displays information about network traffic that flows through nginx-ingress for a website. You can view the following information:

  • Website data of the last 24 hours, including the number of page views (PVs), the number of unique visitors (UVs), inbound and outbound traffic, the average latency, the proportion of mobile users, and the proportions of 5xx errors and 404 errors.
  • Website data of the last 1 minute, including the number of PVs, the number of UVs, the success rate of requests, the average latency, the P95 latency, and the P99 latency.
  • Detailed information about requests within the last 24 hours, including the PV trend of the last 24 hours (based on the PVs within the last 24 hours), the PV trend of the last 7 days (based on the PVs within the last 7 days), regional distribution of request sources, the top N source areas and cities, the proportion of mobile users, and the proportions of Android users and iOS users.
  • Top N URLs within the last 1 hour, including the 10 URLs with the highest PVs, the 10 URLs with the highest latencies, the 10 URLs that return the most 5xx errors, and the 10 URLs that return the most 404 errors.
Ingress overview

Ingress access center

The Ingress access center dashboard displays up-to-date information about requests. You can obtain and analyze the following data to help make business decisions: the numbers of UVs and PVs within the last 24 hours, the regional distribution of PVs and UVs, the top N areas by request, the top N cities by request, the top N browsers with the highest PVs, the top N source IP addresses with the highest PVs, the proportion of mobile users, and the proportions of Android users and iOS users.

Ingress monitoring center

The Ingress monitoring center dashboard provides real-time monitoring and alerting data of the website. You can view the following data within the last 1 hour: the success rate of requests, the proportion of 5xx errors, the proportion of 404 errors, the proportion of requests that are not forwarded, the average latency, the P95 latency, the P99 latency, the P999 latency, the request distribution by status code, the proportion of PVs on each Ingress, the top 10 Services with the highest PVs, the top 10 Services with the highest request failure rates, the top 10 Services with the highest average latencies, and the top 10 Services with the most requests.

Ingress monitoring center

Ingress monitoring center for blue-green deployment

The Ingress monitoring center for blue-green deployment dashboard displays the real-time monitoring data of the Services that are used to expose the old and new application versions. The dashboard also compares the monitoring data of the Services. This allows you to identify exceptions and roll back the application version at the earliest opportunity. You must specify ServiceA and ServiceB for monitoring data comparison. The dashboard displays the following dynamic monitoring data of each Service: the number of PVs, the proportion of 5xx errors, the success rate of requests, the average latency, the P95 latency, the P99 latency, the P999 latency, and the number of requests.

Ingress monitoring center for blue-green deployment

Ingress exceptions center

The Ingress exceptions center automatically detects anomalies in the log of nginx-ingress. This service uses the machine learning algorithms provided by Log Service and the time series analysis algorithms to improve detection efficiency.

Ingress exceptions center

Configure alerting

Log Service enables interactive log analytics and provides visualized dashboards. You can also use Log Service to configure alerting based on data on the dashboards. You can select one or more alert notification methods, such as emails, DingTalk chatbot webhooks, custom webhooks, and (Short Message Service) messages.

For more information about how to configure alerting, see Configure an alert rule.

The following example describes how to configure alerting based on the proportion of 5xx errors. After you configure alerting, the system checks whether the alert conditions are met every 5 minutes. If the proportion of 5xx errors within the specified time range reaches 1%, an alert is triggered.

  1. In the Dashboard section, click the Ingress Monitoring Center dashboard. On the Ingress Monitoring Center page, move the pointer over the More icon in the upper-right corner of the 5XX Proportion chart and click Save as Alert (Old Version).
    Configure alerting
  2. In the Create Alert wizard, set Alert Name, Associated Chart, Frequency, and Trigger Condition. The total field in the query statement indicates the proportion of 5XX errors. Specify total > 1 in the Trigger Condition field.
  3. On the Notifications wizard page, select one or more notification methods based on your requirements and set the parameters. Then, click Submit to complete the configuration.

Subscribe to a dashboard

Log Service allows you to subscribe to dashboards. This feature takes snapshots of a dashboard and sends the snapshots by email or DingTalk group message at a specified interval.

For more information about how to subscribe to a dashboard, see Subscribe to a dashboard.

The following example describes how to subscribe to the Ingress overview dashboard. After you subscribe to the dashboard, a message is sent at 10:00 every day to the specified DingTalk group.

  1. In the Dashboard section, click the Ingress Overview V1.2 dashboard. On the Ingress Overview V1.2 page, choose Subscribe > Create.
  2. In the Create Subscription wizard, select Daily and 10:00 in the Frequency field. Turn off Add Watermark. Then, click Next.
  3. On the Notifications wizard page, select WebHook-DingTalk Bot from the Notifications drop-down list and set Request URL to the webhook URL of your DingTalk chatbot. Then, click Submit to complete the configuration.