Container Service for Kubernetes:Use ACK Net Exporter to troubleshoot network issues

Install and configure ACK Net Exporter

Install ACK Net Exporter

1. Log on to the ACK console and choose Marketplace > Marketplace in the left-side navigation pane.
2. Find and click ack-net-exporter on the Marketplace page.
3. On the ack-net-exporter page, click Deploy in the upper-right corner.
4. In the Basic Information step, specify Cluster and Namespace, and then click Next.
5. In the Parameters step, configure the following parameters and click OK.

   Parameter	Description	Default value
   name	The name of the ACK Net Exporter component.	ack-net-exporter-default
   namespace	The namespace to which ACK Net Exporter belongs.	kube-system
   config.enableEventServer	Specify whether to enable event tracing. Valid values: false: disables event tracing. true: enables event tracing.	false
   config.enableMetricServer	Specify whether to enable metric collection. Valid values: false: disables metric collection. true: enables metric collection.	true
   config.enableLegacyVersion	Specify whether to enable the compatibility mode. Valid values: false: disables the compatibility mode. true: enables the compatibility mode. After you enable this mode, ACK Net Exporter supports more operating systems. However, you cannot use new features provided by ACK Net Exporter.	true
   config.remoteLokiAddress	The Grafana Loki service address to which events are pushed.	By default, this parameter is empty.
   config.metricLabelVerbose	Specify whether to enable metric verbose. Valid values: false: disables metric verbose. true: enables metric verbose. After you enable this feature, pod IP addresses and labels are saved as the label information of metrics.	false
   config.metricServerPort	The port that is used by the metric service to provide HTTP services.	9102
   config.eventServerPort	The port that is used by the event service to provide gRPC streaming services.	19102
   config.metricProbes	The metric probes that you want to enable. For more information, see ACK Net Exporter metrics.	By default, this parameter is empty and only the required metric probes are enabled.
   config.eventProbes	The event probes that you want to enable. For more information, see ACK Net Exporter events.	By default, this parameter is empty and only the required event probes are enabled.

Configure ACK Net Exporter

• You can run the following command to modify the ConfigMap of ACK Net Exporter:

  kubectl edit cm inspector-config -n kube-system

• You can also configure ACK Net Exporter in the ACK console.
  1. Log on to the ACK console and click Clusters in the left-side navigation pane.
  2. On the Clusters page, click the name of a cluster and choose Configurations > ConfigMaps in the left-side navigation pane.
  3. On the ConfigMap page, set Namespace to kube-system, search for inspector-config, and then click Edit in the Actions of inspector-config.
  4. In the Edit panel, configure the parameters and click OK.

     The following table describes the parameters supported by ACK Net Exporter.

     Parameter	Description	Default value
     debugmode	Specify whether to enable the debugging mode. Valid values: false: disables the debugging mode. true: enables the debugging mode. After you enable this feature, debug-level logs, interface debugging, Go pprof, and gops are supported.	false
     event_config.loki_enable	Specify whether to enable the feature of pushing events to Grafana Loki. For more information, see Use Grafana Loki to collect and visualize events. Valid values: false: disables the feature. true: enables the feature.	false
     event_config.loki_address	The Grafana Loki service address. By default, the system automatically discovers a service named grafana-loki in the specified namespace.	By default, this parameter is empty.
     event_config.probes	The event probes that you want to enable. For more information, see ACK Net Exporter events.	By default, this parameter is empty and only the required event probes are enabled.
     event_config.port	The port used by the event service to provide gRPC streaming services.	19102
     metric_config.verbose	Specify whether to enable metric verbose. Valid values: false: disables metric verbose. true: enables metric verbose. After you enable this feature, pod IP addresses and labels are saved as the label information of metrics.	false
     metric_config.port	The port that is used by the metric service to provide HTTP services.	9102
     metric_config.probes	The metric probes that you want to enable. For more information, see ACK Net Exporter metrics.	By default, this parameter is empty and only the required metric probes are enabled.
     metric_config.interval	The interval at which metrics are collected. Metric collection compromises performance. Therefore, ACK Net Exporter caches the periodically collected metrics in memory.	5

In earlier ACK Net Exporter versions, you need to trigger the system to recreate all ACK Net Exporter containers after you modify the configuration of ACK Net Exporter. The modified configuration takes effect after the containers are recreated. You no longer need to perform this operation in ACK Net Exporter 0.2.3 and later versions because these versions support hot updates.

Usage notes for ACK Net Exporter

Use ACK Net Exporter in operating systems other than Alinux

Some key features of ACK Net Exporter rely on eBPF programs to collect information. To meet the requirements of different operating system kernels, ACK Net Exporter uses CO-RE to distribute eBPF programs. When ACK Net Exporter starts up, it needs to load the BTF file that is associated with the operating system kernel. The BTF file stores the metadata of the kernel debug information. If no corresponding BTF file is loaded, the key features become unavailable. Most later operating system versions have built-in BTF files. For more information about the operating systems, see BPF Type Format.

Metrics and metric format supported by ACK Net Exporter

inspector_pod_udprcvbuferrors{namespace="elastic-system",netns="ns402653****",node="iZbp179u0bgzhofjupc****",pod="elastic-operator-0"} 0 1654487977826

Events and event format supported by ACK Net Exporter

ACK Net Exporter can collect events of network exceptions that occur on nodes. This section describes the network exceptions that you may encounter. These exceptions occasionally occur and are difficult to reproduce. Currently, no efficient methods can be used to troubleshoot these exceptions.

• Connection failures and request timeouts caused by data packet loss.
• Performance issues caused by time-consuming data processing.
• Business interruptions that occur due to the anomalies of the stateful connection mechanism, such as TCP or connection tracking.

ACK Net Exporter provides eBPF-based context observability for operating system kernels to help you troubleshoot the preceding issues. ACK Net Exporter can capture the status of the operating system in real time when an exception occurs and then generates an event log. For more information about the events and event probes supported by ACK Net Exporter, see ACK Net Exporter events.

type=TCPRESET_NOSOCK pod=storage-monitor-5775dfdc77-fj767 namespace=kube-system protocol=TCP saddr=100.103.42.233 sport=443 daddr=10.1.17.188 dport=33488

For events that require valid operating system kernel stacking information, ACK Net Exporter captures the stacking context in the operating system kernel when these events occur, such as the following event:

type=PACKETLOSS pod=hostNetwork namespace=hostNetwork protocol=TCP saddr=10.1.17.172 sport=6443 daddr=10.1.17.176 dport=43018  stacktrace:skb_release_data+0xA3 __kfree_skb+0xE tcp_recvmsg+0x61D inet_recvmsg+0x58 sock_read_iter+0x92 new_sync_read+0xE8 vfs_read+0x89 ksys_read+0x5A

ACK Net Exporter allows you to view events by using multiple methods. For more information, see the Collect monitoring data from ACK Net Exporter topic.

Collect monitoring data from ACK Net Exporter

Scenario 1: Export monitoring data to Prometheus or Grafana and visualize the data

# In the following example, only one endpoint is specified for data collection. 
scrape_configs:
# The job=<job_name> label is added to each time series that is collected based on the configuration. In this example, the job name is set to net-exporter_sample. 
- job_name: "net-exporter_sample"
static_configs:
  - targets: ["{kubernetes pod ip}:9102"]
                

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: kube-system
data:
  prometheus.yml: |-
          # Add the following configuration to the Prometheus server: 
      - job_name: 'net-exporter'
        kubernetes_sd_configs:
          - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_endpoints_name]
          regex: 'net-exporter'
          action: keep

      - job_name: 'kubernetes-pods'

        kubernetes_sd_configs:
        - role: pod

        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

After you add the configuration, the Status > Targets page of the Prometheus server shows the ACK Net Exporter pods that run as normal. You can also enter inspector into the search box on the Graph page of the Prometheus server to view the ACK Net Exporter metrics.

You can configure Grafana to visualize the monitoring data that is collected to Prometheus:

1. In the left-side navigation pane of the Grafana page, choose > Dashboard.
2. On the New dashboard page, click Add an empty panel.
3. In the lower part of the Edit Panel page, enter Prometheus into the Data source field. Then, enter the address of the Prometheus server.
4. Click Metric browser and enter inspector. Then, Grafana displays all available ACK Net Exporter metrics. Click Save in the upper-right part. In the dialog box that appears, click Save. Grafana then displays the visualized data, as shown in the following figure.
   
5. You can configure how the metrics are displayed on a Grafana dashboard based on the configurations that are displayed in the preceding figure. For example, you can use the following configurations to display the increment trend of the inspector_pod_tcppassiveopens metric. This metric indicates the total number of sockets that are created due to handshake requests sent by clients to establish TCP connections within a network space after the system is started or the container is created. To view the increment trend of this metric, use the following configurations:

   // Use the rate() method provided by PromQL to calculate the increment trend of the metric. 
   rate(inspector_pod_tcppassiveopens[1m])
   
   // Use the labels provided by net-exporter to configure a legend to display the metric. 
   {{namespace}}/{{pod}}/{{node}}

Scenario 2: Export monitoring data to ARMS and visualize the data

To export monitoring data from ACK Net Exporter to Application Real-Time Monitoring Service (ARMS) and visualize the data, perform the following steps.

1. Enable Prometheus Service.
2. Configure custom ACK Net Exporter metrics.
   1. Log on to the ARMS console. In the left-side navigation pane, choose Prometheus Service > Prometheus Instances.
   2. In the upper-left corner of the Prometheus Monitoring page, select the region in which your ACK cluster is deployed and click the Prometheus instance that you want to manage. Then, you are redirected to the instance details page. In most cases, the name of the Prometheus instance is the same as the name of your cluster.
   3. In the left-side navigation pane, click Service Discovery and then click the Targets tab. In the lower part of the tab, click kubernetes-pods. The information shows that the custom ACK Net Exporter metrics are configured.
      If kubernetes-pods is not displayed, you need to click the Configurations tab and turn on Default Service Discovery.
   4. In the left-side navigation pane, click Dashboards. Click the corresponding dashboard to log on to Grafana. Click Add panel, select Graph, and then enable the data sources related to your cluster in the Data source section.
   5. Click Metric browser and enter inspector. Then, Grafana automatically displays all available ACK Net Exporter metrics. In the upper-right corner, click Save. In the dialog box that appears, click Save. Grafana then displays the visualized data, as shown in the following figure.

Scenario 3: Export monitoring data to Grafana Loki and visualize the data

You can push anomaly events collected by ACK Net Exporter to your pre-configured Grafana Loki service in real time. This helps you manage these events in a centralized manner. To export monitoring data from ACK Net Exporter to Grafana Loki, perform the following steps.

1. Set up Grafana Loki.
   Note Deploy Grafana Loki in a network that is accessible to the ACK Net Exporter pods. ACK Net Exporter can automatically push event logs to Grafana Loki.
2. On the configuration page of ACK Net Exporter, set enableEventServer to true and lokiServerAddress to the address of the Grafana Loki service. You can specify the IP address or domain name of the Grafana Loki service.
3. Run the following command to access the service address and check whether Grafana Loki is ready:

   curl http://[Address of Grafana Loki]:3100/ready

4. When Grafana Loki is ready, add Grafana Loki as a Grafana data source.
   Open Grafana. In the left-side navigation pane, choose Data source > Loki, enter the address of Grafana Loki, and then click Save&test.
5. In the left-side navigation pane, click Explore. On the top of the page, set the data source to Loki and view the events pushed to Grafana Loki.
   You can view the events of a node by selecting the node from the Label filters drop-down list or specify keywords in the Line containers field to search for specific events.

   You can click Add to dashboard on the top of the page to add a configured event panel to the dashboard.

   The content of the events provided by ACK Net Exporter varies based on the event type. You can check the event details to view the relevant content.

   For more information about the LogQL query language supported by Grafana Loki, see LogQL.

Scenario 4: Use the ACK Net Exporter CLI to collect events

The ACK Net Exporter CLI (inspector-cli) is a scenario-specific troubleshooting and analysis tool developed by the ACK team based on ACK Net Exporter. You can use inspector-cli to collect kernel exception events in real time. inspector-cli can help quickly identify the cause of common exceptions in cloud-native scenarios.

# Launch a temporary container to run inspector-cli. You can replace the image with a later version to update inspector-cli. 
docker run -it --name=inspector-cli --network=host registry.cn-hangzhou.aliyuncs.com/acs/inspector:v0.0.1-12-gff0558c-aliyun

which inspector
# /bin/inspector is the working path of inspector-cli. You can directly run inspector-cli in the container. 

# Set '-e' to specify the address of the event service of ACK Net Exporter. 
inspector watch -e 10.1.16.255

# Expected output: 
 INFO  TCP_RCV_RST_ESTAB Namespace=kube-system Pod=kube-proxy-worker-tbv5s Node=iZbp1jesgumdx66l8ym8j8Z Netns=4026531993 10.1.16.255:43186 -> 100.100.27.15:3128
...

You can also log on to the inspector container of ACK Net Exporter to troubleshoot issues.

# When you run the following command, set the -n parameter to the namespace of net-exporter and specify the net-exporter pod that you want to access. 
kubectl exec -it -n kube-system -c inspector net-exporter-2rvfh -- sh

# Run the following command to view the distribution of network entities on the current node. 
inspector list entity

# Run the following command to listen for network exception events and other relevant information in the local network. 
inspector watch -d -v
#{"time":"2023-02-03T09:01:03.402118044Z","level":"INFO","source":"/go/src/net-exporter/cmd/watch.go:63","msg":"TCPRESET_PROCESS","meta":"hostNetwork/hostNetwork node=izbp1dnsn1bwv9oyu2gaupz netns=ns0 ","event":"protocol=TCP saddr=10.1.17.113 sport=6443 daddr=10.1.17.113 dport=44226  state:TCP_OTHER "}

# You can also specify multiple ACK Net Exporter nodes to view the time when the event occurs on these nodes. 
inspector watch -s 10.1.17.113 -s 10.1.18.14 -d -v
            

How to use ACK Net Exporter to troubleshoot occasional container network issues

This section describes how to troubleshoot occasional network issues in cloud-native scenarios. With the help of ACK Net Exporter, you can quickly obtain information that is required for fixing these issues.

DNS timeout issues

A large number of services in cloud-native environments rely on the DNS resolution service provided by CoreDNS. If a DNS issue correlated to CoreDNS occurs, you need to check the metrics of the CoreDNS pod.

Nginx Ingress 499, 502, 503, and 504 issues

You can check the changes of the preceding metrics at the point in time when the issue occurs to narrow down the scope. If you still cannot locate the cause, Submit a ticket and include the preceding information in your ticket to request technical support.

TCP reset issues

Occasional network latency and jitter issues

Case study

The following cases show how to use ACK Net Exporter to help troubleshoot container network issues.

Case 1: Occasional DNS resolution timeout

Symptom

Customer A submitted a ticket to request technical support to handle DNS resolution timeouts that occasionally occur. The application of Customer A is written in PHP. CoreDNS is configured to perform DNS resolution.

Troubleshooting

1. Obtain DNS metrics from the monitoring system of Customer A.
2. The following situations exist based on the obtained metrics:
   • Each time a DNS resolution timeout occurs, the value of inspector_pod_udpnoports increases by 1. The value of this metric is small.
   • The number of __udp4_lib_rcv packet drops indicated by the inspector_pod_packetloss metric increases by 1. However, the change in the number of packet drops is minor.
3. Customer A specifies that the IP address of the DNS server is a public IP address provided by an Internet service provider (ISP). Based on the obtained metrics, the DNS timeouts occurred because the time required to send the response to the client is long. The response is received after the DNS query times out in user mode.

Case 2: Occasional Java application connection failure

Symptom

Customer B submitted a ticket to request technical support to resolve the following issue: Tomcat occasionally becomes unavailable and the issue lasts 5 to 10 seconds each time.

Troubleshooting

1. The log analysis result shows that the Java runtime was performing a garbage collection operation when the issue occurred.
2. Customer B deployed ACK Net Exporter and analyzed the monitoring data. Customer B found that the value of the inspector_pod_tcpextlistendrops metric increased significantly at the time when the issue occurred.
3. The analysis result shows that request processing was slowed down when the Java runtime performed the garbage collection operation. However, new requests are not throttled. As a result, a large number of connections are created and the backlog of the LISTEN socket is overflowed. This causes the value of the inspector_pod_tcpextlistendrops metric to increase.
4. The issue of TCP connections piling up lasts only a short period of time and the issue is not caused due to the request processing capability of TCP. In this scenario, Customer B modified the Tomcat parameters as we recommended and resolved the issue.

Case 3: Occasional network jitter

Symptom

Customer C submitted a ticket to request technical support to resolve the following issue: The round-trip time between the Redis instance and application significantly increases. As a result, timeout errors occur. The issue cannot be reproduced.

Troubleshooting

1. After Customer C analyzed the log, Customer C identified that the response time of Redis requests occasionally exceeds 300 milliseconds.
2. Customer C also identified that the value of the inspector_node_virtsendcmdlat metric increased at the time when the issue occurred. The affected monitoring levels in Prometheus Service are 15 and 18. After calculation, Customer C identified two virtual calls with a long response time. The response time of the call with a monitoring level of 15 exceeds 36 milliseconds and the response time of the call with a monitoring level of 18 exceeds 200 milliseconds.
3. The kernel occupies the CPU when processing virtual calls. In this case, CPU resources cannot be preempted by other operations. As a result, the execution of virtual calls is slowed down when pods are added or deleted in batches, which further causes the response time to increase.

Case 4: NGINX Ingress occasionally fails to pass health checks

Symptom

Customer D submitted a ticket to request technical support to resolve the following issue: The NGINX Ingress occasionally fails to pass health checks. As a result, request failures occur.

Troubleshooting

1. After Customer D deployed ACK Net Exporter, Customer D identified that the following metrics are abnormal:
   1. The values of the inspector_pod_tcpsummarytcprxqueue and inspector_pod_tcpsummarytcptxqueue metrics increased.
   2. The value of the inspector_pod_tcpexttcptimeouts metric increased.
   3. The value of the inspector_pod_tcpsummarytcptimewaitconn metric decreased and the value of the inspector_pod_tcpsummarytcpestablishedconn metric increased.
2. The analysis result shows that the kernel ran as normal when the issue occurred. Connections are created as normal. However, exceptions occurred when the user process handled the packets in the receive socket and sent packets. In this scenario, the health check failure may be caused by a scheduling or rate limiting issue.
3. Customer D checked the monitoring data of the cgroups as we recommended and identified CPU throttling at the point in time when the health check failure occurred. This indicates that the user process occasionally failed to schedule CPU resources due to a cgroup issue.
4. To resolve this issue, refer to CPU Burst and configure CPU Burst for the NGINX Ingress.

References

ACK Net Exporter metrics

The metrics supported by ACK Net Exporter are constantly updated. For more information, see the instructions on the Marketplace page of the ACK console. All metrics and events provide pod-specific information, except net_softirq and virtcmdlat, which are not related to pods.

ACK Net Exporter events

The following table describes the operating system network-related events that can be captured by using the latest ACK Net Exporter version.

Recommended Grafana configuration file

• If you use a Grafana version later than 8.4.0, click ACK Net Exporter-0.2.9.json to download the Grafana configuration file.
• If you use Grafana 8.4.0 or earlier, click ACK Net Exporter-legacy.json to download the Grafana configuration file.