ACK KubeSkoop (formerly ACK Net Exporter) is a network monitoring and troubleshooting suite open-sourced by Alibaba Cloud Container Service for Kubernetes (ACK). It helps you monitor and quickly troubleshoot complex network issues in your clusters. The best practices for using KubeSkoop in ACK managed cluster help you get started and resolve real-world issues.
Background information
KubeSkoop provides eBPF-based capabilities such as in-depth network monitoring, network connectivity diagnostics, packet capture, and latency probing, and exposes Prometheus metrics and anomaly events. KubeSkoop runs on nodes as daemon Pods. It uses eBPF to collect node information and aggregate the information into specific Pods, and provides a standardized interface for observing high-level network information. The following figure shows the core architecture of KubeSkoop.
Install and configure the ACK KubeSkoop add-on
Install the ACK KubeSkoop add-on
-
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of your cluster. In the left navigation pane, click Components and Add-ons .
-
On the Add-ons page, search for ACK KubeSkoop, find the add-on, and click Install.
-
On the InstallACK KubeSkoop page, click Confirm.
Configure the KubeSkoop add-on
-
You can run the following command to configure the KubeSkoop add-on by using a ConfigMap.
kubectl edit cm kubeskoop-config -n ack-kubeskoop -
You can also configure the KubeSkoop add-on in the console.
-
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
On the ConfigMaps page, set Namespace to ack-kubeskoop, and then search for kubeskoop-config. For kubeskoop-config, in the Actions column on the right side, click Edit.
-
In the Edit panel, configure the parameters, and then click OK. KubeSkoop supports the following configuration items:
Parameter
Description
Default value
debugmode
Specifies whether to enable debug mode. Valid values:
-
false: disables debug mode.
-
true: enables debug mode, which supports DEBUG-level logs, debug interfaces, and the Go pprof and gops diagnostic tools.
falseport
The port on which the metric service provides HTTP access.
9102enableController
Specifies whether to enable the controller component. The controller interacts with the Kubernetes API to perform monitoring or management tasks.
truecontrollerAddr
The address of the controller service, which is used to locate the KubeSkoop controller component.
dns:kubeskoop-controller:10263metrics.probes
The list of metric types to collect. Each probe corresponds to a metric category.
metrics: probes: - name: conntrack - name: qdisc - name: netdev - name: io - name: sock - name: tcpsummary - name: tcp - name: tcpext - name: udp - name: rdmaFor more information about each probe, see Probes, Metrics, and Events.
After the ConfigMap is updated, you do not need to restart the ACK KubeSkoop add-on. The add-on automatically hot-reloads the changes and enables or disables the corresponding probes.
-
-
Configure the Application Real-Time Monitoring Service (ARMS) Prometheus dashboard
-
Log on to the ARMS console.
-
In the navigation pane on the left, click Integration Management.
-
On the Integration Management page, click Add Integration, search for KubeSkoop in the search box of the Integration Center, and click ACK KubeSkoop Network Monitor.
-
On the ACK KubeSkoop Network Monitor page that appears, select the Container Service for Kubernetes cluster that you want to integrate, enter an Integration Name, and then click OK to enable KubeSkoop monitoring.
-
Log on to the ACK console. In the left navigation pane, click Clusters.
-
On the Clusters page, click the name of your cluster. In the left navigation pane, click .
-
Click the Others tab. In the dashboard list, you can view the node and Pod monitoring dashboards created by KubeSkoop.On the Prometheus Monitoring page, click the Others tab, and find the Ack KubeSkoop Network Monitor - Node and Ack KubeSkoop Network Monitor - Pod tabs to view the KubeSkoop network monitoring dashboard.
For more information about Managed Service for Prometheus, see Configure Managed Service for Prometheus.
Use KubeSkoop
Manually view KubeSkoop monitoring metrics
KubeSkoop provides monitoring data in Prometheus format. After you install KubeSkoop, you can access the service port of any KubeSkoop Pod instance to obtain all metric information:
-
Run the following command to obtain all KubeSkoop instances.
kubectl get pod -n ack-kubeskoop -o wide | grep kubeskoop-agentExpected output:
kubeskoop-agent-2chvw 1/1 Running 0 43m 172.16.16.xxx cn-hangzhou.172.16.16.xxx <none> <none> kubeskoop-agent-2qtbf 1/1 Running 0 43m 172.16.16.xxx cn-hangzhou.172.16.16.xxx <none> <none> kubeskoop-agent-72pgf 1/1 Running 0 43m 172.16.16.xxx cn-hangzhou.172.16.16.xxx <none> <none> -
Run the following command to obtain the metric information. Replace
172.16.xxx.xxxin the command with the address of the KubeSkoop instance that you obtained in the previous step.curl http://172.16.16.xxx:9102/metrics
The monitoring metric data provided by KubeSkoop is in the following format:
kubeskoop_netdev_rxbytes{k8s_namespace="",k8s_node="cn-hangzhou.172.16.16.xxx",k8s_pod=""} 2.970963745e+09
How to use ACK KubeSkoop to troubleshoot intermittent complex issues in container networks
The following content provides troubleshooting guides for some typical cloud-native issues. With ACK KubeSkoop, these guides help you quickly obtain information related to the issue.
DNS timeout issues
In cloud-native environments, DNS service timeouts cause service access failures. Common causes of DNS access timeouts include:
-
The DNS service responds slowly and cannot complete a DNS query before the application timeout is reached.
-
The sender fails to send the DNS query packet smoothly or in time because of an issue on the sender side.
-
The server responds to the packet in time, but packets are dropped because of issues such as insufficient memory on the sender.
You can use the following metrics to help troubleshoot intermittent DNS timeout issues:
|
Metric |
Description |
|
kubeskoop_pod_udpsndbuferrors |
The number of errors that occur when UDP sends packets through network layer methods. |
|
kubeskoop_pod_udpincsumerrors |
The number of CSUM verification errors that occur when UDP receives packets. |
|
kubeskoop_pod_udpnoports |
The number of times that the network layer call |
|
kubeskoop_pod_udpinerrors |
The number of errors that occur when UDP receives packets. |
|
kubeskoop_pod_udpoutdatagrams |
The number of packets that UDP successfully sends through network layer methods. |
|
kubeskoop_pod_udprcvbuferrors |
The number of errors that occur when UDP copies application layer protocol data to the socket receive queue and the queue has insufficient space. |
In cloud-native environments, many services rely on CoreDNS for DNS resolution. When a DNS issue occurs, if the failed DNS request involves CoreDNS, you must also check the preceding metrics of the CoreDNS Pods for anomalies.
Nginx Ingress HTTP response status code 499/502/503/504 issues
In cloud-native environments, intermittent exceptions in an Ingress gateway or in other proxy services that act as a proxy or broker are common complex issues. For Nginx Ingress and other proxy services built on Nginx, 499/502/503/504 are the four most common types of issues. They indicate the following:
-
499: The client that sends requests to Nginx closes the TCP connection before Nginx responds. Common causes include:-
After the client connection is established, the client sends the request late, so the client timeout is reached while Nginx is responding. This is common in asynchronous request frameworks of Android clients.
-
The server processes the connection slowly after the connection is established, which requires further troubleshooting.
-
The backend responds slowly when the server sends a request to the upstream backend.
-
-
502: This scenario is commonly caused by connection-level issues between Nginx and the upstream backend, such as connection establishment failures or abnormal closures by the backend. Common causes include:-
DNS resolution fails for the domain name that is configured for the backend. This usually occurs when a Kubernetes Service is used as the backend.
-
A connection to the upstream fails to be established.
-
The request or response of the upstream is too large to be allocated memory, which interrupts normal service interactions.
-
-
503: In Nginx, this status code indicates to the client that all upstreams are unavailable. In cloud-native scenarios, it has some specific meanings. Common causes include:-
No backend is available. This situation is rare.
-
The traffic volume is too high and is throttled by the Limit Req feature of the Ingress.
-
-
504: This status code indicates that the service packets related to the upstream time out in Nginx. A common cause is that the service packets returned by the upstream do not arrive in time.
When you encounter the preceding types of issues, first collect some general information to determine the possible scope of the issue and the next troubleshooting direction:
-
The access_log information provided by Nginx, especially
request_time,upstream_connect_time, andupstream_response_time. -
The error_log information provided by Nginx, especially whether abnormal error messages appear when the issue occurs.
-
The health check results, if liveness or readiness health checks are configured.
Based on the preceding information, if connection failures may occur, monitor changes in the following metrics according to the specific symptoms of the issue:
|
Metric |
Description |
|
kubeskoop_tcpext_listenoverflow |
The number of times that the half-open connection queue overflows when a socket in the LISTEN state accepts a connection. |
|
kubeskoop_tcpext_listendrops |
The number of times that a socket in the LISTEN state fails to create a socket in the SYN_RECV state. |
|
kubeskoop_netdev_txdropped |
The number of NIC send errors that result in dropped packets. |
|
kubeskoop_netdev_rxdropped |
The number of NIC receive errors that result in dropped packets. |
|
kubeskoop_tcp_activeopens |
The number of times that TCP in a single Pod successfully initiates the first SYN handshake. SYN retransmissions are not included, but failed connection establishment also increases this metric. |
|
kubeskoop_tcp_passiveopens |
The cumulative number of times that TCP in a single Pod completes the handshake and successfully allocates a socket. This metric can generally be understood as the number of connections that are successfully established. |
|
kubeskoop_tcp_retranssegs |
The total number of retransmitted packets in a single Pod. The count is calculated in advance based on TSO segmentation. |
|
kubeskoop_tcp_estabresets |
The number of times that TCP connections are abnormally closed in a single Pod. This metric counts only the outcome. |
|
kubeskoop_tcp_outrsts |
The number of Reset packets that TCP sends in a single Pod. |
|
kubeskoop_conntrack_invalid |
The number of times that a conntrack (CT) entry fails to be created for various reasons but the packet is not dropped. |
|
kubeskoop_conntrack_drop |
The number of packets that are dropped because a conntrack (CT) entry fails to be created. |
If an issue similar to slow Nginx responses occurs, for example, a timeout occurs but the request_time of Nginx is short, you can monitor changes in the following metrics:
|
Metric |
Description |
|
kubeskoop_tcpsummary_tcpestablishedconn |
The number of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_pod_tcpsummarytcptimewaitconn |
The number of TCP connections that are currently in the TIME_WAIT state. |
|
kubeskoop_tcpsummary_tcptimewaitconn |
The total number of bytes of packets in the send queue of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_tcpsummary_tcprxqueue |
The total number of bytes of packets in the receive queue of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_tcpext_tcpretransfail |
The number of times that a retransmitted packet returns an error other than EBUSY, which indicates that the retransmission cannot complete normally. |
Based on the preceding metric information, you can narrow the troubleshooting scope by examining how the metrics change at the time when the issue occurs.
TCP Reset packet issues
A TCP Reset packet is an action in the TCP protocol that responds to unexpected situations. It usually has the following impacts on user applications:
-
connection reset by peererror, which usually occurs in services that depend on the C library, such as Nginx. -
Broken pipeerror, which usually occurs in services that encapsulate TCP connections, such as Java or Python applications.
In cloud-native network environments, Reset packets have many common causes. The following lists several common causes of Reset packets:
-
Server-side exceptions that prevent normal service delivery, such as insufficient memory allocated to TCP. In these cases, a Reset packet is usually sent actively.
-
Traffic forwarded to an unexpected backend because a stateful mechanism such as Endpoint or conntrack becomes abnormal when a Service or load balancer is used.
-
Connection release caused by security reasons.
-
Protection Against Wrapped Sequence Numbers (PAWS) or sequence number wrapping in scenarios such as NAT environments and high concurrency.
-
TCP Keepalive used to keep a connection alive while no normal service communication occurs for a long time.
To quickly distinguish between the preceding root causes, you can collect some basic information and metrics:
-
Sort out the network topology between the client and the server when the Reset packet is generated.
-
Monitor changes in the following metrics:
Metric
Description
kubeskoop_tcpext_tcpabortontimeout
The number of times that a Reset packet is sent because keepalive, window probe, or retransmission calls exceed the upper limit.
kubeskoop_tcpext_tcpabortonlinger
The number of times that a Reset packet is sent to quickly recycle a connection in the FIN_WAIT2 state after the TCP Linger2 option is enabled.
kubeskoop_tcpext_tcpabortonclose
The number of times that a Reset packet is sent because data remains unread when a TCP connection is closed for reasons outside the state machine.
kubeskoop_tcpext_tcpabortonmemory
The number of times that a Reset packet is sent to end a connection because tcp_check_oom triggers insufficient memory in the logic that allocates tw_sock or tcp_sock.
The number of Reset packets that are sent to quickly recycle a connection through Reset when the Linger or Linger2 option is enabled.
kubeskoop_tcpext_tcpackskippedsynrecv
The number of times that a socket in the SYN_RECV state does not reply with an ACK packet.
kubeskoop_tcpext_tcpackskippedpaws
The number of times that the PAWS mechanism triggers a correction but OOW rate limiting restricts the ACK packet from being sent.
kubeskoop_tcp_estabresets
The number of times that TCP connections are abnormally closed in a single Pod. This metric counts only the outcome.
kubeskoop_tcp_outrsts
The number of Reset packets that TCP sends in a single Pod.
Intermittent network latency jitter issues
Intermittent network latency jitter is the most common and hardest type of issue to locate in cloud-native environments. It has a wide variety of causes, and latency can also trigger the three types of issues described earlier. In container network scenarios, network latency inside a node is usually caused by one of the following reasons:
-
A real-time process managed by the RT scheduler runs for too long, which causes long queues or slow processing in service processes or network kernel threads.
-
The process intermittently makes slow external calls, such as slow cloud disk responses or intermittent increases in the RTT of RDS, which slows down request processing.
-
Node configuration issues cause unbalanced loads across different CPUs or NUMA nodes, and the heavily loaded system stalls.
-
Stateful kernel mechanisms cause latency, such as the conntrack confirm operation or a large number of orphan sockets that affect normal socket lookups.
For this type of issue, although it appears to be a network issue, the root cause usually lies elsewhere in the OS. You can monitor the following metrics to narrow the troubleshooting scope:
|
Metric |
Description |
|
kubeskoop_io_ioreadsyscall |
The number of file system read operations, such as read and pread, that a process performs. |
|
kubeskoop_io_iowritesyscall |
The number of file system write operations, such as write and pwrite, that a process performs. |
|
kubeskoop_io_ioreadbytes |
The number of bytes that a process reads from the file system, usually from a block device. |
|
kubeskoop_io_iowritebytes |
The number of bytes that a process writes to the file system. |
|
kubeskoop_tcpext_tcptimeouts |
The number of retransmissions that are performed when a SYN packet receives no reply. This metric is triggered when the CA state does not enter recovery, loss, or disorder. |
|
kubeskoop_tcpsummary_tcpestablishedconn |
The number of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_tcpsummary_tcptimewaitconn |
The number of TCP connections that are currently in the TIME_WAIT state. |
|
kubeskoop_tcpsummary_tcptxqueue |
The total number of bytes of packets in the send queue of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_tcpsummary_tcprxqueue |
The total number of bytes of packets in the receive queue of TCP connections that are currently in the ESTABLISHED state. |
|
kubeskoop_softnet_processed |
The number of packets that all CPUs in a single Pod process from the CPU backlog that the NIC fills. |
|
kubeskoop_softnet_dropped |
The number of packets that all CPUs in a single Pod drop. |
Customer case studies
The following cases show how some container network customers used ACK KubeSkoop to troubleshoot and analyze complex issues. You can compare the symptoms for reference.
Case 1: Intermittent DNS timeout issues
Symptoms
Customer A detected an anomaly: intermittent DNS resolution timeouts occurred. The customer's service uses PHP, and CoreDNS is configured as the DNS service.
Troubleshooting process
-
Based on the customer's description, communicate with the customer to obtain the DNS-related data from the customer's monitoring system.
-
Analyze the data within the time range when the errors were reported and identify the following issues.
-
Within the error time range,
kubeskoop_udp_noportsincreased by 1, and the overall metric value is small. -
kubeskoop_packetloss_totalincreased by 1, and the number of dropped packets changed slightly.
-
-
The customer reported that the configured DNS address belongs to a service provider on the public network. Based on the monitoring information, the root cause of the customer's errors is that DNS requests return slowly: the DNS response packet arrives after the user-space timeout returns.
Case 2: Intermittent connection failures in a Java application
Symptoms
Customer B detected an anomaly: Tomcat became unavailable intermittently, and each occurrence lasted about 5 to 10 seconds.
Troubleshooting process
-
Log analysis confirmed that the customer's Java Runtime was performing GC when the issue occurred.
-
After KubeSkoop monitoring was deployed, the
kubeskoop_tcpext_listendropsmetric increased significantly at the time of the issue. -
The analysis concluded that when the customer's Java Runtime performed GC, requests were processed more slowly, which slowed down request release. However, the creation of new requests was not limited, so a large number of connections were generated. This filled the backlog of the listen socket and caused an overflow, which led to an increase in
kubeskoop_tcpext_listendrops. -
In this case, the customer's connection accumulation lasted a short time and the processing capacity itself was normal. We suggested that the customer adjust the Tomcat parameters, which resolved the issue.
Case 3: Intermittent network latency jitter for a customer
Symptoms
Customer C detected an anomaly: requests between the customer's application and Redis intermittently experienced increased RTT, which caused service timeouts. The issue could not be reproduced.
Troubleshooting process
-
Log analysis showed that the total response time of Redis requests intermittently exceeded 300 ms for the customer.
-
After KubeSkoop was deployed, monitoring showed that the kubeskoop_virtcmdlatency_latency metric increased when the issue occurred. The increased le values (Level in Prometheus monitoring) were 18 and 15. After conversion, two virtualization calls with high latency occurred at that time: le 15 produced latency higher than 36 ms, and le 18 produced latency higher than 200 ms.
-
Because kernel virtualization calls occupy the CPU while they run and cannot be preempted, the customer's intermittent latency was caused by some virtualization calls that ran too long during the bulk creation and deletion of Pods.
Case 4: Intermittent health check failures on Ingress Nginx
Symptoms
The Ingress machines intermittently failed health checks, which was accompanied by service request failures.
Troubleshooting process
-
After monitoring was deployed, the issue was found to be abnormal changes in multiple metrics.
-
kubeskoop_tcpsummary_tcprxqueueandkubeskoop_tcpsummary_tcptxqueueboth increased. -
kubeskoop_tcpext_tcptimeoutsincreased. -
kubeskoop_tcpsummary_tcptimewaitconndecreased, andkubeskoop_tcpsummary_tcpestablishedconnincreased.
-
-
The analysis confirmed that at the time of the issue, the kernel worked normally and connections were established normally, but the process behaved abnormally when it processed the packets received in sockets and when it performed the actual packet sending operations. We inferred that the user process had a scheduling or throttling issue at that time.
-
After we suggested that the customer check the Cgroup monitoring, the customer found that CPU Throttled occurred at the time of the issue. This confirmed that the user process intermittently failed to be scheduled because of Cgroup.
-
For more information, see Enable the CPU Burst performance optimization policy. After the CPU Burst feature was configured for the Ingress, this type of issue was resolved.