Container Service for Kubernetes:Use KubeSkoop to troubleshoot network issues

Install the ACK KubeSkoop add-on

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. On the Clusters page, click the name of your cluster. In the left navigation pane, click Add-ons.

3. On the Add-ons page, search for ACK KubeSkoop, find the add-on, and click Install.

4. On the Install Component ACK KubeSkoop page, click Confirm.

Configure the KubeSkoop add-on

KubeSkoop is configured through a ConfigMap named kubeskoop-config in the ack-kubeskoop namespace. Changes take effect immediately—no restart is required.

Option 1: Edit via kubectl

kubectl edit cm kubeskoop-config -n ack-kubeskoop

Option 2: Edit via the console

1. Log on to the ACK console. In the left navigation pane, click Clusters.

2. Click the name of your cluster. In the left navigation pane, choose Configurations > ConfigMaps.

3. On the ConfigMaps page, set Namespace to ack-kubeskoop, search for kubeskoop-config, and click Edit in the Actions column.

4. In the Edit panel, update the parameters and click OK.

The following table describes the supported parameters.

Set up the ARMS Prometheus dashboard

1. Log on to the ARMS console. In the left navigation pane, click Integration Management.

2. On the Integration Management page, click Add Integration. Search for KubeSkoop and click ACK KubeSkoop Network Monitoring.

3. In the ACK KubeSkoop Network Monitoring dialog box, select the ACK cluster, enter an Integration Name, and click OK.

4. Log on to the ACK console. Click the cluster name. In the left navigation pane, choose Operations > Prometheus Monitoring.

5. Click the Others tab. The node and Pod monitoring dashboards created by KubeSkoop appear in the dashboard list.

For more information about Alibaba Cloud Prometheus Service, see Use Alibaba Cloud Prometheus Service.

Troubleshoot DNS timeouts

DNS timeouts in container environments typically have one of three root causes:

• The DNS server is slow to respond and misses the client-side timeout.

• The sender cannot dispatch the DNS query packet promptly.

• The server responds in time, but the sender drops the response due to resource constraints such as insufficient memory.

Because most cloud-native workloads rely on CoreDNS, monitor the following metrics for both your application Pods and any CoreDNS Pods.

Troubleshoot Nginx Ingress 499/502/503/504 errors

These four status codes each point to a different failure layer.

Collect baseline information first. Before checking KubeSkoop metrics, gather:

• Nginx access_log, specifically request_time, upstream_connect_time, and upstream_response_time

• Nginx error_log entries around the time of the issue

• Liveness and readiness probe status if configured

If you suspect connection failures, check for changes in these metrics:

If Nginx responses are slow (for example, request_time is long but upstream_response_time is short), check for queue buildup:

Troubleshoot TCP resets

A TCP reset surfaces in applications as a connection reset by peer error (common in C-based apps like Nginx) or a Broken pipe error (common in Java or Python apps that use TCP connection wrappers).

Common causes in cloud-native environments:

• Server-side resource exhaustion, such as insufficient TCP memory, triggers a proactive reset.

• Load Balancing or a Kubernetes Service forwards traffic to an unexpected backend due to anomalies in Endpoint selection or conntrack state.

• Security policies close the connection.

• In NAT environments or under high concurrency, Protection Against Wrapped Sequence Numbers (PAWS) or sequence number wraparound occurs.

• TCP keepalive expires because no business traffic has passed for an extended period.

Start by mapping the network topology between the client and server at the time the reset occurs. Then check the following metrics:

Troubleshoot network latency jitter

Intermittent latency spikes in container networks often have non-obvious causes. Although the symptoms appear as network delays, the root cause is frequently an operating system scheduling or resource issue.

Common causes:

• A real-time process managed by the RT scheduler runs for too long, starving business processes or network kernel threads.

• The workload experiences occasional slow external calls, such as high-latency cloud disk I/O or intermittent RDS Round-Trip Time (RTT) spikes.

• Uneven load between CPUs or NUMA nodes causes some CPUs to lag.

• Stateful kernel mechanisms such as the conntrack confirm operation, or a large number of orphan sockets, slow down socket lookups.

Monitor the following metrics to narrow down the cause:

Case 1: Intermittent DNS timeouts

Problem: A customer running a PHP application experienced intermittent DNS resolution timeouts. The DNS service was CoreDNS, with the configured upstream resolver pointing to a public provider.

Findings: During error windows:

• kubeskoop_udp_noports increased by 1.

• kubeskoop_packetloss_total increased by 1.

Both changes were small in absolute terms.

Root cause: The public DNS server was responding slowly. The response arrived after the PHP application had already timed out on the client side. The small metric increments were consistent with occasional, not systematic, packet delays.

Case 2: Intermittent connection failures in a Java application

Problem: A customer's Tomcat service became unavailable intermittently, with each outage lasting approximately 5 to 10 seconds.

Findings: Log analysis confirmed that a Garbage Collection (GC) pause was occurring at the time of each incident. After deploying KubeSkoop, kubeskoop_tcpext_listendrops increased significantly during GC events.

Root cause: During GC, request processing slowed and connections were not released quickly. Incoming connection requests continued to accumulate, filling the listen socket's backlog and causing it to overflow, which triggered the spike in kubeskoop_tcpext_listendrops.

Resolution: The customer adjusted Tomcat's connection backlog parameters, which resolved the issue.

Case 3: Intermittent RTT spikes to Redis

Problem: A customer observed intermittent Redis requests with total response times exceeding 300 ms, but could not reproduce the issue on demand.

Findings: After deploying KubeSkoop, kubeskoop_virtcmdlatency_latency increased during the latency windows:

• Bucket le=15 (threshold ~36 ms) increased, indicating at least one call exceeded 36 ms.

• Bucket le=18 (threshold ~200 ms) increased, indicating a call exceeding 200 ms.

Root cause: Kernel virtualization calls triggered during batch Pod creation and deletion occupied CPU cores and could not be preempted, causing the customer's intermittent latency.

Case 4: Intermittent health check failures for Nginx Ingress

Problem: Nginx Ingress health checks failed intermittently, accompanied by business request failures.

Findings: At the time of each failure:

• kubeskoop_tcpsummary_tcprxqueue and kubeskoop_tcpsummary_tcptxqueue both increased.

• kubeskoop_tcpext_tcptimeouts increased.

• kubeskoop_tcpsummary_tcptimewaitconn decreased while kubeskoop_tcpsummary_tcpestablishedconn increased.

The kernel and connection establishment paths appeared normal, but the user-space process was not consuming packets from the receive queue or flushing the send queue.

Root cause: cgroup CPU throttling was intermittently preventing the Ingress process from being scheduled, stalling both packet processing and transmission.

Resolution: The customer enabled CPU Burst for the Nginx Ingress Pods by following Enable CPU Burst, which eliminated the throttling.