Diagnose ASM Traffic Anomalies via Packet Capture Tasks - Alibaba Cloud Service Mesh

Service Mesh (ASM) allows you to initiate packet inspection tasks for workloads deployed in an ASM instance to obtain information about TCP packets in pods within a certain period of time and dump the TCP packets to the hosts of the nodes in a Kubernetes cluster for downloading and viewing. This topic describes how to diagnose the traffic in an ASM instance by using a packet inspection task.

Background information

Some traffic may not meet expectations due to the diversity of applications managed by ASM. For example, request headers are too large and requests do not comply with the standard HTTP protocol. Packet inspection tasks can quickly capture traffic information for workloads, helping you diagnose complex traffic issues.

Prerequisites

An ASM instance of Enterprise Edition or Ultimate Edition, running version 1.21.6.72 or later. For more information, see Create an ASM instance or Update an ASM instance.
A Container Service for Kubernetes (ACK) cluster added to the ASM instance. For more information, see Add a cluster to an ASM instance.

Create a packet inspection task

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Mesh Diagnosis.
On the Mesh Diagnosis page, click Service Packet Inspection.

On the Service Packet Inspection tab, click Create Task. In the dialog box, configure the following parameters:

Parameter	Description
Kubernetes Clusters	The ACK cluster where the target pod is deployed.
Namespace	The namespace of the target pod.
Pod Name	The pod to capture packets from.
Duration	How long to capture packets.
tcpdump Parameters	Filters that control which packets are captured. See Common tcpdump filters for examples. For the full parameter reference, see tcpdump(1).
Save File Name	The name of the output pcap file.

Click OK.
Wait until the Status column of the task changes to Complete.
Note the values in the Host of File and File Save Path columns. Connect to the corresponding Kubernetes node and download the pcap file from the specified path. For details, see Use Workbench to upload a file to and download a file from an instance.

Common tcpdump filters

Each filter targets a specific segment of the sidecar traffic flow. The following diagram shows how traffic moves through a pod with a sidecar proxy:

Client pod                                Server pod
+--------------------------+     +--------------------------+
|                          |     |                          |
|  App -(lo)-> Sidecar     |     |  Sidecar -(lo)-> App     |
|        (port 15001)      |     |      (127.0.0.6)         |
|                          |     |                          |
|  Sidecar -(any)->        |     |        <-(any)- Sidecar  |
|   [destination service]  |     |   [client pod]           |
+--------------------------+     +--------------------------+

Use the filters below based on which traffic segment you need to inspect.

Client pod: application to sidecar proxy

Capture traffic between the application and the outbound sidecar proxy listener (port 15001) inside a client pod:

# Replace 172.16.xx.xx with the IP address of the destination service.
-i lo dst port 15001 or src host 172.16.xx.xx

This filter uses the loopback interface (lo) because app-to-sidecar communication happens within the same pod over localhost.

Client pod: sidecar proxy to destination service

Capture traffic leaving the client pod's sidecar proxy toward a destination service:

# Replace 192.168.xx.xx with the IP address of the pod that runs the destination service.
-i any host 192.168.xx.xx

This filter uses -i any because the traffic exits the pod through a network interface.

Server pod: sidecar proxy to application

Capture inbound traffic that the sidecar proxy forwards to the application inside a server pod:

-i lo host 127.0.0.6

Server pod: client pod to sidecar proxy

Capture traffic arriving at a server pod from a specific client pod:

# Replace 192.168.xx.xx with the IP address of the client pod.
-i any host 192.168.xx.xx

Delete a packet inspection task

Log on to the ASM console. In the left-side navigation pane, choose Service Mesh > Mesh Management.
On the Mesh Management page, click the name of the ASM instance. In the left-side navigation pane, choose ASM Instance > Mesh Diagnosis.
On the Mesh Diagnosis page, click the Service Packet Inspection tab.
Find the task and click Delete in the Actions column. In the confirmation message, click OK.

Alibaba Cloud Service Mesh:Diagnose traffic in an ASM instance with packet inspection