Monitor Pod Traffic in Real Time with Terway and Hubble - Container Service for Kubernetes - Alibaba Cloud - Container Service for Kubernetes

Deploying Cilium Hubble in an ACK Terway cluster lets you visualize container network traffic and network policies, providing observability into your network architecture and service topology. This topic describes how to use Cilium Hubble to view network traffic data, such as sources and destinations, for network observability.

Prerequisites

Create an ACK managed cluster.

Note

Cilium Hubble currently supports only the Terway ENI multi-IP network mode, which is based on IPvlan. Therefore, when you create a cluster, you must set the Network Plug-in to Terway and the Terway Mode to IPvlan. Otherwise, you cannot use Cilium Hubble.

Step 1: Configure the eni-config ConfigMap

Console

Edit the eni-config ConfigMap for Terway.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Configurations > ConfigMaps.
On the ConfigMaps page, select kube-system from the Namespace drop-down list. Find eni-config and click Edit YAML in the Actions column.

In the View in YAML panel, find 10-terway.conf, add the following parameters to 10-terway.conf, and click OK.

"cilium_enable_hubble": "true",
"cilium_hubble_listen_address": ":4244",
"cilium_hubble_metrics_server": ":9091",
"cilium_hubble_metrics": "drop,tcp,flow,port-distribution,icmp",

Parameter	Description	Remarks
eniip_virtual_type	Specifies whether to enable IPvlan mode.	If this parameter is not present or its value is not `IPVLAN`, your cluster does not support Cilium Hubble.
cilium_enable_hubble	Specifies whether to enable network traffic analysis with Hubble.	In this topic, this parameter is set to `"true"`.
cilium_hubble_listen_address	The listening address for Hubble network traffic information.	In this topic, this parameter is set to `":4244"`.
cilium_hubble_metrics_server	The listening address for Hubble metrics.	In this topic, this parameter is set to `":9091"`.
cilium_hubble_metrics	The metrics to collect. Separate multiple metrics with a comma (,).	Layer 7 capabilities such as DNS and HTTP are not supported. The complete list of metrics that Hubble supports is `"drop,tcp,flow,port-distribution,icmp"`. Note Enabling too many metrics can affect Hubble performance.

Restart the Terway pods to apply the changes.
1. In the left-side navigation pane, choose Workloads > Pods.
2. On the Pods page, select kube-system from the Namespace drop-down list and enter terway-eniip in the search box. For each matching pod, choose More > Delete in the Actions column.
3. In the confirmation dialog box that appears, click OK.
  On the Pods page, wait for the Status of the new terway-eniip-xxx pod to become Running. This indicates that the pod has restarted successfully.
4. Repeat these steps to delete all Terway pods.

Kubectl

Edit the eni-config ConfigMap.

Run the following command to edit the eni-config ConfigMap:
```
kubectl -n kube-system edit configmap eni-config
```

In the 10-terway.conf data block, add the following parameters. Then, save the changes and exit.

"cilium_enable_hubble": "true",
"cilium_hubble_listen_address": ":4244",
"cilium_hubble_metrics_server": ":9091",
"cilium_hubble_metrics": "drop,tcp,flow,port-distribution,icmp",

Parameter	Description	Remarks
eniip_virtual_type	Specifies whether to enable IPvlan mode.	If this parameter is not present or its value is not `IPvlan`, your cluster does not support Cilium Hubble.
cilium_enable_hubble	Specifies whether to enable network traffic analysis with Hubble.	In this topic, this parameter is set to `"true"`.
cilium_hubble_listen_address	The listening address for Hubble network traffic information.	In this topic, this parameter is set to `":4244"`.
cilium_hubble_metrics_server	The listening address for Hubble metrics.	In this topic, this parameter is set to `":9091"`.
cilium_hubble_metrics	The metrics to collect. Separate multiple metrics with a comma (,).	Layer 7 capabilities such as DNS and HTTP are not supported. The complete list of metrics that Hubble supports is `"drop,tcp,flow,port-distribution,icmp"`. Note Enabling too many metrics can affect Hubble performance.

Restart the Terway pods to apply the changes.

kubectl rollout restart daemonset -n kube-system terway-eniip

Step 2: Install Cilium Hubble

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of your cluster. In the left navigation pane, click Applications > Helm.

On the Helm page, click Deploy. Configure the Basic Information parameters.

Parameter	Example value
Release Name	ack-terway-hubble
Namespace	kube-system
Source	The default value is Marketplace.
Chart	For Use Scenarios, select All. For Supported Architecture, select amd64. In the search box, enter ack-terway-hubble.

Select ack-terway-hubble and click Next.

On the Parameters page, select a Chart Version. In the Parameter area, find the ingress section, set the hosts value to your Hubble UI access domain, and then click OK.

Note

The following parameters are located under the ingress section.

Parameter	Description
enabled	Specifies whether to enable Ingress access for the Hubble UI.
annotations	Custom Ingress annotations.
path	Custom Ingress root path.
hosts	Custom Ingress hosts.
tls	Custom Ingress TLS configuration.

Step 3: Use Cilium Hubble

If you configured the Ingress hosts parameter for Cilium Hubble, you can access the Hubble UI by navigating to that domain on port 80. The UI displays the following information:

Note

If you use a domain name that is not publicly resolvable, such as ingress.local, run the kubectl -n kube-system get svc nginx-ingress-lb command to get the IP address of the Hubble UI. Then, add an entry to your local hosts file to map ingress.local to this IP address.

The top section displays a service map of the communication topology between pods and services across your cluster's namespaces.
The bottom section provides details about the current network traffic, including sources, destinations, port information, and forwarding status.
If you have network policies configured, you can see which traffic flows are dropped by those policies.

可观测

Cilium Hubble exposes network traffic metrics through the hubble-metrics service in the kube-system namespace. The cilium_hubble_metrics field in the eni-config ConfigMap controls which metrics are exposed. You can collect this data using open source Prometheus or Managed Service for Prometheus. For more information, see Use open source Prometheus for monitoring and Connect to and configure Managed Service for Prometheus.

Note

For more information about the metrics exposed by Cilium Hubble, see Hubble Exported Metrics.

Category	Metric	Label	Description
drop	hubble_drop_total	reason, protocol	Total dropped packets.
tcp	hubble_tcp_flags_total	flag, family	Total TCP flags observed.
flow	hubble_flows_processed_total	type, subtype, verdict	Total processed network flows.
port-distribution	hubble_port_distribution_total	protocol, port	Packet distribution by destination port.
icmp	hubble_icmp_total	family, type	Total ICMP packets.