Implement network observability by using ACK Terway and Cilium Hubble - Container Service for Kubernetes

Deploy Cilium Hubble in a Container Service for Kubernetes (ACK) cluster that uses the Terway network plug-in to gain visibility into container network traffic. Cilium Hubble shows packet sources and destinations, pod and Service topologies, network policy enforcement, and flow-level metrics across your cluster.

Prerequisites

Before you begin, make sure that you have:

An ACK managed cluster with the following network settings:
- Network Plug-in: Terway
- Terway Mode: IPvlan (One ENI for Multi-Pod mode, based on IPVLAN)

Important

Cilium Hubble supports only the One ENI for Multi-Pod mode of Terway. If your cluster uses a different Terway mode, Cilium Hubble cannot be enabled. When you create the ACK cluster, select Terway for the Network Plug-in parameter and IPvlan for the Terway Mode parameter.

Limitations

No Layer 7 metrics: HTTP and DNS metrics are not supported. Only Layer 3/4 metrics (drop, tcp, flow, port-distribution, icmp) are available.
Performance impact from excessive metrics: Specifying too many metrics in the cilium_hubble_metrics parameter can degrade Cilium Hubble performance.
Architecture: The ack-terway-hubble Helm chart only supports the amd64 architecture.

Enable Cilium Hubble in the Terway ConfigMap

Enable Cilium Hubble by adding parameters to the 10-terway.conf section of the eni-config ConfigMap in the kube-system namespace.

Add the following parameters to 10-terway.conf:

"cilium_enable_hubble":"true",
"cilium_hubble_listen_address":":4244",
"cilium_hubble_metrics_server":":9091",
"cilium_hubble_metrics":"drop,tcp,flow,port-distribution,icmp",

ConfigMap parameters

Parameter	Description	Remarks
`eniip_virtual_type`	Specifies whether to enable the IPVLAN mode.	If this parameter is absent or not set to `IPVLAN`, the cluster does not support Cilium Hubble.
`cilium_enable_hubble`	Enables Cilium Hubble for network traffic analysis.	In this example, this parameter is set to `"true"`.
`cilium_hubble_listen_address`	Port used to expose the Cilium Hubble Service.	In this example, this parameter is set to `":4244"`.
`cilium_hubble_metrics_server`	Port used to expose the Cilium Hubble metrics server.	In this example, this parameter is set to `":9091"`.
`cilium_hubble_metrics`	Metrics to collect. Separate multiple metrics with commas (`,`).	In this example, this parameter is set to `"drop,tcp,flow,port-distribution,icmp"`.

Console

Log on to the Container Service Management Console . In the navigation pane on the left, click Clusters.
On the Clusters page, click the name of your cluster. In the navigation pane on the left, click Configurations > ConfigMaps.
In the upper part of the ConfigMap page, select kube-system from the Namespace drop-down list, find the eni-config ConfigMap, and click Edit YAML in the Actions column.
In the View in YAML panel, find 10-terway.conf, add the parameters listed above, then click OK.

kubectl

Run the following command to edit the ConfigMap:

kubectl -n kube-system edit configmap eni-config

Add the parameters listed above to 10-terway.conf and save the ConfigMap.

Restart Terway pods

Restart the Terway pods for the modified ConfigMap to take effect.

ACK console:

In the left navigation pane of the cluster details page, choose Workloads > Pods.
In the upper part of the Pods page, select kube-system from the Namespace drop-down list, enter terway-eniip in the search box, and then choose More > Delete in the Actions column for each terway-eniip-xxx pod.
In the message that appears, click OK.
Repeat the preceding steps to delete all Terway pods.

kubectl:

kubectl rollout restart daemonset -n kube-system terway-eniip

Verify the restart

After the pods restart, verify that all Terway pods are in the Running state:

kubectl -n kube-system get pods -l app=terway-eniip

All pods should show a STATUS of Running and all containers should be ready.

Install Cilium Hubble

Log on to the Container Service Management Console . In the navigation pane on the left, click Clusters.
On the Clusters page, click the name of your cluster. In the navigation pane on the left, click Applications > Helm.
On the Helm page, click Deploy. Configure the Basic Information parameters.
Select the ack-terway-hubble card and click Next.
In the Parameters step, configure the Chart Version parameter. In the Parameters section, specify the Ingress parameters to enable access to Hubble UI. Configure other parameters based on your business requirements. Click OK.

Ingress parameters

Configure the following parameters under ingress to enable Hubble UI access:

Parameter	Description
`enabled`	Specifies whether to use an Ingress to access Hubble UI.
`annotations`	The annotations of the Ingress.
`path`	The root path of the Ingress.
`hosts`	The host of the Ingress. Used to access Hubble UI.
`tls`	The Transport Layer Security (TLS) settings of the Ingress.

Access Hubble UI

If you specified the hosts parameter for the Ingress, access Hubble UI by visiting the Ingress host on port 80.

If you use a domain name that does not have an authoritative resolution, such as ingress.local, run the kubectl -n kube-system get svc nginx-ingress-lb command to obtain the external IP address of the Ingress controller. Then, add an entry to your local hosts file that maps the domain name to the obtained IP address.

Hubble UI features

The Hubble UI provides the following views:

Top section: Pod and Service topology for each namespace. Select a namespace from the drop-down list to view its topology.
Bottom section: Network traffic details including sources, destinations, ports, and forwarding states.
Network policy enforcement: Packets dropped due to network policy mismatching are highlighted, helping identify misconfigured policies.

Network observability dashboard showing pod interactions and traffic metrics

Collect and visualize metrics

Cilium Hubble exposes network flow metrics through the hubble-metrics Service in the kube-system namespace. Control which metrics are exposed by setting the cilium_hubble_metrics parameter in the eni-config ConfigMap.

Collect these metrics by using either:

Available metrics

The following table lists the metrics that Cilium Hubble can expose. For detailed metric definitions, see Exported Metrics in the Cilium documentation.

Category	Metric	Labels	Description
drop	`hubble_drop_total`	reason, protocol	Number of dropped packets
tcp	`hubble_tcp_flags_total`	flag, family	Number of TCP flags
flow	`hubble_flows_processed_total`	type, subtype, verdict	Number of processed network flows
port-distribution	`hubble_port_distribution_total`	protocol, port	Distribution of packets on destination ports
icmp	`hubble_icmp_total`	family, type	Number of Internet Control Message Protocol (ICMP) messages