Optimize conntrack configurations in Terway - Container Service for Kubernetes

Tune eBPF conntrack table size and timeout settings to handle high-connection workloads in Terway.

Check the conntrack table size

The eBPF conntrack table size depends on the memory capacity of the node.

To view the conntrack table size on a node:

Retrieve the terway-eniip pod name:

# kubectl describe node <nodename>  | grep terway-eniip
  kube-system                 terway-eniip-xxxx         350m (8%)     1100m (28%)  200Mi (3%)       256Mi (3%)     16h

Query the conntrack table size:

# kubectl exec -it -nkube-system terway-eniip-xxxx -c policy -- cilium status --verbose
...
BPF Maps:   dynamic sizing: on (ratio: 0.002500)
  Name                          Size
  Auth                          524288
  Non-TCP connection tracking   65536
  TCP connection tracking       131072
  Endpoint policy               65535
...

Parameters:

dynamic sizing: on: eBPF maps use dynamic upper limits. If set to off, configure bpf-map-dynamic-size-ratio for dynamic adjustment, or set bpf-ct-global-any-max, bpf-ct-global-tcp-max, and bpf-nat-global-max for fixed sizes.
ratio: 0.002500: maximum eBPF map size as a proportion of total system memory.
Non-TCP connection tracking: maximum entries in a non-TCP conntrack table.
TCP connection tracking: maximum entries in the TCP conntrack table.

Check the number of conntrack entries

By default, each node exposes Prometheus metrics on port 9962.

Metric

Labels and values

cilium_datapath_conntrack_gc_entries

family: IP address type. Valid values: ipv4 and ipv6.

protocol: protocol type. Valid values: non-TCP and TCP.

status: cleanup status. Combine all states to get the pre-cleanup conntrack entry count. Valid values: alived and deleted.

Modify the default monitoring port

Parameter

Description

prometheus-serve-addr

The Prometheus listening port. Set to \"\" to disable.

Terway ≥ 1.14.0: disabled by default
Terway < 1.14.0: set to :9962 by default.

To modify this parameter, see cilium_args in Customize Terway configuration parameters.

Adjust the eBPF conntrack size

eBPF conntrack and Linux conntrack are functionally identical. Default settings suit most scenarios.

Modify the settings in the following scenarios:

Many long-lived connections exist.
Pods are exposed to the Internet, and many half-open connections may exist.

Important

Adjusting the conntrack table size is high-risk. Evaluate the impact before applying changes in production. After adjustment, verify the result as described in Check the conntrack table size.

Parameter	Description
bpf-map-dynamic-size-ratio	Maximum eBPF map size as a proportion of total system memory. `off`: disables auto scaling (fixed map size) `0.0025`: sets map limit to 0.0025 of total system memory
bpf-ct-timeout-regular-any	Timeout for non-TCP conntrack entries. Default value: `1m0s`.
bpf-ct-timeout-regular-tcp	Timeout for established TCP conntrack entries. Default value: `6h0m0s`.
bpf-ct-timeout-regular-tcp-fin	Timeout for closing TCP conntrack entries. Default value: `10s`.
bpf-ct-timeout-regular-tcp-syn	Timeout for establishing TCP conntrack entries. Default value: `1m0s`.
bpf-ct-timeout-service-any	Timeout for non-TCP service conntrack entries. Default value: `1m0s`.
bpf-ct-timeout-service-tcp	Timeout for established TCP service conntrack entries. Default value: `6h0m0s`.
bpf-ct-timeout-service-tcp-grace	Timeout for graceful TCP service shutdown. Default value: `1m0s`.
conntrack-gc-interval	The conntrack table cleanup interval. Default value: `5m0s`. Note The interval at which cilium-agent cleans up the conntrack table. Do not exceed the default value. A shorter interval increases cilium-agent CPU usage but prevents the conntrack table from filling up. Adjust the CPU limit of the Terway container in the ACK console.
bpf-ct-global-any-max	Maximum non-TCP service conntrack table size. Note When dynamic sizing is enabled, use `bpf-map-dynamic-size-ratio` to adjust the table size.
bpf-ct-global-tcp-max	Maximum TCP service conntrack table size. Note When dynamic sizing is enabled, use `bpf-map-dynamic-size-ratio` to adjust the table size.
bpf-nat-global-max	Maximum BPF NAT table size. Note Must be less than the sum of bpf-ct-global-any-max and bpf-ct-global-tcp-max. Default = (bpf-ct-global-any-max + bpf-ct-global-tcp-max) × 2/3 When dynamic sizing is enabled, use `bpf-map-dynamic-size-ratio` to adjust the table size.

To modify these parameters, see cilium_args in Customize Terway configuration parameters.

Map sizes with bpf-map-dynamic-size-ratio set to 0.0025:

Node memory	CT map size (TCP)	CT map size (any)	NAT map size
7.5 GiB	131,072	65,536	131,072
16 GiB	151,765	75,882	151,765
30 GiB	284,560	142,280	284,560
240 GiB	2,276,484	1,138,242	2,276,484

Example

Set bpf-map-dynamic-size-ratio to 0.003 and bpf-ct-timeout-regular-tcp to 1h:

10-terway.conf: |
{
  "cniVersion": "0.4.0",
  "name": "terway",
  "capabilities": {"bandwidth": true},
  "eniip_virtual_type": "IPvlan",
  "cilium_args": "--bpf-map-dynamic-size-ratio=0.003 --bpf-ct-timeout-regular-tcp=1h",
  "type": "terway"
}