Customize node pool kubelet configurations - Container Service for Kubernetes

If the default kubelet configuration does not meet your needs, you can customize kubelet parameters at the node pool level to adjust node behavior. For example, you can fine-tune resource reservations to manage resource usage, set custom node-pressure eviction thresholds to mitigate resource shortages, or change the topology manager policy to improve system performance.

Limitations

Only ACK clusters that run Kubernetes 1.20 or later support custom kubelet parameters. To upgrade a cluster, see Manually upgrade a cluster.
Only ACK Lingjun clusters that run Kubernetes 1.22 or later support custom kubelet parameters. To upgrade a cluster, see Upgrade a cluster.

If your cluster version does not meet these conditions, unexpected behavior may occur.

Usage notes

Custom kubelet parameters are applied to nodes in batches. The changes take effect immediately on existing nodes in the node pool and are automatically applied to new nodes. Applying the configuration restarts the kubelet process, which may affect running nodes and workloads. We recommend that you perform this operation during off-peak hours.
If evictionHard, kubeReserved, or systemReserved is not configured, the system uses default values for resource reservation. For more information about how the default values are calculated, see Node resource reservation policy.
Changes to the resource reservation configuration may reduce a node's allocatable resources. For nodes with high resource usage, this may trigger node eviction.
We strongly recommend that you do not use the command line to define kubelet parameters that are not supported in the console. Using the command line poses significant stability risks. You are responsible for the correctness and compatibility of the user data file content. Incorrect or deprecated parameters can make nodes unavailable.
When the kubelet starts, it merges configurations from different sources based on their priority. If the same configuration item is set in multiple ways, the setting with the higher priority overwrites the one with the lower priority.
Configuration merge order
The following configurations are sorted by priority from lowest to highest:
1. The feature gates specified by the --feature-gates flag in the command line.
2. The kubelet configuration file, which contains parameters customized through the ACK console or OpenAPI.
3. Plugin configurations, which are loaded in alphabetical order from the kubelet configuration directory.
4. Command-line parameters other than feature gates that are specified directly through the command line.

Customize kubelet parameters in the console

Applying custom kubelet parameters restarts the kubelet process, which may affect your services. We recommend that you perform this operation during off-peak hours.

On the ACK Clusters page, click the name of your cluster. In the left navigation pane, click Nodes > Node Pools.
On the Node Pools page, find the target node pool and choose > Kubelet Configuration in the Actions column.
Read the notes on the page. Click Custom Parameters, select the parameters to configure, specify the nodes to upgrade, and set the batch update policy. Then, complete the operation as prompted.

The batch update policy is described as follows:
- Maximum Parallel Nodes per Batch: The number of nodes to update concurrently in each batch. You can monitor and manage the update process in the event list.
- Interval Between Batches: The waiting time between consecutive update batches.
You can pause the update to verify the upgraded nodes. When paused, any in-progress node updates will complete, but new updates will not start until you resume the task.

Complete the custom configuration task as soon as possible. A paused task is automatically canceled after seven days, and the related events and logs are cleared.

In addition to the console, you can also call the ModifyNodePoolNodeConfig API operation to customize kubelet parameters. The following section describes the customizable kubelet parameters supported by ACK.

Customizable kubelet parameters

Parameter	Description	Default	Values
allowedUnsafeSysctls	Specify the allowed unsafe sysctls or sysctl wildcards (patterns ending with ``). Use a comma (,) to separate them. Important* Before you use this parameter, carefully assess the risks and ensure availability.	N/A	Supports sysctl configurations with the following prefixes: kernel.shm kernel.msg kernel.sem fs.mqueue. net.
containerLogMaxFiles	The maximum number of log files for a container. The value must be 2 or greater. The container runtime must be containerd.	10	[2, 10]
containerLogMaxSize	The maximum size of a container log file before it is rotated. The container runtime must be containerd.	100Mi	N/A
cpuCFSQuota	Enables CPU CFS quota enforcement for containers that have CPU limits.	true	Valid values: true false
cpuCFSQuotaPeriod	Sets the CPU CFS quota period value. The CustomCPUCFSQuotaPeriod feature gate must be enabled.	100ms	A value from 1 millisecond to 1 second, inclusive.
cpuManagerPolicy	The CPU manager policy.	none	Valid values: none static
eventBurst	The maximum number of event records in a burst.	10	The value must be an integer from 1 to 100 and greater than or equal to the value of `eventRecordQPS`.
eventRecordQPS	The number of events that can be generated per second.	5	[1, 50]
evictionHard	A set of hard eviction thresholds that trigger pod eviction.	imagefs.available<15%,memory.available<300Mi,nodefs.available<10%,nodefs.inodesFree<5%	None
evictionSoft	A set of soft eviction thresholds.	None	None
evictionSoftGracePeriod	A set of eviction grace periods. Note `evictionSoft` must be set.	None	None
featureGates	The set of feature gates for experimental features. Each gate is represented as `key=value`. For more information, see Feature Gates. In the configuration form, select a key (such as `featureGates`) from the drop-down list on the left, enter the corresponding value in the text box on the right, and then select the checkbox on the right to enable the feature gate. Important Before you use this parameter, carefully assess the risks and ensure availability. Removed feature gates do not take effect. Update them promptly. For more information, see Feature Gates (removed).	N/A	N/A
imageGCHighThresholdPercent	The percentage of disk usage that triggers image garbage collection. Image garbage collection runs continuously when disk usage exceeds this threshold. This value must be greater than the value of imageGCLowThresholdPercent.	85	[60, 95]
imageGCLowThresholdPercent	The percentage of disk usage below which image garbage collection does not run. This value must be less than the value of imageGCHighThresholdPercent.	80	[30, 90]
kubeAPIBurst	The maximum number of burst requests sent to the API server per second.	10	[1, 100], and must be greater than or equal to `kubeAPIQPS`
kubeAPIQPS	The number of queries per second (QPS) to the API server.	5	[1, 50]
kubeReserved	The resource configuration reserved for Kubernetes system daemons.	The value is automatically calculated by default. For more information, see Node resource reservation policy.	N/A
maxPods	The maximum number of pods that can run on a node. Important Modifying the value of `maxPods` does not affect the number of IP addresses that can be allocated to the node. If the value of `maxPods` is too large, pods not using HostNetwork mode may fail to start due to an inability to get an IP address.	N/A. The value depends on physical resource configurations such as machine specifications and container network planning.	N/A
memoryManagerPolicy	The policy for the memory manager.	None	Valid values: None Static Before you use this policy, complete the following operations: The MemoryManager feature gate is enabled. If you manually manage memory resource reservations (systemReserved, kubeReserved, evictionHard, and reservedMemory), you must ensure that the configuration is correct. The configuration must follow the formula `kube-reserved + system-reserved + eviction-hard = sum(reserved-memory)` to prevent nodes from becoming unavailable. For example, in the custom parameter panel, set memoryManagerPolicy to Static and configure the following parameters based on the formula `evictionHard.memory.available + systemReserved.memory + kubeReserved.memory = sum(reservedMemory.limits.memory)`: set the memory.available of evictionHard to `100Mi`, the memory of kubeReserved to `100Mi`, the numaNode of reservedMemory to `0`, the limits.memory of reservedMemory to `300Mi`, and the memory of systemReserved to `100Mi`. For more information, see Reserved memory flag.
podPidsLimit	The maximum number of process IDs (PIDs) that can be used in each pod. A positive integer: Sets the upper limit on the number of PIDs. -1: No limit is set on the number of PIDs.	16384	None
readOnlyPort	The read-only port for the kubelet that does not require authentication.	For clusters that run Kubernetes 1.26 or later, the default value is 0. For clusters that run a Kubernetes version earlier than 1.26, the default value is 10255.	0 For information about the risks of opening the read-only port (10255) for kubelet container monitoring, see [Product Change] Migrate the monitoring port of earlier-version ACK clusters to an authenticated port.
registryBurst	The maximum number of burst image pulls.	10	The value must be in the range [1, 100] and must be greater than or equal to the value of `registryPullQPS`.
registryPullQPS	The maximum QPS for the image registry.	5	[1, 50]
reservedMemory	A list of memory reservations for NUMA nodes.	None	None
serializeImagePulls	Specifies whether to pull images serially.	False	Valid values: False: This is the recommended value. It allows images to be pulled in parallel to accelerate pod startup. True: Pulls images serially.
systemReserved	The resource configuration reserved for system daemons.	The value is automatically calculated by default. For more information, see Node resource reservation policy.	N/A
topologyManagerPolicy	The topology manager policy. With a NUMA architecture, allocating data to the same NUMA node reduces cross-node access and improves performance. The topology manager makes topology-aligned resource allocation decisions. For more information, see Control Topology Management Policies on a node.	none	`none` (default): Does not enforce a topology management policy. `restricted`: The kubelet rejects a pod if its resource requests cannot be satisfied by a single NUMA node. `best-effort`: The kubelet attempts to allocate resources on a single NUMA node, but allows allocations from multiple NUMA nodes if a single-node allocation is not possible. `single-numa-node`: The kubelet only allows CPU and device resources to be allocated from a single NUMA node for a pod.
containerLogMonitorInterval	The cluster must run Kubernetes 1.30 or later. The interval at which container logs are checked for rotation.	10s	[3s, 60s]
containerLogMaxWorkers	The cluster must run Kubernetes 1.30 or later. The maximum number of concurrent workers for log rotation.	1	[1, 20]
tracing	Enables tracing for control plane or data plane components of the cluster. For more information, see Tracing management.	None	endpoint: The recommended setting is `localhost:4317`. `samplingRatePerMillion`: [0, 1000000]
singleProcessOOMKill	Controls the OOM Kill behavior of containers in a cgroup v2 environment. When set to `true`, this parameter disables the `memory.oom.group` flag for the container's cgroup, causing processes inside the container to be OOM-killed individually instead of as a group. This behavior is consistent with cgroup v1. If this parameter is not explicitly configured, the system automatically determines the default value based on the current environment: `true`: Triggers an OOM kill individually. `false`: Triggers an OOM Kill on a per-group basis (the default behavior for cgroup v2).	In a cgroup v2 environment, the default is `false`	Non-Linux: Only `true` is supported. cgroup v1 Linux: Only `true` is supported. cgroup v2 Linux: Supports `true` and `false`
configMapAndSecretChangeDetectionStrategy	The operating mode for the ConfigMap and Secret manager.	`Watch`	`Get`: The kubelet directly retrieves the required objects from the API server. `Cache`: The kubelet uses a TTL to cache objects retrieved from the API server. `Watch`: The kubelet uses the Watch mechanism to monitor changes to relevant objects.
serverTLSBootstrap	Only ACK Managed Pro clusters are supported. Ensure that the `RotateKubeletServerCertificate` feature gate is enabled. Enable Kubelet server certificate bootstrapping. When this feature is enabled, Kubelet requests server certificates from the cluster through the `certificates.k8s.io` API instead of using self-signed certificates. The cluster automatically approves these Certificate Signing Requests (CSRs).	For clusters 1.34 or later: `true` For clusters of version 1.33 and lower: `false`	Valid values: `true` `false`

FAQ

Will custom configurations be deprecated?

As Kubernetes evolves, parameters and feature gates can be deprecated or removed. If a custom parameter managed by ACK is deprecated in a new Kubernetes version, ACK removes the corresponding configuration during a cluster upgrade.

Managing kubelet with a configuration file

Container Service for Kubernetes adjusts how kubelet configurations are managed based on community best practices. For clusters that run Kubernetes 1.20 or later, deprecated kubelet command-line flags are gradually replaced by a configuration file. For more information, see Kubelet Configuration (v1beta1).

New nodes use both the configuration file and the original configuration method. Existing nodes are not affected. To manage the configurations of all nodes in a node pool by using only the configuration file, you can apply a custom configuration as described in Customizable kubelet parameters to standardize the configurations of all nodes in the node pool.

Modifying unsupported kubelet parameters

ACK allows you to write custom parameters to the /etc/kubernetes/kubelet-customized-args.conf file. This file stores custom startup parameters and configuration options for the kubelet. Parameters in this file take precedence over values set through the node pool's custom kubelet configuration feature when the node restarts.

Important

Adjusting kubelet parameters can cause issues such as node registration failures and pod scheduling failures, which affect your services. Before you proceed, fully assess the risks of the change.

(Recommended) For nodes that will be added to the node pool in the future, you can add a script that writes to the custom parameter configuration file to the User Data section of the node pool. This ensures that new nodes use these custom parameter values by default.

Configure the following content in the User Data section of the node pool configuration, and replace ${kubelet_key} and ${kubelet_value} with the actual values.

You can write to the kubelet-customized-args.conf file only by using the > /etc/kubernetes/kubelet-customized-args.conf or > kubelet-customized-args.conf command in User Data. If you modify the file by using any other method in Pre-defined Custom Data, the /etc/kubernetes/kubelet-customized-args.conf file will be overwritten during the ACK initialization process.
```
mkdir -p /etc/kubernetes  
echo 'KUBELET_CUSTOMIZED_ARGS="--${kubelet_key}=${kubelet_value}"' > /etc/kubernetes/kubelet-customized-args.conf
systemctl daemon-reload
systemctl restart kubelet
```
To access the configuration page, see Create and manage node pools.
For existing nodes in the node pool, log on to the node to modify the custom parameter configuration file. Then, run the following commands to apply the configuration.
```
systemctl daemon-reload
systemctl restart kubelet
```

References

For more information about the configuration items for a node pool, see Create and manage node pools.
If you encounter errors or abnormal behavior with your nodes, pods, or the kubelet, see Troubleshoot node exceptions, Pod troubleshooting, and FAQ about nodes and node pools to troubleshoot the issues.