How to increase the etcd storage capacity limit for ACK dedicated clusters with an etcd version earlier than v3.4.3 - Container Service for Kubernetes

ACK dedicated clusters use etcd v3.3.8 by default, which supports up to 2 GB of storage. When etcd storage exceeds 2 GB, writes to etcd fail. Upgrade etcd to v3.4.3 to raise the limit to 100 GB.

Prerequisites

Ensure the following:

The cluster runs etcd earlier than v3.4.3.
Storage requirement exceeds 2 GB. The upgrade is optional if data stays within 2 GB.

How etcd storage quotas work

etcd v3.4.3 sets ETCD_QUOTA_BACKEND_BYTES=100000000000 for up to 100 GB backend storage and ETCD_EXPERIMENTAL_BACKEND_BBOLT_FREELIST_TYPE=map for improved storage reclamation.

When the backend store exceeds the configured quota, writes to etcd fail. Upgrading to v3.4.3 with the higher quota prevents this.

Upgrade etcd to v3.4.3

Upgrade one node at a time. Because etcd is highly available, the cluster remains accessible and application access to etcd is not interrupted during the rolling upgrade.

Important

Wait until each upgraded node reaches the Ready state before moving to the next node.

Verify the current version

SSH into a master node and check the etcd version:

etcd --version

Confirm version 3.3.8. Skip the upgrade if already at v3.4.3 or later.

Download and install the v3.4.3 binaries

On the master node, download and replace the etcd v3.4.3 binaries:

etcdbin=http://aliacs-k8s-cn-hangzhou.oss.aliyuncs.com/etcd/etcd-v3.4.3/etcd
etcdctlbin=http://aliacs-k8s-cn-hangzhou.oss.aliyuncs.com/etcd/etcd-v3.4.3/etcdctl

wget -O etcd ${etcdbin}
wget -O etcdctl ${etcdctlbin}
chmod +x {etcd,etcdctl}
mv etcd /usr/bin/etcd
mv etcdctl /usr/bin/etcdctl

Verify the new version:

etcd --version

Update the etcd configuration

Update the etcd service file with the 100 GB quota and map-based freelist type, then restart etcd:

ETCD_FILE=/lib/systemd/system/etcd.service

# Remove any existing quota and freelist settings to avoid duplicates
sed -i "/ETCD_EXPERIMENTAL_BACKEND_BBOLT_FREELIST_TYPE/ d" ${ETCD_FILE}
sed -i "/ETCD_QUOTA_BACKEND_BYTES/ d" ${ETCD_FILE}

# Set the new 100 GB quota and the map-based freelist for better performance
sed -i "/^\[Service\]/a\Environment=\"ETCD_EXPERIMENTAL_BACKEND_BBOLT_FREELIST_TYPE=map\"" ${ETCD_FILE}
sed -i "/^\[Service\]/a\Environment=\"ETCD_QUOTA_BACKEND_BYTES=100000000000\"" ${ETCD_FILE}

# Update the cluster state from "new" to "existing" for the rejoining node
sed -i "s/initial-cluster-state new/initial-cluster-state existing/g" ${ETCD_FILE}

systemctl daemon-reload
systemctl restart etcd

Note

While a node restarts, remaining members log connection errors for that peer. These resolve automatically once the node rejoins.

Verify the node has rejoined

After restarting etcd, confirm the node appears in the cluster member list:

ENDPOINTS=`ps -eaf|grep etcd-servers|grep -v grep|awk -F "=" '{print $22}'|awk -F " " '{print $1}'`

ETCDCTL_API=3 etcdctl \
    --endpoints=${ENDPOINTS} \
    --cacert=/var/lib/etcd/cert/ca.pem \
    --cert=/var/lib/etcd/cert/etcd-client.pem \
    --key=/var/lib/etcd/cert/etcd-client-key.pem \
    member list

Once confirmed, repeat the download, configuration, and verification steps on the next master node.

Confirm etcd is running

After upgrading all nodes, confirm etcd is running on each node:

ps aux | grep etcd

Next steps

On a master node, check the cluster health:

ENDPOINTS=`ps -eaf|grep etcd-servers|grep -v grep|awk -F "=" '{print $22}'|awk -F " " '{print $1}'`

ETCDCTL_API=3 etcdctl \
    --endpoints=${ENDPOINTS} \
    --cacert=/var/lib/etcd/cert/ca.pem \
    --cert=/var/lib/etcd/cert/etcd-client.pem \
    --key=/var/lib/etcd/cert/etcd-client-key.pem \
    endpoint health

Expected output:

ENDPOINTS is healthy