Renew the etcd certificates on master nodes in your ACK dedicated cluster to maintain service continuity and reduce security risks from certificate leakage or key cracking. ACK sends you a notification two months before expiration—act promptly when you receive it.
ACK managed Pro clusters handle etcd and Kubernetes control plane certificates automatically. To eliminate manual certificate management, migrate your ACK dedicated cluster to an ACK managed Pro cluster. See Hot migrate ACK dedicated clusters to ACK managed Pro clusters.
Before you begin
-
ACK sends internal messages and text message reminders two months before etcd certificates expire, and displays the Update ETCD Certificate button on the Clusters page.
-
During renewal, control plane components on the master nodes—API Server, etcd, kube-controller-manager, and kubelet—are restarted one by one. Persistent connections to the API Server are interrupted during this period. Perform this operation during off-peak hours. The process takes approximately 30 minutes.
-
If you modified the default configuration file directory for etcd or Kubernetes in your ACK dedicated cluster, create a symbolic link to the original directory before renewing the certificates. Otherwise, the renewal fails.
-
If you manually renew the certificates and the Update ETCD Certificate reminder is still displayed in the console, submit a ticketsubmit a ticketsubmit a ticketsubmit a ticket to have it removed.
-
If renewal fails, submit a ticketsubmit a ticketsubmit a ticketsubmit a ticket for assistance.
Choose a renewal method
| Situation | Recommended method |
|---|---|
| Certificates are about to expire (not yet expired), console is accessible | Use the console |
| Certificates are about to expire, console renewal fails or is inaccessible | Manually renew (Scenario 1) |
| Certificates have already expired, API Server may be inaccessible | Manually renew (Scenario 2) |
| Renewal failed and cluster is in a broken state | Roll back the changes |
Renew certificates from the console
-
Log on to the ACK console. In the left navigation pane, click Clusters.
-
Click Update ETCD Certificate next to the cluster whose certificates are about to expire. On the Update Certificate page, click Update Certificate.
The Update ETCD Certificate button appears next to a cluster only when its certificates are due to expire within two months.
-
In the Note dialog box, click OK.
After the renewal completes:
-
The Update Certificate page shows Updated Successfully.
-
The Update ETCD Certificate button is no longer displayed next to the cluster on the Clusters page.
Manually renew certificates
Manual renewal applies when:
-
The console auto-renewal is unavailable.
-
The certificates cannot be renewed by deploying a template.
All scripts in this section must be run as the root user.
Prerequisites
Before running the renewal scripts, configure password-free Secure Shell (SSH) logon between master nodes for the root user.
To verify, SSH from one master node to another. If prompted for a password, set up password-free logon:
# Step 1: Generate an RSA key pair. Skip this step if a key already exists on the node.
ssh-keygen -t rsa
# Step 2: Copy the public key to each other master node.
# Replace $(internal-ip) with the internal IP address of the target master node.
ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)If you skip this step, you must enter the root password each time the script connects to a remote node.
Scenario 1: Renew certificates that are not yet expired
Use this procedure when the etcd certificates are approaching expiration but have not yet expired.
Step 1: Save the scripts
Save the following two scripts in the same folder on any master node. Name them restart-apiserver.sh and rotate-etcd.sh.
Therotate-etcd.shscript retrieves the region by querying the node's metadata service and pulls the rotation image from the nearest registry in that region. To specify the region manually, run the script with--region <region-id>.
Step 2: Run the rotation script
On any master node, run:
bash rotate-etcd.shIf the output ends with etcd CA and certs have successfully rotated!, the certificates on all master nodes and the Kubernetes components have been renewed.
Step 3: Verify the renewal
cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;doneThe renewal is complete if all expiration dates in the output are 50 years in the future.
After manually renewing the certificates, the update button may still appear in the console because the ACK console cannot retrieve the renewal result automatically. Submit a ticketSubmit a ticketSubmit a ticketSubmit a ticket to have it removed.
Scenario 2: Renew certificates that have expired
Use this procedure when the etcd certificates have already expired, including cases where the API Server is inaccessible.
Step 1: Save the scripts
Save the following two scripts in the same folder on any master node. Name them restart-apiserver.sh and rotate-etcd.sh.
Therotate-etcd.shscript retrieves the region by querying the node's metadata service. To specify the region manually, run the script with--region <region-id>.
Step 2: Run the rotation script
On any master node, run:
bash rotate-etcd.shIf the output ends with etcd CA and certs have successfully rotated!, the certificates on all master nodes and the Kubernetes components have been renewed.
Step 3: Verify the renewal
cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;doneThe renewal is complete if all expiration dates in the output are 50 years in the future.
After manually renewing the certificates, the expired status may still appear in the console because the ACK console cannot retrieve the renewal result automatically. Submit a ticketSubmit a ticketSubmit a ticketSubmit a ticket to have it removed.
Roll back when certificate renewal fails
Use this procedure when console renewal or command-line renewal fails. The rollback generates a new set of etcd certificates and updates the etcd server certificate and the kube-apiserver client certificate, restoring the cluster to a functional state.
All scripts must be run as the root user. Configure password-free SSH logon between master nodes before proceeding (see Prerequisites).
Step 1: Save the scripts
Save the following two scripts in the same folder on any master node. Name them restart-apiserver.sh and rollback-etcd.sh.
Therollback-etcd.shscript retrieves the region by querying the node's metadata service. To specify the region manually, run the script with--region <region-id>.
Step 2: Run the rollback script
On any master node, run:
bash rollback-etcd.shIf the output ends with etcd CA and certs have successfully rotated!, the rollback is complete.
Step 3: Verify the rollback
cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;doneThe rollback is complete if the expiration dates in the output are more than 50 years in the future.