All Products
Search

This Product

    Container Service for Kubernetes:Renew etcd certificates for ACK dedicated clusters

    Document Center

    Container Service for Kubernetes:Renew etcd certificates for ACK dedicated clusters

    Last Updated:Jul 29, 2025

    To ensure service continuity and mitigate potential security risks from certificate leakage or key cracking, we recommend that you promptly renew the etcd certificates for the master nodes in your ACK dedicated clusters when you receive system notifications. This topic describes how to renew etcd certificates for the master nodes in an ACK dedicated cluster.

    Background information

    You can migrate workloads from an ACK dedicated cluster to an ACK managed Pro cluster. You can choose to migrate the cluster to an ACK managed Pro cluster. The etcd and Kubernetes control plane certificates of an ACK managed Pro cluster are managed by Alibaba Cloud. After you migrate an ACK dedicated cluster, you do not need to perform the following renewal operations. For more information about how to perform the migration, see Hot migrate ACK dedicated clusters to ACK managed Pro clusters.

    Usage notes

    • Container Service for Kubernetes (ACK) sends you internal messages and text message reminders two months before the etcd certificates expire, and displays the Renew Etcd Certificate button on the Clusters page.

    • During the renewal process, control plane components on the cluster's master nodes, such as the API Server, etcd, kube-controller-manager, and kubelet, are restarted one by one. Persistent connections to the API Server are interrupted during this period. We recommend that you perform this operation during off-peak hours. The renewal process takes approximately 30 minutes to complete.

    • If you have modified the default configuration file directory of etcd or Kubernetes in the ACK dedicated cluster, you must create a symbolic link to the original directory before you renew the certificates. Otherwise, the renewal fails.

    • If you manually renew the certificates and the Update ETCD Certificate expiration reminder is still displayed in the ACK console, you can submit a ticket to have the reminder removed.

    • If the renewal process fails, submit a ticket for assistance.

    Scenario 1: Renew etcd certificates that are not expired

    When the etcd certificates are about to expire and a reminder is displayed, you can renew the certificates using one of the following methods.

    Use the console to automatically renew etcd certificates

    1. Log on to the ACK console. In the navigation pane on the left, click Clusters.

    2. Click Update ETCD Certificate next to the cluster whose etcd certificates are about to expire. On the Update Certificate page, click Update Certificate.

      Note

      If the etcd certificates of a cluster are due to expire within two months, the Update ETCD Certificate button is displayed next to the cluster.

      etcd

    3. In the Note dialog box, click OK.

      After the certificates are renewed, the following changes occur:

      • On the Update Certificate page, a message that reads Updated Successfully is displayed.

      • On the Clusters page, the Update ETCD Certificate button is no longer displayed next to the cluster.

    Manually renew etcd certificates

    Scenarios

    • The etcd certificates in an ACK dedicated cluster are about to expire.

    • The etcd certificates cannot be automatically renewed by deploying a template.

    • The etcd certificates cannot be renewed in the console.

    In these scenarios, a cluster administrator can log on to any master node and run the following script to manually renew the etcd certificates.

    Note

    The following script must be run as the root user.

    1. Make sure that password-free logon between master nodes is configured for the root user.

      Use Secure Shell (SSH) to log on to another master node from a master node. If you are prompted to enter a password, you must configure password-free logon between master nodes.

      # 1. Generate a key. If a logon key already exists on your node, you can skip this step.
ssh-keygen -t rsa

# 2. Use the ssh-copy-id tool to transfer the public key to all other master nodes. $(internal-ip) is the internal IP address of another master node.
ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)
      Note

      If you do not configure password-free logon, you must enter the root password when you run the script.

    2. Copy the following scripts, save them as restart-apiserver.sh and rotate-etcd.sh, and then save the two files in the same folder.

      Note

      The rotate-etcd.sh script attempts to retrieve the region information by accessing the node's metadata service and pulls the rotation image from the nearest location in that region. You can also run the script with the --region xxxx parameter to specify the region.

      View The Restart-apiserver.sh Script

      #! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API Server restarted"

      View The Rotate-etcd.sh Script

      #!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "-----generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "-----generate etcdserver"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "-----generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # Create a CA bundle.
  cat $KUBE_CERT_PATH/etcd/ca.pem >>$dir/bundle_ca.pem
  cat $ETCD_CERT_DIR/ca.pem >>$dir/bundle_ca.pem
  cat $dir/ca.pem >>$dir/bundle_ca.pem

  # Create a peer CA bundle.
  cat $ETCD_CERT_DIR/peer-ca.pem >$dir/bundle_peer-ca.pem
  cat $dir/peer-ca.pem >>$dir/bundle_peer-ca.pem

  current_year=$(date +%Y)
  TARGET_TEAR=$((TARGET_TEAR + 50))

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*
}

function etcd_client_urls() {
  local etcd_hosts=()
  for ip in "${ETCD_HOSTS[@]}"; do
    etcd_hosts+=("https://$ip:2379")
  done
  local result=$(
    IFS=','
    echo "${etcd_hosts[*]}"
  )
  echo "$result"
}

function check_cert_files_exist() {
  REQUIRED_CERTS=("ca.pem" "etcd-server-key.pem" "etcd-server.pem" "peer-ca-key.pem" "peer-ca.pem")
  if [ ! -d "$ETCD_CERT_DIR" ]; then
    echo "Error: Directory $ETCD_CERT_DIR does not exist"
    exit 1
  fi

  for cert_file in "${REQUIRED_CERTS[@]}"; do
    if [ ! -f "$ETCD_CERT_DIR/$cert_file" ]; then
      echo "Error: File $ETCD_CERT_DIR/$cert_file does not exist"
      exit 1
    fi
  done

  echo "All required certificate files exist"
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done
  ready=0
  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      ready=1
      break
    fi
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function rotate_etcd_ca() {
  for ADDR in $ETCD_HOSTS; do
    echo "update etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client-key.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client-key.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem

    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    ssh -o StrictHostKeyChecking=no root@$ADDR chmod 0644 $ETCD_CERT_DIR/*
    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"

    # Check whether etcd is started and whether the cluster is normal.
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
    restart_one_apiserver $ADDR
    echo "apiserver on node $ADDR restarted"
  done
}

function rotate_etcd_certs() {
  for ADDR in $ETCD_HOSTS; do
    echo "update etcd peer certs on node $ADDR"
    scp -o StrictHostKeyChecking=no \
      $dir/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem} root@$ADDR:$ETCD_CERT_DIR/

    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR

    ssh -o StrictHostKeyChecking=no root@$ADDR \
      chmod 0400 $ETCD_CERT_DIR/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem}

    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
  done
}

function recover_etcd_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    echo "replace etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem
    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    echo "restart apiserver on node $ADDR"
    restart_one_apiserver $ADDR
    echo "apiserver on node $ADDR restarted"
    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
    sleep 5
  done
}

function recover_etcd_client_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    echo "replace etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
  done
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function restart_one_apiserver() {
  ADDR=$1
  if [[ -z "${ADDR}" ]]; then
    printf "ADDR is empty,exit."
    exit 1
  fi
  printf "restart apiserver on node %s\n" "${ADDR}"
  scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
echo "${ETCD_HOSTS[@]}"

check_container_runtime

# Update certs on etcd nodes.
echo "---restart runtime and kubelet on master nodes---"
for ADDR in $ETCD_HOSTS; do
  if [ "$cmd" == "docker" ]; then
    echo "restart docker on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
  fi
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
done
sleep 5
echo "---end to restart runtime and kubelet on master nodes---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

echo "---check cert files exist---"
check_cert_files_exist
echo "---end to check cert files exist---"

echo "---check connectivity for etcd nodes---"
check_etcd_cluster_ready
echo "---end to check connectivity for etcd nodes---"

# Update certs on etcd nodes.
for ADDR in $ETCD_HOSTS; do
  scp -o StrictHostKeyChecking=no restart-apiserver.sh root@$ADDR:/tmp/restart-apiserver.sh
  ssh -o StrictHostKeyChecking=no root@$ADDR chmod +x /tmp/restart-apiserver.sh
done

gencerts

echo "---rotate etcd ca and etcd client ca---"
rotate_etcd_ca
echo "---end to rotate etcd ca and etcd client ca---"

echo "---rotate etcd peer and certs---"
rotate_etcd_certs
echo "---end to rotate etcd peer and certs---"

echo "check etcd cluster ready"
check_etcd_cluster_ready

echo "---replace etcd ca---"
recover_etcd_ca
echo "---end to replace etcd ca---"

generate_cm
echo "etcd CA and certs have succesfully rotated!"

    3. On any master node, run bash rotate-etcd.sh.

      If the command output is etcd CA and certs have successfully rotated!, the certificates on all master nodes and the Kubernetes certificates are successfully renewed.

    4. Verify that the certificates are renewed.

      cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
      Note

      • If the expiration date in the output of the preceding script is 50 years in the future, the renewal is complete.

      • After you manually renew the certificates, the update button may still be displayed for the cluster in the console because the ACK console cannot retrieve the renewal result. You can submit a ticket to have the button removed.

    Scenario 2: Renew etcd certificates that have expired

    Scenarios

    • The etcd certificates have expired.

    • If the API Server is inaccessible, you can rotate etcd certificates.

    • The etcd certificates cannot be automatically renewed by deploying a template.

    • The etcd certificates cannot be renewed in the console.

    In these scenarios, a cluster administrator can log on to any master node and run the following script to manually renew the etcd certificates.

    Note

    The following script must be run as the root user.

    1. Make sure that password-free logon between master nodes is configured for the root user.

      Use Secure Shell (SSH) to log on to another master node from a master node. If you are prompted to enter a password, you must configure password-free logon between master nodes.

      # 1. Generate a key. If a logon key already exists on your node, you can skip this step.
ssh-keygen -t rsa

# 2. Use the ssh-copy-id tool to transfer the public key to all other master nodes. $(internal-ip) is the internal IP address of another master node.
ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)
      Note

      If you do not configure password-free logon, you must enter the root password when you run the script.

    2. Copy the following scripts, save them as restart-apiserver.sh and rotate-etcd.sh, and then save the two files in the same folder.

      Note

      The rotate-etcd.sh script attempts to retrieve the region information by accessing the node's metadata service and pulls the rotation image from the nearest location in that region. You can also run the script with the --region xxxx parameter to specify the region.

      View The Restart-apiserver.sh Script

      #! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API Server restarted"

      View The Rotate-etcd.sh Script

      #!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/rollback/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"
  cd $dir

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "generate etcd server certificates"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*

  for ADDR in $ETCD_HOSTS; do
    printf "sync the certificates of node %s" "${ADDR}"
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" mkdir -p "${dir}"
    scp -o StrictHostKeyChecking=no "${dir}"/* root@"${ADDR}":/var/lib/etcd/cert/
    scp -o StrictHostKeyChecking=no "${dir}"/ca.pem "${dir}"/etcd-client.pem "${dir}"/etcd-client-key.pem root@"${ADDR}":/etc/kubernetes/pki/etcd/
  done
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function rotate_etcd() {
  for ADDR in $ETCD_HOSTS; do
    printf "rotate etcd's certificates on node %s\n" "${ADDR}"
      if [ "$cmd" == "docker" ]; then
        echo "restart docker on node $ADDR"
        ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
      fi
    ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
  done
}

function rotate_apiserver() {
  echo "current dir: $currentDir"
  for ADDR in $ETCD_HOSTS; do
    printf "restart apiserver on node %s\n" "${ADDR}"
    scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
  done
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done

  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      break
    fi
    sleep 1
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
printf "ETCD_HOSTS: %s\n" "$ETCD_HOSTS"

gencerts
echo "---generate certificates successfully---"

rotate_etcd
echo "---rotate etcd successfully---"

echo "---check etcd cluster ready---"
check_etcd_cluster_ready

rotate_apiserver
echo "---restart apiserver successfully---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

generate_cm
echo "etcd CA and certs have successfully rotated!"

rm -rf $dir

    1. Verify that the certificates are renewed.

    cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
    Note

    • If the expiration date in the output of the preceding script is 50 years in the future, the renewal is complete.

    • After you manually renew the certificates, the expired status may still be displayed for the cluster in the console because the ACK console cannot retrieve the renewal result. You can submit a ticket to have the status removed.

    Roll back when certificate renewal fails

    Scenarios

    • Certificate renewal from the console fails.

    • Certificate renewal from the command line fails.

    In these scenarios, a cluster administrator can log on to any master node and run the following script to roll back the changes. This operation restores the cluster to a functional state by generating a new set of etcd certificates and updating the etcd server certificate and the kube-apiserver client certificate.

    Note

    The following script must be run as the root user.

    1. Make sure that password-free logon between master nodes is configured for the root user.

      Use Secure Shell (SSH) to log on to another master node from a master node. If you are prompted to enter a password, you must configure password-free logon between master nodes.

      # 1. Generate a key. If a logon key already exists on your node, you can skip this step.
ssh-keygen -t rsa

# 2. Use the ssh-copy-id tool to transfer the public key to all other master nodes. $(internal-ip) is the internal IP address of another master node.
ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)
      Note

      If you do not configure password-free logon, you must enter the root password when you run the script.

    2. Copy the following scripts, save them as restart-apiserver.sh and rollback-etcd.sh, and then save the two files in the same folder

      Note

      The rollback-etcd.sh script attempts to retrieve the region information by accessing the node's metadata service and pulls the rotation image from the nearest location in that region. You can also run the script with the --region xxxx parameter to specify the region.

      View The Restart-apiserver.sh Script

      #! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API Server restarted"

      View The Rollback-etcd.sh Script

      #!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/rollback/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"
  cd $dir

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "generate etcd server certificates"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*

  for ADDR in $ETCD_HOSTS; do
    printf "sync the certificates of node %s" "${ADDR}"
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" mkdir -p "${dir}"
    scp -o StrictHostKeyChecking=no "${dir}"/* root@"${ADDR}":/var/lib/etcd/cert/
    scp -o StrictHostKeyChecking=no "${dir}"/ca.pem "${dir}"/etcd-client.pem "${dir}"/etcd-client-key.pem root@"${ADDR}":/etc/kubernetes/pki/etcd/
  done
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function rotate_etcd() {
  for ADDR in $ETCD_HOSTS; do
    printf "rotate etcd's certificates on node %s\n" "${ADDR}"
      if [ "$cmd" == "docker" ]; then
        echo "restart docker on node $ADDR"
        ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
      fi
    ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
  done
}

function rotate_apiserver() {
  echo "current dir: $currentDir"
  for ADDR in $ETCD_HOSTS; do
    printf "restart apiserver on node %s\n" "${ADDR}"
    scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
  done
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done

  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      break
    fi
    sleep 1
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
printf "ETCD_HOSTS: %s\n" "$ETCD_HOSTS"

gencerts
echo "---generate certificates successfully---"

rotate_etcd
echo "---rotate etcd successfully---"

echo "---check etcd cluster ready---"
check_etcd_cluster_ready

rotate_apiserver
echo "---restart apiserver successfully---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

generate_cm
echo "etcd CA and certs have successfully rotated!"

rm -rf $dir

    3. On any master node, run bash rollback-etcd.sh.

      If the command output is etcd CA and certs have successfully rotated!, the certificates on all master nodes and the Kubernetes certificates are successfully renewed.

    4. Verify that the certificates are renewed.

    cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done
    Note

    The rotation is complete if the time returned by the preceding script is more than 50 years in the future.