Rotate etcd certificates for ACK dedicated master nodes - Container Service for Kubernetes

Renew etcd certificates on master nodes to maintain cluster security and avoid service disruptions.

ACK managed Pro clusters handle etcd and control plane certificates automatically. To avoid manual management, migrate to an ACK managed Pro cluster.

Before you begin

ACK sends in-app and SMS reminders two months before etcd certificates expire and displays Renew Etcd Certificate on the Clusters page.
During renewal, control plane components (API server, etcd, kube-controller-manager, kubelet) restart sequentially on master nodes. Persistent API server connections are interrupted. Perform during off-peak hours. Takes approximately 30 minutes.
If you modified the default etcd or Kubernetes configuration directory, create a symbolic link to the original directory before renewal. Otherwise, renewal fails.
If the Renew Etcd Certificate reminder persists after manual renewal, submit a ticket to remove it.
If renewal fails, submit a ticket for assistance.

Choose a renewal method

Situation	Recommended method
Not yet expired, console accessible	Use the console
Not yet expired, console renewal unavailable	Manually renew (Scenario 1)
Already expired, API server may be inaccessible	Manually renew (Scenario 2)
Renewal failed, cluster in broken state	Roll back the changes

Renew certificates from the console

Log on to the ACK console. In the left navigation pane, click Clusters.
Click Renew Etcd Certificate next to the target cluster. On the Update Certificate page, click Update Certificate.
The Renew Etcd Certificate button appears only when certificates are due to expire within two months.
In the Note dialog box, click OK.

After renewal:

The Update Certificate page shows The certificate has been updated.
The Renew Etcd Certificate button no longer appears on the Clusters page.

Manually renew certificates

Use manual renewal when:

The etcd certificates of the ACK dedicated cluster are about to expire.
Console auto-renewal is unavailable.
Certificates cannot be renewed by deploying a template.

Run all scripts in this section as the root user.

Prerequisites

Configure password-free SSH logon between master nodes for the root user.

To verify, SSH from one master node to another. If prompted, set up password-free logon:

# Step 1: Generate an RSA key pair. Skip this step if a key already exists on the node.
ssh-keygen -t rsa

# Step 2: Copy the public key to each other master node.
# Replace $(internal-ip) with the internal IP address of the target master node.
ssh-copy-id -i ~/.ssh/id_rsa.pub $(internal-ip)

If skipped, you must enter the root password each time the script connects to a remote node.

Scenario 1: Renew certificates that are not yet expired

Use when etcd certificates are approaching expiration but have not yet expired.

Step 1: Save the scripts

Save the following scripts in the same folder on any master node. Name them restart-apiserver.sh and rotate-etcd.sh.

The rotate-etcd.sh script auto-detects the region from the node's metadata service. To specify it manually, use --region <region-id>.

The restart-apiserver.sh script

#! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver Pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver Pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver Pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver Pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API server restarted"

The rotate-etcd.sh script

#!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "-----generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "-----generate etcdserver"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "-----generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # Create a CA bundle.
  cat $KUBE_CERT_PATH/etcd/ca.pem >>$dir/bundle_ca.pem
  cat $ETCD_CERT_DIR/ca.pem >>$dir/bundle_ca.pem
  cat $dir/ca.pem >>$dir/bundle_ca.pem

  # Create a peer CA bundle.
  cat $ETCD_CERT_DIR/peer-ca.pem >$dir/bundle_peer-ca.pem
  cat $dir/peer-ca.pem >>$dir/bundle_peer-ca.pem

  current_year=$(date +%Y)
  TARGET_TEAR=$((TARGET_TEAR + 50))

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*
}

function etcd_client_urls() {
  local etcd_hosts=()
  for ip in "${ETCD_HOSTS[@]}"; do
    etcd_hosts+=("https://$ip:2379")
  done
  local result=$(
    IFS=','
    echo "${etcd_hosts[*]}"
  )
  echo "$result"
}

function check_cert_files_exist() {
  REQUIRED_CERTS=("ca.pem" "etcd-server-key.pem" "etcd-server.pem" "peer-ca-key.pem" "peer-ca.pem")
  if [ ! -d "$ETCD_CERT_DIR" ]; then
    echo "Error: Directory $ETCD_CERT_DIR does not exist"
    exit 1
  fi

  for cert_file in "${REQUIRED_CERTS[@]}"; do
    if [ ! -f "$ETCD_CERT_DIR/$cert_file" ]; then
      echo "Error: File $ETCD_CERT_DIR/$cert_file does not exist"
      exit 1
    fi
  done

  echo "All required certificate files exist"
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done
  ready=0
  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      ready=1
      break
    fi
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function rotate_etcd_ca() {
  for ADDR in $ETCD_HOSTS; do
    echo "update etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client.pem
    scp -o StrictHostKeyChecking=no $dir/etcd-client-key.pem root@$ADDR:$KUBE_CERT_PATH/etcd/etcd-client-key.pem
    scp -o StrictHostKeyChecking=no $dir/bundle_peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem

    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    ssh -o StrictHostKeyChecking=no root@$ADDR chmod 0644 $ETCD_CERT_DIR/*
    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"

    # Check whether etcd is started and whether the cluster is normal.
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
    restart_one_apiserver $ADDR
    echo "apiserver on node $ADDR restarted"
  done
}

function rotate_etcd_certs() {
  for ADDR in $ETCD_HOSTS; do
    echo "update etcd peer certs on node $ADDR"
    scp -o StrictHostKeyChecking=no \
      $dir/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem} root@$ADDR:$ETCD_CERT_DIR/

    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR

    ssh -o StrictHostKeyChecking=no root@$ADDR \
      chmod 0400 $ETCD_CERT_DIR/{peer-ca-key.pem,etcd-server.pem,etcd-server-key.pem,etcd-client.pem,etcd-client-key.pem,ca-key.pem,*-name*.pem}

    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
  done
}

function recover_etcd_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    echo "replace etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$ETCD_CERT_DIR/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/peer-ca.pem root@$ADDR:$ETCD_CERT_DIR/peer-ca.pem
    ssh -o StrictHostKeyChecking=no root@$ADDR chown -R etcd:etcd $ETCD_CERT_DIR
    echo "restart apiserver on node $ADDR"
    restart_one_apiserver $ADDR
    echo "apiserver on node $ADDR restarted"
    echo "restart etcd on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
    echo "etcd on node $ADDR restarted"
    echo "check connectivity for etcd nodes"
    check_etcd_cluster_ready
    echo "end to check connectivity for etcd nodes"
    sleep 5
  done
}

function recover_etcd_client_ca() {
  # Update certs on etcd nodes.
  for ADDR in $ETCD_HOSTS; do
    echo "replace etcd CA on node $ADDR"
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
    scp -o StrictHostKeyChecking=no $dir/ca.pem root@$ADDR:$KUBE_CERT_PATH/etcd/ca.pem
  done
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function restart_one_apiserver() {
  ADDR=$1
  if [[ -z "${ADDR}" ]]; then
    printf "ADDR is empty,exit."
    exit 1
  fi
  printf "restart apiserver on node %s\n" "${ADDR}"
  scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
echo "${ETCD_HOSTS[@]}"

check_container_runtime

# Update certs on etcd nodes.
echo "---restart runtime and kubelet on master nodes---"
for ADDR in $ETCD_HOSTS; do
  if [ "$cmd" == "docker" ]; then
    echo "restart docker on node $ADDR"
    ssh -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
  fi
  ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
done
sleep 5
echo "---end to restart runtime and kubelet on master nodes---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

echo "---check cert files exist---"
check_cert_files_exist
echo "---end to check cert files exist---"

echo "---check connectivity for etcd nodes---"
check_etcd_cluster_ready
echo "---end to check connectivity for etcd nodes---"

# Update certs on etcd nodes.
for ADDR in $ETCD_HOSTS; do
  scp -o StrictHostKeyChecking=no restart-apiserver.sh root@$ADDR:/tmp/restart-apiserver.sh
  ssh -o StrictHostKeyChecking=no root@$ADDR chmod +x /tmp/restart-apiserver.sh
done

gencerts

echo "---rotate etcd ca and etcd client ca---"
rotate_etcd_ca
echo "---end to rotate etcd ca and etcd client ca---"

echo "---rotate etcd peer and certs---"
rotate_etcd_certs
echo "---end to rotate etcd peer and certs---"

echo "check etcd cluster ready"
check_etcd_cluster_ready

echo "---replace etcd ca---"
recover_etcd_ca
echo "---end to replace etcd ca---"

generate_cm
echo "etcd CA and certs have succesfully rotated!"

Step 2: Run the rotation script

On any master node, run:

bash rotate-etcd.sh

If the output ends with etcd CA and certs have successfully rotated!, all master node and Kubernetes component certificates have been renewed.

Step 3: Verify the renewal

cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done

Renewal is complete if all expiration dates are 50 years in the future.

After manual renewal, the update button may persist because the ACK console cannot detect the result automatically. Submit a ticket to have it removed.

Scenario 2: Renew certificates that have expired

Use when etcd certificates have expired, API server is inaccessible, template-based auto-rotation is unavailable, or console renewal is unavailable.

Step 1: Save the scripts

Save the following scripts in the same folder on any master node. Name them restart-apiserver.sh and rotate-etcd.sh.

The rotate-etcd.sh script auto-detects the region from the node's metadata service. To specify it manually, use --region <region-id>.

The restart-apiserver.sh script

#! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver Pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver Pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver Pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver Pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API Server restarted"

The rotate-etcd.sh script

#!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/rollback/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"
  cd $dir

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "generate etcd server certificates"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*

  for ADDR in $ETCD_HOSTS; do
    printf "sync the certificates of node %s" "${ADDR}"
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" mkdir -p "${dir}"
    scp -o StrictHostKeyChecking=no "${dir}"/* root@"${ADDR}":/var/lib/etcd/cert/
    scp -o StrictHostKeyChecking=no "${dir}"/ca.pem "${dir}"/etcd-client.pem "${dir}"/etcd-client-key.pem root@"${ADDR}":/etc/kubernetes/pki/etcd/
  done
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function rotate_etcd() {
  for ADDR in $ETCD_HOSTS; do
    printf "rotate etcd's certificates on node %s\n" "${ADDR}"
      if [ "$cmd" == "docker" ]; then
        echo "restart docker on node $ADDR"
        ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
      fi
    ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
  done
}

function rotate_apiserver() {
  echo "current dir: $currentDir"
  for ADDR in $ETCD_HOSTS; do
    printf "restart apiserver on node %s\n" "${ADDR}"
    scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
  done
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done

  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      break
    fi
    sleep 1
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
printf "ETCD_HOSTS: %s\n" "$ETCD_HOSTS"

gencerts
echo "---generate certificates successfully---"

rotate_etcd
echo "---rotate etcd successfully---"

echo "---check etcd cluster ready---"
check_etcd_cluster_ready

rotate_apiserver
echo "---restart apiserver successfully---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

generate_cm
echo "etcd CA and certs have successfully rotated!"

rm -rf $dir

Step 2: Run the rotation script

On any master node, run:

bash rotate-etcd.sh

If the output ends with etcd CA and certs have successfully rotated!, all master node and Kubernetes component certificates have been renewed.

Step 3: Verify the renewal

cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done

Renewal is complete if all expiration dates are 50 years in the future.

After manual renewal, the expired status may persist because the ACK console cannot detect the result automatically. Submit a ticket to have it removed.

Roll back when certificate renewal fails

Use when console or command-line renewal fails. This procedure generates new etcd certificates and updates the etcd server and kube-apiserver client certificates to restore the cluster.

Run all scripts as the root user. Configure password-free SSH logon between master nodes before proceeding (see Prerequisites).

Step 1: Save the scripts

Save the following scripts in the same folder on any master node. Name them restart-apiserver.sh and rollback-etcd.sh.

The rollback-etcd.sh script auto-detects the region from the node's metadata service. To specify it manually, use --region <region-id>.

The restart-apiserver.sh script

#! /bin/bash

declare -x cmd

k8s::wait_apiserver_ready() {
  set -e
  for i in $(seq 600); do
    if kubectl cluster-info &>/dev/null; then
      return 0
    else
      echo "wait apiserver to be ready, retry ${i}th after 1s"
      sleep 1
    fi
  done
  echo "failed to wait apiserver to be ready"
  return 1
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

function restart_apiserver() {
  # Check the container runtime.
  if [[ $cmd == "docker" ]]; then
    # Use the docker command to restart the kube-apiserver Pod.
    container_id=$(docker ps | grep kube-apiserver | awk '{print $1}' | head -n 1 )
    if [[ -n $container_id ]]; then
      echo "Restarting kube-apiserver Pod using Docker: $container_id"
      docker restart "${container_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  elif [[ $cmd == "crictl" ]]; then
    # Use the crictl command to restart the kube-apiserver Pod.
    pod_id=$(crictl pods --label component=kube-apiserver --latest --state=ready | grep -v "POD ID" | head -n 1 | awk '{print $1}')
    if [[ -n $pod_id ]]; then
      echo "Restarting kube-apiserver Pod using crictl: $pod_id"
      crictl stopp "${pod_id}"
    else
      echo "kube-apiserver Pod not found."
    fi
  else
    echo "Unsupported container runtime: $cmd"
  fi
  k8s::wait_apiserver_ready
}

check_container_runtime
restart_apiserver
echo "API server restarted"

The rollback-etcd.sh script

#!/bin/bash

set -eo pipefail

declare -x TARGET_TEAR
declare -x cmd
dir=/tmp/rollback/etcdcert
KUBE_CERT_PATH=/etc/kubernetes/pki
ETCD_CERT_DIR=/var/lib/etcd/cert
ETCD_HOSTS=""
currentDir="$PWD"

# Renew the Kubernetes certificates. Replace the default image region cn-hangzhou with the region of the cluster.
function get_etcdhosts() {
  name1=$(find "$ETCD_CERT_DIR" -name '*-name-1.pem' -exec basename {} \; | sed 's/-name-1.pem//g')
  name2=$(find "$ETCD_CERT_DIR" -name '*-name-2.pem' -exec basename {} \; | sed 's/-name-2.pem//g')
  name3=$(find "$ETCD_CERT_DIR" -name '*-name-3.pem' -exec basename {} \; | sed 's/-name-3.pem//g')

  echo "hosts: $name1 $name2 $name3"
  ETCD_HOSTS="$name1 $name2 $name3"
}

function gencerts() {
  echo "generate ssl cert ..."
  rm -rf $dir
  mkdir -p "$dir"
  cd $dir

  local hosts
  hosts=$(echo $ETCD_HOSTS | tr -s " " ",")

  echo "generate ca"
  echo '{"CN":"CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' |
    cfssl gencert -initca - | cfssljson -bare $dir/ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/ca-config.json

  echo "generate etcd server certificates"
  export ADDRESS=$hosts,ext1.example.com,coreos1.local,coreos1,127.0.0.1
  export NAME=etcd-server
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME
  export ADDRESS=
  export NAME=etcd-client
  echo '{"CN":"'$NAME'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
    cfssl gencert -config=$dir/ca-config.json -ca=$dir/ca.pem -ca-key=$dir/ca-key.pem -hostname="$ADDRESS" - | cfssljson -bare $dir/$NAME

  # gen peer-ca
  echo "generate peer certificates"
  echo '{"CN":"Peer-CA","key":{"algo":"rsa","size":2048}, "ca": {"expiry": "438000h"}}' | cfssl gencert -initca - | cfssljson -bare $dir/peer-ca -
  echo '{"signing":{"default":{"expiry":"438000h","usages":["signing","key encipherment","server auth","client auth"]}}}' >$dir/peer-ca-config.json
  i=0
  for host in $ETCD_HOSTS; do
    ((i = i + 1))
    export MEMBER=${host}-name-$i
    echo '{"CN":"'${MEMBER}'","hosts":[""],"key":{"algo":"rsa","size":2048}}' |
      cfssl gencert -ca=$dir/peer-ca.pem -ca-key=$dir/peer-ca-key.pem -config=$dir/peer-ca-config.json -profile=peer \
        -hostname="$hosts,${MEMBER}.local,${MEMBER}" - | cfssljson -bare $dir/${MEMBER}
  done

  # chown
  chown -R etcd:etcd $dir
  chmod 0644 $dir/*

  for ADDR in $ETCD_HOSTS; do
    printf "sync the certificates of node %s" "${ADDR}"
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" mkdir -p "${dir}"
    scp -o StrictHostKeyChecking=no "${dir}"/* root@"${ADDR}":/var/lib/etcd/cert/
    scp -o StrictHostKeyChecking=no "${dir}"/ca.pem "${dir}"/etcd-client.pem "${dir}"/etcd-client-key.pem root@"${ADDR}":/etc/kubernetes/pki/etcd/
  done
}

function generate_cm() {
  echo "generate status configmap"

  cat <<-"EOF" >/tmp/ack-rotate-etcd-ca-cm.yaml.tpl
apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-rotate-etcd-status
  namespace: kube-system
data:
  status: "success"
  hosts: "$hosts"
EOF

  sed -e "s#\$hosts#$ETCD_HOSTS#" /tmp/ack-rotate-etcd-ca-cm.yaml.tpl | kubectl apply -f -
}

function rotate_etcd() {
  for ADDR in $ETCD_HOSTS; do
    printf "rotate etcd's certificates on node %s\n" "${ADDR}"
      if [ "$cmd" == "docker" ]; then
        echo "restart docker on node $ADDR"
        ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart docker
      fi
    ssh -e none -o StrictHostKeyChecking=no root@$ADDR systemctl restart etcd
  done
}

function rotate_apiserver() {
  echo "current dir: $currentDir"
  for ADDR in $ETCD_HOSTS; do
    printf "restart apiserver on node %s\n" "${ADDR}"
    scp -o StrictHostKeyChecking=no "${currentDir}"/restart-apiserver.sh root@"${ADDR}":/tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" systemctl restart kubelet
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" chmod +x /tmp/restart-apiserver.sh
    ssh -e none -o StrictHostKeyChecking=no root@"${ADDR}" bash /tmp/restart-apiserver.sh
  done
}

function check_etcd_cluster_ready() {
  local etcd_endpoints=()
  for ip in $ETCD_HOSTS; do
    etcd_endpoints+=("https://$ip:2379")
  done

  for i in $(seq 300); do
    for idx in "${!etcd_endpoints[@]}"; do
      endpoint="${etcd_endpoints[$idx]}"
      local health_output=$(ETCDCTL_API=3 etcdctl --cacert=/var/lib/etcd/cert/ca.pem --cert=/var/lib/etcd/cert/etcd-server.pem --key=/var/lib/etcd/cert/etcd-server-key.pem --endpoints "$endpoint" endpoint health --command-timeout=1s 2>&1)
      if echo "$health_output" | grep -q "successfully committed proposal"; then
          unset 'etcd_endpoints[$idx]'
      else
          echo "etcdctl result: ${health_output}"
          echo "$endpoint is not ready"
      fi
    done
    # shellcheck disable=SC2199
    if [[ -z "${etcd_endpoints[@]}" ]]; then
      echo "ETCD cluster is ready"
      break
    fi
    sleep 1
    printf "wait etcd cluster to be ready, retry %d after 1s,total 300s \n" "$i"
  done
}

function get_region_id() {
    set +e; # close error out
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    local path=100.100.100.200/latest/meta-data/region-id
    for (( i=0; i<3; i++));
    do
        response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" --retry 1 --retry-delay 5 -sSL $path)
        if [[ $? -gt 0 || "x$response" == "x" ]];
        then
            sleep 2; continue
        fi
        if echo "$response"|grep -E "<title>.*</title>" >/dev/null;
        then
            sleep 3; continue
        fi
        echo "$response"
        # return from metadata succeed.
        set -e; return
    done
    set -e # open error out
    # function will return empty string when failed
}

function is_vpc() {
    # Execute the curl command and capture the network-type from ECS meta-server
    local TOKEN=`curl -X PUT "http://100.100.100.200/latest/api/token" \
    -H "X-aliyun-ecs-metadata-token-ttl-seconds:900"`
    response=$(curl -H "X-aliyun-ecs-metadata-token: $TOKEN" -s http://100.100.100.200/latest/meta-data/network-type)
    if [ "$response" = "vpc" ]; then
      return 0
    else
      return 1
    fi
}

function renew_k8s_certs() {
  # try to get region id from meta-server if not given in parameter
  META_REGION=$(get_region_id)
  if [[ -z "$REGION" ]]; then
    if [[ -z "$META_REGION" ]]; then
        echo "failed to get region id from ECS meta-server, please enter the region parameter."
        return 1
    fi
    REGION=$META_REGION
  elif [[ -n "${META_REGION}" && "$REGION" != "$META_REGION" ]] ; then
    echo "switch to use local region id $META_REGION"
    REGION=$META_REGION
  fi
  # Update certs for k8s components and kubeconfig
  for ADDR in $ETCD_HOSTS; do
    echo "renew k8s components cert on node $ADDR"
    #compatible containerd
    set +e
    IMAGE="registry.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    if is_vpc; then
      IMAGE="registry-vpc.$REGION.aliyuncs.com/acs/etcd-rotate:v2.0.0"
    fi
    echo "will pull rotate image $IMAGE"
    ssh -o StrictHostKeyChecking=no root@$ADDR docker run --privileged=true  -v /:/alicoud-k8s-host --pid host --net host \
             $IMAGE /renew/upgrade-k8s.sh --role master
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr image pull $IMAGE
    ssh -o StrictHostKeyChecking=no root@$ADDR ctr run --privileged=true --mount type=bind,src=/,dst=/alicoud-k8s-host,options=rbind:rw \
            --net-host $IMAGE cert-rotate /renew/upgrade-k8s.sh --role master
    set -e
    echo "finished renew k8s components cert on $ADDR"
  done
}

function check_container_runtime() {
  if command -v dockerd &>/dev/null && ps aux | grep -q "[d]ockerd"; then
    cmd=docker
  elif command -v containerd &>/dev/null && ps aux | grep -q "[c]ontainerd"; then
    cmd=crictl
  else
    echo "Neither Dockerd nor Containerd is installed or running."
    exit 1
  fi
}

while
    [[ $# -gt 0 ]]
do
    key="$1"

    case $key in
    --region)
      export REGION=$2
      shift
      ;;
    *)
      echo "unknown option [$key]"
      exit 1
      ;;
    esac
    shift
done

get_etcdhosts
printf "ETCD_HOSTS: %s\n" "$ETCD_HOSTS"

gencerts
echo "---generate certificates successfully---"

rotate_etcd
echo "---rotate etcd successfully---"

echo "---check etcd cluster ready---"
check_etcd_cluster_ready

rotate_apiserver
echo "---restart apiserver successfully---"

echo "---renew k8s components certs---"
renew_k8s_certs
echo "---end to renew k8s components certs---"

generate_cm
echo "etcd CA and certs have successfully rotated!"

rm -rf $dir

Step 2: Run the rollback script

On any master node, run:

bash rollback-etcd.sh

If the output ends with etcd CA and certs have successfully rotated!, the rollback is complete.

Step 3: Verify the rollback

cd /var/lib/etcd/cert
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/etcd
for i in `ls | grep pem| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done


cd /etc/kubernetes/pki/
for i in `ls | grep crt| grep -v key`;do openssl x509 -noout -text -in $i | grep -i after && echo "$i" ;done

The rollback is complete if all expiration dates are more than 50 years in the future.