Update expiring certificates for master and worker nodes in an ACK dedicated cluster - Container Service for Kubernetes

Renew all node certificates in an ACK dedicated cluster via the console, kubectl, or manual Kubernetes jobs.

Note

ACK automatically updates the master node certificates in an ACK managed cluster. No manual operation required.

Update all node certificates in the console

Log on to the ACK console. In the left navigation pane, click Clusters.
Click Update Certificate next to the cluster with an expiring certificate. The Update Certificate page opens.

Note
The Update Certificate button appears for clusters whose certificates expire in about two months.

In the cluster list, click Update Certificate in the Actions column.
On the Update Certificate page, click Update Certificate and follow the prompts.

After the update:
- The Update Certificate page displays a success message such as The certificate has been updated..
- On the Clusters page, the Update Certificate prompt for the cluster disappears.

Automatically update all node certificates with kubectl

Update certificates

On any master node, update all node certificates:

curl http://aliacs-k8s-cn-hangzhou.oss-cn-hangzhou.aliyuncs.com/public/cert-update/renew.sh | bash

Verify the result

Ensure you are connected to the cluster with kubectl. See Connect to a Kubernetes cluster by using kubectl.

View the status of master and worker nodes.

kubectl get nodes

[root@xxx ~]# kubectl get nodes
NAME                      STATUS    ROLES     AGE       VERSION
cn-hangzhou.xxx           Ready     <none>    23d       v1.11.2
cn-hangzhou.xxx           Ready     <none>    23d       v1.11.2
cn-hangzhou.xxx           Ready     master    47d       v1.11.2
cn-hangzhou.xxx           Ready     master    47d       v1.11.2
cn-hangzhou.xxx           Ready     master    47d       v1.11.2
cn-hangzhou.xxx           Ready     <none>    47d       v1.11.2
cn-hangzhou.xxx           Ready     <none>    47d       v1.11.2
[root@xxx ~]#

Check job status. The update is complete when COMPLETIONS for each master node job is 1/1 and the worker node job completions match the total worker nodes in the cluster.

kubectl -n kube-system get job

[root@                              ~]# kubectl get job -nkube-system
NAME                                  COMPLETIONS   DURATION   AGE
aliyun-cert-renew-master-1            1/1           46s        4m49s
aliyun-cert-renew-master-2            1/1           28s        4m19s
aliyun-cert-renew-master-3            1/1           26s        3m48s
aliyun-cert-renew-worker              6/6           46s        3m18s
ingress-nginx-admission-create        1/1           29s        3d
ingress-nginx-admission-patch         1/1           43s        3d
kube-eventer-init-1.5-5e0e7cl-aliyun  1/1           31s        3d

Manually update master node certificates

In any directory, create a job-master.yml file with the following content.

apiVersion: batch/v1
kind: Job
metadata:
  name: ${jobname}
  namespace: kube-system
spec:
  backoffLimit: 0
  completions: 1
  parallelism: 1
  template:
    spec:
      activeDeadlineSeconds: 3600
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - ${hostname}
      containers:
      - command:
        - /renew/upgrade-k8s.sh
        - --role
        - master
        image: registry.cn-hangzhou.aliyuncs.com/acs/cert-rotate:v1.0.0
        imagePullPolicy: Always
        name: ${jobname}
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /alicoud-k8s-host
          name: ${jobname}       
      hostNetwork: true
      hostPID: true
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/master
      volumes:
      - hostPath:
          path: /
          type: Directory
        name: ${jobname}

Obtain the number and names of the master nodes in the cluster.

Method 1: Use the command line

Run the following command:

kubectl get nodes

[root@xxx            ~]# kubectl get nodes
NAME               STATUS    ROLES     AGE    VERSION
cn-hangzhou.ixxx   Ready     <none>    22d    v1.11.2
cn-hangzhou.ixxx   Ready     <none>    22d    v1.11.2
cn-hangzhou.ixxx   Ready     master    46d    v1.11.2
cn-hangzhou.ixxx   Ready     master    46d    v1.11.2
cn-hangzhou.ixxx   Ready     master    46d    v1.11.2
cn-hangzhou.ixxx   Ready     <none>    46d    v1.11.2
cn-hangzhou.ixxx   Ready     <none>    46d    v1.11.2
[root@xxx            ~]#

Method 2: Use the console
1. Log on to the ACK console. In the left navigation pane, click Clusters.
2. On the Clusters page, click the name of the destination cluster or click Details in the Actions column.
3. In the left-side navigation pane of the cluster management page, choose Nodes > Nodes to view master node count, names, IPs, and instance IDs.

Replace the ${jobname} and ${hostname} variables in job-master.yml.
```
sed 's/${jobname}/cert-job-2/g; s/${hostname}/hostname/g' job-master.yml > job-master2.yml
```
Parameters:
- ${jobname}: the job name. In this example, set to cert-job-2.
- ${hostname}: a master node name. Replace hostname with the name obtained in Step 2.
Create the job.
```
kubectl create -f job-master2.yml
```
Check the job status. The certificate is updated when COMPLETIONS is 1/1.
```
kubectl get job -nkube-system
```

Repeat Steps 3-5 for each remaining master node.

[root@xxx ~]# kubectl get job -nkube-system
NAME          COMPLETIONS   DURATION   AGE
cert-job-2    1/1           46s        22m
cert-job-3    1/1           28s        2m
cert-job-4    1/1           26s        1m
[root@xxx ~]#

Manually update worker node certificates

In any directory, create a job-node.yml file with the following content.

apiVersion: batch/v1
kind: Job
metadata:
  name: ${jobname}
  namespace: kube-system
spec:
  backoffLimit: 0
  completions: ${nodesize}
  parallelism: ${nodesize}
  template:
    spec:
      activeDeadlineSeconds: 3600
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: job-name
                operator: In
                values:
                - ${jobname}
            topologyKey: kubernetes.io/hostname
      containers:
      - command:
        - /renew/upgrade-k8s.sh
        - --role
        - node
        - --rootkey
        - ${key}
        image: registry.cn-hangzhou.aliyuncs.com/acs/cert-rotate:v1.0.0
        imagePullPolicy: Always
        name: ${jobname}
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /alicoud-k8s-host
          name: ${jobname}
      hostNetwork: true
      hostPID: true
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext: {}
      volumes:
      - hostPath:
          path: /
          type: Directory
        name: ${jobname}

Note

If a worker node has a Taint, add a tolerations for that Taint to the job-node.yml file. Add the following content between securityContext: {} and volumes: (if you have n worker nodes with Taints, copy this section n times):

      tolerations:
      - effect: NoSchedule
        key: ${key}
        operator: Equal
        value: ${value}

To obtain ${key} and ${value}:

In any directory, create a taint.tml file with the following content.

{{printf "%-50s %-12s\n" "Node" "Taint"}}
                    {{- range .items}}
                    {{- if $taint := (index .spec "taints") }}
                    {{- .metadata.name }}{{ "\t" }}
                    {{- range $taint }}
                    {{- .key }}={{ .value }}:{{ .effect }}{{ "\t" }}
                    {{- end }}
                    {{- "\n" }}
                    {{- end}}
                    {{- end}}

Query the ${name} and ${value} of the worker node with a Taint.

kubectl get nodes -o go-template-file="taint.tml"

[root@xxx                       ~]# kubectl get nodes -o go-template-file="taint.tml"
Node                              Taint
cn-hangzhou.i-xxx                 key1=value1:NoSchedule
cn-hangzhou.i-xxx                 node-role.kubernetes.io/master=<no value>:NoSchedule
cn-hangzhou.i-xxx                 node-role.kubernetes.io/master=<no value>:NoSchedule
cn-hangzhou.i-xxx                 node-role.kubernetes.io/master=<no value>:NoSchedule

Obtain the CA key of the cluster.

sed '1d' /etc/kubernetes/pki/ca.key | base64 -w 0

Replace the ${jobname}, ${nodesize}, and ${key} variables in job-node.yml.
```
sed 's/${jobname}/cert-node-2/g; s/${nodesize}/nodesize/g; s/${key}/key/g' job-node.yml > job-node2.yml
```
Parameters:
- ${jobname}: the job name. In this example, set to cert-node-2.
- ${nodesize}: the worker node count. See Step 2 in Manually update master node certificates to obtain the worker node countManually update master node certificates. Replace nodesize with the actual count.
- ${key} is the cluster CAKey. Replace key with the CAKey from Step 2 of Manually update the certificates of worker nodes.
Create the job.
```
kubectl create -f job-node2.yml
```

Check the job status. The certificates are updated when COMPLETIONS matches the number of worker nodes.

kubectl get job -nkube-system

[root@xxx ~]# kubectl get job -nkube-system
NAME          COMPLETIONS   DURATION   AGE
cert-job-2    1/1           46s        1h
cert-job-3    1/1           28s        47m
cert-job-4    1/1           26s        46m
cert-node-2   4/4           46s        1m
[root@xxx ~]#