How to deploy CCM in a self-managed Kubernetes cluster - Elastic Container Instance

The cloud controller manager (CCM) integrates Kubernetes with Alibaba Cloud infrastructure services, such as Classic Load Balancer (CLB) and Virtual Private Cloud (VPC). It allows you to attach both in-cluster nodes and out-of-cluster servers to the same CLB backend, helping prevent traffic disruption during migration. You can also forward business traffic to multiple Kubernetes clusters for backup and disaster recovery, ensuring high availability for your applications. This topic describes how to deploy CCM in a self-managed Kubernetes cluster.

Prerequisites

A VNode has been deployed in your self-managed Kubernetes cluster.
If your Kubernetes cluster is deployed in an on-premises data center, ensure that network connectivity is established between the data center and Alibaba Cloud.

Background information

CCM is an Alibaba Cloud component that integrates Kubernetes with Alibaba Cloud infrastructure services. It provides the following capabilities:

Manages load balancing
When the type of a Service is set to LoadBalancer, CCM creates and configures an Alibaba Cloud Classic Load Balancer (CLB) for the Service, including resources such as the CLB instance, listeners, and backend server groups. When the Endpoints of the Service or the cluster nodes change, CCM automatically updates the backend server group of the CLB.
Enables cross-node communication
When using Flannel as the cluster network component, CCM establishes network connectivity between containers and nodes. It writes the pod CIDR block of each node to the VPC route table, enabling cross-node communication for containers. This feature is enabled by default and requires no configuration.

For more information, see cloud controller manager.

Note

CCM is open source. For project details, see cloud-provider-alibaba-cloud.

Preparations

Skip this section if your self-managed Kubernetes cluster does not use Elastic Compute Service (ECS) instances as nodes. Otherwise, follow these steps to configure the providerID for your ECS nodes, which allows CCM to manage their routes.

Deploy OpenKruise to use the BroadcastJob controller.
Run the following commands:
```
helm repo add openkruise https://openkruise.github.io/charts/
helm repo update
helm install kruise openkruise/kruise --version 1.3.0
```
For more information, see the OpenKruise documentation.

Configure the providerID for ECS nodes by using a BroadcastJob.

Save the following content as provider.yaml.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ecs-node-initor
rules:
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - patch
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ecs-node-initor
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: ecs-node-initor
subjects:
- kind: ServiceAccount
  name: ecs-node-initor
  namespace: default
roleRef:
  kind: ClusterRole
  name: ecs-node-initor
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: apps.kruise.io/v1alpha1
kind: BroadcastJob
metadata:
  name: create-ecs-node-provider-id
spec:
  template:
    spec:
      serviceAccount: ecs-node-initor
      restartPolicy: OnFailure
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: type
                operator: NotIn
                values:
                - virtual-kubelet
      tolerations:
      - operator: Exists
      containers:
        - name: create-ecs-node-provider-id
          image: registry.cn-beijing.aliyuncs.com/eci-release/provider-initor:v1
          command: [ "/usr/bin/init" ]
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
  completionPolicy:
    type: Never
  failurePolicy:
    type: FailFast
    restartLimit: 3

Deploy the BroadcastJob.
```
kubectl apply -f provider.yaml
```

Check the BroadcastJob results.

kubectl get pods -o wide

If all pods related to create-ecs-node-provider-id are in the Completed state, the providerID for the corresponding ECS nodes has been successfully configured. The following is a sample output:

NAME                              READY   STATUS      RESTARTS   AGE   IP              NODE                        NOMINATED NODE   READINESS GATES
create-ecs-node-provider-id-9jvms 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.206   <none>           1/1
create-ecs-node-provider-id-f5sb8 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.202   <none>           1/1
create-ecs-node-provider-id-f6r2c 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.201   <none>           1/1
create-ecs-node-provider-id-nhrsp 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.203   <none>           1/1
create-ecs-node-provider-id-ntn4d 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.204   <none>           1/1
create-ecs-node-provider-id-rr2bk 0/1     Completed   0          34m   172.23.xxx      cn-shanghai.10.10.156.205   <none>           1/1

(Optional) Clean up the BroadcastJob.
```
kubectl delete -f provider.yaml
```

Procedure

Create a ConfigMap.

Save the AccessKey pair for your Alibaba Cloud account as environment variables.
```
export ACCESS_KEY_ID=LTAI********************
export ACCESS_KEY_SECRET=HAeS**************************
```
For information about how to obtain an AccessKey ID and an AccessKey Secret, see Obtain an AccessKey pair.

Run the following script to create the ConfigMap.

Save the following content as configmap-ccm.sh, replace the region value with your region ID, and then run the script.

#!/bin/bash
## create ConfigMap kube-system/cloud-config for CCM.
accessKeyIDBase64=`echo -n "$ACCESS_KEY_ID" |base64 -w 0`
accessKeySecretBase64=`echo -n "$ACCESS_KEY_SECRET"|base64 -w 0`
cat <<EOF >cloud-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloud-config
  namespace: kube-system
data:
  cloud-config.conf: |-
    {
        "Global": {
            "accessKeyID": "$accessKeyIDBase64",
            "accessKeySecret": "$accessKeySecretBase64",
            "region": "cn-hangzhou"
        }
    }
EOF
kubectl create -f cloud-config.yaml

bash configmap-ccm.sh

After you run the script, a ConfigMap named cloud-config is created in the kube-system namespace.

Deploy CCM.

Modify ${ImageVersion} and {$ClusterCIDR}, and then save the following content as ccm.yaml.

You can obtain the ImageVersion from the CCM changelog. For more information, see cloud controller manager.
You can view the ClusterCIDR by running the kubectl cluster-info dump | grep -m1 cluster-cidr command.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:cloud-controller-manager
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs:
      - get
      - list
      - update
      - create
  - apiGroups:
      - ""
    resources:
      - persistentvolumes
      - services
      - secrets
      - endpoints
      - serviceaccounts
    verbs:
      - get
      - list
      - watch
      - create
      - update
      - patch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
      - delete
      - patch
      - update
  - apiGroups:
      - ""
    resources:
      - services/status
    verbs:
      - update
      - patch
  - apiGroups:
      - ""
    resources:
      - nodes/status
    verbs:
      - patch
      - update
  - apiGroups:
      - ""
    resources:
      - events
      - endpoints
    verbs:
      - create
      - patch
      - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cloud-controller-manager
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:cloud-controller-manager
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: cloud-controller-manager
    namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:shared-informers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: shared-informers
    namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:cloud-node-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: cloud-node-controller
    namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:pvl-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: pvl-controller
    namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:route-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:cloud-controller-manager
subjects:
  - kind: ServiceAccount
    name: route-controller
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: cloud-controller-manager
    tier: control-plane
  name: cloud-controller-manager
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cloud-controller-manager
      tier: control-plane
  template:
    metadata:
      labels:
        app: cloud-controller-manager
        tier: control-plane
      annotations:
        scheduler.alpha.kubernetes.io/critical-pod: ''
    spec:
      serviceAccountName: cloud-controller-manager
      tolerations:
        - effect: NoSchedule
          operator: Exists
          key: node-role.kubernetes.io/master
        - effect: NoSchedule
          operator: Exists
          key: node.cloudprovider.kubernetes.io/uninitialized
      nodeSelector:
        node-role.kubernetes.io/master: ""
      containers:
        - command:
          -  /cloud-controller-manager
          - --leader-elect=true
          - --cloud-provider=alicloud
          - --use-service-account-credentials=true
          - --cloud-config=/etc/kubernetes/config/cloud-config.conf
          - --configure-cloud-routes=true
          - --route-reconciliation-period=3m
          - --leader-elect-resource-lock=endpoints
          # Replace ${ClusterCIDR} with your own cluster CIDR.
          # Example: 172.16.0.0/16
          - --cluster-cidr=${ClusterCIDR}  
          # Replace ${ImageVersion} with the latest release version.
          # Example: v2.1.0
          image: registry.cn-hangzhou.aliyuncs.com/acs/cloud-controller-manager-amd64:${ImageVersion}
          livenessProbe:
            failureThreshold: 8
            httpGet:
              host: 127.0.0.1
              path: /healthz
              port: 10258
              scheme: HTTP
            initialDelaySeconds: 15
            timeoutSeconds: 15
          name: cloud-controller-manager
          resources:
            requests:
              cpu: 200m
          volumeMounts:
            - mountPath: /etc/kubernetes/
              name: k8s
            - mountPath: /etc/ssl/certs
              name: certs
            - mountPath: /etc/pki
              name: pki
            - mountPath: /etc/kubernetes/config
              name: cloud-config
      hostNetwork: true
      volumes:
        - hostPath:
            path: /etc/kubernetes
          name: k8s
        - hostPath:
            path: /etc/ssl/certs
          name: certs
        - hostPath:
            path: /etc/pki
          name: pki
        - configMap:
            defaultMode: 420
            items:
              - key: cloud-config.conf
                path: cloud-config.conf
            name: cloud-config
          name: cloud-config

Run the following command to deploy CCM.
```
kubectl create -f ccm.yaml
```

Verify the deployment

Create a LoadBalancer Service and a sample backend Deployment.

Save the following content as ccm-test.yaml.

To prevent image pull failures, replace the image address with one from your VNode's region.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: default
  annotations:
    service.beta.kubernetes.io/alibaba-cloud-loadbalancer-address-type: "intranet"
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: nginx
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registry-vpc.cn-beijing.aliyuncs.com/eci_open/nginx:1.14.2

Deploy the Service and Deployment.
```
kubectl create -f ccm-test.yaml
```
After deployment, CCM creates and configures an Alibaba Cloud CLB for the Service, including the CLB instance, listeners, and backend server groups.

Verify that the Service works as expected.

Run the curl command with the Service's external IP address. A successful response, like the example below, confirms that the backend Nginx service is accessible.

[root@k8s-master ~]# kubectl get service
NAME           TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)        AGE
kubernetes     ClusterIP      10.xxx.x.1      <none>         443/TCP        7d1h
nginx          LoadBalancer   10.xxx.02.253   172.16.5.xxx   80:32554/TCP   14m
[root@k8s-master ~]# kubectl get pods | grep nginx
test-nginx-84cbc7685b-gvlvq   1/1     Running   0          22s
test-nginx-84cbc7685b-w9np2   1/1     Running   0          22s
[root@k8s-master ~]# curl 172.16.5.xxx
&lt;!DOCTYPE html&gt;
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.&lt;br/&gt;
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

For more information about using Services, see CCM usage.