Community Blog Self-built Kubernetes on Alibaba Cloud

Self-built Kubernetes on Alibaba Cloud

In this article, we will be setting up a self-built kubernetes on Alibaba Cloud Elastic Compute Service instances using Linux flavors (Centos7 and Ubuntu 16.

By Anish Nath, Alibaba Cloud Tech Share Author. Tech Share is Alibaba Cloud's incentive program to encourage the sharing of technical knowledge and best practices within the cloud community.

Alibaba Cloud Container Service for Kubernetes is a fully-managed service compatible with Kubernetes to help users focus on their applications rather than managing container infrastructure. There are two ways to deploy Kubernetes on Alibaba Cloud, one through Container Service (built-in) and the other through an Elastic Compute Service (ECS) instance (self-built). If you are not sure which installation method suits your needs better, then refer to the documentation Alibaba Cloud Kubernetes vs. self-built Kubernetes.

For the most part, choosing Alibaba Cloud Container Service is the preferred choice as it saves time and reduces the complexity to maintain Kubernetes clusters. However, there may be cases where a manual installation is better suited to your needs. In this article, we will be setting up a self-built kubernetes on Alibaba Cloud Elastic Compute Service instances using Linux flavors (Centos7 and Ubuntu 16.04).


  1. A valid Alibaba Cloud account. If you don't have one already, sign up to the Free Trial to enjoy $300 worth in Alibaba Cloud products.
  2. An ECS instance running Ubuntu 16.04 or RHEL7 or Centos7. You can select your preferred region and configurations; this will not affect the outcome of the server setup.
  3. A sudo password for your server.

Kubernetes Security Group Setup

These required ports needs to be opened in the pod security group, to do this:

  1. Log on to the ECS console.
  2. Select a region.
  3. In the left-side navigation pane, select Networks & Security and select Security Group.
  4. Click Create Security Group.
  5. Set the Inbound rules for your pod network as shown below. For my example, is the pod network IP address; you'll need to replace it with your own.
Authorization Object Protocol Type Port Range Authorization Policy TCP 6443/6443 Allow TCP 2379/2379 Allow TCP 2380/2380 Allow TCP 10250/10250 Allow TCP 10251/10251 Allow TCP 10252/10252 Allow TCP 22/22 Allow


  1. Alibaba Cloud Container Service and the pod CIDR block cannot overlap with the VPC CIDR block.
  2. The service CIDR block cannot overlap with the VPC CIDR block or pod CIDR block.
  3. The security group rules is enable communication between kubernetes master and cluster.

Connect to Your Alibaba Cloud Server

Locate the Internet IP address (Public IP address) associated with your Alibaba Cloud ECS Instance. There are other ways to connect to your ECS instance as well. Visit the official ECS documentation to learn more.

Before proceeding, stop and check whether:

  1. ECS public IP is enabled.
  2. ECS is up and running.
  3. ECS security group is pointed to pods-security group.
  4. ECS security group Authorization Object matches with ECS instance.

Setup Environment

This example utilizes two ECS instances running in the Alibaba Could environment and with these hostnames

Master Server name Minion Cluster
kube-master kube-minion-1

Install Kubernetes-master and Kubernetes-minion

These are the minimum requirements to setup kubernetes ECS instance master and minion.

Requirement Kubernetes-master Kubernetes-minion
Disable system swap and SELinux Y Y
remove any swap entry from /etc/fstab Y Y
net.bridge.bridge-nf-call-iptables is set to 1 Y Y
Install Docker & enable on restart Y Y
Install kubeadm Y Y
Install kubelet Y N
Install kubectl Y N
Configure docker cgroupsfs Y N
Creating Network Addons (flannel/Calico) Y N

Disable system swap with swapoff -a. This will immediately disable swap and remove any swap entry from /etc/fstab

Disable SE Linux by setenforce 0

You should ensure net.bridge.bridge-nf-call-iptables is set to 1 in your sysctlconfig

cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
sysctl --system

If you are running CentOS or RHEL7, install Docker and enable on restart with these commands.

yum -y update
yum install -y docker 

If you are running Ubuntu, install Docker and enable on restart with these commands.

apt-get update 
apt-get install -y docker.io

Start and enable Docker and check if docker service is running.

systemctl start docker
systemctl enable docker
systemctl status docker

Install Kubernetes

You will need to install these packages on all of your machines:

  1. kubeadm: the command to bootstrap the cluster.
  2. kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers.
  3. kubectl: the command line util to talk to your cluster.

Kubernetes Installation on CentOS7/RHEL7

Setting up Kubernetes yum repository

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

Install Kubernetes in CentOS7/RHEL7

yum install -y kubelet kubeadm kubectl

Kubernetes Installation on Ubuntu

Setting up Kubernetes apt repository for Ubuntu

apt-get install -y apt-transport-https curl
apt-get install -y docker.io
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb http://apt.kubernetes.io/ kubernetes-xenial main
apt-get update

Install Kubernetes in Ubuntu

apt-get install -y kubelet kubeadm kubectl

Running Kubernetes

Enable & start kublet

systemctl enable kubelet  
systemctl start kubelet

Verify that your Docker cgroup driver matches the kubelet config: (kube-master node)

docker info | grep -i cgroup
Cgroup Driver: cgroupfs

Configuration for cgroup drive is right in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf


Reload daemon and restart kubelet: (kube-master node)

systemctl daemon-reload
systemctl restart kubelet

Flush Reset Kubernetes component (This will wipeout all the cluster config if exist any do not run this after the config create)

kubeadm reset -f 

Set up Kubernetes Network add on

  1. For flannel to work correctly, --pod-network-cidr= has to be passed to kubeadm init.
  2. For Calico to work correctly, --pod-network-cidr= has to be passed to kubeadm init.

Creating flannel Networks

Note apiserver-advertise-address is the IP of the kube-master

kubeadm init --service-cidr --kubernetes-version v1.11.0 --pod-network-cidr --apiserver-advertise-address

You should get information back on initiating commands as a normal user, as well as the network that you need to deploy as well as how to join worker nodes to the cluster.

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You can now join any number of machines by running the following on each node
as root:

Join the Cluster (kube-minnion)

kubeadm join --token 5m8qxr.46rpadiwt8fcka0v --discovery-token-ca-cert-hash sha256:b05a0b8849a57432247c06200864f5ce99d40ffdcae965293c0026204ef33da4

Run kubectl get nodes on the master to see this node join the cluster. It will few seconds to get your cluster in ready state

root@kube-master:kubectl get nodes
NAME          STATUS    ROLES     AGE       VERSION
kube-master   Ready     master    2m        v1.11.0
kube-minion   Ready     <none>    47s       v1.11.0

Apply flannel Addons

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

You will get output like this

clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds created

Enable and restart kubelet engine

systemctl restart kubelet
systemctl status kubelet

Verifying the Installation

Verify the Cluster Information

root@kube-master: kubectl cluster-info 
Kubernetes master is running at
KubeDNS is running at

Verify the Services

root@kube-master:kubectl get services -n kube-system
NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
kube-dns               ClusterIP      <none>        53/UDP,53/TCP   2m

Create Tokens

[root@kube-master ~]kubeadm token create 
I0710 04:08:37.149017    8685 feature_gate.go:230] feature gates: &{map[]}

That's it! You have successfully configured a self-built Kubernetes cluster on Alibaba Cloud. To learn more about Kubernetes on Alibaba Cloud, visit Alibaba Cloud Container Service for Kubernetes.

0 0 0
Share on

Alibaba Clouder

2,605 posts | 746 followers

You may also like