×
Community Blog Provisioning an Alibaba Cloud Kubernetes (ACK) Multi-AZ Kubernetes Cluster Using Terraform

Provisioning an Alibaba Cloud Kubernetes (ACK) Multi-AZ Kubernetes Cluster Using Terraform

In this post, we will show you how to use Terraform to integrate resources on Alibaba Cloud with an application provisioned on ACK cluster.

By Yan Tsz Kin, Solutions Architect

1

The purpose of this article is to present a simple and practical example of provisioning an Alibaba Cloud Kubernetes (ACK) Multi-AZ Kubernetes cluster with a sample application on Alibaba Cloud using Terraform. This is also an example of Infrastructure as Code which is a key example of enabling best practices in DevOps.

I will walk through a sample project that is available in my GitHub repository: https://github.com/kin-alibaba/terraform-kubernetes

Prerequisites

  1. Terraform and providers installed according to https://www.alibabacloud.com/help/doc-detail/67384.htm
  2. Terraform 0.11.11 installed and runs on Linux
  3. Terraform AliCloud provider version 1.29
  4. Terraform Kubernetes provider version 1.5
  5. Docker image(s) is pushed to registry already
  6. A domain name is being managed in Alibaba Cloud DNS (e.g. seyantszkin.xyz)

Target Environment

Our target infrastructure includes:

  1. 1 Multi-AZ Kubernetes cluster in 2 Availability Zones (Max. 3 zones supported)
  2. 3 Kubernetes Master Nodes across 2 Availability Zones with 2 Server Load Balancer instances
  3. 3 Kubernetes Worker Nodes across 2 Availability Zones
  4. 1 NAT Gateway instance for downloading Docker image(s) from Docker Hub
  5. 1 Server Load Balancer (SLB) instance for Nginx Ingress Controller
  6. 1 Kubernetes Service serves sample web application connecting to Alibaba Cloud ApsaraDB RDS for MySQL (RDS)
  7. 1 Alibaba Cloud DNS record will be bind to the Kubernetes Service

Our target architecture is illustrated in the diagram below:

2

Terraform – Infrastructure as Code

Terraform allow us to describe the target environment, and then manage the lifecycle of the environment. In this case, Terraform files are modularized into multiple files for easier management in which variables are externalized to each "config" files.

Before Provisioning

In the accesskey.tf, update the Alibaba Cloud Access Key and Secret.

variable "access_key" {
default = ""
}
variable "secret_key" {
default = ""
}

In vpc_config.tf, update the Region and Availability Zone information, you may check out the Region ID at https://www.alibabacloud.com/help/doc-detail/40654.htm, Zone ID is typically appending –zone name to Region ID, e.g. Hong Kong Zone C is having Region ID cn-hongkong, Zone ID is cn-hongkong-c, Singapore Zone C is ap-southeast1c.

variable "region" {
default = "cn-hongkong"
}

variable "azone1" {
default = "cn-hongkong-b"
}

variable "azone2" {
default = "cn-hongkong-c"
}

variable "azone3" {
default = "cn-hongkong-c"
}

Make sure following files exist and empty:

  1. ~/.kube/cluster-ca-cert.pem
  2. ~/.kube/client-key.pem
  3. ~/.kube/client-cert.pem

This is critical for Terraform to proceed with Kubernetes provider after cluster provisioned.

Start Provisioning

Execute IaC.sh or command "terraform apply –auto-approve"

Create VPC and Vswitches

resource "alicloud_vpc" "vpc" {
name = "${var.vpc_name}"
cidr_block = "${var.vpc_cidr}"
}

resource "alicloud_vswitch" "vswitch1" {
availability_zone = "${var.azone1}"
name = "${var.vswitch_name1}"
cidr_block = "${var.vswitch_cidr1}"
vpc_id = "${alicloud_vpc.vpc.id}"
}

resource "alicloud_vswitch" "vswitch2" {
availability_zone = "${var.azone1}"
name = "${var.vswitch_name2}"
cidr_block = "${var.vswitch_cidr2}"
vpc_id = "${alicloud_vpc.vpc.id}"
}

resource "alicloud_vswitch" "vswitch3" {
availability_zone = "${var.azone3}"
name = "${var.vswitch_name3}"
cidr_block = "${var.vswitch_cidr3}"
vpc_id = "${alicloud_vpc.vpc.id}"
}

Create NAT Gateway and EIP instance and associate to the VPC

resource "alicloud_nat_gateway" "nat_gateway" {
  vpc_id = "${alicloud_vpc.vpc.id}"
  specification   = "Small"
  name   = "kin-k8s-tf-nat-gw"
  depends_on = [
     "alicloud_vswitch.vswitch1",
     "alicloud_vswitch.vswitch2",
     "alicloud_vswitch.vswitch3"
  ]
}

resource "alicloud_eip" "eip1" {
  bandwidth            = "20"
}

resource "alicloud_eip_association" "eip1_asso" {
  allocation_id = "${alicloud_eip.eip1.id}"
  instance_id   = "${alicloud_nat_gateway.nat_gateway.id}"
}

resource "alicloud_snat_entry" "snat1" {
  snat_table_id     = "${alicloud_nat_gateway.nat_gateway.snat_table_ids}"
  source_vswitch_id = "${alicloud_vswitch.vswitch1.id}"
  snat_ip           = "${alicloud_eip.eip1.ip_address}"
}

resource "alicloud_snat_entry" "snat2" {
  snat_table_id     = "${alicloud_nat_gateway.nat_gateway.snat_table_ids}"
  source_vswitch_id = "${alicloud_vswitch.vswitch2.id}"
  snat_ip           = "${alicloud_eip.eip1.ip_address}"
}

resource "alicloud_snat_entry" "snat3" {
  snat_table_id     = "${alicloud_nat_gateway.nat_gateway.snat_table_ids}"
  source_vswitch_id = "${alicloud_vswitch.vswitch3.id}"
  snat_ip           = "${alicloud_eip.eip1.ip_address}"
}

Create RDS instance

resource "alicloud_db_instance" "rdsinstance" {
    engine = "MySQL"
    engine_version = "5.6"
    instance_type = "rds.mysql.t1.small"
    instance_storage = "5"
    vswitch_id = "${alicloud_vswitch.vswitch1.id}"
    security_ips = ["${var.vswitch_cidr1}", "${var.vswitch_cidr2}", "${var.vswitch_cidr3}", "${alicloud_eip.eip1.ip_address}"]
    #depends_on = ["alicloud_cs_kubernetes.k8s-cluster"]
}

resource "alicloud_db_database" "rdsdb" {
    instance_id = "${alicloud_db_instance.rdsinstance.id}"
    name = "demodb"
    character_set = "utf8"
}

resource "alicloud_db_account" "mysqlroot" {
    instance_id = "${alicloud_db_instance.rdsinstance.id}"
    name = "${var.db_credential["username"]}"
    password = "${var.db_credential["password"]}"
}

resource "alicloud_db_account_privilege" "default" {
    instance_id = "${alicloud_db_instance.rdsinstance.id}"
    account_name = "${alicloud_db_account.mysqlroot.name}"
    privilege = "ReadWrite"
    db_names = ["${alicloud_db_database.rdsdb.name}"]
}

Create ACK cluster

resource "alicloud_cs_kubernetes" "k8s-cluster" {
    name = "${var.k8clu_name}"
    vswitch_ids = [                             #Indicates Multiple Availability Zone
        "${alicloud_vswitch.vswitch1.id}",
        "${alicloud_vswitch.vswitch2.id}",
        "${alicloud_vswitch.vswitch3.id}"
    ]
    master_instance_types = ["${var.master_type["zone1"]}","${var.master_type["zone1"]}","${var.master_type["zone2"]}"]
    master_disk_category = "cloud_efficiency"          #cloud_ssd or cloud_efficiency
    master_disk_size = "40"
    worker_instance_types = ["${var.worker_type["zone1"]}","${var.worker_type["zone1"]}","${var.worker_type["zone2"]}"]
    worker_disk_category = "cloud_efficiency"         #cloud_ssd or cloud_efficiency
    worker_disk_size = "40"
    worker_data_disk_category = "cloud_ssd"           #cloud_ssd or cloud_efficiency
    worker_data_disk_size = "40"
    worker_numbers = [1,1,1]
    key_name = "${alicloud_key_pair.k8s-ssh-key.key_name}"  #for ECS ssh key auth, either key_name or password 
    #password = "${var.k8ssh["password"]}"  #for ECS password auth, either key_name or password 
    new_nat_gateway = "false"
    pod_cidr = "172.20.0.0/16"
    service_cidr = "172.30.0.0/16"
    slb_internet_enabled = "true"           #for SLB of K8S API Server
    enable_ssh = "true"                        #SSH login kubernetes
    install_cloud_monitor = "true"
    cluster_network_type = "terway"
    kube_config = "${var.kube_cli["cfg"]}"
    client_cert = "${var.kube_cli["client_cert"]}"
    client_key = "${var.kube_cli["client_key"]}"
    cluster_ca_cert = "${var.kube_cli["k8s_ca"]}"
    depends_on = [
    "alicloud_eip_association.eip1_asso", 
    "alicloud_snat_entry.snat1", 
    "alicloud_snat_entry.snat2", 
    "alicloud_snat_entry.snat3"
    ]
    
}

Deploy application on ACK with Horizontal Pod Autoscaler enabled and expose the deployment to the Internet

variable "app1"{
    default = 
    {
    name = "sample"
    namespace = "default"
    min_replicas = 2
    max_replicas = 10
    cpu_threshold = 80
    #image_repo = "registry-intl.ap-southeast-1.aliyuncs.com/kin-test-acr/demo"
    image_repo = "seyantszkin/demo"            #
    image_ver = "v1.0"
    svc_port = 80
    container_port = 8080
    svc_type = "LoadBalancer"
    }
}


resource "kubernetes_deployment" "app1" {
    
    metadata {
        name = "${var.app1["name"]}"
        labels {
            app = "${var.app1["name"]}"
        }
        namespace = "${var.app1["namespace"]}"
    }
    
    spec {
        replicas = "${var.app1["min_replicas"]}"
        
        selector {
            match_labels {
                app = "${var.app1["name"]}"
            }
        }
        
        template {
            metadata {
                labels {
                    app = "${var.app1["name"]}"
                }
            }
            spec {
                container {
                    env {
                        name = "MYSQL_HOST"
                        value = "${alicloud_db_instance.rdsinstance.connection_string}"
                        }
                    env    {
                        name = "MYSQL_PORT"
                        value = "3306"
                        }
                    env    {
                        name = "DB_USERNAME"
                        value = "${var.db_credential["username"]}"
                        }
                    env    {
                        name = "DB_PASSWORD"
                        value = "${var.db_credential["password"]}"
                        }                        
                    image = "${var.app1["image_repo"]}:${var.app1["image_ver"]}"
                    name = "${var.app1["name"]}"
                    port {
                        container_port = "${var.app1["container_port"]}"
                        protocol = "TCP"
                    }
                    #resources {
                    #    requests {}
                    #}
                }
                #image_pull_secrets {
                #    name = "${kubernetes_secret.reg_secret.metadata.0.name}"
                #}
            
            }
        }
    
    }
    depends_on = [
    "alicloud_cs_kubernetes.k8s-cluster", 
    "alicloud_db_database.rdsdb"
    ]
}


resource "kubernetes_horizontal_pod_autoscaler" "hpa1" {
  
  metadata {
    name = "${var.app1["name"]}"
  }
  spec {
    max_replicas = "${var.app1["max_replicas"]}"
    min_replicas = "${var.app1["min_replicas"]}"
    target_cpu_utilization_percentage = "${var.app1["cpu_threshold"]}"
    scale_target_ref {
      kind = "Deployment"
      name = "${var.app1["name"]}"
    }
  }
  depends_on = ["kubernetes_deployment.app1"]
}

resource "kubernetes_service" "svc1" {
    metadata {
        name = "${var.app1["name"]}-svc"
        namespace = "${var.app1["namespace"]}"
    }

    spec {
        selector {
            app = "${var.app1["name"]}"
        }
        
        port {
            port = "${var.app1["svc_port"]}"
            target_port = "${var.app1["container_port"]}"
            protocol = "TCP"
        }
        
        session_affinity = "None"
        type = "${var.app1["svc_type"]}"
        
    }
    depends_on = ["kubernetes_deployment.app1"]
}

Bind a DNS A record to the Server Load balancer which exposes the application to the Internet

resource "alicloud_dns_record" "svc1" {
  name = "alitest.org"            # A domain name (e.g. alicloud.org) registered under your Alibaba Cloud account
  host_record = "${var.app1["name"]}"
  type = "A"
  value = "${kubernetes_service.svc1.load_balancer_ingress.0.ip}"
}

After Provisioning

The output of the Terraform should look like following screenshot:

3

If the local machine that runs the Terraform has Kubernetes client installed, i.e. kubectl, the ~/.kube/config is downloaded, you may try connect to the Kubernetes cluster directly for further actions or tasks such as create Ingress which is not yet supported by Terraform Kubernetes provider.

Finally, the web application should be up and running as shown below.

4

Troubleshooting Potential Issues

Terraform is not able to repair, fix or retry in case of timeout of the provisioning processes. In addition, it is possible that metadata of Terraform may be corrupted when applying changes to the Terraform managed stack. Therefore, it is better to wrap up the provisioning process or even the Terraform command in a shell script to handle exceptions.

Summary

In this post, I have shown how Terraform can integrate resources on Alibaba Cloud with an application provisioned on ACK cluster. You can extend it to stack lifecycle management processes of your Alibaba Cloud resources.

To learn more about Kubernetes on Alibaba Cloud, visit https://www.alibabacloud.com/product/kubernetes

1 0 0
Share on

Alibaba Clouder

2,190 posts | 506 followers

You may also like

Comments

294618655049306610 April 12, 2019 at 7:39 am

very nice post, learned a lot

Alibaba Clouder

2,190 posts | 506 followers

Related Products

  • Container Service for Kubernetes

    Alibaba Cloud Container Service for Kubernetes is a fully managed cloud container management service that supports native Kubernetes and integrates with other Alibaba Cloud products.

    Learn More
  • Learn More
  • ECI(Elastic Container Instance)

    Elastic Container Instance (ECI) is an agile and secure serverless container instance service. You can easily run containers without managing servers. Also you only pay for the resources that have been consumed by the containers. ECI helps you focus on your business applications instead of managing infrastructure.

    Learn More