In the cloud-native era, an increasing number of users choose to migrate applications and businesses to the cloud. The requirement on the container platform varies based on different business scenarios. To meet business requirements, many users want to create Kubernetes clusters by using custom images. This topic describes how to create a Kubernetes cluster by using a custom image.

Prerequisites

Before you create a Kubernetes cluster by using a custom image, take note of the following limits on the custom images that are supported by Container Service for Kubernetes (ACK):

  • We recommend that you use the latest base images provided by ACK. The base images of ACK can be used to create Kubernetes clusters and have passed the strict tests of the ACK technical team. Custom images that are used to create Kubernetes clusters must meet the following requirements:
    • Alibaba Cloud cloud-init can be installed. For more information, see Install cloud-init.
    • If you want to create an ACK dedicated cluster by using a custom image, you must enable the sshd server and use the default port 22. This allows you to transfer files to cluster nodes after the nodes are enabled. For more information, see Connect to the master nodes of a dedicated Kubernetes cluster by using SSH.
    • Time synchronization is performed by using a Network Time Protocol (NTP) server provided by Alibaba Cloud.
  • If you want to use custom images, Submit a ticket.

Background information

ACK allows you to create Kubernetes clusters by using custom images. However, you may encounter the following issues when you create and package custom images:
  • It is not efficient to manually create images.
  • Records may be missing in the image change history. This makes it difficult to troubleshoot errors.
  • You cannot check whether the custom images meet the requirements of nodes in ACK clusters.
To solve the preceding issues, the ACK technical team launches the open source ack-image-builder project to help you create custom images that meet the requirements of nodes in ACK clusters.

The ack-image-builder project is developed based on the open source tool HashiCorp Packer, and provides default configuration templates and verification scripts that are used to create and verify custom images.

Precautions

If you modify the parameters of the operating system when you use a custom image, nodes may fail to start up or run as normal. For example, if you set kernel.modules_disabled to 1, Docker becomes unavailable. We recommend that you test the custom image in a test environment.

Procedure

To create a custom image by using ack-image-builder, perform the following steps:

  1. Install Packer.
    1. Download Packer from the official website. Make sure that the downloaded version is compatible with your node operating system. Then, install and verify Packer by following the installation instructions of Packer.
    2. Run the following command to check the version of Packer:
      packer version
      If the following output is returned, Packer is installed.
      Packer v1.4.1
  2. Configure a Packer template.
    When you create a custom image by using Packer, you must create a template file in JSON format. In the template file, specify the image builder provided by Alibaba Cloud and the provisioner that is used to create and configure custom images. For more information, refer to Alicloud Image Builder and Provisioners.
    {
      "variables": {
        "region": "cn-hangzhou",
        "image_name": "test_image{{timestamp}}",
        "source_image": "centos_7_06_64_20G_alibase_20190711.vhd",
        "instance_type": "ecs.n1.large",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
      "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "ssh_username": "root",
          "instance_type": "{{user `instance_type`}}",
          "io_optimized": "true"
        }
      ],
      "provisioners": [
        {
          "type": "shell",
          "scripts": [
            "scripts/updateKernel.sh",
            "scripts/reboot.sh",
            "scripts/cleanUpKerneles.sh",
            "config/default.sh",
            "scripts/updateDNS.sh",
            "scripts/verify.sh"
          ],
          "expect_disconnect": true
        }
      ]
    }
    Parameter Description
    access_key The AccessKey ID that is used to create the custom image.
    secret_key The AccessKey secret that is used to create the custom image.
    region The region of the cloud resources that are temporarily used to create the custom image.
    image_name The name of the custom image.
    source_image The name of the base image used to create the custom image. You can obtain the name of a base image from the public image list of Alibaba Cloud.
    instance_type The type of the cloud resources that are temporarily used to create the custom image.
    provisioners The type of the provisioner used to create the custom image.
  3. Create a Resource Access Management (RAM) user and create an AccessKey pair for the RAM user. For more information, see Create a RAM user and Obtain an AccessKey pair.
    Note We recommend that you grant the RAM user the permissions on Packer when you create the RAM user. For more information about the RAM policy, see RAM Policy.
  4. Add the AccessKey pair to the template and create a custom image by using the template.
    1. Run the following commands to add the AccessKey pair to the template:
      export ALICLOUD_ACCESS_KEY=XXXXXX
      export ALICLOUD_SECRET_KEY=XXXXXX
    2. Run the following commands to create a custom image:
      packer build alicloud.json

      Expected output:

      alicloud-ecs output will be in this color.
      
      ==> alicloud-ecs: Prevalidating source region and copied regions...
      ==> alicloud-ecs: Prevalidating image name...
          alicloud-ecs: Found image ID: centos_7_06_64_20G_alibase_20190711.vhd
      ==> alicloud-ecs: Creating temporary keypair: xxxxxx
      ==> alicloud-ecs: Creating vpc...
          alicloud-ecs: Created vpc: xxxxxx
      ==> alicloud-ecs: Creating vswitch...
          alicloud-ecs: Created vswitch: xxxxxx
      ==> alicloud-ecs: Creating security group...
          alicloud-ecs: Created security group: xxxxxx
      ==> alicloud-ecs: Creating instance...
          alicloud-ecs: Created instance: xxxxxx
      ==> alicloud-ecs: Allocating eip...
          alicloud-ecs: Allocated eip: xxxxxx
          alicloud-ecs: Attach keypair xxxxxx to instance: xxxxxx
      ==> alicloud-ecs: Starting instance: xxxxxx
      ==> alicloud-ecs: Using ssh communicator to connect: 47.111.127.54
      ==> alicloud-ecs: Waiting for SSH to become available...
      ==> alicloud-ecs: Connected to SSH!
      ==> alicloud-ecs: Provisioning with shell script: scripts/verify.sh
          alicloud-ecs: [20190726 11:04:10]: Check if kernel version >= 3.10.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if systemd version >= 219.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if sshd is running and listen on port 22.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if cloud-init is installed.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if wget is installed.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if curl is installed.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if kubeadm is cleaned up.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if kubelet is cleaned up.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if kubectl is cleaned up.  Verify Passed!
          alicloud-ecs: [20190726 11:04:10]: Check if kubernetes-cni is cleaned up.  Verify Passed!
      ==> alicloud-ecs: Stopping instance: xxxxxx
      ==> alicloud-ecs: Waiting instance stopped: xxxxxx
      ==> alicloud-ecs: Creating image: test_image1564110199
          alicloud-ecs: Detach keypair xxxxxx from instance: xxxxxxx
      ==> alicloud-ecs: Cleaning up 'EIP'
      ==> alicloud-ecs: Cleaning up 'instance'
      ==> alicloud-ecs: Cleaning up 'security group'
      ==> alicloud-ecs: Cleaning up 'vSwitch'
      ==> alicloud-ecs: Cleaning up 'VPC'
      ==> alicloud-ecs: Deleting temporary keypair...
      Build 'alicloud-ecs' finished.
      
      ==> Builds finished. The artifacts of successful builds are:
      --> alicloud-ecs: Alicloud images were created:
      
      cn-hangzhou: m-bp1aifbnupnaktj00q7s
      scripts/verify.sh specifies the verification result of the custom image.
  5. Use the custom image to create an ACK cluster.
    An ACK Pro cluster is used as an example.
    1. Log on to the ACK console.
    2. In the left-side navigation pane of the ACK console, click Clusters.
    3. In the upper-right corner of the Clusters page, click Create Kubernetes Cluster.
    4. On the Managed Kubernetes tab, configure the cluster based on the custom image that you create. For more information about the cluster parameters, see Create a professional managed Kubernetes cluster.
      1. After you configure the cluster parameters, click Next:Node Pool Configurations.
      2. On the Node Pool Configurations wizard page, click Select in the Custom Image section.
      3. In the Choose Custom Image dialog box, select the custom image and click Use.
      4. Complete other configurations for the cluster.
    5. After the cluster configuration is completed, click Create Cluster.
      After the cluster is created by using the custom image, the custom image is also used in other operations, such as cluster scaling.