Container Service for Kubernetes:Custom images for scaling optimization

Pré-requisitos

• Um cluster gerenciado do ACK criado.

• Cliente kubectl conectado ao cluster ACK. Para mais informações, consulte Conectar-se a um cluster ACK usando kubectl.

Por que usar imagens personalizadas para dimensionamento

Os pools de nós do ACK suportam Auto Scaling com imagens de sistema operacional integradas, como Alibaba Cloud Linux e CentOS, que atendem à maioria dos casos de uso. Para cenários que exigem software pré-instalado ou configurações de alto desempenho, utilize o Alicloud Image Builder para criar imagens personalizadas de sistema operacional.

Envie a tarefa de criação da imagem para o cluster como um Job ou CronJob.

Criar uma imagem personalizada com um Job do ACK

Os exemplos a seguir utilizam um ConfigMap chamado build-config e um Job chamado build.

Etapa 1: Configure parâmetros de criação

Crie um ConfigMap chamado build-config para definir os parâmetros de criação da imagem.

1. Crie um arquivo chamado build-config.yaml com o seguinte conteúdo YAML.

   Visualize detalhes

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: build-config
   data:
     ack.json: |-
   
       {
         "variables": {
           "image_name": "ack-optimized_image-1.30-{{timestamp}}",
           "source_image": <source_image>,
           "instance_type": <instance_type>,
           "region": "{{env `ALICLOUD_REGION`}}",
           "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
           "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
         },
         "builders": [
           {
             "type": "alicloud-ecs",
             "system_disk_mapping": {    
                       "disk_size": 120
                       },   # Specifies the system disk size for the image in GB. The default is 40 GB.
             "access_key": "{{user `access_key`}}",
             "secret_key": "{{user `secret_key`}}",
             "region": "{{user `region`}}",
             "image_name": "{{user `image_name`}}",
             "source_image": "{{user `source_image`}}",
             "ssh_username": "root",
             "vpc_id": "",  # If you do not specify vpc_id, the process creates a temporary virtual private cloud (VPC) and automatically deletes it after completion.
             "vswitch_id": "",  # If you do not specify vswitch_id, the process creates a temporary vSwitch and automatically deletes it after completion.
             "security_group_id": "",   # If you do not specify security_group_id, the process creates a temporary security group and automatically deletes it after completion.
             "instance_type": "{{user `instance_type`}}",
             "skip_image_validation": "true",
             "io_optimized": "true"
           }
         ],
         "provisioners": [{
          "type": "file",
          "source": "scripts/ack-optimized-os-all.sh",    # Applies only to Alibaba Cloud Linux 2 images.
          "destination": "/root/"
           },
           {
          "type": "shell",
          "inline": [
            "export RUNTIME=containerd",    # Specify the container runtime.
            "export RUNTIME_VERSION=1.6.28",
            "export SKIP_SECURITY_FIX=true",
            "export KUBE_VERSION=1.30.1-aliyun.1",
            "export OS_ARCH=amd64",    # Specify amd64 or arm64 based on your requirements.
            "bash /root/ack-optimized-os-all.sh"
         ]
         }]
       }

   As tabelas a seguir listam os parâmetros YAML.

   Tabela 1. Parâmetros no arquivo de configuração do Alicloud Image Builder

   Parâmetro	Exemplo	Descrição
   variables{"<variable1>":"<value>"}	variables{"access_key":"{{env ALICLOUD_ACCESS_KEY}}"}	As variables do Alicloud Image Builder. Nota Codificar informações sensíveis como access_key e secret_key diretamente no arquivo de configuração representa um risco de segurança. Defina-as como variáveis que recebem valores em tempo de execução.
   builders{"type":"<value>"}	builders{"type":"alicloud-ecs"}	O construtor de imagens (builders). Quando type é alicloud-ecs, o sistema cria uma instância ECS temporária para gerar a imagem e a libera posteriormente.
   provisioners{"type":"<value>"}	provisioners{"type":"shell"}	O provisionador de imagens (provisioners) define as operações na instância temporária. Quando type é shell, um script shell executa automaticamente na instância. Por exemplo, yum install redis.x86_64 -y instala o Redis. Consulte Configuração do provisionador.

   Tabela 2. Parâmetros de criação de imagem

   Parâmetro	Exemplo	Descrição	Obrigatório
   access_key	yourAccessKeyID	ID do par AccessKey. Consulte Obter um par AccessKey.	Sim
   secret_key	yourAccessKeySecret	Segredo do par AccessKey.	Sim
   region	cn-beijing	Região da imagem personalizada.	Sim
   image_name	ack-custom_image	Nome da imagem personalizada. Deve ser único entre as imagens existentes.	Sim
   source_image	aliyun_2_1903_x64_20G_alibase_20200904.vhd	ID da imagem pública do Alibaba Cloud com o sistema operacional correspondente. Consulte Imagens de SO suportadas pelo ACK.	Sim
   instance_type	ecs.c6.xlarge	Tipo de instância ECS para a instância temporária de criação. Especifique um tipo acelerado por GPU para imagens GPU.	Sim
   RUNTIME	containerd	Runtime de contêiner. Valores válidos: docker e containerd.	Sim
   RUNTIME_VERSION	1.6.28	Se o runtime de contêiner for docker, o valor padrão de RUNTIME_VERSION é 19.03.15. Se o runtime de contêiner for containerd, o valor padrão de RUNTIME_VERSION é 1.6.20.	Sim
   SKIP_SECURITY_FIX	true	Define se as atualizações de segurança devem ser ignoradas.	Sim
   KUBE_VERSION	1.30.1-aliyun.1	Versão do cluster.	Sim
   PRESET_GPU	true	Indica se o driver GPU deve ser pré-instalado para acelerar a inicialização.	Não
   NVIDIA_DRIVER_VERSION	460.91.03	Versão do driver GPU a ser pré-instalada. Valor padrão: 460.91.03.	Não
   OS_ARCH	amd64	Arquitetura da CPU. Valores válidos: amd64 e arm64.	Sim
   MOUNT_RUNTIME_DATADISK	true	Defina como true se a imagem personalizada tiver imagens de aplicação em cache e for necessário montar um disco de dados na instância ECS durante o uso.	Não

   Importante

   • Antes de atribuir uma imagem personalizada a um pool de nós, verifique se a configuração do pool (versão do cluster, região, runtime de contêiner, tipo de instância GPU) corresponde às definições de criação da imagem. Caso contrário, o nó não conseguirá ingressar no cluster.

   • Para validar, adicione a imagem a um pool de nós regular com parâmetros correspondentes e verifique se suas cargas de trabalho executam conforme o esperado.

2. Implante o Alicloud Image Builder no cluster:

   kubectl apply -f build-config.yaml

Etapa 2: Crie um Job de criação

1. Conceda as permissões necessárias ao par AccessKey com a seguinte política.

   Visualize detalhes

   {
     "Version": "1",
     "Statement": [
       {
         "Action": [
           "ecs:DeleteInstance",
           "ecs:StartInstance",
           "ecs:StopInstance",
           "ecs:DescribeInstances"
         ],
         "Resource": "*",
         "Effect": "Allow"
       },
       {
         "Action": [
           "ecs:CreateInstance",
           "ecs:DescribeImages",
           "ecs:CreateImage",
           "ecs:ModifyImageSharePermission",
           "ecs:CreateKeyPair",
           "ecs:DeleteKeyPairs",
           "ecs:DetachKeyPair",
           "ecs:AttachKeyPair",
           "ecs:CreateSecurityGroup",
           "ecs:DeleteSecurityGroup",
           "ecs:AuthorizeSecurityGroupEgress",
           "ecs:AuthorizeSecurityGroup",
           "ecs:CreateSnapshot",
           "ecs:AttachDisk",
           "ecs:DetachDisk",
           "ecs:DescribeDisks",
           "ecs:CreateDisk",
           "ecs:DeleteDisk",
           "ecs:CreateNetworkInterface",
           "ecs:DescribeNetworkInterfaces",
           "ecs:AttachNetworkInterface",
           "ecs:DetachNetworkInterface",
           "ecs:DeleteNetworkInterface",
           "ecs:DescribeInstanceAttribute"
         ],
         "Resource": "*",
         "Effect": "Allow"
       },
       {
         "Action": [
           "vpc:DescribeVpcs",
           "vpc:DescribeVSwitches",
           "vpc:AllocateEipAddress",
           "vpc:AssociateEipAddress",
           "vpc:UnassociateEipAddress",
           "vpc:DescribeEipAddresses",
           "vpc:ReleaseEipAddress",
           "vpc:CreateVpc",
           "vpc:DeleteVpc",
           "vpc:DescribeVpcs",
           "vpc:CreateVSwitch",
           "vpc:DeleteVSwitch",
           "vpc:DescribeVSwitches",
           "vpc:CreateRouteTable",
           "vpc:DeleteRouteTable",
           "vpc:DescribeRouteTables",
           "vpc:CreateNatGateway",
           "vpc:DeleteNatGateway",
           "vpc:DescribeNatGateways",
           "vpc:CreateSnatEntry",
           "vpc:DeleteSnatEntry",
           "vpc:DescribeSnatTableEntries"
         ],
         "Resource": "*",
         "Effect": "Allow"
       }
     ]
   }

2. Gere strings codificadas em Base64 para seu par AccessKey:

   echo -n "yourAccessKeyID" | base64
   echo -n "yourAccessKeySecret" | base64

3. Crie um Secret chamado my-secret com o seguinte YAML.

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-secret
     namespace: default
   type: Opaque
   data:
     ALICLOUD_ACCESS_KEY: TFRI**************** # The Base64-encoded string from the previous step.
     ALICLOUD_SECRET_KEY: a0zY****************

4. Crie um arquivo chamado build.yaml com o seguinte conteúdo YAML.

   O Job cria uma instância ECS temporária a partir da source_image, executa os scripts de provisioners, gera uma imagem personalizada e a envia para a região especificada.

   Visualize detalhes

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: image-builder
     namespace: default
   spec:
     template:
       metadata:
         name: image-builder
       spec:
         containers:
           - name: image-builder
             image: "registry-cn-hangzhou.ack.aliyuncs.com/acs/image-builder:v3.3"
             imagePullPolicy: Always
             env:
               - name: ALICLOUD_ACCESS_KEY
                 valueFrom:
                   secretKeyRef:
                     name: my-secret
                     key: ALICLOUD_ACCESS_KEY
               - name: ALICLOUD_SECRET_KEY
                 valueFrom:
                   secretKeyRef:
                     name: my-secret
                     key: ALICLOUD_SECRET_KEY
               - name: ALICLOUD_REGION
                 value: cn-hangzhou
             command: ["packer"] 
             args: ["build", "/config/ack.json"]
             volumeMounts:
               - name: config
                 mountPath: /config
         volumes:
           - name: config
             configMap:
               name: build-config
               items:
                 - key: ack.json
                   path: ack.json
         restartPolicy: Never

5. Implante o Job para iniciar a criação da imagem.

   kubectl apply -f build.yaml

Etapa 3: (Opcional) Visualize logs de criação

1. Faça login no console do ACK. No painel de navegação à esquerda, clique em Clusters.

2. Na página Clusters, clique no nome do seu cluster. No painel de navegação à esquerda, clique em Workloads > Jobs.

3. Na lista de Jobs, localize o Job criado. Na coluna Actions, clique em Details e, em seguida, clique na aba Logs para visualizar os logs de criação.

Configuração do provisionador

  "provisioners": [{
      "type": "shell",
      "script": "script.sh"
  }]

  "provisioners": [
    {
    "type": "ansible",
    "playbook_file": "./playbook.yml"
    }
  ]

    {
      "variables": {
        "region": "{{env `ALICLOUD_REGION`}}",
        "image_name": "ack-custom_image",
        "source_image": "centos_7_04_64_20G_alibase_201701015.vhd",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "instance_type": "ecs.c6.xlarge",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
    "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "ssh_username": "root",
          "instance_type": "{{user `instance_type`}}",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
   "provisioners": [{
        "type": "shell",
        "inline": [
            "cd $HOME",
            "wget https://cpfs-client.oss-cn-beijing.aliyuncs.com/kernel/kernel-devel-`uname -r`.rpm",
            "rpm -ivh --replacefiles kernel-devel-`uname -r`.rpm"
        ]
      }
    ]
  }

{
      "variables": {
        "region": "{{env `ALICLOUD_REGION`}}",
        "image_name": "ack-custom_image",
        "source_image": "aliyun_3_arm64_20G_alibase_20240528.vhd",
        "instance_type": "ecs.r8y.xlarge",   # Specify an instance type for the Arm architecture.
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
    "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "instance_type": "{{user `instance_type`}}",
          "ssh_username": "root",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
   "provisioners": [
        {
            "type": "file",
            "source": "scripts/ack-optimized-os-linux3-all.sh",
            "destination": "/root/"
        },
        {
            "type": "shell",
            "inline": [
                "export RUNTIME=containerd",
                "export RUNTIME_VERSION=2.1.6",
                "export SKIP_SECURITY_FIX=true",
                "export KUBE_VERSION=1.28.9-aliyun.1",
                "export OS_ARCH=arm64",
                "bash /root/ack-optimized-os-linux3-all.sh"
            ]
        }
      ]
    }

{
      "variables": {
        "region": "{{env `ALICLOUD_REGION`}}",
        "image_name": "ack-custom_image",
        "source_image": "aliyun_2_1903_x64_20G_alibase_20221102.vhd",
        "instance_type": "ecs.gn6i-c4g1.xlarge",   # To pre-install a GPU driver, set this parameter to a GPU-accelerated instance type.
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
    "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "instance_type": "{{user `instance_type`}}",
          "ssh_username": "root",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
   "provisioners": [
        {
            "type": "file",
            "source": "scripts/ack-optimized-os-all.sh",
            "destination": "/root/"
        },
        {
            "type": "shell",
            "inline": [
                "export RUNTIME=containerd",
                "export RUNTIME_VERSION=1.6.36",
                "export SKIP_SECURITY_FIX=true",
                "export PRESET_GPU=true",          # To pre-install a GPU driver, set PRESET_GPU to true. If you do not want to pre-install a GPU driver, leave this parameter empty or set it to false.
                "export NVIDIA_DRIVER_VERSION=510.47.03",         # Specify the GPU driver version. If you leave this parameter empty, the builder installs version 460.91.03 by default.
                "export KUBE_VERSION=1.22.3-aliyun.1",
                "export OS_ARCH=amd64",
                "bash /root/ack-optimized-os-all.sh"
            ]
        }
      ]
    }

{
      "variables": {
        "image_name": "ack-custom_image",
        "source_image": "aliyun_3_x64_20G_alibase_20240528.vhd",
        "instance_type": "ecs.c6.xlarge",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "region": "{{env `ALICLOUD_REGION`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
    "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "instance_type": "{{user `instance_type`}}",
          "ssh_username": "root",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
   "provisioners": [
        {
            "type": "file",
            "source": "scripts/ack-optimized-os-linux3-all.sh",
            "destination": "/root/"
        },
        {
            "type": "shell",
            "inline": [
                "export RUNTIME=containerd",
                "export RUNTIME_VERSION=2.1.6",
                "export SKIP_SECURITY_FIX=true",
                "export KUBE_VERSION=1.30.1-aliyun.1",
                "export OS_ARCH=amd64",
                "bash /root/ack-optimized-os-linux3-all.sh",
                "ctr -n k8s.io i pull registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/pause:3.9"        # Bake application images into the OS image.
            ]
        }
      ]
    }

{
      "variables": {
        "image_name": "ack-custom_image",
        "source_image": "aliyun_3_x64_20G_alibase_20240528.vhd",
        "instance_type": "ecs.c6.xlarge",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "region": "{{env `ALICLOUD_REGION`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
    "builders": [
        {
          "type": "alicloud-ecs",
          "system_disk_mapping": {    
                    "disk_size": 120,
                    "disk_category": "cloud_essd"
                    },
          "image_disk_mappings": {
                    "disk_size": 40,
                    "disk_category": "cloud_auto"
                    },     # Configure a data disk when you build the custom image. After building the image, the process automatically creates a snapshot of the data disk.
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "instance_type": "{{user `instance_type`}}",
          "ssh_username": "root",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
   "provisioners": [
        {
            "type": "file",
            "source": "scripts/ack-optimized-os-linux3-all.sh",
            "destination": "/root/"
        },
        {
            "type": "shell",
            "inline": [
                "export RUNTIME=containerd",
                "export RUNTIME_VERSION=2.1.6",
                "export SKIP_SECURITY_FIX=true",
                "export KUBE_VERSION=1.30.1-aliyun.1",
                "export OS_ARCH=amd64",
                "export MOUNT_RUNTIME_DATADISK=true",     # Mount container runtime file paths to the data disk.
                "bash /root/ack-optimized-os-linux3-all.sh",
                "ctr -n k8s.io i pull registry-cn-hangzhou-vpc.ack.aliyuncs.com/acs/pause:3.9",        # Bake the application image into the OS image.
                "mv /var/lib/containerd /var/lib/container/containerd"       # Move the image files to the data disk.
            ]
        }
      ]
    }

docker login <image_address> -u user -p password
docker pull nginx

ctr -n k8s.io i pull --user=username:password nginx

apiVersion: v1
kind: ConfigMap
metadata:
  name: build-config
data:
  ack.json: |-
    
    {
      "variables": {
        "image_name": "ack-optimized_image-1.22-{{timestamp}}",
        "source_image":"aliyun_3_x64_20G_alibase_20230110.vhd",  # Base image for Alibaba Cloud Linux 3.
        "instance_type": "ecs.c6.xlarge",
        "region": "{{env `ALICLOUD_REGION`}}",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}"
      },
      "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "ssh_username": "root",  
          "instance_type": "{{user `instance_type`}}",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
      "provisioners": [{
       "type": "file",
       "source": "scripts/ack-optimized-os-linux3-all.sh",
       "destination": "/root/"
        },
        {
       "type": "shell",
       "inline": [
         "export RUNTIME=containerd",
         "export SKIP_SECURITY_FIX=true",
         "export OS_ARCH=amd64",
         "export KUBE_VERSION=1.22.3-aliyun.1",
         "bash /root/ack-optimized-os-linux3-all.sh"
      ]
      }]
    }

apiVersion: v1
kind: ConfigMap
metadata:
  name: build-config
data:
  ack.json: |-
    
    {
      "variables": {
        "image_name": "ack-optimized_image-1.26-{{timestamp}}",
        "source_image": "m-bp1c7zuf8mcabc99babc",       # Base image for Red Hat Enterprise Linux 9.
        "instance_type": "ecs.c6.xlarge",
        "region": "{{env `ALICLOUD_REGION`}}",
        "access_key": "{{env `ALICLOUD_ACCESS_KEY`}}",
        "secret_key": "{{env `ALICLOUD_SECRET_KEY`}}",
        "runtime": "{{env `RUNTIME`}}",
        "skip_security_fix": "{{env `SKIP_SECURITY_FIX`}}"
      },
      "builders": [
        {
          "type": "alicloud-ecs",
          "access_key": "{{user `access_key`}}",
          "secret_key": "{{user `secret_key`}}",
          "region": "{{user `region`}}",
          "image_name": "{{user `image_name`}}",
          "source_image": "{{user `source_image`}}",
          "ssh_username": "root",
          # "vpc_id": "",
          # "vswitch_id": "",
          # "security_group_id": "",      
          "instance_type": "{{user `instance_type`}}",
          "skip_image_validation": "true",
          "io_optimized": "true"
        }
      ],
      "provisioners": [{
       "type": "file",
       "source": "scripts/ack-optimized-os-rhel9-all.sh",
       "destination": "/root/"
        },
        {
       "type": "shell",
       "inline": [
         "export RUNTIME=containerd",
         "export SKIP_SECURITY_FIX=true",
         "export OS_ARCH=amd64",
         "export KUBE_VERSION=1.26.3-aliyun.1",
         "bash /root/ack-optimized-os-rhel9-all.sh"
      ]
      }]
    }

Próximos passos

• Criar um pool de nós a partir de uma imagem personalizada.

• Dimensionar nós a partir de uma imagem personalizada.