Elastic Container Instance:Configure eci-profile

How it works

When you use ECI in an ACK cluster via the Virtual Kubelet (VK), which is deployed as the ack-virtual-node add-on, you may encounter the following issues:

• Application teams must modify YAML files to schedule pods to ECI.

• Application teams must modify YAML files to enable ECI-specific advanced features.

These issues blur the boundary between platform operations and application management. To address this, ECI provides the eci-profile feature, which offers the following three capabilities:

• ECI Scheduler

  In hybrid clusters that use both ECI and regular nodes, you can schedule pods to ECI by configuring pod labels, namespace labels, or ECI elastic scheduling. However, these methods require changes to existing resources and are not truly zero-touch.

  To solve this, the ECI Scheduler introduces a new scheduling mechanism based on a mutating webhook. In the eci-profile, you can define custom selectors that filter pods based on their labels or the labels of their namespace. Pods that match the selector criteria are automatically scheduled to ECI.

• ECI Effect

  Some ECI features, such as specifying an ECS instance type, enabling an image cache, or configuring an NTP service, require you to add annotations or labels to pods. For more information, see ECI Pod Annotation.

  The ECI Effect automates this process by injecting annotations and labels. In an eci-profile, you can define a selector that specifies both the matching criteria and the annotations and labels to inject. Pods that match the selector are automatically scheduled to ECI, and the specified annotations and labels are injected.

• Hot updates

  The eci-profile includes settings such as ClusterIP, hybrid cloud mode, log collection, and vSwitches. You can update these settings as needed.

  • You do not need to restart VK when you update the configuration.

  • The updated configuration takes effect immediately for new ECI pods. For existing ECI pods, you must perform a rolling update to apply the changes.

Prerequisites

• Ensure that the VK (ack-virtual-node add-on) in your cluster is updated to the latest version. For more information about how to upgrade components, see Manage components.

• To use the ECI Scheduler feature, you must enable webhooks. Pods in an ACK Serverless cluster are scheduled to ECI by default and do not require the ECI Scheduler feature.

Configuration

When a pod is created, the system uses the configuration in the data field of the eci-profile ConfigMap in the kube-system namespace to create the pod. You can view the YAML file of the eci-profile by running the kubectl get cm -n kube-system eci-profile -o yaml command. The following is a template of the eci-profile YAML file:

apiVersion: v1
data:
  enableClusterIp: "true"
  enableHybridMode: "false"
  enableLinuxArm64Node: "false" 
  enableLogController: "false"
  enablePVCController: "false"
  enablePrivateZone: "false"
  enableReuseSSLKey: "false"
  featureGates: "WaitForFirstConsumer=false"
  securityGroupId: sg-2zeeyaaxlkq9sppl****
  selectors: ""
  slsMachineGroup: ""
  vSwitchIds: vsw-2ze23nqzig8inprou****,vsw-2ze94pjtfuj9vaymf****
  vpcId: vpc-2zeghwzptn5zii0w7****
kind: ConfigMap
metadata:
  creationTimestamp: "2023-01-11T08:28:14Z"
  name: eci-profile
  namespace: kube-system
  resourceVersion: "356"
  uid: b345fa8c-919e-41fc-a981-57864b1a****

You can modify the data configuration in the eci-profile to configure selectors or update cluster-level parameters.

You can modify the eci-profile in one of the following ways:

• Use the kubectl edit command:

  kubectl edit configmap eci-profile -n kube-system

• Use the ACK console:

  1. Log on to the ACK console.

  2. On the Clusters page, find the target cluster and click its name.

  3. In the left-side navigation pane, choose Configurations > ConfigMaps.

  4. Select kube-system from the Namespace drop-down list.

  5. Find eci-profile and click Edit YAML.

Configure selectors

The selectors field defines the configurations for ECI Scheduler and ECI Effect. When a pod is created, the system evaluates the selectors against the pod. If the pod labels match, the pod is automatically scheduled to ECI, and annotations or labels are injected to enable ECI features.

The selectors field can contain multiple selectors. In each selector, you must declare a name and can optionally include the following fields:

• namespaceSelector: the namespace labels to match.

• objectSelector: the pod labels to match.

• effect: the annotations and labels to dynamically inject.

The following is a template for the selectors configuration:

data:
  selectors: |
    [
      {
        "name": "selector-demo1", # Selector name. Required. Cannot be empty.
        "namespaceSelector": {    # Match by namespace labels. Optional.
          "matchLabels": {        # Namespace labels to match. Multiple labels use AND logic.
            "eci": "true"
          }
        },
        "objectSelector": {       # Match by pod labels. Optional.
          "matchLabels": {        # Pod labels to match. Multiple labels use AND logic.
            "eci": "true"
          }
        },
        "effect": {               # Annotations and labels to inject. Optional.
          "annotations": {
            "k8s.aliyun.com/eci-use-specs": "ecs.c6.xlarge"
          },
          "labels": {
            "created-by-eci": "true"
          }
        }
      },
      {
        "name": "selector-demo2",   
        "objectSelector": {      
          "matchLabels": {     
            "eci": "test"
    	  }
        }
      }
    ]

In the preceding template, the selector named selector-demo1 implements the following logic:

If a pod belongs to a namespace with the eci: true label and the pod itself has the eci: true label, the pod is automatically scheduled to ECI. The system also adds the annotation "k8s.aliyun.com/eci-use-specs": "ecs.c6.xlarge" and the label created-by-eci: true to the pod.

After you configure the selectors, run the following command to verify the configuration. If the returned YAML includes your selectors, the configuration is successful. If not, check whether the selector format is correct.

kubectl get mutatingwebhookconfigurations -o yaml vk-webhook

data:
  selectors: |
    [
      {
        "name":"eci-selector",
        "namespaceSelector":{
          "matchLabels":{
            "type":"eci"
          }
        },
        "objectSelector":{
          "matchLabels":{
            "created-by-eci":"true"
          }
        }
      }
    ]

data:
  selectors: |
    [
      {
        "name":"gpu-namespace-selector",
        "namespaceSelector":{
          "matchLabels":{
            "gpu":"true"
          }
        },
        "effect": {
          "annotations": {
            "k8s.aliyun.com/eci-use-specs":"ecs.gn6v-c8g1.2xlarge"
          },        
          "labels":{
            "gpu":"test"
          }
        }
      }
    ]

data:
  selectors: |
    [
      {
        "name":"autoimc-object-selector",
        "objectSelector":{
          "matchLabels":{
            "imc":"auto"
          }
        },
        "effect": {
          "annotations": {
            "k8s.aliyun.com/eci-auto-imc": "true"
          }
        }
      }
    ]

Update cluster parameters

Cluster-level parameters in the data field, such as vpcId and vSwitchIds, define the VPC and vSwitch settings. You can update these parameters as needed. Updates take effect immediately without requiring a VK restart. The following parameters are supported:

The following is an example configuration:

data:
  enableClusterIp: "true"
  enableHybridMode: "false"
  enableLinuxArm64Node: "false" 
  enableLogController: "false"
  enablePVCController: "false"
  enablePrivateZone: "false"
  enableReuseSSLKey: "false"
  securityGroupId: sg-2zeeyaaxlkq9sppl****
  selectors: ""
  slsMachineGroup: ""
  vSwitchIds: vsw-2ze23nqzig8inprou****,vsw-2ze94pjtfuj9vaymf****
  vpcId: vpc-2zeghwzptn5zii0w7****