為Agent Sandbox配置Sidecar容器自動注入 - Container Compute Service

為簡化Agent Sandbox的儲存掛載及運行時注入的YAML配置，可通過在Sandbox資源中配置 spec.runtimes 欄位，實現自動注入 CSI 儲存掛載和 AgentRuntime 相關配置，來簡化整個部署流程。

準備工作

已完成Agent Sandbox環境搭建，詳情請參見建立Agent Sandbox。
在叢集組件管理中，確認ack-agent-sandbox-controller組件版本為v0.5.10及以上。
安裝組件（v0.5.9及以上版本）時，會在sandbox-system命名空間下建立名為sandbox-injection-config的ConfigMap，包含agent-runtime和csi兩個配置項。
如需特殊定製或配置修改，請在修改前聯絡阿里雲支援人員評估配置的完備性。

功能概述

在Agent Sandbox中配置動態儲存裝置掛載時，由於涉及手動設定CSI及Sidecar容器，YAML配置一般較為複雜。通過Sidecar自動注入功能，您只需定義業務容器，並在SandboxSet或Sandbox資源中聲明spec.runtimes欄位，系統將自動為建立的Sandbox注入所需的CSI和AgentRuntime Sidecar配置。目前支援以下兩種注入類型：

csi：注入CSI儲存掛載相關的init容器和Volume配置，支援NAS、OSS等共用儲存掛載能力。
agent-runtime：注入AgentRuntime init容器（如envd等環境管理工具），並為業務容器添加相應的環境變數和生命週期鉤子。

Sidecar自動注入僅對新建立的Sandbox執行個體生效，存量執行個體不受影響。如需使用Command 和Filesystem 介面，必須注入agent-runtime。

配置Sidecar注入

在SandboxSet或Sandbox資源的spec.runtimes欄位中聲明需要注入的運行時類型。以下分別展示兩種資源的配置樣本。

SandboxSet資源配置樣本

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxSet
metadata:
  name: code-interpreter-inject-test
  namespace: default
spec:
  runtimes:
  - name: csi           # 支援CSI掛載能力，建立的Sandbox會被注入對應的Sidecar
  - name: agent-runtime # 注入envd等環境管理工具
  replicas: 4
  template:
    metadata:
      labels:
        alibabacloud.com/acs: "true"
    spec:
      automountServiceAccountToken: false
      containers:
      - image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/code-interpreter:v1.6 # 替換成實際叢集所在的地區
        imagePullPolicy: IfNotPresent
        name: sandbox
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
      terminationGracePeriodSeconds: 30

Sandbox資源配置樣本

apiVersion: agents.kruise.io/v1alpha1
kind: Sandbox
metadata:
  name: code-interpreter-inject-test-xxx
  namespace: default
spec:
  runtimes:
  - name: csi           # 提供CSI掛載能力
  - name: agent-runtime # 注入envd等環境管理工具
  ...

注入後的Pod效果樣本

# 注意：鏡像地址中的<region-id>會根據叢集所在地區自動替換
apiVersion: v1
kind: Pod
metadata:
  annotations:
    agents.kruise.io/created-by: sandbox
  labels:
    agents.kruise.io/sandbox-pool: code-interpreter-init-xxx
    alibabacloud.com/acs: "true"
  name: code-interpreter-init-xxx
  namespace: default
spec:
  automountServiceAccountToken: false
  containers:
  - env:
    # --- 以下為自動注入的配置 ---
    - name: ENVD_DIR
      value: /mnt/envd
    - name: GODEBUG
      value: multipathtcp=0
    - name: POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    # --- 自動注入配置結束 ---
    image: example:tag
    imagePullPolicy: IfNotPresent
    # --- 以下為自動注入的配置 ---
    lifecycle:
      postStart:
        exec:
          command:
          - bash
          - -c
          - /mnt/envd/envd-run.sh
    # --- 自動注入配置結束 ---
    name: sandbox
    resources:
      limits:
        cpu: "2"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 2Gi
    volumeMounts:
    # --- 以下為自動注入的配置 ---
    - mountPath: /mnt/envd
      name: envd-volume
    - mountPath: /run/csi/mount-root
      mountPropagation: HostToContainer
      name: mount-root
    - mountPath: /var/run/csi/sockets/nasplugin.csi.alibabacloud.com
      name: nas-plugin-dir
    - mountPath: /var/run/csi/sockets/ossplugin.csi.alibabacloud.com
      name: oss-plugin-dir
    # --- 自動注入配置結束 ---
  # --- 以下為自動注入的配置 ---
  initContainers:
  - command:
    - sh
    - /workspace/entrypoint_inner.sh
    env:
    - name: ENVD_DIR
      value: /mnt/envd
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/agent-runtime:v0.0.5
    imagePullPolicy: IfNotPresent
    name: init
    resources: {}
    restartPolicy: Always
    volumeMounts:
    - mountPath: /mnt/envd
      name: envd-volume
  - args:
    - --endpoint=unix://var/run/csi/sockets/driverplugin.csi.alibabacloud.com-replace/csi.sock
    - --driver=nas,oss
    - --v=1
    - --run-controller-service=false
    - --run-node-service=true
    - --feature-gates=AlinasMountProxy=true
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    - name: KUBELET_ROOT_DIR
      value: /
    - name: ALIBABA_CLOUD_NETWORK_TYPE
      value: vpc
    - name: REGION_ID
      value: cn-hangzhou
    - name: OSS_SKIP_GLOBAL_MOUNT
      value: "true"
    - name: KUBE_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-plugin:v1.35.1-2592a4872
    imagePullPolicy: IfNotPresent
    name: csi-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    restartPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /run/csi/mount-root
      mountPropagation: Bidirectional
      name: mount-root
    - mountPath: /var/run/csi/sockets/nasplugin.csi.alibabacloud.com
      name: nas-plugin-dir
    - mountPath: /var/run/csi/sockets/ossplugin.csi.alibabacloud.com
      name: oss-plugin-dir
    - mountPath: /run/cnfs
      name: run-cnfs
    - mountPath: /host/var/run/efc
      name: efc-metrics-dir
    - mountPath: /host/var/run/ossfs
      name: ossfs-metrics-dir
  - args:
    - --socket=/run/cnfs/alinas-mounter.sock
    - --v=4
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.3-cgroupv1-dport-forbidden
    imagePullPolicy: IfNotPresent
    name: csi-agent-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 1Gi
    restartPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /run/csi/mount-root
      mountPropagation: Bidirectional
      name: mount-root
    - mountPath: /sys/fs/cgroup/net_cls/kubepods
      name: cgroup-net-cls
    - mountPath: /etc/aliyun-defaults/cpfs
      name: csi-agent-config
    - mountPath: /etc/aliyun-defaults/alinas
      name: csi-agent-config
    - mountPath: /run/cnfs
      name: run-cnfs
  # --- 自動注入配置結束 ---
  restartPolicy: Always
  volumes:
  # --- 以下為自動注入的配置 ---
  - emptyDir: {}
    name: envd-volume
  - hostPath:
      path: /dev/fuse
      type: CharDevice
    name: fuse-device
  - hostPath:
      path: /sys/fs/cgroup/net_cls/kubepods
      type: Directory
    name: cgroup-net-cls
  - hostPath:
      path: /var/run/csi
      type: DirectoryOrCreate
    name: mount-root
  - emptyDir: {}
    name: nas-plugin-dir
  - emptyDir: {}
    name: oss-plugin-dir
  - emptyDir: {}
    name: run-cnfs
  - emptyDir: {}
    name: efc-metrics-dir
  - emptyDir: {}
    name: ossfs-metrics-dir
  - emptyDir: {}
    name: csi-agent-config
  # --- 自動注入配置結束 ---

注入配置說明

以下為v0.5.9及以上版本預設提供的注入配置，通常無需修改。其中鏡像地址中的<region-id>會根據叢集所在地區自動替換（如cn-zhangjiakou）。

agent-runtime預設注入配置

{
  "mainContainer": {
    "name": "",
    "env": [
      {
        "name": "ENVD_DIR",
        "value": "/mnt/envd"
      },
      {
        "name": "GODEBUG",
        "value": "multipathtcp=0"
      },
      {
        "name": "POD_UID",
        "valueFrom": {
          "fieldRef": {
            "fieldPath": "metadata.uid"
          }
        }
      }
    ],
    "resources": {},
    "volumeMounts": [
      {
        "name": "envd-volume",
        "mountPath": "/mnt/envd"
      }
    ],
    "lifecycle": {
      "postStart": {
        "exec": {
          "command": [
            "bash",
            "-c",
            "/mnt/envd/envd-run.sh"
          ]
        }
      }
    }
  },
  "csiSidecar": [
    {
      "name": "init",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/agent-runtime:v0.0.5",
      "command": [
        "sh",
        "/workspace/entrypoint_inner.sh"
      ],
      "env": [
        {
          "name": "ENVD_DIR",
          "value": "/mnt/envd"
        },
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        }
      ],
      "resources": {},
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "envd-volume",
          "mountPath": "/mnt/envd"
        }
      ],
      "imagePullPolicy": "IfNotPresent"
    }
  ],
  "volume": [
    {
      "name": "envd-volume",
      "emptyDir": {}
    }
  ]
}

csi預設注入配置

{
  "mainContainer": {
    "name": "",
    "resources": {},
    "volumeMounts": [
      {
        "name": "mount-root",
        "mountPath": "/run/csi/mount-root",
        "mountPropagation": "HostToContainer"
      },
      {
        "name": "nas-plugin-dir",
        "mountPath": "/var/run/csi/sockets/nasplugin.csi.alibabacloud.com"
      },
      {
        "name": "oss-plugin-dir",
        "mountPath": "/var/run/csi/sockets/ossplugin.csi.alibabacloud.com"
      }
    ]
  },
  "csiSidecar": [
    {
      "name": "csi-sidecar",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-plugin:v1.35.1-2592a4872",
      "args": [
        "--endpoint=unix://var/run/csi/sockets/driverplugin.csi.alibabacloud.com-replace/csi.sock",
        "--driver=nas,oss",
        "--v=1",
        "--run-controller-service=false",
        "--run-node-service=true",
        "--feature-gates=AlinasMountProxy=true"
      ],
      "env": [
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        },
        {
          "name": "KUBELET_ROOT_DIR",
          "value": "/"
        },
        {
          "name": "ALIBABA_CLOUD_NETWORK_TYPE",
          "value": "vpc"
        },
        {
          "name": "REGION_ID",
          "value": "cn-hangzhou"
        },
        {
          "name": "OSS_SKIP_GLOBAL_MOUNT",
          "value": "true"
        },
        {
          "name": "KUBE_NODE_NAME",
          "valueFrom": {
            "fieldRef": {
              "apiVersion": "v1",
              "fieldPath": "spec.nodeName"
            }
          }
        }
      ],
      "resources": {
        "limits": {
          "cpu": "500m",
          "memory": "1Gi"
        },
        "requests": {
          "cpu": "100m",
          "memory": "128Mi"
        }
      },
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "mount-root",
          "mountPath": "/run/csi/mount-root",
          "mountPropagation": "Bidirectional"
        },
        {
          "name": "nas-plugin-dir",
          "mountPath": "/var/run/csi/sockets/nasplugin.csi.alibabacloud.com"
        },
        {
          "name": "oss-plugin-dir",
          "mountPath": "/var/run/csi/sockets/ossplugin.csi.alibabacloud.com"
        },
        {
          "name": "run-cnfs",
          "mountPath": "/run/cnfs"
        },
        {
          "name": "efc-metrics-dir",
          "mountPath": "/host/var/run/efc"
        },
        {
          "name": "ossfs-metrics-dir",
          "mountPath": "/host/var/run/ossfs"
        }
      ],
      "imagePullPolicy": "IfNotPresent",
      "securityContext": {
        "privileged": true
      }
    },
    {
      "name": "csi-agent-sidecar",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.3-cgroupv1-dport-forbidden",
      "args": [
        "--socket=/run/cnfs/alinas-mounter.sock",
        "--v=4"
      ],
      "env": [
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        }
      ],
      "resources": {
        "limits": {
          "cpu": "500m",
          "memory": "1Gi"
        },
        "requests": {
          "cpu": "500m",
          "memory": "1Gi"
        }
      },
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "mount-root",
          "mountPath": "/run/csi/mount-root",
          "mountPropagation": "Bidirectional"
        },
        {
          "name": "cgroup-net-cls",
          "mountPath": "/sys/fs/cgroup/net_cls/kubepods"
        },
        {
          "name": "csi-agent-config",
          "mountPath": "/etc/aliyun-defaults/cpfs"
        },
        {
          "name": "csi-agent-config",
          "mountPath": "/etc/aliyun-defaults/alinas"
        },
        {
          "name": "run-cnfs",
          "mountPath": "/run/cnfs"
        }
      ],
      "imagePullPolicy": "IfNotPresent",
      "securityContext": {
        "privileged": true
      }
    }
  ],
  "volume": [
    {
      "name": "fuse-device",
      "hostPath": {
        "path": "/dev/fuse",
        "type": "CharDevice"
      }
    },
    {
      "name": "cgroup-net-cls",
      "hostPath": {
        "path": "/sys/fs/cgroup/net_cls/kubepods",
        "type": "Directory"
      }
    },
    {
      "name": "mount-root",
      "hostPath": {
        "path": "/var/run/csi",
        "type": "DirectoryOrCreate"
      }
    },
    {
      "name": "nas-plugin-dir",
      "emptyDir": {}
    },
    {
      "name": "oss-plugin-dir",
      "emptyDir": {}
    },
    {
      "name": "run-cnfs",
      "emptyDir": {}
    },
    {
      "name": "efc-metrics-dir",
      "emptyDir": {}
    },
    {
      "name": "ossfs-metrics-dir",
      "emptyDir": {}
    },
    {
      "name": "csi-agent-config",
      "emptyDir": {}
    }
  ]
}

每個配置項的資料結構包含以下欄位：

欄位	說明
`mainContainer`	主容器注入配置，包括環境變數（`env`）、卷掛載（`volumeMounts`）和生命週期鉤子（`lifecycle`）等。格式與Kubernetes `corev1.Container`一致。
`csiSidecar`	Sidecar容器列表，用於注入CSI外掛程式容器或AgentRuntime初始化容器。格式為`corev1.Container`數組。
`volume`	Volume配置列表，定義注入到Pod層級的卷。格式為`corev1.Volume`數組。