Configure automatic sidecar injection for Agent Sandbox - Container Compute Service

To simplify the YAML configuration for storage mounting and runtime injection in Agent Sandbox, you can configure the spec.runtimes field in a Sandbox resource. This automatically injects CSI storage mount and AgentRuntime configurations, streamlining the deployment process.

Prerequisites

You have set up the Agent Sandbox environment. For more information, see Create an Agent Sandbox.
In the Add-ons section of your cluster, verify that the ack-agent-sandbox-controller component is version v0.5.10 or later.
The component installer creates a ConfigMap object named sandbox-injection-config in the sandbox-system namespace with default configurations. For custom settings or modifications, contact Alibaba Cloud technical support to validate the configuration before making changes.

Overview

When you configure dynamic storage mounts in an Agent Sandbox, the YAML configuration is typically complex because you must manually configure CSI and sidecar containers. With the Sidecar automatic injection feature, you only need to define the application container and declare the spec.runtimes field in the SandboxSet or Sandbox resource. The system then automatically injects the required CSI and AgentRuntime Sidecar configurations into each new Sandbox. The following two injection types are currently supported:

csi: Injects the init container and Volume configurations for mounting storage by using CSI, enabling the mounting of shared storage such as NAS and OSS.
agent-runtime: Injects the AgentRuntime init container (an environment management tool such as envd) and adds corresponding environment variables and lifecycle hooks to the application container.

Automatic sidecar injection affects only new Sandbox instances. To use the Command and Filesystem APIs, you must inject agent-runtime.

Configure sidecar injection

Declare the runtime type to be injected in the spec.runtimes field of a SandboxSet or Sandbox resource. The following examples show the configurations for both types of resources.

Example: SandboxSet resource

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxSet
metadata:
  name: code-interpreter-inject-test
  namespace: default
spec:
  runtimes:
  - name: csi           # Enables CSI mounting capabilities. The corresponding sidecar is injected into new Sandbox instances.
  - name: agent-runtime # Injects environment management tools like envd.
  replicas: 4
  template:
    metadata:
      labels:
        alibabacloud.com/acs: "true"
    spec:
      automountServiceAccountToken: false
      containers:
      - image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/code-interpreter:v1.6 # Replace with your cluster's region ID.
        imagePullPolicy: IfNotPresent
        name: sandbox
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
      terminationGracePeriodSeconds: 30

Example: Sandbox resource

apiVersion: agents.kruise.io/v1alpha1
kind: Sandbox
metadata:
  name: code-interpreter-inject-test-xxx
  namespace: default
spec:
  runtimes:
  - name: csi           # Provides CSI mounting capabilities.
  - name: agent-runtime # Injects environment management tools like envd.
  ...

Example: Pod manifest after injection

# Note: The <region-id> in the image address is automatically replaced with the region ID of your cluster.
apiVersion: v1
kind: Pod
metadata:
  annotations:
    agents.kruise.io/created-by: sandbox
  labels:
    agents.kruise.io/sandbox-pool: code-interpreter-init-xxx
    alibabacloud.com/acs: "true"
  name: code-interpreter-init-xxx
  namespace: default
spec:
  automountServiceAccountToken: false
  containers:
  - env:
    # --- The following configurations are automatically injected ---
    - name: ENVD_DIR
      value: /mnt/envd
    - name: GODEBUG
      value: multipathtcp=0
    - name: POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    # --- End of automatically injected configurations ---
    image: example:tag
    imagePullPolicy: IfNotPresent
    # --- The following configurations are automatically injected ---
    lifecycle:
      postStart:
        exec:
          command:
          - bash
          - -c
          - /mnt/envd/envd-run.sh
    # --- End of automatically injected configurations ---
    name: sandbox
    resources:
      limits:
        cpu: "2"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 2Gi
    volumeMounts:
    # --- The following configurations are automatically injected ---
    - mountPath: /mnt/envd
      name: envd-volume
    - mountPath: /run/csi/mount-root
      mountPropagation: HostToContainer
      name: mount-root
    - mountPath: /var/run/csi/sockets/nasplugin.csi.alibabacloud.com
      name: nas-plugin-dir
    - mountPath: /var/run/csi/sockets/ossplugin.csi.alibabacloud.com
      name: oss-plugin-dir
    # --- End of automatically injected configurations ---
  # --- The following configurations are automatically injected ---
  initContainers:
  - command:
    - sh
    - /workspace/entrypoint_inner.sh
    env:
    - name: ENVD_DIR
      value: /mnt/envd
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/agent-runtime:v0.0.5
    imagePullPolicy: IfNotPresent
    name: init
    resources: {}
    restartPolicy: Always
    volumeMounts:
    - mountPath: /mnt/envd
      name: envd-volume
  - args:
    - --endpoint=unix://var/run/csi/sockets/driverplugin.csi.alibabacloud.com-replace/csi.sock
    - --driver=nas,oss
    - --v=1
    - --run-controller-service=false
    - --run-node-service=true
    - --feature-gates=AlinasMountProxy=true
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    - name: KUBELET_ROOT_DIR
      value: /
    - name: ALIBABA_CLOUD_NETWORK_TYPE
      value: vpc
    - name: REGION_ID
      value: cn-hangzhou
    - name: OSS_SKIP_GLOBAL_MOUNT
      value: "true"
    - name: KUBE_NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-plugin:v1.35.1-2592a4872
    imagePullPolicy: IfNotPresent
    name: csi-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    restartPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /run/csi/mount-root
      mountPropagation: Bidirectional
      name: mount-root
    - mountPath: /var/run/csi/sockets/nasplugin.csi.alibabacloud.com
      name: nas-plugin-dir
    - mountPath: /var/run/csi/sockets/ossplugin.csi.alibabacloud.com
      name: oss-plugin-dir
    - mountPath: /run/cnfs
      name: run-cnfs
    - mountPath: /host/var/run/efc
      name: efc-metrics-dir
    - mountPath: /host/var/run/ossfs
      name: ossfs-metrics-dir
  - args:
    - --socket=/run/cnfs/alinas-mounter.sock
    - --v=4
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.3-cgroupv1-dport-forbidden
    imagePullPolicy: IfNotPresent
    name: csi-agent-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 1Gi
    restartPolicy: Always
    securityContext:
      privileged: true
    volumeMounts:
    - mountPath: /run/csi/mount-root
      mountPropagation: Bidirectional
      name: mount-root
    - mountPath: /sys/fs/cgroup/net_cls/kubepods
      name: cgroup-net-cls
    - mountPath: /etc/aliyun-defaults/cpfs
      name: csi-agent-config
    - mountPath: /etc/aliyun-defaults/alinas
      name: csi-agent-config
    - mountPath: /run/cnfs
      name: run-cnfs
  # --- End of automatically injected configurations ---
  restartPolicy: Always
  volumes:
  # --- The following configurations are automatically injected ---
  - emptyDir: {}
    name: envd-volume
  - hostPath:
      path: /dev/fuse
      type: CharDevice
    name: fuse-device
  - hostPath:
      path: /sys/fs/cgroup/net_cls/kubepods
      type: Directory
    name: cgroup-net-cls
  - hostPath:
      path: /var/run/csi
      type: DirectoryOrCreate
    name: mount-root
  - emptyDir: {}
    name: nas-plugin-dir
  - emptyDir: {}
    name: oss-plugin-dir
  - emptyDir: {}
    name: run-cnfs
  - emptyDir: {}
    name: efc-metrics-dir
  - emptyDir: {}
    name: ossfs-metrics-dir
  - emptyDir: {}
    name: csi-agent-config
  # --- End of automatically injected configurations ---

Injection configuration

The following code shows the default injection configurations provided in v0.5.9 and later, which you typically do not need to modify. The system automatically replaces the <region-id> placeholder in image addresses with the region ID of your cluster (for example, cn-zhangjiakou).

Default agent-runtime injection configuration

{
  "mainContainer": {
    "name": "",
    "env": [
      {
        "name": "ENVD_DIR",
        "value": "/mnt/envd"
      },
      {
        "name": "GODEBUG",
        "value": "multipathtcp=0"
      },
      {
        "name": "POD_UID",
        "valueFrom": {
          "fieldRef": {
            "fieldPath": "metadata.uid"
          }
        }
      }
    ],
    "resources": {},
    "volumeMounts": [
      {
        "name": "envd-volume",
        "mountPath": "/mnt/envd"
      }
    ],
    "lifecycle": {
      "postStart": {
        "exec": {
          "command": [
            "bash",
            "-c",
            "/mnt/envd/envd-run.sh"
          ]
        }
      }
    }
  },
  "csiSidecar": [
    {
      "name": "init",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/agent-runtime:v0.0.5",
      "command": [
        "sh",
        "/workspace/entrypoint_inner.sh"
      ],
      "env": [
        {
          "name": "ENVD_DIR",
          "value": "/mnt/envd"
        },
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        }
      ],
      "resources": {},
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "envd-volume",
          "mountPath": "/mnt/envd"
        }
      ],
      "imagePullPolicy": "IfNotPresent"
    }
  ],
  "volume": [
    {
      "name": "envd-volume",
      "emptyDir": {}
    }
  ]
}

Default csi injection configuration

{
  "mainContainer": {
    "name": "",
    "resources": {},
    "volumeMounts": [
      {
        "name": "mount-root",
        "mountPath": "/run/csi/mount-root",
        "mountPropagation": "HostToContainer"
      },
      {
        "name": "nas-plugin-dir",
        "mountPath": "/var/run/csi/sockets/nasplugin.csi.alibabacloud.com"
      },
      {
        "name": "oss-plugin-dir",
        "mountPath": "/var/run/csi/sockets/ossplugin.csi.alibabacloud.com"
      }
    ]
  },
  "csiSidecar": [
    {
      "name": "csi-sidecar",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-plugin:v1.35.1-2592a4872",
      "args": [
        "--endpoint=unix://var/run/csi/sockets/driverplugin.csi.alibabacloud.com-replace/csi.sock",
        "--driver=nas,oss",
        "--v=1",
        "--run-controller-service=false",
        "--run-node-service=true",
        "--feature-gates=AlinasMountProxy=true"
      ],
      "env": [
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        },
        {
          "name": "KUBELET_ROOT_DIR",
          "value": "/"
        },
        {
          "name": "ALIBABA_CLOUD_NETWORK_TYPE",
          "value": "vpc"
        },
        {
          "name": "REGION_ID",
          "value": "cn-hangzhou"
        },
        {
          "name": "OSS_SKIP_GLOBAL_MOUNT",
          "value": "true"
        },
        {
          "name": "KUBE_NODE_NAME",
          "valueFrom": {
            "fieldRef": {
              "apiVersion": "v1",
              "fieldPath": "spec.nodeName"
            }
          }
        }
      ],
      "resources": {
        "limits": {
          "cpu": "500m",
          "memory": "1Gi"
        },
        "requests": {
          "cpu": "100m",
          "memory": "128Mi"
        }
      },
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "mount-root",
          "mountPath": "/run/csi/mount-root",
          "mountPropagation": "Bidirectional"
        },
        {
          "name": "nas-plugin-dir",
          "mountPath": "/var/run/csi/sockets/nasplugin.csi.alibabacloud.com"
        },
        {
          "name": "oss-plugin-dir",
          "mountPath": "/var/run/csi/sockets/ossplugin.csi.alibabacloud.com"
        },
        {
          "name": "run-cnfs",
          "mountPath": "/run/cnfs"
        },
        {
          "name": "efc-metrics-dir",
          "mountPath": "/host/var/run/efc"
        },
        {
          "name": "ossfs-metrics-dir",
          "mountPath": "/host/var/run/ossfs"
        }
      ],
      "imagePullPolicy": "IfNotPresent",
      "securityContext": {
        "privileged": true
      }
    },
    {
      "name": "csi-agent-sidecar",
      "image": "registry-<region-id>-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.3-cgroupv1-dport-forbidden",
      "args": [
        "--socket=/run/cnfs/alinas-mounter.sock",
        "--v=4"
      ],
      "env": [
        {
          "name": "__IGNORE_RESOURCE__",
          "value": "true"
        }
      ],
      "resources": {
        "limits": {
          "cpu": "500m",
          "memory": "1Gi"
        },
        "requests": {
          "cpu": "500m",
          "memory": "1Gi"
        }
      },
      "restartPolicy": "Always",
      "volumeMounts": [
        {
          "name": "mount-root",
          "mountPath": "/run/csi/mount-root",
          "mountPropagation": "Bidirectional"
        },
        {
          "name": "cgroup-net-cls",
          "mountPath": "/sys/fs/cgroup/net_cls/kubepods"
        },
        {
          "name": "csi-agent-config",
          "mountPath": "/etc/aliyun-defaults/cpfs"
        },
        {
          "name": "csi-agent-config",
          "mountPath": "/etc/aliyun-defaults/alinas"
        },
        {
          "name": "run-cnfs",
          "mountPath": "/run/cnfs"
        }
      ],
      "imagePullPolicy": "IfNotPresent",
      "securityContext": {
        "privileged": true
      }
    }
  ],
  "volume": [
    {
      "name": "fuse-device",
      "hostPath": {
        "path": "/dev/fuse",
        "type": "CharDevice"
      }
    },
    {
      "name": "cgroup-net-cls",
      "hostPath": {
        "path": "/sys/fs/cgroup/net_cls/kubepods",
        "type": "Directory"
      }
    },
    {
      "name": "mount-root",
      "hostPath": {
        "path": "/var/run/csi",
        "type": "DirectoryOrCreate"
      }
    },
    {
      "name": "nas-plugin-dir",
      "emptyDir": {}
    },
    {
      "name": "oss-plugin-dir",
      "emptyDir": {}
    },
    {
      "name": "run-cnfs",
      "emptyDir": {}
    },
    {
      "name": "efc-metrics-dir",
      "emptyDir": {}
    },
    {
      "name": "ossfs-metrics-dir",
      "emptyDir": {}
    },
    {
      "name": "csi-agent-config",
      "emptyDir": {}
    }
  ]
}

The data structure for each configuration item contains the following fields:

Field	Description
`mainContainer`	The injection configuration for the main container includes items such as environment variables (`env`), volume mounts (`volumeMounts`), and lifecycle hooks (`lifecycle`). The format is consistent with the Kubernetes `corev1.Container`.
`csiSidecar`	A list of sidecar containers used to inject CSI plugin containers or AgentRuntime init containers. The format is an array of `corev1.Container`.
`volume`	A list of volumes to be injected at the Pod level, formatted as an array of `corev1.Volume`.