Mount shared storage for an Agent Sandbox - Container Compute Service

This topic describes how to mount an OSS or NAS static storage volume to a sandbox when creating an Agent Sandbox.

Prerequisites

You have set up the basic environment for Agent Sandbox. For more information, see Create an Agent Sandbox.
In the Add-ons section of your cluster, verify that the ack-agent-sandbox-controller component is version v0.5.10 or later.
The component installer creates a ConfigMap object named sandbox-injection-config in the sandbox-system namespace with default configurations. For custom settings or modifications, contact Alibaba Cloud technical support to validate the configuration before making changes.
In the Add-ons section of your cluster, verify that the ack-sandbox-manager component is version v0.5.0 or later.

This is a whitelist feature. To use it, please submit a ticket.

Limitations

The feature supports runtime storage mounting for the create function of the E2B interface, the hibernate/wake up function, and in-place image upgrades. However, you cannot restore storage from a snapshot created with the checkpoint mechanism.
A sandbox with mounted NAS shared storage cannot hibernate.
OSS static storage mounts support only AccessKey authentication.
When configuring a network policy or traffic policy, allow egress traffic to the OSS and NAS control plane endpoints.

Mount shared storage

ACS supports mounting a PersistentVolume (PV), such as Alibaba Cloud NAS, to a sandbox at runtime when you call the Sandbox.create or Sandbox.beta_create API. This feature depends on the agent-runtime component.

Step 1: Configure storage mounting capabilities

ACS provides a configuration solution for dynamically injecting Sidecars. You only need to set the spec.runtimes parameter on a SandboxSet or Sandbox object. The framework then automatically injects the required CSI configuration, including the agent-runtime and storage-related CSI Sidecar configurations, when a Sandbox is created.

The following is an example of a SandboxSet configuration:

apiVersion: agents.kruise.io/v1alpha1
kind: SandboxSet
metadata:
  name: code-interpreter-inject-test
  namespace: default
spec:
  runtimes:
  - name: csi           # Enables CSI mounting. The corresponding Sidecar is injected into new Sandboxes.
  - name: agent-runtime # Injects environment management tools such as envd.
  replicas: 4
  template:
    metadata:
      labels:
        alibabacloud.com/acs: "true"
    spec:
      automountServiceAccountToken: false
      containers:
      - image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/code-interpreter:v1.6
        imagePullPolicy: IfNotPresent
        name: sandbox
        resources:
          limits:
            cpu: "1"
            memory: 1Gi
          requests:
            cpu: "1"
            memory: 1Gi
      terminationGracePeriodSeconds: 30

The following is an example of a Sandbox object configuration:

apiVersion: agents.kruise.io/v1alpha1
kind: Sandbox
metadata:
  name: code-interpreter-inject-test-xxx
  namespace: default
spec:
  runtimes:
  - name: csi           # Provides CSI mounting capabilities.
  - name: agent-runtime # Injects environment management tools such as envd.
  ...

Example of a Pod after injection

The following are the configurations that are automatically injected into the Pod object of a Sandbox after a SandboxSet is scaled out. These configurations include the agent-runtime, csi-sidecar, and csi-agent-sidecar init containers, as well as the related volumes and volumeMounts.

apiVersion: v1
kind: Pod
metadata:
  labels:
    agents.kruise.io/sandbox-pool: code-interpreter-inject-test
    alibabacloud.com/acs: "true"
  name: code-interpreter-inject-test-xxx
  namespace: default
spec:
  containers:
  - env:
    # Injected environment variables
    - name: ENVD_DIR
      value: /mnt/envd
    - name: GODEBUG
      value: multipathtcp=0
    - name: POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    image: example:tag
    imagePullPolicy: IfNotPresent
    # Injected lifecycle hook
    lifecycle:
      postStart:
        exec:
          command:
          - bash
          - -c
          - /mnt/envd/envd-run.sh
    name: sandbox
    resources:
      limits:
        cpu: "2"
        memory: 2Gi
      requests:
        cpu: "2"
        memory: 2Gi
    volumeMounts:
    # Injected volumeMounts
    - mountPath: /mnt/envd
      name: envd-volume
    - mountPath: /run/csi/mount-root
      mountPropagation: HostToContainer
      name: mount-root
    - mountPath: /var/run/csi/sockets/nasplugin.csi.alibabacloud.com
      name: nas-plugin-dir
    - mountPath: /var/run/csi/sockets/ossplugin.csi.alibabacloud.com
      name: oss-plugin-dir
  initContainers:
  # Injected agent-runtime init container
  - command:
    - sh
    - /workspace/entrypoint_inner.sh
    env:
    - name: ENVD_DIR
      value: /mnt/envd
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/agent-runtime:v0.0.5
    name: init
    restartPolicy: Always
  # Injected csi-sidecar container
  - args:
    - --endpoint=unix://var/run/csi/sockets/driverplugin.csi.alibabacloud.com-replace/csi.sock
    - --driver=nas,oss
    - --v=1
    - --run-controller-service=false
    - --run-node-service=true
    - --feature-gates=AlinasMountProxy=true
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    - name: KUBELET_ROOT_DIR
      value: /
    - name: ALIBABA_CLOUD_NETWORK_TYPE
      value: vpc
    - name: REGION_ID
      value: cn-zhangjiakou   # Replace with your actual region ID.
    - name: OSS_SKIP_GLOBAL_MOUNT
      value: "true"
    image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/csi-plugin:v1.35.1-2592a4872
    name: csi-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 128Mi
    restartPolicy: Always
    securityContext:
      privileged: true
  # Injected csi-agent-sidecar container
  - args:
    - --socket=/run/cnfs/alinas-mounter.sock
    - --v=4
    env:
    - name: __IGNORE_RESOURCE__
      value: "true"
    image: registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.4-3c34d4a-aliyun
    name: csi-agent-sidecar
    resources:
      limits:
        cpu: 500m
        memory: 1Gi
      requests:
        cpu: 500m
        memory: 1Gi
    restartPolicy: Always
    securityContext:
      privileged: true
  # Injected volumes
  volumes:
  - emptyDir: {}
    name: envd-volume
  - hostPath:
      path: /dev/fuse
      type: CharDevice
    name: fuse-device
  - hostPath:
      path: /var/run/csi
      type: DirectoryOrCreate
    name: mount-root
  - emptyDir: {}
    name: nas-plugin-dir
  - emptyDir: {}
    name: oss-plugin-dir
  - emptyDir: {}
    name: run-cnfs
  - emptyDir: {}
    name: efc-metrics-dir
  - emptyDir: {}
    name: ossfs-metrics-dir
  - emptyDir: {}
    name: csi-agent-config

Step 2: Create a PersistentVolume

Create the corresponding PV object in your cluster based on your storage requirements. The following sections describe how to create static storage volumes for OSS and NAS.

Create an OSS PersistentVolume

Create a PersistentVolume object that describes the OSS volume and a corresponding Secret object for the access key. For more information, see Use a static OSS volume.

apiVersion: v1
kind: Secret
metadata:
  name: oss-secret
  # The Secret must be created in the sandbox-system namespace.
  namespace: sandbox-system
stringData:
  akId: <YOUR_ACCESS_KEY_ID>       # AccessKey ID used to access OSS
  akSecret: <YOUR_ACCESS_KEY_SECRET> # AccessKey Secret used to access OSS
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    alicloud-pvname: oss-pv-sandbox-system
  name: oss-pv-sandbox-system
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  csi:
    driver: ossplugin.csi.alibabacloud.com
    nodePublishSecretRef:
      name: oss-secret
      namespace: sandbox-system
    volumeAttributes:
      bucket: <OSS_BUCKET>               # Replace with the actual name of your bucket in OSS.
      otherOpts: -o umask=022 -o allow_other
      url: <OSS_ENDPOINT>               # The OSS endpoint. Replace with your actual OSS endpoint address.
    volumeHandle: oss-pv-sandbox-system
  persistentVolumeReclaimPolicy: Retain
  storageClassName: test
  volumeMode: Filesystem

Create a NAS PersistentVolume

Create a PersistentVolume resource for NAS. NAS storage does not support public network mounts. You must configure a mount point in the NAS console in advance. For more information, see Use a static NAS volume.

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    alicloud-pvname: nas-pv-sandbox-system
  name: nas-pv-sandbox-system
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 5Gi
  csi:
    driver: nasplugin.csi.alibabacloud.com
    volumeAttributes:
      # Replace with the actual address of your NAS mount point. The VPC of the mount point must be the same as the VPC of the cluster.
      server: xxx.xxx.extreme.nas.aliyuncs.com
      # The directory path of the NAS mount.
      path: /share
    volumeHandle: nas-pv-sandbox-system
  mountOptions:
  - nolock,tcp,noresvport
  - vers=3
  persistentVolumeReclaimPolicy: Retain
  volumeMode: Filesystem

Step 3: Mount a PersistentVolume with the E2B SDK

Mount a single PersistentVolume

The following Python code shows how to get a Sandbox from the warm pool and mount the PersistentVolume to a specified directory.

Example: Mount an OSS PersistentVolume

from e2b_code_interpreter import Sandbox

sbx = Sandbox.create(template="code-interpreter-inject-test", timeout=300, metadata={
    "e2b.agents.kruise.io/csi-volume-name": "oss-pv-sandbox-system", # The name of the OSS PersistentVolume.
    "e2b.agents.kruise.io/csi-mount-point": "/data-oss",
    "e2b.agents.kruise.io/csi-subpath": "data-subPath"  # A relative path that specifies a subdirectory within the OSS bucket.
})

Example: Mount a NAS PersistentVolume

from e2b_code_interpreter import Sandbox

sbx = Sandbox.create(template="code-interpreter-inject-test", timeout=300, metadata={
    "e2b.agents.kruise.io/csi-volume-name": "nas-pv-sandbox-system",
    "e2b.agents.kruise.io/csi-mount-point": "/data-nas",
    "e2b.agents.kruise.io/csi-subpath": "data-subPath"
})

If you receive a 504 response, navigate to Configurations > ConfigMaps, modify the sandbox-injection-config file by replacing registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.3-cgroupv1-dport-forbidden with registry-cn-zhangjiakou-vpc.ack.aliyuncs.com/acs/csi-agent:v1.35.4-3c34d4a-aliyun, and then create the Sandbox again.

The metadata parameter is described as follows:

Parameter	Description
`e2b.agents.kruise.io/csi-volume-name`	The name of the PersistentVolume object.
`e2b.agents.kruise.io/csi-mount-point`	The mount path for the volume in the container.
`e2b.agents.kruise.io/csi-subpath`	The subdirectory name (relative path) within the remote storage.

Mount multiple PersistentVolumes

If you need to mount multiple PersistentVolumes at once, you can use the e2b.agents.kruise.io/csi-volume-config parameter to specify multiple mount configurations in JSON array format.

If both the e2b.agents.kruise.io/csi-volume-config (new protocol) and the single-volume mount parameters (old protocol) are configured, the new protocol has a higher priority than the old protocol.

The following Python example mounts multiple PersistentVolumes:

from e2b_code_interpreter import Sandbox

sbx = Sandbox.create(template="code-interpreter-inject-test", timeout=600, metadata={
    "e2b.agents.kruise.io/csi-volume-config": '[{"pvName":"pv-nas-fast","mountPath":"/data-nas","subPath":"data-subPath","readOnly":true},{...}]'
})
print(f"sandbox id: {sbx.sandbox_id}")

The configurations for csi-volume-config are as follows:

Field	Type	Description
`pvName`	String	The name of the PersistentVolume object.
`mountPath`	String	The mount path within the container.
`subPath`	String	The subdirectory name (relative path) within the remote storage. Optional.
`readOnly`	Boolean	Whether to mount the volume as read-only. Optional. Default: false.