The ACK One fleet supports unified management and scheduling of Agent Sandbox across multiple clusters. Distribute a SandboxSet to multiple clusters with cluster-specific settings, schedule SandboxClaims across clusters, and access Agent Sandbox through the E2B SDK.
Preparations
Obtain the KubeConfig of the fleet in the ACK One console and connect to the fleet by using kubectl.
See Manage associated clusters to add at least two associated clusters to the fleet.
See Install add-ons to install the Agent Sandbox-related add-ons in all associated clusters.
ImportantThe version of the ack-agent-sandbox-controller add-on must be v0.5.13-release.1 or later. When you install the add-on, set
SandboxMultiClusterNaming=truein FeatureGate to ensure that Sandbox IDs are unique across multiple clusters.To use the E2B capabilities of multi-cluster Agent Sandbox, configure a different E2B domain for each cluster when you install the ack-sandbox-manager add-on in multiple associated clusters.
Install the AMC command-line help tool.
Configure multi-cluster scheduling for SandboxClaim
Even if you use only the SandboxClaim method and skip the optional E2B configuration, configure the sandbox-system/sandbox-scheduler-config ConfigMap on the fleet before you create or schedule SandboxClaims. This ConfigMap declares the Sandbox capacity of each associated cluster and can optionally record its E2B domain.
Edit the ConfigMap by using kubectl:
kubectl edit cm sandbox-scheduler-config -n sandbox-systemConfigure the associated clusters in the ConfigMap:
Set
nameto the ID of each associated cluster.Set
maxSandboxesto the maximum number of Sandboxes that each cluster is expected to run. This limit excludes the Sandbox replicas in SandboxSets.(Optional) Set
e2bDomainto the E2B domain configured when Sandbox Manager was installed in the cluster. The value is then recorded in the sandbox.one.alibabacloud.com/e2b-domain annotation of the Sandbox resource.
For a configuration example and field descriptions, see Use Sandbox resources to manage multi-cluster Agent Sandbox.
Step 1: Create a Sandbox warm pool by using SandboxSet
The SandboxSet warm pool and the Multi-Cluster Gateway load balancer are billable resources. Sandboxes in the warm pool run continuously, so the warm pool continues to incur charges for as long as it exists. To release the warm pool, delete the SandboxSet from the fleet. For more information about billing, see Agent Sandbox.
Save the following sample to sandbox-set.yaml and then run
kubectl apply -f sandbox-set.yamlto create a SandboxSet in the fleet.A SandboxSet defines the template of Sandboxes and the desired number of available replicas.
apiVersion: agents.kruise.io/v1alpha1 kind: SandboxSet metadata: name: code-interpreter spec: persistentContents: - filesystem replicas: 4 runtimes: - name: csi - name: agent-runtime scaleStrategy: {} template: metadata: labels: alibabacloud.com/acs: "true" alibabacloud.com/compute-class: agent-sandbox alibabacloud.com/compute-qos: default spec: automountServiceAccountToken: false containers: - image: registry-cn-zhangjiakou.ack.aliyuncs.com/acs/code-interpreter:v1.6 imagePullPolicy: IfNotPresent name: sandbox resources: limits: cpu: "1" memory: 1Gi requests: cpu: "1" ephemeral-storage: 30Gi memory: 1Gi restartPolicy: Always updateStrategy: maxUnavailable: 20%Save the following sample to code-interpreter-pp.yaml and then run
kubectl apply -f code-interpreter-pp.yaml.When you create a SandboxSet, also create a PropagationPolicy resource and use the
spec.placement.replicaScheduling.replicaSchedulingTypefield to specify the rule for distributing the SandboxSet to associated clusters. Whenspec.resourceSelectors.kindis set toSandboxSet,replicaSchedulingTypesupports only theDuplicatedoption, which means a complete copy of the SandboxSet is created in each associated cluster.When
preserveResourcesOnDeletionis set tofalse, deleting the resources on the fleet also deletes the resources on the associated clusters. To keep the resources on the associated clusters, set it totrue.apiVersion: policy.one.alibabacloud.com/v1alpha1 kind: PropagationPolicy metadata: name: code-interpreter-pp spec: preserveResourcesOnDeletion: false resourceSelectors: - apiVersion: agents.kruise.io/v1alpha1 kind: SandboxSet placement: replicaScheduling: replicaSchedulingType: DuplicatedView the aggregated SandboxSet status across all associated clusters in the fleet.
kubectl get sandboxsetExpected output:
NAME REPLICAS AVAILABLE UPDATEDREPLICAS UPDATEDAVAILABLEREPLICAS UPDATEREVISION AGE code-interpreter 8 8 8 8 74bfb6**** 5dView the SandboxSet status in different associated clusters.
kubectl amc get sandboxset -MExpected output:
NAME CLUSTER CLUSTER_ALIAS REPLICAS AVAILABLE UPDATEDREPLICAS UPDATEDAVAILABLEREPLICAS UPDATEREVISION AGE ADOPTION code-interpreter <cluster-id-1> sandbox-cluster1 4 4 4 4 56fc44**** 5d Y code-interpreter <cluster-id-2> sandbox-cluster2 4 4 4 4 74bfb6**** 5d Y(Optional) Use OverridePolicy to specify different SandboxSet replica counts and resource specifications for different clusters.
The following sample configures different replica counts and specifications in two clusters for the SandboxSet named
code-interpreter:CLUSTER_1_ID: 5 replicas, with 1 CPU and 2 GiB of memory.CLUSTER_2_ID: 2 replicas, with 2 CPUs and 4 GiB of memory.
Save the following sample to code-interpreter-op.yaml and then run
kubectl apply -f code-interpreter-op.yaml.OverridePolicy uses JSON Patch paths to locate the fields to override.
/spec/replicasoverrides the replica count configuration, and/spec/template/spec/containers/0/resourcesoverrides the container resource configuration.apiVersion: policy.one.alibabacloud.com/v1alpha1 kind: OverridePolicy metadata: name: code-interpreter-op spec: resourceSelectors: - apiVersion: agents.kruise.io/v1alpha1 kind: SandboxSet name: code-interpreter overrideRules: - targetCluster: clusterNames: - CLUSTER_1_ID overriders: plaintext: - operator: replace path: /spec/replicas value: 5 - path: /spec/template/spec/containers/0/resources operator: add value: requests: cpu: "1" memory: "2Gi" limits: cpu: "1" memory: "2Gi" - targetCluster: clusterNames: - CLUSTER_2_ID overriders: plaintext: - operator: replace path: /spec/replicas value: 2 - path: /spec/template/spec/containers/0/resources operator: add value: requests: cpu: "2" memory: "4Gi" limits: cpu: "2" memory: "4Gi"
Step 2: Use Agent Sandbox
After the Sandbox warm pool is created, use Agent Sandbox in either of the following ways:
SandboxClaim method: Request Sandbox instances from the warm pool and manage them in a Kubernetes-native way. Use this method for in-cluster workflows driven by kubectl or CRDs.
E2B SDK method: Use the E2B SDK to create and use Sandboxes through the Multi-Cluster Gateway of the fleet. Use this method when you develop AI agent applications.
E2B SDK method
Agent Sandbox supports integration with the E2B ecosystem. After the E2B capabilities of multi-cluster Agent Sandbox are enabled, the fleet provides a unified Multi-Cluster Gateway entry point and multi-cluster scheduling capabilities. Clients can create, connect to, and use Sandboxes through the E2B SDK without needing to know which backend cluster runs a Sandbox.
Obtain an Agent Sandbox
Through the E2B SDK, develop and deploy projects in a Sandbox and view the results, including running code (run_code), reading and writing files (files.write and files.read), and running commands (commands.run). For more information about how to use the SDK, see Use the E2B SDK to connect to Agent Sandbox Container Edition.
Before you use the E2B capabilities, complete all configurations in(Optional) Configure the E2B capabilities of multi-cluster Agent Sandbox and Step 1: Create a Sandbox warm pool by using SandboxSet.
Install Python in your local environment.
Install the E2B Python SDK. Only versions earlier than v2.25.0 are supported.
pip install "e2b-code-interpreter==2.7.0" "e2b==2.24.0"Configure environment variables.
# fleet.your.domain.com is the fleet E2B domain, which resolves to the Multi-Cluster Gateway. It does not require the * prefix. Modify it based on your actual configuration. export E2B_DOMAIN=fleet.your.domain.com # Use the default generated or updated Fleet Admin API Key export E2B_API_KEY=e2b_0000000000000000000000000000000000000000Save the following code as the
main.pyfile.# Import the E2B SDK from e2b_code_interpreter import Sandbox sbx: Sandbox = Sandbox.create(template="code-interpreter") print(f"sandbox id: {sbx.sandbox_id}") result = sbx.run_code("print('hello, world')") print(f"run code result: {result}") text = input("enter some text to be saved to file 'text.txt' inside sandbox: ") sbx.files.write("text.txt", text) print(f"read file from sandbox via files api: [{sbx.files.read('text.txt')}]") print(f"read file from sandbox via commands api: [{sbx.commands.run('cat text.txt')}]") input("press ENTER to kill the sandbox") print(sbx.kill())For more information about the
metadataparameter of thecreateoperation, see Create an Agent Sandbox Container Edition.Run the
main.pyfile to create and verify a Sandbox.After the first prompt appears, enter text such as
acs agent sandboxand press ENTER. The text is written to the/home/user/text.txtfile of the pod namedcode-interpreter-29***. Press ENTER again to delete the current Sandbox.python main.pyExpected output:
sandbox id: default--code-interpreter-29*** run code result: Execution(Results: [], Logs: Logs(stdout: ['hello, world\n'], stderr: []), Error: None) enter some text to be saved to file 'text.txt' inside sandbox: acs agent sandbox read file from sandbox via files api: [acs agent sandbox] read file from sandbox via commands api: [CommandResult(stderr='', stdout='acs agent sandbox', exit_code=0, error='')] press ENTER to kill the sandbox True
SandboxClaim method
Distribute and schedule SandboxClaims across multiple clusters
A SandboxClaim is used to request Sandbox instances from an existing SandboxSet. Use a PropagationPolicy to control how SandboxClaims are scheduled across multiple clusters.
Save the following sample to sandbox-claim.yaml and then run
kubectl apply -f sandbox-claim.yaml.apiVersion: agents.kruise.io/v1alpha1 kind: SandboxClaim metadata: name: code-interpreter namespace: default spec: templateName: code-interpreter replicas: 1 claimTimeout: 5m ttlAfterCompleted: 5mChoose a scheduling mode based on your requirements:
ImportantDo not configure different PropagationPolicies for the same SandboxClaim resource.
Basic scheduling (Gang scheduling): automatically selects one associated cluster and schedules all replicas requested by the SandboxClaim to that cluster. Perform the following operations:
Save the following sample to sandboxclaim-pp.yaml and then run
kubectl apply -f sandboxclaim-pp.yaml.apiVersion: policy.one.alibabacloud.com/v1alpha1 kind: PropagationPolicy metadata: name: sandboxclaim-pp spec: preserveResourcesOnDeletion: false resourceSelectors: - apiVersion: agents.kruise.io/v1alpha1 kind: SandboxClaim placement: replicaScheduling: replicaSchedulingType: Divided customSchedulingType: GangView the SandboxClaim on the fleet.
kubectl get sandboxclaimExpected output:
NAME PHASE TEMPLATE DESIRED CLAIMED AGE code-interpreter Completed code-interpreter 1 1 6sView the SandboxClaim on the associated cluster.
kubectl amc get sandboxclaim -MExpected output:
NAME CLUSTER CLUSTER_ALIAS PHASE TEMPLATE DESIRED CLAIMED AGE ADOPTION code-interpreter <CLUSTER-ID> sandbox-cluster1 Completed code-interpreter 1 1 3s YReplace
CLUSTER_IDin the following command with the cluster ID obtained in the previous step and then run the command to get Sandbox details by using the SandboxClaim name label.kubectl amc get sandbox -m CLUSTER_ID -l agents.kruise.io/claim-name=code-interpreterExpected output:
NAME CLUSTER CLUSTER_ALIAS STATUS AGE CLAIMED SHUTDOWN_TIME PAUSE_TIME MESSAGE ADOPTION code-interpreter-47*** CLUSTER_ID sandbox-cluster1 Running 59m true
Cluster priority scheduling: sets scheduling priorities for clusters. The scheduler selects the cluster with a higher priority and schedules all replicas requested by the SandboxClaim to that cluster. When the resources of the high-priority cluster are insufficient, it automatically falls back to a lower-priority cluster. Perform the following operations:
Save the following sample to sandboxclaim-pp-priority.yaml and then run
kubectl apply -f sandboxclaim-pp-priority.yaml.The order declared in the
spec.placement.clusterAffinitiesfield is the scheduling priority order. The PropagationPolicy in the following sample schedules toCLUSTER_1_ID(primary) first, and schedules toCLUSTER_2_ID(secondary) when resources are insufficient.apiVersion: policy.one.alibabacloud.com/v1alpha1 kind: PropagationPolicy metadata: name: sandboxclaim-pp-priority spec: preserveResourcesOnDeletion: false resourceSelectors: - apiVersion: agents.kruise.io/v1alpha1 kind: SandboxClaim placement: replicaScheduling: customSchedulingType: Gang replicaSchedulingType: Divided clusterAffinities: - affinityName: primary clusterNames: - CLUSTER_1_ID - affinityName: secondary clusterNames: - CLUSTER_2_IDView the SandboxClaim on the fleet.
kubectl get sandboxclaimExpected output:
NAME PHASE TEMPLATE DESIRED CLAIMED AGE code-interpreter Completed code-interpreter 1 1 6sVerify that the SandboxClaim is deployed to CLUSTER_1_ID according to the priority rules.
kubectl amc get sandboxclaim -MExpected output:
NAME CLUSTER CLUSTER_ALIAS PHASE TEMPLATE DESIRED CLAIMED AGE ADOPTION code-interpreter <CLUSTER-1-ID> sandbox-cluster1 Completed code-interpreter 1 1 3s YReplace
CLUSTER_IDin the following command with the cluster ID obtained in the previous step and then run the command to get Sandbox details by using the SandboxClaim name label.kubectl amc get sandbox -m CLUSTER_ID -l agents.kruise.io/claim-name=code-interpreterExpected output:
NAME CLUSTER CLUSTER_ALIAS STATUS AGE CLAIMED SHUTDOWN_TIME PAUSE_TIME MESSAGE ADOPTION code-interpreter-47*** CLUSTER_ID sandbox-cluster1 Running 59m true
Step 3: (Optional) Configure multi-cluster failover
The SandboxClaim link and the E2B link use different multi-cluster failover mechanisms. Configure failover based on the tab that corresponds to the method you use.
E2B link
For E2B scenarios, the ACK One fleet supports the following two methods to determine whether a cluster has failed:
Automatic probing: Automatically probes the health status of the Sandbox Manager add-on in each associated cluster.
Manual configuration: Configure
disabled: truefor a cluster insandbox-system/sandbox-e2b-scheduler-config, which drains the cluster for maintenance.
Failure detection method | Description | Determination rules | Effects |
Automatic probing | Periodically probes the health status of the Sandbox Manager add-on in each associated cluster at an interval of 5 seconds. |
| When a cluster is marked as
|
Manual configuration | Manually configure |
| When a cluster is configured with
|
To manually drain a cluster, add disabled: true to the configuration of the cluster in the sandbox-e2b-scheduler-config ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: sandbox-e2b-scheduler-config
namespace: sandbox-system
data:
config: |
...
# Fields per cluster:
# disabled — bool, optional — drain the cluster (exclude from create scheduling)
clusters:
- name: $clusterid
disabled: true
...To manually restore a cluster, change the configuration of the cluster to disabled: false.
SandboxClaim link
The ACK One fleet uses Open Cluster Management (OCM) to monitor network connectivity between the associated clusters and the fleet. If the fleet does not receive a heartbeat from the OCM agent in an associated cluster for 5 minutes, the fleet automatically adds the NoSelect taint to that cluster. SandboxClaims are then moved to other available associated clusters.
To perform maintenance on an associated cluster, manually add a taint to that cluster:
Adding or removing taints does not affect the Sandbox instances that are already running on the cluster.
Replace
CLUSTER_IDin the following command with the ID of the target associated cluster, and then run the command to add a taint to that cluster. The scheduler then stops assigning SandboxClaims to the tainted cluster:kubectl patch managedcluster CLUSTER_ID --type=merge -p '{ "spec": { "taints": [ {"key": "maintenance", "value": "true", "effect": "NoSelect"} ] } }'After the cluster maintenance is complete, remove the taint to restore scheduling on the associated cluster:
kubectl patch managedcluster CLUSTER_ID --type=json -p '[{"op":"remove","path":"/spec/taints"}]'