All Products
Search
Document Center

Container Service for Kubernetes:Manage Agent Sandbox by using ACK One multi-cluster fleet

Last Updated:Oct 21, 2026

The ACK One fleet supports unified management and scheduling of Agent Sandbox across multiple clusters. Distribute a SandboxSet to multiple clusters with cluster-specific settings, schedule SandboxClaims across clusters, and access Agent Sandbox through the E2B SDK.

Preparations

  1. Enable fleet management.

  2. Obtain the KubeConfig of the fleet in the ACK One console and connect to the fleet by using kubectl.

  3. See Manage associated clusters to add at least two associated clusters to the fleet.

  4. See Install add-ons to install the Agent Sandbox-related add-ons in all associated clusters.

    Important

    The version of the ack-agent-sandbox-controller add-on must be v0.5.13-release.1 or later. When you install the add-on, set SandboxMultiClusterNaming=true in FeatureGate to ensure that Sandbox IDs are unique across multiple clusters.

    To use the E2B capabilities of multi-cluster Agent Sandbox, configure a different E2B domain for each cluster when you install the ack-sandbox-manager add-on in multiple associated clusters.
  5. Install the AMC command-line help tool.

Configure multi-cluster scheduling for SandboxClaim

Even if you use only the SandboxClaim method and skip the optional E2B configuration, configure the sandbox-system/sandbox-scheduler-config ConfigMap on the fleet before you create or schedule SandboxClaims. This ConfigMap declares the Sandbox capacity of each associated cluster and can optionally record its E2B domain.

  1. Edit the ConfigMap by using kubectl:

    kubectl edit cm sandbox-scheduler-config -n sandbox-system
  2. Configure the associated clusters in the ConfigMap:

    • Set name to the ID of each associated cluster.

    • Set maxSandboxes to the maximum number of Sandboxes that each cluster is expected to run. This limit excludes the Sandbox replicas in SandboxSets.

    • (Optional) Set e2bDomain to the E2B domain configured when Sandbox Manager was installed in the cluster. The value is then recorded in the sandbox.one.alibabacloud.com/e2b-domain annotation of the Sandbox resource.

For a configuration example and field descriptions, see Use Sandbox resources to manage multi-cluster Agent Sandbox.

(Optional) Configure the E2B capabilities of multi-cluster Agent Sandbox

If you plan to use Agent Sandbox through the E2B SDK, complete all configurations in this section before you create the Sandbox warm pool. If you use only the SandboxClaim method, skip this section.

Step 1: Enable the E2B capabilities of multi-cluster Agent Sandbox

  1. Log on to the ACK One console. In the left-side navigation pane, choose Fleet > Fleet Information.

  2. In Basic Information, enable Multi-Cluster Agent Sandbox.

Step 2: Configure the Multi-Cluster Gateway certificate and trust chain on the fleet

This step prepares the TLS certificate and the backend CA trust anchor required by the Multi-Cluster Gateway.

  1. Prepare the certificate and the CA bundle. For information about how to obtain certificates, see the certificate application section in Apply for a certificate. Choose one of the following methods based on your environment:

    Important

    To use cert-manager on the fleet to automatically issue certificates, submit a ticket to apply for the related fleet permissions.

    • Generate a self-signed certificate (development and test environments): The Subject Alternative Name (SAN) must cover the fleet domain, such as *.fleet.your.domain.com.

    • Use a certificate issued by a publicly trusted CA (production environments): The Gateway front-end certificate must match the fleet domain so that clients trust it. The CA bundle must contain the CAs that issued the Application Load Balancer (ALB) certificates of all associated clusters. Multiple PEM files can be concatenated.

  2. Create the Gateway front-end TLS certificate. Import the certificate prepared in the previous step as a TLS secret:

    kubectl create secret tls global-sandbox-scheduler-tls \
      --cert=certs/tls.crt \
      --key=certs/tls.key \
      -n sandbox-system
  3. Create the backend CA trust anchor, which is used by BackendTLSPolicy to verify the ALB certificates of associated clusters and by the HTTPS client for health probing. Choose one of the following methods based on the certificate type of the associated clusters:

    • Self-signed certificates (development and test environments): If the associated clusters use different CAs, concatenate the CA PEM files into a single file, and then create a ConfigMap:

      # Concatenate the CAs of multiple associated clusters (skip if the same CA is used)
      cat cluster1-ca.crt cluster2-ca.crt > certs/ca.crt
      
      kubectl create cm sandbox-ca-cm \
        --from-file=certs/ca.crt \
        -n sandbox-system
    • Certificates issued by a publicly trusted CA (production environments): If the domains of all associated clusters use certificates issued by a publicly trusted CA, no CA ConfigMap is required. In Step 3 below, configure wellKnownCACertificates: "System" for each cluster in the sandbox-e2b-scheduler-config ConfigMap:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: sandbox-e2b-scheduler-config
        namespace: sandbox-system
      data:
        config: |
          clusters:
          - name: c727xxxx7843
            wellKnownCACertificates: "System"
          ...

Step 3: Configure the multi-cluster scheduling information on the fleet

The sandbox-e2b-scheduler-configConfigMap on the fleet is the scheduling configuration entry point for the multi-cluster Agent Sandbox E2B link. It contains the configurations of the associated clusters and is used for Sandbox scheduling and routing.

  1. Edit the ConfigMap:

    kubectl edit cm sandbox-e2b-scheduler-config -nsandbox-system
  2. Configure the ConfigMap based on the following example:

    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: sandbox-e2b-scheduler-config
      namespace: sandbox-system
    data:
      config: |
        defaultMaxSandboxesPerCluster: 100000
    
        clusters:
        - name: <YOUR-ACK-CLUSTER-ID>
          maxSandboxes: 100000
          endpoint: <YOUR-CLUSTER-SANDBOX-MANAGER-INGRESS-DOMAIN>
          e2bDomain: <YOUR-CLUSTER-SANDBOX-E2B-DOMAIN>
          e2bApiKey: <YOUR-CLUSTER-SANDBOX-E2B-ADMIN-API-KEY>
        - name: <YOUR-ACK-CLUSTER-ID>
          maxSandboxes: 100000
          endpoint: <YOUR-CLUSTER-SANDBOX-MANAGER-INGRESS-DOMAIN>
          e2bDomain: <YOUR-CLUSTER-SANDBOX-E2B-DOMAIN>
          e2bApiKey: <YOUR-CLUSTER-SANDBOX-E2B-ADMIN-API-KEY>
    
        requestTimeout: 5m
        volumeGC:
          grace: 8h
          reconcileInterval: 24h
  3. The following table describes the configuration fields of the sandbox-e2b-scheduler-config ConfigMap.

    Field

    Sub-field

    Required

    Description

    requestTimeout

    -

    No

    Default value: 5m (5 minutes). The request timeout period of the HTTPRoute rules. Adjust the value based on how long your requests take.

    defaultMaxSandboxesPerCluster

    -

    No

    Default value: 100000. The maximum number of Sandboxes that a single cluster is expected to run (excluding the Sandbox replicas in SandboxSets). If maxSandboxes is not configured in clusters, this value applies.

    clusters

    name

    Yes

    The cluster ID, such as c6cb44842ae2242e6bc5b96cea4f3xxxx.

    maxSandboxes

    No

    The maximum number of Sandboxes that the cluster is expected to run. If this field is not configured, the value of defaultMaxSandboxesPerCluster takes effect.

    endpoint

    Yes

    The Ingress endpoint domain of the cluster's Sandbox Manager. For example, if ALB Ingress Controller is used, the value is in the format of alb-xxxx.cn-hangzhou.alb.aliyuncsslb.com.

    e2bDomain

    Yes

    The E2B domain configured when Sandbox Manager was installed in the cluster, such as cluster1.example.com.

    e2bApiKey

    Yes

    The adminApiKey configured when Sandbox Manager was installed in the cluster, such as e2b_0000000000000000000000000000000000000000.

    qpm

    No

    The creation rate limit of the cluster (Sandboxes per minute). If this field is not configured or the value is less than or equal to 0, no rate limit is applied.

    burst

    No

    The maximum number of Sandboxes that can be created in a single (batch) request. If this field is omitted, the default value equals qpm (the quota of one full minute).

    region

    No

    The region to which the cluster belongs. If this field is not configured, Sandbox scheduling is not limited by region. Cross-region scheduling is triggered only when at least two clusters are configured with different values. If you need to distinguish regions, configure this field for each cluster.

    wellKnownCACertificates

    No

    The value Systemindicates that the cluster uses certificates issued by a publicly trusted CA. If self-signed certificates are used, do not configure this field, and create the CA in sandbox-ca-cm.

    volumeGC

    grace

    No

    Default value: 8h. How long a PersistentVolume that is created in an associated cluster during Sandbox scheduling is retained before it is reclaimed.

    reconcileInterval

    No

    The interval at which the fleet scans associated clusters for reclaimable PersistentVolumes. A value of 0 disables the scan. Default value: 24h.

Step 4: Configure the Multi-Cluster Gateway entry point in DNS

After the preceding steps are complete, configure the Gateway entry point IP of multi-cluster Agent Sandbox in DNS. E2B requests can then be initiated through fleet.your.domain.com (the E2B domain).

  1. Obtain the LB IP of the Multi-Cluster Gateway:

    kubectl -n sandbox-system get gateway global-sandbox-scheduler \
      -o jsonpath='{.status.addresses[0].value}'
  2. In DNS or PrivateZone, resolve the wildcard domain of the fleet to the Multi-Cluster Gateway LB IP obtained in the previous step.

    *.fleet.your.domain.com -> <YOUR-MULTI-CLUSTER-GATEWAY-IP>

Step 5: (Optional) Obtain or generate a Fleet Admin API Key

When you access the fleet through fleet.your.domain.com (the E2B domain), the client must include the E2B_API_KEY configured here. When multi-cluster Agent Sandbox is enabled, a default value is generated on the fleet. The default value can also be replaced with a custom key.

  1. Obtain the Fleet Admin API Key that was generated by default:

    kubectl get secret fleet-api-keys -n sandbox-system \
      -o jsonpath='{.data.admin}' | base64 -d ; echo
  2. Update the Fleet Admin API Key (replace <YOUR-KEY>).

    kubectl patch secret fleet-api-keys \
      -p '{"stringData":{"admin":"<YOUR-KEY>"}}' \
      -n sandbox-system

Step 1: Create a Sandbox warm pool by using SandboxSet

Important

The SandboxSet warm pool and the Multi-Cluster Gateway load balancer are billable resources. Sandboxes in the warm pool run continuously, so the warm pool continues to incur charges for as long as it exists. To release the warm pool, delete the SandboxSet from the fleet. For more information about billing, see Agent Sandbox.

  1. Save the following sample to sandbox-set.yaml and then run kubectl apply -f sandbox-set.yaml to create a SandboxSet in the fleet.

    A SandboxSet defines the template of Sandboxes and the desired number of available replicas.
    apiVersion: agents.kruise.io/v1alpha1
    kind: SandboxSet
    metadata:
      name: code-interpreter
    spec:
      persistentContents:
      - filesystem
      replicas: 4
      runtimes:
      - name: csi
      - name: agent-runtime
      scaleStrategy: {}
      template:
        metadata:
          labels:
            alibabacloud.com/acs: "true"
            alibabacloud.com/compute-class: agent-sandbox
            alibabacloud.com/compute-qos: default
        spec:
          automountServiceAccountToken: false
          containers:
          - image: registry-cn-zhangjiakou.ack.aliyuncs.com/acs/code-interpreter:v1.6
            imagePullPolicy: IfNotPresent
            name: sandbox
            resources:
              limits:
                cpu: "1"
                memory: 1Gi
              requests:
                cpu: "1"
                ephemeral-storage: 30Gi
                memory: 1Gi
          restartPolicy: Always
      updateStrategy:
        maxUnavailable: 20%
  2. Save the following sample to code-interpreter-pp.yaml and then run kubectl apply -f code-interpreter-pp.yaml.

    When you create a SandboxSet, also create a PropagationPolicy resource and use the spec.placement.replicaScheduling.replicaSchedulingType field to specify the rule for distributing the SandboxSet to associated clusters. When spec.resourceSelectors.kind is set to SandboxSet, replicaSchedulingTypesupports only the Duplicated option, which means a complete copy of the SandboxSet is created in each associated cluster.

    When preserveResourcesOnDeletion is set to false, deleting the resources on the fleet also deletes the resources on the associated clusters. To keep the resources on the associated clusters, set it to true.
    apiVersion: policy.one.alibabacloud.com/v1alpha1
    kind: PropagationPolicy
    metadata:
      name: code-interpreter-pp
    spec:
      preserveResourcesOnDeletion: false
      resourceSelectors:
      - apiVersion: agents.kruise.io/v1alpha1
        kind: SandboxSet
      placement:
        replicaScheduling:
          replicaSchedulingType: Duplicated
  3. View the aggregated SandboxSet status across all associated clusters in the fleet.

    kubectl get sandboxset

    Expected output:

    NAME               REPLICAS   AVAILABLE   UPDATEDREPLICAS   UPDATEDAVAILABLEREPLICAS   UPDATEREVISION   AGE
    code-interpreter   8          8           8                 8                          74bfb6****       5d
  4. View the SandboxSet status in different associated clusters.

    kubectl amc get sandboxset -M

    Expected output:

    NAME               CLUSTER          CLUSTER_ALIAS     REPLICAS   AVAILABLE   UPDATEDREPLICAS   UPDATEDAVAILABLEREPLICAS   UPDATEREVISION   AGE   ADOPTION
    code-interpreter   <cluster-id-1>   sandbox-cluster1  4          4           4                 4                          56fc44****       5d    Y
    code-interpreter   <cluster-id-2>   sandbox-cluster2  4          4           4                 4                          74bfb6****       5d    Y
  5. (Optional) Use OverridePolicy to specify different SandboxSet replica counts and resource specifications for different clusters.

    The following sample configures different replica counts and specifications in two clusters for the SandboxSet named code-interpreter:

    • CLUSTER_1_ID: 5 replicas, with 1 CPU and 2 GiB of memory.

    • CLUSTER_2_ID: 2 replicas, with 2 CPUs and 4 GiB of memory.

    Save the following sample to code-interpreter-op.yaml and then run kubectl apply -f code-interpreter-op.yaml.

    OverridePolicy uses JSON Patch paths to locate the fields to override. /spec/replicas overrides the replica count configuration, and /spec/template/spec/containers/0/resources overrides the container resource configuration.
    apiVersion: policy.one.alibabacloud.com/v1alpha1
    kind: OverridePolicy
    metadata:
      name: code-interpreter-op
    spec:
      resourceSelectors:
        - apiVersion: agents.kruise.io/v1alpha1
          kind: SandboxSet
          name: code-interpreter
      overrideRules:
        - targetCluster:
            clusterNames:
              - CLUSTER_1_ID
          overriders:
            plaintext:
              - operator: replace
                path: /spec/replicas
                value: 5
              - path: /spec/template/spec/containers/0/resources
                operator: add
                value:
                  requests:
                    cpu: "1"
                    memory: "2Gi"
                  limits:
                    cpu: "1"
                    memory: "2Gi"
        - targetCluster:
            clusterNames:
              - CLUSTER_2_ID
          overriders:
            plaintext:
              - operator: replace
                path: /spec/replicas
                value: 2
              - path: /spec/template/spec/containers/0/resources
                operator: add
                value:
                  requests:
                    cpu: "2"
                    memory: "4Gi"
                  limits:
                    cpu: "2"
                    memory: "4Gi"

Step 2: Use Agent Sandbox

After the Sandbox warm pool is created, use Agent Sandbox in either of the following ways:

  • SandboxClaim method: Request Sandbox instances from the warm pool and manage them in a Kubernetes-native way. Use this method for in-cluster workflows driven by kubectl or CRDs.

  • E2B SDK method: Use the E2B SDK to create and use Sandboxes through the Multi-Cluster Gateway of the fleet. Use this method when you develop AI agent applications.

E2B SDK method

Agent Sandbox supports integration with the E2B ecosystem. After the E2B capabilities of multi-cluster Agent Sandbox are enabled, the fleet provides a unified Multi-Cluster Gateway entry point and multi-cluster scheduling capabilities. Clients can create, connect to, and use Sandboxes through the E2B SDK without needing to know which backend cluster runs a Sandbox.

Obtain an Agent Sandbox

Through the E2B SDK, develop and deploy projects in a Sandbox and view the results, including running code (run_code), reading and writing files (files.write and files.read), and running commands (commands.run). For more information about how to use the SDK, see Use the E2B SDK to connect to Agent Sandbox Container Edition.

Before you use the E2B capabilities, complete all configurations in(Optional) Configure the E2B capabilities of multi-cluster Agent Sandbox and Step 1: Create a Sandbox warm pool by using SandboxSet.
  1. Install Python in your local environment.

  2. Install the E2B Python SDK. Only versions earlier than v2.25.0 are supported.

    pip install "e2b-code-interpreter==2.7.0" "e2b==2.24.0"
  3. Configure environment variables.

    # fleet.your.domain.com is the fleet E2B domain, which resolves to the Multi-Cluster Gateway. It does not require the * prefix. Modify it based on your actual configuration.
    export E2B_DOMAIN=fleet.your.domain.com
    # Use the default generated or updated Fleet Admin API Key
    export E2B_API_KEY=e2b_0000000000000000000000000000000000000000
  4. Save the following code as the main.py file.

    # Import the E2B SDK
    from e2b_code_interpreter import Sandbox
    
    sbx: Sandbox = Sandbox.create(template="code-interpreter")
    print(f"sandbox id: {sbx.sandbox_id}")
    
    result = sbx.run_code("print('hello, world')")
    print(f"run code result: {result}")
    
    text = input("enter some text to be saved to file 'text.txt' inside sandbox:  ")
    sbx.files.write("text.txt", text)
    print(f"read file from sandbox via files api: [{sbx.files.read('text.txt')}]")
    print(f"read file from sandbox via commands api: [{sbx.commands.run('cat text.txt')}]")
    
    input("press ENTER to kill the sandbox")
    print(sbx.kill())
    For more information about the metadata parameter of the create operation, see Create an Agent Sandbox Container Edition.
  5. Run the main.py file to create and verify a Sandbox.

    After the first prompt appears, enter text such as acs agent sandbox and press ENTER. The text is written to the /home/user/text.txt file of the pod named code-interpreter-29***. Press ENTER again to delete the current Sandbox.

    python main.py

    Expected output:

    sandbox id: default--code-interpreter-29***
    run code result: Execution(Results: [], Logs: Logs(stdout: ['hello, world\n'], stderr: []), Error: None)
    enter some text to be saved to file 'text.txt' inside sandbox:  acs agent sandbox
    read file from sandbox via files api: [acs agent sandbox]
    read file from sandbox via commands api: [CommandResult(stderr='', stdout='acs agent sandbox', exit_code=0, error='')]
    press ENTER to kill the sandbox
    True

SandboxClaim method

Distribute and schedule SandboxClaims across multiple clusters

A SandboxClaim is used to request Sandbox instances from an existing SandboxSet. Use a PropagationPolicy to control how SandboxClaims are scheduled across multiple clusters.

  1. Save the following sample to sandbox-claim.yaml and then run kubectl apply -f sandbox-claim.yaml.

    apiVersion: agents.kruise.io/v1alpha1
    kind: SandboxClaim
    metadata:
      name: code-interpreter
      namespace: default
    spec:
      templateName: code-interpreter
      replicas: 1
      claimTimeout: 5m
      ttlAfterCompleted: 5m
  2. Choose a scheduling mode based on your requirements:

    Important

    Do not configure different PropagationPolicies for the same SandboxClaim resource.

    Basic scheduling (Gang scheduling): automatically selects one associated cluster and schedules all replicas requested by the SandboxClaim to that cluster. Perform the following operations:

    1. Save the following sample to sandboxclaim-pp.yaml and then run kubectl apply -f sandboxclaim-pp.yaml.

      apiVersion: policy.one.alibabacloud.com/v1alpha1
      kind: PropagationPolicy
      metadata:
        name: sandboxclaim-pp
      spec:
        preserveResourcesOnDeletion: false
        resourceSelectors:
        - apiVersion: agents.kruise.io/v1alpha1
          kind: SandboxClaim
        placement:
          replicaScheduling:
            replicaSchedulingType: Divided
            customSchedulingType: Gang
    2. View the SandboxClaim on the fleet.

      kubectl get sandboxclaim

      Expected output:

      NAME               PHASE       TEMPLATE           DESIRED   CLAIMED   AGE
      code-interpreter   Completed   code-interpreter   1         1         6s
    3. View the SandboxClaim on the associated cluster.

      kubectl amc get sandboxclaim -M

      Expected output:

      NAME               CLUSTER        CLUSTER_ALIAS    PHASE       TEMPLATE           DESIRED   CLAIMED   AGE   ADOPTION
      code-interpreter   <CLUSTER-ID>   sandbox-cluster1 Completed   code-interpreter   1         1         3s    Y
    4. Replace CLUSTER_ID in the following command with the cluster ID obtained in the previous step and then run the command to get Sandbox details by using the SandboxClaim name label.

      kubectl amc get sandbox -m CLUSTER_ID -l agents.kruise.io/claim-name=code-interpreter

      Expected output:

      NAME                     CLUSTER      CLUSTER_ALIAS    STATUS    AGE   CLAIMED   SHUTDOWN_TIME   PAUSE_TIME   MESSAGE   ADOPTION
      code-interpreter-47***   CLUSTER_ID   sandbox-cluster1 Running   59m   true

    Cluster priority scheduling: sets scheduling priorities for clusters. The scheduler selects the cluster with a higher priority and schedules all replicas requested by the SandboxClaim to that cluster. When the resources of the high-priority cluster are insufficient, it automatically falls back to a lower-priority cluster. Perform the following operations:

    1. Save the following sample to sandboxclaim-pp-priority.yaml and then run kubectl apply -f sandboxclaim-pp-priority.yaml.

      The order declared in the spec.placement.clusterAffinities field is the scheduling priority order. The PropagationPolicy in the following sample schedules to CLUSTER_1_ID (primary) first, and schedules to CLUSTER_2_ID (secondary) when resources are insufficient.
      apiVersion: policy.one.alibabacloud.com/v1alpha1
      kind: PropagationPolicy
      metadata:
        name: sandboxclaim-pp-priority
      spec:
        preserveResourcesOnDeletion: false
        resourceSelectors:
        - apiVersion: agents.kruise.io/v1alpha1
          kind: SandboxClaim
        placement:
          replicaScheduling:
            customSchedulingType: Gang
            replicaSchedulingType: Divided
          clusterAffinities:
            - affinityName: primary
              clusterNames:
                - CLUSTER_1_ID
            - affinityName: secondary
              clusterNames:
                - CLUSTER_2_ID
    2. View the SandboxClaim on the fleet.

      kubectl get sandboxclaim

      Expected output:

      NAME               PHASE       TEMPLATE           DESIRED   CLAIMED   AGE
      code-interpreter   Completed   code-interpreter   1         1         6s
    3. Verify that the SandboxClaim is deployed to CLUSTER_1_ID according to the priority rules.

      kubectl amc get sandboxclaim -M

      Expected output:

      NAME               CLUSTER          CLUSTER_ALIAS    PHASE       TEMPLATE           DESIRED   CLAIMED   AGE   ADOPTION
      code-interpreter   <CLUSTER-1-ID>   sandbox-cluster1 Completed   code-interpreter   1         1         3s    Y
    4. Replace CLUSTER_ID in the following command with the cluster ID obtained in the previous step and then run the command to get Sandbox details by using the SandboxClaim name label.

      kubectl amc get sandbox -m CLUSTER_ID -l agents.kruise.io/claim-name=code-interpreter

      Expected output:

      NAME                     CLUSTER      CLUSTER_ALIAS    STATUS    AGE   CLAIMED   SHUTDOWN_TIME   PAUSE_TIME   MESSAGE   ADOPTION
      code-interpreter-47***   CLUSTER_ID   sandbox-cluster1 Running   59m   true

Step 3: (Optional) Configure multi-cluster failover

The SandboxClaim link and the E2B link use different multi-cluster failover mechanisms. Configure failover based on the tab that corresponds to the method you use.

E2B link

For E2B scenarios, the ACK One fleet supports the following two methods to determine whether a cluster has failed:

  • Automatic probing: Automatically probes the health status of the Sandbox Manager add-on in each associated cluster.

  • Manual configuration: Configure disabled: true for a cluster in sandbox-system/sandbox-e2b-scheduler-config, which drains the cluster for maintenance.

Failure detection method

Description

Determination rules

Effects

Automatic probing

Periodically probes the health status of the Sandbox Manager add-on in each associated cluster at an interval of 5 seconds.

  • If three consecutive probes fail, the cluster is marked as unhealthy.

  • If two consecutive probes succeed, the cluster is marked as healthy.

When a cluster is marked as unhealthy, the following operations performed through the E2B SDK are affected:

  • During create, the cluster is filtered out, and Sandboxes are automatically scheduled to a healthy cluster. If no healthy cluster is available, the message scheduling failed: no cluster available after filtering is returned.

  • Operations such as connect and pauseare intercepted by the Multi-Cluster Gateway of the fleet, which returns HTTP 503. List operations are not supported.

Manual configuration

Manually configure disabled: true in the sandbox-e2b-scheduler-config ConfigMap. This drains the cluster for maintenance.

disabled: true: The value indicates that the cluster is under maintenance and no new Sandboxes are created on it.

When a cluster is configured with disabled: true:

  • During create, the cluster is filtered out, and Sandboxes are automatically scheduled to a healthy cluster. If no healthy cluster is available, the message scheduling failed: no cluster available after filtering is returned.

  • Operations such as connect and pause continue to work.

To manually drain a cluster, add disabled: true to the configuration of the cluster in the sandbox-e2b-scheduler-config ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: sandbox-e2b-scheduler-config
  namespace: sandbox-system
data:
  config: |
    ...
    # Fields per cluster:
    #   disabled — bool, optional — drain the cluster (exclude from create scheduling)
    clusters:
    - name: $clusterid
      disabled: true
      ...

To manually restore a cluster, change the configuration of the cluster to disabled: false.

SandboxClaim link

The ACK One fleet uses Open Cluster Management (OCM) to monitor network connectivity between the associated clusters and the fleet. If the fleet does not receive a heartbeat from the OCM agent in an associated cluster for 5 minutes, the fleet automatically adds the NoSelect taint to that cluster. SandboxClaims are then moved to other available associated clusters.

To perform maintenance on an associated cluster, manually add a taint to that cluster:

Adding or removing taints does not affect the Sandbox instances that are already running on the cluster.
  1. Replace CLUSTER_ID in the following command with the ID of the target associated cluster, and then run the command to add a taint to that cluster. The scheduler then stops assigning SandboxClaims to the tainted cluster:

    kubectl patch managedcluster CLUSTER_ID --type=merge -p '{
      "spec": {
        "taints": [
          {"key": "maintenance", "value": "true", "effect": "NoSelect"}
        ]
      }
    }'
  2. After the cluster maintenance is complete, remove the taint to restore scheduling on the associated cluster:

    kubectl patch managedcluster CLUSTER_ID --type=json -p '[{"op":"remove","path":"/spec/taints"}]'