Deploy Pods on ACS via ACK One Virtual Nodes - Container Service for Kubernetes

How it works

ACS uses a layered architecture that separates Kubernetes control from container compute. Kubernetes manages business workloads — Deployments, Services, StatefulSets, and CronJobs — while the ACS compute layer handles pod scheduling and resource allocation.

Virtual nodes bridge the two layers. After you install the ack-virtual-node add-on in your registered cluster, virtual nodes appear as schedulable nodes. Pods scheduled to these virtual nodes run as ACS instances — each pod maps to one ACS instance — in secure, isolated container runtime environments. The pods communicate over the network with pods running on regular nodes in the same cluster.

This architecture suits long-running workloads with elastic traffic. Schedule those workloads to virtual nodes during peak periods, then release the pods when traffic subsides to reduce costs. Because ACS manages the underlying capacity, you never need to plan node compute capacity in advance.

Considerations

Before you start, note the following behaviors and constraints:

Default compute type: If you schedule a pod to a virtual node without specifying ACS as the compute type, Elastic Container Instance (ECI) instances are used by default.
GPU availability: ACS GPU and GPU HPN (High-Performance Networking) compute capacity is in invitational preview. Submit a ticketSubmit a ticket to request access before attempting GPU workloads.
GPU HPN capacity reservation: ACS GPU HPN requires a pre-purchased GPU-HPN Capacity Reservation associated with your cluster.
GPU HPN resource requests: GPU HPN pods do not declare GPU resources in their requests and limits fields. Use the actual device model name (such as nvidia.com/gpu) instead.
GPU HPN scheduling: GPU HPN nodes only schedule pods with the gpu-hpn compute class label.

Compute type comparison

Choose the ACS compute type that fits your workload:

Compute type	Compute class label	Availability	Capacity reservation required
CPU	`general-purpose`	Generally available	No
GPU	`gpu`	Invitational preview	No
GPU HPN	`gpu-hpn`	Invitational preview	Yes

Prerequisites

Before you begin, make sure you have:

An ACK One registered cluster connected to a Kubernetes cluster in a data center or third-party cloud. Kubernetes 1.24 or later is recommended. For setup instructions, see Create an ACK One registered cluster.
The ACK Virtual Node add-on version 2.13.0 or later installed. For installation instructions, see Install the ack-virtual-node add-on.

Grant RAM permissions to the ack-virtual-node add-on

The ack-virtual-node add-on needs Resource Access Management (RAM) permissions to create and manage ACS instances on your behalf. Choose one of the following methods.

Option 1: Use onectl (recommended)

Install onectl on your on-premises machine. For instructions, see Use onectl to manage registered clusters.

Run the following command to grant RAM permissions to the add-on.

onectl ram-user grant --addon ack-virtual-node

Expected output:

Ram policy ack-one-registered-cluster-policy-ack-virtual-node granted to ram user ack-one-user-ce313528c3 successfully.

Option 2: Use the console

Create a RAM user.

Grant the RAM user permissions. Attach one of the following to the RAM user:

The system policies AliyunECIFullAccess, AliyunVPCReadOnlyAccess, and AliyunAccFullAccess
A custom policy with the following content

Expand to view the custom policy template

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "vpc:DescribeVSwitches",
                "vpc:DescribeVpcs"
            ],
            "Resource": "*",
            "Effect": "Allow"
        },
        {
            "Action": [
                "eci:CreateContainerGroup",
                "eci:DeleteContainerGroup",
                "eci:DescribeContainerGroups",
                "eci:DescribeContainerGroupStatus",
                "eci:DescribeContainerGroupEvents",
                "eci:DescribeContainerLog",
                "eci:UpdateContainerGroup",
                "eci:UpdateContainerGroupByTemplate",
                "eci:CreateContainerGroupFromTemplate",
                "eci:RestartContainerGroup",
                "eci:ExportContainerGroupTemplate",
                "eci:DescribeContainerGroupMetric",
                "eci:DescribeMultiContainerGroupMetric",
                "eci:ExecContainerCommand",
                "eci:CreateImageCache",
                "eci:DescribeImageCaches",
                "eci:DeleteImageCache",
                "eci:DescribeContainerGroupMetaInfos",
                "eci:UpdateImageCache",
                "eci:RestartContainer",
                "eci:RestartContainers"
            ],
            "Resource": [
                "*"
            ],
            "Effect": "Allow"
        },
        {
            "Action": [
                "acc:RecommendZones",
                "acc:DescribeZones",
                "acc:CreateInstance",
                "acc:UpdateInstance",
                "acc:DeleteInstance",
                "acc:RestartInstance",
                "acc:DescribeInstances",
                "acc:DescribeInstanceStatus",
                "acc:DescribeInstanceEvents",
                "acc:DescribeInstanceDetail",
                "acc:DescribeMultiInstanceMetric",
                "acc:DescribeContainerLog",
                "acc:ResizeInstanceVolume",
                "acc:CreateCustomResource",
                "acc:UpdateCustomResource",
                "acc:DeleteCustomResource",
                "acc:DescribeCustomResources",
                "acc:DescribeCustomResourceDetail",
                "acc:DescribeReservationMetrics"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

Grant permissions to the RAM user.
Create an AccessKey for the RAM user.

Warning
Configure a network access restriction policy for the AccessKey to restrict calls to trusted networks.
Create a secret named alibaba-addon-secret in the registered cluster using the AccessKey.
```
kubectl -n kube-system create secret generic alibaba-addon-secret \
  --from-literal='access-key-id=<your-access-key-id>' \
  --from-literal='access-key-secret=<your-access-key-secret>'
```
Replace <your-access-key-id> and <your-access-key-secret> with the AccessKey ID and AccessKey secret you created in the previous step. After the ack-virtual-node add-on is installed, it automatically uses this secret to access cloud resources.

Deploy ACS CPU workloads

After you install or upgrade the ACK Virtual Node add-on to version 2.13.0 or later, it supports both ACS and ECI compute capacity.

Step 1: Update the security group

Add inbound rules to the control plane security group of your registered cluster so that the cluster nodes can reach the ACS endpoints.

On the cluster Basic Information page, click the security group ID next to Control Plane Security Group.

On the security group details page, click Add Rule and add the following three rules.

Rule type	Protocol	Port range	Source CIDR block	Description
Inbound	TCP	80	CIDR block of the IDC cluster, for example, 192.168.1.0/24	Configure ACS endpoints
Inbound	TCP	443	CIDR block of the IDC cluster, for example, 192.168.1.0/24	Configure ACS endpoints
Inbound	TCP	10250	CIDR block of the IDC cluster, for example, 192.168.1.0/24	Serverless kubelet service

Step 2: Deploy the workload

Create a file named nginx.yaml with the following content. The three alibabacloud.com labels in template.metadata.labels route the pod to ACS compute: acs: "true" enables ACS, compute-class sets the resource type, and compute-qos sets the quality of service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
        alibabacloud.com/acs: "true"                    # Enable ACS compute
        alibabacloud.com/compute-class: general-purpose  # ACS compute class
        alibabacloud.com/compute-qos: default            # ACS compute QoS
    spec:
      containers:
      - name: nginx
        image: mirrors-ssl.aliyuncs.com/nginx:stable-alpine
        ports:
          - containerPort: 80
            protocol: TCP
        resources:
          limits:
            cpu: 2
          requests:
            cpu: 2

Apply the manifest.
```
kubectl apply -f nginx.yaml
```

Verify the deployment

Check that both pods are running on virtual nodes.

kubectl get pods -o wide

Expected output:

NAME                     READY   STATUS    RESTARTS   AGE     IP               NODE                            NOMINATED NODE   READINESS GATES
nginx-54bcbc9b66-****   1/1     Running   0          3m29s   192.168.XX.XXX   virtual-kubelet-cn-shanghai-l   <none>           <none>
nginx-54bcbc9b66-****   1/1     Running   0          3m29s   192.168.XX.XXX   virtual-kubelet-cn-shanghai-l   <none>           <none>

The NODE column shows virtual-kubelet-*, confirming the pods are running on virtual nodes.

Confirm the pods are ACS instances.

kubectl describe pod nginx-54bcbc9b66-****

Expected output (key fields):

Annotations:  ProviderCreate: done
              alibabacloud.com/instance-id: acs-uf6008giwgjxlvn*****
              alibabacloud.com/pod-ephemeral-storage: 30Gi
              alibabacloud.com/pod-use-spec: 2-2Gi
              kubernetes.io/pod-stream-port: 10250
              network.alibabacloud.com/enable-dns-cache: false
              topology.kubernetes.io/region: cn-shanghai

The annotation alibabacloud.com/instance-id: acs-uf6008giwgjxlvn***** confirms the pod is running as an ACS instance.

Deploy ACS GPU workloads

ACS GPU compute capacity is in invitational preview. Submit a ticketSubmit a ticket to request access.

The setup for GPU workloads follows the same security group and permission steps as CPU workloads. The difference is in the pod labels and resource requests.

Use the following labels in template.metadata.labels to request GPU compute:

alibabacloud.com/acs: "true"
alibabacloud.com/compute-class: gpu
alibabacloud.com/compute-qos: default
alibabacloud.com/gpu-model-series: GN8IS  # GPU model. Replace with your actual model.

For supported compute classes and QoS types, see Mapping between compute classes and QoS types. For supported GPU models for gpu-model-series, see Specify GPU model and driver version for ACS GPU pods.

Deploy a GPU workload

Create a GPU Deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dep-node-selector-demo
  labels:
    app: node-selector-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-selector-demo
  template:
    metadata:
      labels:
        app: node-selector-demo
        alibabacloud.com/acs: "true"
        alibabacloud.com/compute-class: gpu
        alibabacloud.com/compute-qos: default
        alibabacloud.com/gpu-model-series: example-model  # Replace with your actual GPU model, for example, T4
    spec:
      containers:
      - name: node-selector-demo
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/stress:v1.0.4
        command:
        - "sleep"
        - "1000h"
        resources:
          limits:
            cpu: 1
            memory: 1Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: 1
            memory: 1Gi
            nvidia.com/gpu: "1"

Check the GPU workload status.

kubectl get pod node-selector-demo-9cdf7bbf9-s**** -oyaml

Expected output (key fields):

    phase: Running

    resources:
      limits:
        #other resources
        nvidia.com/gpu: "1"
      requests:
        #other resources
        nvidia.com/gpu: "1"

Deploy ACS GPU HPN workloads

ACS GPU HPN compute capacity is in invitational preview. Submit a ticketSubmit a ticket to request access.

GPU HPN workloads have two additional prerequisites beyond the standard setup:

Purchase GPU-HPN Capacity Reservation and associate it with your cluster.
Upgrade the ACK Virtual Node component to a version that supports GPU HPN (request access via ticket).

Use the following labels in template.metadata.labels for GPU HPN compute:

alibabacloud.com/compute-class: gpu-hpn  # Fixed value for GPU HPN
alibabacloud.com/compute-qos: default
alibabacloud.com/acs: "true"

For supported compute classes and QoS types, see Mapping between compute classes and QoS types. For other ACS pod configuration options, see ACS pods.

Deploy a GPU HPN workload

GPU HPN nodes only schedule pods with the gpu-hpn compute class. These pods do not declare GPU resources in their resource requests — use the actual device model name in the requests and limits fields instead.

Important

Pay attention to the following field configurations:

Set alibabacloud.com/compute-class: gpu-hpn.
Set the reserved node label: alibabacloud.com/node-type: reserved.
In requests and limits, use the actual device model name, such as nvidia.com/gpu.

Create a GPU HPN Deployment using nodeSelector to target reserved GPU HPN nodes.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: dep-node-selector-demo
  labels:
    app: node-selector-demo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: node-selector-demo
  template:
    metadata:
      labels:
        app: node-selector-demo
        alibabacloud.com/compute-class: gpu-hpn
        alibabacloud.com/compute-qos: default
        alibabacloud.com/acs: "true"
    spec:
      nodeSelector:
        alibabacloud.com/node-type: reserved  # Target reserved GPU HPN nodes
      containers:
      - name: node-selector-demo
        image: registry-cn-hangzhou.ack.aliyuncs.com/acs/stress:v1.0.4
        command:
        - "sleep"
        - "1000h"
        resources:
          limits:
            cpu: 1
            memory: 1Gi
            nvidia.com/gpu: "1"  # Replace with the actual device model name
          requests:
            cpu: 1
            memory: 1Gi
            nvidia.com/gpu: "1"  # Replace with the actual device model name

Check the GPU HPN workload status.

kubectl get pod node-selector-demo-9cdf7bbf9-s**** -oyaml

Expected output (key fields):

    phase: Running

    resources:
      limits:
        #other resources
        nvidia.com/gpu: "1"
      requests:
        #other resources
        nvidia.com/gpu: "1"

What's next

To configure ACS pod resource limits, QoS settings, and other options, see ACS pods.
To understand the mapping between compute classes and QoS types, see Mapping between compute classes and QoS types.
For an overview of ACK One registered clusters, see Overview of registered clusters.