ENS:Deploy QwQ-32B reasoning model on edge cloud

Option A: Use the ENS console

1. Log on to the ENS console.

2. In the left-side navigation pane, choose Resources and Images > Instances.

3. On the Instances page, click Create Instance. For more information, see Create an instance.

4. Configure the instance with the following settings.

   Step	Parameter	Recommended value
   Basic Configurations	Billing Method	Subscription
   	Instance type	x86 Computing
   	Instance Specification	NVIDIA 48GB * 2. For detailed specifications, contact the customer manager.
   	Image	Ubuntu, ubuntu_22_04_x64_20G_alibase_20240926
   Network and Storage	Network	Self-built Network
   	System Disk	Ultra Disk, 80 GB or more
   	Data Disk	Ultra Disk, 1 TB or more
   System Settings	Set Password	Custom Key or Key Pair

5. Click Confirm Order in the lower-right corner. Complete the payment.

After payment, the page redirects to the ENS console. The instance is ready when its status changes to Running.

Option B: Call the RunInstances API

Call the RunInstances operation in OpenAPI Portal. The following JSON shows reference parameters.

{
  "InstanceType": "ens.gnxxxx",                    // Instance type
  "InstanceChargeType": "PrePaid",
  "ImageId": "ubuntu_22_04_x64_20G_alibase_20240926",
  "ScheduleAreaLevel": "Region",
  "EnsRegionId": "cn-your-ens-region",             // Edge node
  "Password": "<YOURPASSWORD>",                    // Password
  "InternetChargeType": "95BandwidthByMonth",
  "SystemDisk": {
    "Size": 80,
    "Category": "cloud_efficiency"
  },
  "DataDisk": [
    {
      "Category": "cloud_efficiency",
      "Size": 1024
    }
  ],
  "InternetMaxBandwidthOut": 5000,
  "Amount": 1,
  "NetWorkId": "n-xxxxxxxxxxxxxxx",
  "VSwitchId": "vsw-xxxxxxxxxxxxxxx",
  "InstanceName": "test",
  "HostName": "test",
  "PublicIpIdentification": true,
  "InstanceChargeStrategy": "instance"             // Billing based on instance
}

Log on to the instance

For more information, see Connect to an instance.

Expand the root partition

After creating or resizing an instance, expand the root partition online without restarting.

# Install the cloud environment toolkit.
sudo apt-get update
sudo apt-get install -y cloud-guest-utils

# Make sure the GPT partitioning tool sgdisk exists.
type sgdisk || sudo apt-get install -y gdisk

# Expand the physical partition.
sudo LC_ALL=en_US.UTF-8 growpart /dev/vda 3

# Resize the file system.
sudo resize2fs /dev/vda3

# Verify the result.
df -h

Mount the data disk

Format and mount the data disk. Adjust the commands based on your environment.

# Identify the new disk.
lsblk

# Format the disk without partitioning it.
sudo mkfs -t ext4 /dev/vdb

# Configure the mount.
sudo mkdir /data
echo "UUID=$(sudo blkid -s UUID -o value /dev/vdb) /data ext4 defaults,nofail 0 0" | sudo tee -a /etc/fstab

# Verify the mount.
sudo mount -a
df -hT /data

# Modify permissions.
sudo chown $USER:$USER /data

Install CUDA

For more information, see CUDA Toolkit 12.4 Downloads.

# Download the CUDA Toolkit.
wget https://developer.download.nvidia.com/compute/cuda/12.4.0/local_installers/cuda_12.4.0_570.124.06_linux.run
chmod +x cuda_12.4.0_570.124.06_linux.run

# Run the installer. This step takes several minutes and requires interactive input.
sudo sh cuda_12.4.0_570.124.06_linux.run

# Add environment variables to ~/.bashrc.
export PATH="$PATH:/usr/local/cuda-12.8/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-12.8/lib64"
source ~/.bashrc

# Verify the installation.
nvcc -V
nvidia-smi

Install auxiliary software (optional)

uv is a Python virtual environment and dependency management tool, useful for clusters that run multiple models. For more information, see Installing uv.

# Install uv. By default, uv installs to ~/.local/bin/.
curl -LsSf https://astral.sh/uv/install.sh | sh

# Add uv to PATH in ~/.bashrc.
export PATH="$PATH:~/.local/bin"
source ~/.bashrc

# Create a virtual environment.
uv venv myenv --python 3.12 --seed
source myenv/bin/activate

If the CUDA environment variables become invalid after activating the virtual environment (for example, nvcc or nvidia-smi cannot be found), add the CUDA paths to the activation script:

# Edit myenv/bin/activate and add the following lines after the existing export PATH statement:
export PATH="$PATH:/usr/local/cuda-12.8/bin"
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/local/cuda-12.8/lib64"

Install the required packages:

# Install vLLM and ModelScope.
uv pip install vllm==0.7.2
uv pip install modelscope

# Install the GPU monitoring tool (optional). You can also use nvidia-smi.
uv pip install nvitop

Download the QwQ-32B model and benchmark script

# Download the model to /data to avoid running out of space on the system disk.
mkdir -p /data/Qwen/QwQ-32B
cd /data/Qwen/QwQ-32B
modelscope download --model Qwen/QwQ-32B --local_dir .

# Optional: download the dataset.
wget https://www.modelscope.cn/datasets/gliang1001/ShareGPT_V3_unfiltered_cleaned_split/resolve/master/ShareGPT_V3_unfiltered_cleaned_split.json

# Install git if needed.
apt update
apt install git -y

# Clone the vLLM repository for benchmark scripts.
git clone https://github.com/vllm-project/vllm.git

Start the vLLM server

vllm serve /data/Qwen/QwQ-32B/ \
  --host 127.0.0.1 \
  --port 8080 \
  --tensor-parallel-size 2 \
  --trust-remote-code \
  --served-model-name qw \
  --gpu-memory-utilization 0.95 \
  --enforce-eager \
  --max-num-batched-tokens 8192 \
  --max-model-len 8192 \
  --enable-prefix-caching

Run the benchmark

python3 ./vllm/benchmarks/benchmark_serving.py \
  --backend vllm \
  --served-model-name qw \
  --model /data/Qwen/QwQ-32B \
  --dataset-name random \
  --random-input 1024 \
  --random-output 4096 \
  --random-range-ratio 1 \
  --max-concurrency 4 \
  --num-prompts 10 \
  --host 127.0.0.1 \
  --port 8080 \
  --save-result \
  --result-dir /data/logs/ \
  --result-filename QwQ-32B-4-1-4.log

Review the results

The following image shows example benchmark output.