Compute Nest:Quickly deploy Qwen3 series models in Compute Nest

Query model deployment parameters

1. On the Logs tab, find ALIYUN::ECS::RunCommand in the Resource Type column, copy and click the Associated ID to go to the ECS Cloud Assistant page.

   

2. On the Command Execution Result tab of the ECS Cloud Assistant page, paste the associated ID and click the search icon.

   

3. Click View in the Actions column. On the Execution Information tab, view model deployment parameters in the Command Content section.

   

Deploy models with custom parameters

To deploy models by specifying custom parameters, perform the following steps to modify and redeploy the service instance:

1. On the Resources tab, click Remote Connection to connect to the ECS instance. In the dialog boxt that appears, click Password-free Logon to log on to the ECS instance.

   

2. Stop the model service.

   Warning

   Stopping the service causes business interruptions. We recommend that you perform this operation during non-peak business hours.

   sudo docker stop vllm
   sudo docker rm vllm

3. Obtain, modify, and run the model deployment command.

   In this example, reference scripts for virtual large language model (vLLM) and SGlang are provided. You can refer to the comments to modify the scripts that you want to run.

   Note

   Redeployment takes about 10 minutes.

   vLLM

   sudo docker run -d -t --net=host \
    --gpus all \ # Allow the container to access all available GPU devices
    --entrypoint /bin/bash \
    --privileged \
    --ipc=host \
    --name vllm \ # Give the container an easy-to-recognize name vllm
    -v /root:/root \ Mount the host's /root directory to the container's /root for data sharing
    egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.7.2-pytorch2.5.1-cuda12.4-ubuntu22.04 \
    -c "pip install --upgrade vllm==0.8.2 && # Customize version, such as pip install vllm==0.7.1
    export GLOO_SOCKET_IFNAME=eth0 && # Environment variable required for VPC network communication, do not delete or modify
    export NCCL_SOCKET_IFNAME=eth0 && # Environment variable required for VPC network communication, do not delete or modify
    vllm serve /root/llm-model/${ModelName} \  # Use service to start the model
    --served-model-name ${ModelName} \  # Specify the model name used in the service
    --gpu-memory-utilization 0.98 \ # GPU utilization rate, too high may cause OOM in other processes. Valid values: 0 and 1.
    --max-model-len ${MaxModelLen} \ # Maximum model length, value range depends on the model itself.
    --enable-chunked-prefill \
    --host=0.0.0.0 \
    --port 8080 \
    --trust-remote-code \
    --api-key "${VLLM_API_KEY}" \ # Optional, set API key, can be removed if not needed.
    --tensor-parallel-size $(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l | awk '{print $1}')" # Number of GPUs to use, default is all GPUs.

   SGlang

    #Download public image containing SGlang
    sudo docker pull egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.7.2-sglang0.4.3.post2-pytorch2.5-cuda12.4-20250224
   
    sudo docker run -d -t --net=host \
    --gpus all \ # Allow the container to access all available GPU devices
    --entrypoint /bin/bash \
    --privileged \
    --ipc=host \
    --name llm-server \
    -v /root:/root \
    egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm:0.7.2-sglang0.4.3.post2-pytorch2.5-cuda12.4-20250224 \ 
    -c "pip install sglang==0.4.3 && # Customize version
    export GLOO_SOCKET_IFNAME=eth0 && # Environment variable required for VPC network communication, do not delete or modify
    export NCCL_SOCKET_IFNAME=eth0 && # Environment variable required for VPC network communication, do not delete or modify
    python3 -m sglang.launch_server \
    --model-path /root/llm-model/${ModelName} \ # Use service to start the model
    --served-model-name ${ModelName} \ # Specify the model name used in the service
    --tp $(nvidia-smi --query-gpu=index --format=csv,noheader | wc -l | awk '{print $1}')" \ # Number of GPUs to use, default is all GPUs.
    --trust-remote-code \
    --host 0.0.0.0 \
    --port 8080 \
    --mem-fraction-static 0.9 # GPU utilization rate, too high may cause OOM in other processes. Valid values: 0 and 1.

   Sample successful request.

   

4. Check whether the model service runs as expected. If the result as shown in the following figure is returned, the model service is successfully redeployed.

   sudo docker ps
   sudo docker logs vllm  

   

   

Performance test examples