inference-nv-pytorch 26.07 - Container Compute Service - Alibaba Cloud Documentation Center

Main features and bug fixes

Main features

The vLLM image is upgraded to vLLM v0.25.1.
The SGLang image is upgraded to SGLang v0.5.15.post1.
The vLLM and SGLang images support the amd64 and aarch64 architectures.

Bug fixes

None

Image name	inference-nv-pytorch
Tag	26.07-vllm0.25.1-pytorch2.11-cu130-20260723-serverless		26.07-sglang0.5.15.post1-pytorch2.11-cu130-20260715-serverless
Supported architectures	amd64	aarch64	amd64	aarch64
Scenario	Large model inference	Large model inference	Large model inference	Large model inference
Framework	pytorch	pytorch	pytorch	pytorch
Requirements	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580
System components	Ubuntu 24.04.3 LTS Python 3.12.7 CUDA 13.0 NCCL 2.30.7-1 torch 2.11.0 torchvision 0.26.0 torchaudio 2.11.0 triton 3.7.1 Transformers 5.14.1 deep_ep 2.1.0+dd758ca flash_attn 2.8.3 flash_attn_3 3.0.0 flashinfer-python 0.6.13 flashinfer-cubin 0.6.13 flash_mla 1.0.0+b7643bd mooncake 0.3.11.post1 nvshmem 3.7.1 xgrammar 0.2.3 vLLM 0.25.1 ray 2.56.1 ljperf 0.1.0+d0e4a408 numpy 2.3.5	Ubuntu 24.04 Python 3.12.7 CUDA 13.0.2 NCCL 2.30.7-1 torch 2.11.0 torchvision 0.26.0 torchaudio 2.11.0 torchao 0.17.0 triton 3.6.0 Transformers 5.12.1 diffusers 0.39.0 decord 3.4.0 deep_ep 2.1.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flashinfer-python 0.6.13 flash_mla 1.0.0 mooncake 0.3.11.post1 nvshmem 3.7.1 xgrammar 0.2.3 vLLM 0.25.1 ray 2.56.0 xfuser 0.4.5 ljperf 0.1.0+477686c5	Ubuntu 24.04 Python 3.12 CUDA 13.0.2 NCCL 2.30.7-1 torch 2.11.0 torchvision 0.26.0 torchaudio 2.11.0 torchao 0.17.0 triton 3.6.0 Transformers 5.12.1 diffusers 0.39.0 decord 0.6.0 deep_ep 2.1.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flash-attn-4 4.0.0b15 flashinfer-python 0.6.12 flash_mla 1.0.0 mooncake 0.3.11.post1 nvshmem 3.7.1 xgrammar 0.2.1 sgl-kernel 0.4.4 sglang 0.5.15.post1 ray 2.56.0 xfuser 0.4.5 ljperf 0.1.0+d0e4a408	Ubuntu 24.04 Python 3.12 CUDA 13.0.2 NCCL 2.30.7-1 torch 2.11.0 torchvision 0.26.0 torchaudio 2.11.0 torchao 0.17.0 triton 3.6.0 transformers 5.12.1 diffusers 0.39.0 decord 3.4.0 deep_ep 2.1.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flash-attn-4 4.0.0b15 flashinfer-python 0.6.12 flash_mla 1.0.0 mooncake 0.3.11.post1 nvshmem 3.7.1 xgrammar 0.2.1 sgl-kernel 0.4.4 sglang 0.5.15.post1 ray 2.56.0 xfuser 0.4.5 ljperf 0.1.0+477686c5

Asset

Public images

CUDA 13.0 asset

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:26.07-vllm0.25.1-pytorch2.11-cu130-20260723-serverless
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:26.07-sglang0.5.15.post1-pytorch2.11-cu130-20260715-serverless

VPC images

To quickly pull ACS AI container images in a VPC, replace the Asset URI of the specified AI container image egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/{image:tag} with acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}.

{region-id}: the region ID of the ACS product. For example, cn-beijing and cn-wulanchabu.
{image:tag}: the name and tag of the AI container image. For example, inference-nv-pytorch:25.10-vllm0.11.0-pytorch2.8-cu128-20251028-serverless and training-nv-pytorch:25.10-serverless.

Note

These images are for ACS and EGS multi-tenant. Do not use them in EGS dedicated environments.

Driver requirements

CUDA 13.0: NVIDIA driver release ≥ 580

Quick start

This example shows how to pull the inference-nv-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

Note

To use the inference-nv-pytorch image in ACS, select the image from the Artifacts Center on the Create Workload page in the console. You can also specify the image reference in a YAML file. For more information, see the following topics about building model inference services with ACS GPU resources:

Pull the inference container image.

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Download the open-source model from ModelScope.

pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

Run the following command to enter the container.

docker run -d -t --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Run an inference test for the vLLM conversational feature.

Start the server service.

python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

Test on the client.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Introduce deep learning."}
    ]}'

For more information about how to use vLLM, see vLLM.

Known issues

The images in this release do not support the deepgpu-comfyui plugin.

Container Compute Service:inference-nv-pytorch 26.07