inference-nv-pytorch 26.06 - Container Compute Service - Alibaba Cloud Documentation Center

This topic provides the release notes for inference-nv-pytorch 26.06.

Main features and bug fixes

Main features

In the vLLM image, vLLM is upgraded to v0.22.0.
In the SGLang image, SGLang is upgraded to v0.5.12.post1.
The vLLM and SGLang images now support amd64 and aarch64 architectures.

Bug fixes

None

Image name	inference-nv-pytorch
Tag	26.06-vllm0.22.0-pytorch2.11-cu130-20260611-serverless		26.06-sglang0.5.12.post1-pytorch2.11-cu130-20260611-serverless
Supported architecture	amd64	aarch64	amd64	aarch64
Use case	large model inference	large model inference	large model inference	large model inference
Framework	pytorch	pytorch	pytorch	pytorch
Requirements	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580	NVIDIA Driver release >= 580
System components	Ubuntu 24.04 Python 3.12.7 Torch 2.11.0 CUDA 13.0.2 NCCL 2.30.7-1 torchvision 0.26.0 TorchAudio 2.11.0 triton 3.6.0 Transformers 5.9.0 diffusers 0.38.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flashinfer-python 0.6.11 xgrammar 0.2.1 vLLM 0.22.0 ray 2.55.1 xfuser 0.4.5 ljperf 0.1.0+d0e4a408	Ubuntu 24.04 Python 3.12.7 Torch 2.11.0 CUDA 13.0.2 NCCL 2.30.7-1 torchvision 0.26.0 TorchAudio 2.11.0 torchao 0.17.0 triton 3.6.0 Transformers 5.11.0 diffusers 0.38.0 decord 0.6.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flashinfer-python 0.6.11 xgrammar 0.2.1 vLLM 0.22.0 ray 2.55.1 xfuser 0.4.5 ljperf 0.1.0+477686c5	Ubuntu 24.04 Python 3.12 Torch 2.11.0 CUDA 13.0.2 NCCL 2.30.7-1 torchvision 0.26.0 TorchAudio 2.11.0 torchao 0.17.0 triton 3.6.0 Transformers 5.9.0 diffusers 0.38.0 decord 0.6.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flash-attn-4 4.0.0b15 flashinfer-python 0.6.11 xgrammar 0.2.0 sgl-kernel 0.4.2 sglang 0.5.12.post1 ray 2.55.1 xfuser 0.4.5 ljperf 0.1.0+d0e4a408 deep_ep 2.0.0+af9a040 flash_mla 1.0.0+b7643bd	Ubuntu 24.04 Python 3.12 Torch 2.11.0 CUDA 13.0.2 NCCL 2.30.7-1 torchvision 0.26.0 TorchAudio 2.11.0 torchao 0.17.0 triton 3.6.0 Transformers 5.6.0 diffusers 0.38.0 decord 3.4.0 imageio-ffmpeg 0.6.0 flash_attn 2.8.3 flash_attn_3 3.0.0 flash-attn-4 4.0.0b15 flashinfer-python 0.6.11 xgrammar 0.2.0 sgl-kernel 0.4.2 sglang 0.5.12.post1 ray 2.55.1 xfuser 0.4.5 ljperf 0.1.0+477686c5 deep_ep 2.0.0+af9a040 flash_mla 1.0.0+b7643bd

Asset

Public image

CUDA 13.0 asset

egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:26.06-vllm0.22.0-pytorch2.11-cu130-20260611-serverless
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:26.06-sglang0.5.12.post1-pytorch2.11-cu130-20260622-serverless

VPC image

Replace the public network registry host in your image URI with the region-specific VPC endpoint.

URI component	Public network	IN-VPC
Registry host	`egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com`	`acs-registry-vpc.{region-id}.cr.aliyuncs.com`
Repository	`egslingjun`	`egslingjun` (unchanged)
Image and tag	`{image:tag}`	`{image:tag}` (unchanged)

Replace {region-id} with the ID of the region where your ACS service runs. For example:

Region	Region ID
China (Beijing)	`cn-beijing`
China (Ulanqab)	`cn-wulanchabu`

For the full list of supported regions, see Regions.

Example {image:tag} values:

inference-nv-pytorch:25.10-vllm0.11.0-pytorch2.8-cu128-20251028-serverless
training-nv-pytorch:25.10-serverless

Note

These images are for ACS and EGS multi-tenant. Do not use them in EGS dedicated environments.

Driver requirements

CUDA 13.0: NVIDIA Driver release >= 580

Quick start

This example shows how to pull the inference-nv-pytorch image using Docker and test the inference service with the Qwen2.5-7B-Instruct model.

Note

To use the inference-nv-pytorch image in ACS, select the image from the Artifacts Center on the Create Workload page in the console. You can also specify the image reference in a YAML file. For more information, see the following topics about building model inference services with ACS GPU resources:

Pull the inference container image.

docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Download the open-source model from ModelScope.

pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

Run the following command to enter the container.

docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

Run an inference test for the vLLM conversational feature.

Start the server service.

python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

Test on the client.

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "/mnt/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Introduce deep learning."}
    ]}'

For more information about how to use vLLM, see vLLM.

Known issues

The images in this release do not support the deepgpu-comfyui plugin.

Container Compute Service:inference-nv-pytorch 26.06