All Products
Search

This Product

    Container Compute Service:inference-nv-pytorch 25.08

    Document Center

    Container Compute Service:inference-nv-pytorch 25.08

    Last Updated:Jun 20, 2026

    These are the release notes for inference-nv-pytorch 25.08.

    Main features and bug fixes

    Main features

    • Upgraded vLLM to v0.10.0.

    • Upgraded SGLang to v0.4.10.post2.

    Bug fixes

    None.

    Contents

    inference-nv-pytorch

    inference-nv-pytorch

    Tag

    25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless

    25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless

    Application scenario

    Large model inference

    Large model inference

    Framework

    pytorch

    pytorch

    Requirements

    NVIDIA Driver release >= 570

    NVIDIA Driver release >= 570

    System components

    • Ubuntu 24.04

    • Python 3.12

    • Torch 2.7.1+cu128

    • CUDA 12.8

    • NCCL 2.27.5

    • diffusers 0.34.0

    • deepgpu-comfyui 1.1.7

    • deepgpu-torch 0.0.24+torch2.7.0cu128

    • flash_attn 2.8.2

    • imageio 2.37.0

    • imageio-ffmpeg 0.6.0

    • ray 2.48.0

    • transformers 4.55.0

    • triton 3.3.1

    • vllm 0.10.0

    • xformers 0.0.31

    • xfuser 0.4.4

    • xgrammar 0.1.21

    • Ubuntu 24.04

    • Python 3.12

    • Torch 2.7.1+cu128

    • CUDA 12.8

    • NCCL 2.27.5

    • diffusers 0.34.0

    • deepgpu-comfyui 1.1.7

    • deepgpu-torch 0.0.24+torch2.7.0cu128

    • flash_attn 2.8.2

    • flash_mla 1.0.0+41b611f

    • flashinfer-python 0.2.9rc2

    • imageio 2.37.0

    • imageio-ffmpeg 0.6.0

    • transformers 4.54.1

    • sgl-kernel 0.2.8

    • sglang 0.4.10.post2

    • xgrammar 0.1.22

    • triton 3.3.1

    • torchao 0.9.0

    Assets

    Public images

    • egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless

    • egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless

    VPC image

    acs-registry-vpc.{region-id}.cr.aliyuncs.com/egslingjun/{image:tag}
    Replace {region-id} with the region where your Alibaba Cloud Container Compute Service (ACS) is activated (for example, cn-beijing or cn-wulanchabu).
    Replace {image:tag} with the name and tag of the image.
    Note

    The inference-nv-pytorch:25.03-vllm0.8.2-pytorch2.6-cu124-20250328-serverless and inference-nv-pytorch:25.03-sglang0.4.4.post1-pytorch2.5-cu124-20250327-serverless images are compatible with ACS and Lingjun multi-tenant deployments only. It is not compatible with Lingjun single-tenant deployments.

    Note

    The inference-nv-pytorch:25.08-vllm0.10.0-pytorch2.7-cu128-20250811-serverless and inference-nv-pytorch:25.08-sglang0.4.10.post2-pytorch2.7-cu128-20250808-serverless images are suitable for ACS and Lingjun multi-tenant deployments. They are not compatible with Lingjun single-tenant deployments.

    Driver requirements

    NVIDIA Driver release >= 570

    Quick start

    This example shows how to use Docker to pull the inference-nv-pytorch image and test its inference service with the Qwen2.5-7B-Instruct model.

    Note

    To use the inference-nv-pytorch image in ACS, you can select it from the Artifact Center page when you create a workload in the console, or specify the image in a YAML file. For more information, see the following topics:

    1. Pull the inference container image.

      docker pull egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

    2. Download the open-source model in ModelScope format.

      pip install modelscope
cd /mnt
modelscope download --model Qwen/Qwen2.5-7B-Instruct --local_dir ./Qwen2.5-7B-Instruct

    3. Run the following command to start and enter the container.

      docker run -it --rm --gpus all --network=host --privileged --init --ipc=host \
--ulimit memlock=-1 --ulimit stack=67108864  \
-v /mnt/:/mnt/ \
egslingjun-registry.cn-wulanchabu.cr.aliyuncs.com/egslingjun/inference-nv-pytorch:[tag]

    4. Test the conversational inference feature of vLLM.

      1. Start the server.

        python3 -m vllm.entrypoints.openai.api_server \
--model /mnt/Qwen2.5-7B-Instruct \
--trust-remote-code --disable-custom-all-reduce \
--tensor-parallel-size 1

      2. Run a test on the client.

        curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",  
    "messages": [
    {"role": "system", "content": "You are a friendly AI assistant."},
    {"role": "user", "content": "Tell me about deep learning."}
    ]}'

        For more information, see vLLM.

    Known issues

    • The deepgpu-comfyui plug-in, which accelerates video generation for the Wanx model, currently supports only GN8IS and G49E.