Container Service for Kubernetes:Quickly deploy a large language model inference service in ACK

Step 1: Download the model files

1. Install Git and Git Large File Storage (LFS). macOS

   brew install git
   brew install git-lfs

   Windows Download and install Git from the official Git website. Git Large File Storage is bundled with Git for Windows — download the latest version. Linux (Red Hat-based)

   yum install git
   yum install git-lfs

   For other Linux distributions, see the official Git website.

2. Clone the Qwen1.5-4B-Chat model repository and pull the large files:

   GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/qwen/Qwen1.5-4B-Chat.git
   cd Qwen1.5-4B-Chat
   git lfs pull

Step 2: Upload model files to OSS

1. Install and configure ossutil.

2. Create a bucket. To reduce model pull latency, create the bucket in the same region as your cluster:

   ossutil mb oss://<your-bucket-name>

3. Create a folder in the bucket for the model files:

   ossutil mkdir oss://<your-bucket-name>/Qwen1.5-4B-Chat

4. Upload the model files:

   ossutil cp -r ./Qwen1.5-4B-Chat oss://<your-bucket-name>/Qwen1.5-4B-Chat

Step 3: Configure a persistent volume (PV)

1. Log on to the ACK console and click the target cluster. In the left navigation pane, choose Volumes > Persistent Volumes.

2. Click Create. In the Create PV dialog box, set the following parameters and click Create:

   Parameter	Value
   PV type	OSS
   Volume name	llm-model
   Capacity	20Gi
   Access mode	ReadOnlyMany
   Access certificate	Select Create Secret
   Optional parameters	-o umask=022 -o max_stat_cache_size=0 -o allow_other
   Bucket ID	Click Select Bucket and select your bucket
   OSS path	/Qwen1.5-4B-Chat
   Endpoint	Select Public Endpoint

Step 4: Configure a persistent volume claim (PVC)

1. In the left navigation pane, choose Volumes > Persistent Volume Claims.

2. On the Persistent Volume Claims page, set the following parameters and click Create:

   Parameter	Value
   PVC type	OSS
   Name	llm-model
   Allocation mode	Select Existing Volumes
   Existing volumes	Select the llm-model PV created in the previous step
   Capacity	20Gi

Step 5: Deploy the inference service

Run the Arena command to deploy the service. The --data flag mounts the PVC containing the pre-loaded model files. Because the model is already on the mounted volume, the pod starts without downloading anything:

arena serve custom \
    --name=modelscope \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --data=llm-model:/Qwen1.5-4B-Chat \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/quick-deploy-llm:v1 \
    "MODEL_ID=/Qwen1.5-4B-Chat python3 server.py"

The following output confirms the inference service was submitted:

service/modelscope-v1 created
deployment.apps/modelscope-v1-custom-serving created
INFO[0001] The Job modelscope has been submitted successfully
INFO[0001] You can run `arena serve get modelscope --type custom-serving -n default` to check the job status

Check the service status:

arena serve get modelscope

Once the pod status shows Running, the inference service is ready.

How can I pull model files from Hugging Face instead of ModelScope?

Make sure the container runtime can reach the Hugging Face repository, then set MODEL_SOURCE=Huggingface in the Arena command. The GPU node needs at least 30 GB of free disk space to accommodate the downloaded files:

arena serve custom \
    --name=huggingface \
    --version=v1 \
    --gpus=1 \
    --replicas=1 \
    --restful-port=8000 \
    --readiness-probe-action="tcpSocket" \
    --readiness-probe-action-option="port: 8000" \
    --readiness-probe-option="initialDelaySeconds: 30" \
    --readiness-probe-option="periodSeconds: 30" \
    --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/quick-deploy-llm:v1 \
    "MODEL_ID=Qwen/Qwen1.5-4B-Chat MODEL_SOURCE=Huggingface python3 server.py"

The following output confirms the resources were created:

service/huggingface-v1 created
deployment.apps/huggingface-v1-custom-serving created
INFO[0003] The Job huggingface has been submitted successfully
INFO[0003] You can run `arena serve get huggingface --type custom-serving -n default` to check the job status

Pull models from Hugging Face

1. Make sure that the container runtime environment can access the Hugging Face repository.

2. Use the Arena client to deploy a custom service and specify the container image for deployment using the --image parameter. For more information about the parameters, see Command parameter reference.

   This method downloads the Hugging Face model files into the container. Ensure that your GPU node has at least 30 GB of available disk space.

   arena serve custom \
       --name=huggingface \
       --version=v1 \
       --gpus=1 \
       --replicas=1 \
       --restful-port=8000 \
       --readiness-probe-action="tcpSocket" \
       --readiness-probe-action-option="port: 8000" \
       --readiness-probe-option="initialDelaySeconds: 30" \
       --readiness-probe-option="periodSeconds: 30" \
       --image=kube-ai-registry.cn-shanghai.cr.aliyuncs.com/kube-ai/quick-deploy-llm:v1 \
       "MODEL_ID=Qwen/Qwen1.5-4B-Chat MODEL_SOURCE=Huggingface python3 server.py"

   The following output indicates that the Kubernetes resources for the huggingface-v1 inference service have been created:

   service/huggingface-v1 created
   deployment.apps/huggingface-v1-custom-serving created
   INFO[0003] The Job huggingface has been submitted successfully 
   INFO[0003] You can run `arena serve get huggingface --type custom-serving -n default` to check the job status