Quickly deploy a scalable and highly available DeepSeek-R1-Distill-Qwen-7B inference service

Solution overview

This solution deploys ECI-based inference instances across multiple zones in a single region for zone-level disaster recovery. ALB handles traffic distribution, and Object Storage Service (OSS) stores model files. The following figure shows the architecture.

1. Plan the cluster network

Create a Virtual Private Cloud (VPC) and vSwitches across multiple zones. This example uses one VPC and two vSwitches in different zones for high availability.

You can reuse an existing VPC and skip this step.

1. In the console, create one VPC and two vSwitches as instructed below.

   ①②: Click Create VPC.

   ③: Region. China (Hangzhou). ④: Name. vpc-ess-hangzhou. ⑤: IPv4 CIDR. Select 192.168.0.0/16.

   vSwitch 1: ⑥: Name. vSwitch-j. ⑦: Zone. Zone J. ⑧: IPv4 CIDR. 192.168.0.0/24. ⑨: Click Add to create vSwitch 2.

   vSwitch 2: ⑩: Name. vSwitch-k. ⑪: Zone. Zone K. ⑫: IPv4 CIDR. 192.168.1.0/24.

2. Click OK and wait for the VPC to be created.

2. Create an OSS bucket

Create an Object Storage Service (OSS) bucket to store model weight files. ECI instances read files from this bucket.

You can reuse an existing OSS bucket and skip this step.

1. In the Object Storage Service console, create a bucket.

   Configure the following key parameters. Keep the defaults for others.

   ①: Click Create Bucket. ②: Bucket. This name is required when you mount the storage space later. ③: Region. Select Specific Region and make sure the region is the same as the region of your ECI instances. This tutorial uses China (Hangzhou). Same-region access from ECI to OSS uses the internal network and incurs no data transfer fees. Access OSS resources from an ECS instance over an internal network.

2. Click OK.

3. Create an instance RAM role

Create an instance RAM role that allows ECI instances to read model weight files from your OSS bucket.

1. In the console, create an instance RAM role with the following settings.

   ①: Click Create Role.

   ②: Principal Type. Select Cloud Service. ③: Principal Name. Select ECS. 4. Click OK and set a name for the instance RAM role as prompted.

2. In the console, create the following policy.

   ①: Click Create Policy. ②: Click JSON Editor. ③: This policy grants full permissions for a specific bucket. Use the following policy script. Important When you configure this policy, replace <bucket_name> with the Bucket of the bucket that you created. { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": "oss:*", "Resource": [ "acs:oss:*:*:<bucket_name>", "acs:oss:*:*:<bucket_name>/*" ] } ] } Click OK and set a name for the policy as prompted.

3. In the console, grant permissions to the instance RAM role.

   ①②: Click Grant Permission. ③: Principal. Select the instance RAM role that you created. ④: Policies. Select the policy that you created. Click OK.

4. Prepare image cache and model files

Create an image cache to accelerate ECI instance startup, and download model weight files to OSS.

1. In the console, create an ECI image cache.

   ①②: Click Create Image Cache.

   ③④: Region and Zone. Must match the VPC region from Step 1. The zone must match one of the vSwitches from Step 1. ⑤⑥: Network. Select the VPC and one of the vSwitches that you created in Step 1. ⑦: Elastic IP Addresses. Select an existing EIP. If none exists, click EIP console to create one, then return, click the refresh icon , and select it. ⑧: Security Group. Select a security group. If none exists, click Create Security Group to create one, then return and select it. ⑨: Image Cache Name. vllm. ⑩: Cache Size. 100 GB. ⑪: Image. egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm. ⑫: Version. 0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04.

   Wait for the image cache to be created (~15 minutes). Check progress on the image cache page of the console.

2. In the console, launch a temporary ECI instance to download the model weight files to the OSS bucket.

   ①②: Click Create Container Group.

   ③: Billing Method. pay-as-you-go. ④: Region. Must match the VPC region from Step 1. This tutorial uses China (Hangzhou). ⑤: VPC. Select the VPC that you created in Step 1. ⑥: vSwitch. Select the vSwitch that you created in Step 1. ⑦: Security Group. Set the parameter as prompted on the page.

   Container group configuration: ⑧: CPU. 2 vCPU. ⑨: Memory. 4 GiB. ⑩: After containers run and exit. Select Upon Failure. Advanced settings: ⑪: Expand Advanced Configuration. ⑫: Turn on Automatically Match Image Cache. ⑬: Select and add OSS Persistence. ⑭: Name. oss-data. ⑮: Bucket. Select the bucket that you created in Step 2. ⑯: RAM Role. Select the instance RAM role that you created in Step 3. ⑰: Ephemeral Storage Size. Set the value to 100 GiB.

   Container configuration: ⑱: Image. egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm. Ⲅ: Image Tag. 0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04. ⑳: Startup Command. Copy the following commands to the corresponding fields as shown in the figure. /bin/bash -c git-lfs clone https://www.modelscope.cn/models/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.git /oss-data/DeepSeek-R1-Distill-Qwen-7B This command uses git-lfs to clone the model from the ModelScope community repository to the /oss-data/DeepSeek-R1-Distill-Qwen-7B directory. Advanced container settings: ㉑: Expand container-1 advanced settings. ㉒: CPU. 2 vCPU. ㉓: Memory. 4 GiB. ㉔: Turn on Storage and Add a volume mount. ㉕: Select oss-data. ㉖: Mount Path. /oss-data. ㉗: Click Next: Other Settings.

   ㉘: Elastic IP Addresses. Select Auto Create. ㉙: Maximum Bandwidth. Set the value to 200 Mbit/s. ㉚: Click Confirm Configuration and complete the creation of the ECI instance as prompted.

   Model weight file download starts automatically after the ECI instance is created. Proceed with Step 5 and Step 6 while the download runs. How to check if the download is complete After the download is complete, the status of the ECI instance changes to Run successfully in the ECI console. In the Object Storage Service console, a folder named DeepSeek-R1-Distill-Qwen-7B is generated in the bucket.

5. Create an ALB instance

Create an Application Load Balancer (ALB) instance as the service endpoint.

1. In the console, create an ALB instance.

   ①②: Click Create ALB.

   ③: Region. Select China (Hangzhou) to be consistent with the VPC that you created in Step 1. ④: Network Type. Select Public to provide services over the internet. ⑤: VPC. Select the VPC that you created in Step 1. ⑥: Zone and vSwitch. Select the vSwitches that you created in Step 1. ⑦: Instance Name. alb-eci-deepseek-7B. ⑧: Click Buy Now and then complete the creation of the ALB instance as prompted.

2. In the console, create a listener and a backend server group for the ALB instance.

   ①②: Find the ALB instance that you created and click Create Listener in the Actions column. If you cannot find the ALB instance, switch to a different region in the upper-left corner of the page.

   ③: Select HTTP as the protocol. ④: Set the listener port to 80. ⑤: Click Next. ⑥: Click Create Server Group. ⑦: Server Group Type. Select Server. ⑧: Server Group Name. eci-deepseek-7B. ⑨: VPC. Select the VPC that you created in Step 1. This parameter is automatically selected. ⑩: Select Backend Server Protocol. HTTP. ⑪⑫: Enable Health Check. ⑬: Health check method. Select GET. ⑭: Health check path. Set the value to /health. After the inference service starts, ALB uses the /health path to determine the service status. ⑮⑯⑰: Click Create, click Next, and then click Submit. Wait for the creation to complete.

6. Create a scaling group

Create a scaling group associated with the ALB instance. The group automatically manages ECI instances and adds them to the ALB backend server group.

1. In the Auto Scaling console, create a scaling group and associate it with the ALB instance.

   ①②③: Click Create Scaling Group.

   ④: Scaling Group Name. deepseek-7B-servers. ⑤: Type. Select ECI. ⑥: Instance Configuration Source. Select Create from Scratch. ⑦: Minimum Number of Instances. Set this parameter to 0. This is the minimum number of instances in the scaling group. ⑧: Maximum Number of Instances. Set this parameter to 10. This is the maximum number of instances in the scaling group. ⑨: VPC. Select the VPC that you created in Step 1. ⑩: vSwitch. Select all the vSwitches that you created in Step 1. ⑪: Click Show Advanced Settings.

   Advanced settings: ⑫⑬: Enable Expected Number of Instances and set the value to 0. This creates an empty scaling group. ⑭: Click Add Server Group and set the Type to ALB. ⑮: Server Group. Select the server group that you created in Step 5. ⑯: Port. Select 30000. The inference service deployed in the ECI instance will use this port to provide services. ⑰: Click Create. Wait for the scaling group to be created. Then, you can create a scaling configuration as prompted.

2. In the Auto Scaling console, create a scaling configuration.

   A scaling configuration is a template for creating instances during scale-out events.

   ①②③: Find the scaling group that you created and click its ID to go to the details page.

   ④⑤⑥: Click Instance Configuration Sources > Scaling Configuration > Create Scaling Configuration.

   ⑦: Billing Method. Select Pay-As-You-Go. ⑧: Select a Security Group.

   Container group configuration: ⑨: Select Specify Instance Type. ⑩: Instance Type. ecs.gn7i-c8g1.2xlarge. ⑪: Select Automatically Match Image Cache. ⑫: Expand Advanced Configuration. ⑬⑭⑮⑯: Select OSS Persistence. Configure the Bucket and instance RAM Role. ⑰: Ephemeral Storage Size. 100 GiB. ⑱: GPU Driver Version. tesla=550.

   Container configuration: Ⲅ: Image. egs-registry.cn-hangzhou.cr.aliyuncs.com/egs/vllm. ⑳: Image Tag. 0.6.4.post1-pytorch2.5.1-cuda12.4-ubuntu22.04. ㉑: Startup Command. Copy the following commands to the corresponding fields as shown in the figure. /bin/bash -c vllm serve /oss-data/DeepSeek-R1-Distill-Qwen-7B --port 30000 --served-model-name DeepSeek-R1-Distill-Qwen-7B --tensor-parallel-size 1 --max-model-len=16384 --enforce-eager --dtype=half --api-key api-key-example-abc123 This command reads the model weight files from OSS and starts the inference service on port 30000. The API key is set to api-key-example-abc123.

   ㉒: Expand container-1 advanced settings. ㉓: CPU. Set the value to 8 vCPU. ㉔: Memory. Set the value to 30 GiB. ㉕: GPU. Set the value to 1. ㉖㉗: Turn on Storage and Add storage. ㉘: Mount Path. /oss-data ㉙: Click Next: Other Settings.

   ㉚: Elastic IP Addresses. Select Auto Create. ㉛: Maximum Bandwidth. 200 Mbit/s. ㉜: Click Confirm Configuration and complete the creation of the scaling configuration as prompted.

   ㉝㉞㉟: Enable the scaling configuration and start the scaling group as prompted.

7. Start the service

Trigger a scale-out by setting the expected instance count. This example scales to five ECI instances.

After you adjust the expected number of instances, there is a delay before the new instances are created. You can monitor the progress on the Scaling Activities tab.

Next steps

Production considerations

Consider the following improvements for production environments.

• Set up auto scaling: Scale instances based on workload to optimize costs. Configure event-triggered tasks for GPU memory-based scaling, or automatically scale based on the queries per second (QPS) metrics of a single ALB instance.

• Use a NAT gateway: Instead of assigning an EIP to each instance, use a NAT gateway to provide shared internet access for ECI instances.

• Add a custom domain and HTTPS: Use a custom domain name for the service endpoint and configure an HTTPS listener for security. Configure a CNAME record for an ALB instance and Add an HTTPS listener.