Auto Scaling:Reduce fine-tuning costs for large models using spot instances

Cost comparison

Assuming a 12-hour training period, the unit price for a spot instance is 3.5 RMB per hour, while a pay-as-you-go instance costs 10 RMB per hour. The following table provides the cost comparison between the two options.

Procedure

1. Create a base image containing the essential training environment.

   This image serves as the startup template for instances in the scaling group. The built-in auto-start script ensures new instances can quickly resume training and run automatically.

2. Create and configure a scaling group.

   A scaling group ensures the training task continues by automatically launching new spot or pay-as-you-go instances when existing ones are interrupted or reclaimed.

3. Start training.

   Once a scaling group is configured, a scale-out event is automatically triggered. This creates new instances, which immediately begin running the training task based on the predefined automation rules.

4. Test: Simulate interruption and reclamation

   Manually trigger instance interruption and reclamation to confirm that the system automatically launches a new instance and resumes the interrupted training task correctly. This validation is critical for ensuring stability and reliability during resource reclamation scenarios.

1.1 Create an instance and build a basic environment

This instance serves as a template for creating an image. Later, Auto Scaling will automatically launch new instances from this image in the scaling group.

1. Go to the ECS console to create a GPU-accelerated instance.

   First, you need to create a pay-as-you-go GPU-accelerated instance to set up the basic environment. This example uses the ecs.gn7i-c8g1.2xlarge instance type, deployed in Zone J of the China (Hangzhou) region. The configuration steps are illustrated in the following figure.

   ① Billing Method: Set the value to Pay-as-you-go. ② Region: Select China (Hangzhou). ③④ Network and Zone: Select the VPC and vSwitches. If no VPCs or vSwitches exist, follow the on-screen instructions to create them.

   ⑤⑥ Instance > All Instance Types: Select ecs.gn7i-c8g1.2xlarge.

   ⑦⑧⑨ Image > Public Images: Select Ubuntu 22.04 64 bit. ⑩ Auto-Install GPU Driver: Specify CUDA Version 12.4.1, Driver Version 550.127.08, and CUDNN Version 9.2.0.82.

   ⑪ System Disk > Size: Enter 60 GiB.

   ⑫ Public IP Address: Select Assign Public IPv4 Address to enable Internet access and file download. ⑬ Bandwidth Billing Method: Select Pay-by-traffic. ⑭ Maximum Bandwidth: Select 100 Mbps. ⑮ Security Group: Click the New Security Group tab. ⑯ Security Group Type: Set the value to Basic Security Group. ⑰ Open IPv4 Ports/Protocols: Select SSH (TCP:22) and ICMP (IPv4) to facilitate subsequent remote connection.

   ⑱⑲⑳ Log Credential: Select Key Pair. This key pair is required for logging on to the ECS instance. You can also set the value to Custom Password. Complete the settings as prompted. ㉑ Instance Name: Enter a name for the ECS instance. Use a clear and memorable instance name to make searching easier. In this example, ess-lora-deepseek7b-template is used.

   Click Confirm Order. Wait until the ECS instance is created.

2. Once the ECS instance is ready, connect to it and wait for the GPU driver installation to complete.

   1. Go to ECS console - Instance.

   2. Locate the ECS instance you created in the previous step, and click Connect in the Actions column. Use Workbench to establish a connection, and log on to the ECS instance as prompted.

      If the ECS instance isn't found, check whether your current region matches the instance's region. You can switch regions by using the dropdown list in the upper-left corner.

      If the ECS instance is stopped, refresh the page and wait for it to start.

      

   3. Once you've connected to the ECS instance, wait until the GPU driver installation is complete. After installation, the system will prompt you to reconnect to the ECS instance.

      If the interface becomes unresponsive, try refreshing the page and reconnecting to the ECS instance.

      

      How can I verify if the installation is complete when I can't see the page content?

      This page does not appear after you connect to the ECS instance, likely because the driver is already installed. To check if the driver is installed, run the following command:

      nvidia-smi

      If the following page appears, the driver has been installed.

      

      If this page doesn't appear, the startup driver is either not installed or has an issue. We recommend creating a new instance and selecting Auto-Install GPU Driver during setup.

      If you want to manually install the driver, refer to Manually install the Tesla driver on a GPU-accelerated compute-optimized Linux instance.

3. Install Python dependencies.

   To install the Python dependencies needed for training, run the following commands:

   This example uses the Ubuntu 22.04 64-bit image, which includes Python 3.10. As a result, you won't need to install Python dependencies separately.

   # The Ubuntu 22.04 64-bit image includes Python 3.10 by default, so no extra installation is needed.
   python3 -m pip install --upgrade pip
   # Switch to Alibaba Cloud internal image repository.
   pip config set global.index-url http://mirrors.cloud.aliyuncs.com/pypi/simple/
   
   pip install modelscope==1.22.3
   pip install openai==1.61.0
   pip install tqdm==4.67.1
   pip install "vllm>=0.5.1" -U
   pip install "lmdeploy>=0.5,<0.6.5" -U --no-deps
   pip install autoawq -U --no-deps
   pip install auto_gptq optimum bitsandbytes -U
   pip install ms-swift[all]
   pip install timm -U
   pip install deepspeed==0.14.* -U
   pip install qwen_vl_utils decord librosa pyav icecream -U

While waiting for the dependencies to install, you can click the icon to enable the multi-terminal feature and proceed with the steps in Step 1.2.

1.2 Create and attach an OSS bucket

To store the model weight file, dataset, and checkpoints generated during training, you must first create an OSS bucket. Once the bucket is created, attach it to your ECS instance as an additional data disk.

1. Go to the OSS console to create a bucket.

   The following figures show the key parameter settings for this example. Retain the default values for any parameters not listed.

   ② Bucket Name: Enter a name for the bucket. The bucket name is required when mounting the bucket. ③ Region: Select a region. The selected region must match the region of the ECS instance. In this example, the region must be China (Hangzhou). You can use an ECS instance to access OSS buckets within the same region over an internal network, and there are no charges for traffic on this network. For more information, see Access to OSS resources from an ECS instance by using an internal endpoint of OSS.

2. Create and bind a RAM role.

   A Resource Access Management (RAM) role is used to grant an ECS instance permissions to access OSS buckets. To create and bind a RAM role, perform the following steps:

   1. In the RAM console, create a RAM role. The following figures show the key parameter settings for this example.

      ② Principal Type: Select Cloud Service. ③ Principal Name: Select Elastic Compute Service, which specifies that the RAM role will be assigned to the ECS instance. ④ Click OK and specify a name for the RAM role as prompted.

   2. In the RAM console, create a custom policy as shown in the following figure.

      ③ This policy grants all the necessary permissions to access an OSS bucket. The policy script is as follows: Important When configuring the custom policy, replace <bucket_name> with the name of the bucket you actually created. { "Version": "1", "Statement": [ { "Effect": "Allow", "Action": "oss:*", "Resource": [ "acs:oss:*:*:<bucket_name>", "acs:oss:*:*:<bucket_name>/*" ] } ] } Click OK and configure the policy name as prompted.

   3. In the RAM console, grant permissions to the RAM role.

      ⑥ Principal: Select the RAM role that you created. ⑦ Policy: Select the custom policy that you created. Click Grant permissions.

   4. Log on to the ECS console and assign the RAM role to the ECS instance.

      If the ECS instance isn't found, check whether your current region matches the instance's region. You can switch regions by using the dropdown list in the upper-left corner.

      

3. Mount the OSS bucket to the ECS instance.

   1. Connect to the ECS instance created in Step 1.1, and run the following commands to install ossfs:

      wget https://gosspublic.alicdn.com/ossfs/ossfs_1.91.5_ubuntu22.04_amd64.deb
      apt-get update
      DEBIAN_FRONTEND=noninteractive apt-get install gdebi-core
      DEBIAN_FRONTEND=noninteractive gdebi -n ossfs_1.91.5_ubuntu22.04_amd64.deb

   2. Run the following commands to mount the OSS bucket: Update the parameter settings in the commands with the following values:

      • <bucket_name>: Replace this with the name of the bucket you created.

        View the name of a bucket

        Go to the OSS console to find the bucket that you created.

      • <ecs_ram_role>: Replace this with the name of the RAM role you created.

        View the name of a RAM role

        Go to the ECS console, find the instance to which the RAM role is assigned, and then follow the steps shown in the following figure to go to the Attach/Detach RAM Role page.

        This page shows the RAM role that has been assigned to the instance.

        If the ECS instance isn't found, check whether your current region matches the instance's region. You can switch regions by using the dropdown list in the upper-left corner.

        

      • <internal_endpoint>: Replace this with oss-cn-hangzhou-internal.aliyuncs.com.

        Important

        In this example, the bucket is located in the China (Hangzhou) region. As a result, the VPC endpoint used is oss-cn-hangzhou-internal.aliyuncs.com.

        How can I obtain the endpoint if my OSS bucket is not located in China (Hangzhou)?

        If you created an OSS bucket in a region other than China (Hangzhou), retrieve the VPC endpoint and replace <internal_endpoint> with the VPC endpoint you obtained.

        

      # Replace the bucket name, VPC endpoint, and RAM role with their respective actual values.
      BUCKET_NAME="<bucket_name>"
      ECS_RAM_ROLE="<ecs_ram_role>"
      INTERNAL_ENDPOINT="<internal_endpoint>"
      
      # The mount directory of the bucket.
      BUCKET_MOUNT_PATH="/mnt/oss-data"
      
      #1. Back up the fstab file before mounting.
      cp /etc/fstab /etc/fstab.bak
      
      #2. Create the mount directory.
      mkdir $BUCKET_MOUNT_PATH
      #3. Mount the bucket to the instance.
      ossfs $BUCKET_NAME $BUCKET_MOUNT_PATH -ourl=$INTERNAL_ENDPOINT -oram_role=http://100.100.100.200/latest/meta-data/ram/security-credentials/$ECS_RAM_ROLE
      #4. Enable automatic mounting upon instance startup.
      echo "ossfs#$BUCKET_NAME $BUCKET_MOUNT_PATH fuse _netdev,url=http://$INTERNAL_ENDPOINT,ram_role=http://100.100.100.200/latest/meta-data/ram/security-credentials/$ECS_RAM_ROLE,allow_other 0 0" | sudo tee -a /etc/fstab

4. Check if the storage space is available.

   1. Upload any file to the OSS bucket.

      

   2. Run the following command in the instance to check if the file is visible in the mount directory:

      ls /mnt/oss-data/

      If it appears, this means the mount was successful.

      

   Why can't I find this file in the OSS bucket?

   This issue may be due to a mount failure. Follow these steps to troubleshoot the issue.

   1. Verify if the ECS instance and the OSS bucket are located in the same region. Also, ensure that <internal_endpoint> in the mount command is correctly replaced with the VPC endpoint of your bucket.

   2. Verify if <bucket_name> in the mount command is correctly replaced with the name of your bucket.

   3. Reassign the RAM role to the ECS instance and verify if <ecs_ram_role> in the mount command matches the name of the RAM role.

   4. Replace <bucket_name>, <internal_endpoint>, and <ecs_ram_role> in the following commands with the appropriate values, and then remount the bucket.

      # Replace the bucket name, VPC endpoint, and RAM role with their respective actual values.
      BUCKET_NAME="<bucket_name>"
      ECS_RAM_ROLE="<ecs_ram_role>"
      INTERNAL_ENDPOINT="<internal_endpoint>"
      
      # The mount directory of the bucket.
      BUCKET_MOUNT_PATH="/mnt/oss-data"
      
      #1. Attempt to unmount the bucket.
      umount $BUCKET_MOUNT_PATH
      
      #2. Restore the /etc/fstab file from the backup.
      cp /etc/fstab.bak /etc/fstab
      
      #3. Remount the bucket to the instance.
      ossfs $BUCKET_NAME $BUCKET_MOUNT_PATH -ourl=$INTERNAL_ENDPOINT -oram_role=http://100.100.100.200/latest/meta-data/ram/security-credentials/$ECS_RAM_ROLE
      #4. Re-enable automatic mounting upon instance startup.
      echo "ossfs#$BUCKET_NAME $BUCKET_MOUNT_PATH fuse _netdev,url=http://$INTERNAL_ENDPOINT,ram_role=http://100.100.100.200/latest/meta-data/ram/security-credentials/$ECS_RAM_ROLE,allow_other 0 0" | sudo tee -a /etc/fstab

   4. Verify if the file from the OSS bucket is visible in the mount directory (/mnt/oss-data).

      ls /mnt/oss-data/

1.3 Prepare a model and dataset

The model weight file and dataset referenced in this topic can be downloaded from the ModelScope community. After connecting to the ECS instance, download the model and dataset to the mount directory of the OSS bucket and wait for all files to finish downloading.

1. Download a dataset

   # The mount directory of the bucket.
   BUCKET_MOUNT_PATH="/mnt/oss-data"
   
   # Download the fine-tuning dataset from the ModelScope community.
   # Use the modelscope tool installed in Step 1.1.
   modelscope download --dataset swift/self-cognition --local_dir $BUCKET_MOUNT_PATH/self-cognition
   modelscope download --dataset AI-ModelScope/alpaca-gpt4-data-zh --local_dir $BUCKET_MOUNT_PATH/alpaca-gpt4-data-zh
   modelscope download --dataset AI-ModelScope/alpaca-gpt4-data-en --local_dir $BUCKET_MOUNT_PATH/alpaca-gpt4-data-en

   If the progress is stuck, try pressing Enter multiple times.

2. Download the model weight file

   Important

   If the model weight file is too large and the download fails or a please try again message appears, simply retry the commands to resume the download.

   # The mount directory of the bucket.
   BUCKET_MOUNT_PATH="/mnt/oss-data"
   
   # Download the DeepSeek-R1-Distill-Qwen-7B model from the ModelScope community.
   modelscope download --model deepseek-ai/DeepSeek-R1-Distill-Qwen-7B --local_dir $BUCKET_MOUNT_PATH/DeepSeek-R1-Distill-Qwen-7B

   If the progress is stuck, try pressing Enter multiple times.

3. Verify if the model weight file is valid

   Once the download is complete, run the following command in the terminal to test the model's functionality and verify if the model weight file is complete.

   # The mount directory of the bucket.
   BUCKET_MOUNT_PATH="/mnt/oss-data"
   
   CUDA_VISIBLE_DEVICES=0 swift infer \
       --model $BUCKET_MOUNT_PATH/DeepSeek-R1-Distill-Qwen-7B \
       --stream true \
       --infer_backend pt \
       --max_new_tokens 2048

   Once the model weight file is loaded, as shown in the following figure, you can begin conversing with the large model. If the model weight file fails to load, try downloading it again.

   

   Enter exit once the test is complete.

1.4 Write an automatic training script

1. Write an automatic training script.

   Run the following commands to create an automatic training script and grant it executable permissions. This script will automatically resume training from the most recent checkpoint and monitor the training progress until completion.

   # Create an automatic training script.
   cat <<EOF > /root/train.sh
   #!/bin/bash
   
   # The mount directory of the bucket.
   BUCKET_MONTH_PATH="/mnt/oss-data"
   
   # The storage directory of the model weight file and dataset.
   MODEL_PATH="\$BUCKET_MONTH_PATH/DeepSeek-R1-Distill-Qwen-7B"
   DATASET_PATH="\$BUCKET_MONTH_PATH/alpaca-gpt4-data-zh#500 \$BUCKET_MONTH_PATH/alpaca-gpt4-data-en#500 \$BUCKET_MONTH_PATH/self-cognition#500"
   
   # Set the output directory.
   OUTPUT_DIR="\$BUCKET_MONTH_PATH/output"
   mkdir -p "\$OUTPUT_DIR"
   
   # Confirm that the training has been completed without requiring any operation.
   if [ -f "\$OUTPUT_DIR/logging.jsonl" ]; then
       last_line=\$(tail -n 1 "\$OUTPUT_DIR/logging.jsonl")
       if echo "\$last_line" | grep -q "last_model_checkpoint" && echo "\$last_line" | grep -q "best_model_checkpoint"; then
           echo "Training already completed. Exiting."
           exit 0
       fi
   fi
   
   # Initialize the recovery parameters.
   RESUME_ARG=""
   
   # Find the most recent checkpoint
   LATEST_CHECKPOINT=\$(ls -dt \$OUTPUT_DIR/checkpoint-* 2>/dev/null | head -1)
   
   if [ -n "\$LATEST_CHECKPOINT" ]; then
       RESUME_ARG="--resume_from_checkpoint \$LATEST_CHECKPOINT"
       echo "Resume training from: \$LATEST_CHECKPOINT"
   else
       echo "No checkpoint found. Starting new training."
   fi
   
   # Start the training command.
   CUDA_VISIBLE_DEVICES=0 swift sft \\
       --model \$MODEL_PATH \\
       --train_type lora \\
       --dataset \$DATASET_PATH \\
       --torch_dtype bfloat16 \\
       --num_train_epochs 1 \\
       --per_device_train_batch_size 1 \\
       --per_device_eval_batch_size 1 \\
       --learning_rate 1e-4 \\
       --lora_rank 8 \\
       --lora_alpha 32 \\
       --target_modules all-linear \\
       --gradient_accumulation_steps 16 \\
       --eval_steps 50 \\
       --save_steps 10 \\
       --save_total_limit 5 \\
       --logging_steps 5 \\
       --max_length 2048 \\
       --output_dir "\$OUTPUT_DIR" \\
       --add_version False \\
       --overwrite_output_dir True \\
       --system 'You are a helpful assistant.' \\
       --warmup_ratio 0.05 \\
       --dataloader_num_workers 4 \\
       --model_author swift \\
       --model_name swift-robot \\
       \$RESUME_ARG
   EOF
   
   # Grant the executable permissions.
   chmod +x /root/train.sh

   Verify if the script runs correctly

   Run the following command to verify if the script runs correctly:

   ./train.sh

   As shown in the following figure, if the model loads correctly after you execute the command and the training begins without issues, this confirms that the model file is valid, and the dependencies are complete. To exit the training, press CTRL + C.

   

   Otherwise, reinstall the Python dependencies as outlined in Step 1.1, and download the model weight file and dataset as instructed in Step 1.3.

2. Set up a Linux service and enable auto-start on system boot.

   Run the following commands to create a Linux service and enable the training script to start automatically on system startup:

   # Create a log storage directory.
   mkdir -p /root/train-service-log
   
   # Write a service configuration file.
   cat <<EOF > /etc/systemd/system/train.service
   [Unit]
   Description=Train AI Model Script
   After=network.target local-fs.target remote-fs.target
   Requires=local-fs.target remote-fs.target
   
   [Service]
   ExecStart=/root/train.sh
   WorkingDirectory=/root/
   User=root
   Environment="PATH=/usr/bin:/usr/local/bin"
   Environment="CUDA_VISIBLE_DEVICES=0"
   StandardOutput=append:/root/train-service-log/train.log
   StandardError=append:/root/train-service-log/train_error.log
   
   [Install]
   WantedBy=multi-user.target
   EOF
   
   # Reload the systemd configurations.
   systemctl daemon-reload
   # Enable the training.service to start automatically on system startup.
   systemctl enable train.service

   Executing the commands produces the following result:

   

1.5 Build an image

Once you've completed all the previous steps, build a custom image from your configured instance. This image will serve as the startup template for scaled-out instances, eliminating the need to reinstall dependencies each time.

1. Go to the ECS console.

2. Create an image following the steps shown in the figure below.

   

3. Wait for the image to be created, which typically takes about 5 minutes. You can monitor the progress in the ECS console.

   

2.1 Create a scaling group

To create a scaling group, perform the following steps:

2.2 Create a scaling configuration

A scaling configuration defines the specifications and image of the instances in a scaling group. After you create a scaling configuration, Auto Scaling uses it to automatically launch new instances in the scaling group based on the defined instance settings. To create a scaling configuration, perform the following steps:

Enable the scaling group and scaling configuration as prompted.

• Integrate CloudMonitor to detect interruptions and reclamations

  For production environments, we recommend that you integrate CloudMonitor into your training code to detect and handle spot instance interruptions and reclamations. By saving checkpoints 5 minutes before interruptions or reclamations, you can minimize progress loss when resuming training. The updated solution architecture is as follows:

  

• Create a comprehensive task recovery mechanism

  In the example in this topic, resuming training will automatically start from the most recent checkpoint. However, the validity of the checkpoint is not automatically verified. In practical applications, it's recommended to implement an anomaly detection mechanism to filter out invalid checkpoints and ensure training resumes from the most recent valid one.

• Enhance the conclusion of the training task

  You can integrate the logic for determining the end of training into the training code. Once training is complete, use the CLI or SDK to call an API operation and set the expected number of instances to 0. Auto Scaling will then automatically release any excess instances in the scaling group, preventing unnecessary costs from resource waste.

  You can also report custom events to CloudMonitor once the training is complete. CloudMonitor will then notify you of the training result via email, text message, or DingTalk chatbot.

• Switch to a more efficient storage model

  When training a model with a large number of parameters, OSS can lead to system bottlenecks. To enhance overall system efficiency, we recommend using a high-throughput, low-latency file system, such as CPFS, for mounting.

• Configure multi-zone vSwitches

  If you configure a vSwitch in only one zone, Auto Scaling will be able to create instances in just that zone for the scaling group. This may lead to a scale-out failure if there are not enough resources available in that zone. We recommend configuring vSwitches across multiple zones. When a spot instance is reclaimed, Auto Scaling automatically launches a new spot instance in a different zone. This increases the likelihood of using spot instances.