Object Storage Service:Use an OSS accelerator to speed up model training

Create a GPU-accelerated instance

1. Go to the ECS instance buy page.

2. Click the Custom Launch tab.

3. Configure the instance. For detailed parameter descriptions, see Parameter descriptions. Key settings for this guide:

   • Region: Select a supported region. This guide uses China (Hangzhou).

   • Instance type: ecs.gn6i-c4g1.xlarge

     

   • Image: Ubuntu 22.04. Select Auto-install GPU Driver and set the CUDA version to 12.4.1. CUDA installs automatically when the instance starts.

   

4. Complete the creation.

Connect to the instance

1. On the Instances page in the ECS console, find the instance by region and ID, then click Connect in the Actions column.

   

2. In the Remote connection dialog, click Sign in now in the Workbench section.

   

3. In the Instance Login dialog, set the authentication method to match what you chose when creating the instance. For a key pair, select SSH Key Authentication and upload or paste the private key file.

   The private key file (.pem) was downloaded to your computer automatically when you created the key pair. Check your browser's download history to find it.

4. After login, wait for CUDA to finish installing automatically.

   

Create a bucket and record the internal endpoint

1. On the Buckets page of the OSS console, click Create Bucket.

2. Follow the on-screen instructions to complete the bucket creation.

3. Go to the Overview page of the bucket. In the Port section, record the Access from ECS over the VPC (virtual private cloud) (internal network) endpoint. This endpoint is used to upload datasets and save checkpoints during training.

   

Create an OSS accelerator

1. On the Buckets page, click the bucket name. In the left navigation tree, choose Bucket Settings > OSS Accelerator.

2. Click Create Accelerator. In the Create Accelerator panel, set the capacity (this guide uses 500 GB), then click Next.

3. Set Acceleration Policy to Paths and add the dataset directory in the bucket to the accelerated paths. Click OK and follow the on-screen instructions to complete the creation.

   

4. Record the accelerated endpoint. This endpoint is used to download datasets from the cache during training.

   

Set up the environment

export OSS_ACCESS_KEY_ID=<ACCESS_KEY_ID>
export OSS_ACCESS_KEY_SECRET=<ACCESS_KEY_SECRET>

Upload datasets

The training data used in this guide is a subset of the ImageNet ILSVRC dataset.

1. Download the training and validation sets to the instance:

      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241216/jsnenr/n04487081.tar
      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241218/dxrciv/n10148035.tar
      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241216/senwji/val.tar

2. Extract and organize the datasets:

      tar -zxvf n10148035.tar && tar -zxvf n04487081.tar && tar -zxvf val.tar
      mkdir dataset && mkdir ./dataset/train && mkdir ./dataset/val
      mv n04487081 ./dataset/train/ && mv n10148035 ./dataset/train/ && mv IL*.JPEG ./dataset/val/

3. Run the upload script to send the datasets to the bucket:

      python3 upload_dataset.py

   The script (upload_dataset.py) transforms each image into a tensor and uploads it to the bucket using the internal endpoint. Replace <YourBucketName> with your actual bucket name before running.

      # upload_dataset.py
   
      from torchvision import transforms
      from PIL import Image
      import oss2
      import os
      from oss2.credentials import EnvironmentVariableCredentialsProvider
   
      # Internal endpoint for the China (Hangzhou) region
      OSS_ENDPOINT = "oss-cn-hangzhou-internal.aliyuncs.com"
      OSS_BUCKET_NAME = "<YourBucketName>"
      BUCKET_REGION = "cn-hangzhou"
   
      # Prefix for dataset objects in the bucket
      OSS_URI_BASE = "dataset/imagenet/ILSVRC/Data"
   
      def to_tensor(img_path):
          IMG_DIM_224 = 224
          compose = transforms.Compose([
                  transforms.RandomResizedCrop(IMG_DIM_224),
                  transforms.RandomHorizontalFlip(),
                  transforms.ToTensor(),
                  transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
              ])
          img = Image.open(img_path).convert('RGB')
          img_tensor = compose(img)
          numpy_data = img_tensor.numpy()
          binary_data = numpy_data.tobytes()
          return binary_data
   
      def list_dir(directory):
          for root, _, files in os.walk(directory):
              rel_root = os.path.relpath(root, start=directory)
              for file in files:
                  rel_filepath = os.path.join(rel_root, file) if rel_root != '.' else file
                  yield rel_filepath
   
      # Local dataset root. Structure must match:
      # ./dataset/train/{class_id}/{image}.JPEG
      # ./dataset/val/ILSVRC2012_val_*.JPEG
      IMG_DIR_BASE = "./dataset"
   
      bucket_api = oss2.Bucket(oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider()), OSS_ENDPOINT, OSS_BUCKET_NAME, region=BUCKET_REGION)
   
      for phase in ["val", "train"]:
          IMG_DIR = "%s/%s" % (IMG_DIR_BASE, phase)
          for _, img_relative_path in enumerate(list_dir(IMG_DIR)):
              img_bin_name = img_relative_path.replace(".JPEG", ".pt")
              object_key = "%s/%s/%s" % (OSS_URI_BASE, phase, img_bin_name)
              bucket_api.put_object(object_key, to_tensor("%s/%s" % (IMG_DIR, img_relative_path)))

4. Download the label files used for dataset mapping:

      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241220/izpskr/imagenet_class_index.json
      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241220/lfilrp/ILSVRC2012_val_labels.json

Start model training

The training uses four Python modules. All examples use the accelerated endpoint to load datasets and the internal endpoint to save checkpoints.

oss_dataloader.py — loads ImageNet datasets from the accelerator cache and creates PyTorch data loaders.

# oss_dataloader.py

import json
import numpy as np
from torch.utils.data import DataLoader
import torch

class ImageCls():
    def __init__(self):
        self.__syn_to_class = {}
        self.__syn_to_label = {}
        with open("imagenet_class_index.json", "rb") as f:
            cls_list = json.load(f)
            for cls, v in cls_list.items():
                syn = v[0]
                label = v[1]
                self.__syn_to_class[syn] = int(cls)
                self.__syn_to_label[int(cls)] = label

    def __len__(self):
        return len(self.__syn_to_label)

    def __getitem__(self, syn):
        cls = self.__syn_to_class[syn]
        return cls

class ImageValSet():
    def __init__(self):
        self.__val_to_syn = {}
        with open("ILSVRC2012_val_labels.json", "rb") as f:
            val_syn_list = json.load(f)
            for val, syn in val_syn_list.items():
                self.__val_to_syn[val] = syn

    def __getitem__(self, val):
        return self.__val_to_syn[val]

imageCls = ImageCls()
imageValSet = ImageValSet()

IMG_DIM_224 = 224
OSS_URI_BASE = "oss://<YourBucketName>/dataset/imagenet/ILSVRC/Data"

# Accelerated endpoint — routes reads through the OSS accelerator cache
ENDPOINT = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com"

def obj_to_tensor(object):
    data = object.read()
    numpy_array_from_binary = np.frombuffer(data, dtype=np.float32).reshape([3, IMG_DIM_224, IMG_DIM_224])
    return torch.from_numpy(numpy_array_from_binary)

def train_tensor_transform(object):
    tensor_from_binary = obj_to_tensor(object)
    key = object.key
    syn = key.split('/')[-2]
    return tensor_from_binary, imageCls[syn]

def val_tensor_transform(object):
    tensor_from_binary = obj_to_tensor(object)
    key = object.key
    image_name = key.split('/')[-1].split('.')[0] + ".JPEG"
    return tensor_from_binary, imageCls[imageValSet[image_name]]

def make_oss_dataloader(dataset, batch_size, num_worker, shuffle):
    image_datasets = {
        'train': dataset.from_prefix(OSS_URI_BASE + "/train/", endpoint=ENDPOINT, transform=train_tensor_transform),
        'val': dataset.from_prefix(OSS_URI_BASE + "/val/", endpoint=ENDPOINT, transform=val_tensor_transform),
    }
    dataloaders = {
        'train': DataLoader(image_datasets['train'], batch_size=batch_size, shuffle=shuffle, num_workers=num_worker),
        'val': DataLoader(image_datasets['val'], batch_size=batch_size, shuffle=shuffle, num_workers=num_worker)
    }
    return dataloaders

pre_trained_model.py — initializes a pretrained ResNet-18 model on the GPU.

# pre_trained_model.py

from torchvision import models
import torch.nn as nn
import torch

def make_resnet_model(cls_count=1000):
    device = torch.device("cuda:0")
    model = models.resnet18(pretrained=True)
    num_ftrs = model.fc.in_features
    model.fc = nn.Linear(num_ftrs, cls_count)

    model = model.to(device)
    if torch.cuda.device_count() > 1:
        model = nn.DataParallel(model)

    return model, device

resnet_train.py — trains the model for a given number of epochs and saves the best checkpoint to OSS.

# resnet_train.py

from osstorchconnector import OssCheckpoint
import torch.optim as optim
import torch
import torch.nn as nn

OSS_CHECKPOINT_URI = "oss://<YourBucketName>/checkpoints/resnet18.pt"

# Internal endpoint for writing checkpoints directly to OSS
ENDPOINT = "oss-cn-hangzhou-internal.aliyuncs.com"

def train(model, dataloaders, device, epoch_num):
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    exp_lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
    criterion = nn.CrossEntropyLoss()

    best_acc = 0.0
    for epoch in range(epoch_num):
        for phase in ['train', 'val']:
            if phase == 'train':
                model.train()
            else:
                model.eval()

            running_loss = 0.0
            running_corrects = 0

            # Iterate over batches
            dataset_size = 0
            for (inputs, labels) in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # Backpropagation runs only during the training phase
                    if phase == 'train':
                        optimizer.zero_grad()
                        loss.backward()
                        optimizer.step()

                # Accumulate statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)
                dataset_size += inputs.size(0)

            if phase == 'train':
                exp_lr_scheduler.step()

            epoch_loss = running_loss / dataset_size
            epoch_acc = running_corrects / dataset_size

            print(f'[Epoch {epoch}/{epoch_num - 1}][{phase}] {dataset_size} imgs {epoch_acc}')

            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                # Save the best checkpoint to OSS
                checkpoint = OssCheckpoint(endpoint=ENDPOINT)
                with checkpoint.writer(OSS_CHECKPOINT_URI) as checkpoint_writer:
                    torch.save(model.state_dict(), checkpoint_writer)

main.py — the entry point that wires everything together and starts training.

# main.py

from oss_dataloader import make_oss_dataloader
from pre_trained_model import make_resnet_model
from osstorchconnector import OssMapDataset
from resnet_train import train

NUM_EPOCHS = 30   # Number of training epochs
BATCH_SIZE = 64   # Batch size
NUM_WORKER = 4    # Number of data loader workers

model, device = make_resnet_model()
dataloaders = make_oss_dataloader(OssMapDataset, BATCH_SIZE, NUM_WORKER, True)
train(model, dataloaders, device, NUM_EPOCHS)

Start training:

python3 main.py

Training output appears as each epoch completes: