使用OSS加速器提升模型訓練速度

步驟一：建立GPU雲端服務器

以下步驟旨在建立並串連一個適用於模型訓練任務的GPU雲端服務器執行個體。該執行個體的規格為ecs.gn6i-c4g1.xlarge，作業系統為Ubuntu 22.04、CUDA版本為12.4.1。需注意，自訂執行個體配置時，CUDA版本請選擇最新版本。

步驟二：建立OSS Bucket並開通OSS加速器

以下步驟旨在與目標GPU雲端服務器同一地區下，建立一個用於儲存資料集的Bucket並開啟該Bucket的OSS加速器空間，以提高資料集的訪問速度。需注意，雲端服務器和Bucket處於同一地區且通過內網網域名稱訪問不會產生任何流量費用。

1. 建立Bucket並擷取內網網域名稱

   重要

   OSS加速器功能目前在華東1（杭州）、華東2（上海）、華北2（北京）、華北6（烏蘭察布）、華南1（深圳）以及新加坡地區開放公測。請確保您建立的Bucket位於上述地區之一，並且與目標GPU雲端服務器處於同一地區。本文將以華東1（杭州）地區為例進行說明。

   1. 進入Object Storage Service控制台的Bucket列表頁面，單擊建立 Bucket。

   2. 在建立 Bucket面板，依據面板所給出的提示資訊來完成Bucket的建立工作。

   3. 在目標Bucket概覽頁面，訪問連接埠地區，複製ECS 的 VPC 網絡訪問（內網）的Endpoint留作備用，以便在後續訓練時上傳資料集和checkpoint到目標Bucket。

2. 開通OSS加速器並擷取加速器網域名稱

   1. 進入Object Storage Service控制台的Bucket列表頁面，選擇目標Bucket後在左側導覽列，選擇Bucket 配置 > OSS 加速器進入OSS 加速器頁面。

   2. 單擊建立OSS加速器，在建立OSS加速器面板設定加速器容量，本次實驗以配置500 GB為例。單擊下一步。

   3. 選擇指定路徑加速，並配置加速路徑為資料集所在目錄後單擊確定，然後根據頁面指引完成加速器的建立。

      加速路徑填寫 dataset/，並勾選同步預熱。開啟同步預熱後，用戶端通過加速器網域名稱寫入的資料會同時寫入 OSS Bucket 和 OSS 加速器，再次讀取時可獲得更低延遲。

   4. 在OSS加速器介面複製加速器網域名稱留作備用，以便在後續訓練時用於從目標Bucket下載資料集。

步驟三：訓練模型

以下步驟旨在GPU雲端服務器上，完成模型訓練環境的配置、資料集的上傳，以及使用OSS加速器網域名稱來加速模型訓練。

1. 配置訓練環境

   1. 準備conda環境並配置依賴項。

      1. 執行以下命令，下載並安裝conda。

         curl -L https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o /tmp/miniconda.sh && bash /tmp/miniconda.sh -b -p /opt/conda/ && rm /tmp/miniconda.sh && /opt/conda/bin/conda clean -tipy && export PATH=/opt/conda/bin:$PATH  && conda init bash && source ~/.bashrc && conda update conda 

      2. 執行vim environment.yaml命令，建立並開啟名為environment.yaml的conda環境檔案，添加以下配置後儲存。

         name: py312
         channels:
           - defaults
           - conda-forge
           - pytorch
         dependencies:
           - python=3.12
           - pytorch>=2.5.0
           - torchvision 
           - torchaudio 
           - transformers 
           - torchdata
           - oss2

      3. 執行以下命令，基於conda環境檔案建立名為py312的conda環境。

         conda env create -f environment.yaml

      4. 執行conda activate py312命令，啟用名為py312的conda環境。即已成功啟用。

         重要

         後續的操作步驟，請在已啟用的conda環境中繼續進行。

   2. 配置環境變數。

      執行以下命令，配置上傳資料集時所需的訪問憑證。請注意，命令中的<ACCESS_KEY_ID>、<ACCESS_KEY_SECRET>請分別替換為RAM使用者的AccessKey ID、AccessKeySecret。有關如何建立AccessKey ID和AccessKeySecret請參見建立AccessKey。

      export OSS_ACCESS_KEY_ID=<ACCESS_KEY_ID>
      export OSS_ACCESS_KEY_SECRET=<ACCESS_KEY_SECRET>

   3. 配置OSS Connector。

      1. 執行以下命令，安裝OSS Connector。

         pip install osstorchconnector

      2. 執行以下命令，建立訪問憑證設定檔。

         mkdir -p /root/.alibabacloud && touch /root/.alibabacloud/credentials

      3. 執行vim /root/.alibabacloud/credentials命令，開啟設定檔添加如下配置並儲存。有關OSS Connector更多配置，請參見配置OSS Connector for AI/ML。

         使用AccessKey ID和AccessKey Secret作為訪問憑證的配置樣本，樣本中的<AccessKeyId>、<AccessKeySecret>請分別替換為RAM使用者的AccessKey ID、AccessKeySecret。如何建立AccessKey ID和AccessKeySecret請參見建立AccessKey。

         {
           "AccessKeyId": "LTAI************************",
           "AccessKeySecret": "At32************************"
         }

      4. 執行以下命令，設定credentials檔案唯讀許可權，以保障AK、SK密鑰安全。

         chmod 400 /root/.alibabacloud/credentials

      5. 執行以下命令，建立OSS Connector設定檔。

         mkdir -p /etc/oss-connector/ && touch /etc/oss-connector/config.json

      6. 執行vim /etc/oss-connector/config.json命令，開啟設定檔添加如下配置並儲存。正常情況下使用此預設配置即可。

         {
             "logLevel": 1,
             "logPath": "/var/log/oss-connector/connector.log",
             "auditPath": "/var/log/oss-connector/audit.log",
             "datasetConfig": {
                 "prefetchConcurrency": 24,
                 "prefetchWorker": 2
             },
             "checkpointConfig": {
                 "prefetchConcurrency": 24,
                 "prefetchWorker": 4,
                 "uploadConcurrency": 64
             }
         }
                         

2. 準備資料

   1. 上傳訓練資料集和驗證資料集到目標Bucket。

      1. 執行以下命令下載訓練資料集和驗證資料集至雲端服務器。需注意，此資料集並非真實實驗情境所採用資料集，僅作測試用途。

         wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241216/jsnenr/n04487081.tar
         wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241218/dxrciv/n10148035.tar
         wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241216/senwji/val.tar

      2. 執行以下命令解壓下載好的資料集，並在當前路徑下建立dataset目錄將資料集放入其中。

         tar -zxvf n10148035.tar && tar -zxvf n04487081.tar && tar -zxvf val.tar
         mkdir dataset && mkdir ./dataset/train && mkdir ./dataset/val
         mv n04487081 ./dataset/train/ && mv n10148035 ./dataset/train/ && mv IL*.JPEG ./dataset/val/

      3. 執行python3 upload_dataset.py命令運行此指令碼，將解壓好的資料集上傳至指定的Bucket中。

         # upload_dataset.py
         from torchvision import transforms
         from PIL import Image
         import oss2
         import os
         from oss2.credentials import EnvironmentVariableCredentialsProvider
         # # 以杭州地區內網網域名稱為例
         OSS_ENDPOINT = "oss-cn-hangzhou-internal.aliyuncs.com"    #OSS內網訪問網域名稱 
         OSS_BUCKET_NAME = "<YourBucketName>"    #目標Bucket名稱 
         BUCKET_REGION = "cn-hangzhou"    #目標Bucket地區 
         # OSS_URI_BASE: 自訂OSS bucket中的儲存首碼
         OSS_URI_BASE = "dataset/imagenet/ILSVRC/Data"
         def to_tensor(img_path):
             IMG_DIM_224 = 224
             compose = transforms.Compose([
                     transforms.RandomResizedCrop(IMG_DIM_224),
                     transforms.RandomHorizontalFlip(),
                     transforms.ToTensor(),
                     transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
                 ])
             img = Image.open(img_path).convert('RGB')
             img_tensor = compose(img)
             numpy_data = img_tensor.numpy()
             binary_data = numpy_data.tobytes()
             return binary_data
         def list_dir(directory):
             for root, _, files in os.walk(directory):
                 rel_root = os.path.relpath(root, start=directory)
                 for file in files:
                     rel_filepath = os.path.join(rel_root, file) if rel_root != '.' else file
                     yield rel_filepath
         IMG_DIR_BASE = "./dataset" 
         """
             IMG_DIR_BASE是本地存放圖片的路徑，絕對路徑或相對路徑都可以
             該路徑下結構應該和實際資料集結構一致，具體結構如下
             {IMG_DIR_BASE}/
                 train/
                     n10148035/
                         n10148035_10034.JPEG
                         n10148035_10217.JPEG
                         ... 
                     n11879895/
                         n11879895_10016.JPEG
                         n11879895_10019.JPEG
                         ...
                     ...
                 val/
                     ILSVRC2012_val_00000001.JPEG
                     ILSVRC2012_val_00000002.JPEG
                     ...
         """
         bucket_api = oss2.Bucket(oss2.ProviderAuthV4(EnvironmentVariableCredentialsProvider()), OSS_ENDPOINT, OSS_BUCKET_NAME, region=BUCKET_REGION)
         for phase in [ "val", "train"]:
             IMG_DIR = "%s/%s" % (IMG_DIR_BASE, phase)
             for _, img_relative_path in enumerate(list_dir(IMG_DIR)):
                 img_bin_name = img_relative_path.replace(".JPEG", ".pt")
                 object_key = "%s/%s/%s" % (OSS_URI_BASE, phase, img_bin_name)
                 bucket_api.put_object(object_key, to_tensor("%s/%s" % (IMG_DIR,img_relative_path)))

   2. 下載映像資料集標籤檔案，用於構建資料集的分類映射關係。

      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241220/izpskr/imagenet_class_index.json
      wget https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241220/lfilrp/ILSVRC2012_val_labels.json

3. 訓練流程

   1. 構建用於處理ImageNet資料集的工具模組。該模組主要通過OSS加速器網域名稱，從OSS加速器空間下載資料集並構建dataloader。

      oss_dataloader.py

      # oss_dataloader.py
      import json
      import numpy as np
      from torch.utils.data import DataLoader
      import torch
      class ImageCls():
          def __init__(self):
              self.__syn_to_class = {}
              self.__syn_to_label = {}
              with open("imagenet_class_index.json", "rb") as f:
                  cls_list = json.load(f)
                  for cls, v in cls_list.items():
                      syn = v[0]
                      label = v[1]
                      self.__syn_to_class[syn] = int(cls)
                      self.__syn_to_label[int(cls)] = label
          def __len__(self):
              return len(self.__syn_to_label)
          def __getitem__(self, syn):
              cls = self.__syn_to_class[syn]
              return cls
      class ImageValSet():
          def __init__(self):
              self.__val_to_syn = {}
              with open("ILSVRC2012_val_labels.json", "rb") as f:
                  val_syn_list = json.load(f)
                  for val, syn in val_syn_list.items():
                      self.__val_to_syn[val] = syn
          def __getitem__(self, val):
              return self.__val_to_syn[val]
      imageCls = ImageCls()
      imageValSet = ImageValSet()
      IMG_DIM_224 = 224
      OSS_URI_BASE = "oss://<YourBucketName>/dataset/imagenet/ILSVRC/Data"
      # OSS加速器網域名稱, 用來下載資料集  請替換為目標OSS加速器空間網域名稱 
      ENDPOINT = "cn-hangzhou-j-internal.oss-data-acc.aliyuncs.com" 
      def obj_to_tensor(object):
          data = object.read()
          numpy_array_from_binary = np.frombuffer(data, dtype=np.float32).reshape([3, IMG_DIM_224, IMG_DIM_224])
          return torch.from_numpy(numpy_array_from_binary)
      def train_tensor_transform(object):
          tensor_from_binary = obj_to_tensor(object)
          key = object.key
          syn = key.split('/')[-2]
          return tensor_from_binary, imageCls[syn]
      def val_tensor_transform(object):
          tensor_from_binary = obj_to_tensor(object)
          key = object.key
          image_name = key.split('/')[-1].split('.')[0] + ".JPEG"
          return tensor_from_binary, imageCls[imageValSet[image_name]]
      def make_oss_dataloader(dataset, batch_size, num_worker, shuffle):
          image_datasets = {
              'train': dataset.from_prefix(OSS_URI_BASE + "/train/", endpoint=ENDPOINT, transform=train_tensor_transform),
              'val': dataset.from_prefix(OSS_URI_BASE + "/val/", endpoint=ENDPOINT, transform=val_tensor_transform),
          }
          dataloaders = {
              'train': DataLoader(image_datasets['train'], batch_size=batch_size, shuffle=shuffle, num_workers=num_worker),
              'val': DataLoader(image_datasets['val'], batch_size=batch_size, shuffle=shuffle, num_workers=num_worker)
          }
          return dataloaders

   2. 構建預訓練ResNet18模型初始化工具模組。

      pre_trained_model.py

      # pre_trained_model.py
      from torchvision import models
      import torch.nn as nn
      import torch
      def make_resnet_model(cls_count=1000):
          device = torch.device("cuda:0")
          model = models.resnet18(pretrained=True)
          num_ftrs = model.fc.in_features
          model.fc = nn.Linear(num_ftrs, cls_count)
          model = model.to(device)
          if torch.cuda.device_count() > 1:
              model = nn.DataParallel(model)
          return model, device

   3. 構建用於訓練ResNet模型的工具模組。該模組通過給定的模型（model）、資料載入器（dataloaders），以及訓練輪數（epoch_num）開展模型訓練工作。

      resnet_train.py

      # resnet_train.py
      from osstorchconnector import OssCheckpoint
      import torch.optim as optim
      import torch
      import torch.nn as nn
      OSS_CHECKPOINT_URI = "oss://<YourBucketName>/checkpoints/resnet18.pt"
      # OSS內網網域名稱 
      ENDPOINT = "oss-cn-hangzhou-internal.aliyuncs.com" 
      def train(model, dataloaders, device, epoch_num):
          optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
          exp_lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)
          criterion = nn.CrossEntropyLoss()
          best_acc = 0.0
          for epoch in range(epoch_num):
              for phase in ['train', 'val']:
                  if phase == 'train':
                      model.train()
                  else:
                      model.eval()
                  running_loss = 0.0
                  running_corrects = 0
                  # 遍曆資料
                  dataset_size = 0
                  for (inputs, labels) in dataloaders[phase]:
                      inputs = inputs.to(device)
                      labels = labels.to(device)
                      with torch.set_grad_enabled(phase == 'train'):
                          outputs = model(inputs)
                          _, preds = torch.max(outputs, 1)
                          loss = criterion(outputs, labels)
                          # 僅在訓練階段反向傳播和最佳化
                          if phase == 'train':
                              optimizer.zero_grad()
                              loss.backward()
                              optimizer.step()
                      # 統計
                      running_loss += loss.item() * inputs.size(0)
                      running_corrects += torch.sum(preds == labels.data)
                      dataset_size += inputs.size(0)
                  if phase == 'train':
                      exp_lr_scheduler.step()
                  epoch_loss = running_loss / dataset_size
                  epoch_acc = running_corrects / dataset_size
                  print(f'[Epoch {epoch}/{epoch_num - 1}][{phase}] {dataset_size} imgs {epoch_acc}')
                  if phase == 'val' and epoch_acc > best_acc:
                      best_acc = epoch_acc
                      # checkpoint上傳到OSS
                      checkpoint = OssCheckpoint(endpoint=ENDPOINT)
                      with checkpoint.writer(OSS_CHECKPOINT_URI) as checkpoint_writer:
                          torch.save(model.state_dict(), checkpoint_writer)

   4. 構建用於整合模型訓練流程的主指令檔。該檔案整合以上多個工具模組開啟模型訓練。

      main.py

      # main.py
      from oss_dataloader import make_oss_dataloader
      from pre_trained_model import make_resnet_model
      from osstorchconnector import OssMapDataset
      from resnet_train import train
      # 訓練基本參數
      NUM_EPOCHS = 30 # epoch number
      BATCH_SIZE = 64 # batch size
      NUM_WORKER = 4 # dataloader worker number
      # 使用預訓練的resnet18模型
      model, device = make_resnet_model()
      # 使用OssMapDataset資料集構造Dataloader
      dataloaders = make_oss_dataloader(OssMapDataset, BATCH_SIZE, NUM_WORKER, True)
      # 調用訓練主流程
      train(model, dataloaders, device, NUM_EPOCHS)

   5. 執行python3 main.py命令來開始模型訓練，如下所示，已成功啟動訓練流程。

      (py312) root@xxx :~# python3 main.py
      /opt/conda/envs/py312/lib/python3.12/site-packages/torchvision/models/_utils.py:208: UserWarning: The parameter 'pretrained' is deprecated since 0.13 and may be removed in the future, please use 'weights' instead.
        warnings.warn()
      /opt/conda/envs/py312/lib/python3.12/site-packages/torchvision/models/_utils.py:223: UserWarning: Arguments other than a weight enum or `None` for 'weights' are deprecated since 0.13 and may be removed in the future. The current behavior is equivalent to passing `weights=ResNet18_Weights.IMAGENET1K_V1`. You can also use `weights=ResNet18_Weights.DEFAULT` to get the most up-to-date weights.
        warnings.warn(msg)
      Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /root/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
      100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 44.7M/44.7M [00:10<00:00, 4.40MB/s]
      2024/12/20 13:36:34.853415| INFO |th=00000000000000000|dataset.cpp:222|new_oss_dataset:new oss dataset, uuid: 92251a5f-7f00-45b5-aaba-b8894ab15b0d id: 0, total: 1, pid: 20655, endpoint: cn-hangzhou-internal.oss-data-acc.aliyuncs.com
      2024/12/20 13:36:34.853437| INFO |th=00000000000000000|dataset.cpp:230|new_oss_dataset:[cred_path=/root/.alibabacloud/credentials][config_path=/etc/oss-connector/config.json]
      2024/12/20 13:36:34.853518| INFO |th=00000000000000000|dataset.cpp:255|new_oss_dataset:set log level: 1
      2024/12/20 13:36:34.853591| INFO |th=00000000000000000|dataset.cpp:262|new_oss_dataset:set log path: /var/log/oss-connector/connector_log_0
      /root/oss_dataloader.py:51: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /home/conda/feedstock_root/build_artifacts/libtorch_1733624414920/work/torch/csrc/utils/tensor_numpy.cpp:206.)
        return torch.from_numpy(numpy_array_from_binary)
      /root/oss_dataloader.py:51: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /home/conda/feedstock_root/build_artifacts/libtorch_1733624414920/work/torch/csrc/utils/tensor_numpy.cpp:206.)
        return torch.from_numpy(numpy_array_from_binary)
      /root/oss_dataloader.py:51: UserWarning: The given NumPy array is not writable, and PyTorch does not support non-writable tensors. This means writing to this tensor will result in undefined behavior. You may want to copy the array to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /home/conda/feedstock_root/build_artifacts/libtorch_1733624414920/work/torch/csrc/utils/tensor_numpy.cpp:206.)
        return torch.from_numpy(numpy_array_from_binary)

4. 結果驗證

   進入Bucket列表選擇目標Bucket，單擊文件管理 > 檔案清單後查看checkpoints目錄下的resnet18.pt檔案。訓練結束後checkpoint檔案已成功上傳至OSS。