Intelligent Computing LINGJUN:Treinamento do ResNet50 com KSpeed

docker run -it --gpus all --name=resnet50_kspeed_test --net=host --ipc host --device=/dev/infiniband/ --ulimit memlock=-1:-1 -v /{path-to-imagenet}:/{path-to-imagenet-in-docker} -v /{path-to-DeepLearningExamples}:/{path-to-DeepLearningExamples-in-docker} eflo-registry.cn-beijing.cr.aliyuncs.com/eflo/ngc-pytorch-kspeed-22.05-py38:v2.2.0

imagenet
├── train
│   ├── n01440764
│   │  ├── n01440764_10026.JPEG
│   │  ├── n01440764_10027.JPEG
│   │  └── ......
│   ├── n01443537
│   └── ......         
└── val                
    ├── n01440764
    │  ├── ILSVRC2012_val_00000293.JPEG
    │  ├── ILSVRC2012_val_00002138.JPEG
    │  └── ......
    ├── n01443537
    └── ......       

#Stay in the DeepLearningExamples directory
cd ./PyTorch/Classification/ConvNets

#Single-machine eight-GPU baseline
bash ./resnet50v1.5/training/AMP/DGXA100_resnet50_AMP_multi.sh pytorch {path-to-imagenet-in-docker}

#Single-machine eight-GPU kspeed
bash ./resnet50v1.5/training/AMP/DGXA100_resnet50_AMP_multi.sh kspeed {path-to-imagenet-in-docker}

#Single-machine eight-GPU dali+kspeed
bash ./resnet50v1.5/training/AMP/DGXA100_resnet50_AMP_multi.sh dali-kspeed {path-to-imagenet-in-docker}

Adição do arquivo do módulo kspeeddataloader

Adicione o arquivo DeepLearningExamples/PyTorch/Classification/ConvNets/image_classification/kspeeddataloader.py, que implementa um Dataloader do PyTorch e um Dataloader do Dali baseados no KSpeed.

Dataloader do PyTorch baseado no KSpeed

Para implementar um Dataloader do PyTorch baseado no KSpeed, modifique o Dataset e integre-o ao Sampler e ao Dataloader nativos do PyTorch:

• Importe o módulo kspeeddataset

  import kspeed.utils.data.kspeeddataset as KSpeedDataset

• Substitua torchvison.datasets.ImageFolder por KSpeedDataset.KSpeedImageFolder para ativar a aceleração de carregamento de dados do KSpeed

  train_dataset = KSpeedDataset.KSpeedImageFolder(
          traindir, None, workers, kspeed_iplist,
          "admin", "admin", transforms.Compose(transforms_list),
      )
  
  val_dataset = KSpeedDataset.KSpeedImageFolder(
          valdir, None, workers, kspeed_iplist,
          "admin", "admin",
          transforms.Compose(
              [
                  transforms.Resize(
                      image_size + crop_padding, interpolation=interpolation
                  ),
                  transforms.CenterCrop(image_size),
              ]
          ),
      )

• Implemente os métodos get_kspeed_train_loader e get_kspeed_val_loader. Consulte as linhas 16–72 e 74–128 do arquivo kspeeddataloader.py para obter detalhes.

Dataloader do Dali baseado no KSpeed

Para implementar um Dataloader do Dali baseado no KSpeed, configure a fonte de dados de entrada do pipeline do Dali como KSpeedCallable:

• KSpeedCallable

  O KSpeedCallable herda de KSpeedDataset.KSpeedFolder e lê amostras do conjunto de dados ImageNet por meio de self.dataset.getBIN(path) (consulte as linhas 164–179 do arquivo kspeeddataloader.py).

  def __call__(self, sample_info):
          
      if self.dataset is None:
          self.load()
      if sample_info.iteration >= self.full_iters:
          raise StopIteration()
      if self.last_seen_epoch != sample_info.epoch_idx:
          self.last_seen_epoch = sample_info.epoch_idx
          self.perm = np.random.default_rng(seed=42 + sample_info.epoch_idx).permutation(len(self.files))
      idx = self.perm[sample_info.idx_in_epoch + self.shard_offset]
          
      path = os.path.join(self.root, self.files[idx])
      dout = self.dataset.getBIN(path)
      sample = np.frombuffer(dout, dtype=np.uint8)
      label = np.int32([self.labels[idx]])
      return sample, label

• Pipeline do Dali baseado no KSpeedCallable

  As linhas 223–229 do arquivo kspeeddataloader.py utilizam o KSpeedCallable como fonte de dados externa para o Pipeline do Dali:

  if kspeed:
      images, labels = fn.external_source(source=kscallable,
                          num_outputs=2,
                          batch=False, 
                          parallel=True, 
                          dtype=[types.UINT8, types.INT32], 
                          device='cpu')

Adição de opções em DATA_BACKEND_CHOICES

Na linha 40 do arquivo DeepLearningExamples/PyTorch/Classification/ConvNets/image_classification/dataloaders.py, altere o código original DATA_BACKEND_CHOICES = ["pytorch", "synthetic"] conforme abaixo:

DATA_BACKEND_CHOICES = ["pytorch", "syntetic", "kspeed", "dali-kspeed",  "dali"]

Adição da opção args.data_backend

No arquivo DeepLearningExamples/PyTorch/Classification/ConvNets/main.py, adicione o seguinte código à ramificação args.data_backend nas linhas 512–520:

elif args.data_backend == "kspeed":
    get_train_loader = get_kspeed_train_loader
    get_val_loader = get_kspeed_val_loader
elif args.data_backend == "dali":
    get_train_loader = get_dali_kspeed_train_loader(dali_cpu=True, kspeed=False)
    get_val_loader = get_dali_kspeed_val_loader(dali_cpu=True, kspeed=False)
elif args.data_backend == "dali-kspeed":
    get_train_loader = get_dali_kspeed_train_loader(dali_cpu=True)
    get_val_loader = get_dali_kspeed_val_loader(dali_cpu=True)