This topic describes how to deploy a YAML file to create containers that share one graphics processing unit (GPU). After you deploy the file, you can use the cGPU solution to isolate GPU memory used by each container. This ensures more efficient use of GPU resources.
Prerequisites
Procedure
Result
- Run the following command to view the log entries that record YAML file deployment
in Step 2.
The log entries provide more information about whether the GPU memory is isolated by cGPU.
kubectl logs binpack-0 --tail=1 2020-03-13 09:14:13.931003: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2832 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:07.0, compute capability: 7.5)
The command output indicates that the container applies for GPU memory of 2,832 MiB.
- Run the following command to log on to the container and view the memory that is allocated
to the container:
kubectl exec -it binpack-0 nvidia-smi Fri Mar 13 09:32:18 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla T4 On | 00000000:00:07.0 Off | 0 | | N/A 41C P0 26W / 70W | 3043MiB / 3231MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| +-----------------------------------------------------------------------------+
The command output indicates that the memory allocated to the container is 3,231 MiB (3 x 1,024 = 3,072).
- Run the following command to view the total memory of the host:
nvidia-smi Fri Mar 13 17:36:24 2020 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 418.87.01 Driver Version: 418.87.01 CUDA Version: 10.1 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Tesla T4 On | 00000000:00:07.0 Off | 0 | | N/A 40C P0 26W / 70W | 3053MiB / 15079MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 8796 C python3 3043MiB | +-----------------------------------------------------------------------------+
The command output indicates that the total memory of the host is 15,079 MiB and the memory that is allocated to the container is 3,053 MiB.