What do I do if Persistence Mode that I enabled does not take effect and the ECC status or the MIG feature fails to be configured after a GPU-accelerated instance is restarted? - Elastic GPU Service

Starting with Tesla driver version 535, NVIDIA deprecated the nvidia-smi -pm 1 method for enabling Persistence Mode. Settings applied with nvidia-smi -pm 1 are not preserved across restarts on instances running Tesla 535 or later. This is why Persistence Mode, ECC status, and multi-instance GPU (MIG) configuration all revert after a reboot — all three issues share the same root cause: the legacy Persistence Mode mechanism no longer keeps the GPU initialized in newer driver versions.

The fix is to switch to the NVIDIA Persistence Daemon (nvidia-persistenced), which is the supported replacement.

Confirm the issue

Run the following command to check your kernel logs:

dmesg | grep -i persistence

If the output contains the following message, your instance is affected:

NVRM: Persistence mode is deprecated and will be removed in a future release. Please use nvidia-persistenced instead.

Solution

Follow the "Enable Persistence Mode by using the NVIDIA Persistence Daemon" step in the Manually install the Tesla driver on a GPU-accelerated compute-optimized Linux instance topic to switch from nvidia-smi -pm 1 to nvidia-persistenced.

After completing the steps in that guide, Persistence Mode, ECC status, and MIG configuration will be preserved across instance restarts.