:Troubleshoot virtio hot-unplug Oops exception

Symptoms

Hot-unplugging a virtio device, such as a disk or NIC, from an ECS instance running a specific recent kernel version can cause an Oops exception, resulting in one of the following outcomes:

• If the instance is configured withkernel.panic_on_oops = 1, a kernel panic occurs.

• If the instance is configured withkernel.panic_on_oops = 0, the kernel becomes unresponsive.

Cause

The Linux upstream community introduced this issue in this commit, which added support for the admin virtqueue for virtio devices.

This commit introduced the following changes:

• An is_avq function pointer was added to the virtio_pci_device definition to check for the existence of an admin virtqueue.

• The virtio_pci_modern_probe function, which is responsible for initializing modern virtio devices, assigns a value to the is_avq function pointer.

  @@ -588,6 +658,7 @@ int virtio_pci_modern_probe(struct virtio_pci_device *vp_dev)
   	vp_dev->config_vector = vp_config_vector;
   	vp_dev->setup_vq = setup_vq;
   	vp_dev->del_vq = del_vq;
  +	vp_dev->is_avq = vp_is_avq;
   	vp_dev->isr = mdev->isr;
   	vp_dev->vdev.id = mdev->id;

• When a virtio device is hot-unplugged, the code checks whether the current queue is an admin virtqueue.

  @@ -236,6 +236,9 @@ void vp_del_vqs(struct virtio_device *vdev)
          int i;
          list_for_entry_safe(vq, n, &vdev->vqs, list) {
  +               if (vp_dev->is_avq(vdev, vq->index))
  +                       continue;
  +
                  if (vp_dev->per_vq_vectors) {
                          int v = vp_dev->vqs[vq->index]->msix_vector;
                  }
          }

However, for legacy virtio devices, the is_avq function pointer is not initialized and remains a null pointer. As a result, when you hot-unplug a legacy virtio device, the call toif (vp_dev->is_avq(vdev, vq->index)) attempts to dereference a null pointer. This triggers an exception, which can cause a system crash.

Affected scope

• Linux upstream community

  The upstream community has resolved this issue in this commit. The fix adds a null check for the is_avq function pointer before calling it.

• Operating system

  • Ubuntu 24

  • Other operating systems with kernel versions around 6.8 that have incorporated admin virtqueue support but lack the is_avq null check patch.

    Note

    You can run theuname -r command to check the kernel version.

• virtio device

  This issue affects legacy virtio devices when they are hot-unplugged from ECS instances.

Solutions

• Solution 1: Switch to an instance family that uses modern virtio devices, such as 8th-generation or later instance families. These instance families are not affected by this issue. For more information, see Change instance types. To learn more about instance families, see Instance families.

• Solution 2:

  1. Upgrade to the latest kernel package and verify that it includes the virtio-pci: Check if is_avq is NULL patch.

  2. (Conditionally required) If the latest kernel does not include the patch, you must apply it manually.

Appendix: Terms

This section explains key terms used in this topic, such as virtio device, admin virtqueue, and virtio_pci_device.