The OS-bundled kdump service on ECS bare metal instances may fail to generate crash dump files due to instance type and OS incompatibility.
Problem description
Crash dump files cannot be generated after device hot-swap on the following instances:
-
Sixth-generation ECS Bare Metal Instance families (including ebmg6, ebmc6, and ebmr6) with the following images:
-
CentOS 8.3 or earlier images
-
Ubuntu 16/18 images
-
Debian 10 images
-
Alibaba Cloud Linux 2 images with a kernel version earlier than 4.19.91-24.al7
-
-
Seventh-generation ECS Bare Metal Instance families (including ebmg7, ebmc7, and ebmr7) with Debian 10 images.
Cause
During the crashkernel phase on an ECS instance, Kdump fails to allocate the pci_resource to the vda block device. As a result, crash dump generation fails. The root cause is an incompatibility between the instance type and the operating system.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=33ead8c1-41b6-4f85-84ec-xxx does not exist. Dropping to a shell!
Solution
Use one of the following methods to resolve this issue:
-
Upgrade the OS kernel to version 5.10.
-
Add the following patches to the OS and rebuild the kernel:
Benjamin Herrenschmidt (1): PCI: Don't auto-realloc if we're preserving firmware config Kelsey Skunberg (1): PCI: Make pci_hotplug_io_size, mem_size, and bus_size private Logan Gunthorpe (1): PCI: Don't disable bridge BARs when assigning bus resources Nicholas Johnson (2): PCI: Add "pci=hpmmiosize" and "pci=hpmmioprefsize" parameters PCI: Avoid double hpmemsize MMIO window assignmentNoteSome kernel versions already include some of these patches. Add only the missing ones. For example, Debian 10 kernel 4.19 includes the first and third patches. You must add the second, fourth, and fifth patches for kdump to work.
Note the following items:
For Debian or Ubuntu kernels, modify the crashkernel parameter to adjust memory reserved for kdump. Recommended settings:
crashkernel=0M-2G:0M,2G-8G:192M,8G-:256M
For some OS images such as Debian 10, you must increase the reserved memory to 384 MB or 512 MB. Otherwise, an OOM exception occurs during the crashkernel phase and crash dump files cannot be generated.
To reserve 256 MB of memory when total system memory exceeds 384 MB, adjust the crashkernel parameter as follows:
-
Open the /kdump-tools.cfg file.
vim /etc/default/grub.d/kdump-tools.cfg -
Press I to enter Insert mode and modify the crashkernel parameter:
GRUB_CMDLINE_LINUX_DEFAULT="$GRUB_CMDLINE_LINUX_DEFAULT crashkernel=384M-:256M"Press Esc, enter
:wq, and press Enter to save and close the file. -
Update the GRUB configuration.
update-grub -
Restart the ECS instance for the configuration to take effect.
Restart ECS instances during off-peak hours to minimize business impact. See Restart an instance.