Linux kernels earlier than 4.15 may trigger a softlockup error during file cache writeback. Upgrade the kernel to resolve the issue.
Symptom
On a Linux ECS instance with a kernel version earlier than 4.15, a softlockup error occurs during file cache writeback. A call stack similar to the following is generated.
Run uname -r to check the kernel version. For example, if the output is 3.10.0-514.26.2.el7.x86_64, the kernel version is 3.10.0. In this topic, version 4.15 refers to 4.15.0.
[3507707.671883] [<ffffffff8127cf7a>] redirty_tail+0x3a/0x40
[3507707.671884] [<ffffffff81280ea4>] __writeback_inodes_wb+0x64/0xc0
[3507707.671885] [<ffffffff81281238>] wb_writeback+0x268/0x300
[3507707.671887] [<ffffffff812819f4>] wb_workfn+0xb4/0x380
[3507707.671889] [<ffffffff810a5dc9>] process_one_work+0x189/0x420
[3507707.671890] [<ffffffff810a625b>] worker_thread+0x1fb/0x4b0
[3507707.671891] [<ffffffff810a6060>] ? process_one_work+0x420/0x420
[3507707.671893] [<ffffffff810ac696>] kthread+0xe6/0x100
[3507707.671894] [<ffffffff810ac5b0>] ? kthread_park+0x60/0x60
[3507707.671897] [<ffffffff81741dd9>] ret_from_fork+0x39/0x50
Cause
When ECS instance memory is insufficient, the kernel frequently calls wakeup_flusher_threads, creating excessive writeback tasks (wb_writeback_work). The writeback threads become stuck processing these tasks, triggering a softlockup error.
Solution
Update the kernel to a version later than 4.15. Alibaba Cloud Linux is unaffected because its kernel version is 4.19. The following steps apply to other Linux distributions.
Before you proceed, create snapshots to back up instance data to prevent data loss. See Create snapshot manually.
-
Connect to the Linux instance.
-
Check the kernel version:
uname -r -
If the kernel version is 4.15 or earlier, update it:
If the kernel version is later than 4.15, the softlockup error does not apply. No further action is needed.
yum update kernelNoteIf
yum update kernelfails for your kernel version, download a kernel RPM package later than 4.15 and upgrade manually. -
Restart the instance after the kernel update.
reboot -
After the restart, verify that the kernel version is later than 4.15:
uname -r