Resolve a softlockup error caused by deleting cgroups - Elastic Compute Service - Alibaba Cloud -

Fix the softlockup error caused by deleting cgroups in an ECS instance by upgrading the kernel version.

Problem description

A softlockup error occurs when you delete cgroups in an ECS instance, with a call stack similar to the following:

[3302742.447940] Kernel panic - not syncing: softlockup: hung tasks
[3302742.448677] CPU: 18 PID: 1 Comm: systemd Kdump: loaded Tainted: G OEL ------------ T 3.10.0-862.14.4.el7.x86_64 #1
[3302742.450167] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS 8a46cfe 04/01/2014
[3302742.462123] [] mem_cgroup_reparent_charges+0x16d/0x3c0
[3302742.463243] [] mem_cgroup_css_offline+0x84/0x140
[3302742.464327] [] cgroup_destroy_locked+0xea/0x370
[3302742.465414] [] cgroup_rmdir+0x22/0x40
[3302742.466434] [] vfs_rmdir+0xdc/0x150
[3302742.467449] [] do_rmdir+0x1f1/0x220
[3302742.468470] [] ? ____fput+0xe/0x10
[3302742.469495] [] ? task_work_run+0xc0/0xe0
[3302742.470578] [] SyS_rmdir+0x16/0x20
[3302742.471628] [] system_call_fastpath+0x22/0x27

Cause

When cgroups are deleted, the kernel re-accounts in-use memory pages to the parent cgroup hierarchy. If the cgroups consume a large amount of memory, this process takes a long time. Because no scheduling points exist during the calculation, a softlockup error occurs.

Solution

Important

Before you proceed, create snapshots to back up instance data to prevent data loss. See Create snapshot manually.

Operations vary based on the instance operating system.

If your instance runs CentOS, upgrade the kernel version.
1. Upgrade the kernel:
```
yum update kernel
```
2. Restart the instance:
```
reboot
```
3. Verify that the kernel version is 3.10.0-1160 or later:
```
uname -r
```
If your instance runs Alibaba Cloud Linux, this issue does not occur.
For other operating systems, upgrade the kernel to version 4.17 or later.