How to troubleshoot high slab_unreclaimable memory usage in Linux - Alibaba Cloud Linux

High slab_unreclaimable memory on a Linux ECS instance may indicate a kernel slab memory leak. This guide helps you identify the root cause and resolve the issue.

Symptoms

Run cat /proc/meminfo | grep "SUnreclaim" on a Linux instance. If the SUnreclaim value is large (for example, SUnreclaim: 6069340 kB), the instance has high slab_unreclaimable memory. If slab_unreclaimable exceeds 10% of total memory, a slab memory leak is likely.

Cause

The kernel's slab allocator caches same-sized memory objects to reduce fragmentation. The slab_unreclaimable portion holds memory the kernel cannot free because it contains active objects such as dentry and inode caches. If these caches grow excessively, high slab_unreclaimable usage can trigger the OOM Killer.

Solution

Connect to the Linux instance.

For more information, see Choose a connection method.
Identify the unreclaimable slab with the most objects or highest memory usage:
1. Find the slab with the most objects or highest memory usage.
```
slabtop -s -a
```
  Record the slab name (the NAME column) with the highest OBJ/SLAB value.
2. Check if the slab memory is unreclaimable.
  
  Replace <slab NAME> with the slab name with the highest OBJ/SLAB value from the previous step.
```
cat /sys/kernel/slab/<slab NAME>/reclaim_account
```
  For example, check whether kmalloc-192 is reclaimable:
```
cat /sys/kernel/slab/kmalloc-192/reclaim_account
```
  Output 0 means unreclaimable; output 1 means reclaimable.
Identify the root cause of the slab memory leak.
Use the crash tool for static analysis or the perf tool for dynamic analysis. The following examples use the kmalloc-192 slab.
Method 1: Use crash to perform static analysis
1. Install the crash tool.
  sudo yum install crash -y
2. Install the kernel-debuginfo tool.
  
  Alibaba Cloud Linux 3
  sudo yum install -y kernel-debuginfo-<kernel version> --enablerepo=alinux3-plus-debug
  
  Note
  Replace kernel version with your actual kernel version. Run uname -r to check.
  
  Alibaba Cloud Linux 2
  sudo yum install kernel-debuginfo -y
3. Start the crash tool.
  sudo crash
4. View memory statistics for kmalloc-192 in crash:
  kmem -S kmalloc-192
  
  To limit output, view only the last few rows. For example, to view the last 10 rows:
  kmem -S kmalloc-192 | tail -n 10
  
  Sample command output:
  SLAB MEMORY NODE TOTAL ALLOCATED FREE ffffea004c94e780 ffff88132539e000 0 42 29 13 ffffea004cbef900 ffff88132fbe4000 0 42 40 2 ffffea000a0e6280 ffff88028398a000 0 42 40 2 ffffea004bfa8000 ffff8812fea00000 0 42 41 1 ffffea006842b380 ffff881a10ace000 0 42 41 1 ffffea0009e7dc80 ffff880279f72000 0 42 34 8 ffffea004e67ae80 ffff881399eba000 0 42 40 2 ffffea00b18d6f80 ffff882c635be000 0 42 42 0
  
  The output shows that ffff88028398a000 has little FREE memory and high ALLOCATED memory.
5. View memory data for ffff88028398a000:
  rd ffff88028398a000 512 -S
  
  If the output is large, display it in pages.
  
  If the put_cred_rcu function repeats multiple times in the output, search for put_cred_rcu in the Linux kernel source code:
  void __put_cred(struct cred *cred) { call_rcu(&cred->rcu, put_cred_rcu); }
  
  put_cred_rcu asynchronously releases the cred struct. The repeated presence of put_cred_rcu at the end of the cred struct indicates a slab memory leak in the kernel.
Method 2: Use perf to perform dynamic analysis
1. Install the perf tool.
  sudo yum install perf -y
2. Use perf to capture unreleased memory in kmalloc-192 over 200 seconds:
  sudo perf record -a -e kmem:kmalloc --filter 'bytes_alloc == 192' -e kmem:kfree --filter ' ptr != 0' sleep 200
3. Save the captured data to a file.
  
  In this example, the file is named testperf.txt:
  sudo perf script > testperf.txt
4. View testperf.txt:
  cat testperf.txt
  
  Identify slabs with no free memory, then trace the responsible function in the kernel source code.
After identifying the function call path or kernel data structure causing the leak, work with kernel developers or professional O&M personnel to pinpoint the source and resolve the issue.

Possible solutions:
- Upgrade the kernel or patch.
- Adjust kernel parameters.
- Restart affected services or modules.
- Optimize applications or drivers.
- Restart the system.

References

Slab memory leaks reduce available memory, cause fragmentation, and can trigger the OOM Killer or performance degradation.

Symptoms

Cause

Solution

Method 1: Use crash to perform static analysis

Method 2: Use perf to perform dynamic analysis

References