Transparent Huge Pages THP performance tuning Alibaba Cloud Linux - Alibaba Cloud Linux

THP configurations

In Alibaba Cloud Linux 2, kernel 4.19.81-17.2 and earlier defaulted transparent_hugepage/enabled to madvise (selective THP). Starting from kernel 4.19.91-18, the default changed to always for compatibility with Red Hat Enterprise Linux 7, CentOS 7, and Amazon Linux 2. The always setting enables THP system-wide.

Global configuration

Configure THP through /sys/kernel/mm/transparent_hugepage/enabled:

always

THP is globally enabled.
never

THP is globally disabled.
madvise

THP is enabled only for memory regions marked with MADV_HUGEPAGE via madvise().

Note
The MADV_HUGEPAGE flag signals to the kernel that an application intends to use huge pages for a specific memory region.

Defragmentation configuration

Beyond global THP settings, two defragmentation mechanisms also apply:

THP defragmentation: Merges scattered small pages into huge pages to reduce memory fragmentation.
khugepaged defragmentation: The khugepaged daemon periodically scans memory and merges scattered small pages into contiguous huge pages.

The khugepaged daemon proactively merges pages in the background. The THP defrag setting controls allocation-time defragmentation strategy.

THP defragmentation

Controls memory behavior during page faults (Direct Reclaim, Background Reclaim, Direct Compaction, Background Compaction). Configure by writing to /sys/kernel/mm/transparent_hugepage/defrag:

always

The system pauses the requesting process and performs synchronous Direct Reclaim and Direct Compaction. If enough contiguous memory becomes available, the THP is allocated.
defer

Falls back to a 4 KB page and wakes kswapd (Background Reclaim) and kcompactd (Background Compaction). Later, khugepaged merges the 4 KB pages into a 2 MB THP when contiguous memory is available.
madvise

For memory regions marked with MADV_HUGEPAGE via madvise(), behaves like always. All other regions fall back to 4 KB pages.

Note
In Alibaba Cloud Linux 2 with kernel version 4.19.81-17.2 and later, the system default for this setting is madvise.
defer+madvise

For MADV_HUGEPAGE regions (via madvise()), behaves like always. For all other regions, behaves like defer.
never

Disables allocation-time defragmentation.

khugepaged defragmentation

Key khugepaged configurations:

Enable or disable the feature

Configuration file: /sys/kernel/mm/transparent_hugepage/khugepaged/defrag. Options:
- 0
  
  Disables khugepaged defragmentation.
- 1
  
  When set to 1, the khugepaged daemon periodically merges scattered 4 KB pages into 2 MB THPs during idle times.
  Note
  - In Alibaba Cloud Linux 2 with kernel version 4.19.91-18 and later, the system default is 1.
  - This acquires memory path locks. If khugepaged scans at suboptimal times, it can degrade application performance.
Retry interval

Time khugepaged waits before retrying after a failed THP allocation. Default: 60000 ms (60 seconds). Configuration file: /sys/kernel/mm/transparent_hugepage/khugepaged/alloc_sleep_millisecs.
Scan Interval

Time between khugepaged wakeups. Default: 10000 ms (10 seconds). Configuration file: /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs.
Number of pages to scan

Pages scanned per khugepaged wakeup. Default: 4096. Configuration file: /sys/kernel/mm/transparent_hugepage/khugepaged/pages_to_scan.

THP configuration recommendations

Impact of using THP

THP improves performance by increasing TLB hit rates and reducing PTE overhead, with no application code changes required. However, when THP allocation fails, the triggered recovery mechanisms can degrade performance. The default configuration does not suit all workloads. For example:

With THP defrag set to always, the system performs synchronous Direct Reclaim or Direct Compaction under memory pressure, causing latency spikes.
With khugepaged enabled (1), the daemon acquires memory path locks during consolidation, which can degrade memory-sensitive applications if triggered at the wrong time.
With THP enabled but both defragmentation features disabled, memory exhausts free pages faster than with 4 KB pages, triggering earlier reclamation and compaction.

Configuration recommendations

THP's performance impact depends on your workload. Adjust the configuration based on your application requirements.

Important

Back up your configuration files or Create a snapshot for a disk before making changes.

For a balance between performance and stability, consider the experimental defer+madvise setting. This lets kswapd, kcompactd, and khugepaged handle compaction in the background without blocking applications.
```
sudo bash -c "echo 'defer+madvise' > /sys/kernel/mm/transparent_hugepage/defrag"
```

If khugepaged CPU usage is high (approaching 100%), increase its scan interval. For example, set it to 30 seconds:

sudo sh -c 'echo 30000 > /sys/kernel/mm/transparent_hugepage/khugepaged/scan_sleep_millisecs'

Alternatively, disable khugepaged:

sudo sh -c 'echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/defrag'

For workloads where stability outweighs throughput — such as high-volume databases, latency-sensitive services, or frequent short-lived allocations — disable THP. To disable at runtime:
```
sudo sh -c 'echo "never" > /sys/kernel/mm/transparent_hugepage/enabled'
```
Note
This change applies only to the current session and reverts on reboot. To disable THP permanently, add a kernel boot parameter:
```
sudo grubby --args="transparent_hugepage=never" --update-kernel="/boot/vmlinuz-$(uname -r)"
sudo reboot
```

Check THP usage

Check THP usage at the system or process level.

System level

THP parameters affect all processes. Check system-wide THP usage:
```
cat /proc/meminfo | grep AnonHugePages
```
Example output:
```
AnonHugePages:    614400 kB
```
Note
A non-zero value indicates that THPs are in use on the system.

Process level: Applications use madvise() with MADV_HUGEPAGE to control THP for their own memory regions without affecting other processes. Check THP usage for a specific process:

sudo cat /proc/<PID>/smaps | grep AnonHugePages

Note

Replace <PID> with the actual process ID.

Example output:

AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB
AnonHugePages:         0 kB

References

THP functionality and risks are documented in Transparent Hugepage Support.
Code Huge Page applies the same consolidation principle to map code segments into 2 MB+ regions, reducing TLB misses. Alibaba Cloud Linux also supports Huge Pages for code.
Red Hat Enterprise Linux 7 THP tuning: Configuring Transparent Huge Pages.