An operation fails in Alibaba Cloud Linux due to a lack of read permissions for the OverlayFS file system - Alibaba Cloud Linux

On Alibaba Cloud Linux 2 and Alibaba Cloud Linux 3, a kernel bug in specific versions causes OverlayFS copy-up operations to fail with "Permission denied" when the process lacks read permissions on files in the lower directory. Install a kernel hotfix or upgrade the kernel to resolve this.

Check whether this issue applies to you

The issue occurs only when all three of the following conditions are met:

Your instance runs one of the affected kernel versions listed in the table below.
In the OverlayFS file system, the upper and lower directories are on the same file system.
A process that triggers a copy-up operation — copying files from the lower directory to the upper directory — does not have read permissions on the source files or directories in the lower directory.

Affected kernel versions

Operating system	Image	Affected kernel versions
Alibaba Cloud Linux 2	Alibaba Cloud Linux 2.1903 LTS 64-bit	4.19.91-27.al7, 4.19.91-27.1.al7, 4.19.91-27.2.al7, 4.19.91-27.3.al7, 4.19.91-27.4.al7
Alibaba Cloud Linux 3	Alibaba Cloud Linux 3.2104 LTS 64-bit	5.10.134-13.al8, 5.10.134-13.1.al8, 5.10.134-14.al8, 5.10.134-14.1.al8

Reproduce the issue

Run the following commands to confirm the issue on your instance:

mkdir -p /root/test/lower/dir /root/test/upper /root/test/work /root/test/mount
chmod 0737 /root/test/lower/dir
chown root:bin /root/test/lower/dir
mount -t overlay -o lowerdir=/root/test/lower,upperdir=/root/test/upper,workdir=/root/test/work overlay /root/test/mount
cd /root/test && sudo -u bin -g bin touch mount/dir/RANDOM

If the bug is present on your instance, the following error is returned:

touch: cannot touch 'mount/dir/RANDOM': Permission denied

Root cause

In the affected kernel versions, when the upper and lower directories share the same file system, the ovl_override_creds() function fails to run for processes that lack read permissions on the target files or directories in the lower directory. Because the process cannot establish the credentials needed to perform the copy-up, the operation is rejected with a permission error.

Fix the issue

Prerequisites

Connect to the Linux instance before proceeding.

Step 1: Check your kernel version

uname -r

Example outputs:

Alibaba Cloud Linux 2: 4.19.91-27.al7.x86_64
Alibaba Cloud Linux 3: 5.10.134-13.1.al8.x86_64

Step 2: Apply a fix

Two options are available:

Option 1: Install a kernel hotfix

For Alibaba Cloud Linux 3 (versions 5.10.134-13.al8, 5.10.134-13.1.al8, 5.10.134-14.al8, and 5.10.134-14.1.al8):

yum install -y kernel-hotfix-13108708-5.10.134-13.1

For Alibaba Cloud Linux 2 (versions 4.19.91-27.al7, 4.19.91-27.1.al7, 4.19.91-27.2.al7, 4.19.91-27.3.al7, and 4.19.91-27.4.al7):

yum install -y kernel-hotfix-13110805-4.19.91-27

Option 2: Upgrade the kernel

yum upgrade kernel

After the upgrade:

For Alibaba Cloud Linux 3: confirm the version is 5.10.134-15.al8 or later.
For Alibaba Cloud Linux 2: confirm the version is 4.19.91-28.al7 or later.

Step 3: Verify the fix

Run the same reproduction commands. If the fix is successful, no error is returned.

mkdir -p /root/test/lower/dir /root/test/upper /root/test/work /root/test/mount
chmod 0737 /root/test/lower/dir
chown root:bin /root/test/lower/dir
mount -t overlay -o lowerdir=/root/test/lower,upperdir=/root/test/upper,workdir=/root/test/work overlay /root/test/mount
cd /root/test && sudo -u bin -g bin touch mount/dir/RANDOM

If no error is returned, the issue is resolved.