File Storage NAS:Troubleshoot file read and write access issues

Server unresponsiveness during concurrent file access

Cause: A bug in the Linux SMB kernel driver causes this issue. When you mount a file system using SMB 2.1 or 3.0, clients might fail to send the expected SMB BreakAck packets in some concurrent access scenarios, causing the server to stop responding for 35 seconds.

Solution 1: Mount the file system by using SMB 2.0.

Solution 2: Follow these steps.

1. Run the following command to disable oplocks when you load the CIFS module:

   # modprobe cifs enable_oplocks=0

2. Run the following command to disable oplocks after the CIFS module is loaded:

   # echo 0 > /sys/module/cifs/parameters/enable_oplocks

3. Run the following command to check the status of oplocks:

   # cat /sys/module/cifs/parameters/enable_oplocks

   In the output, Y indicates enabled, and N indicates disabled.

   Note

   • To apply the changes, unmount and then remount the SMB file system.

   • To make the changes permanent, create the /etc/modprobe.d/cifs.conf file and add the line options cifs enable_oplocks=0.

Unable to create symbolic links

Cause

The SMB file system was mounted on Linux without the mfsymlinks option or by using protocol version 2.0.

Solution

When you mount an SMB file system on Linux, use protocol version 2.1 or 3.0 and add the mfsymlinks option. An example of the mount command is as follows. For descriptions of the parameters in the example, see SMB (Linux) mount command parameter descriptions.

sudo mount -t cifs //file-system-id.region.nas.aliyuncs.com/myshare /mnt -o vers=2.1,guest,uid=0,gid=0,dir_mode=0755,file_mode=0755,mfsymlinks,cache=strict,rsize=1048576,wsize=1048576

Unresponsive SMB mount point

Cause

On Linux distributions with a kernel earlier than 3.10.0-514, the SMB kernel driver may crash in concurrent access scenarios, making the mount point inaccessible. The kernel log contains messages similar to the following:

...
[<ffffffffc03c9bc1>] cifs_oplock_break+0x1f1/0x270 [cifs]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a8d74>] rescuer_thread+0x294/0x3c0
...

Solution

• Remount the file system with the cache=none option. This may affect performance.

• Upgrade the operating system of the ECS instance that runs Linux.

"Bad file descriptor" error when copying large files

Cause

This issue occurs due to a temporary network or backend failure. The SMB clients on some Linux distributions, such as SUSE, have limited support for this type of failover.

Solution

The following table lists the recommended Linux operating system versions for NAS SMB.

Garbled Chinese characters on clients

Symptom

When you write Chinese characters (for example, in filenames or file content) from a Linux client to a NAS file system, they may appear garbled on a Windows client, and vice versa.

Cause

By default, Windows clients use the GBK character set for Chinese encoding and decoding, while Linux clients use the UTF-8 character set. Each platform encodes data using its own character set. When data written by one platform is read by the other, the incompatible character sets cause a decoding failure, and the content appears garbled.

Solution

To avoid platform incompatibility issues, we recommend that you mount NAS file systems over SMB on Windows clients and over NFS on Linux clients.

Slow file operations on Windows with NFS

Cause

On Windows, NFS has compatibility issues between case-sensitive and case-insensitive semantics. The performance of creating files in a directory significantly degrades as the directory size increases. This happens because the directory must be traversed each time you create a file. When the directory contains around 100,000 entries, a single directory traversal can take more than 10 seconds.

Solution

Modify the mount parameters by adding the -o casesensitive=yes option to avoid directory traversal. The following command provides an example:

mount -o nolock -o mtype=hard -o timeout=60 -o casesensitive=yes \\file-system-id.region.nas.aliyuncs.com\! Z:

Replace the drive letter Z: and the mount point address file-system-id.region.nas.aliyuncs.com based on your environment.

How to resolve the invalid device error when you rename a file on an NFS file system from a Windows client?

Renaming a file returns the invalid device error if you mounted a subdirectory of the NFS file system. To resolve this, mount the root directory of the file system. For more information, see Step 2: Mount a general-purpose NAS file system that uses NFS.

File creation latency in NFS

• Symptom:

  ECS-1 created the file abc, but there is a delay before ECS-2 can see the file abc. The delay is sometimes 1 second, and can even be up to 1 minute. Why does this happen?

• Cause:

  This is caused by the Lookup Cache and is expected behavior for a time period T. For example, ECS-2 accesses the file path before ECS-1 creates the file abc. This causes ECS-2 to receive a 'file not found' response, and a record is then cached indicating that the file abc does not exist. Within the time period T, because the FileAttr has not yet expired, when ECS-2 accesses the file path again, it still retrieves the cached record indicating that the file abc does not exist.

• Solution:

  To ensure that ECS-2 can see the file immediately after ECS-1 creates it, use one of the following solutions:

  • Solution 1: Disable the negative lookup cache on ECS-2 to prevent the caching of non-existent files. This solution has the lowest overhead.

    When you mount the file system, add the lookupcache=positive option. The default value is lookupcache=all. The following command provides an example:

    sudo mount -t nfs -o vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,lookupcache=positive file-system-id.region.nas.aliyuncs.com:/ /mnt

  • Solution 2: Disable all caches on ECS-2. This solution can severely degrade performance. Choose this solution only if your workload requires it.

    When you mount the file system, add the actimeo=0 option. The following command provides an example:

    sudo mount -t nfs -o vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,actimeo=0 file-system-id.region.nas.aliyuncs.com:/ /mnt

Data write latency in NFS

• Symptom:

  ECS-1 updates a file named abc. However, when ECS-2 immediately reads the file, it still reads the old content. Why does this happen?

• Cause:

  This issue has the following two causes:

  • After ECS-1 writes data to the file abc, the data does not immediately flush to the server. Instead, the client first caches the data in its page cache. The data is flushed only when the application calls fsync or close.

  • ECS-2 has a file cache and may not immediately retrieve the latest content from the server. For example, if ECS-2 has cached the data when ECS-1 updates the file abc, ECS-2 still uses the cached content when it reads the file again.

• Solution:

  To ensure that ECS-2 can immediately read the latest data after ECS-1 updates a file, use one of the following solutions:

  • Solution 1: Use close-to-open (CTO) consistency. To ensure CTO, after ECS-1 updates the file, it must call close or fsync. Before ECS-2 reads the file, it must reopen the file and then read it.

  • Solution 2: Disable all caches on both ECS-1 and ECS-2. This solution can severely degrade performance. Choose this solution only if your workload requires it.

    • Disable the cache on ECS-1. When you mount the file system, add the noac option to ensure that all written data is immediately persisted. The following command provides an example:

      sudo mount -t nfs -o vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,noac file-system-id.region.nas.aliyuncs.com:/ /mnt

      Note

      • If the fsync call is made after the write operation on ECS-1 is complete, or if you use synchronous writes, you can replace noac with actimeo=0 for slightly better performance.

      • The noac option is equivalent to specifying actimeo=0 and forcing all writes to be synchronous.

    • Disable the cache on ECS-2. When you mount the file system, add the actimeo=0 option to ignore all caches. The following command provides an example:

      sudo mount -t nfs -o vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,actimeo=0 file-system-id.region.nas.aliyuncs.com:/ /mnt

Different file owners on different ECS instances

In a file system, user identity is identified by a UID or GID, not a username. The username displayed as the owner on an ECS instance is derived from this UID.

For example, on ECS instance 1, the admin user creates a file named admin_on_machine1. On ECS instance 2, the admin user creates a file named admin_on_machine2. When you run the ll command on ECS instance 1 to view the files, the output is shown in the following figure: When you run the ll command on ECS instance 2, the output is shown in the following figure: The query results from the two ECS instances show that the same file has different owner usernames.

Then, run the id command on each instance to query the admin user information. On ECS instance 1, the UID of the admin user is 505, as shown in the following figure: On ECS instance 2, the UID of the admin user is 2915, as shown in the following figure: If you run the stat admin_on_machine1 admin_on_machine2 command, the output shows that the two files belong to different UIDs, as shown in the following figure:

Exceptions from concurrent writes to log files

Symptom

Apsara File Storage NAS provides shared file access with a unified namespace for multiple clients. However, when multiple processes or clients concurrently write to the same file, for example, a log file, each process maintains its own file descriptor and write offset context. The NFS protocol lacks atomic append semantics, which can lead to data corruption issues such as overwrites and interleaving.

Solution

• (Recommended) Have different processes or clients write to different files within the same file system, and then merge the files for subsequent analysis. This approach avoids concurrency issues and does not require file locks, so it does not impact performance.

• For scenarios that involve concurrent appends to the same file, such as a log file, you can use a file lock and the seek mechanism to ensure write atomicity and consistency. However, this method can significantly degrade performance. The following section provides a brief overview for your reference.

How to use flock and seek

Because the NFS protocol does not provide atomic append semantics, concurrent writes to the end of the same file, such as a log file, can easily overwrite each other. On Linux, you can use the flock and seek mechanisms to simulate atomic appends on an NFS file system and protect concurrent append writes.

Follow these steps:

1. Call fd=open(filename, O_WRONLY | O_APPEND | O_DIRECT) to open the file in append mode and specify O_DIRECT to bypass the page cache. This returns the file descriptor fd.

2. Call flock(fd, LOCK_EX|LOCK_NB) to try to acquire a file lock. If the call fails (for example, because the lock is already held), it returns an error. You can then retry or handle the error.

3. After you acquire the lock, call lseek(fd, 0, SEEK_END) to move the current write offset of fd to the end of the file.

4. Perform a normal write operation. The data is written to the end of the file, and the file lock prevents concurrent writes from overwriting each other.

5. After the write operation is complete, call flock(fd, LOCK_UN) to release the file lock.

Why does the ls command return a 523 error when run on an NFS file system in a Linux operating system?

Symptom

When you run the ls command in an NFS file system from a Linux client, the following error message is returned.

Cause

If you run the ls command on a file system directory while concurrent rename operations are being performed on it, a 523 error is returned.

Solution

Wait a moment and retry the operation. If the error persists, submit a ticket.

SMB file system mount failures

Symptom

If you use both NFS and SMB file systems and fail to mount an NFS file system by using the net use command, you may also encounter issues when you try to mount the correct SMB file system.

Solution

Verify that you are mounting the correct file system, temporarily stop the mount operation, and try again after 5 minutes. If the operation still fails, submit a ticket.

Mounted directory visible only to Administrator

This issue is due to the user isolation mechanism in Windows. A directory mounted by one user is not visible in another user's session.

To enable sharing among multiple users, create a directory link. For example, run the following command to create a directory named myshare on the C drive:

mklink /D C:\myshare \\xxxxxxx-xxxx.cn-beijing.nas.aliyuncs.com\myshare\

Poor performance of SMB on Linux clients

If an SMB file system performs poorly, you can troubleshoot the issue based on the following potential causes.

• Cause 1: The maximum throughput of an SMB file system scales linearly with its storage capacity.

  Solution: Use the fio tool to test the performance of the SMB file system. For more information, see Test the performance of a NAS file system.

• Cause 2: The network bandwidth of a single ECS instance that runs Linux is low.

  Solution: Use multiple ECS instances that run Linux to achieve the desired overall performance of the file system.

• Cause 3: Client-side caching for the SMB file system is disabled.

  Solution: When you mount an SMB file system, cache=none indicates that caching is disabled. The default option or cache=strict indicates that caching is enabled. You can run the sudo mount | grep cifs command to check whether the correct option is used.

• Cause 4: The I/O size for the SMB client is not properly set.

  Solution: Adjust the rsize and wsize values based on your business requirements. The default value is 1048576.

• Cause 5: The CPU or memory specifications of the ECS instance that runs Linux are too low, or too many resources are occupied by other services.

  Solution: Select appropriate specifications for the ECS instance that runs Linux and check the resources used by other applications on the system to ensure that CPU and memory requirements are met. You can run the top command to check the CPU and memory usage.

• Cause 6: The atime option is used for mounting.

  Solution: If your business is not highly sensitive to file access times (atime), do not use the atime option when you mount the file system.

• Cause 7: The web server scenario involves frequent reads of a large number of small files, infrequent writes, and write notifications.

  Solution: You can configure a specific caching mechanism for the web server, such as Apache, on the client, or contact the Alibaba Cloud NAS team to enable acceleration for web server scenarios.

How do I resolve the Permission denied error when I access an SMB file system from Linux?

Cause: This error can occur if incorrect values for UID, GID, file_mode, or dir_mode are used in the mount command.

Solution: Check whether the mount options such as UID, GID, file_mode, and dir_mode are correctly set. For more information, see Mount an SMB file system.

Changing filename case in SMB

SMB file systems are case-insensitive, which is consistent with Windows systems. However, you cannot rename a file by only changing its case.

As a workaround, you can first rename the file to a different name, and then rename it to the desired name with the correct case.

Unable to change file owner and mode

You cannot dynamically change the owner or mode of files and directories. These attributes can only be specified when you mount the file system. For more information, see Mount an SMB file system.

What generates files with the .nfs extension? How can they be deleted?

When you delete a file that is open in an application, the system creates a temporary file with the .nfs suffix. The system automatically deletes this file when the application process closes.

How do I resolve the bind conn to session failed on NFSv4 server error when accessing files in a NAS file system directory?

• Cause

  This error occurs because Apsara File Storage NAS does not support NFSv4.1. The system reports this error if you try to mount a file system using NFSv4.1.

• Solution

  Remount the file system by using NFSv3 or NFSv4.0 based on your business scenario. For more information, see Usage notes for mounting a file system.

Data synchronization issues with NFS

Symptom

When multiple clients mount the same NAS file system, data changes made on one client might not be immediately visible on other clients.

Cause

By default, the operating system kernel maintains file and directory attributes and creates a metadata cache to reduce the need for NFSPROC_GETATTR remote procedure calls.

Solution

Run the following mount command to disable caching for file and directory attributes.

mount -t nfs4 -o noac file-system-id.region.nas.aliyuncs.com:/ /mnt

Replace file-system-id.region.nas.aliyuncs.com with the mount point address of the file system and /mnt with the local path on your server where you want to mount the file system.

Container writes to old NAS after remount

Cause

If you mount a NAS file system to an ECS instance and then map the mount directory to a container using a local volume (HostPath), the container's mount information becomes independent of the host ECS instance. Consequently, if you unmount the original NAS directory on the host or mount a new one, the running container continues to use the old NAS file system.

Solution

After you mount the new NAS file system on the ECS instance, restart the container Pod.

Files not visible after server restart

If the file system still exists, this issue typically occurs because the server is not configured to automatically mount the NAS file system upon restart.

To manually mount the NAS file system again, see Usage notes for mounting a file system.

To configure automatic mounting of the NAS file system after a restart, see the following topics:

• Automatically mount an NFS file system on a Linux system

• Automatically mount an SMB file system on a Windows system

Slow file migration and copy with SMB on Linux

If you have ruled out file system performance issues, the slowness might be due to a lack of concurrency in your migration or copy process. You can use the following open-source tools for parallel migration or copying.

• GNU Parallel

  Select an appropriate number of threads based on system resources. Example: find * -type f | parallel --will-cite -j 10 cp {} /mnt/smb/ &

• Fpart

• Fpsync

• multi

"Disk quota exceeded" error

• Cause

  This error indicates that the user or group has exceeded the storage space or file count limit defined by a restrictive quota on the target directory. These operations include increasing file length, creating files or directories, and moving files into the directory. An error message such as Disk quota exceeded is returned.

• Solution

  1. Free up space by deleting data or increase the capacity limit of the directory. For more information, see Edit a user quota.

  2. After freeing up space, perform a small write operation (such as touch testfile) in the directory. This helps trigger a faster refresh of the quota statistics. Once the write operation succeeds, restart your application.

"Permission denied" error when accessing NFS

You can follow these steps to configure AnonymousGid and AnonymousUid for your system.

1. Log on to the ECS instance where the file system is mounted.

2. Open Command Prompt and run the regedit command to open Registry Editor.

3. Navigate to .

4. Right-click a blank area, select , and create the following two registry entries.

   • AnonymousGid, with a value of 0.

   • AnonymousUid, with a value of 0.

5. Restart the ECS instance.

6. Remount the general-purpose NAS file system that uses NFS.

   mount -o nolock -o mtype=hard -o timeout=60 \\file-system-id.region.nas.aliyuncs.com\! Z:

   Replace the drive letter Z: and the mount point domain name file-system-id.region.nas.aliyuncs.com with your actual values.

7. Run the mount command to check whether the file system is successfully mounted.

   After the mount is complete, the output must include mount=hard, locking=no, and a timeout value of 10 or greater. Otherwise, the mount failed.

   

Modifying root directory permissions with chown

You cannot modify the permissions of a NAS root directory.

To control permissions on a locally mounted NAS directory, you can mount a subdirectory. For example, if you mount the NAS root directory to /data, you cannot use chown to change the owner and group of the /data directory. If you mount a NAS subdirectory, which must be created in advance, to the local /data directory, you can use chown to change the owner and group of the /data directory. Note that to create a subdirectory in NAS, you must first mount the NAS root directory and then create the subdirectory. For information about how to create and mount a subdirectory, see How do I create and mount a NAS subdirectory on a Linux system?.