File Storage NAS:Troubleshoot file system mount failures

Automatic script to troubleshoot NFS mount failures

Mounting an NFS file system on a Linux server can fail for several reasons. Use this automatic check script to troubleshoot the issue.

1. Log in to the Linux server where the file system failed to mount.

2. Download the automatic check script.

   wget https://nas-client-tools.oss-cn-hangzhou.aliyuncs.com/linux_client/check_alinas_nfs_mount.py -P /tmp/

3. Run the automatic check script and review the configuration in the output.

   python2.7 /tmp/check_alinas_nfs_mount.py file-system-id.region.nas.aliyuncs.com:/ /mnt

   The command contains the following parameters. Replace the placeholders with your actual values.

   • file-system-id.region.nas.aliyuncs.com: The mount target address of the NAS NFS file system. To obtain the address, go to the NAS console, click the target file system, click Mount Targets, and then hover over the icon in the Mount Target Address column.

   • /: The root directory of the NAS file system.

   • /mnt: The local path to mount the file system to.

   After you correct the mount configuration, the automatic check script returns a mount command and a completion message.

4. Copy and run the returned mount command to verify the mount.

Automatic troubleshooting script for SMB mount failures

Mounting a file system on a Windows server using the SMB protocol can fail for multiple reasons. Follow these steps to run an automatic inspection script and identify the cause.

1. Log on to the affected Windows server.

2. In PowerShell or PowerShell ISE, use the following commands to download and run the mount troubleshooting script. Then, apply the solution provided by the script.

   1. Download the troubleshooting script.

      Invoke-WebRequest https://nas-client-tools.oss-cn-hangzhou.aliyuncs.com/windows_client/alinas_smb_windows_inspection.ps1 -OutFile alinas_smb_windows_inspection.ps1

   2. Run the troubleshooting script.

      .\alinas_smb_windows_inspection.ps1 -MountAddress abcde-123.region-id.nas.aliyuncs.com -Locale zh-CN

      In this example, abcde-123.region-id.nas.aliyuncs.com is the mount target. Replace this with the actual value.

Troubleshoot SMB mount failures on Linux

Mounting an SMB file system on a Linux server can fail for several reasons. Follow these steps to troubleshoot the issue, or check the output from /var/log/messages and dmesg.

1. Check the Linux operating system version.

   Linux has limited compatibility with the SMB protocol. NAS SMB file systems support the following Linux distributions and kernel versions.

   Operating system	Version
   CentOS	CentOS 7.6 64-bit: 3.10.0-957.21.3.el7.x86_64 and later
   Alibaba Cloud Linux	Alibaba Cloud Linux 2.1903 64-bit: 4.19.43-13.2.al7.x86_64 and later Alibaba Cloud Linux 3.2104 64-bit: 5.10.23-4.al8.x86_64 and later
   Debian	Debian 9.10 64-bit: 4.9.0-9-amd64 and later
   Ubuntu	Ubuntu 18.04 64-bit: 4.15.0-52-generic and later
   openSUSE	openSUSE 42.3 64-bit: 4.4.90-28-default and later
   SUSE Linux	SUSE Linux Enterprise Server 12 SP2 64-bit: 4.4.74-92.35-default and later
   CoreOS	CoreOS 2079.4.0 64-bit: 4.19.43-coreos and later

2. Check the CIFS client.

   Confirm that the CIFS client (cifs-utils) is installed on the Linux operating system, or that mount.cifs is in a directory specified by the PATH environment variable.

   Run the following commands to check if the CIFS client is installed. If the CIFS client is not installed or is not in the specified directory, reinstall the CIFS client. For more information, see Install a CIFS client.

   Operating system	Command
   Ubuntu	sudo apt list cifs-utils
   Debian
   RHEL	sudo yum list cifs-utils
   CentOS
   OpenSUSE	sudo zypper search -i cifs-utils
   SLES12-SP2
   CoreOS	which mount.cifs

3. Check network connectivity.

   Run the ping <VolumeDomainName> command to check network connectivity.

   1. Confirm that the ECS instance and file system are in the same account.

      If they are not in the same Alibaba Cloud account, create an SMB file system in the same account as the ECS instance, or connect the VPCs by using Cloud Enterprise Network (CEN). For more information, see Mounting NAS across accounts and regions with CEN.

   2. Confirm that the ECS instance and file system are in the same VPC.

      If they are not in the same VPC, create a new SMB file system or connect the VPCs by using CEN. For more information, see Use CEN to mount a NAS file system across VPCs in the same region.

4. Check port 445.

   • Check whether the Linux ECS instance's security group blocks access to the IP address of the SMB file system or port 445.

     Run the telnet <VolumeDomainName> 445 command to check connectivity.

     If port 445 is not open, add a rule to the target ECS instance's security group that allows traffic on port 445. For more information, see Add a security group rule.

   • Confirm that the Linux ECS instance is not attempting to connect by using an unsupported TCP port.

5. Check the permission group.

   Confirm that the permission group for the SMB file system's mount target allows access from the ECS instance.

   If access from the ECS instance is not allowed, modify the permission group configuration. For more information, see Modify the configuration of a permission group.

6. Check administrator mount permissions.

   Confirm that the Linux ECS instance administrator has root or sudo permissions.

7. Check the mount command.

   Confirm that the mount command for the SMB file system is correct. The syntax for a manual mount command is as follows:

   sudo mount -t cifs //file-system-id.region.nas.aliyuncs.com/myshare /mnt -o vers=2.0,guest,uid=0,gid=0,dir_mode=0755,file_mode=0755,mfsymlinks,cache=strict,rsize=1048576,wsize=1048576

   If the mount command parameters are incorrect, modify the command. For more information about the mount parameters for an SMB file system on a Linux server, see Mount parameters for an SMB file system on a Linux server.

8. Check SELinux.

   Confirm that the SELinux settings for the target directory are correct.

9. Check the number of concurrent mounts.

   Check whether the number of mount connections from Linux ECS instances exceeds the per-file system mount limit (1,000).

   • If the limit is exceeded, make adjustments based on your business requirements.

   • If the limit is not exceeded, submit a ticket for assistance.

     In the ticket, provide the file system information, Linux version, the specific mount command, and the output from /var/log/messages and dmesg.

Troubleshoot NFS mount failures on Windows

Mounting a general-purpose NFS file system on a Windows system can fail for several reasons. Use these steps to troubleshoot the issue.

1. Check the account

   Ensure that the ECS instance and the mount target are in the same account. If they are not, use CEN to connect the accounts before you mount the NFS file system. For more information, see Mounting NAS across accounts and regions with CEN.

2. Check the mount target network

   Ensure that the ECS instance and the mount target are in the same VPC. If they are not, use CEN to connect the VPCs before you mount the NFS file system. For more information, see Use CEN to mount a NAS file system across VPCs in the same region.

3. Check ports 2049 and 111

   Ensure that the security group for the ECS instance allows access to ports 2049 and 111 on the mount target. If not, modify the security group rules. For more information, see Modify security group rules.

   You can use the telnet or Test-NetConnection command to test port connectivity.

   • Run the telnet command in Command Prompt.

     telnet $mount_point.$region.nas.aliyuncs.com 2049
     telnet $mount_point.$region.nas.aliyuncs.com 111

   • Run the Test-NetConnection command in PowerShell.

     Test-NetConnection $mount_point.$region.nas.aliyuncs.com -Port 2049
     Test-NetConnection $mount_point.$region.nas.aliyuncs.com -Port 111

4. Check the NFS file system permission group

   Ensure that the permission group of the mount target allows access from the ECS instance. If not, modify the permission group configuration. For more information, see Modify the configuration of a permission group.

"Subdirectory does not exist" error

Error message: mount.nfs: access denied by server while mounting xxxx.nas.aliyuncs.com:/<dir>

This error indicates that the NAS subdirectory <dir> specified in the mount command does not exist. To resolve this, mount the NAS root directory first. Then, create the required subdirectory and mount it. For more information, see How do I create and mount a subdirectory of a NAS file system on a Linux system?.

Duplicate ECS instance name error

Symptom

When you mount a file system using NFSv4.0, the system returns the error message mount.nfs: Operation not permitted or mount.nfs: an incorrect mount option was specified. However, you can mount the file system using NFSv3.

Cause

On certain kernel versions, a conflict occurs if an ECS instance shares a hostname with another instance that has already mounted the same NFS mount target using NFSv4.0.

Solution

1. Run the following command on the affected ECS instance.

   echo 'install nfs /sbin/modprobe --ignore-install nfs nfs4_unique_id=`cat /sys/class/dmi/id/product_uuid`' >> /etc/modprobe.d/nfs.conf

2. Restart the ECS instance during off-peak hours.

   Alternatively, manually unmount all mounted NFS file systems and run the rmmod command to unload the NFSv4.0 client and the NFS kernel module.

3. Mount the NFS file system again. For more information, see Mount an NFS file system.

"mount.nfs: No such device" error when mounting an NFS

Symptom

When you mount an NFS file system on an ECS instance, the following error message appears.

mount.nfs: No such device

[root@ixxx xxx]# mount -t nfs -o vers=4,minorversion=0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport xxx :/ /mnt
mount.nfs: No such device
[root@xxx xxx]#

Cause

This error occurs if the options sunrpc tcp_slot_table_entries=128 setting in the /etc/modprobe.d/sunrpc.conf file is misspelled as options sunrpc tcp_slot_entries=128. This misspelling prevents the sunrpc module from loading correctly into the kernel.

Solution

1. Correct the setting in the /etc/modprobe.d/sunrpc.conf file to options sunrpc tcp_slot_table_entries=128.

2. Run the modprobe sunrpc command to load the sunrpc module.

3. Mount the NFS file system again.

mount: can't find /root/nas in /etc/fstab error

Problem

When you mount a NAS NFS file system on an ECS instance, you receive the following error message.

mount: can't find /root/nas in /etc/fstab

Cause

The command format may be incorrect.

Solution

Ensure that you use the correct command to mount the NAS NFS file system. For details about the command parameters, see Mount command parameters.

• For a General-purpose NAS, run one of the following commands.

  To mount a file system using the NFSv3 protocol:

  sudo mount -t nfs -o vers=3,nolock,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport file-system-id.region.nas.aliyuncs.com:/ /mnt

  To mount a file system using the NFSv4 protocol:

  sudo mount -t nfs -o vers=4,minorversion=0,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport file-system-id.region.nas.aliyuncs.com:/ /mnt

• For an Extreme NAS, run the following command.

  sudo mount -t nfs -o vers=3,nolock,noacl,proto=tcp,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport file-system-id.region.extreme.nas.aliyuncs.com:/share /mnt

In the preceding commands, replace the file-system-id.region.nas.aliyuncs.com placeholder with the mount target of your NFS file system.

Fix file handle error for General-purpose NAS NFS

• Cause

  This error occurs if the Locking registry entry does not exist, or if the Locking entry exists but the Locking value is not set to 1.

• Solution

  1. In Command Prompt, run the regedit command to open Registry Editor.

  2. Navigate to . Right-click a blank area and select .

  3. Rename the new value to Locking. Double-click Locking. In the Edit DWORD (32-bit) Value window, set the value data to 1.

  4. Click OK.

  5. Restart the ECS instance.

  6. Mount the General-purpose NAS NFS file system again.

     mount -o nolock -o mtype=hard -o timeout=60 \\file-system-id.region.nas.aliyuncs.com\! Z:

     Replace the drive letter Z: and the mount target file-system-id.region.nas.aliyuncs.com with your own.

  7. Run the mount command to verify the mount.

     After the file system is mounted, check the output. The mount is successful if the output includes mount=hard, locking=no, and a timeout value of 60.

     C:\Users\Administrator>mount
     Local    Remote                                 Properties
     Z:      \\29xxx.nas~    UID=-2, GID=-2
                               rsize=1048576, wsize=1048576
                               mount=hard, timeout=60.0
                               retry=1, locking=no
                               fileaccess=755, lang=GB2312-80
                               casesensitive=no
                               sec=sys

Troubleshoot system error 53 for Windows SMB mounts

• Symptoms

  The network path was not found.

• Causes

  • The network connection is unavailable.

  • The TCP/IP NetBIOS Helper service is not started.

  • LanmanWorkstation is not correctly configured in the registry.

• Solution

  1. Run the ping <mount target address> command to check the reachability and latency of the mount target.

     • If the network connection is normal, proceed to the next step.

     • If the connection fails, troubleshoot the network issue as follows:

       • Verify that the mount command is correct and free of typos, such as incorrect slashes / or \, extra spaces, or a misspelled myshare name.

         Use the following format to mount an SMB file system:

         net use <target_drive_letter> \\<mount_target_address>\myshare

         Example:

         net use z: \\xxxx.cn-hangzhou.nas.aliyuncs.com\myshare

       • In the NAS console, on the File System List page, verify that the file system's file system type is SMB, and note its mount target address.

       • Verify that the mount target address is correct.

       • Ensure the ECS instance and the mount target are in the same VPC.

       • If you are connecting from a client in a different VPC or through a VPN, ensure the network configuration is correct.

  2. Run the telnet <mount target address> 445 command to verify that the SMB service is available.

  3. Ensure the TCP/IP NetBIOS Helper service is started. For more information, see Mount an SMB file system.

  4. Open the Registry Editor. In the key, verify that the value of ProviderOrder includes LanmanWorkstation. If it does not, add it. In the Edit String dialog box, ensure the Value name is ProviderOrder and the Value data contains LanmanWorkstation (for example, RDPNP,LanmanWorkstation,webclient).

  If the issue persists, check if Windows Defender Firewall is blocking SMB-related network requests.

  In Windows Defender Firewall, under Outbound Rules, find the File and Printer Sharing (SMB-Out) rule and open its Properties dialog box. On the General tab, ensure that the Enabled checkbox is selected and the action is set to Allow the connection.

System error 58 on Windows SMB file system

• Error message

  The specified server cannot perform the requested operation.

• Cause

  The client is using an incompatible SMB protocol version.

• Solution

  Ensure you are using Windows 2008 R2 or a later version. Windows 2008 is not supported.

Troubleshoot Windows system error 64 for SMB mounts

• Error message

  The specified network name is no longer available.

• Causes

  • The NAS permission group does not allow access from the target ECS instance.

  • The private IP address or VPC IP address of the target ECS instance is incorrectly configured in the NAS permission group.

  • Your account has overdue payments.

  • When mounting in a classic network, the ECS instance and the NAS file system belong to different Alibaba Cloud accounts.

  • The file system type is not SMB.

• Solution

  This error indicates that you do not have permission to access the NAS file system resources. To resolve this issue, check the following:

  1. Ensure that the permission group of the file system mount target includes the private IP address or VPC IP address of the instance.

  2. Ensure that your Alibaba Cloud account has no overdue payments.

  3. If you are mounting in a classic network, ensure that the ECS instance and the NAS file system belong to the same Alibaba Cloud account.

  4. Ensure that the file system type is SMB.

Resolve system error 67 for SMB mount

• Error message

  The network name cannot be found.

• Cause

  The required network services are not running.

• Solution

  Start the following services. See Mount an SMB protocol file system for detailed instructions.

  1. Start the Workstation service.

  2. Start the TCP/IP NetBIOS Helper service.

Troubleshoot system error 85 on Windows SMB mounts

• Error message

  The local device name is already in use.

• Cause

  The target drive letter is already in use.

• Solution

  Change the target drive letter and remount the file system.

Resolve system error 1231 for SMB file systems

• Symptom

  The network location cannot be reached.

• Cause

  • Client for Microsoft Networks is not installed or enabled.

  • File and Printer Sharing for Microsoft Networks is not installed or enabled.

• Solution

  Install and enable Client for Microsoft Networks and File and Printer Sharing for Microsoft Networks.

  If these components are installed but disabled, select their checkboxes. To install a missing component, follow the steps below.

  Also, ensure that QoS Packet Scheduler is enabled.

  1. Open the Network and Sharing Center and click your active network connection.

  2. Click Properties.

  3. In the WLAN Properties dialog box, click Install.

     • To install Client for Microsoft Networks:

       1. In the Select Network Feature Type dialog box, select Client, and then click Add.

       2. Select Client for Microsoft Networks, and then click OK.

     • To install File and Printer Sharing for Microsoft Networks:

       1. In the Select Network Feature Type dialog box, select Service, and then click Add.

       2. Select

Resolve system error 1272 for SMB mounts

• Symptom

  You receive the following error message: You can't access this shared folder because your organization's security policies block unauthenticated guest access. These policies help protect your PC from unsafe or malicious devices on the network.

• Cause

  A security policy on the Windows system blocks guest access to the SMB file system.

• Solution

  On Windows Server versions later than 2016, modify the following registry key to allow guest access.

  1. Log on to the ECS instance where you are mounting the file system.

  2. Open Command Prompt, run the regedit command, and press Enter to open Registry Editor.

  3. In the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters path, verify that the AllowInsecureGuestAuth registry key is set to 1.

     1. If the value of the AllowInsecureGuestAuth registry key is 0, right-click AllowInsecureGuestAuth and set Value data (V) to 1.

     2. If the AllowInsecureGuestAuth registry key does not exist, open PowerShell and run the following command to create it.

        New-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters -Name AllowInsecureGuestAuth -PropertyType DWORD -Value 1

  4. Mount the SMB file system again.

     net use Z: \\file-system-id.region.nas.aliyuncs.com\myshare

     If system error 1272 persists, the value of the RequireSecuritySignature registry key might be 1. If so, change its value to 0 and mount the SMB file system again.

     The RequireSecuritySignature registry key is located at the following path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters.

  For more information, see Guest access in SMB2 disabled by default in Windows and Overview of Server Message Block signing.

System error 3227320323 on SMB mount in Windows

• Symptom

  You receive system error 3227320323.

• Cause

  The Microsoft network client: Digitally sign communications (always) policy is enabled in Windows. This policy prevents anonymous mounts because they do not provide the identity information required for digital signing.

• Solution

  Disable the Microsoft network client: Digitally sign communications (always) policy, and then mount the file system again. To disable this policy, follow these steps.

  1. Press Win+R, enter gpedit.msc, and press Enter to open the Local Group Policy Editor.

  2. Navigate to . Double-click Microsoft network client: Digitally sign communications (always), select Disabled, and then click OK.

System error 1312 when mounting an SMB file system

• Symptom

  Windows System Error 1312.

• Cause

  This error occurs if you enter an incorrect username or workgroup when using the New-SmbGlobalMapping PowerShell command to mount the file system.

• Solution

  Find the correct workgroup name in the My Computer properties. Then, when prompted by the mount command, enter the correct username or workgroup. For example, the username is workgroup\administrator and the password is ECSadministrator.

NFS auto-mount failure on server startup

Symptom

An NFS file system configured for automatic mounting fails to mount on server reboot.

Cause

This issue can occur on servers running CentOS 7.0. By default, CentOS 7.0 does not process non-local file systems listed in the /etc/fstab file because the remote-fs.target service is disabled. As a result, the NFS file system fails to mount automatically on server reboot.

Solution

Choose one of the following solutions to automatically mount an NFS file system on server startup. After applying the solution, open the /etc/fstab configuration file and add the mount configuration. For more information about how to add the mount configuration, see Mount an NFS file system.

• Solution 1: Permanent fix

  Run the following commands to start and permanently enable the remote-fs.target service. This ensures that the NFS file system mounts on boot.

  systemctl start remote-fs.target

  systemctl enable remote-fs.target

• Solution 2: Workaround

  As an alternative, run the following command to add a mount command to the /etc/rc.local file. This ensures that all NFS file systems listed in /etc/fstab mount on boot.

  [ ! -f /etc/rc.local ] && echo '#!/bin/bash' > /etc/rc.local; echo "mount -a -t nfs" >> /etc/rc.local; chmod +x /etc/rc.local

IIS on Windows Server 2016 cannot load SMB volumes

To resolve this issue, see Install and configure an AD domain.

Resolve Windows NFS soft mount issues

• Problem description

  When you mount an NFS file system on Windows, it uses a soft mount by default. This mode can compromise data consistency and cause applications to exit unexpectedly.

  • Data consistency: With a soft mount, when a request to the server times out, the client returns an error. The application treats the operation as incomplete, but the server may have already processed the request. This discrepancy can lead to data inconsistency.

  • Unexpected application exit: In a soft mount, when a request times out, the client returns a timeout error. In some programming languages, this error throws an exception. If the application does not handle the exception, it exits unexpectedly.

• Solution

  On Windows, use a hard mount for the NFS file system to avoid these issues.

  1. Run themount command to check the current mount mode.

     • If the output showsmount=soft, follow the steps below.

     • If the output showsmount=hard, no changes are needed.

       Local    Remote                Properties
       -----------------------------------------------------------
       X:       \\xxx                UID=-2, GID=-2
                                     rsize=32768, wsize=32768
                                     mount=hard, timeout=1.6
                                     retry=1, locking=yes
                                     fileaccess=755, lang=ANSI
                                     casesensitive=no
                                     sec=sys

  2. Stop any applications that are currently using this NFS file system.

  3. Unmount the NFS file system.

     umount H:

     Replace the drive letterH: in the umount command with your actual drive letter.

  4. Remount the NFS file system using the hard mount option.

     mount -o nolock -o mtype=hard -o timeout=60 \\xxxxxx.cn-hangzhou.nas.aliyuncs.com\! h:

     Replace xxxxxx.cn-hangzhou.nas.aliyuncs.com with your mount target's domain name and h: with the desired drive letter.

  5. Run themount command to confirm the new settings.

     The mount is successful if the output includes mount=hard, locking=no, and a timeout value of 10 or greater.

NFS mount errors on Windows

• Problem description: Invalid file handle

  Solution: Remount the file system using the correct parameters. For details, see Mount an NFS file system (general-purpose NAS).

• Problem description: Network error - 53

  Solution: Remount the file system using the correct parameters. For details, see Mount an NFS file system (general-purpose NAS).

• Problem description: Network error - 1222

  C:\Users\Administrator>mount -n nolock -n mtype=hard -o timeout=60 \\12dc2a4b1d9-opu25
  Network Error - 1222
  For more information, type "NET HELPMSG 1222".

  Solution: Install the NFS client and remount the file system. For details, see Step 1: Install the NFS client.

NAS mount error in container: access denied by server while mounting <mount_address>

Symptoms

You receive the error access denied by server while mounting <mount_address> when you mount a NAS file system to a container.

Cause

Possible causes include the following:

1. The mount directory does not exist.

2. The user on the host who starts the container lacks permission to modify the NAS file system.

3. The permission group configured for the mount target does not grant access to the container.

Solution

1. Verify that the mount directory exists.

   You can run the cd <mount_directory> command to check if the directory exists.

2. Confirm that the user who starts the container has root permissions.

   If the user who starts the container does not have root permissions, log on to the NAS file system as the root user, change the permissions, and then mount the file system again. This ensures that the container user's permissions match the file system permissions on the mount target. When mounting the file system from within a container, run the container in privileged mode.

3. Confirm that the permission group configured for the mount target includes the container's IP address.

   1. Log on to the NAS console.

   2. In the navigation pane, choose File System > File System List.

   3. In the top navigation bar, select a region.

   4. On the File System List page, find the target file system and click Manage in the Actions column.

   5. On the file system details page, click Mount Targets.

   6. In the list of mount targets, click the permission group name in the Permission Group column to view the rules for that group.

   7. Confirm that the permission group rules include the IP address of the target container.

      If the target container's IP address is not in the permission group rules, add it. For more information, see Add a rule to a permission group.

Troubleshoot Cloud Assistant batch operations

First, confirm that the mount parameters you entered are correct. Next, check the error code in the command output and follow the guidance in the error message. For more information, see Error messages. Finally, use the one-click mount feature to attempt the operation on a single ECS instance. If it fails, use the error message to resolve the issue.

Alternatively, use the NAS console to mount a file system on a single ECS instance. If the mount fails, follow the on-screen instructions to resolve the issue.

File system mount failures in the console

Use the following table to troubleshoot file system mount failures.