ossfs FAQ - Object Storage Service - Alibaba Cloud Documentation Center

This topic describes common issues and solutions when using ossfs. If you encounter an issue, first check your ossfs version. If you are running an older version, such as 1.80.x, we recommend that you upgrade to the latest version. Newer versions provide new features and improved stability.

Overview

Each ossfs error includes a message. To troubleshoot issues such as socket connection failures or HTTP 4xx/5xx status codes, collect these messages to identify the cause. Enable debug logs before you begin.

A 403 error indicates that the request is denied due to insufficient permissions.
A 400 error indicates an incorrect user operation.
A 5xx error is usually related to network jitter or a client-side issue.

Key characteristics of ossfs:

ossfs mounts a remote OSS bucket as a local disk. If your workload is sensitive to file I/O performance, we recommend that you do not use ossfs.
Operations in ossfs do not guarantee atomicity. A local operation might succeed even if the corresponding remote OSS operation fails.

If ossfs does not meet your business requirements, use ossutil.

Permission issues

Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21839][INF]      curl.cpp:PutHeadRequest(2278):   [tpath=/.fuse_hidden0000002700000006]
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21839][INF]      curl.cpp:prepare_url(4030):  URL is http://oss-cn-hangzhou-internal.aliyuncs.com/pgback/.fuse_hidden0000002700000006
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21839][INF]      curl.cpp:prepare_url(4062):  URL changed is http://pgback.oss-cn-hangzhou-internal.aliyuncs.com/.fuse_hidden0000002700000006
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21839][INF]      curl.cpp:PutHeadRequest(2330):   ...fuse_hidden0000002700000001
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21840][INF]      curl.cpp:RequestPerform(63...):  HTTP response code 403 was returned, returning EPERM
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21840][INF]      s3fs:s3fs_flush(2127):  [path=/tmp/7kAcr][fd=6]
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21840][INF]      fdcache.cpp:RowFlush(1302):  [path]=[path=/tmp/7kAcr][fd=6]
Dec 22 17:21:22 rerpdb1601 s3fs[21838]: [tid-21845][INF]      s3fs:s3fs_release(2185):  [path=/tmp/7kAcr][fd=6]

403 error when touching a file

Cause: A 403 error is usually caused by a permission issue. You might receive a 403 error when you run the touch command in the following cases:

The file is an Archive Storage object. Touching this type of object returns a 403 error.
The AccessKey that you use does not have permission to perform operations on the bucket.

Solution:

For Archive Storage objects: Restore the object before accessing it, or enable real-time access for Archive Storage objects in the bucket.
For permission errors: Grant the required permissions to the account that owns the AccessKey.

rm command error: Operation not permitted

Cause: The rm command calls the DeleteObject API operation. If you mounted the bucket as a RAM user, ensure the RAM user has permission to delete objects.

Solution: Grant the required permissions to the RAM user. For more information, see RAM policy overview and Common examples of RAM policies.

Error: Invalid endpoint

Cause: This error indicates that the endpoint is incorrectly specified. This can happen for the following reasons:

The bucket and the endpoint do not match.
The UID of the bucket owner does not match the UID associated with the AccessKey.

Solution: Verify and correct your configuration.

Mount issues

Custom domain name support

No, ossfs does not support mounting with custom domain names.

HTTPS mount fails on CentOS 7.x

Solution:

Add the -ocurldbg option when you mount the bucket and check whether the logs contain NSS error -8023 (SEC_ERROR_PKCS11_DEVICE_ERROR).
If this error is present in the logs, check the locally installed NSS version.

If the NSS version is 3.36, run the yum update nss command to update NSS, and then try mounting the bucket again.

Mounting fails with ECS RAM role

Solution:

Run the curl http://100.100.100.200/latest/meta-data/ram/security-credentials/[your-ecs-ram-role] command to check connectivity.
If the curl command succeeds, add the -o disable_imdsv2 option when you mount the bucket.

Mount error: Transport endpoint is not connected

Cause: The specified directory does not exist.

Solution: Create the directory, and then perform the mount operation.

Mount error: Permission denied

Cause: This is a bug in FUSE. FUSE requires the current user to have read permissions on the current working directory, which is not the mount point.

Solution: Use the cd command to switch to a directory for which you have read permissions, and then run the ossfs command.

Mount error: Mountpoint is not empty

Cause: By default, ossfs can only be mounted to an empty directory. This error occurs when mounting ossfs to a non-empty directory.

Solution: Switch to an empty directory and mount again. If you still need to mount to this directory, add the -ononempty parameter when mounting.

Mount timeout error

Cause: The ossfs mount request timed out.

Solution: ossfs uses the readwrite_timeout option to set the timeout period in seconds for read and write requests. The default value is 60. Increase this value based on your workload.

Mount error: Incorrect credential file permissions

Cause: The permissions for the /etc/passwd-ossfs file are incorrect.

Solution: The /etc/passwd-ossfs file stores credentials, so you must restrict access for other users. Run the chmod 640 /etc/passwd-ossfs command to change the file permissions.

ls error: Operation not permitted

Cause: Check whether your bucket contains objects with names that contain invisible characters. File systems enforce strict limits on file and directory names, which can cause this error.

Solution: Rename these objects using another tool. Then, the ls command can list the directory contents correctly.

Mount error: FUSE device not found

Cause: This error usually occurs within a Docker container because the container does not have permission to access or load the FUSE kernel module.

Solution: When running in a Docker container, you can add the --privileged=true parameter to grant the container higher privileges. This allows processes inside the container to perform operations similar to those on the host machine, such as using the FUSE file system. The following is an example command to start a container by using the --privileged flag:

docker run --privileged=true -d your_image

Mount error: libcrypto.so.1.1 not found

Cause: The installation package version does not match your operating system version.

Solution: To resolve this issue, download the installation package that matches your operating system.

Billing

Avoid costs from background file scans

Cause: When a program scans a directory mounted by ossfs, the scan generates requests to OSS. A high request rate can incur fees.

Solution: Use the auditd tool to identify which processes scan the OSS-mounted directory. Perform the following steps:

Install and start auditd.

sudo apt-get install auditd
sudo service auditd start

Set the OSS mount point as the watch directory. For example, if the mount point is /mnt/ossfs, run the following command:
```
auditctl -w /mnt/ossfs
```
Check the audit log to see which processes accessed the directory.
```
ausearch -i | grep /mnt/ossfs
```
Configure the process to skip scanning the mount point.
For example, if the audit log shows that updatedb scanned the mounted directory, modify the /etc/updatedb.conf file to exclude it. To do so, perform the following steps:
1. In the RUNEFS = parameter, add fuse.ossfs.
2. In the PRUNEPATHS = parameter, add the mount point.

Disk and memory issues

Nov  8 02:58:28 izw9e9xgcy43hdoultcoez kernel: [30273]     0 30273  33017    827   61     0  AliYunDun
Nov  8 02:58:28 izw9e9xgcy43hdoultcoez kernel: [16398]     0 16398  1301998  317459   955     0  java
Nov  8 02:58:28 izw9e9xgcy43hdoultcoez kernel: Out of memory: Kill process 3017 (ossfs) score 58 or sacrifice child
Nov  8 02:58:28 izw9e9xgcy43hdoultcoez kernel: Killed process 3017 (ossfs) total-vm:3840372kB, anon-rss:4787892kB, file-rss:208kB, shmem-rss:0kB
Nov  8 02:58:45 izw9e9xgcy43hdoultcoez s3fs s3fs[2968]: s3fs.cpp:list_bucket(2382): list_bucket returns error(-1).
Nov  8 02:58:45 izw9e9xgcy43hdoultcoez s3fs s3fs[2968]: s3fs.cpp:list_bucket(2459): xmlReadMemory with error...
Nov  8 02:58:45 izw9e9xgcy43hdoultcoez s3fs s3fs[2968]: s3fs.cpp:list_bucket(2382): list_bucket returns error(-1).
Nov  8 02:58:47 izw9e9xgcy43hdoultcoez systemd: Started Session 4614 of user root.

ossfs disconnects intermittently

Cause:

Check the debug logs for out-of-memory (OOM) errors. To do this, enable debug logging by adding the -d -odbglevel=dbg parameter. ossfs writes logs to the default system log file.
- On CentOS systems, logs are written to /var/log/message.
- On Ubuntu systems, logs are written to /var/log/syslog.
The logs may show that high memory usage during listbucket and listobject operations triggered the system's OOM killer, terminating the ossfs process.

Note
The listobject operation sends an HTTP request to OSS to retrieve object metadata. If you have many objects, the ls command can consume a large amount of system memory to fetch the metadata.

Solution:

Increase the stat cache size using the -omax_stat_cache_size=xxx parameter. The first ls operation is slower, but subsequent calls are faster because metadata is cached locally. The default value is 1,000, which consumes about 4 MB of memory. Adjust this value based on your machine's available memory.
During read and write operations, ossfs can write large amounts of temporary cache data to the disk, similar to Nginx. This can exhaust the available disk space. ossfs automatically cleans up temporary files when it exits.
Use ossutil instead of ossfs. ossfs can be used for non-production workloads. For services that require high reliability and stability, we recommend that you use ossutil.

Why does ossfs fill up disk space?

Cause: To improve performance, ossfs by default uses as much disk space as possible to store temporary upload and download data. This can exhaust the available disk space.

Solution: Use the -oensure_diskfree option to reserve disk space. For example, to reserve 20 GB of disk space, run the following command:

ossfs examplebucket /tmp/ossfs -o url=http://oss-cn-hangzhou.aliyuncs.com -oensure_diskfree=20480

The df command shows 256 TB disk size

The disk size shown by the df command is for display purposes only and does not represent the actual bucket capacity. The values of Size (total disk space) and Avail (available disk space) are fixed at 256 TB, and the value of Used (used disk space) is fixed at 0 TB.

OSS provides unlimited bucket capacity. The storage usage depends on your actual usage. For more information about how to query bucket usage, see Query bucket-level usage.

[root@rjhdadmin xxx]# du -sh *
24G    caohh.zip
[root@rjhdadmin lixiao]# cp caohh.zip  /data/ossfs/xxx
cp: writing '/data/ossfs/xxx/caohh.zip': Input/output error
cp: closing '/data/ossfs/xxx.zip': Input/output error
[root@rjhdadmin xxx]#

cp command: Input/output error

Cause: This error indicates a problem with disk I/O. Check whether the disk has a high read/write load when the error occurs.

Solution: Add part-related parameters to control file read and write operations. Run the ossfs -h command to view the available part-related parameters.

cp: writing '/data/tmp/I_201704/12/request.1491926430217.avro': Input/output error
cp: closing '/data/tmp/I_201704/12/request.1491926430217.avro': Input/output error

I/O error when copying large files

Cause: Using ossfs with rsync can cause issues. In this case, a user ran the cp command on a 141 GB file, which placed the disk under a very high load and caused this error.

Solution: To download files from OSS to an ECS instance or upload files from an ECS instance to OSS, use the multipart upload and download features of ossutil.

[root@c653a8832c9eb4d8d9xxx xxx a-node5 ~]# ossfs xxx xxx xxx /xxx/daily -o multipart_size=31457280 -ourl=http://oss-ap-southeast-3-internal.aliyuncs.com
ossfs: There is no enough disk space for used as cache(or temporary) directory by s3fs.
[root@c653a8832c9eb4d8d9xxx a-node5 ~]#

Upload error: Not enough disk space for cache

Cause

The available disk space is less than multipart_size * parallel_count.

The multipart_size parameter specifies the part size. The default unit is MB. The parallel_count parameter specifies the number of parts to upload in parallel. The default value is 5.
Analysis

By default, ossfs uploads large files using multipart upload. During an upload, ossfs writes temporary cache files to the /tmp directory. Before writing data, ossfs checks whether the free space on the disk that contains the /tmp directory is less than multipart_size * parallel_count. If the free space is greater than multipart_size * parallel_count, the write proceeds as normal. If the free space is less than multipart_size * parallel_count, ossfs returns an error indicating insufficient local disk space.

For example, the available disk space is 300 GB and the file to be uploaded is 200 GB. If multipart_size is set to 100000 (100 GB) and parallel_count remains at the default value of 5, ossfs calculates the required space as 100 GB × 5 = 500 GB, which exceeds the local free space.
Solution

If parallel_count remains at the default value of 5, set multipart_size to a reasonable value.
- If the available disk space is 300 GB and the file to be uploaded is 200 GB, set multipart_size to 20.
- If the available disk space is 300 GB and the file to be uploaded is 500 GB, set multipart_size to 50.

Dependency issues

Installation error: FUSE library too old

Cause: The libfuse version used to compile ossfs is higher than the libfuse version linked at runtime. This often happens if you manually install libfuse. On CentOS 5.x and CentOS 6.x systems, the ossfs package provided by Alibaba Cloud includes libfuse-2.8.4. If libfuse-2.8.3 exists in the runtime environment and ossfs links to the older FUSE version, this warning appears.

Run the ldd $(which ossfs) | grep fuse command to confirm which FUSE version ossfs links to at runtime. If the result is /lib64/libfuse.so.2, run the ls -l /lib64/libfuse* command to view the FUSE version.

Solution: Configure ossfs to link to the correct version.

Run the rpm -ql ossfs | grep fuse command to find the libfuse directory.
If the result is /usr/lib/libfuse.so.2, run ossfs by using the LD_LIBRARY_PATH=/usr/lib ossfs … command.

Total size: 6.0 M
Installed size: 6.0 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Check Error:
  file /sbin/mount.fuse from install of ossfs-1.80.3-1.x86_64 conflicts with file from package fuse-2.8.3-5.el6.x86_64
  file /usr/bin/fusermount from install of ossfs-1.80.3-1.x86_64 conflicts with file from package fuse-2.8.3-5.el6.x86_64
  file /usr/bin/ulockmgr_server from install of ossfs-1.80.3-1.x86_64 conflicts with file from package fuse-2.8.3-5.el6.x86_64
Error Summary
-------------
[root@localhost 123]# rpm -qa |grep fuse
fuse-2.8.3-5.el6.x86_64
fuse-ntfs-3g-2013.1.13-2.el6.rf.x86_64
You have new mail in /var/spool/mail/root
[root@localhost 123]#

FUSE dependency installation error

Cause: The FUSE version does not meet the requirements of ossfs.

Solution: Manually download and install the latest version of FUSE. Do not use yum. For more information, see fuse.

ls error: Input/output error

Cause: This issue mainly occurs in CentOS environments. The log shows the NSS error -8023 error. An issue occurs when ossfs uses libcurl for HTTPS communication, possibly because the Network Security Services (NSS) library that libcurl depends on is too old.

Solution: Run the following command to update the NSS library to the latest version:

yum update nss

Installation error: Conflicts with fuse-devel

Cause: An old version of FUSE exists on the system and conflicts with the dependency version in ossfs.

Solution: Uninstall FUSE by using your package manager, and then reinstall ossfs.

Other issues

ossfs is slow with many files or high concurrency

ossfs 1.0 is not recommended for high-concurrency scenarios. If you must use it in these scenarios, consider the following options:

Option 1: Mount the bucket by using ossfs 2.0. Compared with ossfs 1.0, ossfs 2.0 provides major performance improvements for sequential read and write operations and for concurrent reads of small files. For more information, see Performance improvements.
Option 2: Use Cloud Storage Gateway to mount an OSS bucket for better performance. The performance of ossfs 1.0 is not suitable for high-concurrency workloads or large file uploads and downloads. It is best suited for routine operations on small files.

Uploaded files have incorrect Content-Type

Cause: When uploading files, ossfs sets the Content-Type by reading the /etc/mime.types file. If this file does not exist, ossfs defaults to application/octet-stream.

Solution: Check whether this file exists. If not, add it.

Automatically add the mime.types file using a command.
- For Ubuntu systems:
  
  Run the sudo apt-get install mime-support command.
- For CentOS systems:
  
  Run the sudo yum install mailcap command.
Manually add the mime.types file.
1. Create the mime.types file.
```
vi /etc/mime.types
```
2. Add the required formats, one per line. Use the format application/javascript js.

After you add the file, you must remount the OSS bucket.

Folder identified as a file

Scenario 1

Cause: If you create a folder object (an object whose name ends with a forward slash (/)) and set its Content-Type to text/plain, ossfs identifies the object as a file.

Solution: You can add the -ocomplement_stat parameter during mounting. If a folder object has a size of 0 or 1, ossfs recognizes it as a folder.
Scenario 2
Cause: Run the ossutil stat command, where <folder-object> is an object name that ends with a forward slash (/). For example, ossutil stat oss://[bucket]/folder/. After you run the command, check the following fields:
1. Check the Content-Length field, which indicates the object size. If the object size is not 0, the object is identified as a file.
  
  Solution: If you no longer need the content of this folder object, run the ossutil rm oss://[bucket]/folder/ command to delete the object. This operation does not affect the files in the folder. Alternatively, you can use ossutil to upload a zero-byte object with the same name to overwrite the folder object.
2. If the object size is 0, check the Content-Type field. If the value of this field is not application/x-directory, httpd/unix-directory, binary/octet-stream, or application/octet-stream, the object is also identified as a file.
  
  Solution: Run the ossutil rm oss://[bucket]/folder/ command to delete the object. This operation does not affect the files in the folder.

mv operation fails

Cause: The mv operation in ossfs can fail if the source file is an Archive Storage, Cold Archive, or Deep Cold Archive object.

Solution: Before you run the mv operation, you must restore the object. For more information, see Restore objects.

Windows support

No. In Windows, you can mount a bucket by using Rclone. For more information, see Rclone.

Mounting on multiple Linux servers

Yes. You can mount it on multiple Linux ECS servers. For more information, see Mount a bucket.

Inconsistent file information across tools

Cause: By default, ossfs caches file metadata, including the file size and permissions, to improve performance. This avoids sending a request to OSS each time you run the ls command. If you modify files by using other tools such as an SDK, the OSS console, or ossutil, ossfs may not refresh its cache in time, resulting in data inconsistencies.

Solution: Add the -omax_stat_cache_size=0 parameter when you mount the bucket to disable metadata caching. Each time the ls command is executed, a request is sent to OSS to fetch the latest file information.

Slow mount after enabling versioning

Cause: By default, ossfs lists files using the ListObjects (GetBucket) operation. If versioning is enabled and the bucket contains one or more historical versions of objects and a large number of expired delete markers, listing the current objects using ListObjects (GetBucket) can be slow, which degrades mount performance.

Solution: Add the -olistobjectsV2 option to switch ossfs to use the ListObjectsV2 (GetBucketV2) operation for better listing performance.

Mounting over HTTPS

ossfs supports mounting over HTTPS. The following is a sample command for the China (Hangzhou) region:

ossfs examplebucket /tmp/ossfs -o url=https://oss-cn-hangzhou.aliyuncs.com

ls command is slow in large directories

Cause: If a directory contains N files, listing its contents requires at least N OSS HTTP requests. This can cause serious performance issues when N is large.

Solution: Increase the stat cache size using the -omax_stat_cache_size=xxx parameter. The first ls operation is slower, but subsequent calls are faster because metadata is cached locally. Before ossfs 1.91.1, the default value of this parameter is 1,000. In ossfs 1.91.1 and later, the default value is 100,000, which consumes tens of megabytes of memory. Adjust this value based on your machine's available memory.

Unmount error: Device or resource busy

Cause: A process is accessing files under the mount point /mnt/ossfs-bucket, which prevents the directory from being unmounted.

Solution:

Run the lsof /mnt/ossfs-bucket command to find the process that is using the directory.
Use the kill command to terminate the process.
Run the fusermount -u /mnt/ossfs-bucket command to unmount the bucket.

Large number of 404 logs in ossfs 1.0

Background information: When you use ossfs 1.0, it is common to see 404 Not Found records in logs. In most cases, this is not a system error. This is expected behavior as ossfs 1.0 simulates a local file system.

Before operating on a file, the operating system checks whether the target object exists. This process can trigger many probe requests to OSS. If an object does not exist, OSS returns a 404 status code.

The following steps describe how ossfs 1.0 checks whether an object exists:

ossfs sends a HeadObject request to check whether the specified path, such as object, exists as an actual object.

If the object exists, the system returns the object metadata. If the object does not exist, the system returns a 404 error and proceeds to the next step.
After it receives the 404 error, ossfs sends a HeadObject request to check whether an object/ object exists.

If the object exists, the system returns the object metadata. If the object does not exist, the system returns a 404 error and proceeds to the next step.
After it receives the 404 error, ossfs sends a HeadObject request to check whether an object_$folder$ object exists.

If the object exists, the system returns the object metadata. If the object does not exist, the system returns a 404 error and proceeds to the next step.
After it receives the 404 error, ossfs sends a ListObjects request to determine whether the specified path is a directory by listing objects with the prefix object/.

If the result is empty, the path does not exist. If the result is not empty, the directory exists and the system lists the directory contents.

Cause:

If you use commands such as stat to access a file that does not exist, OSS returns a 404 error, which is mapped to the local file system error "No such file or directory".
Before creating files or directories in bulk, the operating system checks if the target exists and sends a create request only if it does not. The 404 errors generated during this check are expected behavior, not a system issue.

Solution: Although a 404 error indicates an expected behavior, frequent probe requests can affect performance in high-concurrency or bulk operation scenarios. You can use the following options to optimize performance:

Important

After you configure these options, ossfs 1.0 cannot detect changes made to objects in OSS until the local cache expires.

Increase the -o stat_cache_expire and -o max_stat_cache_size values to extend the metadata cache duration and increase the number of cache entries.

If you query a file or directory before its metadata expires, ossfs avoids sending repeated requests to OSS.
- -o stat_cache_expire: specifies the expiration time of the metadata cache. The default value is 900s.
- -o max_stat_cache_size: specifies the number of entries in the metadata cache. The default value is 100000.
If you use a version of ossfs earlier than 1.91.6, enable the negative cache using the -o enable_noobj_cache option. In ossfs 1.91.6 and later, the negative cache is enabled by default.

When a file is queried for the first time and does not exist, ossfs caches the result in memory. Before the cache expires, subsequent queries use the local negative cache instead of sending requests to OSS.

Note
In ossfs 1.0, the file negative cache is part of the metadata cache. You can use the -o stat_cache_expire parameter to control the expiration time and the -o max_stat_cache_size parameter to control the maximum number of cache entries.