This topic describes how to use the CLI to manage Cloud Storage Gateway (CSG) software gateways.
Prerequisites
An Alibaba Cloud account is created and real-name verification for the account is complete. For more information, see Create an Alibaba Cloud account.
NoteWe recommend that you manage CSG resources in the CSG console as a RAM user. For more information, see Use RAM to implement account-based access control.
CSG is activated. If CSG is not activated, follow the on-screen instructions in the CSG console to activate CSG.
A client is deployed. The following client requirements apply:
CSG software cloud gateway: A client that runs on an Elastic Compute Service (ECS) instance is available in the region where you want to deploy the CSG software cloud gateway. For more information, see Create an ECS instance.
NoteSoftware cloud gateways are supported in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Shenzhen), China (Heyuan), China (Chengdu), and China (Hong Kong).
Software cloud gateways support clients that run one of the following operating systems:
Linux: CentOS 7, CentOS 8, Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04
Windows: Windows Server 2016 and Windows Server 2019
Software on-premises gateway: A client for a software on-premises gateway must meet the following requirements:
NoteSoftware on-premises gateways support clients that run one of the following operating systems:
Linux: CentOS 7, CentOS 8, Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04
Windows: Windows Server 2016 and Windows Server 2019
The client has access to the Internet or is connected to a virtual private cloud (VPC) over an Alibaba Cloud Express Connect circuit.
An Object Storage Service (OSS) bucket is created. For more information, see Create buckets.
NoteSoftware gateways support Standard and Infrequent Access (IA) OSS buckets.
An AccessKey pair is created for the Alibaba Cloud account. For more information, open the RAM console.
Billing
The CSG software edition is in public review and you are not charged software gateway fees during this period.
Data storage in OSS buckets incurs fees that are calculated in OSS.
Step 1: Download the CSG software installation package
The latest package version is 0.6.2.
Linux
Download links:
Ubuntu 16.04: CsgAgent-latest-ubuntu1604.deb
Ubuntu 18.04: CsgAgent-latest-ubuntu1804.deb
Ubuntu 20.04: CsgAgent-latest-ubuntu2004.deb
Ubuntu 22.04: CsgAgent-latest-ubuntu2204.deb
CentOS 7 and Alibaba Cloud Linux 2: CsgAgent-latest-centos7.rpm
CentOS 8 and Alibaba Cloud Linux 3: CsgAgent-latest-centos8.rpm
Windows
Download links:
Windows Server 2016 and Windows Server 2019: CsgAgent-latest-win64.zip
Step 2: Install the package of the CSG software gateway
If the installation of the CSG software gateway package fails due to the absence of specific dependencies, follow the on-screen instructions to install the dependencies.
CentOS and Alibaba Cloud Linux
This example shows you how to install the package for CentOS 7.
Download the CsgAgent-latest-centos7.rpm installation package to the client.
Run the following commands to install the package:
sudo rpm -i CsgAgent-latest-centos7.rpm sudo /usr/local/bin/Aliyun/csg/scripts/install.sh
Run the
csgfsctl --help
command to check the installation status.
Ubuntu
This example shows you how to install the package for Ubuntu 16.04.
Download the CsgAgent-latest-ubuntu1604.deb installation package to the client.
Run the following commands to install the package:
sudo apt install -y ./CsgAgent-latest-ubuntu1604.deb sudo /usr/local/bin/Aliyun/csg/scripts/install.sh
Run the
csgfsctl --help
command to check the installation status.
Windows
Download the CsgAgent-latest-win64.zip installation package to the client and decompress the package.
NoteThe files in the installation package must be extracted to the same directory.
We recommend that you maintain the directory structure when you decompress the installation package.
Run install.bat as an administrator to install the package.
Run the
csgfsctl --help
command to check the installation status.NoteIf the installation fails, check the client for antivirus software. If antivirus software is running on the client, disable it and install the package again.
Step 3: Activate the software gateway
Before you can use a software gateway, you must activate the gateway.
To use the csgfsctl tool on Windows, you must run cmd.exe as an administrator.
Run
csgfsctl activate --ak=xxx --sk=xxx
to activate a software cloud gateway.NoteTo activate a software gateway, you must provide a valid AccessKey pair. You can use the AccessKey pair of a RAM user that is assigned the AliyunHCSSGWFullAccess policy.
Run the
csgfsctl activate --region=xxx --ak=xxx --sk=xxx
command to activate a software on-premises gateway.NoteTo activate a software on-premises gateway, you must provide a valid AccessKey pair (AccessKey ID and AccessKey secret) and the region where you want to deploy the gateway. The regions supported for software on-premises gateways are the same as those for software cloud gateways. For example, you specify the China (Beijing) region and provide a valid AccessKey pair. You can use the AccessKey pair of a RAM user that is assigned the AliyunHCSSGWFullAccess policy.
After successful activation, you can select the corresponding region in the CSG console to view the information about the gateway.
Step 4: Mount a bucket
This step provides an example on how to mount a bucket on a client that runs on an ECS instance in the China (Beijing) region.
Linux: For example, to mount the test subdirectory of the beijing bucket at the /mnt/test mount point and use /root/cache as the cache path to provide 50 GB of cache (10 GB for metadata storage and 40 GB for file storage), run the following command:
sudo csgfsctl create --mp=/mnt/test --cd=/root/cache --ak=xxx --sk=xxx --ep=oss-cn-beijing-internal.aliyuncs.com --bucket=beijing -c=50 --dc=40 --prefix=test --work-mode=standard
Windows: For example, to mount the test subdirectory of the beijing bucket at the D:\test mount point and use D:\cache as the cache path to provide 50 GB of cache (10 GB for metadata storage and 40 GB for file storage), run the following command:
csgfsctl create --mp=D:\test --cd=D:\cache --ak=xxx --sk=xxx --ep=oss-cn-beijing-internal.aliyuncs.com --bucket=beijing -c=50 --dc=40 --prefix=test --work-mode=standard
After mounting is complete, do not modify or delete files in the cache-dir directory to prevent possible consequences such as mount point unavailability.
You can run the csgfsctl create --help
command to view how to create a mount point. The following table describes the parameters.
Parameter | Required | Description |
--mountpoint (abbreviation: -mp) | Yes | The path of the mount point. Example: /mnt/test/. Note
|
--cache-dir (abbreviation: -cd) | Yes | The cache path. The cache path provides cache storage for metadata and files. Warning
|
--access-key (abbreviation: -ak) | Yes | The AccessKey ID of the account. Enter the AccessKey ID of an Alibaba Cloud account or a RAM user that has the read and write permissions on the bucket. |
--access-key-secret (abbreviation: -sk) | Yes | The AccessKey secret of the account. |
--endpoint (abbreviation: -ep) | Yes | The endpoint of the bucket to which you want to map. Note If you use a client that runs on an ECS instance, take note of the following items:
|
--bucket (abbreviation: -b) | Yes | The name of the OSS bucket. |
--prefix (abbreviation: -p) | No | The subdirectory of the OSS bucket. Note If this parameter is left empty, the root directory of the bucket is mounted on the mount point. If this parameter is not empty, the specified bucket subdirectory is mounted on the mount point. |
--cache-size-gb (abbreviation: -c) | Yes | The disk space for cache storage. Unit: GB. Note
|
--data-cache-size-gb (abbreviation: -dc) | No | The disk space for file cache. Unit: GB. Note
|
--work-mode (abbreviation: -wm) | No | The read/write pattern. The default value is async-write, which indicates the asynchronous write pattern. The asynchronous write pattern uses memory to speed up reads/writes that require small I/O, causing high memory usage. Other valid values:
|
--allow-rename-dir | No | Specifies whether to allow directory renaming. Valid values:
Note OSS does not support renaming. Renaming a directory actually copies all objects in the directory. If your scenario does not require directory renaming, set this parameter to false. |
After you configure the software gateway, you can access a share on the software gateway in the same way as you access an on-premises directory.
The share of a software gateway is synchronized to the connected OSS bucket. Operations on the files in the share are applied to the replicas in the OSS bucket.
Related operations
Query a mount point
To query a mount point, run the sudo csgfsctl get
command. The following table describes the parameters.
You can run the csgfsctl get --help
command to learn how to query a mount point.
Parameter | Required | Description |
--mountpoint (abbreviation: -mp) | No | The path of the mount point. Note If you specify a mount point, information about the mount point is returned. If you do not configure this parameter, information about all the mount points on the client is returned. |
The following table describes some of the parameters returned by the get command.
Parameter | Description |
RemainingMetaSpace | The remaining metadata cache size. Note
|
State | The synchronization status of the mount point. Note One of the following values is returned for the parameter:
|
Healthy | Indicates whether the mount point is normal. One of the following values is returned for the parameter:
|
Modify mount point configurations
Linux: The following sample code enables the reverse synchronization feature for the /mnt/test mount point and sets the reverse synchronization interval to 600 seconds:
sudo csgfsctl set --mountpoint=/mnt/test --reverse-sync=true --rs-interval=600
Windows: The following sample code enables the reverse synchronization feature for the D:\test mount point and sets the reverse synchronization interval to 600 seconds:
csgfsctl set --mountpoint=D:\test --reverse-sync=true --rs-interval=600
You can run the csgfsctl set --help
command to learn how to modify mount point configurations. The following table describes the parameters.
Parameter | Required | Description |
--mountpoint (abbreviation: -mp) | Yes | The path of the mount point. |
--reverse-sync (abbreviation: -rs) | No | Specifies whether to enable the reverse synchronization feature. One of the following values is returned for the parameter:
Note If the reverse synchronization feature is enabled, the gateway synchronizes metadata from the bucket to the local file system. The reverse synchronization feature can decrease the gateway performance. |
--rs-interval (abbreviation: -rsi) | No | The interval for reverse synchronization. Unit: seconds. Note
|
--ignore-delete | No | Specifies whether to enable Ignore Deletions. Valid values:
Note If Ignore Deletions is enabled, the delete operation on the files in the cache is not applied to the objects in the bucket. |
--oss-direct-read | No | Specifies whether to enable Bypass Cache Read. We recommend that you do not set this parameter to true in most cases. Valid values:
Note By default, if data requested in a read request is not found in the cache, the gateway downloads the data from the bucket and stores the data to the on-premises disk cache. This process involves data pre-reading. However, if read requests are random and the file cache size is considerably less than the bucket capacity, data pre-reading performance and downloading performance may be unsatisfactory. In this case, we recommend that you set this parameter to true. |
--allow-rename-dir | No | Specifies whether to allow directory renaming. Valid values:
Note OSS does not support renaming. Renaming a directory actually copies all objects in the directory. We recommend that you forbid directory renaming in most cases for optimized user experience. |
--upload-delay | No | The latency in synchronizing data from the on-premises cache to the bucket. Unit: seconds. Note
|
--upload-bandwidth-limit | No | The maximum upload bandwidth. Unit: MB/s. |
--rw-bandwidth-limit | No | The maximum read/write bandwidth. Unit: MB/s. |
--download-bandwidth-limit | No | The maximum download bandwidth. Unit: MB/s. |
--cache-size-gb (abbreviation: -c) | No | The size of the cache space after cache scaling. Unit: GB. Note
|
--endpoint (abbreviation: -ep) | No | The endpoint of the bucket. If the existing endpoint is no longer valid, you can use this parameter to specify a different endpoint. |
--seq-mode | No | Specifies whether to use the sequential write pattern. Valid values:
Note
|
--silent (abbreviation: -s) | No | Disables confirmation when you change write patterns. If you do not specify this parameter, you are required to enter y or N to confirm whether to switch the write pattern. To skip confirmation, specify this parameter in the command. |
Delete a mount point
Before you delete a mount point, make sure that the mount point or cache path is not being used by any process. This includes checking whether a process is using the mount point path or cache path. If the mount point or cache path is being used by a process, the mount point may fail to be deleted, or incomplete deletion may occur.
Make sure that you do not create new files or directories in the cache path. When you delete the mount point, the cache path is cleared.
You can run the csgfsctl remove --help
command to learn how to delete a mount point. The following table describes the parameters.
Parameter | Required | Description |
--mountpoint (abbreviation: -mp) | Yes | The path of the mount point. |
--force | No |
|
Logging
To collect logs, run the csgfsctl log
command.
Linux: The generated logs are located in the /usr/local/bin/Aliyun/csg/ path.
Windows: The generated logs are located in the C:\Program Files\Aliyun\csg\bin path.
Uninstall the software gateway
You must use the one of the following methods to uninstall the software gateway. Otherwise, you may encounter unexpected issues, such as service unavailability, unexpected bills, and uninstallation failure.
Linux: Run
sudo /usr/local/bin/Aliyun/csg/scripts/uninstall.sh
.Windows: Run uninstall.bat as an administrator.
Before you uninstall a software gateway, you must manually delete all mount points associated with the gateway. Otherwise, the uninstallation fails.
Before you delete a mount point, make sure that the mount point or cache path is not being used by any process. This includes checking whether a process is using the mount point path or cache path. If the mount point or cache path is being used by a process, the mount point may fail to be deleted, or incomplete deletion may occur.
Upgrade the software gateway
If the software gateway is not the latest version, you can upgrade the software gateway.
You can use the
csgfsctl -v
command to check the version number of your software gateway.You do not need to delete mount points associated with the gateway for the upgrade operation. After the upgrade is completed, the mount points are automatically restored.
During the upgrade, the mount points are temporarily unavailable. We recommend that you upgrade your gateway during off-peak business hours.
Before you upgrade your software gateway, make sure that no processes except for csg_agent on Linux are using the mount points or cache path of the software gateway. This includes checking whether a process is using the mount point path or cache path.
On Linux, you can run lsof <mount point path>
to check whether the mount point is being used.
CentOS and Alibaba Cloud Linux (CentOS 7 is used in this example)
Download the latest installation package: CsgAgent-latest-centos7.rpm.
Run the following commands to upgrade the gateway:
sudo rpm -iU CsgAgent-latest-centos7.rpm sudo /usr/local/bin/Aliyun/csg/scripts/upgrade.sh
Run the
csgfsctl -v
command to check whether the upgrade is successful.
Ubuntu (Ubuntu 16.04 is used in this example)
Download the latest installation package: CsgAgent-latest-ubuntu1604.deb.
Run the following commands to upgrade the gateway:
sudo apt install --only-upgrade -y ./CsgAgent-latest-ubuntu1604.deb sudo /usr/local/bin/Aliyun/csg/scripts/upgrade.sh
Run the
csgfsctl -v
command to check whether the upgrade is successful.
Windows
Download and decompress the latest installation package: CsgAgent-latest-win64.zip.
NoteThe files in the installation package must be extracted to the same directory.
We recommend that you maintain the directory structure when you decompress the installation package.
Run upgrade.bat as an administrator to upgrade the gateway.
Run the
csgfsctl -v
command to check whether the upgrade is successful.