All Products
Search
Document Center

Cloud Storage Gateway:Manage CSG software gateways by using the CLI

Last Updated:Mar 11, 2024

This topic describes how to use the CLI to manage Cloud Storage Gateway (CSG) software gateways.

Prerequisites

  1. An Alibaba Cloud account is created and real-name verification for the account is complete. For more information, see Create an Alibaba Cloud account.

    Note

    We recommend that you manage CSG resources in the CSG console as a RAM user. For more information, see Use RAM to implement account-based access control.

  2. CSG is activated. If CSG is not activated, follow the on-screen instructions in the CSG console to activate CSG.

  3. A client is deployed. The following client requirements apply:

    1. CSG software cloud gateway: A client that runs on an Elastic Compute Service (ECS) instance is available in the region where you want to deploy the CSG software cloud gateway. For more information, see Create an ECS instance.

      Note
      • Software cloud gateways are supported in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Zhangjiakou), China (Hohhot), China (Shenzhen), China (Heyuan), China (Chengdu), and China (Hong Kong).

      • Software cloud gateways support clients that run one of the following operating systems:

        • Linux: CentOS 7, CentOS 8, Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04

        • Windows: Windows Server 2016 and Windows Server 2019

    2. Software on-premises gateway: A client for a software on-premises gateway must meet the following requirements:

      Note
      • Software on-premises gateways support clients that run one of the following operating systems:

        • Linux: CentOS 7, CentOS 8, Alibaba Cloud Linux 2, Alibaba Cloud Linux 3, Ubuntu 16.04, Ubuntu 18.04, Ubuntu 20.04, and Ubuntu 22.04

        • Windows: Windows Server 2016 and Windows Server 2019

      • The client has access to the Internet or is connected to a virtual private cloud (VPC) over an Alibaba Cloud Express Connect circuit.

  4. An Object Storage Service (OSS) bucket is created. For more information, see Create buckets.

    Note

    Software gateways support Standard and Infrequent Access (IA) OSS buckets.

  5. An AccessKey pair is created for the Alibaba Cloud account. For more information, open the RAM console.

Billing

The CSG software edition is in public review and you are not charged software gateway fees during this period.

Note

Data storage in OSS buckets incurs fees that are calculated in OSS.

Step 1: Download the CSG software installation package

The latest package version is 0.6.2.

Linux

Download links:

Windows

Download links:

Windows Server 2016 and Windows Server 2019: CsgAgent-latest-win64.zip

Step 2: Install the package of the CSG software gateway

Note

If the installation of the CSG software gateway package fails due to the absence of specific dependencies, follow the on-screen instructions to install the dependencies.

CentOS and Alibaba Cloud Linux

This example shows you how to install the package for CentOS 7.

  1. Download the CsgAgent-latest-centos7.rpm installation package to the client.

  2. Run the following commands to install the package:

    sudo rpm -i CsgAgent-latest-centos7.rpm
    sudo /usr/local/bin/Aliyun/csg/scripts/install.sh
  3. Run the csgfsctl --help command to check the installation status.

Ubuntu

This example shows you how to install the package for Ubuntu 16.04.

  1. Download the CsgAgent-latest-ubuntu1604.deb installation package to the client.

  2. Run the following commands to install the package:

    sudo apt install -y ./CsgAgent-latest-ubuntu1604.deb
    sudo /usr/local/bin/Aliyun/csg/scripts/install.sh
  3. Run the csgfsctl --help command to check the installation status.

Windows

  1. Download the CsgAgent-latest-win64.zip installation package to the client and decompress the package.

    Note
    • The files in the installation package must be extracted to the same directory.

    • We recommend that you maintain the directory structure when you decompress the installation package.

  2. Run install.bat as an administrator to install the package.

  3. Run the csgfsctl --help command to check the installation status.

    Note

    If the installation fails, check the client for antivirus software. If antivirus software is running on the client, disable it and install the package again.

Step 3: Activate the software gateway

Before you can use a software gateway, you must activate the gateway.

Note

To use the csgfsctl tool on Windows, you must run cmd.exe as an administrator.

  • Run csgfsctl activate --ak=xxx --sk=xxx to activate a software cloud gateway.

    Note

    To activate a software gateway, you must provide a valid AccessKey pair. You can use the AccessKey pair of a RAM user that is assigned the AliyunHCSSGWFullAccess policy.

  • Run the csgfsctl activate --region=xxx --ak=xxx --sk=xxx command to activate a software on-premises gateway.

    Note

    To activate a software on-premises gateway, you must provide a valid AccessKey pair (AccessKey ID and AccessKey secret) and the region where you want to deploy the gateway. The regions supported for software on-premises gateways are the same as those for software cloud gateways. For example, you specify the China (Beijing) region and provide a valid AccessKey pair. You can use the AccessKey pair of a RAM user that is assigned the AliyunHCSSGWFullAccess policy.

After successful activation, you can select the corresponding region in the CSG console to view the information about the gateway.

Step 4: Mount a bucket

This step provides an example on how to mount a bucket on a client that runs on an ECS instance in the China (Beijing) region.

  • Linux: For example, to mount the test subdirectory of the beijing bucket at the /mnt/test mount point and use /root/cache as the cache path to provide 50 GB of cache (10 GB for metadata storage and 40 GB for file storage), run the following command:

    sudo csgfsctl create --mp=/mnt/test --cd=/root/cache --ak=xxx --sk=xxx --ep=oss-cn-beijing-internal.aliyuncs.com --bucket=beijing -c=50 --dc=40 --prefix=test --work-mode=standard
  • Windows: For example, to mount the test subdirectory of the beijing bucket at the D:\test mount point and use D:\cache as the cache path to provide 50 GB of cache (10 GB for metadata storage and 40 GB for file storage), run the following command:

    csgfsctl create --mp=D:\test --cd=D:\cache --ak=xxx --sk=xxx --ep=oss-cn-beijing-internal.aliyuncs.com --bucket=beijing -c=50 --dc=40 --prefix=test --work-mode=standard
Warning

After mounting is complete, do not modify or delete files in the cache-dir directory to prevent possible consequences such as mount point unavailability.

You can run the csgfsctl create --help command to view how to create a mount point. The following table describes the parameters.

Parameter

Required

Description

--mountpoint (abbreviation: -mp)

Yes

The path of the mount point. Example: /mnt/test/.

Note
  • When you specify a mount point, make sure that the mount point path does not contain any data.

  • Take note of the following items when you mount a directory on Windows:

    • Do not specify the root directory of an existing drive as the mount point.

    • The mount point can be a drive letter, such as Z:\.

--cache-dir (abbreviation: -cd)

Yes

The cache path. The cache path provides cache storage for metadata and files.

Warning
  • When you configure mounting, make sure that no data exists in the cache path.

  • For Windows, do not use the root directory of an existing drive as the cache path.

  • After mounting is complete, do not create a file or directory in the cache path. When you delete the mount point, the cache path is cleared.

--access-key (abbreviation: -ak)

Yes

The AccessKey ID of the account. Enter the AccessKey ID of an Alibaba Cloud account or a RAM user that has the read and write permissions on the bucket.

--access-key-secret (abbreviation: -sk)

Yes

The AccessKey secret of the account.

--endpoint (abbreviation: -ep)

Yes

The endpoint of the bucket to which you want to map.

Note

If you use a client that runs on an ECS instance, take note of the following items:

  • If you want to implement cross-region access (the client and the OSS bucket reside in different regions), you must use the public endpoint of the bucket.

  • If you do not need cross-region access, we recommend that you use the internal endpoint of the bucket for optimal access performance. Internal endpoints follow the regionID-internal.aliyuncs.com format.

--bucket (abbreviation: -b)

Yes

The name of the OSS bucket.

--prefix (abbreviation: -p)

No

The subdirectory of the OSS bucket.

Note

If this parameter is left empty, the root directory of the bucket is mounted on the mount point. If this parameter is not empty, the specified bucket subdirectory is mounted on the mount point.

--cache-size-gb (abbreviation: -c)

Yes

The disk space for cache storage. Unit: GB.

Note
  • The cache size must be 20 to 32,768 GB.

  • We recommend that you use a disk that is at least 2 GB larger than the cache size that you want to specify.

--data-cache-size-gb (abbreviation: -dc)

No

The disk space for file cache. Unit: GB.

Note
  • The cache size is the disk space for file cache plus the disk space for metadata cache.

  • If you do not configure this parameter, 70% of the cache size is reserved for file storage.

  • The file cache size cannot be less than 14 GB and cannot be greater than the reserved cache storage minus 4 GB.

--work-mode (abbreviation: -wm)

No

The read/write pattern. The default value is async-write, which indicates the asynchronous write pattern. The asynchronous write pattern uses memory to speed up reads/writes that require small I/O, causing high memory usage. Other valid values:

  • standard: provides slow reads/writes for small I/O and reduces memory usage.

  • seq-write: sequential write

  • read-only: read-only

--allow-rename-dir

No

Specifies whether to allow directory renaming. Valid values:

  • true (default)

  • false

Note

OSS does not support renaming. Renaming a directory actually copies all objects in the directory. If your scenario does not require directory renaming, set this parameter to false.

After you configure the software gateway, you can access a share on the software gateway in the same way as you access an on-premises directory.

Note

The share of a software gateway is synchronized to the connected OSS bucket. Operations on the files in the share are applied to the replicas in the OSS bucket.

Related operations

Query a mount point

To query a mount point, run the sudo csgfsctl get command. The following table describes the parameters.

You can run the csgfsctl get --help command to learn how to query a mount point.

Parameter

Required

Description

--mountpoint (abbreviation: -mp)

No

The path of the mount point.

Note

If you specify a mount point, information about the mount point is returned. If you do not configure this parameter, information about all the mount points on the client is returned.

The following table describes some of the parameters returned by the get command.

Parameter

Description

RemainingMetaSpace

The remaining metadata cache size.

Note
  • If the remaining space of the metadata cache is less than 2 GB, a warning message appears to prompt you to increase the cache size. We recommend that you use the set command to increase the cache size. For information about how to increase the cache size, see the description about the cache-size-gb parameter in the "Modify mount point configurations" section of this topic.

  • If the remaining space of the metadata cache is insufficient, an error may occur when you access the file system that is mounted on the mount point.

State

The synchronization status of the mount point.

Note

One of the following values is returned for the parameter:

  • dirty: Data in the mount point path still needs to be synchronized to the bucket. You cannot delete the mount point if the synchronization state is dirty. For more information about how to delete a mount point, see the "Delete a mount point" section of this topic.

  • clean: All data in the mount point path has been synchronized to the bucket.

Healthy

Indicates whether the mount point is normal. One of the following values is returned for the parameter:

  • true

  • false

Modify mount point configurations

  • Linux: The following sample code enables the reverse synchronization feature for the /mnt/test mount point and sets the reverse synchronization interval to 600 seconds:

    sudo csgfsctl set --mountpoint=/mnt/test --reverse-sync=true --rs-interval=600
  • Windows: The following sample code enables the reverse synchronization feature for the D:\test mount point and sets the reverse synchronization interval to 600 seconds:

    csgfsctl set --mountpoint=D:\test --reverse-sync=true --rs-interval=600

You can run the csgfsctl set --help command to learn how to modify mount point configurations. The following table describes the parameters.

Parameter

Required

Description

--mountpoint (abbreviation: -mp)

Yes

The path of the mount point.

--reverse-sync (abbreviation: -rs)

No

Specifies whether to enable the reverse synchronization feature. One of the following values is returned for the parameter:

  • true

  • false

Note

If the reverse synchronization feature is enabled, the gateway synchronizes metadata from the bucket to the local file system. The reverse synchronization feature can decrease the gateway performance.

--rs-interval (abbreviation: -rsi)

No

The interval for reverse synchronization. Unit: seconds.

Note
  • You can configure this parameter only when the reverse synchronization feature is enabled. Valid values: 10 to 36000. Default value: 300.

  • Access to the same directory within a preset time interval for reverse synchronization triggers only one reverse synchronization task.

  • If a directory contains tens of thousands of files, we recommend that you set the reverse synchronization interval to more than 300 seconds.

--ignore-delete

No

Specifies whether to enable Ignore Deletions. Valid values:

  • true

  • false

Note

If Ignore Deletions is enabled, the delete operation on the files in the cache is not applied to the objects in the bucket.

--oss-direct-read

No

Specifies whether to enable Bypass Cache Read. We recommend that you do not set this parameter to true in most cases. Valid values:

  • true

  • false

Note

By default, if data requested in a read request is not found in the cache, the gateway downloads the data from the bucket and stores the data to the on-premises disk cache. This process involves data pre-reading. However, if read requests are random and the file cache size is considerably less than the bucket capacity, data pre-reading performance and downloading performance may be unsatisfactory. In this case, we recommend that you set this parameter to true.

--allow-rename-dir

No

Specifies whether to allow directory renaming. Valid values:

  • true

  • false

Note

OSS does not support renaming. Renaming a directory actually copies all objects in the directory. We recommend that you forbid directory renaming in most cases for optimized user experience.

--upload-delay

No

The latency in synchronizing data from the on-premises cache to the bucket. Unit: seconds.

Note
  • The value of this parameter ranges from 0 to 120. By default, this parameter is set to 2.

  • If you set this parameter to a non-zero value, operations on the files are synchronized from the cache to the bucket after the specified time period elapses. This setting prevents frequent local file operations within a short window of time from generating a large number of parts in the bucket.

--upload-bandwidth-limit

No

The maximum upload bandwidth. Unit: MB/s.

--rw-bandwidth-limit

No

The maximum read/write bandwidth. Unit: MB/s.

--download-bandwidth-limit

No

The maximum download bandwidth. Unit: MB/s.

--cache-size-gb (abbreviation: -c)

No

The size of the cache space after cache scaling. Unit: GB.

Note
  • This parameter takes effect only when the cache size after cache scaling is greater than the existing cache size. You can use the get command to query the existing cache size.

  • Cache scaling is only supported for metadata cache. This feature is not supported for file cache.

--endpoint (abbreviation: -ep)

No

The endpoint of the bucket. If the existing endpoint is no longer valid, you can use this parameter to specify a different endpoint.

--seq-mode

No

Specifies whether to use the sequential write pattern. Valid values:

  • true: switches from the standard write pattern to the sequential write pattern.

  • false: switches from the sequential write pattern to the standard write pattern.

Note
  • If you want to use the sequential write pattern, we recommend that you set the work-mode parameter to seq-write when you perform the mounting operation.

  • The seq-mode parameter changes the access pattern. When you modify a mount point and specify the seq-mode parameter, make sure that the mount point has no ongoing I/O requests.

  • Before the access pattern is switched to the pattern specified by the seq-mode parameter, new I/O requests fail.

  • In the standard access pattern, if the I/O request of the mount point is complete and a large number of files are being uploaded, the seq-mode parameter setting may require a significant amount of time to take effect, or the seq-mode parameter setting may time out.

--silent (abbreviation: -s)

No

Disables confirmation when you change write patterns. If you do not specify this parameter, you are required to enter y or N to confirm whether to switch the write pattern. To skip confirmation, specify this parameter in the command.

Delete a mount point

Warning
  • Before you delete a mount point, make sure that the mount point or cache path is not being used by any process. This includes checking whether a process is using the mount point path or cache path. If the mount point or cache path is being used by a process, the mount point may fail to be deleted, or incomplete deletion may occur.

  • Make sure that you do not create new files or directories in the cache path. When you delete the mount point, the cache path is cleared.

You can run the csgfsctl remove --help command to learn how to delete a mount point. The following table describes the parameters.

Parameter

Required

Description

--mountpoint (abbreviation: -mp)

Yes

The path of the mount point.

--force

No

  • If not all data in a mount point has been synchronized to the bucket (the value of State in the output of the csgfsctl get command is dirty), you cannot directly delete the mount point.

  • You can include the force option to skip this step and forcibly delete the mount point.

  • If the value of the State parameter is dirty, make sure that you have backed up data that remains to be synchronized to OSS before you use the force option to forcibly delete the mount point.

Logging

To collect logs, run the csgfsctl log command.

Note
  • Linux: The generated logs are located in the /usr/local/bin/Aliyun/csg/ path.

  • Windows: The generated logs are located in the C:\Program Files\Aliyun\csg\bin path.

Uninstall the software gateway

You must use the one of the following methods to uninstall the software gateway. Otherwise, you may encounter unexpected issues, such as service unavailability, unexpected bills, and uninstallation failure.

  • Linux: Run sudo /usr/local/bin/Aliyun/csg/scripts/uninstall.sh.

  • Windows: Run uninstall.bat as an administrator.

Warning
  • Before you uninstall a software gateway, you must manually delete all mount points associated with the gateway. Otherwise, the uninstallation fails.

  • Before you delete a mount point, make sure that the mount point or cache path is not being used by any process. This includes checking whether a process is using the mount point path or cache path. If the mount point or cache path is being used by a process, the mount point may fail to be deleted, or incomplete deletion may occur.

Upgrade the software gateway

If the software gateway is not the latest version, you can upgrade the software gateway.

  • You can use the csgfsctl -v command to check the version number of your software gateway.

  • You do not need to delete mount points associated with the gateway for the upgrade operation. After the upgrade is completed, the mount points are automatically restored.

  • During the upgrade, the mount points are temporarily unavailable. We recommend that you upgrade your gateway during off-peak business hours.

Warning

Before you upgrade your software gateway, make sure that no processes except for csg_agent on Linux are using the mount points or cache path of the software gateway. This includes checking whether a process is using the mount point path or cache path.

On Linux, you can run lsof <mount point path> to check whether the mount point is being used.

  • CentOS and Alibaba Cloud Linux (CentOS 7 is used in this example)

    1. Download the latest installation package: CsgAgent-latest-centos7.rpm.

    2. Run the following commands to upgrade the gateway:

      sudo rpm -iU CsgAgent-latest-centos7.rpm
      sudo /usr/local/bin/Aliyun/csg/scripts/upgrade.sh
    3. Run the csgfsctl -v command to check whether the upgrade is successful.

  • Ubuntu (Ubuntu 16.04 is used in this example)

    1. Download the latest installation package: CsgAgent-latest-ubuntu1604.deb.

    2. Run the following commands to upgrade the gateway:

      sudo apt install --only-upgrade -y ./CsgAgent-latest-ubuntu1604.deb
      sudo /usr/local/bin/Aliyun/csg/scripts/upgrade.sh
    3. Run the csgfsctl -v command to check whether the upgrade is successful.

  • Windows

    1. Download and decompress the latest installation package: CsgAgent-latest-win64.zip.

      Note
      • The files in the installation package must be extracted to the same directory.

      • We recommend that you maintain the directory structure when you decompress the installation package.

    2. Run upgrade.bat as an administrator to upgrade the gateway.

    3. Run the csgfsctl -v command to check whether the upgrade is successful.