11.11 The Biggest Deals of the Year. 40% OFF on selected cloud servers with a free 100 GB data transfer! Click here to learn more.
Alibaba Cloud Hybrid Cloud Storage Array allows you to build cloud backup solutions between different server, both on premises and on the cloud. However, there are cases where you may require real-time data protection, such as if you work in a company where everyone uses a shared directory for centralized document storage. If you need these documents to trigger backup within a very short period of time (with short RTO), or need them to be quickly distributed to other users, then the Cloud Storage Gateway's replication mode will meet your needs.
Cloud Storage Gateway is a pure software gateway form of online cloud hosts and offline virtual machines. It provides NFS and CIFS file protocol support for client hosts, and uploads files written to the gateway to Alibaba Cloud Alibaba Cloud Object Storage Service (OSS). Cloud Storage Gateway supports two modes: cache and replication. The difference between these two modes is that the replication mode guarantees both the local data and data on the cloud are full data, while the cache mode only keeps metadata of the hot and cold data locally. When you access cold data, you need to pull the corresponding data from the cloud.
In this article, we will use a backup and a recovery operation to demonstrate how Cloud Storage Gateway quickly backs up a user's local data to Alibaba Cloud OSS, and conveniently reverse-synchronizes the data from OSS to the user's local environment. The deployment and configuration flexibility of the storage gateway can recover the backup data in Alibaba Cloud OSS to Alibaba Cloud, a third-party cloud, or the customer's offline data center during data recovery, as long as the target cloud storage gateway is connected to the Alibaba Cloud OSS network.
Cloud Storage Gateway-based cross-cloud replication architecture diagram
First, we need to deploy a cloud host in the customer's cloud environment to support Cloud Storage Gateway. In order to facilitate the rapid deployment of Cloud Storage Gateway, a virtual machine image file is provided to directly generate a cloud host. First enter the Cloud Storage Gateway page of the Alibaba Cloud console, select the East China 2-Shanghai region, and click Create Gateway Cluster in the upper right corner to create a gateway cluster. A gateway cluster is a concept similar to a label or a group, which makes it easy for users to aggregate multiple gateways into one page for management and maintenance.
Create a gateway cluster
After creating a gateway cluster, go to the cluster page and create a gateway. Note that select Local data center as the Location, and File gateway as the Type. Then go to the next step to download the KVM image. This image is compatible with cloud vendor T's cloud host image, and can be used to create cloud vendor T's cloud host.
Create a gateway and download an image
In the last step, download the certificate. After the download is complete, the steps to create a gateway on the Alibaba Cloud console are completed.
Download the certificate to complete creation of the gateway
Go back to cloud vendor T's console and select the Shanghai region. Choose Image > Custom Image, and then click Import Image.
Enter the image page
The first time you import a custom image, you need to enable the cloud vendor's object storage function. You can open it by following the corresponding steps. Then upload the Cloud Storage Gateway image file to the OSS. Copy the link to the image file.
Link to the file in OSS
Go back to the Import Image page, copy the image file link, and then paste it in to the Image File URL field, and then enter other required fields. Click Start import.
Importing an image
It takes 5–10 minutes to import the image file. As shown in the following picture, we can see this image in the images list.
Image imported successfully
Click Create Cloud Host on the right side of the images list to create a cloud host. The next step is to customize the specifications of the cloud host. Cloud Storage Gateway supports different configurations. You can choose the appropriate configuration according to your business needs. If you have any questions, you can also seek advice from Alibaba Cloud's Cloud Storage Gateway team. Here, let's select the lowest-level configuration to deploy Cloud Storage Gateway.
Cloud Storage Gateway host configuration
After successfully purchasing the cloud host, wait a few minutes for the system to complete initialization. Note that Cloud Storage Gateway's web management and NFS and CIFS protocols require some additional ports. Therefore, we need to enter the security group of the cloud host and open the following ports.
Opening Cloud Storage Gateway ports
By default, the cloud host should have already added a suitable security group policy. Just add the ports that we want to open into the policy.
Then, we can enter https:// IP address>:443 in the browser to open Cloud Storage Gateway's web management interface. You need to provide the AK of your Alibaba Cloud account and set the gateway's administrator username and password. At the same time, you need to upload the certificate that you downloaded when you created the gateway. This certificate has the same role as the certificate of the Hybrid Cloud Backup client.
Cloud Storage Gateway registration interface
After completing the registration, we can see that the gateway status has changed from Initialized to Activated on the Cloud Storage Gateway page of the Alibaba Cloud console. This means the cloud storage gateway is already online, and we can use the Alibaba Cloud console to monitor the resource status of the cloud storage gateway in cloud vendor T's network.
Cloud Storage Gateway activation
In addition, after completing the registration, we will be able to use Cloud Storage Gateway's management interface.
Cloud Storage Gateway's management interface
Next, you can complete Cloud Storage Gateway configuration in three simple steps: cloud resource management, cache settings, and NFS/CIFS mounting.
Cloud resource management connects Alibaba Cloud OSS resources to gateways. Before setting cloud resources, you need to create OSS buckets in East China 2 - Shanghai region. Then Cloud Storage Gateway will automatically pull all available OSS buckets within this region for you to choose from. Note that we need to choose Internet for this region, because the network of cloud vendor T and Alibaba Cloud's Cloud Storage Gateway are connected through the Internet.
Create cloud resources
Cache settings are the process of formatting the data disk we used for creating the cloud host. Selecting this disk completes the cache settings.
Cache settings
The last step, NFS/CIFS mounting. Because we created a Linux host when demonstrating Hybrid Cloud Backup, here we only set up NFS (CIFS settings are similar). We need to enter the NFS name, select the cloud resource that we created earlier, as well as the cache path and user mapping. You can edit the Read-Only Client List and Read-Write Client List fields based on your IT environment conditions. In this demonstration, because the webserver host and the cloud storage gateway that we created earlier are in the same intranet environment, we enter the intranet IP address of the webserver host in the Read-Write Client List field. Select Synchronization mode. In Advanced Settings, select Yes for Ignore Delete to ensure that the local delete operation will not delete OSS files on the cloud. Click OK.
NFS settings
NFS Settings > Advanced Settings
So far, the configuration of Cloud Storage Gateway is complete. Next, we can log on to the webserver cloud host that we created earlier to mount the NFS directory provided by Cloud Storage Gateway. You can use the showmount command to check whether the mount on Cloud Storage Gateway is in place.
View Cloud Storage Gateway's NFS directory
Note that the name of the NFS directory displayed by showmount is not the directory name used when mounting. We should mount the directory using the NFS name we entered when creating NFS. The name of the NFS we created was crosscloudnfs.
Mounting NFS directory to the client
After using the df -h command, we can see that in the replication (synchronization) mode, the directory has a space of 40 GB. The reason is that in the replication mode of Cloud Storage Gateway, the local cache space and the OSS space on the cloud are equal. Therefore, the size of the local cache space determines how much space will be used by OSS on the cloud. In the cache mode, the OSS space on the cloud is an extension of the local cache space, so after mounting NFS to the client, the size we see is 256 TB. You need to take this into consideration when selecting the mode and doing business planning.
Directory space in the replication mode
After completing this step, the user server, Cloud Storage Gateway and Alibaba Cloud OSS are connected. Next, we will perform two replication operations to simulate Cloud Storage Gateway's cloud backup scenario when multiple users share the same storage: one user transfers the db_file directory that was backed up earlier to the cross_cloud_gw directory, and the other user transfers the server_log_new directory.
Transferring two directories to Cloud Storage Gateway
The transfer is in process. You can view the file status of OSS on Alibaba Cloud console. These two directories and the files are uploaded to OSS almost instantly. The reason why the file size of some files is displayed as 0 KB in OSS is that the writing of these files into Cloud Storage Gateway's cache disk has not been completed. After Cloud Storage Gateway completes the writing, these files will be uploaded to Alibaba Cloud OSS synchronously. In addition to providing the OSS upload API, Cloud Storage Gateway also implements file slicing, verification, error retransmission and other mechanisms to ensure efficient and reliable file upload.
OSS file status during the upload
You can view resource usage in the monitoring page of Cloud Storage Gateway's management interface. We can also monitor the use of CPU, memory, disk, network and other resources on Alibaba Cloud console. Alibaba Cloud's Cloud Storage Gateway implements a set of traffic control algorithms associated with front-end and back-end traffic, to ensure the front-end read and write performance when the gateway cache space is sufficient; and to control the front-end write speed when the gateway cache space is highly occupied, so as to ensure that the local cache space of the gateway will not be fully occupied.
Cloud Storage Gateway resource monitoring
During the transfer process, you can also view the space usage of the NFS directory cross_cloud_gw on the webserver host, namely the mount client.
NFS directory space occupation
After all cached data has been uploaded to OSS synchronously, the cache status of the gateway becomes Sync complete.
Sync complete
Next, we will simulate a disaster scenario: when files in a directory of the webserver cloud host are deleted by mistake, or cloud host T's cloud host is unavailable for a long time, how can we use Cloud Storage Gateway to recover OSS files from the cloud to a cloud host directory or to a new cloud host? Here we will demonstrate the latter case: reversely synchronizing OSS files to the cloud storage gateway deployed internally by Alibaba Cloud for access by ECS on Alibaba Cloud.
Creating a cloud storage gateway on Alibaba Cloud is similar to creating an offline cloud storage gateway, and is omitted here. We directly move to the NFS settings of the cloud storage gateway. Unlike the previous step, here we need to select Yes for Reverse sync to reversely synchronize OSS files from the cloud to the local cache of the gateway for access by front-end users, so as to achieve the goal of disaster recovery.
Creating Cloud Storage Gateway's NFS directory
After creating a new NFS directory, you can mount it. In reverse sync mode, the cloud storage gateway's NFS directory automatically synchronizes OSS files to the local cache space. As a result, the files are easily recovered and can be accessed on any host that has mounted the directory. Of course, if there are many large files, the time required for synchronization depends on the bandwidth. When synchronizing files from OSS to Cloud Storage Gateway's cache, there is also a verification algorithm to ensure file consistency.
Reverse synchronization of Cloud Storage Gateway
If you want to reversely synchronize files in OSS back to cloud vendor T's cloud storage gateway, which is the seventh step in the architecture diagram at the beginning of this article, you can simply enable Reverse sync in NFS settings at the cloud storage gateway deployed by cloud vendor T. Then data will be automatically synchronized to the gateway's cache for access by the front-end host.
By using the file backup and reverse sync file recovery functions in the synchronized replication mode of Cloud Storage Gateway, you've already experienced the ease-of-use of Cloud Storage Gateway. Cloud Storage Gateway, as a pure software, boasts flexible deployment and simple configuration, and can be seamlessly connected to your existing business system. The "backup immediately after writing" replication mode is perfect for backing up centralized shared directories. Cloud Storage Gateway supports recovering backup files from Alibaba Cloud OSS to any cloud vendor's host, including Alibaba Cloud's ECS host or even your own offline data center, which perfectly solves problems with file backup in cross-cloud (multi-cloud) scenarios. You can recover data to any locations that have deployed Cloud Storage Gateway. Distributing recovered data using gateways is extremely efficient.
To learn more about Cloud Storage Gateway, visit www.alibabacloud.com/product/sgw
57 posts | 11 followers
FollowAlibaba Cloud Storage - October 31, 2018
Alibaba Cloud Storage - November 1, 2018
Alibaba Clouder - September 4, 2020
Alibaba Clouder - July 3, 2020
Alibaba Cloud Native - November 29, 2023
Alibaba Cloud Community - March 21, 2022
57 posts | 11 followers
Follow
Hybrid Cloud Storage
A cost-effective, efficient and easy-to-manage hybrid cloud storage solution.
Learn More
Cloud Backup
Cloud Backup is an easy-to-use and cost-effective online data management service.
Learn More
Hybrid Cloud Distributed Storage
Provides scalable, distributed, and high-performance block storage and object storage services in a software-defined manner.
Learn More
Database Security Solutions
Protect, backup, and restore your data assets on the cloud with Alibaba Cloud database services.
Learn MoreMore Posts by Alibaba Cloud Storage
Raja_KT February 11, 2019 at 4:02 am
Good one and it will be good if you can provide details on recommendations of DTS, DBS too in contrast with this :)