Elastic Compute Service:Deploy a 3FS cluster based on ECS instances

Step 1: Prepare the environment

Prepare ECS instances for deploying 3FS cluster nodes and configure eRDMA on the instances.

Create an ECS instance to serve as the meta and client nodes, and create five ECS instances to serve as the storage nodes. All nodes can communicate with each other over the internal network. All nodes can access the Internet to download AC2 image resources.

During creation, take note of the following parameters for all ECS instances. For information about other parameters, see Custom launch ECS instances.

• Region: In this example, the China (Hangzhou) region is used.

• Instance type: Select an instance type that supports eRDMA.

  • Meta node: ecs.g8i.48xlarge

  • Storage nodes: ecs.i4.32xlarge

• Image: Select an Ubuntu 22 image that supports eRDMA.

• eRDMA Interface (ERI): Enable ERI for an elastic network interface (ENI) on an instance to enable eRDMA communication mode.

  Important

  The IP addresses in this example are the primary private IP addresses of the ENIs with ERI enabled. In this example, ERI is enabled for the primary ENI of each instance. If you test RDMA communication on secondary ENIs, change the IP addresses accordingly.

  

Step 2: Deploy the meta node

1. Connect to the instance used as the meta node.

   For more information, see Connect to Linux.

2. Install the eRDMA driver:

   curl -O http://mirrors.cloud.aliyuncs.com/erdma/env_setup.sh
   sudo /bin/bash env_setup.sh > /var/log/erdma_install.log 2>&1

   The script automatically installs the required dependencies and then the eRDMA driver. Wait for the script execution to complete.

3. Check whether eRDMA is properly configured:

   ibv_devinfo

   The following command output indicates that eRDMA is properly enabled on the instance (the eRDMA driver is correctly installed and ERI is correctly configured). The installation process of the eRDMA driver requires a period of time to complete. If an error is returned, see Verify the correctness of eRDMA configurations for troubleshooting.

   

4. Set the RDMA connection mode to Compat mode.

   Important

   • By default, CPU-based instance families that support eRDMA install the eRDMA kernel-mode driver in Standard mode. In this mode, only the RDMA_CM connection establishment method is supported.

   • eRDMA provides Compat mode for out of bandwidth (OOB) scenarios. In Compat mode, 16 TCP ports in the range of 30608 to 30623 are additionally occupied.

   sudo sh -c "echo 'options erdma compat_mode=Y' >> /etc/modprobe.d/erdma.conf"
   sudo rmmod erdma
   sudo modprobe erdma compat_mode=Y

5. Install Docker.

   Note

   Before you use AC2 images, you must set up the Docker runtime environment. The Docker installation steps vary based on the operating system. For more information, see Install and use Docker and Docker Compose. In this example, Ubuntu is used.

   apt update
   apt install docker.io -y

   After the installation is complete, run the following command to view the Docker version and check whether Docker is installed.

   

6. Deploy and start ClickHouse to persist cluster runtime metrics, such as node status, resource utilization, and I/O:

   docker run -d --net=host --name clickhouse-server --ulimit nofile=262144:262144 ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/clickhouse:25.3.1.2703-ubuntu22.04

   • --net=host: Sets the communication mode of the container to host. Applications in the container can directly use the network interfaces and network configurations of the host. This provides the same network communication capabilities as the host.

   • --ulimit nofile: specifies the file descriptor limit for processes in the container, which is the maximum number of files a process can open simultaneously. This prevents system instability caused by excessive file handle usage by processes.

7. Deploy and start FoundationDB to store cluster transaction log settings and data:

   docker run -d --network=host --name fdb ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/fdb:7.3.63-ubuntu22.04

8. Deploy and start 3FS monitor to collect and analyze various metrics during system runtime, such as throughput, latency, and resource usage:

   docker run -d --network=host --name monitor --ulimit memlock=-1 --privileged --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/3fs:b71ffc55-fdb7.3.63-fuse3.16.2-ubuntu22.04 ./monitor.sh

   • --ulimit memlock=-1: Sets maxlockedmemory to unlimited. This means there is no limit on the amount of memory that a non-root user can lock. This ensures that eRDMA applications can lock the required amount of memory when run by non-root users, allowing them to use the eRDMA feature effectively.

   • --device=/dev/infiniband/uverbsX and --device=/dev/infiniband/rdma_cm: Expose the eRDMA user mode character devices to the container.

     Run the following command to view the character device name (the X in uverbsX ):

     ls /dev/infiniband | grep uverbs

     

9. Deploy and start the 3FS cluster manager (mgmtd) to manage storage nodes and resource allocation in the cluster:

   docker run -d --network=host --name mgmtd --ulimit memlock=-1 --privileged --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --env FDB_CLUSTER=`docker exec fdb cat /etc/foundationdb/fdb.cluster` --env REMOTE_IP="172.16.20.172:10000" --env MGMTD_SERVER_ADDRESSES="RDMA://172.16.20.172:8000" ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/3fs:b71ffc55-fdb7.3.63-fuse3.16.2-ubuntu22.04 ./mgmtd.sh

   • --env REMOTE_IP: specifies the address of the monitor service, which is the primary private IP address of the node on which the service resides. In this example, the service resides on the same node as the meta service.

   • --env MGMTD_SERVER_ADDRESSES: specifies the address of the mgmtd service, which is the primary private IP address of the node on which the service is started. In this example, the service resides on the same node as the meta and monitor services.

10. Deploy and start the 3FS metadata service (meta):

    docker run -d --network=host --name meta --ulimit memlock=-1 --privileged --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --env FDB_CLUSTER=`docker exec fdb cat /etc/foundationdb/fdb.cluster` --env META_NODE_ID=100 --env REMOTE_IP="172.16.20.172:10000" --env MGMTD_SERVER_ADDRESSES="RDMA://172.16.20.172:8000" ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/3fs:b71ffc55-fdb7.3.63-fuse3.16.2-ubuntu22.04 ./meta.sh

    • --env META_NODE_ID: specifies the sequential number of the meta node. In this example, one meta node is used, and its sequential number is set to 100.

    • --env REMOTE_IP: specifies the address of the monitor service, which is the primary private IP address of the node on which the service resides. In this example, the service resides on the same node as the meta service.

    • --env MGMTD_SERVER_ADDRESSES: specifies the address of the mgmtd service, which is the primary private IP address of the node on which the service is started. In this example, the service resides on the same node as the meta and monitor services.

11. View the started node services (Docker containers):

    docker ps

    

Step 3: Deploy storage nodes

Perform the following steps on each of the five storage nodes:

1. Connect to an instance used as a storage node.

   For more information, see Connect to Linux.

2. Install the eRDMA driver:

   curl -O http://mirrors.cloud.aliyuncs.com/erdma/env_setup.sh
   sudo /bin/bash env_setup.sh > /var/log/erdma_install.log 2>&1

   The script automatically installs the required dependencies and then the eRDMA driver. Wait for the script execution to complete.

3. Check whether eRDMA is correctly configured:

   ibv_devinfo

   The following command output indicates that eRDMA is properly enabled on the instance (the eRDMA driver is correctly installed and ERI is correctly configured). The installation process of the eRDMA driver requires a period of time to complete. If an error is returned, see Verify the correctness of eRDMA configurations for troubleshooting.

   

4. Set the RDMA connection mode to Compat mode.

   Important

   • By default, CPU-based instance families that support eRDMA install the eRDMA kernel-mode driver in Standard mode. In this mode, only the RDMA_CM connection establishment method is supported.

   • eRDMA provides Compat mode for out of bandwidth (OOB) scenarios. In Compat mode, 16 TCP ports in the range of 30608 to 30623 are additionally occupied.

   sudo sh -c "echo 'options erdma compat_mode=Y' >> /etc/modprobe.d/erdma.conf"
   sudo rmmod erdma
   sudo modprobe erdma compat_mode=Y

5. Format and mount NVMe SSDs. In this example, eight local disks on the storage node are mounted and formated to eXtensible File System (XFS) file systems. XFS file systems provide high performance and are suitable for large files.

   mkdir -p /storage/data{0..7}
   for i in {0..7};do mkfs.xfs -L data${i} /dev/nvme${i}n1;mount -o noatime,nodiratime -L data${i} /storage/data${i};done;
   mkdir -p /storage/data{0..7}/3fs

   If you repeat this step, clear the storage space and then specify the -f option to force mount the file systems.

   for i in {0..7};do umount /storage/data${i};done
   for i in {0..7};do mkfs.xfs -f -L data${i} /dev/nvme${i}n1;mount -o noatime,nodiratime -L data${i} /storage/data${i};done;

   After the mounting is complete, run the following command to check the partition status.

   

6. Install Docker.

   Note

   Before you use AC2 images, you must set up the Docker runtime environment. The Docker installation steps vary based on the operating system. For more information, see Install and use Docker and Docker Compose. In this example, Ubuntu is used.

   apt update
   apt install docker.io -y

   After the installation is complete, run the following command to view the Docker version and check whether Docker is installed.

   

7. Start the storage service (storage):

   docker run -d --network=host --name storage --ulimit memlock=-1 --privileged -v /storage:/storage --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm --env STORAGE_NODE_ID=10001 --env TARGET_PATHS="/storage/data0/3fs","/storage/data1/3fs","/storage/data2/3fs","/storage/data3/3fs","/storage/data4/3fs","/storage/data5/3fs","/storage/data6/3fs","/storage/data7/3fs" --env REMOTE_IP="172.16.20.172:10000" --env MGMTD_SERVER_ADDRESSES="RDMA://172.16.20.172:8000" ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/3fs:b71ffc55-fdb7.3.63-fuse3.16.2-ubuntu22.04 ./storage.sh

   • --device=/dev/infiniband/uverbsX and --device=/dev/infiniband/rdma_cm: Expose the eRDMA user mode character devices to the container.

     Run the following command to view the character device name (the X in uverbsX ):

     ls /dev/infiniband | grep uverbs

     

   • --env STORAGE_NODE_ID: specifies the sequential number of the storage node. For the five storage nodes, the numbers are 10001, 10002, 10003, 10004, and 10005, respectively. Replace the values with the actual numbers.

   • --env TARGET_PATHS: specifies the storage directories for 3FS. In this example, the directories created on the local disks in the preceding step are used.

   • --env REMOTE_IP: specifies the address of the monitor service, which is the primary private IP address of the node on which the service resides. In this example, the service resides on the same node as the meta service.

   • --env MGMTD_SERVER_ADDRESSES: specifies the address of the mgmtd service, which is the primary private IP address of the node on which the service is started. In this example, the service resides on the same node as the meta and monitor services.

8. View the started node services (Docker containers):

   docker ps

   

Step 4: View the connected storage nodes on the meta node

1. Connect to the instance used as the meta node.

   For more information, see Connect to Linux.

2. View the status of the nodes connected to the meta node:

   docker exec -it meta  /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.16.20.172:8000"]' "list-nodes"

   Replace the IP address with the IP address of the mgmtd service.

   

   admin_cli is a command-line tool for managing and maintaining 3FS. You can use it for cluster configuration, status monitoring, and troubleshooting. For more information, run the following command:

   docker exec -it meta  /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.16.20.172:8000"]' "help"

Step 5: Configure storage node-related parameters on the meta node

Configure storage node-related information, such as the number of replicas and the number of disks per storage node.

1. Connect to the instance used as the meta node.

   For more information, see Connect to Linux.

2. Configure 3FS:

   docker exec \
     --env STORAGE_NODE_NUM=5 \
     --env STORAGE_NODE_BEGIN=10001 \
     --env STORAGE_NODE_END=10005 \
     --env REPLICATION_FACTOR=3 \
     --env NUM_DISKS_PER_NODE=8 \
     --env MGMTD_SERVER_ADDRESSES="RDMA://172.16.20.172:8000" \
     meta \
     ./config_3fs.sh

   • --env STORAGE_NODE_NUM: specifies the number of storage nodes, which is 5 in this example.

   • --env STORAGE_NODE_BEGIN: specifies the start sequential number of storage nodes, which is 10001 in this example.

   • --env STORAGE_NODE_END: specifies the end sequential number of storage nodes, which is 10005 in this example.

   • --env REPLICATION_FACTOR: specifies the number of replicas for stored data. In this example, 3 is used, which indicates three replicas.

   • --env NUM_DISKS_PER_NODE: specifies the number of disks per storage node, which is 8 in this example.

   • --env MGMTD_SERVER_ADDRESSES: specifies the address of the mgmtd service.

   • --env NUM_TARGETS_PER_DISK: specifies the number of storage targets expected to be created per physical disk. Default value: 12, which indicates that each SSD is divided into 12 storage targets.

   • --env MIN_TARGETS_PER_DISK: specifies the minimum number of storage targets allowed per physical disk. Default value: 12, which indicates that each SSD must maintain at least 12 available storage targets.

   The config_3fs.sh script performs the following operations:

   • Creates the root administrator and generates an authentication token. The token is saved to /opt/3fs/etc/token.txt to authenticate subsequent operations.

   • A data distribution policy is generated based on input parameters such as STORAGE_NODE_NUM and REPLICATION_FACTOR. A chained storage table is then generated from this policy and uploaded to the management server mgmtd to deploy the target and chain structures for the storage nodes.

3. Run the following command to confirm that the chain has been created and its status is normal.

   docker exec -it meta  /opt/3fs/bin/admin_cli -cfg /opt/3fs/etc/admin_cli.toml --config.mgmtd_client.mgmtd_server_addresses '["RDMA://172.16.20.172:8000"]' "list-chains"

   Replace the IP address with the address of the configured mgmtd service.

   The following figure shows the chain information. The chain status must be normal for the subsequent FIO test.

   

Step 6: Configure the FUSE client

Configure the FUSE client on the meta node to allow you to mount remote storage and perform file operations as if operating on a local system.

1. Connect to the instance used as the meta node.

   For more information, see Connect to Linux.

2. Obtain a token to ensure secure client access:

   docker exec meta cat /opt/3fs/etc/token.txt

3. Start the FUSE client:

   docker run -d --network=host --name fuse --shm-size=200g --ulimit memlock=-1 --privileged \
     --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/rdma_cm \
     --env REMOTE_IP="172.16.20.172:10000" \
     --env MGMTD_SERVER_ADDRESSES="RDMA://172.16.20.172:8000" \
     --env TOKEN=${token} \
     ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/3fs:b71ffc55-fdb7.3.63-fuse3.16.2-ubuntu22.04  \
     ./fuse.sh

   • --shm-size: specifies the size of the /dev/shm shared memory area in the container.

     • If FUSE is used, you can set the --shm-size parameter to a smaller value, such as 2g.

     • If USRBIO is used, the test process needs to share I/O buffers with the FUSE process by using shared memory to achieve zero copy. We recommend that you increase the value of the --shm-size parameter, such as 200g.

   • --device=/dev/infiniband/uverbsX and --device=/dev/infiniband/rdma_cm: Expose the eRDMA user mode character devices to the container.

     Run the following command to view the character device name (the X in uverbsX ):

     ls /dev/infiniband | grep uverbs

     

   • --env REMOTE_IP: specifies the address of the monitor service, which is the primary private IP address of the node on which the service resides. In this example, the service resides on the same node as the meta service.

   • --env MGMTD_SERVER_ADDRESSES: specifies the address of the mgmtd service, which is the primary private IP address of the node on which the service is started. In this example, the service resides on the same node as the meta and monitor services.

   • --env TOKEN: specifies the token string obtained in the previous step.

4. View the started node services (Docker containers):

   docker ps

   

5. View the file system mounting information and disk usage in the FUSE container:

   docker exec fuse df -hT | grep 3fs