Elastic High Performance Computing:Use a post-processing script to mount ossfs on an E-HPC cluster

Background information

You can use one of the following methods to configure a post-processing script:

• When you create an E-HPC cluster, configure the Post-Install Script parameter in the Software Configurations step.

• When an E-HPC cluster is running, use OpenAPI Explorer to configure or obtain a post-processing script.

  • Call the SetPostScripts operation to configure a post-processing script.

  • Call the GetPostScripts operation to obtain a post-processing script.

Prerequisites

1. An E-HPC cluster is created. For more information, see Create a cluster by using the wizard.

   In this example, a post-processing script is configured when an E-HPC cluster is running. If you have not created an E-HPC cluster, you can directly configure a post-processing script when you create an E-HPC cluster.

2. An Object Storage Service (OSS) bucket is created to store the installation package of ossfs and a post-processing script. For more information, see Create buckets.

Step 1: Download and configure ossfs

1. Download the installation package of ossfs.

   For more information, see Installation.

2. Upload the installation package to the OSS bucket.

   For more information, see Simple upload. When you upload the installation package, set the File ACL parameter to Public Read.Then, you can access or obtain a post-processing script later. After the installation package is uploaded, record the private download URL of the installation package from the file details page. Sample download URL: https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ossfs_1.80.6_centos7.0_x86_64.rpm.

3. Configure an AccessKey ID and an AccessKey secret for ossfs.

   1. Log on to the E-HPC cluster.

   2. Write the AccessKey pair information to the account configuration file.

      Write the AccessKey ID and AccessKey secret to the shared directory of the cluster in the $bucket_name:$access_key_id:$access_key_secret format. For example, you can write the AccessKey pair information to the /home/passwd-ossfs directory. The directory is used as a parameter in a post-processing script. Run the following command:

      echo "$bucket_name:$access_key_id:$access_key_secret" >  /home/passwd-ossfs

      Note

      • For information about how to obtain an AccessKey ID and an AccessKey secret, see Create an AccessKey pair.

      • Do not leak the AccessKey pair information. For security advice, see Best practices to prevent AccessKey pair leaks.

   3. Change the permissions of the account configuration file.

      Run the following command to set the /home/passwd-ossfs parameter to 600:

      chmod 600 /home/passwd-ossfs

Step 2: Write and upload a post-processing script

1. Write a post-processing script.

   The following example shows how to write a post-processing script named ehpcMountOss.sh. Replace the values of the OssFsConfPath, OssFsUrl, and OssEndpoint parameters based on your business requirements.

   Important

   If a post-processing script fails to be executed, node errors may occur. Make sure that the script content is valid. For more information, see Configure an installation script.

   #!/bin/bash
   # Enter the directory of the account configuration file.
   OssFsConfPath="/home/passwd-ossfs"
   
   # Enter the private download URL of the ossfs installation package.
   OssFsUrl="https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ossfs_1.80.6_centos7.0_x86_64.rpm"
   
   # Enter the internal endpoint of the OSS bucket.
   OssEndpoint="http://oss-cn-shanghai-internal.aliyuncs.com"
   
   wget ${OssFsUrl}
   yum install -y ossfs_1.80.6_centos7.0_x86_64.rpm
   rm -rf ossfs_1.80.6_centos7.0_x86_64.rpm
   
   echo ${BucketName}:${AccessKeyId}:${AccessKeySecret} > /etc/passwd-ossfs
   chmod 640 /etc/passwd-ossfs
   mkdir /tmp/ossfs
   ossfs ${BucketName} /tmp/ossfs -opasswd_file=${OssFsConfPath} -ourl=${OssEndpoint} -oallow_other

   In the preceding script, ossfs is mounted on the E-HPC cluster. For more information about how to configure ossfs, see Advanced configurations.

2. Upload the post-processing script to the OSS bucket.

   For more information, see Simple upload. When you upload the post-processing script, set the File ACL parameter to Public Read.Then, you can access or obtain the post-processing script later. After the post-processing script is uploaded, record the private download URL of the post-processing script from the file details page. Sample download URL: https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ehpcMountOss.sh.

Step 3: Use OpenAPI Explorer to configure the post-processing script

1. Log on to OpenAPI Explorer.

2. Call the SetPostScripts operation to configure the post-processing script.

   Note

   If you have configured a post-processing script for the E-HPC cluster, log on to the cluster and run the rm -rf command to delete the script first. By default, the post-processing script is stored in the /opt/ehpc_user_scripts directory.

   Specify the following parameters. For more information, see SetPostScripts.

   Parameter	Example	Description
   RegionId	cn-shanghai	The ID of the region in which the cluster resides.
   ClusterId	ehpc-sh-R1X01h****	The ID of the cluster.
   PostInstallScripts.0.Url	https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ehpcMountOss.sh	The private download URL of the post-processing script.

3. Call the GetPostScripts operation to view the post-processing script.

   Specify the following parameters. For more information, see GetPostScripts.

   Parameter	Example	Description
   RegionId	cn-shanghai	The ID of the region in which the cluster resides.
   ClusterId	ehpc-sh-R1X01h****	The ID of the cluster.

Step 4: Scale out the E-HPC cluster to test the post-processing script

1. Scale out the E-HPC cluster. For more information, see Manually scale out a cluster.

2. Log on to the added compute nodes and check whether ossfs is mounted on the compute nodes.

   After you log on to the added compute nodes, run the df -h command. The post-processing script is automatically executed on the nodes and ossfs is mounted. The following output is expected.