All Products
Search
Document Center

Elastic High Performance Computing:Use a post-processing script to deploy a scaled out compute node

Last Updated:Jun 14, 2024

After a post-processing script is configured for an Elastic High Performance Computing (E-HPC) cluster, the script is automatically executed on nodes that are added to the E-HPC cluster later. This meets the pre-deployment requirements of compute nodes. You can use a post-processing script to perform custom operations on compute nodes. For example, you can mount ossfs on the compute nodes that are added to an E-HPC cluster later or deploy a software environment. In this topic, a post-processing script is configured to describe how to mount ossfs on an E-HPC cluster.

Background information

You can use one of the following methods to configure a post-processing script:

  • When you create an E-HPC cluster, configure the Post-Install Script parameter in the Software Configurations step.

  • When an E-HPC cluster is running, use OpenAPI Explorer to configure or obtain a post-processing script.

    • Call the SetPostScripts operation to configure a post-processing script.

    • Call the GetPostScripts operation to obtain a post-processing script.

Note

If you configure a post-processing script when you create an E-HPC cluster, the script takes effect on all nodes in the cluster. If you configure a post-processing script when an E-HPC cluster is running, the script takes effect only on the nodes that are added to the cluster later and does not take effect on the existing nodes.

Prerequisites

  1. An E-HPC cluster is created. For more information, see Create a cluster by using the wizard.

    In this example, a post-processing script is configured when an E-HPC cluster is running. If you have not created an E-HPC cluster, you can directly configure a post-processing script when you create an E-HPC cluster.

  2. An Object Storage Service (OSS) bucket is created to store the installation package of ossfs and a post-processing script. For more information, see Create a bucket.

Step 1: Download and configure ossfs

  1. Download the installation package of ossfs.

    For more information, see Installation.

  2. Upload the installation package to the OSS bucket.

    For more information, see Simple upload. When you upload the installation package, set the File ACL parameter to Public Read.Then, you can access or obtain a post-processing script later. After the installation package is uploaded, record the private download URL of the installation package from the file details page. Sample download URL: https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ossfs_1.80.6_centos7.0_x86_64.rpm.

  3. Configure an AccessKey ID and an AccessKey secret for ossfs.

    1. Log on to the E-HPC cluster.

    2. Write the AccessKey pair information to the account configuration file.

      Write the AccessKey ID and AccessKey secret to the shared directory of the cluster in the $bucket_name:$access_key_id:$access_key_secret format. For example, you can write the AccessKey pair information to the /home/passwd-ossfs directory. The directory is used as a parameter in a post-processing script. Sample command:

      echo "$bucket_name:$access_key_id:$access_key_secret" >  /home/passwd-ossfs
      Note
    3. Change the permissions of the account configuration file.

      Run the following command to set the /home/passwd-ossfs parameter to 600:

      chmod 600 /home/passwd-ossfs

Step 2: Write and upload a post-processing script

  1. Write a post-processing script.

    The following example shows how to write a post-processing script named ehpcMountOss.sh. Replace the values of the OssFsConfPath, OssFsUrl, and OssEndpoint parameters based on your business requirements.

    Important

    If a post-processing script fails to be executed, node errors may occur. Make sure that the script content is valid. For more information, see Configure an installation script.

    #!/bin/bash
    # Enter the directory of the account configuration file.
    OssFsConfPath="/home/passwd-ossfs"
    
    # Enter the private download URL of the ossfs installation package.
    OssFsUrl="https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ossfs_1.80.6_centos7.0_x86_64.rpm"
    
    # Enter the internal endpoint of the OSS bucket.
    OssEndpoint="http://oss-cn-shanghai-internal.aliyuncs.com"
    
    wget ${OssFsUrl}
    yum install -y ossfs_1.80.6_centos7.0_x86_64.rpm
    rm -rf ossfs_1.80.6_centos7.0_x86_64.rpm
    
    echo ${BucketName}:${AccessKeyId}:${AccessKeySecret} > /etc/passwd-ossfs
    chmod 640 /etc/passwd-ossfs
    mkdir /tmp/ossfs
    ossfs ${BucketName} /tmp/ossfs -opasswd_file=${OssFsConfPath} -ourl=${OssEndpoint} -oallow_other

    In the preceding script, ossfs is mounted on the E-HPC cluster. For more information about how to configure ossfs, see Advanced configurations.

  2. Upload the post-processing script to the OSS bucket.

    For more information, see Simple upload. When you upload the post-processing script, set the File ACL parameter to Public Read.Then, you can access or obtain the post-processing script later. After the post-processing script is uploaded, record the private download URL of the post-processing script from the file details page. Sample download URL: https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ehpcMountOss.sh.

Step 3: Use OpenAPI Explorer to configure the post-processing script

  1. Log on to OpenAPI Explorer.

  2. Call the SetPostScripts operation to configure the post-processing script.

    Note

    If you have configured a post-processing script for the E-HPC cluster, log on to the cluster and run the rm -rf command to delete the script first. By default, the post-processing script is stored in the /opt/ehpc_user_scripts directory.

    Specify the following parameters. For more information, see SetPostScripts.

    Parameter

    Example

    Description

    RegionId

    cn-shanghai

    The ID of the region in which the cluster resides.

    ClusterId

    ehpc-sh-R1X01h****

    The ID of the E-HPC cluster.

    PostInstallScripts.0.Url

    https://bucket-***.oss-cn-shanghai-internal.aliyuncs.com/demo/ehpcMountOss.sh

    The private download URL of the post-processing script.

  3. Call the GetPostScripts operation to view the post-processing script.

    Specify the following parameters. For more information, see GetPostScripts.

    Parameter

    Example

    Description

    RegionId

    cn-shanghai

    The ID of the region in which the cluster resides.

    ClusterId

    ehpc-sh-R1X01h****

    The ID of the E-HPC cluster.

Step 4: Scale out the E-HPC cluster to test the post-processing script

  1. Scale out the E-HPC cluster. For more information, see Manually scale out an E-HPC cluster.

  2. Log on to the added compute nodes and check whether ossfs is mounted on the compute nodes.

    After you log on to the added compute nodes, run the df -h command to check whether the post-processing script is automatically executed on the nodes and ossfs is mounted. The following output is expected:

    后处理脚本2