This topic describes how to select nodes in batches and run specific scripts on these nodes to meet your business requirements, such as installing third-party software and modifying the running environment of a cluster after you create it, especially a subscription cluster.

Prerequisites

  • A cluster is in the Idle or Running state. Scripts cannot be executed on clusters in other states.
  • Cluster scripts are developed or obtained and uploaded to Object Storage Service (OSS). For more information about the cluster scripts, see Examples.

Overview

Take bootstrap actions to initialize on-demand clusters. A cluster script is similar to a bootstrap action. After a cluster is created, you can use the cluster script feature to install software and services previously unavailable to your cluster. For example:

  • Use YUM to install software already provided.
  • Download public software from the Internet.
  • Install software to read your data from OSS.
  • Install and run a service, such as Flink or Impala. However, the script you want to compile is more complex.

Features

Only one cluster script can run on a cluster at a specific point in time. You cannot submit another cluster script if one is already in progress. You can retain up to 10 cluster script records for each cluster. If more than 10 records exist, you must delete the previous records before you create new cluster scripts.

A cluster script may succeed on some nodes, but fail on others. For example, if you restart a node, the script may fail to run. After you resolve the issue, you can run the cluster script on the failed nodes again. After you scale out a cluster, you can also run cluster scripts on the added nodes.

You can use the cluster script feature to download scripts from OSS to a specific node and run these scripts. If the scripts fail to be executed, you can log on to the node to check the operational log. The operational log for each node is stored in the /var/log/cluster-scripts/clusterScriptId path. If the cluster is configured with an OSS log path, the operational log is also stored in the osslogpath/clusterId/ip/cluster-scripts/clusterScriptId path. To create a cluster script, follow these steps:

  1. Log on to the Alibaba Cloud E-MapReduce console.
  2. In the top navigation bar, select the region where you want to create a cluster. The region of a cluster cannot be changed after the cluster is created.
  3. Click the Cluster Management tab.
  4. Find the target cluster and click Details in the Actions column.
  5. In the left-side navigation pane of the Cluster Overview page that appears, click Cluster Scripts.
  6. On the Cluster Scripts page that appears, click Create and Run in the upper-right corner.
  7. In the Create Script dialog box, specify Name, Script, and Target Nodes. (You can select multiple nodes.)
    Note We recommend that you test the cluster script feature on a single node before you use the feature on the entire cluster.
  8. Click OK.

    After a cluster script is created, it is displayed in the cluster script list and is in the running state.

  9. Click Refresh in the upper-right corner to update the status of the cluster script.

    Click Details in the Actions column to view the running status of the cluster script.

Examples

Similar to a bootstrap action, you can specify the file that you want to download from OSS in the script. In the following example, the oss://yourbucket/myfile.tar.gz file is downloaded to your local machine and decompressed to the /yourdir directory.
#! /bin/bash
osscmd --id=<yourid> --key=<yourkey> --host=oss-cn-hangzhou-internal.aliyuncs.com get oss://<yourbucket>/<myfile>.tar.gz ./<myfile>.tar.gz
mkdir -p /<yourdir>
tar -zxvf <myfile>.tar.gz -C /<yourdir>
The osscmd tool is pre-installed on a specific node. You can use this tool to download the file.
Note The OSS address specified by using host can be an internal, public, or VPC endpoint. If the node is of the classic network type, you must specify an internal endpoint. For example, the internal endpoint for a node that resides in the China (Hangzhou) region is oss-cn-hangzhou-internal.aliyuncs.com. If the node is of the VPC type, you must specify a domain name that you can access from the VPC. For example, the domain name for a node that resides in the China (Hangzhou) region is vpc100-oss-cn-hangzhou.aliyuncs.com.
Additional system software packages can also be installed by using YUM, for example, ld-linux.so.2.
#! /bin/bash
yum install -y ld-linux.so.2

The root account is used to run a specified script on a cluster by default. You can run the su hadoop command in the script to switch to the hadoop user.