This topic describes how to select nodes in batches and run specific scripts on these nodes to meet your business requirements, such as installing third-party software and modifying the running environment of a cluster after you create it, especially a subscription cluster.
- A cluster is in the Idle or Running state. Scripts cannot be executed on clusters in other states.
- Cluster scripts are developed or obtained and uploaded to Object Storage Service (OSS). For more information about the cluster scripts, see Examples.
Take bootstrap actions to initialize on-demand clusters. A cluster script is similar to a bootstrap action. After a cluster is created, you can use the cluster script feature to install software and services previously unavailable to your cluster. For example:
- Use YUM to install software already provided.
- Download public software from the Internet.
- Install software to read your data from OSS.
- Install and run a service, such as Flink or Impala. However, the script you want to compile is more complex.
Only one cluster script can run on a cluster at a specific point in time. You cannot submit another cluster script if one is already in progress. You can retain up to 10 cluster script records for each cluster. If more than 10 records exist, you must delete the previous records before you create new cluster scripts.
A cluster script may succeed on some nodes, but fail on others. For example, if you restart a node, the script may fail to run. After you resolve the issue, you can run the cluster script on the failed nodes again. After you scale out a cluster, you can also run cluster scripts on the added nodes.
You can use the cluster script feature to download scripts from OSS to a specific node and run these scripts. If the scripts fail to be executed, you can log on to the node to check the operational log. The operational log for each node is stored in the /var/log/cluster-scripts/clusterScriptId path. If the cluster is configured with an OSS log path, the operational log is also stored in the osslogpath/clusterId/ip/cluster-scripts/clusterScriptId path. To create a cluster script, follow these steps:
- Log on to the Alibaba Cloud E-MapReduce console.
- In the top navigation bar, select the region where you want to create a cluster. The region of a cluster cannot be changed after the cluster is created.
- Click the Cluster Management tab.
- Find the target cluster and click Details in the Actions column.
- In the left-side navigation pane of the Cluster Overview page that appears, click Cluster Scripts.
- On the Cluster Scripts page that appears, click Create and Run in the upper-right corner.
- In the Create Script dialog box, specify Name, Script, and Target Nodes. (You can select multiple nodes.)
Note We recommend that you test the cluster script feature on a single node before you use the feature on the entire cluster.
- Click OK.
After a cluster script is created, it is displayed in the cluster script list and is in the running state.
- Click Refresh in the upper-right corner to update the status of the cluster script.
Click Details in the Actions column to view the running status of the cluster script.
#! /bin/bash osscmd --id=<yourid> --key=<yourkey> --host=oss-cn-hangzhou-internal.aliyuncs.com get oss://<yourbucket>/<myfile>.tar.gz ./<myfile>.tar.gz mkdir -p /<yourdir> tar -zxvf <myfile>.tar.gz -C /<yourdir>
#! /bin/bash yum install -y ld-linux.so.2
The root account is used to run a specified script on a cluster by default. You can run the su hadoop command in the script to switch to the hadoop user.