After you create a cluster, you can use the manual script execution feature to run a specific script on multiple nodes to meet custom requirements. This topic describes how to add and run a manual script.
Background information
The manual script execution feature lets you immediately run a specific script on multiple existing nodes. This feature is suitable for long-running clusters. For temporary clusters that are created on demand, use bootstrap actions to initialize the cluster. For more information about bootstrap actions, see Use bootstrap actions to run scripts.
Manual execution is similar to a bootstrap action. After you create a cluster, you can use the manual execution feature to install software and services that are not yet supported in the cluster, such as:
Use Yellowdog Updater, Modified (YUM) to install software.
Download software from the Internet.
Read data from Object Storage Service (OSS).
Install and run services, such as the Pig component, which requires more complex scripts.
Prerequisites
An E-MapReduce cluster has been created. For more information, see Create a cluster.
The cluster must be in the Running state. You cannot run cluster scripts if the cluster is in any other state.
You have developed or obtained a cluster script and uploaded it to OSS. For an example, see Example.
Usage notes
Records of manually run scripts are retained for a maximum of 60 days.
Only one cluster script can run in a cluster at a time. If a cluster script is running, you cannot submit a new one.
A cluster script might succeed on some nodes and fail on others. For example, a script might fail because a node restarts. After you resolve the issue, you can rerun the script on only the nodes where it previously failed. After you scale out a cluster, you can also run the script on only the new nodes.
Procedure
Go to the Script Operation tab.
Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select a region and a resource group based on your business requirements.
Find the cluster that you want to manage and click Services in the Actions column.
Click the Script Operation tab.
On the Script Operation page, click the Manual Execution tab.
Click Create And Execute.
In the dialog box that appears, enter a Name, select a location from the Script Location drop-down list, and select a scope from the Execution Scope drop-down list.
NoteWhen you use the cluster script feature, test the script on a single node first. After the test is successful, run the script on the entire cluster.
The script path must be in the oss://**/*.sh format.
After you configure the parameters, click OK.
After the script is created, it is added to the script list and its status is Running. Possible script statuses are Running, Completed, and Submission Failed.
In the Actions column, click Details to view the details of the script.
Click View Execution Result to view the execution status of the script.
The status of the script on a node can be Waiting, Running, Completed, Failed, Submission Failed, or Canceled.
In the Actions column, click Delete to delete the script.
Example
Similar to bootstrap action scripts, you can specify the file to download from OSS in a cluster script. For example, download the sample file oss://<yourBucket>/<myFile>.tar.gz to a local directory and decompress it to the /yourDir directory.
#!/bin/bash
osscmd --id=<yourAccessKeyId> --key=<yourAccessKeySecret> --host=oss-cn-hangzhou-internal.aliyuncs.com get oss://<yourBucketName>/<yourFile>.tar.gz ./<yourFile>.tar.gz
mkdir -p /<yourDir>
tar -zxvf <yourFile>.tar.gz -C /<yourDir>An OSS address can be an internal endpoint, a public endpoint, or a virtual private cloud (VPC) endpoint. If you use the classic network, specify an internal endpoint. For example, the internal endpoint for OSS in the China (Hangzhou) region is oss-cn-hangzhou-internal.aliyuncs.com. If you use a VPC, specify a domain name that is accessible from the VPC. For example, the domain name for OSS in the China (Hangzhou) region is vpc100-oss-cn-hangzhou.aliyuncs.com.
You can also use cluster scripts with YUM to install additional system packages, such as ld-linux.so.2.
#!/bin/bash
yum install -y ld-linux.so.2By default, scripts are run by the root user. You can also use the su hadoop command in the script to switch to the hadoop user.