After you create a cluster, you can use manual execution to run a script on selected nodes. This topic describes how to add and run these scripts.
Background information
Manual execution allows you to immediately run a specified script on selected nodes in an existing cluster. This feature is designed for long-running clusters. For temporary on-demand clusters, use a bootstrap action to initialize the cluster. For more information, see Run scripts by using bootstrap actions.
Similar to a bootstrap action, you can use manual execution to install software and services that are not yet supported by the cluster. For example, you can:
-
Install available software packages by using YUM.
-
Download publicly available software from the internet.
-
Read your data from OSS.
-
Install and run services, such as the Pig component. This may require more complex scripts.
Prerequisites
-
You have created a cluster. For more information, see Create a cluster.
-
The cluster must be in the running state.
-
You have developed or obtained a cluster script and uploaded it to OSS. For an example, see Examples.
Considerations
-
Manual execution records are retained for up to 60 days.
-
Only one cluster script can run on a cluster at a time.
-
A cluster script may succeed on some nodes and fail on others. For example, a node restart can cause the script to fail. After resolving the issue, you can rerun the script on only the failed nodes. After a cluster scale-out, you can also run the cluster script on only the newly added nodes.
Procedure
-
Go to the Script operation page.
-
Log on to the E-MapReduce console.
-
In the top navigation bar, select a region and a resource group.
-
Click Services for the target cluster.
-
Click the Script Operation tab.
-
-
On the Script Operation page, click the Manual Execution tab.
-
Click Create and Execute.
-
In the dialog box, enter a Name, select a script from the Script Address drop-down list, and specify the Execution Scope.
Note-
As a best practice, test the cluster script on a single node before you run it on the entire cluster.
-
The script path must be in the
oss://**/*.shformat.
-
-
After configuring the parameters, click OK.
After a cluster script is created, it appears in the script list and enters the running state. Possible script statuses are Running, Complete, and Submit Failed.
-
To view the details of the manually executed script, click Details in the Actions column.
-
To check the script's execution status, click View Execution Result next to the script name.
The status of the script on each node can be Waiting, Running, Complete, Failed, Submit Failed, or Cancel.
-
To delete the manually executed script, click Delete in the Actions column.
-
Examples
Similar to a bootstrap action script, you can specify files to download from OSS in your cluster script. The following example shows how to download the file oss://<yourBucket>/<myFile>.tar.gz and decompress it to the /yourDir directory.
#!/bin/bash
osscmd --id=<yourAccessKeyId> --key=<yourAccessKeySecret> --host=oss-cn-hangzhou-internal.aliyuncs.com get oss://<yourBucketName>/<yourFile>.tar.gz ./<yourFile>.tar.gz
mkdir -p /<yourDir>
tar -zxvf <yourFile>.tar.gz -C /<yourDir>OSS provides internal, public, and VPC endpoints. If your cluster is in a classic network, you must specify an internal endpoint. For example, the internal endpoint for the China (Hangzhou) region is oss-cn-hangzhou-internal.aliyuncs.com. If your cluster is in a VPC, you must specify an endpoint accessible from the VPC, such as vpc100-oss-cn-hangzhou.aliyuncs.com for the China (Hangzhou) region.
You can also use a cluster script to install extra system packages by using YUM. For example, you can install ld-linux.so.2.
#!/bin/bash
yum install -y ld-linux.so.2By default, the cluster runs your script as the root user. You can also use the su hadoop command to switch to the hadoop user in your script.