Note: Make sure you have completed all the prerequisites.
In this tutorial, you will be able to get a idea of what clusters, jobs, and execution plans play and how they are used in E-MapReduce. You will also be able to create a Spark Pi job and run it successfully in the cluster. Finally, you can see the approximate calculation result of Pi on the console page.
- Create a cluster.
- On the left side of the console, select Cluster and click Create Cluster at the upper right corner.
- Software configurations.
- Use lasted EMR product version.
- Use the default software configuration.
- Hardware configurations.
- Select Pay-As-You-Go.
- If there is no security group, click New and enter the security group name.
- Select 4-core and 8G for the master node.
- Select 4-core and 8G for the Core node (one instance).
- Keep others in default status.
- Basic configurations
- Enter the name of the cluster.
- Select the log path to save job logs and select make sure that the logging feature is on. In the region for the cluster,
create an OSS bucket.
- Enter the password.
- Create a cluster.
- Create a job.
- On the left side of the console, select Job and click Create Job at the upper right corner.
- Enter the job name.
- Select Spark as the job type.
- Enter parameters as follows.
--class org.apache.spark.examples.SparkPi --master yarn-client --driver-memory 512m --num-executors 1 --executor-memory 1g --executor-cores 2 /usr/lib/spark-current/examples/jars/spark-examples_2.11-2.1.1.jar 10
/usr/lib/spark-current/examples/jars/spark-examples_2.11-2.1.1.jarjar file name is decided by Spark version in cluster, for example, if Spark version is 2.1.1, it should be
spark-examples_2.11-2.1.1.jar, if Spark version is 2.2.0, then file name is
- Keep others in default status to create the job.
- Create an execution plan
- When a cluster is created successfully, its status on the list is shown as Idle.
- Select Execution Plan on the left side of the console and click Create Execution Plan at the upper right corner.
- Select Existing Cluster. Choose the newly created cluster and associate it with the execution plan to create.
- Add the job created earlier to the queue.
- Enter the name of the execution plan.
- Choose Manual Execution by default.
- Create an execution plan.
- Run the execution plan.
- On the execution plan list page, click Run Now.
- View job logs and confirm the results.
- Click Management and proceed to the management page. View the Running Log at the bottom of the page.
- Click the right side of the running log to view the job list.
- Click stdout and you can see the approximate calculation result of Pi: 3.14xxxx.