This topic describes how to configure a Hadoop MapReduce job.
- Log on to the Alibaba Cloud E-MapReduce console with an Alibaba Cloud account.
- Click the Data Platform tab.
- In the Projects section, click Edit Job in the row of a project.
- In the left-side navigation pane, right-click the required folder and choose Create Job from the shortcut menu.
Note You can also right-click the folder to create a subfolder, rename the folder, or delete the folder.
- In the dialog box that appears, set the Name and Description parameters, and select MR from the Job Type drop-down list.
This option indicates that a Hadoop MapReduce job will be created. You can use the following command syntax to submit a Hadoop MapReduce job:
hadoop jar xxx.jar [MainClass] -Dxxx ....
- Click OK.
- Specify the command line arguments required to submit the job in the Content field.
Start from the argument that follows
hadoop jar. Enter the path of the JAR file that is used to run the job before setting [MainClass] and other command line arguments.For example, you want to submit a Hadoop sleep job. Instead of reading and writing data, this job only submits some mapper and reducer tasks to the Hadoop cluster, and sleeps for a period of time during the execution of each task. In Hadoop 2.6.0, this job is packaged in hadoop-mapreduce-client-jobclient-2.6.0-tests.jar. You can run the following command to submit the job:
hadoop jar /path/to/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar sleep -m 3 -r 3 -mt 100 -rt 100To configure this job in E-MapReduce, enter the following command in the Content field:
/path/to/hadoop-mapreduce-client-jobclient-2.6.0-tests.jar sleep -m 3 -r 3 -mt 100 -rt 100Note Click Enter an OSS path in the lower part of the page. In the dialog box that appears, set File Prefix to OSSREF and specify the file in File Path. The system automatically completes the path of the Hadoop MapReduce script in OSS.
- Click Save.
In the preceding example, the sleep job does not involve data input and output. To configure a job that reads data and provides processing results, such as a wordcount job, you must specify the data input and output paths. You can read data from and write data to HDFS and OSS in E-MapReduce. To read data from and write data to OSS, set the input and output paths to the paths in OSS. Sample code:
jar ossref://emr/checklist/jars/chengtao/hadoop/hadoop-mapreduce-examples-2.6.0.jar randomtextwriter -D mapreduce.randomtextwriter.totalbytes=320000 oss://emr/checklist/data/chengtao/hadoop/Wordcount/Input