This topic describes how to use MaxCompute Studio to develop a MapReduce program. The development process includes writing, debugging, packaging, uploading, and running a MapReduce program.

Prerequisites

Make sure that the following requirements are met:

Write a MapReduce program

  1. In the Project tool window, expand your MaxCompute Java module and choose src > main > java. Then, right-click java and choose New > MaxCompute Java.
    11
  2. Configure Name, select the Driver type, and then press Enter.
    Create a MaxCompute Java class
    • Name: the name of the MaxCompute Java class. If you have not created a package, specify this parameter in the packagename.classname format. The system automatically generates a package.
    • Select the Driver, Mapper, or Reducer type.
  3. After you create a MaxCompute Java class, develop a Java program in the editor.

    The Java template contains the framework code. You need only to specify the input table, output table, and the Mapper and Reducer classes.

    Develop a program

Run a MapReduce program on your on-premises machine to debug the program

Run the MapReduce program that you wrote on your on-premises machine to debug the program, and check whether the debugging results are as expected.

  1. Right-click the Java script that you wrote and select Run.
  2. In the Run/Debug Configurations dialog box, select the name of the MaxCompute project in which the MapReduce program runs.
    **
  3. Click OK to run the UDF.
    Note
    • The system reads data from the specified table in warehouse as the input during the local run. You can view the log output in the console.
    • The system downloads the table data from the specified MaxCompute project to the warehouse directory. If the data is already downloaded, the system does not perform this step.

Perform unit testing to debug a MapReduce program

You can write a test case based on the test case for WordCount unit testing in the examples folder. Example

Package and upload a MapReduce program

After you debug the MapReduce program that you wrote, package the MapReduce program into a JAR file and upload the file to your MaxCompute project as a resource. For more information, see Package, upload, and register a Java program.

Run a MapReduce program

Run the MapReduce program that you developed on the MaxCompute client.

  1. In the left-side navigation pane, click Project Explorer.
  2. Right-click the name of your MaxCompute project and select Open in Console.
  3. In the Console tool window, run the following command to start the MapReduce program.
    For more information about the command, see Submit a MapReduce job.
    jar-libjars wordcount.jar -classpath D:\odps\clt\wordcount.jar com.aliyun.odps.examples.mr.WordCount wc_in wc_out;