This topic describes how to use MaxCompute Studio to develop a MapReduce program. This includes how to write, debug, package, upload, and run a MapReduce program.

Prerequisites

Before you begin, make sure that you have completed the following operations:

Write a MapReduce program

  1. In the Project tool window, expand your MaxCompute Java module and choose src > main > java. Then, right-click java and choose New > MaxCompute Java.
    11
  2. Specify Name, select the Driver type, and then press Enter.
    Create a MaxCompute java class
    • Name: the name of the MaxCompute Java class. If no package is created, enter packagename.classname. The system automatically creates a package.
    • Select the Driver, Mapper, or Reducer type.
  3. After you create a MaxCompute java class, develop a Java program in the editor.

    The Java template is automatically filled with the framework code. You need only to configure the input table, output table, and the Mapper and Reducer classes.

    Write a program

Perform a local run to debug a MapReduce program

Perform a local run to test your MapReduce program and check whether the results meet your expectations.

  1. Right-click the compiled Java script and select Run.
  2. In the Run/Debug Configurations dialog box, select the name of the MaxCompute project where the MapReduce program runs.
    **
  3. Click OK to run the UDF.
    Note
    • The system reads the specified table data in warehouse as the input during the local run. You can view the log output in the console.
    • The system downloads the table data from the specified MaxCompute project to the warehouse directory. If the data is already downloaded, the system does not perform this step.

Perform unit testing to debug a MapReduce program

Refer to the WordCount unit testing in the examples directory and write your test case. Example

Package and upload a MapReduce program

After you debug a MapReduce program, compress the MapReduce program into a JAR package and upload the package to the MaxCompute server as a resource. For more information, see Package, upload, and register.

Run a MapReduce program

Use the MaxCompute client to run a MapReduce program.

  1. In the left-side navigation pane, click Project Explorer.
  2. Right-click the name of your project and select Open in Console.
  3. In the Console tool window, run the following command to start your MapReduce program.
    For more commands, see JAR command. 
    jar-libjars wordcount.jar -classpath D:\odps\clt\wordcount.jar com.aliyun.odps.examples.mr.WordCount wc_in wc_out;