DataWorks:MaxCompute MR node

Step 1: Develop the MR code

1. Upload, submit, and publish the mapreduce-examples.jar resource. See Resource management.

   For details on the internal logic of mapreduce-examples.jar, see the WordCount example.

2. In the MaxCompute MR node editor, enter the following code:

   Parameter	Description
   --@resource_reference	Auto-generated when you right-click a resource name in the Resource section and choose Reference Resource.
   -resources	The name of the referenced JAR resource file.
   -classpath	The path to the JAR file. When a resource is referenced, the path is always ./ followed by the JAR filename (for example, ./mapreduce-examples.jar).
   com.aliyun.odps.mapred.open.example.WordCount	The main class in the JAR file called at execution. This must exactly match the main class defined in the JAR file.
   wc_in	The input table for the MapReduce job.
   wc_out	The output table for the MapReduce job.

   -- Create the input table.
   CREATE TABLE IF NOT EXISTS wc_in (key STRING, VALUE STRING);
   -- Create the output table.
   CREATE TABLE IF NOT EXISTS wc_out (key STRING, cnt BIGINT);
       --- Create the system dual table.
       DROP TABLE IF EXISTS dual;
       CREATE TABLE dual(id BIGINT); -- If the pseudo table does not exist in the workspace, you must create it and initialize its data.
       --- Initialize data in the system pseudo table.
       INSERT OVERWRITE TABLE dual SELECT count(*) FROM dual;
       --- Insert sample data into the input table wc_in.
       INSERT OVERWRITE TABLE wc_in SELECT * FROM (
       SELECT 'project','val_pro' FROM dual
       UNION ALL
       SELECT 'problem','val_pro' FROM dual
       UNION ALL
       SELECT 'package','val_a' FROM dual
       UNION ALL
       SELECT 'pad','val_a' FROM dual
         ) b;
   -- Reference the JAR resource that you uploaded. Right-click the resource name in the Resource section and choose Reference Resource to auto-generate this statement.
   --@resource_reference{"mapreduce-examples.jar"}
   jar -resources mapreduce-examples.jar -classpath ./mapreduce-examples.jar com.aliyun.odps.mapred.open.example.WordCount wc_in wc_out

   The key parameters in the jar command are:

Multiple JAR files

If the job depends on more than one JAR file, list all paths in -classpath, separated by commas:

-classpath ./xxxx1.jar,./xxxx2.jar

Step 2: Run the MR job

1. In the Run Configuration section on the right, set the Compute Engine Instance, Compute Quota, and Resource Group.

   To access a data source on the Public Network or in a Virtual Private Cloud (VPC), use a scheduling resource group that can connect to the data source. See Network connectivity solutions.

2. In the toolbar, click Run. In the Parameters dialog box, select your MaxCompute data source and run the job.

Step 3: (Optional) Verify the results

Query the output table using a MaxCompute SQL node:

SELECT * FROM wc_out;

Expected output:

+------------+------------+
| key        | cnt        |
+------------+------------+
| package    | 1          |
| pad        | 1          |
| problem    | 1          |
| project    | 1          |
| val_a      | 2          |
| val_pro    | 2          |
+------------+------------+