MaxCompute:Develop a MaxCompute MapReduce task

Step 1: Upload the JAR resource

Download mapreduce-examples.jar, then upload, commit, and deploy it as a MaxCompute resource. See Create and use MaxCompute resources.

Step 2: Write and run the node code

Paste the following code into the ODPS MR node editor and run it.

-- Create an input table.
CREATE TABLE if not exists wc_in (key STRING, value STRING);
-- Create an output table.
CREATE TABLE if not exists wc_out (key STRING, cnt BIGINT);

-- Create a dual table if one does not already exist in the workspace.
drop table if exists dual;
create table dual(id bigint);
-- Initialize the dual table.
insert overwrite table dual select count(*) from dual;

-- Populate the input table with sample data.
insert overwrite table wc_in select * from (
  select 'project',  'val_pro' from dual
  union all
  select 'problem',  'val_pro' from dual
  union all
  select 'package',  'val_a'   from dual
  union all
  select 'pad',      'val_a'   from dual
) b;

-- Reference the uploaded JAR resource. To add this line automatically,
-- right-click the JAR resource in the resource list and select Reference Resources.
--@resource_reference{"mapreduce-examples.jar"}
jar -resources mapreduce-examples.jar -classpath ./mapreduce-examples.jar com.aliyun.odps.mapred.open.example.WordCount wc_in wc_out

The jar command has the following structure:

jar -resources <jar-resource-name> -classpath ./<jar-resource-name> <main-class> <input-table> <output-table>

When the node runs successfully, it returns OK.

Using multiple JAR resources: If a single node needs more than one JAR file, separate the paths of the referenced JAR resources with commas (,), such as -classpath ./xxxx1.jar,./xxxx2.jar.

Step 3: Verify the output

Run the following query on an ODPS SQL node to confirm the results were written correctly.

select * from wc_out;

Expected output:

+------------+------------+
| key        | cnt        |
+------------+------------+
| package    | 1          |
| pad        | 1          |
| problem    | 1          |
| project    | 1          |
| val_a      | 2          |
| val_pro    | 2          |
+------------+------------+