MaxCompute:Use Java UDTFs to read MaxCompute resources

Prerequisites

Before you begin, make sure you have:

• MaxCompute Studio installed and connected to a MaxCompute project, with a Java module created. See Install MaxCompute Studio, Manage project connections, and Create a MaxCompute Java module

• IntelliJ IDEA 2024 and JDK 1.8 installed

How the UDTF works

A MaxCompute Java UDTF handler class uses two lifecycle methods:

Loading resources in setup instead of process means each resource is read once and reused across all rows — this is the recommended pattern for file and table resources.

Resource reading API

The ExecutionContext class provides two methods for reading MaxCompute resources:

UDTF code

The UDTF below takes two string inputs and outputs three columns. In setup, it reads one file resource (file_resource.txt) and two table resources (table_resource1, table_resource2). In process, it forwards the input values alongside the line and record counts loaded during setup.

Input and output parameters:

UDTF class (`UDTFResource`):

package com.aliyun.odps.examples.udf;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.util.Iterator;
import com.aliyun.odps.udf.ExecutionContext;
import com.aliyun.odps.udf.UDFException;
import com.aliyun.odps.udf.UDTF;
import com.aliyun.odps.udf.annotation.Resolve;
/**
 * project: example_project
 * table: wc_in2
 * partitions: p1=2,p2=1
 * columns: cola,colc
 */
@Resolve("string,string->string,bigint,string")
public class UDTFResource extends UDTF {
  ExecutionContext ctx;
  long fileResourceLineCount;
  long tableResource1RecordCount;
  long tableResource2RecordCount;
  @Override
  public void setup(ExecutionContext ctx) throws UDFException {
  this.ctx = ctx;
  try {
   InputStream in = ctx.readResourceFileAsStream("file_resource.txt");
   BufferedReader br = new BufferedReader(new InputStreamReader(in));
   String line;
   fileResourceLineCount = 0;
   while ((line = br.readLine()) != null) {
     fileResourceLineCount++;
   }
   br.close();
   Iterator<Object[]> iterator = ctx.readResourceTable("table_resource1").iterator();
   tableResource1RecordCount = 0;
   while (iterator.hasNext()) {
     tableResource1RecordCount++;
     iterator.next();
   }
   iterator = ctx.readResourceTable("table_resource2").iterator();
   tableResource2RecordCount = 0;
   while (iterator.hasNext()) {
     tableResource2RecordCount++;
     iterator.next();
   }
 } catch (IOException e) {
   throw new UDFException(e);
 }
}
   @Override
   public void process(Object[] args) throws UDFException {
     String a = (String) args[0];
     long b = args[1] == null ? 0 : ((String) args[1]).length();
     forward(a, b, "fileResourceLineCount=" + fileResourceLineCount + "|tableResource1RecordCount="
     + tableResource1RecordCount + "|tableResource2RecordCount=" + tableResource2RecordCount);
    }
}

`pom.xml` dependency for local testing:

<dependency>
    <groupId>com.aliyun.odps</groupId>
    <artifactId>odps-udf-local</artifactId>
    <version>0.48.0-public</version>
</dependency>

Local testing

1. In MaxCompute Studio, create a new Java class of the UDTF type. Name the class UDTFResource and paste the UDTF code from UDTF code.

2. Configure the runtime parameters using the warehouse resource in the Java module.

   The input parameters are the values of the first and third columns (cola and colc) of each row in partition p1=2, p2=1 of the wc_in2 table in the local resource. The code reads file_resource.txt from the file resource, table_resource1 (mapped to wc_in1), and table_resource2 (mapped to wc_in2 with p1=2, p2=1).

   

3. Right-click the UDTFResource class and select Run. The results appear in the output panel.

   

   

Deploy and test with the MaxCompute client

Step 1: Add the file resource

In the upper-left corner of IDEA, click Project Explorer and select Add Resource.

Add file_resource.txt based on the MaxCompute instance information.

CREATE TABLE wc_in1 (
  col1 STRING,
  col2 STRING,
  col3 STRING,
  col4 STRING
);

INSERT INTO wc_in1 VALUES
('A1','A2','A3','A4'),
('A1','A2','A3','A4'),
('A1','A2','A3','A4'),
('A1','A2','A3','A4');

CREATE TABLE wc_in2 (
    cola STRING,
    colb STRING,
    colc STRING
) PARTITIONED BY (p1 STRING, p2 STRING);

ALTER TABLE wc_in2 ADD PARTITION (p1='2',p2='1');

INSERT INTO wc_in2 PARTITION (p1='2',p2='1') VALUES
('three1','three2','three3'),
('three1','three2','three3'),
('three1','three2','three3');

Step 3: Add table resources

Map wc_in1 to table_resource1 and wc_in2 to table_resource2 in Resource.

Add the wc_in1 resource:

Add the wc_in2 resource:

Step 4: Package and deploy the UDTF

Right-click the UDTFResource class and select Deploy to Server... to open the packaging interface. In the Extra resources section, include file_resource.txt, table_resource1, and table_resource2. Set the function name to my_udtf.

Step 5: Call the UDTF from the client

In the upper-left corner of IDEA, click Project Explorer, right-click the MaxCompute project, and select Open Console. Run the following SQL command to call the UDTF:

SELECT my_udtf("10","20") AS (a, b, fileResourceLineCount);

The results appear in the console.

UDTF code requirements

A valid MaxCompute Java UDTF must meet the following requirements:

• The class must extend UDTF.

• The class must be annotated with @Resolve, specifying input and output types in the format "input_types->output_types".

• The class must implement the process(Object[] args) method and call forward() to emit output rows.

• All file and table resources used in the UDTF must be declared in the Extra resources section when deploying with Deploy to Server....