MaxCompute:Developing Java UDFs

• Write a Hive UDF in Java

• Use complex data types in Java UDFs

• Replace strings by using regular expressions

• Obtain the values of strings that have no delimiters

• Obtain the values of strings that have delimiters

• Reference a file resource

• Reference a table resource

• Reference third-party packages in Python UDFs

A Java UDF consists of the following parts:

• Java package: Optional.

  You can group your Java classes into a package for easier reuse and organization.

• Inherit the UDF class: Required.

  The required base class is com.aliyun.odps.udf.UDF. If you need other UDF classes or complex data types, add the required classes from the MaxCompute SDK. For example, the class for the STRUCT data type is com.aliyun.odps.data.Struct.

• @Resolve annotation: Optional.

  The format is @Resolve(<signature>), where signature defines the data types of the input parameters and return value. When you use the STRUCT data type in a UDF, reflection cannot retrieve field names and field types from com.aliyun.odps.data.Struct. In this case, you must use the @Resolve annotation to retrieve them. If you use STRUCT in a UDF, add the @Resolve annotation to the UDF class. The annotation affects only the overloads whose parameters or return values contain com.aliyun.odps.data.Struct. Example: @Resolve("struct<a:string>,string->string"). For a detailed example, see UDF Example: Complex Data Types.

• Custom Java class: Required.

  This is the unit that organizes your UDF code and defines the variables and methods that implement your business logic.

• evaluate method: Required.

  Your custom Java class must include a non-static public evaluate method. The data types of its input parameters and return value define the UDF's SQL signature.

  You can implement multiple evaluate methods. When you call the UDF, MaxCompute selects the matching evaluate method based on the argument types.

  When you write a Java UDF, you can use Java types or Java Writable types. For detailed mappings between MaxCompute data types and Java data types, see Appendix: Data types.

• UDF initialization and cleanup: Optional. You can implement initialization and cleanup using void setup(ExecutionContext ctx) and void close(). The void setup(ExecutionContext ctx) method is called once before the evaluate method and can be used to initialize resources or member objects required for the computation. The void close() method is called once after all evaluate calls are complete and is used for cleanup tasks, such as closing files.

The following examples show two types of UDFs.

• Use Java types

  // Organize the Java class in the org.alidata.odps.udf.examples package.
  package org.alidata.odps.udf.examples;  
  // Inherit the UDF class.
  import com.aliyun.odps.udf.UDF;         
  // Define a custom Java class.
  public final class Lower extends UDF { 
  // The evaluate method defines the UDF's logic. It takes a String and returns a String.
      public String evaluate(String s) { 
          if (s == null) { 
          return null; 
      } 
          return s.toLowerCase(); 
    } 
  }

• Use Java Writable types

  // Organize the Java class in the com.aliyun.odps.udf.example package.
  package com.aliyun.odps.udf.example;
  // Add the required classes for the Java Writable type.
  import com.aliyun.odps.io.Text;
  // Inherit the UDF class.
  import com.aliyun.odps.udf.UDF;
  // Define a custom Java class.
  public class MyConcat extends UDF {
    private Text ret = new Text();
  // Define the evaluate method. `Text` specifies the data type of the input parameters, and the `return` value is also a Text object.
    public Text evaluate(Text a, Text b) {
        if (a == null || b == null) {
        return null;
      }
        ret.clear();
        ret.append(a.getBytes(), 0, a.getLength());
        ret.append(b.getBytes(), 0, b.getLength());
        return ret;
    }
  }

Data type mappings

To ensure that the data types used in a Java UDF are consistent with MaxCompute data types, use the following mappings.

The Java byte[] type is not in the list of supported Java types. If you use byte[] for an input parameter or the return value of an evaluate method, an ODPS-0130071 error is reported. To process binary data in a UDF, use the Java types corresponding to the MaxCompute BINARY type: com.aliyun.odps.data.Binary for the standard Java type, or com.aliyun.odps.io.BytesWritable for the Writable type. You can also convert binary data to a Base64-encoded string and use the String type for the return value.

Hive UDF compatibility

If your MaxCompute project uses the 2.0 data type edition, MaxCompute supports Hive-style UDFs. You can directly use Hive UDFs developed for a compatible Hive version.

The compatible Hive version is 2.1.0, which corresponds to Hadoop 2.7.2. If your UDF was compiled against a different Hive or Hadoop version, recompile the UDF JAR file with Hive 2.1.0 or Hadoop 2.7.2.

For a detailed example of using a Hive UDF in MaxCompute, see UDF Example: Hive Compatibility.