MaxCompute:MaxCompute UDF (Java) FAQ

Common class or dependency issues when calling a MaxCompute UDF:

• Symptom 1: A runtime error occurs with the message ClassNotFoundException or Some dependencies are missing.

  • Causes:

    • The resource JAR package specified when you created the UDF is incorrect.

    • The resource JAR package on which the UDF depends is not uploaded to MaxCompute. For example, a dependent third-party package is not uploaded.

    • The project is incorrect when you call the UDF. The UDF is not in the current MaxCompute project. For example, the UDF is registered in a development project but is called in a production project.

    • The file resource does not exist or the resource type is incorrect. For example, for a PY file, the resource type is PY, but the type required by get_cache_file in the UDF code is FILE.

  • Solutions:

    • Check the JAR package to confirm that it is correct and contains the required classes. Repackage the file and upload it to the MaxCompute project. For more information about packaging, uploading, and registration, see Package, upload, and register a UDF.

    • Upload the third-party package on which the UDF depends to the MaxCompute project as a resource. Then, add this package to the list of dependent resources when you register the function. For more information about adding resources and registering functions, see Add a resource and Register a function.

    • To resolve the issue described in Cause 3, execute the list functions; command in the project where the error occurred using the MaxCompute client. This ensures that the MaxCompute UDF exists and that its class and dependent resources are correct.

    • To resolve the issue described in Cause 4, use the MaxCompute client to run the desc function <function_name>; command to ensure that the Resources list includes all required file resources. If the resource type does not match, you can run the add <file_type> <file_name>; command to add the resource again.

• Symptom 2: A runtime error message is displayed, such as NoClassDefFoundError or NoSuchMethodError, or the error code is ODPS-0123055.

  • Causes:

    • The version of a third-party library in the uploaded JAR package is inconsistent with the version of the built-in third-party library in MaxCompute.

    • Java sandbox restrictions. If the Stderr of the job instance contains the message java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "createClassLoader"), a sandbox restriction is indicated. MaxCompute UDFs are subject to Java sandbox restrictions when they run in a distributed environment. For more information about Java sandbox restrictions, see Java sandbox.

  • Solutions:

    • You can use maven-shade-plugin to resolve the version inconsistency and modify the import path. Repackage the file into a JAR package and upload it to the MaxCompute project. For more information about packaging, uploading, and registration, see Package, upload, and register a UDF.

    • See Java sandbox restriction issues.

Common performance issues when calling a MaxCompute UDF:

• Symptom 1: The runtime error is kInstanceMonitorTimeout.

  • Cause: The UDF processing time exceeds the timeout limit. By default, a batch of records (typically 1,024) must be processed within 1,800 seconds. This limit applies to processing a single batch, not the total runtime of the worker. SQL typically processes data at more than 10,000 records per second. The limit prevents infinite loops in UDFs that would cause prolonged CPU usage.

  • Solutions:

    • If the actual data volume for computation is large, you can call ExecutionContext.claimAlive in the method of the UDF's implementation class to reset the timer.

    • Optimize the UDF code logic. You can also configure the following session-level parameters to adjust UDF runtime and improve processing speed.

      Parameter	Description
      set odps.function.timeout=xxx;	Adjusts the UDF runtime timeout. The default value is 1800s. You can increase this value as needed. The value ranges from 1s to 3600s.
      set odps.stage.mapper.split.size=xxx;	Adjusts the input data volume for a Map worker. The default value is 256 MB. You can decrease this value as needed.
      set odps.sql.executionengine.batch.rowcount=xxx;	Adjusts the number of data rows that MaxCompute processes at a time. The default value is 1024. You can decrease this value as needed.

• Symptom 2: The runtime returns an error message of errMsg:SigKill(OOM) or OutOfMemoryError.

  • Cause: MaxCompute jobs run in three main stages: Map, Reduce, and Join. If the data volume is large, each instance in these stages may require a long processing time.

  • Solutions:

    • If the error is related to fuxi or runtime code, you can set the following resource parameters to improve processing speed.

      Parameter	Description
      set odps.stage.mapper.mem=xxx;	Adjusts the memory size of a Map worker. The default value is 1024 MB. You can increase this value as needed.
      set odps.stage.reducer.mem=xxx;	Adjusts the memory size of a Reduce worker. The default value is 1024 MB. You can increase this value as needed.
      set odps.stage.joiner.mem=xxx;	Adjusts the memory size of a Join worker. The default value is 1024 MB. You can increase this value as needed.
      set odps.stage.mapper.split.size=xxx;	Adjusts the input data volume for a Map worker. The default value is 256 MB. You can increase this value as needed.
      set odps.stage.reducer.num=xxx;	Adjusts the number of workers in the Reduce stage. You can increase this value as needed.
      set odps.stage.joiner.num=xxx;	Adjusts the number of workers in the Join stage. You can increase this value as needed.

    • If the error is in the Java code, you can also increase the JVM memory by setting set odps.sql.udf.jvm.memory=xxx; in addition to the parameters above.

Common issues when calling a Java user-defined table-valued function (UDTF):

• Symptom 1: Runtime error: Semantic analysis exception: only a single expression in the SELECT clause is supported with UDTFs.

  • Cause: When you call a Java UDTF in a SELECT statement, the UDTF is mixed with other columns or expressions. This is not supported. The following example shows incorrect usage.

    select b.*, 'x', udtffunction_name(v) from table lateral view udtffunction_name(v) b as f1, f2;

  • Solution: Use the Java UDTF with LATERAL VIEW. Example:

    select b.*, 'x' from table lateral view udtffunction_name(v) b as f1, f2;

• Symptom 2: Runtime error: Semantic analysis exception - expect 2 aliases but have 0.

  • Cause: The output column names are not specified in the Java UDTF code.

  • Solution: Specify the column names in the as clause of the SELECT statement. Example:

    select udtffunction_name(paramname) as (col1, col2);