MaxCompute:UDF development (Python 3)

You can use MaxCompute Studio to write a UDF in Python 3. The code must contain the following components:

• Module import: required.

  UDF code must include from odps.udf import annotate, which is used to import the function signature. This way, MaxCompute can identify the function signature that is defined in the code. If you want to reference files or tables in UDF code, the UDF code must include from odps.distcache import get_cache_file or from odps.distcache import get_cache_table.

• Function signature: Required.

  The format is @annotate(<signature>), where signature defines the data types of the function's input parameters and return value. For more information about function signatures, see Function signatures and data types.

• Custom Python class: Required.

  The class organizes your UDF code and defines the variables and methods that implement your business logic. You can also reference a built-in third-party library or reference file and table resources in your code. For more information, see Third-party library or Reference resources.

• evaluate method: required.

  The evaluate method is contained in the custom Python class. The evaluate method defines the input parameters and return value of the UDF. Each Python class can contain only one evaluate method.

The following code provides a UDF example.

# Import the function signature module.
from odps.udf import annotate
# Define the function signature.
@annotate("bigint,bigint->bigint")
# Define the custom Python class.
class MyPlus(object):
# Implement the evaluate method.
    def evaluate(self, arg0, arg1):
        if None in (arg0, arg1):
            return None
        return arg0 + arg1

Format of function signatures:

@annotate(<signature>)

The signature parameter is a string that specifies the data types of input parameters and return value. When you run a UDF, the data types of the input parameters and return value of the UDF must be consistent with the data types specified in the function signature. The data type consistency is checked during semantic parsing. If the data types are inconsistent, an error is returned. Format of a signature:

'arg_type_list -> type'

Parameter description:

• arg_type_list: specifies the data types of input parameters. If multiple input parameters are used, their data types are separated by commas (,). The following data types are supported: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, and VARCHAR. Complex data types, such as ARRAY, MAP, and STRUCT, and nested complex data types are also supported.

  arg_type_list can be represented by an asterisk (*) or left empty ('').

  • If arg_type_list is represented by an asterisk (*), a random number of input parameters are allowed.

  • If arg_type_list is left empty (''), no input parameters are used.

• type: specifies the data type of the return value. For a UDF, only one column of values is returned. The following data types are supported: BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, and DECIMAL(precision,scale). Complex data types, such as ARRAY, MAP, and STRUCT, and nested complex data types are also supported.

The following table provides examples of valid function signatures.

The following code shows how to reference a table.

from odps.udf import annotate
from odps.distcache import get_cache_table
@annotate('->string')
class DistCacheTableExample(object):
    def __init__(self):
        self.records = list(get_cache_table('udf_test'))
        self.counter = 0
        self.ln = len(self.records)
    def evaluate(self):
        if self.counter > self.ln - 1:
            return None
        ret = self.records[self.counter]
        self.counter += 1
        return str(ret)

After developing a Python 3 UDF by following the development workflow, you can call it in MaxCompute SQL as follows:

set odps.sql.python.version=cp37;

Python 2 reached its EOL in early 2020. We recommend migrating your projects based on their type:

• New projects: For new MaxCompute projects or projects where you are writing Python UDFs for the first time, use Python 3 for all Python UDFs.

• Existing projects: For projects with many Python 2 UDFs, exercise caution when enabling Python 3. To gradually migrate, use the following methods:

  • New jobs and new UDFs: Use Python 3 to write UDFs and enable Python 3 at the session level. For more information about how to enable Python 3, see Enable Python 3.

  • Python 2 UDFs: Rewrite Python 2 UDFs to make them compatible with both Python 2 and Python 3. For more information about how to rewrite the UDFs, see Porting Python 2 Code to Python 3.

    Note

    If you write public UDFs shared across multiple MaxCompute projects, ensure they are compatible with both Python 2 and Python 3.