MaxCompute:Python 3 UDFs

Quick start

The following minimal example adds two integers and handles NULL inputs:

from odps.udf import annotate

@annotate("bigint,bigint->bigint")
class MyPlus(object):
    def evaluate(self, arg0, arg1):
        if None in (arg0, arg1):
            return None
        return arg0 + arg1

To run a Python 3 UDF, add the following session flag before your SQL statement:

SET odps.sql.python.version=cp37;
SELECT my_plus(col_a, col_b) FROM my_table;

UDF code structure

Every Python 3 UDF requires four components:

Limitations

Internet access (enforced at runtime)

UDFs cannot access the internet by default. To enable internet access, submit a Network Connection Request Form. The MaxCompute technical support team will contact you to complete the setup. For instructions, see Network Connection Request FormNetwork Access Process.

VPC access (enforced at runtime)

UDFs cannot access virtual private clouds (VPCs) by default. To access VPC resources from a UDF, first create a network connection between your MaxCompute project and the target VPC. For more information, see Access resources in a VPC using a UDF.

Reading table data (enforced at runtime)

UDFs, user-defined aggregate functions (UDAFs), and user-defined table-valued functions (UDTFs) cannot read data from the following table types:

• Tables with modified schemas (Schema Evolution)

• Tables that contain complex data types

• Tables that contain the JSON data type

• Transactional tables

Usage notes

Python 2 and Python 3 are not compatible. Do not mix Python 2 and Python 3 UDFs in the same SQL statement.

Python 2 reached end of life (EOL) in early 2020. For guidance on migrating existing UDFs, see Migrate Python 2 UDFs.

NULL handling

Handle NULLs explicitly inside your evaluate method:

def evaluate(self, arg):
    if arg is None:
        return None
    return arg.upper()

Develop a UDF

MaxCompute supports UDF development with MaxCompute Studio, DataWorks, and the MaxCompute client (odpscmd). All three tools follow the same workflow:

1. Write UDF code

2. Upload the Python file and register the function

3. Call the UDF in SQL

The following sections walk through the workflow for each tool, using the same example function GetUrlChar that extracts a URL segment by position.

SET odps.sql.python.version=cp37;
SELECT UDF_GET_URL_CHAR("http://www.taobao.com/a.htm", 1);

+-----+
| _c0 |
+-----+
|  a  |
+-----+

from odps.udf import annotate

@annotate("string,bigint->string")
class GetUrlChar(object):

    def evaluate(self, url, n):
        if n == 0:
            return ""
        try:
            index = url.find(".htm")
            if index < 0:
                return ""
            a = url[:index]
            index = a.rfind("/")
            b = a[index + 1:]
            c = b.split("-")
            if len(c) < n:
                return ""
            return c[-n]
        except Exception:
            return "Internal error"

SET odps.sql.python.version=cp37;
SELECT UDF_GET_URL_CHAR("http://www.taobao.com/a.htm", 1);

from odps.udf import annotate

@annotate("string,bigint->string")
class GetUrlChar(object):

    def evaluate(self, url, n):
        if n == 0:
            return ""
        try:
            index = url.find(".htm")
            if index < 0:
                return ""
            a = url[:index]
            index = a.rfind("/")
            b = a[index + 1:]
            c = b.split("-")
            if len(c) < n:
                return ""
            return c[-n]
        except Exception:
            return "Internal error"

SET odps.sql.python.version=cp37;
SELECT UDF_GET_URL_CHAR("http://www.taobao.com/a.htm", 1);

Third-party libraries

The built-in Python 3 runtime in MaxCompute does not include NumPy. To use NumPy, manually upload the NumPy wheel package. Download the package from PyPI or a mirror — the filename follows the pattern numpy-<version>-cp37-cp37m-manylinux1_x86_64.whl.

For upload instructions, see Resource operations or Use third-party packages in Python UDFs.

For a full list of standard libraries available in the Python 3.7 runtime, see The Python Standard Library.

Function signatures and data types

Before you write your UDF code, decide:

• Which input types your function accepts and which type it returns

• How your function handles NULL inputs (MaxCompute can pass NULLs to any UDF)

The function signature uses the @annotate decorator:

@annotate(<signature>)

The signature string format is:

'arg_type_list -> type'

Input types (`arg_type_list`)

Separate multiple input types with commas. The following types are supported:

BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), CHAR, VARCHAR, and complex types ARRAY, MAP, STRUCT (including nested complex types).

Two special values for arg_type_list:

Return type (`type`)

UDFs return a single column. The supported return types are:

BIGINT, STRING, DOUBLE, BOOLEAN, DATETIME, DECIMAL, FLOAT, BINARY, DATE, DECIMAL(precision,scale), and complex types ARRAY, MAP, STRUCT (including nested complex types).

The available types depend on the MaxCompute data type edition used by your project. For details, see Data type editions.

Signature examples

MaxCompute SQL to Python 3 type mappings

Write your UDF code using these type mappings to ensure consistency:

Reference resources

Reference files or tables inside UDF code using the odps.distcache module.

from odps.udf import annotate
from odps.distcache import get_cache_file

@annotate('bigint->string')
class DistCacheExample(object):
    def __init__(self):
        cache_file = get_cache_file('test_distcache.txt')
        kv = {}
        for line in cache_file:
            line = line.strip()
            if not line:
                continue
            k, v = line.split()
            kv[int(k)] = v
        cache_file.close()
        self.kv = kv

    def evaluate(self, arg):
        return self.kv.get(arg)

from odps.udf import annotate
from odps.distcache import get_cache_table

@annotate('->string')
class DistCacheTableExample(object):
    def __init__(self):
        self.records = list(get_cache_table('udf_test'))
        self.counter = 0
        self.ln = len(self.records)

    def evaluate(self):
        if self.counter > self.ln - 1:
            return None
        ret = self.records[self.counter]
        self.counter += 1
        return str(ret)

Call a UDF

Enable Python 3

MaxCompute projects use Python 2 for UDFs by default. To run a Python 3 UDF, add the following line before your SQL statement:

SET odps.sql.python.version=cp37;

Call within the same project

Call the UDF the same way as a built-in function:

SET odps.sql.python.version=cp37;
SELECT my_udf(column1, column2) FROM my_table;

Call across projects

To use a UDF from another project (for example, to use a UDF from Project B in Project A), prefix the function call with the source project name:

SELECT B:udf_in_other_project(arg0, arg1) AS res FROM table_t;

For more information, see Access resources across projects using packages.

Migrate Python 2 UDFs

Python 2 reached end of life (EOL) in early 2020.

For new projects, write all Python UDFs in Python 3.

For existing projects with Python 2 UDFs, proceed with caution when switching to Python 3. Two approaches are available:

• Write new UDFs in Python 3 and enable Python 3 at the session level for new jobs. For details, see Enable Python 3.

• Rewrite existing Python 2 UDFs to be compatible with both Python 2 and Python 3. For guidance, see Porting Python 2 code to Python 3.

For UDFs shared across multiple projects, write code that is compatible with both Python 2 and Python 3.

What's next

• Write a Hive UDF in Java

• Use complex data types in Java UDFs

• Replace strings by using regular expressions

• Obtain the values of strings that have no delimiters

• Obtain the values of strings that have delimiters

• Reference a file resource

• Reference a table resource

• Reference third-party packages in Python UDFs