MaxCompute:UDF example: Use third-party packages in Python UDFs

Prerequisites

Before you begin, make sure that the following requirements are met:

• Python is installed. Python 3 is recommended.

• The MaxCompute client is installed and configured. For more information, see Install and configure the MaxCompute client.

• For packages that require compilation, ensure that pip, setuptools, and wheel are installed. You can run the pip install setuptools command to install setuptools and run the pip install wheel command to install wheel.

• If you use GDAL 3.0 or later, make sure that PROJ 6 is installed.

• If you use Docker to compile a third-party package, make sure that Docker is installed. For more information, see the Docker installation documentation.

Use the NumPy package (Python 3 UDFs)

MaxCompute's Python 2 environment includes NumPy by default. To use the NumPy package in a Python 3 UDF, you must manually upload the package by following these steps:

1. Taking version 1.19.2 as an example, go to the PyPI page. In the Download files section, download the NumPy package whose filename ends with cp37-cp37m-manylinux1_x86_64.whl.

   Note

   Loading may fail if you use a package with a different filename extension. If you want to use a different version, go to the Navigation section in the upper-left corner of the PyPI page and click Release history to view historical versions.

2. Change the filename extension of the downloaded NumPy package to .zip.

   Example: numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip.

3. Use the MaxCompute client to upload the NumPy package to a MaxCompute project. For more information about how to upload a resource, see Resource operations.

   Sample command:

   ADD ARCHIVE D:\Downloads\numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip -f;

4. Write a Python UDF script and save it as a .py file.

   The following is a sample Python UDF script saved as import_numpy.py:

   from odps.udf import annotate
   
   @annotate("->string")
   class TryImport(object): # The class name is TryImport.
       def __init__(self):
           import sys
           sys.path.insert(0, 'work/numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip') # The NumPy package. You only need to replace the package name that follows work/.
   
       def evaluate(self):
           import numpy
           return "import succeed"

5. Use the MaxCompute client to upload the import_numpy.py script as a resource to your MaxCompute project.

   Sample command:

   ADD PY D:\Desktop\import_numpy.py -f;

6. Use the uploaded import_numpy.py script and NumPy package to register a user-defined function through the MaxCompute client. For more information about how to register a function, see Function operations.

   Assume that the registered user-defined function is named numpy and the function resource is located in the test_project project. Sample command:

   CREATE FUNCTION numpy AS 'import_numpy.TryImport' USING 'test_project/resources/import_numpy.py,numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip';

   Note

   When you register the function, you must add the NumPy package, such as numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip, to the resource list.

7. After registering the function, call it from your SQL statements. Ensure you enable Python 3 for execution. For more information, see Python 3 UDF development specifications and workflow.

Use packages that require compilation

If a third-party package is a TAR.GZ archive from PyPI or a source code package from GitHub, the root directory of the decompressed package may contain a setup.py file. Before you can use this type of package, you must compile the setup.py file to generate a wheel package in an environment that is compatible with MaxCompute. Then, upload the resource and register the function. You can then call the third-party package using a Python UDF. For more information about uploading a resource and registering a function, see Use the NumPy package (Python 3 UDFs).

Once your environment meets the compatibility requirements, follow these steps to generate a wheel package from a setup.py file:

1. Extract the third-party package to a local directory. In the CLI of your system, switch to the folder where the setup.py file is located.

   For example, if you download the GDAL-3.2.0.zip package and extract it, the setup.py file is in the D:\Downloads\GDAL-3.2.0 directory. Sample command:

   cd D:\Downloads\GDAL-3.2.0

2. In the CLI, run the following command to check whether bdist_wheel is in the output.

   Sample command:

   python setup.py --help-command

   • If bdist_wheel is in the output, go to Step 3.

   • If bdist_wheel is not in the output, you must modify the setup.py script. Change from distutils.core import setup to from setuptools import setup. Then, go to Step 3.

3. In the CLI, run the following command to compile and generate a wheel package.

   python setup.py bdist_wheel 

   Note

   The wheel package is in the dist directory.

Use packages with dynamic-link libraries

To use a third-party package that depends on an .so library file in a Python UDF, follow these steps:

1. View the dependencies. You can view the dependencies in the Dependencies section of the PyPI page.

   For example, the dependencies of GDAL 3.0.4 are as follows.

   Dependencies
   
   - libgdal (3.0.4 or greater) and header files (gdal-devel)
   - numpy (1.0.0 or greater) and header files (numpy-devel) (not explicitly required, but many examples and utilities will not work without it)

   Note

   As shown above, the dependencies include libgdal and numpy. libgdal is obtained by compiling the GDAL source code in a container, and numpy is obtained by downloading the NumPy package from the PyPI page or in a Docker container.

2. Download the NumPy package.

   You can download the NumPy package using one of the following methods:

   • On the PyPI page, go to the Download files section and download the NumPy package whose filename ends with cp37-cp37m-manylinux1_x86_64.whl.

     Note

     If you are using a Python 2 environment, you must download a compatible NumPy package. In the Navigation area on the left side of the PyPI page, click Release history, select version 1.16.6 or earlier, and download the package whose filename ends with cp27-cp27m-manylinux1_x86_64.whl.

   • In the quay.io/pypa/manylinux2010_x86_64 Docker container, run the /opt/python/cp37-cp37m/bin/pip download numpy -d ./ command to download the NumPy package to the current directory.

3. Compile the .so library file.

   1. Download the GDAL 3.0.4 source code and extract it to a local directory.

   2. Use Docker to pull the quay.io/pypa/manylinux2010_x86_64 container image and enter the interactive terminal mode.

      Sample commands:

      docker pull quay.io/pypa/manylinux2010_x86_64
      docker run -it quay.io/pypa/manylinux2010_x86_64 /bin/bash

   3. Copy the GDAL 3.0.4 source code to the container.

      Sample command:

      docker cp ./gdal-3.0.4 <CONTAINER ID>:/opt/source/  

      To obtain the CONTAINER ID, see docker ps.

4. Compile GDAL 3.0.4 in the container. For more information, see BuildingOnUnix.

   Sample commands:

   # You must specify the installation directory of PROJ 6 in the configure option.
   ./configure --prefix=/path/to/install/prefix --with-proj=/path/to/install/proj6/prefix
   make
   make install
   export PATH=/path/to/install/prefix/bin:$PATH
   export LD_LIBRARY_PATH=/path/to/install/prefix/lib:$LD_LIBRARY_PATH
   export GDAL_DATA=/path/to/install/prefix/share/gdal
   # Test
   gdalinfo --version

   The following errors may occur during compilation:

   • configure: error: PROJ 6 symbols not found: GDAL 3.0 and later versions depend on PROJ 6. You must download and install PROJ 6.

   • fatal error: zlib.h: No such file or directory: Run the yum install zlib-devel command and compile again.

5. Use the docker cp command to copy the libgdal.so and libproj.so files from the container to your local machine. These are located in the lib folder of their respective installation directories. Ensure you copy the actual files, not their symbolic links.

6. Create a GDAL wheel package in the container. For more information, see BuildingOnUnix.

   Sample commands:

   # If NumPy is required, you must install NumPy first.
   /opt/python/cp37-cp37m/bin/pip install numpy
   # Switch to the directory of the GDAL source code.
   cd swig/python
   # A wheel package is generated in the dist directory. Example: GDAL-3.0.4-cp37-cp37m-linux_x86_64.whl
   /opt/python/cp37-cp37m/bin/python setup.py bdist_wheel

7. Based on the generated .so library files, wheel package, or NumPy package, upload the resources and register the function to use the third-party package in a Python UDF. For more information about how to upload resources and register a function, see Use the NumPy package (Python 3 UDFs).

   Note the following points:

   • When you upload resources, you must upload libgdal.so and libproj.so as FILE resources, and upload numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip and GDAL-3.0.4-cp37-cp37m-linux_x86_64.zip as ARCHIVE resources.

   • When you register the function, you must add libgdal.so, libproj.so, numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip, and GDAL-3.0.4-cp37-cp37m-linux_x86_64.zip to the function's resource list.

   The following is sample code for the Python UDF:

   Note

   This example uses Python 3 code and must be run in a Python 3 environment. If you need to run it in a Python 2 environment, take note of the usage of the get_cache_file parameter. For more information, see Reference resources.

   # coding: utf-8
   from odps.udf import annotate
   from odps.distcache import get_cache_file
   
   def include_file(file_name):
       import os, sys
       so_file = get_cache_file(file_name, 'b')
       
       with open(so_file.name, 'rb') as fp:
           content=fp.read()
           so = open(file_name, "wb")
           so.write(content)
           so.flush()
           so.close()
   
   @annotate("->string")
   class TryImport(object):
       def __init__(self):
           import sys
           include_file('libgdal.so.26')
           include_file('libproj.so.15')
           sys.path.insert(0, 'work/GDAL-3.0.4-cp37-cp37m-linux_x86_64.zip') # The compiled GDAL package. You only need to replace the package name that follows work/.
           sys.path.insert(0, 'work/numpy-1.19.2-cp37-cp37m-manylinux1_x86_64.zip') # The NumPy package. You only need to replace the package name that follows work/.
   
       def evaluate(self):
           from osgeo import gdal
           from osgeo import ogr
           from osgeo import osr
           from osgeo import gdal_array
           from osgeo import gdalconst
           return "import succeed"

   Note

   If you encounter a runtime error that libgdal.so.26 or libproj.so.15 cannot be found, rename the uploaded libgdal.so and libproj.so files to libgdal.so.26 and libproj.so.15, respectively.