MaxCompute:Generate a third-party package for PyODPS

To run the pyodps-pack tool in Docker mode, you must install Docker. The pyodps-pack tool can automatically call Docker that you installed. Therefore, you do not need to manually run the pyodps-pack toll in Docker.

• Linux operating system: For more information about how to install Docker in Linux, see Install Docker Engine.

• macOS or Windows operating system:

  • If you are an individual developer, you can use Docker Desktop.

  • If you are an unauthorized enterprise user, we recommend that you use the open source Rancher Desktop.

After you install PyODPS, you can run the following commands to package pandas and pandas dependencies for Python 3 in the local CLI:

• Package all dependencies in Docker mode.

  pyodps-pack pandas

• Package all dependencies in non-Docker mode.

  pyodps-pack --without-docker pandas

• Specify the pandas version for packaging dependencies.

  pyodps-pack pandas==1.2.5

After you perform the packaging operations, the versions of all dependency packages are displayed in the pyodps-pack tool, as shown in the following code. The packages.tar.gz file is generated in the current directory. The file contains the following dependency packages:

Package         Version
--------------- -------
numpy           1.21.6
pandas          1.2.5
python-dateutil 2.8.2
pytz            2022.6
six             1.16.0

If you want to generate a package in Python 2.7, check whether the generated package is used in MaxCompute or DataWorks. For more information, see PyODPS DataFrame.

• If you want to use the generated package in MaxCompute, run the following command:

  pyodps-pack --mcpy27 pandas

• If you want to use the generated package in DataWorks, run the following command:

  pyodps-pack --dwpy27 pandas

In this example, the pyodps-pack tool is used to package a project that is created based on the pyproject.toml file. The project uses the following directory structure:

test_package_root
├── test_package
│   ├── __init__.py
│   ├── mod1.py
│   └── subpackage
│       ├── __init__.py
│       └── mod2.py
└── pyproject.toml 

The pyproject.toml file may contain the following data:

[project]
name = "test_package"
description = "pyodps-pack example package"
version = "0.1.0"
dependencies = [
    "pandas>=1.0.5"
]

After the package is generated, run the following command to compress the generated package and all dependency packages into the packages.tar.gz file. In the command, replace path_to_package with the upper-level directory of test_package_root.

pyodps-pack /<path_to_package>/test_package_root

You can use the pyodps-pack tool to package the code in a Git repository such as a GitHub repository. For example, you can run the following command to package PyODPS code:

pyodps-pack git+https://github.com/aliyun/aliyun-odps-python-sdk.git

If you want to package a branch or tag, you can run the following command:

pyodps-pack git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

To package code, you may need to install the dependencies that are required for packaging, such as Cython. You can use the --install-requires parameter to specify the dependencies that are required for installation. You can also write a file named install-requires.txt that has the same format as the requirements.txt file and use the --install-requires-file parameter to specify the written file. For example, if you need to install Cython before you package PyODPS code, you can run the following command:

pyodps-pack \
    --install-requires cython \
    git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

You can also run the following command to create a file named install-requires.txt and write content to the file based on the format of the requirements.txt file. Sample file content:

cython>0.29

Sample packaging command:

pyodps-pack \
    --install-requires-file install-requires.txt \
    git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

Some packages contain additional binary dependencies, such as dynamic-link libraries that you need to compile or install. The pyodps-pack tool provides the --run-before parameter to specify the step you need to perform before the packaging operation. In the specified step, you can install the required binary dependencies. The following example describes how to package the Geospatial Data Abstraction Library (GDAL).

1. Determine the binary dependencies that need to be installed during packaging.

   Install libgdal in a version later than 3.6.0 based on the documentation of GDAL 3.6.0 on PyPI. As mentioned in libgdal compilation instructions, the package requires the PROJ package of version 6.0 or later. Both the binary packages need to be compressed into a file by using CMake. Write a script file for compilation of these binary dependencies and save the file as install-gdal.sh.

   #!/bin/bash
   set -e
   
   cd /tmp
   curl -o proj-6.3.2.tar.gz https://download.osgeo.org/proj/proj-6.3.2.tar.gz
   tar xzf proj-6.3.2.tar.gz
   cd proj-6.3.2
   mkdir build && cd build
   cmake ..
   cmake --build .
   cmake --build . --target install
   
   cd /tmp
   curl -o gdal-3.6.0.tar.gz http://download.osgeo.org/gdal/3.6.0/gdal-3.6.0.tar.gz
   tar xzf gdal-3.6.0.tar.gz
   cd gdal-3.6.0
   mkdir build && cd build
   cmake ..
   cmake --build .
   cmake --build . --target install

2. Run the following command to generate a package by using the pyodps-pack tool:

   pyodps-pack --install-requires oldest-supported-numpy --run-before install-gdal.sh gdal==3.6.0

The following table describes the parameters available for pyodps-pack commands. You can use the parameters to control the packaging process.