MaxCompute:Generate a third-party package for PyODPS

Prerequisites

Before you begin, ensure that you have:

• PyODPS V0.11.3 or later installed

• Docker installed and running (required for Docker mode, which is the default)

• Python 3 (recommended; Python 2 packaging may fail for new projects)

Packaging scenarios at a glance

Use this table to find the command for your scenario, then follow the detailed instructions in the sections below.

How it works

pyodps-pack is installed automatically with PyODPS. On Linux and macOS, the executable is in the bin directory under the Python installation path. On Windows, it is in the Scripts directory. Run pyodps-pack commands from the Windows CLI, macOS Terminal, or Linux Shell — not from the DataWorks console, the MaxCompute client (odpscmd), or the Python CLI.

If the bin or Scripts directory is already in your PATH environment variable (set during Python installation or virtual environment activation), no additional setup is needed. Otherwise, navigate to the directory manually or add it to PATH.

After packaging completes, pyodps-pack prints all resolved dependency versions and writes packages.tar.gz to the current directory.

Choose a packaging mode

pyodps-pack supports two modes:

docker run --privileged --rm tonistiigi/binfmt --install arm64

Package PyPI dependencies

Usage notes:

• Use Python 3 for new MaxCompute projects. Packaging with Python 2 may fail.

• For existing Python 2 projects, migrate to Python 3 to simplify maintenance.

• On Linux, prefix pyodps-pack commands with sudo to ensure Docker has the necessary permissions.

• On macOS, do not use sudo with pyodps-pack — doing so may cause permission errors.

Docker mode:

pyodps-pack pandas

Non-Docker mode:

pyodps-pack --without-docker pandas

Specific version:

pyodps-pack pandas==1.2.5

After packaging, pyodps-pack prints the resolved versions of all included packages:

Package         Version
--------------- -------
numpy           1.21.6
pandas          1.2.5
python-dateutil 2.8.2
pytz            2022.6
six             1.16.0

The output archive packages.tar.gz is written to the current directory.

pyodps-pack --mcpy27 pandas

pyodps-pack --dwpy27 pandas

Package custom code

pyodps-pack packages custom Python projects built with setup.py or pyproject.toml. See Build System Interface for details.

The following example packages a project based on pyproject.toml with this directory structure:

test_package_root
├── test_package
│   ├── __init__.py
│   ├── mod1.py
│   └── subpackage
│       ├── __init__.py
│       └── mod2.py
└── pyproject.toml

Example pyproject.toml:

[project]
name = "test_package"
description = "pyodps-pack example package"
version = "0.1.0"
dependencies = [
    "pandas>=1.0.5"
]

Run the following command, replacing <path_to_package> with the parent directory of test_package_root:

pyodps-pack /<path_to_package>/test_package_root

This compresses the project and all its dependencies into packages.tar.gz.

Package code from a Git repository

Pass the Git URL directly:

pyodps-pack git+https://github.com/aliyun/aliyun-odps-python-sdk.git

To package a specific branch or tag:

pyodps-pack git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

If the packaging process requires build-time dependencies (such as Cython), specify them with --install-requires. These dependencies are used during packaging but are not necessarily included in the output archive.

pyodps-pack \
    --install-requires cython \
    git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

Alternatively, list build-time dependencies in a file that follows the requirements.txt format and pass it with --install-requires-file:

cython>0.29

pyodps-pack \
    --install-requires-file install-requires.txt \
    git+https://github.com/aliyun/aliyun-odps-python-sdk.git@v0.11.2.2

Package binary dependencies

Some packages include binary components — such as dynamic-link libraries — that must be compiled before packaging. Use --run-before to run a Bash script that installs these binary components before pyodps-pack runs.

The following example packages GDAL 3.6.0, which requires libgdal (version > 3.6.0) and PROJ (version 6.0 or later), both compiled with CMake.

1. Write a script named install-gdal.sh that compiles and installs PROJ and GDAL:

   #!/bin/bash
   set -e
   
   cd /tmp
   curl -o proj-6.3.2.tar.gz https://download.osgeo.org/proj/proj-6.3.2.tar.gz
   tar xzf proj-6.3.2.tar.gz
   cd proj-6.3.2
   mkdir build && cd build
   cmake ..
   cmake --build .
   cmake --build . --target install
   
   cd /tmp
   curl -o gdal-3.6.0.tar.gz http://download.osgeo.org/gdal/3.6.0/gdal-3.6.0.tar.gz
   tar xzf gdal-3.6.0.tar.gz
   cd gdal-3.6.0
   mkdir build && cd build
   cmake ..
   cmake --build .
   cmake --build . --target install

2. Run pyodps-pack with the script and the oldest-supported-numpy build dependency:

   pyodps-pack --install-requires oldest-supported-numpy --run-before install-gdal.sh gdal==3.6.0

Parameters

Troubleshoot packaging failures

What's next

After generating the package, upload it to MaxCompute as an archive resource and reference it in your PyODPS node. See Reference a third-party package in a PyODPS node.