By Ji Sheng
For binary packages, MaxCompute requires wheel packages with cp27-cp27m contained in their package names. However, for packages that have not been updated for a long time, for example, the crcmod package on which oss2 depends, PyPI does not provide wheel packages. Therefore, you have to create wheel packages yourself. This article describes how to use the quay.io/pypa/manylinux1_x86_64 image to build a wheel package that can be used on MaxCompute.
This article uses https://github.com/pypa/manylinux as a reference. The quay.io/pypa/manylinux1_x86_64 image is also the standard tool that the majority of Python projects currently use for packaging on Travis CI. If you have any further questions, you can study this project.
Many packages have dependencies, such as the devel rpm package or other Python packages. You need to know the dependencies of a specific package before packaging. Information about installation and packaging can usually be found on GitHub. The crcmod package has no other dependencies except gcc. Therefore, you can skip this step.
Older Python packages generally do not support building wheel packages. Specifically, an error is reported when 'python setup.py bdist_wheel' is used for packaging. To build wheel packages, you need to modify setup.py. For some packages, you can simply replace the setup function in distutils with the setup function in setuptools. For packages that require many custom operations in setup.py, you need to analyze the packaging process in detail. This is complicated and thus not discussed in this article.
For example, for crcmod, change
from distutils.core import setup in setup.py
to
from setuptools import setup
After making the preceding change, run the following in the root directory of the project:
python setup.py bdist_wheel
If no errors are reported and the generated wheel package can be used locally, it indicates that setup.py is available.
Create a bin directory in the project and build-wheel.sh under that directory:
mkdir bin && vim bin/build-wheel.sh
Enter the following content:
#! /bin/bash
# modified from https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
set -e -x
# Install a system package required by our library
# Change to the dependency installation command
# Compile wheels
PYBIN=/opt/python/cp27-cp27m/bin
# If dev-requirements.txt is present under the root directory of the package, remove the following comments.
# "${PYBIN}/pip" install -r /io/dev-requirements.txt
"${PYBIN}/pip" wheel /io/ -w wheelhouse/
# Bundle external shared libraries into the wheels
for whl in wheelhouse/*.whl; do
auditwheel repair "$whl" -w /io/wheelhouse/
done
Enter the dependency installation script that you obtained in Step 1 into this script. Make sure to use the version in /opt/python/cp27-cp27m/bin when using Python or pip.
Finally, set the execute permission.
chmod a+x bin/build-wheel.sh
Use Docker to download the required image and package the root directory of the project (Docker is required in this step. Please install it in advance):
docker pull quay.io/pypa/manylinux1_x86_64
docker run --rm -v `pwd`:/io quay.io/pypa/manylinux1_x86_64 /io/bin/build-wheel.sh
The generated wheel package is in the wheelhouse directory under the root directory of the project.
137 posts | 20 followers
FollowAlibaba Cloud MaxCompute - September 18, 2019
Alibaba Cloud MaxCompute - November 15, 2021
- November 23, 2017
Alibaba Cloud MaxCompute - September 12, 2018
Alibaba Cloud MaxCompute - October 18, 2021
Alibaba Clouder - September 10, 2018
137 posts | 20 followers
FollowAlibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn MoreConduct large-scale data warehousing with MaxCompute
Learn MoreAlibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn MoreApsaraDB for HBase is a NoSQL database engine that is highly optimized and 100% compatible with the community edition of HBase.
Learn MoreMore Posts by Alibaba Cloud MaxCompute