Platform For AI:Deploy a TensorFlow model with SDK

Install GCC

This example uses the default GCC 4.8 for CentOS:

yum install -y gcc-c++

Install Python 3 and set up a virtual environment

# Update pip
python3 -m pip install --upgrade pip

# Install and activate a virtual environment
pip3 install virtualenv==16.0
python3 -m virtualenv venv
source venv/bin/activate

Install TensorFlow and download the C++ library

TensorFlow model inference requires two shared libraries: libtensorflow_framework.so and libtensorflow_cc.so. In production, build a TensorFlow Wheel package that matches the same configuration, environment, and compiler version as libtensorflow_cc.so. This example uses the community edition of TensorFlow and a precompiled libtensorflow_cc.so for demonstration only — do not use these files in production.

# Install TensorFlow
pip3 install tensorflow-gpu==1.15.0

# Download libtensorflow_cc.so
wget http://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/demo/sdk/tensorflow/libtensorflow_cc.so

Step 1: Prepare the model

Download the sample optimized model. Alternatively, use your own model optimized with PAI-Blade. See Optimize a TensorFlow model.

wget http://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/demo/asr_frozen.pb

Step 2: Download the inference code

A PAI-Blade optimized model runs the same way as a standard TensorFlow model — no extra code or configuration changes are required. Download the sample inference code:

wget http://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/demo/sdk/tensorflow/tf_sdk_demo.cc

The downloaded tf_sdk_demo.cc contains only general TensorFlow inference logic with no PAI-Blade-specific code.

Step 3: Compile the inference code

To run the PAI-Blade optimized model, link libtf_blade.so and libtao_ops.so from the SDK's /usr/local/lib directory when compiling. Compared with a standard TensorFlow Serving compilation, you must also link these two .so files provided by PAI-Blade that contain optimization operators.

# Retrieve TensorFlow compile flags
TF_COMPILE_FLAGS=$(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))')

# Retrieve TensorFlow link flags
TF_LD_FLAGS=$(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))')

# libtensorflow_cc.so is in the current directory
TF_CC_PATH='.'

g++ -std=c++11 tf_sdk_demo.cc \
    ${TF_COMPILE_FLAGS} \
    ${TF_LD_FLAGS} \
    -L ${TF_CC_PATH} \
    -L /usr/local/lib \
    -ltensorflow_cc \
    -ltf_blade \
    -ltao_ops \
    -o demo_cpp_sdk.bin

Replace these values as needed:

Step 4: Run inference

Set the required environment variables and run the compiled binary against the optimized model. The binary uses BLADE_REGION and BLADE_TOKEN to authenticate with PAI-Blade.

# Required: contact the PAI team to obtain these values
export BLADE_REGION=<region>
export BLADE_TOKEN=<token>

TF_LD_FLAGS=$(python3 -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))')
TF_FRAMEWORK_PATH=`echo $TF_LD_FLAGS | awk '{print $1}' | sed "s/-L//g"`

LD_LIBRARY_PATH=${TF_FRAMEWORK_PATH}:${TF_CC_PATH}:/usr/local/lib:${LD_LIBRARY_PATH} ./demo_cpp_sdk.bin asr_frozen.pb

Replace these values as needed:

When the model loads and inference begins, the output looks similar to the following, which indicates that the model has started to run:

...
2020-11-20 16:41:41.263192: I demo_cpp_sdk.cpp:96] --- Execution uses: 41.995 ms
2020-11-20 16:41:41.305550: I demo_cpp_sdk.cpp:96] --- Execution uses: 42.334 ms
2020-11-20 16:41:41.347772: I demo_cpp_sdk.cpp:96] --- Execution uses: 42.195 ms
2020-11-20 16:41:41.390894: I demo_cpp_sdk.cpp:96] --- Execution uses: 43.09 ms
2020-11-20 16:41:41.434968: I demo_cpp_sdk.cpp:96] --- Execution uses: 44.047 ms
...