Platform For AI:Deploy a PyTorch model with SDK

Prepare the server

Provision a server with the following configuration. This guide uses an Elastic Compute Service (ECS) instance:

Set up Python 3

# Update pip.
python3 -m pip install --upgrade pip

# Install virtualenv and create a virtual environment.
pip3 install virtualenv==16.0
python3 -m virtualenv venv

# Activate the virtual environment.
source venv/bin/activate

Step 1: Download the model and test data

This guide uses a pre-optimized ResNet-50 sample model. Download it along with the test input data:

# Download the optimized sample model.
wget http://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/demo/sdk/pytorch/optimized_resnet50.pt

# Download the test input data.
wget http://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/demo/sdk/pytorch/inputs.pth

To use your own model instead, substitute optimized_resnet50.pt with your PAI-Blade-optimized model file in the steps that follow.

Step 2: Prepare the inference code

The inference code for a PAI-Blade model uses the same PyTorch C++ API (torch::jit::load()) as any standard TorchScript model — no extra code or configuration needed.

Save the following code to a file named torch_app.cc:

#include <torch/script.h>
#include <torch/serialize.h>
#include <chrono>
#include <iostream>
#include <fstream>
#include <memory>

int benchmark(torch::jit::script::Module &module,
             std::vector<torch::jit::IValue> &inputs) {
  // Warm up: run 10 iterations before timing.
  for (int k = 0; k < 10; ++ k) {
    module.forward(inputs);
  }
  auto start = std::chrono::system_clock::now();
  // Benchmark: run 20 iterations.
  for (int k = 0; k < 20; ++ k) {
    module.forward(inputs);
  }
  auto end = std::chrono::system_clock::now();
  std::chrono::duration<double> elapsed_seconds = end-start;
  std::time_t end_time = std::chrono::system_clock::to_time_t(end);

  std::cout << "finished computation at " << std::ctime(&end_time)
            << "\nelapsed time: " << elapsed_seconds.count() << "s"
            << "\navg latency: " << 1000.0 * elapsed_seconds.count()/20 << "ms\n";
  return 0;
}

torch::Tensor load_data(const char* data_file) {
  std::ifstream file(data_file, std::ios::binary);
  std::vector<char> data((std::istreambuf_iterator<char>(file)), std::istreambuf_iterator<char>());
  torch::IValue ivalue = torch::pickle_load(data);
  CHECK(ivalue.isTensor());
  return ivalue.toTensor();
}

int main(int argc, const char* argv[]) {
  if (argc != 3) {
    std::cerr << "usage: example-app <path-to-exported-script-module> <path-to-saved-test-data>\n";
    return -1;
  }

  torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
    auto image_tensor = load_data(argv[2]);

    std::vector<torch::jit::IValue> inputs{image_tensor};
    benchmark(module, inputs);
    auto outputs = module.forward(inputs);
  }
  catch (const c10::Error& e) {
    std::cerr << "error loading the model" << std::endl << e.what();
    return -1;
  }

  std::cout << "ok\n";
}

Step 3: Compile the inference program

Compile torch_app.cc by linking the libtorch libraries together with the PAI-Blade shared libraries (libtorch_blade.so and libral_base_context.so) from the SDK's /usr/local/lib directory.

TORCH_DIR=$(python3 -c "import torch; import os; print(os.path.dirname(torch.__file__))")
g++ torch_app.cc -std=c++14 \
    -D_GLIBCXX_USE_CXX11_ABI=0 \
    -I ${TORCH_DIR}/include \
    -I ${TORCH_DIR}/include/torch/csrc/api/include \
    -Wl,--no-as-needed \
    -L /usr/local/lib \
    -L ${TORCH_DIR}/lib \
    -l torch -l torch_cuda -l torch_cpu -l c10 -l c10_cuda \
    -l torch_blade -l ral_base_context \
    -o torch_app

Key flags explained:

Adjustable parameters:

• torch_app.cc: the inference code filename

• torch_app: the output executable name

Step 4: Run inference

Set the required environment variables and run the compiled executable against the model and test data:

export BLADE_REGION=<region>    # For example: cn-beijing, cn-shanghai.
export BLADE_TOKEN=<token>
export LD_LIBRARY_PATH=/usr/local/lib:${TORCH_DIR}/lib:${LD_LIBRARY_PATH}
./torch_app optimized_resnet50.pt inputs.pth

Replace the following placeholders:

If inference runs successfully, the output looks similar to:

finished computation at Wed Jan 27 20:03:38 2021

elapsed time: 0.513882s
avg latency: 25.6941ms
ok

The exact timing values vary by hardware. The final ok confirms the model loaded and ran without errors.