Platform For AI:PyTorch processor

Prerequisites

Before you begin, make sure you have:

• A PyTorch model exported to TorchScript format (.pt file)

• The eascmd client installed and configured

• An Alibaba Cloud Object Storage Service (OSS) bucket to store the model file

Choose a processor version

EAS provides CPU and GPU processor variants for PyTorch 1.6 through 1.10. Select the processor name that matches your PyTorch version and hardware target.

Processor names follow the pattern pytorch_{cpu|gpu}_{version}. Use a _cpu_ processor for CPU-only instances, and a _gpu_ processor for GPU instances.

Step 1: Deploy a service

Choose one of the following deployment methods:

• eascmd (recommended): Suitable for scripted or automated deployments

• Console: Suitable for one-off deployments or when you prefer a visual interface

{
  "name": "pytorch_resnet_example",
  "model_path": "http://examplebucket.oss-cn-shanghai.aliyuncs.com/models/resnet18.pt",
  "processor": "pytorch_cpu_1.6",
  "metadata": {
    "cpu": 1,
    "instance": 1,
    "memory": 1000
  }
}

Step 2: Call the service

PyTorch services use Protocol Buffers (Protobuf) for input and output. The EAS console's online debugging tool supports only plain text and cannot be used with PyTorch services. Use the EAS SDK or build raw Protobuf requests instead.

The EAS SDK handles request serialization and includes built-in fault tolerance and direct connection support. The following Python example sends ten inference requests to a deployed PyTorch service:

#!/usr/bin/env python

from eas_prediction import PredictClient
from eas_prediction import TorchRequest

if __name__ == '__main__':
    # Replace with your service endpoint and service name
    client = PredictClient('http://182848887922****.cn-shanghai.pai-eas.aliyuncs.com', 'pytorch_gpu_wl')
    client.init()

    req = TorchRequest()
    req.add_feed(0, [1, 3, 224, 224], TorchRequest.DT_FLOAT, [1] * 150528)
    # req.add_fetch(0)
    for x in range(0, 10):
        resp = client.predict(req)
        print(resp.get_tensor_shape(0))

Key SDK methods:

For authentication setup and advanced request patterns, see Use the Python SDK.

Request format

The EAS PyTorch processor uses the Protobuf format for input and output. When using the SDK, requests are serialized automatically — you only need to call add_feed and add_fetch as shown above.

To build raw Protobuf requests without the SDK, generate client code from the following .proto definition. For construction guidance, see Construct a request for a TensorFlow service.

syntax = "proto3";

package pytorch.eas;
option cc_enable_arenas = true;

enum ArrayDataType {
  // Not a legal value for DataType. Used to indicate a DataType field
  // has not been set
  DT_INVALID = 0;

  // Data types that all computation devices are expected to be
  // capable to support
  DT_FLOAT = 1;
  DT_DOUBLE = 2;
  DT_INT32 = 3;
  DT_UINT8 = 4;
  DT_INT16 = 5;
  DT_INT8 = 6;
  DT_STRING = 7;
  DT_COMPLEX64 = 8;  // Single-precision complex
  DT_INT64 = 9;
  DT_BOOL = 10;
  DT_QINT8 = 11;     // Quantized int8
  DT_QUINT8 = 12;    // Quantized uint8
  DT_QINT32 = 13;    // Quantized int32
  DT_BFLOAT16 = 14;  // Float32 truncated to 16 bits.  Only for cast ops
  DT_QINT16 = 15;    // Quantized int16
  DT_QUINT16 = 16;   // Quantized uint16
  DT_UINT16 = 17;
  DT_COMPLEX128 = 18;  // Double-precision complex
  DT_HALF = 19;
  DT_RESOURCE = 20;
  DT_VARIANT = 21;  // Arbitrary C++ data types
}

// Dimensions of an array
message ArrayShape {
  repeated int64 dim = 1 [packed = true];
}

// Protocol buffer representing an array
message ArrayProto {
  // Data Type
  ArrayDataType dtype = 1;

  // Shape of the array.
  ArrayShape array_shape = 2;

  // DT_FLOAT
  repeated float float_val = 3 [packed = true];

  // DT_DOUBLE
  repeated double double_val = 4 [packed = true];

  // DT_INT32, DT_INT16, DT_INT8, DT_UINT8.
  repeated int32 int_val = 5 [packed = true];

  // DT_STRING
  repeated bytes string_val = 6;

  // DT_INT64.
  repeated int64 int64_val = 7 [packed = true];

}


message PredictRequest {

  // Input tensors.
  repeated ArrayProto inputs = 1;

  // Output filter.
  repeated int32 output_filter = 2;
}

// Response for PredictRequest on successful run.
message PredictResponse {
  // Output tensors.
  repeated ArrayProto outputs = 1;
}

What's next

• Use the Python SDK — authentication, connection options, and advanced request patterns

• Service deployment: EASCMD & DSW — full eascmd deployment reference

• Construct a request for a TensorFlow service — building raw Protobuf requests