Develop C++ custom processors for EAS Model Serving - Platform For AI - Alibaba Cloud - Platform For AI

Quick start demo

Download the pai-prediction-example project. The project contains two custom processors:

echo: Requests when the user input is returned as-is, along with the file list in the model.
image_classification: mnist text classification, inputs mnist jpg images, returns image classification categories.

Compilation methods are described in the project's README file. Each processor's local debugging method is described in the README file in its respective directory.

Interface definition

A custom processor must export exactly two C functions: initialize() and process(). Both are required — EAS calls them at different points in the service lifecycle.

Parameter	Direction	Description
`model_entry`	Input	Corresponds to the `model_entry` field in the deployment configuration file, which indicates the entry file of the model package. For more information about this field, see JSON deployment. You can pass in a file name (for example, `randomforest.pmml`) or a directory (for example, `./model`).
`model_config`	Input	Corresponds to the `model_config` field in the deployment configuration file, which indicates the custom model configuration information. For more information about this field, see JSON deployment.
`state`	Output	Model loading status. Set to `0` on success; any other value indicates a loading failure.
Return value		Pointer to the loaded model in memory. EAS passes this pointer to every subsequent `process()` call as `model_buf`.

Parameter	Direction	Description
`model_buf`	Input	Model memory address returned by `initialize()`.
`input_data`	Input	Request payload. Can be a string or binary data.
`input_size`	Input	Length of the input data.
`output_data`	Output	Data returned by the processor. You must allocate the memory on the heap. The model is responsible for releasing the memory.
`output_size`	Output	Length of the output data.
Return value		HTTP status code for the response. Return `0` or `200` for success.

Return values that are not recognized HTTP status codes are automatically converted to HTTP 400. Always use a standard HTTP status code (for example, 400 or 500) when returning an error.

Usage example

The following is a simple usage example. The online prediction code does not load any model data and returns the user input to the client as-is.

Write the prediction code.

#include <stdio.h>
#include <string.h>
extern "C" {
    void *initialize(const char *model_entry, const char *model_config, int *state)
    {
       *state = 0;
        return NULL;
    }
    int process(void *model_buf, const void *input_data, int input_size,
            void **output_data, int *output_size)
    {
        if (input_size == 0) {
            const char *errmsg = "input data should not be empty";
            *output_data = strdup(errmsg);
            *output_size = strlen(errmsg);
            return 400;
        }  
        *output_data = strdup((char *)input_data);
        *output_size = input_size;
        return 200;
    }
}

Compile the code into a .so file by using the following Makefile.

CC=g++
CCFLAGS=-I./ -D_GNU_SOURCE -Wall -g -fPIC
LDFLAGS= -shared -Wl,-rpath=./
OBJS=processor.o
TARGET=libpredictor.so
all: $(TARGET)
$(TARGET): $(OBJS)
	$(CC) -o $(TARGET) $(OBJS) $(LDFLAGS) -L./
%.o: %.cc
	$(CC) $(CCFLAGS) -c $< -o $@
clean:
	rm -f $(TARGET) $(OBJS)

Deploy the service on EAS. The JSON deployment configuration must include the following processor-related fields. processor_entry specifies the main processor file, which is the .so file produced in the previous step.

{
    "name": "test_echo",
    "model_path": "http://*****.oss-cn-shanghai.aliyuncs.com/****/saved_model.tar.gz",
    "processor_path": "oss://path/to/echo_processor_release.tar.gz",
    "processor_entry": "libpredictor.so",
    "processor_type": "cpp",
    "metadata": {
        "instance": 1
    }
}