Platform For AI:Warm up model services

Use the SDK for Python

#!/usr/bin/env python

from eas_prediction import TFRequest

# Build a warm-up request that matches your production request exactly.
# The inputs and outputs must be identical to those used after the service goes live.
req = TFRequest('serving_default')
req.add_feed('sentence1', [200, 15], TFRequest.DT_INT32, [1] * 200 * 15)
req.add_feed('sentence2', [200, 15], TFRequest.DT_INT32, [1] * 200 * 15)
req.add_feed('y', [200, 2], TFRequest.DT_INT32, [2] * 200 * 2)
req.add_feed('keep_rate', [], TFRequest.DT_FLOAT, [0.2])
req.add_feed('images', [1, 784], TFRequest.DT_FLOAT, [1] * 784)
req.add_fetch('sorted_labels')
req.add_fetch('sorted_probs')

# Save the warm-up request to a binary file.
with open("warm_up.bin", "wb") as fw:
    fw.write(req.to_string())

Use the SDK for Java

Add the eas-sdk dependency to your pom.xml:

<dependency>
  <groupId>com.aliyun.openservices.eas</groupId>
  <artifactId>eas-sdk</artifactId>
  <version>2.0.13</version>
</dependency>

Check the Maven repository for the latest version.

import java.io.File;
import com.aliyun.openservices.eas.predict.request.TFDataType;
import com.aliyun.openservices.eas.predict.request.TFRequest;
import org.apache.commons.io.FileUtils;

public class TestTf {

    public static void main(String[] args) throws Exception {
        // Build a warm-up request that matches your production request exactly.
        TFRequest request = new TFRequest();
        request.setSignatureName("predict_images");

        float[] content = new float[784];
        for (int i = 0; i < content.length; i++) {
            content[i] = (float) 0.0;
        }
        request.addFeed("images", TFDataType.DT_FLOAT, new long[]{1, 784}, content);
        request.addFetch("scores");

        // Write the request to a binary file.
        File writename = new File("/path/to/warm_up1.bin");
        FileUtils.writeByteArrayToFile(writename, request.getRequest().toByteArray());
    }
}

Replace /path/to/warm_up1.bin with the actual path where you want to save the file.

Send a test request with curl

curl --data-binary @"</path/to/warmup.bin>" \
  -H 'Authorization: <your-token>' \
  <service-address>

If the response is too large to display in the terminal, add --output <file-path> to save it to a file.

Parse the file directly

Use this method to inspect the request content without a running service.

Python

from eas_prediction import TFRequest

req = TFRequest()
with open('/path/to/warm_up1.bin', 'rb') as wm:
    req.request_data.ParseFromString(wm.read())
    print(req.request_data)

Java

import com.aliyun.openservices.eas.predict.proto.PredictProtos;
import org.apache.commons.io.FileUtils;
import java.io.File;

public class Test {

    public static void main(String[] args) throws Exception {
        File refile = new File("/path/to/warm_up1.bin");
        byte[] data = FileUtils.readFileToByteArray(refile);
        PredictProtos.PredictRequest pb = PredictProtos.PredictRequest.parseFrom(data);
        System.out.println(pb);
    }
}

Why does my TensorFlow model still reload files after warm-up?

Issue

In real business scenarios, updating TensorFlow models can lead to service instability. Even with the warm-up feature added to the Processor, this issue may persist. Every distinct input-output signature can cause the model to reload files for warming up.

Cause

This issue arises because the TensorFlow function session->Run(inputs, output_tensor_names, {}, &outputs) performs hash validation on inputs and output_tensor_names. If there are changes in the inputs or outputs, it triggers a reload.

For example, consider a TensorFlow model with the following inputs and outputs:

Inputs:
  threshold: []; DT_FLOAT
  model_id: []; DT_STRING
  input_holder: [-1]; DT_STRING

Outputs:
  model_version_id: []; DT_STRING
  sorted_labels: [-1, 3]; DT_STRING
  sorted_probs: [-1, 3]; DT_FLOAT

If the warm-up request fetches only sorted_labels and sorted_probs:

request.addFeed("input_holder", TFDataType.DT_STRING, new long[]{1}, input);
request.addFeed("threshold", TFDataType.DT_FLOAT, new long[] {}, th);
request.addFeed("model_id", TFDataType.DT_STRING, new long[]{}, model_name);

request.addFetch("sorted_labels");
request.addFetch("sorted_probs");

But production requests also fetch model_version_id:

request.addFeed("input_holder", TFDataType.DT_STRING, new long[]{1}, input);
request.addFeed("threshold", TFDataType.DT_FLOAT, new long[] {}, th);
request.addFeed("model_id", TFDataType.DT_STRING, new long[]{}, model_name);

request.addFetch("sorted_labels");
request.addFetch("sorted_probs");
request.addFetch("model_version_id"); // An additional output is fetched.

That extra output changes the hash, causing a reload even after warm-up.

Solution

Each service must use real business requests for warming up. Upload a warm-up file built from an actual production request, with the exact same inputs and outputs. It is sufficient for session->Run to successfully execute once with a matching request — you only need one warm-up file.