This tutorial shows how to use PAI-Blade to accelerate a TensorFlow ResNet50 frozen graph for GPU inference. You will install PAI-Blade, run TensorRT-based optimization for both static and dynamic input shapes, benchmark the result, and deploy the optimized model.
Prerequisites
Before you begin, make sure you have:
Linux with Python 3.6 or later and Compute Unified Device Architecture (CUDA) 10.0
TensorFlow 1.15
PAI-Blade 3.17.0 or later
How optimization works
Set up environment → Load model and data → Run blade.optimize → Review report → Benchmark → DeployPAI-Blade applies a one-stop optimization through the blade.optimize API. For ResNet50, the primary optimization is TensorRT (via the Tf2TrtPlus pass), which replaces TensorFlow subgraphs with fused TensorRT engines. After optimization, load the model with import blade.runtime.tensorflow—no other changes to your inference code are required.
Step 1: Set up the environment
Install the PAI-Blade wheel package for TensorFlow 1.15.0 and CUDA 10.0.
pip3 install pai_blade_gpu==3.17.0 -f https://pai-blade.oss-cn-zhangjiakou.aliyuncs.com/release/repo.htmlDownload the TensorFlow ResNet50 model and test data.
wget http://pai-blade.cn-hangzhou.oss.aliyun-inc.com/tutorials/tf_resnet50_v1.5.tar.gzExtract the archive. The archive contains a
frozen.pbfrozen graph for the ResNet50 model and test data for different batch sizes.tar zxvf tf_resnet50_v1.5.tar.gz
Step 2: Optimize the model
Load the model and test data
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
import numpy as np
import time
import tensorflow.compat.v1 as tf
import blade
from blade.model.tf_model import TfModel
def _load_model_and_data():
local_dir = "./tf_resnet50_v1.5/"
model_path = os.path.abspath(os.path.join(local_dir, "frozen.pb"))
data_path = os.path.abspath(os.path.join(local_dir, "test_bc1.npy"))
graph_def = tf.GraphDef()
with open(model_path, 'rb') as f:
graph_def.ParseFromString(f.read())
test_data = np.load(data_path, allow_pickle=True, encoding='bytes').item()
return graph_def, test_data
graph_def, test_data = _load_model_and_data()
print(test_data)Choose an optimization mode
TensorRT optimization supports two modes based on whether input shapes are fixed or variable at inference time.
| Mode | Use when | Behavior when shape is out of range |
|---|---|---|
| Static shape | Input shape is always the same (for example, you restrict all requests to a fixed batch size to minimize latency) | Falls back to the original TensorFlow graph |
| Dynamic shape | Input shape varies across requests (for example, your service supports dynamic batching) | Falls back to the original TensorFlow subgraph if outside the min–max range |
Static shape optimization
Use this mode when all inference requests use the same input shape.
config = blade.Config()
config.gpu_config.aicompiler.enable = False
config.gpu_config.disable_fp16_accuracy_check = True
config.gpu_config.tensorrt.enable = True # TensorRT is enabled by default; set to False to disable.
optimized_model_static, opt_spec_static, report = blade.optimize(
graph_def, # The original model as a TF GraphDef.
'o1', # Optimization level: o1 or o2.
device_type='gpu', # Target device for the optimized model.
config=config,
outputs=['softmax_tensor'], # Output node names. Blade infers them if not provided.
test_data=[test_data]
)
print(report)A sample optimization report:
{
"software_context": [
{"software": "tensorflow", "version": "1.15.0"},
{"software": "cuda", "version": "10.0.0"}
],
"hardware_context": {"device_type": "gpu", "microarchitecture": "T4"},
"user_config": "",
"diagnosis": {
"model": "tmp_graph.pbtxt",
"test_data_source": "user provided",
"shape_variation": "dynamic",
"message": "",
"test_data_info": "input_tensor:0 shape: (1, 224, 224, 3) data type: float32"
},
"optimizations": [
{"name": "Tf2TrtPlus", "status": "effective", "speedup": "3.37", "pre_run": "6.81 ms", "post_run": "2.02 ms"},
{"name": "TfStripUnusedNodes", "status": "effective", "speedup": "na", "pre_run": "na", "post_run": "na"},
{"name": "TfFoldConstants", "status": "effective", "speedup": "na", "pre_run": "na", "post_run": "na"}
],
"overall": {"baseline": "6.98 ms", "optimized": "2.11 ms", "speedup": "3.31"},
"model_info": {"input_format": "frozen_pb"},
"compatibility_list": [{"device_type": "gpu", "microarchitecture": "T4"}],
"model_sdk": {}
}Reading the report:
| Field | What it tells you |
|---|---|
overall.baseline / overall.optimized | End-to-end latency before and after optimization |
overall.speedup | Ratio of baseline to optimized latency |
optimizations[].name = Tf2TrtPlus | TensorRT fusion pass. When status is effective, TensorRT successfully replaced TensorFlow subgraphs. |
optimizations[].name = TfStripUnusedNodes / TfFoldConstants | Graph-cleaning passes; they do not have standalone latency measurements. |
In static shape mode, Blade uses the shape of test_data for optimization. If the input shape at inference time differs, execution falls back to the original TensorFlow graph, which significantly reduces performance.The results above are for this example only. Your actual results may vary. For a full description of report fields, see Optimization report.
Dynamic shape optimization
Use this mode when your service supports dynamic batching and the batch size varies across requests.
To enable dynamic shape optimization, add a dynamic_tuning_shapes configuration to TensorRTConfig. The min and max keys define the allowed input range; opts lists specific shapes that Blade tunes for static-shape precision within that range.
config_dynamic = blade.Config()
config_dynamic.gpu_config.aicompiler.enable = False
config_dynamic.gpu_config.disable_fp16_accuracy_check = True
config_dynamic.gpu_config.tensorrt.enable = True
config_dynamic.gpu_config.tensorrt.dynamic_tuning_shapes = {
"min": [1, 224, 224, 3],
"opts": [
[1, 224, 224, 3],
[2, 224, 224, 3],
[4, 224, 224, 3],
[8, 224, 224, 3],
],
"max": [8, 224, 224, 3],
}
optimized_model_dynamic, opt_spec_dynamic, report = blade.optimize(
graph_def,
'o1',
device_type='gpu',
config=config_dynamic,
outputs=['softmax_tensor'],
test_data=[test_data]
)
print(report)
with tf.gfile.FastGFile('optimized_model_dynamic.pb', mode='wb') as f:
f.write(optimized_model_dynamic.SerializeToString())A sample optimization report:
{
"software_context": [
{"software": "tensorflow", "version": "1.15.0"},
{"software": "cuda", "version": "10.0.0"}
],
"hardware_context": {"device_type": "gpu", "microarchitecture": "T4"},
"user_config": "",
"diagnosis": {
"model": "tmp_graph.pbtxt",
"test_data_source": "user provided",
"shape_variation": "dynamic",
"message": "",
"test_data_info": "input_tensor:0 shape: (1, 224, 224, 3) data type: float32"
},
"optimizations": [
{"name": "Tf2TrtPlus", "status": "effective", "speedup": "3.96", "pre_run": "7.98 ms", "post_run": "2.02 ms"},
{"name": "TfStripUnusedNodes", "status": "effective", "speedup": "na", "pre_run": "na", "post_run": "na"},
{"name": "TfFoldConstants", "status": "effective", "speedup": "na", "pre_run": "na", "post_run": "na"}
],
"overall": {"baseline": "7.87 ms", "optimized": "2.52 ms", "speedup": "3.12"},
"model_info": {"input_format": "frozen_pb"},
"compatibility_list": [{"device_type": "gpu", "microarchitecture": "T4"}],
"model_sdk": {}
}TensorRT optimization applies as long as the input shape at inference falls within the min–max range. Shapes outside that range fall back to the original TensorFlow subgraph.
The results above are for this example only. Your actual results may vary. For a full description of report fields, see Optimization report.
Step 3: Benchmark the optimized model
Run the following script to confirm that the optimized latency matches the overall.optimized value in the report. The script runs 100 warmup iterations before measuring to stabilize GPU state.
import time
with tf.Session(config=TfModel.new_session_config()) as sess, opt_spec_dynamic:
sess.graph.as_default()
tf.import_graph_def(optimized_model_dynamic, name="")
# Warmup
for i in range(0, 100):
sess.run(['softmax_tensor:0'], test_data)
# Benchmark
num_runs = 1000
start = time.time()
for i in range(0, num_runs):
sess.run(['softmax_tensor:0'], test_data)
elapsed = time.time() - start
rt_ms = elapsed / num_runs * 1000.0
print("Latency of optimized model: {:.2f}".format(rt_ms))Expected output:
Latency of optimized model: 2.26The measured latency of 2.26 ms closely matches the "optimized": "2.52 ms" value in the report's overall section. The small difference is normal—the optimization is working because the test input shape falls within the dynamic shape range.
The results above are for this example only. Your actual results may vary.
Step 4: Deploy the optimized model
Blade provides Python and C++ runtime software development kits (SDKs). This section covers the Python SDK. For the C++ SDK, see Use an SDK to deploy a TensorFlow model for inference.
(Optional) During the trial period, set the following environment variable to prevent unexpected exits caused by authentication timeouts.
export BLADE_AUTH_USE_COUNTING=1Authenticate with PAI-Blade.
export BLADE_REGION=<region> export BLADE_TOKEN=<token>Replace the following placeholders:
Placeholder Description <region>The region where you use PAI-Blade. Get the list of supported regions from the PAI-Blade DingTalk group. For the group QR code, see Install PAI-Blade. <token>The authentication token for PAI-Blade. Get the token from the PAI-Blade DingTalk group. For the group QR code, see Install PAI-Blade. Load and run the optimized model. To integrate Blade, add only
import blade.runtime.tensorflow—no other changes to your existing inference code are needed.import tensorflow.compat.v1 as tf import blade.runtime.tensorflow infer_data = np.load('./tf_resnet50_v1.5/test_bc1.npy', allow_pickle=True, encoding='bytes').item() # Optimized model produced by blade.optimize model_path = './optimized_model_dynamic.pb' graph_def = tf.GraphDef() with open(model_path, 'rb') as f: graph_def.ParseFromString(f.read()) with tf.Session() as sess: sess.graph.as_default() tf.import_graph_def(graph_def, name="") print(sess.run(['softmax_tensor:0'], infer_data))
Appendix: TensorRTConfig
TensorRTConfig controls TensorRT-specific behavior within blade.Config.
class TensorRTConfig():
def __init__(self) -> None:
self.enable = True
self.dynamic_tuning_shapes: Dict[str, List[List[Any]]] = dict()
......| Parameter | Type | Description |
|---|---|---|
enable | bool | Controls whether TensorRT optimization is applied. TensorRT is attempted by default when optimizing on a GPU device. |
dynamic_tuning_shapes | Dict[str, List[List[Any]]] | Defines the input shape range for dynamic shape optimization. Required when your service receives inputs of varying sizes. |
The dynamic_tuning_shapes dictionary has three keys:
| Key | Type | Description |
|---|---|---|
min | List[List[int]] | Lower bound of the dynamic input range for each model input. |
max | List[List[int]] | Upper bound of the dynamic input range for each model input. |
opts | List[List[List[int]]] | Specific shapes to optimize at static-shape precision. All opts values must fall between min and max. |
All shapes in opts must be within the min–max range. If any value falls outside this range, TensorRT optimization fails with Dim value in 'opts' is not between min_dim and max_dim.
Example with two model inputs:
{
"min": [[1, 3, 224, 224], [1, 50]],
"opts": [
[[1, 3, 512, 512], [1, 60]],
[[1, 3, 320, 320], [1, 55]]
],
"max": [[1, 3, 1024, 1024], [1, 70]]
}What's next
Optimization report — understand every field in the output report
Use an SDK to deploy a TensorFlow model for inference — integrate the C++ runtime SDK
Install PAI-Blade — get authentication credentials and supported regions