Triton Inference Server model registration and inference in Flink SQL - Realtime Compute for Apache Flink

Background

Realtime Compute for Apache Flink (VVR 11.7+) supports registering models on NVIDIA Triton Inference Server with the CREATE MODEL statement. Use ML_PREDICT in SQL jobs to run inference on real-time data streams.

NVIDIA Triton Inference Server is a high-performance, open-source inference service from NVIDIA that supports TensorFlow, PyTorch, ONNX, TensorRT, and other frameworks. You can use a self-managed Triton service or deploy one on PAI-EAS. Deploy a service by using a Triton Inference Server image.

Usage notes

This feature is supported only in VVR engine version 11.7 and later.
Requests to an external Triton server use the public internet and require public network access.
If the Triton server is deployed on PAI, requests use a private network. To obtain the endpoint, check the FAQ section.
Triton server resources, network conditions, and model performance affect inference throughput. Overloaded or rate-limited Triton services can cause back pressure in Flink jobs. Severe rate limiting can cause operator timeouts and job restarts.

Syntax

CREATE MODEL [catalog_name.][db_name.]model_name
INPUT (
  input_column input_type
)
OUTPUT (
  output_column output_type
)
WITH (
  'provider' = 'triton',
  'endpoint' = '<endpoint>',
  'auth-token' = '<authentication_token>'
  'model-name' = '<model_name>',
  'model-version' = '<model_version>'
);

WITH parameters

General parameters

Parameter	Description	Type	Required	Default	Notes
provider	The model service type.	String	Yes	None	The value must be `triton`.
endpoint	The HTTP endpoint of the Triton Inference Server.	String	Yes	None	Ensure network connectivity between the Flink workspace and the Triton service. Network connection options. For PAI-EAS deployments, obtain the endpoint from the service invocation information on the model inference service page in the PAI console.
model-name	The name of the model on the Triton server.	String	Yes	None	This name must match the model name in the Triton model repository.
model-version	The version of the Triton model.	String	No	latest	You can specify a version, such as `1`.
timeout	The timeout for an HTTP request.	Duration	No	30s	Applies to connection, read, and write operations. Specify in duration format, such as `10s` or `30000ms`.
flatten-batch-dim	Whether to flatten the batch dimension of an array input.	Boolean	No	false	Default array input shape is `[1, N]`. Set to `true` to flatten to `[N]` when the model expects one-dimensional input.
priority	The request priority.	Integer	No	None	Range: `0` to `255`. Higher values indicate higher priority. Passed through to Triton request parameters.
compression	The compression algorithm for the request body.	String	No	None	Currently, only `gzip` is supported.
auth-token	The authentication token for the Triton model.	String	No	None	Adds the `Authorization: Bearer <auth-token>` header to requests. For PAI-EAS deployments, obtain the token from the service invocation information on the model inference service page in the PAI console.
custom-headers	Custom HTTP request headers.	Map	No	None	Example: 'X-Trace-Id:abc,Authorization:token'.

Stateful model parameters

These parameters apply to stateful Triton models (RNN, LSTM) that maintain state across requests.

Parameter	Description	Type	Required	Default	Notes
sequence-id	The sequence ID.	String	No	None	Triton routes requests with the same sequence ID to the same model instance.
sequence-start	Marks the request as the start of a sequence.	Boolean	No	false	If set to `true`, Triton initializes the model state before processing this request.
sequence-end	Marks the request as the end of a sequence.	Boolean	No	false	If set to `true`, Triton releases the model state after processing this request.

Type mapping

Flink column types must match the data_type in the model's config.pbtxt on the Triton server.

Flink type	Triton dtype	Description
BOOLEAN	BOOL	Boolean type.
TINYINT	INT8	8-bit signed integer.
SMALLINT	INT16	16-bit signed integer.
INT	INT32	32-bit signed integer.
BIGINT	INT64	64-bit signed integer.
FLOAT	FP32	32-bit floating-point number.
DOUBLE	FP64	64-bit floating-point number.
STRING / VARCHAR	BYTES	Text type.
ARRAY<T>	Corresponds to the element type `T`.	Only one-dimensional arrays are supported. `T` must be one of the scalar types listed above.

Shape rules:

The shape of a scalar input is [1].
The default shape of an ARRAY<T> input is [1, N], where N is the length of the array. If the Triton model expects a shape of [N], set 'flatten-batch-dim' = 'true'.

FAQ

How do I obtain the endpoint and token for a Triton service deployed on the Platform for AI (PAI)?

Log on to the Platform for AI (PAI) console.
In the left-side navigation pane, choose Elastic Algorithm Service (EAS) > Inference Service, and then click the name of the target service to open the Overview page.
In the Basic Information section, click View Invocation Information.
In the Invocation Information panel, copy the endpoint and the token.

How do I resolve a shape mismatch error?

Ensure the Flink input column type matches the dims in the model's config.pbtxt on the Triton server. For ARRAY<T> inputs, Flink sends shape [1, N] by default. If the model expects [N], set:

'flatten-batch-dim' = 'true'

Nested arrays are not supported. For high-dimensional tensors, flatten to a one-dimensional ARRAY<T> and restore the shape on the model side.

Are models with multiple inputs or outputs supported?

Only single input and single output columns are supported. For multiple inputs, pack numerical features into one ARRAY<T> or serialize complex structures as a JSON string and parse it on the model side. For multiple outputs, merge them into one output tensor or JSON string on the model side.