This topic describes how to use the machine learning function CLUSTER_SERVING to implement online inference. Tensorflow SavedModel with the format of input_shape=(?,128) is used in this topic.

Limits

This topic applies only to Ververica Platform (VVP) 2.4.2 and later.

Background information

Analytics Zoo is an open source big data analytics and artificial intelligence (AI) platform developed by Intel®. Analytics Zoo supports machine learning training and inference of the distributed TensorFlow and PyTorch models. It improves the performance of deep learning workloads by using the Intel® OpenVINO tool suite and Deep Learning Boost instruction set. Cluster Serving is a distributed inference solution of Analytics Zoo. It can be deployed on Apache Flink clusters for distributed computing. Cluster Serving can implement end-to-end distributed machine learning inference without the need to manually write machine learning code. For more information about Analytics Zoo, see Analytics Zoo.

Cluster Serving integrates some of its features into the built-in functions of Flink and supports various inference models, such as TensorFlow, PyTorch, and OpenVINO. You can directly call the CLUSTER_SERVING function in the console of fully managed Flink to perform distributed machine learning inference. When you use the CLUSTER_SERVING function, you need only to provide model files and data files to complete the end-to-end application of machine learning.

Syntax

CLUSTER_SERVING(uri, `data`)

Input parameters

Parameter Data type Description
uri String The data ID.
data String The returned data.

Example

  • Test data
    Table 1. input.csv
    uri(String) data(String)
    my-data 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
  • Test procedure
    1. Log on to the Object Storage Service (OSS) console and upload the test data file input.csv to the oss://***/ directory.
    2. Log on to the console of fully managed Flink. Create a source table and a result table, load the specified model, and then insert the online inference data of the CLUSTER_SERVING function into the result table.
      CREATE TEMPORARY TABLE ClusterServingInput(
         uri STRING,
        `data` STRING
      )  WITH  (
         'connector' = 'filesystem',
         'path' = 'oss://alink-vvp/artifacts/namespaces/test-module-default/input.csv',
         'format' = 'csv'
      );
      
      CREATE TEMPORARY TABLE ClusterServingOutput (
         uri1 STRING,  
         data1 STRING
       ) WITH (
         'connector' = 'print'
      );
      
      LOAD MODULE `cluster-serving`;
      INSERT INTO ClusterServingOutput SELECT cluster_serving(uri, `data`) FROM ClusterServingInput;
    3. Add the following configuration to the Additional Configuration section on the Advanced tab of the job and click Save. In addition, turn on Use for SQL Editor previews on the Session Clusters page, add the following configuration to the Additional Configuration section, and then click Save.
      pipeline.global-job-parameters: '"modelPath:""[Directory of your model file]"""'

      modelPath indicates the OSS directory in which your model file is stored, such as oss://***/tf_auto/.

    4. On the Deployments page, find the required job and click Start in the Actions column.
  • Test result
    uri1 data1
    0.03932024,0.017788976,-2.2538425E-4,0.021853039,0.030905273,0.052324444,0.053021688,0.08776946,0.0625467,0.12149326,0.048770625,0.04533424,0.09271586,0.023119276,0.0037812046,0.052707616,0.029312208,0.023519568,0.07501668,0.037939064,0.06363185,0.04994791,0.08504477,0.06940596,0.052632406,0.028651983,0.02205597,0.05861686,0.040175162,0.04682718,0.05283319,0.01967535,0.111778766,0.03182444,0.04509241,0.09942829,0.009491239,0.038903005,0.11488753,0.019503327,0.0346709,0.060727723,0.03718308,0.1295206,0.04292037,0.13272661,0.009957914,0.050370887,0.03016096,0.067269124,0.09981682,0.08606887,0.085193545,0.041775055,0.02956512,0.027142545,0.09836763,0.029330501,0.040107135,0.0753366,0.068517864,0.020111244,0.086271614,0.04449262,0.107297346,0.07932708,0.036529146,0.014892701,0.08245985,0.04272034,0.076421194,0.035283472,0.028554386,0.019374546,0.048012834,0.011391987,0.028220229,0.054405782,0.037807927,0.085761145,0.07974813,0.04150249,-0.017132144,0.08219319,0.019887058,0.034193676,0.06221051,0.07516215,0.0598385,0.09159884,0.038117558,0.05329266,0.021038251,0.072046585,0.05561088,0.017538168,0.025691401,0.08580602,0.02225845,0.117060736,0.060743354,0.11840888,0.065381594,0.08059649,0.045638822,0.06399781,0.011646094,0.043574452,0.123603456,0.045708194,0.05672055,0.08402837,0.118181005,0.09657015,0.08259893,0.0010145458,0.031954847,0.07388758,0.035380267,0.070080444,0.06764534,0.058731187,0.022044828,0.094359346,0.05559277,0.032255664,0.07360537,0.11992536 128