Platform For AI:Menyebarkan layanan model menggunakan gambar TensorFlow Serving

Prasyarat

model_config_list {
  config {
    name: 'modelA'
    base_path: '/models/modelA/'
    model_platform: 'tensorflow'
    model_version_policy{
        all: {}
    }
  }
  config {
    name: 'modelB'
    base_path: '/models/modelB/'
    model_platform: 'tensorflow'
    model_version_policy{
        specific {
            versions: 1
            versions: 2
        }
    }
    version_labels {
      	key: 'stable'
      	value: 1
    }
    version_labels {
      	key: 'canary'
      	value: 2
    }
  }
  config {
    name: 'modelC'
    base_path: '/models/modelC/'
    model_platform: 'tensorflow'
    model_version_policy{
        latest {
            num_versions: 2
        }
    }
  }
}

Menyebar layanan

Anda dapat menyebar gambar TensorFlow Serving dengan salah satu dari dua cara berikut.

• Penerapan model berbasis skenario: Cocok untuk skenario penerapan dasar. Anda cukup mengonfigurasi beberapa parameter untuk menyebar layanan model TensorFlow Serving dengan sekali klik.

• Penerapan model kustom: Opsi ini memungkinkan Anda menyesuaikan lebih banyak opsi konfigurasi, seperti mengubah port atau mengatur periode polling untuk file model.

Kirim permintaan layanan

Layanan mendukung protokol permintaan HTTP dan gRPC, tergantung pada nomor port yang dikonfigurasi dalam perintah run saat penerapan layanan. Bagian berikut memberikan contoh permintaan layanan untuk modelA.

1. Persiapkan data uji

   modelA adalah model klasifikasi gambar. Set data pelatihan adalah Fashion-MNIST, dan sampelnya berupa gambar grayscale berukuran 28x28. Output model adalah probabilitas bahwa suatu sampel termasuk ke salah satu dari 10 kelas. Untuk tujuan pengujian, kami menggunakan [[[[1.0]] * 28] * 28] sebagai data uji untuk permintaan layanan modelA.

2. Contoh permintaan

   Permintaan HTTP

   Jika nomor port diatur ke 8501, layanan mendukung permintaan HTTP. Tabel berikut merangkum jalur permintaan HTTP untuk penerapan model tunggal dan multi-model:

   Model tunggal

   Multi-model

   Format jalur: <service_url>/v1/models/<model_name>:predict Di mana: Penerapan berbasis skenario: <model_name> tidak dapat dikustomisasi dan default-nya adalah model. Penerapan kustom: <model_name> adalah nama model yang dikonfigurasi dalam Command to Run. Jika tidak ada nama model yang dikonfigurasi, default-nya adalah model.

   Permintaan tanpa menentukan versi maupun permintaan yang menentukan versi model didukung. Format jalurnya sebagai berikut: Tidak menentukan versi (memuat versi terbaru secara default): <service_url>/v1/models/<model_name>:predict Menentukan versi model: <service_url>/v1/models/<model_name>/versions/<version_num>:predict Jika version_labels diatur: <service_url>/v1/models/<model_name>/labels/<version label>:predict Dengan, <model_name> adalah nama model yang dikonfigurasi dalam file konfigurasi model.

   Parameter <service_url> adalah titik akhir layanan yang telah Anda sebar. Untuk menemukan titik akhir tersebut, buka halaman Elastic Algorithm Service (EAS) dan klik Invocation Information di kolom Service Type layanan target. Saat Anda melakukan debugging online di konsol, jalur ini akan diisi secara otomatis.

   Ambil contoh penerapan berbasis skenario untuk model tunggal modelA. Jalur permintaan HTTP-nya adalah <service_url>/v1/models/model:predict.

   Contoh berikut menunjukkan cara mengirim permintaan layanan melalui debugging online di konsol dan menggunakan kode Python:

   Debugging daring

   Setelah layanan diterapkan, klik kolom Actions layanan tersebut, lalu klik Online Debugging. Parameter Request Parameter Online Tuning sudah diisi dengan <service_url>. Tambahkan jalur /v1/models/model:predict ke alamat tersebut dan konfigurasikan data permintaan layanan di bagian Body:

   {"signature_name": "serving_default", "instances": [[[[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]], [[1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0], [1.0]]]]}

   Setelah mengonfigurasi parameter, klik Send Request untuk mendapatkan output yang mirip dengan berikut.

   

   Kirim permintaan HTTP menggunakan kode Python

   Berikut ini contoh kode Python:

   from urllib import request
   import json
   
   # Ganti dengan titik akhir layanan dan token Anda.
   # Anda dapat menemukannya di tab Shared Gateway - Internet Endpoint pada halaman Call Info.
   service_url = '<service_url>'
   token = '<token>'
   # Untuk penerapan model tunggal berbasis skenario, nama model adalah 'model'. Untuk kasus lain, lihat tabel deskripsi jalur di atas.
   model_name = "model"
   url = "{}/v1/models/{}:predict".format(service_url, model_name)
   
   # Buat permintaan HTTP.
   req = request.Request(url, method="POST")
   req.add_header('authorization', token)
   data = {
       'signature_name': 'serving_default',
       'instances': [[[[1.0]] * 28] * 28]
   }
   
   # Panggil layanan.
   response = request.urlopen(req, data=json.dumps(data).encode('utf-8')).read()
   
   # Lihat hasilnya.
   response = json.loads(response)
   print(response)

   Permintaan gRPC

   Jika nomor port diatur ke 8500 dan konfigurasi terkait gRPC ditambahkan, layanan mendukung permintaan gRPC. Berikut ini contoh kode Python:

   import grpc
   import tensorflow as tf
   from tensorflow_serving.apis import predict_pb2
   from tensorflow_serving.apis import prediction_service_pb2_grpc
   from tensorflow.core.framework import tensor_shape_pb2
   
   # Titik akhir layanan. Untuk formatnya, lihat deskripsi parameter host di bawah.
   host = "tf-serving-multi-grpc-test.166233998075****.cn-hangzhou.pai-eas.aliyuncs.com:80"
   # Ganti <token> dengan token layanan. Anda dapat menemukan token di tab Shared Gateway - Internet Endpoint.
   token = "<token>"
   
   # Nama model. Lihat deskripsi parameter name di bawah.
   name = "<model_name>"
   signature_name = "serving_default"
   # Atur ke nomor versi model. Anda hanya dapat mengirim permintaan ke satu versi model dalam satu waktu.
   version = "<version_num>"
   
   # Buat permintaan gRPC.
   request = predict_pb2.PredictRequest()
   request.model_spec.name = name
   request.model_spec.signature_name = signature_name
   request.model_spec.version.value = version
   request.inputs["keras_tensor"].CopyFrom(tf.make_tensor_proto([[[[1.0]] * 28] * 28]))
   
   # Panggil layanan.
   channel = grpc.insecure_channel(host)
   stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
   metadata = (("authorization", token),)
   response, _ = stub.Predict.with_call(request, metadata=metadata)
   
   print(response)

   Parameter utama dikonfigurasi sebagai berikut:

   Parameter	Deskripsi
   host	Anda harus mengonfigurasinya sebagai titik akhir layanan. Titik akhir harus menghilangkan http:// dan diakhiri dengan :80. Anda dapat menemukan titik akhir di halaman Elastic Algorithm Service (EAS) dengan mengklik Invocation Information di kolom Service Type untuk layanan yang ingin Anda panggil.
   name	Kirim permintaan gRPC untuk satu model: Penerapan berbasis skenario: Atur ke model. Penerapan kustom: Nama model dikonfigurasi sebagai nama model yang ditentukan dalam konfigurasi Command to Run. Jika tidak ada nama model yang ditentukan dalam Command to Run, default-nya adalah model. Kirim permintaan gRPC untuk beberapa model: Atur ke nama model yang dikonfigurasi dalam file konfigurasi model.
   version	Atur ke nomor versi model. Anda hanya dapat mengirim permintaan ke satu versi model dalam satu waktu.
   metadata	Token layanan. Anda dapat melihat token di halaman Invocation Information.

Referensi

• Untuk mempelajari cara menyebar layanan Elastic Algorithm Service (EAS) berdasarkan mesin Triton Inference Server, lihat Menyebar layanan menggunakan gambar Triton Inference Server.

• Anda juga dapat mengembangkan gambar kustom dan menggunakannya untuk menyebar layanan EAS. Untuk informasi selengkapnya, lihat Gambar kustom.