Multimodal embedding API - OpenSearch - Alibaba Cloud Documentation Center

A multimodal embedding service based on Qwen2-VL that converts text, images, and combined inputs into semantic vectors.

Model ID (service_id)	Dimension	Service description	QPS limit
ops-m2-encoder	768 dimensions	Bilingual (Chinese-English) multimodal service trained on BM-6B with 6 billion image-text pairs (3 billion Chinese, 3 billion English). Supports cross-modal retrieval (text-to-image, image-to-text) and image classification. Note This model does not accept both text and an image in the same input object.	10 Note To request a higher API QPS limit, submit a ticket to technical support.
ops-m2-encoder-large	1024 dimensions	Bilingual (Chinese-English) multimodal service with 1 billion parameters. Offers stronger performance than `ops-m2-encoder` for multimodal tasks. Note This model does not accept both text and an image in the same input object.
ops-gme-qwen2-vl-2b-instruct	1536 dimensions	Multimodal embedding service based on Qwen2-VL MLLMs. Supports single-modal and multimodal inputs for text, image, and combined data.
ops-mm-embedding-v1-2b	1536 dimensions	Multimodal embedding model developed by the Alibaba Cloud AI Search Open Platform team. Fine-tuned on Qwen2-VL 2B Instruct, it supports text, image, and video inputs, encoding them into unified semantic vectors for cross-modal retrieval and understanding tasks.
ops-mm-embedding-v1-7b	3584 dimensions	Multimodal embedding model developed by the Alibaba Cloud AI Search Open Platform team. Fine-tuned on Qwen2-VL 7B Instruct, it supports text, image, and video inputs, encoding them into unified semantic vectors for cross-modal retrieval and understanding tasks.
ops-mm-embedding-face-001	512 dimensions	Face retrieval model that encodes single or multiple image inputs into high-dimensional semantic vectors for face search and comparison.

Prerequisites

Get authentication credentials

The AI Search open platform requires an API key for authentication. For instructions, see Get an API key.
Get the service endpoint

You can call the service via the public network or a VPC. For details, see Get the service endpoint.

Request

Request body limit

The request body cannot exceed 8 MB.

Request method

POST

URL

{host}/v3/openapi/workspaces/{workspace_name}/multi-modal-embedding/{service_id}

host: The service endpoint, accessible over the public network or VPC. Get service endpoints.

In the left navigation pane, select your workspace (for example, default) and click API Keys to find the public API endpoint and private API endpoint.
workspace_name: Workspace name, such as default.
service_id: Model ID, such as ops-m2-encoder.

Request parameters

Header parameters

API key authentication

Parameter	Type	Required	Description	Example
Content-Type	String	Yes	Set to `application/json`.	application/json
Authorization	String	Yes	The API key for authentication.	Bearer OS-d1**2a

Body parameters

Parameter	Type	Required	Description	Example
input	List[ContentObject]	Yes	Content to convert into embeddings. Up to 32 objects per request.	`[ { "text":"Science and technology are the primary productive forces" }, { "image":"http://***/a.jpg" } ]`

ContentObject

Parameter

Type

Required

Description

Example

text

String

Text to convert into an embedding.

{
  "text":"Text input"
}

image

String

Image to convert into an embedding, provided as a URL or Base64-encoded string.

URLs must be publicly accessible.
For Base64 input, pass data to the image parameter in the format data:image/{format};base64,{base64_image}, where:

image/{format}: Image format, such as image/jpeg.

base64_image: The Base64-encoded image data.

{
  "image":"http://xxxxx/a.jpg"
}

{
  "image":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wCEAAoHCB..."
}

Response parameters

Parameter	Type	Description	Example
result.embeddings	List	Array of embedding objects, each corresponding to an item in the `input` array.	`[ { "index": 0, "embedding": [0.003143,0.009750,...,-0.017395] }, {} ]`
result.embeddings[].index	Int	The index of the corresponding input object.	0
result.embeddings[].embedding	List[Double]	The resulting embedding vector.	[0.003143,0.009750,...,-0.017395]

cURL request example

curl -X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer YOUR_API_KEY" \
"http://****-hangzhou.opensearch.aliyuncs.com/v3/openapi/workspaces/default/multi-modal-embedding/ops-m2-encoder" \
-d '{
"input":[
  {
    "image":"http://***/a.jpg"
  }
]
}'

Response examples

Sample success response

{
    "request_id": "B4AB89C8-B135-****-A6F8-2BAB801A2CE4",
    "latency": 38,
    "usage": {
        "image":1,
        "token_count":28
    },
    "result": {
        "embeddings": [
            {
                "index": 0,
                "embedding": [
                   -0.033447265625,
                   0.10577392578125,
                   -0.0015211105346679688,
                   -0.044189453125,
                    ...
                   0.004688262939453125,
                   -4.5239925384521484E-5
                ]
            }
        ]
    }
}

Sample error response

Error responses include a code and message indicating the cause.

{
    "request_id": "651B3087-8A07-****-B931-9C4E7B60F52D",
    "latency": 0,
    "code": "InvalidParameter",
    "message": "JSON parse error: Cannot deserialize value of type `InputType` from String \"xxx\""
}

Status codes

All status codes are documented in Status codes.