Sparse text embedding - OpenSearch - Alibaba Cloud Documentation Center

OpenSearch AI Search Open Platform allows you to use the sparse text embedding service by calling APIs. You can use this service to convert text data into sparse vectors. Sparse vectors occupy a smaller storage space and are commonly used to express keywords and word frequency information. They can be combined with dense vectors for hybrid retrieval to improve the retrieval effect.

Service	ID	Description	QPS limit for API calls (Alibaba Cloud account and RAM users)
OpenSearch sparse text embedding	ops-text-sparse-embedding-001	Supported languages: more than 100 languages Maximum input text length: 8,192 tokens	50 Note To apply for higher QPS, submit a ticket.

Prerequisites

The authentication information is obtained.
When you call an AI Search Open Platform service by using an API, you need to authenticate the caller's identity.
The service access address is obtained.
You can call a service over the Internet or a virtual private cloud (VPC). For more information, see Get service registration address.

Usage notes

The request body cannot exceed 8 MB in size.

Request method

POST

URL

{host}/v3/openapi/workspaces/{workspace_name}/text-sparse-embedding/{service_id}

URL parameters

host: the endpoint that is used to call the API operation. You can call the API operation over the Internet or a virtual private cloud (VPC). For more information about how to obtain the endpoint, see Query service endpoint.
workspace_name: the name of the workspace. Example: default.
service_id: the ID of the service that you want to use. Example: ops-text-sparse-embedding-001.

Request parameters

Header parameters

API key authentication

Parameter	Type	Required	Description	Example
Content-Type	String	Yes	The request type. Set the value to application/json.	application/json
Authorization	String	Yes	The API key.	Bearer OS-d1**2a

Body parameters

Parameter	Type	Required	Description	Example
input	Array/String	Yes	The input text entries. Each request can contain up to 32 entries. The length of an entry is determined by the model that you select. Empty strings are not supported.	["Science and technology are the primary productive forces","OpenSearch product documentation"]
input_type	String	No	The data type of the input. Valid values: query document Default value: document.	document
return_token	boolean	No	Specifies whether to return the embeddings. Valid values: true false Default value: false.	false

Response parameters

Parameter	Type	Description	Example
request_id	String	The request ID.	B4AB89C8-B135-****-A6F8-2BAB801A2CE4
latency	Float/Int	The time consumed for the request, in milliseconds.	10
usage	Object	The metering information about the current call.	"usage": { "token_count": 11 }
usage.token_count	Int	The number of tokens.	11
result.sparse_emebddings	List	The output of the algorithm used in the request. The value is a list of arrays. Each array contains the output of the algorithm for an input text entry.	[ { "index": 0, "embedding": [{ "tokenId": 6, "weight": 0.10137939453125 }] }, { "index": 1, "embedding": [{ "tokenId": 9803, "weight": 0.1951904296875 }] } ]
result.sparse_embeddings[].index	Int	The sequence number of the input text entry in the request.	0
result.sparse_embeddings[].embedding	List	The sparse embedding result.	[ { "token":"test", "token_id": 900, "weight":0.423 }]
result.sparse_embeddings[].embedding[].token	String	The text token. This parameter is returned if return_token is set to true.	"xxx"
result.sparse_embeddings[].embedding[].token_id	Int	The ID of the token.	123
result.sparse_embeddings[].embedding[].weight	Float	The weight.	0.121

Sample Curl request

curl -XPOST -H"Content-Type: application/json" 
"http://****-hangzhou.opensearch.aliyuncs.com/v3/openapi/workspaces/default/text-sparse-embedding/ops-text-sparse-embedding-001" 
-H "Authorization: Bearer <Your API key>" 
-d "{
    \"input\": [
          \"Science and technology are the primary productive forces\", 
          "OpenSearch product documentation"
    ], 
    \"input_type\": \"query\", 
    \"return_token\": false
}"

Sample responses

Sample success response

{
	"request_id": "75C50B5B-E79E-4930-****-F48DBB392231",
	"latency": 22,
	"usage": {
		"token_count": 11
	},
	"result": {
		"sparse_embeddings": [
			{
				"index": 0,
				"embedding": [
					{
						"tokenId": 6,
						"weight": 0.10137939453125
					},
					{
						"tokenId": 163040,
						"weight": 0.2841796875
					},
					{
						"tokenId": 354,
						"weight": 0.1431884765625
					},
					{
						"tokenId": 5998,
						"weight": 0.161376953125
					},
					{
						"tokenId": 8550,
						"weight": 0.2388916015625
					},
					{
						"tokenId": 2017,
						"weight": 0.1614990234375
					}
				]
			},
			{
				"index": 1,
				"embedding": [
					{
						"tokenId": 9803,
						"weight": 0.1951904296875
					},
					{
						"tokenId": 86250,
						"weight": 0.317138671875
					},
					{
						"tokenId": 5889,
						"weight": 0.17529296875
					},
					{
						"tokenId": 2564,
						"weight": 0.11614990234375
					},
					{
						"tokenId": 59529,
						"weight": 0.1666259765625
					}
				]
			}
		]
	}
}

Sample error response

If an error occurs for the API request, the corresponding error code and error message are returned by using the code and message parameters.

{
    "request_id": "45C8C9E5-6BCB-****-80D3-E298F788512B",
    "latency": 0,
    "code": "InvalidParameter",
    "message": "JSON parse error: Unexpected character ..."
}

Error codes

For a list of error codes, see Status codes.