Push documents in batches by using the API - OpenSearch - Alibaba Cloud Documentation Center

OpenSearch LLM-Based Conversational Search Edition lets you push documents in batches using an API.

Prerequisites

Obtain an API key to authenticate your API calls. For more information, see Manage API keys.
Obtain a service endpoint, which is required for all API calls. For more information, see Get service endpoint.

Request details

Request method	Request protocol	Request data format
POST	HTTP	JSON

Request URL

{host}/v3/openapi/apps/[app_group_identity]/actions/knowledge-bulk

{host}: The service endpoint. You can call the API over the public network or a VPC. For more information, see Get service endpoint.
{app_group_identity}: The name of your application. To find the name, sign in to the OpenSearch LLM-Based Conversational Search Edition console and locate your instance on the Instance Management page.

Request parameters

Header parameters

Parameter	Type	Required	Description	Example
Content-Type	string	Yes	The format of the request body. This must be 'application/json'.	application/json
Authorization	string	Yes	The API key for request authentication. The value must be prefixed with `Bearer` and a space.	Bearer OS-d1**2a

Body parameters

Parameter	Type	Required	Description	Example
cmd	string	Yes	The action to perform on the document.	ADD/DELETE
fields	map	Yes	The document fields as a key-value map.
fields.id	string	Yes	The primary key of the document.	13579
fields.title	string	No	The document title.	Something interesting
fields.url	string	No	The URL of the document.	https://www.aliyun.com
fields.content	string	Yes	The document content.	No, is not.

Example requests

1. Upload a document to the main table

['{
    "cmd": "ADD",
    "fields": {
      "id": "13579",
      "title": "Something interesting",
      "url": "https://www.aliyun.com",
      "content": "No, is not."
      }
}']

Upload an unstructured document (PDF, DOC, TXT, or HTML)

['{
    "cmd" : "URL/BASE64",
    "fields": {
      "id": "Document ID (optional)",
      "type": "File type, such as pdf, doc, txt, or html",
      "title": "File name (optional)",
      "content": "URL or Base64-encoded data",
      "url": "Link (optional)"
    }
 }']

Note

The url parameter only supports data access URLs for OSS objects generated using the ossClient.generatePresignedUrl method.

2. Upload a document to a custom table

['{
    "cmd" : "ADD",
    "fields": {					# The data object. Define fields based on the table schema.
      "key1": "value1",
      "key2": "value2",
      "key3": "value3"
    },
    "table_name": "custom_table_name"	# The table name. If left empty, the operation is performed on the main table.
  }']

3. Delete a document

['{
    "cmd" : "DELETE",
    "fields": {
      "id": "13579333"
    }
}']

cmd: Required. Specifies the action to perform on the document. Valid values are ADD and DELETE. We recommend batching multiple operations in a single request to improve network and processing efficiency. ADD creates a new document or overwrites an existing document with the same primary key. DELETE removes a document. The operation is considered successful even if the specified document does not exist.
fields: Required. The document's fields. The primary key is always required. For a DELETE action, only the document's primary key is required.
Note: The request body must be a JSON array to support batch operations on multiple documents.

Response parameters

Parameter	Type	Description
errors	list	A list of errors.
status	string	The execution status. `OK` indicates success, and `FAIL` indicates failure. If the request fails, use the returned error code to troubleshoot.
request_id	string	The ID of the request.
result	boolean	Returns `true` on successful execution. This parameter is omitted on failure.
total	int	The total number of documents in the request.
success	int	The number of successfully uploaded documents.
failure	int	The number of documents that failed to upload.
failed_ids	array	The IDs of documents that failed to upload.

Example response

{
  "request_id" : "abc123-ABC",
  "result" : {
    "total": 100,
    "success": 50,
    "failure": 50,
    "failed_ids": [
      "id1",
      "id2",
      "id3",
      "..."
    ]
  }
  "errors" : [
    {
      "code" : "The error code, if an error occurred.",
      "message" : "The error message, if an error occurred."
    }
  ]
}

Notes

Field names are case-insensitive when you push data using the API or an SDK.
Data pushes using the API or an SDK are subject to frequency and size limits. The upload size limit cannot be modified. Limits vary by application. For more information, see Limits.
To prevent data loss, check the response after each upload and retry requests that fail with specific error codes, especially 3007. Data processing is asynchronous. A response of OK only indicates that the system has successfully received the data. Data processing errors are displayed in the console. Check for error messages regularly.
The POST request body cannot exceed 2 MB before encoding. The server rejects requests that exceed this limit.
If the request body of a POST operation contains Chinese characters, it must be UTF-8 encoded. Likewise, the value of the Content-MD5 header must be calculated from the UTF-8 encoded data. Otherwise, the push operation will fail.
If an API call returns the error code 4016, the RAM user has insufficient permissions. Grant the AliyunOpenSearchFullAccess permission to the RAM user. For instructions, see Grant permissions to a RAM user.