OpenSearch:OSS + API data source

The procedure consists of the following two steps:

1. Activate OSS

2. Add an OSS + API data source

---

Step 1: Activate OSS

1. Activate the service

Activate OSS using the same Alibaba Cloud account and in the same region as your purchased OpenSearch service.

On the OSS resource plan purchase page, set Product Type to OSS Resource Plan, Resource Plan Type to Standard - Zone-redundant Storage (ZRS), and Region to Chinese Mainland General. The region must match the region of your purchased OpenSearch service.

2. Create a bucket

Before you upload files to OSS, create a bucket to store the files. For more information, see Create a bucket.

Path: OSS console → Buckets → Create bucket

In the left-side navigation pane of the OSS console, click Buckets, and then click Create Bucket.

Open the bucket that you created and add an OpenSearch tag.

Path: Bucket configuration → Bucket tags → Create tag

Set both the tag key and value to opensearch.

3. Configure the OSS file format

a. OSS file format

To use the uploaded files as a data source for index building, make sure that:

• The files use UTF-8 encoding.

• The data in the files is converted to HA3 or JSON format.

ⅰ. JSON format

The following example contains multiple records:

{"field_double": ["100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300"], "title": "Huawei Mate 9 Kirin 960 processor Leica dual cameras", "color": "Red", "empty_int32": "", "price": "3599", "CMD": "add", "nid": "1", "gather_cn_str": "", "desc": ["str1", "str2", "str3"], "brand": "Huawei", "size": "5.9","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}
{"field_double": ["100.0", "221.123", "500.3333333", "100.0", "221.123", "500.3333333"], "field_int32": ["100", "200", "300", "100", "200", "300"], "title": "Huawei P10 Plus all-network smartphone", "color": "Blue", "empty_int32": "", "price": "4388", "CMD": "add", "nid": "2", "gather_cn_str": "colorBlue", "desc": ["str1", "str2", "str3", "str1", "str2", "str3"], "brand": "Huawei", "size": "5.5","__subdocs__":[{"sub_pk":"100","sub_field1":"200","sub_field2":["100","200","300"]},{"sub_pk":"200","sub_field1":"200","sub_field2":["100","200","300"]}]}

ⅱ. HA3 format

1. The following example shows the complete contents of the standard_sample.data file.

CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_CMD=add^_
PK=12345321^_
url=http://www.aliyun.com/index.html^_
title=Alibaba Cloud Computing Co., Ltd.^_
body=xxxxxx xxx^_
time=3123423421^_
multi_value_field=1234^]324^]342^_
bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
^^
CMD=delete^_
PK=12345321^_

2. File delimiters: The preceding data file contains two types of commands: add and delete. Each command consists of multiple lines, with one key-value pair on each line. Commands are separated by '^^\n', key-value pairs by '^_\n', and multiple values by '^]'. The following table describes the delimiters.

C++ encoding	ASCII Hexadecimal	Description	Display in emacs/vi	Input in emacs	Input in vi
"\x1F\n"	1F0A	Key-value pair delimiter	^_ (followed by a line break)	C-q C-7	C-v C-7
"\x1E\n"	1E0A	Command delimiter	^^ (followed by a line break)	C-q C-6	C-v C-6
"\x1D"	1D	Multi-value delimiter	^]	C-q C-5	C-v C-5
"\x1C"	1C	Section weight indicator	^\	C-q C-4	C-v C-4
"\x1D"	1D	Section delimiter	^]	C-q C-5	C-v C-5
"\x03"	03	Subdocument field delimiter	^C	C-q C-c	C-v C-c

3. Command formats

   Add command format: The add command adds new content to the index. The first line must be CMD=add, followed by the fields of the document. The field order can match the order in the schema. All fields that appear must be defined in the fields section of the schema.

   CMD=add^_
   PK=12345321^_
   url=http://www.aliyun.com/index.html^_
   title=Alibaba Cloud Computing Co., Ltd.^_
   body=xxxxxx xxx^_
   time=3123423421^_
   multi_value_field=1234^]324^]342^_
   bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
   ^^CMD=add^_
   PK=12345321^_
   url=http://www.aliyun.com/index.html^_
   title=Alibaba Cloud Computing Co., Ltd.^_
   body=xxxxxx xxx^_
   time=3123423421^_
   multi_value_field=1234^]324^]342^_
   bidwords=mp3^\price=35.8^Ptime=13867236221^]mp4^\price=32.8^Ptime=13867236221^_
   ^^

   Delete command format: The delete command deletes specified content from the index. The first line must be CMD=delete, followed by the field defined as the primary key in the index schema and the field used for partition hashing. If these are the same field, specify it only once.

   CMD=delete^_
   PK=12345321^_
   ^^CMD=delete^_
   PK=12345321^_
   ^^

b. Upload files to the bucket

Return to the page of the bucket that you created. In the left-side navigation pane, choose Object management → Objects → Upload object.

In the Files to upload section, select a method to scan for files, select the content to upload, and then click Upload object to upload the files.

After the upload is complete, the file status in the Task List panel changes to Uploaded.

Step 2: Add an OSS + API data source

Purchase an instance: For more information, see Purchase an OpenSearch Vector Search Edition instance.

1. Basic table information

On the Instances page, find the OpenSearch Vector Search Edition instance that you purchased and click Manage in the Actions column. In the left-side navigation pane, choose Table management → Add table.

Configure the basic table information and click Next.

After you select the Vector: Image Search template, select a retrieval mode: Dense Vector Retrieval or Dense & Sparse Vector Hybrid Retrieval.

Configuration:

• Table name: Specify a custom name.

• Number of data shards: Enter a positive integer no greater than 256 to improve full index building speed and query performance.

• Number of data update resources: The number of resources used to update data. Each index includes two free update resources, each with 4 vCPUs and 8 GB of memory. Resources beyond the free quota incur charges. For more information, see Billing overview of OpenSearch Vector Search Edition.

• Scenario template: Three templates are available: 1. General-purpose template, 2. Vector: Image search template, and 3. Vector: Semantic text search template.

2. Data synchronization

Set Full data source to OSS + API and configure the other settings. After Data source validation succeeds, click Next.

Configuration:

• OSS path: The path to the OSS files. The path must start with / and cannot contain ?, =, or &. Files cannot be stored in the root directory. Store the files in a folder and specify the path to that folder.

• OSS bucket: The name of the OSS bucket.

On the Buckets page of the OSS console, click the name of the target bucket, such as zhyq-sz, to open its details page.

• Data format: The files must be in HA3 or JSON format. Otherwise, the data cannot be transferred.

• Data source validation: You can proceed to the next step only after validation succeeds.

3. Field configuration

OpenSearch Vector Search Edition presets fields based on the scenario template that you select and automatically imports fields from the full data source into the following list. Complete Field configuration and Data preprocessing - Configure, and then click Next.

Field descriptions:

• id (primary key)

• source_image (source image)

• namespace (namespace)

• source_image_vector (source image vector)

Configuration:

• The primary key field and vector field are required. The primary key field must be of the int or string type, and you must select the primary key option. The vector field must be of the float type, and you must select the vector field option.

• By default, the vector field is a multi-value float field. Multiple values are separated by commas (,) by default. You can also specify a custom multi-value delimiter.

• If a field is missing or empty, the system automatically fills it with a default value: 0 for numeric fields and an empty string for STRING fields. You can specify custom default values.

• Data preprocessing - Configure: Click Configure to open the data preprocessing settings for the source_image field.

  Specific configuration values in this example:

  • OSS Endpoint: oss-cn-shenzhen-internal.aliyuncs.com

  • OSS Bucket: zhyq-sz

  • In the service list, select the Built-in General Image-to-Vector Model - 512-dimension CLIP model and set the generated field to source_image_vector.

  1. Data source: Two options are available: OSS and Base64 encoding. Select OSS in this example.

     • OSS: Specify the OSS path. Store the images in an OSS folder and import them directly from OSS.

     • Base64 encoding: Encode the images in Base64, and then store them in a database or transfer them directly through an API.

  2. Preprocessing template: The available templates depend on the data type to preprocess, such as text or images. This example preprocesses images, so three templates are available: 1. Image embedding, 2. Optical character recognition (OCR), and 3. OCR + Image embedding. Select the Image Embedding preprocessing template in this example.

  3. Service list:

     • After you select a preprocessing template, the service list automatically displays the model types used by the template.

     • Available model sources:

       • Built-in models: A limited selection of model types and models is available free of charge.

       • Custom models: You can customize models to meet your needs. On the Vector Search Edition page, choose Models > Custom Models and click Create Model. For more information, see Custom models.

4. Index schema

Configure the index schema and click Next.

On the Vector Index tab, configure the following parameters for this example:

• Index Name: source_image_vector

• Hybrid Retrieval: Disabled

• Vector Dimensions: 512

• Distance Type: SquaredEuclidean

• Real-time Index: true

• Vector Index Algorithm: Qc

Configuration:

• Included fields: The primary key field and vector field are required. The namespace field is optional and can be left empty.

• Set the vector dimensions, real-time index, distance type, and vector index algorithm based on your business requirements.

• More advanced settings: You can use the default values or adjust them based on your business requirements. For more information, see General vector index configuration. After you configure the Index schema page, click Next to go to the Confirm creation page.

The advanced settings include the following parameters:

• Real-time Index Parameters

• Index Building Parameters

• Index Retrieval Parameters

• Linear Building Threshold

• Minimum Number of Recall Candidates

• Recall Rate Metric Reporting

• Ignore Invalid Vector Data

5. Confirm creation

Click Confirm creation and wait 2 minutes. Return to the Instances page. After the instance status changes to Normal, click Query test in the Actions column to perform searches and tests.