OpenSearch:Quick start for single-node instances

Purchase an instance

1. Go to the OpenSearch console and switch to OpenSearch Vector Search Edition in the upper-left corner. Then, on the instance list page, click Create Instance.

2. Set the product edition to Vector Search Edition, select a region, configure Data Nodes, Data Node Specifications, and Total Storage per Data Node, and set VPC and vSwitch. Finally, set a username and password as prompted (used to verify permissions during queries; this is not your Alibaba Cloud account password), and click Buy Now:

   Complete the following configurations on the purchase page:

   • Product Edition: Select Vector Search Edition.

   • Region and Zone: Select the target region, such as China (Hangzhou).

   • Data Nodes: Set this based on your business requirements, for example, 1.

   • Data Node Specification Family: Select Local SSD or Cloud Disk.

   • Data Node Specifications: Select the required specifications, for example, 2 cores and 16 GB.

   null

   • Plan the number and specifications of data nodes to purchase based on your business. After you determine the specifications, the actual cost is automatically generated on the buy page.

   • For the single-node solution, you do not need to purchase query nodes (that is, the number of query nodes is 0), and the query node specification family does not take effect.

   • The VPC and vSwitch configurations must be the same as those of the ECS instance that accesses the Vector Search Edition instance. Otherwise, the error {'errors':{'code':'403','message':'Forbidden'}} is reported when you access the Vector Search Edition instance.

   • The SSD data node specification family includes a free quota of 50 GB, and additional storage is charged in increments of 50 GB. The cloud disk type has no free quota, and storage is charged in increments of 50 GB.

3. On the Confirm Order page, review the service agreement. After you confirm that everything is correct, click Activate Now.

4. After the purchase succeeds, click Console to view the purchased Vector Search Edition instance on the Instances page.

Configure the instance

For a newly purchased instance, the instance status on its details page is Pending Configuration, and an empty instance consistent with the number and specifications of the purchased data nodes is automatically deployed. You then need to configure the basic table information, data synchronization, fields, and index schema for the instance. After the index is rebuilt, you can search normally.

1. Basic table information

In table management, click Add Table, enter a table name, set the number of data shards and number of resources for data updates, select the required scenario template, and click Next:

Configuration description:

• Table Name: Can be customized.

• Number of Data Shards: Because this is a single-node instance, the value is fixed at the default value of 1 and cannot be changed. The current instance supports only the single-shard solution. You can scale out to support multiple nodes. For more information, see the scaling documentation.

• Number of Resources for Data Updates: The number of resources used for data updates. Each index is provided with two free update resources of 4 cores and 8 GB by default. Resources that exceed the free quota incur fees. For more information, see Billing overview of OpenSearch Vector Search Edition

• Scenario Template: Vector Search Edition provides three built-in templates: General, Vector-Image Search, and Vector-Text Semantics.

2. Data synchronization

Select the full data source (supported data sources include MaxCompute+API, Object Storage Service (OSS)+API, Synchronize vector data from OpenLake-DLF to Alibaba Cloud OpenSearch, and API). This topic uses MaxCompute+API as an example. After you configure the parameters and they pass the verification, click Next.

After you select MaxCompute+API, specify the following parameters:

• AccessKey and AccessKey Secret

• Project: Must start with a letter, can contain only letters, digits, and underscores, and must be 3 to 28 characters in length.

• Table: Must be 1 to 128 characters in length.

• Partition: If the source table has multi-level partitions, separate the partitions with commas.

• Automatic Index Rebuilding: If you select Yes, make sure that a done table has been created in MaxCompute. Otherwise, automatic index rebuilding does not take effect.

After you complete the settings, click the Verify button to verify the data source.

3. Field configuration

OpenSearch presets relevant fields based on the scenario template that you select, and automatically imports the fields from the full data source (if any) into the field list:

Field configuration description:

• Required fields: primary key field and vector field. The primary key field must be of the int or string type and the Primary Key option must be selected. The vector field must be of the float type and the Vector Field option must be selected.

• By default, the vector field is a multi-value float field. Multiple values are split by the system default delimiter, the comma ",". You can also enter a custom multi-value delimiter.

• If a field is missing from the data or a field is empty, the system automatically fills in a default value: 0 for numeric types and an empty string for the STRING type. Custom default values are supported.

4. Index schema

4.1. Vector index

On the Vector Index tab, enter an index name (such as message_vector) and specify whether to enable hybrid retrieval. The top of the page indicates that OpenSearch has automatically built indexes with the same names for all fields except vector fields.

• The primary key field and vector field are required. The namespace field is optional and can be left empty.

• Only the three fixed fields can be selected. Adding fields is not supported.

• Vector dimension: Select based on the vectors generated by your model.

• Distance type: InnerProduct (inner product distance; a higher score indicates higher relevance) and SquareEuclidean (Euclidean distance; a lower score indicates higher relevance) are supported.

• Vector index algorithm: Linear, HNSW, QGraph, and QC are supported. Instances with GPU specifications additionally support the CAGRA algorithm. For more information about the algorithms, see Introduction to vectors.

• Real-time index: Specifies whether vector indexes are built in real time for incremental data from the API. Default value: true.

• For more advanced configurations, click to expand the section. For parameter descriptions, see General vector index configuration.

  The advanced configurations include the following parameters:

  • Real-time Index Parameters: Default value: {"proxima.oswg.streamer.segment_size":2048}

  • Index Retrieval Parameters: Default value: {"proxima.qc.searcher.scan_ratio":0.01}

  • Index Building Parameters: Default value: {"proxima.qc.builder.quantizer_class":"Int8Quant"...}

  • Minimum Number of Retrieval Candidates: Default value: 20000

  • Linear Build Threshold: Default value: 5000

  • Ignore Invalid Vector Data: Default value: true

  • Recall Rate Metric Reporting: Default value: true

4.2. Other index types

OpenSearch has automatically built indexes with the same names for all fields except vector fields, and you can manage them here. Vector indexes must be configured manually.

On the Other Index Types tab, the index configuration table contains the following columns: Index Name, Included Fields, Build Index (controlled by a switch), and Index Type (primary key index or normal index). You can enable or disable index building for each field and select the corresponding index type based on your business needs.

4.3. Global index configuration

This feature is disabled by default. After you enable it, you can set an expiration time for an integer field. If the current time minus the document time > the expiration time, the document is automatically cleared.

The configuration items include Document Time Field (select an integer field that represents the document time and specify the time unit) and Expiration Time (valid values: 1 to 49710 days).

5. Confirm creation

After the index configuration is complete, click Confirm Creation. The table status changes to being created.

6. Change history

In Instance Management > Change History, you can view change records for the last 3, 7, or 30 days, such as all processes for creating a table, adding an index, scaling out, and full data import. After all processes are complete, the engine is built and you can start query tests.

7. Query test

After you create the table and add data, you can run queries in Vector Management>Query Test. Query tests support both form mode and developer mode.

7.1 Form mode

Configure Form Mode → Table Name → Query type in sequence. You can set the query type to Vector or Text. In this example, Text is selected. Enter the query text in the text field and click Search. The results are displayed in the Search Results section.

7.2 Developer mode

Configure Form Mode → Table Name → Search Box in sequence. Enter the query parameters in the search box and click Search. The results are displayed in the Search Results section.

• vector: the vector to query

• topK: the number of top results (K) to return

• includeVector: specifies whether to return the vector information in the documents

For the detailed query syntax, see the syntax description below.

Syntax description

• Vector query

• Primary key-based query

• Filter expressions

Use vector search in SDKs

• Query data (perform vector and primary key queries by using SDKs)

• Update data (add and delete records by using SDKs)

Upgrade from a single node to multiple nodes

To upgrade a single-node instance to multiple nodes, perform the following operations:

1. On the instance list page, find the instance and click Upgrade/Downgrade in the Actions column on the right.

2. On the configuration change page, select the number and specifications of query nodes to scale out or scale in, select the service agreement, and then click Buy Now.