OpenSearch:Product overview

System architecture

Online system

From the user's perspective, the online system is divided by table. Users can query the data of each vector table separately through APIs, and can also manage each table separately, such as configuring table fields, configuring vector indexes, configuring data sources, and controlling the real-time data concurrency of each table.

On the backend, the online system uses a distributed architecture with multiple rows and columns that consists of query nodes and data nodes:

• Query nodes: A query node parses, validates, or rewrites incoming query requests, forwards the parsed requests to data nodes for execution, collects and merges the results returned by the data nodes, and returns the processed results to the user. Query nodes are compute nodes that do not load user data and generally do not require much memory. However, they consume a large amount of memory when a large number of documents are returned or when statistics generate a large number of entries. If the processing capacity of the query nodes reaches a bottleneck, you can increase the number of query node replicas or upgrade the query node specifications.

• Data nodes: A data node loads the user's index data and, based on queries, retrieves documents and performs operations such as filtering, statistics, and sorting on them. Indexes on data nodes can be sharded. Sharding means hashing the sharding field into the range [0,65535] and dividing this range into a specified number of shards (specified when the index is built). For clusters with large data volumes or high query performance requirements, sharding can improve the processing performance of a single request. To improve the processing capacity of the entire cluster (for example, from 1,000 QPS to 10,000 QPS), you can add replicas. Adding a replica does not mean adding only one Searcher node, but adding multiple Searcher nodes that together hold all the data (the shards must together cover the complete range [0,65535]).

Tables share data node resources (CPU, memory, and disk):

The preceding figure shows two vector tables, one with 4 shards and one with 1 shard. For the 4-shard table, data is evenly distributed to each shard based on the generated index. The 1-shard vector table becomes a broadcast table, and each data node stores a complete copy of the index data of vector table 2.

Data sources

By configuring a data source, users can rebuild indexes (full data) from raw data and import the rebuilt index data into tables in the online system for data retrieval. Data sources and tables have a one-to-one relationship. Currently, the system supports the following data sources: MaxCompute, Saro, and API. API is an empty data source, and all data must be pushed by users to tables in the online system through APIs.

Management and control system

The management and control system is the O&M platform for an OpenSearch Vector Search Edition instance. This platform greatly reduces O&M costs.

Data synchronization process

Full data process

OpenSearch Vector Search Edition supports multiple index versions. Each index version is built based on a copy of raw data (the API data source contains no data by default). Each triggered index rebuild is a full data process. The full data process is a non-resident task: after data processing is complete, a full index is generated and the full data process ends. The generated full index is then switched to the online cluster through a full switchover to provide retrieval services.

Support for multiple index versions ensures the stability of data changes. When the index schema or the data structure changes, the new index generated through the full data process is completely isolated from the index of the previous version. If a change causes problems, you can roll it back promptly.

Generating a full index involves processes such as data processing, index building, and index merging. At each stage, you can set the concurrency of index processing to speed up full index generation.

Real-time incremental process

After a full index is generated, each full version has a resident incremental process, and incremental data synchronization is performed by data update nodes.

MaxCompute, OSS, and API data sources all push real-time incremental data through APIs. The incremental data is pushed to tables through APIs, consumed by data update nodes, and finally indexed in real time by data nodes for user queries.

The incremental process is a resident task. Each full version of each table corresponds to one incremental process. You can control the number of data update nodes to improve real-time data processing capacity.