OpenSearch:End-to-end image search solution

You can import image source data directly into OpenSearch to vectorize images and perform vector searches. This enables various image retrieval capabilities, such as search-by-image and text-to-image search.

Solution architecture

You can upload images and build an image search engine in four ways:

• Object Storage Service (OSS) + MaxCompute + OpenSearch Vector Search Edition: Upload images to OSS. Then, store business data and the corresponding image path for each data entry in a MaxCompute table. The image path is the OSS path, such as /image/1.jpg.

• MaxCompute + OpenSearch Vector Search Edition: Store Base64-encoded images and the corresponding table data in a MaxCompute table.

• API + OpenSearch Vector Search Edition: Call the data push API to push Base64-encoded images and the corresponding table data to an OpenSearch Vector Search Edition instance.

• Data Lake Formation (DLF) + OpenSearch Vector Search Edition: Migrate image data to a data table in the DLF 2.0 Catalog.

This topic demonstrates how to build an image search engine using OSS, MaxCompute, and OpenSearch Vector Search Edition.

Prerequisites

1. Create an AccessKey pair

When you activate an Alibaba Cloud account and log on to the console for the first time, you are prompted to create an AccessKey pair before you can proceed.

• An AccessKey pair for your Alibaba Cloud account is required to create and use applications.

• After you create an AccessKey pair for your Alibaba Cloud account, you can also create an AccessKey pair for a RAM user to access services. You must grant the required access permissions to the RAM user. For more information, see Create and authorize a RAM user.

2. Create an OSS bucket

1. Activate OSS

2. Get started with the OSS console.

3. Get started with the OSS console

4. Add the opensearch tag. Set both the key and value to opensearch.

This topic uses 1,000 images uploaded to OSS as an example.

The following are some common image types:

Purchase an OpenSearch Vector Search Edition instance

To purchase an instance, see Purchase an OpenSearch Vector Search Edition instance.

Instance configuration

On the Instance List page, find the newly purchased instance. The instance is in the Pending Configuration state. You must configure a table for the instance before you can perform searches.

1. Basic table information

Configuration description:

• Table Name: A custom name for the table.

• Number of data shards: Enter a positive integer no greater than 256. Sharding improves the speed of full builds and the performance of single queries. For some existing instances, all index tables must have the same number of shards. Alternatively, one index table can have a single shard, and the other index tables must have the same number of shards.

• Number of resources for data updates: The number of resources used for data updates. By default, each index is provided with two 4-core 8 GB update resources free of charge. You are charged for resources that exceed the free quota. For more information, see Billing overview of Vector Search Edition.

• Scenario Template: Select Vector: Image Search.

2. Data synchronization

For the full data source, select MaxCompute + API. Configure the parameters sequentially. The data source must pass validation before you can proceed to the next step.

Parameter description:

• AccessKey ID: The AccessKey ID of your Alibaba Cloud account or a RAM user.

• AccessKey Secret: The AccessKey secret that corresponds to the AccessKey ID.

• Project Name (Project): The name of the target MaxCompute project.

• Table Name (Table): The name of the target MaxCompute table.

• Table Data Partition (Partition): A partition key is required for a MaxCompute data source. Example: ds=20170626.

• Automatic Reindexing: Specifies whether to enable automatic reindexing. If enabled, the system automatically rebuilds the index for the table when a change is detected in its data source.

3. Field configuration

Vector Search Edition provides preset fields based on the selected scenario template. It also automatically imports fields from the full data source into the field list. The following figure shows the four preset fields for the Vector: Image Search template.

Field configuration description:

• Preset fields: 1. id (primary key), 2. source_image (source image), 3. namespace (namespace), and 4. source_image_vector (source image vector).

• Required fields: A primary key field and a vector field are required. The primary key field must be of the INT or STRING type and designated as the primary key. The vector field must be of the FLOAT type and designated as the vector field.

• By default, the vector field is a multi-value FLOAT field, and its values are separated by commas (,). You can specify a custom separator.

• For the field that requires image vectorization, select Configure in the Data Pre-processing Required column to configure the source_image preset field.

• The fields must be defined in a specific order: primary key field, namespace field (optional), and vector field, as shown in the preceding figure.

Configuration description for source_image field data pre-processing:

• Data source: The options are Object Storage Service (OSS) and Base64 encoding.

  • Object Storage Service (OSS): The images are stored in an OSS folder and imported directly. You must provide the OSS path.

  • Base64 encoding: The images must be encoded first and then stored in a database or transferred directly using an API.

• Pre-processing template: The available templates depend on the data type. For an image field, the available pre-processing templates are: 1. Image Vectorization, 2. OCR Image Text Recognition, and 3. OCR Image Text Recognition + Image Vectorization.

• Service list:

  • After you select a pre-processing template, the service list for that template appears automatically, showing the model types used in the template.

  • Models are available from two sources:

    • Built-in model: A limited number of model types are available and can be called free of charge.

    • AI Search Open Platform: A rich set of model capabilities are available. You are charged based on the number of calls. For billing details, see Billing methods and billable items. Before you use the AI Search Open Platform, you must activate a workspace and an API key.

    • Custom model: A custom model created by the user. To create one, see Custom models.

  • AI Platform OCR Image Text Recognition/Text Embedding Service

    • OCR Image Text Recognition: For image data such as architecture diagrams and analysis charts, the AI Search Open Platform provides the Image Content Recognition Service. This service uses a large multi-modal model to parse, understand, and recognize text in images. It can also use OCR capabilities to recognize text in images and extract text information for use in scenarios such as image retrieval and Q&A pairs.

    • Text Embedding: This service converts text data into a dense vector representation. It supports multiple text embedding models with different languages, input lengths, and output dimensions. It can be used for scenarios such as information retrieval, text classification, and similarity comparison.

• Generated fields: After the model processes the source image data, the data is automatically converted into other formats, which generates new fields.

  • For services that perform embedding, a generated field is required by default.

  • For OCR services, you can choose whether to generate a field.

• API call configuration: To successfully access the AI Search Open Platform, set the workspace, API key, and connection domain name.

4. Index schema

Configuration description:

• The vector index name is the same as the vector field name.

• The included fields are the primary key and the vector field. The namespace field is optional.

• Configure the advanced configuration parameters as needed. For more information, see General configurations for vector indexes.

5. Confirm creation

Click Confirm Creation to create the table. In the navigation pane on the left, choose Log Records > Change History to view the table creation progress. When the status changes to Complete, you can perform query tests.

Query tests

1. Perform query tests in the console

In the navigation pane on the left, choose Vector Management > Query Test. The query test feature in the console supports Form mode and Developer mode. For more information about how to use query tests in the console, see Query tests.

1.1 Form mode

Select Form Mode and then select a Table Name. For Query Type, select Image for this example (Vector, Text, and Image are supported). For Image Provider, select Upload Image File. After you upload the query image, click Search to view the results in the search results pane.

1.2 Developer mode

Select Developer Mode and then select a Table Name. Enter the query parameters in the search box. Then, click Search to view the query results in the results pane on the right.

Field description:

• vector: The specific vector to query.

• topK: The number of top K results to return.

• includeVector: Specifies whether to return the vector information in the document.

For detailed query syntax, see the syntax description below.

2. SDK query

Query example: For more information, see Prediction query.

Results:

{
    "totalCount": 5,
    "result": [
        {
            "id": 5,
            "score": 1.103209137916565
        },
        {
            "id": 3,
            "score": 1.1278988122940064
        },
        {
            "id": 2,
            "score": 1.1326735019683838
        }
    ],
    "totalTime": 242.615
}

The result field contains the returned results.

Syntax description

• Image prediction query

• Primary key query

• filter expression

Use vector search with an SDK

• Query data (Perform vector and primary key queries using an SDK)

• Update data (Add and delete records using an SDK)