The AISearch feature in PolarDB for MySQL integrates vector similarity search into the database kernel. While storing and processing structured data, you can also perform efficient similarity search on vectors generated from unstructured data, such as text, images, and audio. You can build AI applications like semantic search, intelligent recommendations, and image search within your PolarDB cluster without building and maintaining separate vector databases or complex data synchronization pipelines. PolarDB provides two vector search protocols to suit different technology stacks and business scenarios: a fully MySQL-compatible protocol, and the OpenSearch protocol (PolarSearch), which is compatible with the mainstream search ecosystem.
Core concepts
Vector embedding: A technique that converts unstructured data, such as text and images, into numerical arrays (vectors) using pre-trained embedding models. These vectors capture the deep semantic information of the data, enabling machines to understand and compare their similarity.
Similarity search (k-NN): Also known as k-nearest neighbor search. Its core objective is to find the k vectors in a massive dataset that are "closest" to a given query vector. In this context, "distance" is calculated using various mathematical formulas and represents the semantic similarity between data points.
Vector index: A vector index is created to avoid full scans of massive datasets. An index is a data structure optimized for efficient queries. It narrows the search space based on vector distribution, reducing search latency from seconds to milliseconds while ensuring a high recall rate.
Vector index algorithm comparison: PolarDB supports multiple mainstream vector index algorithms. The two most common are HNSW and IVF, each with trade-offs in performance, resource consumption, and applicable scenarios.
HNSW (Hierarchical Navigable Small World): A graph-based index that provides high performance and a high recall rate but has significant memory overhead. It is ideal for scenarios that require extremely low query latency and high precision, where the dataset fits within available memory.
IVF (Inverted File): A cluster-based inverted index with lower memory consumption, making it well-suited for very large datasets in memory-constrained scenarios. However, its search precision is typically slightly lower than HNSW.
Key advantages
The PolarDB vector engine combines the simplicity of relational databases with the high performance of dedicated vector databases, offering significant advantages over other technology options.
Feature | Traditional relational database | Dedicated vector database | PolarDB vector engine |
SQL support | Supported | Not supported | Supported |
Vector search | Poor performance | High performance | High performance |
Learning curve | Low | High | Low |
Ecosystem compatibility | Rich | Limited | Dual-ecosystem |
Scalability | Limited | Good | Excellent |
Operational complexity | Simple | Complex | Simple |
In addition, it offers the following core advantages:
All-in-one solution: Process both business and vector data within PolarDB without a separate vector database. This simplifies the system architecture and reduces operational costs.
Enterprise-grade reliability: Fully supports ACID transactions to ensure data consistency. Its distributed architecture provides high availability and automatic switchover to ensure business continuity.
Close integration with LLMs: Built-in inference capabilities for the Qwen large language model (LLM) simplify the development pipeline for AI applications.
Core capabilities and performance metrics
Search performance
Latency: P99 (99% of requests) < 10 ms, P95 (95% of requests) < 5 ms.
Throughput: > 10,000 QPS per node.
Accuracy: recall rate > 99%.
Scalability
Data scale: Supports petabytes of vector data and similarity search across billions of vectors.
Concurrency: Handles tens of thousands of concurrent queries.
Cluster scale: Supports dynamic scale-out to hundreds of nodes with intelligent data sharding strategies.
Resource efficiency
Storage compression: Vector data compression ratio > 50%.
Memory usage: Effectively supports terabyte-scale graph indexes through layered caching technology.
CPU utilization: CPU utilization > 80% with multi-core parallel computing optimizations.
Use cases
PolarDB provides a dual-protocol design. You can choose the protocol that best fits your team's tech stack, business requirements, and performance expectations.
Comparison | MySQL protocol | OpenSearch protocol |
Access method | Standard SQL | RESTful API (compatible with Elasticsearch/OpenSearch) |
Core advantages | Integrates with business data, supports adding vector columns to existing tables, supports ACID transactions, and has a low learning curve. | Offers hybrid search capabilities that support combined queries with vectors, full-text, and scalar data in a mature ecosystem. |
Underlying dependency | Relies on In-Memory Column Index (IMCI). Vector searches run on an IMCI read-only node to isolate analytical and transactional workloads. | Runs on a dedicated search node (PolarSearch), providing a search engine-like service. |
Data synchronization | Not required. Data written to the primary node is automatically visible to the IMCI read-only node. | Not required. Data written to the primary node is automatically visible to the search node. |
Intelligent Q&A and customer service bots
Business pain point: Traditional keyword matching often fails to understand user intent, leading to inaccurate answers.
Solution: Convert question-answer pairs from your knowledge base into vectors and store them. When a user asks a question, convert it into a query vector and use similarity search to find the most relevant knowledge points and provide a more accurate answer.
Protocol recommendation:
MySQL protocol: If you only need basic semantic matching and want to quickly integrate it into an existing MySQL application, this protocol is a convenient choice.
OpenSearch protocol: If you need to combine keywords, category labels, or other criteria for complex filtering, the hybrid search capabilities of this protocol are ideal.
Personalized recommendation systems
Business pain point: Recommending relevant products or content based on a user's historical behavior, such as views, clicks, and purchases.
Solution: Represent both users and items (such as products, articles, and videos) as vectors. By calculating the similarity between user and item vectors, you can retrieve a candidate set of items that the user might be interested in. This set can then be refined by using other ranking strategies.
Protocol recommendation:
OpenSearch protocol: The retrieval layer of a recommendation system often handles a massive amount of data and is sensitive to performance and cost. The IVF index and Product Quantization (PQ) technology provided by the OpenSearch protocol are well-suited for handling massive datasets while controlling memory costs.
Image search and multimodal retrieval
Business pain point: Finding similar images by uploading a sample image, or searching for images using a text description.
Solution: Convert both images and text into vectors and store them in PolarDB. Whether the input is an image or text, it is converted into a query vector to perform a similarity search in the database.
Protocol recommendation:
MySQL protocol: For scenarios that require strong consistency between image feature vectors and business data such as product IDs and prices, the transactional capabilities of the MySQL protocol are an essential safeguard.