Vector analysis

Introduction to vector databases

Most real-world data is unstructured, such as images, audio, videos, and text. The volume of this unstructured data is growing rapidly due to applications like smart cities, short videos, personalized product recommendations, and visual product searches. To process such data, artificial intelligence (AI) techniques are commonly used to extract features and convert them into feature vectors. These vectors are then analyzed and retrieved to work with the original unstructured data. A database capable of storing, analyzing, and retrieving feature vectors is known as a vector database.

Vector databases use vector index technology to enable fast retrieval of feature vectors. Vector indexing is a form of Approximate Nearest Neighbor Search (ANNS), which focuses on identifying likely neighbors rather than exact matches. This approach improves retrieval efficiency by sacrificing a small amount of precision within an acceptable range, distinguishing vector databases significantly from traditional databases.

To simplify the use of ANNS vector indexing in production environments, the industry has adopted two main approaches.

• One approach offers ANNS vector indexing as a standalone service for creating and retrieving vector indexes, resulting in a dedicated vector database.

• The other approach integrates ANNS vector indexing into a traditional structured database, creating a DBMS with vector retrieval capabilities.

The vector database feature in AnalyticDB for PostgreSQL is a DBMS that integrates the self-developed vector search engine FastANN. It supports vector retrieval while providing comprehensive database capabilities, including ease of use, transactions, high availability, and high scalability. You can process vectors directly using SQL.

Features

Vector databases use AI algorithms to extract features from unstructured data, with the resulting feature vectors serving as unique identifiers. The distance between vectors reflects the similarity of the underlying unstructured data. In AnalyticDB for PostgreSQL, vector retrieval and analysis are built on a Massively Parallel Processing (MPP) query architecture, allowing you to retrieve unstructured data through a SQL interface and perform association analysis with structured data.

Common scenarios

You can use vector analysis in AnalyticDB for PostgreSQL to easily build various intelligent applications.

• Search-by-image services that retrieve images based on an input image.

• Video retrieval services that locate videos by searching for specific frames within them.

• Voiceprint retrieval services that match audio clips against other audio clips.

• Recommendation system services that match user features to deliver personalized recommendations.

• Semantic-based text retrieval and recommendation that identifies similar texts.

• Q&A chatbots built by integrating with a Large Language Model (LLM) to deliver efficient question-and-answer services.

• File deduplication that removes duplicate files using file fingerprint features.

Advantages

Powered by the self-developed vector engine FastANN, the vector analysis capabilities of the AnalyticDB for PostgreSQL vector database are widely used across many services, including Alibaba's data mid-end, Alibaba's e-commerce and new retail businesses, Alibaba Cloud's Urban Intelligence, and Q&A services built with the Qwen LLM.

Compared to other vector databases, it offers the following advantages:

• Hybrid analysis of structured and unstructured data.

  AnalyticDB for PostgreSQL combines traditional database capabilities to support hybrid analysis of unstructured, structured, and semi-structured data, fully leveraging its indexing capabilities for structured and semi-structured data.

• Dual-channel retrieval with vector search and full-text search.

  AnalyticDB for PostgreSQL supports both vector indexes and full-text indexes, enabling dual-channel retrieval through vector search and full-text search to significantly improve recall accuracy for vector data.

• Real-time data updates and queries.

  AnalyticDB for PostgreSQL supports stream ingestion and real-time construction of vector data.

• Ease of use.

  AnalyticDB for PostgreSQL is ready to use upon request, supports standard SQL syntax, and simplifies development significantly.

• Low cost.

  The AnalyticDB for PostgreSQL vector database supports compressing FP32 to FP16, reducing storage costs by half. In addition, the vector index in the AnalyticDB for PostgreSQL vector database is built on segmented page storage. This enables it to fully utilize PostgreSQL's shared_buffer for caching. Consequently, the vector index in AnalyticDB for PostgreSQL can support vector storage exceeding the available memory size.