Object Storage Service:Overview of vector buckets

Core concepts

• Vector bucket: A new bucket type and cloud resource for managing large-scale vector data.

• Vector index: You can create vector indexes in a vector bucket. A vector index is an index table that stores vector data. You can create multiple vector indexes in the same vector bucket for different business types to store their respective vector data. When you initiate a retrieval query, the results are returned based on the similarity of the vector data in the vector index.

• Vector data: High-dimensional numerical arrays converted from unstructured data, such as images, videos, and documents, using a vector model. These arrays represent content features. Vector retrieval returns results based on the similarity of vector data. You can use any vectorization service, such as ECS, PAI, or Alibaba Cloud Model Studio, to generate vectors. Then, you can write them to a specified vector index using the OSS API, a software development kit (SDK), or the ossutil tool. When writing data, you can also attach metadata for subsequent scalar filter queries.

Benefits

• Low cost: Vector data has become essential infrastructure for various AI applications and is growing exponentially. Vector buckets use a simple and user-friendly billing model. Billing is based on only two items: vector data storage capacity and the amount of data scanned during retrieval. This model can reduce costs by more than 90% compared to traditional methods.

• Large scale: OSS vector buckets are designed with an architecture for large-scale vector data storage. They can handle the storage requirements for massive amounts of vector data. OSS uses a serverless architecture that scales elastically. After you start using a vector bucket, you do not need to worry about scaling.

• Easy to use: OSS vector buckets provide a complete API, SDK, and the ossutil command line interface. You can also manage and perform read and write operations on vector data in the OSS console, such as retrieving, adding, and performing a bulk insert of vector data.

• Unified management: You can manage vector buckets and buckets that store massive raw data in the same way. For example, you can configure the same bucket policy for permission management or set the same log export path for auditing.

• Semantic retrieval: You can use the `QueryVectors` API provided by vector buckets to query vector data in an index table and obtain retrieval results sorted by similarity. OSS vector buckets also support filtering queries based on scalar metadata. When you write vector data to an OSS vector bucket, you can include scalar metadata to enable post-filtering. When creating a vector index, you can also set non-filterable metadata. Non-filterable metadata cannot be used as a post-filtering condition but is returned with the retrieval results as descriptive information for the vector results.

Scenarios

Scenario 1: Build a low-cost RAG application

As AI services develop, the volume of vector data grows exponentially, which increases the pressure on storage and retrieval costs. For multi-modal retrieval scenarios, such as knowledge bases, AI assistants, and medical image retrieval, user toleration for retrieval latency is increasing from tens to hundreds of milliseconds. In this case, you can use a vector bucket as the vector storage foundation for your RAG application to meet business needs at a very low cost.

Scenario 2: Build an AI agent with layered retrieval

Different AI agents have varying requirements for retrieval performance. You can store the full volume of vector data centrally in a low-cost OSS vector bucket. For business scenarios that require high performance and low latency, you can synchronize hot spot data to other products, such as Tablestore, for high-performance retrieval. This method helps you build a layered retrieval architecture for your AI agent application.

Scenario 3: Build an AI content management platform with unified data management

AI applications generate massive amounts of unstructured content, such as user-generated content (UGC), internal documents, and AI-generated content, along with their corresponding vectorized results. This process can lead to fragmented storage and retrieval systems. By storing raw data in a standard OSS bucket and vector data in an OSS vector bucket, you can build an efficient AI data management platform, for example, for AIGC data management. You can use a single set of APIs and SDKs to manage and access both raw files and vector indexes. This makes it easy to build an efficient and unified AI content management platform.

Enterprise-grade features

Domain name access

Vector buckets provide separate public and internal network endpoints that are isolated from standard OSS buckets.

• Public endpoint: $bucketname-$uid.regionID.oss-vectors.aliyuncs.com

• Internal network endpoint: $bucketname-$uid.regionID-internal.oss-vectors.aliyuncs.com

Note: Except for the ListVectorBuckets operation, all other operations must use a third-level domain.

Secure transmission

Vector buckets support encrypted transmission over the HTTPS protocol to ensure data security during transit.

Access control

• Bucket policy: Supports resource-based authorization policies. You can control permissions with granularity down to the vector bucket level or the level of a single vector index or multiple vector indexes.

• RAM policy: Supports identity-based RAM authorization policies. This allows for fine-grained permission control over vector buckets, vector indexes, and data operations. This also supports cross-account access authorization.

Logs

• Access log storage: Supports storing access logs in a specified bucket in real time or near real time.

• Unified log format: The log format is fully compatible with standard OSS logs. It includes an additional BucketARN field to uniquely identify the vector bucket resource, which simplifies unified log analysis.

Billing

This feature is currently in invitational preview and is free of charge. To request a trial, go to the Vector Buckets page.