Vector Retrieval Service for Milvus:Manage External Collections

Step 1: Link a lake table

Select a DLF instance, database, and target table. The page displays the schema (field name and type) of the lake table. Unsupported types are highlighted in red. For detailed mapping rules, see Schema mapping rules below.

Step 2: Configure the external collection

Configure the following settings:

• Name and Description: The name and description of the collection.

• Primary key configuration: The primary key supports INT64 and VarChar types. Auto ID is not supported.

• Vector field: Supports Float, Binary, Float16, and BFloat16 vector types. You must specify the dimension.

• Scalar field configuration: Configure the scalar field mapping based on the fields of the lake table.

Step 3: Confirm data mapping

Confirm the mapping between DLF lake table fields and Milvus collection fields, and then click Create to create the Milvus collection.

Sync operations

• Auto mode: Automatically retrieves the latest snapshot of the DLF table (preferring the COMPACT commit type), creates a tag, and syncs it to the Milvus External Collection. You can configure a snapshot prefix filter.

• Manual mode: Select an existing tag name to sync.

Scheduled sync configuration

Enable scheduled sync and configure a sync policy:

• Simple interval mode: Set a sync interval in minutes, hours, or days.

• Cron expression mode: Use a cron expression to customize the sync schedule. You can also configure a snapshot prefix.

Sync task list

Filter tasks by status or tag and browse results with pagination. For failed tasks, you can view the failure reason.

Schema limitations

Unsupported field types:

• auto_id=True is not supported. External Collections must use an external primary key.

• TIMESTAMPZ and GEOMETRY types are not supported.

• SparseFloatVector type is not supported.

• Nested complex types (such as Array>) are not supported.

• DynamicField is not supported.

• PartitionKey is not supported.

Field naming limitations:

• Field names are case-sensitive.

• Field names must exactly match the Paimon table field names.

• Field name mapping is not supported (for example, mapping user_id in Milvus to userId in Paimon).

Other limitations:

• Schema evolution is not supported. The schema cannot be changed after creation.

• Field name remapping is not supported.

• is_function_output=True (function output fields) is not supported.

Data operation limitations

Unsupported operations:

• insert(), upsert(), delete(), and flush() operations are not supported.

• Direct data modification is not supported. Data must be modified on the Paimon table side.

Supported operations:

• search(): Vector search.

• query(): Scalar query.

• create_index(): Create an index.

• load()/release(): Load or release a collection.

• bulk_import(): Trigger a refresh (incremental sync).

Index limitations

Supported index types:

• Vector indexes: HNSW, HNSW_SQ, HNSW_PQ, IVF_FLAT, IVF_SQ8, IVF_PQ, IVF_RABITQ, and SCANN.

• Scalar indexes: INVERTED, BITMAP, and STL_SORT.

Index behavior:

• Indexes are built on the Milvus side and stored in the Milvus object storage.

• Indexes do not affect the Paimon table data.

• Dropping a collection also deletes its indexes.

Performance limitations

Cold read performance: Without indexes, queries must scan the corresponding column data in the Paimon table. Create indexes for all mapped fields to improve query performance.

Sync operation performance: Sync operation speed depends on the snapshot size of the Paimon table, network bandwidth, and catalog response speed.

Consistency limitations

Tag consistency:

• External Collections provide consistency guarantees based on DLF tags. You are responsible for managing the lifecycle of tags.

• Query results reflect the state of the tag specified during the last sync. During a sync operation, you can query data from the previously refreshed tag.

• To query the latest data, you must manually trigger a refresh.

Concurrency limitations:

• Only one sync task can run at a time for a single External Collection.

• Queries can be performed normally during a sync operation. The queries return data from the last completed sync.

Scalar type mapping

Vector type mapping

Array type mapping