This topic describes how to use the AI Function capability of Alibaba Cloud Milvus to process intelligent-driving frames: create a collection once with multiple functions attached, and when a frame address is written, multimodal vectorization, scene classification, structured extraction of traffic elements, and named entity recognition are performed automatically. You can then perform text-to-frame search, image-to-frame search, corner case mining with structured filtering, and multimodal reranking.
Solution overview
In the intelligent-driving R&D pipeline, the vehicle-side camera is the first entry point for data. A test vehicle is typically equipped with 6 to 12 surround-view cameras that continuously capture driving footage at 20 to 30 FPS, so a single vehicle can generate several TB of video in a single day of road testing. These videos are the core assets for training perception models, reproducing accidents, and iterating on planning and control strategies, but they are massive, unstructured, and hard to retrieve.
What engineering teams care about is not a given video clip itself, but what happens inside the video. Performing scene understanding frame by frame requires identifying traffic participants (pedestrians, non-motorized vehicles, the vehicle ahead), traffic control elements (traffic light status, lane lines, speed limit signs), the road environment (intersections, highways, tunnels, construction zones), and abnormal events (cut-ins, running red lights, hard braking, scattered objects) from the image. Only by persisting these understanding results in a structured form can the following high-value scenarios be supported:
Data replay: Replay real-world traffic frames into the simulation and training pipelines to continuously iterate perception models.
Corner case mining: Precisely retrieve long-tail scenarios such as "a construction zone at a tunnel entrance on a rainy night" from massive frame sets. These are exactly the cases where models are most likely to fail and where samples are scarcest.
Automatic labeling: Use a large model to coarsely label frames, replacing a large amount of manual bounding-box labeling. Humans only review and refine the results.
Road test reports: Aggregate statistics by scene and event to quickly produce quantifiable periodic reports.
Without a unified platform, you typically have to build your own pipeline of "frame extraction service → object detection/multimodal model → vector database → metadata database". You must maintain scaling and failure recovery for the GPU inference cluster yourself, and frame data is repeatedly moved between object storage, the inference cluster, and the vector database, resulting in a long pipeline with many points of failure. Manual labeling also suffers from inconsistent criteria, which introduces significant label noise.
The AI Function capability of Alibaba Cloud Milvus 2.6 converges these capabilities into a single vector database: inference as query. Model inference is no longer a separate external pipeline; instead, it is a function call automatically triggered inside Milvus during insert and search. For frame analysis, this topic uses the following functions:
Function | Purpose | Purpose in frame analysis |
| Use | Supports text-to-frame search and image-to-frame search, so semantic requirements such as "a construction zone in the rain" can be matched by vector retrieval. |
| Selects the best-matching item from a preset label set for each frame and writes it to the classification field. | Tags each frame with a scene label such as intersection, highway, tunnel, or construction zone. |
| Extracts elements from a frame based on specified labels and writes them to a structured field as JSON. | Extracts the traffic light status, lane count, weather, and abnormal events for precise filtering. |
| Identifies named entities that explicitly appear in a frame. | Extracts place names, road names, and speed limit values from signs. |
| After recalling the top N results with vector retrieval, performs secondary scoring with | Reranks by consistency of subject, action, and scene to improve the fine-ranking quality of corner case mining. |
The first four functions are executed automatically at write time and form the main "inference on write" pipeline. AI_RERANK takes effect during the retrieval phase and is an optional fine-ranking enhancement.
Prerequisites
A Milvus 2.6 instance has been created. AI Function relies on the 2.6 kernel, and no separate model service binding is required after creation.
To access the instance over the Internet, you have enabled Internet access on the Security Configuration tab of the instance details page and added the client egress IP address to the Internet access whitelist.
pymilvus has been installed. The examples in this topic are verified with pymilvus 3.0.0. If you want to extract frames yourself, you must also install ffmpeg.
Frame images are prepared and uploaded to a publicly accessible address that the model can reach (such as OSS).
The RESTful interface shares port 19530 with gRPC, so you must explicitly include the port when making calls, for example, http://c-xxx.milvus.aliyuncs.com:19530. If you omit the port, port 80 is used by default and the connection times out.
Procedure
Step 1: Extract frames from video
Vehicle-side cameras capture continuous video, so you must first use ffmpeg to extract frames, obtain frame images, and upload them to an address that the model can access (such as OSS). Then pass frame_url to the subsequent code for ingestion. During frame extraction, also record the clip_id and ts_ms (video clip ID and frame timestamp) of each frame, so that you can locate the original video after a retrieval hit.
# Timed frame extraction: take one frame every 1 second and scale it to a width of 960
ffmpeg -i clip001.mp4 -vf "fps=1,scale=960:-1" -q:v 3 frames/clip001_%04d.jpg
# Take only key frames (I-frames), which carry more information and less redundancy
ffmpeg -i clip001.mp4 -vf "select='eq(pict_type,I)',scale=960:-1" -fps_mode vfr -q:v 3 frames/clip001_key_%04d.jpg
# Capture a single frame at a specified time point (for example, at 5.2s, corresponding to ts_ms=5200)
ffmpeg -ss 5.2 -i clip001.mp4 -frames:v 1 -vf scale=960:-1 -q:v 3 frames/clip001_5200ms.jpgIn scale=960:-1, -1 indicates that the height is automatically calculated based on the original aspect ratio. If you use ffmpeg 8.x or later, use -fps_mode vfr to take I-frames. The legacy syntax -vsync vfr still works but triggers a deprecation warning.
Step 2: Prepare common code
The following code contains the connection configuration, the REST call utility, and a compatibility wrapper for the TEXTTRANSFORM function type. Replace MILVUS_URI and MILVUS_TOKEN with the information of your actual instance.
from __future__ import annotations
import json
from typing import Any
from urllib.error import HTTPError
from urllib.request import Request, urlopen
from pymilvus import DataType, Function, FunctionType, MilvusClient
MILVUS_URI = "http://c-xxxx.milvus.aliyuncs.com:19530"
MILVUS_TOKEN = "root:xxx"
def post_json(path: str, body: dict[str, Any], timeout: int = 120) -> tuple[int, dict[str, Any]]:
request = Request(
f"{MILVUS_URI.rstrip('/')}{path}",
data=json.dumps(body, ensure_ascii=False).encode("utf-8"),
headers={"Authorization": f"Bearer {MILVUS_TOKEN}", "Content-Type": "application/json"},
method="POST",
)
try:
with urlopen(request, timeout=timeout) as response:
return response.status, json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
return exc.code, json.loads(exc.read().decode("utf-8"))
# Alibaba Cloud Milvus exposes TEXTTRANSFORM as a managed extension, whose function type value is 9.
# The FunctionType enum in some pymilvus versions does not include this member, so a compatibility wrapper is provided below.
TEXTTRANSFORM_FUNCTION_TYPE = 9
def texttransform_function_type() -> Any:
for type_name in ("TEXTTRANSFORM", "TEXT_TRANSFORM", "TextTransform"):
function_type = getattr(FunctionType, type_name, None)
if function_type is not None:
return function_type
existing = getattr(FunctionType, "_value2member_map_", {}).get(TEXTTRANSFORM_FUNCTION_TYPE)
if existing is not None:
return existing
extension = int.__new__(FunctionType, TEXTTRANSFORM_FUNCTION_TYPE)
extension._name_ = "TEXTTRANSFORM"
extension._value_ = TEXTTRANSFORM_FUNCTION_TYPE
FunctionType._value2member_map_[TEXTTRANSFORM_FUNCTION_TYPE] = extension
FunctionType._member_map_["TEXTTRANSFORM"] = extension
return extension
VECTOR_DIM = 2560
EMBED_MODEL = "qwen3-vl-embedding"
VLM_MODEL = "qwen3.7-plus" # Multimodal understanding model (classification/extraction/entities)
client = MilvusClient(uri=MILVUS_URI, token=MILVUS_TOKEN)AI_CLASSIFY, AI_EXTRACT, and AI_ENTITY_EXTRACT are all functions of the TEXTTRANSFORM type (function type value 9), and the specific task is distinguished by the task parameter. The FunctionType enum in some pymilvus versions does not include this member, so the compatibility wrapper above is required.
Step 3: Create a collection with four AI Functions attached at once
The vector field is used for semantic retrieval, and the other three fields respectively hold the results of classification, structured extraction, and entity recognition. All four functions take frame_url as input and are executed automatically at write time.
# ===== Create the collection: attach 4 AI Functions in a single creation =====
collection_name = "driving_frames"
if client.has_collection(collection_name):
client.drop_collection(collection_name)
schema = MilvusClient.create_schema(auto_id=True, enable_dynamic_field=False)
schema.add_field("id", DataType.INT64, is_primary=True)
schema.add_field("frame_url", DataType.VARCHAR, max_length=4096) # Frame image address
schema.add_field("clip_id", DataType.VARCHAR, max_length=128) # ID of the video clip it belongs to
schema.add_field("ts_ms", DataType.INT64) # Frame timestamp (ms)
schema.add_field("embedding", DataType.FLOAT_VECTOR, dim=VECTOR_DIM)
schema.add_field("scene", DataType.VARCHAR, max_length=64) # AI_CLASSIFY output
schema.add_field("attributes", DataType.JSON) # AI_EXTRACT output
schema.add_field("entities", DataType.JSON) # AI_ENTITY_EXTRACT output
# 1) Multimodal vectorization: frame image -> 2560-dimensional vector
schema.add_function(Function(
name="embed_frame", function_type=FunctionType.TEXTEMBEDDING,
input_field_names=["frame_url"], output_field_names=["embedding"],
params={"provider": "aliyun_milvus", "model_name": EMBED_MODEL,
"dim": VECTOR_DIM, "is_multimodal": "true"}))
# 2) Scene classification: intersection/highway/tunnel/construction zone/normal road
schema.add_function(Function(
name="classify_scene", function_type=texttransform_function_type(),
input_field_names=["frame_url"], output_field_names=["scene"],
params={"provider": "aliyun_milvus", "model_name": VLM_MODEL,
"task": "ai_classify", "media_type": "image",
"labels": "intersection,highway,tunnel,construction zone,normal road",
"prompt": "Classify by the road environment in which the driving image is captured.", "temperature": "0"}))
# 3) Structured extraction: traffic light/lane count/weather/abnormal event
# Note: specify a default value for every label to avoid the model returning null (see the description below)
schema.add_function(Function(
name="extract_traffic", function_type=texttransform_function_type(),
input_field_names=["frame_url"], output_field_names=["attributes"],
params={"provider": "aliyun_milvus", "model_name": VLM_MODEL,
"task": "ai_extract", "media_type": "image",
"labels": "traffic_light,lane_count,weather,anomaly_event",
"prompt": "traffic_light takes the value red/green/yellow/none; "
"weather takes the value sunny/rainy/cloudy/night/unknown, and fill in unknown when it cannot be determined; "
"lane_count takes an integer, and fill in 0 when it cannot be determined; "
"anomaly_event describes anomalies such as cut-ins, running a red light, or accidents, and fill in none if there is no anomaly.",
"temperature": "0"}))
# 4) Named entities: road names/place names/speed limit values
schema.add_function(Function(
name="extract_entities", function_type=texttransform_function_type(),
input_field_names=["frame_url"], output_field_names=["entities"],
params={"provider": "aliyun_milvus", "model_name": VLM_MODEL,
"task": "ai_entity_extract", "media_type": "image",
"entity_types": "LOCATION,PRODUCT",
"prompt": "Extract only the place names, road names, and speed limit values that explicitly appear on traffic signs in the image. Do not guess based on appearance.",
"temperature": "0"}))
index_params = client.prepare_index_params()
index_params.add_index(field_name="embedding", index_type="AUTOINDEX", metric_type="COSINE")
client.create_collection(collection_name=collection_name, schema=schema,
index_params=index_params)A multimodal function must include "is_multimodal": "true", and the dim of the vector field must be consistent with the dim in the function parameters (2560).
Be sure to specify a default value for each extraction label. In tests, if no default value is specified, the model returns null when it cannot make a determination (for example, for a frame inside a tunnel where the sky is not visible, weather is judged as null). When a JSON field is null, the != condition does not match that row. For example, running attributes["weather"] != "rainy" against 6 frames of data returns only 4 records, and the row with null is silently excluded. In corner case mining scenarios, such silently missed frames have serious consequences, so the prompt above provides a default value for every label.
Step 4: Ingest frames (inference on write)
You only need to write the three fields frame_url, clip_id, and ts_ms. The other four fields are automatically populated by the AI Functions, with no manual labeling required.
# ===== Ingest frames: inference on write, with embedding/scene/attributes/entities automatically populated =====
# Replace frame_url with your own publicly accessible image addresses that the model can reach;
# clip_id and ts_ms record the source clip and timestamp of the frame, making it easy to locate the original video after retrieval.
frames = [
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_dashcam_5000.jpg",
"clip_id": "clip_dashcam", "ts_ms": 5000},
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_dashcam_12000.jpg",
"clip_id": "clip_dashcam", "ts_ms": 12000},
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_highway_3000.jpg",
"clip_id": "clip_highway", "ts_ms": 3000},
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_highway_9000.jpg",
"clip_id": "clip_highway", "ts_ms": 9000},
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_urban_2000.jpg",
"clip_id": "clip_urban", "ts_ms": 2000},
{"frame_url": "https://<your-bucket>.oss-cn-hangzhou.aliyuncs.com/frames/clip_urban_8000.jpg",
"clip_id": "clip_urban", "ts_ms": 8000},
]
# The multimodal batch limit of qwen3-vl-embedding is 10; exceeding it reports image batch size can should be [1, 10]
_BATCH = 8
for _i in range(0, len(frames), _BATCH):
client.insert(collection_name, frames[_i:_i + _BATCH])
client.flush(collection_name)
client.load_collection(collection_name)
# After the write completes, query the structured results directly to verify
ingested_rows = client.query(
collection_name, filter="",
output_fields=["frame_url", "clip_id", "ts_ms", "scene", "attributes", "entities"],
limit=100)
for row in ingested_rows:
print(f"{row['clip_id']}@{row['ts_ms']}ms scene={row['scene']} "
f"attributes={json.dumps(row['attributes'], ensure_ascii=False)} "
f"entities={json.dumps(row['entities'], ensure_ascii=False)}")The multimodal batch limit of qwen3-vl-embedding is 10. If a single batch exceeds this limit, the error image batch size can should be [1, 10] is reported, so data is written in batches. After the write, you must call flush(); otherwise, an immediately following retrieval may return empty results.
In tests, writing 6 frames plus flush and load took about 12 seconds in total. Examples of the structured results:
Frame | scene | attributes | entities |
dashcam@5000ms | Intersection | traffic_light=green, lane_count=4, weather=sunny, anomaly_event=none | [] |
dashcam@12000ms | Construction zone | traffic_light=none, lane_count=3, weather=rainy, anomaly_event=none | [] |
highway@3000ms | Highway | traffic_light=none, lane_count=4, weather=sunny, anomaly_event=none | [{"text":"EXIT 111","type":"LOCATION"}] |
highway@9000ms | Tunnel | traffic_light=none, lane_count=2, anomaly_event=none | [] |
urban@2000ms | Normal road | weather=cloudy, anomaly_event=pedestrians and animals on the road | [] |
urban@8000ms | Intersection | traffic_light=red, lane_count=4, weather=sunny, anomaly_event=none | [] |
The scene classifications of all six frames are fully consistent with the image content, and the extracted traffic light status, weather, and lane count also match the images. Named entities were extracted only for frames containing readable sign text; the model did not invent place names based on image appearance, which complies with the "do not guess based on appearance" constraint in the prompt.
Step 5: Text-to-frame search / structured filtering / image-to-frame search
The vectors and structured fields produced during the write phase can be combined during retrieval: vectors handle semantic recall, and structured fields handle precise filtering.
# ===== Text-to-frame search / structured filtering / image-to-frame search =====
# 1) Text-to-frame search: the query text is mapped into the image vector space by the same multimodal model and recalled directly
query = "Construction zone in the rain, with traffic cones and construction signs on the road"
text_search = client.search(
collection_name=collection_name, data=[query], anns_field="embedding",
limit=10, output_fields=["frame_url", "scene", "attributes"])
for hit in text_search[0]:
print(f"score={hit['distance']:.4f} scene={hit['entity']['scene']}")
# 2) Vector recall + structured filtering (corner case mining)
# The structured fields come from AI_EXTRACT / AI_CLASSIFY at write time and can be used directly in filter
corner_query = "Abnormal events on the road at night"
corner_search = client.search(
collection_name=collection_name, data=[corner_query], anns_field="embedding",
limit=10, filter='attributes["anomaly_event"] != "none"',
output_fields=["frame_url", "scene", "attributes"])
print(f"Matched {len(corner_search[0])} long-tail scenario records")
# 3) Image-to-frame search: simply replace data with an image URL; everything else stays the same
image_query_url = frames[0]["frame_url"]
image_search = client.search(
collection_name=collection_name, data=[image_query_url],
anns_field="embedding", limit=10, output_fields=["frame_url", "scene"])
for hit in image_search[0]:
print(f"score={hit['distance']:.4f} scene={hit['entity']['scene']}")Test results:
Retrieval method | Query | Result |
Text-to-frame search | Construction zone in the rain, with traffic cones and construction signs on the road | The construction zone frame ranks first with 0.5904, while the runner-up scores only 0.1797, showing clear discrimination |
Image-to-frame search | Using the URL of an intersection frame as the query | The query image itself scores 1.0000, a similar intersection frame scores 0.5534, and the least relevant frame scores 0.1162 |
Structured filtering | attributes["anomaly_event"] != "none" | Matches frames in the data that genuinely contain abnormal events |
In image-to-frame search, the similarity of the query image to itself is 1.0000, which can be used to verify the consistency of image encoding.
Structured filtering supports string equality and numeric comparison. For example, attributes["lane_count"] >= 4 filters out frames with four or more lanes. When combining conditions, note that if the value of a JSON field is null, the != condition does not match that row, which may cause long-tail samples to be silently missed. This is also the reason why a default value is specified for each label in Step 3.
The combined conditions for corner case mining must match the actual content of your material library. For example, scene == "tunnel" and attributes["anomaly_event"] != "none" requires both a "tunnel scene" and "an abnormal event". If no such frame exists in the material library, the query normally returns 0 records. An empty result means that the current library contains no such long-tail scenario, which is itself a valid conclusion of corner case mining rather than a malfunction. During debugging, we recommend that you first use a single condition to confirm that the pipeline works, and then gradually add conditions to narrow the scope.
Step 6: Multimodal reranking with AI_RERANK (optional)
Vector similarity measures semantic closeness, which is not entirely equivalent to "how relevant a result is to the query intent". You can use qwen3-vl-rerank to perform secondary fine-ranking on the recalled frames to improve the ranking quality of long-tail scenarios. Two usages are provided: attach a ranker in search, or rerank an existing batch of frame URLs independently through the REST interface.
# ===== Multimodal reranking with AI_RERANK (optional) =====
QUERY = "Cut-in behavior at a highway ramp"
# Method 1: attach a ranker in search, so the top N vector-recalled results are fine-ranked by qwen3-vl-rerank
reranker = Function(
name="rerank_frames", function_type=FunctionType.RERANK,
input_field_names=["frame_url"],
params={"reranker": "model", "provider": "aliyun_milvus",
"model_name": "qwen3-vl-rerank", "queries": [QUERY],
"is_multimodal": "true",
"instruct": "Rank candidate frames by relevance to the query, "
"prioritizing subject, action, scene and fine-grained visual details.",
"timeout_sec": 10})
rerank_search = client.search(
collection_name=collection_name, data=[QUERY], anns_field="embedding",
limit=20, output_fields=["frame_url", "scene"], ranker=reranker)
for hit in rerank_search[0]:
print(f"rerank_score={hit['distance']:.4f} scene={hit['entity']['scene']}")
# Method 2: rerank only an already recalled batch of frame URLs through REST /v2/vectordb/ai/rerank
rest_docs = [f["frame_url"] for f in frames[:3]]
status, data = post_json("/v2/vectordb/ai/rerank", {
"model_name": "qwen3-vl-rerank", "query": QUERY,
"documents": rest_docs,
"params": {"is_multimodal": True,
"instruct": "Rank candidate frames by relevance to the query, "
"prioritizing subject, action, scene and fine-grained visual details.",
"timeout_sec": 10}})
assert status == 200 and data.get("code") == 0, data
for item in sorted(data["data"]["output"]["results"],
key=lambda x: x["relevance_score"], reverse=True):
print(f"index={item['index']} relevance_score={item['relevance_score']:.4f}")Reranking results measured for the query "Cut-in behavior at a highway ramp":
Rank | Frame scene | Rerank score |
1 | Highway | 0.5898 |
2 | Construction zone | 0.5018 |
3 | Tunnel | 0.4858 |
4 | Intersection | 0.4830 |
5 | Intersection | 0.4089 |
6 | Normal road | 0.3819 |
The highway frame ranks first, which matches the query semantics. The two invocation methods give exactly the same score for the same frame, so you can choose based on engineering needs: use a ranker when you want "recall + fine-ranking" completed in one step, and use the REST interface when you already have a candidate frame list and only want to rerank it.
Multimodal reranking requires is_multimodal in params, and you can use instruct to specify the ranking emphasis (for example, prioritizing subject, action, scene, and fine-grained visual details). The documents of the REST interface is an array of frame URL strings, and top_n is not supported, so a score is returned for every candidate.
Solution value
Dimension | Traditional self-built solution | Milvus AI Function |
Development cycle | Requires joint debugging of multiple systems: frame extraction, inference, vector database, and metadata database | Create the collection once with functions attached, then use it directly for writes and retrieval |
Inference OM | Self-built GPU inference cluster requiring scaling and failure recovery | Invocation is managed by Milvus, with zero O&M on the business side |
Data flow | Frames are moved multiple times between object storage, the inference cluster, and the vector database | Inference on write; data never leaves the Milvus instance |
Multimodal retrieval | Requires building your own alignment between the text and image vector spaces |
|
Labeling criteria | Manual labeling is slow and costly, with inconsistent judgment criteria |
|
For intelligent-driving teams, the direct value of this solution is:
Faster corner case mining: With text-to-frame search plus structured filtering, further enhanced by multimodal reranking, long-tail scenarios can be retrieved with a single natural-language sentence.
Lower automatic labeling costs:
AI_CLASSIFY,AI_EXTRACT, andAI_ENTITY_EXTRACTproduce labels with unified criteria at write time, so humans only review and refine them.Converged pipeline: Inference is consolidated into the vector database. Writing performs vectorization and structured understanding, retrieval performs multimodal semantic recall, and driving data never leaves the Milvus instance throughout the process.