Hybrid Scalar-Vector Retrieval with Alibaba Cloud EMR Serverless Spark

About Alibaba Cloud E-MapReduce (EMR), an open-source big data platform, is a fully managed cloud-native big data platform built on the open-source ecosystem.

About Alibaba Cloud E-MapReduce (EMR), an open-source big data platform, is a fully managed cloud-native big data platform built on the open-source ecosystem. It is deeply compatible with Hadoop, Spark, StarRocks, Presto, Hudi, Iceberg, and Paimon, and provides three flexible deployment models: Serverless, on ECS, and on ACK.

About Alibaba Cloud E-MapReduce: Serverless Open-Source Big Data Platform

Abstract

EMR Serverless Spark natively supports hybrid scalar-vector retrieval through DLF Global Index, enabling multi-dimensional queries in a single SQL statement. By combining the vector_search UDF with standard WHERE filters, Spark can jointly use the Lumina vector index and B-tree scalar index for efficient execution.

This approach makes access simple, removes index operations overhead, and supports serverless pay-as-you-go billing with no idle cost. Built-in Celeborn and the Fusion engine also deliver up to 3x better performance than open-source Spark. It enables in-lake search with strong consistency and low latency, and is well suited for autonomous driving, multimodal e-commerce search, RAG knowledge bases, and offline vector batch processing.

Traditionally, semantic search and structured filtering required separate systems, leading to data movement, higher cost, and consistency challenges. With EMR Serverless Spark, both can be done in one SQL query.

What Is Hybrid Scalar-Vector Retrieval, and Why Is It Hard for Traditional Architectures?

Traditional data retrieval faces a dilemma:

Pure scalar queries (WHERE weather='heavy_rain' AND speed>80): precise, but unable to find semantically similar scenarios
Pure vector queries (Top-K nearest neighbors): can find records that look similar, but cannot enforce business constraints

Take autonomous driving as an example. Engineers often need queries like:

"From historical data, find the Top-10 most similar cases where the weather is heavy rain and the road type is urban road."

Under a traditional architecture, this requires two steps: first retrieve Top-K candidates from the vector database, then filter them again in the business database. That means two queries, data movement, and possibly fewer than K valid results.

The value of hybrid retrieval is that it adds scalar filters while performing vector nearest-neighbor search, delivering results that are both similar and compliant in one step.

How EMR Serverless Spark Implements Hybrid Scalar-Vector Retrieval

Alibaba Cloud EMR Serverless Spark leverages the Global Index capability of DLF Paimon tables in the data lake to unify vector indexes and B-tree indexes within Spark SQL’s query execution framework, enabling joint execution of scalar filtering and vector nearest-neighbor retrieval.

Core Architecture

Both types of indexes can be created on the same Paimon table. When Spark executes a query, it automatically coordinates both index paths without requiring users to manage the underlying scheduling.

Building the Vector Index

To enable a vector index on a Paimon table, you only need to specify table properties during table creation. After Spark writes data, the index becomes effective automatically:

CREATE TABLE ai_dataset.scene_vectors (
    id BIGINT,
    path STRING,
    weather STRING,
    road_type STRING,
    speed_range STRING,
    embedding ARRAY<FLOAT>
) USING paimon
TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'morax.lumina-index.enabled' = 'true',
    'global-index.lumina.index-column' = 'embedding',
    'lumina.index.dimension' = '1152'
);

When Index Construction Happens

Automatic construction: When morax.* parameters are declared in TBLPROPERTIES, DLF automatically schedules index building based on the vector column in the table.
Manual trigger: If you want the index created immediately after data ingestion, or if morax.* parameters were not declared during table creation, you can manually trigger index construction.

`vector_search`: The Vector Retrieval Function in Spark SQL

Spark SQL introduces the vector_search table function, allowing vector nearest-neighbor retrieval directly in SQL:

SELECT * FROM vector_search(
    'ai_dataset.scene_vectors',
    'embedding',
    array(0.12F, 0.34F, ...),
    10
);

This query finds the Top-K records most similar to the input query vector.

Hybrid Scalar-Vector Retrieval in One Step

By combining the results of vector_search with scalar WHERE conditions, you can perform hybrid retrieval:

SELECT id, path, weather, road_type, speed_range
FROM vector_search(
    'ai_dataset.scene_vectors',
    'embedding',
    array(0.12F, 0.34F, ...),
    10
)
WHERE weather = 'heavy_rain'
  AND road_type = 'urban';

Spark uses the vector index to retrieve nearest-neighbor candidates and the B-tree index to filter scalar conditions, completing both steps collaboratively without cross-system data movement.

Practice: Autonomous Driving Scenario Recall for Training Data Preparation

Perception models often perform poorly in adverse weather conditions and need a large amount of scenario-specific data for retraining and fine-tuning. However, such scenarios make up only a tiny fraction of road-collected data, and manual frame-by-frame filtering is highly inefficient.

The following example shows how EMR Serverless Spark SQL can support the full workflow of data ingestion into the lake → hybrid retrieval → training set export.

Step 1: Ingest Road-Collected Data into the Lake, with AI-Automated Labels and Embeddings

A road-test vehicle generates large numbers of video frames every day, stored in OSS. With Alibaba Cloud EMR Serverless Spark AI Function, a single SQL statement can read images, extract labels, and generate embeddings:

CREATE TABLE IF NOT EXISTS ad_dataset.driving_scenes (
    id          BIGINT,
    path        STRING,
    weather     STRING,
    lighting    STRING,
    road_type   STRING,
    objects     ARRAY<STRING>,
    risks       ARRAY<STRING>,
    scene_tag   STRING,
    sensor_type STRING,
    embedding   ARRAY<FLOAT>
) USING paimon
TBLPROPERTIES (
    'row-tracking.enabled'               = 'true',
    'data-evolution.enabled'             = 'true',
    'morax.lumina-index.enabled'         = 'true',
    'global-index.lumina.index-column'   = 'embedding',
    'lumina.index.dimension'             = '1152',
    'global-index.btree.index-columns'   = 'weather,road_type,lighting,objects,risks,scene_tag'
);

WITH raw AS (
    SELECT
        monotonically_increasing_id() AS id,
        path,
        ai_query(
            'You are an autonomous driving data analysis assistant. '
            || 'Based on the input road scene image, output a JSON object with the following structure: '
            || '{"weather": "sunny/cloudy/rainy/snowy/foggy/other", '
            || '"lighting": "daytime/nighttime/dusk/tunnel/other", '
            || '"road_type": "urban/expressway/rural/intersection/ramp/parking_lot/other", '
            || '"objects": ["car", "pedestrian", "bicycle", "motorcycle", "bus", "truck", '
            || '"traffic_light", "traffic_sign", "cone", "construction_equipment"], '
            || '"risks": ["construction", "congestion", "occlusion", "accident_signs", '
            || '"wrong_way", "illegal_parking", "water_logging", "ice", "other"]}. '
            || 'All field values must exactly match the enum options above (case-sensitive). '
            || 'objects and risks must be arrays of strings; use empty array [ ] if none. '
            || 'Do not output any extra text, explanation, Markdown, or code block. '
            || 'Output only valid JSON.',
            data => content
        ) AS scene_json,
        ai_embedding_multimodal(content) AS embedding
    FROM read_files(
        'oss://ad-team-raw/camera_front/2025-*/',
        suffix => 'jpg,png'
    )
)
INSERT INTO ad_dataset.driving_scenes
SELECT
    id,
    path,
    get_json_object(scene_json, '$.weather')                        AS weather,
    get_json_object(scene_json, '$.lighting')                       AS lighting,
    get_json_object(scene_json, '$.road_type')                      AS road_type,
    from_json(get_json_object(scene_json, '$.objects'), 'ARRAY<STRING>') AS objects,
    from_json(get_json_object(scene_json, '$.risks'),   'ARRAY<STRING>') AS risks,
    'normal'                                                        AS scene_tag,
    'camera_front'                                                  AS sensor_type,
    embedding
FROM raw;

One INSERT statement completes three tasks: reading OSS images, automatically labeling and vectorizing them through AI functions, and writing them into a Paimon table with dual indexes. Subsequent queries can then benefit from vector and B-tree acceleration.

Step 2: Recall Target Scenarios with Hybrid Retrieval

If the perception model has a high error rate in "heavy rain + urban road" scenarios, we can retrieve similar historical cases for retraining:

# Use the embedding of a typical misjudged scene as the query vector
query_vec = spark.sql("""
    SELECT embedding
    FROM ad_dataset.driving_scenes
    WHERE path = 'oss://ad-team-raw/camera_front/2025-10-15/frame_CF_003812.jpg'
""").collect()[0]["embedding"]

vec_literal = "array(" + ",".join(f"{v}f" for v in query_vec) + ")"

# Retrieve historical cases most similar to a typical misjudged scene under rainy urban road conditions
result = spark.sql(f"""
    SELECT id, path, weather, road_type, lighting, objects, risks
    FROM vector_search(
        'ad_dataset.driving_scenes',
        'embedding',
        {vec_literal},
        500
    )
    WHERE weather   = 'rainy'
      AND road_type = 'urban'
      AND lighting  = 'nighttime'
""")

result.show(truncate=False)

The advantage of hybrid retrieval is that vector search first finds semantically similar scenes from massive data, and then B-tree indexes accurately filter by weather, road type, and time of day. The two paths work together in a single step. Traditional solutions require retrieving candidates from a vector store first and then filtering again in the business database, while also failing to guarantee the final result count.

Step 3: Batch Recall Multiple Corner Cases to Build the Training Set

Training does not require just one type of scenario. With Spark SQL batch processing, you can recall multiple adverse scenarios in one job and write them directly into the training dataset:

from functools import reduce
from pyspark.sql import DataFrame

SCENES = [
    (
        "oss://ad-team-raw/camera_front/2025-10-15/frame_CF_003812.jpg",
        {"weather": "rainy", "road_type": "urban"},
        500,
        "rainy_urban",
    ),
    (
        "oss://ad-team-raw/camera_front/2025-11-02/frame_CF_001547.jpg",
        {"weather": "foggy", "road_type": "expressway"},
        500,
        "foggy_expressway",
    ),
    (
        "oss://ad-team-raw/camera_front/2025-12-08/frame_CF_000923.jpg",
        {"weather": "snowy", "road_type": "rural", "lighting": "nighttime"},
        300,
        "snowy_rural_nighttime",
    ),
]

def recall_scene(anchor_path: str, filters: dict, top_k: int, source_label: str) -> DataFrame:
    vec = spark.sql(f"""
        SELECT embedding FROM ad_dataset.driving_scenes
        WHERE path = '{anchor_path}'
    """).collect()[0]["embedding"]

    vec_literal = "array(" + ",".join(f"{v}f" for v in query_vec) + ")"

    where_clause = " AND ".join(f"{col} = '{val}'" for col, val in filters.items())
    return spark.sql(f"""
        SELECT
            id, path, weather, lighting, road_type,
            objects, risks, scene_tag, sensor_type,
            '{source_label}' AS source_query
        FROM vector_search(
            'ad_dataset.driving_scenes',
            'embedding',
            {vec_literal},
            {top_k}
        )
        WHERE {where_clause}
    """)

frames = [recall_scene(*scene) for scene in SCENES]
result = reduce(DataFrame.unionByName, frames)

spark.sql("""
    CREATE TABLE IF NOT EXISTS ad_dataset.training_set_corner_cases (
        id          BIGINT,
        path        STRING,
        weather     STRING,
        lighting    STRING,
        road_type   STRING,
        objects     ARRAY<STRING>,
        risks       ARRAY<STRING>,
        scene_tag   STRING,
        sensor_type STRING,
        source_query STRING
    ) USING paimon
""")

result.writeTo("ad_dataset.training_set_corner_cases").append()

print(f"Total records written: {result.count()}")

Key value: one SQL workflow completes multiple scenario recalls and merges them, while the source_query field records the recall condition for each sample, making it easier to apply weighted sampling during training.

Step 4: Training Set Quality Analysis and Deduplication

After retrieval, the training set distribution needs to be analyzed and duplicates removed — all of this can be done directly in Spark SQL:

-- Analyze the distribution of recalled samples by scenario
SELECT source_query, COUNT(*) AS sample_count
FROM ad_dataset.training_set_corner_cases
GROUP BY source_query
ORDER BY sample_count ASC;

-- Join with a dimension table to count coverage by city
SELECT
    t.source_query,
    r.city,
    COUNT(*) AS sample_count
FROM ad_dataset.training_set_corner_cases t
JOIN dim_road_info r ON t.path = r.image_path
GROUP BY t.source_query, r.city
HAVING COUNT(*) > 5
ORDER BY t.source_query, sample_count DESC;

Applicable Business Scenarios

Intelligent Driving: Corner Case Mining and Data Feedback Loop

The core challenge in autonomous driving iteration is that long-tail scenarios are difficult to mine efficiently. Traditional solutions depend on manual annotation and frame-by-frame labeling, which is slow and costly.