Community

Blog Events Webinars Tutorials Forum
  1. Blog
  2. Events
  3. Webinars
  4. Tutorials
  5. Forum
Create Account
×
Community Blog Alibaba Cloud EMR Serverless Spark AI Function Multimodal & Autonomous Driving Practice

Alibaba Cloud EMR Serverless Spark AI Function Multimodal & Autonomous Driving Practice

EMR Serverless Spark enables hybrid SQL retrieval in data lake for multimodal AI scenarios.

Alibaba Cloud E-MapReduce (EMR), an open-source big data platform, is a fully managed cloud-native big data platform built on the open-source ecosystem. It is deeply compatible with Hadoop, Spark, StarRocks, Presto, Hudi, Iceberg, and Paimon, and provides three flexible deployment models: Serverless, on ECS, and on ACK.

About Alibaba Cloud E-MapReduce: Serverless Open-Source Big Data Platform

Overview

This article provides a detailed introduction to best practices for multimodal processing with Alibaba Cloud EMR Serverless Spark AI Function. Focusing on autonomous driving scenarios, it demonstrates through a real-world intelligent driving data preprocessing case how multimodal AI Function can automatically identify complex road conditions, such as roundabouts, and long-tail special vehicles, such as police cars and crane trucks. This solution transforms the traditional high-cost, labor-intensive annotation process into an efficient automated ETL pipeline. It not only enables second-level analysis of high-resolution images, but also verifies accuracy in handling non-standard visual features through structured outputs, offering a highly valuable reference for intelligent data-cleaning engines in the autonomous driving data closed loop.

AI Function Expands Multimodal Capabilities

EMR Serverless Spark AI Function greatly lowers the barrier to using large models by natively integrating LLM inference capabilities into the compute engine in the form of SQL functions. In the latest esr-4.7.0 and later versions (Spark 3.5.2), AI Function has comprehensively expanded support for multimodal input. This is mainly reflected in the multimodal usage of ai_query and the new ai_embedding_multimodal function.

ai_query: Several Modes for Image Understanding

Mode 1: Read image content from a DLF Blob field This is suitable for scenarios where image binary data is stored directly in the table.

CREATE DATABASE ai_dataset;
CREATE TABLE ai_dataset.test_blob (
    path STRING,
    data BINARY
) TBLPROPERTIES (
    'row-tracking.enabled' = 'true',
    'data-evolution.enabled' = 'true',
    'blob-field' = 'data'
);

INSERT INTO ai_dataset.test_blob
SELECT path, content FROM read_files("oss://bucket_name/images/ccpd_1m_100/", suffix => "jpg");

SELECT path, 
  ai_query(
  'Please describe the main content of the image',
  service_name => 'qwen3.6-plus',
  data => data
  ) 
  FROM ai_dataset.test_blob;

Mode 2: Path column + URI reading When images are stored in OSS, OSS-HDFS, or other Hadoop-compatible file systems, and the table only stores the file paths, you can specify data_type => 'uri'. The function will read the file internally by path and perform inference. This mode is suitable for existing datasets where image paths are already stored in a table.

-- License plate recognition: batch inference from a path table
SELECT
  path,
  ai_query(
    'Return the license plate number in the image. IMPORTANT: only reply with the plate information; if the image is blurry and cannot be recognized, reply "Unrecognizable".',
    service_name => 'qwen3.6-plus',
    data => path,
    data_type => 'uri',
    options => '{"enable_thinking": false}'
  ) AS plate_number
FROM (
  SELECT /*+ REPARTITION(10) */ path
  FROM ccpd_1m_1000
) t;
  • data => path means the path column is passed in.
  • data_type => 'uri' means the function reads the file internally by path, supporting OSS and file paths. For file paths, Spark-managed mounting must be used, for example file:///mnt/data/images/xxx.jpg.
  • If data_type => 'binary', the function accepts binary image content as well as http(s) URLs.
  • The number of partitions is recommended to be controlled in the inner subquery via the REPARTITION hint, rather than directly in the outer projection containing the AI function.
  • If options.enable_thinking is not set, the default is false.

Mode 3: read_files + binary reading By combining the read_files function, images can be batch-read directly from a directory, and the binary content (BINARY type) can be used as multimodal input. This removes the need to create a path table in advance, making it suitable for batch processing newly added data.

-- Directly read images in bulk from an OSS directory and run inference
SELECT
  path,
  ai_query(
    'Please describe the main content of the image',
    service_name => 'qwen3.6-plus',
    data => content
  ) AS description
FROM read_files(
  'oss://bucket/path/to/images/',
  suffix => 'jpg,jpeg,png'
);

Tip: read_files supports the suffix parameter for extension filtering. The filter is pushed down to the file scan stage, avoiding reading other large files in the directory first and significantly reducing irrelevant file scans and downstream AI request volume. If recursive reading of subdirectories is needed, you can add recursive => true.

ai_embedding_multimodal: Image Vectorization

For image-to-image search and multimodal retrieval scenarios, the ai_embedding_multimodal function can efficiently convert images into vectors (embeddings), laying the foundation for subsequent vector search and similarity computation.

Method 1: Path column + URI reading

SELECT
  path,
  ai_embedding_multimodal(
    path,
    service_name => 'tongyi-embedding-vision-plus',
    data_type => 'uri'
  ) AS embedding
FROM image_paths_table;

Method 2: Directly pass a binary column and write vectors into DLF Paimon

create table embedding_result(
  path string, 
  embedding array<double>) 
using paimon;

insert overwrite embedding_result
SELECT
  path,
  ai_embedding_multimodal(
    content,
    service_name => 'tongyi-embedding-vision-plus'
  ) AS embedding
FROM read_files('oss://bucket/path/to/images/', suffix => 'jpg,png');

The image embeddings generated by ai_embedding_multimodal can be written directly into vector databases such as Milvus to build multimodal retrieval systems that support advanced scenarios like cross-modal text-image search.

Support for Mainstream Models: Open Ecosystem, Flexible Integration

EMR Serverless Spark AI Function not only has built-in Qwen series models, but also supports integration with mainstream external models through a unified model service registration mechanism, including PAI-EAS, DeepSeek, KIMI, GLM, MiniMax, and others, covering today’s most popular LLM ecosystems.

In terms of integration, the product provides two core paths to cover all scenarios from rapid validation to enterprise-grade production:

Quick Validation: Connect to Alibaba Cloud Model Studio (out-of-the-box) For teams that want to quickly validate business scenarios, perform PoC verification, or explore data, Alibaba Cloud Model Studio is the most efficient choice. Users do not need to worry about model deployment, autoscaling, or API authentication details; they only need to obtain the key from the Model Studio console and can then directly call cloud-hosted large models in SQL. This “zero-code, zero-ops” experience reduces PoC time from weeks to hours.

Enterprise Customization: Connect to PAI-EAS (secure and controllable) For industries such as finance and healthcare, where data privacy is extremely sensitive, or for enterprises with self-developed fine-tuned models, PAI-EAS (Model Online Service) provides a solid foundation. Users can deploy private models on PAI-EAS and connect them to EMR Serverless Spark through the VPC internal network. Data never needs to leave the secure VPC environment; inference is completed directly inside the lakehouse, meeting strict compliance requirements while benefiting from the extreme performance of Serverless Spark elastic computing.

Model Access List

The following models are built into Serverless Spark and can be called directly with AI Function without additional registration:

Model Service Name Model Name
qwen3.6-plus qwen3.6-plus
qwen3.5-plus qwen3.5-plus
qwen-plus qwen-plus
text-embedding-v4 text-embedding-v4
tongyi-embedding-vision-plus tongyi-embedding-vision-plus

Practice Case: Image Understanding for Multimodal Intelligent Driving Data

Scenario Background

In the R&D and testing of autonomous driving and intelligent transportation systems, road test vehicles generate dashcam video data every day. Traditional processing workflows often rely on manual annotation teams to label video frames frame by frame, which is not only costly and time-consuming, but also difficult to handle long-tail scenarios such as rare special vehicles and complex road structures.

This case requires a smart driving team to quickly analyze a batch of raw images collected by a front camera, with the goal of automatically identifying whether they contain the following key elements:

  • Special road condition: roundabout, which tests the perception algorithm’s understanding of non-linear roads.
  • Special vehicles: police car, pickup truck, crane/lift truck.

Multimodal Solution Based on Spark AI Functions

We use the ai_query function integrated in EMR Serverless Spark to invoke the multimodal capabilities of the Qwen (Qwen3.6-plus) large model. The entire process is simplified into a standard ETL pipeline: read OSS files -> define prompts in SQL -> call the AI function -> output structured results.

img

Code Implementation

-- Define a prompt variable containing detailed visual feature descriptions
WITH vars AS (
  SELECT '''
# Please determine which of the following labels the image belongs to: Roundabout | Police Car | Pickup Truck | Crane/Lift Truck | None of the Above

* Roundabout
Visual features: There is a circular intersection directly in front of the vehicle, and there are white guide arrows on the road surface or roadside.

* Police Car
Visual features: The vehicle body has clearly visible words such as "公安", "警察", or "POLICE", or official police insignia, and the body livery matches a standard police car style (such as black-and-white, blue-and-white, etc.).

* Pickup Truck
Visual features: A light truck with a separate cab and an open cargo bed at the rear.

* Crane/Lift Truck
Visual features: A heavy-duty special-purpose engineering vehicle with a large folding crane arm or a tall telescopic ladder mounted on the chassis.

# Do not include any explanatory text in the output. Format: ["Label 1", "Label 2", ...]
''' AS prompt
)

-- Run visual inference
SELECT 
  t.path, 
  ai_query(
    v.prompt, 
    data => t.content,
    service_name => "qwen3.6-plus"
  ) as result
FROM read_files('oss://baobao-wlcb/camera_front/', 'jpg,png') t
CROSS JOIN vars v;

Results and Case Analysis

After the task was submitted, EMR Serverless Spark automatically and elastically scaled compute resources, completing the analysis of five high-resolution intelligent driving images within seconds.

img

Input data overview We prepared five typical test images in the oss://baobao-wlcb/camera_front/ directory, covering different levels of difficulty:

  • Image 1: Output labels: ["Police Car", "Pickup Truck"]

img

  • Image 2: Output labels: ["Roundabout"]

img

  • Image 3: Output labels: ["Crane/Lift Truck"]

img

  • Image 4: Output labels: ["Pickup Truck"]

img

  • Image 5: Output labels: [] (corresponding to “None of the Above”)

img

Summary

Through this case, we verified the advantages of EMR Serverless Spark AI Functions in intelligent driving data preprocessing:

  • Out-of-the-box, no coding required, and multiple times more efficient: the hours-long process of “download data -> write Python scripts -> deploy environment -> run inference” is reduced to minutes of “write SQL -> submit and run.”
  • High-throughput batch processing with no bandwidth pressure: deeply optimized for large-scale datasets, throughput is significantly better than UDF solutions. The embedding function supports automatically merging multiple rows into a single LLM request, significantly reducing latency.
  • Production-grade stability guarantees: built-in rate limiting with automatic backoff, timeout retries, service exception circuit breaking, and other mechanisms mean users do not need to handle exception scenarios themselves, ensuring stable completion of large-scale data processing tasks.
  • Unified model service management: users do not need to manage credentials in code and can flexibly switch between different models; the platform centrally configures model service endpoints and keys for security and compliance.
  • Extreme elastic cost efficiency: the Serverless architecture means no charges are incurred when there are no tasks. Billing is based on the actual number of images processed and compute time, making it ideal for intermittent data processing needs.

In the future, this architecture can be extended to video-level behavior analysis, such as “Is the driver fatigued?” and “Are pedestrians crossing illegally?”, becoming an indispensable intelligent cleaning engine in the autonomous driving data closed loop.

Contact Us

Want to see how EMR powers elastic, fully managed big data analytics with Spark, StarRocks, and lakehouse-native AI? 👉 Try EMR on Alibaba Cloud or talk to our solution architect to explore how you can build data lakes, real-time warehouses, and Data + AI workflows with seamless scalability.

More Resources：

E-MapReduce Document

E-MapReduce Serverless Spark Free Trial:1000 CU*H 3 months !

E-MapReduce Big Data DataLake Big Data and AI EMR Serverless Spark AI Function
0 0 0
Share on

Read previous post:

Hybrid Scalar-Vector Retrieval with Alibaba Cloud EMR Serverless Spark

Alibaba Cloud Big Data and AI

22 posts | 0 followers

You may also like

Comments

Alibaba Cloud Big Data and AI

22 posts | 0 followers

Related Products

More Posts by Alibaba Cloud Big Data and AI

See All