Enterprise data is often scattered across different systems: data warehouses, data lakes, and online analytical databases each stand on their own, while business computations frequently need to bring them together. In the past, cross-system processing usually meant multiple sets of connection configurations, staging tables for data movement, and a string of interdependent tasks.
The multi-Catalog capability of Alibaba Cloud EMR Serverless Spark changes this premise. By connecting multiple Catalogs to a workspace, a single Spark Session can access data spaces such as DLF, Hive Metastore, MaxCompute, Hologres, StarRocks, Paimon, and Iceberg at the same time, converging cross-system reads, joins, and writes into one Spark SQL statement.
Starting from a real cross-Catalog processing scenario, this article introduces the scope of Catalog support in Serverless Spark and how it simplifies data pipelines and the development experience.
Suppose a data team needs to produce a daily user operations report. Order details and membership levels are stored in separate data spaces; each table can be queried independently, but the report needs to join and compute them together.
The actual business logic is not complex: filter the day's orders, join the membership dimension, and write the results into a new detail table. The trouble usually appears before the SQL runs — preparing multiple connections, moving data, creating staging tables, and then stringing the pipeline together with several tasks.
EMR Serverless Spark multi-Catalog brings these data spaces into the same Spark Session. The order table, the membership table, and the result table can appear directly in the same Spark SQL, and developers only need to finish the join and write logic for the report.
As Catalog support expands, a Spark Session can directly access more and more data. Table names still follow the same format:
<catalog>.<database>.<table>New Catalogs extend Spark's data scope, and existing business SQL can keep working as-is.
Currently, Catalogs such as DLF, Hive Metastore, Hologres, and StarRocks can be added directly in the workspace's Data Catalog. The Data Catalog is the unified entry point for displaying and managing Catalogs in a workspace, and it does not affect the availability of the Catalogs themselves. This entry point will continue to expand.
Serverless Spark also supports MaxCompute, Paimon, Iceberg, and custom Catalogs. The complete support scope includes:
In the workspace's Data Catalog, select a Catalog type and fill in the connection information to complete the setup. Once connected, tasks and sessions in the workspace can all reuse it. Business code references stable logical table names, while connection addresses, network, and authentication information are managed centrally by the platform. If a Catalog type cannot yet be added in the Data Catalog, you can also inject access yourself through the spark.sql.catalog configuration.
Common data sources and lake format capabilities such as Paimon and Iceberg are built in. Developers can skip finding packages, packaging, uploading JARs, and resolving ClassPath conflicts — open Spark SQL and start writing business logic.
A single Spark Session can see tables like these at the same time:
dlf_prod.retail.orders
hms_warehouse.ods.user_events
odps.retail_dw.dwd_order_di
holo_realtime.public.dim_user_level
sr_analytics.ads.product_report
paimon_curated.ads.user_pay_detail
iceberg_archive.history.closed_ordersThese tables can appear directly in the same query. The Catalog name indicates where the data comes from, while the filters, joins, aggregations, window functions, and table writes that follow still use standard Spark syntax.
Writing the daily user operations report from the beginning as SQL: paid orders come from the MaxCompute Catalog, membership levels come from the Hologres Catalog, and the results are written to a Paimon Catalog. The target Paimon table has already been created, and Spark handles the read, join, and write directly:
INSERT INTO paimon_curated.ads.user_pay_detail
SELECT
'${bizdate}' AS dt,
o.order_id,
o.user_id,
COALESCE(u.member_level, 'NORMAL') AS member_level,
o.pay_amount
FROM odps.retail_dw.dwd_order_di o
LEFT JOIN holo_realtime.public.dim_user_level u
ON o.user_id = u.user_id
WHERE o.ds = '${bizdate}'
AND o.order_status = 'PAID';The three Catalogs odps, holo_realtime, and paimon_curated together make up a single Spark SQL statement. There are no sync tables or data movement scripts in between — the entire pipeline submits just one Spark job.
If the membership table is migrated to StarRocks, HMS, or DLF, you only need to replace the fully qualified table name, and the existing business logic keeps working. The target table can also be switched to a DLF-managed Paimon or Iceberg table, with reading, computing, and writing still handled by Spark.
Tables from different Catalogs enter the same Spark logical plan. With the full SQL in view, Spark can arrange data scans, predicate pushdown, partition pruning, joins, and shuffles, and then write the results directly to the target Catalog.
Developers no longer need to split a piece of business logic into multiple client scripts or chain several compute tasks together with temporary tables. Code, resources, and runtime status are all consolidated in a single Spark job.
The Serverless platform provisions compute resources on demand, Spark UI shows the complete execution process, and Driver and Executor logs are retained in one place. Cross-Catalog jobs offer the same development, tuning, and troubleshooting experience as ordinary Spark jobs.
Multi-Catalog lets a single Spark Session access more data spaces. Developers keep using their existing SQL and DataFrame code to complete cross-Catalog reads, computations, and writes.
As the Data Catalog continues to expand, more Catalogs will be available directly in the workspace. Users still work with the familiar Spark jobs, Spark UI, and runtime logs.
Want to see how EMR powers elastic, fully managed big data analytics with Spark, StarRocks, Ray, and lakehouse-native AI? 👉 Try EMR on Alibaba Cloud or talk to our solution architect to explore how you can build data lakes, real-time warehouses, and Data + AI workflows with seamless scalability.
More Resources：
E-MapReduce Serverless Spark Free Trial:1000 CU*H 3 months !
30 posts | 0 followers
FollowAlibaba EMR - August 5, 2024
Alibaba EMR - November 14, 2024
Alibaba Cloud Big Data and AI - July 6, 2026
Alibaba Cloud Big Data and AI - April 24, 2026
Alibaba Cloud Big Data and AI - July 9, 2026
Alibaba EMR - February 15, 2023
30 posts | 0 followers
Follow
Big Data Consulting for Data Technology Solution
Alibaba Cloud provides big data consulting services to help enterprises leverage advanced data technology.
Learn More
Big Data Consulting Services for Retail Solution
Alibaba Cloud experts provide retailers with a lightweight and customized big data consulting service to help you assess your big data maturity and plan your big data journey.
Learn More
Quick BI
A new generation of business Intelligence services on the cloud
Learn More
ApsaraDB for HBase
ApsaraDB for HBase is a NoSQL database engine that is highly optimized and 100% compatible with the community edition of HBase.
Learn MoreMore Posts by Alibaba Cloud Big Data and AI