Hologres data lake acceleration - read and write OSS data with foreign tables - Hologres

Background information

As object storage matures, data lake solutions are shifting toward cloud-native architectures. In the Alibaba Cloud lakehouse architecture, OSS serves as unified data lake storage, providing a secure, cost-effective, highly reliable, and scalable foundation.

The real-time data lake solution focuses on streaming and real-time data processing within the lakehouse architecture. Hologres supports real-time writes, updates, and analysis. By integrating with DLF, HMS, OSS, and various ecosystem capabilities, Hologres uses foreign tables to accelerate reads and writes of data in OSS without data migration. Foreign tables map fields rather than store data, reducing development and O&M costs and breaking down data silos.

The following table lists the Alibaba Cloud services used in the real-time data lake solution.

Service	Description	Reference
DLF	A fully managed service for building cloud data lakes and lakehouses. DLF provides centralized metadata management, centralized permission and security management, and convenient data ingestion and exploration capabilities.	What is Data Lake Formation (DLF) 1.0?
HMS	A core component of Apache Hive that manages metadata for Hive and Spark tables, including storage locations, table schemas, table names, column names, data types, and partition information. HMS supports Hive and Spark data queries.	Hive Metastore Server
OSS	OSS is the unified storage for cloud data lakes in DLF. It is a secure, cost-effective, and highly reliable service capable of storing large amounts of data and all types of files. OSS delivers 99.9999999999% data durability and is the de facto standard for data lake storage.	What is OSS?
OSS	OSS-HDFS (JindoFS) is a cloud-native data lake storage service that integrates with Hadoop ecosystem compute engines. It is fully compatible with HDFS APIs and POSIX, and outperforms native OSS for Hive/Spark-based ETL workloads.	What is OSS-HDFS?

Architecture

The following figure shows the recommended Hologres data lake architecture. It covers the full data lifecycle from collection and storage to management and application, providing an end-to-end lakehouse solution with flexible auto scaling.

Unified metadata management

Supports DLF and HMS metadata services.
Supports one-click mapping to external catalogs through external databases.
Supports automatic discovery and refresh of metadata changes in the data lake.
Supports creating databases and tables in the data lake through DDL statements. Paimon is supported.

Unified computing resources and pattern

Uses Dynamic Table for lakehouse data layering and processing.
Supports high-performance OLAP analysis and write-back of data lake data.
Uses virtual warehouse instances for resource elasticity and isolation.
Supports serverless tasks with zero ownership cost and pay-as-you-go billing.
Integrates with mainstream BI tools.

Support for open lakehouse formats

Supports accelerated queries of Paimon, Iceberg, Hudi, Delta, ORC, and Parquet lake tables.
Supports write-back in Paimon, Iceberg, ORC, and Parquet formats.

Usage notes

Hologres provides the following methods to map external data sources.

Mapping method	Syntax	Description	Supported data source	Version requirement	Scenario
CREATE EXTERNAL DATABASE	CREATE EXTERNAL DATABASE CREATE EXTERNAL SCHEMA CREATE EXTERNAL TABLE	Creates an external database on a Hologres instance to load metadata from an external data source. This enables centralized management of internal and external data in the lakehouse architecture.	DLF 1.0 DLF 2.0 MaxCompute	V3.0	Map an entire database and all its tables from a catalog in an external data source to Hologres.
FOREIGN TABLE	IMPORT FOREIGN SCHEMA	Creates multiple foreign tables in a Hologres schema at once to automatically map tables from an external data source.	DLF 1.0 DLF 2.0 HMS MaxCompute Hologres	V0.8	Map all tables in a database or schema of an external data source to a Hologres schema.
FOREIGN TABLE	CREATE FOREIGN TABLE	Manually creates a foreign table in Hologres to map a table or specific fields from an external data source.	DLF 1.0 DLF 2.0 HMS MaxCompute Hologres	V0.8	Map specific tables or fields to Hologres.

Table formats and file formats

Table formats

Table format	Supported version	Supported compression method
Hudi	Data reads supported in Hologres V1.3 and later	UNCOMPRESSED GZIP SNAPPY BROTLI LZ4 ZSTD LZ4_RAW None ZLIB
Delta Lake	Data reads supported in Hologres V1.3 and later	UNCOMPRESSED GZIP SNAPPY BROTLI LZ4 ZSTD LZ4_RAW
Apache Paimon	Data reads supported in Hologres V2.1 and later Data reads from lake tables and data writes to Apache Paimon append-only tables based on DLF 2.0 supported in Hologres V3.0 and later	PARQUET UNCOMPRESSED SNAPPY GZIP LZO BROTLI LZ4 ZSTD ORC NONE ZLIB SNAPPY LZO LZ4
Iceberg	Data reads from Iceberg tables of V1 and V2 based on DLF 1.0 and HMS supported in Hologres V3.0	PARQUET UNCOMPRESSED SNAPPY GZIP LZO BROTLI LZ4 ZSTD ORC NONE ZLIB SNAPPY LZO LZ4

File formats

File format	Supported version	Supported compression method
CSV	Data reads and writes supported in Hologres V1.3 and later	COMPRESSION_CODEC BZip2Codec DefaultCodec GzipCodec SnappyCodec
Parquet	Data reads and writes supported in Hologres V1.3 and later	UNCOMPRESSED GZIP SNAPPY BROTLI LZ4 ZSTD LZ4_RAW
ORC	Data reads and writes supported in Hologres V1.3 and later	None ZLIB SNAPPY
SequenceFile	Data reads and writes supported in Hologres V1.3 and later	COMPRESSION_CODEC BZip2Codec DefaultCodec GzipCodec SnappyCodec COMPRESSION_TYPE NONE RECORD BLOCK

Data type mappings

For DLF-to-Hologres data type mappings, see Data types overview.

Feature overview

In Hologres V1.1 and later, you can read ORC, Parquet, CSV, and SequenceFile data from OSS. In Hologres V1.3 and later, you can write data in these formats to OSS, and read Apache Hudi or Delta Lake tables in OSS.

Note
You can view the version of your Hologres instance on the instance details page in the Hologres console. If your instance version is earlier than V1.1, upgrade it in the console or contact the Hologres DingTalk group. To upgrade manually, see Instance upgrades. To join the DingTalk group, see How to get online support?.
In Hologres V1.3.25 and later, the DLF multi-catalog feature isolates metadata across test, development, and cross-department instances, helping ensure the security of your business. Catalog.
In Hologres V1.3.26 and later, you can read and write data in OSS-HDFS. Hologres integrates with Hadoop ecosystem compute engines to accelerate real-time analysis of data in OSS-HDFS, supporting federated queries across data lakes for big data and AI workloads.
In Hologres V2.1.0 and later, you can read Apache Paimon foreign tables. Apache Paimon is a unified lake storage platform for streaming and batch processing, supporting high-throughput writes and low-latency queries, enabling real-time data flow in data lakes. Users can integrate real-time and offline data lake processing using Apache Paimon. Apache Paimon.
In Hologres V2.2 and later, the new foreign table architecture enables HQE to read ORC and Parquet files directly with SSD-based cache acceleration, improving performance by more than 5x. You can use HMS to access data in OSS and OSS-HDFS. Access OSS data with Hive Metastore (beta).

Note
If your Hologres instance is V2.1 or earlier, contact Hologres technical support to upgrade.
In Hologres V3.0 and later, the following features are added:
- External databases support catalog-level metadata mapping for DLF and MaxCompute data sources, improving metadata and data management capabilities of data lakes. CREATE EXTERNAL DATABASE.
- External schemas and external tables allow you to create databases and tables in a DLF catalog for aggregated data write-back. CREATE EXTERNAL SCHEMA and CREATE EXTERNAL TABLE.
- High-performance writes to Apache Paimon append-only tables for data forwarding between lakes and warehouses.
- Paimon deletion vector optimization improves query performance when large deletions occur before compaction.
- Reconstructed Delta Lake readers significantly improve read performance.
- Iceberg-based data lake reads expand the data lake ecosystem.
- HMS metadata mapping accelerates data queries in EMR clusters. See Use the HMS to access data in OSS data lakes (beta).
- Enhanced security: DLF 2.0 access uses the service-linked role by default, with optional RAM role access.