To seamlessly integrate with your data lake, Hologres V3.0 and later enables you to create external tables for Paimon data sources in Data Lake Formation (DLF). Building on this, Hologres V4.0 introduces external dynamic tables, which consolidate all three steps — data transformation, target table creation, and write-back to the data lake — into a single table definition. The table automatically sets up Paimon external tables, incrementally processes data, and writes results back to the data lake, so you can build efficient lakehouse pipelines without orchestrating multiple separate jobs.
Architecture
The following diagram shows how an external dynamic table reads from source tables, incrementally processes only new or changed data, and writes the results back to Paimon in the data lake.
Benefits
Unified data pipeline
A single external dynamic table handles data transformation, target table creation in the data lake, and processed data write-back. This eliminates the need to orchestrate multiple separate jobs.
Efficient incremental writes
Incremental refresh mode processes and writes only new or changed data to the data lake. Each refresh cycle touches less data, consuming fewer compute resources and completing faster than a full refresh.
Reduced costs with serverless
External dynamic tables run on serverless resources with pay-as-you-go pricing. Compute resources are allocated only during a refresh cycle and released immediately when it completes, so you pay nothing for idle time. For details, see Introduction to serverless instances.
Use cases
Cost-effective, near real-time data lake queries
Use a Paimon table in your data lake as the source. The external dynamic table incrementally transforms the data and writes it back to Paimon — the data never leaves the data lake. Query the processed results through a Hologres external table for near real-time insights.
Build a data lakehouse
Use a Hologres or MaxCompute table as the source. The external dynamic table transforms the data and automatically writes the results to Paimon for centralized storage. This reduces storage costs, facilitates data processing, and bridges your data warehouse and data lake into a unified lakehouse.