DataWorks, MaxCompute, and Hologres form a Data Lakehouse platform that unifies batch and real-time workloads, reducing data analysis cycles from days to minutes or seconds.
What is a Data Lakehouse?
Traditional data architectures force a choice between two models:
|
Architecture |
Strengths |
Limitations |
|
Data warehouse |
Structured data, fast SQL queries, strong governance |
Expensive at scale, rigid schema, no streaming support |
|
Data lake |
Low-cost storage, flexible formats, supports ML workloads |
Poor query performance, weak governance, no ACID guarantees |
Data Lakehouse merges both models: warehouse-grade query performance and governance with lake-level cost efficiency and flexibility. One storage layer serves batch and real-time workloads without separate systems.
Challenges of a dual-stack approach
Most enterprises run two separate technology stacks -- one for batch processing (Hive, Spark) and another for real-time streams (Flink, Kafka). This dual-stack approach creates four problems:
|
Challenge |
Description |
|
Architecture fragmentation |
Two separate stacks increase development and operational costs and make cross-system data consistency difficult. |
|
Delayed insights |
Warehouse data is unavailable for ad hoc queries until the next batch cycle completes — often hours or a full day. Correlating real-time events with large historical datasets is especially hard. |
|
Low resource efficiency |
Reserving capacity for peak batch and real-time loads leads to low utilization and high TCO. |
|
Staffing overhead |
Operating two separate big data systems requires a large, highly skilled team. |
Architecture
The platform follows a four-stage data flow, from ingestion through unified analytics.
Stage 1: Unified data ingestion and layering
Data Integration ingests data from multiple source types into a unified cloud data lake or data warehouse:
|
Source type |
Examples |
|
Structured databases |
MySQL, PostgreSQL, Oracle |
|
Log files |
Application logs, access logs |
|
Real-time message queues |
Kafka, other streaming sources |
Ingested data follows a standard layering model (ODS, DWD, DWS, ADS), so one copy serves both batch and real-time computing — eliminating data silos at the source.
Stage 2: Batch processing
MaxCompute SQL nodes in Data Studio handle large-scale processing. The scheduling system automatically runs ETL tasks daily after midnight, processing TB-to-PB-scale historical data for:
-
Decision analysis
-
User profiling
-
Machine learning
Stage 3: Real-time and near-real-time computing
The platform supports two latency tiers:
|
Processing mode |
Engine |
Latency |
Use cases |
|
Real-time |
Realtime Compute for Apache Flink (Flink SQL nodes) |
Milliseconds |
Real-time risk control, live dashboards, real-time recommendations |
|
Near-real-time (ad hoc) |
Hologres |
Seconds |
Interactive drill-downs, self-service exploration via BI tools |
Hologres runs interactive queries on massive datasets in the data lake or data warehouse. Business analysts and operations staff can perform multi-dimensional drill-downs on the latest data directly, without waiting for scheduled reports.
Stage 4: Integrated analytics and unified services
Hologres directly accelerates queries on MaxCompute data, enabling federated analysis across real-time and historical datasets without duplicating data between systems.
DataService Studio packages analysis results into standard APIs, providing a single data service endpoint for:
-
Business applications
-
BI reports and dashboards
-
Downstream systems
Component summary
|
Component |
Role |
Connects to |
|
Data Integration |
Ingests batch and streaming data from external sources |
MaxCompute, Hologres |
|
MaxCompute |
Stores and batch-processes historical data (TB/PB scale) |
Hologres (for accelerated queries) |
|
Hologres |
Runs real-time interactive queries on both live and historical data |
MaxCompute, DataService Studio |
|
Flink SQL |
Processes data streams with millisecond latency |
Hologres, MaxCompute |
|
Data Studio |
Development environment for authoring and scheduling SQL nodes |
MaxCompute, Flink SQL |
|
DataService Studio |
Exposes query results as standard APIs |
Business applications, BI tools |
Benefits
|
Benefit |
Details |
|
Lower TCO |
One storage layer, one development platform, and multiple compute engines cut development and operational complexity, lowering TCO by over 50%. |
|
Faster time to insight |
Data analysis cycles drop from days to minutes or seconds, shifting decisions from periodic reviews to real-time insights. |
|
Self-service analytics |
High-performance interactive queries let business users explore data independently, reducing ad hoc requests to analysts. |
|
Data-driven innovation |
A unified real-time data foundation powers user behavior analysis, precision marketing, financial risk control, and intelligent supply chains. |
Customer case study
Financial services: A data lakehouse implementation at an Internet finance company