MaxCompute provides a data lakehouse solution that breaks down the barriers between data lakes and data warehouses. This solution combines the flexibility and rich ecosystem of data lakes with the enterprise-grade deployment capabilities of data warehouses, which lets you build an integrated data management platform.
Applicability
Data Lakehouse Solution 1.0 is available only in the following regions: China (Hangzhou), China (Shanghai), China (Beijing), China (Zhangjiakou), China (Shenzhen), China (Hong Kong), Singapore, and Germany (Frankfurt).
Implementation methods
The MaxCompute data lakehouse solution integrates data warehouses and data lakes. The following implementation methods are supported:
Build a data lakehouse using MaxCompute, Data Lake Formation, and Object Storage Service: The metadata (schema) of the data lake is stored in Data Lake Formation (DLF). MaxCompute uses the metadata management capabilities of DLF to improve data processing for semi-structured formats in Object Storage Service (OSS). Supported formats include Delta Lake, Hudi, AVRO, CSV, JSON, PARQUET, and ORC.
Build a data lakehouse using MaxCompute and Hadoop: This method supports deployments in on-premises data centers, on cloud-based virtual machines, or through Alibaba Cloud E-MapReduce. After the VPCs for the MaxCompute and Hadoop platforms are connected, MaxCompute can directly access the Hive metadata service. The metadata is then mapped to an external project in MaxCompute.
References
After an external project is created, the account that created the project owns the tables within it. To grant permissions to other users, see Users and permissions for external projects.
For information about how to manage external projects using SQL, see Manage external projects using SQL.
Spark on MaxCompute supports access to external data sources in a data lakehouse. To change the environment for data processing jobs from Spark to MaxCompute, see Access external data sources in a data lakehouse from Spark.
For information about distributed file system and data storage solutions, see Process unstructured data using MaxCompute external volumes.