The MaxCompute metadata mapping feature in DataWorks lets you query MaxCompute data at Hologres speed without copying it. DataWorks creates Hologres foreign tables that reference your MaxCompute tables in place, so queries run against Hologres while the data stays in MaxCompute.
How it works
DataWorks supports three levels of metadata mapping, each using a different Hologres mechanism:
| Mapping level | Hologres object created | Underlying mechanism | Metadata refresh |
|---|---|---|---|
| Project | External database (same name as MaxCompute project) | CREATE EXTERNAL DATABASE | Real-time — changes in MaxCompute are reflected automatically |
| Schema | Foreign tables for all tables in a schema | Auto Load | One-time — delete the schema and re-create the mapping to refresh |
| Table | Single foreign table | CREATE FOREIGN TABLE | One-time — delete the foreign table and re-create the mapping to refresh |
Choose the mapping level based on your scope:
Project-level: Map an entire MaxCompute project at once. Use when you want real-time metadata sync across all tables in a project.
Schema-level: Map all tables in a specific schema. Use when you need a scoped subset and one-time creation is acceptable.
Table-level: Map a single table with full control over the foreign table name, schema path, and field mappings. Use when you need fine-grained configuration.
For background on accelerating queries on MaxCompute data using Hologres, see Accelerated queries on MaxCompute data.
Limits
Only MaxCompute internal projects support metadata mapping to Hologres.
Only Hologres V3.0.7 and later support this feature.
MaxCompute metadata can only be mapped to Hologres exclusive instances.
MaxCompute and Hologres support different data types, and some types cannot be mapped. Review Data type mappings between MaxCompute and Hologres before proceeding.
Prerequisites
Before you begin, make sure you have:
A Hologres exclusive instance running V3.0.7 or later
Super user permissions on the target Hologres instance
Read and write permissions on the source MaxCompute tables
(For schema-level mapping) The schema feature enabled on the MaxCompute project — see Schema-related operations
(If using a DataWorks data source) The O&M or Workspace Administrator role in the workspace
Map a project
Project-level mapping creates an external database in Hologres with the same name as the MaxCompute project. The mapping is live — any changes to the MaxCompute project metadata are reflected in Hologres automatically.
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select the target region. Find your workspace and click Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane of the Data Studio page, click the
icon to open the DATA CATALOG pane.In the MaxCompute directory, find the project to map. Click the
icon, then select Metadata Mapping to Hologres.On the Metadata Mapping to Hologres tab, configure the parameters. For Instance Search Method, select one of the following:
From DataWorks Data Sources: Select a Hologres data source associated with the current workspace. Configure the Data Source parameter. The access identity for the data source must have super user permissions on the Hologres instance and read and write permissions on the source MaxCompute tables. > Note: Only users with the O&M or Workspace Administrator role can select data sources added in the production environment.
I Am Authorized: Select a Hologres instance accessible under the current Alibaba Cloud account. Configure the Hologres Instance parameter. Make sure you have super user permissions on the instance and read and write permissions on the source MaxCompute tables.
MaxCompute source
Parameter Description Source Object Type Fixed as MaxCompute Project.Source Object Name Fixed as the name of the current MaxCompute project. Hologres destination
Parameter Description Instance Search Method How to select the target Hologres instance. See the options below. External Database Name of the external database to create in the Hologres exclusive instance. The MaxCompute project metadata is mapped to this external database in real-time. Delete the external database to stop the mapping. Click Run in the top toolbar. DataWorks executes CREATE EXTERNAL DATABASE to create the external database and map the MaxCompute project metadata to it.
Map a schema
Schema-level mapping creates Hologres foreign tables for all tables in a MaxCompute schema. This is a one-time operation — metadata is not automatically refreshed after creation. To refresh, delete the schema in Hologres and re-create the mapping.
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select the target region. Find your workspace and click Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane of the Data Studio page, click the
icon to open the DATA CATALOG pane.In the MaxCompute directory, find the schema to map. Click the
icon, then select Metadata Mapping to Hologres.On the Metadata Mapping to Hologres tab, configure the parameters. For Instance Search Method, select one of the following:
From DataWorks Data Sources: Select a Hologres data source associated with the current workspace. Configure the Data Source and Schema parameters. The access identity must have read and write permissions on the source MaxCompute tables and the Hologres instance. > Note: Only users with the O&M or Workspace Administrator role can select data sources added in the production environment.
I Am Authorized: Select a Hologres instance accessible under the current Alibaba Cloud account. Configure the Hologres Instance, Database, and Schema parameters. Make sure you have read and write permissions on the source MaxCompute tables and the Hologres instance.
MaxCompute source
Parameter Description Source Object Type Fixed as MaxCompute Schema.Source Object Name Fixed as the name of the current MaxCompute schema. Hologres destination
Parameter Description Instance Search Method How to select the target Hologres instance. See the options below. Database An existing database in the Hologres instance. Schema Name of the schema to create in the Hologres exclusive instance. The MaxCompute schema metadata is mapped to this Hologres schema. Click Run in the top toolbar. DataWorks uses the Auto Load feature to create Hologres foreign tables and map the MaxCompute schema metadata.
Map a table
Table-level mapping creates a single Hologres foreign table for a MaxCompute table. You control the foreign table name, schema path, field mappings, and partition configuration. Like schema-level mapping, this is a one-time operation — delete the foreign table and re-create the mapping to refresh metadata.
Go to the Workspaces page in the DataWorks console. In the top navigation bar, select the target region. Find your workspace and click Shortcuts > Data Studio in the Actions column.
In the left-side navigation pane of the Data Studio page, click the
icon to open the DATA CATALOG pane.In the MaxCompute directory, find the table to map. Right-click the table name and select Metadata Mapping to Hologres.
On the Metadata Mapping to Hologres tab, configure the parameters. For Instance Search Method, select one of the following:
From DataWorks Data Sources: Select a Hologres data source associated with the current workspace. Configure the Data Source, Schema, and External Table parameters. The access identity must have read and write permissions on the source MaxCompute table and the Hologres instance. > Note: Only users with the O&M or Workspace Administrator role can select data sources added in the production environment.
I Am Authorized: Select a Hologres instance accessible under the current Alibaba Cloud account. Configure the Hologres Instance, Database, Schema, and External Table parameters. Make sure you have read and write permissions on the source MaxCompute table and the Hologres instance.
MaxCompute source
Parameter Description Source Object Type Fixed as MaxCompute Table.Source Object Name Fixed as the name of the current MaxCompute table. Hologres destination
Parameter Description Instance Search Method How to select the target Hologres instance. See the options below. Database An existing database in the Hologres instance. Schema An existing schema in the specified database. External Table Name of the foreign table to create in the Hologres exclusive instance. Field Configure field mappings and partition settings on the Mapping Field and Partition Configurations tabs. Action for Table Name Conflicts What to do if a table with the same name already exists: Skip and Create No Table, Overwrite, Delete Existing Table and Create Another Table, or Report Error. Click Run in the top toolbar. DataWorks executes CREATE FOREIGN TABLE to create the Hologres foreign table and map the MaxCompute table metadata.
What's next
After the mapping completes:
View the foreign tables: In the DATA CATALOG pane, open the Hologres directory to see the created foreign tables.
Run a query: Create a Hologres SQL node in Data Studio and query the foreign table to verify the mapping. For example:
SELECT * FROM <schema_name>.<foreign_table_name> LIMIT 10;Replace
<schema_name>and<foreign_table_name>with the actual Hologres schema and foreign table names.