Boost MaxCompute Query Speed with Hologres CTP - Hologres

Limitations

Hologres can only read the latest snapshot of a MaxCompute Delta Table.
Writing data from Hologres back to a MaxCompute Delta Table is not supported.

How it works

When you submit a query against a MaxCompute foreign table or external table, Hologres routes the request through the configured pathway. The Common Table pathway reads data directly from MaxCompute storage files using a C++ native reader, bypassing the SQE/HQE engine. This enables higher throughput and supports table types that the SQE/HQE pathway does not handle.

Feature comparison

Feature	SQE/HQE pathway	Common Table pathway
Performance	Strong	Very high. 33% faster than SQE/HQE; more than 200% faster than reading via the open Storage API.
Dynamic data masking	Rules must be reconfigured in Hologres.	Inherits masking rules from MaxCompute. Query results match MaxCompute query results.
Schema Evolution	Not supported. Does not support direct file reads.	Supported. Supports high-performance direct file reads.
Encryption at rest (KMS key)	Supported	Supported
Encryption at rest (default key)	Not supported	Supported. Requires Hologres V3.2 or later.
ACID 1.0 tables	Not supported	Supported
PK Delta Table	Not supported	Supported
Append Delta Table	Not supported	Supported

Prerequisites

Before you begin, make sure you have:

A Hologres instance running V3.0.38 or later
MaxCompute permissions to access the target project and tables (see Manage user permissions using commands)

Required permissions

The Common Table pathway enforces the same permissions as MaxCompute. Starting from Hologres V4.0, permission requirements changed to align with MaxCompute query permissions.

Access method	Operation	V3.0–V3.2 permissions	V4.0+ permissions
Foreign table method	Batch create tables with IMPORT FOREIGN SCHEMA	Describe, List	Describe, List
Foreign table method	Query data	Describe, Select	Select
External database method	View table metadata	Describe	Describe
External database method	Query data	Describe, Select	Select

Enable the Common Table pathway

Set the hg_experimental_external_catalog_routing parameter to 'odps:common_table,dlf:hqe'. Apply the setting at the scope that fits your needs.

Note

Session-level settings must be executed in the same statement as a query or DML statement.

Session level

SET hg_experimental_external_catalog_routing = 'odps:common_table,dlf:hqe';

Database level

ALTER DATABASE <database_name> SET hg_experimental_external_catalog_routing = 'odps:common_table,dlf:hqe';

Instance level

ALTER ROLE ALL SET hg_experimental_external_catalog_routing = 'odps:common_table,dlf:hqe';

Verify the active pathway

Run the following query to check which pathway is active:

SHOW hg_experimental_external_catalog_routing;

If the result is odps:common_table,dlf:hqe, the Common Table pathway is active.

To check which pathway was used for a past query, look up the query_extinfo field in the slow query log. If the value contains driver:CommonTable, the query used the Common Table pathway.

Disable the Common Table pathway

To revert to the SQE/HQE pathway, set the parameter to 'odps:holo_native,dlf:hqe'.

Session level

SET hg_experimental_external_catalog_routing = 'odps:holo_native,dlf:hqe';

Database level

ALTER DATABASE <database_name> SET hg_experimental_external_catalog_routing = 'odps:holo_native,dlf:hqe';

Instance level

ALTER ROLE ALL SET hg_experimental_external_catalog_routing = 'odps:holo_native,dlf:hqe';

Configure Auto Split Size

Auto Split Size automatically determines the optimal data split size based on the instance type. It is disabled by default.

The parameter name differs by Hologres version:

Version	Parameter
V4.0 and later	`hg_experimental_enable_maxcompute_sdk_auto_split_size`
Earlier than V4.0	`hg_experimental_enable_common_table_auto_split_size`

Use 'on' to enable and 'off' to disable. Apply the setting at session, database, or instance level using the same scope commands as the pathway setting.

V4.0 and later

Enable

-- Session level
SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'on';

-- Database level
ALTER DATABASE <database_name> SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'on';

-- Instance level
ALTER ROLE ALL SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'on';

Disable

-- Session level
SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'off';

-- Database level
ALTER DATABASE <database_name> SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'off';

-- Instance level
ALTER ROLE ALL SET hg_experimental_enable_maxcompute_sdk_auto_split_size = 'off';

Earlier than V4.0

Enable

-- Session level
SET hg_experimental_enable_common_table_auto_split_size = 'on';

-- Database level
ALTER DATABASE <database_name> SET hg_experimental_enable_common_table_auto_split_size = 'on';

-- Instance level
ALTER ROLE ALL SET hg_experimental_enable_common_table_auto_split_size = 'on';

Disable

-- Session level
SET hg_experimental_enable_common_table_auto_split_size = 'off';

-- Database level
ALTER DATABASE <database_name> SET hg_experimental_enable_common_table_auto_split_size = 'off';

-- Instance level
ALTER ROLE ALL SET hg_experimental_enable_common_table_auto_split_size = 'off';

Feature updates

Feature	Supported versions
JSON data type in MaxCompute	>=4.1.1
Predicate pushdown for equality filters on Delta Table attributes (clustering keys, primary keys)	>=4.0.6
Append Delta Tables	>=3.0.40, >=3.1.10, >=3.2.0, >=4.0.0
spillRead for Delta Tables — resolves the following error: `MaxCompute csdk open reader failed: storage/formats/orcfile/change_log_reader_util.cpp(255): InvalidArgument: ODPS-0010000:InvalidArgument:Need spill read but TmpDir or TmpCapability no specified.`	>=3.0.41, >=3.1.10, >=3.2.0, >=4.0.0
TIMESTAMP_NTZ data type	>=3.0.41, >=3.1.10, >=3.2.0, >=4.0.0
Tables with columns that have default values	>=3.0.40, >=3.1.10, >=3.2.0, >=4.0.0
Auto Split Size	>=3.1.33, >=3.2.0, >=4.0.0