Hologres:MaxCompute

Hologres vs. MaxCompute

Foreign tables vs. internal tables

• Foreign tables do not store data. They access data directly from MaxCompute. Without indexes, they rely solely on CPU for computation. Use foreign tables for small datasets with low QPS (queries per second).

• Internal tables store data within Hologres. For data updates, complex queries, or high-QPS scenarios, import data into internal tables to leverage Hologres performance.

Missing SELECT privilege on a MaxCompute table

You have NO privilege 'odps:Select' on {table}

or:

You have NO privilege 'MaxCompute:Select' on {table}

Cause: Your account lacks SELECT permission on the MaxCompute table.

Solution: Ask a MaxCompute administrator to grant SELECT privilege on the table. For details, see MaxCompute permissions.

Missing SELECT privilege on a cross-project table

You have NO privilege 'odps:Select' on {table}

This error also appears when you query a MaxCompute table in a different project, even if your account has permissions through package-based access control.

Cause: Hologres needs to know which MaxCompute project context to use for authorization in cross-project scenarios.

Solution: Set the current project name before your query:

-- For Hologres V0.8+:
set hg_experimental_odps_current_project_name = 'holoprojectname';

-- For Hologres V0.7:
set seahawks.seahawks_internal_current_odps_project = 'holoprojectname';

Column-level sensitive label mismatch

The sensitive label of column '{column}' is 2, but your effective label is 0

Cause: Your account has permissions on only some columns in the MaxCompute table. Hologres versions earlier than V0.8 did not fully support column-level permissions.

Solution: Choose one of the following:

• Upgrade to V0.8 or later (recommended). These versions handle column-level permissions properly.

• Modify your query to include only columns your account can access.

• Request permissions for all columns. See MaxCompute permissions.

• Workaround for older versions: Set the following GUC (Grand Unified Configuration) parameters before your query:

set hg_experimental_enable_odps_executor = on;
set hg_experimental_enable_query_master = on;

If you are on a newer version and still see this error, it may be a defect. Use this alternative workaround:

set hg_experimental_enable_MaxCompute_executor = on;
set hg_experimental_enable_query_master = on;

Missing LIST privilege (HoloWeb or DataStudio)

You have NO privilege 'odps:List' on {project}

Cause: Creating a foreign table through HoloWeb or DataStudio requires List permission in MaxCompute to display available tables.

Solution:

• Ask a MaxCompute administrator to grant List permission. See MaxCompute permissions.

• Alternatively, create the foreign table directly with a CREATE FOREIGN TABLE SQL statement, which does not require List permission. See Accelerate queries of MaxCompute data based on foreign tables.

Access denied by IP whitelist

Access denied by project ip white list

Cause: The target MaxCompute project has an IP whitelist enabled, and the HoloWeb server IP address (sourceIP in the error message) is not on that list.

Solution: Add the sourceIP from the error message to the IP whitelist of the target MaxCompute project.

Account not in MaxCompute project

You don't exist in project {project}

Cause: Your account has not been added as a member to the specified MaxCompute project.

Solution: Verify the project name in the error is correct. If it is, ask a MaxCompute administrator to add your account to the project. See Permission overview.

Missing encryption role authorization

query next from foreign table executor failed validate userinfo

Cause: Hologres was not granted the AliyunHologresEncryptionDefaultRole role. This error can also appear intermittently due to caching if the authorization was granted less than three hours ago.

Solution: Grant the AliyunHologresEncryptionDefaultRolePolicy policy to the account. See Query encrypted MaxCompute data.

Not an ORC file

status { code: SERVER_INTERNAL_ERROR message: "hos_exception: Invalid argument: not an ORC file" }

Cause: Hologres foreign tables cannot directly access MaxCompute tables that are being stream-loaded. The data is not yet in ORC (Optimized Row Columnar) format.

Solution: Add the following GUC parameters before your SQL statement:

set hg_experimental_enable_access_odps_with_table_api = on;
set hg_experimental_enable_access_odps_orc_via_holo = off;

ORC schema mismatch for DECIMAL type

Open ORC file failed for schema mismatch. Reader schema

Cause: The MaxCompute table is in ORC format, and the storage method for its DECIMAL type has changed. This typically happens when a new DECIMAL field is added or a canary configuration change is made in MaxCompute.

Solution:

• Run the following command in MaxCompute and re-import the data:

set MaxCompute.storage.orc.enable.binary.decimal = false;

• Or change the DECIMAL type to DOUBLE in the MaxCompute table and refresh the data.

Timestamp overflow

Timestamp overflow detected while converting timestamp from orc VectorBatch to arrow

Cause: The MaxCompute table contains a TIMESTAMP type. After data loading through Tunnel, the precision becomes nanoseconds. Hologres does not support nanosecond-precision timestamps.

Solution:

• Convert the TIMESTAMP type to DATETIME in MaxCompute.

• Or upgrade your Hologres instance to V1.1.70 or later.

Schema evolution not enabled

failed to import foreign schema: Failed to get MaxCompute table: Not enable schema evolution

Cause: The metadata of the MaxCompute table has been modified (columns added or removed), putting it into a schema evolution status that Hologres cannot read.

Solution:

1. Upgrade your Hologres instance to V1.3 or later.

2. Run IMPORT FOREIGN SCHEMA to refresh the foreign table schema after a MaxCompute table schema change.

3. If the error persists, recreate the MaxCompute table and then the foreign table.

Transactional (ACID) table not supported

failed to import foreign schema: Failed to get MaxCompute table: Not enable acid table

Cause: The MaxCompute table is a transactional table (ACID -- Atomicity, Consistency, Isolation, Durability).

Solution: Convert the table to a standard MaxCompute table. Transactional tables are not supported.

Partition limit exceeded (512)

Specified partitions count in MaxCompute table: exceeds the limitation of 512

or:

Build desc failed: Exceeds the partition limitation of 512, current match {n} partitions

Cause: By default, Hologres queries a maximum of 512 partitions from a foreign table per query.

Solution:

• Add a partition filter to your query to reduce the number of scanned partitions.

• Import the data into an internal table, which has no partition limit. See Import data from MaxCompute using SQL.

• Adjust the partition limit with a GUC parameter. The default is 512 and the maximum is 1024. Setting this value too high can degrade query performance.

-- For V1.1 and later:
set hg_foreign_table_max_partition_limit = 128;

-- For V0.10:
set hg_experimental_foreign_table_max_partition_limit = 128;

Scan size limit exceeded (200 GB)

Build desc failed: Exceeds the scan limitation of 200 GB, current scan {n} GB

Cause: By default, Hologres limits the underlying data scan to 200 GB per query. This limit applies to data scanned from the selected MaxCompute partitions, not the total stored data.

Solution:

• Add filters to reduce the partitions accessed, keeping the scanned data under 200 GB.

• Import the data into Hologres before querying. See Import data from MaxCompute using SQL.

• (Not recommended) Increase the scan limit. Replace the value with the required size in GB. Increasing this limit excessively can degrade performance and cause OOM (out-of-memory) errors.

set hg_experimental_foreign_table_max_scan_size = 400;

Resources exhausted (server busy)

Request denied, may caused by server busy

Cause: Foreign table query resources are exhausted and CPU usage has been exceeded.

Solution:

• Optimize your SQL. See Optimize query performance for MaxCompute foreign tables.

• Reduce the DOP (Degree of Parallelism). DOP controls the concurrency for reading data from a foreign table at a single execution node. The default is 256, with a range of 0--1024. Setting it too high causes OOM errors and possible instance restarts. Setting it too low degrades foreign table query performance.

-- Check the current DOP:
show hg_foreign_table_executor_max_dop;

-- Set it to half the current value (example):
set hg_foreign_table_executor_max_dop = 18;

• Import data into an internal table. Internal tables support indexes for better query performance. See Import data from MaxCompute using SQL.

Memory limit exceeded during import

Query executor exceeded total memory limitation {limit}: {used} bytes used

Cause: The query exceeded the computation memory limit. Each Hologres instance node has a standard memory limit of 64 GB, split roughly into thirds: one-third for computation, one-third for caching, and one-third for metadata.

Solution: Try these steps in order:

1. Check the execution plan. Run explain analyze <sql>; to review the row counts. If the import query involves subqueries and some tables have not been analyzed (or data has changed since the last analysis), the optimizer may choose a suboptimal join order. Run analyze <tablename>; on all involved internal and foreign tables to update statistics.

2. Reduce the batch size. When a table has many columns or large rows, a single batch may consume too much memory. Reduce the rows per batch:

      set hg_experimental_query_batch_size = 1024;  -- Default: 8192
      insert into holo_table select * from mc_table;

3. Reduce the import DOP. The hg_foreign_table_executor_max_dop parameter defaults to the number of CUs (compute units) in the instance. Set a smaller value during import:

      set hg_foreign_table_executor_max_dop = 8;
      insert into holo_table select * from mc_table;

4. Deduplicate data. If you use insert on conflict and the foreign table has excessive duplicate rows, deduplicate in MaxCompute before import to reduce memory pressure. See Merge multiple rows of data into a single row.

5. Upgrade Hologres. Starting from V1.1.24, Hologres dynamically adjusts memory by refreshing the current memory level in real time and allocating more memory for computation when available. See Instance upgrade.

6. Scale up the instance. If all other approaches fail, increase your instance resources. See Upgrade.

CFile table type not supported

query next from foreign table executor failed, GetRecordBatch() is not implemented

Cause: The MaxCompute table is a CFile-type table, which Hologres does not support through the default access path.

Solution: Add the following GUC parameter before your SQL statement:

set hg_experimental_enable_access_odps_with_table_api = on;

Streaming Tunnel data not readable

Query next from foreign table executor failed, not implemented

Cause: The MaxCompute table data was loaded through the Streaming Tunnel using tunnel.createStreamUploadSession. Hologres requires specific GUC parameters to read this data.

Solution (Hologres V1.3 and later): Add the following parameters before your query:

set hg_experimental_enable_access_odps_with_table_api = on;
set hg_experimental_enable_access_odps_orc_via_holo = off;

Workaround for versions earlier than V1.3: Stop the streaming load in MaxCompute, then merge the data:

set odps.merge.task.mode = sql;
set odps.merge.lock.expire.time = 0;
ALTER TABLE tablename [PARTITION] MERGE SMALLFILES;

MaxCompute views not supported

Build desc failed: failed to check permission: Currently not supported table type "view"

Cause: Hologres does not support MaxCompute views as foreign tables.

Metadata retrieval failure (Pangu)

Build desc failed: failed to get foreign table split: MaxCompute-0010000: System internal error - get input pangu dir meta fail

Cause: The Hologres configuration for reading from MaxCompute was not updated promptly.

Solution: Wait a few minutes and retry. If the issue persists, contact technical support.

RPC connection closed due to small files

Build desc failed: failed to get foreign table split: ERPC_ERROR_CONNECTION_CLOSED

Cause: The MaxCompute table has too many small files, causing the metadata request to exceed the 1 GB limit of the RPC (Remote Procedure Call) protocol.

Solution:

• Merge the small files in MaxCompute:

set MaxCompute.merge.task.mode = sql;
set MaxCompute.merge.lock.expire.time = 0;
ALTER TABLE <tablename> [PARTITION] MERGE SMALLFILES;

• Upgrade your Hologres instance to V0.10.21 or later, where this issue has been fixed. See Instance upgrade.

• If the data volume is small, import the data into Hologres. Contact MaxCompute technical support if the issue persists.

Encrypted MaxCompute data access failure

status { code: SERVER_INTERNAL_ERROR message: "hos_exception: IO error: Failed to execute pangu open normal file, err: PanguParameterInvalidException" }

Cause: The HQE engine cannot directly access encrypted MaxCompute data stored on Pangu.

Solution: Switch the execution engine to SQE, which can access encrypted data. Set this at the database level (applies to new connections):

ALTER DATABASE <dbname> SET hg_experimental_enable_access_odps_orc_via_holo = false;

Or at the session level:

SET hg_experimental_enable_access_odps_orc_via_holo = false;

Query slowdown after foreign table schema evolution

Cause: By default, Hologres uses HQE to accelerate foreign table queries. When the foreign table schema evolves (for example, due to MaxCompute schema changes), Hologres falls back to SQE, which is less efficient.

Solution:

1. Query hologres.hg_query_log to identify slow queries.

2. Check the engine_type field. If it shows SQE, the performance drop is caused by the engine fallback.

3. Recreate the affected foreign table with the updated schema in Hologres.

Slow foreign table queries (general)

The most effective approach is to optimize your SQL statements. See Optimize query performance for MaxCompute foreign tables.