MaxCompute:StarRocks connector

Syntax

CREATE EXTERNAL CATALOG <catalog_name>
[COMMENT <comment>]
PROPERTIES
(
    "type" = "odps",
    CatalogParams,
    ScanParams,
    CachingMetaParams
)

Parameters

Example

The following example creates a catalog named odps_catalog for the MaxCompute project mf_mc_bj.

CREATE EXTERNAL CATALOG odps_catalog
PROPERTIES (
    "type"              = "odps",
    "odps.access.id"    = "<yourAccessKeyId>",
    "odps.access.key"   = "<yourAccessKeySecret>",
    "odps.endpoint"     = "http://service.cn-beijing.maxcompute.aliyun.com/api",
    "odps.tunnel.quota" = "pay-as-you-go",
    "odps.project"      = "mf_mc_bj"
);

Replace the placeholder values:

View catalogs

List all catalogs in the current StarRocks cluster:

SHOW CATALOGS;

Expected output:

+-----------------+----------+------------------------------------------------------------------+
| Catalog         | Type     | Comment                                                          |
+-----------------+----------+------------------------------------------------------------------+
| default_catalog | Internal | An internal catalog contains this cluster's self-managed tables. |
| odps_catalog    | Odps     | NULL                                                             |
+-----------------+----------+------------------------------------------------------------------+

View the creation statement for a specific catalog:

SHOW CREATE CATALOG odps_catalog;

View the schema of a MaxCompute table

View the column definitions of a table:

-- Syntax
DESC[RIBE] <catalog_name>.<database_name>.<table_name>;

-- Example
DESC odps_catalog.mf_mc_bj.src;

Expected output:

+-------+------+------+-------+---------+-------+
| Field | Type | Null | Key   | Default | Extra |
+-------+------+------+-------+---------+-------+
| id    | INT  | Yes  | false | NULL    |       |
| a     | INT  | Yes  | false | NULL    |       |
+-------+------+------+-------+---------+-------+

View the schema and creation statement of a table:

-- Syntax
SHOW CREATE TABLE <catalog_name>.<database_name>.<table_name>;

-- Example
SHOW CREATE TABLE odps_catalog.mf_mc_bj.src;

Delete a catalog

DROP CATALOG odps_catalog;

Collect statistics for the cost-based optimizer (CBO)

StarRocks does not automatically collect statistics for MaxCompute tables. Run the following command to trigger manual collection:

-- Replace <table_name> with the name of the MaxCompute table.
ANALYZE TABLE <table_name>;

The cost-based optimizer (CBO) uses statistical information to estimate execution costs and select the most efficient query plan. For details, see Gather statistics for CBO.

Refresh cached metadata

By default, StarRocks caches MaxCompute metadata to improve query performance. If you change a table schema or update data, run the following command to force a metadata refresh:

-- Replace <table_name> with the name of the MaxCompute table.
REFRESH EXTERNAL TABLE <table_name>;

"Your slot quota is exceeded" error

This error occurs when the number of concurrent read slots exceeds the quota of your Data Transmission Service resource group. When StarRocks reads data, it splits the table into shards and distributes them to backend (BE) nodes, which read data in parallel. The number of occupied slots is approximately Max(Number of shards, Number of BEs × Concurrency of the BE thread pool).

To resolve this, try one or more of the following:

• Purchase an exclusive resource group for Data Transmission Service that supports higher concurrency. See Purchase and use an exclusive resource group for Data Transmission Service.

• Reduce the number of shards by adjusting ScanParams. If you use the row_offset sharding policy, increase the value of odps.split.row.count to merge smaller shards.

• Modify the number of threads in the StarRocks BE thread pool by changing scanner_thread_pool_thread_num or scanner_thread_pool_queue_size. See StarRocks configuration.