全部產品
Search
文件中心

MaxCompute:Append表 - Hash Cluster(邀測)

更新時間:Sep 02, 2026

MaxCompute在Append Delta Table表格式中進一步支援Hash Cluster,在支援增量資料處理的同時提升查詢效能。本文介紹其與其他表類型的差異、建表文法、SQL及資料通道 SDK 使用樣本。

適用情境

推薦在以下情境使用 Hash Cluster:

  • 等值過濾查詢:按固定列做點查或等值過濾,減少掃描量。

  • 等值JOIN / GROUP BY:多表按相同KEY關聯或彙總,減少 Shuffle。

與相似表類型對比

表類型

聚簇方式

增量寫入(ACID)

Hash Clustered Table

Hash

不支援

有 Hash Clustering(Shuffle + Sort)最佳化,但不支援 ACID。

PK Delta Table

Hash

支援

有 ACID 能力與 Hash Clustering 最佳化,適用於有主鍵資料。讀寫效能弱於無主鍵的表。

Append Delta Table - Range Cluster

Range

支援

有 ACID 能力與重聚簇支援,但聚簇方式為 Range,寫入效能弱於 Hash。

Append Delta Table - Hash Cluster

Hash

支援

集 Hash Clustering 最佳化(Shuffle + Sort)與完整 ACID 能力於一體,同時支援後台增量與全量重聚簇。是上述三類表中綜合能力最完整的類型。

前提條件

開啟以下 Session 參數後再建表:

SET odps.table.append2.enable=true;
SET odps.table.hash.delta.enable=true; -- hash delta 建表試用開關

建表文法

CREATE TABLE [IF NOT EXISTS] <table_name>
             [(<col_name> <data_type> [comment <col_comment>], ...)]
             [PARTITIONED BY (<col_name> <data_type> [comment <col_comment>], ...)]
             CLUSTERED BY (<col_name> [, <col_name>, ...])
             [SORTED BY (<col_name> [, <col_name>, ...])] -- 僅支援升序
             INTO <number_of_buckets> BUCKETS
             TBLPROPERTIES ('table.format.version' = '2'); 

參數說明

參數

說明

CLUSTERED BY

指定 Hash 分桶列。建議選擇在等值過濾、JOIN、GROUP BY 或 WINDOW PARTITION BY 中頻繁使用的列,且該列的取值種類盡量多,以充分發揮分桶效果。

SORTED BY

可選。指定桶內排序列。當前僅支援升序排序。建議選擇範圍/等值過濾列、視窗計算或版本時間列。

INTO ... BUCKETS

指定邏輯 Bucket 數。建議結合資料量、查詢並發和分桶列基數設定。 Bucket 數量和寫表時並發度,讀表 Shuffle 最佳化時並發度相關。

table.format.version

設定為 2,即表示建立適用於Hash Cluster的表資料格式。

SQL 使用樣本

本樣本以商品狀態和價格版本表為例。業務上通常會按 item_id 點查或關聯商品,因此將 item_id 作為 Hash 分桶列;版本生效時間 event_time 用於查看歷史變化,因此將 event_time 作為桶內排序列。該設計適用於緩慢變化維表、商品價格版本表、狀態變更明細表等情境。

準備工作

SET odps.sql.type.system.odps2=true;
SET odps.table.append2.enable=true;
SET odps.table.hash.delta.enable=true;

建表

非分區表

CREATE TABLE hash_delta_sales_demo (
  item_id BIGINT,
  event_time TIMESTAMP,
  price DOUBLE,
  status STRING
)
CLUSTERED BY (item_id)
SORTED BY (event_time)
INTO 256 BUCKETS 
TBLPROPERTIES ('table.format.version' = '2');

分區表

如果資料需要按日期管理,也可以建立分區表:

CREATE TABLE hash_delta_sales_demo_pt (
  item_id BIGINT,
  event_time TIMESTAMP,
  price DOUBLE,
  status STRING
)
PARTITIONED BY (ds STRING)
CLUSTERED BY (item_id)
SORTED BY (event_time)
INTO 256 BUCKETS
TBLPROPERTIES ('table.format.version' = '2');

通過DESC EXTENDED hash_delta_sales_demo;查看錶資訊。表的分桶和排序定義如下:

ClusterType:              hash
BucketNum:                256
ClusterColumns:           [item_id]
SortColumns:              [event_time ASC]

增量寫入

以下以非分區表為例,示範增量寫入和重聚簇過程。

  • 初始寫入資料

    INSERT INTO TABLE hash_delta_sales_demo VALUES
      (1001, TIMESTAMP '2026-05-01 10:00:00', 10.00, 'active'),
      (1001, TIMESTAMP '2026-05-03 10:00:00', 13.00, 'active'),
      (1002, TIMESTAMP '2026-05-01 11:00:00', 20.00, 'active');
    
    DESC EXTENDED hash_delta_sales_demo;

    單擊查看執行結果樣本

    關鍵字段:

    • DataPhysicalClustered: true -- 資料已按 item_id 物理分桶

    • DataFullySorted: true -- 桶內資料已按 event_time 全量排序

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:39:38                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 4823                                               |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |  |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             14469                                                    |
    | FileNum:                  5                                                        |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          true                                                     |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:39:38                                  |
    +------------------------------------------------------------------------------------+
  • 刪除操作

    刪除一部分資料後查看錶狀態:

    DELETE FROM hash_delta_sales_demo WHERE item_id = 1002;
    
    DESC EXTENDED hash_delta_sales_demo;

    單擊查看執行結果樣本

    關鍵字段:

    • DataPhysicalClustered: true -- 資料已按 item_id 物理分桶

    • DataFullySorted: true -- 刪除操作不破壞已有資料的排序狀態

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:41:02                                      |
    | LastAccessTime:           2026-07-08 16:40:57                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 7082                                               |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |  |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             21246                                                    |
    | FileNum:                  10                                                       |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          true                                                     |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:41:02                                  |
    +------------------------------------------------------------------------------------+
    
  • 補寫歷史資料

    補寫一條歷史版本。該版本的 event_time 位於 item_id=1001 已有時間範圍中間:

    INSERT INTO TABLE hash_delta_sales_demo VALUES
      (1001, TIMESTAMP '2026-05-02 09:00:00', 12.00, 'active');
    
    DESC EXTENDED hash_delta_sales_demo;

    單擊查看執行結果樣本

    關鍵字段:

    • DataPhysicalClustered: true -- 資料已按 item_id 物理分桶

    • DataFullySorted: false -- 新增檔案使桶內資料不再全量有序

    +------------------------------------------------------------------------------------+
    | Owner:                    ALIYUN$***_com                                           |
    | Project:                  test                                                     |
    | TableComment:                                                                      |
    +------------------------------------------------------------------------------------+
    | CreateTime:               2026-07-08 16:38:34                                      |
    | LastDDLTime:              2026-07-08 16:38:34                                      |
    | LastModifiedTime:         2026-07-08 16:42:54                                      |
    | LastAccessTime:           2026-07-08 16:40:57                                      |
    +------------------------------------------------------------------------------------+
    | InternalTable: YES      | Size: 10705                                              |
    +------------------------------------------------------------------------------------+
    | Native Columns:                                                                    |
    +------------------------------------------------------------------------------------+
    | Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
    +------------------------------------------------------------------------------------+
    | item_id  | bigint |       |               | true     | NULL         |              |
    | event_time | timestamp |       |               | true     | NULL         |              |
    | price    | double |       |               | true     | NULL         |              |
    | status   | string |       |               | true     | NULL         |              |
    +------------------------------------------------------------------------------------+
    | Extended Info:                                                                     |
    +------------------------------------------------------------------------------------+
    | TableID:                  65**8e                                                   |
    | IsArchived:               false                                                    |
    | PhysicalSize:             32115                                                    |
    | FileNum:                  13                                                       |
    | ColdStorageStatus:        N/A                                                      |
    | CompressionStrategy:      normal                                                   |
    | DataFullySorted:          false                                                    |
    | DataPhysicalClustered:    true                                                     |
    | IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
    | OverlapDepth:             2                                                        |
    | OverlapRatio:             1.000000                                                 |
    | StoredAs:                 AliOrc                                                   |
    | Transactional:            true                                                     |
    | encryption_enable:        false                                                    |
    | odps.timemachine.retention.days: 1                                                        |
    | ClusterType:              hash                                                     |
    | BucketNum:                256                                                      |
    | ClusterColumns:           [item_id]                                                |
    | SortColumns:              [event_time ASC]                                         |
    | StorageTier:              Standard                                                 |
    | StorageTierLastModifiedTime:  2026-07-08 16:42:54                                  |
    +------------------------------------------------------------------------------------+
    
    

Bucket Pruning

對分桶列做等值查詢時,MaxCompute 會根據 Hash 分布直接定位目標 Bucket,跳過其餘 Bucket 的掃描。以下查詢按 item_id = 1001 過濾,實際唯讀取該值所在的 1 個邏輯Bucket,而非全表掃描:

SELECT * FROM hash_delta_sales_demo
  WHERE item_id = 1001
  ORDER BY event_time
  LIMIT 10;

-- 返回結果:
+------------+---------------------+------------+--------+
| item_id    | event_time          | price      | status |
+------------+---------------------+------------+--------+
| 1001       | 2026-05-01 10:00:00 | 10.0       | active |
| 1001       | 2026-05-02 09:00:00 | 12.0       | active |
| 1001       | 2026-05-03 10:00:00 | 13.0       | active |
+------------+---------------------+------------+--------+

全量重聚簇

如果需要重新整理存量資料,執行 RECLUSTER FULL。該操作會保留表中的歷史資料語義,並按照當前表定義重新組織儲存資料。

ALTER TABLE hash_delta_sales_demo RECLUSTER FULL;

DESC EXTENDED hash_delta_sales_demo;

單擊查看執行結果樣本

關鍵字段:

  • DataPhysicalClustered: true -- 資料已按 item_id 物理分桶

  • DataFullySorted: true -- 桶內資料已全量排序

+------------------------------------------------------------------------------------+
| Owner:                    ALIYUN$***_com                                           |
| Project:                  test                                                     |
| TableComment:                                                                      |
+------------------------------------------------------------------------------------+
| CreateTime:               2026-07-08 16:38:34                                      |
| LastDDLTime:              2026-07-08 16:38:34                                      |
| LastModifiedTime:         2026-07-08 16:42:54                                      |
| LastAccessTime:           2026-07-08 16:40:57                                      |
+------------------------------------------------------------------------------------+
| InternalTable: YES      | Size: 20218                                              |
+------------------------------------------------------------------------------------+
| Native Columns:                                                                    |
+------------------------------------------------------------------------------------+
| Field    | Type   | Label | ExtendedLabel | Nullable | DefaultValue | Comment      |
+------------------------------------------------------------------------------------+
| item_id  | bigint |       |               | true     | NULL         |              |
| event_time | timestamp |       |               | true     | NULL         |              |
| price    | double |       |               | true     | NULL         |              |
| status   | string |       |               | true     | NULL         |              |
+------------------------------------------------------------------------------------+
| Extended Info:                                                                     |
+------------------------------------------------------------------------------------+
| TableID:                  65**8e                                                   |
| IsArchived:               false                                                    |
| PhysicalSize:             60654                                                    |
| FileNum:                  20                                                       |
| ColdStorageStatus:        N/A                                                      |
| CompressionStrategy:      normal                                                   |
| DataFullySorted:          true                                                     |
| DataPhysicalClustered:    true                                                     |
| IsolationMin:             NONSTRICT_SNAPSHOT_ISOLATION                             |
| OverlapDepth:             2                                                        |
| OverlapRatio:             1.000000                                                 |
| StoredAs:                 AliOrc                                                   |
| Transactional:            true                                                     |
| encryption_enable:        false                                                    |
| odps.timemachine.retention.days: 1                                                        |
| ClusterType:              hash                                                     |
| BucketNum:                256                                                      |
| ClusterColumns:           [item_id]                                                |
| SortColumns:              [event_time ASC]                                         |
| StorageTier:              Standard                                                 |
| StorageTierLastModifiedTime:  2026-07-08 16:42:54                                  |
+------------------------------------------------------------------------------------+

Hash Delta 表支援 INSERT、UPDATE、DELETE、MERGE INTO 等增量寫入,同時保留 Hash 分布資訊。最佳化器基於當前資料狀態選擇執行計畫:資料有序時利用有序儲存,資料變為非全量有序後仍可利用 Hash 分桶,必要時通過RECLUSTER FULL 恢複全量排序。

資料通道使用樣本

以上文中 hash_delta_sales_demo 表為例,介紹通過Data Transmission Service SDK 進行 Hash Clustered Delta Table 的資料上傳/下載。

  1. 匯入SDK依賴包

    建議使用最新版本,至少需要升級到0.59版本及以上。詳情參見版本更新記錄

  2. 範例程式碼

    單擊查看範例程式碼

    /**
     * 使用 MaxStorageClient 對 hash_delta_sales_demo 表進行上傳和下載的範例程式碼。
     *
     * 表結構:
     * CREATE TABLE hash_delta_sales_demo (
     *   item_id BIGINT,
     *   event_time TIMESTAMP,
     *   price DOUBLE,
     *   status STRING
     * );
     */
    public class MaxStorageClientExample {
    
        private static final String ENDPOINT = "<your-endpoint>";
        private static final String TUNNEL_ENDPOINT = "<your-tunnel-endpoint>";
        private static final String PROJECT = "<your-project>";
        private static final String ACCESS_ID = "<your-access-id>";
        private static final String ACCESS_KEY = "<your-access-key>";
        private static final String TABLE_NAME = "hash_delta_sales_demo";
    
    
        public static void main(String[] args) throws Exception {
            RootAllocator allocator = new RootAllocator(Long.MAX_VALUE);
    
            // 1. 構建 MaxStorageClient
            MaxStorageClient client = MaxStorageClient.builder()
                    .endpoint(ENDPOINT)
                    .tunnelEndpoint(TUNNEL_ENDPOINT)
                    .credentialsProvider(
                            new StaticCredentialProvider(new AliyunAccount(ACCESS_ID, ACCESS_KEY).getCredentials()))
                    .project(PROJECT)
                    .bufferAllocator(allocator)
                    .build();
    
            try {
                // 2. 上傳資料
                uploadData(client);
                Thread.sleep(5000);
    
                // 3. 下載資料
                downloadData(client);
            } finally {
                client.close();
                allocator.close();
            }
        }
    
        /**
         * 上傳資料到 hash_delta_sales_demo 表
         */
        private static void uploadData(MaxStorageClient client) throws Exception {
            TableIdentifier tableId = TableIdentifier.of(PROJECT, TABLE_NAME);
    
            // 建立寫入會話,withOverwrite(true) 表示覆蓋寫入
            TableWriteSession writeSession = client
                    .createTableWriteSessionBuilder(tableId)
                    .withOverwrite(true)
                    .build();
    
            System.out.println("Write session created: " + writeSession.getId());
    
            // 使用 RecordWriter 寫入資料(進階 API,按行寫入)
            try (RecordWriter writer = writeSession.createWriterBuilder("stream-1", 1)
                    .build()
                    .getAsRecordWriter(1024)) {
    
                for (int i = 0; i < 1000; i++) {
                    Record record = writer.newRecord(false);
                    record.set(0, (long) i);                                    // item_id: BIGINT
                    record.set(1, LocalDateTime.of(2025, 5, 18, 10, 30, i % 60).atZone(ZoneId.systemDefault())
                            .toInstant()); // event_time: TIMESTAMP
                    record.set(2, 99.9 + i * 0.1);                             // price: DOUBLE
                    record.set(3, i % 2 == 0 ? "paid" : "pending");            // status: STRING
                    writer.write(record);
                }
            }
    
            // 提交會話,資料對外可見
            writeSession.commit();
            System.out.println("Successfully uploaded 1000 records to " + TABLE_NAME);
        }
    
        /**
         * 從 hash_delta_sales_demo 表下載資料
         */
        private static void downloadData(MaxStorageClient client) throws Exception {
            TableIdentifier tableId = TableIdentifier.of(PROJECT, TABLE_NAME);
    
            // 建立讀取會話,可以選擇列和過濾條件
            TableReadSession readSession = client.createTableReadSessionBuilder(tableId)
                    .withColumns(Arrays.asList("item_id", "event_time", "price", "status"))
                    .withSplitOptions(SplitOptions.newBuilder()
                            .withSplitMode(SplitMode.ROW_OFFSET)
                            .build())
                    .build();
    
            System.out.println("Read session created: " + readSession.getId());
    
            // 擷取資料分區
            List<InputSplit> splits = readSession.getSplits();
            System.out.println("Total splits: " + splits.size());
    
            int totalRecords = 0;
    
            // 遍曆每個分區讀取資料
            for (InputSplit split : splits) {
                try (ArrowReader reader = readSession.createReaderBuilder(split).build()) {
                    Schema schema = reader.getSchema();
                    System.out.println("Schema: " + schema);
    
                    while (reader.nextBatch()) {
                        VectorSchemaRoot root = reader.getCurrentValue();
                        int rowCount = root.getRowCount();
                        totalRecords += rowCount;
    
                        // 列印前 5 行作為樣本
                        int printCount = Math.min(rowCount, 5);
                        for (int i = 0; i < printCount; i++) {
                            System.out.printf("  item_id=%s, event_time=%s, price=%s, status=%s%n",
                                    root.getVector("item_id").getObject(i),
                                    root.getVector("event_time").getObject(i),
                                    root.getVector("price").getObject(i),
                                    root.getVector("status").getObject(i));
                        }
                        if (rowCount > 5) {
                            System.out.println("  ... (" + (rowCount - 5) + " more rows in this batch)");
                        }
                    }
                }
            }
    
            System.out.println("Total records downloaded: " + totalRecords);
        }
    }

常見問題

如何選擇合適的單 Bucket 儲存量?

建議將單 Bucket 儲存量控制在數百 MB 到數十 GB 之間。

  • Bucket 過小:儲存開銷上升,Shuffle 代價增大。

  • Bucket 過大:寫表時間較長,查詢時 Bucket Pruning 的過濾效果下降, Shuffle 最佳化效果下降。

建議根據預期資料增長量(而非當前量)來設定 Bucket 數,避免頻繁修改表結構。

如果業務資料量確實偏大,單 Bucket 也可支援更大儲存,也可以將 Bucket Num 設定的更大,但需結合實際寫入與查詢效能綜合評估。