MaxCompute:ODPS-0123031

Error: maximum 60000 partitions allowed

Error message

ODPS-0123031:ODPS partition exception - maximum 60000 partitions allowed

Cause

Each MaxCompute partitioned table can hold a maximum of 60,000 partitions. This is a hard limit and cannot be modified.

Solution

Use one of the following approaches to reduce the partition count:

• Configure a lifecycle. A lifecycle automatically deletes expired partitions and their data. See Lifecycle management operations.

• Merge historical partitions. Consolidate data from multiple historical partitions into one. See Merge partitions.

• Split the table. Back up historical partition data to a new table, then delete the historical partitions from the original table. See Delete partitions.

Error: a single instance cannot output data to more than 10000 partitions

Error message

ODPS-0123031:Partition exception - a single instance cannot output data to more than 10000 partitions

Cause

During an INSERT INTO or INSERT OVERWRITE statement, a single job can write to a maximum of 10,000 output partitions, even though a table can hold up to 60,000 partitions in total.

The most common cause is a high-cardinality partition field. For example, using an ID column as a partition field generates a distinct partition for each ID, quickly exceeding the per-job limit.

Solution

If your job outputs to thousands of dynamic partitions and the count exceeds 10,000, first check for errors in your business logic or SQL syntax. If the logic is correct, use one of the following approaches:

• Modify the partition fields. Replace high-cardinality fields with lower-cardinality fields such as date or category to reduce the number of partitions generated per job.

• Split the business logic into multiple jobs. Divide the data into smaller batches and run multiple jobs, each writing to fewer than 10,000 partitions.

Error: invalid dynamic partition value

Error message

ODPS-0123031:Partition exception - invalid dynamic partition value: province=Shanghai

Cause

The partition field contains special characters or Chinese characters, which are not valid for dynamic partitioning. Dynamic partitions are created based on the values in the specified partition field, so invalid characters in that field trigger this error.

Solution

Before inserting data into dynamic partitions, make sure your partition configuration follows these rules:

• In a distributed environment, a maximum of 512 dynamic partitions can exist in a single process.

• A single SQL statement for dynamic partitioning cannot generate more than 2,000 dynamic partitions.

• Partition values that are dynamically generated cannot be NULL.

• If the table contains multi-level partitions, any static partitions specified in an INSERT statement must be high-level partitions.

Error: java.lang.ClassNotFoundException: HiveInputFormatWrapper

Error message

ODPS-0123131:User defined function exception - internal error - Fatal Error Happended

java.lang.RuntimeException: java.lang.ClassNotFoundException: com.aliyun.odps.hive.wrapper.HiveInputFormatWrapper

Cause

Your query uses a Hive-related resolver, but the Hive-compatible data type edition is not enabled.

Solution

Run the following command to enable the Hive-compatible data type edition:

SET odps.sql.hive.compatible=true;