Resolve task run failures - - Alibaba Cloud Documentation Center

This topic covers common task failures in Proxima CE on MaxCompute, with causes and solutions for each error.

Duplicate primary keys in the seek phase

Error message

2019-11-19 22:52:51 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:52:59 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:07 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:15 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:24 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:32 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:40 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:48 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:53:56 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:1/12000/12000(+1 backups)[0%]
2019-11-19 22:54:05 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:0/12000/12000(+1 backups)[0%]
2019-11-19 22:54:14 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:0/12000/12000(+1 backups)[0%]
2019-11-19 22:54:22 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:0/12000/12000(+1 backups)[0%]
2019-11-19 22:54:30 M1_job0:0/650/650[100%]      R2_1_job0:0/8004/8000(+4 backups)[100%] R3_2_job0:0/12000/12000(+1 backups)[0%]
...
FAILED: ODPS-0123144: Fuxi job failed - WorkerRestart errCode:252,errMsg:kInstanceMonitorTimeout, usually caused by bad udf performance.
CentauriException(code=20005, msg=odps任务运行失败, detailMsg=SeekJobWorker Job running error.)
        at com.alibaba.proxima.pipeline.worker.odps.SeekJobWorker.apply(SeekJobWorker.java:90)
        at com.alibaba.proxima.pipeline.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:40)
        at com.alibaba.proxima.pipeline.CentauriPipelineExecutor.execute(CentauriPipelineExecutor.java:30)
        at com.alibaba.proxima.CentauriRunner.runWithNormal(CentauriRunner.java:35)
        at com.alibaba.proxima.CentauriRunner.main(CentauriRunner.java:208)

Most common cause: The doc table contains vectors that share the same primary key but have different values. During the Mapper-Reducer1-Reducer2 (MRR) process in the seek phase, all records with the same primary key are routed to the same reducer. If the data is not deduplicated, one reducer processes far more records than others, causing data skew that terminates the reduce task.

To diagnose the issue:

Open LogView for the failing MaxCompute task. For instructions, see Use LogView to view job information.

On the Fuxi task monitoring page, filter the failed instances and click the corresponding StdOut to view the standard output log. For example, the StdOut log for instance R3_2#48 shows the following output:

[2019-11-19 21:48:38.306215]            ---------- Run Task: R3_2#48 ----------
[2019-11-19 21:48:50.571175]    total input size is 2327967432708 bytes.
[2019-11-19 21:48:50.571175]    total input record number is 793592600.
Task ID: R3_2_000048

If the input record count for one instance is much higher than other instances, the data in your doc table has not been deduplicated.

Deduplicate the data to ensure each primary key maps to a single vector value.

GetSmallCategoryDocNum returns empty in the search preparation phase

Error message

java.lang.NullPointerException
        at com.alibaba.proxima.utils.OdpsUtil.getSmallCategoryDocNum(OdpsUtil.java:719)
        at com.alibaba.proxima.pipeline.worker.odps.SmallCategoryPrepareWorker.apply(SmallCategoryPrepareWorker.java:62)
        at com.alibaba.proxima.pipeline.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:40)
        at com.alibaba.proxima.pipeline.CentauriPipelineExecutor.execute(CentauriPipelineExecutor.java:30)
        at com.alibaba.proxima.CentauriRunner.runWithMultiCategory(CentauriRunner.java:62)
        at com.alibaba.proxima.CentauriRunner.main(CentauriRunner.java:210)

A field — typically category or pk — contains an empty value in the table. Use SQL statements to delete the records where the field is empty.

Schema validation fails in the search phase for small categories

Error message

2020-07-28 16:58:15.221 [main] INFO  p.a.p.p.ProximaCEPipelineExecutor - [] - execute SmallCategorySeek worker start ..........
[400] com.aliyun.odps.OdpsException: ODPS-0420031: Invalid xml in HTTP request body - The request body is malformed or the server version doesn't match this sdk/client. XML Schema validation failed: Element 'Value': [facet 'maxLength'] The value has a length of '4952955'; this exceeds the allowed maximum length of '2097152'.
Element 'Value': '{"MaxCompute.pipeline.0.output.key.schema":"instId:BIGINT,type:BIGINT,pk:STRING,category:BIGINT","MaxCompute.pipeline.0.output.value.schema":"v

The schema payload exceeds the 2,097,152-character limit because the doc table contains too many categories. Split the doc table into multiple tables, one per category group, and run separate tasks for each.

"-nan" appears during table data parsing

Error message

ODPS-0123131:User defined function exception — Traceback:
java.lang.NumberFormatException: For input string: "-nan"
	at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043)
	at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110)
	at java.lang.Double.parseDouble(Double.java:538)
	at java.lang.Double.valueOf(Double.java:502)
	at com.alibaba.proxima.mr.SeekMergeReducer.reduce(SeekMergeReducer.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runReducer(MapReduceUtils.java:160)
	at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run(LotReducerUDTF.java:330)
	at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run(BatchStandaloneUDTFEvaluator.java:53)

The doc or query table contains an invalid vector value — either a number too large to represent (overflow) or a value so close to zero that division fails. Examples of invalid values:

1.23~4.56~7.89~nan~4.21
1.1~2.2~127197893781729178311928739179222121.23128767846816278193456789087654~0.000000000000000000000000000000000000000001~5.5

Use MaxCompute SQL user-defined functions (UDFs) to scan both the doc and query tables for invalid values, then remove or correct them before rerunning.

JNI call exception in a multi-category search (doc table has 0 documents for a category)

Error message

2021-08-16 10:36:31 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/0/3000[0%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:36:39 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/0/3000[0%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:36:47 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/0/3000[0%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:36:56 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:351/0/3000[0%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:04 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:2766/234/3000[95%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:12 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:2766/234/3000[95%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:21 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:1431/313/3000[100%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:31 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:1323/317/3000[100%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:35 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/317/3000[99%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:43 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/317/3000[99%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:37:51 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/317/3000[99%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:38:00 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/317/3000[99%]   R3_2_job0:0/0/4500[0%]
2021-08-16 10:38:08 M1_U1_job0:0/1/1[100%]        M1_U0_job0:0/24067/24067[100%]   R2_1_job0:0/317/3000[99%]   R3_2_job0:0/0/4500[0%]
...
FAILED: ODPS-0123131:User defined function exception - Traceback:
CentauriException(code=20006, msg=jni调用异常, detailMsg=linearJni init method call exception , code:-2)
	at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.initLinearJni(SmallCategorySeekTopnReducer.java:162)
	at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.reduce(SmallCategorySeekTopnReducer.java:111)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runReducer(MapReduceUtils.java:160)
	at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run(LotReducerUDTF.java:330)
	at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run(BatchStandaloneUDTFEvaluator.java:53)
CentauriException(code=20005, msg=odps任务运行失败, detailMsg=SmallCategorySeekWorker Job running error.)
	at com.alibaba.proxima.pipeline.worker.odps.SmallCategorySeekWorker.apply(SmallCategorySeekWorker.java:116)
	at com.alibaba.proxima.pipeline.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:41)
	at com.alibaba.proxima.pipeline.CentauriPipelineExecutor.execute(CentauriPipelineExecutor.java:30)
	at com.alibaba.proxima.CentauriRunner.runWithMultiCategory(CentauriRunner.java:60)
	at com.alibaba.proxima.CentauriRunner.main(CentauriRunner.java:232)
2021-08-16 10:38:14.133 [main] INFO  c.a.p.p.CentauriPipelineExecutor - [] - execute CleanUpWorker worker start .........
2021-08-16 10:38:14.279 [main] INFO  com.alibaba.proxima.utils.OdpsUtil - [] - drop table dmp_mind_shop_user_norm_doc_table_0_tmp_1629081006420 start ...
2021-08-16 10:38:15 INFO  - DQC Hook:begin
2021-08-16 10:38:16 INFO  - getJsonSummary is null, retry 3
2021-08-16 10:38:19 INFO  - getJsonSummary is null, retry 2
2021-08-16 10:38:22 INFO  - getJsonSummary is null, retry 1
2021-08-16 10:38:25 INFO  - 该Job没有发生数据变化，所以不需要DQC校验。(Code:1)
2021-08-16 10:38:25 INFO  - DQC Hook:end
2021-08-16 10:38:25.737 [main] INFO  com.alibaba.proxima.utils.OdpsUtil - [] - drop table dmp_mind_shop_user_norm_doc_table_0_tmp_1629081006420 end ...
2021-08-16 10:38:25.893 [main] INFO  com.alibaba.proxima.utils.OdpsUtil - [] - drop table dmp_mind_shop_norm_query_table2_tmp_1629081166805 start ...
2021-08-16 10:38:27 INFO  - DQC Hook:begin
2021-08-16 10:38:27 INFO  - getJsonSummary is null, retry 3

The key error message in the StdErr log is linear params category_info is invalid, causing CentauriException(code=20006, msg=jni调用异常). The full log is as follows.

ERROR: ld.so: object '${LD_PRELOAD}' from LD_PRELOAD cannot be preloaded: ignored.
Heap Size: 1024M
native library 'libcentauri-1.1.2.so' loaded
[JVM Heap Memory] maximum = 10379526144
[JVM Heap Memory] used: 103077912, init stat info
SeekJni System.loadLibrary(centauri-1.1.2) start ...
SeekJni System.loadLibrary(centauri-1.1.2) end ...
[JVM Heap Memory] used: 137437232, after processed 1 keys, 1 values
[ERROR] 2021-08-16 18:37:06 140363264742864 linear_singleton.cc:273] linear params category_info is invalid
[ERROR] 2021-08-16 18:37:15 140363264742864 linear_singleton.cc:19] validate linear params failed.
[ERROR] 2021-08-16 18:37:15 140363264742864 linear_jni.cc:23] LinearSingleton init failed with ret -2.
Exception in thread "main" CentauriException(code=20006, msg=jni调用异常, detailMsg=linearJni init method call exception , code:-2)
    at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.initLinearJni(SmallCategorySeekTopnReducer.java:162)
    at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.reduce(SmallCategorySeekTopnReducer.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runCounter(MapReduceUtils.java:160)
    at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run(LotReducerUDTF.java:330)
    at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run(BatchStandaloneUDTFEvaluator.java:53)
ERROR: ld.so: object '${LD_PRELOAD}' from LD_PRELOAD cannot be preloaded: ignored.
Heap Size: 1024M
native library 'libcentauri-1.1.2.so' loaded
[JVM Heap Memory] maximum = 10379526144
[JVM Heap Memory] used: 103077912, init stat info
SeekJni System.loadLibrary(centauri-1.1.2) start ...
SeekJni System.loadLibrary(centauri-1.1.2) end ...
[JVM Heap Memory] used: 137437144, after processed 1 keys, 1 values
[ERROR] 2021-08-16 18:37:15 140475475331776 linear_singleton.cc:273] linear params category_info is invalid
[ERROR] 2021-08-16 18:37:15 140475475331776 linear_singleton.cc:19] validate linear params failed.
[ERROR] 2021-08-16 18:37:15 140475475331776 linear_jni.cc:23] LinearSingleton init failed with ret -2.
Exception in thread "main" CentauriException(code=20006, msg=jni调用异常, detailMsg=linearJni init method call exception , code:-2)
    at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.initLinearJni(SmallCategorySeekTopnReducer.java:162)
    at com.alibaba.proxima.mr.SmallCategorySeekTopnReducer.reduce(SmallCategorySeekTopnReducer.java:111)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

The query table references a category that has zero documents in the doc table. Proxima CE treats this as a user input error and terminates the task rather than silently skipping the category — continuing would return no recall results for that category, creating a hidden bug.

Remove the empty category from the doc table or query table, then rerun.

Access to a MaxCompute Tunnel endpoint fails

Error message

Failed to create DownloadSession ErrorCode=Local Error, ErrorMessage=Failed to create download session with tunnel endpoint

Proxima CE calls the MaxCompute Tunnel interface to read table data. This error has two common causes:

Network issue: The download session failed transiently. Recreate the download session and retry.
Invalid Tunnel endpoint: The configured endpoint does not match your network environment (for example, a classic-network endpoint used with a public-network project). For details, see Why does my MaxCompute project use the public Tunnel endpoint when I configured the classic-network Tunnel endpoint? Add the -tunnel_endpoint startup parameter to specify a valid Tunnel endpoint, then rerun.

"java.lang.NullPointerException" in a MapReduce task

There are three distinct error patterns, each with a different cause.

Error example 1 (due to setKey())

2021-11-17 14:56:15 M1_job0:2/3/5[100%] R2_1_job0:0/0/5000[0%]  R3_2_job0:0/0/7500[0%]
2021-11-17 14:56:21 M1_job0:2/3/5[100%] R2_1_job0:0/0/5000[0%]  R3_2_job0:0/0/7500[0%]
2021-11-17 14:56:30 M1_job0:0/5/5[100%] R2_1_job0:0/0/5000[0%]  R3_2_job0:0/0/7500[0%]
2021-11-17 14:56:38 M1_job0:0/5/5[100%] R2_1_job0:11/0/5000[0%] R3_2_job0:0/0/7500[0%]
2021-11-17 14:56:47 M1_job0:0/5/5[100%] R2_1_job0:109/0/5000[0%]     R3_2_job0:0/0/7500[0%]
...
FAILED: ODPS-0123131:User defined function exception - Traceback:
java.lang.NullPointerException
        at com.alibaba.proxima.proto.ParamsInfo$InputKVSBuilder.setKey(ParamsInfo.java:408)
        at com.alibaba.proxima.mr.SeekTopnReducer.reduce(SeekTopnReducer.java:114)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runReducer(MapReduceUtils.java:160)
        at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run(LotReducerUDTF.java:330)
        at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run(BatchStandaloneUDTFEvaluator.java:53)
CentauriException(code=20005, msg=odps任务运行失败, detailMsg=SeekJobWorker Job running error.)
        at com.alibaba.proxima.pipeline.worker.odps.SeekJobWorker.apply(SeekJobWorker.java:90)
        at com.alibaba.proxima.pipeline.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:41)
        at com.alibaba.proxima.pipeline.CentauriPipelineExecutor.execute(CentauriPipelineExecutor.java:30)
        at com.alibaba.proxima.CentauriRunner.runWithNormal(CentauriRunner.java:33)
        at com.alibaba.proxima.CentauriRunner.main(CentauriRunner.java:244)

Error example 2 (due to VectorConvert.convert())

2021-11-15 17:40:08 M1_job0:0/0/45[0%]  R2_1_job0:0/0/2[0%]
2021-11-15 17:40:16 M1_job0:18/0/45[0%] R2_1_job0:0/0/2[0%]
2021-11-15 17:40:25 M1_job0:18/2/45[0%] R2_1_job0:0/0/2[0%]
...
FAILED: ODPS-0123131:User defined function exception – Traceback:
REPORT:
https://dm.guide/report/ODPS-0123131?data-dm-guide-action=4&data-dm-guide-extra-msg=ID:750eebc1-b999-4e90-8820-1d006aadd0c7
java.lang.NullPointerException
	at com.alibaba.proxima.utils.VectorConvert.convert(VectorConvert.java:15)
	at com.alibaba.proxima.mr.BuildMapper.map(BuildMapper.java:63)
	at sun.reflect.GeneratedMethodAccessor1.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runMapper(MapReduceUtils.java:120)
	at com.aliyun.odps.mapred.bridge.LotMapperUDTF.run(LotMapperUDTF.java:807)
	at com.aliyun.odps.udf.impl.batch.BatchStandaloneUDTFEvaluator.run(BatchStandaloneUDTFEvaluator.java:53)

CentauriException(code=20005, msg=odps任务运行失败, detailMsg=BuildJobWorker Job running error.)
	at com.alibaba.proxima.pipeline.worker.odps.BuildJobWorker.apply(BuildJobWorker.java:81)
	at com.alibaba.proxima.pipeline.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:41)
	at com.alibaba.proxima.pipeline.CentauriPipelineExecutor.execute(CentauriPipelineExecutor.java:30)
	at com.alibaba.proxima.CentauriRunner.runWithNormal(CentauriRunner.java:30)
	at com.alibaba.proxima.CentauriRunner.main(CentauriRunner.java:230)
2021-11-15 17:40:30.777 [main] INFO  c.a.p.CentauriPipelineExecutor – [] – execute CleanUpWorker worker start ..........
2021-11-15 17:40:30.949 [main] INFO  com.alibaba.proxima.utils.OdpsUtil – [] – drop table
tmp_mumei_proxima_doc_split_tmp_1636968696798_start

Error example 3 (due to getInputVolumeFileSystem())

2021-12-03 13:12:26 M1_job_0:0/8/8[100%]        R2_1_job_0:3/0/3[0%]    R3_2_job_0:0/0/3[0%]
2021-12-03 13:12:34 M1_job_0:0/8/8[100%]        R2_1_job_0:3/0/3[0%]    R3_2_job_0:0/0/3[0%]
...
FAILED: ODPS-0123131: Fuxi job failed — Failed in UDF/UDTF/UDAF com.aliyun.odps.mapred.bridge.LotReducerUDTF class, at query location of line 21, column 12
ODPS-0123131:User defined function exception — Traceback:
java.lang.NullPointerException
        at com.aliyun.odps.udf.impl.NativeExecutionContext.getInputVolumeFileSystem(NativeExecutionContext.java:373)
        at com.aliyun.odps.mapred.bridge.UDTFTaskContextImpl.getInputVolumeFileSystem(UDTFTaskContextImpl.java:481)
        at com.alibaba.proxima2.ce.pipeline.odps.mr.SeekTopNReducer.setup(SeekTopNReducer.java:58)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runReducer(MapReduceUtils.java:155)
        at com.aliyun.odps.mapred.bridge.LotReducerUDTF.run(LotReducerUDTF.java:330)
        at com.aliyun.odps.udf.impl.batch.StandaloneUserThreadForEE.run(StandaloneUserThreadForEE.java:22)
ProximaCEException(code=20005, msg=odps任务运行失败, detailMsg=SeekJobWorker Job running error.)
        at com.alibaba.proxima2.ce.pipeline.odps.worker.SeekJobWorker.apply(SeekJobWorker.java:90)
        at com.alibaba.proxima2.ce.pipeline.odps.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:31)
        at com.alibaba.proxima2.ce.pipeline.PipelineExecutor.execute(PipelineExecutor.java:27)
        at com.alibaba.proxima2.ce.ProximaCERunner.runWithNormal(ProximaCERunner.java:28)
        at com.alibaba.proxima2.ce.ProximaCERunner.main(ProximaCERunner.java:147)
13:12:40.884 [main] INFO com.alibaba.proxima2.ce.pipeline.PipelineExecutor — Start to execute CleanUpWorker worker...
13:12:41.038 [main] INFO com.alibaba.proxima2.ce.utils.OdpsUtil — drop table
cp_crowd_lookalike_result_doc_b364d0605b5a4f348725785c911b56ad_split_tmp_AevgmUxjpN_665 start ...

Error examples 1 and 2

Both errors occur when a MapReduce task cannot read a column in the input table. Possible causes:

The column does not exist in the table.
A row in the column has a null value.
A row in the column has an invalid value that causes a parsing error.

The error log tells you which phase failed:

Build phase (Mapper-Reducer process): check the doc table for the issues above.
Seek phase (Mapper-Reducer1-Reducer2 process): check the query table.

Error example 2 is recorded during the Mapper-Reducer (build) phase; error example 1 is recorded during the Mapper-Reducer1-Reducer2 (seek) phase.

Error example 3

This error occurs when a mapper task cannot read volume or volume partition information in the seek phase. Possible causes:

Concurrent tasks sharing the same doc table: Volume partitions are named based on the doc table name and partition names. If multiple tasks use the same doc table simultaneously, their index files may overwrite or delete each other, causing volume reads to fail and indexes to fail to load. Give each task a unique doc table name to prevent conflicts.
-column_num or -row_num set too high: The number of reducer instances in MRR tasks equals column_num x row_num. MaxCompute supports a maximum of 99,999 reducer instances. Exceeding this limit causes unpredictable errors where specific reducer instances cannot locate the correct volume partition. Set valid values for both parameters — see Cluster sharding for recommended values.

Partition key column `pt` not found by `getPartitionColumn`

Error message

13:09:30.929 [main] INFO com.alibaba.proxima2.ce.utils.OdpsUtil - check doc result false
13:09:31.237 [main] INFO com.alibaba.proxima2.ce.utils.OdpsUtil - check table:proxima_doc_table , column: pk , odpsType:BIGINT.
13:09:31.237 [main] INFO com.alibaba.proxima2.ce.utils.OdpsUtil - check table:proxima_doc_table , column: vector , odpsType:STRING pass.
java.lang.NullPointerException
        at com.aliyun.odps.TableSchema.getPartitionColumnIndex(TableSchema.java:206)
        at com.aliyun.odps.TableSchema.getPartitionColumn(TableSchema.java:195)
        at com.alibaba.proxima2.ce.utils.OdpsUtil.checkDocQueryTable(OdpsUtil.java:677)
        at com.alibaba.proxima2.ce.utils.ConfigParser.checkTableInfo(ConfigParser.java:227)
        at com.alibaba.proxima2.ce.utils.ConfigParser.parse(ConfigParser.java:46)
        at com.alibaba.proxima2.ce.ProximaCERunner.main(ProximaCERunner.java:139)
FAILED: Run job failed.
REPORT:
https://dm.guide/report/Run job fail?data-dm-guide-action=4&data-dm-guide-extra-msg=ID:e6962785-f4eb-4c27-bc76-2600f2ff8d8d
2021-12-28 13:09:31 INFO ========================================================================
2021-12-28 13:09:31 INFO Exit code of the Shell command 1
2021-12-28 13:09:31 INFO --- Invocation of Shell command completed ---

The doc and query table schemas enforce a strict requirement: the partition key column must be named pt with the STRING data type. If pt is defined as a regular column instead of a partition key column, this error occurs.

Make sure pt is specified as a partition key column. For the required schema format, see Import input table data.

"ShuffleServiceMode: Dump checkpoint failed"

Error message

0010000:System internal error - fuxi job failed, caused by: ShuffleServiceMode: Dump checkpoint failed

The output of a single mapper or reducer instance exceeded 400 GB. This happens when a single instance processes too much data, or when the internal logic of a mapper or reducer task causes excessive data bloat.

Set the -mapper_split_size parameter (in MB) to reduce the amount of data each mapper processes.

"MaxCompute-0430071: Rejected by policy - rejected by system throttling rule"

Error message

FAILED: MaxCompute-0430071: Rejected by policy - rejected by system throttling rule

MaxCompute throttled the task execution request. This typically happens when the cluster is running other workloads (such as stress tests) or when task execution is restricted for a period due to security policy.

Wait and rerun the task. Contact the project owner or the cluster administrator to find out when the restriction lifts.

"java.lang.ArrayIndexOutOfBoundException" due to a table reading failure

Error message

...
FAILED: ODPS-0123131: Fuxi job failed – Failed in UDF/UDTF/UDAF com.aliyun.odps.mapred.bridge.LotMapperUDTF class, at query location of line 8, column 26
ODPS-0123131:User defined function exception – Traceback:
java.lang.ArrayIndexOutOfBoundsException: 5560
	at com.alibaba.proxima2.ce.pipeline.odps.mr.GeneratePkMapper.map(GeneratePkMapper.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.aliyun.odps.mapred.bridge.utils.MapReduceUtils.runMapper(MapReduceUtils.java:120)
	at com.aliyun.odps.mapred.bridge.LotMapperUDTF.run(LotMapperUDTF.java:807)
	at com.aliyun.odps.udf.impl.batch.StandaloneUserThreadForEE.run(StandaloneUserThreadForEE.java:22)
ProximaCEException(code=20005, msg=odps任务运行失败, detailMsg=PrimaryKeyProcess Job running error.)
	at com.alibaba.proxima2.ce.pipeline.odps.worker.TmpTableJobWorker.apply(TmpTableJobWorker.java:103)
	at com.alibaba.proxima2.ce.pipeline.odps.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:31)
	at com.alibaba.proxima2.ce.pipeline.PipelineExecutor.execute(PipelineExecutor.java:27)
	at com.alibaba.proxima2.ce.ProximaCERunner.runWithNormal(ProximaCERunner.java:25)
	at com.alibaba.proxima2.ce.ProximaCERunner.main(ProximaCERunner.java:147)
17:24:49.914  [main]  INFO com.alibaba.proxima2.ce.pipeline.PipelineExecutor – Start to execute CleanUpWorker worker...
17:24:50.025  [main]  INFO com.alibaba.proxima2.ce.utils.OdpsUtil – drop table dwd_gx_pkg_kg_C2M_PDD_alldoc_by_swintransformerv2_df_split_tmp_MZzSCqbREB_214 start ...
2022-09-28 17:24:51  INFO  – DQC Hook:begin

The GeneratePkMapper task failed to read the table and could not get a valid array length. Possible causes:

Unstable cluster network (timeout): The read timed out. Retry the task a few times, or wait until the network stabilizes before rerunning.

16:53:55.898 [main] INFO com.alibaba.proxima2.ce.utils.OdpsUtil - getDocSplitCountArray : dwd_gx_pkg_kg_
[ODPS] 获取ODPS表记录失败 java.io.IOException: ErrorCode=Local Error, ErrorMessage=Read timed out
17:22:14.380 [main] INFO com.alibaba.proxima2.ce.pipeline.odps.worker.TmpTableJobWorker - PrimaryKey job
gbPFR_214

Tunnel exception: Proxima CE reads tables via Tunnel commands, so a Tunnel error also causes this failure. See Access to a MaxCompute Tunnel endpoint fails for troubleshooting steps.

Timeout during cluster sharding

Error message

FAILED: ODPS-0010000:System internal error - Timeout when graph master wait all workers start
java.io.IOException: Job failed!
    at com.aliyun.odps.graph.GraphJob.run(GraphJob.java:429)
    at com.alibaba.proxima2.ce.pipeline.odps.worker.KmeansGraphJobWorker.apply(KmeansGraphJobWorker.java:131)
    at com.alibaba.proxima2.ce.pipeline.odps.runner.OdpsPipelineRunner.run(OdpsPipelineRunner.java:31)
    at com.alibaba.proxima2.ce.pipeline.PipelineExecutor.execute(PipelineExecutor.java:27)
    at com.alibaba.proxima2.ce.ProximaCERunner.runWithNormal(ProximaCERunner.java:28)
    at com.alibaba.proxima2.ce.ProximaCERunner.main(ProximaCERunner.java:149)

During cluster sharding, Proxima CE uses the Graph engine to cluster data. GraphJob requests a set number of worker nodes by default. If the cluster does not have enough available resources, the job times out waiting for workers to start.

Limit the number of worker nodes GraphJob can request by adding the following parameter before the Proxima CE startup command:

set odps.graph.worker.num=400; -- 400 is an example. Set this based on your cluster's available resources.

Complete command example:

set odps.graph.worker.num=400;
jar -resources kmeans_center_resource_cl,proxima-ce-aliyun-1.0.0.jar
-classpath http://schedule@{env}inside.cheetah.alibaba-inc.com/scheduler/res?id=251678818 com.alibaba.proxima2.ce com.alibaba.proxima2.ce.ProximaCERunner
-doc_table doc_table_pailitao
-doc_table_partition 20210707
-query_table query_table_pailitao
-query_table_partition 20210707
-output_table output_table_pailitao_cluster_2000w
-output_table_partition 20210707
-data_type float
-dimension 512
-oss_access_id xxx
-oss_access_key xxx
-oss_endpoint xxx
-oss_bucket xxx
-owner_id 123456
-vector_separator blank
-pk_type int64
-row_num 10
-column_num 10
-job_mode build:seek:recall
-topk 1,50,100,200
-sharding_mode cluster
-kmeans_resource_name kmeans_center_resource_cl
-kmeans_cluster_num 1000;

Duplicate primary keys in the seek phase

GetSmallCategoryDocNum returns empty in the search preparation phase

Schema validation fails in the search phase for small categories

"-nan" appears during table data parsing

JNI call exception in a multi-category search (doc table has 0 documents for a category)

Access to a MaxCompute Tunnel endpoint fails

"java.lang.NullPointerException" in a MapReduce task

Error examples 1 and 2

Error example 3

Partition key column pt not found by getPartitionColumn

"ShuffleServiceMode: Dump checkpoint failed"

"MaxCompute-0430071: Rejected by policy - rejected by system throttling rule"

"java.lang.ArrayIndexOutOfBoundException" due to a table reading failure

Timeout during cluster sharding

Partition key column `pt` not found by `getPartitionColumn`