MaxCompute:FAQ about running failures

Open Logview that runs the MaxCompute task and view the value in the StdOut column. For more information about how to use Logview, see Use LogView to view job information. If the input record number of this instance that is shown in the preceding figure is much greater than the input record number of other instances, check whether the data contains vectors that have the same primary key but different values. If the data contains vectors that have the same primary key but different values, the data in the database is not deduplicated. From Reducer1 to Reducer2 during the Mapper-Reducer1-Reducer2 process in the seek phase, the data in which vectors have the same primary key but different values is processed by the same reducer. As a result, data skew occurs and a reduce task may be unexpectedly terminated.

In most cases, this issue occurs because Proxima CE fails to call an interface of MaxCompute Tunnel due to one of the following reasons:

• Network issue: In this case, recreate a download session.

• Cross-network access issue: The number of MaxCompute tables fails to be obtained because the configured Tunnel endpoint is invalid. For more information, see When I use the Tunnel Upload command to upload data, I configure the Tunnel endpoint of the classic network for my MaxCompute project but the project is connected to the public Tunnel endpoint. Why?. To resolve this issue, add the -tunnel_endpoint startup parameter to the code to specify a valid Tunnel endpoint and rerun the code.

Error example 1 and error example 2: In most cases, these errors occur because a MapReduce task fails to read data from a column in the input table in the build or seek phase. Possible causes:

• The column does not exist.

• The value of a row in the column is null.

• The value of a row in the column is invalid. As a result, a parsing error occurs.

If the task fails in the build phase, check whether the preceding issues exist in the doc table. If the task fails in the seek phase, check whether the preceding issues exist in the query table. You can determine whether the task fails in the build or seek phase based on the error message.

• The MapReduce tasks that are generated in the Mapper-Reducer process run in the build phase. The output that is shown in Error example 2 is recorded in run logs.

• MRR tasks that are generated in the Mapper-Reducer1-Reducer2 process run in the seek phase. The output that is shown in Error example 1 is recorded in run logs.

Error example 3: In most cases, this error occurs when a mapper task fails to obtain information about a volume and its volume partition of MaxCompute in the seek phase. Possible causes:

• Multiple tasks that use the same doc table run at the same time. The output that is shown in Error example 3 is recorded in run logs. For Proxima CE of the current version, the volume partition that stores indexes is based on the name of the doc table and partition names. If multiple tasks that use the same doc table run at the same time, index files in the same volume of the tasks may be overwritten or deleted. In this case, the volume of MaxCompute fails to be read and indexes may fail to be loaded. You can change the names of the doc tables that are used by the tasks to ensure that the names of the doc tables of the tasks are different. This helps ensure that each task runs as expected.

• The row and column parameters are incorrectly configured. MaxCompute supports a maximum of 99,999 reducer instances. If the -column_num and -row_num parameters are set to excessively large values, the number of instances of MRR tasks in the seek phase exceeds the upper limit. The number of instances is the product of column_num multiplied by row_num. In this case, an uncontrollable error may occur and specific reducer instances in the seek phase cannot find the correct volume partition that stores indexes. To prevent this issue, specify valid values for the row and column parameters. For more information, see Cluster sharding.

The mapper task GeneratePkMapper fails to obtain a valid array length. In most cases, an SQL task that runs before the mapper task reads table data to obtain the array length. This issue occurs due to a table reading failure.

• Timeout: The network of the cluster to which the project belongs is unstable. In this case, retry the task several times or rerun the task after a specific period of time.

• Tunnel exception: When Proxima CE reads a table, Proxima CE obtains the table records by using Tunnel commands. Therefore, a Tunnel exception also causes a table reading failure. For more information about how to handle Tunnel errors, see What do I do if access to a Tunnel endpoint of MaxCompute fails?.

When Proxima CE builds an index during cluster sharding, Proxima CE clusters data based on the Graph engine. By default, GraphJob runs to apply for a specific number of worker nodes. If the cluster resources are insufficient, this issue occurs. You can use the following command to limit the number of worker nodes that can execute GraphJob. This helps limit resource requests. Add the following parameter configuration before the startup command of Proxima CE.

set odps.graph.worker.num=400; -- In this example, 400 is a sample value. You can specify the value based on the cluster resource configuration.

Complete commands:

set odps.graph.worker.num=400;
jar -resources kmeans_center_resource_cl,proxima-ce-aliyun-1.0.0.jar
-classpath http://schedule@{env}inside.cheetah.alibaba-inc.com/scheduler/res?id=251678818 com.alibaba.proxima2.ce com.alibaba.proxima2.ce.ProximaCERunner
-doc_table doc_table_pailitao
-doc_table_partition 20210707
-query_table query_table_pailitao
-query_table_partition 20210707
-output_table output_table_pailitao_cluster_2000w
-output_table_partition 20210707
-data_type float
-dimension 512
-oss_access_id xxx
-oss_access_key xxx
-oss_endpoint xxx
-oss_bucket xxx
-owner_id 123456
-vector_separator blank
-pk_type int64
-row_num 10
-column_num 10
-job_mode build:seek:recall
-topk 1,50,100,200
-sharding_mode cluster
-kmeans_resource_name kmeans_center_resource_cl
-kmeans_cluster_num 1000;