MaxCompute:FAQ

No, vectors in an input table cannot be of the BINARY type. Proxima CE creates indexes by converting vector columns in the doc table into indexes. By default, vector columns in the doc table support only the STRING type. The BINARY type is not supported. However, Proxima CE provides the -binary_to_int command-line parameter. This parameter specifies whether to convert data of the BINARY type into the INT type. For example, if commas (,) are used as delimiters of input data, this parameter takes effect in the following ways:

• If -binary_to_int is set to false, input data can be 1,1,1,1,1,1,…..

• If -binary_to_int is set to true, input data can be 12345,13423,13325,…..

The data type conversion feature allows you to convert N binary values made up of 0 and 1 into N 32-bit integers. This way, the sizes of created indexes are reduced.

For users:

• Columns affect the build process. If a large number of columns are created, the index size of each column is small, and the search speed of a single column is fast. This accelerates the build process but increases the amount of cluster machine resources that are used.

• Rows affect the seek process. If a large number of rows are created, the number of queries for each row is small, and the search speed of a single row is fast. This accelerates the seek process but increases the amount of cluster machine resources that are used.

Take note of the following limits on row and column division:

• Default limits on the usage of cluster resources. You must contact the owner of the MaxCompute project to which your account belongs to learn about the default limits on the usage of cluster resources.

• Limits on MapReduce instances. In MaxCompute MapReduce, a maximum number of 99,999 instances can be used to run reduce tasks. In the build process, the number of instances is specified by the column_num parameter. In the seek process, the number of instances is determined by using the following formula: column_num × row_num. You must make sure that the value calculated by using the preceding formula is less than 99999.

If you have confirmed that the input data is valid and the system runs as expected, the issue may lie in the data of the doc table. By default, Proxima CE uses the Hierarchical Navigable Small World (HNSW) graph algorithm to create indexes. In this case, objects in the graph may not be linked. As a result, the number of retrieved results is less than the specified number. Solutions:

• Decrease the recall rate. You cannot completely resolve the issue by using this method. If the graph linking issue is caused by the underlying algorithm, the number of retrieved results cannot be the same as the specified number (200 in this example) regardless of the recall rate. If you decrease the recall rate in special cases, the retrieval of other vectors may also be affected. You must evaluate the impact before you use this method.

• Change the index creation algorithm. For example, if you use Hierarchical Clustering (HC) to create a graph, you can configure the -algo_model command-line parameter to specify the index creation algorithm.

• Complement top K results. For versions later than Proxima 2.4, if you use the HNSW graph algorithm, you can add the configuration {"proxima.hnsw.searcher.force_padding_result_enable" : True} to complement the specified top K results based on the search results. This method has a negative impact on the similarity search in extreme cases. You must evaluate the impact based on actual business before you use this method.

In Proxima CE, you cannot execute tasks that involve the same doc table at the same time. If you execute such tasks at the same time, the index overwrite issue may occur. For example, if an index is created for Task A, the index of Task B is overwritten by the created index. As a result, various issues occur. Common issues:

• Errors related to the underlying OSS volume file system are reported.

• The build process fails and the errors related to Java Native Interface (JNI)-based index writes are reported.

• The seek process fails and the errors related to JNI-based index loading are reported.

The common cause is that offline and online tasks run in the same cluster. In common cases, different types of MaxCompute jobs are executed in clusters in a hybrid manner. The underlying layer of clusters is shared by offline and online tasks. Therefore, offline tasks may occupy large amounts of cluster resources. As a result, online tasks cannot obtain sufficient resources, and these tasks slowly run and even fail. Solutions:

• Restrict the resource usage of offline tasks. You can limit the concurrency of offline tasks by specifying the number of rows and columns of Proxima offline tasks. You can also impose limits on your MaxCompute project in the MaxCompute console. These operations slow down the execution of offline tasks.

• Try to run online and offline tasks during different periods of time. You can also try to apply for new resources.

The following or similar information is logged.A task may be killed due to the following reasons:

• The task is run for a long period of time. If an SQL task is run in MaxCompute for more than 24 hours, it is automatically killed. You can set odps.sql.job.max.time.hours to 72. This way, you can run the task for up to 72 hours.

  set odps.sql.job.max.time.hours=72;

• The cluster is overloaded and resources are preempted by the task for a long time. As a result, the task is killed.

• The task is manually killed. You can check whether the task is killed by the project owner or other users assigned with the project administrator role.

In most cases, if a task is killed, you can rerun it. However, if the cluster is overloaded, we recommend that you use one of the following methods:

• Prioritize the task by configuring the -odps_task_priority parameter. For more information, see Optional parameters.

  Important

  This method is risky. If you use this method, the task may preempt the resources that are allocated to other high-priority jobs. You need to contact the project owner or the relevant personnel and ask them to confirm that no high-priority online or offline tasks run in the cluster of the project.

• Wait until the resource usage of high-priority tasks decreases. Then, rerun the Proxima CE task.