OpenSearch:Sort policies

Search engines require high retrieval performance. To meet this demand, the system provides a two-stage sorting process: rough sort and fine sort. The rough sort stage quickly selects a subset of high-quality documents (the top N results) from the initial retrieval. These documents then undergo fine sorting for detailed scoring to return the most relevant results. The rough sort primarily impacts performance, while the fine sort determines the final result quality. Therefore, a rough sort should be simple and efficient, using only the key factors from the fine sort. Both stages are configured using sort expressions.

A sort expression (also known as a ranking formula) lets you customize how search results are ranked for your application. You specify an expression in the query request to sort the results. A sort expression supports basic operations (arithmetic, relational, logical, bitwise, and conditional), mathematical functions, and fine sort functions. For common applications such as forums and news sites, OpenSearch provides best practices for relevance sorting. You can use these as templates, modify them based on your data characteristics, and create your own expressions.

Before you configure relevance-based ranking (fine sort), you must understand the system's default sorting behavior. After documents that match the query are found, they enter the sorting stage. For more information, see the sort clause. If you do not specify a sort clause, or if the sort clause explicitly includes RANK, the relevance scoring stage begins.

The design of sort expressions for rough sort and fine sort depends on your search requirements. The article on relevance sorting provides a detailed guide on how to design and select ranking factors for several typical scenarios.

Basic operations

Operation	Operator	Description
Unary operation	-	The minus sign (-). Negates the value of an expression. Examples: -1, -max(width).
Arithmetic operation	+, -, *, /	Example: width / 10
Relational operation	==, !=, >, <, >=, <=	Example: width>=400
Logical operation	and, or, !	Examples: width>=400 and height >= 300, !(a > 1 and b < 2)
Bitwise operation	&, |, ^	Example: 3 & (price ^ pubtime) + (price | pubtime)
Conditional operation	if(cond, thenValue, elseValue)	If the cond expression evaluates to a non-zero value, the function returns thenValue; otherwise, it returns elseValue. For example, if(2, 3, 5) returns 3, and if(0, 3, 5) returns 5. Note: String field types such as literal or text are not supported. The returned value must be within the int32 range.
in operation	i in [value1, value2, …, valuen]	If i is in the set [value1, value2, ..., valuen], the expression returns 1; otherwise, it returns 0. For example, 2 in [2, 4, 6] returns 1, and 3 in [2, 4, 6] returns 0.

Mathematical functions

Function	Description
max(a, b)	Returns the greater of a and b.
min(a, b)	Returns the lesser of a and b.
ln(a)	Returns the natural logarithm of a.
log2(a)	Returns the base-2 logarithm of a.
log10(a)	Returns the base-10 logarithm of a.
sin(a)	Returns the sine of a.
cos(a)	Returns the cosine of a.
tan(a)	Returns the tangent of a.
asin(a)	Returns the arcsine of a.
acos(a)	Returns the arccosine of a.
atan(a)	Returns the arctangent of a.
ceil(a)	Returns the smallest integer that is greater than or equal to a. For example, ceil(4.2) returns 5.
floor(a)	Returns the largest integer that is less than or equal to a. For example, floor(4.6) returns 4.
sqrt(a)	Returns the square root of a. For example, sqrt(4) returns 2.
pow(a,b)	Returns the value of a raised to the power of b. For example, pow(2, 3) returns 8.
now()	Returns the current time in seconds since the Epoch (00:00:00 UTC, January 1, 1970).
random()	Returns a random value between 0 and 1.

Built-in feature functions

OpenSearch provides a variety of rough sort functions, including LBS, text, and timeliness functions. You can combine these in a sort expression to implement powerful relevance sorting.

Cava plugin

Cava is a high-performance programming language developed by the OpenSearch engine team based on LLVM. Its syntax is similar to Java, and its performance is comparable to C++. Cava is an object-oriented language that supports just-in-time (JIT) compilation and various security checks to ensure program robustness. Using Cava and the provided Cava libraries, you can create custom sort plugins in OpenSearch. Compared to using a standard sort expression, a Cava plugin offers the following advantages:

• Greater customization: Cava provides richer syntax features than expressions, such as for loops and function and class definitions, which lets you implement complex business requirements.

• Easier maintenance: A Cava plugin is more readable and easier to maintain than a complex sort expression.

• Lower learning curve: Cava syntax is similar to Java, making it easy for Java developers to learn.

Note: You can configure the Cava plugin only for an exclusive application.

Procedure

This section uses a text relevance function as an example to show how to configure rough sort and fine sort policies.

1. Create a rough sort policy. In the OpenSearch console, select OpenSearch High-performance Search Edition in the upper-left corner. Navigate to Search Configuration Center > Sort Configuration > Policy Management, and click Create. Enter a Policy Name, set Scope to Rough Sort and Type to Expression, and then click Next.

   Select "static_bm25" as the scoring feature and set the weight to "10". A weight of 10 means the score is multiplied by 10 during calculation. You can also add a search field to the sort. The field must be an attribute field and must be a numeric type, such as INT, DOUBLE, or FLOAT. When you set a weight for the field, the value of the field multiplied by its weight is added to the sort score.

   For example, add the sale_price field to the sort and set the weight for the pid field to 0.08.

   After the configuration is complete, return to the Policy Management page. The message "Sort policy configured successfully" appears. Click Search Test to verify the sort configuration.

2. Create a fine sort policy. In the OpenSearch console, click Create. Enter a Policy Name, set Scope to Fine Sort and Type to Expression, and then click Next.

   Select 'text_relevance' from the built-in functions, enter the name of a field from the index in the parentheses, and click Complete. In the Sort Configuration step, use the Field and Built-in Function drop-down lists to generate the sort expression text_relevance(brand), and then click Complete. The configuration is complete, and you are returned to the Policy Management page.

3. View the sort results. On the Search Test page, configure the parameters for rough sort and fine sort, and enable the Show Sort Details option. Set the first_rank_name parameter to test_1 (rough sort) and the second_rank_name parameter to test_2 (fine sort). You can then view the score calculation for each function.

   Sort details appear below each search result. This includes the total sort score (for example, 10000.2259030193) and the individual results of each expression, such as static_bm250, static_bm250*10, and text_relevance(brand).

   Note

   Document scoring occurs in two stages: rough sort and fine sort. Documents retrieved by a query and passed through a filter first undergo rough sort. High-scoring documents are selected based on the rough sort expression. Then, the top N results are precisely scored using the fine sort expression to return the optimal results.

   • If only a rough sort policy is configured, the document score is (10000 + the result of the rough sort expression). If the calculated score exceeds 20,000, it is capped at 20,000.

   • If only a fine sort policy is configured, the document score is (10000 + the result of the fine sort expression). There is no upper limit for the score.

   • If both rough and fine sort policies are configured, the final score for documents that enter the fine sort stage is (10000 + the result of the fine sort expression). The final score for the remaining documents is (10000 + the result of the rough sort expression). If this score exceeds 20,000, it is capped at 20,000.

   • You can create multiple rough sort and fine sort rules. However, a query request can use only one rough sort rule and one fine sort rule at a time.

   Important

   • The first_rank_name parameter supports only one sort expression name. You cannot use multiple rough sort expressions at the same time.

   • The second_rank_name parameter supports only one sort expression name. You cannot use multiple fine sort expressions at the same time.

SDK configuration examples

Java SDK example:

// Set the rough and fine sort expressions. This example uses the default policies.
Rank rank = new Rank();
rank.setFirstRankName("default"); // The name of the rough sort policy.
rank.setSecondRankName("default"); // The name of the fine sort policy.
rank.setReRankSize(5); // The number of documents for fine sorting.

PHP SDK example:

// Specify the rough sort expression.
$params->setFirstRankName('default');
// Specify the fine sort expression.
$params->setSecondRankName('default');

Notes:

• If you configure default rough and fine sort policies in the console but also specify different policies in your code, the policies defined in the code take precedence at query time.

• To view sort details in your code:

  Add the format:fulljson parameter to the config clause.

  In the result, sortExprValues contains the document's score information.

  "sortExprValues": [
      "10000.0399786383"
  ],
  "property": { ... }

  sortExprValues is an array that contains the values of the sort fields specified in the sort clause. For example, for the clause:

  sort=-price;-RANK

  sortExprValues would contain the value of the price field followed by the final document score.

  If you do not specify a sort clause, sortExprValues contains only the final document score.