When you query data using a search index, you can define a sort order in advance or specify one at query time to retrieve results in a specific order. If the result set is large, you can use offset-based or token-based paging to quickly locate the data you need.
Index sort
By default, a search index sorts data based on its index sort. When you query data with a search index, this IndexSort setting determines the default sort order.
When you create a search index, you can define a custom IndexSort. If you do not specify one, the search index defaults to sorting by primary key.
Index sort only supports
PrimaryKeySort(sort by primary key) andFieldSort(sort by field value).A search index with a field of the nested type does not support index sort.
Sort at query time
You can only sort by fields where enableSortAndAgg is set to true.
You can specify a sort order for each query. A search index supports the following four types of sorters (Sorter). You can also combine sorters to create a multi-level sort.
ScoreSort
Sorts results by their relevance score, which is calculated using the BM25 algorithm. This is suitable for scenarios like full-text search.
If you need to sort results by relevance score, you must explicitly specify ScoreSort. Otherwise, the results are sorted based on the search index's IndexSort configuration.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new ScoreSort())));PrimaryKeySort
Sorts results by primary key.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new PrimaryKeySort()))); // Ascending order.
//searchQuery.setSort(new Sort(Arrays.asList(new PrimaryKeySort(SortOrder.DESC)))); // Descending order.FieldSort
Sorts results by the value of a specified column.
Single-column sort
Sorts results by the values in a single column.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(new FieldSort("col", SortOrder.ASC))));Multi-column sort
Sorts results by the values in one column, and then by the values in another column.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setSort(new Sort(Arrays.asList(
new FieldSort("col1", SortOrder.ASC), new FieldSort("col2", SortOrder.ASC))));Fallback sort
When sorting by a Long, Double, or Date column, you can use the missingField parameter to specify a fallback column of the same type. This fallback is used for any row that is missing a value in the primary sort column.
/**
* Sorts by `Col_Long` in descending order. If a row is missing a `Col_Long` value, it uses the value from `Col_Long_sec` instead.
*/
SearchQuery searchQuery = new SearchQuery();
FieldSort fieldSort = new FieldSort("Col_Long");
// Specifies `Col_Long_sec` as the fallback for missing values in `Col_Long`.
fieldSort.setMissingField("Col_Long_sec");
fieldSort.setOrder(SortOrder.DESC); Missing value sort
If a document is missing the sort field, you can use the missingValue parameter to control its position in the sorted results.
The sorting behavior is as follows:
When
missingValueis set toFieldSort.FIRST_WHEN_MISSING, documents with missing values are always placed at the beginning of the results, regardless of the sort order (ascending or descending).When
missingValueis set toFieldSort.LAST_WHEN_MISSINGor is not set (null), documents with missing values are always placed at the end of the results, regardless of the sort order./** * Sorts by `Col_Long` in descending order. Places documents with a missing `Col_Long` value at the beginning of the results. */ SearchQuery searchQuery = new SearchQuery(); FieldSort fieldSort = new FieldSort("Col_Long"); // Places documents with missing values first. fieldSort.setMissingValue(FieldSort.FIRST_WHEN_MISSING); fieldSort.setOrder(SortOrder.DESC); searchQuery.setSort(new Sort(Arrays.asList(fieldSort)));
Multi-value sort
For multi-valued fields, such as array or nested type fields, you can use the mode parameter to specify which value in the collection to use for sorting.
Sort by a specified value within an array.
// Rows doc1 and doc2 contain an array field named field1. In doc1, the value of field1 is [2,3]. In doc2, the value is [1,3,4].
// You can set the mode parameter to specify which value in the array to use for sorting.
{
// If mode is set to SortMode.MAX, the results are doc2 (sorted by 4) and then doc1 (sorted by 3).
FieldSort fieldSort = new FieldSort("field1", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
}
{
// If mode is set to SortMode.MIN, the results are doc1 (sorted by 2) and then doc2 (sorted by 1).
FieldSort fieldSort = new FieldSort("field1", SortOrder.DESC);
fieldSort.setMode(SortMode.MIN);
}You can also sort by values within sub-fields of a nested type.
// Rows doc1 and doc2 contain a nested type field named field1.
// In doc1, the value of field1 is [{"name":"b", "age":1},{"name":"a", "age":7}].
// In doc2, the value of field1 is [{"name":"a", "age":1},{"name":"c", "age":1},{"name":"d", "age":5}].
{
// Sort all sub-rows and use the mode parameter to specify which value to sort by.
// If you set mode to SortMode.MAX and sort by the age field, the result is doc1 (sorted by 7) and then doc2 (sorted by 5).
FieldSort fieldSort = new FieldSort("field1.age", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.matchAll().build());
fieldSort.setNestedFilter(nestedFilter);
}
{
// Sort only the sub-rows where age=1 and use the mode parameter to specify which value to sort by.
{
// If you set mode to SortMode.MAX and sort by the name field, the result is doc2 (sorted by "c") and then doc1 (sorted by "b").
FieldSort fieldSort = new FieldSort("field1.name", SortOrder.DESC);
fieldSort.setMode(SortMode.MAX);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.term("field1.age",1).build());
fieldSort.setNestedFilter(nestedFilter);
}
{
// If you set mode to SortMode.MIN and sort by the name field, the result is doc1 (sorted by "b") and then doc2 (sorted by "a").
FieldSort fieldSort = new FieldSort("field1.name", SortOrder.DESC);
fieldSort.setMode(SortMode.MIN);
String path = "field1";
NestedFilter nestedFilter = new NestedFilter(path, QueryBuilders.term("field1.age",1).build());
fieldSort.setNestedFilter(nestedFilter);
}
}GeoDistanceSort
Sorts results by the distance from a geographic point.
SearchQuery searchQuery = new SearchQuery();
// Sorts results by distance from the `geo` (Geopoint) column value to the point "0,0".
Sort.Sorter sorter = new GeoDistanceSort("geo", Arrays.asList("0, 0"));
searchQuery.setSort(new Sort(Arrays.asList(sorter)));Paging methods
When retrieving results, you can use either the limit and offset parameters or a token for paging.
Limit and offset paging
You can use limit and offset for paging, but the sum of limit and offset must not exceed 100,000, and the maximum value for limit is 100.
To learn how to increase the limit to 1,000, see How can I increase the query result limit of the Search API to 1,000 for a search index?.
If unset, limit defaults to 10 and offset defaults to 0.
SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(new MatchAllQuery());
searchQuery.setLimit(100);
searchQuery.setOffset(100);Token-based paging
We recommend using token-based paging for deep paging because it has no depth limitations.
If a query has more results to return, the response includes a nextToken. Use this token in the next request to retrieve the following page of results.
By default, token-based paging only allows you to move forward through the results. However, because tokens remain valid for the duration of the paging session, you can cache previous tokens to implement backward paging.
To persist a nextToken or send it to a front-end application, you must Base64-encode it. The nextToken is a byte array, not a string. Converting it directly with new String(nextToken) will corrupt the token and cause data loss.
A token preserves the sort order from the previous request, whether it was the default index sort or a custom sort order. Therefore, you cannot specify a Sort parameter in a token-based request. You also cannot use the offset parameter, as the token itself dictates the starting position.
A search index with a field of the nested type does not support index sort. To paginate results from such a search index, you must specify a sort order in your query. Otherwise, the server will not return a nextToken even if more results are available.
private static void readMoreRowsWithToken(SyncClient client) {
SearchQuery searchQuery = new SearchQuery();
searchQuery.setQuery(new MatchAllQuery());
searchQuery.setGetTotalCount(true);// Set this to true to return the total number of matched rows.
// Specify the table name (e.g., sampleTable) and the search index name (e.g., sampleSearchIndex). You can find the search index name on the Indexes tab of your table in the Tablestore console or by listing search indexes using the SDK.
SearchRequest searchRequest = new SearchRequest("<TABLE_NAME>", "<SEARCH_INDEX_NAME>", searchQuery);
SearchResponse resp = client.search(searchRequest);
if (!resp.isAllSuccess()) {
throw new RuntimeException("not all success");
}
List<Row> rows = resp.getRows();
while (resp.getNextToken()!=null) { // A null `nextToken` means all data has been retrieved.
// Get the nextToken.
byte[] nextToken = resp.getNextToken();
{
// If you need to persist the nextToken or send it to a front-end application, use Base64 encoding to convert it to a string.
// The token itself is not a string. Directly converting it using new String(nextToken) will corrupt the token.
String tokenAsString = Base64.toBase64String(nextToken);
// Decode the string back to a byte array.
byte[] tokenAsByte = Base64.fromBase64String(tokenAsString);
}
// Set the token for the next request.
searchRequest.getSearchQuery().setToken(nextToken);
resp = client.search(searchRequest);
if (!resp.isAllSuccess()) {
throw new RuntimeException("not all success");
}
rows.addAll(resp.getRows());
}
System.out.println("RowSize: " + rows.size());
System.out.println("TotalCount: " + resp.getTotalCount());// Prints the total count of matched rows, not the number of rows returned in this response.
}FAQ
References
A search index supports various query types, including Term query, Terms query, MatchAll query, Match query, MatchPhrase query, Prefix query, Range query, Wildcard query, Geo query, Vector query, Boolean query, Nested query, and Exists query. You can choose the appropriate query type for multi-dimensional data retrieval.
To sort or paginate the result set, use the sorting and paging features. For more information, see Sort and paginate results.
To group the result set by a specific column and return only one document for each group, use the collapsing feature. For more information, see Collapsing.
For data analysis, such as finding the maximum or minimum value, calculating a sum, or counting rows, you can use the aggregation feature of the Search API or SQL queries. For more information, see Aggregation and SQL query.
For fast data exports where the overall result order is not important, you can use the ParallelScan and ComputeSplits APIs to export data in parallel. For more information, see Export data in parallel.