How to export documents in batches by using scroll - OpenSearch

Scenarios

The main goal of traditional search scenarios is to retrieve the most relevant results in the shortest possible time. Therefore, search results are limited. For example, the search method can retrieve a maximum of 5,000 documents. In some scenarios, you need more results for analysis. In these cases, you can use the scroll operation to obtain more results.

Parameters

Search parameters:

Parameter	Type	Required	Valid values	Default value	Description
scroll	STRING	Yes	Weeks, days, hours, minutes, and seconds		The validity period for the next scroll request. You must set this parameter in every request. For example, 1m indicates 1 minute. Supported time units: w=Week, d=Day, h=Hour, m=minute, s=second.
search_type	STRING	Yes	scan		Required in the first query. You do not need to specify it in subsequent queries, which are performed by specifying scroll_id.
scroll_id	string	Yes			The first call to the scroll method returns a scroll_id but no data. In each subsequent search, you must specify the scroll_id returned by the previous search. Each subsequent search result returns a scroll_id and the corresponding matched data. This parameter is required in subsequent queries.
The fetch_fields parameter	string				Specifies the application fields whose content is retrieved.

Response parameters:

Parameter	Type	Description
status	string	The execution result. OK indicates success and FAIL indicates failure. If the request fails, troubleshoot based on the returned error code.
request_id	string	The ID of the query record, which is mainly used for troubleshooting.
result	string	The actual returned results, including the query time (searchtime), the total number of results in the engine (total), the number of results returned for this request (num), the maximum number of results that can be returned for this query (viewtotal), the query results (items), the statistical results (facet), and the scroll ID (scroll_id).
errors	string	The error details. error_message indicates the error message. For the meaning of error_code, see the Error codes topic.

Note

Note: The scroll results can be returned only in the fulljson or json format.

Sample results

Result of the first request:

{
    "status": "OK",
    "request_id": "150150574119953661605242",
    "result": {
        "searchtime": 0.005029,
        "total": 1,
        "num": 0,
        "viewtotal": 1,
        "scroll_id": "eJxtUMtuhDAM/BrvOYQC5cABdulvRFFIirsm2TpBavv1Ndut1EMlS36NZ0Y2ZHMxbueceAjIuWCMnrPjRITLyfzZm83y9V QVGT8x80U3PxQNUqieVZV1/an4ItbTUBPSx5wgXqKdvOSbmuKR8ZYjGWWirB4tvToAiX7u3G2eCNK77vnz8GlGPAV6suKBeqxAn0OiTd7NGEnesspyoyFLF6hecn4JUKjVgp0K3FnkfMfIyPoDuYWegX9GeYOpicY9TG8gwOSuBL04X1 MMg3ROwCesLlG6X7a2o=",
        "items": [],
        "facet": []
    },
    "errors": [],
    "tracer": ""
}

Result of subsequent requests:

{
    "status": "OK",
    "request_id": "150150574119952551519970",
    "result": {
        "searchtime": 0.006293,
        "total": 1,
        "num": 1,
        "viewtotal": 1,
        "scroll_id": "eJxNT9tugzAM/RrznIRC4YEHaNlvRFFIhteQtE6Qtn39TNdJk2z5dnx8rIPJRdudcqKhl60Uir2Vp06ISv8b6s3QbZCVzpaCdp93XXBzg2wEW9MJ2dWq8q7YVXt0YckDLlBP0WyOw31N8YgYizZEnAUsjkx4VT4k8zexpjiNS/XYHX0NNkWP71BfVyxQjxLUxSfazFH4PYSPnCL3iMniDZq3jN98aFRCgGrZniy8/itkBHWGuYVeQH+B+QzTCUZ1NJ9gj4FVMfrQPr8Y+Hk+dgU14fIDVhtfTw==",
        "items": [
            {
                "fields": {
                    "cate_id": "0",
                    "float_arr": "0",
                    "id": "1",
                    "int_arr": "0",
                    "literal_arr": "search",
                    "name": "search",
                    "phone": "1381111****",
                    "index_name": "app_schema_demo"
                },
                "property": {},
                "attribute": {},
                "variableValue": {},
                "sortExprValues": [
                    "1"
                ]
            }
        ],
        "facet": []
    },
    "errors": [],
    "tracer": ""
}

Usage notes

sort clause (supports only a single field of the INT type, and is available only in API V3 and the corresponding SDKs).
scroll supports only exporting all data. It does not support the aggregate and distinct clauses, rough sort and fine sort expressions, or query analysis.
The start parameter in the config clause does not take effect in scroll queries. Its default value is 0, which means that page skipping is not supported. The value of hit is limited to [0,500].
A scroll query uses the hit value of the first query (the query that returns the scroll ID). Any subsequent changes to the hit value do not take effect.
The first execution returns only the scroll_id value and no document data. Data is returned when you set scroll_id in the second query.
To determine whether a search failed, check the code and message for errors. Do not rely on status. Error codes
If the error Scroll_id is expired is returned, the validity period of the scroll request has expired. Adjust the scroll parameter.

SDK sample demos

Note

Note:

1. start in the config clause does not take effect. Use the hit value to specify the number of documents retrieved each time.

2. aggregate, distinct, and rough sort and fine sort expressions do not take effect. The sort clause supports sorting only by a single field of the INT type.

3. scroll queries across multiple applications are not supported.

4. If the specified scroll_id is invalid, an error is reported during the query.

5. Retrieved results can be returned only in the fulljson or json format.

6. The first query returns only scroll_id and no document data. To retrieve data, you must run another query and set the scroll_id returned by the previous query.