When a query statement yields numerous results, the display speed decreases, affecting the user experience. Simple Log Service offers a paged query feature to limit the number of logs returned per query. This topic explains how to paginate query and analysis results.
Overview of pagination methods
Simple Log Service supports pagination for query and analysis results when using the GetLogs API. Different pagination methods apply to query results and analysis results. To determine the total number of log lines beforehand, see GetHistograms.
-
Search statement: This queries the raw log data using keywords. The offset and line parameters in the GetLogs API facilitate pagination. For more information, see the query overview.
-
Analytic statement: This analyzes query results using SQL syntax for statistical outcomes. The LIMIT clause in SQL is used for pagination. For more information, see Overview of Queries and Analysis and LIMIT Clause.
Examples of pagination methods
Below are examples demonstrating how to paginate query and analysis results. Select the one that fits your requirements:
-
Example of paginating query results
For a paged query, increment the offset parameter value until all logs are retrieved. Once the value reaches a certain threshold, 0 is returned, indicating completion. At this point, all required data has been read.
-
Sample Code for Pagination
offset = 0 # Specify the starting line for reading query results. In this case, start from line 0. line = 100 # Specify the number of lines to read in the current request. The maximum value is 100. If the value is greater than 100, only 100 lines are returned. In this case, 100 lines are read at a time. query = "status:200" # Query all logs in which the value of the status field is 200. while True: response = get_logstore_logs(query, offset, line) # Call the operation to read logs. process (response) # Call custom logic to process the returned result. if response.get_count() == 0 && response.is_complete() break; # The read process is complete, and the current loop ends. else offset += 100 # Increase the offset by 100 to read the next 100 lines.
Python code example
For more information, see Python SDK Overview.
# The service endpoint of Simple Log Service. endpoint = '' # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '') accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '') # The name of the project. project = '' # The name of the logstore. logstore = '' client = LogClient(endpoint, accessKeyId, accessKey) topic = "" From = int(time.time()) - 600 To = int(time.time()) log_line = 100 offset = 0 while True: res4 = None for retry_time in range(0, 3): req4 = GetLogsRequest(project, logstore, From, To, topic=topic, line=log_line, offset=offset) res4 = client.get_logs(req4) if res4 is not None and res4.is_completed(): break time.sleep(1) offset += 100 if res4.is_completed() and res4.get_count() == 0: break; if res4 is not None: # Display the execution result. res4.log_print()
Java code example
For more information, see Java SDK Overview.
int log_offset = 0; int log_line = 100; // The number of rows that are read at a time. A maximum of 100 rows can be returned. If you want to read more than 100 rows, use the offset parameter. The offset and line parameters take effect only for a search statement that uses keywords. If you use an analytic statement, these parameters are invalid. If you want an analytic statement to return more than 100 rows, use the LIMIT clause. while (true) { GetLogsResponse res4 = null; // For each offset, 100 rows are read at a time. If the read operation fails, a maximum of three retries are allowed. for (int retry_time = 0; retry_time < 3; retry_time++) { GetLogsRequest req4 = new GetLogsRequest(project, logstore, from, to, topic, query, log_offset, log_line, false); res4 = client.GetLogs(req4); if (res4 != null && res4.IsCompleted()) { break; } Thread.sleep(200); } System.out.println("Read log count:" + String.valueOf(res4.GetCount())); log_offset += log_line; if (res4.IsCompleted() && res4.GetCount() == 0) { break; } }
-
-
Example of paginating analysis results
Use the LIMIT clause in SQL to paginate analysis results. For instance, the statement
* | select count(1), url group by url
can be used to return 1,000 log rows. Specify reading 500 rows at a time, completing the task in two iterations. The example below illustrates pagination of analysis results:* | select count(1) , url group by url limit 0, 500 * | select count(1) , url group by url limit 500, 500
-
Sample Code for Pagination
offset = 0 // Specify the starting line for reading query results. In this case, start from line 0. line = 500 // Specify the number of lines to read in the current request. The maximum value is 1,000,000. If many lines are read at a time, the network delay increases, and the client-side processing speed decreases. In this case, 500 lines are read at a time. query = "* | select count(1) , url group by url limit " while True: real_query = query + offset + "," + line response = get_logstore_logs(real_query) // Call the operation to read logs. process (response) // Call custom logic to process the returned result. if response.get_count() == 0 break; // The read process is complete, and the current loop ends. else offset += 500 // Increase the offset by 500 to read the next 500 lines.
Python code example
For more information, see Python SDK Overview.
# The service endpoint of Simple Log Service endpoint = '' # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. accessKeyId = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '') accessKey = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '') # The name of the project. project = '' # The name of the logstore. logstore = '' client = LogClient(endpoint, accessKeyId, accessKey) topic = "" origin_query = "* | select * limit " From = int(time.time()) - 600 To = int(time.time()) log_line = 100 offset = 0 while True: res4 = None query = origin_query + str(offset) + " , " + str(log_line) for retry_time in range(0, 3): req4 = GetLogsRequest(project, logstore, From, To, topic=topic, query=query) res4 = client.get_logs(req4) if res4 is not None and res4.is_completed(): break time.sleep(1) offset += 100 if res4.is_completed() and res4.get_count() == 0: break; if res4 is not None: # Display the execution result. res4.log_print()
Java code example
For more information, see Java SDK Overview.
int log_offset = 0; int log_line = 500; String origin_query = "* | select count(1) , url group by url limit " while (true) { GetLogsResponse res4 = null; // For each offset, 500 rows are read at a time. If the read operation fails, a maximum of three retries are allowed. query = origin_query + log_offset + "," + log_line; for (int retry_time = 0; retry_time < 3; retry_time++) { GetLogsRequest req4 = new GetLogsRequest(project, logstore, from, to, topic, query); res4 = client.GetLogs(req4); if (res4 != null && res4.IsCompleted()) { break; } Thread.sleep(200); } System.out.println("Read log count:" + String.valueOf(res4.GetCount())); log_offset += log_line; if (res4.GetCount() == 0) { break; } }
-