If a query statement returns a large number of query and analysis results, the results are displayed at a lower speed. Log Service provides the paged query feature to limit the number of logs that can be returned for each query. This topic describes the paging methods of query and analysis results.

Paging methods

Log Service provides the query and analysis feature that allows you to execute a query statement to query logs by using keywords and analyze the query results by using SQL syntax. You can also call the GetLogs operation to query the raw data of logs by using keywords and analyze the query results by using SQL syntax. A query statement can contain a search statement and an analytic statement. The paging methods vary between the search statement and analytic statement. For more information, see GetLogs.
  • Search statement: queries the raw data of logs by using keywords. You can configure the offset and line parameters in the GetLogs operation to perform a paged query. For more information, see Query statements.
  • Analytic statement: analyzes query results by using SQL syntax. You can use the LIMIT clause to perform a paged query. For more information, see Analytic statements and LIMIT syntax.

Paging of query results

The following list describes the offset and line parameters in the GetLogs operation:
  • offset: the line from which query results are returned.
  • line: the number of lines that are returned for the current API request. A maximum of 100 lines can be returned. If you set this parameter to a value greater than 100, only 100 lines are returned.

When a paged query is performed, the value of the offset parameter increases until all logs are read. When the value reaches a specific number, 0 is returned, and the progress is complete. In this case, all the required data is read.

  • Sample code for paging implementation
    offset = 0                           // Read logs from line 0. 
    line = 100                          // Read 100 lines at a time. 
    query = "status:200"                 // Read the logs whose value of the status field is 200. 
    while True:
         response = get_logstore_logs(query, offset, line) // Call the operation to read logs. 
         process (response)                                 // Call custom logic to process the returned result. 
         if response.get_count() == 0 && response.is_complete()   
             The read process is complete, and the current loop ends.
         else
            offset += 100                          // The value of the offset parameter increases to 100. The next 100 lines are read. 
  • Python code example
    For more information, see Overview.
        endpoint = ''      // The Log Service endpoint. For more information, see Endpoints. 
        accessKeyId = ''   // The AccessKey ID of your Alibaba Cloud account. For more information, see AccessKey pair. An Alibaba Cloud account has permissions to call all API operations. If you use the AccessKey pair of an Alibaba Cloud account, security risks may occur. We recommend that you create and use a RAM user to call API operations or perform routine O&M. 
        accessKey = ''     // The AccessKey secret of your Alibaba Cloud account. 
        project = ''       // The name of the project. 
        logstore = ''      // The name of the Logstore. 
        client = LogClient(endpoint, accessKeyId, accessKey)
        topic = ""
        From = int(time.time()) - 600
        To = int(time.time())
        log_line = 100
        offset = 0
        while True:
            res4 = None
        for retry_time in range(0, 3):
                req4 = GetLogsRequest(project, logstore, From, To, topic=topic, line=log_line, offset=offset)
                res4 = self.client.get_logs(req4)
                if res4 is not None and res4.is_completed():
                    break
                time.sleep(1)
                offset += 100
                if res4.is_completed() and res4.get_count() == 0:
                  break;
                if res4 is not None:
                res4.log_print()  // Display the execution result. 
  • Java code example

    For more information, see Overview.

            int log_offset = 0;
            int log_line = 100;   // The number of lines that are read at a time. A maximum of 100 lines can be returned. If you want to read more than 100 lines, use the offset parameter. The offset and line parameters take effect only for a search statement that uses keywords. If you use an analytic statement, these parameters are invalid. If you want an analytic statement to return more than 100 lines, use the LIMIT clause. 
            while (true) {
                GetLogsResponse res4 = null;
                // For each offset, 100 lines are read at a time. If the read operation fails, a maximum of three retries are allowed. 
                for (int retry_time = 0; retry_time < 3; retry_time++) {
                    GetLogsRequest req4 = new GetLogsRequest(project, logstore, from, to, topic, query, log_offset,
                            log_line, false);
                    res4 = client.GetLogs(req4);
    
                    if (res4 != null && res4.IsCompleted()) {
                        break;
                    }
                    Thread.sleep(200);
                }
                System.out.println("Read log count:" + String.valueOf(res4.GetCount()));
                log_offset += log_line;
                if (res4.IsCompleted() && res4.GetCount() == 0) {
                            break;
                }
    
            }
                        

Paging of analysis results

You can use the LIMIT clause for the paging of analysis results. Example:
limit Offset, Line

The following list describes the offset and line parameters:

  • offset: the line from which analysis results are returned.
  • line: the number of lines that are returned for the current API request. A maximum of 1,000,000 lines can be returned. If a large number of lines are read at a time, the network delay increases, and the client-side processing speed decreases.

For example, if you want to use the * | select count(1) , url group by url statement to return 2,000 lines, you can execute the following statements to perform 4 paged queries and query 500 lines each time:

* | select count(1) , url  group by url  limit 0, 500
* | select count(1) , url  group by url  limit 500, 500
* | select count(1) , url  group by url  limit 1000, 500
* | select count(1) , url  group by url  limit 1500, 500
  • Sample code for paging implementation
        offset = 0     // Read logs from line 0. 
        line = 500    // Read 500 lines at a time. 
        query = "* | select count(1) , url  group by url  limit "
        while True:
        real_query = query + offset + "," +  lines
        response = get_logstore_logs(real_query) // Call the operation to read logs. 
        process (response)                       // Call custom logic to process the returned result. 
        if response.get_count() == 0   
             The read process is complete, and the current loop ends.
        else
            offset += 500                        // The value of the offset parameter increases to 500. The next 500 lines are read. 
  • Python code example

    For more information, see Overview.

        endpoint = ''      // The Log Service endpoint. For more information, see Endpoints.  
        accessKeyId = ''   // The AccessKey ID of your Alibaba Cloud account. For more information, see AccessKey pair. An Alibaba Cloud account has permissions to call all API operations. If you use the AccessKey pair of an Alibaba Cloud account, security risks may occur. We recommend that you create and use a RAM user to call API operations or perform routine O&M. 
        accessKey = ''     // The AccessKey secret of your Alibaba Cloud account. 
        project = ''       // The name of the project. 
        logstore = ''      // The name of the Logstore. 
        client = LogClient(endpoint, accessKeyId, accessKey)
        topic = ""
        origin_query = "* | select *  limit "
        From = int(time.time()) - 600
        To = int(time.time())
        log_line = 100
        offset = 0
        while True:
            res4 = None
            query = origin_query + str(offset) + " , " + str(log_line)
            for retry_time in range(0, 3):
                req4 = GetLogsRequest(project, logstore, From, To, topic=topic, query)
                res4 = self.client.get_logs(req4)
                if res4 is not None and res4.is_completed():
                    break
                time.sleep(1)
                offset += 100
                if res4.is_completed() and res4.get_count() == 0:
                  break;
                if res4 is not None:
                res4.log_print()  // Display the execution result. 
  • Java code example

    For more information, see Overview.

            int log_offset = 0;
            int log_line = 500;
            String origin_query = "* | select count(1) , url  group by url  limit "
            while (true) {
                GetLogsResponse res4 = null;
                // For each offset, 500 lines are read at a time. If the read operation fails, a maximum of three retries are allowed. 
                query = origin_query + log_offset + "," + log_line;
                for (int retry_time = 0; retry_time < 3; retry_time++) {
                    GetLogsRequest req4 = new GetLogsRequest(project, logstore, from, to, topic, query);
                    res4 = client.GetLogs(req4);
    
                    if (res4 != null && res4.IsCompleted()) {
                        break;
                    }
                    Thread.sleep(200);
                }
                System.out.println("Read log count:" + String.valueOf(res4.GetCount()));
                log_offset += log_line;
                if (res4.GetCount() == 0) {
                            break;
                }
    
            }