Use GetLogs to query logs - Simple Log Service - Alibaba Cloud Documentation Center

After you collect logs, you can call the GetLogs operation to query the collected logs. This topic provides examples on how to query collected logs by calling the GetLogs operation.

Prerequisites

Logs are collected. For more information, see Data collection overview.
A Resource Access Management (RAM) user is created, and the required permissions are granted to the RAM user. For more information, see Create a RAM user and grant permissions to the RAM user.
The ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables are configured. For more information, see Configure environment variables.
Important
- The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. We recommend that you use the AccessKey pair of a RAM user to call API operations or perform routine O&M.
- We recommend that you do not save the AccessKey ID or AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and the security of all resources within your account may be compromised.
Simple Log Service SDK for Java is installed. For more information, see Install Simple Log Service SDK for Java.
You are familiar with the parameters of the GetLogs operation. For more information, see GetLogs.
The sample code that is used in this topic is based on the SDK version aliyun-log-0.6.69. If the system reports one of the following errors, we recommend that you update the SDK to the latest version or change the version of the SDK: an error that indicates a version conflict or an error indicating that no operations are found during debugging, such as an error indicating that no GetLogs operation is found.

Usage notes

In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used, which is https://cn-hangzhou.log.aliyuncs.com. If you want to access Simple Log Service by using other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is https://cn-hangzhou-intranet.log.aliyuncs.com. For more information about the mappings between the endpoints and regions that are supported by Simple Log Service, see Endpoints.
You can call the IsCompleted() method in the response object of the SDK code to check whether the query result is accurate.
- If the IsCompleted() method returns true, the query is successful and the query results are accurate and complete.
- If the IsCompleted() method returns false, the query is successful, but the query results are inaccurate and incomplete. To obtain the complete results, you must repeat the request. For more information about inaccurate query results, see Cause.

Raw log

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Examples of log query and analysis

The following sample Java code provides examples on how to query and analyze logs.

Important

When you call the GetLogs operation by using Simple Log Service SDK for Java, take note of the following items:

If you set query to only a search statement, for example, path-0/file-5, the line parameter is valid and is used to specify the maximum number of logs that can be returned. The maximum value of line is 100. To obtain more logs, you can use an SQL LIMIT clause. For more information, see LIMIT clause.
If you set query to a query statement, for example, request_method:POST | SELECT host, COUNT(*) AS pv GROUP BY host LIMIT 5, the line parameter is invalid. You must use an SQL LIMIT clause to specify the maximum number of logs that can be returned. For more information, see LIMIT clause.

For more information about query statements, see Basic syntax.

Example 1: Query logs by using a keyword

In this example, a file named GetLogsTest.java is created. The file is used to query logs that contain the keyword path-0/file-5. The line parameter specifies the maximum number of logs that can be returned. In this example, line is set to 3. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
         // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute a query statement in the specified Logstore. 
        try {
            // Use the keyword path-0/file-5 to query logs. 
            String query = "path-0/file-5";

            int from = (int) (new Date().getTime() / 1000 - 300);
            int to = (int) (new Date().getTime() / 1000);

            // In this example, the query parameter is set to a search statement and the line parameter is set to 3. The line parameter specifies the maximum number of logs that can be returned. Maximum value of line: 100. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, 3, 0,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("Returned query result count :" + logsResponse.GetCount());
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("log time : " + item.mLogTime);
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

Response:

-------------Query is started.-------------
Returned query result count :3
from time is :1644573549
to time is :1644573849
log time : 1644573808
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
log time : 1644573808
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
log time : 1644573788
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","request_uri":"/request/path-0/file-5"...}
-------------Query is finished.-------------

Process finished with exit code 0

Example 2: Query logs by specifying a field

In this example, a file named GetLogsTest.java is created. The file is used to query logs whose request method is POST. The line parameter specifies the maximum number of logs that can be returned. In this example, line is set to 3. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
        // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute an SQL statement in the specified Logstore. 
        try {
            // Query the logs whose request method is POST. 
            String query = "request_method:POST";

            int from = (int) (new Date().getTime() / 1000 - 300);
            int to = (int) (new Date().getTime() / 1000);

            // In this example, the query parameter is set to a search statement and the line parameter is set to 3. The line parameter specifies the maximum number of logs that can be returned. Maximum value of line: 100. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, 3, 0,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("Returned query result count :" + logsResponse.GetCount());
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("log time : " + item.mLogTime);
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

Response:

-------------Query is started.-------------
Returned query result count :3
from time is :1644574151
to time is :1644574451
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"3604","request_method":"POST"...}
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"3369","request_method":"POST"...}
log time : 1644574438
Jsonstring : {"remote_addr":"203.0.XX.XX","__topic__":"nginx_access_log","body_bytes_sent":"12714","request_method":"POST"...}
-------------Query is finished.-------------

Process finished with exit code 0

Example 3: Analyze logs by using an SQL statement

In this example, a file named GetLogsTest.java is created. The file is used to query logs whose request method is POST and count the number of page views (PVs) for POST requests. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
        // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute an SQL statement in the specified Logstore. 
        try {
            // Query logs whose request method is POST and count the number of PVs for POST requests. 
            String query = "request_method:POST|select COUNT(*) as pv";

            int from = (int) (new Date().getTime() / 1000 - 300);
            int to = (int) (new Date().getTime() / 1000);

            // In this example, the query parameter is set to a query statement and the line parameter is invalid. The maximum number of logs that can be returned is determined by the query parameter. Only one log can be returned. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, 3, 0,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("Returned query result count :" + logsResponse.GetCount());
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("log time : " + item.mLogTime);
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

Response:

-------------Query is started.-------------
Returned query result count :1
from time is :1644574354
to time is :1644574654
log time : 1644574354
Jsonstring : {"pv":"162","logtime":1644574354}
-------------Query is finished.-------------

Process finished with exit code 0

Example 4: Analyze logs by using the GROUP BY clause

In this example, a file named GetLogsTest.java is created. The file is used to query logs whose request method is POST and group the obtained logs by host. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
        // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute an SQL statement in the specified Logstore. 
        try {
            // Query logs whose request method is POST and group the obtained logs by host. 
            // Use a LIMIT clause to specify the maximum number of logs that can be returned to 5. 
            String query = "request_method:POST|select host, COUNT(*) as pv group by host limit 5";

            int from = (int) (new Date().getTime() / 1000 - 300);
            int to = (int) (new Date().getTime() / 1000);

            // In this example, the query parameter is set to a query statement and the line parameter is invalid. The maximum number of logs that can be returned is determined by the query parameter. A maximum of five logs can be returned. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, 3, 0,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("Returned query result count :" + logsResponse.GetCount());
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("log time : " + item.mLogTime);
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

Response:

-------------Query is started.-------------
Returned query result count :5
from time is :1644574445
to time is :1644574745
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example1.com","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.org","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.net","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.edu","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.aliyundoc.com","logtime":1644574445}
-------------Query is finished.-------------

Process finished with exit code 0

Example 5: Analyze logs by using the GROUP BY clause (200 logs returned)

In this example, a file named GetLogsTest.java is created. The file is used to query logs whose request method is POST, group the obtained logs by host, and return a maximum of 200 logs. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
        // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute an SQL statement in the specified Logstore. 
        try {
            // Query logs whose request method is POST and group the obtained logs by host. 
            // Use a LIMIT clause to specify the maximum number of logs that can be returned. 
            String old_query = "request_method:POST|select host, COUNT(*) as pv group by host limit ";

            int from = (int) (new Date().getTime() / 1000 - 300);
            int to = (int) (new Date().getTime() / 1000);
            int log_offset = 0;
            int log_line = 200;
            
            String query = old_query + log_offset + "," + log_line;

            // In this example, the query parameter is set to a query statement and the line parameter is invalid. The maximum number of logs that can be returned is determined by the query parameter. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, 10, 0 ,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("Returned query result count :" + logsResponse.GetCount());
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("log time : " + item.mLogTime);
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

Response:

-------------Query is started.-------------
Returned query result count :200
from time is :1644574445
to time is :1644574745
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example1.com","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.org","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.net","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.example.edu","logtime":1644574445}
log time : 1644574445
Jsonstring : {"pv":"1","host":"www.aliyundoc.com","logtime":1644574445}
......
-------------Query is finished.-------------

Process finished with exit code 0

Example 6: Query the total number of logs within the previous hour by using an SQL statement

In this example, a file named GetLogsTest.java is created. The file is used to query the total number of logs within the previous hour by using the *|select count(*) as count query statement. Example:

import com.aliyun.openservices.log.Client;
import com.aliyun.openservices.log.common.LogItem;
import com.aliyun.openservices.log.common.QueriedLog;
import com.aliyun.openservices.log.exception.LogException;
import com.aliyun.openservices.log.response.GetLogsResponse;

import java.util.Date;

public class GetLogsTest {

    public static void main(String[] args) throws LogException {
        // In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
        String accessId = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_ID");
        String accessKey = System.getenv("ALIBABA_CLOUD_ACCESS_KEY_SECRET");
        // The name of the project. 
        String project = "your-project-name";
        // The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
        String host = "cn-hangzhou.log.aliyuncs.com";
        // The name of the Logstore. 
        String logStore = "your-logstore-name";

        // Create a Simple Log Service client. 
        Client client = new Client(host, accessId, accessKey);

        // Execute an SQL statement in the specified Logstore. 
        try {
            // Query the total number of logs. 
            String query = "*|select count(*) as count";
            // Set the query time range to 3600 in seconds, which is equivalent to 1 hour. 
            int from = (int) (new Date().getTime() / 1000 - 3600);
            int to = (int) (new Date().getTime() / 1000);
            int log_offset = 0;
            int log_line = 200;

            // In this example, the query statement specified in the query parameter is used to query the total number of logs within the specified time range. 
            GetLogsResponse logsResponse = client.GetLogs(project, logStore, from, to, "", query, log_line, log_offset,true);
            System.out.println("-------------Query is started.-------------");
            System.out.println("from time is :" + from);
            System.out.println("to time is :" + to);
            System.out.println("Returned query result count :" + logsResponse.GetCount());

            for (QueriedLog log : logsResponse.getLogs()) {
                LogItem item = log.GetLogItem();
                System.out.println("Jsonstring : " + item.ToJsonString());
            }
            System.out.println("-------------Query is finished.-------------");

        } catch (LogException e) {
            System.out.println("LogException e :" + e.toString());
            System.out.println("error code :" + e.GetErrorCode());
            System.out.println("error message :" + e.GetErrorMessage());
            throw e;
        }
    }
}

The returned result shows that the total number of logs within the previous hour is 19,051. Response:

from time is :1675041679
to time is :1675045279
Returned sql result count :1
Jsonstring : {"count":"19051","logtime":1675041679}
-------------Query is finished.-------------

References

If the response that is returned by Log Service contains error information after you call an API operation, the call fails. You can handle errors based on the error codes that are returned when API calls fail. For more information, see Error codes.
Alibaba Cloud OpenAPI Explorer provides debugging capabilities, SDKs, examples, and related documents. You can use OpenAPI Explorer to debug Log Service API operations without the need to manually encapsulate or sign requests. For more information, visit OpenAPI Portal.
Log Service provides the command-line interface (CLI) to meet the requirements for automated configurations in Log Service. For more information, see Log Service CLI.
For more information about sample code, see Alibaba Cloud Log Service SDK for Java on GitHub.
For more information about sample code, see Alibaba Cloud Log Service SDK for Python on GitHub.