Use GetLogs to query logs - Simple Log Service - Alibaba Cloud Documentation Center

After you collect logs, you can call the GetLogs operation to query the collected logs. This topic provides examples to show how to query collected logs by calling the GetLogs operation.

Prerequisites

A Resource Access Management (RAM) user is created, and the required permissions are granted to the RAM user. For more information, see Create a RAM user and grant permissions to the RAM user.
The ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables are configured. For more information, see Configure environment variables.
Important
- The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. We recommend that you use the AccessKey pair of a RAM user to call API operations or perform routine O&M.
- We recommend that you do not save the AccessKey ID or AccessKey secret in your project code. Otherwise, the AccessKey pair may be leaked, and the security of all resources within your account may be compromised.
Simple Log Service SDK for Python is installed. For more information, see Install Simple Log Service SDK for Python.
Logs are collected. For more information, see Data collection overview.
You are familiar with the parameters of the GetLogs operation. For more information, see GetLogs.

Usage notes

In this example, the public Simple Log Service endpoint for the China (Hangzhou) region is used, which is https://cn-hangzhou.log.aliyuncs.com. If you want to access Simple Log Service by using other Alibaba Cloud services that reside in the same region as your project, you can use the internal Simple Log Service endpoint, which is https://cn-hangzhou-intranet.log.aliyuncs.com. For more information about the mappings between the endpoints and regions that are supported by Simple Log Service, see Endpoints.
You can call the is_completed() method in the response object of the SDK code to check whether the query result is accurate.
- If the is_completed() method returns true, the query is successful and the query results are accurate and complete.
- If the is_completed() method returns false, the query is successful, but the query results are inaccurate and incomplete. To obtain the complete results, you must repeat the request. For more information about inaccurate query results, see Cause.

Raw log

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Examples

The following sample Python code provides examples on how to query and analyze logs.

Important

You can configure the query parameter in the GetLogs operation to specify a query statement. Take note of the following items:

If you set query to only a search statement, the line parameter is valid and is used to specify the maximum number of logs that can be returned.
If you set query to a query statement, the line parameter is invalid. You must use a LIMIT clause to specify the maximum number of logs that can be returned. For more information, see LIMIT clause.

For more information about query statements, see Basic syntax.

Example 1: Query logs by using a keyword

Use the search statement path-0/file-5 to query logs. Example:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The Simple Log Service endpoint. For more information, see Endpoints. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the project and the Logstore. 
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Simple Log Service client. 
    client = LogClient(endpoint, access_key_id, access_key)

    # Use the keyword path-0/file-5 to query logs. 
    query = 'path-0/file-5'

    # Configure the from_time and to_time parameters to specify the start time and end time of the time range within which you want to query logs. Set the values to UNIX timestamps. 
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter is set to a search statement and the line parameter is set to 3. The line parameter specifies the maximum number of logs that can be returned. 
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Display the query result. 
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------

Process finished with exit code 0

Example 2: Query logs by specifying a field

Use the search statement request_method:POST to query logs whose request method is POST. The line parameter specifies the maximum number of logs that can be returned. In this example, line is set to 3. Example:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the project and the Logstore. 
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Simple Log Service client. 
    client = LogClient(endpoint, access_key_id, access_key)

    # Specify a field to query logs. 
    # Query the logs whose request method is POST. 
    query = 'request_method:POST'

    # Configure the from_time and to_time parameters to specify the start time and end time of the time range within which you want to query logs. Set the values to UNIX timestamps. 
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter is set to a search statement and the line parameter is set to 3. The line parameter specifies the maximum number of logs that can be returned. 
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Display the query result. 
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'tv0m'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', '6joc'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', 'da8'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
-------------Query is finished.-------------

Process finished with exit code 0

Example 3: Analyze logs by using an SQL statement

Use the query statement request_method:POST|select COUNT(*) as pv to query logs whose request method is POST and count the number of page views (PVs) for POST requests. Example:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the project and the Logstore. 
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Simple Log Service client. 
    client = LogClient(endpoint, access_key_id, access_key)

    # Use an SQL statement to analyze logs. 
    # Query the logs whose request method is POST and count the number of PVs for POST requests. 
    query = 'request_method:POST|select COUNT(*) as pv'

    # Configure the from_time and to_time parameters to specify the start time and end time of the time range within which you want to query logs. Set the values to UNIX timestamps. 
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter is set to a query statement and the line parameter is invalid. The maximum number of logs that can be returned is determined by the query parameter. Only one log can be returned. 
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Display the query result. 
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('pv', '2918')])
-------------Query is finished.-------------

Process finished with exit code 0

Example 4: Analyze logs by using the GROUP BY clause

Use the query statement request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5 to query logs whose request method is POST, group the obtained logs by host, and sort the obtained logs by PV. Example:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the project and the Logstore. 
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Simple Log Service client. 
    client = LogClient(endpoint, access_key_id, access_key)

    # Query the logs whose request method is POST, group the obtained logs by host, and sort the obtained logs by PV. 
    # Use the LIMIT clause to limit the maximum number of logs that can be returned to 5. 
    query = 'request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5'

    # Configure the from_time and to_time parameters to specify the start time and end time of the time range within which you want to query logs. Set the values to UNIX timestamps. 
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter is set to a query statement and the line parameter is invalid. The maximum number of logs that can be returned is determined by the query parameter. A maximum of five logs can be returned. The reverse parameter is invalid. Logs are sorted based on the order that is specified in the SQL statement. 
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Display the query result. 
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('host', 'www.example.com'), ('pv', '7')])
dict_items([('host', 'www.example.org'), ('pv', '6')])
dict_items([('host', 'www.example.net'), ('pv', '6')])
dict_items([('host', 'www.example.edu'), ('pv', '5')])
dict_items([('host', 'www.aliyundoc.com'), ('pv', '4')])
-------------Query is finished.-------------

Process finished with exit code 0

Example 5: Write specific fields in obtained logs to a local file

Use the search statement path-0/file-5 to query logs and write the values of a specific field in the obtained logs to a local file named log.txt. Example:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The Simple Log Service endpoint. In this example, the Simple Log Service endpoint for the China (Hangzhou) region is used. Replace the parameter value with the actual endpoint. 
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # In this example, the AccessKey ID and AccessKey secret are obtained from environment variables. 
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the project and the Logstore. 
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Simple Log Service client. 
    client = LogClient(endpoint, access_key_id, access_key)

    # Use the keyword path-0/file-5 to query logs. 
    query = 'path-0/file-5'

    # Configure the from_time and to_time parameters to specify the start time and end time of the time range within which you want to query logs. Set the values to UNIX timestamps. 
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter is set to a search statement and the line parameter is set to 3. The line parameter specifies the maximum number of logs that can be returned. 
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Display the query result. 
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')
    
    # Extract the values of the key field from the obtained logs and save the values to a local file. 
    print('-------------Start writing logs to local files.-------------')
    for loglocal in response.get_logs():
        filename = 'log.txt'
        with open(filename, mode='a') as fileobject:
            fileobject.write(loglocal.contents.get('remote_user')+'\n')
    print('-------------Finishing writing logs to local files.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------
-------------Start writing logs to local files.-------------
-------------Finishing writing logs to local files.-------------

Process finished with exit code 0

The following example shows the content of the log.txt file that is generated in the directory of the GetLogsTest.py file:

nhf3g
ysu
l_k

References

If the response that is returned by Log Service contains error information after you call an API operation, the call fails. You can handle errors based on the error codes that are returned when API calls fail. For more information, see Error codes.
Alibaba Cloud OpenAPI Explorer provides debugging capabilities, SDKs, examples, and related documents. You can use OpenAPI Explorer to debug Log Service API operations without the need to manually encapsulate or sign requests. For more information, visit OpenAPI Portal.
Log Service provides the command-line interface (CLI) to meet the requirements for automated configurations in Log Service. For more information, see Log Service CLI.
For more information about sample code, see Alibaba Cloud Simple Log Service SDK for Python on GitHub.