Query logs with the GetLogs API - Simple Log Service - Alibaba Cloud Documentation Center

Call the GetLogs operation to query collected logs. This topic provides Python SDK examples for common query and analysis scenarios.

Prerequisites

A RAM user is created and granted the required permissions. Create a RAM user and grant permissions.
The ALIBABA_CLOUD_ACCESS_KEY_ID and ALIBABA_CLOUD_ACCESS_KEY_SECRET environment variables are configured. Configure environment variables on Linux, macOS, and Windows.
Important
- The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Use a RAM user's AccessKey pair for API calls and daily operations.
- Do not hard-code your AccessKey ID or AccessKey Secret in project code. Exposed credentials compromise all resources in your account.
The Log Service SDK for Python is installed. Install Log Service SDK for Python.
Logs are collected to the target Logstore. Data collection overview.
You are familiar with the GetLogs operation parameters. GetLogs.

Usage notes

This topic uses the public endpoint for the China (Hangzhou) region as an example: https://cn-hangzhou.log.aliyuncs.com.
If you access Log Service from other Alibaba Cloud services in the same region as your Project, use the internal endpoint: https://cn-hangzhou-intranet.log.aliyuncs.com.
Supported regions and endpoints: Endpoints.
Call is_completed() on the response object to check whether the query results are precise.
- If is_completed() returns true, the query is complete and the results are accurate.
- If is_completed() returns false, the results are imprecise and incomplete. Repeat the request to obtain complete results. Possible causes of imprecise queries.

Sample raw log

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Examples

The following Python examples demonstrate common log query and analysis scenarios.

Important

The query parameter in GetLogs accepts search and analysis statements:

When query contains only a search statement, line controls the number of returned logs.
When query contains a search and analysis statement, line is ignored. Use LIMIT to control the number of returned rows. LIMIT clause.

Search statement syntax: Basic syntax.

Example 1: Query logs by using a keyword

Search logs by keyword path-0/file-5:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. For more information, see Endpoints. 
    # This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------

Process finished with exit code 0

Example 2: Query logs by specifying a field

Search for logs where request_method is POST, with line set to 3:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by a specific field.
    # Query for logs where the request method is POST.
    query = 'request_method:POST'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'tv0m'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', '6joc'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', 'da8'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
-------------Query is finished.-------------

Process finished with exit code 0

Example 3: Analyze logs by using an SQL statement

Count POST requests using request_method:POST|select COUNT(*) as pv:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Analyze logs by using an analysis statement.
    # Query for logs where the request method is POST and count the number of page views (PVs).
    query = 'request_method:POST|select COUNT(*) as pv'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies a search and analysis statement. The line parameter is ignored. 
    # The number of results is determined by the analysis statement, which returns one row.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('pv', '2918')])
-------------Query is finished.-------------

Process finished with exit code 0

Example 4: Analyze logs by using the GROUP BY clause

Count POST requests grouped by host, sorted by PV in descending order, using request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Count POST requests, group them by host, and sort them by PV count.
    # The LIMIT clause in the analysis statement restricts the output to 5 results.
    query = 'request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # The query parameter specifies the search and analysis statement. 
    # The line and reverse parameters are ignored. The number and order of results are determined by the analysis statement.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('host', 'www.example.com'), ('pv', '7')])
dict_items([('host', 'www.example.org'), ('pv', '6')])
dict_items([('host', 'www.example.net'), ('pv', '6')])
dict_items([('host', 'www.example.edu'), ('pv', '5')])
dict_items([('host', 'www.aliyundoc.com'), ('pv', '4')])
-------------Query is finished.-------------

Process finished with exit code 0

Example 5: Write specific fields in obtained logs to a local file

Query logs by keyword path-0/file-5 and write a specific field value to a local file (log.txt):

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')
    
    # Extract the value of the 'remote_user' key from each log and save it to a local file.
    print('-------------Start writing logs to a local file.-------------')
    for loglocal in response.get_logs():
        filename = 'log.txt'
        with open(filename, mode='a') as fileobject:
            fileobject.write(loglocal.contents.get('remote_user')+'\n')
    print('-------------Finished writing logs to the local file.-------------')

if __name__ == '__main__':
    main()

Response:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------
-------------Start writing logs to a local file.-------------
-------------Finished writing logs to the local file.-------------

Process finished with exit code 0

Content of log.txt generated in the same directory as GetLogsTest.py:

nhf3g
ysu
l_k

References

If an API call fails, see API error codes to resolve the error.
Log Service also supports Alibaba Cloud common SDKs. Log Service SDK Center - Alibaba Cloud OpenAPI Portal.
Log Service provides a CLI for automating configurations. Log Service command-line interface (CLI).