Consultar logs com a API GetLogs - Simple Log Service: Central de Documentação do Alibaba Cloud

Chame a operação GetLogs para consultar logs coletados. Este tópico apresenta exemplos do SDK para Python em cenários comuns de consulta e análise.

Pré-requisitos

Crie um usuário RAM e conceda as permissões necessárias. Criar um usuário RAM e conceder permissões.
Configure as variáveis de ambiente ALIBABA_CLOUD_ACCESS_KEY_ID e ALIBABA_CLOUD_ACCESS_KEY_SECRET. Configurar variáveis de ambiente no Linux, macOS e Windows.
Importante
- O par de AccessKey de uma conta Alibaba Cloud tem permissões em todas as operações de API. Use o par de AccessKey de um usuário RAM para chamadas de API e operações diárias.
- Não codifique seu AccessKey ID ou AccessKey Secret diretamente no código do projeto. Credenciais expostas comprometem todos os recursos da sua conta.
Instale o Log Service SDK for Python. Instalar o Log Service SDK for Python.
Colete logs no Logstore de destino. Visão geral da coleta de dados.
Conheça os parâmetros da operação GetLogs. GetLogs.

Observações de uso

Este tópico usa o endpoint público da região China (Hangzhou) como exemplo: https://cn-hangzhou.log.aliyuncs.com.

Para acessar o Log Service a partir de outros serviços da Alibaba Cloud na mesma região do seu Project, use o endpoint interno: https://cn-hangzhou-intranet.log.aliyuncs.com.

Regiões e endpoints compatíveis: Endpoints.
Chame is_completed() no objeto de resposta para verificar a precisão dos resultados da consulta.
- Se is_completed() retornar true, a consulta foi concluída e os resultados são exatos.
- Caso is_completed() retorne false, os resultados estarão imprecisos e incompletos. Repita a solicitação para obter os resultados completos. Possíveis causas de consultas imprecisas.

Exemplo de log bruto

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Exemplos

Os exemplos em Python a seguir demonstram cenários comuns de consulta e análise de logs.

Importante

O parâmetro query no GetLogs aceita instruções de pesquisa e análise:

Quando query contém apenas uma instrução de pesquisa, line controla o número de logs retornados.
Se query incluir uma instrução de pesquisa e análise, line será ignorado. Use LIMIT para controlar o número de linhas retornadas. Cláusula LIMIT.

Sintaxe da instrução de pesquisa: Sintaxe básica.

Exemplo 1: Consultar logs usando uma palavra-chave

Pesquise logs pela palavra-chave path-0/file-5:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. For more information, see Endpoints. 
    # This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Resposta:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------

Process finished with exit code 0

Exemplo 2: Consultar logs especificando um campo

Pesquise logs em que request_method seja POST, com line definido como 3:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by a specific field.
    # Query for logs where the request method is POST.
    query = 'request_method:POST'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Resposta:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'tv0m'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', '6joc'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', 'da8'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
-------------Query is finished.-------------

Process finished with exit code 0

Exemplo 3: Analisar logs usando uma instrução SQL

Conte solicitações POST usando request_method:POST|select COUNT(*) as pv:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Analyze logs by using an analysis statement.
    # Query for logs where the request method is POST and count the number of page views (PVs).
    query = 'request_method:POST|select COUNT(*) as pv'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies a search and analysis statement. The line parameter is ignored. 
    # The number of results is determined by the analysis statement, which returns one row.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Resposta:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('pv', '2918')])
-------------Query is finished.-------------

Process finished with exit code 0

Exemplo 4: Analisar logs usando a cláusula GROUP BY

Conte solicitações POST agrupadas por host, ordenadas por PV em ordem decrescente, usando request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5:

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Count POST requests, group them by host, and sort them by PV count.
    # The LIMIT clause in the analysis statement restricts the output to 5 results.
    query = 'request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # The query parameter specifies the search and analysis statement. 
    # The line and reverse parameters are ignored. The number and order of results are determined by the analysis statement.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Resposta:

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('host', 'www.example.com'), ('pv', '7')])
dict_items([('host', 'www.example.org'), ('pv', '6')])
dict_items([('host', 'www.example.net'), ('pv', '6')])
dict_items([('host', 'www.example.edu'), ('pv', '5')])
dict_items([('host', 'www.aliyundoc.com'), ('pv', '4')])
-------------Query is finished.-------------

Process finished with exit code 0

Exemplo 5: Gravar campos específicos dos logs obtidos em um arquivo local

Consulte logs pela palavra-chave path-0/file-5 e grave o valor de um campo específico em um arquivo local (log.txt):

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')
    
    # Extract the value of the 'remote_user' key from each log and save it to a local file.
    print('-------------Start writing logs to a local file.-------------')
    for loglocal in response.get_logs():
        filename = 'log.txt'
        with open(filename, mode='a') as fileobject:
            fileobject.write(loglocal.contents.get('remote_user')+'\n')
    print('-------------Finished writing logs to the local file.-------------')

if __name__ == '__main__':
    main()

Resposta:

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------
-------------Start writing logs to a local file.-------------
-------------Finished writing logs to the local file.-------------

Process finished with exit code 0

Conteúdo do arquivo log.txt gerado no mesmo diretório que GetLogsTest.py:

nhf3g
ysu
l_k

Referências

Se uma chamada de API falhar, consulte Códigos de erro da API para resolver o erro.
O Log Service também é compatível com SDKs comuns da Alibaba Cloud. Log Service SDK Center - Alibaba Cloud OpenAPI Portal.
O Log Service oferece uma CLI para automatizar configurações. Interface de linha de comando (CLI) do Log Service.