Interroger les journaux avec l'API GetLogs - Simple Log Service

Appelez l'opération GetLogs pour interroger les journaux collectés. Cette rubrique fournit des exemples de SDK Python pour les scénarios courants d'interrogation et d'analyse.

Prérequis

Un utilisateur RAM est créé et dispose des autorisations requises. Créer un utilisateur RAM et accorder des autorisations.
Les variables d'environnement ALIBABA_CLOUD_ACCESS_KEY_ID et ALIBABA_CLOUD_ACCESS_KEY_SECRET sont configurées. Configurer les variables d'environnement sous Linux, macOS et Windows.
Important
- La paire AccessKey d'un compte Alibaba Cloud dispose d'autorisations sur toutes les opérations API. Utilisez la paire AccessKey d'un utilisateur RAM pour les appels API et les opérations quotidiennes.
- Ne codez pas en dur votre AccessKey ID ou AccessKey Secret dans le code du projet. L'exposition des identifiants compromet toutes les ressources de votre compte.
Le SDK Log Service pour Python est installé. Installer le SDK Log Service pour Python.
Les journaux sont collectés dans le Logstore cible. Présentation de la collecte de données.
Vous connaissez les paramètres de l'opération GetLogs. GetLogs.

Remarques sur l'utilisation

Cette rubrique utilise l'endpoint public de la région Chine (Hangzhou) à titre d'exemple : https://cn-hangzhou.log.aliyuncs.com.

Si vous accédez à Log Service depuis d'autres services Alibaba Cloud situés dans la même région que votre Project, utilisez l'endpoint interne : https://cn-hangzhou-intranet.log.aliyuncs.com.

Régions et endpoints pris en charge : Endpoints.
Appelez is_completed() sur l'objet de réponse pour vérifier si les résultats de la requête sont précis.
- Si is_completed() renvoie true, la requête est terminée et les résultats sont exacts.
- Si is_completed() renvoie false, les résultats sont imprécis et incomplets. Répétez la demande pour obtenir des résultats complets. Causes possibles de requêtes imprécises.

Exemple de journal brut

body_bytes_sent:1750
host:www.example.com
http_referer:www.example.com
http_user_agent:Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; it-it) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27
http_x_forwarded_for:203.0.XX.XX
remote_addr:203.0.XX.XX
remote_user:p288
request_length:13741
request_method:GET
request_time:71
request_uri:/request/path-1/file-1
http_code:200
time_local:11/Aug/2021:06:52:27
upstream_response_time:0.66

Exemples

Les exemples Python suivants illustrent des scénarios courants d'interrogation et d'analyse des journaux.

Important

Le paramètre query dans GetLogs accepte les instructions de recherche et d'analyse :

Lorsque query contient uniquement une instruction de recherche, line contrôle le nombre de journaux renvoyés.
Lorsque query contient une instruction de recherche et d'analyse, line est ignoré. Utilisez LIMIT pour contrôler le nombre de lignes renvoyées. Clause LIMIT.

Syntaxe de l'instruction de recherche : Syntaxe de base.

Exemple 1 : Interroger les journaux à l'aide d'un mot-clé

Recherchez les journaux par mot-clé path-0/file-5 :

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. For more information, see Endpoints. 
    # This example uses the endpoint for the China (Hangzhou) region. Replace it with your actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Réponse :

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------

Process finished with exit code 0

Exemple 2 : Interroger les journaux en spécifiant un champ

Recherchez les journaux où request_method est POST, avec line défini sur 3 :

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by a specific field.
    # Query for logs where the request method is POST.
    query = 'request_method:POST'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Réponse :

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'tv0m'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', '6joc'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
dict_items([ ('remote_user', 'da8'), ('time_local', '14/Feb/2022:06:59:08'), ('request_method', 'POST')...])
-------------Query is finished.-------------

Process finished with exit code 0

Exemple 3 : Analyser les journaux à l'aide d'une instruction SQL

Comptez les requêtes POST en utilisant request_method:POST|select COUNT(*) as pv :

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Analyze logs by using an analysis statement.
    # Query for logs where the request method is POST and count the number of page views (PVs).
    query = 'request_method:POST|select COUNT(*) as pv'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies a search and analysis statement. The line parameter is ignored. 
    # The number of results is determined by the analysis statement, which returns one row.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Réponse :

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('pv', '2918')])
-------------Query is finished.-------------

Process finished with exit code 0

Exemple 4 : Analyser les journaux à l'aide de la clause GROUP BY

Comptez les requêtes POST groupées par hôte, triées par PV dans l'ordre décroissant, en utilisant request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5 :

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Count POST requests, group them by host, and sort them by PV count.
    # The LIMIT clause in the analysis statement restricts the output to 5 results.
    query = 'request_method:POST|select host, COUNT(*) as pv group by host order by pv desc limit 5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # The query parameter specifies the search and analysis statement. 
    # The line and reverse parameters are ignored. The number and order of results are determined by the analysis statement.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')

if __name__ == '__main__':
    main()

Réponse :

ready to query logs from logstore nginx-moni
-------------Query is started.-------------
dict_items([('host', 'www.example.com'), ('pv', '7')])
dict_items([('host', 'www.example.org'), ('pv', '6')])
dict_items([('host', 'www.example.net'), ('pv', '6')])
dict_items([('host', 'www.example.edu'), ('pv', '5')])
dict_items([('host', 'www.aliyundoc.com'), ('pv', '4')])
-------------Query is finished.-------------

Process finished with exit code 0

Exemple 5 : Écrire des champs spécifiques des journaux obtenus dans un fichier local

Interrogez les journaux par mot-clé path-0/file-5 et écrivez une valeur de champ spécifique dans un fichier local (log.txt) :

# encoding: utf-8
import time
import os
from aliyun.log import *

def main():
    # The endpoint for Log Service. This example uses the endpoint for the China (Hangzhou) region. 
    # Replace the value with the actual endpoint.
    endpoint = 'cn-hangzhou.log.aliyuncs.com'
    # This example retrieves the AccessKey ID and AccessKey Secret from environment variables.
    access_key_id = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_ID', '')
    access_key = os.environ.get('ALIBABA_CLOUD_ACCESS_KEY_SECRET', '')

    # The names of the Project and Logstore.
    project = 'your-project-name'
    logstore = 'your-logstore-name'

    # Create a Log Service client.
    client = LogClient(endpoint, access_key_id, access_key)

    # Query logs by using the keyword 'path-0/file-5'.
    query = 'path-0/file-5'

    # The from_time and to_time parameters specify the time range for the query in UNIX timestamp format.
    from_time = int(time.time()) - 3600
    to_time = time.time() + 3600

    print("ready to query logs from logstore %s" % logstore)

    # In this example, the query parameter specifies the search statement, and the line parameter limits the results to 3 log entries.
    request = GetLogsRequest(project, logstore, from_time, to_time, '', query=query, line=3, offset=0, reverse=False)
    response = client.get_logs(request)
    # Print the query results.
    print('-------------Query is started.-------------')
    for log in response.get_logs():
        print(log.contents.items())
    print('-------------Query is finished.-------------')
    
    # Extract the value of the 'remote_user' key from each log and save it to a local file.
    print('-------------Start writing logs to a local file.-------------')
    for loglocal in response.get_logs():
        filename = 'log.txt'
        with open(filename, mode='a') as fileobject:
            fileobject.write(loglocal.contents.get('remote_user')+'\n')
    print('-------------Finished writing logs to the local file.-------------')

if __name__ == '__main__':
    main()

Réponse :

ready to query logs from logstore your-logstore-name
-------------Query is started.-------------
dict_items([ ('remote_user', 'nhf3g'), ('time_local', '14/Feb/2022:06:49:28'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'ysu'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
dict_items([ ('remote_user', 'l_k'), ('time_local', '14/Feb/2022:06:49:38'),  ('request_uri', '/request/path-0/file-5')...])
-------------Query is finished.-------------
-------------Start writing logs to a local file.-------------
-------------Finished writing logs to the local file.-------------

Process finished with exit code 0

Contenu du fichier log.txt généré dans le même répertoire que GetLogsTest.py :

nhf3g
ysu
l_k

Références

En cas d'échec d'un appel API, consultez les codes d'erreur API pour résoudre l'erreur.
Log Service prend également en charge les SDK communs Alibaba Cloud. Centre SDK Log Service - Portail OpenAPI Alibaba Cloud.
Log Service propose une CLI pour automatiser les configurations. Interface de ligne de commande (CLI) Log Service.