Simple Log Service:Use custom query statements to analyze alert logs

Log topics

The topic of an alert log varies based on the status of the alert log. You can identify the status of an alert log based on the value of the __topic__ field. The following table describes common log topics.

Log topic	Description
__topic__: alert_state	Data is evaluated based on the alert monitoring rule.
__topic__: alert_received	The alert management system receives alerts.
__topic__: alert_routed	Alerts are merged by route based on alert policies after the alerts are received.
__topic__: alert_pre_filter	Alerts enter the suppression or silence stage.
__topic__: alert_silenced	Alerts are suppressed or silenced.
__topic__: alert_pre_notify	Alerts enter the notification sending stage.
__topic__: alert_notified	Alert notifications are sent.
__topic__: system_config	Logs are generated when configuration errors occur.

Procedure

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you want to manage. Example: sls-alert-13****47-cn-hangzhou.

3. On the Log Storage > Logstores tab, click internal-alert-center-log.

4. On the query and analysis page for the Logstore, run custom queries to analyze your alert logs.

   A query statement consists of a search statement and an analytic statement in the Search statement|Analytic statement format. For more information about the syntax of query statements, see Search syntax and functions and SQL analysis syntax.

   • Example 1: Calculate the number of alert monitoring rules that trigger alerts within a specific period of time and calculate the number of triggers for each alert monitoring rule within the period of time.

     • Query statement

       __topic__: alert_received |
       select
         "alert.project" as project,
         "alert.alert_name" as alert_name,
         count(*) as cnt
       group by
         project,
         alert_name
       order by
         cnt desc

     • The query results show the project, the name of the alert monitoring rule, and its corresponding trigger count.

   • Example 2: Calculate the number of times that alert notifications failed to be sent by using each notification method within a specific period of time.

     • Query statement

       __topic__: alert_notified and level: error |
       select
         "notifierConfig.type" as notificationType,
         count(*) as cnt
       group by
         notificationType
       order by
         cnt desc

     • The query results show each notification type and the corresponding count, such as 77 for Webhook and 39 for DingTalk.

   • Example 3: View the causes for alert notification failures.

     • Query statement

       __topic__: system_config
       and alert.alert_id: alert -1626423664 -868572 |
       select
         level,
         error,
         msg,
         "desc"

     • Query and analysis results

       • If the query returns results, this indicates a configuration problem. For details about configuration errors, see Configuration errors. The results include the level, error, msg, and desc columns. For example, a result with level as warning, error as UserGroupEmpty, msg as group sls.app.audit.builtin is empty, and desc as this group will not be notified indicates that the user group sls.app.audit.builtin is empty and will not receive notifications.

       • If no data is contained in the query and analysis results, the specified notification method may be invalid. For example, the specified webhook URL is invalid or the specified DingTalk chatbot is deleted. You can execute the following query statement to view detailed causes:

         __topic__: alert_notified
         and level: error
         and alert.alert_id: alert -1626423664 -868572 |
         select
           error

         In this example, the following messages are returned. The returned messages indicate that alert notifications failed to be sent because the specified webhook URL is invalid.

         Unexpected status code 400, response: {"code":"InvalidParameter","message":"invalid uri:http://localhost Failed to connect to localhost port 80: Connection refused"}.