The intelligent inspection feature of Log Service allows you to inspect data such as logs and metrics and identify exceptions in the data in an automated, intelligent, and adaptive manner. This topic describes how to use the intelligent inspection feature to inspect log data from multiple dimensions and label the alerts that are generated.

Prerequisites

  • Access log data is collected from Server Load Balancer (SLB) Layer 7 and stored in a Logstore named slb-log. For more information, see Data collection overview.
  • Indexes are configured for the source Logstore. For more information, see Configure indexes.

Background information

The intelligent inspection feature allows you to inspect non-metric data, such as log data. In most cases, non-metric data refers to regular log data. You can use SQL statements to convert log data into metric data that can be inspected by the intelligent inspection feature.

Data example

Company A ingests its access log data from SLB Layer 7 into Log Service. Then, Company A uses SQL statements to calculate the values of metrics such as queries per second (QPS), latency, volume of inbound traffic, volume of outbound traffic, and success rate. In this case, Company A must set the Entity parameter to host and set the Feature parameter to qps, success_date, inflow, outflow, and latency. If the specified host becomes faulty, Log Service displays changes in the values of the specified metrics in the Log Service console and notifies O&M engineers by using DingTalk. O&M engineers can immediately troubleshoot the faults based on the fault information.

The following figure shows an example of the access log from SLB Layer 7.

Intelligent inspection on log data from multiple dimensions

Step 1: Create an intelligent inspection task

  1. Log on to the Log Service console.
  2. In the Projects section, click the name of the project that you want to view.
  3. In the left navigation sidebar, choose Jobs > Intelligent Inspection.
  4. In the Intelligent Inspection pane, click the plus icon.
  5. In the Basic Information step of the Create Intelligent Inspection Task wizard, configure the following parameters and click Next.
    Basic Information step of the Create Intelligent Inspection Task wizard
  6. In the Algorithm Configurations step of the Create Intelligent Inspection Task wizard, complete the following operations:
    1. In the Data Feature Settings section, set the Data Type parameter to Non-indexed Data. Then, configure the other parameters.
      Non-indexed data
      The following code snippet shows an example of the query statement in the Data Feature Settings section:
      * |
      select
        __time__-__time__ % 60 as time,
        'slb-total' as host,
        COUNT(*) as qps,
        round(1.0 * count_if(status < 400) / COUNT(*), 2) as success_rate,
        sum(request_length) as inflow,
        sum(body_bytes_sent) as outflow,
        round(avg(upstream_response_time) * 1000, 3) as latency
      FROM  log
      group by
        time
      order by
        time asc
      limit
        10000
    2. In the Algorithm Configurations section, configure the following parameters, select an entity from the Data Sampling drop-down list, and then click Sample Data Preview to check whether the parameter settings are suitable for the source data and whether expected results can be obtained.
      Basic Information step of the Create Intelligent Inspection Task wizard
    3. In the Scheduling Settings section, specify the date and time at which you want to start the intelligent inspection task.
      Note After an intelligent inspection task is created, the task starts at the date and time that you specify.
    4. Click Next.
  7. In the Alert Configuration step of the Create Intelligent Inspection Task wizard, configure the following parameters and click Complete.
    For more information about how to obtain the webhook URL of a DingTalk group, see DingTalk-Custom.
    Alert
    If the value of the result.score field for a sample exceeds 0.75, Log Service considers the sample abnormal and sends an alert to the specified DingTalk group.

Step 2: Label alerts

You can label each alert that you receive in the specified DingTalk group.

  • If the alert is positive, click Confirm.
  • If the alert is false, click False Positive.
The following figure shows a sample alert.Label an alert