Log Service allows you to search and analyze large amounts of log data in real time. This topic describes the benefits of the search and analytics feature, the methods to use the feature, and the statement format that is used by the feature.

Benefits

  • Real-time: You can analyze logs immediately after they are collected.
  • Fast:
    • Search: Billions of log entries can be processed and searched within one second (with five conditions).
    • Analytics: Hundreds of millions of log entries can be aggregated and analyzed within one second (with aggregation based on five dimensions and with the GROUP BY condition).
  • Flexible: Search and analytics statements can be changed based on your requirements and results are returned in real time.
  • All-in-one: The search and analytics feature seamlessly integrates with Grafana, DataV, and Jaeger and supports RESTful APIs and JDBC. In addition, the feature also supports multiple sub-features in the console. For example, the search and analytics feature allows you to visualize search and analysis results in charts and dashboards and provides quick analysis based on predefined rules.

Index

You can use the index feature of Log Service to sort log entries based on one or more columns. This feature ensures quick location of log data. For more information, see Enable and configure the index feature for a Logstore.

Log Service supports two types of indexes: full-text index and field index.
  • Full-text index: The full-text index stores the key and value of each field as text. This index allows you to search the keys and values for a keyword.
  • Field index: If you add a field to the index, you can specify the key of the field and a keyword to search the field for the keyword. In Log Service, you can add the fields of the following data types to a field index: text, JSON, long, and double.

Methods

Note Before you use the search and analytics feature, make sure that you have collected logs and enabled and configured the index feature.
  • Use the console

    You can specify a time range and query statement on a search and analytics page in the Log Service console to query log data. For more information, see Query logs and Query syntax.

  • Use the API

    You can call the GetLogs or GetHistograms API operation to query log data.

Statement format

You can execute a query statement to search and analyze log data that is collected by Log Service. Each query statement consists of a search statement and an analytic statement. They are separated by a vertical bar (|). 
Search statement|Analytic statement
Statement Required Description
Search statement No A search statement contains one or more search conditions. A condition can be a keyword, a pattern that contains wildcard characters, a value, or a value range.

If you leave the search statement unspecified or specify an asterisk (*) as the search statement, it indicates that no condition is specified and all log data is returned. For more information, see Query syntax.

Analytic statement No An analytic statement is used to aggregate or analyze a search result.

If you leave the analytic statement unspecified, it indicates that the search result is returned without being analyzed. For more information, see Real-time analysis.

Additional considerations

If you need to run a query statement to search a large number of log entries (for example, you need to search more than 10 billion log entries), Log Service may fail to return the entire result. Instead, Log Service returns a partial result and notifies you that the returned result is incomplete. At the same time, the Log Service server caches the partial query result for 15 minutes. If you run the query statement again and part of the query result hits the cache, Log Service continues to search for the matching log entries that are not cached. Log Service then merges and returns the query results to you. Therefore, you can repeatedly run the query statement to obtain the complete result.