Log Service provides the LogSearch/Analytics function to query and analyze large amounts of logs in real time. You can use this function by enabling the index and field statistics.

Functional advantages

  • Real-time: Logs can be analyzed immediately after they are written.
  • Fast:
    • Query: Billions of data can be processed and queried within one second (with five conditions).
    • Analysis: Hundreds of millions of data can be aggregated and analyzed within one second (with aggregation by five dimensions and the GroupBy condition).
  • Flexible: Query and analysis conditions can be changed as required to obtain results in real time.
  • Ecologic: Besides functions such as reports, dashboards, and quick analysis provided in the console, Log Service seamlessly interconnects with products such as Grafana, DataV, and Jaeger, and supports protocols such as RESTful  API and JDBC.

Basic concepts

Without enabling the LogSearch/Analytics (index) function, raw data is consumed according to the sequence in the shard, which is similar to Kafka. With the LogSearch/Analytics (index) function enabled, besides the consumption in sequence, you can also count and query the logs. For the difference between log consumption and log query, see Differences between log consumption and log query. 

Enable an index

  1. Log on to the Log Service console. On the Project List page, click the project name.
  2. Select the Logstore, and click Search.  Then, click Enable Index in the upper-right corner. If you have enabled the index before, click Index Attributes > Modify.. 
    • After enabling the query and statistics, data is indexed in the backend. Traffic and storage space for the index are required.
    • ◦If this function is not required, click Disable to disable it.
  3. Enter the Settings menu to complete configuration.
Data types

You can configure the type of each key in a log (full text index is a special key, whose value is the log). Currently, Log Service supports the following data types.

Category Type Description Query example
Basic TEXT The text type that supports keyword and fuzzy match.  uri:"login*" method:"post" 
Basic Long The value type that supports interval query. status>200, status in [200, 500]
Basic Double The value type with a float. price>28.95, t in [20.0, 37]
Combination JSON The content is a JSON field, which is of the text type by default and supports the nested model.  You can configure indexes of text, long,  and double type for element b under a by using the path format such as a.b . The field type after the configuration is subject to the configuration. level0.key>29.95 level0.key2:"action"
Combination Full text Use a log as the text for query.  error and "login fail" 

Query and analysis syntax

Real-time query and analysis is composed of Search and Analytics, which are separated with a vertical line ( | ):

$Search |$Analytics
  • •Search: The query condition, which is generated by using keywords, fuzzy match conditions, values, ranges, and combination conditions.  If Search is empty or an asterisk (*), all data is queried.
  • Analytics: Calculate and count the query results or the full data.
Note
Both Search and Analytics are optional. If Search is empty, all the data in the specified period is not filtered and the results are counted directly.  If Analytics is empty, the query results are returned and no statistics are collected.
Note

For more information, see Query syntax, Syntax description.

Query examples

Besides time, the following log also contains four key values.

Sequence number Key Type
0 time -
1 class text
2 status Long
3 Latency double
4 message json 
0. time:2018-01-01 12:00:00
  1. class:central-log
  2. status:200
  3. latency:68.75
  4. message:
    
      "methodName": "getProjectInfo",
      "success": true,
      "remoteAddress": "1.1.1.1:11111",
      "usedTime": 48,
      "param": {
              "projectName": "ali-log-test-project",
              "requestId": "d3f0c96a-51b0-4166-a850-f4175dde7323"
      
      "result": {
          "message": "successful",
          "code": "200",
          "data": {
              "clusterRegion": "ap-southeast-1",
              "ProjectName": "ali-log-test-project",
              "CreateTime": "2017-06-08 20:22:41"
          
          "success": true
      }
Configuration is as follows:
Figure 1. Index settings


Where:

  • ① indicates that all the data of the string type and bool type in the JSON field can be queried.
  • ② indicates that data of the long type can be queried.
  • ③ indicates that you can analyze the configured field by using SQL statements.
Example 1: Query string, bool type 
class : cental*
message.traceInfo.requestId : 92.137_1518139699935_5599
message.param.projectName : ali-log-test-project
message.success : true
Note
  • No configurations in the JSON field are needed.
  • JSON Map and Array are auto scaling and support multi-level nesting. Each layer is separated  with a period (.).
Example 2:  Query double、long type 
latency>40
message.usedTime > 40
Note
You configure JSON fields independently. The fields must not be in array.
Example 3: Combined query 
class : cental* and message.usedTime > 40 not message.param.projectName:ali-log-test-project

Other information

If you query a large amount of log data (such as a long query time span, where the data volume is over 10 billion), one request cannot query all the data. In this case, Log Service returns the existing data and notifies you that the query result is incomplete.

At the same time, the server caches the results of the query within 15  minutes.  When the query result is partially cached, the server continues to scan log data that has not been cached. To reduce the workload of merging multiple query results, Log Service merges the result of the cache hit with the result of the new query and returns it to you.

Therefore, Log Service enables you to get the final result by calling the interface repeatedly with the same parameters.