edit-icon download-icon

Query logs

Last Updated: Mar 26, 2018

Log Service supports querying many data types, such as text, value, and JSON. Fuzzy match is also supported.

Note: You must enable the LogSearch/Analytics function before using Log Service to query data. To enable the LogSearch/Analytics function, see Overview.

Instructions

  1. Make sure you have enabled the index and analysis functions before querying and analyzing logs.
    • Enable index: Click Enable in the upper-right of the query page.
    • Enable analysis: Turn on the Enable Analytics switch for a specific field on the Search & Analysis page after you enable the index.
  2. The full text index and key/value index cannot be disabled at the same time.
  3. The Case Sensitive and Token attributes are unavailable if the index type is long or double.
  4. You can perform fast statistics and analysis by using SQL statements only after enabling analytics for the corresponding column.
  5. You must specify the Logstore and time range to be queried when querying logs. The Logstore must belong to a specified project and the time range cannot exceed the log storage period of the Logstore.

Supported data types

1. Text type

Similar to search engines, data of text type is queried based on terms. Therefore, you must configure the Case Sensitive and Token.

1.1 Case sensitive

Determine whether or not to support case sensitive when querying raw logs. For example, the raw log is internalError.

  • After turning off the Case Sensitive switch, the sample log can be queried based on the keyword INTERNALERROR or internalerror.
  • After turning on the Case Sensitive switch, the sample log can only be queried based on the keyword internalError.

1.2 Token

You can separate the contents of a raw log into several keywords by using a token.

For example, the raw log is /url/pic/abc.gif.

  • If no token is set, the string is considered as an individual word /url/pic/abc.gif. You can only query this log by using the complete string or fuzzy match such as /url/pic/*.
  • If / is set as the token, the raw log is separated into three words: url, pic, and abc.gif. You can query this log by using any of the three words or fuzzy match, for example, url, abc.gif, or pi*. You can also use /url/pic/abc.gif to query this log (/url/pic/abc.gif is separated into the following three conditions during the query: url, pic, and abc.gif).
  • If /. is set as the token, the raw log is separated into four words: url, pic, abc, and gif.

Note: You can broaden the query range by setting appropriate tokens.

2. Value type

You can query logs of long (integer) type and double (decimal) type. With long or double selected as the Type, you can only query the key by using a value range. For example, you can use the following keyword to query the longkey whose key range is between 1000 and 2000.

  1. longKey > 1000 and longKey < 2000

3. JSON (combination type)

JSON is of combination type and is composed of text, boolean, value, array, and map.

3.1 Text type

JSON fields can automatically recognize fields of text type and bool type.

  1. jsonkey: {
  2. key1:text_value,
  3. key2:true,
  4. key3:3.14
  5. }

For example, the following jsonkey can be queried by using the conditions such as jsonkey.key1:"text_value" and jsonkey.key2:true.

Value type

You can query the data of double type or long type that is not in the JSON array by setting the type and specifying the path.

  1. The type of the jsonkey.key3 field is double.
  2. Query: jsonkey.key3 > 3

3.3 JSON field including invalid content

Log Service attempts to parse the valid contents until the invalid content appears.

Example:

  1. "json_string":
  2. {
  3. "key_1" : "value_1",
  4. "key_map" :
  5. {
  6. "key_2" : "value_2",
  7. "key_3" : "valu

Data after key_3 is truncated and lost. The field json_string.key_map.key_2 and contents before this field can be successfully parsed.

Note:

  • JSON object type and JSON array type are not supported.
  • The field cannot be in a JSON array.
  • Field of bool type can be converted to the text type.

4. Full text query (combination type)

By default, full text query (index) considers that a log is of text type and does not need to specify keys. For example, the following log is composed of four fields (time/status/level/message):

  1. [20180102 12:00:00] status:200,level:"error",message:"some thing is error in this field"

After enabling full text index, all the keys and values in the log are separated into individual words.

  1. [20180102 12:00:00] status 200 level error message some thing is error in this field

Enter the keyword error during the query, both the error in the level field and that in the message field are queried.

Example

The following log includes four key values besides the time.

Sequence number Key Type
0 time
1 class text
2 status long
3 latency double
4 message json
  1. time:2018-01-01 12:00:00
  2. 1. class:central-log
  3. 2. status:200
  4. 3. latency:68.75
  5. 4. message:
  6. {
  7. "methodName": "getProjectInfo",
  8. "success": true,
  9. "remoteAddress": "1.1.1.1:11111",
  10. "usedTime": 48,
  11. "param": [
  12. {
  13. "projectName": "ali-log-test-project",
  14. "requestId": "d3f0c96a-51b0-4166-a850-f4175dde7323"
  15. }
  16. ],
  17. "result": {
  18. "message": "successful",
  19. "code": "200",
  20. "data": {
  21. "clusterRegion": "ap-southeast-1",
  22. "ProjectName": "ali-log-test-project",
  23. "CreateTime": "2017-06-08 20:22:41"
  24. },
  25. "success": true
  26. }
  27. }

See the following configurations.

1

Where:

  • ① indicates that all the data of the string type and bool type in the JSON field can be queried.
  • ② indicates that data of the long type can be queried.
  • ③ indicates that you can analyze the configured field by using SQL statements.

Example 1: Query data of string type and bool type

  1. class : cental*
  2. message.traceInfo.requestId : 92.137_1518139699935_5599
  3. message.param.projectName : ali-log-test-project
  4. message.success : true

Note:

  • No configuration in the JSON field is needed.
  • JSON map and array are automatically expanded and support multiple levels of nesting, using a dot (.) to separate each level.

Example 2: Query data of double type and long type

  1. latency>40
  2. message.usedTime > 40

Note: Configure the JSON field independently and the JSON field cannot be in the array.

Example 3: Query data of combination type

  1. class : cental* and message.usedTime > 40 not message.param.projectName:ali-log-test-project

Query and analysis syntax

For more information about the query syntax, see Query syntax. For how to analyze the query results, see Analysis syntax.

Others

If the log volume you are about to query is large (for example, the time span is large and tens of billions of data exists), you cannot retrieve all the data for one query request. In this situation, Log Service returns existing data to you and tells you that the query results are not complete in the returned result. At the same time, Log Service caches the query results within 15 minutes. If some query request results are the same as those in the cache, Log Service continues to scan the logs that are not in the cache for this request. To reduce the workload of merging multiple query results, Log Service merges the query results that are the same as those in the cache and the results newly scanned in this query, and then returns them to you. Therefore, Log Service allows you to call the API multiple times with the same parameter to obtain the final complete results.

Thank you! We've received your feedback.