Log Service supports querying many data types, such as text, value, and JSON. Fuzzy match is also supported.
Note: You must enable the LogSearch/Analytics function before using Log Service to query data. To enable the LogSearch/Analytics function, see Overview.
- Make sure you have enabled the index and analysis functions before querying and analyzing logs.
- Enable index: Click Enable in the upper-right of the query page.
- Enable analysis: Turn on the Enable Analytics switch for a specific field on the Search & Analysis page after you enable the index.
- The full text index and key/value index cannot be disabled at the same time.
- The Case Sensitive and Token attributes are unavailable if the index type is long or double.
- You can perform fast statistics and analysis by using SQL statements only after enabling analytics for the corresponding column.
- You must specify the Logstore and time range to be queried when querying logs. The Logstore must belong to a specified project and the time range cannot exceed the log storage period of the Logstore.
Similar to search engines, data of text type is queried based on terms. Therefore, you must configure the Case Sensitive and Token.
Determine whether or not to support case sensitive when querying raw logs. For example, the raw log is
- After turning off the Case Sensitive switch, the sample log can be queried based on the keyword
- After turning on the Case Sensitive switch, the sample log can only be queried based on the keyword
You can separate the contents of a raw log into several keywords by using a token.
For example, the raw log is
- If no token is set, the string is considered as an individual word
/url/pic/abc.gif. You can only query this log by using the complete string or fuzzy match such as
/is set as the token, the raw log is separated into three words:
abc.gif. You can query this log by using any of the three words or fuzzy match, for example,
pi*. You can also use
/url/pic/abc.gifto query this log (
/url/pic/abc.gifis separated into the following three conditions during the query:
/.is set as the token, the raw log is separated into four words:
Note: You can broaden the query range by setting appropriate tokens.
You can query logs of
long (integer) type and
double (decimal) type. With long or double selected as the Type, you can only query the key by using a value range. For example, you can use the following keyword to query the longkey whose key range is between 1000 and 2000.
longKey > 1000 and longKey < 2000
JSON is of combination type and is composed of text, boolean, value, array, and map.
JSON fields can automatically recognize fields of text type and bool type.
For example, the following jsonkey can be queried by using the conditions such as
You can query the data of double type or long type that is not in the JSON array by setting the type and specifying the path.
The type of the jsonkey.key3 field is double.
Query: jsonkey.key3 > 3
Log Service attempts to parse the valid contents until the invalid content appears.
"key_1" : "value_1",
"key_2" : "value_2",
"key_3" : "valu
Data after key_3 is truncated and lost. The field
json_string.key_map.key_2 and contents before this field can be successfully parsed.
- JSON object type and JSON array type are not supported.
- The field cannot be in a JSON array.
- Field of bool type can be converted to the text type.
By default, full text query (index) considers that a log is of text type and does not need to specify keys. For example, the following log is composed of four fields (time/status/level/message):
[20180102 12:00:00] status:200,level:"error",message:"some thing is error in this field"
After enabling full text index, all the keys and values in the log are separated into individual words.
[20180102 12:00:00] status 200 level error message some thing is error in this field
Enter the keyword error during the query, both the error in the level field and that in the message field are queried.
The following log includes four key values besides the time.
"CreateTime": "2017-06-08 20:22:41"
See the following configurations.
- ① indicates that all the data of the string type and bool type in the JSON field can be queried.
- ② indicates that data of the long type can be queried.
- ③ indicates that you can analyze the configured field by using SQL statements.
class : cental*
message.traceInfo.requestId : 92.137_1518139699935_5599
message.param.projectName : ali-log-test-project
message.success : true
- No configuration in the JSON field is needed.
- JSON map and array are automatically expanded and support multiple levels of nesting, using a dot (.) to separate each level.
message.usedTime > 40
Note: Configure the JSON field independently and the JSON field cannot be in the array.
class : cental* and message.usedTime > 40 not message.param.projectName:ali-log-test-project
If the log volume you are about to query is large (for example, the time span is large and tens of billions of data exists), you cannot retrieve all the data for one query request. In this situation, Log Service returns existing data to you and tells you that the query results are not complete in the returned result. At the same time, Log Service caches the query results within 15 minutes. If some query request results are the same as those in the cache, Log Service continues to scan the logs that are not in the cache for this request. To reduce the workload of merging multiple query results, Log Service merges the query results that are the same as those in the cache and the results newly scanned in this query, and then returns them to you. Therefore, Log Service allows you to call the API multiple times with the same parameter to obtain the final complete results.