This topic describes how to enable and configure the index feature for a Logstore.

Prerequisites

Log data is collected. For more information, see Data collection.

Background information

You must enable and configure the index feature for a Logstore before you can query the log data that is stored in the Logstore. We recommend that you configure the index feature for your Logstore based on your business requirements.
Note
  • After you enable the index feature, the indexes occupy extra storage space and transferring the indexes occupies extra bandwidth.
  • After you enable and configure the index feature for a Logstore, the update applies only to the log data that is newly written to the Logstore.
Log Service adds multiple system fields as key-value pairs to a log entry when writing the log entry to a Logstore. If the index feature of a Logstore is enabled and configured, you can search and analyze the log data in the Logstore based on the system fields.
System field Description
__topic__ The topic of a log entry. If a topic is specified for a log entry, Log Service adds this field to the log entry. The key of the field is __topic__ and the value is the log topic. For more information, see Configure a log topic.
__source__ The source of a log entry.
__time__ The timestamp of a log entry.
Note By default, no delimiter to segment the values of the __topic__ and __source__ fields is defined. Therefore, only exact match is supported for the two fields.

Procedure

  1. Log on to the Log Service console.
  2. In the Projects section, click a project.
  3. On the Log Management > Logstores tab, choose Management icon > Search & Analysis to the right of a Logstore.
  4. In the upper-right corner, click Enable.
  5. Configure indexes.
    Note If you configure both the full-text index and the field index, the full-text index does not take effect for the fields in the field index.
    Index type Description
    Full-text index The full-text index stores the key and value of each field as text. This index allows you to search the keys and values for a keyword.
    Field index If you add a field to the index, you can specify the key of the field and a keyword to search the field for the keyword.

    To use the automatic indexing feature, click Automatic Index Generation. After the index is automatically configured, you can modify the index.

    • Configure the full-text index.
      Parameter Description
      LogReduce Specifies whether to aggregate text logs that have the same pattern. If you turn on the LogReduce switch, Log Service aggregates text logs that have the same pattern. You can then view log entries by pattern. For more information, see LogReduce.
      Full Text Index Specifies whether to enable the full-text index. If you turn on the Full Text Index switch, the full-text index is enabled.
      Case Sensitive Specifies whether searches are case-sensitive.
      • If you turn off the Case Sensitive switch, searches are not case-sensitive. For example, if a log entry contains internalError, you can retrieve the log entry by using the INTERNALERROR or internalerror keyword.
      • If you turn on the Case Sensitive switch, searches are case-sensitive. For example, if a log contains internalError, you can retrieve the log entry only by using the internalError keyword.
      Delimiter The delimiters that are used to segment the content of a log entry into multiple words. Each character that you specify is a separate delimiter. For example, the content of a log entry is /url/pic/abc.gif.
      • If no delimiter is specified, the entire /url/pic/abc.gif string is considered a word. The log entry is returned only if the keyword is the entire string or a wildcard pattern such as /url/pic/*.
      • If you specify a forward slash (/) as the delimiter, Log Service segments the log content into three words: url, pic, and abc.gif. The log entry is returned if the keyword is one of the words or a wildcard pattern that matches one of the words, for example, url, abc.gif, or pi*. The log entry is also returned if the keyword is /url/pic/abc.gif. The /url/pic/abc.gif keyword is equivalent to url and pic and abc.gif.
      • If you specify a forward slash (/) and a period (.) as the delimiters, Log Service segments the log content into four words: url, pic, abc, and gif.
    • Configure the field index.
      Parameter Description
      Field Name The name of the log field, for example, _address_.
      Note
      • If you need to add a tag field (such as an Internet IP address or Unix timestamp) to the index, you must set the Field Name parameter in the format of __tag__:key, for example, __tag__:__receive_time__. For information about tag fields, see Logs.
      • In the Field Search section, you must set the Type parameter of a tag field to text. Numeric data types are not supported.
      Type The data type of the field. The following data types are supported. For more information, see Data types of indexes.
      • text: Each value of the field is in the text format.
      • long: Each value of the field is a long integer. You must use a value range to query a field of this data type.
      • double: Each value of the field is a double-precision floating-point number. You must use a value range to query a field of this data type.
      • json: Each value of the field is in the JSON format.
      Note The Case Sensitive and Delimiter parameters are not available for fields of the long and double types.
      Alias The alias of a column, for example, address.

      The alias is used only in analytic statements. The field name is stored and used for query. For more information, see Column aliases.

      Case Sensitive Specifies whether searches are case-sensitive.
      • If you turn off the Case Sensitive switch, searches are not case-sensitive. For example, if a log entry contains internalError, you can retrieve the log entry by using the INTERNALERROR or internalerror keyword.
      • If you turn on the Case Sensitive switch, searches are case-sensitive. For example, if a log contains internalError, you can retrieve the log entry only by using the internalError keyword.
      Delimiter The delimiters that are used to segment the content of a log entry into multiple words. Each character that you specify is a separate delimiter. For example, the content of a log entry is /url/pic/abc.gif.
      • If no delimiter is specified, the entire /url/pic/abc.gif string is considered a word. The log entry is returned only if the keyword is the entire string or a wildcard pattern such as /url/pic/*.
      • If you specify a forward slash (/) as the delimiter, Log Service segments the log content into three words: url, pic, and abc.gif. The log entry is returned if the keyword is one of the words or a wildcard pattern that matches one of the words, for example, url, abc.gif, or pi*. The log entry is also returned if the keyword is /url/pic/abc.gif. The /url/pic/abc.gif keyword is equivalent to url and pic and abc.gif.
      • If you specify a forward slash (/) and a period (.) as the delimiters, Log Service segments the log content into four words: url, pic, abc, and gif.
      Enable Analytics Specifies whether to enable the real-time analysis feature. By default, the feature is enabled.

      If the feature is enabled, you can use an analytic statement to analyze the result that is returned by a search statement.

  6. Click OK.
    The index configurations applies within 1 minute.

What to do next

Query logs