An index is an inverted storage structure that consists of keywords and logical pointers that can refer to actual data. You can use an index to quickly locate data rows based on keywords. An index is similar to a data catalog. You can query and analyze log data only after you configure indexes.

Prerequisites

Logs are collected. For more information, see Data collection overview.
Important If you want to analyze logs, you must store the logs in a Standard Logstore. For more information, see Manage a Logstore.

Usage notes

  • After you enable the indexing feature, you are charged for the index traffic and storage space occupied by indexes. For more information, see Billable items.
  • After you disable the indexing feature for a Logstore, the storage space occupied by historical indexes is automatically released if the data retention period of the Logstore elapses.
  • The indexing feature takes effect only on the log data that is written after you configure indexes. If you want to query and analyze historical data, you must use the reindexing feature. For more information, see Reindex logs for a Logstore.
  • By default, indexes are configured for specific reserved fields in Log Service. For more information, see Reserved fields. No delimiters are specified for the indexes of the __topic__ and __source__ fields. When you search for the fields, only exact match is supported.
  • If you want to search for the fields that are prefixed with __tag__, you must configure field indexes. Full-text indexes are not supported.

Index types

The following table describes the index types supported by Log Service.

Note Query and analysis results vary based on index configurations. Therefore, you must configure indexes based on your business requirements. If you configure both full-text indexes and field indexes, the configurations of the field indexes take precedence.
Index type Description
Full-text index Log Service splits an entire log into multiple words based on specified delimiters to create indexes. In a search statement, the field names (keys) and field values (values) are both plain text. For example, the search statement error returns the logs that contain the keyword error.
Field index After you configure field indexes, you can specify field names and field values in the Key:Value format to search for logs. For example, the search statement level:error returns the logs whose level field value contains error.

If you want to use the analysis feature, you must configure field indexes and turn on Enable Analytics for the required fields. If you turn on Enable Analytics, no additional index traffic is generated, and no additional storage space is occupied.

Configure full-text indexes

  1. Log on to the Log Service console.
  2. In the Projects section, click the project that you want to view.
  3. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
  4. Go to the index configuration page.
    • If you have not enabled the indexing feature, click Enable on the search and analysis page of the Logstore. Enable the indexing feature
    • If you have enabled the indexing feature, choose Index Attributes > Attributes on the search and analysis page of the Logstore. Configure indexes
  5. In the Search & Analysis panel, configure the following parameters and click OK.
    Note The index configurations take effect within 1 minute.
    Parameter Description
    LogReduce If you turn on LogReduce, Log Service automatically aggregates text logs that have the same pattern during log collection. This way, you can obtain the overall information about logs. For more information, see LogReduce.
    Full Text Index If you turn on Full Text Index, the full-text indexing feature is enabled.
    Case Sensitive Specifies whether searches are case-sensitive.
    • If you turn on Case Sensitive, searches are case-sensitive. For example, if a log contains internalError, you can search for the log only by using the keyword internalError.
    • If you turn off Case Sensitive, searches are not case-sensitive. For example, if a log contains internalError, you can search for the log by using the keyword INTERNALERROR or internalerror.
    Include Chinese Specifies whether to distinguish between Chinese content and English content in searches.
    • After you turn on Include Chinese, if a log contains Chinese characters, the Chinese content is split based on the Chinese grammar. The English content is split based on specified delimiters.
      Important When the Chinese content is split, the write speed is reduced. Proceed with caution.
    • If you turn off Include Chinese, all the content in a log is split based on specified delimiters.
    Delimiter The delimiters that are used to split the content of a log into multiple words. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If the default delimiters cannot meet your requirements, you can specify custom delimiters. All ASCII codes can be defined as delimiters.
    For example, the content of a log is /url/pic/abc.gif.
    • If you do not specify a delimiter, the log is regarded as a single word /url/pic/abc.gif. You can search for the log only by using the keyword /url/pic/abc.gif or by using /url/pic/* to perform a fuzzy search.
    • If you set Delimiter to a forward slash (/), the content of the log is split into the following three words: url, pic, and abc.gif. You can search for the log by using the keyword url, abc.gif, or /url/pic/abc.gif, or by using pi* to perform a fuzzy search.
    • If you set Delimiter to a forward slash (/) and a period (.), the content of the log is split into the following four words: url, pic, abc, and gif.
    Maximum Statistics Field Length The maximum length of a field value that can be retained. Default value: 2048. Unit: bytes. The default value is equal to 2 KB. You can change the value of the Maximum Statistics Field Length parameter. Valid values: 64 to 16384.
    Important If the length of a field value exceeds the value of this parameter, the field value is truncated, and the excess part is not involved in analysis.

Configure field indexes

Note When you configure field indexes, you can add a maximum of 500 fields. When you configure indexes for a JSON field, you can add a maximum of 100 subfields.
  1. Log on to the Log Service console.
  2. In the Projects section, click the project that you want to view.
  3. Choose Log Storage > Logstores. On the Logstores tab, click the Logstore that you want to view.
  4. Go to the index configuration page.
    • If you have not enabled the indexing feature, click Enable on the search and analysis page of the Logstore. Enable the indexing feature
    • If you have enabled the indexing feature, choose Index Attributes > Attributes on the search and analysis page of the Logstore. Configure indexes
  5. In the Search & Analysis panel, configure the following parameters and click OK.
    Note The index configurations take effect within 1 minute.
    Parameter Description
    Key Name The name of the log field. Example: client_ip.
    Note
    • If you configure an index for a tag field, you must set the Key Name parameter in the __tag__:KEY format. For example, you can set the parameter to __tag__:__receive_time__. Different tag fields are supported. For example, a tag field that indicates a public IP address or a UNIX timestamp is supported. For more information, see Reserved fields.
    • When you configure an index for a tag field, numeric data types are not supported. You must set the Type parameter for each tag field to text.
    Type The data type of the log field value. Valid values: text, long, double, and json. For more information, see Data types.
    Note If you set the data type for a field to long or double, you cannot configure the Case Sensitive, Include Chinese, or Delimiter parameter for the field.
    Alias The alias of the field. Example: ip.

    An alias is used only in analytic statements. You must use the original field name in search statements. For more information, see Column aliases.

    Case Sensitive Specifies whether searches are case-sensitive.
    • If you turn on Case Sensitive, searches are case-sensitive. For example, if a log contains internalError, you can search for the log only by using the keyword internalError.
    • If you turn off Case Sensitive, searches are not case-sensitive. For example, if a log contains internalError, you can search for the log by using the keyword INTERNALERROR or internalerror.
    Delimiter The delimiters that are used to split the content of a log into multiple words. The default delimiters are , '";=()[]{}?@&<>/:\n\t\r. If the default delimiters cannot meet your requirements, you can specify custom delimiters. All ASCII codes can be defined as delimiters.
    For example, the content of a log is /url/pic/abc.gif.
    • If you do not specify a delimiter, the log is regarded as a single word /url/pic/abc.gif. You can search for the log only by using the keyword /url/pic/abc.gif or by using /url/pic/* to perform a fuzzy search.
    • If you set Delimiter to a forward slash (/), the content of the log is split into the following three words: url, pic, and abc.gif. You can search for the log by using the keyword url, abc.gif, or /url/pic/abc.gif, or by using pi* to perform a fuzzy search.
    • If you set Delimiter to a forward slash (/) and a period (.), the content of the log is split into the following four words: url, pic, abc, and gif.
    Include Chinese Specifies whether to distinguish between Chinese content and English content in searches.
    • After you turn on Include Chinese, if a log contains Chinese characters, the Chinese content is split based on the Chinese grammar. The English content is split based on specified delimiters.
      Important When the Chinese content is split, the write speed is reduced. Proceed with caution.
    • If you turn off Include Chinese, all the content in a log is split based on specified delimiters.
    Enable Analytics Before you can use the analysis feature, you must turn on Enable Analytics.
    Maximum Statistics Field Length The maximum length of a field value that can be retained. Default value: 2048. Unit: bytes. The default value is equal to 2 KB. You can change the value of the Maximum Statistics Field Length parameter. Valid values: 64 to 16384.
    Important If the length of a field value exceeds the value of this parameter, the field value is truncated, and the excess part is not involved in analysis.

Enable the automatic update of indexes

If a Logstore is a dedicated Logstore for a cloud service or an internal Logstore, Auto Update is turned on by default. The built-in indexes of the Logstore are automatically updated to the latest version.

Important If you delete the indexes of a dedicated Logstore for a cloud service, features such as reports and alerting that are enabled for the Logstore may be affected.
If you want to configure custom indexes, turn off Auto Update in the Search & Analysis panel. Enable the automatic update of indexes

Index traffic descriptions

After you configure indexes, index traffic is generated.

Index type Description
Full-text index All field names and field values are stored as text. The field names and field values are both included in the index traffic.
Field index The method that is used to calculate index traffic varies based on the data type of a field.
  • text: Field names and field values are both included in the index traffic.
  • long and double: Field names are not included in the index traffic. Each field value is counted 8 bytes in the index traffic.

    For example, if you configure an index for the status field of the long type and the field value is 400, the string status is not included in the index traffic, and the 400 value is counted 8 bytes in the index traffic.

  • json: Field names and field values are both included in the index traffic. The subfields that are not indexed are also included. For more information, see How do I calculate index traffic for a JSON field?
    • If a subfield is not indexed, the index traffic is calculated by regarding the data type of the subfield as text.
    • If a subfield is indexed, the index traffic is calculated based on the data type of the subfield. The data type can be text, long, or double.