Create and manage indexes in Simple Log Service - Simple Log Service

Why indexes are required

Keywords retrieve content from raw logs. For example, a log entry curl/7.74.0 contains the keyword curl, but without parsing, the entry is a single unit that does not match curl exactly. Simple Log Service cannot retrieve it.

To make logs searchable, split them into terms with delimiters. For example, applying \n\t\r,;[]{}()&^*#@~=<>/\?:'" produces the terms curl and 7.74.0. Simple Log Service builds an index from these terms. Logs can be queried and analyzed only after an index is created.

Simple Log Service supports both a full-text index and a field index. If you configure both for the same Logstore, the field index configuration takes precedence.

Index types

Full-text index

A full-text index splits an entire log entry into text-type terms by Delimiter. You can then query logs by keyword. For example, Chrome or Safari finds logs containing Chrome or Safari.

Important

The Delimiter does not support Chinese characters. To handle Chinese text, enable Include Chinese. SLS then automatically tokenizes Chinese content based on grammar.
If you configure only a full-text index, you can use full-text search syntax. Query syntax and functions

Field index

A field index splits logs by field name (KEY) and tokenizes each field's content. Four data types are supported: text, long, double, and JSON. Query by field name and value (Key:Value) or with a SELECT statement. Query syntax and features

Important

To query or analyze a specific field with a SELECT statement, you must create a field index for it. A field index takes precedence over a full-text index when both are configured for the same field.
text fields support full-text queries, field-specific queries, and analytic statements (SELECT).
- Without a full-text index, a full-text query searches only across all text-type fields.
- With a full-text index enabled, full-text queries search all logs.
long and double fields support field-specific queries and analytic statements (SELECT).

Create an index

Important

Index configurations directly affect query and analysis results. New indexes take about one minute to take effect.
Indexes apply only to newly ingested data. To query historical data, reindex it.
SLS automatically creates indexes for some reserved fields.

The delimiters for __topic__ and __source__ are empty, so queries on these fields must use exact match.
Fields prefixed with __tag__ do not support full-text indexing. Create a text-type field index to query and analyze them, for example: *| select "__tag__:__receive_time__".
If a log contains duplicate field names (for example, request_time), the console displays one as request_time_0. The underlying name remains request_time. Always use the original name when creating indexes, querying, analyzing, shipping, or transforming data.

Console

Log on to the Simple Log Service console.
In the Projects section, click the project you want.
On the Log Storage > Logstores tab, click the logstore you want.
On the Search & Analysis page of the Logstore, click Enable Index.

Note
You can query the latest data about one minute after enabling indexing.
(Optional) Turn off auto update.

For Logstores used by other Alibaba Cloud services or for internal use, Auto Update is enabled by default to keep built-in indexes up to date. To customize indexes, turn off Auto Update on the Query and Analysis panel.

Warning
Deleting indexes of a Logstore dedicated to another cloud service affects related features such as reports and alerts.

Create an index.

Full-text index

After you click Enable Index, Full-text Index is enabled by default. You can toggle LogReduce, Case Sensitive, and Include Chinese as needed, and specify Delimiter or customize Delimiter.

Configuration page:

Parameters

Parameter	Description
LogReduce	When you turn on LogReduce, SLS clusters similar text logs during collection and extracts common patterns for quick overview. LogReduce
Case Sensitive	Controls query case sensitivity: With Case Sensitive enabled, queries are case-sensitive. For example, if a log contains `internalError`, you must use `internalError` to find it. With Case Sensitive disabled, queries are case-insensitive. For example, if a log contains `internalError`, you can find it with `INTERNALERROR` or `internalerror`.
Include Chinese	Controls how Chinese content is handled in searches: With Include Chinese enabled, Chinese content is tokenized based on Chinese grammar, while other content is split by the specified delimiter. Important Chinese tokenization can slightly reduce ingestion speed. Enable only when necessary. With Include Chinese disabled, all content is split by the specified delimiter.
Delimiter	Log content is split into terms by delimiters. Defaults: `, '";=()[]{}?@&<>/:\n\t\r`. All ASCII characters are valid delimiters. If Delimiter is empty, the entire field value is treated as a single term, searchable only by the full string or a wildcard query. For example, the log content is `/url/pic/abc.gif`. Without delimiters, an entire log entry such as `/url/pic/abc.gif` is treated as a single word. You can find it only by the full string `/url/pic/abc.gif` or a wildcard query such as `/url/pic/`. With the delimiter set to forward slash (/), the log is split into three words: `url`, `pic`, and `abc.gif`. You can find the log by any of these words or a fuzzy search such as `url`, `abc.gif`, `pi`, or `/url/pic/abc.gif`. With delimiters set to forward slash (/) and period (.), the log is split into four words: `url`, `pic`, `abc`, and `gif`. You can find the log by any of these words or a fuzzy search.

Field index

After you click Enable Index, click Automatic Index Generation on the Query and Analysis page to auto-generate a field index from the first entry in preview data. To add a custom field, click +. Parameter descriptions

Initial view:

Field index configuration:

Parameters

Parameter	Description
Field Name	Log field name (KEY), such as `client_ip`. Field names can contain only letters, digits, and underscores (`_`), and must start with a letter or underscore (`_`). Important For a public IP address, Unix timestamp, or other `__tag__` field, set Field Name to the `__tag__:KEY` format, such as `__tag__:__receive_time__`. Reserved fields `__tag__` fields do not support numeric indexes. Set Type to text for all `__tag__` fields.
Type	Data type of the field value: text, long, double, or json. long and double types do not support Case Sensitive, Include Chinese, or Delimiter settings.
Alias	Field alias. For example, alias `client_ip` as `ip`. Aliases can contain only letters, digits, and underscores (`_`), and must start with a letter or underscore (`_`). Important Aliases work only in analytic statements (SELECT). Use the original field name in search queries. Column aliases
Case Sensitive	Controls query case sensitivity: With Case Sensitive enabled, queries are case-sensitive. For example, if a log contains `internalError`, you must use `internalError` to find it. With Case Sensitive disabled, queries are case-insensitive. For example, if a log contains `internalError`, you can find it with `INTERNALERROR` or `internalerror`.
Delimiter	Log content is split into terms by delimiters. Defaults: `, '";=()[]{}?@&<>/:\n\t\r`. All ASCII characters are valid delimiters. If Delimiter is empty, the entire field value is treated as a single term, searchable only by the full string or a wildcard query. For example, the log content is `/url/pic/abc.gif`. Without delimiters, an entire log entry such as `/url/pic/abc.gif` is treated as a single word. You can find it only by the full string `/url/pic/abc.gif` or a wildcard query such as `/url/pic/`. With the delimiter set to forward slash (/), the log is split into three words: `url`, `pic`, and `abc.gif`. You can find the log by any of these words or a fuzzy search such as `url`, `abc.gif`, `pi`, or `/url/pic/abc.gif`. With delimiters set to forward slash (/) and period (.), the log is split into four words: `url`, `pic`, `abc`, and `gif`. You can find the log by any of these words or a fuzzy search.
Include Chinese	Controls how Chinese content is handled in searches: With Include Chinese enabled, Chinese content is tokenized based on Chinese grammar, while other content is split by the specified delimiter. Important Chinese tokenization can slightly reduce ingestion speed. Enable only when necessary. With Include Chinese disabled, all content is split by the specified delimiter.
Enable Analytics	Turn on Enable Analytics to enable statistical analysis on this field.

(Optional) Set the maximum field length.

During SQL analysis, Simple Log Service truncates field values at 2048 bytes (2 KB) by default. Adjust this limit (64–16384 bytes) with Maximum Field Length at the bottom of the Query and Analysis page.
Important
- Index configuration updates apply only to newly ingested data.
- If a field value exceeds the maximum length, the excess is truncated and excluded from analysis.

API

Manage indexes by calling Simple Log Service API operations:

SDK

Manage indexes with Simple Log Service SDKs. The following sections list commonly used SDKs. SDK reference overview.

Java

Manage indexes with the Simple Log Service SDK for Java. Use the Log Service SDK for Java to manage indexes.

Python

Manage indexes with the Simple Log Service SDK for Python. Manage indexes with the Python SDK.

In addition to its native SDK, SLS also supports the common Alibaba Cloud SDKs. For more information, see Simple Log Service_SDK Center_Alibaba Cloud OpenAPI Explorer.

CLI

Manage indexes through the Simple Log Service CLI:

Update an index

Procedure

On the Search & Analysis page of the target Logstore, choose Index Attributes > Attributes. Update indexes carefully. Changes take effect in about one minute.

Disable an index

Important

After you Disable an index, SLS automatically releases the storage for historical index data when the Logstore's retention period expires.

Procedure

On the Search & Analysis page of the target Logstore, choose Index Attributes > Disable.

Index configuration examples

Example 1

The log content contains the request_time field. You can execute the query request_time>100.

With only a full-text index, the search returns logs that contain the three terms request_time, > (not a delimiter), and 100.
With field indexes of the double and long types, the query returns logs where request_time is greater than 100.
If you create a full-text index and field indexes for the double and long types, the full-text index for request_time is ignored, and the query returns logs where request_time is greater than 100.

Example 2

Because the log content contains the request_time field, you can run the full-text query request_time.

If you create only a field index of the double or long data type, no relevant logs are returned.
You can create only a full-text index to query for logs that contain request_time from all log text.
Query for logs that contain request_time from fields that have a field index of the text type.

Example 3

The log content contains the status field. Run the analysis statement * | SELECT status, count(*) AS PV GROUP BY status.

If you create only a full-text index, no relevant logs are returned.
Creating a field index for status returns the different status codes and their corresponding total PVs.

Index traffic

Full-text index

All field names and values count toward index traffic.

Field index

Index traffic calculation varies by field data type:

text type: Both the field name and value count toward index traffic.
long and double types: The field name does not count toward index traffic. Each field value contributes a fixed 8 bytes.

For example, if an index is created on the status field (long type) with value 200, the string status is not counted. The index traffic for 200 is a fixed 8 bytes.
JSON type: Both field names and values count toward index traffic, including sub-keys not explicitly indexed. How is the index traffic of JSON fields calculated?
- Non-indexed sub-keys: index traffic is calculated as text type.
- Indexed sub-keys: index traffic is calculated based on the specified data type (text, long, or double).

Billing

Logstores in pay-by-ingested-data mode

Indexes occupy storage space. Manage intelligent tiered storage
Reindexing does not incur fees.
Index traffic charges: Billable items for the pay-by-ingested-data mode

Logstores in pay-by-feature mode

Indexes occupy storage space. Manage intelligent tiered storage
Creating an index generates billable index traffic. Billable items for the pay-by-feature mode How do I reduce index traffic fees?
Reindexing incurs fees. Billable items and prices are the same as for creating an index.

Next steps

FAQ

Why can't I find my logs after I import them?
- Check whether your configured delimiter is correct.
- Index configurations apply only to newly ingested data. To query historical data, reindex the data.
How do I perform a query with two conditions?

Include both conditions in your search statement. For example, not OK not Unknown finds logs where the status is neither OK nor Unknown.
How do I query logs that contain multiple keywords?
For example, to find logs where http_user_agent contains "like Gecko":
- Phrase search: http_user_agent:"like Gecko". Phrase search.
- LIKE syntax: * | SELECT * WHERE http_user_agent LIKE '%like Gecko%'
How do I search for a keyword that contains spaces?

Searching POS version returns logs containing either POS or version. Wrap the phrase in double quotes — "POS version" — to match the exact phrase.
FAQ about log queries
Common errors in log query and analysis
How do I perform a fuzzy query on logs?
FAQ about querying and analyzing JSON logs
How do I download logs to a local device?
Why does Log Service truncate field values during query and analysis?