Auto-discover field indexes continuously identifies fields in newly written logs that do not have field indexes configured and automatically creates text-type field indexes for eligible fields. Enable this feature to eliminate manual index maintenance when your log fields change frequently or are too numerous to enumerate in advance.
Background information
Simple Log Service uses index technology to provide query and analysis capabilities. When you use field indexes, you typically need to determine field names in advance and configure the index type, tokenizer, and statistics properties for each field.
However, maintaining a complete field index configuration in advance is difficult in the following scenarios:
The number of log fields is large, and you cannot obtain a complete field list in advance.
As your business iterates continuously, new fields are added to logs. Manually updating index configurations incurs high maintenance costs.
Some fields appear only in specific scenarios and may not appear in the initial log samples used for auto-generating indexes.
In these scenarios, you often discover that certain fields have no indexes only when you need to analyze them. Newly created indexes take effect only on newly written data, and reindexing has time range limitations. Analyzing historical data without indexes is inconvenient.
To address these issues, Simple Log Service provides the auto-discover field indexes feature. After you enable this feature, Simple Log Service continuously identifies top-level fields in newly written logs that do not have field indexes configured, and automatically creates text-type field indexes with statistics enabled. After the fields take effect, you can use field_name:field_value for field queries, and use SQL statements for filtering, grouping, and aggregation and analysis.
Benefits
Reduced maintenance costs — You do not need to enumerate all fields in advance or manually update index configurations each time a new log field is added.
Continuous adaptation to field changes — After you enable this feature, it continuously discovers fields in subsequently written logs, without being limited to the log samples available when the index was initially enabled.
Fewer historical data gaps — By discovering and creating indexes for new fields as logs are written, this feature reduces issues where historical data cannot be queried or analyzed due to delayed discovery.
Fast query and analysis — Auto-discovered fields use field indexes for queries and support SQL statistical analysis.
Fine-grained configuration — For stable fields that require numeric range queries, specific tokenizers, or other types, you can further configure them as specified field indexes.
Prerequisites
This feature is currently available through an allowlist. After the allowlist is enabled for your account, the corresponding toggle appears in the console. To request access, submit a ticket.
You have created a Standard Logstore. Query Logstores do not support auto-discover field indexes.
You have enabled full-text indexing for the Logstore. Auto-discover field indexes depends on the full-text index configuration. For more information, see Create indexes.
Your account has permissions to modify Logstore index configurations.
Enable auto-discover field indexes
Log on to the Simple Log Service console.
In the Projects list, click the target project.
In the tab, click the target Logstore.
On the Search & Analysis page, choose .
If you have not created indexes, click Enable Index and make sure that full-text indexing is enabled.
Select the Auto-discover unspecified fields (full-text index required) toggle, and then click OK.
After a few minutes, refresh the index configuration page and verify that the Auto-discovered fields section appears.
Auto-discover unspecified fields and Auto-generate indexes on the console page are different features:
Auto-generate indexes — Generates recommended field index configurations based on preview logs as a one-time operation. The configuration takes effect only after user confirmation.
Auto-discover unspecified fields — Continuously discovers new fields in newly written logs and automatically creates field indexes after it is enabled. No user confirmation is required for each new field.
Disabling the toggle only stops the discovery of new fields. It does not automatically delete the field indexes that have already been created.
View and manage auto-discovered fields
After you enable this feature, Simple Log Service continuously checks top-level fields in newly written logs. The field indexes automatically discovered and created are displayed in the Auto-discovered fields section on the index configuration page. These field indexes are all text type, share case sensitivity, tokenizer, and Chinese inclusion properties with the full-text index, and have statistics enabled.
Auto-discovered fields already have index data created and can be used directly for query and analysis.
To promote an auto-discovered field to a specified field index, in the Auto-discovered fields section, locate the target field and click the + icon in the Actions column. The field moves to the Specified Field Search section above, where you can modify the field type, case sensitivity, tokenizer, or Chinese inclusion properties.
Query and analyze auto-discovered fields
On the log search page, auto-discovered fields support field queries, quick analysis, and standard SQL analysis. The usage is consistent with manually created field indexes. Auto-discovered fields are fixed as text type and inherit the tokenizer configuration from the full-text index. To modify the type or index properties, promote the field to a specified field index.
You can perform index queries directly on these fields.
Auto-discovered fields are always text type in the index configuration. In SQL scenarios where numeric intent is clear and standard type analysis cannot pass, the SQL engine attempts to convert field values to the target numeric type. Values that cannot be converted are treated as NULL. This behavior does not modify the field index type, and expressions that can be executed as strings retain string semantics. The following SQL statement shows an example:
* | SELECT customer_city, product_category, sum(order_amount) AS total_amount GROUP BY customer_city, product_categoryIn the preceding SQL example, customer_city, order_amount, and product_category are all auto-discovered fields. You can use sum(order_amount) directly, treating order_amount as a numeric type for analysis.
Limits
| Item | Description |
| Logstore type | Only Standard Logstores are supported. Query Logstores are not supported. |
| Field level | Only top-level fields in logs are auto-discovered. Sub-fields in JSON strings are not automatically expanded. Configure JSON-type field indexes manually for JSON sub-fields. |
| Field type | All auto-discovered fields are uniformly created as text type. For long, double, or JSON types, configure field indexes manually. For more information, see Data types. |
| Field name length | Field names cannot exceed 128 characters. |
| Field name characters | Field names can contain only letters, digits, and _, ., :, @, #, $, -, /, [, ]. System-reserved fields are not automatically indexed. |
| Field quantity | A maximum of 1,024 field indexes can be automatically created per Logstore. No new fields are discovered after the limit is reached. |
| Case sensitivity | If logs contain field names that differ only in case, such as UserId and userid, both are discovered. In pure index queries, they can be used separately. However, SQL column names are case-insensitive, so such fields cannot be reliably distinguished in SQL. Promoting both to specified field indexes is also not supported. |
| Effective scope | Takes effect only on data written after the feature is enabled. For historical data, you must reindex logs. |
| Effective time | After Auto-discover unspecified fields is enabled and new fields appear in logs, the underlying system immediately generates corresponding index data. However, the discovered field configurations appear on the page after a few minutes and a page refresh. |
When a field name starts with a digit or contains special characters, enclose it in double quotation marks in SQL statements. The following example shows the syntax:
* | SELECT "service-name", count(*) GROUP BY "service-name"FAQ
Why are no new fields discovered after I enable auto-discover field indexes?
Check the following items in order:
Whether the Logstore is Standard type.
Whether full-text indexing and auto-discover are enabled.
Whether new logs containing the field have been written after auto-discover was enabled.
Whether the field name meets the length and character requirements and is not a system-reserved field.
Whether the auto-discover field limit or total index field limit has been reached.
Whether you have waited a few minutes and refreshed the index configuration page after writing.
Why can't numeric fields be queried with comparison operators directly?
Auto-discover cannot reliably infer the long-term data type of a field from partial samples, so all auto-discovered fields are created as text type. Pure index queries can only query by text matching, not by key > 3. In SQL, the engine automatically performs on-demand numeric type conversion for discovered fields, so you can use * | SELECT xxx WHERE key > 3 directly in SQL. For confirmed numeric fields, manually configure long or double type field indexes.
Can I still query auto-discovered fields after disabling the feature?
Yes. Disabling the toggle only stops the discovery of new fields. It does not automatically delete the field indexes that have already been created, and you can continue to query and analyze them.
How is auto-discover field indexes billed?
The auto-discover field indexes feature itself does not incur separate charges. After you enable it, Simple Log Service automatically creates text-type field indexes for newly discovered fields, and the associated costs are calculated based on existing index billing rules. Because this feature depends on the already-enabled full-text index, it typically does not generate significant additional costs. Actual costs are subject to the index storage volume and the Logstore billing mode.