SLS Elasticsearch compatibility, Elasticsearch-compatible API, Simple Log Service - Simple Log Service

Simple Log Service provides an Elasticsearch-compatible API that translates Elasticsearch DSL queries into SLS index queries and SQL analysis queries. This topic covers compatibility principles, supported API operations, and supported DSL syntax.

Important

This document is proprietary to Alibaba Cloud. It describes how Alibaba Cloud services interact with third-party products and may refer to the names of third-party companies or products.

Compatibility principle

Elasticsearch is an open-source, distributed search and analysis engine based on Apache Lucene. When you migrate from Elasticsearch to Simple Log Service, you may need to adjust workflows and modify upstream and downstream programs. Simple Log Service provides an Elasticsearch-compatible API to simplify this migration.

The API translates Elasticsearch DSL queries into Simple Log Service index queries and SQL analysis queries, and returns results in Elasticsearch API format.

Note

The Elasticsearch-compatible API is based on the Elasticsearch 7.10 specification.
The Elasticsearch-compatible API supports only Standard Logstores, and you must create at least one field index.

Elasticsearch-compatible API endpoints

The endpoint format is https://${project}.${sls-endpoint}/es/. For example, https://etl-guangzhou.cn-guangzhou.log.aliyuncs.com/es/. ${project} is the project name and ${sls-endpoint} is the Simple Log Service endpoint. For more information, see Endpoints.

Supported Elasticsearch API operations

The Elasticsearch-compatible API supports the following operations.

Request method	API path	Description
Post	/${index}/_search	Queries the specified index. The index maps to a Simple Log Service project and Logstore in the format `${project}.${logstore}`.
Post	/${index}/_async_search	Performs a batch query.
Post	/_msearch	Performs a batch query.
Post	/${index}/_bulk	Writes data to the specified index. Important The _bulk operation supports only data writes and does not support data updates.
Get	/${index}/_mapping	Obtains the mapping of the specified index.

Supported Elasticsearch DSL

The Elasticsearch-compatible API supports the following Elasticsearch DSL.

Query-related DSL

DSL	Description
bool	Performs a Boolean operation.
match_all	Matches all logs.
match	Matches an index.
match_phrase	Phrase match. Translated into a SQL LIKE clause. Returns only indexed fields.
multi_match	Matches multiple fields.
prefix	Performs a prefix query.
range	Performs a range query.
term	Performs a term query. Translated into Simple Log Service index query syntax.
terms	Performs a query for multiple terms. Translated into Simple Log Service index query syntax.
wildcard	Matches a wildcard pattern. Translated into Simple Log Service SQL analysis syntax.
regexp	Matches a regular expression. Translated into SQL regexp_like syntax. Returns only indexed fields.
exists	Checks whether a field exists.
ids	Queries the ID field.

Aggregation-related DSL

Category	DSL	Description
Group statistics	Date histogram	Groups statistics by the `@timestamp` field.
	Histogram	Groups statistics by a numeric interval.
	Range	Groups statistics by a numeric range.
	Terms	Groups statistics by the unique values of a field.
	Filter	Groups statistics by a single filter condition.
	Filters	Groups statistics by multiple filter conditions.
Metric statistics	Min	Calculates the minimum value.
	Max	Calculates the maximum value.
	Avg	Calculates the average value.
	Sum	Calculates the sum.
	Count	Calculates the count.
	Cardinality	Calculates the count of unique values.
	Percentiles	Calculates the percentile.
Returned log	Top hits	Returns top matching logs per group.

Differences from standard Elasticsearch

The Elasticsearch-compatible API does not support data updates.
The Elasticsearch-compatible API does not support script-based queries.
A single Logstore supports petabytes of data, so index rollovers are not required.
Indexes are not automatically created. Create them through the API or console. Create an index.

The following fields have specific mappings in the Elasticsearch DSL.

Elasticsearch	Simple Log Service	Description
`@timestamp`	`__time__`	The `@timestamp` field maps to the Simple Log Service `__time__` field in queries.
`_id`	None	Maps to the `_id` field if it exists in Simple Log Service data. If the `_id` field does not exist, it is auto-populated with `uuid` or `_pack_meta`.

RAM authorization

Grant a Resource Access Management (RAM) user query and write permissions for the Elasticsearch-compatible API by using one of the following access policies.

System policies: easy to configure.
Custom policies: provide fine-grained permissions but are complex to configure.

Simple authorization

Use system policies to grant a RAM user query and write permissions for the Elasticsearch-compatible API. Manage the permissions of a RAM user.

System policy name	Query permission	Write permission
AliyunLogFullAccess	Yes	Yes
AliyunLogReadOnlyAccess	Yes	No

Custom policies

Use custom policies for fine-grained permissions. Create a custom policy. Manage the permissions of a RAM user.

Grant query and write permissions for the Elasticsearch-compatible API.

{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "log:GetIndex",
                "log:GetLogStoreContextLogs",
                "log:ListLogStores",
                "log:GetLogStoreHistogram",
                "log:GetLogstoreLogs",
                "log:GetLogStoreContextLogs",
                "log:GetCursorOrData",
                "log:GetLogstore",
                "log:PostLogStoreLogs"
            ],
            "Resource": "acs:log:*:*:project/*"
        }
    ]
}

Grant query-only permission for the Elasticsearch-compatible API.

{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "log:GetIndex",
                "log:GetLogStoreContextLogs",
                "log:ListLogStores",
                "log:GetLogStoreHistogram",
                "log:GetLogstoreLogs",
                "log:GetLogStoreContextLogs",
                "log:GetCursorOrData",
                "log:GetLogstore"
            ],
            "Resource": "acs:log:*:*:project/*"
        }
    ]
}

Grant write-only permission for the Elasticsearch-compatible API.

{
    "Version": "1",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "log:PostLogStoreLogs"
            ],
            "Resource": "acs:log:*:*:project/*"
        }
    ]
}