Simple Log Service:Overview of data views (StoreViews)

What is a StoreView

In a Simple Log Service project, StoreView, LogStore, and MetricStore are all important resources. However, a StoreView is not a physical resource for storing logs. It is a virtual resource created based on Logstores and Metricstores to manage and save associations among multiple Logstores or Metricstores. Using a StoreView, you can perform unified queries on logs from different Logstores or Metricstores. However, StoreViews do not support log modification operations. StoreViews are categorized into log-type StoreViews and time-series-type StoreViews. A log-type StoreView supports querying and analyzing data from different Logstores and predefining associated stores using Structured Process Language (SPL) syntax. A time-series-type StoreView only supports querying data across different Metricstores.

In a log-type StoreView/time-series-type StoreView, you can perform associated queries on log content from different Logstores/Metricstores. For example, if the age field is included in Logstore1 and Logstore2, and the two Logstores are associated with StoreView1, a query for the age field in StoreView1 returns all results from Logstore1 and Logstore2.

Permissions

If you use StoreViews as a Resource Access Management (RAM) user or by assuming a RAM role, you must have the required permissions. For more information about how to use an Alibaba Cloud account to grant permissions to a RAM user or RAM role, see Create a RAM user and grant permissions, Create a RAM role for a trusted Alibaba Cloud service and grant permissions, and Use a RAM role to access resources across Alibaba Cloud accounts.

You can grant system policies or custom policies to the user.

• System policy: Created and maintained by Alibaba Cloud. You can use system policies but cannot modify them.

  • The read-only permissions on Simple Log Service: AliyunLogReadOnlyAccess.

  • The management permissions on Simple Log Service: AliyunLogFullAccess.

• Custom policy: You can create, update, and delete custom policies. You must maintain the policy versions yourself.

  • When you create a StoreView or query data in a StoreView, the RAM user or RAM role must have the permissions to query logs in a Logstore for all associated Logstores and Metricstores. To grant the permissions, add the following content to a custom policy:

    Important

    • The Logstore in the access policy includes Logstores and Metricstores. This policy also applies when your operation object is a Metricstore.

    • The storeview in the access policy includes log-type StoreViews and time-series-type StoreViews.

    {
      "Version": "1",
      "Statement": [
        {
          "Action": "log:GetLogStoreLogs",
          "Resource":[
            "acs:log:*:*:project/${projectName}/Logstore/*",
            "acs:log:*:*:project/${projectName}/storeview/*"
          ],
          "Effect": "Allow"
        }
      ]
    }

Limits

• Each Simple Log Service project supports a maximum of 10 log-type StoreViews/time-series-type StoreViews in total.

• Each log-type StoreView/time-series-type StoreView can be associated with a maximum of 50 Logstores/Metricstores.

• All Logstores/Metricstores in a log-type StoreView/time-series-type StoreView must belong to the same Alibaba Cloud account.

• StoreViews support association analysis only between regions in the Chinese mainland. Cross-region analysis in cross-border scenarios is not supported.

Predefinition capabilities

Concept

A log-type StoreView supports predefining associated stores using SPL syntax when you create the StoreView. The predefined SPL supports only SPL instructions and functions and SPL instruction and function keywords. You can use predefinition for scenarios such as filtering some data or resolving inconsistent field names in aggregation and analysis. Aggregation and analysis require consistent field keys. For example, to aggregate the a field from Logstore1 and the b field from Logstore2, you must use extend a = b to make the column names consistent. For more information, see Examples.

Limits

Currently, log-type StoreViews support predefinition. Time-series-type StoreViews do not support predefinition.

Supported instructions

The predefinition for a log-type StoreView supports SPL instructions and functions and SPL instruction and function operators.

For more information about the functions that SQL and SPL support, see Function overview.

Examples

To create or modify a log-type StoreView, see Create a log-type data view.

For example, to aggregate the a field from Logstore1 and the b field from Logstore2, you can use extend a = b (predefinition) when you create or modify the StoreView to make the column names consistent.

Example 1

Predefine * | where a>1 in Logstore1. This means that only logs where a>1 are hit in the StoreView and used for subsequent analysis.

Example 2

The predefined * | extend b = a SPL statement for LogStore 1 in StoreView adds the b field whose values are the same as the values of the a field to LogStore 1 for subsequent aggregation.

Query capabilities

Log-type StoreView

A log-type StoreView supports cross-Logstore queries. For example, if a log-type StoreView is associated with two stores, Logstore-1 and Logstore-2, and both stores contain the request_method field, the query statement (request_method:GET or request_method:POST) and status in [200 299] returns the matching log results from both Logstore-1 and Logstore-2. For more information about queries in a log-type StoreView, see Query syntax and functions.

Assume a log-type StoreView contains two Logstores with identical log content. When you query for a keyword, all results from the different Logstores are returned. You can use the tag field to determine which Logstore a log comes from.

Time-series-type StoreView

A time-series-type StoreView supports cross-store queries. For example, if a time-series-type StoreView is associated with two stores, metricstore1 and metricstore2, you can enter a Prometheus Query Language (PromQL) statement directly in the query box. The backend computing engine automatically completes the cross-Metricstore computation. For more information about queries in a time-series-type StoreView, see Syntax for time series data query and analysis. To create or modify a time-series-type StoreView, see Query data across Metricstores.

Analysis capabilities

Concept

Data View supports cross-Store aggregation and statistical analysis. For example, if a StoreView is associated with LogStore-1 and LogStore-2, and both Logstores contain the field a, the query statement * | select a, count(1) group by a will return the total count of field `a` from both LogStore-1 and LogStore-2.

Because aggregation and analysis require consistent field keys, you can use predefinition to filter some data and rename fields for scenarios where log content is the same but key names are different. For more information, see Predefinition capabilities.

Limits

Currently, log-type StoreViews support data analytics. Time-series-type StoreViews do not support data analytics.

Supported functions

For more information about the functions that SQL and SPL support, see Function overview.

Examples

Example 1

A log-type StoreView is associated with different Logstores from the same service, and the log content in the Logstores is identical. In this scenario, first check whether the index configurations of the different Logstores are the same. For example, check if they all have full-text indexes or if the required field index configurations are consistent. Then, add the Logstores that you want to associate to the log-type StoreView.

After you create the log-type StoreView, you can use SQL for analysis as usual. The following figure shows that the statistics for different status values in the last 15 minutes are an aggregation of all logs from the two Logstores.

Example 2

A log-type StoreView is associated with different Logstores from the same service, but some log fields in the Logstores have different names. In this case, you can use a predefined SPL statement to redefine a new field to align with the key name of the other Logstore. As shown in the following figure, use the SPL statement *|extend request_method = method to rename the method field to request_method. Click Preview Data to preview the predefined result data.

You can then use request_method for query and analysis.

Differences between a StoreView and a regular store

References

• Query and analyze data across Logstores

• Query data across Metricstores