Log Service provides the log analysis feature. This feature works with the log search feature and is implemented by using SQL syntax.

Syntax

Each query statement consists of a search statement and an analytic statement. The search statement and analytic statement are separated by a vertical bar (|). A search statement can be separately executed. However, an analytic statement must be executed together with a search statement. The log analysis feature is used to analyze search results or all data in a Logstore.
Note Analytic statements are not case-sensitive.
  • Format of a query statement 
    Search statement|Analytic statement
    Statement Description
    Search statement A search statement specifies one or more search conditions and returns the logs that meet the specified conditions.

    A search statement can be a keyword, numeric value, numeric value range, space, or asterisk (*). If you specify the search statement as a space or asterisk (*), no conditions are specified, and all logs are returned. For more information, see Search syntax.

    Analytic statement An analytic statement performs computing or collects statistics on either search results or all logs.
  • Example 
    * | SELECT status, count(*) AS PV GROUP BY status

Syntax description

Log Service allows you to analyze logs by using the standard SQL-92 syntax. When you use an analytic statement in Log Service, take note of the following instructions:

  • You do not need to use a semicolon (;) at the end of the analytic statement as the statement terminator.
  • If you do not use an SQL nested subquery, you do not need to specify the FROM or WHERE clause in the analytic statement. By default, all logs of the current Logstore are analyzed.
  • You can use an SQL nested subquery to perform complex data analysis. If you use an SQL nested subquery, you must specify the FROM clause. 
    * | SELECT sum(pv) FROM (SELECT count(*) AS pv FROM log GROUP BY method)
  • A column name that is specified in the analytic statement can contain only letters, digits, and underscores (_). The column name must start with a letter.
    If you specify a column name that does not comply with the SQL-92 syntax when you collect logs, you must specify an alias for the column name when you configure indexes. Aliases are used only for SQL analysis. Original column names are used in storage. You must use the original column names in search statements. For more information about how to specify an alias, see Configure indexes. Column aliases

Limits

Item Standard instance Dedicated instance
Number of concurrent analytic statements Each project supports a maximum of 15 concurrent analytic statements at a time.

For example, 15 users can execute analytic statements in all Logstores of a project at the same time.

Each project supports a maximum of 150 concurrent analytic statements at a time.

For example, 150 users can execute analytic statements in all Logstores of a project at the same time.

Data volume Each shard supports only 1 GB of data for a single analytic statement. A single analysis allows you to scan up to 200 billion rows of data at a time.
Method to enable Standard instances are enabled by default. Dedicated SQL instances are enabled by using a switch. For more information, see Enable Dedicated SQL.
Resource usage fee Free of charge. You are charged based on the actual CPU time.
Applicable scope You can analyze only the data that is written to Log Service after the log analysis feature is enabled.

If you want to analyze historical data, you must re-index the historical data. For more information, see Reindex logs for a Logstore.

You can analyze only the data that is written to Log Service after the log analysis feature is enabled.

If you want to analyze historical data, you must re-index the historical data. For more information, see Reindex logs for a Logstore.

Returned result After you execute an analytic statement, a maximum of 100 rows of data are returned by default.

If you require more data, use the LIMIT clause. For more information, see LIMIT syntax.

After you execute an analytic statement, a maximum of 100 rows of data are returned by default.

If you require more data, use the LIMIT clause. For more information, see LIMIT syntax.

Size of a field value The maximum size of a field value is 16 KB. If the size of a field value exceeds 16 KB, the excess content is not analyzed. The maximum size of a field value is 16 KB. If the size of a field value exceeds 16 KB, the excess content is not analyzed.
Timeout period The maximum timeout period for a single analytic statement is 55 seconds. The maximum timeout period for a single analytic statement is 55 seconds.
Number of digits that consists of a field value of the double type Each field value of the double type consists of a maximum of 52 digits.

If the number of digits is greater than 52, the accuracy of the field value is compromised.

Each field value of the double type consists of a maximum of 52 digits.

If the number of digits after the decimal point is greater than 52, the accuracy is compromised for those digits.

Implementation methods

Notice
  • If you want to use the log analysis feature, you must turn on Enable Analytics for the required fields when you configure indexes. For more information, see Configure indexes.
  • Log Service provides reserved fields. For more information about how to analyze reserved fields, see Reserved fields.
  • Use the Log Service console

    Log on to the Log Service console. On the query and analysis page of a Logstore, specify a time range and execute a query statement. For more information, see Query logs.

  • Use the API or an SDK

    Call the GetLogs or GetHistograms operation to query and analyze logs.

Analytic functions and syntax

This section lists the analytic functions and syntax supported by Log Service.

Visualization of analysis results

The following figure shows a sample dashboard that displays the analysis results. For more information, see Visualization overview.

Analysis results