Log Service provides the log analysis feature. This feature works together with the log search feature and is implemented by using the SQL syntax.
Syntax
- Syntax
Search statement|Analytic statement
Statement Description Search statement A search statement specifies one or more search conditions, and then returns the logs that meet the specified conditions. A search statement can be a keyword, a numeric value, a numeric value range, a space character, or an asterisk (*). If you specify a space character or an asterisk (*) as the search statement, no conditions are specified and all logs are returned. For more information, see Search syntax.
Analytic statement An analytic statement is used to aggregate or analyze all log data or the log data that meets the specified search conditions in a Logstore. - Example
* | SELECT status, count(*) AS PV GROUP BY status
Syntax description
Log Service allows you to analyze logs by using the standard SQL-92 syntax. When you use an analytic statement in Log Service, take note of the following points:
- You do not need to add a semicolon (;) at the end of the analytic statement to specify the end of the statement.
- If you do not want to use an SQL nested subquery, you do not need to specify a FROM or WHERE clause in the analytic statement. By default, all log data of the current Logstore is analyzed.
- You can use an SQL nested subquery to perform complex data analysis. If you want to
use an SQL nested subquery, you must specify a FROM clause.
* | SELECT sum(pv) FROM (SELECT count(*) AS pv FROM log GROUP BY method)
- If you want to use a string in an analytic statement, you must enclose the string in single quotation marks (''). Strings that are not enclosed or enclosed in double quotation marks ("") indicate field names or column names. For example, 'status' indicates the status string, and status or "status" indicates the status log field.
- A column name that you specify in the analytic statement can contain only letters,
digits, and underscores (_). The column name must start with a letter.
If you specify a column name that does not comply with the SQL-92 syntax when you collect logs, you must specify an alias for the column name when you configure indexes. For more information about how to configure indexes, see Configure indexes.Notice Aliases are used only for SQL analysis. The original column names are used for storage. You must use the original column names in search statements and use aliases in analytic statements.
Limits
Item | Standard SQL | Dedicated SQL |
---|---|---|
Number of concurrent analytic statements | Each project supports a maximum of 15 concurrent analytic statements.
For example, 15 users can concurrently execute analytic statements in all Logstores of a project. |
Each project supports a maximum of 150 concurrent analytic statements.
For example, 150 users can concurrently execute analytic statements in all Logstores of a project. |
Data volume | Each shard supports only 1 GB of data for a single analytic statement. | An analytic statement can scan a maximum of 200 billion rows of data at the same time. |
Method to enable | By default, Standard SQL is enabled. | A switch is provided for you to manually enable Dedicated SQL. For more information, see Enable Dedicated SQL. |
Resource usage fee | Free of charge. | You are charged based on the actual CPU time. For more information, see Billable items. |
Applicable scope | You can analyze only the data that is written to Log Service after the log analysis
feature is enabled.
If you want to analyze historical data, you must reindex the historical data. For more information, see Reindex logs for a Logstore. |
You can analyze only the data that is written to Log Service after the log analysis
feature is enabled.
If you want to analyze historical data, you must reindex the historical data. For more information, see Reindex logs for a Logstore. |
Returned result | By default, an analytic statement returns a maximum of 100 rows of data.
If you want to view more data, use a LIMIT clause. For more information, see LIMIT clause. |
By default, an analytic statement returns a maximum of 100 rows of data.
If you want to view more data, use a LIMIT clause. For more information, see LIMIT clause. |
Size of a field value | The log analysis feature can analyze a maximum of 16,384 bytes (16 KB) of data in
the value of each field. If the size of a field value exceeds 16 KB, the excess content
is not analyzed.
You can change the maximum size for each field value when you configure indexes. Valid values: 64 to 16384. Unit: bytes. For more information, see Configure indexes. |
The log analysis feature can analyze a maximum of 16,384 bytes (16 KB) of data in
the value of each field. If the size of a field value exceeds 16 KB, the excess content
is not analyzed.
You can change the maximum size for each field value when you configure indexes. Valid values: 64 to 16384. Unit: bytes. For more information, see Configure indexes. |
Timeout period | The maximum timeout period for a single analytic statement is 55 seconds. | The maximum timeout period for a single analytic statement is 55 seconds. |
Number of decimal places in the value of a double-type field | The value of a double-type field can contain a maximum of 52 decimal places.
If the number of decimal places is greater than 52, the accuracy of the field value is compromised. |
The value of a double-type field can contain a maximum of 52 decimal places.
If the number of decimal places is greater than 52, the accuracy of the field value is compromised. |
Analysis methods
- If you want to use the log analysis feature, you must turn on Enable Analytics for the required fields when you configure indexes. For more information, see Configure indexes.
- Log Service provides reserved fields. For more information about how to analyze reserved fields, see Reserved fields.
- Use the Log Service console
Log on to the Log Service console. On the Search & Analysis page of a Logstore, specify a time range and execute a query statement. For more information, see Query and analyze logs.
- Call the API or use an SDK
Call the GetLogs or GetHistograms operation to query and analyze logs.
Analytic functions and syntax
This section lists the analytic functions and syntax that are supported by Log Service.
- SQL functions
- Aggregate function
- String functions
- Date and time functions
- JSON functions
- Regular expression functions
- Interval-valued comparison and periodicity-valued comparison functions
- Array functions and operators
- Map functions and operators
- Mathematical calculation functions
- Mathematical statistics functions
- Data type conversion functions
- Security check functions
- Window functions
- IP functions
- URL functions
- Approximate functions
- Binary functions
- Bitwise functions
- Geospatial functions
- Geo functions
- Color functions
- HyperLogLog functions
- Comparison operators
- Logical operators
- Unit conversion functions
- Window funnel function
- Lambda expressions
- Conditional expressions
- Machine learning syntax and functions
- Smooth functions
- Multi-period estimation functions
- Change point detection functions
- Maximum value detection functions
- Prediction and anomaly detection functions
- Sequence decomposition function
- Time series clustering functions
- Frequent pattern statistical function
- Differential pattern statistical function
- Request URL classification function
- Root cause analysis function
- Correlation analysis functions
- Kernal density estimation functions
- Time series padding function
- Anomaly comparison function
- SQL syntax
Sample analysis results
The following figure shows a sample dashboard that displays the analysis results. For more information, see Visualization overview.
