Queries logs that are generated within a specified time range from a specified Logstore of a specified project.

Description

Note Log Service allows you to create a Scheduled SQL job. For more information, see Create a Scheduled SQL job.
  • Host consists of a project name and a Log Service endpoint. You must specify a project in Host.
  • If the number of logs in a Logstore significantly changes, Log Service cannot predict the number of times that you must call this operation to obtain the complete results. In this case, you must check the value of the x-log-progress parameter in the response of each request and determine whether to call this operation one more time to obtain the complete results. Each time you call this operation, the same number of charge units (CUs) are consumed.
  • After a log is written to a Logstore, you can call the GetHistograms or the GetLogs operation to query the log. The latency of the query varies based on the type of the log. Log Service classifies logs into the following types based on log timestamps:
    • Real-time data: The difference between the time record in a log of this type and the current time on Log Service is within the interval (-180 seconds,900 seconds]. For example, if a log was generated at 12:03:00, September 25, 2014 (UTC) and Log Service received the log at 12:05:00, September 25, 2014 (UTC), Log Service processes the log as real-time data. This type of log is usually generated in common scenarios.

    • Historical data: The difference between the time record in a log of this type and the current time on Log Service is within the interval [-604,800 seconds,-180 seconds). For example, if a log was generated at 12:00:00, September 25, 2014 (UTC) and Log Service received the log at 12:05:00, September 25, 2014 (UTC), Log Service processes the log as historical data. This type of log is usually generated in data backfill scenarios.

      After real-time data is written to a Logstore, the data can be queried with a maximum latency of 3 seconds. For 99.9% of queries, the latency is no more than 1 second.

Note Log Service calculates the difference between the log time that is specified by the __time__ field and the receiving time that is specified by the __tag__:__receive_time__ field for each log. The receiving time indicates when Log Service receives the log. If the difference is within the interval (-180 seconds,900 seconds], Log Service processes the log as real-time data. If the difference is within the interval [-604,800 seconds,-180 seconds), Log Service processes the log as historical data.

Debugging

OpenAPI Explorer automatically calculates the signature value. For your convenience, we recommend that you call this operation in OpenAPI Explorer. OpenAPI Explorer dynamically generates the sample code of the operation for different SDKs.

Request headers

This operation uses only common request headers. For more information, see Common request headers.

Request syntax

GET /logstores/{logstore}?type=log HTTP/1.1

Request parameters

ParameterTypePositionRequiredExampleDescription
projectStringHostYesali-test-project

The name of the project.

logstoreStringPathYesexample-logstore

The Logstore whose data you want to query.

fromLongQueryYes1627268185

The beginning of the time range to query. The value is the log time that is specified when log data is written.

  • The time range that is specified in this operation is a left-closed, right-open interval. The interval includes the start time specified by the from parameter, but does not include the end time specified by the to parameter. If you specify the same value for the from and to parameters, the interval is invalid, and an error message is returned.
  • The value is a UNIX timestamp representing the number of seconds that have elapsed since the epoch time January 1, 1970, 00:00:00 UTC.
Note To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis.

If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:

  • * | SELECT * FROM log WHERE from_unixtime(__time__) > from_unixtime(1664186624) AND from_unixtime(__time__) < now()
  • * | SELECT * FROM log WHERE __time__ > to_unixtime(date_parse('2022-10-19 15:46:05', '%Y-%m-%d %H:%i:%s')) AND __time__ < to_unixtime(now())
toLongQueryYes1627269085

The end of the time range to query. The value is the log time that is specified when log data is written.

  • The time range that is specified in this operation is a left-closed, right-open interval. The interval includes the start time specified by the from parameter, but does not include the end time specified by the to parameter. If you specify the same value for the from and to parameters, the interval is invalid, and an error message is returned.
  • The value is a UNIX timestamp representing the number of seconds that have elapsed since the epoch time January 1, 1970, 00:00:00 UTC.
Note To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis.

If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:

  • * | SELECT * FROM log WHERE from_unixtime(__time__) > from_unixtime(1664186624) AND from_unixtime(__time__) < now()
  • * | SELECT * FROM log WHERE __time__ > to_unixtime(date_parse('2022-10-19 15:46:05', '%Y-%m-%d %H:%i:%s')) AND __time__ < to_unixtime(now())
queryStringQueryNostatus: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5

The search statement or the query statement. For more information, see Log search overview and Log analysis overview.

If you add set session parallel_sql=true; to the analytic statement in the query parameter, Dedicated SQL is used. For example, you can set the query parameter to * | set session parallel_sql=true; select count(*) as pv. For more information about common errors that may occur during log query and analysis, see How do I resolve common errors that occur when I query and analyze log data?

Note If you specify an analytic statement in the query parameter, the line and offset parameters do not take effect in this operation. In this case, we recommend that you set the line and offset parameters to 0 and use the LIMIT clause to limit the number of logs to return on each page. For more information, see Paged query.
topicStringQueryNotopic

The topic of the logs. The default value is double quotation marks (""). For more information, see Topic.

lineLongQueryNo100

The maximum number of logs to return for the request. This parameter takes effect only when the query parameter is set to a search statement. Minimum value: 0. Maximum value: 100. Default value: 100.

offsetLongQueryNo0

The row from which the query starts. This parameter takes effect only when the query parameter is set to a search statement. Default value: 0.

reverseBooleanQueryNofalse

Specifies whether to return logs in reverse chronological order of log timestamps. The log timestamps are accurate to the minute. Valid values:

  • true: Logs are returned in reverse chronological order of log timestamps.
  • false: Logs are returned in chronological order of log timestamps. This is the default value.
Important
  • The reverse parameter takes effect only when the query parameter is set to a search statement. The reverse parameter specifies the method used to sort returned logs.
  • If the query parameter is set to a query statement, the reverse parameter does not take effect. The method used to sort returned logs is specified by the ORDER BY clause in the analytic statement. If you use the keyword asc in the ORDER BY clause, the logs are sorted in chronological order. If you use the keyword desc in the ORDER BY clause, the logs are sorted in reverse chronological order. By default, asc is used in the ORDER BY clause.
powerSqlBooleanQueryNofalse

Specifies whether to use Dedicated SQL. For more information, see Enable Dedicated SQL. Valid values:

  • true: uses Dedicated SQL.
  • false: uses Standard SQL. This is the default value.

You can use the powerSql or query parameter to configure Dedicated SQL.

Response parameters

ParameterTypeExampleDescription
x-log-progressStringComplete

The status of the query and analysis results. Valid values:

  • Complete: The query is successful, and the complete query and analysis results are returned.
  • Incomplete: The query is successful, but the query and analysis results are incomplete. To obtain the complete results, you must repeat the request.
Note You can use the *|select count(*) as count statement to obtain the total number of logs. You can also call the GetHistogram operation to obtain the number of logs that are generated within each subinterval of a time range and add the numbers to obtain the total number of logs. If you do not need to obtain the total number of logs, you can modify the offset parameter in the request and repeat the request multiple times. If the value of the x-log-progress parameter is Complete and the number of returned rows is less than the number of requested rows, all logs are read.
x-log-countLong100

The number of rows that are returned.

Note The total number of logs that meet the specified conditions is not returned. To query the total number of logs that meet the specified conditions, you must specify a query statement. For example, the query statement request_method:GET|select count(*) as count can be used to query the total number of logs whose request_method is GET. For more information, see Best practices for query and analysis and Examples for querying and analyzing logs by using SDK for Java.
x-log-processed-rowsLong10000

The number of lines that are processed based on the request.

x-log-elapsed-millisecondLong5

The time that is consumed by the request. Unit: milliseconds.

ServerStringnginx

The name of the server.

Content-TypeStringapplication/json

The content type of the response body.

Content-LengthString0

The content length of the response body.

ConnectionStringclose

Indicates whether the connection is persistent. Valid values:

  • close: The connection is non-persistent. A new TCP connection is established for each HTTP request.
  • keep-alive: The connection is persistent. After a TCP connection is established, the connection remains open, and no more time or bandwidth is consumed to establish new connections.
DateStringSun, 27 May 2018 08:25:04 GMT

The time at which the response was returned.

x-log-requestidString5B0A6B60BB6EE39764D458B5

The ID of the request. This is a unique ID generated by Log Service.

Array of Object[{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}]

An array of logs. Each element in the array indicates a log.

Examples

Sample requests

GET /logstores/{logstore}?type=log?from=1627268185&to=1627269085&query=status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5&topic=topic&line=100&offset=0&reverse=false&powerSql=false HTTP/1.1
Host:ali-test-project.cn-hangzhou.log.aliyuncs.com
Content-Type:application/json

Description of the sample requests

### Sample raw log
```json
    {
        "remote_addr": "203.0.*.*",
        "__time__": 1649904328,
        "__topic__": "nginx_access_log",
        "__source__": "127.0.0.1",
        "body_bytes_sent": "3841",
        "__tag__:__receive_time__": "1649904334",
        "time_local": "14/Apr/2022:02:45:28",
        "request_method": "HEAD",
        "request_uri": "/request/path-2/file-2",
        "http_user_agent": "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:21.0.0) Gecko/20121011 Firefox/21.0.0",
        "remote_user": "jlqd8",
        "request_time": "17",
        "__pack_meta__": "1|MTYyODY2Mjk0NzQ1ODU1Mjk4Mg==|197|165",
        "request_length": "4184",
        "http_referer": "www.example.com",
        "__tag__:__receive_time___0": "1649904334",
        "host": "www.example.com",
        "http_x_forwarded_for": "198.51.100.48",
        "upstream_response_time": "0.23",
        "status": "401"
    }
```

Sample success responses

JSON format

HTTP/1.1 200 OK
Content-Type:application/json

{
  "x-log-progress" : "Complete",
  "x-log-count" : 5,
  "logs" : "[{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}]"
}

Error codes

For a list of error codes, see Service error codes.

HTTP status code

Error code

Error message

Description

404

ProjectNotExist

Project does not exist.

The error message returned because the specified project does not exist.

404

LogStoreNotExist

Logstore does not exist.

The error message returned because the specified Logstore does not exist.

400

InvalidTimeRange

Request time range is invalid.

The error message returned because the specified time range is invalid.

400

InvalidQueryString

Query string is invalid.

The error message returned because the specified query statement is invalid.

400

InvalidOffset

Offset is invalid.

The error message returned because the specified offset parameter is invalid.

400

InvalidLine

Line is invalid.

The error message returned because the specified line parameter is invalid.

400

InvalidReverse

Reverse value is invalid.

The error message returned because the specified reverse parameter is invalid.

400

IndexConfigNotExist

Logstore without index config.

The error message returned because the indexing feature is not enabled for the Logstore.

400

ParameterInvalid

ErrorType:OLSQueryParseError.ErrorMessage:offset is not available for pagination in sql query, please use limit x,y syntax for pagination.

The error message returned because the specified analytic statement in the query parameter of the request is invalid. If the query parameter is set to a query statement that contains an analytic statement, we recommend that you set the line and offset parameters to 0 and specify a LIMIT clause for pagination.

For more information about how to troubleshoot the errors that are related to the SQL statement specified by the query parameter, see How do I resolve common errors that occur when I query and analyze log data?

500

InternalServerError

Specified Server Error Message.

The error message returned because an internal server error has occurred.

For more information, see Common error codes.