Queries logs that are generated within a specified time range from a specified Logstore of a specified project.
Description
- Host consists of a project name and a Log Service endpoint. You must specify a project in Host.
- If the number of logs in a Logstore significantly changes, Log Service cannot predict the number of times that you must call this operation to obtain the complete results. In this case, you must check the value of the x-log-progress parameter in the response of each request and determine whether to call this operation one more time to obtain the complete results. Each time you call this operation, the same number of charge units (CUs) are consumed.
- After a log is written to a Logstore, you can call the GetHistograms or the GetLogs operation to query the log. The latency of the query varies based on the type of the log. Log Service classifies logs into the following types based on log timestamps:
- Real-time data: The difference between the time record in a log of this type and the current time on Log Service is within the interval (-180 seconds,900 seconds]. For example, if a log was generated at 12:03:00, September 25, 2014 (UTC) and Log Service received the log at 12:05:00, September 25, 2014 (UTC), Log Service processes the log as real-time data. This type of log is usually generated in common scenarios.
Historical data: The difference between the time record in a log of this type and the current time on Log Service is within the interval [-604,800 seconds,-180 seconds). For example, if a log was generated at 12:00:00, September 25, 2014 (UTC) and Log Service received the log at 12:05:00, September 25, 2014 (UTC), Log Service processes the log as historical data. This type of log is usually generated in data backfill scenarios.
After real-time data is written to a Logstore, the data can be queried with a maximum latency of 3 seconds. For 99.9% of queries, the latency is no more than 1 second.
- Log Service provides examples on how to call the GetLogs operation by using Log Service SDK for Java and Log Service SDK for Python. For more information, see Examples of calling the GetLogs operation by using Log Service SDK for Java and Examples of calling the GetLogs operation by using Log Service SDK for Python.
Debugging
Request headers
This operation uses only common request headers. For more information, see Common request headers.
Request syntax
GET /logstores/{logstore}?type=log HTTP/1.1
Request parameters
Parameter | Type | Position | Required | Example | Description |
---|---|---|---|---|---|
project | String | Host | Yes | ali-test-project | The name of the project. |
logstore | String | Path | Yes | example-logstore | The Logstore whose data you want to query. |
from | Long | Query | Yes | 1627268185 | The beginning of the time range to query. The value is the log time that is specified when log data is written.
Note To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis. If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:
|
to | Long | Query | Yes | 1627269085 | The end of the time range to query. The value is the log time that is specified when log data is written.
Note To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis. If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:
|
query | String | Query | No | status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5 | The search statement or the query statement. For more information, see Log search overview and Log analysis overview. If you add Note If you specify an analytic statement in the query parameter, the line and offset parameters do not take effect in this operation. In this case, we recommend that you set the line and offset parameters to 0 and use the LIMIT clause to limit the number of logs to return on each page. For more information, see Paged query. |
topic | String | Query | No | topic | The topic of the logs. The default value is double quotation marks (""). For more information, see Topic. |
line | Long | Query | No | 100 | The maximum number of logs to return for the request. This parameter takes effect only when the query parameter is set to a search statement. Minimum value: 0. Maximum value: 100. Default value: 100. |
offset | Long | Query | No | 0 | The row from which the query starts. This parameter takes effect only when the query parameter is set to a search statement. Default value: 0. |
reverse | Boolean | Query | No | false | Specifies whether to return logs in reverse chronological order of log timestamps. The log timestamps are accurate to the minute. Valid values:
Important
|
powerSql | Boolean | Query | No | false | Specifies whether to use Dedicated SQL. For more information, see Enable Dedicated SQL. Valid values:
You can use the powerSql or query parameter to configure Dedicated SQL. |
Response parameters
Parameter | Type | Example | Description |
---|---|---|---|
x-log-progress | String | Complete | The status of the query and analysis results. Valid values:
Note You can use the *|select count(*) as count statement to obtain the total number of logs. You can also call the GetHistogram operation to obtain the number of logs that are generated within each subinterval of a time range and add the numbers to obtain the total number of logs. If you do not need to obtain the total number of logs, you can modify the offset parameter in the request and repeat the request multiple times. If the value of the x-log-progress parameter is Complete and the number of returned rows is less than the number of requested rows, all logs are read. |
x-log-count | Long | 100 | The number of rows that are returned. Note The total number of logs that meet the specified conditions is not returned. To query the total number of logs that meet the specified conditions, you must specify a query statement. For example, the query statement request_method:GET|select count(*) as count can be used to query the total number of logs whose request_method is GET. For more information, see Best practices for query and analysis and Examples for querying and analyzing logs by using SDK for Java. |
x-log-processed-rows | Long | 10000 | The number of lines that are processed based on the request. |
x-log-elapsed-millisecond | Long | 5 | The time that is consumed by the request. Unit: milliseconds. |
Server | String | nginx | The name of the server. |
Content-Type | String | application/json | The content type of the response body. |
Content-Length | String | 0 | The content length of the response body. |
Connection | String | close | Indicates whether the connection is persistent. Valid values:
|
Date | String | Sun, 27 May 2018 08:25:04 GMT | The time at which the response was returned. |
x-log-requestid | String | 5B0A6B60BB6EE39764D458B5 | The ID of the request. This is a unique ID generated by Log Service. |
Array of Object | [{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}] | An array of logs. Each element in the array indicates a log. |
Examples
Sample requests
GET /logstores/{logstore}?type=log?from=1627268185&to=1627269085&query=status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5&topic=topic&line=100&offset=0&reverse=false&powerSql=false HTTP/1.1
Host:ali-test-project.cn-hangzhou.log.aliyuncs.com
Content-Type:application/json
Description of the sample requests
### Sample raw log
```json
{
"remote_addr": "203.0.*.*",
"__time__": 1649904328,
"__topic__": "nginx_access_log",
"__source__": "127.0.0.1",
"body_bytes_sent": "3841",
"__tag__:__receive_time__": "1649904334",
"time_local": "14/Apr/2022:02:45:28",
"request_method": "HEAD",
"request_uri": "/request/path-2/file-2",
"http_user_agent": "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:21.0.0) Gecko/20121011 Firefox/21.0.0",
"remote_user": "jlqd8",
"request_time": "17",
"__pack_meta__": "1|MTYyODY2Mjk0NzQ1ODU1Mjk4Mg==|197|165",
"request_length": "4184",
"http_referer": "www.example.com",
"__tag__:__receive_time___0": "1649904334",
"host": "www.example.com",
"http_x_forwarded_for": "198.51.100.48",
"upstream_response_time": "0.23",
"status": "401"
}
```
Sample success responses
JSON
format
HTTP/1.1 200 OK
Content-Type:application/json
{
"x-log-progress" : "Complete",
"x-log-count" : 5,
"logs" : "[{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}]"
}
Error codes
For a list of error codes, see Service error codes.
HTTP status code | Error code | Error message | Description |
---|---|---|---|
404 | ProjectNotExist | Project does not exist. | The error message returned because the specified project does not exist. |
404 | LogStoreNotExist | Logstore does not exist. | The error message returned because the specified Logstore does not exist. |
400 | InvalidTimeRange | Request time range is invalid. | The error message returned because the specified time range is invalid. |
400 | InvalidQueryString | Query string is invalid. | The error message returned because the specified query statement is invalid. |
400 | InvalidOffset | Offset is invalid. | The error message returned because the specified offset parameter is invalid. |
400 | InvalidLine | Line is invalid. | The error message returned because the specified line parameter is invalid. |
400 | InvalidReverse | Reverse value is invalid. | The error message returned because the specified reverse parameter is invalid. |
400 | IndexConfigNotExist | Logstore without index config. | The error message returned because the indexing feature is not enabled for the Logstore. |
400 | ParameterInvalid | ErrorType:OLSQueryParseError.ErrorMessage:offset is not available for pagination in sql query, please use limit x,y syntax for pagination. | The error message returned because the specified analytic statement in the query parameter of the request is invalid. If the query parameter is set to a query statement that contains an analytic statement, we recommend that you set the line and offset parameters to 0 and specify a LIMIT clause for pagination. For more information about how to troubleshoot the errors that are related to the SQL statement specified by the query parameter, see How do I resolve common errors that occur when I query and analyze log data? |
500 | InternalServerError | Specified Server Error Message. | The error message returned because an internal server error has occurred. |
For more information, see Common error codes.