All Products
Search
Document Center

Simple Log Service:GetLogs

Last Updated:Feb 22, 2024

Queries logs that are generated within a specified time range from a specified Logstore of a specified project.

Usage notes

Note

Simple Log Service allows you to create a Scheduled SQL job. For more information, see Create a Scheduled SQL job.

  • Host consists of a project name and a Simple Log Service endpoint. You must specify a project in Host.

  • An AccessKey pair is created and obtained. For more information, see AccessKey pair.

    The AccessKey pair of an Alibaba Cloud account has permissions on all API operations. Using these credentials to perform operations in Simple Log Service is a high-risk operation. We recommend that you use a RAM user to call API operations or perform routine O&M. To create a RAM user, log on to the RAM console. Make sure that the RAM user has the management permissions on Simple Log Service resources. For more information, see Create a RAM user and authorize the RAM user to access Simple Log Service.

  • The information that is required to query logs is obtained. The information includes the name of the project to which the logs belong, the region of the project, and the name of the Logstore to which the logs belong. For more information, see Manage a project and Manage a Logstore.

  • Limits are imposed when you use Simple Log Service to query logs. We recommend that you specify query statements and query time ranges based on the limits. For more information, see Log search overview and Log analysis overview.

  • Indexes are configured before you query logs. For more information, see Create indexes.

  • If the number of logs in a Logstore significantly changes, Simple Log Service cannot predict the number of times that you must call this operation to obtain the complete results. In this case, you must check the value of the x-log-progress parameter in the response of each request and determine whether to call this operation one more time to obtain the complete results. Each time you call this operation, the same number of charge units (CUs) are consumed.

  • After a log is written to a Logstore, you can call the GetHistograms or the GetLogs operation to query the log. The latency of the query varies based on the type of the log. Simple Log Service classifies logs into the following types based on log timestamps:

    • Real-time data: The difference between the time record in a log of this type and the current time on Simple Log Service is within the interval (-180 seconds,900 seconds]. For example, if a log was generated at 12:03:00, September 25, 2014 (UTC) and Simple Log Service received the log at 12:05:00, September 25, 2014 (UTC), Simple Log Service processes the log as real-time data. This type of log is usually generated in common scenarios.

    • Historical data: The difference between the time record in a log of this type and the current time on Simple Log Service is within the interval [-604,800 seconds,-180 seconds). For example, if a log was generated at 12:00:00, September 25, 2014 (UTC) and Simple Log Service received the log at 12:05:00, September 25, 2014 (UTC), Simple Log Service processes the log as historical data. This type of log is usually generated in data backfill scenarios.

      After real-time data is written to a Logstore, the data can be queried with a maximum latency of 3 seconds. For 99.9% of queries, the latency is no more than 1 second.

Note

Simple Log Service calculates the difference between the log time that is specified by the __time__ field and the receiving time that is specified by the __tag__:__receive_time__ field for each log. The receiving time indicates when Simple Log Service receives the log. If the difference is within the interval (-180 seconds,900 seconds], Simple Log Service processes the log as real-time data. If the difference is within the interval [-604,800 seconds,-180 seconds), Simple Log Service processes the log as historical data.

Authentication resources

The following table describes the authorization information that is required for this operation. You can add the information to the Action element of a RAM policy statement to grant a RAM user or a RAM role the permissions to call this operation.

Action

Resource

log:GetLogStoreLogs

acs:log:{#regionId}:{#accountId}:project/{#ProjectName}/logstore/{#LogstoreName}

Debugging

OpenAPI Explorer automatically calculates the signature value. For your convenience, we recommend that you call this operation in OpenAPI Explorer. OpenAPI Explorer dynamically generates the sample code of the operation for different SDKs.

Request headers

This operation uses only common request headers. For more information, see Common request headers.

Request syntax

GET /logstores/{logstore}?type=log HTTP/1.1

Request parameters

Parameter

Type

Position

Required

Example

Description

project

String

Host

Yes

ali-test-project

The name of the project.

logstore

String

Path

Yes

example-logstore

The Logstore whose data you want to query.

from

Integer

Query

Yes

1627268185

The beginning of the time range to query. The value is the log time that is specified when log data is written.

  • The time range that is specified in this operation is a left-closed, right-open interval. The interval includes the start time specified by the from parameter, but does not include the end time specified by the to parameter. If you specify the same value for the from and to parameters, the interval is invalid, and an error message is returned.

  • The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

Note

To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis.

If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:

  • * | SELECT * FROM log WHERE from_unixtime(__time__) > from_unixtime(1664186624) AND from_unixtime(__time__) < now()

  • * | SELECT * FROM log WHERE __time__ > to_unixtime(date_parse('2022-10-19 15:46:05', '%Y-%m-%d %H:%i:%s')) AND __time__ < to_unixtime(now())

to

Integer

Query

Yes

1627269085

The end of the time range to query. The value is the log time that is specified when log data is written.

  • The time range that is specified in this operation is a left-closed, right-open interval. The interval includes the start time specified by the from parameter, but does not include the end time specified by the to parameter. If you specify the same value for the from and to parameters, the interval is invalid, and an error message is returned.

  • The value is a UNIX timestamp representing the number of seconds that have elapsed since January 1, 1970, 00:00:00 UTC.

Note

To ensure that full data can be queried, specify a query time range that is accurate to the minute. If you also specify a time range in an analytic statement, the time range is used for query and analysis.

If you want to specify a time range that is accurate to the second, you must use the from_unixtime or to_unixtime function in your analytic statement to convert the time format. For more information about the functions, see from_unixtime function and to_unixtime function. Examples:

  • * | SELECT * FROM log WHERE from_unixtime(__time__) > from_unixtime(1664186624) AND from_unixtime(__time__) < now()

  • * | SELECT * FROM log WHERE __time__ > to_unixtime(date_parse('2022-10-19 15:46:05', '%Y-%m-%d %H:%i:%s')) AND __time__ < to_unixtime(now())

query

String

Query

No

status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5

The search statement or the query statement. For more information, see Log search overview and Log analysis overview.

If you add set session parallel_sql=true; to the analytic statement in the query parameter, Dedicated SQL is used. For example, you can set the query parameter to * | set session parallel_sql=true; select count(*) as pv. For more information about common errors that may occur during log query and analysis, see How do I resolve common errors that may occur when I query and analyze logs?

Note

If you specify an analytic statement in the query parameter, the line and offset parameters do not take effect in this operation. In this case, we recommend that you set the line and offset parameters to 0 and use the LIMIT clause to limit the number of logs to return on each page. For more information, see Perform paged queries.

topic

String

Query

No

topic

The topic of the logs. The default value is an empty string. For more information, see Topic.

line

Long

Query

No

100

The maximum number of logs to return for the request. This parameter takes effect only when the query parameter is set to a search statement. Minimum value: 0. Maximum value: 100. Default value: 100. For more information, see Perform paged queries.

offset

Long

Query

No

0

The row from which the query starts. This parameter takes effect only when the query parameter is set to a search statement. Default value: 0. Default value: 0. For more information, see Perform paged queries.

reverse

Boolean

Query

No

false

Specifies whether to return logs in reverse chronological order of log timestamps. The log timestamps are accurate to the minute. Valid values:

  • true: Logs are returned in reverse chronological order of log timestamps.

  • false (default): Logs are returned in chronological order of log timestamps.

Important
  • The reverse parameter takes effect only when the query parameter is set to a search statement. The reverse parameter specifies the method used to sort returned logs.

  • If the query parameter is set to a query statement, the reverse parameter does not take effect. The method used to sort returned logs is specified by the ORDER BY clause in the analytic statement. If you use the keyword asc in the ORDER BY clause, the logs are sorted in chronological order. If you use the keyword desc in the ORDER BY clause, the logs are sorted in reverse chronological order. By default, asc is used in the ORDER BY clause.

powerSql

Boolean

Query

No

false

Specifies whether to use Dedicated SQL. For more information, see Enable Dedicated SQL. Valid values:

  • true: uses Dedicated SQL.

  • false (default): uses Standard SQL.

You can use the powerSql or query parameter to configure Dedicated SQL.

Response parameters

Parameter

Type

Example

Description

x-log-progress

String

Complete

The status of the query and analysis results that are returned. Valid values:

  • Complete: The query is successful, and the complete query and analysis results are returned.

  • Incomplete: The query is successful, but the returned query and analysis results are incomplete. To obtain the complete results, you must repeat the request.

Note

You can use the *|select count(*) as count statement to obtain the total number of logs. You can also call the GetHistogram operation to obtain the number of logs that are generated within each subinterval of a time range and add the numbers to obtain the total number of logs. If you do not need to obtain the total number of logs, you can modify the offset parameter in the request and repeat the request multiple times. If the value of the x-log-progress parameter is Complete and the number of returned rows is less than the number of requested rows, all logs are read.

x-log-count

Long

100

The number of rows that are returned.

Note

The total number of logs that meet the specified conditions is not returned. To query the total number of logs that meet the specified conditions, you must specify a query statement. For example, the query statement request_method:GET|select count(*) as count can be used to query the total number of logs whose request_method is GET. For more information, see Best practices for query and analysis and Examples for querying and analyzing logs by using SDK for Java.

x-log-processed-rows

Long

10000

The number of lines that are processed based on the request.

x-log-elapsed-millisecond

Long

5

The time that is consumed by the request. Unit: milliseconds.

Server

String

nginx

The name of the server.

Content-Type

String

application/json

The content type of the response body.

Content-Length

String

0

The content length of the response body.

Connection

String

close

Indicates whether the connection is persistent. Valid values:

  • close: The connection is non-persistent. A new TCP connection is established for each HTTP request.

  • keep-alive: The connection is persistent. After a TCP connection is established, the connection remains open, and no more time or bandwidth is consumed to establish new connections.

Date

String

Sun, 27 May 2018 08:25:04 GMT

The time when the response was returned.

x-log-requestid

String

5B0A6B60BB6EE39764D458B5

The unique ID of the request. The ID is generated by the server.

Array of Object

[{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}]

An array of logs. Each element in the array indicates a log.

Examples

Sample requests

GET /logstores/{logstore}?type=log&from=1627268185&to=1627269085&query=status: 401 | SELECT remote_addr,COUNT(*) as pv GROUP by remote_addr ORDER by pv desc limit 5&topic=topic&line=100&offset=0&reverse=false&powerSql=false HTTP/1.1
Host:ali-test-project.cn-hangzhou.log.aliyuncs.com
Content-Type:application/json

Description of the sample requests

### Sample raw log
```json
    {
    	"remote_addr": "203.0.*.*",
    	"__time__": 1649904328,
    	"__topic__": "nginx_access_log",
    	"__source__": "127.0.0.1",
    	"body_bytes_sent": "3841",
    	"__tag__:__receive_time__": "1649904334",
    	"time_local": "14/Apr/2022:02:45:28",
    	"request_method": "HEAD",
    	"request_uri": "/request/path-2/file-2",
    	"http_user_agent": "Mozilla/5.0 (Windows NT 6.2; Win64; x64; rv:21.0.0) Gecko/20121011 Firefox/21.0.0",
    	"remote_user": "jlqd8",
    	"request_time": "17",
    	"__pack_meta__": "1|MTYyODY2Mjk0NzQ1ODU1Mjk4Mg==|197|165",
    	"request_length": "4184",
    	"http_referer": "www.example.com",
    	"__tag__:__receive_time___0": "1649904334",
    	"host": "www.example.com",
    	"http_x_forwarded_for": "198.51.100.48",
    	"upstream_response_time": "0.23",
    	"status": "401"
    }
```

Sample success responses

JSON format

HTTP/1.1 200 OK
Content-Type:application/json

{
  "x-log-progress" : "Complete",
  "x-log-count" : 5,
  "logs" : "[{'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.XXX.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}, {'remote_addr': '198.51.100.XXX', 'pv': '1', '__source__': '', '__time__': '1649902984'}]"
}

Error codes

For a list of error codes, see Service error codes.

HTTP status code

Error code

Error message

Description

404

ProjectNotExist

Project does not exist.

The specified project does not exist.

404

LogStoreNotExist

Logstore does not exist.

The specified Logstore does not exist.

400

InvalidTimeRange

Request time range is invalid.

The specified time range is invalid.

400

InvalidQueryString

Query string is invalid.

The specified query statement is invalid.

400

InvalidOffset

Offset is invalid.

The specified offset parameter is invalid.

400

InvalidLine

Line is invalid.

The specified line parameter is invalid.

400

InvalidReverse

Reverse value is invalid.

The specified reverse parameter is invalid.

400

IndexConfigNotExist

Logstore without index config.

The indexing feature is not enabled for the Logstore.

400

ParameterInvalid

ErrorType:OLSQueryParseError.ErrorMessage:offset is not available for pagination in sql query, please use limit x,y syntax for pagination.

The specified analytic statement in the query parameter of the request is invalid. If the query parameter is set to a query statement that contains an analytic statement, we recommend that you set the line and offset parameters to 0 and specify a LIMIT clause for pagination.

For more information about how to troubleshoot the errors that are related to the SQL statement specified by the query parameter, see How do I resolve common errors that may occur when I query and analyze logs?

500

InternalServerError

Specified Server Error Message.

An internal server error has occurred.

For more information, see Common error codes.