Query and analyze real-time OSS access logs - Object Storage Service

Prerequisites

Real-time log query is enabled for the bucket.
Simple Log Service is authorized to access OSS. If not, grant authorization on the Cloud Resource Access Authorization page.

Features

Pushes logs to Simple Log Service within 3 minutes for real-time viewing in the OSS console.
Provides built-in log analysis and common reports.
Queries and analyzes raw logs in real time. Supports filtering by bucket, object name, API operation, or time range.

Billing rules

If your Logstore uses the pay-by-feature billing mode, Simple Log Service does not charge for log storage or index traffic if the storage duration of OSS logs is 7 days or less and the daily write traffic (after compression) or index traffic does not exceed 900 GB. For reference, if a single access log is 1 KB, 900 GB is equivalent to approximately 900 million logs. If these limits are exceeded, Simple Log Service charges for the excess log storage and index traffic. For more information, see Billable items of the pay-by-feature mode.
Reading, processing, and delivering log data from the dedicated Logstore incurs charges billed through Simple Log Service.
If your Logstore uses the pay-by-feature billing mode, the real-time log query feature provides a monthly free quota of 16 × 31 shard-days. Simple Log Service charges for usage that exceeds this quota. For more information, see Billable items of the pay-by-feature mode.

Procedure

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
On the Real-time Log Query tab, enter a query statement and an analysis statement based on the query syntax and analysis syntax to analyze log fields.

For example, analyze the operation field to count GetObject operations on bucket examplebucket over the past seven days:

Sample statement:
```
* and __topic__: oss_access_log AND bucket:examplebucket AND operation:GetObject | select
  count(*) AS GetObjectCount
```
Sample results:

For more information, see Common examples of real-time OSS log queries.

Related operations

Enable real-time log query

You can enable real-time log query when you create a bucket or for an existing bucket.

Enable real-time log query when you create a bucket

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, click Create Bucket.
In the Create Bucket panel, turn on Real-time Log Query and configure other parameters.

For more information about how to configure other parameters, see Create buckets.
Click Create. In the pop-up confirmation box, click Confirm.

Enable real-time log query after you create a bucket

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click Enable Now.

Modify the retention period of logs

After you enable real-time log query, OSS provides free log queries for the last seven days. Click Set Log Retention Period in the upper-right corner to change the retention period.

Reset the index fields of logs

Log fields may change over time, but these changes are not automatically reflected on the real-time log query page. Manually refresh indexed fields by using one of the following methods.

Important

Resetting restores the built-in index and invalidates custom indexed fields. If custom fields are critical to your business, manually re-add them after the reset. For more information, see Create indexes.

Use the OSS console

Disable real-time log query, and then enable it.

Use the Simple Log Service console

Log on to the Log Service console.
In the Log Application section, click the Cloud Service Lens tab. Then, click CloudLens for OSS.
On the Data Import Management page, click the Destination Logstore tab. Find the Logstore that stores the access logs and click Reset in the Index Reset column.
In the message that appears, click OK.

Note
Index resetting takes approximately 1 minute to take effect.

Use LogReduce

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click the LogReduce tab. If you have not enabled LogReduce, click Enable LogReduce.
In the message that appears, click OK.

For more information about LogReduce, see LogReduce.

Disable real-time log query

To disable real-time log query:

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
Click Disable Real-time Log Query in the upper-right corner.

Important
Enabling real-time log query automatically creates Simple Log Service projects, but disabling the feature does not delete them. To avoid unexpected charges, manually delete these projects in the Simple Log Service console. Delete a project.