All Products
Search
Document Center

Object Storage Service:Real-time OSS log query

Last Updated:Feb 09, 2024

A large number of logs are generated when Object Storage Service (OSS) resources are accessed. OSS uses Simple Log Service to help you query and collect statistics for OSS access logs and audit access to OSS in the OSS console, track exception events, and troubleshoot problems. This helps you improve work efficiency and make informed decisions.

Prerequisites

  • Real-time log query is enabled for an OSS bucket. For more information, see Real-time log query.

  • Simple Log Service is authorized to access OSS. If you have not authorized Simple Log Service to access OSS, follow the instructions on the Cloud Resource Access Authorization page to complete the authorization.

Features

  • Pushes logs to Simple Log Service within 3 minutes and allows you to view real-time logs in the OSS console.

  • Provides log analysis and common analysis reports for easy data queries.

  • Allows you to query and analyze raw logs in real time and filter logs by bucket, object name, API operation, or time.

Billing rules

  • If your Logstore uses the pay-by-feature billing mode, you are not charged log storage or index traffic fees if OSS access logs are stored for no more than 7 days and the compressed write traffic or index traffic per day is no more than 900 GB. If the size of an OSS access log is 1 KB, 900 GB is equivalent to 900 million logs. If the limits are exceeded, you are charged log storage and index traffic fees for the excess logs. The fees are included in the bills of Simple Log Service. For more information, see Billable items of pay-by-feature.

  • You are charged when you read data from a dedicated Logstore over the Internet. You are also charged when you transform and ship data.

  • If your Logstore uses the pay-by-feature billing mode, the real-time log query feature provides a free quota for shards per month. The free quota is calculated by using the following formula: 16 × 31. The unit of the free quota is shard-day. After the free quota is exhausted, you are charged fees, and the fees are included in the bills of Simple Log Service. For more information, see Billable items of pay-by-feature.

Procedure

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Logging > Real-time Log Query.

  4. On the Real-time Log Query tab, enter a query statement and an analysis statement based on the query syntax and analysis syntax to analyze the OSS logs. For more information, see Query syntax, Analysis syntax and Log fields.

    For example, you can analyze the number of the GetObject operations on the bucket named examplebucket in the past seven days based on the operation field of OSS access logs.

    Sample query statement and analysis statement:

    * and __topic__: oss_access_log AND bucket:examplebucket AND operation:GetObject | select
      count(*) AS GetObjectCount

    Sample query and analysis results:

    getobjectcount.png

    For more information, see Examples of real-time log query.

Related operations

Enable real-time log query

You can enable real-time log query when you create a bucket or after you create a bucket.

Enable real-time log query when you create a bucket

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, click Create Bucket.

  3. In the Real-time Log Query section of the Create Bucket panel, turn on Real-time Log Query and configure other parameters.

    For more information about how to configure other parameters, see Create buckets.

  4. Click OK.

Enable real-time log query after you create a bucket

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Logging > Real-time Log Query.

  4. Click Activate Now.

Modify the retention period of logs

After you enable real-time log query, OSS allows you to log queries of the last seven days free of charge. You can click Set Log Retention Period in the upper-right corner to modify the retention period of logs.

Reset the index fields of logs

When you query real-time logs, fields may change. For example, a field is added or a field is changed to an index. However, the field on the real-time log query page is not automatically synchronized. You must manually refresh the real-time log index field by using one of the following methods.

Important

The following methods restore the index to the latest version of the built-in index, and custom indexes are lost. Check whether the loss of custom indexes affects your business scenarios. If the loss does not affect your business scenarios, perform the following procedure. If the loss affects your business scenarios, manually add the missing indexes. For more information, see Create indexes.

Use the OSS console

Disable real-time log query, and then enable it.

Use the Simple Log Service console

  1. Log on to the Log Service console.
  2. In the Log Application section, click the Cloud Service Lens tab. Then, click CloudLens for OSS.

  3. On the Destination Logstore tab of the Data Import Management page, find the Logstore that you want to manage and click Reset in the Index Reset column.

  4. In the message that appears, click OK.

    Note

    The change takes effect in 1 minute.

Use LogReduce

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Logging > Real-time Log Query.

  4. Click the LogReduce tab. If you have not enabled LogReduce, click Enable LogReduce.

  5. In the message that appears, click OK.

    For more information about LogReduce, see LogReduce.

Disable real-time log query

If you no longer need to retain log data, perform the following steps to disable real-time log query:

  1. Log on to the OSS console.

  2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

  3. In the left-side navigation tree, choose Logging > Real-time Log Query.

  4. Click Disable Real-time Log Query in the upper-right corner.

    Important

    When you enable real-time log query, Simple Log Service projects are automatically created. However, the projects are not automatically deleted when you disable real-time log query. Therefore, after real-time log query is disabled, you must log on to the Simple Log Service console to delete the projects that are automatically created to avoid unexpected charges. For more information, see Delete a project.