Configure custom log fields for request headers and query parameters - Object Storage Service

OSS access logs include a standard set of fields. To track additional request details, configure custom log fields to record specific request headers and query parameters. Custom log fields are stored in the user_defined_log_fields field.

Background information

HTTP requests are defined by their headers and query parameters. By default, OSS access logs capture only a subset of these. For more information about OSS access logs, see Access logs.

You can configure OSS to record additional request headers and query parameters as defined in RFC 2616.

Scenarios

Proxy analysis: OSS logs do not record the x-forwarded-for header by default. Configure a custom log field to capture it for intermediary analysis.
Query parameter categorization: OSS logs do not record URL query parameters by default. To log specific parameters such as rrr and eee from a URL like http://bucketname.aliyuncs.com/hu.jpg?yyy=45&eee=88&rrr=huyanbin, configure custom log fields.

Usage notes

You can specify up to six request headers and query parameters, one per line.
Logged data is stored as Base64-encoded JSON in the user_defined_log_fields field. The JSON contains three keys: "truncated" (whether data was truncated), "headers" (request headers), and "querys" (query parameters).
Request header keys allow hyphens (-) but not underscores (_). Query parameter keys allow underscores (_).
Request header keys must contain only ASCII printable characters (33–126), excluding underscores (_) and colons (:).
The combined key-value size of all custom fields cannot exceed 1,024 bytes. Data beyond this limit is truncated. Keys are converted to lowercase in logs; values retain their original case.

Prerequisites

Real-time log query is enabled for the bucket. For more information, see Real-time log query.
Requests sent to the bucket include the target request headers or query parameters. Initiate a request.

Methods

Use the OSS console

Log on to the OSS console.
In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.
In the left-side navigation tree, choose Logging > Real-time Log Query.
On the Real-time Log Query tab, click Add next to Request Headers or Query Parameters in Logs.
In the Configure Request Header or Query Parameter in Logs dialog box, click Add next to Specify Log Field.
Select Request Header or Query Parameter, enter a value, and click OK.

Note
Custom field settings take effect within 15 minutes.

Use ossutil

Use ossutil to configure request headers in logs. For information about its installation, see Install ossutil.

Run the following command to add custom request headers or query parameters to the user_defined_log_fields field for examplebucket:

ossutil api put-user-defined-log-fields-config --bucket examplebucket --user-defined-log-fields-configuration "{\"HeaderSet\":{\"header\":[\"header1\",\"header2\",\"header3\"]},\"ParamSet\":{\"parameter\":[\"param1\",\"param2\"]}}"

Related API operation

These methods are built on a RESTful API that you can call directly. Direct API calls require signature calculation in your code. For more information, see PutUserDefinedLogFieldsConfig.

Reference

Analyze intermediaries based on custom log fields