All Products
Search

This Product

    Object Storage Service:Log request headers and query parameters

    Document Center

    Object Storage Service:Log request headers and query parameters

    Last Updated:Feb 23, 2025

    Object Storage Service (OSS) logs a standard set of fields in access logs for basic analytical purposes. In some scenarios, you may want to log custom fields to implement fine-grained user behavior tracking and analysis or tailor information monitoring. In this case, you can configure custom log fields to record request headers and query parameters that you want to monitor. Custom log fields are aggregated in the user_defined_log_fields field.

    Background information

    An HTTP request is largely characterized by its request headers and query parameters. OSS access logs record only a portion of the request characteristics by default. For more information about OSS access logs, see Access logs.

    To monitor data of concern to you, you can configure OSS to additionally record request headers and query parameters. For more information about request headers and query parameters, see RFC 2616.

    Scenarios

    • Proxy analysis: Standard OSS logs do not record the x-forwarded-for header. In cases where you want to monitor the header for intermediary analysis, you can configure a custom log field to log the header.

    • Request categorization based on query parameters: Standard OSS logs do not record query parameters in a request URL. If you want to log query parameters, such as rrr and eee in the http://bucketname.aliyuncs.com/hu.jpg?yyy=45&eee=88&rrr=huyanbin URL, you can configure custom log fields to log the parameters.

    Usage notes

    • Each line holds only one request header or query parameter. You can specify a total of six request headers and query parameters.

    • Your logging configuration information is encoded and populated in the user_defined_log_fields field. The value of the user_defined_log_fields field is Base64-encoded JSON data. The JSON data includes the "truncated" field that indicates whether the JSON data was truncated, the "headers" field that contains the specified request headers, and the "querys" field that contains the specified query parameters.

    • Custom field keys for request headers can contain hyphens (-) but cannot contain underscores (_). Custom field keys for query parameters can contain underscores (_).

    • Custom field keys for request headers must comply with the HTTP protocol and can contain only ASCII printable characters (from 33 to 126) except for underscores (_) and colons (:).

    • The total key and value length of all custom fields for request headers and query parameters cannot exceed the limit of 1,024 bytes. Trailing data beyond the limit is truncated. The keys of custom log fields are converted to lower case in logs. No case conversion is applied to the values of custom log fields. Log field values represent information sent in the requests and are truncated if the length limit is exceeded.

    Prerequisites

    • The real-time log query feature is enabled for the bucket. For more information, see Real-time log query.

    • The requests sent to the bucket contain the related request headers or query parameters. For more information, see Initiate a request.

    Methods

    Use the OSS console

    1. Log on to the OSS console.

    2. In the left-side navigation pane, click Buckets. On the Buckets page, find and click the desired bucket.

    3. In the left-side navigation tree, choose Logging > Real-time Log Query.

    4. On the Real-time Log Query tab, click Add next to Request Headers or Query Parameters in Logs.

    5. In the Configure Request Header or Query Parameter in Logs dialog box, click Add next to Specify Log Field.

    6. Select Request Header or Query Parameter, enter a value, and click OK.

      Note

      Your custom field settings take effect in 15 minutes.

    Use ossutil

    You can use ossutil to configure request headers in logs. For information about its installation, see Install ossutil.

    Run the following command to add custom request headers or query parameters to dynamically modify the user_defined_log_fields field, enabling detailed analysis of requests related to examplebucket.

    ossutil api put-user-defined-log-fields-config --bucket examplebucket --user-defined-log-fields-configuration "{\"HeaderSet\":{\"header\":[\"header1\",\"header2\",\"header3\"]},\"ParamSet\":{\"parameter\":[\"param1\",\"param2\"]}}"

    Related API operation

    The methods described above are fundamentally implemented based on the RESTful API, which you can directly call if your business requires a high level of customization. To directly call an API, you must include the signature calculation in your code. For more information, see PutUserDefinedLogFieldsConfig.

    Reference

    Analyze intermediaries based on custom log fields