Object Storage Service:Analyze intermediaries based on custom log fields

Prerequisites

Before you begin, ensure that you have:

• Real-time log query enabled for the bucket. See Real-time log query.

• Requests sent to the bucket that include the headers or query parameters you want to log. See x-oss-additional-headers.

Step 1: Configure the custom log field

1. Log on to the OSS console.OSS console

2. In the left-side navigation pane, click Buckets. Find and click the target bucket.

3. In the left-side navigation tree, choose Logging > Real-time Log Query.

4. On the Real-time Log Query tab, click Add next to Request Headers or Query Parameters in Logs.

5. In the Configure Request Header or Query Parameter in Logs dialog box, click Add next to Specify Log Field.

6. Select Request Header, enter x-forwarded-for, and click OK.

Step 2: Send a request through a proxy

Requests routed through a proxy — such as Alibaba Cloud CDN — include the X-Forwarded-For header automatically. The following example uses OSS SDK for Python to simulate a proxied request:

# -*- coding: utf-8 -*-
import oss2
from oss2.credentials import EnvironmentVariableCredentialsProvider

# Prepare resources.
bucketName = "examplebucket"
endpoint = "oss-cn-hangzhou.aliyuncs.com"
# Obtain access credentials from environment variables. Before you run the sample code, make sure that the OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET environment variables are configured.
auth = oss2.ProviderAuth(EnvironmentVariableCredentialsProvider())
# Initialize the bucket instance.
bucket = oss2.Bucket(auth, endpoint, bucketName)
# Simulate a request by using a proxy.
headers = dict()
headers["x-forwarded-for"] = "10.0.0.1"
result = bucket.put_object('exampleobject.txt', 'Hello OSS', headers)
# Show the HTTP status code.
print('http status: {0}'.format(result.status))
# Show the request ID. A request ID uniquely identifies the request. Add this parameter to your logs.
print('request_id: {0}'.format(result.request_id))

Replace the placeholder values:

Step 3: Query logs to identify intermediaries

The user_defined_log_fields field is not indexed by default because it was added after the Real-time Log Query feature. If your queries return no results or indexing errors, manually refresh the field index first. See Reset the field index.

OSS stores the captured header values in user_defined_log_fields as a Base64-encoded JSON string. For example, a request with x-forwarded-for: 10.0.0.1 produces a raw field value similar to:

eyJoZWFkZXJzIjp7IngtZm9yd2FyZGVkLWZvciI6IjEwLjAuMC4xIn19

Decoded, this becomes:

{"headers":{"x-forwarded-for":"10.0.0.1"}}

Both queries below use from_utf8(from_base64(...)) to decode the field before extracting the header value.

On the Real-time Log Query tab, enter a query and click Search & Analyze.

Extract the X-Forwarded-For value for all PutObject requests

* and bucket: examplebucket and operation:PutObject | select request_id,
json_format(json_extract(from_utf8(from_base64(user_defined_log_fields)),
'$.headers["x-forwarded-for"]')) as myheader

This returns each request_id alongside the decoded x-forwarded-for value (myheader) from every PutObject request in the bucket.

Filter requests from a specific proxy IP

To narrow results to a single proxy — for example, the IP address 10.0.0.2:

* and bucket: examplebucket and operation:PutObject | select request_id,
json_format(json_extract(from_utf8(from_base64(user_defined_log_fields)),
'$.headers["x-forwarded-for"]')) as myheader having myheader like '%10.0.0.2%'

The having myheader like '%10.0.0.2%' clause filters the result set to rows where the decoded header value contains the target IP.

FAQ