Customize access log headers for an ASM ingress gateway - Alibaba Cloud Service Mesh

When debugging traffic issues or correlating requests across microservices, the default Envoy access log format may not capture the headers you need. For example, user IDs, correlation tokens, or custom trace headers are not logged by default. To add these fields to your ingress gateway access logs, create an EnvoyFilter that defines a custom log_format.

Note EnvoyFilter configurations are tightly coupled to Envoy internals. Incorrect configurations can destabilize your mesh. Re-validate your EnvoyFilter after every ASM or Istio proxy version upgrade to make sure deprecated fields are updated.

Prerequisites

Before you begin, make sure that you have:

An ASM instance (version 1.8 or later recommended). For ASM versions earlier than 1.8, see Envoy v2 API configuration
kubectl installed and connected to a serverless Kubernetes cluster. For more information, see Select a type of cluster credentials

The ingress gateway pod name stored in an environment variable:

  export GATEWAY_POD=$(kubectl get pod -n istio-system -l app=istio-ingressgateway -o jsonpath='{.items[0].metadata.name}')

How access log format operators work

EnvoyFilter configurations use command operators to extract request and response metadata into log fields:

Operator pattern	Description	Example
`%REQ(<header>)%`	Value of the specified request header	`%REQ(USER-AGENT)%` returns the User-Agent string
`%RESP(<header>)%`	Value of the specified response header	`%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%` returns upstream latency
`%<field>%`	Built-in Envoy field	`%DURATION%` returns total request duration in milliseconds

If a header is not present in the request, the log records "-" for that field.

For a full list of available operators, see the Envoy access log command operators documentation.

Step 1: Create the EnvoyFilter

The following EnvoyFilter adds a user_id field to the ingress gateway access logs in JSON format by extracting the value from the USER_ID request header.

For more information about managing Envoy filters, see Manage Envoy filters.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: gateway-accesslog-userdefine
  namespace: istio-system
spec:
  configPatches:
    - applyTo: NETWORK_FILTER
      match:
        context: ANY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: MERGE
        value:
          typed_config:
            '@type': >-
              type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
            access_log:
              - name: envoy.access_loggers.file
                typed_config:
                  '@type': >-
                    type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                  log_format:
                    json_format:
                      authority: '%REQ(:AUTHORITY)%'
                      bytes_received: '%BYTES_RECEIVED%'
                      bytes_sent: '%BYTES_SENT%'
                      downstream_local_address: '%DOWNSTREAM_LOCAL_ADDRESS%'
                      downstream_remote_address: '%DOWNSTREAM_REMOTE_ADDRESS%'
                      duration: '%DURATION%'
                      method: '%REQ(:METHOD)%'
                      path: '%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%'
                      protocol: '%PROTOCOL%'
                      request_id: '%REQ(X-REQUEST-ID)%'
                      requested_server_name: '%REQUESTED_SERVER_NAME%'
                      response_code: '%RESPONSE_CODE%'
                      response_flags: '%RESPONSE_FLAGS%'
                      route_name: '%ROUTE_NAME%'
                      start_time: '%START_TIME%'
                      upstream_cluster: '%UPSTREAM_CLUSTER%'
                      upstream_host: '%UPSTREAM_HOST%'
                      upstream_local_address: '%UPSTREAM_LOCAL_ADDRESS%'
                      upstream_service_time: '%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%'
                      upstream_transport_failure_reason: '%UPSTREAM_TRANSPORT_FAILURE_REASON%'
                      user_agent: '%REQ(USER-AGENT)%'
                      user_id: '%REQ(USER_ID)%'
                      x_forwarded_for: '%REQ(X-FORWARDED-FOR)%'
                  path: /dev/stdout
  workloadSelector:
    labels:
      app: istio-ingressgateway

Configuration parameters

Parameter	Description
`namespace`	Namespace where the EnvoyFilter takes effect. Set this to the namespace of your ingress gateway, typically `istio-system`.
`workloadSelector`	Targets a specific ingress gateway. Format: `app: istio-<ingress gateway name>`. This example targets the gateway named `ingressgateway`.
`log_format`	Defines the structure and fields of the access log output. If omitted, Envoy uses its default log format.
`user_id: '%REQ(USER_ID)%'`	Custom field added in this example. Extracts the `USER_ID` header from each incoming request.

Add other custom headers

To log additional custom headers, add entries to the json_format block using the %REQ(<header>)% operator:

my_custom_header: '%REQ(MY_CUSTOM_HEADER)%'

When a request includes the MY_CUSTOM_HEADER header, the log entry contains its value:

{"protocol": "HTTP/1.1", "duration": "123", "my_custom_header": "value_of_MY_CUSTOM_HEADER"}

Step 2: Verify the configuration

After applying the EnvoyFilter, confirm that the custom header appears in both the gateway's active configuration and its log output.

Check the Envoy config dump

Run the following command to verify that the user_id field is present in the gateway's active configuration:

kubectl exec $GATEWAY_POD -n istio-system -- curl -s localhost:15000/config_dump | grep -5 user_id

The output should show the user_id field inside the log_format section of the access log configuration.

Send a test request

Send a request to the ingress gateway with a custom user_id header:

curl -H 'user_id: 8888' <gateway-ip>/some_path

Replace <gateway-ip> with the external IP address of your ingress gateway.

Check the gateway logs

View the most recent ingress gateway logs:

kubectl -n istio-system logs -f $GATEWAY_POD --tail=10

Look for the user_id field in the JSON output. A correctly applied configuration produces a log entry similar to:

{"user_id": "8888", "protocol": "HTTP/1.1", "duration": "5", "method": "GET", ...}

Cleanup

To remove the custom access log configuration and revert to the default Envoy log format, delete the EnvoyFilter:

kubectl delete envoyfilter gateway-accesslog-userdefine -n istio-system

Envoy v2 API for ASM versions earlier than 1.8

For ASM instances on versions earlier than 1.8, use the Envoy v2 API in the typed_config parameter. The following example shows the equivalent EnvoyFilter configuration:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: enable-accesslog
  namespace: gateway-accesslog-userdefine
spec:
  configPatches:
  - applyTo: NETWORK_FILTER
    match:
      context: ANY
      listener:
        filterChain:
          filter:
            name: envoy.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          "@type": "type.googleapis.com/envoy.config.filter.network.http_connection_manager.v2.HttpConnectionManager"
          access_log:
          - name: envoy.file_access_log
            config:
              path: /dev/stdout
              log_format:
                 ....
  workloadSelector:
    labels:
      app: istio-ingressgateway

Differences from the v3 API:

Parameter	v3 API	v2 API
HTTP connection manager type	`...http_connection_manager.v3.HttpConnectionManager`	`...http_connection_manager.v2.HttpConnectionManager`
Access logger name	`envoy.access_loggers.file`	`envoy.file_access_log`
Access log sub-configuration key	`typed_config`	`config`