Use the Lua plug-in to log request and response headers and bodies - Microservices Engine

Prerequisites

Before you begin, make sure that you have:

An MSE cloud-native gateway running version 1.2.11 or later

How it works

The Lua plug-in provides two entry-point functions that Envoy calls during request processing:

envoy_on_request(request_handle) -- Called when a request arrives at the gateway, before it is forwarded to the upstream service. Use this function to read or modify request headers, read the request body, or set dynamic metadata.
envoy_on_response(response_handle) -- Called when a response is received from the upstream service, before it is sent back to the client. Use this function to read or modify response headers, read the response body, or log aggregated request and response data.

Rule precedence

Plug-in rules are evaluated in the following order. A higher-precedence rule takes effect when multiple rules match the same request.

Route-level plug-in rules (highest precedence)
Domain-level plug-in rules
Instance-level plug-in rules (lowest precedence)

Configure plug-in rules

Log on to the MSE console and select a region in the top navigation bar.
In the left-side navigation pane, choose Cloud-native Gateway > Gateways. On the Gateways page, click the ID of the gateway.
In the left-side navigation pane of the gateway details page, click Plug-in Marketplace.
On the All Plug-ins tab, click the Custom sub-tab and then click the lua resource card.
On the Plug-in Configuration tab, choose a rule level and follow the corresponding steps.
Route-level rules
1. Click Add Rule.
2. Turn on the switch, select a target route, and enter your Lua script in the Configure Rule editor.
3. Click OK.
Domain-level rules
1. Click Add Rule.
2. Turn on the switch, select a target domain, and enter your Lua script in the Configure Rule editor.
3. Click OK.
Important
Wildcard domain names (those containing *) are not supported in Lua plug-in rules.
Instance-level rules

An instance-level rule is preconfigured for the Lua plug-in by default. To edit it, remove the edit lock first.
1. Turn on the switch and enter your Lua script in the Configure Rule editor.
2. Click Save.

API reference

For the complete stream handle API, see the Envoy Lua filter documentation.

JSON helpers

MSE provides built-in JSON serialization and deserialization:

-- Serialize a Lua table to a JSON string
local json_str = json.encode(obj)

-- Deserialize a JSON string to a Lua table
local obj = json.decode(json_str)

If an error occurs during serialization or deserialization, Lua calls the error function and terminates the current script execution.

Script examples

Log request and response details to plug-in logs

This script captures request and response headers and bodies, then writes them to the plug-in log with logInfo(). Bodies are only captured when the content-type is readable, and any body larger than 1024 bytes is truncated.

Both logging examples below use a shared check_content_readable helper that filters for three content types: application/x-www-form-urlencoded, application/json, and text/plain.

Step 1: Configure the Lua script

Add the following script as a route-level, domain-level, or instance-level rule:

local maxBodySize = 1024

-- Check whether the content-type indicates a human-readable body
function check_content_readable(type)
  if type == nil then
    return false
  end
  if string.find(type, "application/x-www-form-urlencoded",1,true) or string.find(type, "application/json",1,true) or string.find(type, "text/plain",1,true) then
     return true
  end
  return false
end

function envoy_on_request(request_handle)
  -- Iterate over all request headers and build a key=value string
  local headers = request_handle:headers()
  local headersStr = ""
  local contentType
  for key, value in pairs(headers) do
    if key == "content-type" then
       contentType = value
    end
    headersStr = headersStr  .. key .. "=" .. value .. ", "
  end
  -- Store request headers in dynamic metadata so envoy_on_response can access them
  request_handle:streamInfo():dynamicMetadata():set("envoy.lua","request_headers",headersStr)

  -- Read the request body if the content-type is readable
  local requestBody = ""
  if check_content_readable(contentType) then
    for chunk in request_handle:bodyChunks() do
      if (chunk:length() > 0) then
        requestBody = requestBody .. chunk:getBytes(0, chunk:length())
      end
      -- Truncate if the body exceeds maxBodySize
      if (#requestBody > maxBodySize) then
         requestBody = requestBody .. "<truncated>"
         break
      end
    end
  end
  -- Store request body in dynamic metadata, replacing newlines for single-line log output
  request_handle:streamInfo():dynamicMetadata():set("envoy.lua","request_body",string.gsub(requestBody,"\n","\\n"))
end

function envoy_on_response(response_handle)
  -- Iterate over all response headers
  local headers = response_handle:headers()
  local headersStr = ""
  local contentType
  local contentEncoding = false
  for key, value in pairs(headers) do
    if key == "content-type" then
       contentType = value
    elseif key == "content-encoding" then
       -- Skip body reading for compressed responses (gzip, br, etc.)
       contentEncoding = true
    end
    headersStr = headersStr .. key .. "=" .. value .. ", "
  end

  -- Read the response body if it is readable and not compressed
  local responseBody = ""
  if check_content_readable(contentType) and not contentEncoding then
    for chunk in response_handle:bodyChunks() do
      if (chunk:length() > 0) then
        responseBody = responseBody .. chunk:getBytes(0, chunk:length())
      end
      if (#responseBody > maxBodySize) then
         responseBody = responseBody .. "<truncated>"
         break
      end
    end
  end

  -- Retrieve the request headers and body stored during envoy_on_request
  local reqHeaders = ""
  local reqBody = ""
  local metadata = response_handle:streamInfo():dynamicMetadata():get("envoy.lua")
  if metadata ~= nil then
    local headers = response_handle:streamInfo():dynamicMetadata():get("envoy.lua")["request_headers"]
    if headers ~= nil then
      reqHeaders = headers
    end
    local body = response_handle:streamInfo():dynamicMetadata():get("envoy.lua")["request_body"]
    if body ~= nil then
      reqBody = body
    end
  end

  -- Write a single log line with all request and response details
  response_handle:logInfo("request Headers: [" .. reqHeaders .. "] request Body: [" .. string.gsub(reqBody,"\n","\\n") .. "] response Headers: [" .. headersStr .. "] response Body: [" .. string.gsub(responseBody,"\n","\\n") .. "]")
end

Step 2: Enable log shipping and view logs

This script writes to plug-in logs, which requires log shipping to be enabled for your gateway.

If log shipping is not enabled, click Immediately Enable Log Shipping in the console. Once enabled, plug-in logs are shipped to Simple Log Service.
To locate a specific request, find the request-id in the gateway access log and search for the same request-id in the Lua plug-in log. The plug-in log entry contains the complete request and response headers and bodies.

Use the request-id from the access log to find the corresponding plug-in log entry

Add request and response details to the access log

Instead of writing to a separate plug-in log, this script stores request and response data as Envoy dynamic metadata. Include this metadata in the gateway access log format for unified logging.

The script sets four metadata keys under the envoy.lua namespace:

Metadata key	Content
`envoy.lua:request_headers`	All request headers as `key=value` pairs
`envoy.lua:request_body`	Request body (truncated at 1024 bytes)
`envoy.lua:response_headers`	All response headers as `key=value` pairs
`envoy.lua:response_body`	Response body (truncated at 1024 bytes)

Step 1: Configure the Lua script

local maxBodySize = 1024

-- Check whether the content-type indicates a human-readable body
function check_content_readable(type)
  if type == nil then
    return false
  end
  if string.find(type, "application/x-www-form-urlencoded",1,true) or string.find(type, "application/json",1,true) or string.find(type, "text/plain",1,true) then
     return true
  end
  return false
end

function envoy_on_request(request_handle)
  -- Iterate over all request headers and build a key=value string
  local headers = request_handle:headers()
  local headersStr = ""
  local contentType
  for key, value in pairs(headers) do
    if key == "content-type" then
       contentType = value
    end
    headersStr = headersStr  .. key .. "=" .. value .. ", "
  end
  -- Store request headers as dynamic metadata for access log output
  request_handle:streamInfo():dynamicMetadata():set("envoy.lua","request_headers",headersStr)

  -- Read the request body if the content-type is readable
  local requestBody = ""
  if check_content_readable(contentType) then
    for chunk in request_handle:bodyChunks() do
      if (chunk:length() > 0) then
        requestBody = requestBody .. chunk:getBytes(0, chunk:length())
      end
      if (#requestBody > maxBodySize) then
         requestBody = requestBody .. "<truncated>"
         break
      end
    end
  end
  -- Store request body as dynamic metadata
  request_handle:streamInfo():dynamicMetadata():set("envoy.lua","request_body",string.gsub(requestBody,"\n","\\n"))
end

function envoy_on_response(response_handle)
  -- Iterate over all response headers
  local headers = response_handle:headers()
  local headersStr = ""
  local contentType
  local contentEncoding = false
  for key, value in pairs(headers) do
    if key == "content-type" then
       contentType = value
    elseif key == "content-encoding" then
       contentEncoding = true
    end
    headersStr = headersStr .. key .. "=" .. value .. ", "
  end
  -- Store response headers as dynamic metadata for access log output
  response_handle:streamInfo():dynamicMetadata():set("envoy.lua","response_headers",headersStr)

  -- Read the response body if it is readable and not compressed
  local responseBody = ""
  if check_content_readable(contentType) and not contentEncoding then
    for chunk in response_handle:bodyChunks() do
      if (chunk:length() > 0) then
        responseBody = responseBody .. chunk:getBytes(0, chunk:length())
      end
      if (#responseBody > maxBodySize) then
         responseBody = responseBody .. "<truncated>"
         break
      end
    end
  end
  -- Store response body as dynamic metadata for access log output
  response_handle:streamInfo():dynamicMetadata():set("envoy.lua","response_body",string.gsub(responseBody,"\n","\\n"))
end

Step 2: Add metadata fields to the access log format

In the left-side navigation pane of the gateway details page, click Parameter Settings.
Click Default Format (Manual Input) and add the metadata keys to the custom log fields. The metadata appears in subsequent access log entries.

Add custom metadata fields to the access log format in Parameter Settings

Disabled Lua libraries and functions

For security, the following Lua libraries and functions are disabled on MSE cloud-native gateways by default:

Library or function

debug.debug

debug.getfenv

debug.getregistry

dofile

io

loadfile

os.execute

os.getenv

os.remove

os.rename

os.tmpname

Envoy Lua filter stream handle API -- Complete reference for all available Lua API methods.

Prerequisites

How it works

Rule precedence

Configure plug-in rules

Route-level rules

Domain-level rules

Instance-level rules

API reference

JSON helpers

Script examples

Log request and response details to plug-in logs

Step 1: Configure the Lua script

Step 2: Enable log shipping and view logs

Add request and response details to the access log

Step 1: Configure the Lua script

Step 2: Add metadata fields to the access log format

Disabled Lua libraries and functions

Related resources