Managed Service for OpenTelemetry:Report trace data from Kong Gateway by using OpenTelemetry

Method 1: Environment variables

Set the following environment variables:

KONG_TRACING_INSTRUMENTATIONS=all
KONG_TRACING_SAMPLING_RATE=1

Method 2: kong.conf file

Add the following settings to your kong.conf file:

# kong.conf
tracing_instrumentations = all
tracing_sampling_rate = 1.0

Method 1: kong.yaml configuration file

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Method 2: Kong Admin API

curl -X POST http://localhost:8001/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "opentelemetry",
      "config": {
        "traces_endpoint": "<HTTP_ENDPOINT>",
        "resource_attributes": {
          "service.name": "<SERVICE_NAME>"
        }
      }
    }
    '

curl -X POST http://localhost:8001/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "opentelemetry",
      "config": {
        "endpoint": "<HTTP_ENDPOINT>",
        "resource_attributes": {
          "service.name": "<SERVICE_NAME>"
        }
      }
    }
    '

Method 3: Kubernetes KongClusterPlugin resource

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-opentelemetry>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
config:
  traces_endpoint: <HTTP_ENDPOINT>
  resource_attributes:
    service.name: <SERVICE_NAME>
plugin: opentelemetry

Configure trace context propagation

By default, the Kong OpenTelemetry plug-in uses W3C as the trace propagation format. It extracts only W3C headers from incoming requests and injects W3C headers into outgoing requests.

To support additional formats, configure the propagation settings. The following example extracts trace context from request headers in this order: W3C, B3 (Zipkin), Jaeger, and OpenTracing. The preserve inject mode passes the extracted data as-is to downstream nodes. If no context is found, the plug-in generates a new trace in the W3C format.

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      # ...
      propagation:
        extract: [ w3c, b3, jaeger, ot ]
        inject: [ preserve ]
        default_format: "w3c"

Scope the plug-in to specific entities

By default, the OpenTelemetry plug-in applies globally. To scope it to a specific service, route, or consumer, add the corresponding field to the plug-in configuration.

Scope to a service:

Replace <SERVICE_NAME|ID> with the service name or ID.

# kong.yaml
plugins:
  - name: opentelemetry
    service: <SERVICE_NAME|ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Scope to a route:

Replace <ROUTE_NAME|ID> with the route name or ID.

# kong.yaml
plugins:
  - name: opentelemetry
    route: <ROUTE_NAME|ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Scope to a consumer:

Replace <CONSUMER_NAME|ID> with the consumer name or ID.

# kong.yaml
plugins:
  - name: opentelemetry
    consumer: <CONSUMER_NAME|ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Instrumentation types

Control the granularity of tracing by setting tracing_instrumentations (or the KONG_TRACING_INSTRUMENTATIONS environment variable) to one or more of the following values:

1. Clone the project

git clone https://github.com/Kong/docker-kong.git && cd docker-kong/compose

2. Create a backend service

Create a Node.js application that uses the Express framework, listens on port 7001, and exposes a /hello endpoint.

1. Create the application directory:

      mkdir backend && cd backend

2. Create the package.json file:

      {
        "name": "backend",
        "version": "1.0.0",
        "main": "index.js",
        "scripts": {},
        "keywords": [],
        "author": "",
        "license": "ISC",
        "description": "",
        "dependencies": {
          "@opentelemetry/api": "^1.9.0",
          "@opentelemetry/auto-instrumentations-node": "^0.52.0",
          "axios": "^1.7.7",
          "express": "^4.21.1"
        }
      }

3. Create the main.js file:

      "use strict";
      const axios = require("axios").default;
      const express = require("express");
      const app = express();
      app.get("/", async (req, res) => {
        const result = await axios.get("http://localhost:7001/hello");
        return res.status(201).send(result.data);
      });
      app.get("/hello", async (req, res) => {
        console.log("hello world!")
        res.json({ code: 200, msg: "success" });
      });
      app.use(express.json());
      app.listen(7001, () => {
        console.log("Listening on http://localhost:7001");
      });

4. Create the Dockerfile:

   Note Replace <HTTP_ENDPOINT> with the HTTP endpoint you obtained earlier.

      FROM node:20.16.0
      WORKDIR /app
      COPY package*.json ./
      RUN npm install
      COPY . .
      ENV OTEL_TRACES_EXPORTER="otlp"
      ENV OTEL_LOGS_EXPORTER=none
      ENV OTEL_METRICS_EXPORTER=none
      ENV OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="<HTTP_ENDPOINT>"
      ENV OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
      ENV OTEL_SERVICE_NAME="ot-nodejs-demo"
      ENV NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
      EXPOSE 7001
      CMD ["node", "main.js"]

3. Configure Docker Compose

Return to the compose directory:

cd ..

1. In docker-compose.yml, add environment variables to enable tracing for Kong Gateway:

      services:
        kong:
          # ...existing configuration...
          environment:
            # ...existing variables...
            KONG_TRACING_INSTRUMENTATIONS: all
            KONG_TRACING_SAMPLING_RATE: 1.0

2. In the same file, add the backend service definition:

      services:
        # ...existing services...
        backend-api:
          build:
            context: ./backend
            dockerfile: Dockerfile
          environment:
            NODE_ENV: production
          ports:
            - "7001:7001"
          networks:
            - kong-net

4. Configure Kong Gateway routing and the OpenTelemetry plug-in

1. Navigate to the config directory:

      cd config

2. Append the following to kong.yaml. Replace <HTTP_ENDPOINT> with your endpoint.

      services:
        - name: backend-api
          url: http://backend-api:7001
          routes:
            - name: main-route
              paths:
                - /hello
   
      plugins:
        - name: opentelemetry
          config:
            traces_endpoint: <HTTP_ENDPOINT>
            headers:
              X-Auth-Token: secret-token
            resource_attributes:
              service.name: kong-dev

5. Start the services

Return to the compose directory and start all services:

cd ..
docker compose up -d

6. Send a test request

curl http://localhost:8000/hello

Expected output:

{"code":200,"msg":"success"}

7. View traces in the console

Log on to the Managed Service for OpenTelemetry console to view traces from both Kong Gateway and the backend service.