Application Real-Time Monitoring Service:Report trace data from Kong Gateway by using OpenTelemetry

Option A: Environment variables

KONG_TRACING_INSTRUMENTATIONS=all
KONG_TRACING_SAMPLING_RATE=1.0

Option B: kong.conf

# kong.conf
tracing_instrumentations = all
tracing_sampling_rate = 1.0

Method 1: Declarative configuration (kong.yaml)

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Method 2: Kong Admin API

curl -X POST http://localhost:8001/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "opentelemetry",
      "config": {
        "traces_endpoint": "<HTTP_ENDPOINT>",
        "resource_attributes": {
          "service.name": "<SERVICE_NAME>"
        }
      }
    }'

curl -X POST http://localhost:8001/plugins/ \
    --header "accept: application/json" \
    --header "Content-Type: application/json" \
    --data '
    {
      "name": "opentelemetry",
      "config": {
        "endpoint": "<HTTP_ENDPOINT>",
        "resource_attributes": {
          "service.name": "<SERVICE_NAME>"
        }
      }
    }'

Method 3: Kubernetes (KongClusterPlugin)

apiVersion: configuration.konghq.com/v1
kind: KongClusterPlugin
metadata:
  name: <global-opentelemetry>
  annotations:
    kubernetes.io/ingress.class: kong
  labels:
    global: "true"
config:
  traces_endpoint: <HTTP_ENDPOINT>
  resource_attributes:
    service.name: <SERVICE_NAME>
plugin: opentelemetry

Context propagation format

The Kong OpenTelemetry plug-in defaults to W3C for context propagation. It extracts W3C headers from upstream requests and injects them into downstream requests.

To support additional formats, configure the propagation block. The following example extracts context in order of W3C, b3 (Zipkin), Jaeger, and OpenTracing, then preserves the extracted format when injecting downstream. If no context is found, the plug-in generates a new W3C trace.

# kong.yaml
plugins:
  - name: opentelemetry
    config:
      # ...
      propagation:
        extract: [ w3c, b3, jaeger, ot ]
        inject: [ preserve ]
        default_format: "w3c"

Plug-in scope

The OpenTelemetry plug-in applies globally by default. Narrow the scope to a specific service, route, or consumer by adding the corresponding field:

Service scope

# kong.yaml
plugins:
  - name: opentelemetry
    service: <SERVICE_NAME_OR_ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Route scope

# kong.yaml
plugins:
  - name: opentelemetry
    route: <ROUTE_NAME_OR_ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Consumer scope

# kong.yaml
plugins:
  - name: opentelemetry
    consumer: <CONSUMER_NAME_OR_ID>
    config:
      traces_endpoint: <HTTP_ENDPOINT>
      resource_attributes:
        service.name: <SERVICE_NAME>

Trace types

Control which types of traces Kong Gateway collects by setting tracing_instrumentations or KONG_TRACING_INSTRUMENTATIONS:

Before you begin

• Git, Docker, and Docker Compose are installed. Docker version 20.10.0 or later is required.

• A Managed Service for OpenTelemetry HTTP endpoint is available (see Prerequisites).

Project structure

docker-kong/compose/
├── docker-compose.yml          # Docker Compose configuration
├── config/
│   └── kong.yaml               # Kong Gateway declarative configuration
└── backend/
    ├── Dockerfile              # Backend service image
    ├── main.js                 # Express application
    ├── package.json
    └── package-lock.json

1. Clone the Kong Docker repository

git clone https://github.com/Kong/docker-kong.git && cd docker-kong/compose

2. Create the backend service

Build a Node.js Express application that listens on port 7001 and exposes a /hello endpoint.

1. Create the application directory:

      mkdir backend && cd backend

2. Create package.json:

      cat << EOF > package.json
      {
        "name": "backend",
        "version": "1.0.0",
        "main": "index.js",
        "scripts": {},
        "keywords": [],
        "author": "",
        "license": "ISC",
        "description": "",
        "dependencies": {
          "@opentelemetry/api": "^1.9.0",
          "@opentelemetry/auto-instrumentations-node": "^0.52.0",
          "axios": "^1.7.7",
          "express": "^4.21.1"
        }
      }
      EOF

3. Create main.js:

      cat << EOF > main.js
      "use strict";
      const axios = require("axios").default;
      const express = require("express");
      const app = express();
      app.get("/", async (req, res) => {
        const result = await axios.get("http://localhost:7001/hello");
        return res.status(201).send(result.data);
      });
      app.get("/hello", async (req, res) => {
        console.log("hello world!")
        res.json({ code: 200, msg: "success" });
      });
      app.use(express.json());
      app.listen(7001, () => {
        console.log("Listening on http://localhost:7001");
      });
      EOF

4. Create the Dockerfile. Replace <HTTP_ENDPOINT> with the endpoint you obtained in the Prerequisites section.

      FROM node:20.16.0
      WORKDIR /app
      COPY package*.json ./
      RUN npm install
      COPY . .
      ENV OTEL_TRACES_EXPORTER="otlp"
      ENV OTEL_LOGS_EXPORTER=none
      ENV OTEL_METRICS_EXPORTER=none
      ENV OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="<HTTP_ENDPOINT>"
      ENV OTEL_NODE_RESOURCE_DETECTORS="env,host,os"
      ENV OTEL_SERVICE_NAME="ot-nodejs-demo"
      ENV NODE_OPTIONS="--require @opentelemetry/auto-instrumentations-node/register"
      EXPOSE 7001
      CMD ["node", "main.js"]

3. Configure Docker Compose

1. Return to the compose directory:

      cd ..

2. Open docker-compose.yml and add the tracing environment variables to the kong service:

      services:
        kong:
          # ... existing configuration ...
          environment:
            # ... existing variables ...
            KONG_TRACING_INSTRUMENTATIONS: all
            KONG_TRACING_SAMPLING_RATE: 1.0

3. In the same file, add a backend-api service:

      services:
        # ... existing services ...
        backend-api:
          build:
            context: ./backend
            dockerfile: Dockerfile
          environment:
            NODE_ENV: production
          ports:
            - "7001:7001"
          networks:
            - kong-net

4. Configure Kong Gateway routing and the OpenTelemetry plug-in

1. Go to the config directory:

      cd config

2. Append the following to kong.yaml. Replace <HTTP_ENDPOINT> with the endpoint from the Prerequisites section.

      cat << EOF >> kong.yaml
   
      services:
        - name: backend-api
          url: http://backend-api:7001
          routes:
            - name: main-route
              paths:
                - /hello
   
      plugins:
        - name: opentelemetry
          config:
            traces_endpoint: <HTTP_ENDPOINT>
            headers:
              X-Auth-Token: secret-token
            resource_attributes:
              service.name: kong-dev
      EOF

5. Start the services

Return to the compose directory and start all containers:

cd ..
docker compose up -d

6. Validate

Send a request through Kong Gateway:

curl http://localhost:8000/hello

Expected output:

{"code":200,"msg":"success"}

Log on to the Managed Service for OpenTelemetry console and check the Applications page. You should see two applications:

• kong-dev -- the Kong Gateway application

• ot-nodejs-demo -- the Node.js backend service

Open a trace on the Trace Explorer page to view the full request path from Kong Gateway to the backend.

7. Clean up

Stop and remove the containers when you no longer need them:

docker compose down