Add custom instrumentation to Java applications using OpenTelemetry Java Instrumentation

Starting with version 4.x, the ARMS Java agent is refactored based on opentelemetry-java-instrumentation. Therefore, the ARMS 4.x agent inherits the custom instrumentation capabilities of opentelemetry-java-instrumentation. This topic uses custom instrumentation for the Motan remote procedure call (RPC) framework as an example to demonstrate the process.

OpenTelemetry Java Instrumentation lets you specify the path to a custom extension JAR package by setting the -Dotel.javaagent.extensions parameter or the OTEL_JAVAAGENT_EXTENSIONS environment variable. You can implement the following features in the JAR package:

Custom TraceId generator
Custom trace context propagation protocol
Custom trace sampler
Custom SpanProcessor
Custom instrumentation logic

For stability and security reasons, the 4.x agent currently supports only the custom instrumentation feature.

Procedure

Step 1: Download and compile the custom instrumentation code

Run the following command to download the custom instrumentation demo project.

This project provides a simple implementation of client-side and server-side instrumentation for the Motan framework. It generates a span when an RPC call is initiated on the client and when the RPC call is received on the server. It also uses the Motan framework's protocol header to pass the trace context.
```
git clone git@github.com:aliyun-observability/aliyun-javaagent-extension-demo.git
```

Navigate to the project's root directory and run the ./gradlew build command.

Note

The Java Development Kit (JDK) version for the current compilation environment must be JDK 17.

After the command runs, the compiled JAR file, which is the custom extension JAR package, is in the .build/libs directory.

opentelemetry-extension-demo ~/Code/java/opentelemetry-extension-demo
  .gradle
  .idea
  build
    classes
    generated
    libs
      opentelemetry-extension-demo-1.0.jar
      opentelemetry-extension-demo-1.0-all.jar
      opentelemetry-javaagent.jar
    tmp
  gradle
  src
  build.gradle
  gradlew
  gradlew.bat
  settings.gradle

Step 2: Download the ARMS agent

Log on to the ARMS console.
In the left-side navigation pane, choose Application Monitoring > Agent Management. In the top navigation bar, select a region.
Click the Agent Version Release Notes tab. Download an agent of version 4.x or later and save it to a suitable directory.

Step 3: Download and compile the Motan demo application

Download the Motan demo application.

git clone https://github.com/weibocom/motan.git

Navigate to the project's root directory and run the cd motan-demo && mvn clean package command.

After the command runs, the motan-demo-server and motan-demo-client sample applications are in the ${motan-root}/motan-demo/motan-demo-client/target and ${motan-root}/motan-demo/motan-demo-server/target directories.

Step 4: Load the custom extension package when starting the application

Start the server application: java -javaagent:/path/to/aliyun-java-agent.jar -Darms.appName=motan-client -Darms.licenseKey=${your licenseKey} -Dotel.javaagent.extensions=/path/to/opentelemetry-extension-demo-1.0.jar -jar /path/to/motan-demo-server.jar
Start the client application: java -javaagent:/path/to/aliyun-java-agent.jar -Darms.appName=motan-server -Darms.licenseKey=${your licenseKey} -Dotel.javaagent.extensions=/path/to/opentelemetry-extension-demo-1.0.jar -jar /path/to/motan-demo-client.jar

Replace the following parameters as needed:

/path/to/aliyun-java-agent.jar: The agent you downloaded in Step 2.
${your licenseKey}: You can obtain the LicenseKey by calling the DescribeTraceLicenseKey OpenAPI operation.
/path/to/opentelemetry-extension-demo-1.0.jar: The custom extension JAR package you obtained in Step 1.
/path/to/motan-demo-server.jar: The motan-demo-server sample application you downloaded in Step 3.
/path/to/motan-demo-client.jar: The motan-demo-client sample application you downloaded in Step 3.

Step 5: View the monitoring results

Run the curl http://localhost:8080/ command multiple times to report data to ARMS.
In the ARMS console, view the trace details. You can see the generated spans for the Motan client and Motan server.

The trace waterfall chart shows a Layer 3 call chain: the root span / (motan-client SERVER, 9.12 ms), the child span MotanDemoService@hello (motan-client CLIENT, 4.68 ms), and the grandchild span MotanDemoService@hello (motan-server SERVER, 258 μs). The Span Details panel displays basic information such as the interface name, duration, and status code. Additional properties include fields such as motan.request.parameters, motan.request.url, otel.scope.name, and thread information.

Notes

When you add custom instrumentation, you must import OpenTelemetry dependencies. For optimal compatibility, import the OpenTelemetry dependency versions based on the following mapping.

ARMS Java version

OpenTelemetry dependency version

4.1.0 to latest

1.28
The package name for custom instrumentation must not start with io.opentelemetry.javaagent.instrumentation. This prevents class conflicts with the existing instrumentation code of the ARMS agent.
In your custom instrumentation, avoid importing frameworks other than OpenTelemetry and the instrumented framework. This prevents class conflicts between the instrumentation code and the application code.

ARMS Java version	OpenTelemetry dependency version
4.1.0 to latest	1.28