The Application Real-Time Monitoring Service (ARMS) agent automatically instruments common Java frameworks to collect trace data without code changes. To capture trace data that reflects your specific business logic, you can add custom instrumentation using the OpenTelemetry SDK for Java. This topic describes how to use the OpenTelemetry SDK for Java to add custom instrumentation, access the trace context, define custom Baggage, and set custom attributes.
For information about components and frameworks supported by the ARMS agent, see Java components and frameworks supported by ARMS.
Prerequisites
-
You have connected your application to Application Real-Time Monitoring Service (ARMS). For more information, see Application Monitoring integration overview.
-
Your ARMS agent version is 2.9.1.2 or later. To upgrade the agent, see Upgrade the ARMS agent.
Add dependencies
Add the following Maven dependencies to your project. For more information, see the official OpenTelemetry documentation.
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-trace</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom</artifactId>
<version>1.23.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
ARMS agent compatibility with OpenTelemetry instrumentation
Key concepts
This section describes only common terms. For more information about other terms, see the OpenTelemetry Specification.
-
span: a specific operation in a request, such as a remote call entry point or an internal method call.
-
SpanContext: the context of a trace, including information such as the trace ID and span ID.
-
attribute: an additional field on a span that records key information.
-
Baggage: key-value pairs that are propagated throughout the entire trace.
Use the OpenTelemetry SDK for Java
You can use the OpenTelemetry SDK to perform the following operations:
-
Add instrumentation to generate spans.
-
Add attributes to spans.
-
Propagate Baggage in the trace context.
-
Get the current trace context and print information such as the trace ID and span ID.
The following sample code shows how to use the OpenTelemetry SDK to perform these operations.
Important: You must obtain the OpenTelemetry instance by calling GlobalOpenTelemetry.get(). Do not use an instance you build manually with the OpenTelemetry SDK. Otherwise, in ARMS agent v4.x, spans generated by the SDK instrumentation will not be visible.
@RestController
@RequestMapping("/ot")
public class OpenTelemetryController {
private Tracer tracer;
private ScheduledExecutorService ses = Executors.newSingleThreadScheduledExecutor();
@PostConstruct
public void init() {
OpenTelemetrySdk.builder()
.setPropagators(ContextPropagators.create(W3CTraceContextPropagator.getInstance()))
.buildAndRegisterGlobal();
tracer = GlobalOpenTelemetry.get().getTracer("manual-sdk", "1.0.0");
ses.scheduleAtFixedRate(new Runnable() {
@Override
public void run() {
Span span = tracer.spanBuilder("schedule")
.setAttribute("schedule.time", System.currentTimeMillis())
.startSpan();
try (Scope scope = span.makeCurrent()) {
System.out.println("scheduled!");
Thread.sleep(500L);
span.setAttribute("schedule.success", true);
System.out.println(Span.current().getSpanContext().getTraceId()); // Get the trace ID
} catch (Throwable t) {
span.setStatus(StatusCode.ERROR, t.getMessage());
} finally {
span.end();
}
}
}, 10, 30, TimeUnit.SECONDS);
}
@ResponseBody
@RequestMapping("/parent")
public String parent() {
Span span = tracer.spanBuilder("parent").setSpanKind(SpanKind.SERVER).startSpan();
try (Scope scope = span.makeCurrent()) {
// Use Baggage to propagate custom business tags.
Baggage baggage = Baggage.current().toBuilder()
.put("user.id", "1")
.put("user.name", "name")
.build();
try (Scope baggageScope = baggage.storeInContext(Context.current()).makeCurrent()) {
child();
}
span.setAttribute("http.method", "GET");
span.setAttribute("http.uri", "/parent");
} finally {
span.end();
}
return "parent";
}
private void child() {
Span span = tracer.spanBuilder("child").startSpan();
try (Scope scope = span.makeCurrent()) {
System.out.println("current traceId = " + Span.current().getSpanContext().getTraceId());
System.out.println("userId in baggage = " + Baggage.current().getEntryValue("user.id"));
Thread.sleep(1000);
} catch (Throwable e) {
span.setStatus(StatusCode.ERROR, e.getMessage());
} finally {
span.end();
}
}
}
Example walkthrough:
-
In the
initmethod ofOpenTelemetryController, a scheduled task starts. A span is created at the beginning of each execution and ended when it finishes. -
In the
parentmethod ofOpenTelemetryController, several OpenTelemetry SDK methods are called.-
Each time the method is called, a span named
parentis created and is ended when the method finishes. -
The Baggage SDK is used to add two Baggage items:
user.idanduser.name. These items propagate to downstream applications. -
Two attributes are added to the span created in step 2.a.
-
-
The
childmethod ofOpenTelemetryControllerperforms the following operations:
Differences between ARMS agent versions
Support for the operations in the preceding code differs between ARMS agent versions.
|
Step |
ARMS agent v4.x and later |
ARMS agent v3.x and earlier |
|
1 |
Supported. A new span is generated. |
Supported. A new span is generated. |
|
2.a |
Supported |
Supported |
|
2.b |
Supported |
Not supported |
|
2.c |
Supported |
Supported |
|
3.a |
Supported |
Supported. This span appears as a method stack within the span created in step 2.a. |
|
3.b |
Supported. The printed trace ID is the same as the trace ID in ARMS. |
Not supported. The printed trace ID is different from the trace ID in the ARMS agent. |
|
3.c |
Supported |
Supported |
Instrumentation results
v4.x and later
-
Instrumentation result of Step 1:
You can see the span generated by the OpenTelemetry SDK.
In the Span Details of a trace in the ARMS console, you can view basic information, such as the application name (for example,
elastic-search-8), operation name (for example,schedule), span type (INTERNAL), and duration. On the Attributes tab, you can view OpenTelemetry attributes, such asotel.scope.name=manual-sdkandotel.scope.version=1.0.0, and custom business attributes, such asschedule.success=true, which confirms that the instrumentation is working. -
Instrumentation results of Step 2.x and Step 3.x:
Spans generated by the OpenTelemetry SDK appear in the same trace as the Tomcat spans generated by the agent. Additionally, the related attributes for the SDK-generated spans are set as expected.
In the trace details, the operation name of the span generated by the Tomcat agent instrumentation is
/opentelemetry/parent. The spans generated by the OpenTelemetry SDK are namedparentand its child spanchild. In the span attributes,http.methodisGET,http.uriis/parent, and the custom attributeotel.scope.nameismanual-sdk.
v3.x and earlier
-
Instrumentation result of Step 1:
On the trace details page in the ARMS console, you can view the trace data collected by instrumentation. The waterfall chart on the left shows span call relationships. For example, the
schedulemethod of theelastic-search-9application is captured as a span with a component type ofuser_methodand a total response time of 567 ms. The Span Details panel on the right displays basic information such as application name, span name, IP address, span ID, and status code. The Attributes section below displays additional attributes in groups, including HTTP Information (http.path,http.status_code), RPC Information (rpc.type), and Built-in Information (component.name=user_method,slow=1). -
Instrumentation results of Step 2.x and Step 3.x:
The
childspan is displayed within the method stack of theparentspan. Additionally, the attributes for the SDK-generated spans are set as expected.In the method stack view, the trace hierarchy shows the OpenTelemetry Entry Span calling the parent span, which in turn calls the child span. These three nested calls each take about 1.01 seconds. The attributes of the child method include
line=-1andrpc.type=98.
Related documents
Correlating trace ID information with your application's business logs allows you to quickly find associated logs for troubleshooting when an issue occurs. For more information, see Associate trace IDs with business logs for Java applications.