Trace provides end-to-end call chain tracing and analysis for AI Agent requests, helping you pinpoint performance bottlenecks, troubleshoot anomalies, and monitor agent health.
Prerequisites
-
Cloud Monitor 2.0 is activated and a workspace is created.
-
Data ingestion is configured for your AI Agent application.
Procedure
Log in to the Cloud Monitor 2.0 console, select your workspace, and in the left-side navigation pane, choose AI Agent Observability. Click the Trace tab.
Features
Search and filter
Enter query conditions in the search bar at the top and press Enter.
Press Shift+Enter to enter multiline queries for more precise searches.
The Aggregate button and aggregation dimension selector to the right of the search bar let you run aggregated analysis on search results for a quick statistical overview.
Overview charts
Three trend charts below the search area show the overall status of AI Agent calls at a glance:
|
Name |
Description |
|
Trace count |
Trend of trace count over the selected time period, reflecting request volume fluctuations. |
|
Average latency |
Trend of average request duration, helping you spot performance degradation or abnormal delays. |
|
Token consumption |
Trend of total token consumption (total_tokens) for monitoring model resource usage. |
View modes
Use the view switcher in the upper-right corner to choose from three view modes:
|
View mode |
Description |
|
List View (default) |
Displays traces in a table, suitable for quick browsing and bulk filtering. |
|
Card View |
Displays trace information as cards with richer content previews. |
|
Trace View |
Displays traces as a timeline or waterfall chart, ideal for analyzing timing relationships and duration distribution across a call chain. |
List fields
In List View, each trace record includes the following fields:
|
Field |
Description |
|
Trace ID |
Unique identifier of the call chain. Use it to retrieve a specific request. |
|
Input |
Original user input to the AI Agent. |
|
Output |
Response from the AI Agent. |
|
Duration |
Total end-to-end duration of the request, in milliseconds (ms). |
|
Total tokens |
Total tokens consumed by the request, including both input and output tokens. |
|
Entry app name |
Name of the application that initiated the request, used to distinguish between Agent application sources. |
|
User ID |
Identifier of the user who initiated the request. |
|
Session ID |
Session identifier. Multiple requests within the same session share this ID. |
|
Start time |
Start time of the request, accurate to the millisecond. |
|
Actions |
Click the Details button to view the full trace details, including execution information for each Span. |
Pagination
The bottom of the list shows the total number of matching traces and provides pagination controls to browse more results.
Trace details
Find the target trace in the trace list and click Details in the Actions column to open the Trace Details page.
You can also navigate from the Session Analysis page to a specific session's details, and then click a trace ID in the Trace List to access the trace details.
Information bar
The top of the page displays core metadata of the current trace for quick identification and sharing.
|
Field |
Description |
|
Trace ID |
Unique identifier for the current call chain. |
|
Start time |
Time the request was initiated, accurate to the millisecond. |
|
Duration |
End-to-end duration, in milliseconds (ms). |
|
Agents |
Number of AI Agents involved in this trace. |
|
Total token consumption |
Total tokens consumed by this request. |
|
Session ID |
Session identifier, used to correlate other requests in the same session. |
|
User ID |
Identifier of the user who initiated the request. |
|
LLM calls |
Number of LLM calls in this trace. |
|
Tool calls |
Number of tool calls in this trace. |
The upper-right corner provides options such as Open in New Page, Password-free Sharing, and Switch to Old Version for viewing and sharing trace information.
Below the information bar, summaries of the request's input and output help you quickly understand the request semantics without drilling into individual Span details.
View modes
The Trace Details page offers five view modes. Switch between them using the tabs at the top:
|
View |
Description |
|
Call Tree |
Displays the Span hierarchy in a tree structure. Each node shows the type (ENTRY, AGENT, LLM, Tool, etc.), Span name, key attributes (such as total tokens and model name), and a duration Gantt chart. Ideal for understanding the overall call structure and duration distribution. |
|
Trace Graph |
Displays Span nodes and their call relationships in a topology graph. Each node shows core metrics like duration, calls, and status. Supports switching between Trajectory View and Aggregated View, suitable for analyzing call topology and node dependencies. |
|
Timeline |
The left side shows a Span list as a waterfall chart (including duration bars), while the right side shows full details of the selected Span (with subtabs like Info, Attributes, Resources, Details, Events, and Links). The most commonly used view for detailed analysis. |
|
Reasoning Trajectory |
Groups conversation trajectories by Agent and uses colored bubbles to distinguish System, User, Assistant, and Tool messages. Reconstructs the end-to-end dialogue during the reasoning process, ideal for analyzing LLM reasoning logic. |
|
Trace Analysis |
Aggregates key performance metrics for the current trace and visualizes performance and call distribution. Use this view to assess overall trace health. |
The Microservice View link in the upper-right corner opens the corresponding trace view in Microservices Observability for coordinated troubleshooting with the underlying infrastructure.
Call tree
Displays the Span hierarchy in an indented tree structure. Each node includes:
-
Span type label (such as ENTRY, AGENT, LLM, and Tool).
-
Span name (such as
enter_ai_application_system,invoke_agent xxx, andgenerate_content xxx). -
Key attributes (such as
total tokens: 2andmodel: cosyvoice-v1). -
A duration Gantt chart bar and the duration value on the right.
Use the search box, Highlight only switch, Expand/Collapse buttons, and View Configuration to quickly locate key Spans in large traces.
Trace graph
Displays call relationships between Spans in a node-link topology graph. Each node card shows:
-
Span name
-
duration
-
calls
-
status (execution status, such as OK or Error)
-
Key attributes (such as model name and total tokens)
Switch between Trajectory View and Aggregated View in the upper-right corner. Trajectory View preserves the original call order to show the complete trace, while Aggregated View merges identical Spans to help identify hot nodes and call patterns.
Timeline
Provides the most detailed single-Span analysis capabilities.
-
Left-side Span list: Shows all Spans in chronological order as a waterfall chart. Each Span displays its type label, name, duration, and Gantt chart bar, illustrating the timing and duration relationships between Spans.
-
Right-side Span details: Click any Span on the left to view its full details on the right, which includes the following subtabs:
|
Subtab |
Description |
|
Info |
Displays core metadata (start time, end time, duration, status, etc.) and input/output content. Input and output are grouped by role (User, Assistant, Tool) and can be toggled between Pretty and Raw formats. |
|
Attributes |
Standard OpenTelemetry attributes and extended GenAI semantic convention attributes. The number in parentheses indicates the attribute count. |
|
Resources |
Resource metadata (such as service, host, and SDK version) to identify the Span's origin. |
|
Details |
Extended detailed information for the Span. |
|
Events |
Events associated with the Span. |
|
Links |
Links associated with the Span, used for cross-trace correlation analysis. |
The details panel provides shortcuts in the upper-right corner, such as Query Logs and Configure Events, for quick access to related logs or event views.
Reasoning trajectory
Organizes the conversation flow by Agent, using colored bubbles to distinguish four message types and reconstruct the LLM reasoning process.
|
Message role |
Description |
|
System |
System prompts or session context settings. |
|
User |
User input or upstream requests. |
|
Assistant |
Model replies or Agent decisions. |
|
Tool |
Input parameters or returned results of a tool call. |
Each message appears as a bubble showing the full text or JSON content (such as structured fields like mime_type, modality, and uri), with the message duration in the upper-right corner.
Trace analysis
Aggregates and displays performance and call metrics for the current trace.
Core metrics
Six core metric cards at the top of the page:
|
Metric |
Description |
|
Total duration |
Total execution duration of this trace. |
|
TT |
Total tokens consumed by the trace. |
|
Token consumption |
Token consumption trend within the trace's time window. |
|
LLI |
Number of LLM calls in the trace. |
|
TO |
Number of tool calls in the trace. |
|
SK |
Number of skill loads in the trace. |
Performance analysis
-
LLM and tool call duration percentage: A pie chart showing the duration percentage of LLM and tool calls in this trace.
-
Top LLM calls by duration: Lists the longest-running LLM call Spans, with fields including spanName, spanId, duration, TTFT, and input tokens.
-
Top tool calls by duration: Lists the longest-running tool call Spans.
LLM call analysis
Four trend charts display fine-grained LLM call performance:
-
LLM turn duration trend (ttft / total_duration)
-
LLM turn token trend (input_tokens / output_tokens)
-
LLM turn token output rate (output_tokens/s)
-
LLM cache hit rate (cache_hit)
Tool call analysis
Three charts show tool call performance:
-
Tool call duration trend
-
Tool call input/output size
-
Call count per tool
Skill load analysis
Shows skill loading records to help analyze which skills the Agent loaded during execution and their durations.
Use cases
-
Pinpoint performance bottlenecks: In the Call Tree or Timeline view, sort by duration to identify the longest Spans. In the Trace Analysis view, review the duration share and top LLM/tool calls to find optimization targets.
-
Troubleshoot abnormal requests: In the Call Tree view, find Spans with an Error status. Then use the Timeline view to inspect their Attributes and Events, including error stacks and details, to identify the root cause.
-
Audit the reasoning process: Use the Reasoning Trajectory view to replay the full User-Assistant-Tool dialogue flow and evaluate the model's decision path and tool usage.
-
Analyze token usage: In the Trace Analysis view, review token consumption distribution and cache hit rate to identify high-consumption nodes and optimize prompt design.