LLM application overview

After you install the Python agent for your Large Language Model (LLM) application, Application Real-Time Monitoring Service (ARMS) begins monitoring the application. On the Overview page, you can view key metrics for your LLM application, such as the number of model calls, token usage, trace count, and session count.

Prerequisites

The agent must be installed for your LLM application. For more information, see Connect an LLM application or inference service to ARMS.

Log on to the ARMS console. In the left-side navigation pane, choose LLM Application Monitoring > Application list.
On the Application list page, select a region at the top of the page, and then click the name of your target application.
In the top navigation bar, click Overview.

Dashboard

Panel	Description
Number of model calls	The number of times the application calls the LLM in the selected time range.
Token usage	The number of tokens the application consumes in the selected time range.
Trace count	The number of traces generated by the application in the selected time range.
Span count	The number of spans generated by the application in the selected time range.
Session count	The number of sessions initiated by the application in the selected time range.
Number of users	The total number of unique users interacting with the application in the selected time range.
Operation type distribution	CHAIN: A tool that connects an LLM with other components to perform complex tasks. A CHAIN can include retrievals, embeddings, LLM calls, and even other nested CHAINs. EMBEDDING: An embedding process, such as an operation that converts text into a vector representation using an embedding model. This allows for similarity searches to find relevant context. RETRIEVER: An operation that accesses a vector storage or database to fetch data. This is typically used to supplement the context provided to the LLM to improve response accuracy and relevance. RERANKER: An operation that reorders a set of input documents based on their relevance to a given query. It may return the top K most relevant documents to be used as context for an LLM. LLM: An invocation of a Large Language Model. For example, calling an LLM through an SDK or API to perform inference or text generation. TOOL: An invocation of an external tool, such as a calculator or a weather API, to obtain specific information, such as the current weather. AGENT: An intelligent agent, which is a complex CHAIN that uses the LLM's reasoning to decide the next action. This may involve multiple calls to LLMs and TOOLs to progressively reach a final answer. TASK: A custom, user-defined method within the application. For example, calling a local function to apply specific logic.
Avg LLM call per request	The average number of LLM calls per request, calculated over a one-minute interval.
Request count trend	Tracks the number of requests to the LLM application per minute.
Model call ranking	Lists the top 5 most frequently called LLMs in the application.
User ranking by request count	Lists the top 5 users who made the most requests in the application.
Session count trend	Tracks the session count for the LLM application per minute.

Application Real-Time Monitoring Service:Overview

Prerequisites

LLM application overview

Dashboard

Number of model calls

Trace count

Session count

Number of users

Operation type distribution

Avg LLM call per request

Model call ranking

User ranking by request count

Session count trend

Related documents