This article describes how to use AgentSight. - Alibaba Cloud Linux

AgentSight is an AI Agent observability tool based on eBPF. It realizes fine-grained data collection and correlation analysis of the entire Agent operation link without intruding business logic.

How to use AgentSight

Product introduction

AgentSight is an AI Agent observability tool based on eBPF. It realizes fine-grained data collection and correlation analysis of the entire Agent operation link without intruding business logic. There is no need to modify the Agent code or configure the agent. After installation, it can automatically discover the AI Agent running on the system and collect its large model calls, token consumption and process behavior.

core competencies

AgentSight mainly includes the following capabilities:

Token consumption analysis: Comprehensive measurement and attribution of Token consumption during Agent operation. Supports flexible query by time period or the last N hours, and can automatically perform comparisons. It supports splitting consumption sources by multiple dimensions such as agents, tasks, roles, etc. The analysis granularity can be accurate to a single LLM call, and supports separate accounting of cached tokens.
behavioral audit: Full-link recording and tracking of Agent’s LLM calls and process execution behaviors. During data collection, key metadata such as provider and model version of each LLM call are completely retained, and the command line parameters of the process are synchronously captured. In addition, the system supports multi-dimensional flexible filtering by time dimension, process identification and event type, and provides visual summary statistical analysis capabilities.
Session interruption diagnostics: Automatically detect and attribute session exceptions, covering 18 types such as current limit, authentication failure, streaming response truncation, context overrun, process crash, retry storm, infinite loop, etc., and mark the severity level, which can be processed on the Dashboard or queried through the CLI.
Optimization analysis: Built-in optimization analysis workspace, performs in-depth analysis of the three dimensions of specified session running accuracy, performance, and cost (some dimensions require the configuration of a large model for analysis), outputs problem location and optimization suggestions, and the analysis history can be reviewed persistently.
Trajectory collection and export: Supports converting Agent session tracks into standard ATIF format (v1.7) for storage and export for offline analysis, playback and evaluation; Dashboard provides track list and sub-agent topology view.
Dashboard visualization: Web visual interface that provides intuitive display of Token consumption trends, Agent status monitoring, session interruption processing, Session details, optimization analysis, and Token savings. Token authentication is enabled by default and can be securely deployed on remote servers and accessed directly through local browsers.

Scope

AgentSight automatically discovers and tracks the following Agents through built-in rules: OpenClaw, Copilot Shell (cosh and cosh-ng, non-AK/SK authentication scenarios), Claude Code, Codex CLI, QwenCode, Hermes, AgentScope, etc. If you need to track other Agents, you can customize process matching rules through the configuration file.

Installation method

For details, please refer toQuick start. After the system service is installed, eBPF tracing (trace) and API service (serve) are started with the system by default, and there is no need to manually execute them.

Conversational interaction method

AgentSight provides conversational interaction skills that can be installed and used in various AI Agents. Users do not need to memorize CLI commands and can complete operations directly through natural language:

View Token consumption: Such as "How many tokens were used today?"
Query audit logs: Such as "Help me check today's LLM call record"
Troubleshoot session exceptions:For example, "Have any sessions been interrupted recently? Please help me find out why."

If you are using Copilot Shell (cosh), the skill is built-in and you can directly use the above natural language instructions. The system will automatically call AgentSight to complete the query and return the analysis conclusion.

CLI command details

agentsight trace — start eBPF tracing

Note: This service has been started by default in the system and does not need to be executed manually.

Start eBPF-based AI Agent activity tracking.

sudo agentsight trace

agentsight serve — start API and Dashboard

Note: This service has been started in the system by default and is bound to 0.0.0.0:7396 by default. There is no need to execute it manually.

Start an HTTP API server to provide an embedded Dashboard UI.

sudo agentsight serve --host 0.0.0.0 --port 7396

This command will bind all network interfaces and can be accessed through the server’s public IP:http://<server public network IP>:7396

Please ensure that the server firewall/security group has released port 7396。

agentsight summary — One-stop status overview

One command summarizes session and token usage in the latest time window, interruption events by severity, and token saving data. It is suitable for daily inspections or to quickly understand the overall operating status of the Agent on the machine. When any data source is unavailable, it is degraded independently without affecting the overall output.

agentsight summary            # Overview of the last 24 hours
agentsight summary --last 48  # last 48 hours
agentsight summary --json     # JSON output

agentsight token — Query Token usage

Query Token usage data.

# Check today’s usage
agentsight token

agentsight audit — Query audit events

Query audit events (LLM calls, process operations).

# View recent events
agentsight audit
# Filter by PID and type
agentsight audit --pid 12345 --type llm
# Summary statistics
agentsight audit --summary

agentsight discover — Scan Agent

Discover the AI Agents running on the system.

# Scan Agent
agentsight discover
# List known types
agentsight discover --list-known

agentsight interruption — Query and manage session interruptions

agentsight interruption list                    # List outage events in the last 24 hours
agentsight interruption list --severity high    # Filter by severity
agentsight interruption stats                   # Statistics by type
agentsight interruption get <ID>                # View individual outage details
agentsight interruption resolve <ID>            # Mark as solved

agentsight dashboard — Get Dashboard access information

Dashboard access address, authentication status, and login token are displayed; in the ECS environment, security group release guidelines are also output.

agentsight dashboard

Dashboard visual interface

Dashboard is AgentSight's web visual interface for viewing conversation history, Trace details, and Token statistics. Dashboard enables token authentication by default. When accessing for the first time, you need to obtain the login token through the agentsight dashboard command;

Dashboard function

Dashboard provides the following core functionality:

Token consumption overview: View the token consumption of the current machine within the selected time period. A time range selector is provided at the top of the Dashboard to switch between different time periods; the input Token, output Token and total Token usage are displayed in the form of statistical cards below.
Agent status: The status bar on the right can view the current Agent process status and provide the Agent process hang and restart function.
Session interruption diagnostics: Independent interruption processing page. The session list identifies exceptions in the form of labels. The interruption details panel displays the type, severity level and error information. It supports Resolve or Hide processing.
Session details: Click "Details" to view the token usage details of each session and trace
Optimization analysis: Select session running accuracy, performance, and cost dimension analysis. The cost dimension supports waste analysis based on "detour", and the analysis history can be reviewed.
Token Save: Linked with the Tokenless component (automatically effective after both are installed), it displays the total consumption, reduced Tokens and reduction rate, supports baseline comparison, decomposition by optimization strategy, and row-level comparison before and after optimization.

Data management

Database management

Automatic capacity limit and cleanup: To prevent the database from growing indefinitely and occupying too much disk space, the system defaults to setting the maximum database capacity to 200 MB. When the database size reaches the upper limit, the cleanup process will be automatically triggered.

Users can customize the maximum capacity (unit: MB) through the environment variable AGENTSIGHT_GENAI_DB_MAX_SIZE_MB, for example, set it to 500 MB.

export AGENTSIGHT_GENAI_DB_MAX_SIZE_MB=500

The interrupt event library is retained for 30 days by default and has an upper limit of 100 MB, which can be adjusted in the configuration file.

Clean historical data

To clean historical data, perform the following operations:

rm -rf /var/log/sysak/.agentsight

Then restart AgentSight.

Configuration management

Default path: /etc/agentsight/config.json (other paths can be specified via runtime --config).

Configuration block	illustrate
cmdline.allow / cmdline.deny	Agent process command line matching rules to determine which processes to track
https	Domain name filtering rules, only capture large model traffic matching domain names
features	Function switch, interrupt detection, trajectory collection, auditing and other capabilities can be started and stopped on demand
runtime_limits	Memory and buffer upper limit to prevent unlimited growth of resource usage

Note: User profiles completely replace the built-in default rules, rather than being merged and appended. If the rules for an Agent are missing from the custom configuration, the Agent will not be discovered; please make sure to retain all Agent rules that need to be monitored before modifying them. Configuration format upgrades are automatically managed by the schema_version field, and the program will automatically back up and migrate when the version becomes outdated.

FAQ

Q1: Why can’t I obtain OpenClaw’s Token consumption data?
A: AgentSight monitors openclaw-gateway daemon. Please check whether the connection status between the client and Gateway is normal. If the following exception log appears, it indicates that pairing was unsuccessful:
Gateway agent failed; falling back to embedded: Error: gateway closed (1008): pairing required
It is recommended to execute the command openclaw devices approve Complete device pairing.

Q2: Why does the Token saving page not display the current Session ID, or the displayed Token saving amount is 0?
A： It may be caused by the following two reasons:

1. The current version does not support cosh’s AK/SK authentication method;

2. The Session ID format is non-standard UUID, causing system matching to fail.

Q3: Why is the "Saved amount of optimization items" displayed on the Token saving page greater than the difference between "Number of Tokens before optimization" minus "Number of Tokens after optimization"?

A： This is because the Agent contextualizes historical messages with every conversation. Therefore, the statistical results of the current conversation include the optimization benefits of historical messages, causing the cumulative savings to be greater than the immediate difference of a single conversation.

Q4: Why does the browser prompt that I need to log in when opening the Dashboard?

A: The new version enables token authentication by default to protect data security. Execute agentsight dashboard on the server to obtain the login token and enter it on the login page.