DataLens AI

DataLens AI provides end-to-end observability for LLM and agent applications. Use tracing, prompt management, and automated evaluation to locate execution errors, quantify token costs, and assess output quality in production. It addresses the challenges of unpredictable outputs and hidden costs when running agents in production.

Core capabilities

LLM observability

LLM applications involve complex, non-deterministic interactions that traditional monitoring cannot cover. DataLens AI traces token consumption, call latency, and tool invocations across your LLM applications, with dimensional slicing.

End-to-end tracing: Breaks down execution into three layers for full observability.
- Session: A complete record of multi-turn user interactions. Review the full context to pinpoint where the agent hallucinates or drifts.
- Trace: A single interaction from user input to agent output, with a breakdown of inputs, outputs, execution time, and token consumption. Visualizes the execution path in tree and graph formats to identify bottlenecks and high-cost operations.
- Generation/span: All atomic operations within a trace, including execution time, tokens consumed, and intermediate results for each step. Enables targeted optimization.
Session and user tracking: Track multi-turn conversations as sessions and link them to user information.
Agent visualization: Visualize the execution flow of an agent in tree or graph formats.
Flexible data collection: Zero-code integration with Dify, Ragflow, and other frameworks. Also supports native Python/JS SDKs and OpenTelemetry, covering 14+ AI frameworks.

Prompt management

Prompt management is critical for developing and iterating on agent applications. DataLens AI separates prompts from code, enabling version control and team collaboration.

Decouple prompts from code: Manage and deploy prompts independently from application code.
Agile iteration: Adjust and optimize prompts without code changes.
A/B testing: Use version tags to A/B test prompts and validate optimizations.
Playground: Test prompts in an online environment.

Evaluation

DataLens AI provides a flexible evaluation system to assess LLM output quality across multiple dimensions and continuously improve agent performance.

Offline and online evaluation: Run dataset-based offline evaluation or online evaluation in production.
Multiple scoring types: Numerical, categorical, boolean, and text-based scoring methods.
LLM as a Judge: Automatically score results with an LLM for quality assessment at scale.
Human scoring and annotation: Create annotation queues for human evaluators to score and label results.
Programmatic scoring with SDKs: Define custom scoring logic with an SDK and integrate it into CI/CD pipelines.

Benefits

100% compatible with the open-source Langfuse ecosystem: Fully compatible with the Langfuse SDK and API. Connect existing Langfuse integrations without code changes.
Fully managed: Activate with a single click. Required resources are created automatically — no infrastructure to deploy or maintain.
14+ AI framework integrations: Zero-code or low-code integration with Dify, Ragflow, LangChain, LlamaIndex, the OpenAI SDK, and more.
Flexible evaluation system: A complete evaluation workflow — from automated scoring to human annotation — for quality assurance at every stage.

Use cases

Scenario	Description	Core value
Development and debugging	Trace errors and performance bottlenecks in agent execution with end-to-end tracing.	Faster troubleshooting and iteration.
Prompt management	Centrally manage and iterate on prompts with version control and A/B testing.	Decouple prompts from code for efficient team collaboration.
Quality evaluation	Assess agent output quality through automated evaluation and human annotation.	Quantify output quality to drive continuous improvement.
Production monitoring	Track token consumption, latency, and error rates in production in real time.	Cost control and anomaly awareness.
Team collaboration	Share observability data across teams with organization and project management features.	Unified observability reduces collaboration overhead.

Limitations

Item	Description
Instance version requirement	See documentation for required component versions.
Service status	Currently in invite-only preview. Apply for access to enable the service.
Public internet access	Features that require public internet access, such as LLM as a Judge, need SNAT configured for the VPC where your Langfuse instance is deployed. For configuration details, see Internet NAT Gateway.