云监控：仿真数据Demo

适用人群

• 新用户：通过 Demo 1~2 快速理解 Pipeline 基本能力。

• 算法工程师：通过 Demo 3 评估 Agent 输出质量。

• 数据平台团队：通过 Demo 4 了解全流程编排模式。

目标资源

与 OT-AI Trace 格式的差异

本模板的仿真数据与其他模板使用的 OT-AI Trace 数据在格式上存在以下差异：

字段可用性矩阵

22 个字段在不同 event_type 下的取值情况如下表所示。

示例数据概览

Demo 1：用户提问三级去重

利用仿真数据中带 dedup_tag 标记的重复数据（exact / fuzzy / semantic），演示三级去重链的逐级过滤效果。无 LLM 和 GPU 成本。

定制建议

Demo 2：问题聚类 + 场景标注

对去重后的用户提问进行语义聚类，按簇采样后用 LLM 自动标注意图、复杂度等维度，发现 Agent 使用的场景分布。包含 1 轮 LLM 调用。

SPL 语法

* | project question=question,
          trace_id=trace_id,
          agent=agent_name,
          user_id=user_id,
          event_type=event_type
  | where event_type = 'user_query' AND length(question) > 0
  | dedup-exact -field=question
  | dedup-semantic -field=question -threshold='0.1'
  | semantic-cluster -field=__dedup_emb -n=5
  | sample -n=3 by __cluster_id
  | llm-call -prompt='@anno/scene-label.md' -fields=question -format=json as anno

标注输出示例

{
  "意图类型": "问题诊断",
  "任务复杂度": "中等",
  "业务场景": "安全审计",
  "补充标签": ["日志分析", "异常检测", "IP维度"]
}

定制建议

Demo 3：Agent 回答质量评估

将离散事件日志按 trace_id 聚合为"一问一答"实例，通过 LLM 多维度评估 Agent 回答质量。包含 1 轮 LLM 调用。

SPL 语法

* | project question=question,
          output=output,
          model=model,
          tool_name=tool_name,
          token_input=token_input,
          token_output=token_output,
          latency_ms=latency_ms,
          event_type=event_type,
          trace_id=trace_id,
          session_id=session_id,
          agent_name=agent_name
  | where event_type IN ('user_query','tool_call','tool_result','assistant_content','completion')
  | make-instance
      question=first(question),
      answer=last(output),
      model=any(model),
      agent=any(agent_name),
      tool_chain=join(tool_name, ' → '),
      tool_count=count_if(event_type = 'tool_call'),
      total_tokens=sum(cast(NULLIF(token_input, '') as bigint)),
      latency=max(cast(NULLIF(latency_ms, '') as bigint))
      by session_id,trace_id
  | where question IS NOT NULL AND answer IS NOT NULL
  | dedup-exact -field=question
  | sample -n=20
  | llm-call -prompt='@eval/agent-quality.md' -fields=question,answer,tool_chain -format=json as eval
  | doc-stats -field=answer

评估输出示例

{
  "需求理解": {"score": 5, "reason": "准确理解了用户要求创建基于IP网段的告警规则"},
  "回答质量": {"score": 4, "reason": "告警规则配置完整，但未说明触发频率限制"},
  "逻辑连贯": {"score": 5, "reason": "从查询到验证到创建，步骤清晰"},
  "格式规范": {"score": 5, "reason": "告警配置以结构化格式呈现"},
  "安全合规": {"score": 5, "reason": "未泄露敏感信息"}
}

定制建议

Demo 4：端到端全流程

数据治理全流程流水线：字段提取、事件聚合、指标派生、三级去重、聚类采样、质量评估 + 场景标注、文档统计。覆盖全部算子，包含 2 轮 LLM 调用。

SPL 语法

* | project question=question,
          output=output,
          model=model,
          tool_name=tool_name,
          tool_args=tool_args,
          tool_success=tool_success,
          token_input=token_input,
          token_output=token_output,
          latency_ms=latency_ms,
          status=status,
          event_type=event_type,
          trace_id=trace_id,
          session_id=session_id,
          conversation_id=conversation_id,
          agent_name=agent_name,
          user_id=user_id,
          region_id=region_id
  | where event_type IN ('user_query','system_prompt','tool_call','tool_result','assistant_content','completion')
  | make-instance
      question=first(question),
      answer=last(output),
      model=any(model),
      agent=any(agent_name),
      user_id=any(user_id),
      region=any(region_id),
      tool_chain=join(tool_name, ' → '),
      tools=array_distinct(tool_name),
      tool_count=count_if(event_type = 'tool_call'),
      has_error=bool_or(status = 'error'),
      total_input_tokens=sum(cast(NULLIF(token_input, '') as bigint)),
      total_output_tokens=sum(cast(NULLIF(token_output, '') as bigint)),
      latency=max(cast(NULLIF(latency_ms, '') as bigint))
      by session_id,trace_id,conversation_id
  | where question IS NOT NULL AND answer IS NOT NULL
  | extend token_total=total_input_tokens + total_output_tokens,
          answer_preview=substr(answer, 1, 500)
  | dedup-exact -field=question
  | dedup-fuzzy -field=question -threshold='3'
  | dedup-semantic -field=question -threshold='0.1'
  | semantic-cluster -field=__dedup_emb -n=5
  | sample -n=3 by __cluster_id
  | llm-call -prompt='@eval/agent-quality.md' -fields=question,answer,tool_chain -format=json as eval
  | llm-call -prompt='@anno/scene-label.md' -fields=question -format=json as anno
  | doc-stats -field=answer

API 配置（JSON）

{
  "name": "mock_data_demo_full",
  "description": "仿真数据端到端全流程 Demo：聚合、去重、聚类、评估、标注，覆盖全部算子能力",
  "source": {
    "type": "logstore",
    "logstore": {
      "project": "ali-pub-cn-hangzhou-staging-sls-admin",
      "logstore": "ai_test",
      "query": "*"
    }
  },
  "pipeline": {
    "nodes": [
      {"id": "extract", "type": "project", "parameters": {"question": "question", "output": "output", "model": "model", "tool_name": "tool_name", "tool_args": "tool_args", "tool_success": "tool_success", "token_input": "token_input", "token_output": "token_output", "latency_ms": "latency_ms", "status": "status", "event_type": "event_type", "trace_id": "trace_id", "session_id": "session_id", "conversation_id": "conversation_id", "agent_name": "agent_name", "user_id": "user_id", "region_id": "region_id"}},
      {"id": "filter_events", "type": "where", "parameters": {"filter": "event_type IN ('user_query','system_prompt','tool_call','tool_result','assistant_content','completion')"}},
      {"id": "assemble", "type": "make-instance", "parameters": {"question": "first(question)", "answer": "last(output)", "model": "any(model)", "agent": "any(agent_name)", "user_id": "any(user_id)", "region": "any(region_id)", "tool_chain": "join(tool_name, ' → ')", "tools": "array_distinct(tool_name)", "tool_count": "count_if(event_type = 'tool_call')", "has_error": "bool_or(status = 'error')", "total_input_tokens": "sum(cast(NULLIF(token_input, '') as bigint))", "total_output_tokens": "sum(cast(NULLIF(token_output, '') as bigint))", "latency": "max(cast(NULLIF(latency_ms, '') as bigint))", "by": "session_id,trace_id,conversation_id"}},
      {"id": "filter_valid", "type": "where", "parameters": {"filter": "question IS NOT NULL AND answer IS NOT NULL"}},
      {"id": "derive_metrics", "type": "extend", "parameters": {"token_total": "total_input_tokens + total_output_tokens", "answer_preview": "substr(answer, 1, 500)"}},
      {"id": "exact_dedup", "type": "dedup-exact", "parameters": {"field": "question"}},
      {"id": "fuzzy_dedup", "type": "dedup-fuzzy", "parameters": {"field": "question", "threshold": "3"}},
      {"id": "semantic_dedup", "type": "dedup-semantic", "parameters": {"field": "question", "threshold": "0.1"}},
      {"id": "cluster", "type": "semantic-cluster", "parameters": {"field": "__dedup_emb", "n": 5}},
      {"id": "sample_per_cluster", "type": "sample", "parameters": {"n": 3, "by": "__cluster_id"}},
      {"id": "evaluate", "type": "llm-call", "parameters": {"prompt": "@eval/agent-quality.md", "fields": "question,answer,tool_chain", "format": "json", "as": "eval"}},
      {"id": "annotate", "type": "llm-call", "parameters": {"prompt": "@anno/scene-label.md", "fields": "question", "format": "json", "as": "anno"}},
      {"id": "text_stats", "type": "doc-stats", "parameters": {"field": "answer"}}
    ]
  },
  "sink": {
    "type": "dataset",
    "dataset": {"workspace": "<your-workspace-name>", "dataset": "mock_demo_full"}
  },
  "executePolicy": {
    "mode": "run_once",
    "run_once": {"fromTime": 1772150000, "toTime": 1772240000}
  }
}

全流程数据量变化

常见问题

在使用 Demo 3 和 Demo 4 时，可能遇到以下常见问题：