今回の試みは、技術系のグループチャットでの議論がきっかけでした。「洗車に行きたいのですが、洗車店はわずか50メートル先にあります。車で行くべきでしょうか、それとも歩いて行くべきでしょうか？」50メートルは確かに近い距離ですが、目的は洗車なので、車を洗車店まで持っていく必要があります。距離や燃費、歩いて行く便利さだけを分析すると、答えを実際に左右する条件を見落としがちです。そのとき、同僚たちが異なるLLMモデルを試したところ、異なる結論が得られました。
このことから、各モデルにはそれぞれ回答の特徴があり、条件の見落としや推論の誤り、ハルシネーションも起こり得ると考えました。ある問題に対して「最もよいモデル」を選ぶのではなく、複数のモデルに独立して回答させ、その結論や相違点を比較できないでしょうか。あるモデルが重要な条件を見落としても、別のモデルがそれに気づけば、回答を突き合わせることで誤りを修正する機会が得られるかもしれません。
そこで、エンドツーエンドのWebプロトタイプを作りました。同じ質問を3つのモデルに並列で渡し、元の回答を保持したうえで、テキストの一致と意味的一致に基づいて、候補をそのまま採用するか、二次統合を行うかを決めます。最後に、モデルによる評価を通じて、候補と最終回答の関係を示します。これは、複数モデルのアンサンブル（Multi-model Ensemble）の一つの実装です。本記事では、呼び出しの流れ、3つのデモ事例、実装の適用範囲と限界を取り上げ、最後に完全な参考コードを掲載します。読者の皆さんがご自身の実験を続ける際に活用できるようにしています。
免責事項：本構成はテスト環境での学習を目的としたものであり、変更を加えずに本番環境のベストプラクティスに適合することを保証するものではありません。
プロトタイプは、ブラウザのフロントエンド、FastAPIバックエンド、ConsensusEngineで構成されています。
● ブラウザは質問の送信と結果の表示を担当します。
● バックエンドはリクエストを受け取ります。
● エンジンは複数モデルによる回答処理を調整します。
● DeepSeek、GLM、Kimiは、最初のラウンドの回答候補を生成します。
● Qwenは、一致性の判定、回答の統合、最終評価を担当します。
● 4つのモデルはいずれもModel Studioを通じて利用します。
図1：複数モデルの合意に基づく回答システムのアーキテクチャ。
ブラウザから質問が送信されると、ConsensusEngineは3つのモデルを並列で呼び出して回答候補を取得し、最終回答を作成します。その後、Qwenが最終回答を評価し、評価結果を元の候補とともにページに返します。一致性の判定、統合、採点は、同じQwenモデルが担当します。
エンジンはまず、3つのモデルのうち、いくつが正常に回答を返したかを確認します。有効な回答がなければ、失敗のメッセージを返し、後続のモデルは呼び出しません。有効な回答が1件だけの場合は合意とみなさず、意味的一致の確認を行わずに、質問とその候補を直接Qwenに渡して統合します。有効な回答が少なくとも2件ある場合は、次の順に判定します。
● 全文が一致しているか：書式や句読点などの違いを無視したうえで、有効な回答すべての全文が同じであれば、最初の元の回答をそのまま採用します。
● 主要な結論が一致しているか：全文が異なる場合は、Qwenが元の質問と回答候補を比較します。有効な回答すべての主要な結論が一致していれば、その中で最も長い元の回答を採用します。
ここで必要なのは、有効な回答すべてが一致することであり、多数決で選ぶことではありません。3件の回答が2対1に分かれた場合も、統合が必要です。1件の呼び出しが失敗し、残りの2件が一致している場合は、そのまま採用できます。全体の一致が得られない限り、Qwenが元の質問と有効な候補すべてを踏まえて最終回答を生成します。
図2：Consensusエンジンの回答選択ロジック。
最終回答の選択後、QwenはLLM-as-a-Judgeとして、次の2種類のスコアを0–100で付けます。
● 意味的類似度：回答候補と最終回答が、主要な結論や重要な事実においてどの程度近いかを表します。
● Confidence Level：評価モデルが最終回答をどの程度支持するかについての評価です。
これらのスコアは回答間の関係を観察するためのものであり、最終回答の再選択や書き換えは行いません。
本記事では、Mac上でのローカル実験を前提としています。実行前に、次の条件を整えてください。
● Python 3.11以降をインストールし、以降のコマンドはmacOSのターミナルで実行します。
● Model Studioの利用を有効にし、シンガポールのエンドポイントに対応するAPI Keyを用意します。また、現在のワークスペースで、下表の4つのモデルを呼び出せることを確認します。
● Section 6に従って、プロジェクトのディレクトリ構成をすべて作成します。参考実装はFastAPI 0.104.1、Uvicorn 0.24.0、httpx 0.25.1、Pydantic 2.5.0に依存しており、OpenAI SDKへの追加の依存はありません。
モデル呼び出し時のパラメータを、参考として下表に示します。
役割 |
モデル設定 |
temperature |
max_tokens |
最初のラウンドの回答候補 |
deepseek-v4-pro-0813 |
0.7 |
1000 |
最初のラウンドの回答候補 |
glm-5.2 |
0.7 |
2048 |
最初のラウンドの回答候補 |
kimi-k3 |
0.6 |
1000 |
回答の統合 |
qwen3.8-max |
0.7 |
1600 |
意味的一致の確認と最終評価 |
qwen3.8-max |
0.0 |
600 |
まず、Section 6に従ってプロジェクトのディレクトリを作成し、各ファイルの内容をすべてコピーします。プロジェクトのルートディレクトリで、以下のコマンドを実行して仮想環境を作成します。
python3 -m venv .venvMacのターミナルで仮想環境を有効にします。
source .venv/bin/activate参考実装の依存パッケージをインストールします。
python -m pip install -r requirements.txt同じターミナル、同じプロジェクトのルートディレクトリで起動コマンドを実行し、入力内容が表示されないプロンプトで、ご自身のModel Studio API Keyを入力します。
python -c "import getpass, os, uvicorn; os.environ['MODELSTUDIO_API_KEY'] = getpass.getpass('Model Studio API Key: '); uvicorn.run('main:app', host='127.0.0.1', port=8000)"ブラウザでローカルアドレス127.0.0.1:8000/をHTTPで開くと、ページにアクセスできます。質問を送信すると、最終回答、各モデルの候補、処理経路、スコアを確認できます。候補の呼び出しが失敗した場合は、カードに対応する診断情報が表示されます。
以下の3つの事例は、エンジンの動作を説明するために、3回のデモでの入力、最終回答、判定経路を記録したものです。毎回同じ出力が得られるわけではありません。モデルの応答やスコアは変わる可能性があるため、実際の結果はその回のページ表示で確認してください。
入力した質問：「洗車に行きたいのですが、洗車店は50メートル先にあります。車で行くべきでしょうか、それとも歩いて行くべきでしょうか？」
Appの最終回答は、車を店まで持っていく必要があるため、洗車には車で行くというものでした。用品を買ったり、話をしたりするだけなら、歩いて行くこともできます。
3つの候補はいずれも「車を店まで持っていく必要がある」という重要な条件を捉えており、補足内容だけが異なっていました。そのため、全文一致や二次統合ではなく、意味的一致を確認してそのまま採用する経路が選ばれました。最終的に表示された本文は、GLMの候補と同じでした。この事例は、異なる表現でも同じ主要な結論を伝えられることを示しています。
図3：洗車の事例。
入力した質問：あなたは分岐器の切り替えを担当する鉄道作業員です。200人を乗せた列車が進んでおり、その先の橋が崩落しています。分岐器を切り替えなければ乗客が死亡します。切り替えれば乗客を救えますが、別の線路にいる1人の保守作業員が列車にひかれて死亡します。このとき、どのような行動を取るべきでしょうか？
Appの最終回答は、直ちに分岐器を切り替えて乗客を救い、同時に、可能な限り通信や拡声、汽笛で保守作業員に危険を避けるよう警告するというものでした。職業上の責任や緊急避難などの観点からも説明していました。
3つの候補はいずれも、分岐器を切り替えて保守作業員に警告する方向の回答でした。倫理面や操作面の論拠には違いがありましたが、同じく意味的一致を確認してそのまま採用する経路が選ばれ、最終的に表示された本文はGLMの候補と同じでした。この種の問題には真の意味で正しい結論がなく、AIの論理的な判断能力がより厳しく問われます。
図4：トロッコ問題の事例。
入力した質問：「ランダムな3桁の数値を1つ生成してください。」
Appの最終回答は482でした。
この質問の目的は、最初のラウンドの3つのモデルに異なる回答を強制的に出させ、統合判定の経路を実行させることでテストを完了することです。
図5：ランダムな数値の事例。
今回の実験では、複数モデルによる並列回答、一致性の判定、回答の採用または統合、最終評価、ページへの表示を一つの動作するシステムにつなげ、この考え方をエンドツーエンドで実践できることを確認しました。洗車とトロッコ問題の事例では、意味的一致を確認してそのまま採用する経路を示し、ランダムな数値の事例では統合の経路を示しました。元の候補と処理経路を保持することで、最終回答がどのように作られたかも観察できるようになりました。
これを出発点として、読者の皆さんはご自身の業務上の課題を実験に持ち込み、異なるモデルの組み合わせや判定基準、事実確認の方法を試すことができます。目的に応じた改善と検証を重ねれば、このプロトタイプは、一度の回答デモにとどまらず、業務上の課題を実際に解決する仕組みへと発展する可能性があります。今後は、固定の問題セット、単一モデルのベースライン、繰り返し実行、人による確認を用いて、より詳しい分析とテストの改善を進められます。
必要に応じて、以下の外国語版をご参照ください。
以上、お読みいただきありがとうございます。本記事が皆さんの仕事や学習に役立つ情報をお届けできれば幸いです。ありがとうございました。🙏
consensus-agent/
├── requirements.txt
├── main.py
├── index.html
└── static/
├── css/
│ └── style.css
└── js/
└── main.jsルートディレクトリconsensus-agent/と、その下にstatic/css/、static/js/のサブディレクトリを作成します。requirements.txt → main.py → index.html → static/css/style.css → static/js/main.jsの順に、以下の5つの完全なファイルを、対応する相対パスにコピーしてください。起動時の作業ディレクトリはconsensus-agent/です。実行手順はSection 4を参照してください。埋め込まれたコードには実際のAPI Keyは含まれておらず、実行時にMODELSTUDIO_API_KEYを通じて注入します。
fastapi==0.104.1
uvicorn==0.24.0
httpx==0.25.1
pydantic==2.5.0"""
Multi-LLM Consensus Agent Backend Server
All four models are served by Alibaba Cloud ModelStudio (DashScope) through its
OpenAI-compatible endpoint. Three panel models answer the question once each
from different vendors, then qwen3.8-max synthesises the consensus answer and
scores every panel answer against it.
"""
import asyncio
import json
import logging
import os
import re
from dataclasses import dataclass
from difflib import SequenceMatcher
from typing import Dict, List, Optional, Tuple
import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
# HTTPX INFO logs include complete request URLs, which carry credentials.
logging.basicConfig(level=logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("httpcore").setLevel(logging.WARNING)
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ModelSpec:
"""Everything the panel models disagree about, in one place.
temperature: kimi-k3 rejects 0.7 and accepts only {0.0, 0.6, 1.0}.
max_tokens: glm-5.2 truncates mid-answer at 1000, so it needs 2048.
"""
name: str
temperature: float
max_tokens: int
class Config:
# Environment only, so this file is safe to copy or share as it stands.
API_KEY = os.getenv("MODELSTUDIO_API_KEY", "")
API_URL = (
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions"
)
# Per-call budget: kimi-k3 alone measured 27.99-30.86s, and the synthesis
# step has to read three full answers before it writes one.
TIMEOUT = 90
# Round 1: one answer each from three different vendors.
PANEL: Tuple[ModelSpec, ...] = (
ModelSpec("deepseek-v4-pro-0813", temperature=0.7, max_tokens=1000),
ModelSpec("glm-5.2", temperature=0.7, max_tokens=2048),
ModelSpec("kimi-k3", temperature=0.6, max_tokens=1000),
)
# Round 2: disagreement is resolved by this model.
SYNTHESIS = ModelSpec("qwen3.8-max", temperature=0.7, max_tokens=1600)
# Round 3: scores the answers. Temperature 0 measured stable across eight
# runs; the reply is one short JSON object, so the budget stays small.
JUDGE = ModelSpec("qwen3.8-max", temperature=0.0, max_tokens=600)
# Fail at import time: an empty key would otherwise surface as a 401 on the first
# request, which reads like a network fault rather than a missing setting.
if not Config.API_KEY:
raise RuntimeError(
"MODELSTUDIO_API_KEY is not set. Export it before starting the app, "
"for example: export MODELSTUDIO_API_KEY=your-modelstudio-key"
)
app = FastAPI(
title="Multi-LLM Consensus Agent API",
description="Three ModelStudio panel models answer, qwen3.8-max synthesises",
version="2.0.0",
)
class QueryRequest(BaseModel):
query: str
class ModelAnswer(BaseModel):
ok: bool
answer: str = ""
error: str = ""
class QueryResponse(BaseModel):
final_answer: str
responses: Dict[str, ModelAnswer]
consensus_info: Dict
def safe_diagnostic(value, *extra_keys) -> str:
"""Redact configured and instance credentials before limiting output."""
text = str(value)
keys = (Config.API_KEY,) + extra_keys
for key in sorted((key for key in keys if key), key=len, reverse=True):
text = text.replace(key, "<REDACTED>")
text = re.sub(
r"(?i)([?&](?:key|api_key|apikey|access_token)=)[^&\s\"'<>]+",
r"\1<REDACTED>",
text,
)
text = re.sub(r"(?i)\bBearer\s+[^\s\"'<>]+", "Bearer <REDACTED>", text)
# Keep remote messages on one log line and avoid unbounded error output.
return " ".join(text.split())[:1500]
class ModelStudioClient:
"""One OpenAI-compatible client; ModelStudio hosts every model we use."""
def __init__(self, spec: ModelSpec):
self.spec = spec
self.name = spec.name
def error_text(self, exc: Exception) -> str:
"""Return an actionable diagnostic without exposing key material."""
if isinstance(exc, httpx.HTTPStatusError):
response = exc.response
try:
body = response.json()
except ValueError:
body = {}
body = body if isinstance(body, dict) else {}
error = body.get("error")
error = error if isinstance(error, dict) else {}
request_id = (
response.headers.get("x-request-id")
or body.get("request_id")
or "unavailable"
)
detail = (
f"HTTP {response.status_code}; model={self.name}; "
f"request_id={request_id}; "
f"code={error.get('code') or error.get('type') or 'unknown'}; "
f"message={error.get('message') or response.text or 'empty body'}"
)
elif isinstance(exc, httpx.TimeoutException):
detail = (
f"{type(exc).__name__}; model={self.name}; "
f"timeout={Config.TIMEOUT}s"
)
else:
detail = f"{type(exc).__name__}; model={self.name}; {exc}"
detail = safe_diagnostic(detail)
logger.error("%s API error: %s", safe_diagnostic(self.name), detail)
return detail
@staticmethod
def extract(response: httpx.Response) -> Tuple[str, str]:
"""Pull the answer out of an OpenAI-shaped response, rejecting empties."""
data = response.json()
if not isinstance(data, dict):
raise ValueError(f"HTTP {response.status_code}; invalid JSON object")
choices = data.get("choices")
if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict):
raise ValueError(f"HTTP {response.status_code}; missing choices[0]")
choice = choices[0]
message = choice.get("message")
message = message if isinstance(message, dict) else {}
finish_reason = str(choice.get("finish_reason"))
text = (message.get("content") or "").strip()
if not text:
# Thinking models spend max_tokens on reasoning_content first, so an
# HTTP 200 can still carry an empty answer. Say so explicitly.
spent = len(message.get("reasoning_content") or "")
raise ValueError(
f"HTTP {response.status_code}; empty content; "
f"finish_reason={finish_reason}; reasoning_characters={spent}"
)
return text, finish_reason
async def answer(self, prompt: str, json_mode: bool = False) -> ModelAnswer:
headers = {
"Authorization": f"Bearer {Config.API_KEY}",
"Content-Type": "application/json",
}
payload = {
"model": self.name,
"messages": [{"role": "user", "content": prompt}],
"temperature": self.spec.temperature,
"max_tokens": self.spec.max_tokens,
"enable_thinking": False,
"stream": False,
}
if json_mode:
payload["response_format"] = {"type": "json_object"}
try:
async with httpx.AsyncClient(timeout=Config.TIMEOUT) as client:
response = await client.post(Config.API_URL, headers=headers, json=payload)
response.raise_for_status()
text, finish_reason = self.extract(response)
except Exception as exc: # noqa: BLE001 - normalised into ModelAnswer
return ModelAnswer(ok=False, error=self.error_text(exc))
logger.info(
"%s API success: HTTP %s; characters=%d; finish_reason=%s",
safe_diagnostic(self.name),
response.status_code,
len(text),
finish_reason,
)
return ModelAnswer(ok=True, answer=text)
def is_score(value) -> bool:
"""Accept a JSON number in 0-100. bool is an int subclass, so reject it first."""
return (
isinstance(value, (int, float))
and not isinstance(value, bool)
and 0 <= value <= 100
)
def normalise_answer(text: str) -> str:
"""Strip the differences that carry no meaning to an exact comparison.
Markdown emphasis, punctuation, case and whitespace are noise: "**Elephant**"
and "elephant" are the same answer.
"""
text = re.sub(r"[*_`#>]+", " ", text)
text = re.sub(r"[^\w\s]", " ", text)
return " ".join(text.lower().split())
def character_similarity(text1: str, text2: str) -> float:
"""Deterministic fallback score: character overlap, which ignores meaning.
Character overlap ranks a reworded but identical conclusion far below a
copied but contradictory one, so this only backs up :class:`Judge`.
"""
if not text1 or not text2:
return 0.0
clean1 = re.sub(r"\s+", " ", text1.strip().lower())
clean2 = re.sub(r"\s+", " ", text2.strip().lower())
return round(SequenceMatcher(None, clean1, clean2).ratio() * 100, 1)
class Judge:
"""Has the synthesis model score the panel answers and its own confidence.
Both numbers must survive a failed or malformed judge call, so every path
here returns a complete result and degrades to ``character_similarity``.
"""
def __init__(self):
self.client = ModelStudioClient(Config.JUDGE)
async def review(
self, query: str, final_answer: str, responses: Dict[str, ModelAnswer]
) -> Dict:
answered = {name: a.answer for name, a in responses.items() if a.ok}
failed = [name for name, a in responses.items() if not a.ok]
if not answered:
return {
"similarity_scores": {name: 0.0 for name in responses},
"confidence_level": 0,
"confidence_rationale": "No panel model returned an answer.",
"similarity_source": "character",
}
verdict = await self.client.answer(
self.build_prompt(query, final_answer, answered, failed), json_mode=True
)
if not verdict.ok:
return self.estimate(final_answer, responses, verdict.error)
parsed = self.parse(verdict.answer, list(answered))
if parsed is None:
return self.estimate(
final_answer,
responses,
f"unparseable judge output: {verdict.answer[:200]}",
)
scores, confidence, rationale = parsed
scores.update({name: 0.0 for name in failed})
return {
"similarity_scores": scores,
"confidence_level": confidence,
"confidence_rationale": rationale,
"similarity_source": "semantic",
}
@staticmethod
def build_prompt(
query: str, final_answer: str, answered: Dict[str, str], failed: List[str]
) -> str:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in answered.items())
template = ", ".join(f'"{name}": <int 0-100>' for name in answered)
absence = ""
if failed:
absence = (
"\nNOTE: these panel members failed and produced no answer: "
f"{', '.join(failed)}. Their absence must lower your confidence.\n"
)
return f"""A panel of models answered one question, then a final answer was produced.
QUESTION:
{query}
FINAL ANSWER:
{final_answer}
PANEL ANSWERS:
{sections}
{absence}
TASK 1 - semantic similarity of each panel answer to the FINAL ANSWER, 0-100:
- 90-100: identical conclusions and key facts, wording may differ
- 70-89: same conclusions, minor omissions or added detail
- 40-69: partially aligned, important differences in scope or emphasis
- 10-39: mostly different, or contradicts on a key point
- 0-9: unrelated, empty, or failed
Judge meaning; ignore wording, length and formatting.
TASK 2 - CONFIDENCE 0-100 in the FINAL ANSWER:
- 90-100: every panel supports the core conclusion; the answer is specific and actionable
- 75-89: a majority supports the core conclusion; minor gaps or unsupported additions
- 50-74: panels split on a key point, or parts of the question stay unanswered
- 25-49: the final answer conflicts with most panels, or makes questionable claims
- 0-24: unrelated, empty, or contradicts a clear consensus
The final answer is not automatically right because it is the final answer. Penalise
omissions and unsupported claims. Do not average the similarity scores.
Reply with JSON only, no prose:
{{"similarity": {{{template}}}, "confidence": <int 0-100>, "confidence_rationale": "<one sentence, max 30 words>"}}"""
@staticmethod
def parse(raw: str, models: List[str]):
"""Read the judge JSON, tolerating a code fence or surrounding prose."""
start, end = raw.find("{"), raw.rfind("}")
if start == -1 or end <= start:
return None
try:
data = json.loads(raw[start : end + 1])
except ValueError:
return None
if not isinstance(data, dict):
return None
raw_scores = data.get("similarity")
confidence = data.get("confidence")
if not isinstance(raw_scores, dict) or not is_score(confidence):
return None
# A partial answer is treated as malformed: mixing judged and estimated
# scores in one run would be impossible for a reader to interpret.
scores: Dict[str, float] = {}
for name in models:
value = raw_scores.get(name)
if not is_score(value):
return None
scores[name] = float(value)
rationale = " ".join(str(data.get("confidence_rationale") or "").split())
return scores, int(confidence), rationale[:300] or "No rationale returned."
async def agreement(self, query: str, answered: Dict[str, str]) -> Optional[Dict]:
"""Ask whether the panel reached one conclusion, wording aside.
Cheap answers look different only because they are longer or shorter; this
decides on meaning so an agreeing panel skips the synthesis call.
"""
if len(answered) < 2:
return None
verdict = await self.client.answer(
self.build_agreement_prompt(query, answered), json_mode=True
)
if not verdict.ok:
logger.warning(
"agreement check unavailable, falling back to synthesis: %s",
safe_diagnostic(verdict.error),
)
return None
return self.parse_agreement(verdict.answer, list(answered))
@staticmethod
def build_agreement_prompt(query: str, answered: Dict[str, str]) -> str:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in answered.items())
template = ", ".join(f'"{name}"' for name in answered)
return f"""A panel of models answered one question independently.
QUESTION:
{query}
PANEL ANSWERS:
{sections}
TASK - decide whether these answers reached the same core conclusion.
- "unanimous": every answer states the same conclusion, and none contradicts another
- "majority": two or more answers share the core conclusion, at least one differs
- "split": no two answers share the core conclusion
Wording, length, formatting, examples and extra detail do NOT count as disagreement.
Only a different conclusion, a contradiction of a key fact, or an answer that fails to
address the question counts.
Reply with JSON only, no prose:
{{"verdict": "unanimous|majority|split", "agreeing": [{template}], "conclusion": "<the shared conclusion in one sentence>"}}"""
@staticmethod
def parse_agreement(raw: str, models: List[str]) -> Optional[Dict]:
"""Derive the agreeing group, or an empty one when the reply is odd.
Empty means "no shared conclusion", which makes the caller synthesise.
"""
start, end = raw.find("{"), raw.rfind("}")
if start == -1 or end <= start:
return None
try:
data = json.loads(raw[start : end + 1])
except ValueError:
return None
if not isinstance(data, dict):
return None
verdict = data.get("verdict")
members = data.get("agreeing")
if verdict not in {"unanimous", "majority", "split"}:
return None
if not isinstance(members, list):
return None
# Keep panel order so the group is reproducible across runs.
agreeing = [name for name in models if name in members]
if verdict == "split" or len(agreeing) < 2:
return {"agreeing": [], "conclusion": ""}
conclusion = " ".join(str(data.get("conclusion") or "").split())[:300]
return {"agreeing": agreeing, "conclusion": conclusion}
@staticmethod
def estimate(
final_answer: str, responses: Dict[str, ModelAnswer], error: str
) -> Dict:
"""Degraded result: similarity survives, confidence does not.
Only the model can rate trust, and a character-overlap number would sit
near zero even for an excellent answer, so no substitute is invented.
"""
scores = {
name: character_similarity(final_answer, a.answer) if a.ok else 0.0
for name, a in responses.items()
}
reason = safe_diagnostic(error).rstrip(".")
return {
"similarity_scores": scores,
"confidence_level": None,
"confidence_rationale": (
f"Confidence not scored because the judge call failed: {reason}. "
"The similarity figures below are character overlap, not meaning."
),
"similarity_source": "character",
}
class ConsensusEngine:
def __init__(self):
self.panel = [ModelStudioClient(spec) for spec in Config.PANEL]
self.synthesizer = ModelStudioClient(Config.SYNTHESIS)
self.judge = Judge()
async def get_consensus_response(self, query: str) -> QueryResponse:
logger.info("Processing query; characters=%d", len(query))
answers = await asyncio.gather(*(client.answer(query) for client in self.panel))
responses = {
client.name: answer for client, answer in zip(self.panel, answers)
}
final_answer, consensus_info = await self.build_consensus(query, responses)
# Scoring runs last: every score is a comparison against the final text.
consensus_info.update(await self.judge.review(query, final_answer, responses))
return QueryResponse(
final_answer=final_answer,
responses=responses,
consensus_info=consensus_info,
)
async def build_consensus(
self, query: str, responses: Dict[str, ModelAnswer]
) -> Tuple[str, Dict]:
valid = {name: a.answer for name, a in responses.items() if a.ok}
total = len(responses)
# Every branch below sets the status, so no default is needed.
info: Dict = {}
if not valid:
info["consistency_status"] = "ALL_FAILED"
info["process_description"] = (
"Every panel model failed; see the individual errors"
)
return "No panel model returned an answer.", info
# Free path: answers that are identical once wording noise is removed.
# Unanimity is required: a 2-1 split goes to synthesis so that the dissenting
# view is weighed instead of being outvoted by a character match.
groups: Dict[str, List[str]] = {}
for name, text in valid.items():
groups.setdefault(normalise_answer(text), []).append(name)
_, winner_names = max(groups.items(), key=lambda item: len(item[1]))
if len(valid) >= 2 and len(winner_names) == len(valid):
info["consistency_status"] = "MAJORITY"
info["agreement_method"] = "exact"
info["process_description"] = (
f"Identical answers: all {len(valid)}/{total} responding models "
f"returned the same text ({', '.join(winner_names)})"
)
return valid[winner_names[0]], info
# Paraphrased answers never match textually, so ask the judge whether they
# reached the same conclusion before paying for a synthesis call. Same rule:
# only a unanimous conclusion skips synthesis.
verdict = await self.judge.agreement(query, valid)
if verdict and len(valid) >= 2 and len(verdict["agreeing"]) == len(valid):
agreeing = verdict["agreeing"]
info["consistency_status"] = "MAJORITY"
info["agreement_method"] = "semantic"
info["process_description"] = (
f"Same conclusion in different words: all {len(valid)}/{total} "
f"responding models agree ({', '.join(agreeing)}); showing the most "
"complete answer"
)
if verdict["conclusion"]:
info["shared_conclusion"] = verdict["conclusion"]
return max((valid[name] for name in agreeing), key=len), info
synthesis = await self.synthesise(query, valid)
if synthesis.ok:
info["consistency_status"] = "SYNTHESISED"
info["process_description"] = (
"Panel did not reach unanimity; "
f"{Config.SYNTHESIS.name} merged the responses"
)
return synthesis.answer, info
# Never present a panel answer as if it were a consensus.
info["consistency_status"] = "SYNTHESIS_FAILED"
info["process_description"] = (
"Synthesis failed; showing the most complete panel answer instead. "
f"error: {synthesis.error}"
)
return max(valid.values(), key=len), info
async def synthesise(self, query: str, valid: Dict[str, str]) -> ModelAnswer:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in valid.items())
# The question must be repeated: short answers like "290" carry no context of
# their own, and without it the synthesis turns into a meta-commentary.
prompt = (
"Several AI models answered the question below. Write one comprehensive, "
"accurate answer that keeps the best elements of each and resolves any "
"contradictions. Answer the question directly; never describe the models "
"or comment on the answers themselves.\n\n"
f"QUESTION:\n{query}\n\n"
f"PANEL ANSWERS:\n{sections}"
)
return await self.synthesizer.answer(prompt)
consensus_engine = ConsensusEngine()
app.mount("/static", StaticFiles(directory="static"), name="static")
@app.get("/", response_class=HTMLResponse)
async def read_root():
with open("index.html", "r", encoding="utf-8") as file:
# The shell must revalidate, otherwise a stale asset list pins old JS.
return HTMLResponse(file.read(), headers={"Cache-Control": "no-cache"})
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"panel": [spec.name for spec in Config.PANEL],
"synthesis": Config.SYNTHESIS.name,
"judge": Config.JUDGE.name,
}
@app.post("/api/consensus", response_model=QueryResponse)
async def get_consensus_answer(request: QueryRequest):
if not request.query.strip():
raise HTTPException(status_code=400, detail="Query cannot be empty")
try:
return await consensus_engine.get_consensus_response(request.query)
except Exception as exc: # noqa: BLE001 - never echo arbitrary exception text
detail = f"Unexpected consensus error: {type(exc).__name__}"
logger.error("Error processing consensus request: %s", detail)
raise HTTPException(status_code=500, detail=detail) from None
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000, log_level="info")<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Multi-LLM Consensus Agent</title>
<!-- Bump ?v= whenever these files change, or browsers keep serving the old copy. -->
<link rel="stylesheet" href="static/css/style.css?v=6">
</head>
<body>
<div class="container">
<header>
<h1>Multi-LLM Consensus Agent</h1>
<p>Get reliable answers through consensus of multiple LLMs</p>
</header>
<main>
<div class="query-section">
<div class="input-group">
<label for="user-query">Enter your question:</label>
<textarea id="user-query" placeholder="Ask anything..."></textarea>
</div>
<button id="submit-query" class="submit-btn">Get Consensus Answer</button>
</div>
<div class="loading-section" id="loading-section" style="display: none;">
<div class="spinner"></div>
<p>Querying three panel models, then synthesising and scoring the result...</p>
<p class="loading-hint">Three rounds run in sequence, so expect 40-60 seconds.</p>
</div>
<div class="results-section" id="results-section" style="display: none;">
<h2>Consensus Results</h2>
<div class="consensus-answer">
<h3>Final Answer</h3>
<div id="final-answer" class="answer-content"></div>
</div>
<div class="individual-responses">
<h3>Individual Model Responses</h3>
<!-- Cards are rendered from the API response, one per model. -->
<div id="response-grid" class="response-grid"></div>
</div>
<div class="consensus-info">
<h3>Consensus Process</h3>
<div id="consensus-details"></div>
</div>
</div>
</main>
<footer>
<p>Multi-LLM Consensus Agent System</p>
</footer>
</div>
<script src="static/js/main.js?v=6"></script>
</body>
</html>/* Global Styles */
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
line-height: 1.6;
color: #333;
background-color: #f5f5f5;
}
.container {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
/* Header Styles */
header {
text-align: center;
margin-bottom: 40px;
padding: 30px 0;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border-radius: 10px;
box-shadow: 0 4px 15px rgba(0,0,0,0.1);
}
header h1 {
font-size: 2.5rem;
margin-bottom: 10px;
}
header p {
font-size: 1.2rem;
opacity: 0.9;
}
/* Main Content */
main {
background: white;
border-radius: 10px;
padding: 30px;
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
}
/* Query Section */
.query-section {
margin-bottom: 30px;
}
.input-group {
margin-bottom: 20px;
}
.input-group label {
display: block;
margin-bottom: 10px;
font-weight: 600;
color: #555;
}
#user-query {
width: 100%;
min-height: 120px;
padding: 15px;
border: 2px solid #ddd;
border-radius: 8px;
font-size: 16px;
font-family: inherit;
resize: vertical;
transition: border-color 0.3s;
}
#user-query:focus {
outline: none;
border-color: #667eea;
box-shadow: 0 0 0 3px rgba(102, 126, 234, 0.1);
}
.submit-btn {
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border: none;
padding: 15px 30px;
font-size: 18px;
border-radius: 8px;
cursor: pointer;
transition: transform 0.2s, box-shadow 0.2s;
font-weight: 600;
}
.submit-btn:hover {
transform: translateY(-2px);
box-shadow: 0 5px 15px rgba(102, 126, 234, 0.4);
}
.submit-btn:active {
transform: translateY(0);
}
.submit-btn:disabled {
background: #ccc;
cursor: not-allowed;
transform: none;
box-shadow: none;
}
/* Loading Section */
.loading-section {
text-align: center;
padding: 40px;
}
.loading-hint {
color: #6c757d;
font-size: 13px;
}
.spinner {
width: 50px;
height: 50px;
border: 5px solid #f3f3f3;
border-top: 5px solid #667eea;
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 20px;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
/* Results Section */
.results-section {
margin-top: 30px;
}
.results-section h2 {
color: #333;
margin-bottom: 25px;
text-align: center;
font-size: 2rem;
}
/* Consensus Answer */
.consensus-answer {
background: linear-gradient(135deg, #f0f8ff 0%, #e6f3ff 100%);
border: 2px solid #667eea;
border-radius: 10px;
padding: 25px;
margin-bottom: 30px;
}
.consensus-answer h3 {
color: #667eea;
margin-bottom: 15px;
font-size: 1.5rem;
}
.answer-content {
font-size: 18px;
line-height: 1.7;
color: #444;
}
/* Individual Responses */
.individual-responses {
margin-bottom: 30px;
}
.individual-responses h3 {
color: #333;
margin-bottom: 20px;
font-size: 1.5rem;
}
.response-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 20px;
}
.response-card {
background: #f9f9f9;
border: 1px solid #ddd;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 5px rgba(0,0,0,0.05);
transition: transform 0.2s;
}
.response-card:hover {
transform: translateY(-3px);
box-shadow: 0 5px 15px rgba(0,0,0,0.1);
}
.response-card h4 {
color: #555;
margin-bottom: 15px;
padding-bottom: 10px;
border-bottom: 2px solid #eee;
font-size: 1.3rem;
}
.response-content {
font-size: 16px;
line-height: 1.6;
color: #666;
}
/* Consensus Info */
.consensus-info {
background: #f8f9fa;
border: 1px solid #dee2e6;
border-radius: 8px;
padding: 20px;
}
.consensus-info h3 {
color: #495057;
margin-bottom: 15px;
font-size: 1.3rem;
}
/* Footer */
footer {
text-align: center;
margin-top: 40px;
padding: 20px;
color: #666;
font-size: 14px;
}
/* Responsive Design */
@media (max-width: 768px) {
.container {
padding: 10px;
}
header h1 {
font-size: 2rem;
}
.response-grid {
grid-template-columns: 1fr;
}
main {
padding: 20px;
}
}
/* Status Indicators; only the four statuses the backend can emit */
.status-majority {
color: #ffc107;
font-weight: 600;
}
.status-synthesised {
color: #667eea;
font-weight: 600;
}
.status-synthesis_failed,
.status-all_failed {
color: #dc3545;
font-weight: 600;
}
/* Per-model failure detail, shown instead of hiding the diagnostic */
.model-error {
background-color: #f8d7da;
border: 1px solid #f5c6cb;
border-radius: 6px;
color: #721c24;
font-size: 14px;
padding: 12px;
word-break: break-word;
}
/* Confidence verdict plus the judge's one-line justification */
.confidence {
margin-top: 12px;
font-size: 15px;
}
.confidence-rationale {
color: #6c757d;
font-size: 14px;
font-style: italic;
margin-top: 6px;
padding-left: 12px;
border-left: 3px solid #dee2e6;
}
/* Similarity Score Styles */
.similarity-score {
margin-top: 15px;
padding: 12px;
border-radius: 6px;
font-size: 14px;
}
.high-similarity {
background-color: #d4edda;
border: 1px solid #c3e6cb;
color: #155724;
}
.medium-similarity {
background-color: #fff3cd;
border: 1px solid #ffeaa7;
color: #856404;
}
.low-similarity {
background-color: #f8d7da;
border: 1px solid #f5c6cb;
color: #721c24;
}
.similarity-bar {
width: 100%;
height: 8px;
background-color: #e9ecef;
border-radius: 4px;
margin-top: 8px;
overflow: hidden;
}
.similarity-fill {
height: 100%;
background: linear-gradient(90deg, #28a745, #20c997);
border-radius: 4px;
transition: width 0.3s ease;
}
.no-response {
color: #6c757d;
font-style: italic;
}// Multi-LLM Consensus Agent Frontend
class ConsensusAgent {
constructor() {
this.apiUrl = '/api/consensus';
this.initializeEventListeners();
}
initializeEventListeners() {
const submitBtn = document.getElementById('submit-query');
const queryInput = document.getElementById('user-query');
submitBtn.addEventListener('click', () => this.handleSubmit());
queryInput.addEventListener('keypress', (e) => {
if (e.key === 'Enter' && e.ctrlKey) {
this.handleSubmit();
}
});
}
async handleSubmit() {
const query = document.getElementById('user-query').value.trim();
if (!query) {
alert('Please enter a question');
return;
}
this.showLoading();
this.hideResults();
try {
const response = await this.queryConsensusAgent(query);
this.displayResults(response);
} catch (error) {
console.error('Error:', error);
this.showError(error.message);
} finally {
this.hideLoading();
}
}
async queryConsensusAgent(query) {
const response = await fetch(this.apiUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ query: query })
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return await response.json();
}
showLoading() {
document.getElementById('loading-section').style.display = 'block';
document.getElementById('submit-query').disabled = true;
}
hideLoading() {
document.getElementById('loading-section').style.display = 'none';
document.getElementById('submit-query').disabled = false;
}
showResults() {
document.getElementById('results-section').style.display = 'block';
}
hideResults() {
document.getElementById('results-section').style.display = 'none';
}
displayResults(data) {
const info = data.consensus_info || {};
const similarityScores = info.similarity_scores || {};
// "semantic" means the judge model scored it; anything else is the
// character-overlap estimate used when the judge call failed.
const similaritySource = info.similarity_source || 'character';
document.getElementById('final-answer').innerHTML = this.formatText(data.final_answer);
// One card per model the backend actually answered with, so the model
// lineup lives only in Config.PANEL.
const grid = document.getElementById('response-grid');
grid.innerHTML = Object.entries(data.responses || {})
.map(([model, result]) =>
this.renderModelCard(model, result, similarityScores[model], similaritySource))
.join('');
document.getElementById('consensus-details').innerHTML =
this.formatConsensusDetails(info);
this.showResults();
}
renderModelCard(model, result, similarityScore, similaritySource) {
let body;
if (result.ok) {
body = this.formatText(result.answer)
+ this.renderSimilarity(similarityScore, similaritySource);
} else {
// Surface the provider's diagnostic: hiding it made every failure
// look like a network problem.
body = `<p class="model-error">${this.escapeHtml(result.error || 'Request failed')}</p>`;
}
return `<div class="response-card">
<h4>${this.escapeHtml(model)}</h4>
<div class="response-content">${body}</div>
</div>`;
}
renderSimilarity(similarityScore, similaritySource) {
if (similarityScore === undefined || similarityScore === null) {
return '';
}
const semantic = similaritySource === 'semantic';
const label = semantic
? 'Semantic similarity to final answer'
: 'Similarity to final answer (estimated)';
const explanation = semantic
? 'Scored by the judge model against a rubric, not by character overlap.'
: 'Character overlap only: the semantic scoring call was unavailable.';
const scoreClass = similarityScore >= 80 ? 'high-similarity' :
similarityScore >= 60 ? 'medium-similarity' : 'low-similarity';
return `<div class="similarity-score ${scoreClass}" title="${this.escapeHtml(explanation)}">
<strong>${label}:</strong> ${similarityScore}%
<div class="similarity-bar">
<div class="similarity-fill" style="width: ${similarityScore}%"></div>
</div>
</div>`;
}
formatText(text) {
if (!text) return '<p class="no-response">No content returned</p>';
return `<p>${this.escapeHtml(text).replace(/\n/g, '<br>')}</p>`;
}
formatConsensusDetails(info) {
if (!info) return '<p>No consensus details available</p>';
const status = String(info.consistency_status || 'unknown');
let html = '<div class="consensus-details">';
html += `<div class="status-info">
<strong>Consistency Status:</strong>
<span class="status-${this.escapeHtml(status.toLowerCase())}">${this.escapeHtml(status)}</span>
</div>`;
if (info.process_description) {
html += `<div class="process-description">
<strong>Process:</strong> ${this.escapeHtml(info.process_description)}
</div>`;
}
// 0 is a real verdict, null means the judge call failed; never hide either.
const confidence = info.confidence_level === null || info.confidence_level === undefined
? 'unavailable'
: `${info.confidence_level}%`;
html += `<div class="confidence">
<strong>Confidence Level:</strong> ${confidence}
</div>`;
if (info.confidence_rationale) {
html += `<div class="confidence-rationale">${this.escapeHtml(info.confidence_rationale)}</div>`;
}
html += '</div>';
return html;
}
showError(message) {
const errorDiv = document.createElement('div');
errorDiv.className = 'error-message';
errorDiv.innerHTML = `<p><strong>Error:</strong> ${this.escapeHtml(message)}</p>`;
errorDiv.style.cssText = `
background: #f8d7da;
color: #721c24;
border: 1px solid #f5c6cb;
border-radius: 5px;
padding: 15px;
margin: 20px 0;
`;
const main = document.querySelector('main');
main.insertBefore(errorDiv, main.querySelector('.query-section').nextSibling);
// Remove error after 5 seconds
setTimeout(() => {
if (errorDiv.parentNode) {
errorDiv.parentNode.removeChild(errorDiv);
}
}, 5000);
}
escapeHtml(unsafe) {
return String(unsafe)
.replace(/&/g, '&')
.replace(/</g, '<')
.replace(/>/g, '>')
.replace(/"/g, '"')
.replace(/'/g, ''');
}
}
// Initialize the application
document.addEventListener('DOMContentLoaded', () => {
new ConsensusAgent();
});Alibaba Cloud Native Community - February 26, 2025
Regional Content Hub - July 7, 2025
Regional Content Hub - August 5, 2024
Regional Content Hub - February 26, 2024
Regional Content Hub - March 8, 2024
Regional Content Hub - June 23, 2026
QwenWork
QwenWork is dedicated to helping employees strengthen their professional competitiveness in the AI era and to enabling enterprises to improve organizational effectiveness.
Learn More
Qwen
Full-range, open-source, multimodal, and multi-functional
Learn More
AgentCore
AgentCore unifies agent construction, governance, and execution capabilities so enterprises can deploy diverse agents with confidence.
Learn More
Qoder
From Assisted to Autonomous — Adapting to Every Programming Need
Learn MoreMore Posts by Wei Ying
