This experiment grew out of a discussion in a technical chat group: “I want to get my car washed. The car wash is only 50 meters away. Should I drive there or walk?” A distance of 50 meters is indeed very short, but the task is to wash the car, which means bringing it to the car wash. An analysis focused only on distance, fuel consumption, and the convenience of walking can easily miss the condition that actually determines the answer. At the time, colleagues tested different LLM models and obtained different conclusions.
This got me thinking: every model has its own response characteristics, and any model can overlook conditions, make reasoning errors, or hallucinate. Rather than choosing the “best model” for a particular question, could we have several models answer independently and then compare their conclusions and disagreements? If one model overlooks a key condition and another notices it, comparing their answers may provide an opportunity to correct the error.
I therefore built an end-to-end Web prototype: it sends the same question to three models in parallel, preserves their original responses, and uses text matching and semantic consistency to decide whether to adopt a candidate directly or perform a second-stage synthesis. A model-based review then shows how the candidates relate to the final answer. This is one implementation of a Multi-model Ensemble. This article covers the call chain, three demonstration cases, and the implementation’s boundaries, and concludes with complete reference code so readers can continue with their own experiments.
Disclaimer: This approach is intended for learning in a test environment. It is not guaranteed to align with production best practices without modification.
The prototype consists of a browser frontend, a FastAPI backend, and ConsensusEngine.
● The browser submits questions and displays results.
● The backend receives requests.
● The engine coordinates responses from multiple models.
● DeepSeek, GLM, and Kimi provide the first-round candidates.
● Qwen handles consistency assessment, answer synthesis, and final review.
● All four models are accessed through Model Studio.
Figure 1: Architecture of the multi-model consensus-based answering system.
After the browser submits a question, ConsensusEngine calls three models in parallel to obtain candidate answers and then produces the final answer. Qwen subsequently reviews the final answer, and the review results are returned to the page alongside the original candidates. The same Qwen model handles consistency assessment, synthesis, and scoring.
The engine first checks how many of the three models successfully returned answers. If there are no valid responses, it returns a failure message and makes no further model calls. If there is only one valid response, it does not treat that as consensus; instead, it sends the question and that candidate directly to Qwen for synthesis, without checking semantic consistency. When there are at least two valid responses, it checks the following in order:
● Whether the full texts match: If all valid responses have identical full texts after differences such as formatting and punctuation are ignored, the first original response is adopted directly.
● Whether the core conclusions agree: If the full texts differ, Qwen compares the original question and candidate responses. If the core conclusions of all valid responses agree, the longest original response is adopted.
The requirement here is agreement among all valid responses, not a majority vote. A 2-to-1 split among three responses still requires synthesis; if one call fails and the other two responses agree, a candidate can be adopted directly. Whenever unanimous agreement has not been established, Qwen generates the final answer using the original question and all valid candidates.
Figure 2: The Consensus engine’s answer selection logic.
Once the final answer has been selected, Qwen acts as an LLM-as-a-Judge to provide two types of scores on a 0–100 scale:
● Semantic similarity: How closely a candidate response matches the final answer in its core conclusion and key facts.
● Confidence Level: The reviewing model’s assessment of how strongly it supports the final answer.
These scores help illustrate the relationships between answers; they do not reselect or rewrite the final answer.
This article assumes a local experiment on a Mac. Prepare the following before running it:
● Install Python 3.11 or later, and run the subsequent commands in the macOS terminal.
● Activate Model Studio, prepare an API Key that matches the Singapore endpoint, and confirm that the current workspace can call the four models in the table below.
● Set up the complete project directory as described in Section 6. The reference implementation depends on FastAPI 0.104.1, Uvicorn 0.24.0, httpx 0.25.1, and Pydantic 2.5.0, with no additional dependency on the OpenAI SDK.
The model call parameters in the table below are provided for reference:
Role |
Model configuration |
temperature |
max_tokens |
First-round candidates |
deepseek-v4-pro-0813 |
0.7 |
1000 |
First-round candidates |
glm-5.2 |
0.7 |
2048 |
First-round candidates |
kimi-k3 |
0.6 |
1000 |
Answer synthesis |
qwen3.8-max |
0.7 |
1600 |
Semantic consistency check and final review |
qwen3.8-max |
0.0 |
600 |
First, set up the project directory and copy the complete files as described in Section 6. Run the following command in the project root directory to create a virtual environment.
python3 -m venv .venvActivate the virtual environment in the Mac terminal.
source .venv/bin/activateInstall the dependencies for the reference implementation.
python -m pip install -r requirements.txtRun the startup command in the same terminal and project root directory, and enter your own Model Studio API Key at the prompt, where input is hidden.
python -c "import getpass, os, uvicorn; os.environ['MODELSTUDIO_API_KEY'] = getpass.getpass('Model Studio API Key: '); uvicorn.run('main:app', host='127.0.0.1', port=8000)"Open the local address 127.0.0.1:8000/ in a browser using HTTP to access the page. After submitting a question, you can view the final answer, each model’s candidate, the processing path, and the scores. If a candidate call fails, its card displays the corresponding diagnostics.
The following three cases record the inputs, final answers, and decision paths from three demonstrations to illustrate the engine’s behavior. They are not fixed outputs for every run; model responses and scores may vary, so the actual results should be taken from the page for that run.
Question entered: “I want to get my car washed. The car wash is 50 meters away. Should I drive there or walk?”
The App’s final answer was to drive to the car wash because the car needs to be brought to the shop; if the visit is only to buy supplies or chat, walking is an option.
All three candidates identified the key condition that “the car needs to be brought to the shop,” differing only in their additional details. The engine therefore followed the path of direct adoption after establishing semantic consistency, rather than full-text matching or second-stage synthesis. The final displayed text was identical to the GLM candidate. This case shows that different wording can express the same core conclusion.
Figure 3: The car wash case.
Question entered: As a railway worker responsible for operating a track switch, you face a train carrying 200 people and a collapsed bridge ahead. If you do not switch tracks, the passengers will die; switching will save them, but a maintenance worker on the other track will be struck and killed. What action should you take?
The App’s final answer was to switch tracks immediately to save the passengers, while doing everything possible to warn the maintenance worker to move out of danger through communication, loudspeakers, or a horn. It elaborated on the decision from perspectives such as professional responsibility and the necessity of acting in an emergency.
All three candidates favored switching tracks and warning the maintenance worker, though their ethical and operational arguments differed. The engine therefore again followed the path of direct adoption after establishing semantic consistency, and the final displayed text was identical to the GLM candidate. Questions of this kind have no truly correct conclusion and pose an even greater challenge to AI’s logical judgment.
Figure 4: The trolley dilemma case.
Question entered: “Generate a random 3-digit number for me.”
The App’s final answer was 482.
The purpose of this question was to force the three models to give different answers in the first round, triggering the synthesis decision path to complete the test.
Figure 5: The random number case.
This experiment combined parallel responses from multiple models, consistency assessment, answer adoption or synthesis, final review, and page display into a working system, demonstrating the practical feasibility of implementing this idea end to end. The car wash and trolley cases demonstrated direct adoption after establishing semantic consistency, while the random number case demonstrated the synthesis path. Preserving the original candidates and processing paths also made the process of forming the final answer observable.
Using this as a starting point, readers can bring their own business questions into the experiment and try different model combinations, decision criteria, and fact-checking methods. With targeted improvements and validation, this prototype has the potential to evolve into a solution that genuinely addresses business pain points, rather than remaining just an answering demonstration. Future work can use a fixed question set, single-model baselines, repeated runs, and manual checks to support more detailed analysis and improvements to testing.
Please refer to the following other-language versions as needed:
● Japanese version: 「洗車問題」から複数モデルの合意に基づく回答システムへ：Model Studio連携のエンドツーエンド実践
That concludes the article. Thank you all for reading; I hope it provides valuable information for your work and learning. Thank you. 🙏
consensus-agent/
├── requirements.txt
├── main.py
├── index.html
└── static/
├── css/
│ └── style.css
└── js/
└── main.jsCreate the root directory consensus-agent/ and the subdirectories static/css/ and static/js/. Copy the five complete files below to their corresponding relative paths in the order requirements.txt → main.py → index.html → static/css/style.css → static/js/main.js. The working directory for startup is consensus-agent/; see Section 4 for the steps to run the project. The embedded code contains no real API Key; it is injected at runtime through MODELSTUDIO_API_KEY.
fastapi==0.104.1
uvicorn==0.24.0
httpx==0.25.1
pydantic==2.5.0"""
Multi-LLM Consensus Agent Backend Server
All four models are served by Alibaba Cloud ModelStudio (DashScope) through its
OpenAI-compatible endpoint. Three panel models answer the question once each
from different vendors, then qwen3.8-max synthesises the consensus answer and
scores every panel answer against it.
"""
import asyncio
import json
import logging
import os
import re
from dataclasses import dataclass
from difflib import SequenceMatcher
from typing import Dict, List, Optional, Tuple
import httpx
from fastapi import FastAPI, HTTPException
from fastapi.responses import HTMLResponse
from fastapi.staticfiles import StaticFiles
from pydantic import BaseModel
# HTTPX INFO logs include complete request URLs, which carry credentials.
logging.basicConfig(level=logging.INFO)
logging.getLogger("httpx").setLevel(logging.WARNING)
logging.getLogger("httpcore").setLevel(logging.WARNING)
logger = logging.getLogger(__name__)
@dataclass(frozen=True)
class ModelSpec:
"""Everything the panel models disagree about, in one place.
temperature: kimi-k3 rejects 0.7 and accepts only {0.0, 0.6, 1.0}.
max_tokens: glm-5.2 truncates mid-answer at 1000, so it needs 2048.
"""
name: str
temperature: float
max_tokens: int
class Config:
# Environment only, so this file is safe to copy or share as it stands.
API_KEY = os.getenv("MODELSTUDIO_API_KEY", "")
API_URL = (
"https://dashscope-intl.aliyuncs.com/compatible-mode/v1/chat/completions"
)
# Per-call budget: kimi-k3 alone measured 27.99-30.86s, and the synthesis
# step has to read three full answers before it writes one.
TIMEOUT = 90
# Round 1: one answer each from three different vendors.
PANEL: Tuple[ModelSpec, ...] = (
ModelSpec("deepseek-v4-pro-0813", temperature=0.7, max_tokens=1000),
ModelSpec("glm-5.2", temperature=0.7, max_tokens=2048),
ModelSpec("kimi-k3", temperature=0.6, max_tokens=1000),
)
# Round 2: disagreement is resolved by this model.
SYNTHESIS = ModelSpec("qwen3.8-max", temperature=0.7, max_tokens=1600)
# Round 3: scores the answers. Temperature 0 measured stable across eight
# runs; the reply is one short JSON object, so the budget stays small.
JUDGE = ModelSpec("qwen3.8-max", temperature=0.0, max_tokens=600)
# Fail at import time: an empty key would otherwise surface as a 401 on the first
# request, which reads like a network fault rather than a missing setting.
if not Config.API_KEY:
raise RuntimeError(
"MODELSTUDIO_API_KEY is not set. Export it before starting the app, "
"for example: export MODELSTUDIO_API_KEY=your-modelstudio-key"
)
app = FastAPI(
title="Multi-LLM Consensus Agent API",
description="Three ModelStudio panel models answer, qwen3.8-max synthesises",
version="2.0.0",
)
class QueryRequest(BaseModel):
query: str
class ModelAnswer(BaseModel):
ok: bool
answer: str = ""
error: str = ""
class QueryResponse(BaseModel):
final_answer: str
responses: Dict[str, ModelAnswer]
consensus_info: Dict
def safe_diagnostic(value, *extra_keys) -> str:
"""Redact configured and instance credentials before limiting output."""
text = str(value)
keys = (Config.API_KEY,) + extra_keys
for key in sorted((key for key in keys if key), key=len, reverse=True):
text = text.replace(key, "<REDACTED>")
text = re.sub(
r"(?i)([?&](?:key|api_key|apikey|access_token)=)[^&\s\"'<>]+",
r"\1<REDACTED>",
text,
)
text = re.sub(r"(?i)\bBearer\s+[^\s\"'<>]+", "Bearer <REDACTED>", text)
# Keep remote messages on one log line and avoid unbounded error output.
return " ".join(text.split())[:1500]
class ModelStudioClient:
"""One OpenAI-compatible client; ModelStudio hosts every model we use."""
def __init__(self, spec: ModelSpec):
self.spec = spec
self.name = spec.name
def error_text(self, exc: Exception) -> str:
"""Return an actionable diagnostic without exposing key material."""
if isinstance(exc, httpx.HTTPStatusError):
response = exc.response
try:
body = response.json()
except ValueError:
body = {}
body = body if isinstance(body, dict) else {}
error = body.get("error")
error = error if isinstance(error, dict) else {}
request_id = (
response.headers.get("x-request-id")
or body.get("request_id")
or "unavailable"
)
detail = (
f"HTTP {response.status_code}; model={self.name}; "
f"request_id={request_id}; "
f"code={error.get('code') or error.get('type') or 'unknown'}; "
f"message={error.get('message') or response.text or 'empty body'}"
)
elif isinstance(exc, httpx.TimeoutException):
detail = (
f"{type(exc).__name__}; model={self.name}; "
f"timeout={Config.TIMEOUT}s"
)
else:
detail = f"{type(exc).__name__}; model={self.name}; {exc}"
detail = safe_diagnostic(detail)
logger.error("%s API error: %s", safe_diagnostic(self.name), detail)
return detail
@staticmethod
def extract(response: httpx.Response) -> Tuple[str, str]:
"""Pull the answer out of an OpenAI-shaped response, rejecting empties."""
data = response.json()
if not isinstance(data, dict):
raise ValueError(f"HTTP {response.status_code}; invalid JSON object")
choices = data.get("choices")
if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict):
raise ValueError(f"HTTP {response.status_code}; missing choices[0]")
choice = choices[0]
message = choice.get("message")
message = message if isinstance(message, dict) else {}
finish_reason = str(choice.get("finish_reason"))
text = (message.get("content") or "").strip()
if not text:
# Thinking models spend max_tokens on reasoning_content first, so an
# HTTP 200 can still carry an empty answer. Say so explicitly.
spent = len(message.get("reasoning_content") or "")
raise ValueError(
f"HTTP {response.status_code}; empty content; "
f"finish_reason={finish_reason}; reasoning_characters={spent}"
)
return text, finish_reason
async def answer(self, prompt: str, json_mode: bool = False) -> ModelAnswer:
headers = {
"Authorization": f"Bearer {Config.API_KEY}",
"Content-Type": "application/json",
}
payload = {
"model": self.name,
"messages": [{"role": "user", "content": prompt}],
"temperature": self.spec.temperature,
"max_tokens": self.spec.max_tokens,
"enable_thinking": False,
"stream": False,
}
if json_mode:
payload["response_format"] = {"type": "json_object"}
try:
async with httpx.AsyncClient(timeout=Config.TIMEOUT) as client:
response = await client.post(Config.API_URL, headers=headers, json=payload)
response.raise_for_status()
text, finish_reason = self.extract(response)
except Exception as exc: # noqa: BLE001 - normalised into ModelAnswer
return ModelAnswer(ok=False, error=self.error_text(exc))
logger.info(
"%s API success: HTTP %s; characters=%d; finish_reason=%s",
safe_diagnostic(self.name),
response.status_code,
len(text),
finish_reason,
)
return ModelAnswer(ok=True, answer=text)
def is_score(value) -> bool:
"""Accept a JSON number in 0-100. bool is an int subclass, so reject it first."""
return (
isinstance(value, (int, float))
and not isinstance(value, bool)
and 0 <= value <= 100
)
def normalise_answer(text: str) -> str:
"""Strip the differences that carry no meaning to an exact comparison.
Markdown emphasis, punctuation, case and whitespace are noise: "**Elephant**"
and "elephant" are the same answer.
"""
text = re.sub(r"[*_`#>]+", " ", text)
text = re.sub(r"[^\w\s]", " ", text)
return " ".join(text.lower().split())
def character_similarity(text1: str, text2: str) -> float:
"""Deterministic fallback score: character overlap, which ignores meaning.
Character overlap ranks a reworded but identical conclusion far below a
copied but contradictory one, so this only backs up :class:`Judge`.
"""
if not text1 or not text2:
return 0.0
clean1 = re.sub(r"\s+", " ", text1.strip().lower())
clean2 = re.sub(r"\s+", " ", text2.strip().lower())
return round(SequenceMatcher(None, clean1, clean2).ratio() * 100, 1)
class Judge:
"""Has the synthesis model score the panel answers and its own confidence.
Both numbers must survive a failed or malformed judge call, so every path
here returns a complete result and degrades to ``character_similarity``.
"""
def __init__(self):
self.client = ModelStudioClient(Config.JUDGE)
async def review(
self, query: str, final_answer: str, responses: Dict[str, ModelAnswer]
) -> Dict:
answered = {name: a.answer for name, a in responses.items() if a.ok}
failed = [name for name, a in responses.items() if not a.ok]
if not answered:
return {
"similarity_scores": {name: 0.0 for name in responses},
"confidence_level": 0,
"confidence_rationale": "No panel model returned an answer.",
"similarity_source": "character",
}
verdict = await self.client.answer(
self.build_prompt(query, final_answer, answered, failed), json_mode=True
)
if not verdict.ok:
return self.estimate(final_answer, responses, verdict.error)
parsed = self.parse(verdict.answer, list(answered))
if parsed is None:
return self.estimate(
final_answer,
responses,
f"unparseable judge output: {verdict.answer[:200]}",
)
scores, confidence, rationale = parsed
scores.update({name: 0.0 for name in failed})
return {
"similarity_scores": scores,
"confidence_level": confidence,
"confidence_rationale": rationale,
"similarity_source": "semantic",
}
@staticmethod
def build_prompt(
query: str, final_answer: str, answered: Dict[str, str], failed: List[str]
) -> str:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in answered.items())
template = ", ".join(f'"{name}": <int 0-100>' for name in answered)
absence = ""
if failed:
absence = (
"\nNOTE: these panel members failed and produced no answer: "
f"{', '.join(failed)}. Their absence must lower your confidence.\n"
)
return f"""A panel of models answered one question, then a final answer was produced.
QUESTION:
{query}
FINAL ANSWER:
{final_answer}
PANEL ANSWERS:
{sections}
{absence}
TASK 1 - semantic similarity of each panel answer to the FINAL ANSWER, 0-100:
- 90-100: identical conclusions and key facts, wording may differ
- 70-89: same conclusions, minor omissions or added detail
- 40-69: partially aligned, important differences in scope or emphasis
- 10-39: mostly different, or contradicts on a key point
- 0-9: unrelated, empty, or failed
Judge meaning; ignore wording, length and formatting.
TASK 2 - CONFIDENCE 0-100 in the FINAL ANSWER:
- 90-100: every panel supports the core conclusion; the answer is specific and actionable
- 75-89: a majority supports the core conclusion; minor gaps or unsupported additions
- 50-74: panels split on a key point, or parts of the question stay unanswered
- 25-49: the final answer conflicts with most panels, or makes questionable claims
- 0-24: unrelated, empty, or contradicts a clear consensus
The final answer is not automatically right because it is the final answer. Penalise
omissions and unsupported claims. Do not average the similarity scores.
Reply with JSON only, no prose:
{{"similarity": {{{template}}}, "confidence": <int 0-100>, "confidence_rationale": "<one sentence, max 30 words>"}}"""
@staticmethod
def parse(raw: str, models: List[str]):
"""Read the judge JSON, tolerating a code fence or surrounding prose."""
start, end = raw.find("{"), raw.rfind("}")
if start == -1 or end <= start:
return None
try:
data = json.loads(raw[start : end + 1])
except ValueError:
return None
if not isinstance(data, dict):
return None
raw_scores = data.get("similarity")
confidence = data.get("confidence")
if not isinstance(raw_scores, dict) or not is_score(confidence):
return None
# A partial answer is treated as malformed: mixing judged and estimated
# scores in one run would be impossible for a reader to interpret.
scores: Dict[str, float] = {}
for name in models:
value = raw_scores.get(name)
if not is_score(value):
return None
scores[name] = float(value)
rationale = " ".join(str(data.get("confidence_rationale") or "").split())
return scores, int(confidence), rationale[:300] or "No rationale returned."
async def agreement(self, query: str, answered: Dict[str, str]) -> Optional[Dict]:
"""Ask whether the panel reached one conclusion, wording aside.
Cheap answers look different only because they are longer or shorter; this
decides on meaning so an agreeing panel skips the synthesis call.
"""
if len(answered) < 2:
return None
verdict = await self.client.answer(
self.build_agreement_prompt(query, answered), json_mode=True
)
if not verdict.ok:
logger.warning(
"agreement check unavailable, falling back to synthesis: %s",
safe_diagnostic(verdict.error),
)
return None
return self.parse_agreement(verdict.answer, list(answered))
@staticmethod
def build_agreement_prompt(query: str, answered: Dict[str, str]) -> str:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in answered.items())
template = ", ".join(f'"{name}"' for name in answered)
return f"""A panel of models answered one question independently.
QUESTION:
{query}
PANEL ANSWERS:
{sections}
TASK - decide whether these answers reached the same core conclusion.
- "unanimous": every answer states the same conclusion, and none contradicts another
- "majority": two or more answers share the core conclusion, at least one differs
- "split": no two answers share the core conclusion
Wording, length, formatting, examples and extra detail do NOT count as disagreement.
Only a different conclusion, a contradiction of a key fact, or an answer that fails to
address the question counts.
Reply with JSON only, no prose:
{{"verdict": "unanimous|majority|split", "agreeing": [{template}], "conclusion": "<the shared conclusion in one sentence>"}}"""
@staticmethod
def parse_agreement(raw: str, models: List[str]) -> Optional[Dict]:
"""Derive the agreeing group, or an empty one when the reply is odd.
Empty means "no shared conclusion", which makes the caller synthesise.
"""
start, end = raw.find("{"), raw.rfind("}")
if start == -1 or end <= start:
return None
try:
data = json.loads(raw[start : end + 1])
except ValueError:
return None
if not isinstance(data, dict):
return None
verdict = data.get("verdict")
members = data.get("agreeing")
if verdict not in {"unanimous", "majority", "split"}:
return None
if not isinstance(members, list):
return None
# Keep panel order so the group is reproducible across runs.
agreeing = [name for name in models if name in members]
if verdict == "split" or len(agreeing) < 2:
return {"agreeing": [], "conclusion": ""}
conclusion = " ".join(str(data.get("conclusion") or "").split())[:300]
return {"agreeing": agreeing, "conclusion": conclusion}
@staticmethod
def estimate(
final_answer: str, responses: Dict[str, ModelAnswer], error: str
) -> Dict:
"""Degraded result: similarity survives, confidence does not.
Only the model can rate trust, and a character-overlap number would sit
near zero even for an excellent answer, so no substitute is invented.
"""
scores = {
name: character_similarity(final_answer, a.answer) if a.ok else 0.0
for name, a in responses.items()
}
reason = safe_diagnostic(error).rstrip(".")
return {
"similarity_scores": scores,
"confidence_level": None,
"confidence_rationale": (
f"Confidence not scored because the judge call failed: {reason}. "
"The similarity figures below are character overlap, not meaning."
),
"similarity_source": "character",
}
class ConsensusEngine:
def __init__(self):
self.panel = [ModelStudioClient(spec) for spec in Config.PANEL]
self.synthesizer = ModelStudioClient(Config.SYNTHESIS)
self.judge = Judge()
async def get_consensus_response(self, query: str) -> QueryResponse:
logger.info("Processing query; characters=%d", len(query))
answers = await asyncio.gather(*(client.answer(query) for client in self.panel))
responses = {
client.name: answer for client, answer in zip(self.panel, answers)
}
final_answer, consensus_info = await self.build_consensus(query, responses)
# Scoring runs last: every score is a comparison against the final text.
consensus_info.update(await self.judge.review(query, final_answer, responses))
return QueryResponse(
final_answer=final_answer,
responses=responses,
consensus_info=consensus_info,
)
async def build_consensus(
self, query: str, responses: Dict[str, ModelAnswer]
) -> Tuple[str, Dict]:
valid = {name: a.answer for name, a in responses.items() if a.ok}
total = len(responses)
# Every branch below sets the status, so no default is needed.
info: Dict = {}
if not valid:
info["consistency_status"] = "ALL_FAILED"
info["process_description"] = (
"Every panel model failed; see the individual errors"
)
return "No panel model returned an answer.", info
# Free path: answers that are identical once wording noise is removed.
# Unanimity is required: a 2-1 split goes to synthesis so that the dissenting
# view is weighed instead of being outvoted by a character match.
groups: Dict[str, List[str]] = {}
for name, text in valid.items():
groups.setdefault(normalise_answer(text), []).append(name)
_, winner_names = max(groups.items(), key=lambda item: len(item[1]))
if len(valid) >= 2 and len(winner_names) == len(valid):
info["consistency_status"] = "MAJORITY"
info["agreement_method"] = "exact"
info["process_description"] = (
f"Identical answers: all {len(valid)}/{total} responding models "
f"returned the same text ({', '.join(winner_names)})"
)
return valid[winner_names[0]], info
# Paraphrased answers never match textually, so ask the judge whether they
# reached the same conclusion before paying for a synthesis call. Same rule:
# only a unanimous conclusion skips synthesis.
verdict = await self.judge.agreement(query, valid)
if verdict and len(valid) >= 2 and len(verdict["agreeing"]) == len(valid):
agreeing = verdict["agreeing"]
info["consistency_status"] = "MAJORITY"
info["agreement_method"] = "semantic"
info["process_description"] = (
f"Same conclusion in different words: all {len(valid)}/{total} "
f"responding models agree ({', '.join(agreeing)}); showing the most "
"complete answer"
)
if verdict["conclusion"]:
info["shared_conclusion"] = verdict["conclusion"]
return max((valid[name] for name in agreeing), key=len), info
synthesis = await self.synthesise(query, valid)
if synthesis.ok:
info["consistency_status"] = "SYNTHESISED"
info["process_description"] = (
"Panel did not reach unanimity; "
f"{Config.SYNTHESIS.name} merged the responses"
)
return synthesis.answer, info
# Never present a panel answer as if it were a consensus.
info["consistency_status"] = "SYNTHESIS_FAILED"
info["process_description"] = (
"Synthesis failed; showing the most complete panel answer instead. "
f"error: {synthesis.error}"
)
return max(valid.values(), key=len), info
async def synthesise(self, query: str, valid: Dict[str, str]) -> ModelAnswer:
sections = "\n\n".join(f"[{name}]\n{text}" for name, text in valid.items())
# The question must be repeated: short answers like "290" carry no context of
# their own, and without it the synthesis turns into a meta-commentary.
prompt = (
"Several AI models answered the question below. Write one comprehensive, "
"accurate answer that keeps the best elements of each and resolves any "
"contradictions. Answer the question directly; never describe the models "
"or comment on the answers themselves.\n\n"
f"QUESTION:\n{query}\n\n"
f"PANEL ANSWERS:\n{sections}"
)
return await self.synthesizer.answer(prompt)
consensus_engine = ConsensusEngine()
app.mount("/static", StaticFiles(directory="static"), name="static")
@app.get("/", response_class=HTMLResponse)
async def read_root():
with open("index.html", "r", encoding="utf-8") as file:
# The shell must revalidate, otherwise a stale asset list pins old JS.
return HTMLResponse(file.read(), headers={"Cache-Control": "no-cache"})
@app.get("/health")
async def health_check():
return {
"status": "healthy",
"panel": [spec.name for spec in Config.PANEL],
"synthesis": Config.SYNTHESIS.name,
"judge": Config.JUDGE.name,
}
@app.post("/api/consensus", response_model=QueryResponse)
async def get_consensus_answer(request: QueryRequest):
if not request.query.strip():
raise HTTPException(status_code=400, detail="Query cannot be empty")
try:
return await consensus_engine.get_consensus_response(request.query)
except Exception as exc: # noqa: BLE001 - never echo arbitrary exception text
detail = f"Unexpected consensus error: {type(exc).__name__}"
logger.error("Error processing consensus request: %s", detail)
raise HTTPException(status_code=500, detail=detail) from None
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000, log_level="info")<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Multi-LLM Consensus Agent</title>
<!-- Bump ?v= whenever these files change, or browsers keep serving the old copy. -->
<link rel="stylesheet" href="static/css/style.css?v=6">
</head>
<body>
<div class="container">
<header>
<h1>Multi-LLM Consensus Agent</h1>
<p>Get reliable answers through consensus of multiple LLMs</p>
</header>
<main>
<div class="query-section">
<div class="input-group">
<label for="user-query">Enter your question:</label>
<textarea id="user-query" placeholder="Ask anything..."></textarea>
</div>
<button id="submit-query" class="submit-btn">Get Consensus Answer</button>
</div>
<div class="loading-section" id="loading-section" style="display: none;">
<div class="spinner"></div>
<p>Querying three panel models, then synthesising and scoring the result...</p>
<p class="loading-hint">Three rounds run in sequence, so expect 40-60 seconds.</p>
</div>
<div class="results-section" id="results-section" style="display: none;">
<h2>Consensus Results</h2>
<div class="consensus-answer">
<h3>Final Answer</h3>
<div id="final-answer" class="answer-content"></div>
</div>
<div class="individual-responses">
<h3>Individual Model Responses</h3>
<!-- Cards are rendered from the API response, one per model. -->
<div id="response-grid" class="response-grid"></div>
</div>
<div class="consensus-info">
<h3>Consensus Process</h3>
<div id="consensus-details"></div>
</div>
</div>
</main>
<footer>
<p>Multi-LLM Consensus Agent System</p>
</footer>
</div>
<script src="static/js/main.js?v=6"></script>
</body>
</html>/* Global Styles */
* {
margin: 0;
padding: 0;
box-sizing: border-box;
}
body {
font-family: 'Segoe UI', Tahoma, Geneva, Verdana, sans-serif;
line-height: 1.6;
color: #333;
background-color: #f5f5f5;
}
.container {
max-width: 1200px;
margin: 0 auto;
padding: 20px;
}
/* Header Styles */
header {
text-align: center;
margin-bottom: 40px;
padding: 30px 0;
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border-radius: 10px;
box-shadow: 0 4px 15px rgba(0,0,0,0.1);
}
header h1 {
font-size: 2.5rem;
margin-bottom: 10px;
}
header p {
font-size: 1.2rem;
opacity: 0.9;
}
/* Main Content */
main {
background: white;
border-radius: 10px;
padding: 30px;
box-shadow: 0 2px 10px rgba(0,0,0,0.1);
}
/* Query Section */
.query-section {
margin-bottom: 30px;
}
.input-group {
margin-bottom: 20px;
}
.input-group label {
display: block;
margin-bottom: 10px;
font-weight: 600;
color: #555;
}
#user-query {
width: 100%;
min-height: 120px;
padding: 15px;
border: 2px solid #ddd;
border-radius: 8px;
font-size: 16px;
font-family: inherit;
resize: vertical;
transition: border-color 0.3s;
}
#user-query:focus {
outline: none;
border-color: #667eea;
box-shadow: 0 0 0 3px rgba(102, 126, 234, 0.1);
}
.submit-btn {
background: linear-gradient(135deg, #667eea 0%, #764ba2 100%);
color: white;
border: none;
padding: 15px 30px;
font-size: 18px;
border-radius: 8px;
cursor: pointer;
transition: transform 0.2s, box-shadow 0.2s;
font-weight: 600;
}
.submit-btn:hover {
transform: translateY(-2px);
box-shadow: 0 5px 15px rgba(102, 126, 234, 0.4);
}
.submit-btn:active {
transform: translateY(0);
}
.submit-btn:disabled {
background: #ccc;
cursor: not-allowed;
transform: none;
box-shadow: none;
}
/* Loading Section */
.loading-section {
text-align: center;
padding: 40px;
}
.loading-hint {
color: #6c757d;
font-size: 13px;
}
.spinner {
width: 50px;
height: 50px;
border: 5px solid #f3f3f3;
border-top: 5px solid #667eea;
border-radius: 50%;
animation: spin 1s linear infinite;
margin: 0 auto 20px;
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
/* Results Section */
.results-section {
margin-top: 30px;
}
.results-section h2 {
color: #333;
margin-bottom: 25px;
text-align: center;
font-size: 2rem;
}
/* Consensus Answer */
.consensus-answer {
background: linear-gradient(135deg, #f0f8ff 0%, #e6f3ff 100%);
border: 2px solid #667eea;
border-radius: 10px;
padding: 25px;
margin-bottom: 30px;
}
.consensus-answer h3 {
color: #667eea;
margin-bottom: 15px;
font-size: 1.5rem;
}
.answer-content {
font-size: 18px;
line-height: 1.7;
color: #444;
}
/* Individual Responses */
.individual-responses {
margin-bottom: 30px;
}
.individual-responses h3 {
color: #333;
margin-bottom: 20px;
font-size: 1.5rem;
}
.response-grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 20px;
}
.response-card {
background: #f9f9f9;
border: 1px solid #ddd;
border-radius: 8px;
padding: 20px;
box-shadow: 0 2px 5px rgba(0,0,0,0.05);
transition: transform 0.2s;
}
.response-card:hover {
transform: translateY(-3px);
box-shadow: 0 5px 15px rgba(0,0,0,0.1);
}
.response-card h4 {
color: #555;
margin-bottom: 15px;
padding-bottom: 10px;
border-bottom: 2px solid #eee;
font-size: 1.3rem;
}
.response-content {
font-size: 16px;
line-height: 1.6;
color: #666;
}
/* Consensus Info */
.consensus-info {
background: #f8f9fa;
border: 1px solid #dee2e6;
border-radius: 8px;
padding: 20px;
}
.consensus-info h3 {
color: #495057;
margin-bottom: 15px;
font-size: 1.3rem;
}
/* Footer */
footer {
text-align: center;
margin-top: 40px;
padding: 20px;
color: #666;
font-size: 14px;
}
/* Responsive Design */
@media (max-width: 768px) {
.container {
padding: 10px;
}
header h1 {
font-size: 2rem;
}
.response-grid {
grid-template-columns: 1fr;
}
main {
padding: 20px;
}
}
/* Status Indicators; only the four statuses the backend can emit */
.status-majority {
color: #ffc107;
font-weight: 600;
}
.status-synthesised {
color: #667eea;
font-weight: 600;
}
.status-synthesis_failed,
.status-all_failed {
color: #dc3545;
font-weight: 600;
}
/* Per-model failure detail, shown instead of hiding the diagnostic */
.model-error {
background-color: #f8d7da;
border: 1px solid #f5c6cb;
border-radius: 6px;
color: #721c24;
font-size: 14px;
padding: 12px;
word-break: break-word;
}
/* Confidence verdict plus the judge's one-line justification */
.confidence {
margin-top: 12px;
font-size: 15px;
}
.confidence-rationale {
color: #6c757d;
font-size: 14px;
font-style: italic;
margin-top: 6px;
padding-left: 12px;
border-left: 3px solid #dee2e6;
}
/* Similarity Score Styles */
.similarity-score {
margin-top: 15px;
padding: 12px;
border-radius: 6px;
font-size: 14px;
}
.high-similarity {
background-color: #d4edda;
border: 1px solid #c3e6cb;
color: #155724;
}
.medium-similarity {
background-color: #fff3cd;
border: 1px solid #ffeaa7;
color: #856404;
}
.low-similarity {
background-color: #f8d7da;
border: 1px solid #f5c6cb;
color: #721c24;
}
.similarity-bar {
width: 100%;
height: 8px;
background-color: #e9ecef;
border-radius: 4px;
margin-top: 8px;
overflow: hidden;
}
.similarity-fill {
height: 100%;
background: linear-gradient(90deg, #28a745, #20c997);
border-radius: 4px;
transition: width 0.3s ease;
}
.no-response {
color: #6c757d;
font-style: italic;
}// Multi-LLM Consensus Agent Frontend
class ConsensusAgent {
constructor() {
this.apiUrl = '/api/consensus';
this.initializeEventListeners();
}
initializeEventListeners() {
const submitBtn = document.getElementById('submit-query');
const queryInput = document.getElementById('user-query');
submitBtn.addEventListener('click', () => this.handleSubmit());
queryInput.addEventListener('keypress', (e) => {
if (e.key === 'Enter' && e.ctrlKey) {
this.handleSubmit();
}
});
}
async handleSubmit() {
const query = document.getElementById('user-query').value.trim();
if (!query) {
alert('Please enter a question');
return;
}
this.showLoading();
this.hideResults();
try {
const response = await this.queryConsensusAgent(query);
this.displayResults(response);
} catch (error) {
console.error('Error:', error);
this.showError(error.message);
} finally {
this.hideLoading();
}
}
async queryConsensusAgent(query) {
const response = await fetch(this.apiUrl, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify({ query: query })
});
if (!response.ok) {
throw new Error(`HTTP error! status: ${response.status}`);
}
return await response.json();
}
showLoading() {
document.getElementById('loading-section').style.display = 'block';
document.getElementById('submit-query').disabled = true;
}
hideLoading() {
document.getElementById('loading-section').style.display = 'none';
document.getElementById('submit-query').disabled = false;
}
showResults() {
document.getElementById('results-section').style.display = 'block';
}
hideResults() {
document.getElementById('results-section').style.display = 'none';
}
displayResults(data) {
const info = data.consensus_info || {};
const similarityScores = info.similarity_scores || {};
// "semantic" means the judge model scored it; anything else is the
// character-overlap estimate used when the judge call failed.
const similaritySource = info.similarity_source || 'character';
document.getElementById('final-answer').innerHTML = this.formatText(data.final_answer);
// One card per model the backend actually answered with, so the model
// lineup lives only in Config.PANEL.
const grid = document.getElementById('response-grid');
grid.innerHTML = Object.entries(data.responses || {})
.map(([model, result]) =>
this.renderModelCard(model, result, similarityScores[model], similaritySource))
.join('');
document.getElementById('consensus-details').innerHTML =
this.formatConsensusDetails(info);
this.showResults();
}
renderModelCard(model, result, similarityScore, similaritySource) {
let body;
if (result.ok) {
body = this.formatText(result.answer)
+ this.renderSimilarity(similarityScore, similaritySource);
} else {
// Surface the provider's diagnostic: hiding it made every failure
// look like a network problem.
body = `<p class="model-error">${this.escapeHtml(result.error || 'Request failed')}</p>`;
}
return `<div class="response-card">
<h4>${this.escapeHtml(model)}</h4>
<div class="response-content">${body}</div>
</div>`;
}
renderSimilarity(similarityScore, similaritySource) {
if (similarityScore === undefined || similarityScore === null) {
return '';
}
const semantic = similaritySource === 'semantic';
const label = semantic
? 'Semantic similarity to final answer'
: 'Similarity to final answer (estimated)';
const explanation = semantic
? 'Scored by the judge model against a rubric, not by character overlap.'
: 'Character overlap only: the semantic scoring call was unavailable.';
const scoreClass = similarityScore >= 80 ? 'high-similarity' :
similarityScore >= 60 ? 'medium-similarity' : 'low-similarity';
return `<div class="similarity-score ${scoreClass}" title="${this.escapeHtml(explanation)}">
<strong>${label}:</strong> ${similarityScore}%
<div class="similarity-bar">
<div class="similarity-fill" style="width: ${similarityScore}%"></div>
</div>
</div>`;
}
formatText(text) {
if (!text) return '<p class="no-response">No content returned</p>';
return `<p>${this.escapeHtml(text).replace(/\n/g, '<br>')}</p>`;
}
formatConsensusDetails(info) {
if (!info) return '<p>No consensus details available</p>';
const status = String(info.consistency_status || 'unknown');
let html = '<div class="consensus-details">';
html += `<div class="status-info">
<strong>Consistency Status:</strong>
<span class="status-${this.escapeHtml(status.toLowerCase())}">${this.escapeHtml(status)}</span>
</div>`;
if (info.process_description) {
html += `<div class="process-description">
<strong>Process:</strong> ${this.escapeHtml(info.process_description)}
</div>`;
}
// 0 is a real verdict, null means the judge call failed; never hide either.
const confidence = info.confidence_level === null || info.confidence_level === undefined
? 'unavailable'
: `${info.confidence_level}%`;
html += `<div class="confidence">
<strong>Confidence Level:</strong> ${confidence}
</div>`;
if (info.confidence_rationale) {
html += `<div class="confidence-rationale">${this.escapeHtml(info.confidence_rationale)}</div>`;
}
html += '</div>';
return html;
}
showError(message) {
const errorDiv = document.createElement('div');
errorDiv.className = 'error-message';
errorDiv.innerHTML = `<p><strong>Error:</strong> ${this.escapeHtml(message)}</p>`;
errorDiv.style.cssText = `
background: #f8d7da;
color: #721c24;
border: 1px solid #f5c6cb;
border-radius: 5px;
padding: 15px;
margin: 20px 0;
`;
const main = document.querySelector('main');
main.insertBefore(errorDiv, main.querySelector('.query-section').nextSibling);
// Remove error after 5 seconds
setTimeout(() => {
if (errorDiv.parentNode) {
errorDiv.parentNode.removeChild(errorDiv);
}
}, 5000);
}
escapeHtml(unsafe) {
return String(unsafe)
.replace(/&/g, '&')
.replace(/</g, '<')
.replace(/>/g, '>')
.replace(/"/g, '"')
.replace(/'/g, ''');
}
}
// Initialize the application
document.addEventListener('DOMContentLoaded', () => {
new ConsensusAgent();
});Wei Ying - November 5, 2026
Alibaba Cloud Community - August 3, 2026
Alibaba Cloud Community - October 9, 2024
Yongbin Li - September 17, 2020
Alibaba Cloud Community - June 3, 2026
Kidd Ip - August 31, 2026
QwenWork
QwenWork is dedicated to helping employees strengthen their professional competitiveness in the AI era and to enabling enterprises to improve organizational effectiveness.
Learn More
Qwen
Full-range, open-source, multimodal, and multi-functional
Learn More
AgentCore
AgentCore unifies agent construction, governance, and execution capabilities so enterprises can deploy diverse agents with confidence.
Learn More
Qoder
From Assisted to Autonomous — Adapting to Every Programming Need
Learn More