Content

Target customers: Customers who need to evaluate and optimize large language models (LLMs), including AI service providers, enterprises that develop LLM, enterprises that use LLM, AI researchers, and research institutions. New features/specifications: The judge model service of PAI uses a fine-tuned LLM based on Qwen2 as a judge to score responses from evaluated models. This service is suitable for open-ended and complex scenarios. Main advantages: 1. Accuracy: The judge model can classify subjective questions into scenarios such as open-ended discussions, creative writing, code generation, and role-playing. It then develop tailored criteria for each scenario, significantly enhancing evaluation accuracy. 2. Efficiency: Without the need for manual data labeling, the judge model can independently analyze and evaluate LLMs based on questions and model answers, greatly boosting evaluation efficiency. 3. Ease of use: PAI offers various usage methods, such as task creation in the console, API calls, and SDK calls. This allows for both quick trials and flexible integration for developers. 4. Cost-effectiveness: The judge model provides performance evaluation at a competitive price. Its performance is comparable to that of ChatGPT-4 in Chinese language scenarios.