Platform For AI:API feature description

Chat completions

Sample code

1. Request example

   Python

   import os
   from openai import OpenAI
   
   
   def main():
       base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
       judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
   
       client = OpenAI(
           api_key=f'Authorization: Bearer {judge_model_token}',
           base_url=base_url
       )
       completion = client.chat.completions.create(
           model='pai-judge',
           messages=[
               {
                   "role": "user",
                   "content": [
                       {
                           "mode": "single",
                           "type": "json",
                           "json": {
                               "question": "According to the first couplet, give the second couplet. first couplet: To climb the mountain, reach the peak",
                               "answer": "To cross the river, find the creek."
                           }
                       }
                   ]
               }
           ]
       )
       print(completion.model_dump())
   
   
   if __name__ == '__main__':
       main()

   curl

   $ curl -X POST https://aiservice.cn-hangzhou.aliyuncs.com/v1/chat/completions \
     -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}" \
     -H "Content-Type: application/json" \
     -d '{
       "model": "pai-judge",
       "messages": [
           {
               "role": "user",
               "content": [
                   {
                       "mode": "single",
                       "type": "json",
                       "json": {
                           "question": "According to the first couplet, give the second couplet. first couplet: To climb the mountain, reach the peak",
                           "answer": "To cross the river, find the creek."
                       }
                   }
               ]
           }
       ]
   }'

2. Response example

   {
       "id": "3b7c3822-1e51-4dc9-b2ad-18b9649a7f19",
       "choices": [
           {
               "finish_reason": "stop",
               "index": 0,
               "logprobs": null,
               "message": {
                   "content": "I think the overall score of the answer is [[2]] due to the following reasons:\n Advantages of the answer:\n1. Relevance: The answer directly addresses the user's question, providing a second couplet that corresponds to the first couplet. ThiS meets the relevance criteria.[[4]]\n2. Harmlessness: The answer does not contain offensive content. This meets the harmlessness criteria.[[5]]\n\n Disadvantages of the answer: \n1. Accuracy: The content \"To cross the river, find the creek\" does not fully align with the logical sequence of \"climbing\" and \"reaching the peak\" in the user's question, which affects the accuracy of the answer.[[2]]\n2. Completeness: The answer does not fully address all aspects of the question because the answer does not provide a complete story or completely align with the question. This affects the completeness of the answer.[[2]]\n3. Source reliability: The answer does not provide source information. Although the source information may not be necessary in some scenarios, the information can enhance the credibility of the answer.[[3]]\n4. Clarity and structure: Although the answer is simple in structure, its clarity and comprehensibility are affected because the answer does not fully correspond to the question.[[3]]\n5. Adaptability to the user level: Although the answer directly addresses the question, the answer may not be completely suitable for users who have a certain understanding of couplets or traditional literature due to inaccuracy.[[3]]\n\n In summary, although the answer performs well in relevance and harmlessness, the answer shows shortcomings in accuracy, completeness, source reliability, clarity and structure, and adaptability to the user level, which results in an overall rating of 2.",
                   "role": "assistant",
                   "function_call": null,
                   "tool_calls": null,
                   "refusal": ""
               }
           }
       ],
       "created": 1733260,
       "model": "pai-judge",
       "object": "chat.completion",
       "service_tier": "",
       "system_fingerprint": "",
       "usage": {
           "completion_tokens": 333,
           "prompt_tokens": 790,
           "total_tokens": 1123
       }
   }
   

Request parameters

The APIs of judge models are compatible with those of OpenAI. You can configure the parameters described in the following table. For information about other parameters, see OpenAI documentation.

messages parameter

The messages parameter specifies the information received by a judge model. The following sample code provides an example:

messages=[
    {
        "role": "user",
        "content": [
            {
                "mode": "single",
                "type": "json",
                "json": {
                    "question": "Provide the second couplet to \"To climb the mountain, reach the peak\"",
                    "answer": "To cross the river, find the creek"
                }
            }
        ]
    }
]

The messages parameter is in the array format. Each element is in the {"role":role, "content": content} format. role specifies the user. content specifies the content to be evaluated by the model. The content includes the following information:

• mode: the evaluation mode. single specifies single-model evaluation. pairwise specifies dual-model competition.

• type: Set the value to json.

• json: the details of the content to be evaluated.

The following table describes the json parameter.

The parameters in content are used to fill in and generate prompt templates. The single-model evaluation and dual-model competition request examples use the following templates to construct requests. The content of content is automatically filled in the corresponding positions of the template.

Your task is to evaluate the quality of an AI assistant's answer.

You clearly understand that when a user provides a question about the [${scene}] scenario (defined as: ${scene_desc}), an AI assistant's answer should meet the following criteria (listed in order of importance from highest to lowest):
[Start of criteria]
${metric}
[End of criteria]

The scoring system uses a ${max_score}-level scale (1-${max_score}), with each score level defined as follows:
[Start of score description]
${score_desc}
[End of score description]

We have collected the following answer from an AI assistant to a user question.
Please comprehensively evaluate the answer based on the criteria that you use for the current scenario. The following items describe the question and answer from an AI assistant:
[Start of data] 
***
[User question]: ${question}
***
[Answer]: ${answer}
***
[Reference answer]: ${ref_answer}
***
[End of data]

You must evaluate the preceding answer based on the following process:
${steps}

Think carefully and then provide your conclusion.

Your task is to evaluate the quality of AI assistants' answers.

You clearly understand that when a user provides a question about the [${scene}] scenario (defined as: ${scene_desc}), an AI assistant's answer should meet the following criteria (listed in order of importance from highest to lowest):
[Start of criteria]
${metric}
[End of criteria]

The scoring system uses a ${max_score}-level scale (1-${max_score}), with each score level defined as follows:
[Start of score description]
${score_desc}
[End of score description]

For a user question in the [${scene}] scenario, we have collected answers from two AI assistants.
Please comprehensively evaluate the answers and determine which answer is better or whether the quality of the two answers is the same based on the criteria that you use for the current scenario. The following items describe a question and the answers from the AI assistants:
[Start of data]
***
[User question]: ${question}
***
[Answer 1]: ${answer1}
***
[Answer 2]: ${answer2}
***
[Reference answer]: ${ref_answer}
***
[End of data]

You must evaluate and compare the two answers based on the following process:
{steps}

Think carefully and then provide your conclusion.

Response parameters

Status codes

{
  "content": [
    {
      "type": "json", 
      "mode": "[single / pairwise]", 
      "json": {
        "question": "<question>", 
        "answer": "<answer>",
        ...
      }
    }
  ]
}

Files

Upload a file: POST /v1/files

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      upload_files = client.files.create(
          file=open("/home/xxx/input.jsonl", "rb"),
          purpose="batch",
      )
      print(upload_files.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XPOST https://aiservice.cn-hangzhou.aliyuncs.com/v1/files \
    -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}" \
    -F purpose="batch"  \
    -F file="@/home/xxx/input.jsonl"
  

• Response example

  {
      "id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      "object": "file",
      "bytes": 698,
      "created_at": 1742454203,
      "filename": "input.jsonl",
      "purpose": "batch"
  }

• Request parameters

  Parameter	Type	Required	Description
  file	file	Yes	The file.
  purpose	string	Yes	The expected usage of the file. assistants: the assistant and message files. vision: the assistant image file. batch: the API file for batch processing. fine-tune: the fine-tuning file.

• Response parameters

  For more information, see File description.

Query a list of files: GET /v1/files

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      list_files = client.files.list(
          purpose="batch",
          order="desc",
          limit=10,
          after=""
      )
      print(list_files.model_dump_json(indent=4))
  
      
  if __name__ == '__main__':
      main()

  curl

  $ curl -XGET https://aiservice.cn-hangzhou.aliyuncs.com/v1/files \
    -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "object": "list",
      "data": [
          {
              "id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
              "object": "file",
              "bytes": 698,
              "created_at": 1742454203,
              "filename": "input.jsonl",
              "purpose": "batch"
          },
          {
              "id": "file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
              "object": "file",
              "bytes": 1420,
              "created_at": 1742455638,
              "filename": "file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022_success.jsonl",
              "purpose": "batch_output"
          }
      ]
  }

• Request parameters

  Parameter	Type	Required	Description
  purpose	string	No	Returns only files with the specific purpose.
  limit	string	No	The expected usage of the file. Only API files for batch processing are supported. Default value: 10000.
  order	string	No	The order in which the created_at parameter is used to sort the query results. asc desc (default)
  after	string	No	The cursor for pagination. This parameter defines an object ID that indicates a position in the query results. For example, if you send a request and receive 100 objects ending with obj_foo, you can set after to obj_foo in a subsequent call to obtain the query results after obj_foo.

• Response parameters

  Parameter	Type	Description
  object	string	Returns only files with the specific purpose.
  data	array	For more information, see File description.

Query a file: GET /v1/files/{file_id}

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      retrieve_files = client.files.retrieve(
          file_id="file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      )
      print(retrieve_files.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XGET https://aiservice.cn-hangzhou.aliyuncs.com/v1/files/file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713 \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      "object": "file",
      "bytes": 698,
      "created_at": 1742454203,
      "filename": "input.jsonl",
      "purpose": "batch"
  }

• Request parameters

  Parameter	Type	Required	Description
  file_id	string	Yes	The ID of the file.

• Response parameters

  For more information, see File description.

Query or download the file content: GET /v1/files/{file_id}/content

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      content_files = client.files.content(
          file_id="file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
      )
      print(content_files)
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XGET https://aiservice.cn-hangzhou.aliyuncs.com/v1/files/file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022/content \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}" > output.jsonl

• Response example

  {"id":"dcee3584-6f30-9541-a855-873a6d86b7d9","custom_id":"request-1","response":{"status_code":200,"request_id":"dcee3584-6f30-9541-a855-873a6d86b7d9","body":{"created":1737446797,"usage":{"completion_tokens":7,"prompt_tokens":26,"total_tokens":33},"model":"qwen-max","id":"chatcmpl-dcee3584-6f30-9541-a855-873a6d86b7d9","choices":[{"finish_reason":"stop","index":0,"message":{"content":"2+2 equals 4."}}],"object":"chat.completion"}},"error":null}
  {"id":"dcee3584-6f30-9541-a855-873a6d86b7d9","custom_id":"request-2","response":{"status_code":200,"request_id":"dcee3584-6f30-9541-a855-873a6d86b7d9","body":{"created":1737446797,"usage":{"completion_tokens":7,"prompt_tokens":26,"total_tokens":33},"model":"qwen-max","id":"chatcmpl-dcee3584-6f30-9541-a855-873a6d86b7d9","choices":[{"finish_reason":"stop","index":0,"message":{"content":"2+2 equals 4."}}],"object":"chat.completion"}},"error":null}

• Request parameters

  Parameter	Type	Required	Description
  file_id	string	Yes	The ID of the file.

• Response parameters

  For more information, see File description.

Delete a file: DELETE /v1/files/{file_id}

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      delete_files = client.files.delete(
          file_id="file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
      )
      print(delete_files)
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XDELETE https://aiservice.cn-hangzhou.aliyuncs.com/v1/files/file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022 \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      "object": "file",
      "deleted": "true"
  }

• Request parameters

  Parameter	Type	Required	Description
  file_id	string	Yes	The ID of the file.

• Response parameters

  Parameter	Type	Description
  id	string	The ID of the file that was deleted.
  object	string	The type of the deleted file.
  deleted	string	Indicates whether the file was deleted.

File description

Batch

Create a batch task: POST /v1/batches

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      create_batches = client.batches.create(
          endpoint="/v1/chat/completions",
          input_file_id="file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
          completion_window="24h",
      )
      print(create_batches.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XPOST https://aiservice.cn-hangzhou.aliyuncs.com/v1/batches \ 
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}" \
      -d '{
          "input_file_id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
          "endpoint": "/v1/chat/completions",
          "completion_window": "24h"
   }'

• Response example

  {
      "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "errors": null,
      "input_file_id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      "completion_window": "24h",
      "status": "Creating",
      "output_file_id": null,
      "error_file_id": null,
      "created_at": 1742455213,
      "in_process_at": null,
      "expires_at": null,
      "FinalizingAt": null,
      "completed_at": null,
      "failed_at": null,
      "expired_at": null,
      "cancelling_at": null,
      "cancelled_at": null,
      "request_counts": {
          "total": 3,
          "completed": 0,
          "failed": 0
      },
      "metadata": null
  }

• Request parameters

  Parameter	Type	Required	Description
  input_file_id	string	Yes	The ID of the file that is formatted as a .jsonl file and must be uploaded with the target batch task. The file can contain up to 50,000 requests and be up to 20 MB in size.
  endpoint	string	Yes	The endpoint to use for all requests in a batch task. Only /v1/chat/completions is supported.
  completion_window	string	Yes	The time range in which a batch of files are processed. Set the value to 24h.
  completion_window	object	No	The custom metadata for the batch task.

• Response parameters

  For more information, see Batch task description.

Query a list of batche tasks: GET /v1/files

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "http://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      list_batches = client.batches.list(
          after="batch_66f245a0-88d1-458c-8e1c-a819a5943022",
          limit=10,
      )
      print(list_batches.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XGET https://aiservice.cn-hangzhou.aliyuncs.com/v1/batches \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "object": "list",
      "data": [
          {
              "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
              "object": "batch",
              "endpoint": "/v1/chat/completions",
              "errors": null,
              "input_file_id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
              "completion_window": "24h",
              "status": "Succeeded",
              "output_file_id": "file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
              "error_file_id": null,
              "created_at": 1742455213,
              "in_process_at": 1742455640,
              "expires_at": 1742455640,
              "FinalizingAt": 1742455889,
              "completed_at": 1742455889,
              "failed_at": null,
              "expired_at": null,
              "cancelling_at": null,
              "cancelled_at": null,
              "request_counts": {
                  "total": 3,
                  "completed": 3,
                  "failed": 0
              },
              "metadata": null
          }
      ],
      "first_id": "",
      "last_id": "",
      "has_more": false
  }

• Request parameters

  Parameter	Type	Required	Description
  purpose	string	No	Returns only files with the specific purpose.
  limit	string	No	The expected usage of the file. assistants: the assistant and message files. vision: the assistant image file. batch: the API file for batch processing. fine-tune: the fine-tuning file.
  order	string	No	The order in which the created_at parameter is used to sort the query results. asc desc (default)
  after	string	No	The cursor for pagination. This parameter defines an object ID that indicates a position in the query results. For example, if you send a request and receive 100 objects ending with obj_foo, you can set after to obj_foo in a subsequent call to obtain the query results after obj_foo.

• Response parameters

  Parameter	Type	Description
  object	string	The object type.
  data	array	The batch task description.

Query a batch task: GET /v1/batches/{batch_id}

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "http://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      retrieve_batches = client.batches.retrieve(
          batch_id="batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      )
      print(retrieve_batches.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XGET https://aiservice.cn-hangzhou.aliyuncs.com/v1/batches/batch_66f245a0-88d1-458c-8e1c-a819a5943022 \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "errors": null,
      "input_file_id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      "completion_window": "24h",
      "status": "Succeeded",
      "output_file_id": "file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
      "error_file_id": null,
      "created_at": 1742455213,
      "in_process_at": 1742455640,
      "expires_at": 1742455640,
      "FinalizingAt": 1742455889,
      "completed_at": 1742455889,
      "failed_at": null,
      "expired_at": null,
      "cancelling_at": null,
      "cancelled_at": null,
      "request_counts": {
          "total": 3,
          "completed": 3,
          "failed": 0
      },
      "metadata": null
  }

• Request parameters

  Parameter	Type	Required	Description
  batch_id	string	Yes	The ID of the batch task.

• Response parameters

  For more information, see Batch task description.

Cancel a batch task: POST /v1/batches/{batch_id}/cancel

Cancels an in-progress batch task. A batch task maintains the cancelling state for up to 10 minutes before changing to the cancelled state. In this case, if the batch task generates outputs, they are available in the output file.

• Request example

  Python

  import os
  from openai import OpenAI
  
  
  def main():
      base_url = "https://aiservice.cn-hangzhou.aliyuncs.com/v1"
      judge_model_token = os.getenv("JUDGE_MODEL_TOKEN")
  
      client = OpenAI(
          api_key=f'Authorization: Bearer {judge_model_token}',
          base_url=base_url
      )
      cancel_batches = client.batches.cancel(
          batch_id="batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      )
      print(cancel_batches.model_dump_json(indent=4))
  
  
  if __name__ == '__main__':
      main()

  curl

  $ curl -XPOST https://aiservice.cn-hangzhou.aliyuncs.com/v1/batches/batch_66f245a0-88d1-458c-8e1c-a819a5943022/cancel \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "errors": null,
      "input_file_id": "file-batch-EC043540BE1C7BE3F9F2F0A8F47D1713",
      "completion_window": "24h",
      "status": "Stopping",
      "output_file_id": "file-batch_output-66f245a0-88d1-458c-8e1c-a819a5943022",
      "error_file_id": null,
      "created_at": 1742455213,
      "in_process_at": 1742455640,
      "expires_at": 1742455640,
      "FinalizingAt": 1742455889,
      "completed_at": 1742455889,
      "failed_at": null,
      "expired_at": null,
      "cancelling_at": null,
      "cancelled_at": null,
      "request_counts": {
          "total": 3,
          "completed": 3,
          "failed": 0
      },
      "metadata": null
  }

• Request parameters

  Parameter	Type	Required	Description
  batch_id	string	Yes	The ID of the batch task.

• Response parameters

  For more information, see Batch task description.

Delete a batch task: DELETE /v1/batches/{batch_id}

• Request example

  $ curl -XDELETE https://aiservice.cn-hangzhou.aliyuncs.com/v1/batches/batch_66f245a0-88d1-458c-8e1c-a819a5943022 \
      -H "Authorization: Bearer ${JUDGE_MODEL_TOKEN}"

• Response example

  {
      "id": "batch_66f245a0-88d1-458c-8e1c-a819a5943022",
      "object": "batch",
      "deleted": "true"
  }

• Request parameters

  Parameter	Type	Required	Description
  batch_id	string	Yes	The ID of the batch task.

• Response parameters

  Parameter	Type	Description
  id	string	The ID of the batch task.
  object	string	The type of the deleted batch task.
  deleted	string	Indicates whether the batch task was deleted.

Batch task description