Alibaba Cloud Model Studio:OpenAI compatible - Batch

Before you start a formal batch task, you can use the test model batch-test-model to perform a complete end-to-end test. This test includes verifying input data, creating a task, querying the task, and downloading the result file. Note the following:

• The test file must meet the requirements for an input file. The file size cannot exceed 1 MB, and the number of lines cannot exceed 100.

• Concurrency limit: The maximum number of parallel tasks is 2.

• Resource usage: The test model does not perform the inference process, so no model inference fees are incurred.

The procedure is as follows:

1. Prepare a test file

   • Download the sample file test_model.jsonl, which contains request information, to your local machine. Make sure that the file is in the same directory as the Python script that is described later in this topic.

   • Sample content: The model parameter is set to batch-test-model, and the url parameter is set to /v1/chat/ds-test.

     {"custom_id":"1","method":"POST","url":"/v1/chat/ds-test","body":{"model":"batch-test-model","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello! How can I help you?"}]}}
     {"custom_id":"2","method":"POST","url":"/v1/chat/ds-test","body":{"model":"batch-test-model","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"}]}}

2. Run the script

   • Execute the following Python script.

     You can modify the code as needed to adjust the file path or other parameters.

     import os
     from pathlib import Path
     from openai import OpenAI
     import time
     
     # Initialize the client.
     client = OpenAI(
         # If the environment variable is not set, you can replace the following line with api_key="sk-xxx". 
         # However, we do not recommend hard-coding the API key in your code in a production environment to reduce the risk of leakage.
         api_key=os.getenv("DASHSCOPE_API_KEY"),
         base_url="https://dashscope-intl.aliyuncs.com/compatible-mode/v1"  # The base_url of Alibaba Cloud Model Studio.
     )
     
     def upload_file(file_path):
         print(f"Uploading the JSONL file that contains the request information...")
         file_object = client.files.create(file=Path(file_path), purpose="batch")
         print(f"File uploaded successfully. File ID: {file_object.id}\n")
         return file_object.id
     
     def create_batch_job(input_file_id):
         print(f"Creating a batch task based on the file ID...")
         # Note: The value of the endpoint parameter must be the same as the value of the url field in the input file. 
         # For the test model (batch-test-model), set this to /v1/chat/ds-test. For other models, set this to /v1/chat/completions.
         batch = client.batches.create(input_file_id=input_file_id, endpoint="/v1/chat/ds-test", completion_window="24h")
         print(f"Batch task created. Batch task ID: {batch.id}\n")
         return batch.id
     
     def check_job_status(batch_id):
         print(f"Checking the batch task status...")
         batch = client.batches.retrieve(batch_id=batch_id)
         print(f"Batch task status: {batch.status}\n")
         return batch.status
     
     def get_output_id(batch_id):
         print(f"Getting the output file ID for successful requests in the batch task...")
         batch = client.batches.retrieve(batch_id=batch_id)
         print(f"Output file ID: {batch.output_file_id}\n")
         return batch.output_file_id
     
     def get_error_id(batch_id):
         print(f"Getting the error file ID for failed requests in the batch task...")
         batch = client.batches.retrieve(batch_id=batch_id)
         print(f"Error file ID: {batch.error_file_id}\n")
         return batch.error_file_id
     
     def download_results(output_file_id, output_file_path):
         print(f"Printing and downloading the results of successful requests in the batch task...")
         content = client.files.content(output_file_id)
         # Print some of the content for testing.
         print(f"Printing the first 1,000 characters of the successful results: {content.text[:1000]}...\n")
         # Save the result file to your local machine.
         content.write_to_file(output_file_path)
         print(f"The complete output results have been saved to the local output file result.jsonl\n")
     
     def download_errors(error_file_id, error_file_path):
         print(f"Printing and downloading the information of failed requests in the batch task...")
         content = client.files.content(error_file_id)
         # Print some of the content for testing.
         print(f"Printing the first 1,000 characters of the failure information: {content.text[:1000]}...\n")
         # Save the error information file to your local machine.
         content.write_to_file(error_file_path)
         print(f"The complete failure information has been saved to the local error file error.jsonl\n")
     
     def main():
         # File paths
         input_file_path = "test_model.jsonl"  # Replace with your input file path.
         output_file_path = "result.jsonl"  # Replace with your output file path.
         error_file_path = "error.jsonl"  # Replace with your error file path.
         try:
             # Step 1: Upload the JSONL file that contains the request information to get the input file ID.
             input_file_id = upload_file(input_file_path)
             # Step 2: Create a batch task based on the input file ID.
             batch_id = create_batch_job(input_file_id)
             # Step 3: Check the batch task status until it is complete.
             status = ""
             while status not in ["completed", "failed", "expired", "cancelled"]:
                 status = check_job_status(batch_id)
                 print(f"Waiting for the task to complete...")
                 time.sleep(10)  # Wait 10 seconds and query the status again.
             # If the task fails, print the error message and exit.
             if status == "failed":
                 batch = client.batches.retrieve(batch_id)
                 print(f"Batch task failed. Error message:{batch.errors}\n")
                 print(f"For more information, see Error codes: https://www.alibabacloud.com/help/en/model-studio/developer-reference/error-code")
                 return
             # Step 4: Download the results. If an output file ID exists, print the first 1,000 characters of the successful results and download the complete results to a local output file.
             # If an error file ID exists, print the first 1,000 characters of the failure information and download the complete information to a local error file.
             output_file_id = get_output_id(batch_id)
             if output_file_id:
                 download_results(output_file_id, output_file_path)
             error_file_id = get_error_id(batch_id)
             if error_file_id:
                 download_errors(error_file_id, error_file_path)
                 print(f"For more information, see Error codes: https://www.alibabacloud.com/help/en/model-studio/developer-reference/error-code")
         except Exception as e:
             print(f"An error occurred: {e}")
             print(f"For more information, see Error codes: https://www.alibabacloud.com/help/en/model-studio/developer-reference/error-code")
     
     if __name__ == "__main__":
         main()

3. Verify the test results

   • The task status is completed.

   • The result file result.jsonl contains the fixed response {"content":"This is a test result."}.

     {"id":"a2b1ae25-21f4-4d9a-8634-99a29926486c","custom_id":"1","response":{"status_code":200,"request_id":"a2b1ae25-21f4-4d9a-8634-99a29926486c","body":{"created":1743562621,"usage":{"completion_tokens":6,"prompt_tokens":20,"total_tokens":26},"model":"batch-test-model","id":"chatcmpl-bca7295b-67c3-4b1f-8239-d78323bb669f","choices":[{"finish_reason":"stop","index":0,"message":{"content":"This is a test result."}}],"object":"chat.completion"}},"error":null}
     {"id":"39b74f09-a902-434f-b9ea-2aaaeebc59e0","custom_id":"2","response":{"status_code":200,"request_id":"39b74f09-a902-434f-b9ea-2aaaeebc59e0","body":{"created":1743562621,"usage":{"completion_tokens":6,"prompt_tokens":20,"total_tokens":26},"model":"batch-test-model","id":"chatcmpl-1e32a8ba-2b69-4dc4-be42-e2897eac9e84","choices":[{"finish_reason":"stop","index":0,"message":{"content":"This is a test result."}}],"object":"chat.completion"}},"error":null}

     If an error occurs, see Error messages for a solution.

After you verify the test, perform the following steps to run a formal batch task:

1. Prepare an input file that meets the requirements described in Input file. In the file, set the model parameter to a supported model and set the url parameter to /v1/chat/completions.

2. Replace the endpoint in the Python script.

   Important

   Make sure that the endpoint in the script is the same as the url parameter in the input file.

3. Run the script and wait for the task to complete. If the task is successful, an output file named result.jsonl is generated in the same directory.

   If the task fails, the program exits and prints an error message.

   If an error file ID exists, an error file named error.jsonl is generated in the same directory for you to review.

   Exceptions that occur during the process are caught and an error message is printed.

{"custom_id":"1","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen-max","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello!"}]}}
{"custom_id":"2","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen-max","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"}]}}

JSONL batch generation tool

You can use the tool below to quickly generate JSONL files. To prevent performance issues, process no more than 10,000 lines at a time. For larger amounts of data, process them in batches.

Input parameter settings

Response example

For more information, see the response example for Create a batch task.

Response parameters

For more information, see the response parameters for Create a batch task.

You can retrieve the content of the files specified by output_file_id and error_file_id in the response parameters by downloading the batch result file.

Input parameter settings

{
  "object": "list",
  "data": [
    {
      "id": "batch_xxx",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "errors": null,
      "input_file_id": "file-batch-xxx",
      "completion_window": "24h",
      "status": "completed",
      "output_file_id": "file-batch_output-xxx",
      "error_file_id": null,
      "created_at": 1722234109,
      "in_progress_at": 1722234109,
      "expires_at": null,
      "finalizing_at": 1722234165,
      "completed_at": 1722234165,
      "failed_at": null,
      "expired_at": null,
      "cancelling_at": null,
      "cancelled_at": null,
      "request_counts": {
        "total": 100,
        "completed": 95,
        "failed": 5
      },
      "metadata": {}
    },
    { ... }
  ],
  "first_id": "batch_xxx",
  "last_id": "batch_xxx",
  "has_more": true
}

Input parameter settings

Response example

For more information, see the response example for Create a batch task.

Response parameters

For more information, see the response parameters for Create a batch task.