Alibaba Cloud Model Studio:Batch inference

Step 1: Prepare the input file

Before creating a task, prepare a JSONL file that meets the following requirements:

• Format: UTF-8 encoded JSONL (one JSON object per line).

• Scale limits: Up to 50,000 requests per file, and a maximum file size of 500 MB.

  If your dataset exceeds these limits, split it into multiple files and submit them as separate tasks.

• Per-line limit: Each JSON object can be up to 6 MB and must not exceed the model's context window.

• Consistency: All requests within the same file must use the same model.

• Unique identifier: Each request must include a custom_id field that is unique within the file for result matching. The custom_id supports a maximum of 256 characters. If this limit is exceeded, the task validation fails. To return a longer identifier, use a custom field in the metadata parameter when creating the task. For more information, see Use metadata to return custom identifiers.

Each JSON object must adhere to the following schema:

{"custom_id":"1","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen-max","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"Hello!"}]}}
{"custom_id":"2","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen-max","messages":[{"role":"system","content":"You are a helpful assistant."},{"role":"user","content":"What is 2+2?"}]}}

{"custom_id":"request-1","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen3.5-plus","enable_thinking":false,"messages":[{"role":"user","content":"Hello"}]}}

{"custom_id":"request-2","method":"POST","url":"/v1/chat/completions","body":{"model":"qwen3.5-plus","enable_thinking":true,"thinking_budget":50,"messages":[{"role":"user","content":"Please analyze the following question"}]}}

Step 2: Create a batch inference task

1. On the Batch Inference** page, click Create Batch.

2. In the dialog box that appears, enter a Task Name and Description, set the Maximum Waiting Time (from 1 to 14 days), and upload your JSONL file.

   You can click Download Sample File to obtain the template.

   

3. When you are done, click Confirm.

Step 3: Monitor and manage tasks

• View:

  • On the task list page, view the task's progress (processed requests/total requests) and Status.

  • Search by task name or ID, or filter by workspace to quickly locate a specific task.

• Manage:

  • Cancel: You can cancel tasks that are in the Executing state in the Actions column.

  • Troubleshoot: For a failed task, hover over the status to view an error summary or download the error file for details.

Step 4: Download results

When the task is complete, click View Results to download the output file:

• Result file: Records all successful requests and their response results.

• Error file (if any): Records all failed requests and their error details.

Both files contain a custom_id field, which is used to match with the original input data, associate results, or locate errors.

Step 5: View usage statistics (optional)

On the Model Monitoring page, you can filter and view usage statistics for batch inference.

Metadata fields

metadata is an optional parameter for creating a Batch task. It supports the following fields:

• ds_name: The name of the task. This name is displayed in the Task Name column on the console.

• ds_description: The description of the task. This description is displayed in the Task Description column in the console.

• Custom fields: In addition to the official fields, the metadata object also supports any custom fields, and their values are not limited to 256 characters. When you query task details, all custom fields are returned in full.

Code example

The following example shows how to use a custom field in metadata to pass back an identifier that is longer than 256 characters:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("DASHSCOPE_API_KEY"),
    base_url="https://dashscope.aliyuncs.com/compatible-mode/v1",
)

batch = client.batches.create(
    input_file_id="file-batch-xxxxxxxxxxxxxxxxxxxx",
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={
        "ds_name": "my_batch_task",
        "ds_description": "A description for my batch inference task",
        "my_custom_field": "The value of this field can exceed 256 characters and is used to pass back longer identifier information..."
    }
)
print(batch)

After a task is successfully created, you can call the GET /v1/batches/{batch_id} operation to retrieve the complete metadata information, which includes all custom fields and their full content.

In a production environment, use the OpenAI-compatible API to automate the creation and management of batch tasks. The core workflow is as follows:

1. Upload a file

   Call POST /v1/files to upload a file. Record the file ID that is returned.

2. To create a task, pass in the file ID , call POST /v1/batches, and record the returned batch_id.

3. Poll status by using batch_id to poll GET /v1/batches/{batch_id}. When the status changes to completed, record the output_file_id and stop polling.

4. To download the result file, use the output_file_id to call GET /v1/files/{output_file_id}/content.

• Pricing: For all successful requests, both input and output tokens are priced at 50% of the real-time inference price for the corresponding model. For details, see Models and Pricing.

• Billing scope:

  • You are billed only for successfully executed requests within a task.

  • File parsing failures, task execution failures, or line-level request errors do not incur charges.

  • For cancelled tasks, any requests that were successfully completed before cancellation are billed normally.