How to create and query a batch processing job - Object Storage Service

Use batch processing to perform large-scale operations on existing objects in a bucket. For example, you can convert all .docx objects in a specific directory to the PDF format, or transcode all MOV objects to the MP4 format.

Note

The new data processing APIs and SDKs are available in regions with an IMM endpoint.

Precautions

A batch processing job generates a corresponding data processing task for each object that meets your specified conditions. We recommend that you test your configuration on a small set of objects using a more specific prefix to verify the outcome.
Suspending a batch processing job immediately stops the creation of new data processing tasks. However, tasks that have already been submitted will continue to run until they are complete.
To prevent unexpected behavior such as recursive processing and overwriting of source objects, the input path must not be a prefix of the output path.
The output path cannot end with a forward slash (/). You must specify a full output object path, including the extension. The object name can include variables. For more information about variables, see Variables.
If the output path does not end with /, the filename must contain at least one variable. Otherwise, multiple output files will overwrite each other.
When you use the {key}, {barename}, or {basename} variables in the output path, ensure that the output path is different from the input path. Otherwise, source objects might be overwritten or trigger recursive processing.

Create a batch processing job

Log on to the OSS console.
In the left-side navigation pane, click Buckets, and then click the target bucket.
In the left-side navigation pane, choose data processing.
Based on the type of objects you want to process, choose Document Processing, Media Processing, or another data processing feature provided by Intelligent Media Management (IMM). This topic uses document processing as an example.
Click the Batch Processing tab, and then click Create Batch Processing.

In the Create Batch Processing panel, configure the settings in the Input Configuration, Output Configuration, Permission Configuration, and Notification Configuration sections.

Parameter	Description
Input bucket	The bucket that contains the source objects for the batch processing job.
mode	You can select either Select File or Select Prefix. Select File: Process one or more specific objects. Select Prefix: Process all objects in a specified directory that match a filter rule.
Input path	If you set mode to Select File, select one or more source objects to process.
Path prefix	If you set mode to Select Prefix, specify a path prefix to identify the directory that contains the source objects. You can use a filter rule to further refine which objects are processed.
Filter rule	If you set mode to Select Prefix, you can use a filter rule along with the path prefix to select the objects to process. All: Process all objects in the directory that is specified by the path prefix. File Prefix: Process objects in the specified directory whose names match a specific prefix. To specify multiple prefixes, separate them with commas (,). File Suffix: Process objects in the specified directory whose names match a specific suffix. To specify multiple suffixes, separate them with commas (,).
Style	Select a processing style for the objects. If no styles are available, click Add Style in the drop-down list to create a new one.
Output path	Specify the storage path for the processed objects. The path cannot end with a forward slash (`/`). You must use variables in the output path to prevent the outputs of multiple tasks from overwriting each other. For more information, see Variables.
service role	The service role that grants Intelligent Media Management (IMM) permission to access other cloud resources, such as OSS. We recommend that you use the `AliyunIMMBatchTriggerRole` service role. If this role does not exist, you can create it on the Cloud Resource Access Authorization page.
message queue	When a task is complete, a notification with the processing result is sent to the specified MNS topic. For more information, see Overview.

Click OK.

After you create a batch processing job, the system generates a data processing task for each source object. You are charged for the execution of these tasks. For more information, see Billing.

Query a batch processing job

Log on to the OSS console.
In the left-side navigation pane, click Buckets, and then click the target bucket.
In the left-side navigation pane, choose data processing.
Based on the object type, choose Document Processing, Media Processing, or another data processing feature provided by Intelligent Media Management (IMM).

Click the Batch Processing tab to view the list of batch processing jobs. You can use the pagination controls at the bottom of the page to navigate through the list.

The following table describes the job statuses.

Status	Description
Ready	The job is ready to run. Newly created jobs are in this state.
Running	The job is running.
Succeeded	The job has completed successfully.
Suspended	The job is suspended.
Failed	The job failed due to an unrecoverable error.

In the Batch Processing ID search box, enter a job ID and click the search icon to find a specific job.
In the actions column, click View Details to view the details of a batch processing job.
Click the expand icon next to a job ID to view all the tasks within that job. You can search for tasks by source object path or status. In the actions column for a task, click View Details to view its details.