PutDataPipelineConfiguration - Object Storage Service - Alibaba Cloud Documentation Center

Calls the PutDataPipelineConfiguration operation to create a data pipeline (DataPipeline) rule. After the rule is created, OSS automatically calls Alibaba Cloud Model Studio to vectorize the raw data in the source bucket and writes the generated vector data to the specified index in the vector bucket.

Note

This feature is currently available to whitelisted users. To use it, contact technical support to submit an application.

Usage notes

A single Alibaba Cloud account (UID) can create up to 1,000 data pipeline rules in a single region.
DataPipelineName must be unique within the same account and region.
A rule cannot be modified after it is created. To adjust the configuration, delete the rule and create a new one.
The source standard bucket and the destination vector bucket must be located in the same region. Data pipelines do not support cross-region operations.
A single rule currently supports writing to only one vector index (VectorIndexNames is a list type, reserved for future multi-index capabilities). When the destination index is full, the rule is automatically paused (Paused).
Vectorization calls your own Alibaba Cloud Model Studio service. Embedding fees are charged by Model Studio.
When you use a RAM user (instead of the Alibaba Cloud account) to operate AI VibeFlow, you must complete the following permission configuration in advance.

Request syntax

POST /?dataPipeline&action=putDataPipelineConfiguration&dataPipelineName=my-data-pipeline&role=acs:ram::<AccountId>:role/my-data-pipeline-role HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
Date: GMT Date
Authorization: SignatureValue
Content-Type: application/xml

<?xml version="1.0" encoding="UTF-8" ?>
<DataPipelineConfiguration>
  <DataPipelineDescription>Use a Model Studio multimodal model to vectorize business data</DataPipelineDescription>
  <Sources>
      <InputBucket>my-bucket</InputBucket>
      <InputDataScope>All</InputDataScope>
      <IgnoreDelete>true</IgnoreDelete>
      <FilterConfiguration>
          <PrefixSet>prefix1/</PrefixSet>
          <PrefixSet>prefix2/prefix3/</PrefixSet>
          <ObjectMediaTypes>text</ObjectMediaTypes>
          <ObjectMediaTypes>image</ObjectMediaTypes>
          <ObjectMediaTypes>video</ObjectMediaTypes>
      </FilterConfiguration>
  </Sources>
  <DataPipelineEmbeddingConfiguration>
      <EmbeddingProvider>bailian</EmbeddingProvider>
      <ApiKey>sk-xxxx</ApiKey>
      <Model>qwen2.5-vl-embedding</Model>
      <FPS>1</FPS>
  </DataPipelineEmbeddingConfiguration>
  <Destination>
      <VectorBucketName>my-vector-bucket</VectorBucketName>
      <VectorIndexNames>my-index</VectorIndexNames>
      <VectorKeyPrefix></VectorKeyPrefix>
      <ObjectTagToMetadata>key1</ObjectTagToMetadata>
      <ObjectTagToMetadata>key2</ObjectTagToMetadata>
      <UsermetaToMetadata>x-oss-meta-key1</UsermetaToMetadata>
  </Destination>
  <DataPipelineError>
      <ErrorMode>ignoreAndRecord</ErrorMode>
      <ErrorBucket>my-error-bucket</ErrorBucket>
      <ErrorPrefix>error-output/</ErrorPrefix>
  </DataPipelineError>
</DataPipelineConfiguration>

Request headers

This operation uses only common request headers. For more information, see Common Request Headers.

Request parameters

In the request, dataPipelineName and role are passed as query parameters, and the other configurations are passed in the request body (XML).

Name	Type	Required	Description	Example
DataPipelineName	string	Yes	The rule name. It can be up to 64 characters in length and can contain uppercase and lowercase letters, digits, and special characters (- _). The name must be unique within an account and region.	my-data-pipeline
role	string	Yes	The role used to create the data pipeline. The role must have the required permissions on the source bucket, the destination vector bucket, and the optional error output bucket.	acs:ram::<AccountId>:role/my-data-pipeline-role
DataPipelineDescription	string	No	The rule description. It can be up to 200 characters in length.	Image vectorization task for the business department
Sources	container	Yes	The data source information. Currently, only one Source is supported.	-
InputBucket	string	Yes	The name of the standard bucket that stores the raw data. Parent node: Source.	mybucket
InputDataScope	string	Yes	The processing mode. Valid values: Historical (existing data only), Incremental (incremental data only), and All (existing and incremental data). Parent node: Source.	All
IgnoreDelete	bool	No	Specifies whether to retain the written vectors when an object is deleted. true (default): The vectors are not deleted. false: The corresponding vectors are deleted when the object is deleted. Parent node: Source.	true
FilterConfiguration	container	No	The object filter rule. If it is not set, all files in the bucket are vectorized. Parent node: Source.	-
PrefixSet	[]string	No	The set of object prefixes. Only objects that match the prefix are processed. The maximum length of a single prefix is 1,023 charactersUp to 1. Parent node: FilterConfiguration.	["source/images/"]
ObjectMediaTypes	[]string	No	The types of objects to process. A maximum of 3 types are supported. Valid values: text (.txt .log .md .csv .json .xml .yaml .yml), image, and video (the file name extension must meet the requirements of the Model Studio model). Parent node: FilterConfiguration.	["text", "image"]
DataPipelineEmbeddingConfiguration	container	Yes	The vectorization configuration.	-
EmbeddingProvider	string	Yes	The vectorization provider. Currently, only bailian is supported. Parent node: DataPipelineEmbeddingConfiguration.	bailian
ApiKey	string	Yes	The API key of Alibaba Cloud Model Studio. It is prefixed with sk-, is 35 to 128 characters in length, and contains only letters and digits except for the prefix. Parent node: DataPipelineEmbeddingConfiguration.	sk-xxxx
Model	string	Yes	The name of the Model Studio vector model. Parent node: DataPipelineEmbeddingConfiguration.	qwen2.5-vl-embedding
FPS	float	No	The frame rate at which Model Studio extracts frames from a video during video vectorization. Valid values: (0,1]. Default value: 1.0. Parent node: DataPipelineEmbeddingConfiguration.	1
Destination	container	Yes	The result output configuration.	-
VectorBucketName	string	Yes	The name of the vector bucket that stores the vector data. Parent node: Destination.	my-vector-bucket
VectorIndexNames	[]string	Yes	The list of vector index names that store the vector data. Currently, only one index is supported. Parent node: Destination.	["my-index"]
VectorKeyPrefix	string	No	The naming method of the destination vector key. If it is empty, the vector key is the same as the object key. If it is not empty, the prefix is added before the object key. Parent node: Destination.	-
ObjectTagToMetadata	[]string	No	The object tags to write as metadata of the vectors. Parent node: Destination.	["tag1", "tag2"]
UsermetaToMetadata	[]string	No	The user metadata to write as metadata of the vectors. Parent node: Destination.	["x-oss-meta-key1"]
DataPipelineError	container	Yes	The conversion error handling configuration. Note: Throttling errors or 5xx errors are retried indefinitely, with a minimum retry interval of 1s and a maximum of 30m.	-
ErrorMode	string	Yes	The handling mode when object vectorization fails. Valid values: ignore, ignoreAndRecord (ignore the failure and record the failure information under ErrorPrefix in ErrorBucket), and abort (stop when an error occurs). Parent node: DataPipelineError.	ignoreAndRecord
ErrorBucket	string	No	The error output bucket. This parameter is required when ErrorMode=ignoreAndRecord, and it cannot be the same as the Source bucket. Parent node: DataPipelineError.	my-error-bucket
ErrorPrefix	string	No	The error output prefix. This parameter is required when ErrorMode=ignoreAndRecord. If the conversion of an object with the key path/a.jpg fails, the error information (including ErrorCode, ErrorMessage, and RequestId) is written to ErrorPrefix/<DataPipelineName>/path/a.jpg-<timestamp>.log in ErrorBucket. Parent node: DataPipelineError.	error-output/

Response headers

This operation uses only common response headers. For more information, see Common Response Headers.

Examples

Sample request:

POST /?dataPipeline&action=putDataPipelineConfiguration&dataPipelineName=my-data-pipeline&role=acs:ram::<AccountId>:role/my-data-pipeline-role HTTP/1.1
Host: oss-cn-hangzhou.aliyuncs.com
Date: Thu, 11 Dec 2025 13:08:38 GMT
Authorization: OSS4-HMAC-SHA256 Credential=LTAI********/20251211/cn-hangzhou/oss/aliyun_v4_request,Signature=********
Content-Type: application/xml

<?xml version="1.0" encoding="UTF-8" ?>
<DataPipelineConfiguration>
  <Sources>
      <InputBucket>my-bucket</InputBucket>
      <InputDataScope>All</InputDataScope>
  </Sources>
  <DataPipelineEmbeddingConfiguration>
      <EmbeddingProvider>bailian</EmbeddingProvider>
      <ApiKey>sk-xxxx</ApiKey>
      <Model>qwen2.5-vl-embedding</Model>
  </DataPipelineEmbeddingConfiguration>
  <Destination>
      <VectorBucketName>my-vector-bucket</VectorBucketName>
      <VectorIndexNames>my-index</VectorIndexNames>
  </Destination>
  <DataPipelineError>
      <ErrorMode>ignore</ErrorMode>
  </DataPipelineError>
</DataPipelineConfiguration>

Sample response:

HTTP/1.1 200 OK
x-oss-request-id: E71AFFDC-25BA-5C4C-AA5D-311029******
Date: Thu, 11 Dec 2025 13:08:38 GMT
Content-Type: application/xml

SDK

For SDK examples of data pipeline (DataPipeline) operations in each language, see Create a data pipeline (Java SDK V2) and Create a data pipeline (Go SDK V2).

ossutil command-line tool

For the ossutil command that corresponds to this operation, see Create a data pipeline.

Error codes

Error code	HTTP status code	Description
InvalidArgument	400	The DataPipelineConfiguration parameter provided in the request is invalid. For example, the model does not support the dimensions of the index, the model does not support the ObjectMediaTypes, or the input and output regions are inconsistent.
DataPipelineConfigurationExceedLimit	400	The number of created data pipelines has reached the upper limit. A single user can create up to 1,000 rules in a region.
DataPipelineNameAlreadyExist	409	The specified DataPipelineConfiguration already exists and cannot be created again.
AccessDenied	403	No user authentication information is provided in the request, or you do not have the permissions to perform the operation.
QpsLimitExceeded	429	QPS throttling (the request rate is limited). When the HTTP status code is 5xx, only part of the data in the same batch of requests may be written successfully. Writes are not atomic.