Create CPFS Dataflow Tasks via CreateDataFlowTask API - Cloud Parallel File Storage - Alibaba Cloud - Cloud Parallel File Storage

Creates a data flow task.

Operation description

Usage notes for CPFS
- Only CPFS 2.2.0 and later support data flow. You can view the version information on the file system details page in the console.
- Data flow tasks are executed asynchronously. You can call DescribeDataFlowTasks to query the task execution status. The time required for a task depends on the amount of data to be imported or exported. If a large amount of data is involved, split the data into multiple tasks.
- You can create a data flow task only on a data flow that is in the Running state.
- When you manually execute a data flow task, the automatic data update task of the data flow is interrupted and waits.
- When you create an export task, make sure that the total length of the absolute path of the file to be exported from CPFS does not exceed 1,023 characters.
Usage notes for CPFS for Lingjun
- Only CPFS for Lingjun 2.4.0 and later support data flow. You can view the version information on the file system details page in the console.
- Data flow tasks are executed asynchronously. You can call DescribeDataFlowTasks to query the task execution status. The time required for a task depends on the amount of data to be imported or exported. If a large amount of data is involved, split the data into multiple tasks.
- You can create a data flow task only on a data flow that is in the Running state.
- When you create an export task, make sure that the total length of the absolute path of the file to be exported from CPFS for Lingjun does not exceed 1,023 characters.
- CPFS for Lingjun supports two task types: batch tasks and streaming tasks. For more information, see Task type overview.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlowTask

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

None

Request parameters

Parameter	Type	Required	Description	Example
FileSystemId	string	Yes	The file system ID. General-purpose CPFS: The ID must start with `cpfs-`, such as cpfs-125487**. CPFS for Lingjun: The ID must start with `bmcpfs-`, such as bmcpfs-0015**.	cpfs-099394bd928c****
DataFlowId	string	Yes	The data flow ID.	df-194433a5be31****
SrcTaskId	string	No	When you specify SrcTaskId, enter the data flow task ID. The system copies the TaskAction, DataType, and EntryList parameter information from the specified data flow task, and you do not need to specify these parameters separately. Note Data flow streaming tasks are not supported.	task-29ee8e890f45****
TaskAction	string	No	The client token that is used to ensure the idempotence of the request. You can use the client to generate the token, but you must make sure that the token is unique among different requests. The token can contain only ASCII characters and cannot exceed 64 characters in length. For more information, see How to ensure idempotence. Note If you do not specify this parameter, the system automatically uses the value of RequestId as the value of ClientToken. The value of RequestId may be different for each API request.	123e4567-e89b-12d3-a456-42665544****
DataType	string	No	Specifies whether to perform a dry run. A dry run checks parameter validity, verifies inventory, and performs other checks without actually creating the instance or incurring fees. Valid values: true: performs a dry run without creating the instance. The system checks whether the required parameters are specified, whether the request format is valid, whether business restrictions are met, and whether NAS resources are available. If the check fails, the corresponding error is returned. If the check succeeds, the HTTP status code 200 is returned, but TaskId is empty. false (default): performs a dry run and sends the request. If the check succeeds, the instance is created.	false
Directory	string	No	The data flow node type. Valid values: Import: performs data import of specified data from the source storage to CPFS. Export: exports specified data from CPFS to the source storage. StreamImport: performs batch export of specified data from the source storage to CPFS as a data stream. StreamExport: performs batch export of specified data from CPFS to the source storage as a data stream. Evict: releases data blocks of files on CPFS. After the release, only the metadata of the files is retained on CPFS. You can still query the files, but the data blocks are purged and do not occupy storage capacity on CPFS. When you access the file data, the data is loaded from the source storage on demand. Inventory: obtains the file checklist managed by data flow on CPFS, which provides the cache status of files in the data flow. Note CPFS for Lingjun supports only Import, Export, StreamImport, and StreamExport. StreamImport and StreamExport are supported only in CPFS for Lingjun 2.6.0 and later.	Import
EntryList	string	No	The type of data on which the data flow task operates. Valid values: Metadata: the metadata of files, including the timestamp, ownership, permission, and other attributes. If you select Metadata, only the metadata of files is imported. You can view the files, but when you access the file data, the data is loaded from the source storage on demand. Data: the data blocks of files. MetaAndData: the metadata and data blocks of files.	Metadata
DryRun	boolean	No	The source directory of the data. Limits: The directory must be 1 to 1,023 characters in length. The directory must be encoded in UTF-8. The directory must start and end with a forward slash (/). Only one directory can be specified at a time. When TaskAction is set to Export, this directory must be a relative path within FileSystemPath. When TaskAction is set to Import, this directory must be a relative path within SourceStoragePath. When TaskAction is set to StreamExport, this directory must be a relative path within FileSystemPath. When TaskAction is set to StreamImport, this directory must be a relative path within SourceStoragePath. Note StreamImport and StreamExport are supported only in CPFS for Lingjun 2.6.0 and later.	/path_in_cpfs/
ClientToken	string	No	The list of files on which the data flow task is executed. Limits: The list must be encoded in UTF-8. The total length of the file list must be less than 64 KB. The file list is in JSON format. Each file path must be 1 to 1,023 characters in length and must start with a forward slash (/). When TaskAction is set to Import, each element in the list represents an OSS object name. When TaskAction is set to Export, each element in the list represents a CPFS file path.	["/path_in_cpfs/file1", "/path_in_cpfs/file2"]
ConflictPolicy	string	No	The OSS directory that contains CSV files whose content specifies the data to be synchronized. Limits: The directory must start and end with a forward slash (/). The directory is case-sensitive. The directory must be 1 to 1,023 characters in length. The directory must be encoded in UTF-8. Note TransferFileListPath, Directory, and EntryList are mutually exclusive parameters. You can specify only one of them. This parameter specifies an existing path in OSS. The *.csv files in the path are stored in OSS. TransferFileListPath supports only Import and Export. For Import, the files or directories specified in the CSV files are imported from OSS to CPFS. For Export, the files or directories specified in the CSV files are exported from CPFS to OSS. The CSV file format must include the Name and Type columns. Name is a relative path, and Type supports two values: dir and file. If Type is dir, the Name value must end with a forward slash (/). Only CPFS for Lingjun supports this feature.	/test_oss_path/
DstDirectory	string	No	Filters directories under the specified directory and transfers the content of the included folders. Note This parameter takes effect only when the Directory parameter is specified. Each folder path must be 1 to 1,023 characters in length and must start and end with a forward slash (/). The total length must not exceed 3,000 characters. Only CPFS for Lingjun supports this feature.	["/test/","/test1/"]
CreateDirIfNotExist	boolean	No	The target directory to which the data flow task maps. Limits: The directory must start and end with a forward slash (/). /../ is not supported. The directory must be 1 to 1,023 characters in length. The directory must be encoded in UTF-8. Only one directory can be specified at a time. When TaskAction is set to Export, this directory must be a relative path within SourceStoragePath. When TaskAction is set to Import, this directory must be a relative path within FileSystemPath. When TaskAction is set to StreamExport, this directory must be a relative path within SourceStoragePath. When TaskAction is set to StreamImport, this directory must be a relative path within FileSystemPath. Note StreamImport and StreamExport are supported only in CPFS for Lingjun 2.6.0 and later.	/path_in_cpfs/
Includes	string	No	Specifies whether to enable automatic creation of the folder if it does not exist. Valid values: true: enables automatic creation of the folder. false (default): does not enable automatic creation of the folder. Note This parameter takes effect when TaskAction is set to Import. Only CPFS for Lingjun 2.6.0 and later support this parameter.	false
TransferFileListPath	string	No	The conflict policy for files with the same name. Valid values: SKIP_THE_FILE: skips files with the same name. KEEP_LATEST: compares the update time and keeps the latest version. OVERWRITE_EXISTING: forcibly overwrites files with the same name. Note This parameter is required when the file system type is CPFS for Lingjun.	SKIP_THE_FILE

Response elements

Element	Type	Description	Example
	object
RequestId	string	The request ID.	2D69A58F-345C-4FDE-88E4-BF518948****
TaskId	string	The data flow task ID.	task-38aa8e890f45****

Examples

Success response

JSON format

{
  "RequestId": "2D69A58F-345C-4FDE-88E4-BF518948****",
  "TaskId": "task-38aa8e890f45****"
}

Error codes

HTTP status code	Error code	Error message	Description
400	IllegalCharacters	The parameter contains illegal characters.	The parameter contains illegal characters.
400	MissingFileSystemId	FileSystemId is mandatory for this action.	The FileSystemId parameter is not specified.
400	MissingDataFlowId	DataFlowId is mandatory for this action.	The DataFlowId parameter is not specified.
400	InvalidFilesystemVersion.NotSupport	This Api does not support this fileSystem version.	This Api does not support this fileSystem version.
403	OperationDenied.InvalidState	The operation is not permitted when the status is processing.	The operation is not permitted when the status is processing.
403	OperationDenied.DependencyViolation	The operation is denied due to dependancy violation.	The conditions for this operation are not met.
403	OperationDenied.DataFlowNotSupported	The operation is not supported.	The file system does not support dataflow.
404	InvalidParameter.InvalidFormat	The EntryList format is invalid.	The format of the specified EntryList does not meet the requirements.
404	InvalidParameter.SizeTooLarge	The specified EntryList size exceeds 64 KB.	The size of the specified EntryList exceeds 64 KB.
404	InvalidDataFlow.NotFound	The specified data flow does not exist.	The specified data flow does not exist.
404	InvalidTaskAction.NotSupported	The task action is not supported.	The specified task type is not supported.
404	InvalidTaskAction.PermissionDenied	The task action is not allowed.	The specified task type is not allowed, such as system replication.
404	InvalidSrcTaskId.NotFound	The SrcTaskId is not found.	The specified SrcTaskId does not exist.
404	InvalidDataType.NotSupported	The data type is not supported.	The specified data type is not supported.
404	InvalidSrcTaskId.TaskIdInvalid	Source task ID is invalid.	The specified task ID to be copied is invalid.
404	InvalidSrcTaskId.TaskIdNotFound	Source task ID is not found.	The specified task ID to be copied does not exist.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.