CreateDataFlow API reference - Cloud Parallel File Storage - Alibaba Cloud - Cloud Parallel File Storage

Creates a data flow between a General-purpose CPFS or CPFS for Lingjun file system and a source storage.

Operation description

This operation is applicable to the following services:

Service	File system ID format	Minimum version that supports data flows
General-purpose CPFS	Starts with `cpfs-`, such as cpfs-125487****	2.2.0 and later
CPFS for Lingjun	Starts with `bmcpfs-`, such as bmcpfs-0015****	2.4.0 and later

Note

: General-purpose CPFS and CPFS for Lingjun share the same set of API operations, but differ in parameter values and supported features. Refer to the corresponding section based on the service you use.

Basic operations
- You can create a data flow only when the General-purpose CPFS or CPFS for Lingjun file system is in the Running state.
- You can create a maximum of 10 data flows for a General-purpose CPFS or CPFS for Lingjun file system.
- Creating a data flow typically takes 2 to 5 minutes. You can call the DescribeDataFlows operation to check whether the data flow is created.
Permissions.

When you create a data flow, Cloud Parallel File Storage assumes the AliyunServiceRoleForNasOssDataflow and AliyunServiceRoleForNasEventNotification service-linked roles to associate the service for the data stream. For more information, see Service-linked roles for Cloud Parallel File Storage.
Usage notes for General-purpose CPFS.

This section applies to General-purpose CPFS file systems whose IDs start with cpfs-.
- Billing
  - You are billed for a data flow based on the data flow bandwidth. For more information, see Billing of General-purpose CPFS.
  - If you enable auto-refresh, EventBridge is used to collect object modification events from the source OSS bucket. You are billed for these events. For more information, see Billing of EventBridge.
- Data flow specifications
  - The data flow bandwidth (Throughput) supports three specifications: 600 MB/s, 1200 MB/s, and 1500 MB/s. The data flow bandwidth is the maximum transmission bandwidth that the data flow can achieve when importing or exporting data.
  - Creating a data flow consumes one vSwitch IP address used by a General-purpose Cloud Parallel File Storage mount target. Make sure that the vSwitch has sufficient IP addresses.
  - Resource check: If you set the DryRun Settings parameter to true, the system checks whether the resources required to create a data flow (data stream) of the specified specification are sufficient.
- Fileset
  - The destination of a data flow is a fileset in a General-purpose CPFS file system. A fileset is a new directory tree structure in a General-purpose CPFS file system. It is a small file system within the parent file system and has an independent inode space and management capabilities.
  - The fileset must already exist when you create a data flow, and it cannot be nested with other filesets. You can create only one data flow on a fileset, which corresponds to one source storage.
  - A fileset can contain a maximum of 1 million files. If the number of files imported from the OSS bucket exceeds this limit, a no space error is reported when you create new files.
Note
If the fileset already contains data, the existing data in the fileset is cleared and replaced with the data synchronized from the OSS bucket after the data flow is created.
- Auto-refresh
  - After you configure auto-refresh, when the data in the source storage changes, the changed metadata undergoes automatic synchronization to the General-purpose Cloud Parallel File Storage file system. The changed data is loaded on demand when users access the files, or you can start a data flow node to load the data.
  - Auto-refresh relies on EventBridge to collect object modification events from the source OSS bucket. You must first activate EventBridge.
  - The scope of auto-refresh is a prefix, which is specified by the RefreshPath parameter. You can configure a maximum of five auto-refresh folders for a data flow (data stream).
  - The auto-refresh interval (AutoRefreshInterval) is the time interval configured in the Settings by General-purpose Cloud Parallel File Storage to check whether data updates exist within the prefix in the source OSS bucket. If data updates exist, an auto-refresh node is started. If the frequency of object modification events in the source OSS bucket exceeds the processing capability of the General-purpose Cloud Parallel File Storage data flow, auto-refresh nodes accumulate, metadata updates are delayed, and the status of the data flow becomes Misconfigured. To resolve this issue, you can upgrade the data flow specification or reduce the modification frequency in the OSS bucket.
  - When you configure auto-refresh for a prefix in a General-purpose Cloud Parallel File Storage data flow, an event bus is created on the user side, and an event rule is created for the prefix of the source OSS bucket. When an object in the prefix of the source OSS bucket is modified, an OSS event is generated in EventBridge and processed by the General-purpose Cloud Parallel File Storage data flow.
  Note
  The event bus and event rule created by General-purpose Cloud Parallel File Storage in EventBridge contain the Create for cpfs auto refresh description. You cannot modify or delete the event bus or event rule. Otherwise, auto-refresh cannot remain in a Normal state.
- Source storage
  - Only OSS is supported Cloud Parallel File Storage. The source storage (SourceStorage) of a data flow must be an OSS bucket.
  - General-purpose CPFS data flows support encrypted and unencrypted access to OSS. If you select the encrypted (SSL) access method, make sure that encryption in transit is enabled for the OSS bucket.
  - If multiple General-purpose CPFS data flows or multiple data flows of the same General-purpose CPFS file system use the same OSS bucket Cloud Parallel File Storage, you must enable versioning for the OSS bucket to prevent data conflicts when multiple General-purpose CPFS file systems export data to the same source.
  - Cross-region OSS data flows are not supported. The OSS bucket must be in the same region as the General-purpose CPFS file system.
    Note
    Before you create a data flow, you must add a tag (key: cpfs-dataflow, value: true) to the source OSS bucket so that the General-purpose CPFS data flow can access the data in the bucket. Do not delete or modify this tag while the data flow is in use. Otherwise, the General-purpose CPFS data flow cannot access the data in the bucket.
Usage notes for CPFS for Lingjun.

This section applies to CPFS for Lingjun file systems whose IDs start with bmcpfs-.
- Source storage
  - Only OSS is supported as the source storage. The source storage (SourceStorage) of a data flow must be an OSS bucket.
  - CPFS for Lingjun data flows support encrypted and unencrypted access to OSS. If you select the encrypted (SSL) access method, make sure that encryption in transit is enabled for the OSS bucket.
  - If multiple CPFS for Lingjun data flows or multiple data flows of the same CPFS for Lingjun file system use the same OSS bucket as the source storage, you must enable versioning for the OSS bucket to prevent data conflicts when multiple CPFS for Lingjun file systems export data to the same source.
  - Cross-region OSS data flows are not supported. The OSS bucket must be in the same region as the CPFS for Lingjun file system.
  - CPFS for Lingjun 2.6.0 and later support creating data flows by using cross-account OSS buckets.
  - You need to set the account id parameter only when you use a cross-account OSS bucket.
  - Before you use a cross-account OSS bucket, you must grant permissions to the account. For more information, see Authorize cross-account data flows.
    Note
    Before you create a data flow, you must add a tag (key: cpfs-dataflow, value: true) to the source OSS bucket so that the CPFS for Lingjun data flow can access the data in the bucket. Do not delete or modify this tag while the data flow is in use. Otherwise, the CPFS for Lingjun data flow cannot access the data in the bucket.
- Limits of data flows on file systems.
  - You cannot rename a non-empty directory in the file system path associated with the data flow. Otherwise, a Permission Denied or Directory Not Empty error is reported.
  - Use special characters in directory and file names with caution. Supported characters include uppercase and lowercase letters, digits, exclamation marks (!), hyphens (-), underscores (_), periods (.), asterisks (*), and parentheses (()).
  - Ultra-long paths are not supported. The maximum path length supported by a data flow is 1,023 characters.
- Limits on data flow imports.
  - When a symbolic link file is imported into CPFS for Lingjun, it is converted into a regular file containing data, and the symbolic link information is lost.
  - If an OSS bucket has multiple versions, only the latest version is copied.
  - File or subdirectory names longer than 255 bytes are not supported.
- Data flow export limits
- Limits on data flow exports
  - After a symlink file is synchronized to OSS, the file pointed to by the symlink is not synchronized. Instead, the symlink file becomes a regular empty object without data.
  - Hardlink files are synchronized to OSS only as regular files.
  - When socket, device, or pipe files are exported to an OSS bucket, they become regular empty objects without data.
  - Directory paths longer than 1,023 characters are not supported.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlow

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

None

Request parameters

Parameter	Type	Required	Description	Example
FileSystemId	string	Yes	The file system ID. General-purpose CPFS: The ID must start with `cpfs-`, such as cpfs-125487**. CPFS for Lingjun: The ID must start with `bmcpfs-`, such as bmcpfs-0015**.	cpfs-099394bd928c****
ClientToken	string	No	The client token that is used to ensure the idempotence of the request. You can generate the token on your client to ensure that the token is unique across different requests. The client token can contain only ASCII characters and cannot exceed 64 characters in length. For more information, see Ensure idempotence. Note If you do not specify this parameter, the system automatically uses the RequestId of the API request as the client token. The RequestId of each API request may be different.	123e4567-e89b-12d3-a456-42665544****
DryRun	boolean	No	Specifies whether to perform a dry run on the creation request. A dry run helps you check parameter validity and resource availability without actually creating the instance or incurring fees. Valid values: true: A dry run is performed and no instance is created. The system checks whether required parameters are specified, the request format, business limits, and NAS resource availability. If the request fails the dry run, an error is returned. If the request passes the dry run, the HTTP status code 200 is returned, but the FileSystemId parameter is empty. false: A regular request is sent. If the request passes the dry run, the instance is created. This is the default value.	false
Description	string	No	The description of the data flow. Limits: The description must be 2 to 128 characters in length. The description must start with a letter and cannot start with `http://` or `https://`. The description can contain digits, colons (:), underscores (_), and hyphens (-).	Bucket01 DataFlow
SourceSecurityType	string	No	The security protection type of the source storage. If the source storage must be accessed through security protection, specify the security protection type. Valid values: None: The source storage does not require security protection for access. This is the default value. SSL: Access is protected by an SSL certificate.	SSL
SourceStorage	string	Yes	The access address of the source storage. Format: `<storage type>://[<account id>:]<path>`. Where: storage type: Only oss is supported. account id: Optional. The UID of the account to which the source storage belongs. This parameter is required when you use a cross-account OSS bucket. path: The name of the OSS bucket. Limits: The name can contain only lowercase letters, digits, and hyphens (-), and must start and end with a lowercase letter or a digit. The name can be up to 128 characters in length. The name must be encoded in UTF-8. Note The OSS bucket must already exist in the region. The account id parameter is supported only in CPFS for Lingjun 2.6.0 and later.	oss://178321033379****:bucket-01
SourceStoragePath	string	No	The access path in the source storage bucket. Limits: The path must start and end with a forward slash (/). The path is case-sensitive. The path must be 1 to 1,023 characters in length. The path must be encoded in UTF-8. Note This parameter is required when the file system type is CPFS for Lingjun.	/prefix/
FsetId	string	No	The fileset ID. Note This parameter is required when the file system type is General-purpose CPFS.	fset-1902718ea0ae****
FileSystemPath	string	No	The directory in the CPFS for Lingjun file system. Limits: The directory must start and end with a forward slash (/). The directory must already exist in the CPFS for Lingjun file system. The directory must be 1 to 1,023 characters in length. The directory must be encoded in UTF-8. Note This parameter is required when the file system type is CPFS for Lingjun.	/path/
AutoRefreshs	array<object>	No	The auto-refresh configurations. Note This parameter takes effect only when the file system type is General-purpose CPFS.
	object	No
RefreshPath	string	No	The auto-refresh directory. General-purpose CPFS registers data modification events of the source storage, checks whether the source data in this directory is updated, and automatically imports the updated data. By default, this parameter is empty. Data updates in the source storage are not automatically imported to General-purpose CPFS. You must create a manual task to import the updates. Limits: The directory must be 2 to 1,024 characters in length. The directory must be encoded in UTF-8. The directory must start and end with a forward slash (/). The directory must already exist in General-purpose CPFS and must be located in the fileset directory of the data flow.	/prefix1/prefix2/
AutoRefreshPolicy	string	No	The auto-refresh policy. This policy determines how data updates from the source storage are imported to General-purpose CPFS. None: Data updates from the source storage are not automatically imported to General-purpose CPFS. You can use a data flow task to import the updates. This is the default value. ImportChanged: Data updates from the source storage are automatically imported to General-purpose CPFS. Note This parameter takes effect only when the file system type is General-purpose CPFS. Valid values: ImportChanged : Data updates from the source storage are automatically imported to General-purpose CPFS. None : Data updates from the source storage are not automatically imported to General-purpose CPFS.	None
AutoRefreshInterval	integer	No	The auto-refresh interval. General-purpose CPFS checks whether data updates exist in the directory at this interval. If data updates exist, an auto-refresh task is started. Unit: minutes. Valid values: 10 to 525600. Default value: 10. Note This parameter takes effect only when the file system type is General-purpose CPFS.	10
Throughput	integer	No	The maximum transmission bandwidth of the data flow. Unit: MB/s. Valid values: 600 1200 1500 Note The transmission bandwidth of the data flow must be less than the I/O bandwidth of the file system. This parameter is required when the file system type is General-purpose CPFS. Valid values: 600 : 600MB/s 1200 : 1200MB/s 1500 : 1500MB/s 2000 : 2000MB/s	600

Response elements

Element	Type	Description	Example
	object
DataFlowId	string	The data flow ID.	df-194433a5be31****
RequestId	string	The request ID.	473469C7-AA6F-4DC5-B3DB-A3DC0D****3E

Examples

Success response

JSON format

{
  "DataFlowId": "df-194433a5be31****",
  "RequestId": "473469C7-AA6F-4DC5-B3DB-A3DC0D****3E"
}

Error codes

HTTP status code	Error code	Error message	Description
400	IllegalCharacters	The parameter contains illegal characters.	The parameter contains illegal characters.
400	MissingFsetId	FsetId is mandatory for this action.	The Fileset ID parameter is not specified.
400	MissingSourceStorage	SourceStorage is mandatory for this action.	The SourceStorage parameter is not specified.
400	MissingThroughput	Throughput is mandatory for this action.	The Throughput parameter is not specified.
400	MissingFileSystemId	FileSystemId is mandatory for this action.	The FileSystemId parameter is not specified.
400	InvalidFilesystemVersion.NotSupport	This Api does not support this fileSystem version.	This Api does not support this fileSystem version.
400	DataFlow.Bucket.RegionUnmatched	The bucket and file system are not in the same region.	The bucket is inconsistent with the filesystem region.
400	Oss.Path.Conflict	The OSS directory conflicts and is occupied by other data flow tasks.	The OSS directory conflicts and is occupied by other data flow tasks.
403	OperationDenied.DependencyViolation	The operation is denied due to dependancy violation.	The conditions for performing this operation are not met.
403	OperationDenied.NoStock	The operation is denied due to no stock.
403	OperationDenied.DependFset	The operation is denied due to invalid fileset state.	The data flow creation failed due to an abnormal fileset status.
403	OperationDenied.ConflictOperation	The operation is denied due to a conflict with an ongoing operation.	A conflict with current file system operations prevents the creation of the data flow.
403	OperationDenied.DependMountpoint	The operation is denied because no mount point is found.	The data flow creation failed because the file system does not have a mount target.
403	OperationDenied.FsetAlreadyInUse	The Fset is already bound to another data flow.	The Fset is already in use.
403	OperationDenied.AutoRefreshNotSupport	The operation is denied. Auto refresh is not supported.	The file system does not support automatic updates.
403	OperationDenied.DependBucketTag	The operation is denied. The OSS Bucket tag cpfs-dataflow is missing.	The data flow creation failed because the OSS bucket is not tagged with cpfs-dataflow.
403	OperationDenied.DataFlowNotSupported	The operation is not supported.	The file system does not support data flows.
403	InvalidOperation.DeletionProtection	The operation is not allowed due to resource is protected by deletion protection.	The instance has deletion protection enabled and cannot be released through the console or API operations.
403	DataFlow.Bucket.AccessDenied	Bucket access denied.	Data flow authentication error.
404	InvalidFileSystem.NotFound	The specified file system does not exist.	The specified file system does not exist.
404	InvalidThroughput.OutOfBounds	Throughput is out of bounds.	The specified throughput is not within the valid range.
404	InvalidDescription.InvalidFormat	Description format is invalid.	The specified description format is invalid.
404	InvalidRefreshPath.InvalidParameter	Refresh path is invalid.	The path of the specified auto-refresh directory is invalid.
404	InvalidRefreshPath.Duplicated	Refresh path is duplicated.	The path of the specified auto-refresh directory is duplicated.
404	InvalidRefreshPath.NotFound	Refresh path does not exist.	The specified auto-refresh directory does not exist.
404	InvalidRefreshPolicy.InvalidParameter	Refresh policy is invalid.	The auto-refresh policy is invalid.
404	InvalidRefreshInterval.OutOfBounds	Refresh interval is out of bounds.	The specified RefreshInterval is not within the valid range.
404	InvalidSourceStorage.Unreachable	Source storage cannot be accessed.	The source storage is inaccessible.
404	InvalidSourceStorage.NotFound	Source storage is not found.	The specified source storage does not exist.
404	InvalidSourceStorage.NotSupport	Source storage type is not supported.	The type of the specified source storage is not supported.
404	InvalidSourceStorage.PermissionDenied	The source storage access permission is denied.	You do not have access permissions for the source storage. Check the service-linked role authorization and the access tags of the source storage.
404	InvalidSourceStorage.InvalidRegion	Source storage region is invalid.	The region of the source storage is abnormal. Check whether the source storage and CPFS are in the same region.
404	InvalidSourceStorage.InvalidParameter	Source storage has invalid parameters.	The specified source storage path is invalid.
404	InvalidSourceSecurityType.NotSupport	The source security type is not supported.	The security protection type of the specified source storage is not supported.
404	InvalidAutoRefresh.TooManyAutoRefreshes	The number of auto refreshes exceeds the limit.	The number of specified auto-renewal configurations exceeds the limit.
404	InvalidSourceStorage.NeedVersioning	Source storage must enable versioning.	The versioning feature must be enabled for the source storage.
404	InvalidFsetId.NotFound	The specified Fileset ID does not exist.	The specified Fileset does not exist.
404	DataFlow.Bucket.NotExist	Bucket does not exist.	The bucket does not exist.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.