Create a CPFS Dataflow via CreateDataFlow API - Cloud Parallel File Storage - Alibaba Cloud - Cloud Parallel File Storage

Creates a data flow between a General-purpose CPFS or CPFS for Lingjun file system and the source storage.

Operation description

This operation applies to the following services:

Service	File system ID format	Minimum version that supports data flows
General-purpose CPFS	Starts with `cpfs-`, for example, cpfs-125487****	2.2.0 and later
CPFS for Lingjun	Starts with `bmcpfs-`, for example, bmcpfs-0015****	2.4.0 and later

Note

Note: General-purpose CPFS and CPFS for Lingjun share the same API operations but have differences in parameter values and feature support. Refer to the corresponding section based on your service type.

Basic operations
- You can create a data flow only when the General-purpose CPFS or CPFS for Lingjun file system is in the Running state.
- You can create a maximum of 10 data flows for a single General-purpose CPFS or CPFS for Lingjun file system.
- Creating a data flow typically takes 2 to 5 minutes. You can call the DescribeDataFlows operation to check whether the data flow is created.
Permissions.

When you create a data flow, Cloud Parallel File Storage (CPFS) assumes the AliyunServiceRoleForNasOssDataflow and AliyunServiceRoleForNasEventNotification service-linked roles. For more information, see Service-linked roles for CPFS.
Usage notes for General-purpose CPFS.

This section applies to General-purpose CPFS file systems whose IDs start with cpfs-.
- Billing
- Creating a data flow is billed based on the data flow bandwidth. For more information, see Billing of General-purpose CPFS.
  - If you enable auto-refresh (AutoRefresh), EventBridge is used to collect object modification events from the source OSS bucket, which incurs fees. For more information, see Billing of EventBridge.
- Data flow specifications
  - The data flow bandwidth (Throughput) supports three specifications: 600 MB/s, 1200 MB/s, and 1500 MB/s. The data flow bandwidth refers to the maximum transmission bandwidth that the data flow can achieve when importing or exporting data.
  - Creating a data flow consumes one vSwitch IP address used by a General-purpose Cloud Parallel File Storage mount target. Make sure that the vSwitch has sufficient IP addresses.
  - Inventory check: When the DryRun parameter is set to true, the system checks whether the resources required to create the specified data flow are sufficient.
- Fileset
  - The destination of a data flow is a Fileset in the General-purpose CPFS file system. A Fileset is a new directory tree structure in General-purpose CPFS. It is a small file system within the parent file system and has an independent inode space and management capabilities.
  - The Fileset must already exist when you create a data flow, and it cannot be nested with other Filesets. You can create only one data flow on a Fileset, which corresponds to one source storage.
  - The maximum number of files in a Fileset is 1 million. If the number of files imported from the OSS bucket exceeds this limit, a no space error is reported when you create new files.
Note
If the Fileset already contains data, the existing data in the Fileset is cleared and replaced with the data synchronized from OSS after the data flow is created.
- Auto-refresh
  - After you configure auto-refresh (AutoRefresh), when data in the source storage changes, the changed metadata is automatically synchronized to the General-purpose CPFS file system. The changed data is loaded on demand when users access files, or you can start a data flow task to load the data.
  - Auto-refresh relies on EventBridge to collect object modification events from the source OSS bucket. You must first activate EventBridge.
  - The scope of auto-refresh is the prefix, which is specified by the RefreshPath parameter. You can configure a maximum of five auto-refresh directories for a single data flow.
  - The auto-refresh interval (AutoRefreshInterval) specifies the auto-refresh time configured in General-purpose CPFS to check whether data updates exist within the prefix in the source OSS bucket. If data updates exist, an auto-refresh task is started. If the frequency of object modification events in the source OSS bucket exceeds the processing capability of the General-purpose CPFS data flow, auto-refresh tasks accumulate, metadata updates are delayed, and the data flow status becomes Misconfigured. You can resolve this by upgrading the data flow specifications or reducing the OSS modification frequency.
  - When you configure auto-refresh for a prefix in a General-purpose Cloud Parallel File Storage data flow, an event bus is created on the user side, and an event rule is created for the prefix of the source OSS bucket. When an object is modified within the prefix of the source OSS bucket, an OSS event is generated in EventBridge and processed by the General-purpose CPFS data flow.
  Note
  The event bus and event rule created by General-purpose Cloud Parallel File Storage in EventBridge contain the description Create for cpfs auto refresh. You cannot modify or delete the event bus or event rule. Otherwise, auto-refresh cannot function normally.
- Source storage
  - The source storage only supports OSS. The source storage (SourceStorage) of a data flow must be an OSS bucket.
  - General-purpose CPFS data flows support accessing OSS in both encrypted and unencrypted modes. If you select the encrypted (SSL) mode to access OSS, you must confirm that the encryption in transit of the OSS bucket supports encrypted access.
  - If multiple General-purpose CPFS data flows or multiple data flows of the same General-purpose Cloud Parallel File Storage file system use the same OSS bucket as the source storage, you must enable versioning for the OSS bucket to prevent data conflicts when multiple General-purpose CPFS file systems export data to the same source.
  - Cross-region OSS data flows are not supported. The OSS bucket must be in the same region as the General-purpose CPFS file system.
    Note
    Before you create a data flow, you must add a tag (key: cpfs-dataflow, value: true) to the source OSS bucket so that the General-purpose CPFS data flow can access the data in the bucket. During the usage of the data flow, do not delete or modify this tag. Otherwise, the General-purpose CPFS data flow cannot access the data in the bucket.
Usage notes for CPFS for Lingjun.

This section applies to CPFS for Lingjun file systems whose IDs start with bmcpfs-.
- Source storage
  - The source storage only supports OSS. The source storage (SourceStorage) of a data flow must be an OSS bucket.
  - CPFS for Lingjun data flows support accessing OSS in both encrypted and unencrypted modes. If you select the encrypted (SSL) mode to access OSS, you must confirm that the encryption in transit of the OSS bucket supports encrypted access.
  - If multiple CPFS for Lingjun data flows or multiple data flows of the same CPFS for Lingjun file system use the same OSS bucket as the source storage, you must enable versioning for the OSS bucket to prevent data conflicts when multiple CPFS for Lingjun file systems export data to the same source.
  - Cross-region OSS data flows are not supported. The OSS bucket must be in the same region as the CPFS for Lingjun file system.
  - CPFS for Lingjun 2.6.0 and later support creating data flows by using cross-account OSS buckets.
  - The account id parameter is required only when you use a cross-account OSS bucket.
  - Before you use a cross-account OSS bucket, you must authorize the account. For specific authorization instructions, see Cross-account data flow authorization.
    Note
    Before you create a data flow, you must add a tag (key: cpfs-dataflow, value: true) to the source OSS bucket so that the CPFS for Lingjun data flow can access the data in the bucket. During the usage of the data flow, do not delete or modify this tag. Otherwise, the CPFS for Lingjun data flow cannot access the data in the bucket.
- Limits of data flows on file systems.
  - In the file system path associated with the data flow, you cannot rename a non-empty directory. Otherwise, a Permission Denied error is reported or the directory is not empty.
  - Use special characters in directory and file names with caution. Supported characters include uppercase and lowercase letters, digits, exclamation marks (!), hyphens (-), underscores (_), periods (.), asterisks (*), and half-width parentheses (()).
  - Ultra-long paths are not supported. The maximum path length supported by data flows is 1,023 characters.
- Limits on data flow import.
  - After a symlink file is imported to CPFS for Lingjun, it is converted into a regular file containing data, and the symlink information is lost.
  - If the OSS bucket has multiple versions, only the latest version is copied.
  - File names or subdirectory names longer than 255 bytes are not supported.
- Limits on data flow export
  - After a symlink file is synchronized to OSS, the file pointed to by the symlink is not synchronized. Instead, it becomes a regular empty object without data.
  - Hard link files are synchronized to OSS only as regular files.
  - When socket, device, or pipe files are exported to an OSS bucket, they become regular empty objects without data.
  - Directory paths longer than 1,023 characters are not supported.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlow

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

None

Request parameters

Parameter	Type	Required	Description	Example
FileSystemId	string	Yes	The file system ID. General-purpose CPFS: The ID must start with `cpfs-`, for example, cpfs-125487**. CPFS for Lingjun: The ID must start with `bmcpfs-`, for example, bmcpfs-0015**.	cpfs-099394bd928c****
ClientToken	string	No	The client token that is used to ensure the idempotence of the request. You can generate a value for this parameter on your client to ensure that the value is unique across different requests. The client token can contain only ASCII characters and cannot exceed 64 characters in length. For more information, see How to ensure idempotence. Note If you do not specify this parameter, the system automatically uses the RequestId of the API request as the client token. The RequestId of each API request may be different.	123e4567-e89b-12d3-a456-42665544****
DryRun	boolean	No	Specifies whether to perform a dry run on the creation request. A dry run helps you check parameter validity and inventory without actually creating the instance or incurring fees. Valid values: true: A dry run is performed, and the instance is not created. The system checks whether required parameters are specified, the request format is valid, service limits are met, and NAS inventory is sufficient. If the request fails the dry run, the corresponding error is returned. If the request passes the dry run, the HTTP status code 200 is returned, but the FileSystemId is empty. false (default): A normal request is sent. If the request passes the check, the instance is created.	false
Description	string	No	The description of the data flow. Limits: The description must be 2 to 128 characters in length. It must start with a letter or a Chinese character and cannot start with `http://` or `https://`. It can contain digits, colons (:), underscores (_), and hyphens (-).	Bucket01 DataFlow
SourceSecurityType	string	No	The security protection type of the source storage. If the source storage must be accessed through security protection, specify the security protection type. Valid values: None (default): The source storage does not require security protection for access. SSL: Access is protected by an SSL certificate.	SSL
SourceStorage	string	Yes	The access address of the source storage. Format: `<storage type>://[<account id>:]<path>`. Where: storage type: Only oss is supported. account id: Optional. The UID of the account to which the source storage belongs. This parameter is required when you use a cross-account OSS bucket. path: The name of the OSS bucket. Limits: The name can contain only lowercase letters, digits, and hyphens (-), and must start and end with a lowercase letter or a digit. The name can be up to 128 characters in length. The name must be encoded in UTF-8. Note The OSS bucket must already exist in the region. The account id parameter is supported only in CPFS for Lingjun 2.6.0 and later.	oss://178321033379****:bucket-01
SourceStoragePath	string	No	The access path in the source storage bucket. Limits: The path must start and end with a forward slash (/). The path is case-sensitive. The path must be 1 to 1,023 characters in length. The path must be encoded in UTF-8. Note This parameter is required when the file system type is CPFS for Lingjun.	/prefix/
FsetId	string	No	The Fileset ID. Note This parameter is required when the file system type is General-purpose CPFS.	fset-1902718ea0ae****
FileSystemPath	string	No	The directory in the CPFS for Lingjun file system. Limits: The path must start and end with a forward slash (/). The directory must already exist in CPFS for Lingjun. The path must be 1 to 1,023 characters in length. The path must be encoded in UTF-8. Note This parameter is required when the file system type is CPFS for Lingjun.	/path/
AutoRefreshs	array<object>	No	The collection of auto-refresh configurations. Note This parameter takes effect only when the file system type is General-purpose CPFS.
	object	No
RefreshPath	string	No	The auto-refresh directory. General-purpose CPFS registers data modification events of the source storage and checks whether the source data in this directory is updated to automatically import the updated data. Default value: empty. Data updates in the source storage are not automatically imported to General-purpose CPFS. You must manually create tasks to import updates. Limits: The path must be 2 to 1,024 characters in length. The path must be encoded in UTF-8. The path must start and end with a forward slash (/). The directory must already exist in General-purpose CPFS and must be located in the Fileset directory of the data flow.	/prefix1/prefix2/
AutoRefreshPolicy	string	No	The auto-refresh policy. This policy determines how data updates from the source are imported to General-purpose CPFS after the source data is updated. None (default): Data updates from the source are not automatically imported to General-purpose CPFS. You can import source data updates by using data flow tasks. ImportChanged: Data updates from the source are automatically imported to General-purpose CPFS. Note This parameter takes effect only when the file system type is General-purpose CPFS. Valid values: ImportChanged : Data updates from the source are automatically imported to General-purpose CPFS. None : Data updates from the source are not automatically imported to General-purpose CPFS.	None
AutoRefreshInterval	integer	No	The auto-refresh interval. General-purpose CPFS checks whether data updates exist in the directory at this interval. If data updates exist, an auto-refresh task is started. Unit: minutes. Valid values: 10 to 525600. Default value: 10. Note This parameter takes effect only when the file system type is General-purpose CPFS.	10
Throughput	integer	No	The maximum transmission bandwidth of the data flow. Unit: MB/s. Valid values: 600 1200 1500 Note The transmission bandwidth of the data flow must be less than the I/O bandwidth of the file system. This parameter is required when the file system type is General-purpose CPFS. Valid values: 600 : 600MB/s 1200 : 1200MB/s 1500 : 1500MB/s 2000 : 2000MB/s	600

Response elements

Element	Type	Description	Example
	object
DataFlowId	string	The data flow ID.	df-194433a5be31****
RequestId	string	The request ID.	473469C7-AA6F-4DC5-B3DB-A3DC0D****3E

Examples

Success response

JSON format

{
  "DataFlowId": "df-194433a5be31****",
  "RequestId": "473469C7-AA6F-4DC5-B3DB-A3DC0D****3E"
}

Error codes

HTTP status code	Error code	Error message	Description
400	IllegalCharacters	The parameter contains illegal characters.	The parameter contains illegal characters.
400	MissingFsetId	FsetId is mandatory for this action.	The Fileset ID parameter is not specified.
400	MissingSourceStorage	SourceStorage is mandatory for this action.	The SourceStorage parameter is not specified.
400	MissingThroughput	Throughput is mandatory for this action.	The Throughput parameter is not specified.
400	MissingFileSystemId	FileSystemId is mandatory for this action.	The FileSystemId parameter is not specified.
400	InvalidFilesystemVersion.NotSupport	This Api does not support this fileSystem version.	This Api does not support this fileSystem version.
400	DataFlow.Bucket.RegionUnmatched	The bucket and file system are not in the same region.	The bucket is inconsistent with the filesystem region.
400	Oss.Path.Conflict	The OSS directory conflicts and is occupied by other data flow tasks.	The OSS directory conflicts and is occupied by other data flow tasks.
403	OperationDenied.DependencyViolation	The operation is denied due to dependancy violation.	The conditions for performing this operation are not met.
403	OperationDenied.NoStock	The operation is denied due to no stock.
403	OperationDenied.DependFset	The operation is denied due to invalid fileset state.	The data flow creation failed due to an abnormal fileset status.
403	OperationDenied.ConflictOperation	The operation is denied due to a conflict with an ongoing operation.	A conflict with current file system operations prevents the creation of the data flow.
403	OperationDenied.DependMountpoint	The operation is denied because no mount point is found.	The data flow creation failed because the file system does not have a mount target.
403	OperationDenied.FsetAlreadyInUse	The Fset is already bound to another data flow.	The Fset is already in use.
403	OperationDenied.AutoRefreshNotSupport	The operation is denied. Auto refresh is not supported.	The file system does not support automatic updates.
403	OperationDenied.DependBucketTag	The operation is denied. The OSS Bucket tag cpfs-dataflow is missing.	The data flow creation failed because the OSS bucket is not tagged with cpfs-dataflow.
403	OperationDenied.DataFlowNotSupported	The operation is not supported.	The file system does not support data flows.
403	InvalidOperation.DeletionProtection	The operation is not allowed due to resource is protected by deletion protection.	The instance has deletion protection enabled and cannot be released through the console or API operations.
403	DataFlow.Bucket.AccessDenied	Bucket access denied.	Data flow authentication error.
404	InvalidFileSystem.NotFound	The specified file system does not exist.	The specified file system does not exist.
404	InvalidThroughput.OutOfBounds	Throughput is out of bounds.	The specified throughput is not within the valid range.
404	InvalidDescription.InvalidFormat	Description format is invalid.	The specified description format is invalid.
404	InvalidRefreshPath.InvalidParameter	Refresh path is invalid.	The path of the specified auto-refresh directory is invalid.
404	InvalidRefreshPath.Duplicated	Refresh path is duplicated.	The path of the specified auto-refresh directory is duplicated.
404	InvalidRefreshPath.NotFound	Refresh path does not exist.	The specified auto-refresh directory does not exist.
404	InvalidRefreshPolicy.InvalidParameter	Refresh policy is invalid.	The auto-refresh policy is invalid.
404	InvalidRefreshInterval.OutOfBounds	Refresh interval is out of bounds.	The specified RefreshInterval is not within the valid range.
404	InvalidSourceStorage.Unreachable	Source storage cannot be accessed.	The source storage is inaccessible.
404	InvalidSourceStorage.NotFound	Source storage is not found.	The specified source storage does not exist.
404	InvalidSourceStorage.NotSupport	Source storage type is not supported.	The type of the specified source storage is not supported.
404	InvalidSourceStorage.PermissionDenied	The source storage access permission is denied.	You do not have access permissions for the source storage. Check the service-linked role authorization and the access tags of the source storage.
404	InvalidSourceStorage.InvalidRegion	Source storage region is invalid.	The region of the source storage is abnormal. Check whether the source storage and CPFS are in the same region.
404	InvalidSourceStorage.InvalidParameter	Source storage has invalid parameters.	The specified source storage path is invalid.
404	InvalidSourceSecurityType.NotSupport	The source security type is not supported.	The security protection type of the specified source storage is not supported.
404	InvalidAutoRefresh.TooManyAutoRefreshes	The number of auto refreshes exceeds the limit.	The number of specified auto-renewal configurations exceeds the limit.
404	InvalidSourceStorage.NeedVersioning	Source storage must enable versioning.	The versioning feature must be enabled for the source storage.
404	InvalidFsetId.NotFound	The specified Fileset ID does not exist.	The specified Fileset does not exist.
404	DataFlow.Bucket.NotExist	Bucket does not exist.	The bucket does not exist.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.