Create a CPFS Dataflow via CreateDataFlow API - Cloud Parallel File Storage - Alibaba Cloud - Cloud Parallel File Storage

Creates a data flow between a CPFS General-purpose or CPFS for Lingjun file system and source storage.

Operation description

This operation applies to the following products:

Product	File system ID format	Minimum version that supports data flows
CPFS General-purpose	Starts with `cpfs-`, such as cpfs-125487****	2.2.0 and later
CPFS for Lingjun	Starts with `bmcpfs-`, such as bmcpfs-0015****	2.4.0 and later

Note

CPFS General-purpose and CPFS for Lingjun share the same set of API operations, but differ in parameter values and feature support. Refer to the corresponding section based on your product type.

Basic operations
- You can create a data flow only when the CPFS General-purpose or CPFS for Lingjun file system is in the Running state.
- A maximum of 10 data flows can be created for a single CPFS General-purpose or CPFS for Lingjun file system.
- Creating a data flow typically takes 2 to 5 minutes. You can call DescribeDataFlows to check whether the data flow is created.
Permissions.

When you create a data flow, Cloud Parallel File Storage (CPFS) obtains two service-linked roles: AliyunServiceRoleForNasOssDataflow and AliyunServiceRoleForNasEventNotification. For more information, see CPFS service-linked roles.
Usage notes for CPFS General-purpose.

This section applies to CPFS General-purpose file systems whose IDs start with cpfs-.
- Billing
  - Creating a data flow incurs charges based on the data flow bandwidth. For more information, see Billing of CPFS General-purpose.
  - When you use the automatic update (AutoRefresh) feature, EventBridge collects object modification events from the source OSS storage, which incurs fees. For more information, see EventBridge billing.
- Data flow specifications
  - The data flow bandwidth (Throughput) supports three specifications: 600 MB/s, 1200 MB/s, and 1500 MB/s. The data flow bandwidth refers to the maximum transfer bandwidth that the data stream can achieve during data import or export.
  - Creating a data flow consumes one vSwitch IP address used by the Cloud Parallel File Storage General-purpose mount target. Make sure that sufficient vSwitch IP resources are available.
  - Inventory check: When DryRun is set to true, you can verify whether the resources required to create a data flow of the specified specification are available.
- Fileset
  - The destination of a data flow is a Fileset in the CPFS General-purpose file system. A Fileset is a new directory tree structure in the CPFS General-purpose file system. It is a small file system within the parent file system that has an independent inode space and management capability.
  - The Fileset must already exist when you create a data flow, and it cannot be nested with other Filesets. Only one data flow can be created on a Fileset, corresponding to one source storage.
  - The maximum number of files in a Fileset is 1,000,000. If the number of files imported from the OSS bucket exceeds this limit, creating new files returns a no space error.
Note
If data already exists in the Fileset, the existing data in the Fileset is cleared and replaced with data synchronized from OSS after the data flow is created.
- Automatic update
  - After you configure automatic update (AutoRefresh), when data in the source storage changes, the changed metadata is automatically synchronized to the Cloud Parallel File Storage General-purpose file system. The changed data is loaded on demand when a user accesses the file, or loaded by starting a data flow node to load data.
  - Automatic synchronization relies on EventBridge to collect object modification events from the source OSS storage. You must first activate EventBridge.
  - The scope of automatic update is a prefix, specified by the RefreshPath parameter. A maximum of 5 automatic update folders can be configured for a data stream.
  - The automatic update interval (AutoRefreshInterval) is the automatic update time set by Cloud Parallel File Storage General-purpose to check whether data updates exist in the specified prefix of the source OSS bucket. If data updates exist, an automatic synchronization node is started. When the frequency of object modification events in the source OSS exceeds the processing capacity of the Cloud Parallel File Storage General-purpose data stream, automatic synchronization nodes accumulate, metadata updates are delayed, and the data stream status changes to Misconfigured. You can resolve this issue by upgrading the data stream specification or reducing the OSS modification frequency.
  - When Cloud Parallel File Storage General-purpose configures automatic update for a prefix, an event bus is created on the user side, and an event rule is created for the prefix of the source OSS bucket. When objects are modified in the prefix of the source OSS bucket, OSS events are generated in EventBridge and processed by the Cloud Parallel File Storage General-purpose data stream.
  Note
  The event buses and event rules created by Cloud Parallel File Storage General-purpose in EventBridge have the description Create for cpfs auto refresh. Do not modify or delete these event buses or event rules. Otherwise, automatic synchronization does not work properly.
- Source storage
  - Only OSS is supported as source storage. The source storage (SourceStorage) of a data stream must be an OSS bucket.
  - Cloud Parallel File Storage General-purpose data streams support both encrypted and non-encrypted access to OSS. When you select encrypted (SSL) access to OSS, confirm that the encryption in transit settings of the OSS bucket support encrypted access.
  - If multiple Cloud Parallel File Storage General-purpose data streams, or multiple data streams of the same Cloud Parallel File Storage General-purpose file system, use the same OSS bucket as the source storage, enable versioning for the OSS bucket to prevent data conflicts when multiple Cloud Parallel File Storage General-purpose file systems export data to the same source. This procedure is required to avoid data conflicts.
  - Cross-region OSS data streams are not supported. The OSS bucket must be in the same region as the Cloud Parallel File Storage General-purpose file system.
    Note
    Before creating a data stream, set a label (key: cpfs-dataflow, value: true) on the source OSS bucket so that the Cloud Parallel File Storage General-purpose data stream can access the data in the bucket. Do not delete or modify this label while the data stream is in use. Otherwise, the Cloud Parallel File Storage General-purpose data stream cannot access the data in the bucket.
Usage notes for CPFS for Lingjun.

This section applies to CPFS for Lingjun file systems whose IDs start with bmcpfs-.
- Source storage
  - Only OSS is supported as source storage. The source storage (SourceStorage) of a data flow must be an OSS bucket.
  - CPFS for Lingjun data flows support both encrypted and non-encrypted access to OSS. When you select encrypted (SSL) access to OSS, make sure that the encryption in transit setting of the OSS bucket supports encrypted access.
  - If multiple CPFS for Lingjun data flows, or multiple data flows of the same CPFS for Lingjun file system, use the same OSS bucket as the source storage, enable versioning for the OSS bucket to prevent data conflicts when multiple CPFS for Lingjun file systems export data to the same source.
  - Cross-region OSS data flows are not supported. The OSS bucket must be in the same region as the CPFS for Lingjun file system.
  - CPFS for Lingjun 2.6.0 and later support creating data flows with cross-account OSS.
  - The account ID parameter is required only when you use cross-account OSS.
  - When you use cross-account OSS, authorize the accounts first. For more information, see Cross-account data flow authorization.
    Note
    Before creating a data flow, set a tag (key: cpfs-dataflow, value: true) on the source OSS bucket so that the CPFS for Lingjun data flow can access the data in the bucket. Do not delete or modify this tag while the data flow is in use. Otherwise, the CPFS for Lingjun data flow cannot access the data in the bucket.
- Data flow restrictions on file systems.
  - In the file system path associated with a data flow, you cannot rename non-empty directories. Otherwise, a Permission Denied or directory not empty error is returned.
  - Use special characters in directory and file names with caution. The following characters are supported: uppercase and lowercase letters, digits, exclamation marks (!), hyphens (-), underscores (_), periods (.), asterisks (*), and parentheses (()).
  - Excessively long paths are not supported. The maximum path length supported by data flows is 1023 characters.
- Data flow import restrictions.
  - After symlink files are imported to CPFS for Lingjun, they are converted to regular files that contain data, and the symlink information is lost.
  - If the OSS bucket has multiple versions, only the latest version is copied.
  - File names or subdirectory names longer than 255 bytes are not supported.
- Data flow export restrictions
  - After symlink files are synchronized to OSS, the files that the symlinks point to are not synchronized. Instead, the symlinks become regular empty objects with no data.
  - Hardlink files are synchronized to OSS as regular files only.
  - Socket, Device, and Pipe files become regular empty objects with no data when exported to an OSS bucket.
  - Directory paths longer than 1023 characters are not supported.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:CreateDataFlow

create

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

None

Request parameters

Parameter	Type	Required	Description	Example
FileSystemId	string	Yes	The file system ID. CPFS General-purpose: must start with `cpfs-`, such as cpfs-125487**. CPFS for Lingjun: must start with `bmcpfs-`, such as bmcpfs-0015**.	cpfs-099394bd928c****
FsetId	string	No	The Fileset ID. Note This parameter is required when the file system type is CPFS General-purpose.	fset-1902718ea0ae****
SourceStorage	string	Yes	The access address of the source storage. Format: `<storage type>://[<account id>:]<path>`. Where: storage type: only oss is supported. account id: optional. The UID of the account that owns the source storage. This parameter is required when you use cross-account OSS. path: the name of the OSS bucket. Limits: Only lowercase letters, digits, and hyphens (-) are supported. The name must start and end with a lowercase letter or digit. The maximum length is 128 characters. UTF-8 encoding is used. Note The OSS bucket must be an existing bucket in the region. The account id parameter is supported only by CPFS for Lingjun 2.6.0 and later.	oss://178321033379****:bucket-01
SourceSecurityType	string	No	The security protection type of the source storage. If the source storage must be accessed through security protection, specify the security protection type. Valid values: None (default): The source storage does not require security protection for access. SSL: Access is protected by SSL certificates.	SSL
Throughput	integer	No	The maximum transfer bandwidth of the data flow. Unit: MB/s. Valid values: 600 1200 1500 Note The transfer bandwidth of the data flow must be less than the I/O bandwidth of the file system. This parameter is required when the file system type is CPFS General-purpose. Valid values: 600 : 600MB/s 1200 : 1200MB/s 1500 : 1500MB/s 2000 : 2000MB/s	600
Description	string	No	The description of the data flow. Limits: The description must be 2 to 128 characters in length. The description must start with a letter and cannot start with `http://` or `https://`. The description can contain digits, colons (:), underscores (_), and hyphens (-).	Bucket01 DataFlow
AutoRefreshs	array<object>	No	The automatic update configurations. Note This parameter takes effect only when the file system type is CPFS General-purpose.
	object	No
RefreshPath	string	No	The automatic update directory. CPFS General-purpose registers data modification events from the source storage and checks whether data in this directory has been updated, then automatically imports the updated data. The default value is empty, which means that data updates in the source storage are not automatically imported to CPFS General-purpose. You must manually create a task to import updates. Limits: The path must be 2 to 1024 characters in length. UTF-8 encoding is used. The path must start and end with a forward slash (/). The directory must be an existing directory in the CPFS General-purpose file system and must be located within the Fileset directory of the data flow.	/prefix1/prefix2/
AutoRefreshPolicy	string	No	The automatic update policy that specifies how data updates from the source storage are imported to CPFS General-purpose. None (default): Data updates from the source storage are not automatically imported to CPFS General-purpose. You can import data updates from the source storage by using data flow tasks. ImportChanged: Data updates from the source storage are automatically imported to CPFS General-purpose. Note This parameter takes effect only when the file system type is CPFS General-purpose. Valid values: ImportChanged : Data updates from the source storage are automatically imported to CPFS General-purpose. None : Data updates from the source storage are not automatically imported to CPFS General-purpose.	None
AutoRefreshInterval	integer	No	The automatic update interval. CPFS General-purpose checks the directory for data updates at this interval and starts an automatic update task if data updates exist. Unit: minutes. Valid values: 10 to 525600. Default value: 10. Note This parameter takes effect only when the file system type is CPFS General-purpose.	10
DryRun	boolean	No	Specifies whether to perform a dry run for this request. A dry run checks parameter validity and resource availability without actually creating an instance or incurring charges. Valid values: true: Sends a dry run request without creating an instance. The check items include required parameters, request format, business restrictions, and NAS inventory. If the check fails, the corresponding error is returned. If the check succeeds, HTTP status code 200 is returned, but FileSystemId is empty. false (default): Sends a normal request and creates an instance after the check is passed.	false
ClientToken	string	No	Ensures the idempotence of the request. Generate a parameter value from your client to ensure that the value is unique across different requests. ClientToken supports only ASCII characters and cannot exceed 64 characters in length. For more information, see How to ensure idempotence. Note If you do not specify this parameter, the system automatically uses the RequestId of the API request as the ClientToken. The RequestId may differ for each API request.	123e4567-e89b-12d3-a456-42665544****
SourceStoragePath	string	No	The access path within the source storage bucket. Limits: The path must start and end with a forward slash (/). The path is case-sensitive. The path must be 1 to 1023 characters in length. UTF-8 encoding is used. Note This parameter is required when the file system type is CPFS for Lingjun.	/prefix/
FileSystemPath	string	No	The directory within the CPFS for Lingjun file system. Limits: The path must start and end with a forward slash (/). The directory must be an existing directory in the CPFS for Lingjun file system. The path must be 1 to 1023 characters in length. UTF-8 encoding is used. Note This parameter is required when the file system type is CPFS for Lingjun.	/path/

Response elements

Element	Type	Description	Example
	object
RequestId	string	The request ID.	473469C7-AA6F-4DC5-B3DB-A3DC0D****3E
DataFlowId	string	The data flow ID.	df-194433a5be31****

Examples

Success response

JSON format

{
  "RequestId": "473469C7-AA6F-4DC5-B3DB-A3DC0D****3E",
  "DataFlowId": "df-194433a5be31****"
}

Error codes

HTTP status code	Error code	Error message	Description
400	IllegalCharacters	The parameter contains illegal characters.	The parameter contains illegal characters.
400	MissingFsetId	FsetId is mandatory for this action.
400	MissingSourceStorage	SourceStorage is mandatory for this action.
400	MissingThroughput	Throughput is mandatory for this action.
400	MissingFileSystemId	FileSystemId is mandatory for this action.
400	InvalidFilesystemVersion.NotSupport	This Api does not support this fileSystem version.	This Api does not support this fileSystem version.
400	DataFlow.Bucket.RegionUnmatched	The bucket and file system are not in the same region.	The bucket is inconsistent with the filesystem region.
403	OperationDenied.DependencyViolation	The operation is denied due to dependancy violation.
403	OperationDenied.NoStock	The operation is denied due to no stock.
403	OperationDenied.DependFset	The operation is denied due to invalid fileset state.
403	OperationDenied.ConflictOperation	The operation is denied due to a conflict with an ongoing operation.
403	OperationDenied.DependMountpoint	The operation is denied because no mount point is found.
403	OperationDenied.FsetAlreadyInUse	The Fset is already bound to another data flow.
403	OperationDenied.AutoRefreshNotSupport	The operation is denied. Auto refresh is not supported.
403	OperationDenied.DependBucketTag	The operation is denied. The OSS Bucket tag cpfs-dataflow is missing.
403	OperationDenied.DataFlowNotSupported	The operation is not supported.
403	InvalidOperation.DeletionProtection	The operation is not allowed due to resource is protected by deletion protection.
403	DataFlow.Bucket.AccessDenied	Bucket access denied.	Data flow authentication error.
404	InvalidFileSystem.NotFound	The specified file system does not exist.	The specified file system does not exist.
404	InvalidThroughput.OutOfBounds	Throughput is out of bounds.
404	InvalidDescription.InvalidFormat	Description format is invalid.
404	InvalidRefreshPath.InvalidParameter	Refresh path is invalid.
404	InvalidRefreshPath.Duplicated	Refresh path is duplicated.
404	InvalidRefreshPath.NotFound	Refresh path does not exist.
404	InvalidRefreshPolicy.InvalidParameter	Refresh policy is invalid.
404	InvalidRefreshInterval.OutOfBounds	Refresh interval is out of bounds.
404	InvalidSourceStorage.Unreachable	Source storage cannot be accessed.
404	InvalidSourceStorage.NotFound	Source storage is not found.
404	InvalidSourceStorage.NotSupport	Source storage type is not supported.
404	InvalidSourceStorage.PermissionDenied	The source storage access permission is denied.
404	InvalidSourceStorage.InvalidRegion	Source storage region is invalid.
404	InvalidSourceStorage.InvalidParameter	Source storage has invalid parameters.
404	InvalidSourceSecurityType.NotSupport	The source security type is not supported.
404	InvalidAutoRefresh.TooManyAutoRefreshes	The number of auto refreshes exceeds the limit.
404	InvalidSourceStorage.NeedVersioning	Source storage must enable versioning.
404	InvalidFsetId.NotFound	The specified Fileset ID does not exist.
404	DataFlow.Bucket.NotExist	Bucket does not exist.	The bucket does not exist.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.