Cloud Parallel File Storage:CreateDataFlow

Basic operations

• Only Cloud Parallel File Storage (CPFS) V2.2.0 and later and CPFS for Lingjun V2.4.0 and later support data flows.
• You can create a data flow only when a CPFS or CPFS for Lingjun file system is in the Running state.
• A maximum of 10 data flows can be created for a CPFS or CPFS for Lingjun file system.
• It generally takes 2 to 5 minutes to create a data flow. You can call the DescribeDataFlows operation to check whether the data flow has been created.

Permission

When you create a data flow, CPFS obtains the following two service-linked roles: AliyunServiceRoleForNasOssDataflow and AliyunServiceRoleForNasEventNotification. For more information, see CPFS service-linked roles.

CPFS usage notes

• Billing

  • If you create a data flow, you are charged for using the data flow throughput. For more information, see Billing of CPFS.
  • When you configure the AutoRefresh feature for a data flow, CPFS must use EventBridge to collect object modification events from the source Object Storage Service (OSS) bucket. Event fees are incurred. For more information, see Billing of EventBridge.

• Data flow specifications

  • The data flow throughput supports the following specifications: 600 MB/s, 1,200 MB/s, and 1,500 MB/s. The data flow throughput is the maximum transmission bandwidth that can be reached when data is imported or exported for a data flow.
  • When you create a data flow, the vSwitch IP addresses used by a CPFS mount target are consumed. Make sure that the vSwitch can provide sufficient IP addresses.
  • Inventory query: If you set the DryRun parameter to true, you can check whether the resources for the data flow whose throughput is changed meet the requirements.

• Fileset

  • The destination for a data flow is a fileset in the CPFS file system. A fileset is a new directory tree structure (a small file directory) in a CPFS file system. Each fileset independently manages an inode space.
  • When you create a data flow for a CPFS file system, the related fileset must already exist and cannot be nested with other filesets. Only one data flow can be created in a fileset, which corresponds to one source storage.
  • A fileset supports a maximum of one million files. If the number of files imported from an OSS bucket into the fileset exceeds the upper limit, the no space error message is returned when you add new files.

**

**Note **If data already exists in the fileset, after you create a data flow, the existing data in the fileset is cleared and replaced with the data synchronized from the OSS bucket.

• AutoRefresh

  • After AutoRefresh is configured, if the data in the source OSS bucket is updated, the updated metadata is automatically synchronized to the CPFS file system. You can load the updated data when you access files, or run a data flow task to load the updated data.
  • AutoRefresh depends on the object modification events collected by EventBridge from the source OSS bucket. You must first activate EventBridge.
  • The AutoRefresh configuration applies only to the prefix and is specified by the RefreshPath parameter. You can configure a maximum of five AutoRefresh directories for a data flow.
  • AutoRefreshInterval refers to the interval at which CPFS checks whether data is updated in the prefix of the source OSS bucket. If data is updated, CPFS runs an AutoRefresh task. If the frequency of triggering the object modification event in the source OSS bucket exceeds the processing capability of the CPFS data flow, AutoRefresh tasks are accumulated, metadata updates are delayed, and the data flow status becomes Misconfigured. To resolve these issues, you can increase the data flow specifications or reduce the frequency of triggering the object modification event.
  • When you add an AutoRefresh configuration to the prefix for a CPFS data flow, an event bus is created at the user side and an event rule is created for the prefix of the source OSS bucket. When an object is modified in the prefix of the source OSS bucket, an OSS event is generated in the EventBridge console. The event is processed by the CPFS data flow.

  **

  **Note **The event buses and event rules created for CPFS in the EventBridge console contain the Create for cpfs auto refresh description. The event buses and event rules cannot be modified or deleted. Otherwise, AutoRefresh cannot work properly.

• Source storage

  • The source storage is an OSS bucket. SourceStorage for a data flow must be an OSS bucket.
  • CPFS data flows support both encrypted and unencrypted access to OSS. If you select SSL-encrypted access to OSS, make sure that encryption in transit for OSS buckets supports encrypted access.
  • If data flows for multiple CPFS file systems or multiple data flows for the same CPFS file system are stored in the same OSS bucket, you must enable versioning for the OSS bucket to prevent data conflicts caused by data export from multiple CPFS file systems to one OSS bucket.
  • Data flows are not supported for OSS buckets across regions. The OSS bucket must reside in the same region as the CPFS file system.

  **

  **Note **Before you create a data flow, you must configure a tag (key: cpfs-dataflow, value: true) for the source OSS bucket. This way, the created data flow can access the data in the OSS bucket. When a data flow is being used, do not delete or modify the tag. Otherwise, the data flow for CPFS cannot access the data in the OSS bucket.

CPFS for Lingjun usage notes

• Source storage

  • The source storage is an OSS bucket. SourceStorage for a data flow must be an OSS bucket.

  • CPFS for Lingjun data flows support both encrypted and unencrypted access to OSS. If you select SSL-encrypted access to OSS, make sure that encryption in transit for OSS buckets supports encrypted access.

  • If data flows for multiple CPFS for Lingjun file systems or multiple data flows for the same CPFS for Lingjun file system are stored in the same OSS bucket, you must enable versioning for the OSS bucket to prevent data conflicts caused by data export from multiple CPFS for Lingjun file systems to one OSS bucket.

  • Data flows are not supported for OSS buckets across regions. The OSS bucket must reside in the same region as the CPFS file system.

  • CPFS for Lingjun V2.6.0 and later allow you to create data flows for OSS buckets across accounts.

  • The account id parameter is required only when you use OSS buckets across accounts.

  • To use OSS buckets across accounts, you must first grant permissions to the related accounts. For more information, see Cross-account authorization on data flows.

    **

    **Note **Before you create a data flow, you must configure a tag (key: cpfs-dataflow, value: true) for the source OSS bucket. This way, the created data flow can access the data in the OSS bucket. When a data flow is being used, do not delete or modify the tag. Otherwise, the data flow for CPFS for Lingjun cannot access the data in the OSS bucket.

• Limits of data flows on file systems

  • You cannot rename a non-empty directory in a path that is associated with a data flow. Otherwise, the Permission Denied error message or an error message indicating that the directory is not empty is returned.
  • Proceed with caution when you use special characters in the names of directories and files. The following characters are supported: letters, digits, exclamation points (!), hyphens (-), underscores (_), periods (.), asterisks (*), and parentheses (()).
  • The path can be up to 1,023 characters in length.

• Limits of data flows on import

  • After a symbolic link is imported to CPFS for Lingjun, the symbolic link is converted into a common data file that contains no symbolic link information.
  • If an OSS bucket has multiple versions, only data of the latest version is used.
  • The name of a file or a subdirectory can be up to 255 bytes in length.

• Limits of data flows on export

  • After a symbolic link is synchronized to OSS, the file that the symbolic link points to is not synchronized to OSS. In this case, the symbolic link is converted into a common object that contains no data.
  • Hard links can be synchronized to OSS only as common files that contain no link information.
  • After a file of the Socket, Device, or Pipe type is exported to an OSS bucket, the file is converted into a common object that contains no data.
  • The directory path can be up to 1,023 characters in length.