Cloud Parallel File Storage:Manage dataflow tasks

Task overview

• Task types

  • Tasks are classified into three types based on their data operations: import, export, and evict.

    Type	Description
    Import	Imports data from a source storage to a CPFS file system. Import type: You can import two types of data: metadata and data. Metadata: Imports only the metadata of files. Data: Imports both the metadata and data of files. Import path: The path of the source file in the OSS bucket. If an imported file or directory does not have POSIX metadata attributes, the default owner is root and the default permission is 0770.
    Export	Exports a specified directory or file from a dataflow fileset to an OSS bucket. Export path: The path of the file or directory in the CPFS file system. Empty directories, hard links, and symbolic links cannot be exported to OSS. Metadata export: You can export the CreateTime, ModifyTime, Ownership, and Permission attributes of a file to an OSS bucket. However, the ChangeTime attribute is not exported to the OSS bucket. Warning CPFS exports metadata named x-oss-meta-afm-xxx to the custom metadata of an OSS bucket. Do not delete or modify this metadata. Otherwise, the metadata in the file system will be incorrect.
    Evict	Releases a file's data from the CPFS file system. After eviction, only the file's metadata remains. While the file is still visible, its data blocks are cleared and no longer occupy storage space. When you access the file again, its data is loaded on demand from the source storage, such as OSS. Note Before you evict a file, make sure that the latest version of the file exists in the OSS bucket.

  • Tasks are also classified by initiator as either user tasks or system tasks.

    Type	Description
    User task	A dataflow task created in the console or by calling the CreateDataFlowTask API operation. You can view this on the Dataflow > Task Management pane in the console. After a user task completes, CPFS generates a task report and saves it to the .dataflow_report directory on the CPFS file system.
    System task	A task that CPFS automatically generates after you enable Automatic Metadata Update. This task synchronizes updated file metadata from an OSS bucket to CPFS. CPFS automatically generates system tasks at the interval specified in Metadata Refresh Interval (minutes) to synchronize updated file metadata from the OSS bucket. You can find information on the Dataflow > Task Management panel in the console. System tasks do not generate task reports.

• Task execution scope

  The scope of a task can be a directory or a specified file list (EntryList). If the scope is a directory, the task traverses all files in the directory tree.

Create a dataflow task

1. Log on to the NAS console.

2. In the left-side navigation pane, choose File System > File System List.

3. In the top navigation bar, select a region.

4. On the File System List page, click the name of the file system.

5. On the details page of the file system, click Dataflow.

6. On the Dataflow tab, find the target dataflow and click Task Management.

7. In the Task Management panel, click Create Job.

8. In the Create Job panel, you can create various types of tasks and configure their parameters.

   Import data

   Parameter	Description
   Data Type	Select the type of data to import. Data: Imports both the data blocks and metadata of files. Metadata: Imports only the metadata of files. When you import only file metadata, you can query only the filenames. When you access the file, its data is loaded on demand from the source storage.
   Import Objects from OSS	Select the directory or file list that the dataflow task will process. Import Objects from OSS: The specified OSS directory must start and end with a forward slash (/). Import Listed Objects: Each line in the file specifies the path of an object in the OSS bucket. Directories are not supported.

   Export data

   • Empty directories, hard links, and symbolic links cannot be exported to an OSS bucket.

   • You can export the CreateTime, ModifyTime, Ownership, and Permission attributes of a file to an OSS bucket. However, the ChangeTime attribute is not exported to the OSS bucket.

   • CPFS exports metadata to the custom metadata of an OSS bucket under the name x-oss-meta-afm-xxx. You must not delete or modify this metadata. Otherwise, the metadata in the file system will be incorrect.

     Parameter	Description
     Specify CPFS subdirectory	Select the directory or file list that the dataflow task will process. Export Files from CPFS: The directory path must be the path in the CPFS file system and must start and end with a forward slash (/). Export Listed Files: Each line in the file specifies the path of a file in the CPFS file system. Directories are not supported.

   Evict data

   Parameter	Description
   Delete File	Select the directory or file list that the dataflow task will process. Delete Files from CPFS: The directory path must start and end with a forward slash (/). Delete Listed Files: Each line in the file specifies the path of a file in the CPFS file system. Directories are not supported.

9. Review the settings and click OK.

   Note

   While a dataflow task is running, CPFS suspends the automatic data update task for that dataflow.

View a task report

1. Log on to the NAS console.

2. In the left-side navigation pane, choose File System > File System List.

3. In the top navigation bar, select a region.

4. On the File System List page, click the name of the file system.

5. On the details page of the file system, click Dataflow.

6. On the Dataflow tab, find the target dataflow and click Task Management.

7. In the Task Management panel, find the target task and in the Actions column, choose > Report.

8. Obtain the full path of the target task report and download the report.

   Note

   • Only user tasks generate task reports; system tasks do not.

   • After a user task is complete, the report is saved to the .dataflow_report directory on the CPFS file system.

   The following is a sample task report:

   SUMMARY,dataflowId,taskId,userId,fsId,startDate,endData,total,succ,skip,failed,throughput_MBps
   FILE,path,status,size
   
   SUMMARY,df-0001,task-0001,1001,cpfs-1234,1632477577,1632477677,18,10,1,7,0.01
   FILE,test1/object1,cached,131072
   FILE,test1/object2,cached,131072

   Category	Field	Description
   Task statistics (SUMMARY)	dataflowId	The dataflow ID.
   	taskId	The task ID.
   	userId	The user ID.
   	fsId	The file system ID.
   	startDate	The time when the task started, in seconds since the epoch.
   	endDate	The time when the task ended, in seconds since the epoch.
   	total	Total number of files that the task processed.
   	succ	Number of files that the task successfully processed.
   	skip	Number of files that the task skipped, such as files that already existed for an import task.
   	failed	Number of files that the task failed to process.
   	throughput_MBps	The average throughput during task execution, in MB/s.
   File information (FILE)	path	The path of the file in the fileset.
   	status	The status of the file. cached: The file is imported or exported. uncached: The file is not imported. dirty: The file was modified in the CPFS file system and has not been exported. NA: The file does not exist.
   	size	The file size, in bytes.

Related operations