DescribeDataFlowTasks API Reference for CPFS Dataflow - CPFS - Alibaba Cloud - Cloud Parallel File Storage

Queries the details of data flow tasks.

Operation description

Only CPFS 2.2.0 and later and CPFS for Lingjun 2.4.0 and later support querying data flow tasks. You can view the version information on the file system details page in the console.

Try it now

Try this API in OpenAPI Explorer, no manual signing needed. Successful calls auto-generate SDK code matching your parameters. Download it with built-in credential security for local usage.

Test

RAM authorization

The table below describes the authorization required to call this API. You can define it in a Resource Access Management (RAM) policy. The table's columns are detailed below:

Action: The actions can be used in the Action element of RAM permission policy statements to grant permissions to perform the operation.
API: The API that you can call to perform the action.
Access level: The predefined level of access granted for each API. Valid values: create, list, get, update, and delete.
Resource type: The type of the resource that supports authorization to perform the action. It indicates if the action supports resource-level permission. The specified resource must be compatible with the action. Otherwise, the policy will be ineffective.
- For APIs with resource-level permissions, required resource types are marked with an asterisk (*). Specify the corresponding Alibaba Cloud Resource Name (ARN) in the Resource element of the policy.
- For APIs without resource-level permissions, it is shown as All Resources. Use an asterisk (*) in the Resource element of the policy.
Condition key: The condition keys defined by the service. The key allows for granular control, applying to either actions alone or actions associated with specific resources. In addition to service-specific condition keys, Alibaba Cloud provides a set of common condition keys applicable across all RAM-supported services.
Dependent action: The dependent actions required to run the action. To complete the action, the RAM user or the RAM role must have the permissions to perform all dependent actions.

Action

Access level

Resource type

Condition key

Dependent action

nas:DescribeDataFlowTasks

get

*DataFlow

acs:nas:{#regionId}:{#accountId}:filesystem/{#filesystemId}

None

Request parameters

Parameter	Type	Required	Description	Example
FileSystemId	string	Yes	The file system ID. General-purpose CPFS: must start with `cpfs-`, such as cpfs-099394bd928c**. CPFS for Lingjun: must start with `bmcpfs-`, such as bmcpfs-290w65p03ok64ya**.	bmcpfs-290w65p03ok64ya****
Filters	array<object>	No	The filter conditions.
	object	No
Key	string	No	The name of the filter key. Valid values: DataFlowIds: filters by data flow ID. TaskIds: filters by data flow task ID. Originator: filters by the initiator of the data flow task. TaskActions: filters by the type of the data flow task. DataTypes: filters by the data type of the data flow task. Status: filters by the data flow status. CreateTimeBegin: filters data flow tasks created after the specified time. CreateTimeEnd: filters data flow tasks created before the specified time. StartTimeBegin: filters data flow tasks started after the specified time. StartTimeEnd: filters data flow tasks started before the specified time. EndTimeBegin: filters data flow tasks ended after the specified time. EndTimeEnd: filters data flow tasks ended before the specified time. Valid values: Status : filters by the data flow status. TaskIds : filters by data flow task ID. Originator : filters by the initiator of the data flow task. CreateTimeEnd : filters data flow tasks created before the specified time. EndTimeBegin : filters data flow tasks ended after the specified time. DataFlowIds : filters by data flow ID. StartTimeEnd : filters data flow tasks started before the specified time. TaskActions : filters by the type of the data flow task. EndTimeEnd : filters data flow tasks ended before the specified time. StartTimeBegin : filters data flow tasks started after the specified time. CreateTimeBegin : filters data flow tasks created after the specified time. DataTypes : filters by the data type of the data flow task.	DataFlowIds
Value	string	No	The value of the filter key. Wildcards are not supported. If Key is set to DataFlowIds, Value is set to a data flow ID or part of a data flow ID. You can specify one or more data flow IDs. A maximum of 10 data flow IDs can be specified. Example: `df-194433a5be31**` or `df-194433a5be31,df-234533a5be31`. If Key is set to TaskId, Value is set to a data flow task ID or part of a data flow task ID. You can specify one or more data flow task IDs. A maximum of 10 data flow task IDs can be specified. Example: `task-38aa8e890f45` or `task-38aa8e890f45,task-29ae8e890f45`. If Key is set to TaskActions, Value is set to the type of the data flow task, including Import, Export, Evict, Inventory, StreamImport, and StreamExport**. Combined queries are supported. CPFS for Lingjun supports only Import, Export, StreamImport, and StreamExport. StreamImport and StreamExport are supported only by CPFS for Lingjun 2.6.0 and later. If Key is set to DataTypes, Value is set to the data type of the data flow task, including MetaAndData, Metadata, and Data. Combined queries are supported. If Key is set to Originator, Value is set to the initiator of the data flow task, including User and System. If Key is set to Status, Value is set to the status of the data flow task, including Pending, Executing, Failed, Completed, Canceling, and Canceled. Combined queries are supported. If Key is set to CreateTimeBegin, Value is set to the earliest creation time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`. If Key is set to CreateTimeEnd, Value is set to the latest creation time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`. If Key is set to StartTimeBegin, Value is set to the earliest start time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`. If Key is set to StartTimeEnd, Value is set to the latest start time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`. If Key is set to EndTimeBegin, Value is set to the earliest end time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`. If Key is set to EndTimeEnd, Value is set to the latest end time of the data flow task. Format: `yyyy-MM-ddThh:mmZ`.	df-194433a5be31****
NextToken	string	No	The pagination token that is used in the next request to retrieve a new page of results. If the return results are truncated, you can use NextToken to initiate a new request to retrieve the content after the current truncation point.	TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=
MaxResults	integer	No	The number of results for each query. Valid values: 10 to 100. Default value: 20.	20
WithReports	boolean	No	Specifies whether to query report information. True (default): queries reports. False: does not query reports. Note Setting this parameter to False speeds up the query. Only CPFS for Lingjun is supported.	True

Response elements

Element	Type	Description	Example
	object
RequestId	string	The request ID.	2D69A58F-345C-4FDE-88E4-BF518948****
NextToken	string	The pagination token that is used in the next request to retrieve a new page of results. If the return results are truncated, you can use NextToken to initiate a new request to retrieve the content after the current truncation point.	TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=
TaskInfo	object
Task	array<object>	The collection of data flow task information.
	array<object>
FilesystemId	string	The file system ID.	bmcpfs-290w65p03ok64ya****
DataFlowId	string	The data flow ID.	df-194433a5be31****
TaskId	string	The data flow task ID.	task-38aa8e890f45****
SourceStorage	string	The access address of the source storage. Format: `<storage type>://[<account id>:]<path>`. Where: storage type: only oss is supported. account id: the UID of the account that owns the source storage. path: the name of the OSS bucket. Limits: Only lowercase letters, digits, and hyphens (-) are supported. The name must start and end with a lowercase letter or digit. The maximum length is 128 characters. UTF-8 encoding is used. Note The OSS bucket must be an existing bucket in the region. The account id parameter is supported only by CPFS for Lingjun 2.6.0 and later.	oss://178321033379****:bucket-01
FileSystemPath	string	The path of the fileset in the CPFS file system. Limits: The length is 2 to 1,024 characters. UTF-8 encoding is used. The path must start and end with a forward slash (/). The directory must be a fileset directory on CPFS. Note Only CPFS supports this parameter.	/a/b/c/
Originator	string	The initiator of the data flow task. Valid values: User: the user initiated the task. System: the task is an automatic update task that CPFS automatically initiated based on the automatic update interval. Note Only CPFS supports this parameter.	User
TaskAction	string	The type of the data flow node. Valid values: Import: performs data import of specified data from the source storage to CPFS. Export: exports specified data from CPFS to the source storage. StreamImport: performs data import of specified data from the source storage to CPFS as a data stream node. StreamExport: exports specified data from CPFS to the source storage as a data stream node. Evict: releases the data blocks of files on CPFS. After the release, only metadata is retained on CPFS. You can still query the file, but the data blocks are purged and do not occupy storage capacity on CPFS. When the file data is accessed, it is loaded from the source storage on demand. Inventory: obtains the checklist of files managed by data flow on CPFS, which provides the cache status of files in the data flow. Note StreamImport and StreamExport are supported only by CPFS for Lingjun 2.6.0 and later.	Import
DataType	string	The data type of the data flow task. Valid values: Metadata: the metadata of files, including timestamp, ownership, permission, and other attribute information. If Metadata is selected, only the metadata of files is imported. Users can see the file, but when accessing the file data, the data is loaded from the source storage on demand. Data: the data blocks of files. MetaAndData: the metadata and data blocks of files. Note CPFS for Lingjun supports only the MetaAndData type.	MetaAndData
Progress	integer	The progress of the data flow task. The number of operations that have been performed by the current data flow task.	50
Status	string	The status of the data flow task. Valid values: Pending: the data flow task has been created and is queued but has not started. Executing: the data flow task is being executed. Failed: the data flow task failed. You can view the data flow task report to check the failure reason. Completed: the data flow task is completed. You can view the data flow task report to verify that all files have been transferred correctly. Canceled: the data flow task was canceled and did not complete. Canceling: the data flow task is being canceled.	Failed
ReportPath `deprecated`	string	The path where the data flow task report is stored in the CPFS file system. For CPFS, the task report is generated in the `.dataflow_report` directory of the CPFS file system. For CPFS for Lingjun, an OSS download link is returned. You can use the link to download the task report.	/path_in_cpfs/reportfile.cvs
CreateTime	string	The time when the task was created. The time follows the ISO 8601 standard. Format: yyyy-MM-dd HH:mm:ss.	2021-08-04 18:27:35
StartTime	string	The time when the task started. The time follows the ISO 8601 standard. Format: yyyy-MM-dd HH:mm:ss.	2021-08-04 18:27:35
EndTime	string	The time when the task ended. The time follows the ISO 8601 standard. Format: yyyy-MM-dd HH:mm:ss.	2021-08-04 18:27:35
FsPath	string	The intelligent directory path.	/aa/
ConflictPolicy	string	The file name conflict policy. SKIP_THE_FILE: skips files with the same name. KEEP_LATEST: compares the update time and keeps the latest version. OVERWRITE_EXISTING: forcibly overwrites files with the same name. Valid values: KEEP_LATEST : Compares the update time and keeps the latest version. OVERWRITE_EXISTING : Forcibly overwrites files with the same name. SKIP_THE_FILE : Skips files with the same name.	KEEP_LATEST
Directory	string	The directory in which the data flow task is executed.	/path_in_cpfs/
DstDirectory	string	The destination directory to which the data flow task is mapped.	/path_in_cpfs/
ErrorMsg	string	The error message of the task. Note If this field is not returned or is empty, no error occurred.	{"ErrorKey":"PATH_NOT_ACCESSIBLE","ErrorDetail":"lstat /cpfs/370lx1ev9ss27o****/test/abcdfnotfound: no such file or directory"}
ProgressStats	object	The progress information of the data flow task.
FilesTotal	integer	The number of files scanned at the source.	3
FilesDone	integer	The number of files for which data flow is completed (including skipped files).	3
ActualFiles	integer	The number of files actually transferred.	3
BytesTotal	integer	The amount of data scanned at the source. Unit: bytes.	131092971520
BytesDone	integer	The amount of data for which data flow is completed (including skipped data). Unit: bytes.	131092971520
ActualBytes	integer	The amount of data actually transferred. Unit: bytes.	131092971520
RemainTime	integer	The estimated remaining time. Unit: seconds.	437
AverageSpeed	integer	The average transfer speed. Unit: bytes per second.	342279299
Reports	object
Report	array<object>	The report list. Note Streaming tasks do not support reports. If WithReport is set to True, this parameter returns report data for CPFS for Lingjun. Only CPFS for Lingjun supports the WithReport parameter.
	object
Name	string	The report name. CPFS: TotalFilesReport: the task report. CPFS for Lingjun: FailedFilesReport: the report of failed files. SkippedFilesReport: the report of skipped files. SuccessFilesReport: the report of successful files.	TotalFilesReport
Path	string	The report link.	https://a-hbr-temp-cn-hangzhou-staging.oss-cn-hangzhou.aliyuncs.com/temp/report/162319438359**/job-000bb6fwqficjbxk/job-000bb6fwqficjbxk_failed.zip?Expires=1721201422&OSSAccessKeyId=LTA**************&Signature=Fp%2BvauORTIVxooXY2tec6z0T%2Bp4%3D
Includes	string	The directories under the specified directory to filter. The content of the filtered directories is transferred. Note Only CPFS for Lingjun supports this feature.	["/test/","/test1/"]
TransferFileListPath	string	The specified OSS directory. Data is synchronized based on the content of the CSV files in the OSS directory. Note Only CPFS for Lingjun supports this feature.	/path_in_cpfs/
EntryList	string		["/path_in_cpfs/file1", "/path_in_cpfs/file2"]

Examples

Success response

JSON format

{
  "RequestId": "2D69A58F-345C-4FDE-88E4-BF518948****",
  "NextToken": "TGlzdFJlc291cmNlU****mVzJjE1MTI2NjY4NzY5MTAzOTEmMiZORnI4NDhVeEtrUT0=",
  "TaskInfo": {
    "Task": [
      {
        "FilesystemId": "bmcpfs-290w65p03ok64ya****",
        "DataFlowId": "df-194433a5be31****",
        "TaskId": "task-38aa8e890f45****",
        "SourceStorage": "oss://178321033379****:bucket-01",
        "FileSystemPath": "/a/b/c/",
        "Originator": "User",
        "TaskAction": "Import",
        "DataType": "MetaAndData ",
        "Progress": 50,
        "Status": "Failed",
        "ReportPath": "/path_in_cpfs/reportfile.cvs",
        "CreateTime": "2021-08-04 18:27:35",
        "StartTime": "2021-08-04 18:27:35",
        "EndTime": "2021-08-04 18:27:35",
        "FsPath": "/aa/",
        "ConflictPolicy": "KEEP_LATEST",
        "Directory": "/path_in_cpfs/",
        "DstDirectory": "/path_in_cpfs/\n",
        "ErrorMsg": "{\"ErrorKey\":\"PATH_NOT_ACCESSIBLE\",\"ErrorDetail\":\"lstat /cpfs/370lx1ev9ss27o****/test/abcdfnotfound: no such file or directory\"}",
        "ProgressStats": {
          "FilesTotal": 3,
          "FilesDone": 3,
          "ActualFiles": 3,
          "BytesTotal": 131092971520,
          "BytesDone": 131092971520,
          "ActualBytes": 131092971520,
          "RemainTime": 437,
          "AverageSpeed": 342279299
        },
        "Reports": {
          "Report": [
            {
              "Name": "TotalFilesReport",
              "Path": "https://a-hbr-temp-cn-hangzhou-staging.oss-cn-hangzhou.aliyuncs.com/temp/report/162319438359****/job-000bb6fwqficjbxk****/job-000bb6fwqficjbxk****_failed.zip?Expires=1721201422&OSSAccessKeyId=LTA****************&Signature=Fp%2BvauORTIVxooXY2tec6z0T%2Bp4%3D"
            }
          ]
        },
        "Includes": "[\"/test/\",\"/test1/\"]\n",
        "TransferFileListPath": "/path_in_cpfs/\n",
        "EntryList": "[\"/path_in_cpfs/file1\", \"/path_in_cpfs/file2\"]"
      }
    ]
  }
}

Error codes

HTTP status code	Error code	Error message	Description
400	IllegalCharacters	The parameter contains illegal characters.	The parameter contains illegal characters.
400	MissingFileSystemId	FileSystemId is mandatory for this action.	The FileSystemId parameter is not specified.
400	InvalidFilesystemVersion.NotSupport	This Api does not support this fileSystem version.	This Api does not support this fileSystem version.
403	OperationDenied.DataFlowNotSupported	The operation is not supported.	The file system does not support data flows.
404	InvalidParameter.InvalidNextToken	The specified NextToken is invaild.
404	InvalidFileSystem.NotFound	The specified file system does not exist.	The specified file system does not exist.
404	InvalidDataFlow.NotFound	The specified data flow does not exist.	The specified dataflow does not exist.
404	InvalidParameter.InvalidMaxResults	The specified MaxResults is invalid.	The specified MaxResults is invalid.
404	InvalidFilterParam	The specified Filter.N.Key is invalid.	The specified Filter.N.Key parameter is invalid.

See Error Codes for a complete list.

Release notes

See Release Notes for a complete list.