Manage CPFS for Lingjun data flows - Cloud Parallel File Storage

Prerequisites

Enable versioning for the OSS bucket to prevent data conflicts when multiple data flows export data to the same bucket. For more information, see Introduction to versioning.

Limitations

Same-account dataflows require CPFS for Lingjun 2.4.0 or later. Cross-account dataflows require 2.6.0 or later.
Maximum 10 dataflows per CPFS for Lingjun file system.
Each file system path can link to only one OSS bucket.
Dataflows cannot span different regions.

For more information about data flow limits, see Limits.

Create a same-account data flow

Log on to the NAS console.
In the left-side navigation pane, choose File System > File System List.
In the top navigation bar, select a region.
On the File System List page, click the name of the file system.
On the file system details page, click the Dataflow tab.
On the Dataflow tab, click Create Dataflow.

In the Create Dataflow dialog box, configure the following parameters.

Parameter	Description
CPFS File System Path	The path in the CPFS for Lingjun file system to associate with the OSS bucket. Constraints: The path must be 1 to 1,023 characters in length. The path must start and end with a forward slash (/).
OSS Bucket	The OSS bucket to associate with the CPFS for Lingjun file system path. Select Select a bucket in the current account and then select the target OSS bucket from the drop-down list.
OSS Object Prefix	The path within the source OSS bucket. Constraints: The prefix must be 1 to 1,023 characters in length. The prefix must start and end with a forward slash (/). The prefix must exist in the OSS bucket.
OSS Bucket SSL	Specifies whether to use HTTPS to access the OSS bucket.
OSS Bucket Tag	To create a data flow, you must tag the OSS bucket with `cpfs-dataflow: true`. While the data flow is active, do not delete or modify this tag. Otherwise, the data flow cannot access the bucket. How to add a tag: Go to the target bucket. In the left-side navigation pane, choose Bucket Settings > Bucket Tagging. Click Create Tag, set the tag key to `cpfs-dataflow` and the tag value to `true`. Click Save, and then return to the NAS console to create the data flow.
SLR Authorization	When you create your first data flow, you must authorize CPFS to assume a Create Dataflow to access Object Storage Service (OSS) resources. For more information, see Service-linked role for Apsara File Storage NAS.

Click OK.

After you click OK, the system validates your settings. This process takes 1 to 2 minutes. The dialog box automatically closes after the validation is complete. Do not manually close the dialog box.

Create a cross-account data flow

When the source OSS bucket is in Account B and the CPFS for Lingjun file system is in Account A, the process has two main steps. First, log on to Account B to authorize the AliyunNasCrossAccountDataFlowDefaultRole role and add the ID of Account A to the role's trust policy. Second, log on to Account A to create the cross-account data flow, as well as import and export tasks.

This section provides an example where the CPFS for Lingjun file system is in Account A and the OSS bucket is in Account B.

Procedure

Authorize the account that owns the source OSS bucket.
1. Log on to the NAS console by using Account B.
2. On the Overview page, find the References section and click Authorization Management.
3. In the Authorization Management panel, find the Cross-account Authorization on Data Flows section and click Authorize Now.
4. Click Authorize.
5. Return to the Authorization Management panel. In the Cross-account Authorization on Data Flows section, click View Details to go to the details page of the AliyunNasCrossAccountDataFlowDefaultRole role.
6. On the Trust Policy tab, click Edit Trust Policy.
7. Change the Service field to the <Account-ID>@nas.aliyuncs.com format.
  
  For example, if the ID of Account A is 178321033379****, change nas.aliyuncs.com in the Service field to 178321033379****@nas.aliyuncs.com. This grants the data flow service in Account A (178321033379****@nas.aliyuncs.com) permission to assume this role.
```
{
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "178321033379****@nas.aliyuncs.com" 
        ]
      }
    }
  ],
  "Version": "1"
}
```

Create the cross-account data flow.

Log on to the NAS console by using Account A.
In the left-side navigation pane, choose File System > File System List.
In the top navigation bar, select a region.
On the File System List page, click the name of the file system.
On the file system details page, click the Dataflow tab.
On the Dataflow tab, click Create Dataflow.

In the Create Dataflow dialog box, configure the following parameters.

Parameter	Description
CPFS File System Path	The path in the CPFS for Lingjun file system to associate with the OSS bucket. Constraints: The path must be 1 to 1,023 characters in length. The path must start and end with a forward slash (/).
OSS Bucket	The OSS bucket to associate with the CPFS for Lingjun file system path. Select Specify a bucket in another account. In the Account ID field, enter the ID of the account that owns the bucket. In the Bucket Name field, enter the bucket's name.
OSS Object Prefix	The path within the source OSS bucket. Constraints: The prefix must be 1 to 1,023 characters in length. The prefix must start and end with a forward slash (/). The prefix must exist in the OSS bucket.
OSS Bucket SSL	Specifies whether to use HTTPS to access the OSS bucket.

Click OK.

After you click OK, the system validates your settings. This process takes 1 to 2 minutes. The dialog box automatically closes after the validation is complete. Do not manually close the dialog box.

Related operations

After you create a data flow, you can view, modify, or delete it in the console.

Actions	Description	Steps
View a data flow	View existing data flows and create tasks for them.	On the Dataflow tab, you can view the configuration information of a specified data flow.
Modify a data flow	You can only modify the description of a data flow.	On the Dataflow tab, find the target data flow. Click Modify and update the description. Click OK.
Delete a data flow	Deletes the data flow and all its associated tasks, and disables data synchronization. Important You cannot delete a data flow if it has running streaming jobs or batch jobs.	On the Dataflow tab, find the target data flow. Click Delete and confirm the deletion. Click OK.

What's next

After you create a data flow, create export or import tasks to transfer data between the CPFS for Lingjun file system and the OSS bucket. For more information, see Create a task.