DataWorks:Data Lake Formation

Limitations

• A DLF data source can only be used in Data Integration.

• A Serverless resource group is required.

Prerequisites

Before you begin, ensure that you have:

• A Serverless resource group created in your DataWorks workspace DLF Data Catalog.

• If you plan to use a RAM user or RAM role as the access identity, the following permissions granted in advance:

  • In the RAM console, attach the AliyunDataWorksDIAccessDLF system policy to the RAM user or RAM role. For details, see Grant permissions to a RAM user.

  • In the Data Lake Formation console, grant the Data Editor permission on the target tables to the RAM user or RAM role.

Add a DLF data source

1. Log on to the DataWorks console. In the top navigation bar, select the desired region. In the left-side navigation pane, choose More > Management Center. Select the desired workspace from the drop-down list and click Go to Management Center.

2. In the left-side navigation pane of the SettingCenter page, click Data Sources.

3. Click Add Data Source, search for and select DLF, then configure the following parameters:

   Parameter	Description
   Data Source Name	Enter a custom name. The name must be unique within the workspace and can contain only letters, digits, and underscores (_). It cannot start with a digit or an underscore.
   Configuration Mode	Only Alibaba Cloud Instance Mode is supported.
   Endpoint	Select the endpoint of the DLF engine instance from the drop-down list.
   Access Identity	Select Alibaba Cloud Account, Alibaba Cloud RAM User, or Alibaba Cloud RAM Role. If you select RAM User or RAM Role, complete the permission setup described in Prerequisites before proceeding.
   DLF Data Catalog	Select a DLF Data Catalog in the same region as your DataWorks workspace.
   Database Name	Select a database in the Data Catalog.

4. Test the connectivity between the data source and the Serverless resource group. If the test passes, click Complete Modification. If the test fails, see Network connectivity configuration for troubleshooting.

Create a data integration task

To use the DLF data source in a Data Integration task, see Data synchronization to Data Lake Formation.

Appendix: Script mode reference

When configuring an offline task in script mode, format the script parameters as shown below. For an overview of script mode, see Use the code editor.

All examples use "stepType": "dlf" to identify the DLF Reader or Writer step.

{
   "type": "job",
   "version": "2.0",
   "steps": [
      {
         "stepType": "dlf",
         "parameter": {
            "datasource": "guxuan_dlf",
            "table": "auto_ob_3088545_0523",
            "column": [
               "id",
               "col1",
               "col2",
               "col3"
            ],
            "tableType": "table",
            "where": "id > 1"
         },
         "name": "Reader",
         "category": "reader"
      },
      {
         "stepType": "stream",
         "parameter": {
            "print": false
         },
         "name": "Writer",
         "category": "writer"
      }
   ],
   "setting": {
      "errorLimit": {
         "record": ""
      },
      "speed": {
         "throttle": true,
         "concurrent": 20,
         "mbps": "12"
      }
   },
   "order": {
      "hops": [
         {
            "from": "Reader",
            "to": "Writer"
         }
      ]
   }
}

{
   "type": "job",
   "version": "2.0",
   "steps": [
      {
         "stepType": "stream",
         "parameter": {
         },
         "name": "Reader",
         "category": "reader"
      },
      {
         "stepType": "dlf",
         "parameter": {
            "datasource": "guxuan_dlf",
            "column": [
               "id",
               "col1",
               "col2",
               "col3"
            ],
            "tableType": "table",
            "table": "auto_ob_3088545_0523"
         },
         "name": "Writer",
         "category": "writer"
      }
   ],
   "setting": {
      "errorLimit": {
         "record": ""
      },
      "speed": {
         "throttle": true,
         "concurrent": 20,
         "mbps": "12"
      }
   },
   "order": {
      "hops": [
         {
            "from": "Reader",
            "to": "Writer"
         }
      ]
   }
}