Elasticsearch:Synchronize Hadoop data to Alibaba Cloud Elasticsearch

Prerequisites

• Create an Alibaba Cloud Elasticsearch cluster and enable its auto-indexing feature. For more information, see Create an Alibaba Cloud Elasticsearch cluster and Configure YML parameters.

  Note

  You can synchronize data only to an Alibaba Cloud Elasticsearch cluster. Self-managed Elasticsearch clusters are not supported.

• A DataWorks workspace is created. For more information, see Create a workspace.

Billing

• For information about the fees for an Alibaba Cloud Elasticsearch instance, see Elasticsearch billable items.

• For information about the fees for a Data Integration resource group, see Resource group fees.

Step 1: Create an exclusive resource group

To ensure fast and stable data transmission, purchase an exclusive resource group for Data Integration and associate it with a workspace and Virtual Private Cloud (VPC).

1. Log on to the DataWorks console.

2. In the top navigation bar, select a region. In the left-side navigation pane, click Resource Groups.

3. On the Exclusive Resource Group tab, click Create legacy resource group > Resource Group for Data Integration.

4. On the DataWorks Exclusive Resources (Subscription) purchase page, set Exclusive Resource Type to exclusive resource group for Data Integration, enter a name for the resource group, and then click Buy Now.

   For more information, see Step 1: Purchase a resource group.

5. Find the created exclusive resource group and click Network Settings in the Actions column to associate the resource group with a VPC. For more information, see Bind a VPC.

   Note

   This topic uses an exclusive resource group for Data Integration to synchronize data over a VPC as an example. For information about how to synchronize data over the internet, see Configure an IP address whitelist.

   The exclusive resource group must be connected to the VPCs of both the Hadoop and Elasticsearch clusters to allow data synchronization between them. Therefore, you must associate the exclusive resource group with the Virtual Private Cloud (VPC), Zone, and VSwitch of both the Hadoop cluster and the Elasticsearch cluster. To view the VPC, zone, and vSwitch of the Elasticsearch cluster, see View the basic information of an Elasticsearch cluster.

   Important

   After you associate the exclusive resource group with a VPC, you must add the vSwitch CIDR Block to the VPC private access whitelists of the Hadoop and Elasticsearch clusters. For more information, see Configure a public or private IP address whitelist for an Elasticsearch cluster.

6. In the upper-left corner of the page, click the back icon to return to the Resource List page.

7. Find the created exclusive resource group and click Associate Workspace in the Actions column to associate the resource group with the destination workspace.

   For more information, see Step 2: Associate a workspace with the resource group.

Step 2: Add data sources

1. Go to the Data Integration page.

   1. Log on to the DataWorks console .

   2. In the left-side navigation pane, click Workspace.

   3. Find the target workspace and choose Shortcuts > Data Integration in the Actions column.

2. In the left-side navigation pane, click Data Source.

3. Add an HDFS data source.

   1. On the Data Sources page, click Add Data Source.

   2. In the Add Data Source dialog box, search for and select HDFS.

   3. On the Add HDFS Data Source page, configure the data source parameters.

      For more information, see Configure an HDFS data source.

   4. Click Test Connectivity. If Connected is displayed, the resource group is connected to the data source.

   5. Click Complete.

4. Similarly, add an Elasticsearch data source. For more information, see Elasticsearch data source.

Step 3: Run a batch synchronization task

A batch synchronization task runs on an exclusive resource group. The resource group obtains data from the data source in Data Integration and writes the data to the destination Elasticsearch cluster.

1. Go to the Data Development page.

   1. Log on to the DataWorks console.

   2. In the left-side navigation pane, click Workspace.

   3. In the Actions column for the destination workspace, choose Shortcuts > Data development.

2. Create an offline synchronization node.

   1. On the Data Development () tab, choose Create > New business process.

   2. Right-click the workflow that you created and choose Create Node > Data Integration > Batch Synchronization.

   3. In the Create Node dialog box, enter a name for the node and click OK.

3. Configure the network and resources.

   1. In the Source section, set Source to HDFS and Data Source Name to the name of the HDFS data source that you want to synchronize.

   2. In the Resource Group section, select the exclusive resource group.

   3. In the Destination section, set Destination to Elasticsearch and Data Source Name to the name of the Elasticsearch data source.

4. Click Next.

5. Configure the task.

   1. In the Source section, select the table that you want to synchronize.

   2. In the Destination section, configure the parameters.

   3. In the Field Mapping section, configure mappings between Source Column and Target Column.

   4. In the Channel Control section, configure the channel parameters.

   For more information, see Configure a batch synchronization task in wizard mode.

6. Run the task.

   1. (Optional) Configure scheduling properties for the task. In the right-side pane, click Properties and configure the parameters as required. For more information about the parameters, see Scheduling configuration.

   2. In the toolbar of the node configuration tab, click the Save icon.

   3. In the toolbar of the node configuration tab, click the Commit icon.

      If you configured scheduling properties, the task runs periodically. You can also click the Run icon in the toolbar to run the task immediately.

      A Shell run successfully! message in the run log indicates that the task completed successfully.

Step 4: Verify the result

1. Log on to the Kibana console of your destination Alibaba Cloud Elasticsearch cluster.

   For more information, see Log on to the Kibana console.

2. In the left-side navigation pane, click Dev Tools.

3. In the Console, run the following command to view the synchronized data.

   POST /hive_esdoc_good_sale/_search?pretty
   {
   "query": { "match_all": {}}
   }

   Note

   hive_esdoc_good_sale is the value you specified for the index parameter in the data synchronization script.

   If the data was synchronized successfully, a result similar to the following is returned.

   
   {
     "took" : 6,
     "timed_out" : false,
     "_shards" : {
       "total" : 5,
       "successful" : 5,
       "skipped" : 0,
       "failed" : 0
     },
     "hits" : {
       "total" : 4,
       "max_score" : 1.0,
       "hits" : [
         {
           "_index" : "hive_esdoc_good_sale",
           "_type" : "_doc",
           "_id" : "2018-08-21 00:00:00",
           "_score" : 1.0,
           "_source" : {
             "trans_num" : 3,
             "click_cnt" : 7,
             "category" : "Outerwear",
             "buyer_id" : "lilei",
             "trans_amount" : 500.6,
             "brand" : "Brand A"
           }
         },
         {
           "_index" : "hive_esdoc_good_sale",
           "_type" : "_doc",
           "_id" : "2018-08-23 00:00:00",
           "_score" : 1.0,
           "_source" : {
             "trans_num" : 5,
             "click_cnt" : 4,
             "category" : "Outerwear",
             "buyer_id" : "jimmy",
             "trans_amount" : 100.2,
             "brand" : "Brand E"
           }
         },
         {
           "_index" : "hive_esdoc_good_sale",
           "_type" : "_doc",
           "_id" : "2018-08-22 00:00:00",
           "_score" : 1.0,
           "_source" : {
             "trans_num" : 2,
             "click_cnt" : 3
           }
         }
       ]
     }
   }