Lindorm:Add a Spark data source

Prerequisites

Before you begin, make sure you have:

• A Lindorm Tunnel Service (LTS) instance

• A Lindorm instance with Lindorm Distributed Processing System (LDPS) activated. See Create an instance

• The Lindorm instance and the LTS instance in the same virtual private cloud (VPC). See Connect VPCs if they are in different VPCs

• (LTS web UI method only) The HDFS namenode hostnames for your Lindorm instance. To get these values, submit a ticket

Add a Spark data source from the Lindorm console

1. Log on to the Lindorm console.

2. On the Instances page, click the ID of the instance whose engine type is LTS.

3. In the left-side navigation pane, click Data Sources.

4. Click the Compute Engine Data Source tab, then click Add Data Source.

5. In the Add Data Source dialog box, configure the following parameters.

   Parameter	Description
   Instance type	Select Lindorm.
   Region	Select the region where your Lindorm instance is deployed.
   Instance ID	Select the ID of your Lindorm instance. LDPS must be activated for the selected instance. See Activate LDPS and modify the configurations.

6. Click OK. When the Spark data source status shows Associated, the data source is added successfully.

Add a Spark data source from the LTS web UI

1. Log on to the LTS web UI. See Activate and log on to LTS.

2. In the left-side navigation pane, choose Data Source Manage > Add Data Source.

3. On the Add data source page, configure the following parameters.

   Parameter	Description
   Name	Enter lts_bulkload_spark.
   Data Source Type	Select Spark.
   Parameters	Enter the JSON configuration below. Replace each placeholder with your actual value.

   {
       "virtualClusterName": "<ldps-access-token>",
       "hdfsUri": "hdfs://<namenode1-hostname>:8020,<namenode2-hostname>:8020",
       "sparkEndpoint": "http://192.168.XX.XX:10099"
   }

   Parameter	Description	How to get the value
   virtualClusterName	The access token for the LDPS JAR address.	On the instance details page, go to Database Connections in the left-side navigation pane, then click the Compute Engine tab.
   hdfsUri	The HDFS endpoint of the Lindorm instance. Format: hdfs://<namenode1-hostname>:8020,<namenode2-hostname>:8020	Submit a ticket to get the namenode hostnames (<namenode1-hostname> and <namenode2-hostname>).
   sparkEndpoint	The JAR VPC address of LDPS. Format: http://<ip-address>:10099	On the instance details page, go to Database Connections in the left-side navigation pane, then click the Compute Engine tab.

4. Click Add.