E-MapReduce:Connecting EMR Serverless Spark to other VPCs

Prerequisites

A data source has been prepared. This example uses a data lake cluster created on the EMR on ECS page. The cluster includes the Hive service, and its Metadata is stored in Built-in MySQL. For more information, see Create a cluster.

Usage limits

vSwitches are supported only in the following availability zones.

• Mainland China

  Region name	Region ID	Availability zone name
  China (Hangzhou)	cn-hangzhou	Zone H Zone I Zone J
  China (Shanghai)	cn-shanghai	Zone L Zone F Zone G
  China (Beijing)	cn-beijing	Zone F Zone G Zone H Zone I Zone K
  China (Shenzhen)	cn-shenzhen	Zone E Zone F
  China (Hong Kong)	cn-hongkong	Zone B Zone C

• Other Countries and Regions

  Region name	Region ID	Availability zone name
  Germany (Frankfurt)	eu-central-1	Zone A Zone B
  Indonesia (Jakarta)	ap-southeast-5	Zone A Zone B
  Singapore	ap-southeast-1	Zone B Zone C
  US (Virginia)	us-east-1	Zone A Zone B
  US (Silicon Valley)	us-west-1	Zone A Zone B
  Japan (Tokyo)	ap-northeast-1	Zone B Zone C
  Mexico (Queretaro)	na-south-1	Zone A
  Philippines (Manila)	ap-southeast-6	Zone A Zone B

Step 1: Create a network connection

1. Go to the Network Connection page.

   1. Log on to the EMR console.

   2. In the left navigation pane, choose EMR Serverless > Spark.

   3. On the Spark page, click the target workspace name.

   4. On the EMR Serverless Spark page, in the left navigation pane, click Normal Network Connection.

2. On the Normal Network Connection page, click Create Network Connection.

3. In the Create Network Connection dialog box, configure the following parameters and click OK.

   Parameter	Description
   Name	Enter a name for the connection.
   VPC	Select the same VPC as the EMR cluster. If no VPC is available, click Create VPC to create one in the VPC console. For more information, see VPCs and vSwitches. Note If your Serverless Spark application needs internet access, ensure the network connection provides it. For example, you can deploy a public NAT gateway in the VPC to enable internet access for Serverless Spark instances. For more information, see Public NAT Gateway.
   vSwitch	Select a vSwitch that is in the same VPC as the EMR cluster. If no vSwitch is available in the current zone, click vSwitch to create one in the VPC console. For more information, see Create and manage vSwitches. Important You can select vSwitches only in specific zones. For more information, see Limitations.

   The network connection is created when its Status changes to Succeeded.

   

Step 2: Add a security group rule

1. Get the CIDR block of the vSwitch used by the network connection.

   Log on to the VPC console and go to the vSwitch page to locate the CIDR block of the vSwitch.

   

2. Add a security group rule.

   1. Log on to the EMR on ECS console.

   2. On the EMR on ECS page, click the cluster ID of the target cluster.

   3. On the Basic Information page, in the Security section, click the link next to Cluster Security Group.

   4. On the Security Group Details page, in the Rules section, click Add Rule, configure the following parameters, and then click OK.

      Parameter	Description
      Protocol	The allowed network protocol. Default: TCP. Note If your network connection uses Kerberos authentication, we recommend that you select UDP and open port 88. For more information about Kerberos authentication, see Enable Kerberos authentication.
      Source	Enter the vSwitch CIDR block from the previous step. Important To avoid security risks, do not set Authorization Object to 0.0.0.0/0.
      Destination (Current Instance)	The destination port to open. For example, 9083.

(Optional) Step 3: Connect to Hive and query data

Skip this step if you already have a Hive table.

1. Log on to the master node of your cluster using SSH. For more information, see Log on to a cluster.

2. Start the Hive command line.

   hive

3. Create a table.

   CREATE TABLE my_table (id INT,name STRING);

4. Insert data into the table.

   INSERT INTO my_table VALUES (1, 'John'); 
   INSERT INTO my_table VALUES (2, 'Jane');

5. Query the data.

   SELECT * FROM my_table;

(Optional) Step 4: Build and upload JAR artifact

To run a JAR job, you must first prepare an artifact. Skip this step for Spark SQL jobs.

1. Create a Maven project on your local machine.

   The project contains the following:

   package com.example;
   
   import org.apache.spark.sql.Dataset;
   import org.apache.spark.sql.Row;
   import org.apache.spark.sql.SparkSession;
   
   public class DataFrameExample {
       public static void main(String[] args) {
           // Create a SparkSession with Hive support enabled.
           SparkSession spark = SparkSession.builder()
                   .appName("HMSQueryExample")
                   .enableHiveSupport()
                   .getOrCreate();
   
           // Execute a query.
           Dataset<Row> result = spark.sql("SELECT * FROM default.my_table");
   
           // Print the query results.
           result.show();
   
           // Stop the SparkSession.
           spark.stop();
       }
   }

   The pom.xml file contains the following:

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
   
       <groupId>org.example</groupId>
       <artifactId>sparkDataFrame</artifactId>
       <version>1.0-SNAPSHOT</version>
   
       <properties>
           <maven.compiler.source>8</maven.compiler.source>
           <maven.compiler.target>8</maven.compiler.target>
           <spark.version>3.3.1</spark.version>
           <scala.binary.version>2.12</scala.binary.version>
       </properties>
   
       <dependencies>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-core_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-sql_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-hive_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
       </dependencies>
   </project>

2. Run the mvn package command to package the project. This command generates the sparkDataFrame-1.0-SNAPSHOT.jar file.

3. On the EMR Serverless Spark page for your workspace, click Artifacts in the left navigation pane.

4. On the Artifacts page, click Upload File.

5. Upload the sparkDataFrame-1.0-SNAPSHOT.jar file from your local machine.

Step 5: Create and run a job