E-MapReduce:Establish network connectivity between EMR Serverless Spark and other VPCs

Prerequisites

Before you begin, ensure that you have:

• A DataLake cluster on the EMR on ECS page that includes the Hive service and uses Built-in MySQL for Metadata. For more information, see Create a cluster.

Supported zones

vSwitches are only available in specific zones.

• China Region

  Region name	Region ID	Supported zones
  China (Hangzhou)	cn-hangzhou	Zone H, Zone I, Zone J
  China (Shanghai)	cn-shanghai	Zone B, Zone L, Zone F, Zone G
  China (Beijing)	cn-beijing	Zone F, Zone G, Zone H, Zone K
  China (Shenzhen)	cn-shenzhen	Zone E, Zone F
  China (Hong Kong)	cn-hongkong	Zone B, Zone C

• Other countries and regions

  Region name	Region ID	Supported zones
  Germany (Frankfurt)	eu-central-1	Zone A, Zone B
  Indonesia (Jakarta)	ap-southeast-5	Zone A, Zone B
  Singapore	ap-southeast-1	Zone B, Zone C
  US (Virginia)	us-east-1	Zone A, Zone B
  US (Silicon Valley)	us-west-1	Zone A, Zone B
  Japan (Tokyo)	ap-northeast-1	Zone B, Zone C

Step 1: Add a network connection

1. Log on to the EMR console.

2. In the left navigation pane, choose EMR Serverless > Spark.

3. On the Spark page, click the workspace name.

4. In the left navigation pane, click Network Connection.

5. On the Network Connection page, click Create Network Connection.

6. In the Create Network Connection dialog box, configure the following parameters and click OK. The connection is ready when its Status changes to Succeeded.

   Parameter	Description
   Name	Enter a name for the connection.
   VPC	Select the same VPC as your EMR cluster. If no VPC is available, click Create VPC to create one in the VPC console. For more information, see VPCs and vSwitches.
   vSwitch	Select a vSwitch in the same VPC as your EMR cluster. The vSwitch must be in a supported zone (see Supported zones). If no vSwitch is available in the required zone, create one in the VPC console. For more information, see Create and manage vSwitches.

   

Step 2: Add a security group rule to the EMR cluster

When a Spark job runs, traffic flows from the vSwitch CIDR block into your VPC. To allow that traffic to reach the HMS service, add an inbound rule to the EMR cluster's security group.

1. Get the CIDR block of the vSwitch you selected in Step 1. Log on to the VPC console and go to the vSwitches page to find the CIDR block.

   

2. Log on to the EMR on ECS console.

3. On the EMR on ECS page, click the cluster ID.

4. On the Basic Information tab, in the Security section, click the link next to Cluster Security Group.

5. On the Security Group Details page, go to the Rules section and click Add Rule. Configure the following parameters and click OK.

   Important

   Do not set Authorization Object to 0.0.0.0/0. An overly permissive rule exposes your cluster to attacks from the public internet.

   Parameter	Value
   Protocol	TCP (default). For Kerberos authentication, select UDP and open port 88 instead. For more information, see Enable Kerberos authentication.
   Source	The CIDR block of the vSwitch from step 1.
   Destination (Current Instance)	Specify the destination port to allow access. For example, 9083.

(Optional) Step 3: Set up a Hive table

Skip this step if you already have a Hive table to query.

1. Use SSH to log on to the master node of your EMR cluster. For more information, see Log on to a cluster.

2. Enter the Hive command line:

   hive

3. Create a table:

   CREATE TABLE my_table (id INT, name STRING);

4. Insert sample data:

   INSERT INTO my_table VALUES (1, 'John');
   INSERT INTO my_table VALUES (2, 'Jane');

5. Verify the data:

   SELECT * FROM my_table;

(Optional) Step 4: Build and upload a JAR artifact

Skip this step if you plan to use a Spark SQL job.

1. Create a Maven project on your local machine with the following class:

   The project contains the following content:

   package com.example;
   
   import org.apache.spark.sql.Dataset;
   import org.apache.spark.sql.Row;
   import org.apache.spark.sql.SparkSession;
   
   public class DataFrameExample {
       public static void main(String[] args) {
           // Create a SparkSession with Hive support enabled.
           SparkSession spark = SparkSession.builder()
                   .appName("HMSQueryExample")
                   .enableHiveSupport()
                   .getOrCreate();
   
           // Query the HMS table.
           Dataset<Row> result = spark.sql("SELECT * FROM default.my_table");
   
           // Print the results.
           result.show();
   
           spark.stop();
       }
   }

   The pom.xml file contains the following content:

   <?xml version="1.0" encoding="UTF-8"?>
   <project xmlns="http://maven.apache.org/POM/4.0.0"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
       <modelVersion>4.0.0</modelVersion>
   
       <groupId>org.example</groupId>
       <artifactId>sparkDataFrame</artifactId>
       <version>1.0-SNAPSHOT</version>
   
       <properties>
           <maven.compiler.source>8</maven.compiler.source>
           <maven.compiler.target>8</maven.compiler.target>
           <spark.version>3.3.1</spark.version>
           <scala.binary.version>2.12</scala.binary.version>
       </properties>
   
       <dependencies>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-core_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-sql_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
           <dependency>
               <groupId>org.apache.spark</groupId>
               <artifactId>spark-hive_${scala.binary.version}</artifactId>
               <version>${spark.version}</version>
           </dependency>
       </dependencies>
   </project>

2. Build the JAR:

   mvn package

   This produces sparkDataFrame-1.0-SNAPSHOT.jar in the target/ directory.

3. Upload the JAR to your workspace:

   1. On the EMR Serverless Spark page for your workspace, click Artifacts in the left navigation pane.

   2. On the Artifacts page, click Upload File and upload sparkDataFrame-1.0-SNAPSHOT.jar.

Step 5: Create and run a job