AnalyticDB:Configure public internet access for Spark applications

Background

An Internet NAT Gateway provides network address translation (NAT). You can create an Internet NAT Gateway for a Virtual Private Cloud (VPC) to allow instances within the VPC to access the internet. For more information, see What is an Internet NAT gateway?.

Usage notes

If your external data source uses network security controls, such as a firewall or an IP allowlist, you must add the public IP address of the SNAT entry or the IP CIDR block of the vSwitch to those controls. Otherwise, the Spark application cannot access the data source.

Procedure

1. Create an Internet NAT Gateway.

   The Internet NAT Gateway must be in the same region as your AnalyticDB for MySQL instance.

2. Associate an Elastic IP address (EIP).

3. Create an SNAT entry.

   We recommend creating an SNAT entry at Select vSwitch. You can select any vSwitch for the entry.

4. In your Spark application, configure the following key parameters as shown in the example below.

   Note

   This configuration is not required if you develop Spark jobs in a Notebook.

   {
       "comments": ["-- Here is just an example of SparkPi. Modify the content and run your spark program."],
       "args": ["1000"],
       "file": "local:///tmp/spark-examples.jar",
       "name": "SparkPi",
       "className": "org.apache.spark.examples.SparkPi",
       "conf": {
           "spark.driver.resourceSpec": "small",
           "spark.executor.instances": 1,
           "spark.executor.resourceSpec": "small",
           "spark.adb.eni.enabled": "true",
           "spark.adb.eni.vswitchId": "vsw-bp1ghmwrkeaw3xvnd****",
           "spark.adb.eni.securityGroupId": "sg-bp1airvjxl5vpr2****"	
       }
   }

   The following table describes the key parameters.

   Parameter	Description
   spark.adb.eni.enabled	Specifies whether to use an elastic network interface (ENI) to access external data sources through external tables. Set this parameter to true.
   spark.adb.eni.vswitchId	The ID of the vSwitch for the elastic network interface. Set this parameter to the vSwitch ID specified in the SNAT entry.
   spark.adb.eni.securityGroupId	The ID of the security group for the elastic network interface. Set this parameter to the ID of a security group in the VPC where the Internet NAT gateway resides.

   For more information about Spark applications, see Overview of Spark application development.