AnalyticDB:Access a MongoDB data source

Prerequisites

• You have an AnalyticDB for MySQL cluster of the Enterprise Edition, Basic Edition, or Data Lakehouse Edition.

• A database account is created for the AnalyticDB for MySQL cluster.

  • If you use an Alibaba Cloud account, you need to only create a privileged account.

  • If you use a Resource Access Management (RAM) user, you must create a privileged account and a standard account and associate the standard account with the RAM user.

• A job resource group is created.

• An ApsaraDB for MongoDB instance has been created in the same AnalyticDB for MySQL as the AnalyticDB for MySQL cluster. In the ApsaraDB for MongoDB instance, a database and a collection have been created, and data has been written to the collection. For more information, see MongoDB quick start.

• You have added the CIDR block of the VSwitch for your ApsaraDB for MongoDB instance to the instance's allowlist.

  Note

  You can find the VSwitch ID on the Basic Information page of the ApsaraDB for MongoDB console. Log on to the Virtual Private Cloud (VPC) console to view the CIDR block of the target VSwitch.

• You have added the ApsaraDB for MongoDB instance to a security group and configured its inbound and outbound rules to allow access on the MongoDB port. For more information, see Add a security group and Add a security group rule.

Procedure

1. Download the JAR packages required by Spark in AnalyticDB for MySQL to access ApsaraDB for MongoDB. Download links: mongo-spark-connector_2.12-10.1.1.jar, mongodb-driver-sync-4.8.2.jar, bson-4.8.2.jar, bson-record-codec-4.8.2.jar, and mongodb-driver-core-4.8.2.jar.

2. Add the following dependency to the dependencies section of your pom.xml file.

     <dependency>
       <groupId>org.mongodb.spark</groupId>
       <artifactId>mongo-spark-connector_2.12</artifactId>
       <version>10.1.1</version>
     </dependency>

3. Write the following sample code to access ApsaraDB for MongoDB, and then compile and package the code. In this example, name the generated JAR package spark-mongodb.jar. Sample code:

   package com.aliyun.spark
   
   import org.apache.spark.sql.SparkSession
   
   object SparkOnMongoDB {
     def main(args: Array[String]): Unit = {
       // The VPC connection URI for the ApsaraDB for MongoDB instance. You can find this URI on the Database Connection page in the ApsaraDB for MongoDB console.
       val connectionUri = args(0)
       // The name of the ApsaraDB for MongoDB database.
       val database = args(1)
       // The name of the ApsaraDB for MongoDB collection.
       val collection = args(2)
       
       val spark = SparkSession.builder()
         .appName("MongoSparkConnectorIntro")
         .config("spark.mongodb.read.connection.uri", connectionUri)
         .config("spark.mongodb.write.connection.uri", connectionUri)
         .getOrCreate()
   
       val df = spark.read.format("mongodb").option("database", database).option("collection", collection).load()
       
       df.show()
       
       spark.stop()
     }
   }

   Note

   For more Spark-to-MongoDB configuration options, see Configuration Options. For more code examples, see Write to MongoDB and Read from MongoDB.

4. Log on to the AnalyticDB for MySQL console. In the upper-left corner of the console, select a region. In the left-side navigation pane, click Clusters. Find the cluster that you want to manage and click the cluster ID.

5. In the left-side navigation pane, choose Job Development > Spark JAR Development.

6. Above the editor, select a job resource group and a job type. This example uses the Batch job type.

7. Enter the following job configuration in the editor.

   Important

   • Spark in AnalyticDB for MySQL supports access to ApsaraDB for MongoDB over a Virtual Private Cloud (VPC) or the internet.

   • A VPC connection is recommended.

   {
     "args": [
       -- The VPC connection URI for the ApsaraDB for MongoDB instance. You can find this URI on the Database Connection page in the ApsaraDB for MongoDB console.
   	  "mongodb://<username>:<password>@<host1>:<port1>,<host2>:<port2>,...,<hostN>:<portN>/<database_name>",
       -- The name of the ApsaraDB for MongoDB database.
       "<database_name>",
       -- The name of the ApsaraDB for MongoDB collection.
       "<collection_name>"
   	],
     "file": "oss://<bucket_name>/spark-mongodb.jar",
     "jars": [
       "oss://<bucket_name>/mongo-spark-connector_2.12-10.1.1.jar",
       "oss://<bucket_name>/mongodb-driver-sync-4.8.2.jar",
       "oss://<bucket_name>/bson-4.8.2.jar",
       "oss://<bucket_name>/bson-record-codec-4.8.2.jar",
       "oss://<bucket_name>/mongodb-driver-core-4.8.2.jar"
   	],
     "name": "MongoSparkConnectorIntro",
     "className": "com.aliyun.spark.SparkOnMongoDB",
     "conf": {
       "spark.driver.resourceSpec": "medium",
       "spark.executor.instances": 2,
       "spark.executor.resourceSpec": "medium",
       "spark.adb.eni.enabled": "true",
       "spark.adb.eni.vswitchId": "vsw-bp14pj8h0****",
       "spark.adb.eni.securityGroupId": "sg-bp11m93k021tp****"
     }
   }

   The following table describes the parameters.

   Parameter	Description
   args	The arguments for your JAR package. Separate multiple arguments with a comma (,).
   file	The OSS path to the spark-mongodb.jar sample program.
   jars	The OSS paths to the required dependency JARs.
   name	The name of the Spark job.
   className	Not required for Python programs.
   spark.adb.eni.enabled	Specifies whether to enable access over an elastic network interface (ENI). To use Spark in a Data Lakehouse Edition cluster to access an ApsaraDB for MongoDB data source, enable ENI-based access.
   spark.adb.eni.vswitchId	The VSwitch ID. You can find this ID on the Basic Information page of the ApsaraDB for MongoDB console.
   spark.adb.eni.securityGroupId	The ID of the security group that contains the ApsaraDB for MongoDB instance. If you have not added a security group, see Add a security group.
   Other conf parameters	The configurations are basically consistent with those in Apache Spark, where parameters are in the key:value format and multiple parameters are separated by commas (,). For more conf parameters, see Configuration parameters.

8. Click Run Now.

9. In the Applications, wait for the application's status to become Completed. Then, in the Actions column, click Logs to view the data of the MongoDB collection.