E-MapReduce:Manage Spark Thrift Server sessions

Connect to Spark Thrift Server by using Python

1. Run the following command to install the PyHive and Thrift packages:

   pip install pyhive thrift

2. Write a Python script to connect to Spark Thrift Server.

   The following is an example Python script that shows how to connect to Hive and display a list of databases. Based on your network environment (public or internal), you can choose the appropriate connection method.

   Connect by using a public endpoint

   from pyhive import hive
   
   if __name__ == '__main__':
       # Replace <endpoint>, <username>, and <token> with your actual information.
       cursor = hive.connect('<endpoint>', port=443, scheme='https', username='<username>', password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

   Connect by using an internal same-region endpoint

   from pyhive import hive
   
   if __name__ == '__main__':
       # Replace <endpoint>, <username>, and <token> with your actual information.
       cursor = hive.connect('<endpoint>', port=80, scheme='http', username='<username>', password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

Connect to Spark Thrift Server by using Java

1. Add the following Maven dependencies to your pom.xml file:

   <dependencies>
           <dependency>
               <groupId>org.apache.hadoop</groupId>
               <artifactId>hadoop-common</artifactId>
               <version>3.0.0</version>
           </dependency>
           <dependency>
               <groupId>org.apache.hive</groupId>
               <artifactId>hive-jdbc</artifactId>
               <version>2.1.0</version>
           </dependency>
       </dependencies>
   

   Note

   The built-in Hive version in Serverless Spark is 2.x. Therefore, only hive-jdbc 2.x versions are supported.

2. Write Java code to connect to Spark Thrift Server.

   The following is a sample Java code that connects to Spark Thrift Server and queries the database list.

   Connect by using a public endpoint

   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.ResultSetMetaData;
   import org.apache.hive.jdbc.HiveStatement;
   
   public class Main {
       public static void main(String[] args) throws Exception {
           String url = "jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token>";
           Class.forName("org.apache.hive.jdbc.HiveDriver");
           Connection conn = DriverManager.getConnection(url);
           HiveStatement stmt = (HiveStatement) conn.createStatement();
   
           String sql = "show databases";
           System.out.println("Running " + sql);
           ResultSet res = stmt.executeQuery(sql);
   
           ResultSetMetaData md = res.getMetaData();
           String[] columns = new String[md.getColumnCount()];
           for (int i = 0; i < columns.length; i++) {
               columns[i] = md.getColumnName(i + 1);
           }
           while (res.next()) {
               System.out.print("Row " + res.getRow() + "=[");
               for (int i = 0; i < columns.length; i++) {
                   if (i != 0) {
                       System.out.print(", ");
                   }
                   System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
               }
               System.out.println(")]");
           }
           
           conn.close();
       }
   }

   Connect by using an internal same-region endpoint

   import java.sql.Connection;
   import java.sql.DriverManager;
   import java.sql.ResultSet;
   import java.sql.ResultSetMetaData;
   import org.apache.hive.jdbc.HiveStatement;
   
   public class Main {
       public static void main(String[] args) throws Exception {
           String url = "jdbc:hive2://<endpoint>:80/;transportMode=http;httpPath=cliservice/token/<token>";
           Class.forName("org.apache.hive.jdbc.HiveDriver");
           Connection conn = DriverManager.getConnection(url);
           HiveStatement stmt = (HiveStatement) conn.createStatement();
   
           String sql = "show databases";
           System.out.println("Running " + sql);
           ResultSet res = stmt.executeQuery(sql);
   
           ResultSetMetaData md = res.getMetaData();
           String[] columns = new String[md.getColumnCount()];
           for (int i = 0; i < columns.length; i++) {
               columns[i] = md.getColumnName(i + 1);
           }
           while (res.next()) {
               System.out.print("Row " + res.getRow() + "=[");
               for (int i = 0; i < columns.length; i++) {
                   if (i != 0) {
                       System.out.print(", ");
                   }
                   System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
               }
               System.out.println(")]");
           }
           
           conn.close();
       }
   }

Connect to Spark Thrift Server by using Spark Beeline

• If you are using a self-built cluster, you need to first go to the bin directory of Spark, and then use beeline to connect to Spark Thrift Server.

  Connect by using a public endpoint

  cd /opt/apps/SPARK3/spark-3.4.2-hadoop3.2-1.0.3/bin/
  
  ./beeline -u "jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token>"

  Connect by using an internal same-region endpoint

  cd /opt/apps/SPARK3/spark-3.4.2-hadoop3.2-1.0.3/bin/
  
  ./beeline -u "jdbc:hive2://<endpoint>:80/;transportMode=http;httpPath=cliservice/token/<token>"

  Note

  The code uses /opt/apps/SPARK3/spark-3.4.2-hadoop3.2-1.0.3 as an example of the Spark installation path in an EMR on ECS cluster. You need to adjust the path based on the actual Spark installation path on your client. If you are not sure about the Spark installation path, you can use the env | grep SPARK_HOME command to find it.

• If you are using an EMR on ECS cluster, you can directly use the Spark Beeline client to connect to Spark Thrift Server.

  Connect by using a public endpoint

  spark-beeline -u "jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token>"

  Connect by using an internal same-region endpoint

  spark-beeline -u "jdbc:hive2://<endpoint>:80/;transportMode=http;httpPath=cliservice/token/<token>"

When you use Hive Beeline to connect to Serverless Spark Thrift Server, if the following error occurs, it is usually because the version of Hive Beeline is incompatible with Spark Thrift Server. Therefore, we recommend that you use Hive Beeline 2.x.

24/08/22 15:09:11 [main]: ERROR jdbc.HiveConnection: Error opening session
org.apache.thrift.transport.TTransportException: HTTP Response code: 404

Connect to Spark Thrift Server by configuring Apache Superset

Apache Superset is a modern data exploration and visualization platform with a rich set of chart forms, from simple line charts to highly detailed geospatial charts. For more information about Superset, see Superset.

1. Install dependencies.

   Make sure that you have installed the thrift package version 0.20.0. If not, you can install it by using the following command:

   pip install thrift==0.20.0

2. Start Superset and go to the Superset interface.

   For more information about how to start Superset, see Superset documentation.

3. Click DATABASE in the upper-right corner of the page to go to the Connect a database page.

4. On the Connect a database page, select Apache Spark SQL.

   

5. Enter the connection string and configure the related data source parameters.

   Connect by using a public endpoint

   hive+https://<username>:<token>@<endpoint>:443/<db_name>

   Connect by using an internal same-region endpoint

   hive+http://<username>:<token>@<endpoint>:80/<db_name>

6. Click FINISH to confirm successful connection and verification.

Connect to Spark Thrift Server by configuring Hue

Hue is a popular open-source web interface for interacting with the Hadoop ecosystem. For more information about Hue, see Hue official documentation.

1. Install dependencies.

   Make sure that you have installed the thrift package version 0.20.0. If not, you can install it by using the following command:

   pip install thrift==0.20.0

2. Add a Spark SQL connection string to the Hue configuration file.

   Find the Hue configuration file (usually located at /etc/hue/hue.conf) and add the following content to the file:

   Connect by using a public endpoint

   [[[sparksql]]]
        name = Spark Sql
        interface=sqlalchemy
        options='{"url": "hive+https://<username>:<token>@<endpoint>:443/"}'

   Connect by using an internal same-region endpoint

   [[[sparksql]]]
        name = Spark Sql
        interface=sqlalchemy
        options='{"url": "hive+http://<username>:<token>@<endpoint>:80/"}'

3. Restart Hue.

   After you modify the configuration, you need to run the following command to restart the Hue service for the changes to take effect:

   sudo service hue restart

4. Verify the connection.

   After Hue is successfully restarted, access the Hue interface and find the Spark SQL option. If the configuration is correct, you should be able to successfully connect to Spark Thrift Server and execute SQL queries.

   

Connect to Spark Thrift Server by using DataGrip

DataGrip is a database management environment for developers, designed to facilitate the querying, creation, and management of databases. Databases can run locally, on a server, or in the cloud. For more information about DataGrip, see DataGrip.

1. Install DataGrip. For more information, see Install DataGrip.

   The DataGrip version used in this example is 2025.1.2.

2. Open the DataGrip client and go to the DataGrip interface.

3. Create a project.

   1. Click and select Flie > New > > Project.

      

   2. In the New Project dialog box, enter a project name, such as Spark, and click OK.

4. Click the icon in the Database Explorer menu bar. Select Data Source > Other > Apache Spark.

   

5. In the Data Sources and Drivers dialog box, configure the following parameters:

   

   Tab	Parameter	Description
   General	Name	The custom connection name. For example, spark_thrift_server.
   	Authentication	The authentication method. In this example, No auth is selected. In a production environment, select User & Password to ensure that only authorized users can submit SQL tasks, which improves system security.
   	Driver	Click Apache Spark, and then click Go to Driver to confirm that the Driver version is ver. 1.2.2. Note Because the current Serverless Spark engine version is 3.x, the Driver version must be 1.2.2 to ensure system stability and feature compatibility.
   	URL	The URL for connecting to Spark Thrift Server. Based on your network environment (public or internal), you can choose the appropriate connection method. Connect by using a public endpoint jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token> Connect by using an internal same-region endpoint jdbc:hive2://<endpoint>:80/;transportMode=http;httpPath=cliservice/token/<token>
   Options	Run keep-alive query	This parameter is optional. Select this parameter to prevent automatic disconnection due to timeout.

6. Click Test Connection to confirm that the data source is configured successfully.

   

7. Click OK to complete the configuration.

8. Use DataGrip to manage Spark Thrift Server.

   After DataGrip successfully connects to Spark Thrift Server, you can perform data development. For more information, see DataGrip Help.

   For example, under the created connection, you can right-click a target table and select New > > Query Console. Then, you can write and run SQL scripts in the SQL editor on the right to view table data information.