E-MapReduce:Manage Kyuubi Gateways

Manage tokens

1. On the Kyuubi Gateway page, locate the target gateway and click Token Management in the Actions column.

2. Click Create Token.

3. In the Create Token dialog box, configure the parameters and click OK.

   Parameter	Description
   Name	The name of the new token.
   Expired At	Set the time-to-live (TTL) for the token. The number of days must be greater than or equal to 1. By default, this feature is enabled and the token expires after 365 days.
   Assign Object	Note If you use DLF (formerly DLF 2.5) by default on the Catalogs tab, you must configure this parameter. Make sure that the configured Resource Access Management (RAM) user or RAM role has the permissions to access DLF. For more information about how to grant permissions, see Add an authorization. The RAM user or RAM role must be added to the Spark workspace before it appears in the Assign Object drop-down list. For more information, see Manage users and roles. From the drop-down list, select the RAM user or RAM role that you added in Access Control. Specify the RAM user or RAM role to which the token is assigned. This user or role is used to access DLF when you connect to the Kyuubi Gateway to submit a Spark task.

4. Copy the token information.

   Important

   After a token is created, you must copy the token information immediately because you cannot view it again later. If a token expires or is lost, you must create or reset the token.

When you connect to a Kyuubi Gateway, replace the placeholders in the JDBC URL with your actual information:

• <endpoint>: The endpoint that you can obtain from the Overview tab.

• <port>: The port number. Use port 443 for public endpoint access and port 80 for internal same-region endpoint access.

• <token>: The token that you copied from the Token Management tab.

• <tokenname>: The token name. You can obtain the name from the Token Management tab.

• <UserName/RoleName>: The RAM user or RAM role that you added in Access Control.

Connect using Beeline

When you connect to a Kyuubi Gateway, ensure that your Beeline version is compatible with the Kyuubi server-side version. If you have not installed Beeline, see Getting Started - Apache Kyuubi.

Perform one of the following operations based on the default catalog that is used on the Catalogs page.

beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;user=<UserName/RoleName>;httpPath=cliservice/token/<token>"

beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>"

When you connect using Beeline, you can modify session parameters. For example, beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>;#spark.sql.shuffle.partitions=100;spark.executor.instances=2;".

Connect using Java

• Update the pom.xml file.

  Replace hadoop-common and hive-jdbc with the appropriate dependency versions.

  <dependencies>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-common</artifactId>
              <version>3.0.0</version>
          </dependency>
          <dependency>
              <groupId>org.apache.hive</groupId>
              <artifactId>hive-jdbc</artifactId>
              <version>2.3.9</version>
          </dependency>
      </dependencies>

• Write Java code to connect to the Kyuubi Gateway.

  Perform one of the following operations based on the default catalog that is used on the Catalogs page.

  Use DLF (formerly DLF 2.5)

  import org.apache.hive.jdbc.HiveStatement;
  import java.sql.Connection;
  import java.sql.DriverManager;
  import java.sql.ResultSet;
  import java.sql.ResultSetMetaData;
  
  public class Main {
      public static void main(String[] args) throws Exception {
          String url = "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>;user=<UserName/RoleName>";
          Class.forName("org.apache.hive.jdbc.HiveDriver");
          Connection conn = DriverManager.getConnection(url);
          HiveStatement stmt = (HiveStatement) conn.createStatement();
  
  
          String sql = "select * from students;";
          System.out.println("Running " + sql);
          ResultSet res = stmt.executeQuery(sql);
  
          ResultSetMetaData md = res.getMetaData();
          String[] columns = new String[md.getColumnCount()];
          for (int i = 0; i < columns.length; i++) {
              columns[i] = md.getColumnName(i + 1);
          }
          while (res.next()) {
              System.out.print("Row " + res.getRow() + "=[");
              for (int i = 0; i < columns.length; i++) {
                  if (i != 0) {
                      System.out.print(", ");
                  }
                  System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
              }
              System.out.println(")]");
          }
  
          conn.close();
      }
  }

  Use other catalogs

  import org.apache.hive.jdbc.HiveStatement;
  import java.sql.Connection;
  import java.sql.DriverManager;
  import java.sql.ResultSet;
  import java.sql.ResultSetMetaData;
  
  public class Main {
      public static void main(String[] args) throws Exception {
          String url = "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>";
          Class.forName("org.apache.hive.jdbc.HiveDriver");
          Connection conn = DriverManager.getConnection(url);
          HiveStatement stmt = (HiveStatement) conn.createStatement();
  
  
          String sql = "select * from students;";
          System.out.println("Running " + sql);
          ResultSet res = stmt.executeQuery(sql);
  
          ResultSetMetaData md = res.getMetaData();
          String[] columns = new String[md.getColumnCount()];
          for (int i = 0; i < columns.length; i++) {
              columns[i] = md.getColumnName(i + 1);
          }
          while (res.next()) {
              System.out.print("Row " + res.getRow() + "=[");
              for (int i = 0; i < columns.length; i++) {
                  if (i != 0) {
                      System.out.print(", ");
                  }
                  System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
              }
              System.out.println(")]");
          }
  
          conn.close();
      }
  }

Connect using Python

1. Run the following command to install the PyHive and Thrift packages.

   pip3 install pyhive thrift

2. Write a Python script to connect to the Kyuubi Gateway.

   The following example shows a Python script that connects to a Kyuubi Gateway and displays a list of databases.

   Perform one of the following operations based on the default catalog that is used on the Catalogs page.

   Use DLF (formerly DLF 2.5)

   from pyhive import hive
   
   if __name__ == '__main__':
       cursor = hive.connect('<endpoint>',
                             port="<port>",
                             scheme='http',
                             username='<UserName/RoleName>',
                             password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

   Use other catalogs

   from pyhive import hive
   
   if __name__ == '__main__':
       cursor = hive.connect('<endpoint>',
                             port="<port>",
                             scheme='http',
                             username='<tokenname>',
                             password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

Connect using the REST API

Kyuubi Gateway provides open-source-compatible REST APIs that support interaction with the Kyuubi service over HTTP. Currently, only the following API paths are supported:

• /api/v1/sessions/*

• /api/v1/operations/*

• /api/v1/batches/*

The following examples show how to use the REST API to connect to a Kyuubi Gateway.

• Example 1: Start a session and perform an SQL query.

  1. Create a session and specify Spark configurations.

     Perform one of the following operations based on the default catalog that is used on the Catalogs page.

     Note

     • spark.emr.serverless.kyuubi.engine.queue specifies the queue that is used by the Spark task at runtime. Replace <dev_queue> with the actual queue name.

     • Replace <UserName/Rolename> with the actual username or role name.

     • <password> is a placeholder. You can enter any value.

     Use DLF (formerly DLF 2.5)

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -u '<UserName/Rolename>:<password>' \
       -d '{
       "configs": {
         "set:hivevar:spark.emr.serverless.kyuubi.engine.queue": "<dev_queue>"
       }
     }'

     Use other catalogs

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -d '{
       "configs": {
         "set:hivevar:spark.emr.serverless.kyuubi.engine.queue": "<dev_queue>"
       }
     }'

     A message similar to the following is returned. In the message, identifier is the Kyuubi session handle, which uniquely identifies a session. In this topic, this value is referred to as <sessionHandle>.

     {"identifier":"619e6ded-xxxx-xxxx-xxxx-c2a43f6fac46","kyuubiInstance":"0.0.0.0:10099"}

  2. Create a statement.

     Use DLF (formerly DLF 2.5)

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/<sessionHandle>/operations/statement/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -u '<UserName/RoleName>:<password>' \
       -d '{
       "statement": "select * from test;",
       "runAsync": true,
       "queryTimeout": 0,
       "confOverlay": {
         "additionalProp1": "string",
         "additionalProp2": "string"
       }
     }'

     Use other catalogs

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/<sessionHandle>/operations/statement/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -d '{
       "statement": "select * from test;",
       "runAsync": true,
       "queryTimeout": 0,
       "confOverlay": {
         "additionalProp1": "string",
         "additionalProp2": "string"
       }
     }'

     A message similar to the following is returned. In the message, identifier is the Kyuubi operation handle, which uniquely identifies a specific operation. In this topic, this value is referred to as <operationHandle>.

     {"identifier":"a743e8ff-xxxx-xxxx-xxxx-a66fec66cfa4"}

  3. Retrieve the statement status.

     Use DLF (formerly DLF 2.5)

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/event/token/<token>' \
       -H 'accept: application/json' \
       -u '<UserName/RoleName>:<password>'

     Use other catalogs

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/event/token/<token>' \
       -H 'accept: application/json'

  4. Retrieve the statement result.

     Use DLF (formerly DLF 2.5)

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/rowset/token/<token>/?maxrows=100&fetchorientation=FETCH_NEXT' \
       -H 'accept: application/json'  \
       -u '<UserName/RoleName>:<password>'

     Use other catalogs

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/rowset/token/<token>/?maxrows=100&fetchorientation=FETCH_NEXT' \
       -H 'accept: application/json'

• Example 2: Use the batches API to submit a batch job.

  Submit a Spark batch processing job to the Kyuubi Gateway using the REST API. The Kyuubi Gateway starts a Spark application and executes the specified task based on the parameters in the request.

  In this example, in addition to replacing information such as <endpoint>, <port>, and <token>, you must also click spark-examples_2.12-3.3.1.jar to download the test JAR package.

  Note

  This JAR package is a simple example that is provided with Spark and is used to calculate the value of Pi (π).

  Use DLF (formerly DLF 2.5)

  curl --location \
    --request POST 'http://<endpoint>:<port>/api/v1/batches/token/<token>' \
    --user '<UserName/RoleName>:<password>' \
    --form 'batchRequest="{
      \"batchType\": \"SPARK\",
      \"className\": \"org.apache.spark.examples.SparkPi\",
      \"name\": \"kyuubi-spark-pi\",
      \"resource\": \"oss://bucket/path/to/spark-examples_2.12-3.3.1.jar\"
    }";type=application/json'

  Use other catalogs

  curl --location \
    --request POST 'http://<endpoint>:<port>/api/v1/batches/token/<token>' \
    --form 'batchRequest="{
      \"batchType\": \"SPARK\",
      \"className\": \"org.apache.spark.examples.SparkPi\",
      \"name\": \"kyuubi-spark-pi\",
      \"resource\": \"oss://bucket/path/to/spark-examples_2.12-3.3.1.jar\"
    }";type=application/json'

Connect using Apache Superset

Apache Superset is a modern data exploration and visualization platform with a rich variety of charts. For more information about Superset, see the Superset documentation.

pip install thrift==0.20.0

1. Start Superset and go to the Superset interface. For more information about how to start Superset, see the Superset documentation.

2. In the upper-right corner of the page, click DATABASE to go to the Connect a database page.

3. On the Connect a database page, select Apache Spark SQL.

4. In the SQLAlchemy URI field, enter the connection string that corresponds to the access method.

   Public network access

   hive+https://<username>:<token>@<endpoint>:443/<db_name>

   VPC network access

   hive+http://<username>:<token>@<endpoint>:80/<db_name>

   The following table describes the parameters.

   Parameter	Description
   username	The username. You can enter any value.
   Token	The access token.
   Endpoint	The access endpoint of the Kyuubi Gateway. You must distinguish between the public endpoint and the internal endpoint.
   db_name	The database name.

5. Click Connect to complete the connection configuration.

Connect using HUE

pip install thrift==0.20.0

1. Edit the HUE configuration file /etc/hue/hue.conf and add the configuration that corresponds to your access method.

   Public network access

   [[[sparksql]]]
        name = Spark Sql
        interface=sqlalchemy
        options='{"url": "hive+https://<username>:<token>@<endpoint>:443/"}'

   VPC network access

   [[[sparksql]]]
        name = Spark Sql
        interface=sqlalchemy
        options='{"url": "hive+http://<username>:<token>@<endpoint>:80/"}'

   The parameters are described in the following table.

   Parameter	Description
   username	The username. You can enter any value.
   Token	The access token.
   Endpoint	The access endpoint of the Kyuubi Gateway.

2. Restart the HUE service.

   sudo service hue restart

3. Log on to the HUE web UI. In the navigation pane on the left, choose Query > Editor > SparkSql to verify the connection.

Connect using DataGrip

1. Open DataGrip and create a new project.

2. In the Database Explorer panel on the right, click the + icon and choose Data Source > Other > Apache Spark.

3. In the Data Sources and Drivers dialog box, configure the following parameters.

   Parameter	Description
   Name	A custom connection name.
   Authentication	Select No auth or User & Password. You can enter any username.
   Driver	Select Apache Spark ver. 1.2.2.
   URL	Public network access jdbc:hive2://<endpoint>:443/;transportMode=http;httpPath=cliservice/token/<token> VPC network access jdbc:hive2://<endpoint>:80/;transportMode=http;httpPath=cliservice/token/<token> Where <endpoint> is the access endpoint of the Kyuubi Gateway, and <token> is the access token.

4. Click Test Connection to verify the connection.

5. Click OK to save the configuration.