E-MapReduce:Manage Kyuubi Gateways

Manage tokens

1. On the Kyuubi Gateway page, find the target gateway and click Tokens in the Actions column.

2. Click Create Token.

3. In the Create Token dialog box, configure the parameters and click OK.

   Parameter	Description
   Name	The name of the new token.
   Expired At	Set the expiration time for the token. The number of days must be 1 or greater. By default, this is enabled and the token expires after 365 days.
   Assigned To	Note If you use DLF (formerly DLF 2.5) by default on the Catalog tab, you must configure this parameter. Make sure the configured Resource Access Management (RAM) user or RAM role has the permissions to access DLF. For more information about how to grant permissions, see Add an authorization. From the drop-down list, select the RAM user or RAM role that you added in Access control. Specify the RAM user or RAM role to which the token is assigned. This is used to access DLF when you connect to the Kyuubi Gateway to submit a Spark job.

4. Copy the token information.

   Important

   You must immediately copy the token information after the token is created because you cannot retrieve it later. If your token expires or is lost, you must create a new token or reset the token.

When you connect to a Kyuubi Gateway, replace the placeholders in the JDBC URL:

• <endpoint>: The endpoint information that you can obtain from the Overview tab.

• <port>: The port number. The port number is 443 for public endpoints and 80 for internal same-region endpoints.

• <token>: The token information that you copied from the Token Management page.

• <tokenname>: The token name. You can obtain it from the Token Management page.

• <UserName/RoleName>: The RAM user or RAM role that you added to Access control.

Connect using Beeline

Before you connect to a Kyuubi Gateway, make sure that your Beeline version is compatible with the Kyuubi server version. If you do not have Beeline installed, see Getting Started - Apache Kyuubi.

Select one of the following methods based on the default catalog that is configured on the Catalog page.

beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;user=<UserName/RoleName>;httpPath=cliservice/token/<token>"

beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>"

When you connect using Beeline, you can modify session parameters. For example: beeline -u "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>;#spark.sql.shuffle.partitions=100;spark.executor.instances=2;".

Connect using Java

• Update the pom.xml file.

  Replace hadoop-common and hive-jdbc with the appropriate dependency versions.

  <dependencies>
          <dependency>
              <groupId>org.apache.hadoop</groupId>
              <artifactId>hadoop-common</artifactId>
              <version>3.0.0</version>
          </dependency>
          <dependency>
              <groupId>org.apache.hive</groupId>
              <artifactId>hive-jdbc</artifactId>
              <version>2.3.9</version>
          </dependency>
      </dependencies>

• Write Java code to connect to the Kyuubi Gateway.

  Select one of the following methods based on the default catalog that is configured on the Catalog page.

  Use DLF (formerly DLF 2.5)

  import org.apache.hive.jdbc.HiveStatement;
  import java.sql.Connection;
  import java.sql.DriverManager;
  import java.sql.ResultSet;
  import java.sql.ResultSetMetaData;
  
  public class Main {
      public static void main(String[] args) throws Exception {
          String url = "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>;user=<UserName/RoleName>";
          Class.forName("org.apache.hive.jdbc.HiveDriver");
          Connection conn = DriverManager.getConnection(url);
          HiveStatement stmt = (HiveStatement) conn.createStatement();
  
  
          String sql = "select * from students;";
          System.out.println("Running " + sql);
          ResultSet res = stmt.executeQuery(sql);
  
          ResultSetMetaData md = res.getMetaData();
          String[] columns = new String[md.getColumnCount()];
          for (int i = 0; i < columns.length; i++) {
              columns[i] = md.getColumnName(i + 1);
          }
          while (res.next()) {
              System.out.print("Row " + res.getRow() + "=[");
              for (int i = 0; i < columns.length; i++) {
                  if (i != 0) {
                      System.out.print(", ");
                  }
                  System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
              }
              System.out.println(")]");
          }
  
          conn.close();
      }
  }

  Use another catalog

  import org.apache.hive.jdbc.HiveStatement;
  import java.sql.Connection;
  import java.sql.DriverManager;
  import java.sql.ResultSet;
  import java.sql.ResultSetMetaData;
  
  public class Main {
      public static void main(String[] args) throws Exception {
          String url = "jdbc:hive2://<endpoint>:<port>/;transportMode=http;httpPath=cliservice/token/<token>";
          Class.forName("org.apache.hive.jdbc.HiveDriver");
          Connection conn = DriverManager.getConnection(url);
          HiveStatement stmt = (HiveStatement) conn.createStatement();
  
  
          String sql = "select * from students;";
          System.out.println("Running " + sql);
          ResultSet res = stmt.executeQuery(sql);
  
          ResultSetMetaData md = res.getMetaData();
          String[] columns = new String[md.getColumnCount()];
          for (int i = 0; i < columns.length; i++) {
              columns[i] = md.getColumnName(i + 1);
          }
          while (res.next()) {
              System.out.print("Row " + res.getRow() + "=[");
              for (int i = 0; i < columns.length; i++) {
                  if (i != 0) {
                      System.out.print(", ");
                  }
                  System.out.print(columns[i] + "='" + res.getObject(i + 1) + "'");
              }
              System.out.println(")]");
          }
  
          conn.close();
      }
  }

Connect using Python

1. Run the following command to install the PyHive and Thrift packages.

   pip3 install pyhive thrift

2. Write a Python script to connect to the Kyuubi Gateway.

   The following Python script is an example of how to connect to a Kyuubi Gateway and display a list of databases.

   Select one of the following methods based on the default catalog that is configured on the Catalog page.

   Use DLF (formerly DLF 2.5)

   from pyhive import hive
   
   if __name__ == '__main__':
       cursor = hive.connect('<endpoint>',
                             port="<port>",
                             scheme='http',
                             username='<UserName/RoleName>',
                             password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

   Use another catalog

   from pyhive import hive
   
   if __name__ == '__main__':
       cursor = hive.connect('<endpoint>',
                             port="<port>",
                             scheme='http',
                             username='<tokenname>',
                             password='<token>').cursor()
       cursor.execute('show databases')
       print(cursor.fetchall())
       cursor.close()

Connect using the REST API

Kyuubi Gateway provides open source-compatible Representational State Transfer (REST) APIs that support interaction with the Kyuubi service over HTTP. Currently, only the following API paths are supported:

• /api/v1/sessions/*

• /api/v1/operations/*

• /api/v1/batches/*

The following examples show how to connect to a Kyuubi Gateway using the REST API.

• Example 1: Start a session and run an SQL query.

  1. Create a session and specify Spark configurations.

     Select one of the following methods based on the default catalog that is configured on the Catalog page.

     Note

     • spark.emr.serverless.kyuubi.engine.queue specifies the queue that the Spark job uses at runtime. Replace <dev_queue> with the actual queue name.

     • <UserName/Rolename>: Replace this with the actual username or role name.

     • <password>: This is a placeholder. You can enter any value.

     Use DLF (formerly DLF 2.5)

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -u '<UserName/Rolename>:<password>' \
       -d '{
       "configs": {
         "set:hivevar:spark.emr.serverless.kyuubi.engine.queue": "<dev_queue>"
       }
     }'

     Use another catalog

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -d '{
       "configs": {
         "set:hivevar:spark.emr.serverless.kyuubi.engine.queue": "<dev_queue>"
       }
     }'

     A message similar to the following is returned. In the message, identifier indicates the Kyuubi session handle, which uniquely identifies a session. In this topic, this value is referred to as <sessionHandle>.

     {"identifier":"619e6ded-xxxx-xxxx-xxxx-c2a43f6fac46","kyuubiInstance":"0.0.0.0:10099"}

  2. Create a statement.

     Use DLF (formerly DLF 2.5)

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/<sessionHandle>/operations/statement/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -u '<UserName/RoleName>:<password>' \
       -d '{
       "statement": "select * from test;",
       "runAsync": true,
       "queryTimeout": 0,
       "confOverlay": {
         "additionalProp1": "string",
         "additionalProp2": "string"
       }
     }'

     Use another catalog

     curl -X 'POST' \
       'http://<endpoint>:<port>/api/v1/sessions/<sessionHandle>/operations/statement/token/<token>' \
       -H 'accept: application/json' \
       -H 'Content-Type: application/json' \
       -d '{
       "statement": "select * from test;",
       "runAsync": true,
       "queryTimeout": 0,
       "confOverlay": {
         "additionalProp1": "string",
         "additionalProp2": "string"
       }
     }'

     A message similar to the following is returned. Here, identifier indicates the Kyuubi operation handle, which uniquely identifies a specific operation. In this topic, this value is referred to as <operationHandle>.

     {"identifier":"a743e8ff-xxxx-xxxx-xxxx-a66fec66cfa4"}

  3. Retrieve the statement status.

     Use DLF (formerly DLF 2.5)

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/event/token/<token>' \
       -H 'accept: application/json' \
       -u '<UserName/RoleName>:<password>'

     Use another catalog

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/event/token/<token>' \
       -H 'accept: application/json'

  4. Retrieve the statement result.

     Use DLF (formerly DLF 2.5)

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/rowset/token/<token>/?maxrows=100&fetchorientation=FETCH_NEXT' \
       -H 'accept: application/json'  \
       -u '<UserName/RoleName>:<password>'

     Use another catalog

     curl --location -X 'GET' \
       'http://<endpoint>:<port>/api/v1/operations/<operationHandle>/rowset/token/<token>/?maxrows=100&fetchorientation=FETCH_NEXT' \
       -H 'accept: application/json'

• Example 2: Use the batches API to submit a batch job.

  You can submit a Spark batch processing job to the Kyuubi Gateway using the REST API. The Kyuubi Gateway starts a Spark application and runs the specified task based on the parameters in the request.

  In this example, in addition to replacing information such as <endpoint>, <port>, and <token>, you must also download the test JAR package by clicking spark-examples_2.12-3.3.1.jar.

  Note

  This JAR package is a simple example that comes with Spark. It is used to calculate the value of Pi (π).

  Use DLF (formerly DLF 2.5)

  curl --location \
    --request POST 'http://<endpoint>:<port>/api/v1/batches/token/<token>' \
    --user '<UserName/RoleName>:<password>' \
    --form 'batchRequest="{
      \"batchType\": \"SPARK\",
      \"className\": \"org.apache.spark.examples.SparkPi\",
      \"name\": \"kyuubi-spark-pi\",
      \"resource\": \"oss://bucket/path/to/spark-examples_2.12-3.3.1.jar\"
    }";type=application/json'

  Use another catalog

  curl --location \
    --request POST 'http://<endpoint>:<port>/api/v1/batches/token/<token>' \
    --form 'batchRequest="{
      \"batchType\": \"SPARK\",
      \"className\": \"org.apache.spark.examples.SparkPi\",
      \"name\": \"kyuubi-spark-pi\",
      \"resource\": \"oss://bucket/path/to/spark-examples_2.12-3.3.1.jar\"
    }";type=application/json'