E-MapReduce:Manage Kyuubi engines

Supported engines

Installing Kyuubi in an E-MapReduce cluster also installs YARN and Spark 3 as dependencies. Kyuubi on E-MapReduce supports only Spark 3 on YARN. Flink, Trino, and Spark 2 are not supported. Each Spark engine corresponds to one Spark application on YARN. For more information, see Examples.

Share levels

Examples

The following examples use the USER share level (kyuubi.engine.share.level set to USER) and assume all users are authenticated through Lightweight Directory Access Protocol (LDAP) or Kerberos.

1. Launch an engine on demand

   When a new user, user1, submits a job using the kyuubi-beeline tool, the Kyuubi server launches a new Spark engine to process it.

   kyuubi-beeline -n user1 \
     -u "jdbc:hive2://master-1-1:10009/tpcds_parquet_1000" \
     -f query1.sql

   If another user, user2, needs to specify resources for the Spark engine, they can use one of the following methods:

   • Method 1: Set resources, such as Spark application executors, in the JDBC URL.

     # Set User config via JDBC Connection URL
     
     kyuubi-beeline -n user2 \
       -u "jdbc:hive2://master-1-1:10009/tpcds_parquet_1000?spark.dynamicAllocation.enabled=false;spark.executor.cores=2;spark.executor.memory=4g;spark.executor.instances=4" \
       -f query1.sql

   • Method 2: Set the resources for the Spark engine in the kyuubi-defaults.conf configuration file.

     # Set User default config in kyuubi-defatuls.conf
     # ___user2___.spark.dynamicAllocation.enabled=false
     # ___user2___.spark.executor.memory=5g
     # ___user2___.spark.executor.cores=2
     # ___user2___.spark.executor.instances=10
     
     kyuubi-beeline -n user2 \
       -u "jdbc:hive2://master-1-1:10009/tpcds_parquet_1000" \
       -f query1.sql

2. Submit jobs to a specific engine

   After a job completes, the Spark engine that Kyuubi launched remains active for a configurable period. New jobs can reuse this idle engine, improving performance by avoiding the need to launch a new YARN application. If no new jobs are submitted within this period, the engine automatically terminates. The kyuubi.session.engine.idle.timeout parameter specifies the idle timeout period, which defaults to PT30M (30 minutes). You can configure this parameter on the kyuubi-defaults.conf tab.

   Kyuubi lets you create subdomains within the same share level. For example, if a user, user4, needs different engine resources for different business scenarios, they can set the kyuubi.engine.share.level.subdomain parameter in the JDBC URL to submit jobs to different engines.

   kyuubi-beeline -n user4 \
     -u "jdbc:hive2://master-1-1:10009/biz1?kyuubi.engine.share.level.subdomain=biz1" \
     -f query1.sql
   
   kyuubi-beeline -n user4 \
     -u "jdbc:hive2://master-1-1:10009/biz2?kyuubi.engine.share.level.subdomain=biz2" \
     -f query2.sql
   
    kyuubi-beeline -n user4 \
     -u "jdbc:hive2://master-1-1:10009/biz3?kyuubi.engine.share.level.subdomain=biz3" \
     -f query3.sql

3. A single engine supports multiple Spark sessions

   Multiple Spark sessions can use a single Spark engine concurrently. For example, if a user, user1, submits two jobs from two different terminals at the same time, both jobs run on the same Spark engine. Spark allocates executor resources to jobs based on its internal default scheduling mechanism.

   # Console 1
   kyuubi-beeline -n user1 \
     -u "jdbc:hive2://master-1-1:10009/biz1" \
     -f query1.sql
   
   # Console 2
   kyuubi-beeline -n user1 \
     -u "jdbc:hive2://master-1-1:10009/biz2" \
     -f query2.sql

References

• The Share Level Of Kyuubi Engines

• The TTL Of Kyuubi Engines