Enable native query acceleration - E-MapReduce - Alibaba Cloud Documentation Center

JindoTable uses a native engine to accelerate queries on ORC or Parquet files in Spark, Hive, or Presto. This topic explains how to enable native query acceleration to improve the performance of Spark, Hive, and Presto.

Prerequisites

You must have an E-MapReduce cluster with your ORC or Parquet files stored in JindoFS or OSS. For more information, see Create a cluster.

Limitations

Binary type files cannot be accelerated.
Partitioned tables are not supported if the values of their partition key columns are stored in the data files.
E-MapReduce clusters of version EMR-5.X and later are not supported.
Using spark.read.schema(userDefinedSchema) is not supported.
Date types are supported only within the range of 1400-01-01 to 9999-12-31.
Tables that contain column names differing only in case (for example, NAME and name) are not supported. Column names are treated case-insensitively during native query acceleration.

The supported engines and storage formats for Spark, Hive, and Presto are as follows.

Engine	ORC	Parquet
Spark2	Supported	Supported
Spark3	Supported	Supported
Presto	Supported	Supported
Hive2	Not supported	Supported
Hive3	Not supported	Supported

The supported engines and storage file systems for Spark, Hive, and Presto are as follows.

Engine	OSS	JindoFS	HDFS
Spark2	Supported	Supported	Supported
Presto	Supported	Supported	Supported
Hive2	Supported	Supported	Not supported
Hive3	Supported	Supported	Not supported

Improve Spark performance

Enable JindoTable ORC or Parquet acceleration.
Note
- Native query acceleration uses off-heap memory. We recommend adding the --conf spark.executor.memoryOverhead=4g configuration to your Spark job to allocate additional resources for acceleration.
- To read ORC or Parquet files in Spark, you must use the DataFrame API or Spark SQL.
- Global settings
  1. Go to the cluster details page.
    1. Log on to the E-MapReduce console.
    2. In the top navigation bar, select a region and resource group as needed.
    3. Click the Clusters tab.
    4. On the Clusters page, find the cluster and click Details in the Actions column.
  2. Modify the configuration.
    1. In the left-side navigation pane, choose Services > Spark.
    2. On the Spark service page, click the Configure tab.
    3. In the search box, locate the spark.sql.extensions parameter and set its value to io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension.
  3. Save the configuration.
    1. Click Save.
    2. In the Confirm dialog box, enter a Execution Reason and click OK.
  4. Restart the ThriftServer.
    1. In the upper-right corner, choose Actions > Restart ThriftServer.
    2. In the Cluster Activities dialog box, enter a Execution Reason and click OK.
    3. In the Confirm dialog box, click OK.
- Job-level settings
  When you run spark-shell or spark-sql, add the following Spark startup parameter:
```
--conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension
```
  For more information about jobs, see Configure a Spark Shell job or Configure a Spark SQL job.
Verify that acceleration is enabled.
1. Log on to the Spark History Server UI.
2. On the SQL page of the Spark UI, view the execution job.
  
  If JindoDataSourceV2Scan appears, acceleration is enabled. If not, check the configurations in Step 1. On the Details for Query 10 page, the Directed Acyclic Graph (DAG) shows the execution plan. For example, JindoDataSourceV2Scan can scan 15.6 GB of data (about 14 billion rows), which is then filtered to 2.7 billion rows. A final HashAggregate operation outputs 4,722 rows, and the total query takes 2.3 minutes.

Improve Presto performance

Important

Presto queries have high concurrency, and native query acceleration uses off-heap memory. Therefore, you must allocate more than 10 GB of memory for query acceleration.

Presto has a built-in catalog for JindoTable native acceleration named hive-acc. You can use the hive-acc catalog to enable native query acceleration.

Example:

presto --server emr-header-1:9090 --catalog hive-acc --schema default

Note

Native query acceleration for Presto does not currently support reading complex data types such as Map, Struct, or Array.

Improve Hive performance

Important

If strict job stability is required, we recommend not enabling native query acceleration.

You can improve Hive performance in one of the following ways:

Use the console

In the console, go to the Configure page of the Hive service. Locate the hive.jindotable.native.enabled custom parameter and set its value to true. Save the configuration and restart the service to apply the change. This method applies to both Hive on MR and Hive on Tez.
Use the CLI
In the CLI, set hive.jindotable.native.enabled to true to enable native query acceleration. Because E-MapReduce V3.35.0 and later include the JindoTable Parquet acceleration plug-in, you can set this parameter directly.
```
set hive.jindotable.native.enabled=true;
```

Note

Native query acceleration for Hive does not currently support reading complex data types such as Map, Struct, or Array.