E-MapReduce:Enable native query acceleration

Limitations

• The Binary data type is not supported.

• Partitioned tables with partition key columns stored in files are not supported.

• E-MapReduce clusters of version 5.X or later are not supported.

• The spark.read.schema(userDefinedSchema) method is not supported because a user-defined schema can be inconsistent with the file schema.

• The supported date range for the Date data type is from 1400-01-01 to 9999-12-31.

• Query acceleration is not supported for tables that contain columns with case-insensitive duplicate names.

Improve Spark performance

1. Enable JindoTable query acceleration for ORC or Parquet files.

   Note

   • Add the --conf spark.executor.memoryOverhead=4g configuration to your Spark job to request additional resources for acceleration.

   • To enable acceleration, Spark must use the native engine when reading ORC or Parquet files.

   • Global settings

     For more information, see Configure global settings for Spark.

   • Job-level settings

     You can add the following Spark startup parameter when you use Spark Shell or Spark SQL.

     spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension

     For more information about job configuration, see Configure a Spark Shell job or Configure a Spark SQL job.

2. Verify that acceleration is enabled.

   1. Open the Spark History Server UI.

   2. On the SQL tab of the Spark History Server UI, view the running job.

      If JindoDataSourceV2Scan appears, the feature is enabled successfully. Otherwise, check the operations in Step 1. In Details for Query 10 on the Spark SQL page, the query takes 2.3 min. The DAG execution plan shows that the JindoDataSourceV2Scan node scanned 15.6 GB of data from 4,722 partitions, read approximately 14 billion rows (read output rows 14,076,201,649), and had approximately 2.7 billion rows remaining after filtering (filtered output rows 2,718,064,652). The data then flows from the Project node to the HashAggregate node, which outputs a final 4,722 aggregated rows.

Improve Presto performance

Presto includes the built-in hive-acc catalog for JindoTable native acceleration. You can use the hive-acc catalog to enable query acceleration directly.

presto --server emr-header-1:9090 --catalog hive-acc --schema default

Improve Hive performance

EMR Hive 2.3.7 (in EMR-3.35.0) includes a built-in JindoTable plugin for Parquet acceleration. You can enable query acceleration by setting the hive.jindotable.native.enabled parameter. Alternatively, go to the hive-site.xml tab on the configuration page of the console, add the hive.jindotable.native.enabled custom parameter, and set it to true. Then, restart Hive. This method works for both Hive on MapReduce and Hive on Tez.

set hive.jindotable.native.enabled=true;

Configure Spark globally

1. Go to the Spark service page

   1. Log on to the Alibaba Cloud EMR console.

   2. In the top navigation bar, select the appropriate region and resource group.

   3. Click the Clusters tab.

   4. On the Clusters page, find the target cluster and click Details in the Actions column.

   5. In the left-side navigation pane, choose Services > Spark.

2. Click the Configure tab

3. Search for the spark.sql.extensions parameter and set its value to io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension

4. Save the configuration

   1. Click Save.

   2. In the Confirm dialog box, enter an Execution Reason and click OK.

5. Restart ThriftServer

   1. In the upper-right corner, choose Actions > Restart ThriftServer.

   2. In the Execute Cluster Operation dialog box, enter an Execution Reason and click OK.

   3. In the Confirm dialog box, click OK.