E-MapReduce:Enable query acceleration based on a native engine

Limits

• Binary files are not supported.

• Partitioned tables whose values of partition key columns are stored in files are not supported.

• EMR clusters of V5.X.X or a later version are not supported.

• spark.read.schema (userDefinedSchema) is not supported.

• Data of the DATE type must be in the YYYY-MM-DD format and range from 1400-01-01 to 9999-12-31.

• If a table contains two columns that have the same column name with different letter cases, such as NAME and name, queries on the columns cannot be accelerated.

• The following table lists the supported Spark, Hive, and Presto engines and the file formats supported by each engine.

  Engine	ORC	Parquet
  Spark2	Supported	Supported
  Spark3	Supported	Supported
  Presto	Supported	Supported
  Hive2	Not supported	Supported
  Hive3	Not supported	Supported

• The following table lists the supported Spark, Hive, and Presto engines and the file systems supported by each engine.

  Engine	OSS	JindoFS	HDFS
  Spark2	Supported	Supported	Supported
  Presto	Supported	Supported	Supported
  Hive2	Supported	Supported	Not supported
  Hive3	Supported	Supported	Not supported

Improve the performance of Spark

1. Enable query acceleration for ORC or Parquet files in JindoTable.

   Note

   • Query acceleration consumes off-heap memory. We recommend that you add --conf spark.executor.memoryOverhead=4g to a Spark task to apply for additional resources for query acceleration.

   • If you use Spark to read data from an ORC or Parquet file, the DataFrame API or Spark SQL is required.

   • Global configuration

     1. Go to the Cluster Overview page of your cluster.

        1. Log on to the Alibaba Cloud EMR console.

        2. In the top navigation bar, select the region where you want to create a cluster. The region of a cluster cannot be changed after the cluster is created.

        3. Click the Cluster Management tab.

        4. On the Cluster Management page, find your cluster and click Details in the Actions column.

     2. Modify the related parameter.

        1. In the left-side navigation pane of the page that appears, choose Cluster Service > Spark.

        2. On the Spark service page, click the Configure tab.

        3. Search for the spark.sql.extensions parameter and change its value to io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension.

     3. Save the configuration.

        1. Click Save in the upper-right corner.

        2. In the Confirm Changes dialog box, specify Description and click OK.

     4. Restart ThriftServer.

        1. Choose Actions > Restart ThriftServer in the upper-right corner.

        2. In the Cluster Activities dialog box, specify Description and click OK.

        3. In the Confirm message, click OK.

   • Job-level configuration

     When you configure a Spark Shell or Spark SQL job, add the following configuration to the code:

     --conf spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension,com.aliyun.emr.sql.JindoTableExtension

     For more information about how to configure a Spark job, see Configure a Spark Shell job or Configure a Spark SQL job.

2. Check whether query acceleration is enabled.

   1. Access the web UI of Spark History Server.

   2. On the SQL tab of Spark, view information about the related task.

      If JindoDataSourceV2Scan appears, query acceleration is enabled. Otherwise, check the configurations in Step 1.

Improve the performance of Presto

presto --server emr-header-1:9090 --catalog hive-acc --schema default

Improve the performance of Hive