All Products
Search

This Product

    DataWorks:Best practices for DataWorks on EMR cluster configuration

    Document Center

    DataWorks:Best practices for DataWorks on EMR cluster configuration

    Last Updated:Aug 20, 2026

    DataWorks allows you to register an EMR DataLake cluster as an EMR compute engine. You can then create nodes for services such as Hive, MapReduce, Presto, and Spark SQL. This enables you to configure workflows, schedule jobs, and manage metadata for your EMR tasks. This topic covers the best practices for configuring a DataLake cluster to run EMR tasks in DataWorks.

    Background

    • When running EMR tasks in DataWorks, you can use various EMR components, each with its own optimal configuration. For more information, see Configure EMR components.

    • The metadata storage options available for your EMR tasks depend on the mode of your DataWorks workspace. For more information, see Choose a metadata storage solution.

    Refer to the DataWorks on EMR user guide for the development process and key considerations when using DataLake clusters with EMR jobs in DataWorks.

    EMR component configuration

    • Kyuubi

      For a production environment, we recommend setting the memory for kyuubi_java_opts to 10g or higher and for kyuubi_beeline_opts to 2g or higher.

    • Spark

      • By default, the Spark component has a small memory allocation. You can add memory settings to the spark-submit command to adjust this allocation for your workload.

      • Based on the size of your EMR cluster, adjust the following Spark properties to appropriate values: spark.driver.memory, spark.driver.memoryOverhead, and spark.executor.memory.

      Important

      • In DataWorks, lineage is supported only for EMR Hive, EMR Spark, and EMR Spark SQL nodes. While EMR Hive nodes support both table-level and column-level lineage, Spark-based nodes support only table-level lineage.

      • For Spark, only Spark 2.x supports lineage. Spark 3.x does not.

      For more information about Spark configuration, see Spark Memory Management.

    • HDFS

      Based on the size of your EMR cluster, adjust the following HDFS parameters to appropriate values: hadoop_namenode_heapsize, hadoop_datanode_heapsize, hadoop_secondary_namenode_heapsize, and hadoop_namenode_opts.

    Metadata storage selection

    To isolate development from production environments in a standard mode DataWorks workspace, you must register two different EMR clusters in the SettingCenter > Data Sources section of DataWorks: one for the development environment and one for the production environment. Additionally, for data isolation, you must store the metadata for the two clusters in separate ApsaraDB RDS databases.