Configure HDFS, YARN, and Spark Settings at Cluster Level - E-MapReduce

E-MapReduce (EMR) lets you view, modify, and add configuration items for services such as Hadoop Distributed File System (HDFS), YARN, and Spark directly in the EMR console.

Prerequisites

Before you begin, ensure that you have:

An EMR cluster. For more information, see Create a cluster.

How it works

EMR manages configuration items at three levels: cluster, node group, and node. When the same configuration item is set at multiple levels, the value at the highest-priority level takes effect. Priority order: node > node group > cluster.

By default, the Configure tab shows cluster-level configuration items. To view or modify node group or node level items, switch the scope using the Default Cluster Configuration drop-down list:

Node Group Configuration — applies the setting to all nodes in a node group
Independent Node Configuration — applies the setting to a specific node

Node group and node level items can only be viewed when Default Cluster Configuration is selected. To modify them, switch to the appropriate scope first.

View configuration items

Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select a region and a resource group.
Find the cluster and click Services in the Actions column.
On the Services tab, find the service and click Configure.
In the search box, enter the name of the configuration item.

If a configuration item at the node group or node level is modified, or the settings of a configuration item are inconsistent with the default settings at the cluster level, the settings of the configuration item at the node group or node level are displayed on the Configure tab, with Default Cluster Configuration selected.

To view a configuration item at the node group or node level, select Node Group Configuration or Independent Node Configuration from the Default Cluster Configuration drop-down list, then select the specific node group or node.

Modify configuration items

Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select a region and a resource group.
Find the cluster and click Services in the Actions column.
On the Services tab, find the service and click Configure.
(Optional) To modify a node group or node level item, select Node Group Configuration or Independent Node Configuration from the Default Cluster Configuration drop-down list.
In the search box, enter the name of the configuration item and click the search icon.
Change the value of the configuration item. > Tip: For example, to increase NodeManager memory, set yarn.nodemanager.resource.memory-mb to a higher value such as 8192.
Click Save. In the Save dialog box, set Execution Reason and click Save.
The Save and Deliver Configuration switch is turned on by default. When enabled, configurations are sent to the client immediately after saving, and you can activate them using prompt mode. Turn off this switch to activate configurations manually using manual mode instead.
Activate the configurations. See Activate configurations.

Add configuration items

Log on to the EMR console. In the left-side navigation pane, click EMR on ECS.
In the top navigation bar, select a region and a resource group.
Find the cluster and click Services in the Actions column.
On the Services tab, find the service and click Configure.
Click the tab where you want to add configuration items, then click Add Configuration Item.
Fill in the configuration details. You can add multiple items at the same time.
Field Description
Key The name of the configuration item
Value The value of the configuration item
Description A description of the configuration item
Actions Remove the configuration item
Click OK. In the dialog box, set Execution Reason and click Save.
Activate the configurations. See Activate configurations.

Field	Description
Key	The name of the configuration item
Value	The value of the configuration item
Description	A description of the configuration item
Actions	Remove the configuration item

Activate configurations

After saving, activate the configurations based on their type and your preferred mode.

Prompt mode

Prompt mode is available only for EMR V5.12.1, EMR V3.46.1, and later minor versions.

Client-side configurations

After saving, click the (To Be Delivered) prompt.
In the Configurations to Be Delivered message, click Deliver.
For the YARN service, if the modified items include queue-related configuration items, click the (Not Effective Yet) prompt or click Deploy on the Edit Resource Queue tab of the YARN service page to activate the changes.

Server-side configurations

After saving, click the (Not Effective Yet) prompt.
In the Configurations to Take Effect dialog box, activate each component based on its type:
- Configurations That Require Custom Operations: Click the corresponding action in the Actions column for each component.
- Configurations That Require Restart: Click restart in the Actions column for a component, or select multiple components and click Batch Restart. In the dialog box, set Execution Reason and click OK.

Manual mode

Client-side configurations

Click Deploy Client Configuration next to Save at the bottom of the page.
In the dialog box, set Execution Reason and click OK.
In the Confirm message, click OK.

Server-side configurations

Choose More > Restart in the upper-right corner of the Configure tab.
In the dialog box, set Execution Reason and click OK.
In the Confirm message, click OK.

Modifiable configuration items

The following table lists the configuration items that can be modified at the node and node group levels in EMR V5.17.1 clusters.

Kerberos-related configuration items are available only if Kerberos authentication is enabled.

<table> <thead> <tr> <td>Service name</td> <td>File</td> <td>Configuration item</td> </tr> </thead> <colgroup></colgroup> <colgroup></colgroup> <colgroup></colgroup> <tbody> <tr> <td>Hadoop-Common</td> <td>core-site.xml</td> <td>fs.oss.tmp.data.dirshadoop.tmp.dir</td> </tr> <tr> <td>HDFS</td> <td>hdfs-env.sh</td> <td>hadoop_datanode_heapsizehadoop_secondarynamenode_optshadoop_namenode_heapsize</td> </tr> <tr> <td>hdfs-site.xml</td> <td>dfs.datanode.data.dirdfs.datanode.failed.volumes.tolerateddfs.datanode.du.reserveddfs.datanode.balance.max.concurrent.moves</td> </tr> <tr> <td>OSS-HDFS</td> <td>None</td> <td>None</td> </tr> <tr> <td>Hive</td> <td>hive-env.sh</td> <td>hive_metastore_heapsizehive_server2_heapsize</td> </tr> <tr> <td>Spark2</td> <td>hiveserver2-site.xml</td> <td>hive.server2.authentication.kerberos.principal</td> </tr> <tr> <td>spark-env.sh</td> <td>spark_history_daemon_memoryspark_thrift_daemon_memory</td> </tr> <tr> <td>spark-thriftserver.conf</td> <td>spark.yarn.historyServer.addressspark.hadoop.hive.server2.thrift.bind.hostspark.yarn.principal</td> </tr> <tr> <td>spark-defaults.conf</td> <td>spark.yarn.historyServer.addressspark.history.kerberos.principal</td> </tr> <tr> <td>Spark3</td> <td>hiveserver2-site.xml</td> <td>hive.server2.authentication.kerberos.principal</td> </tr> <tr> <td>spark-env.sh</td> <td>spark_history_daemon_memoryspark_thrift_daemon_memory</td> </tr> <tr> <td>spark-thriftserver.conf</td> <td>spark.yarn.historyServer.addressspark.hadoop.hive.server2.thrift.bind.hostspark.kerberos.principal</td> </tr> <tr> <td>spark-defaults.conf</td> <td>spark.yarn.historyServer.addressspark.history.kerberos.principal</td> </tr> <tr> <td>Tez</td> <td>None</td> <td>None</td> </tr> <tr> <td>Trino</td> <td>iceberg.properties</td> <td>hive.hdfs.trino.principalhive.metastore.client.principal</td> </tr> <tr> <td>delta.properties</td> <td>hive.hdfs.trino.principalhive.metastore.client.principal</td> </tr> <tr> <td>config.properties</td> <td>coordinatornode-scheduler.include-coordinatorquery.max-memoryquery.max-total-memoryquery.max-memory-per-nodehttp-server.authentication.typehttp-server.authentication.krb5.user-mapping.patternhttp-server.authentication.krb5.service-namehttp-server.authentication.krb5.keytabhttp.authentication.krb5.confighttp-server.https.enabledhttp-server.https.porthttp-server.https.keystore.keyhttp-server.https.keystore.pathevent-listener.config-files <div> <div> </div> <div> Note event-listener.config-files specifies the path where the configuration file of an event listener is stored. This configuration item is available only if you turn on EmrEventListener. </div> </div></td> </tr> <tr> <td>jvm.config</td> <td>jvm parameter</td> </tr> <tr> <td>hudi.properties</td> <td>hive.hdfs.trino.principalhive.metastore.client.principal</td> </tr> <tr> <td>password-authenticator.properties</td> <td>ldap.urlldap.user-bind-pattern</td> </tr> <tr> <td>hive.properties</td> <td>hive.hdfs.trino.principalhive.metastore.client.principal</td> </tr> <tr> <td>DeltaLake</td> <td>None</td> <td>None</td> </tr> <tr> <td>Hudi</td> <td>None</td> <td>None</td> </tr> <tr> <td>Iceberg</td> <td>None</td> <td>None</td> </tr> <tr> <td>JindoData</td> <td>storage.yaml</td> <td>jindofsx.storage.cache-modestorage.watermark.high.ratiostorage.watermark.low.ratiostorage.handler.threads <div> <div> </div> <div> Note <ul> <li>JindoData applies to clusters of EMR V5.14.0 or a later minor version and clusters of EMR V3.48.0 or a later minor version. </li> <li>JindoData is unavailable for clusters of EMR V5.15.0 or a later minor version and clusters of EMR V3.49.0 or a later minor version. You can use <a href="https://www.alibabacloud.com/help/en/document_detail/2579701.html">JindoCache</a> for data caching and <a href="https://www.alibabacloud.com/help/en/document_detail/455181.html">DLF-Auth</a> for authentication. </li> </ul> </div> </div></td> </tr> <tr> <td>Flume</td> <td>flume-conf.properties</td> <td>agent_nameflume-conf.properties</td> </tr> <tr> <td>Kyuubi</td> <td>kyuubi-env.sh</td> <td>kyuubi_java_opts</td> </tr> <tr> <td>YARN</td> <td>yarn-site.xml</td> <td>yarn.nodemanager.resource.memory-mbyarn.nodemanager.local-dirsyarn.nodemanager.log-dirsyarn.nodemanager.resource.cpu-vcoresyarn.nodemanager.addressyarn.nodemanager.node-labels.provider.configured-node-partition</td> </tr> <tr> <td>yarn-env.sh</td> <td>YARN_RESOURCEMANAGER_HEAPSIZEYARN_TIMELINESERVER_HEAPSIZEYARN_PROXYSERVER_HEAPSIZEYARN_NODEMANAGER_HEAPSIZEYARN_RESOURCEMANAGER_HEAPSIZE_MINYARN_TIMELINESERVER_HEAPSIZE_MINYARN_PROXYSERVER_HEAPSIZE_MINYARN_NODEMANAGER_HEAPSIZE_MIN</td> </tr> <tr> <td>mapred-env.sh</td> <td>HADOOP_JOB_HISTORYSERVER_HEAPSIZE</td> </tr> <tr> <td>mapred-site.xml</td> <td>mapreduce.cluster.local.dir</td> </tr> <tr> <td>Impala</td> <td>None</td> <td>None</td> </tr> <tr> <td>OpenLDAP</td> <td>None</td> <td>None</td> </tr> <tr> <td>Ranger</td> <td>None</td> <td>None</td> </tr> <tr> <td>Ranger-Plugin</td> <td>None</td> <td>None</td> </tr> <tr> <td>DLF-Auth</td> <td>None</td> <td>None</td> </tr> <tr> <td>Presto</td> <td>iceberg.properties</td> <td>hive.hdfs.presto.principalhive.metastore.client.principal</td> </tr> <tr> <td>delta.properties</td> <td>hive.hdfs.presto.principalhive.metastore.client.principal</td> </tr> <tr> <td>hive.properties</td> <td>hive.hdfs.presto.principalhive.metastore.client.principal</td> </tr> <tr> <td>config.properties</td> <td>coordinatornode-scheduler.include-coordinatorquery.max-memory-per-nodequery.max-total-memory-per-nodehttp-server.authentication.typehttp.authentication.krb5.principal-hostnamehttp.server.authentication.krb5.service-namehttp.server.authentication.krb5.keytabhttp.authentication.krb5.confighttp-server.https.enabledhttp-server.https.porthttp-server.https.keystore.keyhttp-server.https.keystore.path</td> </tr> <tr> <td>jvm.config</td> <td>jvm parameter</td> </tr> <tr> <td>hudi.properties</td> <td>hive.hdfs.presto.principalhive.metastore.client.principal</td> </tr> <tr> <td>password-authenticator.properties</td> <td>ldap.urlldap.user-bind-pattern</td> </tr> <tr> <td>Starrocks2</td> <td>fe.conf</td> <td>JAVA_OPTSmeta_dir</td> </tr> <tr> <td>be.conf</td> <td>storage_root_pathJAVA_OPTS</td> </tr> <tr> <td>Starrocks3</td> <td>fe.conf</td> <td>JAVA_OPTSmeta_dir</td> </tr> <tr> <td>be.conf</td> <td>storage_root_pathJAVA_OPTS</td> </tr> <tr> <td>Doris</td> <td>fe.conf</td> <td>JAVA_OPTSJAVA_OPTS_FOR_JDK_9meta_dir</td> </tr> <tr> <td>be.conf</td> <td>storage_root_path</td> </tr> <tr> <td>ClickHouse</td> <td>server-config</td> <td>interserver_http_host</td> </tr> <tr> <td>server-metrika</td> <td>macros.shardmacros.replica</td> </tr> <tr> <td>ZooKeeper</td> <td>None</td> <td>None</td> </tr> <tr> <td>Sqoop</td> <td>None</td> <td>None</td> </tr> <tr> <td>Knox</td> <td>None</td> <td>None</td> </tr> <tr> <td>Celeborn</td> <td>celeborn-env.sh</td> <td>CELEBORN_WORKER_MEMORYCELEBORN_WORKER_OFFHEAP_MEMORYCELEBORN_MASTER_MEMORY</td> </tr> <tr> <td>celeborn-defaults.conf</td> <td>celeborn.worker.storage.dirsceleborn.worker.flusher.threads</td> </tr> <tr> <td>Flink</td> <td>flink-conf.yaml</td> <td>security.kerberos.login.principalsecurity.kerberos.login.keytab</td> </tr> <tr> <td>HBase</td> <td>hbase-env.sh</td> <td>hbase_master_optshbase_thrift_optshbase_rest_optshbase_regionserver_opts</td> </tr> <tr> <td>hbase-site.xml</td> <td>hbase.regionserver.handler.counthbase.regionserver.global.memstore.sizehbase.regionserver.global.memstore.lowerLimithbase.regionserver.thread.compaction.throttlehbase.regionserver.thread.compaction.largehbase.regionserver.thread.compaction.small</td> </tr> <tr> <td>HBASE-HDFS</td> <td>hdfs-env.sh</td> <td>hadoop_secondarynamenode_optshadoop_namenode_heapsizehadoop_datanode_heapsize</td> </tr> <tr> <td>hdfs-site.xml</td> <td>dfs.datanode.data.dirdfs.datanode.failed.volumes.tolerateddfs.datanode.du.reserveddfs.datanode.balance.max.concurrent.moves</td> </tr> <tr> <td>JindoCache</td> <td>None</td> <td>None</td> </tr> <tr> <td>Kafka</td> <td>server.properties</td> <td>broker.idnum.network.threadsnum.io.threadskafka.heap.optslog.dirskafka.public-access.iplistenersadvertised.listeners <div> <div> </div> <div> Note <code>kafka.public-access.ip</code> specifies the public IP address of a Kafka broker. You can use this configuration item to configure listeners that have public IP addresses. </div> </div></td> </tr> <tr> <td>kafka-internal-config</td> <td>broker_id</td> </tr> <tr> <td>user_params</td> <td>is_local_disk_instance</td> </tr> <tr> <td>Kudu</td> <td>master.gflags</td> <td>fs_data_dirsfs_wal_dirfs_metadata_dirlog_dir</td> </tr> <tr> <td>tserver.gflags</td> <td>fs_data_dirsfs_wal_dirfs_metadata_dirlog_dir</td> </tr> <tr> <td>Paimon</td> <td>None</td> <td>None</td> </tr> <tr> <td>Phoenix</td> <td>None</td> <td>None</td> </tr> </tbody> </table>