This topic describes how to manage configuration items for services such as HDFS, YARN, and Spark in the E-MapReduce console.
Prerequisites
An existing cluster is required. See Create a cluster for instructions.
Configuration items
-
Log on to the E-MapReduce console.
-
In the top navigation bar, select a region and resource group based on your requirements.
-
On the EMR on ECS page, find the target cluster and click Services in the Actions column.
-
Find the target service and click Configure.
-
Use the search box to find a specific configuration item.
Note-
If a configuration item is modified at the node group level or node level, its value at that level overrides the cluster-level default. This overridden value is then displayed on the cluster-level configuration page.
-
To view configuration values at the node group or node level, select the desired node group or node.
-
Configuration values are applied at three levels. The override priority is as follows: node level > node group level > cluster level.
By default, the Configure tab displays configuration items at the cluster level.
To modify configuration items at the node group level or node level, select Node group level configuration or Node-level configuration from the Cluster-level configuration drop-down list, then modify and save the settings. To see which configuration items can be modified for specific nodes and node groups, see Modifiable configuration items.
To compare configuration differences across all levels (cluster, node group, and node), click Switch to full configuration page.
-
Modify parameters
-
Go to the configuration page of a service.
-
Log on to the E-MapReduce console.
-
In the top navigation bar, select a region and a resource group as needed.
-
On the EMR on ECS page, find the target cluster and click Services in the Actions column.
-
Find the desired service and click Configure.
-
-
Modify the configuration.
-
In the search box, enter the name of the desired configuration item.
-
Find the desired configuration item and change its parameter value.
-
-
Save the configuration.
-
On the Configure page, click Save.
-
In the Save dialog box, enter an Execution Reason and click Save.
NoteIn the Save dialog box, the Save and Deliver Configuration switch is on by default. With the switch enabled, configuration items are delivered to the client after saving. You can apply the changes in manual mode. If you turn off the Save and Deliver Configuration switch in the Save dialog box, you can apply the changes in prompt mode.
-
-
Apply the changes.
To apply the changes, perform the following steps based on the type of configuration item you modified.
Prompt mode
NoteThis mode is available only for EMR-5.12.1 and later, and EMR-3.46.1 and later.
-
Client-side configurations
-
After you save the configuration, a
prompt appears. -
Click the To Be Delivered prompt.
-
In the dialog box, click Deliver.
NoteFor the YARN service, if the delivered configuration items include queue-related configuration items, you must also click Deploy or the
prompt after the delivery to apply the changes.
-
-
Server-side configurations
-
After you save the configuration, a
prompt appears. -
Click the Not Effective Yet prompt.
-
In the dialog box, apply the changes based on the required method.
-
Configurations That Require Custom Operations
Manually click the action for each component.
-
Configurations That Require Restart
-
Click Restart or batch.
-
In the dialog box, enter an Execution Reason and click OK.
-
-
-
Manual mode
-
Client-side configurations
-
After you save the configuration, click Deploy Client Configuration next to the Save button.
-
In the dialog box, enter an Execution Reason and click OK.
-
In the Confirm dialog box, click OK.
-
-
Server-side configurations
-
After you save the configuration, on the Configure page, choose .
-
In the dialog box, enter an Execution Reason and click OK.
-
In the Confirm dialog box, click OK.
-
-
Add configuration items
-
Go to the configuration page of a service.
-
Log on to the E-MapReduce console.
-
In the top navigation bar, select a region and a resource group.
-
On the EMR on ECS page, find your cluster and click Services in the Actions column.
-
On the Services tab, find your service and click Configure.
-
-
Add configuration items.
-
Click the tab where you want to add the configuration items.
-
Click Add Configuration Item.
-
Add the configuration items.
You can add multiple configuration items at a time.
Parameter
Description
Key
The key for the configuration item.
Value
The value for the configuration item.
Description
The description for the configuration item.
Actions
Allows you to delete the configuration item.
-
After you add the configuration items, click OK.
-
Enter an Execution Reason and click Save.
-
-
Apply the changes.
To apply the changes, perform the following steps based on the type of configuration item you modified.
Prompt mode
NoteThis mode is available only for EMR-5.12.1 and later, and EMR-3.46.1 and later.
-
Client-side configurations
-
After you save the configuration, a
prompt appears. -
Click the To Be Delivered prompt.
-
In the dialog box, click Deliver.
NoteFor the YARN service, if the delivered configuration items include queue-related configuration items, you must also click Deploy or the
prompt after the delivery to apply the changes.
-
-
Server-side configurations
-
After you save the configuration, a
prompt appears. -
Click the Not Effective Yet prompt.
-
In the dialog box, apply the changes based on the required method.
-
Configurations That Require Custom Operations
Manually click the action for each component.
-
Configurations That Require Restart
-
Click Restart or batch.
-
In the dialog box, enter an Execution Reason and click OK.
-
-
-
Manual mode
-
Client-side configurations
-
After you save the configuration, click Deploy Client Configuration next to the Save button.
-
In the dialog box, enter an Execution Reason and click OK.
-
In the Confirm dialog box, click OK.
-
-
Server-side configurations
-
After you save the configuration, on the Configure page, choose .
-
In the dialog box, enter an Execution Reason and click OK.
-
In the Confirm dialog box, click OK.
-
-
Node-level modifiable parameters
This topic uses EMR-5.17.1 as an example.
Kerberos-related configuration items are displayed only when Kerberos authentication is enabled.
|
Service name |
File name |
Modifiable parameters |
|
Hadoop-Common |
core-site.xml |
fs.oss.tmp.data.dirs hadoop.tmp.dir |
|
HDFS |
hdfs-env.sh |
hadoop_datanode_heapsize hadoop_secondarynamenode_opts hadoop_namenode_heapsize |
|
hdfs-site.xml |
dfs.datanode.data.dir dfs.datanode.failed.volumes.tolerated dfs.datanode.du.reserved dfs.datanode.balance.max.concurrent.moves |
|
|
OSS-HDFS |
None |
None |
|
Hive |
hive-env.sh |
hive_metastore_heapsize hive_server2_heapsize |
|
Spark2 |
hiveserver2-site.xml |
hive.server2.authentication.kerberos.principal |
|
spark-env.sh |
spark_history_daemon_memory spark_thrift_daemon_memory |
|
|
spark-thriftserver.conf |
spark.yarn.historyServer.address spark.hadoop.hive.server2.thrift.bind.host spark.yarn.principal |
|
|
spark-defaults.conf |
spark.yarn.historyServer.address spark.history.kerberos.principal |
|
|
Spark3 |
hiveserver2-site.xml |
hive.server2.authentication.kerberos.principal |
|
spark-env.sh |
spark_history_daemon_memory spark_thrift_daemon_memory |
|
|
spark-thriftserver.conf |
spark.yarn.historyServer.address spark.hadoop.hive.server2.thrift.bind.host spark.kerberos.principal |
|
|
spark-defaults.conf |
spark.yarn.historyServer.address spark.history.kerberos.principal |
|
|
Tez |
None |
None |
|
Trino |
iceberg.properties |
hive.hdfs.trino.principal hive.metastore.client.principal |
|
delta.properties |
hive.hdfs.trino.principal hive.metastore.client.principal |
|
|
config.properties |
coordinator node-scheduler.include-coordinator query.max-memory query.max-total-memory query.max-memory-per-node http-server.authentication.type http-server.authentication.krb5.user-mapping.pattern http-server.authentication.krb5.service-name http-server.authentication.krb5.keytab http.authentication.krb5.config http-server.https.enabled http-server.https.port http-server.https.keystore.key http-server.https.keystore.path event-listener.config-files Note
The |
|
|
jvm.config |
JVM parameters |
|
|
hudi.properties |
hive.hdfs.trino.principal hive.metastore.client.principal |
|
|
password-authenticator.properties |
ldap.url ldap.user-bind-pattern |
|
|
hive.properties |
hive.hdfs.trino.principal hive.metastore.client.principal |
|
|
DeltaLake |
None |
None |
|
Hudi |
None |
None |
|
Iceberg |
None |
None |
|
JindoData |
storage.yaml |
jindofsx.storage.cache-mode storage.watermark.high.ratio storage.watermark.low.ratio storage.handler.threads Note
|
|
Flume |
flume-conf.properties |
agent_name flume-conf.properties |
|
Kyuubi |
kyuubi-env.sh |
kyuubi_java_opts |
|
YARN |
yarn-site.xml |
yarn.nodemanager.resource.memory-mb yarn.nodemanager.local-dirs yarn.nodemanager.log-dirs yarn.nodemanager.resource.cpu-vcores yarn.nodemanager.address yarn.nodemanager.node-labels.provider.configured-node-partition |
|
yarn-env.sh |
YARN_RESOURCEMANAGER_HEAPSIZE YARN_TIMELINESERVER_HEAPSIZE YARN_PROXYSERVER_HEAPSIZE YARN_NODEMANAGER_HEAPSIZE YARN_RESOURCEMANAGER_HEAPSIZE_MIN YARN_TIMELINESERVER_HEAPSIZE_MIN YARN_PROXYSERVER_HEAPSIZE_MIN YARN_NODEMANAGER_HEAPSIZE_MIN |
|
|
mapred-env.sh |
HADOOP_JOB_HISTORYSERVER_HEAPSIZE |
|
|
mapred-site.xml |
mapreduce.cluster.local.dir |
|
|
Impala |
None |
None |
|
OpenLDAP |
None |
None |
|
Ranger |
None |
None |
|
Ranger-Plugin |
None |
None |
|
DLF-Auth |
None |
None |
|
Presto |
iceberg.properties |
hive.hdfs.presto.principal hive.metastore.client.principal |
|
delta.properties |
hive.hdfs.presto.principal hive.metastore.client.principal |
|
|
hive.properties |
hive.hdfs.presto.principal hive.metastore.client.principal |
|
|
config.properties |
coordinator node-scheduler.include-coordinator query.max-memory-per-node query.max-total-memory-per-node http-server.authentication.type http.authentication.krb5.principal-hostname http.server.authentication.krb5.service-name http.server.authentication.krb5.keytab http.authentication.krb5.config http-server.https.enabled http-server.https.port http-server.https.keystore.key http-server.https.keystore.path |
|
|
jvm.config |
JVM parameters |
|
|
hudi.properties |
hive.hdfs.presto.principal hive.metastore.client.principal |
|
|
password-authenticator.properties |
ldap.url ldap.user-bind-pattern |
|
|
StarRocks2 |
fe.conf |
JAVA_OPTS meta_dir |
|
be.conf |
storage_root_path JAVA_OPTS |
|
|
StarRocks3 |
fe.conf |
JAVA_OPTS meta_dir |
|
be.conf |
storage_root_path JAVA_OPTS |
|
|
Doris |
fe.conf |
JAVA_OPTS JAVA_OPTS_FOR_JDK_9 meta_dir |
|
be.conf |
storage_root_path |
|
|
ClickHouse |
server-config |
interserver_http_host |
|
server-metrika |
macros.shard macros.replica |
|
|
ZooKeeper |
None |
None |
|
Sqoop |
None |
None |
|
Knox |
None |
None |
|
Celeborn |
celeborn-env.sh |
CELEBORN_WORKER_MEMORY CELEBORN_WORKER_OFFHEAP_MEMORY CELEBORN_MASTER_MEMORY |
|
celeborn-defaults.conf |
celeborn.worker.storage.dirs celeborn.worker.flusher.threads |
|
|
Flink |
flink-conf.yaml |
security.kerberos.login.principal security.kerberos.login.keytab |
|
HBase |
hbase-env.sh |
hbase_master_opts hbase_thrift_opts hbase_rest_opts hbase_regionserver_opts |
|
hbase-site.xml |
hbase.regionserver.handler.count hbase.regionserver.global.memstore.size hbase.regionserver.global.memstore.lowerLimit hbase.regionserver.thread.compaction.throttle hbase.regionserver.thread.compaction.large hbase.regionserver.thread.compaction.small |
|
|
HBASE-HDFS |
hdfs-env.sh |
hadoop_secondarynamenode_opts hadoop_namenode_heapsize hadoop_datanode_heapsize |
|
hdfs-site.xml |
dfs.datanode.data.dir dfs.datanode.failed.volumes.tolerated dfs.datanode.du.reserved dfs.datanode.balance.max.concurrent.moves |
|
|
JindoCache |
None |
None |
|
Kafka |
server.properties |
broker.id num.network.threads num.io.threads kafka.heap.opts log.dirs kafka.public-access.ip listeners advertised.listeners Note
The |
|
kafka-internal-config |
broker_id |
|
|
user_params |
is_local_disk_instance |
|
|
Kudu |
master.gflags |
fs_data_dirs fs_wal_dir fs_metadata_dir log_dir |
|
tserver.gflags |
fs_data_dirs fs_wal_dir fs_metadata_dir log_dir |
|
|
Paimon |
None |
None |
|
Phoenix |
None |
None |