This topic describes how to call the E-MapReduce (EMR) API to modify the configurations of a cluster.
Prerequisites
- An EMR cluster is created. For more information, see Create a cluster.
- The cluster ID is obtained. For more information, see View the cluster list and cluster details.
- An AccessKey pair is created. For more information, see Obtain an AccessKey pair.
- The required SDK is obtained. To obtain SDK for Java, see Download SDKs. To obtain SDK for Python, see Install SDK.
Modify the configurations of a cluster
Call the ModifyClusterServiceConfig operation to modify existing configurations or
add parameters.
- Modify existing configurations.
Parameter Type Required Example Description RegionId String Yes cn-hangzhou The ID of the region. ClusterId String Yes C-C0D50B0FA24A**** The ID of the cluster. ServiceName String Yes HDFS The name of the service. For more information, see Service details. ConfigParams String Yes null The specific modifications. Set this parameter to a JSON string. Comment String No modify hdfs config The comment about the modifications. Sample code:#!/usr/bin/env python #coding=utf-8 from aliyunsdkcore.client import AcsClient from aliyunsdkcore.acs_exception.exceptions import ClientException from aliyunsdkcore.acs_exception.exceptions import ServerException from aliyunsdkemr.request.v20160408.ModifyClusterServiceConfigRequest import ModifyClusterServiceConfigRequest client = AcsClient('<accessKeyId>', '<accessSecret>', 'cn-hangzhou') request = ModifyClusterServiceConfigRequest() request.set_accept_format('json') request.set_ClusterId("C-C0D50B0FA24A****") request.set_ServiceName("HDFS") request.set_ConfigParams("{\"hdfs-site\":{\"dfs.namenode.resource.check.interval\":\"6000\"}}") request.set_Comment("modify hdfs config") response = client.do_action_with_exception(request) # python2: print(response) print(str(response, encoding='utf-8'))
- Add parameters.
Add parameters to a specific instance of a cluster.
Parameter Type Required Example Description RegionId String Yes cn-hangzhou The ID of the region. ClusterId String Yes C-2FD9050E78C8**** The ID of the cluster. ServiceName String Yes HDFS The name of the service. For more information, see Service details. ConfigParams String Yes null {} indicates that new parameters are added. Comment String No add new config The comment about the added parameters. CustomConfigParams String No null Configurations added to custom configuration items. HostInstanceId String No i-bp146tnrkq4tcso2**** The ID of the instance. In most cases, this parameter is set to the ID of an ECS instance. Sample code:#!/usr/bin/env python #coding=utf-8 from aliyunsdkcore.client import AcsClient from aliyunsdkcore.acs_exception.exceptions import ClientException from aliyunsdkcore.acs_exception.exceptions import ServerException from aliyunsdkemr.request.v20160408.ModifyClusterServiceConfigRequest import ModifyClusterServiceConfigRequest client = AcsClient('<accessKeyId>', '<accessSecret>', 'cn-hangzhou') request = ModifyClusterServiceConfigRequest() request.set_accept_format('json') request.set_ClusterId("C-2FD9050E78C8****") request.set_ServiceName("HDFS") request.set_ConfigParams("{}") request.set_Comment("add new config") request.set_CustomConfigParams("{\"hdfs-site\":{\"newkey\":{\"Value\":\"customValue\"}}}") request.set_HostInstanceId("i-bp146tnrkq4tcso2****") response = client.do_action_with_exception(request) # python2: print(response) print(str(response, encoding='utf-8'))
Make the configurations of a cluster take effect
- Service parameters
After you modify service parameters, you must restart the related service for the modifications to take effect. You can restart the service in the EMR console or call the RunClusterServiceAction operation to restart the service.
Parameter Type Required Example Description RegionId String Yes cn-hangzhou The ID of the region. ClusterId String Yes C-C0D50B0FA24A**** The ID of the cluster. ServiceName String Yes HDFS The name of the service. For more information, see Service details. ServiceActionName String Yes RESTART The name of the action. Valid values: - START: Start the service.
- RESTART: Restart the service.
- STOP: Stop the service.
- CONFIGURE: Send configurations to a specific instance.
Sample code:#!/usr/bin/env python #coding=utf-8 from aliyunsdkcore.client import AcsClient from aliyunsdkcore.acs_exception.exceptions import ClientException from aliyunsdkcore.acs_exception.exceptions import ServerException from aliyunsdkemr.request.v20160408.RunClusterServiceActionRequest import RunClusterServiceActionRequest client = AcsClient('<accessKeyId>', '<accessSecret>', 'cn-hangzhou') request = RunClusterServiceActionRequest() request.set_accept_format('json') request.set_ClusterId("C-C0D50B0FA24A****") request.set_ServiceName("HDFS") request.set_ServiceActionName("RESTART") response = client.do_action_with_exception(request) # python2: print(response) print(str(response, encoding='utf-8'))
- Client parameters
After you modify client parameters, you must call the RunClusterServiceAction operation to send the configurations to a specific instance of the cluster and then deploy the configurations.
Parameter Type Required Example Description RegionId String Yes cn-hangzhou The ID of the region. ClusterId String Yes C-ADB1FDB7AA82**** The ID of the cluster. ServiceName String Yes HDFS The name of the service. For more information, see Service details. ServiceActionName String Yes CONFIGURE The name of the action. Valid values: - START: Start the service.
- RESTART: Restart the service.
- STOP: Stop the service.
- CONFIGURE: Send configurations to a specific instance.
Sample code:#!/usr/bin/env python #coding=utf-8 from aliyunsdkcore.client import AcsClient from aliyunsdkcore.acs_exception.exceptions import ClientException from aliyunsdkcore.acs_exception.exceptions import ServerException from aliyunsdkemr.request.v20160408.RunClusterServiceActionRequest import RunClusterServiceActionRequest client = AcsClient('<accessKeyId>', '<accessSecret>', 'cn-hangzhou') request = RunClusterServiceActionRequest() request.set_accept_format('json') request.set_ClusterId("C-ADB1FDB7AA82A****") request.set_ServiceName("HDFS") request.set_ServiceActionName("CONFIGURE") response = client.do_action_with_exception(request) # python2: print(response) print(str(response, encoding='utf-8'))
Related API operations
- ModifyClusterServiceConfig: For more information, see Modifies the configuration of a specified service of a cluster.
- RunClusterServiceAction: For more information, see Run specified actions.
Service details
- Services
SPARK KAFKA HIVE TEZ STORM ZEPPELIN HBASE YARN HDFS PRESTO IMPALA DRUID KNOX KAFKA-MANAGER GANGLIA HUE PHOENIX FLINK RANGER APACHEDS HAS SQOOP PIG ZOOKEEPER OOZIE TENSORFLOW SUPERSET TENSORFLOW-ON-YARN LIVY JUPYTER ANALYTICS-ZOO FLUME SMARTDATA BIGBOOT HIVE-CDH KUDU OPENLDAP TENSORFLOW-ON-SPARK GRAFANA PROMETHEUS ALERTMANAGER CLICKHOUSE
- Service components
ANALYTICS-ZOO AnalyticsZooPython AnalyticsZooScala GANGLIA GMETAD GMOND HBASE HMaster HRegionServer ThriftServer HDFS KMS HttpFS NameNode DataNode SecondaryNameNode HIVE HiveServer HiveMetaStore IMPALA ImpalaRuntime ImpalaCatalogd ImpalaStatestored Impalad KUDU KuduMaster KuduTserver PRESTO PrestoMaster PrestoWorker RANGER RangerAdmin RangerUserSync RangerPlugin SPARK SparkHistory YARN ResourceManager JobHistory TimeLineServer WebAppProxyServer NodeManager