Create alert rules

Background information

Alerting is provided by CloudMonitor. To manage alert rules or access additional monitoring features, go to the CloudMonitor console. For more information, see What is CloudMonitor?.

Prerequisites

A cluster is created. For more information, see Create a cluster.

Limits

RAM users must have the following permissions. To grant permissions to a RAM user, see Grant permissions to a RAM user.

{
    "Version": "1",
    "Statement": [
        {
            "Action": [
                "cms:DescribeContactGroupList",
                "cms:DescribeMetricMetaList",
                "cms:PutResourceMetricRules",
                "cms:DescribeMetricRuleList"
            ],
            "Resource": "*",
            "Effect": "Allow"
        }
    ]
}

From template

Go to the Alert Management page.
1. Log on to the E-MapReduce console.
2. In the top navigation bar, select a region and resource group based on your requirements.
3. On the EMR on ECS page, click the ID of your target Cluster ID.
4. Click the Monitoring and Diagnostics tab.
5. Click Alert Management.
On the Alert Management page, click Create Alert Rules.
In the Create Alert Rules panel, find the target service and click Create Alert Rules in the Actions column.

Configure the parameters and click create.

Parameter	Description
Rule Description	Lists the alert rules in the template. You can view metric names and modify default thresholds. For information about the service scope and thresholds, see Template service scope.
Mute Period	The interval at which alert notifications are re-sent for unresolved alerts.
Validity Period	The time window during which the alert rule is active. CloudMonitor evaluates metrics only within this window.
Alert Contact Group	The alert contact group that receives notifications.
Notification method	The notification method and email subject. The following notification methods are available: Phone call, Text message, Email, and DingTalk Chatbot Text message, Email, and DingTalk Chatbot Email and DingTalk Chatbot Alert Email Subject (Optional): If you specify a subject, it is included in the alert notification email.
Callback URL (optional)	A publicly accessible callback URL. CloudMonitor sends a POST request with alert information to this URL. Only HTTP is supported.

After you create the alert rules, they take effect on the current instance. You can view them on the Alert Management page.

On the Alert Management page, click Manage Alert Rules to open the CloudMonitor console, where you can view or modify the rules.

Custom

Go to the Alert Management page.
1. Log on to the E-MapReduce console.
2. In the top navigation bar, select a region and resource group based on your requirements.
3. On the EMR on ECS page, click the ID of your target Cluster ID.
4. Click the Monitoring and Diagnostics tab.
5. Click Alert Management.
On the Alert Management page, click Create Alert Rules.
In the Create Alert Rules panel, click the Create Custom Rule tab.

Configure the parameters and click create.

Parameter	Description
Alert rule	The alert rule name and triggering condition. The condition defines the metric thresholds and criteria that trigger the alert. Note For a list of EMR metrics applicable to alert rules, see CloudMonitor metrics. You can click Add Alert Rule to specify multiple rule conditions.
Mute period	The interval at which alert notifications are re-sent for unresolved alerts.
Effective period	The time window during which the alert rule is active. CloudMonitor evaluates metrics only within this window.
Alert contact group	The alert contact group that receives notifications.
Notification method	The notification method and email subject. The following notification methods are available: Phone call, Text message, Email, and DingTalk Chatbot Text message, Email, and DingTalk Chatbot Email and DingTalk Chatbot Alert Email Subject (Optional): If you specify a subject, it is included in the alert notification email.
Callback URL (optional)	A publicly accessible callback URL. CloudMonitor sends a POST request with alert information to this URL. Only HTTP is supported.

After you create the alert rule, it takes effect on the current instance. You can view it on the Alert Management page.

On the Alert Management page, click Manage Alert Rules to open the CloudMonitor console, where you can view or modify the rules.

View alert rules

The Alert Management page lists all alert rules you created.

Parameter	Description
Rule Name	The CloudMonitor alert rule name.
Status	The alert rule status in CloudMonitor. Valid values: OK, Alert, Insufficient Data, Disabled, and Enabled.
Rule Description	The triggering conditions of the alert rule. An alert is triggered when metric data meets these conditions.
Alert Contact Group	The contact group that receives notifications.
Actions	Details: Opens the CloudMonitor console to view the rule details, alert contact group, alert history, and affected resources. Edit Rule: Opens the CloudMonitor console to modify the alert rule parameters.

Template service scope

Service	Component	Metric	Description
Node (host)	Disk	`emr_node_part_max_used`	An alert is triggered if the average value is greater than 80% for 2 consecutive periods. The metric is checked every minute.
	CPU	`emr_node_cpu_idle`	An alert is triggered if the average value is less than 5% for 5 consecutive periods. The metric is checked every minute.
	Memory	`emr_node_mem_used_percent`	An alert is triggered if the average value is greater than 90% for 2 consecutive periods. The metric is checked every minute.
HDFS	NameNode	`hdfs_namenode_jvm_MemHeapUsedM` / `hdfs_namenode_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
	NameNode	`hdfs_namenode_rpc_service_activity_CallQueueLength`	An alert is triggered if the average value is greater than 1,000 for 2 consecutive periods. The metric is checked every minute.
	NameNode	`hdfs_namenode_fsnamesystem_CorruptBlocks`	An alert is triggered if the average value is greater than 1 for 2 consecutive periods. The metric is checked every minute.
	NameNode	`hdfs_namenode_safemode_status`	An alert is triggered if the component enters safe mode. The metric is checked every minute.
	DataNode	`hdfs_datanode_jvm_MemHeapUsedM` / `hdfs_datanode_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
Spark	SparkHistoryServer	`spark_history_jvm_old_space_utilization`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods. The metric is checked every minute.
Spark	SparkThriftServer	`spark_thrift_driver_jvm_heap_used`/`spark_thrift_driver_jvm_heap_max`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods. The metric is checked every minute.
Hive	HiveMetaStore	`hive_metastore_memory_heap_used`/`hive_metastore_memory_heap_max`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods. The metric is checked every minute.
	HiveMetaStore	`hive_metastore_threads_blocked_count`	An alert is triggered if the average value is greater than 50% for 2 consecutive periods. The metric is checked every minute.
	HiveServer2	`hive_server_memory_heap_used`/`hive_server_memory_heap_max`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods. The metric is checked every minute.
	HiveServer2	`hive_server_threads_deadlock_count`	An alert is triggered if the average value is greater than 50% for 2 consecutive periods. The metric is checked every minute.
YARN	ResourceManager	`yarn_cluster_status`	An alert is triggered if two or more high availability (HA) switchovers occur within 5 minutes, if a node has a status of 1, or if a node's status remains -1.
	ResourceManager	`yarn_resourcemanager_jvm_MemHeapUsedM` / `yarn_resourcemanager_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
	NodeManager	`yarn_cluster_unhealthyNodes`	An alert is triggered if the average value is greater than 1 for 2 consecutive periods. The metric is checked every minute.
	NodeManager	`yarn_nodemanager_jvm_MemHeapUsedM` / `yarn_nodemanager_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
	TimelineServer	`yarn_timelineserver_jvm_MemHeapUsedM` / `yarn_timelineserver_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
	MRHistoryServer	`yarn_jobhistory_jvm_MemHeapUsedM` / `yarn_jobhistory_jvm_MemHeapMaxM`	An alert is triggered if the average value is greater than 95% for 2 consecutive periods, or if no metric data is generated. The metric is checked every minute.
ZooKeeper	ZooKeeper	`zk_znode_count`	An alert is triggered if the average value is greater than or equal to 10,000 for 2 consecutive periods. The metric is checked every minute.
ZooKeeper	ZooKeeper	`zk_watch_count`	An alert is triggered if the average value is greater than or equal to 1,000 for 2 consecutive periods. The metric is checked every minute.
Kafka	KafkaBroker	`Kafka_Broker_kafka_log_LogManager_OfflineLogDirectoryCount`	An alert is triggered if the average value is greater than 0 for 2 consecutive periods. The metric is checked every minute.
Kafka		`Kafka_Broker_kafka_server_ReplicaManager_UnderReplicatedPartitions`	An alert is triggered if the average value is greater than 0 for 2 consecutive periods. The metric is checked every minute.
Presto/Trino	Trino	`trino_QueryManager_FailedQueries_OneMinute_Count`	An alert is triggered if the average value is greater than or equal to 1 for 2 consecutive periods. The metric is checked every minute.
	Trino	`trino_ClusterMemoryPool_name_general_BlockedNodes`	An alert is triggered if the average value is greater than 0 for 2 consecutive periods. The metric is checked every minute.
	Presto	`presto_QueryManager_FailedQueries_OneMinute_Count`	An alert is triggered if the average value is greater than or equal to 1 for 2 consecutive periods. The metric is checked every minute.
	Presto	`presto_ClusterMemoryPool_name_general_BlockedNodes`	An alert is triggered if the average value is greater than 0 for 2 consecutive periods. The metric is checked every minute.
Impala	Impalad	`num_waiting_queries`	An alert is triggered if the average value is greater than or equal to 10 for 2 consecutive periods. The metric is checked every minute. Note You can adjust the threshold based on your cluster's query concurrency.
Kudu	kudu-master	`kudu_cluster_replica_skew`	An alert is triggered if the average value is greater than or equal to 1,000 for 2 consecutive periods. The metric is checked every minute. Note You can adjust the threshold based on your workload.