The intelligent denoising feature of the alert management module in Application Real-Time Monitoring Service (ARMS) helps you filter critical events from a large number of historical events. You can specify a threshold value for noise events to filter out and block specific noise events. The threshold value is also called the threshold value of entropy. The intelligent denoising feature is out-of-the-box. You can enable and use the feature with only a few clicks. This topic describes how to enable and configure the intelligent denoising feature.
Background information
Most monitoring tools allow users to specify threshold values or dynamic threshold values to identify exceptions. If an exception occurs, an alert is triggered and an alert event is generated. Monitoring tools also allow users to configure rules. If a rule is matched, specific event, such as machine resetting, is triggered. In most cases, an O&M team uses multiple monitoring tools to configure different alert rules. However, if multiple monitoring tools are used to monitor multiple sources, an exception may match different alert rules, and a large number of duplicate and redundant alert events may be generated. If multiple exceptions occur, an alert storm may even occur. In this case, O&M personnel cannot identify critical alert events in a quick and effective manner.
- Multiple monitoring and alerting sources and frequent false positives lead to a large number of duplicate and redundant events. Critical events are hard to be identified.
- A large number of exceptions lead to an alert storm.
- Dirty data, such as test events, exists.
How intelligent denoising works
The alert management module allows you to configure different alerting sources in the ARMS console. This way, you can process alert events in a centralized manner. The alert management module provides features, such as the feature of event processing flow and the intelligent denoising feature. The intelligent denoising feature uses natural language processing (NLP) algorithms to process events. The feature uses the following two terms to indicate the importance of events: the amount of information and entropy in information theory. Entropy is the expected value of the self-information of a variable. Entropy is used to indicate the amount of information in an event and is measured in bits or nats. This feature helps you identify critical events from a large number of historical events. You can also specify the threshold value of entropy to filter out and block noise events. The intelligent denoising feature is out-of-the-box. You can enable and use the feature with only a few clicks. The intelligent denoising model is updated once a week to adapt to the changes of the event types and content. You do not need to perform any operations.
- Vectorizes words of event content based on NLP and domain-specific lexicons. This helps measure events at the minimum granularity.
- Constructs entropy values of word vectors and a model to measure event importance based on entropy in information theory and the term frequency-inverse document frequency (TF-IDF) model.
- Uses the sigmoid function and a non-linear normalization method to measure the entropy values of events.
- Trains the intelligent denoising model to implement automatic model iteration based on the processing records and feedback of historical events.
Enable the intelligent denoising feature
- On the Events page, turn on Intelligent Noise Reduction.
Note After you enable the intelligent denoising feature, the alert management module pulls the historical events within the last month for intelligent model training. If a large number of historical events are generated in the last month, the module pulls some of the events.
View details on the Intelligent Noise Reduction Details page
| Parameter | Description |
|---|---|
| Analyzed events | The number of the events that are analyzed by the intelligent denoising feature. |
| Identified Noise Events | The number of the events whose entropy values are lower than the specified threshold value of entropy. The default threshold value of entropy is 0. |
| Noise Event Identification Ratio | The ratio of the noise events to the analyzed events. |
| Event Information Entropy Distribution Map | The distributions of noise events and non-noise events. |
| Analysis results | The Analysis results section displays the list of the analyzed events. You can specify the Noise or Information entropy parameter to filter events based on your business requirements. You can click an event name in the list to view the event details. |
| Top 50 Common Words | The intelligent denoising model saves a word frequency table that contains the words of events based on the statistics of historical events. The top 50 common words are the 50 words that occur most frequently. You can use common words to obtain more detailed information about alert events that belong to the current account. You can also specify keywords to filter noise events or non-noise events based on the common words. |
Configure parameters for the intelligent denoising model
On the Intelligent Noise Reduction Details page, you can configure the Noise Event Threshold, Priority, and Shielding word parameters.
| Parameter | Description |
|---|---|
| Noise Event Threshold | After you enable the intelligent denoising feature, the alert management module calculates the entropy value of each new event. The value of the Noise Event Threshold parameter specifies whether an event is a noise event or a non-noise event. The default threshold for noise events is 0. |
| Priority | In the Configuration of keywords section, you can specify keywords, such as critical, to help filter non-noise events. If the name or content of an event contains a keyword that you specify, the priority of the event is increased to prevent the event from being identified as a noise event. |
| Shielding word | In the Configuration of keywords section, you can also specify keywords, such as test, to help filter noise events. If the name or content of an event contains a keyword that you specify, the entropy value of the event is determined as 0. If you specify the threshold value of noise events to a value that is greater than 0, the event is determined as a noise event. |
FAQ
- When do I need to enable the intelligent denoising feature?
If a large number of duplicate and redundant historical events exist, and critical events are difficult to be filtered from historical events, you can enable the intelligent denoising feature.
If more than 1,000 historical events exist, you can enable the intelligent denoising feature. If the number of historical events exceeds 1,000, the intelligent denoising feature can more accurately identify noise events.
- Do I need to modify the parameter values of the intelligent denoising model?
No, you do not need to modify the parameter values of the intelligent denoising model if it is the first time that you use the intelligent denoising feature. We recommend that you use the default values. The parameters of the intelligent denoising model are the Noise Event Threshold parameter, the Priority parameter, and the Shielding word parameter. After you are more familiar with the feature, you can modify the values based on your business requirements.