Monitor Abnormal Metrics in Real Time via SLS Dashboard Alerts - Simple Log Service

Overview

To quickly identify abnormal metrics on dashboards, configure thresholds in a chart's Field Configuration settings. This lets you set custom metric thresholds with different visual effects to monitor if data exceeds predefined limits. For example, you can set a success rate chart's background to turn red when the rate falls below 70% and trigger an alert. The following sections detail these steps.

Configure chart threshold highlighting: In a sample dashboard, add a success rate chart. Configure a threshold and background color for the success_rate field. The background turns green if the rate is above 70% and red otherwise.
Integrate alerts: Add an alert rule for the success rate chart. When the success rate falls below 70%, an alert is automatically sent to a DingTalk group.
Configure a chart interaction: Configure a custom HTTP URL interaction occurrence for the success rate. When you click the success_rate value, you are redirected to the alert details page.

1. Configure chart threshold highlighting

1.1 Add a dashboard and a chart

Add a dashboard: Log on to the Simple Log Service console. Select the target project. In the left-side navigation pane, choose Dashboards. On the page that appears, add a dashboard named sample.
Add a chart: Click Add Chart to add a chart to the dashboard that displays the success rate.

1.2 Configure the chart

Configure general settings for the chart: In the right-side pane, configure Chart Types and Title. In the left-side pane, set the time range, Logstore, and query. Then, click Apply at the top of the page to view the result.

Query:
```
(*)|select round(sum(if(status < 300, 1, 0)) * 100.0 / count(1), 2) as success_rate
```
Configure field settings for the chart: Set the threshold for the 'success_rate' field to 70, and configure the chart's background to turn green when the rate is above 70 and red when it is below.
1. In the right-side pane, click the Field Configuration tab and select A > success_rate.
2. Configure the unit, font size, threshold, and colors for the 'success_rate' field and the chart.
  - Standard Configurations > Unit: Sets the unit for the success_rate field of query A to %.
  - Chart Style > Value Font Size: Sets the font size for the value of the success_rate field of query A to 70.
  - Threshold > Threshold: Sets the data threshold for the success_rate field of query A to 70.
  - Standard Configurations > Color Scheme: Sets the color scheme for the chart based on the configured threshold. If the value of the success_rate field for query A is less than 70, the chart background turns red. Otherwise, it turns green.
  - Chart Style > Color Mode: Sets the color mode for the chart. None means no color is applied. Value sets the font color. Background sets the background color.
  - Chart Style > Font Color: Sets the font color for the value of the success_rate field of query A to white.
  - For more information about the configurations, see Attributes of charts.

1.3 Save the chart and dashboard

In the upper-right corner of the chart editing page, click OK.
In the upper-right corner of the dashboard editing page, click Save.

2. Configure alert rules

2.1 Add an alert rule

In the upper-right corner of the chart, choose > Save as Alert to add a new alert rule for the chart.

2.2 Set the alert condition

In the alert monitoring rule panel, set the trigger condition to When data matches, success_rate < 70. Keep the default values for other parameters.

2.3 Configure the alert policy

Configure notifications to be sent to a DingTalk group as shown in the following figure. Keep the default values for other parameters.

2.4 View the triggered alert

View the triggered alert's details and copy the URL from the details page. You will need this URL later to configure the chart's interaction occurrence.

In the left-side navigation pane, choose Alert Center > Alert Rules. On the page that appears, click the name of the target rule.
In the Alert History section, view the alert notification records. Copy the URL from your browser's address bar for a later step.

3. Configure a chart interaction

3.1 Open the chart in edit mode

Enter dashboard edit mode: In the dashboard list, click sample. Then, in the upper-right corner of the dashboard page, click Edit.
Enter chart edit mode: In the upper-right corner of the success rate chart, choose > Edit.

3.2 Configure success_rate field interaction

In the right-side pane, on the Interaction Occurrences tab, click Add Field and select A > success_rate from the drop-down list.

3.3 Configure the interaction event

Click Add Event and select Create Custom HTTP URL from the drop-down list. In the dialog box, configure the parameters as shown, and then click OK.

3.4 Save the chart and dashboard

In the upper-right corner of the chart editing page, click OK.
In the upper-right corner of the dashboard editing page, click Save.

3.5 Verify the interaction event

Click the 'success_rate' value in the chart to open the alert details page in a new tab.