When Microservices Engine (MSE) throttles or circuit-breaks requests, your operations team needs to know immediately. After you report MSE traffic protection events to Simple Log Service (SLS), configure alert rules to automatically notify your team. This helps them respond before application stability is affected.
How it works
MSE logs each throttling or circuit breaking event and sends it to an SLS Logstore. An SLS alert monitoring rule periodically queries these logs. When the query returns results that match your trigger condition, SLS sends a notification through the channel you configure.
MSE traffic protection event --> SLS Logstore --> Alert monitoring rule --> NotificationEach traffic protection log entry contains the following fields. Understanding these fields helps you write effective alert queries and design useful notification templates.
| Field | Description | Example values |
|---|---|---|
resource | The protected API endpoint or interface | com.example.service.UserAPI |
expType | The type of protection that triggered | Throttling, circuit breaking |
ruleId | The ID of the traffic protection rule that matched | 12345 |
blockNum | The number of requests blocked per second (blocked QPS) | 61 |
time | The timestamp when the protection event occurred | 2024-06-20 16:35:10 |
appName | The name of the application where the event occurred | order-service |
namespace | The environment or namespace of the application | production |
Prerequisites
Before you begin, make sure that you have:
An activated SLS instance with a Logstore created and log collection configured. See "Step 1: Activate Simple Log Service" in Getting started
MSE Microservices Governance traffic protection enabled with traffic protection rules configured
Traffic protection events reported to SLS. See Report MSE traffic protection events to Simple Log Service.
Create an alert monitoring rule
The following steps cover the main configuration. For all available options, see Configure an alert monitoring rule in Simple Log Service.
Step 1: Define the alert query
Log on to the Simple Log Service console.
Select the project that you created or used in Report MSE traffic protection events to Simple Log Service.
In the left-side navigation pane, click the
icon to go to the Logstores page.Click the target Logstore, then click the
icon in the upper-right corner.
In the Alert Monitoring Rule panel, click the
icon next to Query Statistics and enter the following query in the Query field:* | select resource,expType,ruleId,blockNum,time,appName,namespaceThis query retrieves all traffic protection events with their fields: the protected endpoint, protection type, matched rule, blocked QPS, timestamp, application name, and namespace.
Step 2: Set the alert destination and policy
Set Destination to Simple Log Service Notification and turn on the Enable switch.
Set Alert Policy to Simple Mode and configure the alert policy.
After you save the rule, it appears in Alert Center.
Customize the notification template
Define what information appears in the alert notification when a traffic protection event triggers it.
In the left-side navigation pane, click the
icon to go to Alert Center.Click the Notification Policy tab, then click the Alert Template subtab.
Click Create. In the Add Alert Template dialog box, configure the ID, Name, and Content parameters.
The following sections provide two template options depending on how many events you want to display.
First triggered event only
Use this template to display details of the first event that triggered the alert:
- Alert name: {{ alert.alert_name }}
- Alert severity: {{ alert.severity }}
- Trigger condition: {{ alert.condition }}
- Environment: {{ alert.annotations.namespace }}
- Application: {{ alert.annotations.appName }}
- Interface: {{ alert.annotations.resource }}
- Type: {{ alert.annotations.expType }}
- Hit rule: {{ alert.annotations.ruleId }}
- QPS for throttling: {{alert.annotations.blockNum}}
- Hit time: {{ alert.annotations.time }}
- Details: [[Details]({{ alert.query_url }})]All triggered events
Use this template to display details of every event that triggered the alert within the evaluation window. This option works well when multiple traffic protection rules fire simultaneously across different services.
- Alert name: {{ alert.alert_name }}
- Alert severity: {{ alert.severity }}
- Trigger condition: {{ alert.condition }}
{%- for result in alert.fire_results %}
- Application: {{ result.appName }}
- Interface: {{ result.resource }}
- Type: {{ result.expType }}
- Hit rule: {{ result.ruleId }}
- QPS for throttling: {{result.blockNum}}
- Hit time: {{ result.time }}
{%- endfor %}
- Details: [[Details]({{ alert.query_url }})]Verify the alert
After you save the alert monitoring rule and notification template:
In Alert Center, confirm the new rule appears in the rule list.
Trigger a traffic protection event (or wait for one to occur naturally) and check that the alert notification is delivered with the expected content.