If the number of requests sent by users exceeds the threshold that is configured for Object Storage Service (OSS), subsequent OSS requests are throttled. You can configure basic monitoring settings in the CloudMonitor console to monitor requests sent to OSS in real time and receive alert notifications immediately after throttling is triggered.
Background information
OSS provides bandwidth throttling and queries per second (QPS) throttling at the user and bucket levels. When the QPS or bandwidth usage of requests sent to OSS exceeds the threshold, access to OSS is throttled and the access speed is limited. If bandwidth throttling is triggered, the access latency increases. If QPS throttling is triggered, OSS discards some requests. For more information about bandwidth throttling and QPS throttling, see Limits.
In the CloudMonitor console, you can create an alert rule for OSS throttling events and specify that SMS, email, and DingTalk chatbot notifications are sent to specified contact groups if throttling is triggered or the configured alert thresholds are reached.
Prerequisites
A contact group is created and contacts who are expected to receive throttling alerts are added to the contact group. For more information, see Create an alert contact or alert contact group.
Create an alert rule
Log on to the CloudMonitor console.
In the left-side navigation pane, choose .
On the Event Monitoring tab, click Save as Alert Rule.
In the Create/Modify Event-triggered Alert Rule panel, configure the following parameters, retain the default values for other parameters, and click OK.
Parameter | Description |
Alert Rule Name | Specify a name for the alert rule. In this example, set this parameter to rule1. |
Product Type | Select Object Storage Service. |
Event Type | Select All Types. |
Event Level | Select WARN and INFO. |
Event Name | Select All Events. For more information about OSS throttling events supported by CloudMonitor, see OSS. |
Alert Contact Group | Select Alert Notification for Notification Method. Then, select the created contract group for Alert Contact Group. |
Notification Method | Select Warning (SMS Message + Email + Webhook). |
After the alert rule is configured, if OSS throttling is triggered or alert thresholds are reached, CloudMonitor automatically sends notifications to the specified contacts. An alert notification includes information, such as the alert resource, event name, event type, and event details. For more information about alert notifications, see Alert notifications.
The system checks whether throttling is performed at an interval of 1 minute. If throttling is performed for a duration that is equal to or greater than 30 seconds within an interval, the system sends a notification to the specified contacts. The system checks whether configured alert thresholds are reached at an interval of 10 minutes. If configured alert thresholds are reached or throttling is triggered, the system sends a notification to the specified contacts within 1 second.
Alert notifications
If a throttling alert notification is received, check the alert notification to obtain the details of the alert event. The following tables describe the events, their causes, impact, solutions, and notification details.
If you want to view the traffic usage of all buckets that belong to the current user after you receive a user-level alert notification, create an OSS monitoring dashboard in advance. For more information, see Create a system preset dashboard.
Event names in alert notifications
The alert thresholds in the following table are calculated based on the formula: Alert threshold = Throttling threshold × 0.8.
Event name | Cause | Impact | Solution |
BucketIngressBandwidthThresholdExceeded |
| Requests sent to upload objects are throttled and experience increased latency. | Reduce the number of concurrent upload requests. |
BucketEgressBandwidthThresholdExceeded |
| Requests sent to download objects are throttled and experience increased latency. | Reduce the number of concurrent download requests. |
BucketQpsThresholdExceeded |
| OSS does not respond to some requests and returns HTTP status code 503. | Reduce the number of requests sent per second. |
UserIngressBandwidthThresholdExceeded |
| Requests sent to upload objects are throttled and experience increased latency. | Reduce the number of concurrent upload requests. |
UserEgressBandwidthThresholdExceeded |
| Requests sent to download objects are throttled and experience increased latency. | Reduce the number of concurrent download requests. |
UserQpsThresholdExceeded |
| OSS does not respond to some requests. | Reduce the number of requests sent per second. |
BucketImageCpuThresholdExceeded | The number of CPU cores consumed by image processing (IMG) requests sent to the bucket exceeds the throttling threshold. | IMG requests experience increased latency. | Reduce the number of concurrent IMG requests. |
UserImageCpuThresholdExceeded | The number of CPU cores consumed by IMG requests sent to all buckets of the user exceeds the throttling threshold. | IMG requests experience increased latency. | Reduce the number of concurrent IMG requests. |
BucketMirrorIngressBandwidthThresholdExceeded | The bandwidth of the bucket consumed by mirroring-based back-to-origin requests exceeds the throttling threshold. | Mirroring-based back-to-origin requests experience increased latency. | Reduce the number of concurrent mirroring-based back-to-origin requests. |
BucketMirrorQpsThresholdExceeded | The total number of mirroring-based back-to-origin requests sent to the bucket per second exceeds the throttling threshold. | OSS does not respond to some mirroring-based back-to-origin requests. | Reduce the number of mirroring-based back-to-origin requests sent per second. |
UserMirrorIngressBandwidthThresholdExceeded | The total bandwidth consumed by all buckets of the user to send mirroring-based back-to-origin requests exceeds the throttling threshold. | Mirroring-based back-to-origin requests experience increased latency. | Reduce the number of concurrent mirroring-based back-to-origin requests. |
UserMirrorQpsThresholdExceeded | The total number of mirroring-based back-to-origin requests of the user sent to the bucket per second exceeds the throttling threshold. | OSS does not respond to some mirroring-based back-to-origin requests. | Reduce the number of mirroring-based back-to-origin requests sent per second. |
BucketIngressBandwidth |
| Requests sent to upload objects to buckets experience increased latency. | Reduce the number of concurrent upload requests. |
BucketEgressBandwidth |
| Requests sent to download objects from buckets experience increased latency. | Reduce the number of concurrent download requests. |
UserIngressBandwidth |
| Requests sent to upload objects to buckets from the user experience increased latency. | Reduce the number of concurrent upload requests. |
UserEgressBandwidth |
| Requests sent to download objects from buckets from the user experience increased latency. | Reduce the number of concurrent download requests. |
Alert notification details
Example:
{
"AvgSeverity": "10",
"QosType": "IngressBandwidth",
"TrafficSource": "net_all",
"TargetRate": "10737418240",
"Requester": "300xxxx",
"ResourcePool": "resource-pool-for-ai"
}The following table describes the preceding parameters.
Parameter | Description | Example |
AvgSeverity | The degree of throttling. A higher value results in higher latency. Valid values: 0 to 100. | 10 |
QosType | The type of throttling. Valid values:
| IngressBandwidth |
TrafficSource | The source of the traffic that triggers the throttling. Valid values:
| net_all |
TargetRate | The value of the bandwidth and QPS.
| 10737418240 |
Requester | If the bandwidth of the bucket of the requester or the resource pool reaches the throttling threshold, the Requester field is returned in the event details to identify the requester by displaying its UID. | 300xxxx |
ResourcePool | If the bandwidth of the resource pool or the resource pool of the requester reaches the throttling threshold, the ResourcePool field is returned in the event details to identify the resource pool by displaying its name. | resource-pool-for-ai |
How do I view the total traffic usage of a user?
For example, you receive a UserEgressBandwidthThresholdExceeded event alert. You can perform the following steps to view the traffic usage on all buckets of the user.
Log on to the CloudMonitor console.
In the left-side navigation pane, choose .
On the Dashboards page, find the dashboard and click View on the right side of the OSS dashboard.
In the Traffic Monitoring section, view the buckets that consume a large amount of traffic.