When your request volume exceeds OSS usage limits, throttling can impact your applications. You can use CloudMonitor to configure simple rules that monitor OSS request metrics in real time and send you alert notifications when throttling events occur.
Background
OSS throttles bandwidth and QPS at the user and bucket levels. If bandwidth throttling is triggered, the latency for accessing OSS increases. If QPS throttling is triggered, OSS drops some requests. For more information about the limits for bandwidth and QPS throttling, see Usage limits and performance metrics.
You can configure alert notifications to be sent to a designated contact group through SMS messages, emails, or DingTalk webhooks when request volume triggers throttling or reaches the reporting threshold.
Prerequisites
Create a contact group to receive throttling alert notifications and add contacts to it. For more information, see Create an alert contact or a contact group.
Create an alert rule
Log on to the CloudMonitor console.
In the left-side navigation pane, choose .
On the Event Monitoring tab, click Save as alert rule.
In the Create/Modify Event-triggered Alert Rule panel, set the following parameters, leave the other parameters at their default values, and then click OK.
Parameter | Description |
Alert Rule Name | Set this to rule1. |
Product type | Select Object Storage Service. |
Event type | Select All Types. |
Event Level | Select Warning and INFO. |
Event name | Select All Events. For information about the OSS throttling events that are supported by CloudMonitor, see OSS throttling events supported by CloudMonitor. |
Alert Contact Group | Select Alert Notification, and then select the contact group that you created. |
Notification Method | Select Warning (SMS + Email + Webhook). |
After you configure the event alert rule, CloudMonitor automatically sends an alert notification to the specified contacts if requests trigger OSS throttling or reach the reporting threshold. The alert notification contains information such as the alerting resource, event name, event type, and event details. For more information about alert notifications, see Alert notifications.
Throttling alerts are evaluated once per minute. An alert is triggered if throttling occurs for 30 seconds or longer within a one-minute interval. Reporting thresholds are evaluated once every 10 minutes. An alert is triggered if the threshold is reached for at least 1 second within the ten-minute interval.
Alert notifications
If you receive a throttling alert notification, refer to the following tables to understand the cause, impact, solution, and event details for each type of throttling event.
To view the traffic usage of all buckets that belong to your account after you receive a user-level alert, you must first create an OSS monitoring dashboard. For more information, see Create a system-predefined dashboard.
Event names
In the following table, Reporting threshold = Throttling threshold × 0.8.
Event name | Cause | Impact | Solution |
BucketIngressBandwidthThresholdExceeded |
| Upload requests are throttled and request latency increases. | Reduce the number of concurrent upload requests to a reasonable level. |
BucketEgressBandwidthThresholdExceeded |
| Download requests are throttled and request latency increases. | Reduce the number of concurrent download requests to a reasonable level. |
BucketQpsThresholdExceeded |
| OSS rejects some requests and returns the HTTP 503 status code. | Reduce the number of requests per second to a reasonable level. |
UserIngressBandwidthThresholdExceeded |
| Upload requests are throttled and request latency increases. | Reduce the number of concurrent upload requests to a reasonable level. |
UserEgressBandwidthThresholdExceeded |
| Download requests are throttled and request latency increases. | Reduce the number of concurrent download requests to a reasonable level. |
UserQpsThresholdExceeded |
| OSS rejects some requests. | Reduce the number of requests per second to a reasonable level. |
BucketImageCpuThresholdExceeded | The number of CPU cores used for image processing requests for the bucket exceeds the CPU core throttling threshold for the bucket. | The latency of image processing requests increases. | Reduce the number of concurrent image processing requests to a reasonable level. |
UserImageCpuThresholdExceeded | The total number of CPU cores used for image processing requests for all buckets of the user exceeds the CPU core throttling threshold for the user. | The latency of image processing requests increases. | Reduce the number of concurrent image processing requests to a reasonable level. |
BucketMirrorIngressBandwidthThresholdExceeded | The bandwidth used by mirroring-based back-to-origin requests for the bucket exceeds the bandwidth throttling threshold for such requests. | The latency of mirroring-based back-to-origin requests increases. | Reduce the number of concurrent mirroring-based back-to-origin requests to a reasonable level. |
BucketMirrorQpsThresholdExceeded | The QPS of mirroring-based back-to-origin requests for the bucket exceeds the QPS throttling threshold for such requests. | OSS rejects some mirroring-based back-to-origin requests. | Reduce the QPS of mirroring-based back-to-origin requests to a reasonable level. |
UserMirrorIngressBandwidthThresholdExceeded | The total inbound bandwidth of mirroring-based back-to-origin requests for all buckets of the user exceeds the bandwidth throttling threshold for such requests for the user. | The latency of mirroring-based back-to-origin requests increases. | Reduce the number of concurrent mirroring-based back-to-origin requests to a reasonable level. |
UserMirrorQpsThresholdExceeded | The total QPS of mirroring-based back-to-origin requests for all buckets of the user exceeds the throttling threshold for such requests for the user. | OSS rejects some mirroring-based back-to-origin requests. | Reduce the QPS of mirroring-based back-to-origin requests to a reasonable level. |
BucketIngressBandwidth |
| The latency of upload requests to the bucket increases. | Reduce the number of concurrent upload requests to a reasonable level. |
BucketEgressBandwidth |
| The latency of download requests from the bucket increases. | Reduce the number of concurrent download requests to a reasonable level. |
UserIngressBandwidth |
| The latency of upload requests for the user increases. | Reduce the number of concurrent upload requests to a reasonable level. |
UserEgressBandwidth |
| The latency of download requests for the user increases. | Reduce the number of concurrent download requests to a reasonable level. |
Notification details
The following code provides a sample alert notification:
{
"AvgSeverity": "10",
"QosType": "IngressBandwidth",
"TrafficSource": "net_all",
"TargetRate": "10737418240",
"Requester": "300xxxx",
"BucketGroup": "test-group",
"ResourcePool": "resource-pool-for-ai"
}The following table describes the parameters in the preceding notification.
Parameter | Description | Example |
AvgSeverity | The severity of throttling. A higher value indicates more severe throttling and higher latency. Valid values: 0 to 100. | 10 |
QosType | The type of traffic that triggered the throttling. Valid values:
| IngressBandwidth |
TrafficSource | The source of the traffic that triggered throttling. Valid values:
| net_all |
TargetRate | The bandwidth or QPS value.
| 10737418240 |
Requester | If throttling is triggered by a specific requester's bandwidth usage, the | 300xxxx |
BucketGroup | If the bandwidth of a BucketGroup in a resource pool reaches the throttling threshold, the | test-group |
ResourcePool | If the bandwidth of a resource pool reaches the throttling threshold, the | resource-pool-for-ai |
View user-level traffic usage
This section uses the UserEgressBandwidthThresholdExceeded event as an example to show how to view the traffic usage for each of your buckets.
-
Log on to the Cloud Monitor console.
-
In the left-side navigation pane, choose .
On the Dashboards page, click View on the right side of the OSS monitoring dashboard.
In the Traffic Monitoring section, check which buckets are consuming high traffic based on the throttling alert type.