When request volume exceeds Object Storage Service (OSS) limits, OSS throttles subsequent requests — increasing upload and download latency or dropping requests and returning HTTP 503. This guide shows you how to set up CloudMonitor alert rules so you're notified the moment throttling occurs.
How OSS throttling works
OSS applies two types of throttling at both the user level and the bucket level:
Bandwidth throttling: When inbound or outbound bandwidth exceeds the threshold, OSS slows down transfers. Affected requests experience increased latency.
QPS throttling: When queries per second (QPS) exceeds the threshold, OSS drops some requests and returns HTTP 503.
Alert thresholds in CloudMonitor are set at 80% of the actual throttling threshold:
Alert threshold = Throttling threshold × 0.8This gives you advance warning before throttling fully kicks in, so you can reduce load proactively.
For the full list of OSS limits, see Limits.
Prerequisites
Before you begin, ensure that you have:
A CloudMonitor contact group with the contacts who should receive throttling alerts. To create one, see Create an alert contact or alert contact group
Create an alert rule
Log on to the CloudMonitor console.
In the left-side navigation pane, choose Event Center > System Event.
On the Event Monitoring tab, click Save as Alert Rule.
In the Create/Modify Event-triggered Alert Rule panel, configure the following parameters, keep the default values for all other parameters, and click Ok.
Parameter Value Alert Rule Name A name for the rule. Example: rule1.Product Type Select Object Storage Service. Event Type Select All Types. Event Level Select WARN and INFO. Event Name Select All Events. For the full list of OSS throttling events, see OSS. Alert Contact Group For Notification Method, select Alert Notification. Then select your contact group. Notification Method Select Warning (SMS Message + Email + Webhook).
After you save the rule, CloudMonitor monitors for throttling events and threshold breaches. When either occurs, it sends a notification to the specified contacts. Each notification includes the alert resource, event name, event type, and event details.
The system checks for throttling every 1 minute. If throttling lasts 30 seconds or more within a check interval, the system sends a notification. The system checks configured alert thresholds every 10 minutes. Notifications are sent within 1 second of detecting a threshold breach or throttling event.
Respond to throttling alerts
When you receive a throttling alert, check the event name to understand the impact and how to respond.
If you receive a user-level alert (for example, UserEgressBandwidthThresholdExceeded) and want to identify which buckets are consuming the most traffic, set up an OSS monitoring dashboard first. See Create a system preset dashboard.
Throttling events
The following events are triggered when a throttling threshold is exceeded.
| Impact | Event name | Cause | Solution |
|---|---|---|---|
| Upload requests experience increased latency. | BucketIngressBandwidthThresholdExceeded | The bucket's total inbound bandwidth, the requester's inbound bandwidth for the bucket, or the bucket group's inbound bandwidth exceeds the throttling threshold. | Reduce the number of concurrent upload requests. |
| Download requests experience increased latency. | BucketEgressBandwidthThresholdExceeded | The bucket's total outbound bandwidth, the requester's outbound bandwidth for the bucket, or the bucket group's outbound bandwidth exceeds the throttling threshold. | Reduce the number of concurrent download requests. |
| OSS returns HTTP 503 for some requests. | BucketQpsThresholdExceeded | The total QPS to the bucket, the requester's QPS to the bucket, or the bucket group's QPS exceeds the throttling threshold. | Reduce the number of requests sent per second. |
| Upload requests experience increased latency. | UserIngressBandwidthThresholdExceeded | The total inbound bandwidth of all buckets in the user account, the resource pool, or the requester's resource pool exceeds the throttling threshold. | Reduce the number of concurrent upload requests. |
| Download requests experience increased latency. | UserEgressBandwidthThresholdExceeded | The total outbound bandwidth of all buckets in the user account, the resource pool, or the requester's resource pool exceeds the throttling threshold. | Reduce the number of concurrent download requests. |
| OSS does not respond to some requests. | UserQpsThresholdExceeded | The total QPS to all buckets in the user account, the resource pool, or the requester's resource pool exceeds the throttling threshold. | Reduce the number of requests sent per second. |
| Image Processing (IMG) requests experience increased latency. | BucketImageCpuThresholdExceeded | The CPU cores consumed by IMG requests to the bucket exceed the throttling threshold. | Reduce the number of concurrent IMG requests. |
| IMG requests experience increased latency. | UserImageCpuThresholdExceeded | The CPU cores consumed by IMG requests to all buckets of the user exceed the throttling threshold. | Reduce the number of concurrent IMG requests. |
| Mirroring-based back-to-origin requests experience increased latency. | BucketMirrorIngressBandwidthThresholdExceeded | The bandwidth consumed by mirroring-based back-to-origin requests for the bucket exceeds the throttling threshold. | Reduce the number of concurrent mirroring-based back-to-origin requests. |
| OSS does not respond to some mirroring-based back-to-origin requests. | BucketMirrorQpsThresholdExceeded | The total QPS of mirroring-based back-to-origin requests to the bucket exceeds the throttling threshold. | Reduce the number of mirroring-based back-to-origin requests sent per second. |
| Mirroring-based back-to-origin requests experience increased latency. | UserMirrorIngressBandwidthThresholdExceeded | The total bandwidth consumed by mirroring-based back-to-origin requests across all buckets of the user exceeds the throttling threshold. | Reduce the number of concurrent mirroring-based back-to-origin requests. |
| OSS does not respond to some mirroring-based back-to-origin requests. | UserMirrorQpsThresholdExceeded | The total QPS of mirroring-based back-to-origin requests from the user exceeds the throttling threshold. | Reduce the number of mirroring-based back-to-origin requests sent per second. |
Alert threshold events
The following events are triggered when traffic reaches 80% of the throttling threshold, giving you early warning before throttling begins.
| Impact | Event name | Cause | Solution |
|---|---|---|---|
| Upload requests experience increased latency. | BucketIngressBandwidth | The bucket's total inbound bandwidth, the user's inbound bandwidth for the bucket, or the bucket group's inbound bandwidth exceeds the alert threshold. | Reduce the number of concurrent upload requests. |
| Download requests experience increased latency. | BucketEgressBandwidth | The bucket's total outbound bandwidth, the requester's outbound bandwidth for the bucket, or the bucket group's outbound bandwidth exceeds the alert threshold. | Reduce the number of concurrent download requests. |
| Upload requests experience increased latency. | UserIngressBandwidth | The total inbound bandwidth of all buckets in the user account, the resource pool, or all buckets in the resource pool exceeds the alert threshold. | Reduce the number of concurrent upload requests. |
| Download requests experience increased latency. | UserEgressBandwidth | The total outbound bandwidth of all buckets in the user account, the resource pool, or the requester's resource pool exceeds the alert threshold. | Reduce the number of concurrent download requests. |
Alert notification details
Each alert notification includes a JSON payload with details about the throttling event. Example:
{
"AvgSeverity": "10",
"QosType": "IngressBandwidth",
"TrafficSource": "net_all",
"TargetRate": "10737418240",
"Requester": "300xxxx",
"BucketGroup": "test-group",
"ResourcePool": "resource-pool-for-ai"
}| Field | Description | Example |
|---|---|---|
AvgSeverity | The degree of throttling on a scale of 0–100. A higher value means higher latency. | 10 |
QosType | The type of throttling: IngressBandwidth, EgressBandWidth, or Qps. | IngressBandwidth |
TrafficSource | The traffic source: intranet (internal network), extranet (Internet), or net_all (both). | net_all |
TargetRate | The bandwidth value in bit/s, or the QPS value as a request count. | 10737418240 |
Requester | The UID of the requester. Returned when the requester's bucket or resource pool bandwidth hits the throttling threshold. | 300xxxx |
BucketGroup | The name of the bucket group. Returned when the bucket group bandwidth hits the throttling threshold. | test-group |
ResourcePool | The name of the resource pool. Returned when the resource pool bandwidth hits the throttling threshold. | resource-pool-for-ai |
View traffic usage across buckets
After receiving a user-level alert (for example, UserEgressBandwidthThresholdExceeded), use the OSS monitoring dashboard to identify which buckets are driving the traffic spike.
Log on to the CloudMonitor console.
In the left-side navigation pane, choose Hybrid Cloud Monitoring > Dashboards.
On the Dashboards page, find your OSS dashboard and click View on the right side of the OSS dashboard.
In the Traffic Monitoring section, identify the buckets consuming the most traffic.