If the number of requests sent by users exceeds a threshold that you configure for Object Storage Service (OSS), OSS throttling is triggered. After OSS throttling is triggered, users may fail to send more requests. To address this issue, you need to only complete simple configurations in the CloudMonitor console. This way, you can monitor requests sent to OSS in real time and receive notifications immediately after throttling is triggered.

Background information

OSS provides throttling at the user and bucket levels such as bandwidth throttling and queries per second (QPS) throttling. When the QPS or bandwidth usage of requests sent to OSS exceeds the threshold that you configure for OSS, access to OSS is throttled and the access speed is limited. If bandwidth throttling is triggered, latencies of access to OSS increase. If QPS throttling is triggered, OSS discards specific requests. For more information about bandwidth throttling and QPS throttling, see Limits.

In the CloudMonitor console, you can create an alert rule for OSS throttling events and configure contact groups to receive notifications by text message, email, and DingTalk chatbot if OSS QPS throttling or bandwidth throttling is triggered, or alert thresholds are reached.

Prerequisites

A contact group is created to receive notifications immediately after throttling is triggered. Contacts are added to the contact group. For more information, see Create an alert contact or alert contact group.

Create an alert rule

  1. Log on to the CloudMonitor console.
  2. In the left-side navigation pane, choose Event Monitoring > System Event.
  3. Click the Event Alert tab.
  4. Click Create Alert Rule.
  5. In the Create / Modify Event Alert panel, configure the following parameters. Retain default values for other parameters. Then, click OK.
    Parameter Description
    Alert Rule Name Set this parameter to rule1.
    Product Type Select Object Storage Service.
    Event Type Select All Events.
    Event Level Select WARN and INFO.
    Event Name Select All Events. For more information about OSS throttling events supported by CloudMonitor, see OSS.
    Contact Group Select Alert Notification for Notification Method. Then, select the created contract group.
    Notification Method Select Warning (Message+Email ID+DingTalk Robot).

    After the alert rule is configured, if OSS throttling is triggered or alert thresholds are reached, CloudMonitor automatically sends notifications to the specified contacts. A notification includes information such as the alert resource, event name, event type, and event details. For more information about alert notifications, see Alert notification.

    Notice The system checks whether the monitored items reach the throttling threshold at an interval of 1 minute. If the duration in which the monitored items exceed the throttling threshold is equal to or larger than 30 seconds within an interval, the system sends a notification to the specified contacts. The system checks whether the monitored items reach the alert threshold at an interval of 10 minutes. If the duration in which the monitored items exceed the throttling threshold is equal to or larger than 1 second within an interval, the system sends a notification to the specified contacts.

Alert notification

If the specified contacts receive an alert notification, refer to the following tables for detailed information about the notification and the event name, cause, impact, and solution.

Notice If you want to view the traffic usage of all buckets that belong to the current user after you receive a user-level alert notification, create an OSS monitoring dashboard in advance. For more information about how to create an OSS monitoring dashboard, see Create a system preset dashboard.

Event names in alert notifications

Note The alert thresholds in the following table are calculated based on the formula: Alert threshold = Throttling threshold × 0.8.
Event name Cause impact Solution
BucketIngressBandwidthThresholdExceeded The total upstream bandwidth consumed by the current bucket exceeds the throttling threshold. Requests sent to upload objects are throttled, and latencies increase. Reduce the number of concurrent upload requests.
BucketEgressBandwidthThresholdExceeded The total downstream bandwidth consumed by the current bucket exceeds the throttling threshold. Requests sent to download objects are throttled, and latencies increase. Reduce the number of concurrent download requests.
BucketQpsThresholdExceeded The total number of requests received by the current bucket per second exceeds the throttling threshold. OSS does not respond to specific requests and returns HTTP status code 503. Reduce the number of requests sent per second.
UserIngressBandwidthThresholdExceeded The total upstream bandwidth consumed by all buckets of the current user exceeds the throttling threshold. Requests sent to upload objects are throttled, and latencies increase. Reduce the number of concurrent upload requests.
UserEgressBandwidthThresholdExceeded The total downstream bandwidth consumed by all buckets of the current user exceeds the throttling threshold. Requests sent to download objects are throttled, and latencies increase. Reduce the number of concurrent download requests.
UserQpsThresholdExceeded The total number of requests received by all buckets of the current user per second exceeds the throttling threshold. OSS does not respond to specific requests. Reduce the number of requests sent per second.
BucketImageCpuThresholdExceeded The number of CPU cores consumed by the current bucket to process Image Processing (IMG) requests exceeds the throttling threshold. Latencies increase after requests are sent to implement IMG. Reduce the number of concurrent requests sent to implement IMG.
UserImageCpuThresholdExceeded The number of CPU cores consumed by all buckets of the current user to process IMG requests exceeds the throttling threshold. Latencies increase after requests are sent to implement IMG. Reduce the number of concurrent requests sent to implement IMG.
BucketMirrorIngressBandwidthThresholdExceeded The bandwidth consumed by the current bucket to send mirroring-based back-to-origin requests exceeds the throttling threshold. Latencies increase after mirroring-based back-to-origin requests are sent. Reduce the number of concurrent mirroring-based back-to-origin requests.
BucketMirrorQpsThresholdExceeded The total number of requests sent by the current bucket per second to implement mirroring-based back-to-origin exceeds the throttling threshold. OSS does not respond to specific mirroring-based back-to-origin requests. Reduce the number of mirroring-based back-to-origin requests sent per second.
UserMirrorIngressBandwidthThresholdExceeded The total traffic consumed by all buckets of the current user to send mirroring-based back-to-origin requests exceeds the throttling threshold. Latencies increase after upload requests are sent to implement mirroring-based back-to-origin. Reduce the number of concurrent mirroring-based back-to-origin requests.
UserMirrorQpsThresholdExceeded The total number of requests sent by all buckets of the current user per second to implement mirroring-based back-to-origin exceeds the throttling threshold. OSS does not respond to specific mirroring-based back-to-origin requests. Reduce the number of mirroring-based back-to-origin requests sent per second.
BucketIngressBandwidth The total upstream bandwidth consumed by the current bucket exceeds the alert threshold. Latencies increase after upstream requests are sent to the bucket. Reduce the number of concurrent upstream requests.
BucketEgressBandwidth The total downstream bandwidth consumed by the current bucket exceeds the alert threshold. Latencies increase after downstream requests are sent to the bucket. Reduce the number of concurrent downstream requests.
UserIngressBandwidth The total upstream bandwidth consumed by all buckets of the current user exceeds the alert threshold. Latencies increase after upstream requests are sent to the bucket. Reduce the number of concurrent upstream requests.
UserEgressBandwidth The total downstream bandwidth consumed by all buckets of the current user exceeds the alert threshold. Latencies increase after downstream requests are sent to the bucket. Reduce the number of concurrent downstream requests.

Detailed information about alert notifications

Parameter Description Example
AvgSeverity The degree of throttling. The greater the value is, the higher the latency becomes. Valid values: 0 to 100. 10
QosType The type of the throttling that is triggered. Valid values:
  • IngressBandwidth: the upstream bandwidth.
  • EgressBandWidth: the downstream bandwidth.
  • Qps: the number of requests per second.
IngressBandwidth
TrafficSource The source of the traffic that triggers the throttling. Valid values:
  • intranet: requests that are sent over the internal network.
  • extranet: requests that are sent over the Internet.
  • net_all: requests that are sent over the internal network and the Internet.
net_all