Security and Monitoring Practices – Alibaba Cloud Storage Solutions

Part 3 of this article series will continue discussing error diagnosis, using network logging tools, E2E tracking/diagnosis, troubleshooting, and OSS sandboxing.

By Shantanu Kaushik

Cloud storage plays an important role in every aspect of cloud computing. Storage is the basis of any successful application deployment, database solution, information sharing, or any other cloud storage-based scenario. In the previous article of this series, we discussed the need for proper storage optimization and the cloud monitoring solution by Alibaba Cloud.

Optimizations enroll the need for Cloud Monitor to gather important metrics and make adjustments to your cloud service for a better and more productive event cycle. In this article, we will continue discussing error diagnosis, using network logging tools, E2E tracking/diagnosis, troubleshooting, and OSS sandboxing.

Diagnosing Errors

Storage errors could boil down to a large-scale outage. To prevent this, you need to be as proactive as possible with error diagnosis, including server return with error information to the client when the client-side applications are faulty. Alibaba Cloud Monitor records these errors and displays detailed reports to outline and showcase the complete error information when required and requested.

You can always retrieve the information for specific errors by filtering for the required data type. You can also include individual requests from server log, network log, and client log. The best way to diagnose an error is to combine the error information, including the Object Storage Service (OSS) error code, HTTS status code, and OSS error detail.

You can read more about OSS error responses here. This will give you a detailed explanation of error codes and possible causes.

Logging

The Object Storage Service (OSS) provides server and network logging. You can log user requests to track end-to-end requests and the details associated with these requests. Various issues can be backtracked using the network logging function. You can retrieve the user application data, storage, and network logs. These logs can provide detailed information to assist with diagnosing issues related to traffic conditions between the server and the client.

End-to-End Tracking

An end-to-end process starts with a client request that is processed over a network and sent to the Alibaba Cloud OSS server for processing and execution. This end-to-end tracking process ensures the diagnosis of potential problems and troubleshooting.

Troubleshooting

Let’s outline some common performance issues and their solutions:

1. Low End-to-End Latency (Average)

There could be an issue where end-to-end latency is low, but the client request latency is too high. In this scenario, the client-side will experience request delay. This could be due to:

Multiple Client Requests for the Same Process
If the same process request is executed multiple times from the client-side, it will cause request delay and the user will experience high-request latency.
Multiple Requests With Less Execution Resources
If there are multiple requests in the pipeline and not enough thread to process them, the client-side will suffer from latency issues.

2. High Average End-to-End Latency

High end-to-end latency may be caused by the slow performance of client applications. This could be the result of slow network speeds or a limited number of available connections.

Solution

Check to see if there are a large number of connections in the using TIME_WAIT. If there are, you can adjust the core parameters to overcome this issue. Alternatively, when the number of available connections is insufficient, you need to address the parameters that might be affecting the client CPU, memory, and network resources. You should increase the number of threads or connections to overcome this issue.

You can also try and optimize the client-side application and adjust the application configuration to implement an asynchronous access method. You can analyze the application performance and optimize it if necessary.

Insufficient Resources

System monitoring is a vital tool to find a cause and fix it. You can analyze and pinpoint the resource on the client-side that is causing the bottlenecks. The second step would be to optimize the resource usage or increase client resources, such as CPU, memory, and network bandwidth.

Network Latency

High end-to-end latency due to network factors is generally short-lived. You can use tools like Wireshark to investigate these network problems and fix them.

Sudden and Outrageous Increases in Storage Capacity

If you notice a sudden spike in storage capacity and you cannot find a valid reason, such as an increase in upload requests, it could be due to:

Regular Storage Optimization to Free Up Space
Check to see if the valid request rate has decreased. Continuously failed delete requests could cause issues with storage object deletion and increase the storage. You need to pinpoint the specific cause for this by analyzing the request error types and comparing client logs to gather detailed error information.
When a Lifecycle Policy is Used to Delete Storage Objects
You can use the OSS console or call an API to check if the bucket lifecycle values have changed. If not, you can modify the configuration based on the data gathered using the server logs.

Object Storage Service (OSS) Sandbox

When your Alibaba OSS bucket comes under attack, the Object Storage Service automatically sends the bucket to a sandbox. Let me explain: Here, Sandbox is an isolation mechanism for buckets that are under attack or are compromised. It is a part of the safety operations developed by Alibaba Cloud to ensure that your operations and data will remain safe, even if a bucket has been compromised.

If your bucket is under attack, OSS automatically adds the attacked bucket to the sandbox. The bucket in the sandbox can still respond to requests, but the service quality of your application will not be normal.

To prevent the Object Storage Service from adding your bucket to the sandbox, you can use Anti-DDoS Pro.

Quick Tip: A distributed denial of service (DDoS) attack uses a multiple computer system to launch a coordinated and parallel attack regime against one or more targets through malicious programs. The attack undermines the performance and consumes network bandwidth to make the target servers unresponsive.
Alibaba Cloud Anti-DDoS service prevents this from happening. I will write a more comprehensive article on Alibaba Cloud Anti-DDoS Service soon. The architecture of Alibaba Cloud Anti-DDoS Premium and Pro Service architecture are shown below.

Another solution is to configure a reverse proxy using Alibaba Cloud Elastic Compute Service (ECS) and configure the Anti-DDoS Pro instance. The IP address resolved from the default domain of a bucket changes dynamically for added security measures. To use a fixed IP address to access the bucket, an ECS instance can be used to set up a reverse proxy and attach the elastic IP of an ECS instance to an Anti-DDoS Pro instance to ward off DDoS attacks and HTTP floods.

Wrapping Up

Cloud storage is highly elastic and offers enhanced performance when compared to traditional storage practices. When undergoing a technological paradigm shift, the inclusion of more agile practices is required. Storage plays a crucial role. Maintaining that storage by monitoring it closely and troubleshooting problems is highly recommended.

When it comes to containerization and microservices with DevOps, a lot of things depend on storage. Alibaba Cloud has devised architectures to support and uplift the overall service quality to ensure uninterrupted service and high-scalability usage scenarios.

Upcoming Articles

Apsara File Storage NAS – What and How?

We will discuss the complete architecture and usage scenarios of the Apsara File Storage NAS solution by Alibaba Cloud.

Community

Security and Monitoring Practices – Alibaba Cloud Storage Solutions – Part 3

Diagnosing Errors

Logging

End-to-End Tracking

Troubleshooting

Solution

Insufficient Resources

Network Latency

Sudden and Outrageous Increases in Storage Capacity

Object Storage Service (OSS) Sandbox

Wrapping Up

Upcoming Articles

Read previous post:

Read next post:

Alibaba Clouder

You may also like

Comments

Alibaba Clouder

Related Products

Quick Starts

Hybrid Cloud Distributed Storage

OSS(Object Storage Service)

Storage Capacity Unit