Alibaba Cloud Tech Share - Troubleshooting Alibaba Cloud OSS Development Problems Using Wireshark
Created#More Posted time:Jan 6, 2017 13:32 PM
This article is for $500 USD Coupon - Alibaba Cloud Tech Share
The cloud storage era has come, developers need to transition to the cloud. Many developers complained that they encountered problems when utilizing OSS. However, most of these problems turned out to be some basic problems, such as the incorrect entry of the endpoint domain name, bucket name and object name and incorrect authorization calculation. In fact, these problems are easy to resolve but simply pasting problematic code without describing actual problems makes it difficult. For this reason, the following provides my experience in resolving such problems.
First, let me introduce myself briefly. The company that I work for is an entrepreneurial company and its products are mainly designed for implementing cross-country internet acceleration and network QoS protection during data transmission. Therefore, our business involves the fields with massive data transmission and exchange. Our company started to get access to public cloud services and also Alibaba Cloud OSS from 2016. At that time, we wanted to test if it's possible to apply our transmission acceleration products to OSS.
I am a C/C++ programmer, but Alibaba Cloud OSS provides no C/C++ SDK. As a result, I had to turn to official HTTP API manuals for Alibaba Cloud OSS to become familiar with the product, and this required significant effort as I knew little about Web and HTTP protocols. However, considering my development background in the digital communication industry, it's rational for me to read the RFC standard and the official API documentation for the product. This took me 2 days to go through the API documentation while testing all the interfaces with the officially-provided python osscmd tool. Also, I captured HTTP packets from each OSS interface using Wireshark and reviewed the contents of each OSS HTTP packet. After that, I had a solid grasp of OSS.
When I look back to that time, there was sufficient quality and quantity of Alibaba Cloud OSS documentation, as you could find example HTTP packets and details for each interface. However, the complaints about the product in the forum seemed to be from developers having a high expectation on Alibaba Cloud without actually looking into OSS, instead hoping for OSS integration through simply copying or revising the provided example code. Given that the technical system of OSS cloud storage is quite different from that of previous file system storage products, developers must learn about certain required concepts to get started with OSS.
The following describes the details about the solution: Troubleshooting Alibaba Cloud OSS development problems using Wireshark.
Wireshark is an open-source packet capturing tool for Windows. This tool can be downloaded on many websites and its GUI is very user-friendly.
For OSS HTTP interfaces, OSS API Document.
For any Alibaba Cloud OSS operations, they are eventually represented by the interaction between clients and Alibaba Cloud OSS. The operations that I mention here refer to general PutObject/HeadObject operations but not programming specific SDK APIs.
The following describes an example of capturing a packet using Wireshark, where the packet involves two operations:
Operation 1: Obtain the list of objects under the specified bucket by executing GET BUCKET. This operation is successfully responded to by Alibaba Cloud OSS.
Operation 2: Obtain the property information about the specified object by executing HEAD OBJECT. This operation fails with a 403 failure response.
Packet requesting diagram of operation 1
From the diagram of operation 1, we can see "200 OK" from the returned message and this means that the operation has succeeded.
Packet requesting diagram of operation 2
From the diagram of operation 2, we can see that the "403 Forbidden" error was returned after executing HEAD OBJECT.
The packet capturing comments above are for describing how to troubleshoot an actual problem by capturing packets when developing cloudfs in C++.
The phenomenon of the problem is that thousands of files were attempting to perform the head object operation while the file named libc++-6.dll failed with an error. By capturing packets, I found that the error returned by OSS is "403 Forbidden", which normally means an object is inaccessible as OSS fails to locate the name of the desired object. With this in mind, I found that the major difference between the name of the problematic file "libc++-6.dll" and that of other files lies in the dash ("-") sign. By searching for the usage of the dash sign and reviewing the OSS API manual, I found that the object name must be encoded with UTF-8 and the URL must be escaped so that the problem can be resolved.
However, you can ignore such problems as long as you are using the official SDKs provided by Alibaba Cloud such as Python, Java and .Net, which already eliminate those problems within the SDKs.
Despite that, you can effectively troubleshoot a problem when invoking the official SDKs by capturing packets to learn about the interaction process and the returned OSS response. If this doesn't resolves your problem, you can also paste the information about the captured packets in the forum so that others can provide you with more helpful information.
I hope that sharing the method above can help other developers who want to utilize OSS cloud storage but have thus far failed to achieve it.
[Blanche edited the post at Jan 6, 2017 14:36 PM]
1st Reply#Posted time:Mar 15, 2017 21:56 PM
This may help you to a certain extent!