Problem description
After enabling Alibaba Cloud CDN, your website is slow. Slow performance can be caused by many factors. This topic describes how to analyze, diagnose, and resolve the issue to optimize your website's speed.
Solution
Alibaba Cloud reminds you:
Before performing risky operations, such as modifying instances or data, ensure you have disaster recovery and fault tolerance policies to protect your data.
Before modifying the configuration or data of an instance (such as an ECS or RDS instance), we recommend creating a snapshot or enabling RDS log backup.
If you have submitted your logon credentials to Alibaba Cloud, we recommend that you change your password promptly.
If you experience slow website access after enabling Alibaba Cloud CDN, follow the steps in this topic to resolve the issue.
Analysis approach
Before you troubleshoot and analyze issues, you need to understand the CDN acceleration principle. For more information, see What is Alibaba Cloud CDN. This will help you think about and analyze the possible causes of a problem. In simple terms, CDN works primarily by adding a new layer of cache nodes to the existing network and publishing resources from your web server to the network nodes closest to your users. This allows clients to directly access the nearest CDN node and hit the resource, which reduces back-to-origin requests and improves website access speed. Therefore, the possible causes of slow access can be summarized into the following categories:
Client-side network factors, such as insufficient download bandwidth or incorrect DNS configuration.
Poor network connection and high latency between the client and the CDN node.
A malfunctioning CDN node that responds slowly.
Large resource files that take a long time to download.
Poor network quality during back-to-origin requests.
Slow response from the origin server itself.
Gathering information helps you narrow down potential causes and focus your investigation.
First, determine if the slow access is a global issue or if it affects only individual users in a specific region or on a particular ISP. Alibaba Cloud provides tools like Application Real-Time Monitoring Service (ARMS) and Cloud Monitor. You can use these services to configure monitoring tasks from specific regions and ISP networks for more accurate results:
If only a few users experience slow performance, the issue is likely related to their local network.
Check if the affected users are concentrated in a specific area. For example, if a large number of China Mobile users in a city experience slow access while China Unicom and China Telecom users in the same city do not, the issue may be related to the local ISP's network. Use network monitoring tools to run tests from machines in that region.
It is highly unlikely that all CDN nodes or regional networks would fail simultaneously, so focus on issues like an incorrect acceleration region, dynamic or non-cacheable requests, or a slow origin server.
Determine if slow or abnormal requests are served from the CDN cache.
If a request hits the CDN cache, no back-to-origin request occurs. The CDN serves the cached data directly to the client. In this case, the origin server is not involved.
If a request misses the cache, you need to determine whether the bottleneck is the connection from the client to the CDN or the response time of the origin server.
Key metrics
Monitoring these metrics helps you evaluate CDN effectiveness, understand your business's service usage, and make better decisions. For a detailed introduction to CDN metrics, see CDN metrics.
Information gathering
A complete HTTP request involves several steps: DNS resolution > TCP connection establishment > SSL handshake (for HTTPS) > client sends request > server responds to request. Understanding this process is key to an in-depth analysis. You must gather the following information from the client.
Collect client network information and the CDN node IP address
Use the ping command to test the accelerated domain name. This verifies that the domain resolves correctly to the CDN and checks the network connectivity and latency between the client and the CDN node. If you cannot ping the domain, you need to perform further link diagnostics. For more information, see Link diagnostic methods.
Collect the client IP address and Local DNS
Therefore, ensure the client's Local DNS is configured correctly. Visit the Ali Kunlun User Diagnosis Tool to get the client IP address and Local DNS server.
Find the slow URL
Open your browser's developer tools and go to the Network tab. After entering the URL, the Network tab displays all HTTP requests made by the browser. Click the Time column to sort requests by duration and identify which resources are loading slowly. In the Domain column, find the slow URLs under your CDN-accelerated domain name.
A website often loads resources from multiple sources. Slow-loading resources not accelerated by CDN can slow down your entire website, even if CDN-accelerated content loads quickly. Sort by the Time column to pinpoint which specific URLs are slow.
Collect the HTTP request and response headers
In the Network tab, click the Name of a slow HTTP request. The Headers tab shows the General, response headers, and request headers information for this request. These headers can tell you if the request was for a static resource and whether it hit the cache.
General
Request URL: http://xxx
Request Method: GET
Status Code: 200 OK
Remote Address: 114.80.187.101:80
Referrer Policy: no-referrer-when-downgrade
Response Headers
Ali-Swift-Global-Savetime: 1578328423
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Mon, 06 Jan 2020 16:33:43 GMT
EagleId: 7250bb18157832842266571547e
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Server: TengineCapturing network packets on a mobile device is difficult. If you experience slow performance on a 4G network, a workaround is to enable the mobile hotspot, connect a PC, and gather the required information there.
Collect HTTP request Timing information
The Timing tab shows the time spent in each phase of the request lifecycle.
The phases in the Timing tab include: Resource Scheduling (Queueing), Connection Start (Stalled), and Request/Response (Request sent, Waiting (TTFB), Content Download). If Content Download takes the most time (for example, 10.61s out of a total of 12.77s), it indicates the download phase is the bottleneck.
Common scenarios
By understanding CDN acceleration principles and the HTTP request process, you can analyze symptoms and use the collected client-side information to identify and locate issues. The following sections describe some common problem scenarios.
Scenario 1: Poor client-to-node network quality
The client uses the ping command to test the accelerated domain and finds high network latency or even packet loss. In this case, collect the client's IP address, DNS server, and screenshots of ping and MTR results. Because CDN routes requests based on the client's DNS, you can use the client IP address, DNS, and CDN node information to determine if routing is abnormal. The ping and MTR screenshots show network latency and pinpoint which network link is causing the delay. This section introduces two examples.
Incorrect acceleration region setting
Users in the Chinese mainland are routed to overseas nodes, or overseas users are routed to nodes in the Chinese mainland. This can happen in the following scenarios:
In this case, we recommend setting the acceleration region to "Global".
If the acceleration region for the CDN is set to "Chinese Mainland Only", the domain's service area is limited to CDN nodes within the Chinese mainland. When overseas users access the domain, they will be routed to a CDN node inside the Chinese mainland.
If the acceleration region is set to "Global (Excluding Chinese Mainland)", the domain's service area includes only CDN nodes outside the Chinese mainland. Users in the Chinese mainland will be routed to overseas CDN nodes.
The choice of acceleration region is restricted if your domain name has not completed ICP filing. An ICP filing is required to select "Chinese Mainland Only" or "Global" as the acceleration region. Domains without an ICP filing can only select "Global (Excluding Chinese Mainland)", which provides acceleration through overseas CDN nodes. When users in the Chinese mainland access content through overseas nodes, the long network path can result in poor access speeds.
For scenarios where you need to accelerate content for users in the Chinese mainland with a domain that does not have an ICP filing, consider using Edge Security Acceleration (ESA) instead of CDN. ESA supports domains without ICP filing and uses network optimization to reduce cross-border access latency. For details, see Overseas cross-region security acceleration.
Incorrect client DNS settings
If the client's DNS is set incorrectly, the user needs to change it to the DNS server corresponding to their location and ISP:
A China Mobile user in Guangdong who uses a China Unicom DNS server will be routed to a China Unicom CDN node. This long-distance routing increases network latency.
A China Mobile user in Guangdong who uses a China Mobile DNS server from Harbin will be routed to a CDN node in Harbin, which also increases network latency due to the long distance.
If the acceleration region and DNS are set correctly, but network quality is still poor despite proper CDN routing, collect traceroute and MTR information for further diagnosis.
Scenario 2: Low cache hit ratio or frequent back-to-origin
In static resource acceleration, CDN caches static resources on nodes closest to the client. When a user requests a resource, it is served directly from the cache, avoiding long-distance back-to-origin requests. Therefore, the cache hit ratio directly impacts user experience. Maintaining a high ratio is a core CDN objective. You can optimize the CDN cache hit ratio by addressing its specific causes. For more information, see Improve the CDN cache hit ratio. You can use the X-Cache field in the CDN response header to determine if a request hit the cache.
The following is an example of an HTTP response header for a CDN cache miss. The X-Cache field is MISS and the X-Swift-CacheTime is 0, indicating that the resource was not cached by the CDN:
Cache-Control: no-store, no-cache, must-revalidate
Connection: keep-alive
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Mon, 06 Jan 2020 16:33:36 GMT
EagleId: 7250bb1d15783284155045411e
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Pragma: no-cache
Server: Tengine
Timing-Allow-Origin: *
Transfer-Encoding: chunked
Vary: Accept-Encoding
Vary: Accept-Encoding
Via: cache46.l2cn1829[954,200-0,M], cache5.l2cn1829[955,0], kunlun1.cn2364[3814,200-0,M], kunlun9.cn2364[3816,0]
X-Cache: MISS TCP_MISS dirn:-2:-2
X-Swift-CacheTime: 0
X-Swift-SaveTime: Mon, 06 Jan 2020 16:33:39 GMTX-Cache: If the value is MISS, the request was not found in the cache and a back-to-origin request was required. If the value is HIT, the request was served from the CDN cache.
X-Swift-CacheTime: This field indicates the allowed cache duration on the CDN node, meaning how long the file can be cached. A value of 0 indicates that the request cannot be cached.
The following are common causes and optimization strategies for a low cache hit ratio or frequent back-to-origin requests:
A resource's first access can be slow because the CDN node must fetch the uncached content from the origin server. In this case, we recommend using the URL prefetch feature. See Refresh and prefetch resources. This proactively pushes content from your origin server to CDN nodes, so users can get a cache hit on their first visit, improving load times.
If a resource is not accessed frequently, it may not be effectively cached. CDN nodes are shared resources for all CDN users, and a configured cache rule only specifies the maximum cache duration for a resource. If your accelerated domain has low traffic, its files may be evicted from the cache prematurely. Caches often operate on a least-recently-used (LRU) basis, where popularity is determined by access frequency. Unpopular files are evicted sooner.
Improper cache configurations and short cache times can cause frequent back-to-origin requests. Scenarios include:
If no cache rules are configured on the CDN and a static file does not return ETag and Last-modified response headers, the file cannot be cached on CDN nodes. The solution is to configure these response headers on the origin server or set up cache rules in the CDN. For more information, see Configure cache policies.
If no cache rules are configured, CDN uses its default cache policy. For details, see Alibaba Cloud CDN default cache rules and priorities. The default cache duration is short (no more than 3600 seconds), which can lead to frequent expiration and back-to-origin requests. We recommend setting a reasonable cache duration in the CDN based on your business needs.
If the origin server sends headers that enforce no caching, CDN will not cache the resource, even if you have configured cache rules. This is because these headers override CDN cache rules. If the origin server sends any of the following directives: "s-maxage=0", "max-age=0", "no-cache", "no-store", "private", or "Pragma: no-cache", the CDN cannot cache the content. You must modify these headers on your origin server to allow caching, for example, by using "Public". For details, see Set Nginx HTTP cache policies and Set Apache cache policies.
The URL contains variable parameters. If the URL for a resource contains parameters that change with each request, the CDN treats each unique URL as a new request. This happens even if the different URLs point to the same file that is already cached. The CDN will still fetch the content from the origin server. To prevent this, enable the ignore parameters feature. For more information, see ignore parameters.
Range back-to-origin for large files. For caching large files, we recommend enabling the Range back-to-origin feature to optimize back-to-origin requests. For more information, see Configure Range back-to-origin.
Scenario 3: Slow dynamic requests
For dynamic requests, a CDN node simply forwards the request to the origin server, providing no acceleration. If your website or application has a lot of dynamic content, such as API calls, consider the following solutions.
Implement separation of dynamic and static content on your origin server. Use a CDN domain to accelerate static resources, and use a separate domain that resolves directly to the origin server for dynamic requests.
Consider using Edge Security Acceleration (ESA) to accelerate dynamic requests. For more information, see What is ESA?. Note that ESA accelerates dynamic requests through Layer 4 link optimization, such as route and transport optimization, to fetch data from your origin server as fast as possible. However, if your origin server itself is slow to respond, you will still need to optimize the origin server.
Scenario 4: Slow cross-border back-to-origin
When your origin server and primary user base are in different countries or regions, CDN performance may be affected by submarine cable capacity and public network congestion. This can lead to fluctuations in back-to-origin quality, causing slow retrieval of original content from the origin server and high back-to-origin failure rates, which in turn affects CDN caching and content delivery. We recommend using Global Accelerator (GA) to accelerate traffic to your origin server for specific countries and configuring regional back-to-origin policies in your CDN. This improves back-to-origin quality for cache misses.
Based on your origin server type, refer to the following documents to configure GA acceleration:
To accelerate an ECS origin server, see Use GA with CDN to accelerate back-to-origin.
To accelerate an ALB origin server, see Use GA to accelerate applications behind an ALB.
To accelerate an OSS origin server, see Access OSS through transfer acceleration.
Scenario 5: Slow homepage loading
When a client visits a website like http://www.example.com/, the browser requests the homepage. Upon a successful response, the server returns HTML code to the browser. The browser then parses this code and requests additional resources it references, such as images, JS, and CSS files. If the homepage is a dynamic or non-cacheable resource, every request for it will trigger a back-to-origin request from the CDN. If the origin server responds slowly, the homepage loads slowly, and the request remains in a "Pending" state in the browser's Network tab. To determine if the request is hitting the cache, see Scenario 2: Low cache hit ratio or frequent back-to-origin.
In this scenario, where an uncached homepage loads slowly, the primary symptom is that the homepage request remains in a "Pending" state. Once the homepage data is finally received, the static resources will load quickly.
Scenario 6: Loading large resources
If your website loads large resources, you can enable performance optimization features for your accelerated domain. For more information, see Performance optimization. These features can reduce file sizes, improving acceleration and page readability. Intelligent compression currently supports the following content formats: text/html, text/xml, text/plain, text/css, application/javascript, application/x-javascript, application/rss+xml, text/javascript, image/tiff, image/svg+xml, application/json, and application/xmltext.
Scenario 7: Slow access for specific regions or ISPs
Sometimes, performance issues are specific to a group of users with common characteristics. For example, during a specific time period, a large number of China Mobile users in a certain city report slow or abnormal access, while China Unicom and China Telecom users are unaffected. This type of problem is likely related to the local ISP's network or the specific CDN node that these users are routed to. The typical troubleshooting method is to collect ping information from the affected users to first check the network latency between the client and the CDN node.
Based on the CDN node IP address that the user is routed to, you can bind your hosts file to that node for testing. This method is similar to binding to an origin server for testing; just replace the IP address with the CDN node's IP. Before testing, first verify whether the node itself is responding slowly. If it is, check if the request is hitting the cache and if the resource being loaded is too large. Analyze the issue using the scenarios described above. If you cannot identify the problem, submit a ticket to Alibaba Cloud.