CDN:Analyze user access data with real-time CDN logs

Overview

The real-time log feature, a collaboration between Alibaba Cloud CDN and Simple Log Service (SLS), provides low-latency log processing. This feature pushes user access logs from Alibaba Cloud CDN points of presence (POPs) to Simple Log Service with a typical latency of about 3 minutes. You can then use SLS to store and analyze user access data. When you use Alibaba Cloud CDN to accelerate your website, it records access logs for resources like images and videos. You can analyze these logs to understand your user base, monitor access quality, or quickly identify and resolve issues.

Prerequisites

• You have activated Alibaba Cloud CDN and added an accelerated domain name. For more information, see Activate Alibaba Cloud CDN.

• You have activated Simple Log Service. For more information, see Get started with Simple Log Service.

• You have configured real-time log delivery for the accelerated domain name whose user access data you want to analyze. For more information, see Configure real-time log delivery.

Differences between real-time logs and offline logs

• Log latency

  Real-time logs have a latency of three minutes or less. In contrast, offline logs typically have a data latency of up to 24 hours.

• Log analysis

  The real-time log feature integrates the log storage and analysis capabilities of Simple Log Service. It offers four preset analysis reports: Basic Data, Access Errors, Frequently Requested Resources, and User Composition. The feature also supports custom log analysis, providing a one-stop solution for log storage and analysis. In comparison, offline logs support only log shipping to Object Storage Service and do not have built-in analysis capabilities.

Benefits

• Low latency

  Log data is available within three minutes, which allows you to quickly analyze access logs, identify issues, and respond promptly.

• One-stop service

  Offline log analysis is multi-step: you must download logs, upload them to a data warehouse, and then cleanse and model the data before analysis. This process is time-consuming and labor-intensive. The real-time log feature integrates the storage and analysis capabilities of Simple Log Service to streamline this workflow.

Limits

• By default, a dedicated Logstore is of the Standard type and has a data storage duration of 7 days. You cannot write other data to the Logstore. Features such as query and analysis, alerting, and consumption do not have special limits.

• You can modify the indexes of a dedicated Logstore. If you cannot modify the indexes, contact technical support.

• Currently, you can push logs only to Simple Log Service Projects in the following regions: China (Hangzhou), China (Shanghai), China (Qingdao), China (Beijing), China (Shenzhen), China (Hong Kong), and Asia Pacific SE 1 (Singapore).

Use cases

Real-time logs help you analyze issues with accelerated domain names and understand user access patterns. Alibaba Cloud CDN provides pre-built log analysis reports and supports custom analysis policies.

Pre-built log analysis reports

On the real-time log page, find the Project that you want to analyze and click View Report to open the report template.

The data query template page displays data for all domain names by default. You can also filter by a specific domain name or URI.

For more information, see the following topics:

• Pre-built report: Basic data

• Pre-built report: Access errors

• Pre-built report: Frequently requested resources

• Pre-built report: User composition

• Subscribe to report template data

Custom log analysis

If the pre-built reports do not meet your needs, use Simple Log Service (SLS) for custom log analysis.

For example, you can view rankings of domain names that return the HTTP status code 499 or 502.

On the real-time log page, find the Project whose logs you want to analyze, and then click Log analysis to go to the custom log analysis page.

On the custom analysis page, enter a query statement in the search box for complex queries. You can also click a log field in the Raw Logs panel on the left to filter logs directly.

In addition, you can use the time selector in the upper-right corner to set the time range for your log query, such as the last 15 minutes. Filterable index fields include client_ip, content_type, domain, hit_info, method, and return_code.

For more information, see Custom reports.

Create a real-time log delivery Project

You can create a Simple Log Service Project to store real-time logs for a specific accelerated domain name, such as aliyun.example.com. For more information, see Configure real-time log delivery.

After the Project is created, the Project name is project-example, the Logstore name is project-example, and the log storage region is China (Hangzhou) (cn-hangzhou).

Pre-built report: Basic data

Provides key metrics such as cache hit ratio, access latency, and download speed to help you assess overall service quality and address issues promptly.

This report includes the following data. You can view the overall data or filter by domain name or URI.

• Health: The percentage of requests that have normal HTTP status codes.

• Cache hit ratio: The average cache hit ratio, calculated by byte.

• Download speed: The average download speed of resources.

• Access status: The percentage of each HTTP status code. This helps you quickly view the proportion of error status codes.

• Access latency distribution: The percentage of requests in each latency range.

• Bandwidth: The bandwidth value at a 1-minute granularity.

• PV/UV: The number of page views and unique visitors.

• Request hit ratio: The cache hit ratio calculated by the number of requests instead of by byte.

• Access latency: The average access latency for downloading a resource.

Pre-built report: Access errors

Helps you pinpoint the cause of access errors, such as problems with specific URIs, origin server failures, unavailable POPs, or network issues in certain regions or ISPs.

This report includes the following data. You can view the overall data or filter by domain name or URI.

• Top 10 error domain names: A ranking of top domain names by the percentage of access errors.

• Top 10 error URIs: A ranking of top URIs by the percentage of access errors.

• Request error percentage: The percentage of 4xx and 5xx HTTP status codes over time.

• Error request distribution: The count and percentage of each HTTP status code.

• Errors by ISP: The number of 4xx and 5xx HTTP status codes for each ISP.

• Errors by province: The number of 4xx and 5xx HTTP status codes for each province.

• Error details (4xx): The count and percentage of 4xx HTTP status codes by province and ISP.

• Error details (5xx): The count and percentage of 5xx HTTP status codes by province and ISP.

• Error distribution by client: The count and percentage of 4xx and 5xx HTTP status codes corresponding to different client User-Agents.

Pre-built report: Frequently requested resources

Identifies your most popular resources by domain name, URI, province, and ISP. You can also use this data to validate campaigns and check whether traffic growth during peak periods meets expectations.

This report includes the following data. You can view the overall data or filter by domain name or URI.

• Top domain names by visit: A ranking of top domain names by their percentage of total visits.

• Top domain names by download traffic: A ranking of top domain names by their percentage of total download traffic.

• Frequently requested URIs: Statistics for each URI, including visit count, number of unique visitors, and total download volume.

• Popular access sources: A ranking of popular Referer source domains, along with their visit count, UV count, and percentage.

• Visits across the Chinese mainland: The average number of visits for each province.

• Download speed across the Chinese mainland: The average download speed for each province.

• Statistics by province: The total number of visits, total download traffic, and average download speed for each province.

• Traffic and download speed by ISP: The total download volume and average download speed for each ISP.

• Statistics by ISP: The total number of visits, total download traffic, and average download speed for each ISP.

Pre-built report: User composition

Provides insight into your user base by client type, province, and ISP, and identifies top users by visit count and download volume.

This report includes the following data. You can view the overall data or filter by domain name or URI.

• PV: Total number of page views.

• UV: Total number of unique visitors.

• Source region distribution: The number of visits and their percentage for each province.

• Visits by client: The number of visits and their percentage for each client type.

• Visits by ISP: The total number of visits and their percentage for each ISP.

• Top users by download volume: A breakdown by IP address, showing total visits, error visits, and total download volume.

• Top users by valid visits: A ranking of top users by the number of valid visits (requests not returning 4xx or 5xx status codes). The report breaks down data by IP address, showing total visits, error visits, and download volume.

Subscribe to report template data

To receive report data from Simple Log Service on a regular schedule, use the subscription feature.

Procedure

1. Take the CDN Basic Data page as an example. Click Subscribe in the upper-right corner, and then click Create. On the CDN Log Management page, select the CDN Basic Data tab, and click the Subscribe button in the upper-right corner.

2. In the slide-out configuration panel, enter the Subscription Name, Frequency, and Global Time, and then click Next. On the Subscription Configuration tab, enter a Subscription Name (for example, "CDN Basic Data Report"), set the Frequency (such as Daily) and an execution time, select a Global Time (Preset or Custom), and enable the Add Watermark switch as needed.

3. From the Notifications drop-down list, select a notification method, enter the required information, and then click Submit.

   Supported notification methods include Email, WebHook-DingTalk Bot, WebHook-Feishu Chat Bot, WebHook-WeCom Bot, and WeChat.

Custom reports

• Example 1: View the ranking of domain names by the count of 499 HTTP status codes in the last 30 days.

  Log analysis statement: return_code = 499| select domain , count(*) as c group by domain order by c desc limit 10

• Example 2: View the ranking of domain names by the count of 502 HTTP status codes in the last 30 days.

  Log analysis statement: return_code = 502| select domain , count(*) as c group by domain order by c desc limit 10

• Example 3: View log data for the URI /cpu in the last 30 days.

  You can click the URI field in the Raw Logs panel on the left and then click /cpu to view matching logs.

  After filtering, the Quick Analysis panel shows that requests for /cpu account for 53% of the total requests, with a total of 1,556,698 log entries. The raw log details show fields such as return_code, client_ip, and method.