This topic describes how to use the real-time log feature to analyze user access logs.
Overview
The real-time log feature, a collaboration between Alibaba Cloud CDN and Simple Log Service (SLS), provides low-latency log processing. This feature pushes user access logs from Alibaba Cloud CDN points of presence (POPs) to Simple Log Service with a typical latency of about 3 minutes. You can then use SLS to store and analyze user access data. When you use Alibaba Cloud CDN to accelerate your website, it records access logs for resources like images and videos. You can analyze these logs to understand your user base, monitor access quality, or quickly identify and resolve issues.
Prerequisites
You have activated Alibaba Cloud CDN and added an accelerated domain name. For more information, see Activate Alibaba Cloud CDN.
You have activated Simple Log Service. For more information, see Get started with Simple Log Service.
You have configured real-time log delivery for the accelerated domain name whose user access data you want to analyze. For more information, see Configure real-time log delivery.
Differences between real-time logs and offline logs
Log latency
Real-time logs have a latency of three minutes or less. In contrast, offline logs typically have a data latency of up to 24 hours.
Log analysis
The real-time log feature integrates the log storage and analysis capabilities of Simple Log Service. It offers four preset analysis reports: Basic Data, Access Errors, Frequently Requested Resources, and User Composition. The feature also supports custom log analysis, providing a one-stop solution for log storage and analysis. In comparison, offline logs support only log shipping to Object Storage Service and do not have built-in analysis capabilities.
Benefits
Low latency
Log data is available within three minutes, which allows you to quickly analyze access logs, identify issues, and respond promptly.
One-stop service
Traditional offline log analysis is a multi-step process: you must download logs, upload them to a data warehouse, and then cleanse and model the data before analysis. This process is time-consuming and labor-intensive. The real-time log feature integrates the storage and analysis capabilities of Simple Log Service to streamline this workflow.
Use cases
You can use real-time logs to troubleshoot issues with your accelerated domain names and to understand user access patterns. The real-time log feature of Alibaba Cloud CDN provides preset reports and supports custom analysis to meet various requirements.
Preset log analysis reports
Analysis scenario | Description |
CDN Basic Data | Helps you understand the overall service quality of Alibaba Cloud CDN and the access efficiency for end users, such as cache hit ratio, access latency, and download speed. This allows you to respond quickly to service quality issues. |
CDN Access Errors | Helps you quickly locate the source of access exceptions, such as issues with specific URIs, origin server failures, unavailable POPs, or network problems in certain regions or with specific ISPs. |
CDN Frequently Requested Resources | Provides insights into popular resources by identifying top domain names, URIs, provinces, and ISPs. This data helps you evaluate the effectiveness of marketing campaigns and determine if traffic during peak hours meets expectations, enabling you to adjust your operational strategy. |
CDN User Composition | Helps you understand the composition of your website's user base, including client types, provinces, and ISPs. This report can also identify top users by the number of visits or the amount of data downloaded. |
On the real-time logs page, find the Project that you want to analyze and click View Report to open the report template page.
By default, the report template page displays data for all domain names. You can also query data for a specific domain name or URI.
For more information, see the following topics:
Custom log analysis
If the preset reports do not meet your needs, you can use the powerful analysis features of Simple Log Service to perform custom analysis.
For example, you can rank domain names by the number of requests that returned an HTTP status code of 499 or 502.
On the real-time logs page, find the Project that you want to analyze and click Log analysis to go to the custom analysis page.
On the custom analysis page, you can enter a query in the search box for complex queries. For simple filtering, you can click log fields in the Raw Logs panel.
To the right of the search bar, you can specify a time range by using the time picker, for example, 15 Minutes (Relative). Then, click Query / Analysis to run the query. The Quick Analysis panel on the left lists filterable log fields, including client_ip, content_type, domain, hit_info, and method.
For more information, see Custom reports.
Create a real-time log delivery Project
You can create a Simple Log Service Project to store real-time logs for a specific accelerated domain name, such as aliyun.example.com. For more information, see Configure real-time log delivery.
After the Project is created, the Project name is project-example, the Logstore name is project-example, and the log storage region is China (Hangzhou) (cn-hangzhou).
Preset report: CDN Basic Data
This report helps you understand the overall service quality of Alibaba Cloud CDN and the access efficiency for end users, such as cache hit ratio, access latency, and download speed. This allows you to respond quickly to service quality issues.
This report provides the following data, which you can view in aggregate or filter by domain name or URI:
Health: The percentage of requests that returned a normal HTTP status code.
Cache hit ratio: The average cache hit ratio for resources, calculated by byte.
Download speed: The average download speed for resources.
Access status: The percentage of each HTTP status code, which helps you quickly check the proportion of abnormal status codes.
Access latency distribution: The proportion of requests in each latency range.
Request bandwidth: The bandwidth value at a one-minute granularity.
Page views/Unique visitors: The total number of page views (PV) and unique visitors (UV).
Request hit ratio: The cache hit ratio calculated by the number of requests.
Access latency: The average latency for downloading resources.
Preset report: CDN Access Errors
This report helps you quickly locate the source of access exceptions, such as issues with specific URIs, origin server failures, unavailable POPs, or network problems in certain regions or with specific ISPs.
This report provides the following data, which you can view in aggregate or filter by domain name or URI:
Top 10 domain names by error: The top domain names ranked by the percentage of access errors.
Top 10 URIs by error: The top URIs ranked by the percentage of access errors.
Request error percentage: The percentage of requests with 4xx and 5xx HTTP status codes over time.
Error request distribution: The count and proportion of each HTTP status code.
Errors by ISP: The number of requests with 4xx and 5xx HTTP status codes, grouped by ISP.
Errors by province: The number of requests with 4xx and 5xx HTTP status codes, grouped by province.
Error details (4xx): The count and proportion of requests with 4xx HTTP status codes, grouped by province and ISP.
Error details (5xx): The count and proportion of requests with 5xx HTTP status codes, grouped by province and ISP.
Error distribution by client: The count and proportion of requests with 4xx and 5xx HTTP status codes, grouped by the client User-Agent.
Preset report: CDN Frequently Requested Resources
This report provides insights into popular resources by identifying top domain names, URIs, provinces, and ISPs. This data helps you evaluate the effectiveness of marketing campaigns and determine if traffic during peak hours meets expectations, enabling you to adjust your operational strategy.
This report provides the following data, which you can view in aggregate or filter by domain name or URI:
Top domain names by visits: The top domain names ranked by the percentage of total visits.
Top domain names by download traffic: The top domain names ranked by the percentage of total download traffic.
Frequently requested URIs: The number of visits, unique visitors, and total downloads for each URI.
Popular access sources: The top Referer source domain names, with visit counts, unique visitors, and percentages.
Visits across the Chinese mainland: The average number of visits, grouped by province.
Nationwide download speed: The average download speed for each province.
Statistics by province: The total visits, total download traffic, and average download speed for each province.
Traffic and speed by ISP: The total download traffic and average download speed for each ISP.
Statistics by ISP: The total visits, total download traffic, and average download speed for each ISP.
Preset report: CDN User Composition
This report helps you understand the composition of your website's user base, including client types, provinces, and ISPs. This report can also identify top users by the number of visits or the amount of data downloaded.
This report provides the following data, which you can view in aggregate or filter by domain name or URI:
Page views: The total number of page views (PVs).
Unique visitors: The total number of unique visitors (UVs).
Source region distribution: The number and proportion of visits from each province.
Visits by client: The number and proportion of visits from each client type.
Visits by ISP: The number and proportion of visits from each ISP.
Top users by downloads: The total visits, error visits, and total downloads for each user IP address.
Top users by valid visits: The total visits, error visits, and total downloads for each user IP address, excluding invalid requests (those with 4xx and 5xx status codes).
Subscribe to report template data
You can use the subscription feature to have Simple Log Service periodically send you data from a report template.
Procedure
For example, on the CDN Basic Data page, click Subscribe in the upper-right corner, and then click Create. In the left-side navigation pane of the CDN console, choose Log Management. On the Log Reports page, click the Subscribe button in the upper-right corner.
In the pane that appears, configure Subscription Name, Frequency, and Global Time, and then click Next.
From the Notification List drop-down list, select a notification method, specify the required information, and then click Submit.
Supported notification methods include email, WebHook-DingTalk Bot, WebHook-Feishu Chat Bot, Webhook-WeCom Bot, and WeChat.
Custom reports
Example 1: View the top domain names with the most HTTP 499 status codes in the last 30 days.
Log analysis query:
return_code = 499| select domain , count(*) as c group by domain order by c desc limit 10Example 2: View the top domain names with the most HTTP 502 status codes in the last 30 days.
Log analysis query:
return_code = 502| select domain , count(*) as c group by domain order by c desc limit 10Example 3: View log data for the URI /cpu from the last 30 days.
In the Raw Logs panel, click the URI field, and then click /cpu to filter the logs.
After filtering, the search bar automatically generates the query
* and uri : "/cpu". The Quick Analysis panel on the left shows that the uri field with the value/cpuaccounts for 53% of the results.