This topic describes how to use the real-time log delivery feature to ship logs to Simple Log Service, and perform analysis on the logs in Simple Log Service.
Overview
The real-time log delivery feature is a log data processing service that is jointly developed by Alibaba Cloud CDN and Log Service. Featuring low-latency (typically within 3 minutes) shipping, you can use this feature to push access logs from points of presence (POPs) all over the world to Log Service. You can then use Simple Log Service to store and analyze user access data. Requests that flow through Alibaba Cloud CDN are logged, and a trove of data is provided for analysis. You can analyze these logs to gain insights into user composition and access speed, as well as to locate and troubleshoot content delivery issues.
Prerequisites
Alibaba Cloud CDN is activated and a domain name is added for acceleration. For more information, see Activate Alibaba Cloud CDN.
Simple Log Service is activated. For more information, see Getting Started.
Real-time log delivery is configured for the accelerated domain name for which you want to analyze user access data. For more information, see Configure real-time log delivery.
Differences between real-time log shipping and the offline log feature
Log latency
The real-time log delivery feature collects log data in real time, and logs are generated within 3 minutes after an event occurs. Offline log data is generated within 24 hours.
Log analysis
The real-time log delivery feature integrates the log storage and log analysis capabilities of Simple Log Service. The feature provides four preset analysis report templates, including templates for basic data, error analysis, frequently requested resources, and user analysis. The feature also supports custom log analysis policies. In comparison, the offline log feature only allows you to ship logs to Object Storage Service (OSS), and does not integrate log analysis capabilities.
Benefits
Low latency
Log data is generated in 3 minutes after an event occurs. This allows you to analyze access logs, identify issues, and resolve them in a timely manner.
End-to-end
Traditional offline log analysis solutions typically require you to first download logs, upload them to a data warehouse, and subsequently clean the data and provide a definition for the data model. Only then will you be able to perform data analysis. This results in high labor costs and extends the time that is required for analysis. The real-time log delivery feature integrates log storage and log analysis capabilities of Simple Log Service to eliminate the tedious process of the traditional offline log analysis.
Scenarios
You can use real-time log delivery to troubleshoot issues that may occur when you use Alibaba Cloud CDN to accelerate the delivery of your content. You can also use the collected log data to analyze user access statistics. The real-time log delivery feature of Alibaba Cloud CDN provides preset log analysis reports and supports custom log analysis policies to meet diverse requirements.
Preset log analysis reports
Log analysis report | Description |
Basic data | This report provides information about the overall performance of Alibaba Cloud CDN and the user access efficiency, such as hit ratio, access latency, and download speed. It also allows you to quickly identify and handle service quality issues. |
Access errors | This report helps you quickly locate issues when application access exceptions occur, such as URI issues, origin server failures, unavailable POPs, regional network issues, and Internet service provider (ISP) network issues. |
Frequently requested resources | This report provides information about frequently requested resources and helps you gain insight into popular domain names and URIs, as well as the provinces where most of your customers are located, and the ISPs of the customers. The data is also useful in helping you gain a better understanding of your business, such as whether operational activities attract the expected traffic volume, and whether the traffic during peak hours is higher or lower than expected. The information can help you make adjustments to the overall operations of your business. |
User composition | This report provides information about the user composition of your website, including the client or device, general location, and ISPs of the users. You can analyze top users with most visits or most downloads. |
On the Real-time Logs page, find the project for which you want to analyze logs, and click View Report.
On the data query template page, the data of all domain names is queried by default. You can also query the data of a specified domain name or URI.
For more information, see the following sections:
Custom log analysis
If the preset log analysis reports are unable to satisfy your requirements, you can use the log analysis feature of Simple Log Service to implement custom log analysis.
For example, you can view the ranking of domain names in requests whose HTTP status code is 499 or 502.
On the Real-time Logs page, find the project for which you want to analyze logs, and click Log analysis.
On the custom log analysis page, you can enter a query statement in the search box to query log data by using complex query conditions, or click log fields on the Raw Logs tab to filter logs by using simple filter conditions.
For more information, see Custom reports.
Create a real-time log delivery project
You can create a Log Service project to store real-time logs of an accelerated domain name, such as aliyun.example.com. For more information, see Configure real-time log delivery.
The following figure shows a created project, where the project name is project-example, the Logstore name is project-example, and the log storage region is China (Hangzhou) whose ID is cn-hangzhou.
Preset report: Basic data
This report provides information about the overall performance of Alibaba Cloud CDN and the user access efficiency, such as hit ratio, access latency, and download speed. It also allows you to quickly identify and handle service quality issues.
This report includes the following data. You can view all data, or filter data by domain name or URI.
Health: the proportion of requests with a normal HTTP status code
Cache hit ratio: the average cache hit ratio by the number of bytes
Download speed: the average download speed of resources
Access status: the percentages of HTTP status codes, including abnormal HTTP status codes
Access latency distribution: the proportion of each latency segment
Bandwidth: bandwidth data that is collected every minute
PV/UV: page views and unique visitors
Request hit ratio: the hit ratio by the number of requests
Access latency: the average latency for resource download
Preset report: Access errors
This report helps you quickly locate issues when application access exceptions occur, such as URI issues, origin server failures, unavailable POPs, regional network issues, and ISP network issues.
This report includes the following data. You can view all data, or filter data by domain name or URI.
Top 10 error domain names: the top 10 domain names with most access errors
Top 10 error URIs: the top 10 URIs with most access errors
Request error percentage: the percentage of HTTP 4xx or 5xx status code by time
Error request distribution: the number and proportion of each HTTP status code
Errors by ISP: the number of HTTP 4xx and 5xx status codes by ISP
Errors by province: the number of HTTP 4xx and 5xx status codes by province
Error details (4xx): the number and proportion of HTTP 4xx status codes by province and ISP
Error details (5xx): the number and proportion of HTTP 5xx status codes by province and ISP
Error distribution by client: the number and proportion of HTTP 4xx or 5xx status codes corresponding to User-Agent information about each client
Preset report: Frequently requested resources
This report provides information about frequently requested resources and helps you gain insight into popular domain names and URIs, as well as the provinces where most of your customers are located, and the ISPs of the customers. The data is also useful in helping you gain a better understanding of your business, such as whether operational activities attract the expected traffic volume, and whether the traffic during peak hours is higher or lower than expected. The information can help you make adjustments to the overall operations of your business.
This report includes the following data. You can view all data, or filter data by domain name or URI.
Top domain names by visit: the top domain names by the proportion of total visits
Top domain names by download traffic: the top domain names by the percentage of total download traffic
Frequently requested URIs: the numbers of visits, unique visitors, and downloads for each URI
Popular access sources: popular Referer source domain names, the numbers of visits and unique visitors, and percentage
Visits across the Chinese mainland: the average number of visits by province
Download speed across the Chinese mainland: the average download speed by province
Statistics by province: the total number of visits, total download traffic, and average download speed by province
Traffic and download speed by ISP: the total download traffic and average download speed by ISP
Statistics by ISP: the total number of visits, total download traffic, and average download speed for each ISP
Preset report: User composition
This report provides information about the user composition of your website, including the client or device, general location, and ISPs of the users. You can analyze top users with most visits or most downloads.
This report includes the following data. You can view all data, or filter data by domain name or URI.
PV: page views
UV: unique visitors
Source region distribution: the number and proportion of visits for each province
Visits by client: the number and proportion of visits by client type
Visits by ISP: the number and proportion of visits by ISP
Top users with most downloads: the total number of visits, number of error visits, and total downloads by IP address
Top users with most valid visits: the total number of visits, number of error visits, and downloads by IP address, excluding invalid visits with HTTP status codes such as 4xx and 5xx
Subscribe to report template data
If you want Log Service to periodically send data from a report template to you, you can use the subscription feature.
Procedure
Basic data is used as an example. Click Subscribe in the upper-right corner of the page, and then click Create.
In the dialog box that appears, configure Subscription Name, Frequency, and Global Time, and then click Next.
Select a notification method from the Notification drop-down list, specify relevant information, and then click Submit.
The notification methods include Email, WebHook-DingTalk Bot, WebHook-Feishu Chat Bot, Webhook-WeCom Bot, and WeChat.
Custom reports
Example 1: View the ranking of domain names with most HTTP 499 status codes in the last 30 days.
Log analysis statement:
return_code = 499| select domain , count(*) as c group by domain order by c desc limit 10
Example 2: View the ranking of domain names with most HTTP 502 status codes in the last 30 days.
Log analysis statement:
return_code = 502| select domain , count(*) as c group by domain order by c desc limit 10
Example 3: View the log data where URI is /cpu in the last 30 days.
You can directly click the uri field on the Raw Logs tab on the left and click /cpu to filter the required logs.
