Website access logs are vital to webmasters. The access information of a website includes the page views (PVs), unique visitors (UVs), regions of visitors who access the website, and top 10 pages with most visits. Application logs are essential to application developers. Analysis and optimization of the SQL ORDER BY and LIMIT statements can improve application quality. O&M engineers can monitor server logs and locate exceptions. By monitoring logs in real time, engineers can obtain the server response time changes in the last hour and check whether the traffic is normal when a request is sent from a client to a machine for load balancing. Engineers can also view the monitoring data and obtain key information through the dashboard.
To meet the requirements of the preceding scenarios, Log Service provides a one-stop solution for log data collection and analysis. The LogSearch and Analytics features allow you to analyze logs in real time by using the query statements and SQL-92 statements. Log Service supports multiple visualization methods, including the built-in dashboards and open-source visualization tools such as DataV, Grafana, Tableau through Java Database Connectivity (JDBC), and QuickBI.
Log Service provides data analysis charts to display the real-time log query and analysis results. In addition, Log Service provides dashboards for displaying the log data analysis result in different scenarios.
- No need to define charts in advance: You can apply any computing method and any filter condition to logs generated in any time period. The corresponding chart appears within seconds.
- Interactive analysis: Seamless switching is supported between charts and raw logs.
- Scenario-specific solution: You can view the analysis dashboard through the data import wizard without complex configurations.
The following figure shows the chart types that are supported by the visualization feature of Log Service.
Process and architecture
- Collect data. Log Service supports multiple methods to collect logs, such as by using clients, webpages, protocols, and SDKs or APIs (for mobile apps and games). All collection methods are implemented based on RESTful APIs. You can also implement new collection methods by using APIs and SDKs.
- Set indexes and use query and analysis statements to query and analyze logs.
- Visually display data. Log Service provides RESTful APIs. You can use an appropriate method to visualize your log data. The following section describes the visualization and dashboard features of Log Service.
This section describes the typical use cases of different chart types. Before query and analysis, enable and set indexes. For more information, see Enable and configure the index feature for a Logstore.
A table, as the most common visual representation of data, consists of one or more groups of cells. It is used to display numbers and other items for quick reference and analysis. The items in a table are organized as rows and columns. The first row of a table is called the header, indicating the content and meaning of each column in the table.
In Log Service, the result returned by the query and analysis statements is displayed in tables by default.
Run the following statement to view the distribution of source IP addresses in the current time range in descending order:
* | SELECT sourceIPs, count(*) as count GROUP BY sourceIPs ORDER BY count DESC
The following figure shows the result table. You can click the sort icon in the header to sort a column in a certain order.
- Line chart
A line chart analyzes trends. It is typically used to indicate the changes of a group of data based on an ordered data type (successive time intervals in most cases) for analyzing the trend of data changes.Run the following statement to analyze the changes of PVs, UVs, and average response time in the last 15 minutes:
* | select date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as minutes, approx_distinct(remote_addr) as uv, count(1) as pv, avg(request_time) as avg group by minutes order by minutes asc limit 100000
Select minutes for X Axis, PV and UV for Left Y Axis, avg for Right Y Axis, and UV for Column Marker. The following figure shows a sample line chart.
- Column chart
A column chart uses vertical or horizontal columns to compare the numeric data among different types. A line chart describes the ordered data, while a column chart describes different types of data and counts the number in each data type. For more information about the line chart, see Line chart.Run the following statement to analyze the number of visits for each
http_refererin the last 15 minutes:
* | select http_referer, count(1) as count group by http_referer
- Bar chart
A bar chart, also known as a horizontal bar chart, is suitable for analyzing the ranking of different categories.Run the following statement to analyze the top 10
request_uriwith the most visits in the last 15 minutes:
* | select request_uri, count(1) as count group by request_uri order by count desc limit 10
- Pie chart
A pie chart is used to indicate the ratios of different data types and compare different data types based on the arc length. A pie chart is divided into multiple sectors based on the percentages of various data types. The entire chart indicates the sum of data. Each sector (arc-shaped) indicates the ratio of a data type to the sum. The sum of percentages in all sectors is equal to 100%.Run the following statement to analyze the visit distribution of different request URIs in the last 15 minutes:
* | select requestURI as uri , count(1) as c group by uri limit 10Pie chart:Donut chart:Polar area chart:
- Single value chart
A single value chart, as the easiest and most intuitive display type of data, shows the data on a point clearly and intuitively, and is generally used to indicate the key information on a time point.Run the following statement to analyze the PVs in the last 15 minutes:
* | select count(1) as PV
- Area chart
Based on a line chart, an area chart fills the section between a line and the axis with color. The filled section is an area block and the color highlights the trend.Run the following statement to collect statistics about the PVs of the IP address
10.0.XX.XXon the last day:
remote_addr: 10.0.XX.XX | select date_format(date_trunc('hour', __time__), '%m-%d %H:%i') as time, count(1) as PV group by time order by time limit 1000
- Map chart
You can add color blocks and marks to a map to display geographic data. Log Service provides three kinds of maps: Map of China, World Map, and AMap. Among them, AMap offers the scatter chart and heat map.You can use the remote_addr parameter in the statements to make three types of map charts. Run the following statements to display the top 10 regions with the most visits:
- Map of China
* | select ip_to_province(remote_addr) as address, count(1) as count group by address order by count desc limit 10
- World Map
* | select ip_to_country(remote_addr) as address, count(1) as count group by address order by count desc limit 10
* | select ip_to_geo(remote_addr) as address, count(1) as count group by address order by count desc limit 10
- Map of China
- Flow chart
The banded branches with different colors indicate different types of information. The band width indicates the corresponding numeric value. In addition, the centralized time attribute of the original data maps to the X-axis, which forms a three-dimensional relationship.Run the following statement to display the trends of the requests sent through different methods in the last 15 minutes:
* | select date_format(from_unixtime(__time__ - __time__% 60), '%H:%i:%S') as minute, count(1) as c, request_method group by minute, request_method order by minute asc limit 100000
Select minute for X Axis, c for Y Axis, and request_method for Aggregate Column.
- Sankey diagram
A Sankey diagram is a specific type of flow chart. It is used to describe the flow from one set of values to another. Sankey diagrams are applicable to scenarios such as network flow data. Generally, a Sankey diagram contains three sets of values: source, target, and value. The source and target describes the edge relationship between nodes, and value describes the relationship between source and target.Run the following statement to analyze the flow data in a load balancing scenario:
* | select sourceValue, targetValue, streamValue group by sourceValue, targetValue, streamValue order by streamValue
- Word cloud
A word cloud visualizes text data. It is a cloud-like and colored image composed of words. It can be used to display a large amount of text data. The importance of each word is shown with its font size or color. This allows you to quickly perceive the weight of some keywords.Run the following statement to analyze the visits to different request URIs in the last 15 minutes:
* | select requestURI as uri , count(1) as c group by uri limit 100
Add charts to a dashboard
All charts obtained through the query and analysis statement can be saved in a dashboard. Then, you can adjust the layout of these charts to get a comprehensive dashboard.
You can click Add to New Dashboard to create a dashboard. After creating a dashboard, you can open the dashboard based on tags to view data in real time.