Processing-Website Access Log Statistics and Analysis

Last Updated: Aug 08, 2017

Requirements

  • Nginx/Apache access logs must be connected to the Alibaba Cloud Log Service.

  • The Log Service is activated.

  • The Container Service is activated.

  • There is an Alibaba Cloud AccessKey that can be used to access the Log Service (which can be a key authorized to a sub-account).

Sub-account permission

If you have the sub-account permission, replace ${your_project} with your project name and change ${your_logstore} to the LogStore where your access locates based on the following permission configurations.

  1. {
  2. "Version": "1",
  3. "Statement": [
  4. {
  5. "Action": [
  6. "log:Get*",
  7. "log:List*"
  8. ],
  9. "Resource": "acs:log:*:*:project/${your_project}/logstore/${your_logstore}",
  10. "Effect": "Allow"
  11. },
  12. {
  13. "Action": [
  14. "log:CreateConsumerGroup",
  15. "log:ListConsumerGroup",
  16. "log:ConsumerGroupUpdateCheckPoint",
  17. "log:ConsumerGroupHeartBeat",
  18. "log:GetConsumerGroupCheckPoint"
  19. ],
  20. "Resource": "acs:log:*:*:project/${your_project}/logstore/${your_logstore}/consumergroup/*",
  21. "Effect": "Allow"
  22. }
  23. ]
  24. }

Options

  • Self-built MySQL or RDS (use MySQL in a Docker image by default)

Implementation steps

We use the Ubuntu operating system as an example to describe the implementation steps. Refer to this document for implementation in other systems.

  1. For specific steps of connecting the access logs to the Alibaba Cloud Log Service, refer to the Alibaba Cloud Log Service documentation.

  2. Activate the Container Service.

  3. In the Container Service, refer to Create a Cluster to create a cluster, while selecting “Ubuntu” as the operating system.

  4. In the Container Service, refer to Create an Application to create an application.

    1. Enter “Dashboard” (or a user-defined name) as the application name.
    2. Select the newly created cluster as the deployment cluster.
    3. Click Create from Image.
    4. Click Select an Image, and then select the Alibaba Cloud image “access_log_stat_dashboard”. Select the displayed image “aliyunlog/access_log_stat_dashboard”.
    5. In Web Routing Rules, enter “80” as the container port and “Dashboard” as the domain name, and then click Add when done.
    6. Click OK.
  5. In “Service”, find the service you just created (“Dashboard”, or a user-defined service name), click the service name of the service, and then find the access endpoint in the displayed basic information as in “Access Endpoint”.

  6. Open and use the above URL in a browser.
  7. When you open the Dashboard for the first time, you need to log on with the account you use for the Log Service, including region, project, AccessID and AccessKey.
  8. After logging in, you need to configure the following if you are using the Dashboard for the first time:
    1. Log information, including region, project, AccessId, AccessKey and LogStore. You do not need to provide the LogStore information when you log in.
    2. Map the log content fields. A Docker image uses some default names to access some fields of the access logs. If you are using non-default field names when accessing the Log Service, map the fields here to ensure that your fields can be recognized by the analysis program. Take the latency field as an example. If you configure the field name as “request_time” when accessing the Log Service, you need to fill in “request_time”. Below is a log sample:
  1. 192.168.1.101 - - [17/Mar/2016:10:28:30 +0800] "GET /fonts/fontawesome-webfont.woff?v=4.2.0 HTTP/1.1" 0.021 1207 304 0 "https://sls.console.aliyun.com/css/lib.css" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36"
Backend processing key Field sample
ip 192.168.1.101
method GET
path /fonts/fontawesome-webfont.woff?v=4.2.0
latency 0.021
request_length 1207
status 304
response_length 0
referer https://sls.console.aliyun.com/css/lib.css
user_agent Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.87 Safari/537.36

Advanced options

The data of image computing results is stored in the image MySQL. If you release your container, historical computing results will be lost. In order to keep all historical computing results, use your own MySQL:

  • Create a MySQL user and authorize the user to access this MySQL from Docker.

    1. grant all privileges on *.* to loguser@'%' identified by '123456789';
    2. flush privileges;
  • You can define the user name and password based on your own needs.

  • Modify “/etc/mysql/my.conf” by commenting out the line “bind-address 0.0.0.0”.
  • Restart MySQL

    1. sudo service mysql restart
  • Click The computing results will be temporarily stored in the Docker container. Please change to another database, if you want to store the results permanently on the top of the Dashboard page, to go to MySQL configuration. Fill in the MySQL address and account information, and then click Migrate.

Use ECS to activate docker

We have discussed how to use the Container Service to activate Docker in the section above. ECS users can use their own virtual machine to activate Docker. To activate the image on ECS, execute the following command:

  1. pull registry.aliyuncs.com/aliyunlog/access_log_stat_dashboard
  2. docker run --dns 223.5.5.5 -p 80:80 -d registry.aliyuncs.com/aliyunlog/access_log_stat_dashboard
Thank you! We've received your feedback.