All Products
Search
Document Center

Simple Log Service:Connect Simple Log Service to Grafana

Last Updated:Oct 26, 2023

This topic describes how to use Grafana to display and analyze NGINX logs that are collected by Simple Log Service.

Prerequisites

  • NGINX logs are collected. For more information, see Collect logs in NGINX configuration mode.

  • The indexing feature is enabled, and indexes are created. For more information, see Collect and analyze NGINX access logs.

  • The aliyun-log-grafana-datasource-plugin software package is downloaded. For more information, visit https://github.com/aliyun/aliyun-log-grafana-datasource-plugin.

    Note

    In this topic, aliyun-log-grafana-datasource-plugin V2.9 is used.

  • Grafana is installed. For more information, see Grafana documentation.

    Note
    • In this topic, Grafana 8.0.6 is used.

    • If you install Grafana on your computer, you must enable port 3000 in your browser settings.

    • If you want to use pie charts, you can run the following command to install the Pie Chart plug-in:

      grafana-cli plugins install grafana-piechart-panel

Version compatibility between Grafana and aliyun-log-grafana-datasource-plugin

The following table describes the version compatibility between Grafana and aliyun-log-grafana-datasource-plugin.

Grafana

aliyun-log-grafana-datasource-plugin

8.0.0 and later

V2.x

Earlier than 8.0.0

V1.0

Step 1: Install the Simple Log Service plug-in

The following procedure describes how to install the Simple Log Service plug-in for Grafana:

  1. Run the following commands to decompress the project package to the plug-in directory of Grafana.

    • If Grafana is installed by using a YUM repository or an RPM package, run the following command:

      unzip aliyun-log-grafana-datasource-plugin-master.zip -d /var/lib/grafana/plugins
    • If Grafana is installed by using a .tar.gz file, run the following command:

      {PATH_TO} specifies the installation directory of Grafana.

      unzip aliyun-log-grafana-datasource-plugin-master.zip -d {PATH_TO}/grafana-8.0.6/data/plugins
  2. Modify the configuration file of Grafana.

    1. Open the configuration file.

      • If Grafana is installed by using a YUM repository or an RPM package, open the /etc/grafana/grafana.ini file.

      • If Grafana is installed by using a .tar.gz file, open the {PATH_TO}/grafana-8.0.6/conf/defaults.ini file.

    2. Find [plugins] in the configuration file to configure the allow_loading_unsigned_plugins parameter.

      allow_loading_unsigned_plugins = aliyun-log-service-datasource
  3. Restart the Grafana service.

    1. Run the kill command to terminate the Grafana process.

    2. Run the following commands to start the Grafana service:

      • If Grafana is installed by using a YUM repository or an RPM package, run the following command:

        systemctl restart grafana-server
      • If Grafana is installed by using a .tar.gz file, run the following command:

        ./bin/grafana-server web

Step 2: Add a data source for Grafana

The following procedure describes how to add the Simple Log Service plug-in as a data source for Grafana:

  1. Log on to Grafana.

  2. In the left-side navigation pane, choose G1 > Data Sources.

  3. On the Data Sources tab, click Add data source.

  4. On the Add data source page, click Select in the LogService card.

  5. Configure the data source.

    The following table describes the parameters.

    Parameter

    Description

    Name

    The name of the data source.

    Default

    The Default switch. In this example, turn on the switch.

    Endpoint

    The endpoint of the Simple Log Service project. Example: http://cn-qingdao.log.aliyuncs.com. Enter an endpoint based on your business requirements. For more information, see Endpoints.

    Project

    The name of the project.

    Logstore

    The name of the Logstore.

    AccessKeyId

    The AccessKey ID provided by Alibaba Cloud. The AccessKey ID is used to identify the user. To ensure the security of your account, we recommend that you use the AccessKey pair of a RAM user. For more information about how to obtain an AccessKey pair, see AccessKey pair.

    AccessKeySecret

    The AccessKey secret provided by Alibaba Cloud. The AccessKey secret is used to authenticate the key of the user. To ensure the security of your account, we recommend that you use the AccessKey pair of a RAM user. For more information about how to obtain an AccessKey pair, see AccessKey pair.

  6. Click Save & Test.

Step 3: Add a dashboard

Perform the following steps to add a dashboard for Grafana:

  1. In the left-side navigation pane, choose Icon 1 > Dashboards.

  2. In the New Panel panel, click Choose Visualization.

  3. Configure template variables.

    After you configure template variables in Grafana, you can select different variable values to view different results in the same panel.

    1. Configure a template variable for a time interval.

      1. In the upper-right corner of the New dashboard page, select a time interval and click the Icon 2 icon.

      2. Click Variables.

      3. Click Add variable.

      4. Configure the parameters for the template variable. Then, click Add.

        The following table describes the parameters.

        Parameter

        Description

        Name

        Enter the name of the template variable. Example: myinterval. If you use a template variable in a query criterion, you must add two dollar signs ($$) preceding the name of the template variable. Example: $$myinterval.

        Type

        Select Interval.

        Label

        Enter time interval.

        Values

        Enter 1m,10m,30m,1h,6h,12h,1d,7d,14d,30d.

        Auto Option

        Turn on Auto Option and retain the default values of the other parameters.

    2. Configure a template variable for a domain.

      1. On the Variables page, click New.

      2. Configure the parameters for the template variable. Then, click Add. The following table describes the parameters.

        Parameter

        Description

        Name

        Enter the name of the template variable. Example: hostname. If you use a template variable in a query criterion, you must add a dollar sign ($) preceding the name of the template variable. Example: $hostname.

        Type

        Select Custom.

        Label

        Enter the name of the domain.

        Custom Options

        Enter *,example.com,example.org,example.net in the "Values separated by comma" field. This value specifies that you can view information about access to all domains. You can also view information about access to the example.com, example.org, or example.net domain.

        Selection Options

        Retain the default value of the parameter.

    3. In the left-side navigation pane, click Save.

  4. Add panels based on your business requirements.

    • Add a graph panel to display page views (PVs) and unique visitors (UVs).

      1. In the upper-right corner of the page, click the Icon 3 icon.

      2. In the New Panel panel, click Add Query.

      3. Click the Icon 4 icon and select Graph from the Visualization drop-down list.

      4. Click the Icon 5 icon. Then, enter UV&PV in the Title field.

      5. Click the Icon 6 icon, select Logservice from the Query drop-down list, and then configure the following parameters. grafana-pv&uv01

        Parameter

        Description

        Query

        Enter the query statement that you want to execute. Example:

        $hostname| select approx_distinct(remote_addr) as uv ,count(1) as pv , __time__ - __time__ % $$myinterval as time group by time order by time limit 1000

        In the query result, $hostname is replaced by the name of the domain that you specify, and $$myinterval is replaced by the time interval that you specify.

        Important

        Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.

        X-Column

        Enter time. The value is a UNIX timestamp with second precision.

        Y-Column

        Enter uv,pv.

      6. If the difference between the value of UV and the value of PV is significant, use a dual Y-axis panel to display UV and PV statistics. grafana-pv&uv02

      7. Click the Save icon icon to save your settings.

    • Add a graph panel to display inbound traffic and outbound traffic.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)." section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select sum(body_byte_sent) as net_out, sum(request_length) as net_in,__time__ - __time__ % $$myinterval as time group by __time__ - __time__ % $$myinterval limit 10000

      In the query result, $hostname is replaced by the name of the domain that you specify, and $$myinterval is replaced by the time interval that you specify.

      Important

      Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.

      X-Column

      Enter time. The value is a UNIX timestamp with second precision.

      Y-Column

      Enter net_in,net_out.

    • Add a pie chart panel to display the proportion of requests that are sent by using each HTTP request method.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select count(1) as pv ,method group by method

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter pie.

      Y-Column

      Enter method,pv.

    • Add a pie chart panel to display the proportion of responses that carry each HTTP status code.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select count(1) as pv ,status group by status

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter pie.

      Y-Column

      Enter status,pv.

    • Add a pie chart panel to display the request sources of frequently accessed pages.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select count(1) as pv , referer group by referer order by pv desc

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter pie.

      Y-Column

      Enter referer,pv.

    • Add a table panel to display the pages that are loaded at the highest latencies.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select URL as top_latency_URL ,request_time order by request_time desc limit 10

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      You do not need to configure this parameter.

      Y-Column

      Enter top_latency_url,request_time.

    • Add a table panel to display frequently accessed pages.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select count(1) as pv, split_part(URL,'?',1) as path group by split_part(URL,'?',1) order by pv desc limit 20

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      You do not need to configure this parameter.

      Y-Column

      Enter path,pv.

    • Add a table panel to display the pages that have the most access requests to which HTTP status code 200 is not returned.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname not status:200| select count(1) as pv , url group by url order by pv desc

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      You do not need to configure this parameter.

      Y-Column

      Enter url,pv.

    • Add a singlestat panel to display the average latency.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname | select avg(request_time) as response_time, avg(upstream_response_time) as upstream_response_time ,__time__ - __time__ % $$myinterval as time group by __time__ -  __time__ % $$myinterval limit 10000

      In the query result, $hostname is replaced by the name of the domain that you specify, and $$myinterval is replaced by the time interval that you specify.

      Important

      Two dollar signs ($$) must precede myinterval, and one dollar sign ($) must precede hostname.

      X-Column

      Enter time.

      Y-Column

      Enter upstream_response_time,response_time.

    • Add a logs panel to display the details about logs.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Note

      Each page can display a maximum of 100 logs. Therefore, the maximum value of the Logs Per Page parameter is 100.

    • Add a pie chart panel to display client statistics.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname  | select count(1) as pv, case when  regexp_like(http_user_agent , 'okhttp') then 'okhttp' when  regexp_like(http_user_agent ,  'iPhone') then 'iPhone' when regexp_like(http_user_agent ,  'Android')  then 'Android' else 'unKnown' end as http_user_agent group by  http_user_agent order by pv desc limit 10

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter pie.

      Y-Column

      Enter http_user_agent,pv.

    • Add a graph panel to display the number of requests to which each HTTP status code is returned over a 1-minute time range.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname  | select to_unixtime(time) as time,status,count from (select time_series(__time__, '1m', '%Y-%m-%d %H:%i', '0')  as time,status,count(*) as count from log group by status,time order by time limit 10000)

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter time.

      Y-Column

      Enter col1#:#col2. col1 is the aggregated column, and col2 is one of the other columns.

    • Add a worldmap panel to display the distribution of source IP addresses.

      For more information about how to add a panel, see the "Add a graph panel to display page views (PVs) and unique visitors (UVs)" section of this topic. The following table describes the parameters.

      Parameter

      Description

      Query

      Enter the query statement that you want to execute. Example:

      $hostname  | select   count(1) as pv ,geohash(ip_to_geo(arbitrary(remote_addr))) as geo,ip_to_country(remote_addr) as country  from log group by country having geo <>'' limit 1000

      In the query result, $hostname is replaced by the name of the domain that you specify.

      X-Column

      Enter map.

      Y-Column

      Enter country,geo,pv.

      Parameter

      Description

      Location Data

      Select geohash.

      Location Name Field

      Enter country.

      geo_point/geohash Field

      Enter geo.

      Metric Field

      Enter pv.

  5. View the results.

    In the upper section of the Dashboard page, you can select a time interval. You can also filter the results by using the time interval and hostname query criteria.

FAQ

  • Where are Grafana logs stored?

    Grafana logs are stored in the following directories:

    • macOS: /usr/local/var/log/grafana

    • Linux: /var/log/grafana

  • What do I do if aliyun-log-plugin_linux_amd64: permission denied appears in logs?

    Grant the EXECUTE permission on the dist/aliyun-log-plugin_linux_amd64 directory of the plug-in directory.