Use Metricbeat to collect system data and NGINX service data - Elasticsearch

This topic describes how to use the Alibaba Cloud Metricbeat collector to collect system data, such as CPU usage, memory, disk I/O, and network I/O statistics, and Nginx service data, and then generate visual charts.

Prerequisites

You have completed the following operations:

Create an Alibaba Cloud Elasticsearch (ES) instance.
For more information, see Create an Alibaba Cloud Elasticsearch instance.
Note
The ES instance version must be from 6.0.x to 7.17.x. Version 7.10.0 is recommended to prevent compatibility issues.
Enable the automatic index creation feature for your Alibaba Cloud ES instance.
By default, Alibaba Cloud ES disables automatic index creation for security reasons. However, this feature is required by Beats. If you set the collector output to Elasticsearch, you must enable automatic index creation. For more information, see Quick access and configuration.
Create an Alibaba Cloud ECS instance. The ECS instance must be in the same virtual private cloud (VPC) as the Alibaba Cloud ES instance.
For more information, see Create an instance using the wizard.
Important
Beats supports only the Alibaba Cloud Linux (Alinux), Red Hat, and CentOS operating systems.
Install Cloud Assistant Agent and the Docker service on the target ECS instance.
For more information, see Install the Cloud Assistant Agent and Deploy and use Docker.

Collect system data using Metricbeat

Log on to the Alibaba Cloud Elasticsearch console.
In the left navigation pane, click Beats Data Shippers.
In the Create Shipper area, click Metricbeat.
Install and configure the collector.
For more information, see Collect ECS service logs and Collector YML configuration. The configuration for this topic is shown in the following figure.
Note
- Select Enable Kibana Monitoring to start monitoring the Metricbeat service in the Kibana console.
- Select Enable Kibana Dashboard. This option generates charts in the Kibana console without requiring additional YML configuration. Because Alibaba Cloud Kibana is configured within a VPC, you must enable internal-facing access on the Kibana configuration page to access the console. For more information, see Connect to a cluster through Kibana.
- Because the system module is enabled by default, you can skip Shipper YML Configuration.
Click Next.
Select the ECS instance where you want to install the collector.
Note
If you are creating a collector for the first time, click Authorize Now and follow the prompts to grant Alibaba Cloud ES access to Alibaba Cloud ECS.
Start the collector and check its installation status.
1. Click Start.
  The Start Shipper dialog box appears after the collector starts.
2. Click Back to Beats Shippers to go to the Beats Data Shippers page. The Metricbeat collector that you started is displayed in the Manage Shippers area.
3. Wait for the Shipper Status to change to Enabled. Then, click View Instances in the Actions column.
4. On the View Instances page, check the Installed Shippers. The installation is successful if the status is Normal Heartbeat.
View the results.
1. Log on to the Kibana console of the target Alibaba Cloud ES instance.
  For more information, see Log on to the Kibana console.
2. In the navigation pane, click Dashboard.
3. In the Dashboard list, click [Metricbeat System] Overview. Then, click a system to view its monitoring dashboard.

Collect Nginx service data using Metricbeat

Prerequisite: You must enable stub_status for the Nginx service. The ngx_http_stub_status_module module in Nginx counts the number of requests that the Nginx service receives and processes. To use this module, you must enable stub_status in the nginx.conf file. For information about how to modify the nginx.conf file, see How to modify the listener port address for web services such as Nginx and Tomcat on an ECS instance that runs Linux.

location /status {
           stub_status on;
           access_log off;
        }

Important

The server_status_path configured in the metricbeat.yml file must match the status path in the nginx.conf file.

Log on to the Alibaba Cloud Elasticsearch console.
In the navigation pane on the left, click Beats Data Shippers.
In the Create Shipper area, click Metricbeat.
Install and configure the collector.
For more information, see Collect ECS service logs and Collector YML configuration.
Add the following script to the metricbeat.yml file.
```
metricbeat.modules:
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s
  # Nginx hosts
  hosts: ["http://121.41.**.**"]
  # Path to server status. Default server-status
  server_status_path: "status"
```
Note
- Select Enable Kibana Monitoring to monitor the Metricbeat service in the Kibana console.
- Select Enable Kibana Dashboard. This automatically generates charts in the Kibana console without requiring manual YML configuration. Because Alibaba Cloud Kibana is configured within a VPC, you must first enable internal-facing access on the Kibana configuration page. For more information, see Connect to a cluster through Kibana.
Click Next.
Select the ECS instance where you want to install the collector.
Note
If you are creating a collector for the first time, click Authorize Now and follow the prompts to grant Alibaba Cloud ES access to Alibaba Cloud ECS.
Start the collector and check its installation status.
For more information, see Collect system data using Metricbeat.
View the results.
1. In a browser, go to <Nginx hosts>/status to view the monitoring page.
2. Log on to the Kibana console of the target Alibaba Cloud ES instance.
  For more information, see Log on to the Kibana console.
3. In the left navigation pane, click Dashboard.
4. In the Dashboard list, click [Metricbeat Nginx] Overview to view the Nginx monitoring dashboard.