Collect system metrics and Nginx data with Metricbeat | Elasticsearch - Elasticsearch

Use a Metricbeat shipper to collect system metrics (CPU usage, memory, disk I/O, and network I/O) and Nginx service data, then visualize the data in Kibana dashboards.

Prerequisites

The following requirements must be met:

An Alibaba Cloud Elasticsearch instance is created.
Create an Alibaba Cloud Elasticsearch instance.
Note
The instance must run version 6.0.x to 7.17.x. Version 7.10.0 is recommended.
Automatic index creation is enabled for your Elasticsearch instance.
Alibaba Cloud Elasticsearch disables automatic index creation by default, but Beats requires it. Enable this feature if Elasticsearch is the output for your shipper. Access and configure an instance.
An ECS instance is created in the same VPC as the Elasticsearch instance.
Create an instance by using the wizard.
Important
Beats supports only the following operating systems: Alibaba Cloud Linux (Alinux), Red Hat, and CentOS.
Cloud Assistant Agent and Docker are installed on the target ECS instance.
Install the Cloud Assistant Agent and Deploy and use Docker.

Collect system data with Metricbeat

Log on to the Alibaba Cloud Elasticsearch console.
In the navigation pane on the left, click Beats Data Shippers.
In the Create Shipper section, click Metricbeat.
Install and configure the shipper.
Configure the shipper by following Use Filebeat to collect logs from an ECS instance and Shipper YML Configuration. The following figure shows the configuration used in this topic.
Note
- Select Enable Kibana Monitoring to monitor the Metricbeat service on the Kibana console.
- Select Enable Kibana Dashboard to auto-generate Kibana dashboards without extra YML configuration. Kibana runs in a VPC, so enable private network access first. Connect to an instance by using Kibana.
- The system module is enabled by default. No configuration is needed for the Shipper YML Configuration.
Click Next.
Select the target ECS instance.
Note
If this is your first shipper, click Authorize Now to grant Elasticsearch permission to access your ECS instances.
Start the shipper and check its installation status.
1. Click Start.
  After a successful startup, the Start Shipper dialog box appears.
2. Click Back to Beats Shippers. On the Beats Data Shippers page, find the Metricbeat shipper in the Manage Shippers section.
3. Wait until the Shipper Status changes to Enabled. Then, in the Actions column, click View Instances.
4. On the View Instances page, verify that the Installed Shippers is Normal Heartbeat.
Verify the results.
1. Log on to the Kibana console of your Elasticsearch instance.
  Log on to the Kibana console.
2. In the navigation pane on the left, click Dashboard.
3. In the Dashboard list, click [Metricbeat System] Overview. The monitoring dashboard for your Metricbeat system is displayed.

Collect Nginx service data with Metricbeat

Prerequisite: Enable stub_status for your Nginx service. The ngx_http_stub_status_module counts requests received and processed by Nginx. Use the stub_status configuration block shown below. Modify the listener port of web services such as Nginx and Tomcat on an ECS instance that runs Linux.

location /status {
           stub_status on;
           access_log off;
        }

Important

The server_status_path configured in the metricbeat.yml file must match the path for status in the nginx.conf file.

Log on to the Alibaba Cloud Elasticsearch console.
In the navigation pane on the left, click Beats Data Shippers.
In the Create Shipper section, click Metricbeat.
Install and configure the shipper.
Configure the shipper by following Use Filebeat to collect logs from an ECS instance and Shipper YML Configuration.
Add the following configuration to metricbeat.yml.
```
metricbeat.modules:
- module: nginx
  metricsets: ["stubstatus"]
  enabled: true
  period: 10s
  # Nginx hosts
  hosts: ["http://121.41.**.**"]
  # Path to server status. Default server-status
  server_status_path: "status"
```
Note
- Select Enable Kibana Monitoring to monitor the Metricbeat service on the Kibana console.
- Select Enable Kibana Dashboard to auto-generate Kibana dashboards without extra YML configuration. Kibana runs in a VPC, so enable private network access first. Connect to an instance by using Kibana.
Click Next.
Select the target ECS instance.
Note
If this is your first shipper, click Authorize Now to grant Elasticsearch permission to access your ECS instances.
Start the shipper and check its installation status.
The steps are the same as in Collect system data using Metricbeat.
Verify the results.
1. Navigate to <Nginx hosts>/status in a browser to view the monitoring page.
2. Log on to the Kibana console of your Elasticsearch instance.
  Log on to the Kibana console.
3. In the navigation pane on the left, click Dashboard.
4. In the Dashboard list, click [Metricbeat Nginx] Overview.