Configure the YML file for each Beats shipper to define how it collects data and where it sends output. This topic covers the key parameters for Filebeat, Metricbeat, Heartbeat, and Auditbeat.
Prerequisites
Before you begin, make sure that you have:
An Alibaba Cloud Elasticsearch cluster with the Auto Indexing feature enabled. For details, see Create an Alibaba Cloud Elasticsearch cluster.
Alibaba Cloud Elasticsearch disables Auto Indexing by default for security reasons. Beats requires this feature. If you select Elasticsearch as the output when installing a shipper, enable Auto Indexing first. For details, see Configure the YML file.
Before you configure
Output section: If you specified Output when you installed a shipper, do not add an output section to Shipper YML Configuration. Adding one causes a shipper installation error.
Module configuration: Open source Beats provides multiple modules, but Alibaba Cloud Beats does not support separate configuration for individual modules. To use a module, add its configuration directly to the shipper's YML file.
For example, to enable the system module for Metricbeat, add the following to metricbeat.yml:
metricbeat.modules:
- module: system
metricsets: ["diskio","network"]
diskio.include_devices: []
period: 1sFilebeat configuration
Filebeat collects log files and forwards them to Elasticsearch or Logstash. Use filebeat.inputs in filebeat.yml to define which log sources to collect.
filebeat.inputs:
- type: log
enabled: true
paths:
- /opt/test/logs/t1.log
- /opt/test/logs/t2/*
fields:
alilogtype: usercenter_serverlogEach input source starts with a hyphen (-). Add multiple hyphens to define multiple input sources.
| Parameter | Description |
|---|---|
type | The input type. Default: log. Other valid values: stdin, redis, tcp, syslog. |
paths | The paths of the logs to monitor. Accepts a file path (for example, /log/nginx.log) or a directory with a wildcard (for example, /log/*). The path is mapped to Docker. When specifying a directory, include * for fuzzy match, and make sure all files in the directory match the type you want to collect. |
enabled | Whether the input takes effect. Default: true (active). Set to false to disable. |
fields | Optional. Additional fields to attach to each log event. Indent field names with two spaces under this key. For example, alilogtype: usercenter_serverlog tags each event with the log source type. If logs are shipped to Logstash, Logstash can use this field to route and process logs by type. |
For full input options, see Log input in the Filebeat documentation.
Metricbeat configuration
Metricbeat collects system and service metrics. Use metricbeat.modules in metricbeat.yml to configure which modules and metricsets to run.
The following example collects disk I/O and network metrics from a system host every 10 seconds:
metricbeat.modules:
- module: system
metricsets: ["diskio","network"]
enabled: true
hosts: ["http://XX.XX.XX.XX/"]
period: 10s
fields:
dc: west
tags: ["tag"]| Parameter | Description |
|---|---|
module | The name of the module to run. For supported modules, see Modules. |
metricsets | The metricsets to collect within the module. For available metricsets per module, see Modules. |
enabled | Whether the module takes effect. Default: true (active). Set to false to disable. |
period | How often Metricbeat runs the metricsets. If the target system is unreachable, Metricbeat logs an error for each interval. |
hosts | Optional. The hosts to collect metrics from. |
fields | Optional. Additional fields to send with each metricset event. |
tags | Optional. Tags to attach to each metricset event. |
For full configuration options, see the Metricbeat documentation.
Heartbeat configuration
Heartbeat periodically checks whether your services are reachable and responding. Unlike Metricbeat — which checks whether a service process is running — Heartbeat verifies that a service is actually available to end users.
Use heartbeat.monitors in heartbeat.yml to define which services to monitor. For high availability, deploy Heartbeat on at least two Elastic Compute Service (ECS) instances.
heartbeat.monitors:
- type: http
name: ecs_monitor
enabled: true
urls: ["http://localhost:9200"]
schedule: '@every 5s'
fields:
dc: west| Parameter | Description |
|---|---|
type | The monitor type. Valid values: icmp, tcp, http. |
name | The monitor name. This value appears in the monitor's Exported fields and is used as the job name. The type value is used as the job type. |
enabled | Whether the monitor takes effect. Default: true (active). Set to false to disable. |
urls | Optional. The URLs or hosts to connect to. |
schedule | The check schedule. @every 5s runs the check every 5 seconds from startup. A cron expression such as */5 * * * * * * runs the check every 5 seconds on a fixed schedule. |
fields | Optional. Additional fields to include in the monitor output. |
For full configuration options, see the Heartbeat documentation.
Auditbeat configuration
Auditbeat collects audit logs from the Linux audit framework and monitors file integrity. It groups related events into structured records that integrate with Logstash, Elasticsearch, and Kibana.
Auditbeat requires Linux kernel version 3.14 or later, and the Auditd service must be stopped. Run service auditd status to check the service state.
Use auditbeat.modules in auditbeat.yml to enable modules. The file has two sections: module (what to collect) and output (where to send data).
The following example enables the auditd module to watch for permission and access changes on /etc/passwd, and the file_integrity module to track changes across system binary directories:
auditbeat.modules:
- module: auditd
audit_rules: |
-w /etc/passwd -p wa -k identity
-a always,exit -F arch=b32 -S open,create,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
- module: file_integrity
paths:
- /bin
- /usr/bin
- /sbin
- /usr/sbin
- /etcFor full YML configuration options, see the Auditbeat documentation. For module-specific configuration, see Modules.