You can modify and enable the YML configuration of a shipper to complete specific data collection tasks. This topic discusses specific parameters in the YML configuration files and describes how to modify the YML configuration.

Preparations

The Auto Indexing feature is enabled for your Alibaba Cloud Elasticsearch cluster.

For security purposes, Alibaba Cloud Elasticsearch disables Auto Indexing by default. However, Beats depends on this feature. If you select Elasticsearch for Output when you create a shipper, you must enable the Auto Indexing feature. For more information, see Enable auto indexing.

Note Open-source Beats provides many modules, but Alibaba Cloud Beats does not provide separate configuration for these modules. If you want to use them, you must configure them in the configuration files of different shippers. For example, if you want to enable the system module in a Metricbeat shipper, add the following script to metricbeat.yml: 
metricbeat.modules:
- module: system
metricsets: ["diskio","network"]
diskio.include_devices: []
period: 1s

Filebeat configuration

You can specify filebeat.inputs in filebeat.yml to determine how to search for or handle input data sources. The following figure shows an example of simple input configuration.Filebeat configuration 
filebeat.inputs:
- type: log
  paths:
    - /opt/test/logs/t1.log
    - /opt/test/logs/t2/*
  fields:
    alilogtype: usercenter_serverlog
Notice
  • If you specify Output when you install a shipper, you do not need to specify it again in Shipper YML Configuration. If you specify it again, the system prompts a shipper installation error. For more information, see Install a shipper.
  • An input data source starts with a hyphen (-). You can use multiple hyphens to specify multiple input data sources.
Parameter Description
type The input type. Examples of valid values: stdin, redis, tcp, and syslog. Default value: log.
paths The paths of logs you want to monitor. You can specify either a file or a directory to map to Docker.
enabled Specifies whether the configuration takes effect. The value true indicates that the configuration takes effect. The value false indicates that the configuration does not take effect.
fields The optional fields. Below this parameter, you can indent with two spaces to add fields. For example, enter alilogtype: usercenter_serverlog to add this field in each output log to identify the type of the log source. If logs are shipped to Logstash, they can be classified and processed based on this field.

For more information, visit Log input in the open-source Filebeat documentation.

Metricbeat configuration

Metricbeat delivers system and service statistics in a lightweight manner. You can specify metricbeat.modules in metricbeat.yml to configure a module.Metricbeat configuration 
metricbeat.modules:
- module: system
  metricsets: ["diskio","network"]
  hosts: ["http://XX.XX.XX.XX/"]
  period: 10s
  fields:
    dc: west
  tags: ["tag"]
Notice If you specify Output when you install a shipper, you do not need to specify it again in Shipper YML Configuration. If you specify it again, the system prompts a shipper installation error. For more information, see Install a shipper.
Parameter Description
module The name of the module you want to run. For more information about supported modules, visit Modules.
metricsets The list of metricsets you want to execute. For more information about metricsets, visit Modules.
enabled Specifies whether the configuration takes effect. The value true indicates that the configuration takes effect. The value false indicates that the configuration does not take effect.
period Specifies how often the metricsets are executed. If the system is not reachable, Metricbeat returns an error for each period.
hosts Optional. The list of hosts from which you want to fetch information.
fields Optional. The fields that are sent with the metricset event.
tags Optional. The list of tags that are sent with the metricset event.

For more information, visit Configuration in the open-source Metricbeat documentation.

Heartbeat configuration

Heartbeat can be installed on a remote server in a lightweight manner to periodically check the status of your services and determine whether they are available. Heartbeat is unlike Metricbeat. Heartbeat checks whether your services are reachable but Metricbeat checks whether your services are running.

You can specify heartbeat.monitors in heartbeat.yml to specify the services you want to monitor.
Note Heartbeat requires only the configuration of the services you want to monitor. To ensure availability, we recommend that you deploy at least two Elastic Compute Service (ECS) instances.
Heartbeat configuration 
heartbeat.monitors:
- type: http
  name: ecs_monitor
  enabled: true
  urls: ["http://localhost:9200"]
  schedule: '@every 5s'
  fields:
    dc: west
Notice If you specify Output when you install a shipper, you do not need to specify it again in Shipper YML Configuration. If you specify it again, the system prompts a shipper installation error. For more information, see Install a shipper.
Parameter Description
type The monitor type. Valid values: icmp, tcp, and http.
name The monitor name. This value appears in the exported fields of the monitor field, and is considered as the job name. The type field is considered as the job type.
enabled Specifies whether the configuration takes effect. The value true indicates that the configuration takes effect. The value false indicates that the configuration does not take effect.
urls Optional. The list of servers you want to ping.
schedule The task schedule. If you set the value to @every 5s, the system runs the task every 5 seconds from the time Heartbeat is started. If you set the value to */5 * * * * * *, the system runs the task every 5 seconds.
fields The optional fields that you can add to the output as additional information.

For more information, visit Configuration in the open-source Heartbeat documentation.

Auditbeat configuration

Auditbeat is a lightweight shipper that collects audit logs from the Linux audit framework and monitors file integrity. Auditbeat combines relevant messages into an event to generate structured data for analytics. It can also be seamlessly integrated with Logstash, Elasticsearch, and Kibana.
Notice Auditbeat is based on the Linux audit framework and requires an OS kernel version of 3.14 or later. The state of the Auditd service must be stop. You can run the service auditd status command to query the service status.

You can specify auditbeat.modules in auditbeat.yml to configure the Auditbeat shipper. auditbeat.yml consists of two parts: module and output. If you want to enable the auditd and file_integrity modules, add the following script to auditbeat.yml: 

auditbeat.modules:
- module: auditd
  audit_rules: |
    -w /etc/passwd -p wa -k identity
    -a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
Notice If you specify Output when you install a shipper, you do not need to specify it again in Shipper YML Configuration. If you specify it again, the system prompts a shipper installation error. For more information, see Install a shipper.

For more information about auditbeat.yml configuration, visit Configuration in the open-source Auditbeat documentation. For more information about module configuration, visit Modules.