Shipper YML configuration - Elasticsearch - Alibaba Cloud Documentation Center

Prerequisites

Create an Alibaba Cloud Elasticsearch cluster and enable Auto Indexing. For details, see Create an Alibaba Cloud Elasticsearch cluster.

By default, Alibaba Cloud Elasticsearch disables Auto Indexing for security reasons. Because Beats requires this feature, you must enable Auto Indexing if the shipper Output is set to Elasticsearch. For more information, see Configure YML parameters.

Note Alibaba Cloud does not provide separate configuration files for the many modules in Beats. To use a module, add its configuration directly to the main YML file of the corresponding shipper. For example, to enable the system module in Metricbeat, add the following configuration to the metricbeat.yml file.

metricbeat.modules:
- module: system
metricsets: ["diskio","network"]
diskio.include_devices: []
period: 1s

Filebeat configuration

In filebeat.yml, use filebeat.inputs to define your input sources. The following example shows a basic input configuration.

#========================== Filebeat inputs ================================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/test/logs/t1.log
    - /opt/test/logs/t2/*
  fields:
    alilogtype: usercenter_serverlog
  #- c:\programdata\elasticsearch\logs\*

filebeat.inputs:
- type: log
  enabled:true
  paths:
    - /opt/test/logs/t1.log
    - /opt/test/logs/t2/*
  fields:
    alilogtype: usercenter_serverlog

Important

If you specified the shipper Output when you collect ECS service logs by using Filebeat, do not configure the output separately in the Shipper YML Configuration. Otherwise, the ECS shipper installation will fail.
Each input source starts with a hyphen (-). For multiple sources, precede each additional source with a hyphen (-).

Parameter	Description
`type`	The input type. The default value is `log`. Other supported values include `stdin`, `redis`, `tcp`, and `syslog`.
`paths`	The logs to monitor. You can specify a file, such as `/log/nginx.log`, or a directory, such as `log/`. The specified files or directories are mapped to a Docker directory. Important* To specify a directory, you must use a wildcard pattern such as `/*`. Make sure that the file types in the directory are consistent with the collection type.
`enabled`	Specifies whether the configuration is enabled: `true`: Enabled `false`: Disabled
`fields`	Specifies optional fields to add to the output. Indent the fields that you want to add by two spaces under this parameter. For example, if you set this parameter to `alilogtype: usercenter_serverlog`, this field is added to each log entry to identify the log source. Logstash can then use this field to classify and process logs.

For more details, see the official Log input documentation.

Metricbeat configuration

Metricbeat is a lightweight shipper that sends various system and service statistics. In metricbeat.yml, specify metricbeat.modules to configure module settings.

#========================== Modules configuration ============================

metricbeat.config.modules:
    # Glob pattern for configuration loading
    path: ${path.config}/modules.d/*.yml

    # Set to true to enable config reloading
    reload.enabled: false
metricbeat.modules:
  - module: system
    metricsets: ["diskio","network"]
    enabled: true
    hosts: ["http://XX.XX.XX.XX/"]
    period: 10s
    fields:
      dc: west
    tags: ["tag"]
    # Period on which files under path should be checked for changes
    #reload.period: 10s

metricbeat.modules:
- module: system
  metricsets: ["diskio","network"]
  enabled: true
  hosts: ["http://XX.XX.XX.XX/"]
  period: 10s
  fields:
    dc: west
  tags: ["tag"]

Important If you specified the shipper Output when you collect ECS metrics with Metricbeat, do not configure the output separately in the Shipper YML Configuration. Otherwise, the ECS shipper installation will fail.

Parameter	Description
`module`	The name of the module to run. For information about supported modules and their descriptions, see Modules.
`metricsets`	A list of metricsets to run. For more information about the metricsets, see Modules.
`enabled`	Specifies whether this configuration is enabled. `true` indicates that the configuration is enabled, and `false` indicates that the configuration is disabled.
`period`	The frequency at which to run the metricsets. If a monitored system is inaccessible, Metricbeat returns an error for that collection period.
`hosts`	Optional. A list of hosts to fetch information from.
`fields`	Optional fields to send with metricset events.
`tags`	Optional. A list of tags to send with each metricset event.

For more details, see the official Metricbeat documentation.

Heartbeat configuration

Heartbeat is a lightweight shipper that you install on a remote server to periodically check if services are available. Unlike Metricbeat, which reports if a service is running, Heartbeat checks if the service is reachable.

In heartbeat.yml, you specify heartbeat.monitors to define the services to monitor.

Note The primary configuration for Heartbeat is defining which services to monitor. For high availability, we recommend deploying Heartbeat on at least two ECS instances.

# Define a directory to load monitor definitions from. Definitions take the form
# of individual yaml files.
heartbeat.config.monitors:
  # Directory + glob pattern to search for configuration files
  path: ${path.config}/monitors.d/*.yml
  # If enabled, heartbeat will periodically check the config.monitors path for changes
  reload.enabled: false
  # How often to check for changes
  reload.period: 5s

  # Configure monitors inline
heartbeat.monitors:
- type: http
  name: ecs_monitor
  enabled: true
  urls: ["http://localhost:9200"]
  schedule: '@every 5s'
  fields:
    dc: west
#- type: http

heartbeat.monitors:
- type: http
  name: ecs_monitor
  enabled: true
  urls: ["http://localhost:9200"]
  schedule: '@every 5s'
  fields:
    dc: west

Important If you specified the shipper Output when you monitor ECS services with Heartbeat, do not configure the output separately in the Shipper YML Configuration. Otherwise, the ECS shipper installation will fail.

Parameter	Description
`type`	The monitor type. Valid values: `icmp`, `tcp`, and `http`.
`name`	The name of the monitor. This value appears under the monitor field in Exported fields as the job name. The `type` field is used as the job type.
`enabled`	Specifies whether this configuration is enabled. `true` indicates that the configuration is enabled, and `false` indicates that the configuration is disabled.
`urls`	Optional. A list of servers to connect to.
`schedule`	The task schedule. For example, `@every 5s` runs the task every 5 seconds after Heartbeat starts. A cron expression such as `/5 * * * * *` also runs the task every 5 seconds.
`fields`	Optional fields to add to the output.

For more details, see the official Heartbeat documentation.

Auditbeat configuration

Auditbeat is a lightweight shipper that collects audit data from the Linux audit framework and monitors file integrity. It can combine related messages into single events and structure the data for analysis. It also integrates seamlessly with Logstash, Elasticsearch, and Kibana.

Important Auditbeat depends on the Linux audit framework and requires Linux kernel version 3.14 or later. Ensure the Auditd service is stopped (check its status by running service auditd status).

In the auditbeat.yml file, use the auditbeat.modules section to configure the Auditbeat shipper. The auditbeat.yml file contains module configurations and output settings. Enable a module by adding its parameters to the auditbeat.yml file. The following example shows the configurations for the auditd and file_integrity modules.

auditbeat.modules:
- module: auditd
  audit_rules: |
    -w /etc/passwd -p wa -k identity
    -a always,exit -F arch=b32 -S open,create,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc

Important If you specified the shipper Output when you collect ECS audit data with Auditbeat, do not configure the output separately in the Shipper YML Configuration. Otherwise, the ECS shipper installation will fail.

For more information about configuring auditbeat.yml, see the official Auditbeat documentation. For details about how to configure each module, see Modules.