Troubleshoot Beats shipper installation failures and abnormal heartbeats - Elasticsearch

Use this guide to diagnose and resolve installation failures or abnormal heartbeats for a Beats shipper on Alibaba Cloud Elasticsearch.

Prerequisites

Before you begin, ensure that you have:

Access to the Elastic Compute Service (ECS) instance where Beats is installed
Permission to connect to the ECS instance and run shell commands

Troubleshoot installation failures

Work through the following checks in order. Each check addresses a distinct root cause.

Step 1: Verify the operating system

Beats shippers on Alibaba Cloud Elasticsearch require a supported OS. Confirm the ECS instance runs one of the following:

Alibaba Cloud Linux
Red Hat Enterprise Linux (RHEL)
CentOS

If the instance runs a different OS, migrate Beats to a supported instance.

Step 2: Verify network connectivity

The ECS instance and your Elasticsearch or Logstash cluster must be in the same virtual private cloud (VPC). If they are in different VPCs, Beats cannot connect to the output and the installation will fail.

Step 3: Check Cloud Assistant and Docker

Beats runs inside a Docker container managed by Cloud Assistant. If either service is missing or stopped, the shipper cannot start.

Connect to the ECS instance, then run the following commands.

Check Cloud Assistant:

systemctl status aliyun.service

A healthy Cloud Assistant returns output similar to the following:

If Cloud Assistant is not running, see Install the Cloud Assistant client.

Check Docker:

systemctl status docker

A healthy Docker service returns output similar to the following:

If Docker is not running, see Deploy and use Docker on Alibaba Cloud Linux 2 instances.

Step 4: Check the YAML configuration file

Misconfigured parameters in the shipper's YAML configuration file are a common cause of installation failures. Verify that the file includes the following:

- type: log
  # Change to true to enable this input configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob-based paths.
  paths:
    - /var/log/*.log

Parameter	Description
`enabled`	Set to `true` to activate the input. The default value is `false`.
`paths`	The path to the log files to collect. Supports wildcards, such as `*.log`.

Important

Note the following constraints:

paths vs. Filebeat Log File Path: The paths value in the YAML file and the Filebeat Log File Path on the configuration page serve different purposes. Docker maps the path specified by Filebeat Log File Path; only after that mapping does data reach the location specified by paths. Set both to the same value to avoid collection gaps.
Duplicate Output configuration: If you have already specified Output on the configuration page, do not add an output section in the YAML file. Duplicate output configuration triggers an installation error.
Commented-out parameters: Exercise caution when uncommenting parameters that are commented out with #, such as X-Pack-related parameters. Incorrect values in those parameters can prevent the shipper from installing.

Step 5: Verify the Beats instance directory

After a successful installation, Beats creates a directory at /opt/aliyunbeats/ with three subfolders: conf, data, and logs.

Connect to the ECS instance and confirm the directory and folders exist:

If the directory is missing, the installation did not complete. Review earlier steps to identify the cause.

If the directory exists, check the logs folder for error messages that can help pinpoint the problem:

Step 6: Check the Docker container status

If the shipper still does not work after the earlier checks, inspect the Docker container directly.

Check the container status:
```
docker ps -a | grep filebeat
```
If the container is in the exited state, retrieve its logs to identify the error:
```
docker logs -f <container-id>
```
Replace <container-id> with the container ID from the previous command's output.