Troubleshoot Beats installation failures - Elasticsearch

Verify that the ECS instance for the Beats service runs Alibaba Cloud Linux (Alinux), Red Hat Enterprise Linux (RHEL), or CentOS.

Verify that the ECS instance for the Beats service is in the same VPC as your Elasticsearch or Logstash instance.

Verify that Cloud Assistant and Docker are installed on the ECS instance.

Connect to the ECS instance and run the following commands to verify the installations.

Check the status of the Cloud Assistant service

systemctl status aliyun.service

If the service is running normally, output similar to the following is displayed.

[root@VM01 ~]# systemctl status aliyun.service
● aliyun.service - aliyun-assist
   Loaded: loaded (/etc/systemd/system/aliyun.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2020-08-15 15:38:25 CST; 4 days ago
 Main PID: 20311 (aliyun-service)
   CGroup: /system.slice/aliyun.service
           └─20311 /usr/sbin/aliyun-service

Aug 15 15:38:25 VM01 systemd[1]: Stopped aliyun-assist.
Aug 15 15:38:25 VM01 systemd[1]: Started aliyun-assist.

If Cloud Assistant is not installed, see Install the Cloud Assistant client.

Check the status of Docker

systemctl status docker

If Docker is running normally, output similar to the following is displayed.

[root@VM01 ~]# systemctl status docker
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-08-19 16:45:21 CST; 1min 39s ago
     Docs: http://docs.docker.com
 Main PID: 14625 (dockerd-current)
   CGroup: /system.slice/docker.service
           ├─14625 /usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runti...
           └─14632 /usr/bin/docker-containerd-current -l unix:///var/run/docker/libcontainerd/docker-containerd.sock --metric...

Aug 19 16:45:20 VM01 dockerd-current[14625]: time="2020-08-19T16:45:20.973350772+08:00" level=warning msg="Docker could...yste...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.005411644+08:00" level=info msg="Graph migration...cond...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.006055965+08:00" level=info msg="Loading contain...tar...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.136526271+08:00" level=info msg="Firewalld runni...fals...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.239151591+08:00" level=info msg="Default bridge ...dres...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.280253025+08:00" level=info msg="Loading contain...done...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442183479+08:00" level=info msg="Daemon has comp...atio...
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.442217221+08:00" level=info msg="Docker daemon" ...1.13...
Aug 19 16:45:21 VM01 systemd[1]: Started Docker Application Container Engine.
Aug 19 16:45:21 VM01 dockerd-current[14625]: time="2020-08-19T16:45:21.450179119+08:00" level=info msg="API listen on /...soc...
Hint: Some lines were ellipsized, use -l to show in full.

If Docker is not installed, see Deploy and use Docker (Alibaba Cloud Linux 2).

Check the shipper's YML configuration to ensure that the following parameters are set.

- type: log
  # Change to true to enable this input configuration.
  enabled: true              
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log

Parameter	Description
enabled	The default value is `false`. You must set this parameter to `true`.
paths	The path of the log file. You can use wildcards, such as *.log.

Important

The paths parameter is different from the Filebeat file directory that you specify on the configuration page. The Filebeat file directory is a directory that is mapped by Docker. The files specified in paths can be collected only if this directory is mapped to the collection directory. We recommend that you keep the two consistent.
If you specified an Output on the configuration page, do not specify it again in the YML configuration. Otherwise, an installation error occurs.
Use caution when you modify parameters that are commented out (#) by default in the shipper's YML configuration, such as parameters related to X-Pack. Incorrect changes can cause the installation to fail.

Connect to the ECS instance and check the /opt/aliyunbeats/ directory. Verify that a directory for the Beats instance exists and contains the conf, data, and logs subdirectories.

[root@PVM1 ~]# cd /opt/aliyumbeats
[root@PVM1 aliyumbeats]# ls
ct-cn-77uqof2s7rgxxx
[root@PVM1 aliyumbeats]# cd ct-cn-77uqof2s7rgxxx
[root@PVM1 ct-cn-77uqof2s7rgxxx]# ls
filebeat
[root@PVM1 ct-cn-77uqof2s7rgxxx]# cd filebeat
[root@PVM1 filebeat]# ls
conf  data  logs

Also, check the Beats logs in the logs directory to identify the issue.

[root@VM01 logs]# tail -n 2 filebeat
2020-08-19T09:25:52.871Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring":
"metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":1}},"total":{"ticks":390,"time":{"ms":5},"value":390},"user":{"ti
cks":260,"time":{"ms":4}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849-
e-9dc9-f17xxx","uptime":{"ms":15900027}},"memstats":{"gc_next":4285312,"memory_alloc":2157688,"memory_total":367547441},
libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even
t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.17,"15":0.11,"5":0.09,"norm":{"1":0.17,"15":0.11
":0.09}}}}}}
2020-08-19T09:26:22.872Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring":
"metrics": {"beat":{"cpu":{"system":{"ticks":130,"time":{"ms":3}},"total":{"ticks":390,"time":{"ms":4},"value":390},"user":{"ti
cks":260,"time":{"ms":1}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"f960431f-2849-
e-9dc9-f17xxx","uptime":{"ms":16200027}},"memstats":{"gc_next":4285312,"memory_alloc":2449992,"memory_total":370470483},
libbeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"even
t":{"active":0}}},"registrar":{"states":{"current":1}},"system":{"load":{"1":0.11,"15":0.11,"5":0.08,"norm":{"1":0.11,"15":0.11
":0.08}}}}}}

Check the status of the Beats container and analyze its logs to identify the issue.

Check the status of the Docker container.

docker ps -a | grep filebeat

[root@VM01 logs]# docker ps -a | grep filebeat
922d7f8xxx   registry-vpc.cn-hangzhou.aliyuncs.com/elasticsearch/aliyun-elasticsearch-beats:filebeat-6.8.5   "/usr/local/bin/do..."   33 minutes ago   Up 33 minutes   ct-cn-77uqof2s7rg   ___FILEBEAT

If the container status is exited, check the container's output logs.
```
docker logs -f <container_id>
```

Elasticsearch:Troubleshoot Beats installation failures

Procedure