You can use Beats to collect the data of an Elastic Compute Service (ECS) instance. The data includes logs, network data, and metrics. Beats then sends the collected data to Alibaba Cloud Elasticsearch or Logstash for further processing, such as monitoring and analysis. This topic describes how to use a Filebeat shipper to collect the logs of an ECS instance.
Prerequisites
- An Alibaba Cloud Elasticsearch or Logstash cluster is created. In this example, a V6.7.0 cluster is used.
For more information, see Create an Alibaba Cloud Elasticsearch cluster or Create an Alibaba Cloud Logstash cluster.
- The Auto Indexing feature is enabled for the Elasticsearch cluster.
For security purposes, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. However, Beats depends on this feature when it collects the logs of ECS instances. If you want to select Elasticsearch for Output, you must enable the Auto Indexing feature for the Elasticsearch cluster. For more information, see Configure the YML file.
- An ECS instance is created in the same virtual private cloud (VPC) as the Elasticsearch or Logstash cluster.
When you create the ECS instance, select one of the following types of operating systems: Alibaba Cloud Linux, Red Hat Enterprise Linux (RHEL), and Community Enterprise Operating System (CentOS). Beats supports only the three types of operating systems. For more information, see Create an instance by using the wizard
Important The default installation directory of Beats is /opt/aliyunbeats/. After you install Beats, the conf, logs, and data directories are generated on the ECS instance. The conf directory contains the configuration file, the logs directory contains the Beats log file, and the data directory contains the Beats data file. We recommend that you do not delete or modify these files. Otherwise, errors may occur, or data may be incorrect. If an error occurs, you can view the Beats logs in the logs directory to locate the error. - Cloud Assistant and Docker are installed on the ECS instance. For more information, see Install the Cloud Assistant client and Deploy and use Docker on Alibaba Cloud Linux 2 instances.Important You must use Docker Community Edition (Docker-CE). If you use Docker other than Docker-CE, such as podman-docker, Beats will fail to be deployed.
Procedure
- Log on to the Alibaba Cloud Elasticsearch console.
- Navigate to the Beats Data Shippers page.
- Configure and enable a shipper to collect the logs of the ECS instance. Note In this example, a Filebeat shipper is used.
- View the running ECS instance. After the shipper is created, you can view the running ECS instance to check whether shipper installation on the ECS instance succeeds and handle exceptions as prompted.
- Optional:View monitoring information and dashboards. If you select Enable Kibana Monitoring or Enable Kibana Dashboard in the Configure Shipper step, you can view the monitoring information or dashboards in the Kibana console of the destination Elasticsearch cluster after the shipper is started.
- In the Manage Shippers section, find the shipper that you created, move the pointer over
in the Actions column, and then select Dashboard.
- On the logon page of the Kibana console, enter the username and password, and click Log in.
- In the left-side navigation pane, click Dashboard and click a metric whose dashboard you want to view. Then, you can view the dashboard of the metric.
- In the left-side navigation pane, click Monitoring and select a monitoring item whose information you want to view. Then, you can view the information of the monitoring item.
- In the Manage Shippers section, find the shipper that you created, move the pointer over
References
API operation for creating a shipper: CreateCollector
FAQ
- How do I use a Filebeat shipper to collect logs from multiple directories? When you configure the Filebeat Log File Path parameter for your Filebeat shipper, click Add, enter the directory from which you want to collect logs, and then add the directory to the value of the paths parameter in Shipper YML Configuration.
- What do I do if I fail to install a Beats shipper on an ECS instance?
Troubleshoot the issue by following the instructions provided in Installation failures of Beats shippers.