This topic uses Filebeat as an example to describe how to install and manage a Beats data shipper. A shipper collects the data of your Elastic Compute Service (ECS) instance, which includes log files, network data, and server metrics. Then, the shipper sends the data to your Alibaba Cloud Elasticsearch or Logstash cluster for further operations, such as monitoring and analytics.

Prerequisites

You have completed the following operations:
  • An Alibaba Cloud Elasticsearch cluster is created.

    For more information, see Create an Alibaba Cloud Elasticsearch instance.

  • The Auto Indexing feature is enabled for the Alibaba Cloud Elasticsearch cluster.

    For security purposes, Alibaba Cloud Elasticsearch disables Auto Indexing by default. However, Beats depends on this feature. If you select Elasticsearch for Output when you create a shipper, you must enable the Auto Indexing feature. For more information, see Enable auto indexing.

  • An Alibaba Cloud ECS instance is created, and it is in the same Virtual Private Cloud (VPC) as the Alibaba Cloud Elasticsearch or Logstash cluster.

    For more information, see Create an instance by using the provided wizard.

    Notice
    • The default installation directory of Beats is /opt/aliyunbeats/. After you install Beats, the conf, logs, and data directories are generated on the ECS instance. The conf directory contains the configuration file. The logs directory contains the Beats log file. The data directory contains the Beats data file. We recommend that you do not delete or modify the content of these files. Otherwise, errors may occur or data may be altered. If an error occurs, you can view Beats logs in the logs directory to locate the error.
    • Beats now supports only Aliyun Linux, Red Hat, and CentOS.
  • Cloud Assistant and Docker are installed on the ECS instance.

    For more information, see Install the Cloud Assistant client and Build Docker.

Procedure

  1. Log on to the Alibaba Cloud Beats console.
  2. In the Create Shipper section, click Filebeat.
    Create Shipper section
  3. In the Configure Shipper step, specify parameters as required.
    Configure Shipper
    Table 1. Parameters used to create a Filebeat shipper
    Parameter Description
    Shipper Name Enter a name for the shipper.
    Version Select 6.8.5, which is the only version supported by Filebeat.
    Output Select a destination for the data collected by Filebeat. The system provides Elasticsearch and Logstash for you to select. The access protocol must be consistent with that of the Alibaba Cloud Elasticsearch cluster.
    Username/Password If you select Elasticsearch for Output, enter the username and password for Filebeat to write data to the Alibaba Cloud Elasticsearch cluster.
    Enable Kibana Monitoring Determine whether to monitor the metrics of Filebeat. If you select Elasticsearch for Output, the Kibana monitor uses the same Alibaba Cloud Elasticsearch cluster as Output. If you select Logstash for Output, you must separately configure a monitor in the configuration file.
    Enable Kibana Dashboard Determine whether to enable the default Kibana dashboard. Alibaba Cloud Kibana is configured in a VPC. You must enable the Kibana private network access feature on the Kibana configuration page. For more information, see Network access configuration.
    Filebeat Log File Path This parameter is specific to Filebeat. Alibaba Cloud deploys Beats with Docker. You must map the directory from which logs are collected to Docker. We recommend that you enter a directory that is consistent with input.path in filebeat.yml.
    Shipper YML Configuration Prepare configuration files for the shipper. You can modify the YML configuration files based on your business requirements. For more information, see Prepare the YML configuration files for a shipper.
    Notice If you already specify Output, you do not need to specify it again in Shipper YML Configuration. If you specify it again, the system prompts a shipper installation error.
  4. Click Next.
  5. In the Install Shipper step, select the target ECS instance.
    Install Shipper
    Notice The instance list displays all ECS instances that are in the same VPC as the Alibaba Cloud Elasticsearch or Logstash cluster that you select for Output and have Cloud Assistant and Docker installed.
  6. Click Enable.
  7. In the Enable Shipper dialog box, click Back to Beats Shippers to view the created shipper.
    After the Status of the shipper changes to Enabled, the shipper is created. The two numbers following Enabled indicate the number of ECS instances where the shipper is installed and the number of total target ECS instances. If the shipper is installed on all the ECS instances, the two numbers are equal.Status of the effective shipper
  8. View running ECS instances.
    After the shipper is created, you can view running ECS instances to check whether the shipper installation succeeds and handle exceptions as prompted.
    1. In the Manage Shippers section, find the target shipper and click View Instances in the Actions column.
    2. In the View Instances pane, check whether the shipper installation on the ECS instance succeeds.
      The Installed Shippers column provides the values Heartbeat Normal, Heartbeat Abnormal, and Installation Failed to indicate whether the shipper installation on an ECS instance succeeds. If the value of Installed Shippers is Heartbeat Abnormal or Installation Failed, you can remove the instance or retry the installation on the instance. If the retry also fails, check whether the prerequisites are met.View running ECS instances
    3. Click Add Instance to add ECS instances where you want to install a shipper with the same configuration and type as the created shipper.
  9. Optional:View monitoring and dashboard information.
    If you select Enable Kibana Monitoring or Enable Kibana Dashboard when you create the shipper, you can view the monitoring and dashboard information in the Kibana console after Beats is started.
    1. In the Manage Shippers section, find the target shipper and choose More > Dashboard in the Actions column.
    2. On the logon page of the Kibana console, enter the username and password, and click Log in.
    3. In the left-side navigation pane, click Dashboard to view the dashboard tables and graphs.
      View dashboard
    4. In the left-side navigation pane, click Monitoring to view monitoring information.
      View monitoring information