This topic uses Filebeat as an example to describe how to install and manage a Beats data shipper. A shipper can collect the data of your Elastic Compute Service (ECS) instance, which includes log files, network data, and server metrics. The shipper then sends the data to Alibaba Cloud Elasticsearch or Logstash for further processing, such as monitoring and analytics.

Prerequisites

You have completed the following operations:
  • An Alibaba Cloud Elasticsearch cluster is created.

    For more information, see Create an Alibaba Cloud Elasticsearch cluster.

  • The Auto Indexing feature is enabled for the Elasticsearch cluster.

    For security purposes, Alibaba Cloud Elasticsearch disables the Auto Indexing feature by default. However, Beats depends on this feature. If you select Elasticsearch for Output when you create a shipper, you must enable the Auto Indexing feature. For more information, see Enable the Auto Indexing feature.

  • An Alibaba Cloud ECS instance is created in the same virtual private cloud (VPC) as the Alibaba Cloud Elasticsearch cluster.

    For more information, see Create an instance by using the wizard.

    Notice
    • The default installation directory of Beats is /opt/aliyunbeats/. After you install Beats, the conf, logs, and data directories are generated on the ECS instance. The conf directory contains the configuration file. The logs directory contains the Beats log file. The data directory contains the Beats data file. We recommend that you do not delete or modify the content of these files. Otherwise, errors may occur or data may be altered. If an error occurs, you can view the Beats logs in the logs directory to locate the error.
    • Beats is available for only Aliyun Linux, Red Hat Enterprise Linux (RHEL), and CentOS.
  • Cloud Assistant and Docker are installed on the ECS instance.

    For more information, see Install the Cloud Assistant client and Deploy and use Docker.

Procedure

  1. Log on to the Alibaba Cloud Elasticsearch console. In the left-side navigation pane, click Beats Data Shippers.
  2. In the Create Shipper section, click Filebeat.
    Create Filebeat page
  3. In the Configure Shipper step, configure the parameters.
    Configure Shipper
    Parameter Description
    Shipper Name Enter a name for the shipper. The name must be 1 to 30 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.
    Version Set Version to 6.8.5, which is the only version supported by Filebeat.
    Output Select a destination for the data collected by Filebeat. The destination is the Elasticsearch cluster you created. The protocol must be the same as that of the selected Elasticsearch cluster.
    Username/Password If you select Elasticsearch for Output, enter the username and password for Filebeat to write data to the Alibaba Cloud Elasticsearch cluster.
    Enable Kibana Monitoring Used to monitor the metrics of Filebeat. If you select Elasticsearch for Output, the Kibana monitor uses the same Alibaba Cloud Elasticsearch cluster as Output. If you select Logstash for Output, you must configure a monitor in the configuration file.
    Enable Kibana Dashboard Used to enable the default Kibana dashboard. Alibaba Cloud Kibana is configured in a VPC. You must enable private network access for Kibana on the Kibana configuration page. For more information, see Configure an IP address whitelist for access to the Kibana console over the Internet or an internal network.
    Filebeat Log File Path This parameter is specific to Filebeat. Alibaba Cloud deploys Beats with Docker. You must map the directory from which logs are collected to Docker. We recommend that you specify a directory that is consistent with input.path in filebeat.yml.
    Shipper YML Configuration Prepare configuration files for the shipper. You can modify the YML configuration files based on your business requirements. For more information, see Prepare the YML configuration files for a shipper.
    Notice If you already specify Output, you do not need to specify it again in Shipper YML Configuration. Otherwise, the system prompts a shipper installation error.
  4. Click Next.
  5. In the Install Shipper step, select the ECS instance on which you want to install a shipper.
    Install Shipper
    Notice The system displays all ECS instances under your account that reside in the same VPC as the Elasticsearch cluster selected for Output. A shipper can be installed only on an ECS instance on which Cloud Assistant and Docker are installed.
  6. Click Enable.
  7. In the Enable Shipper dialog box, click Back to Beats Shippers to view the created shipper.
    After the Status of the shipper changes to Enabled, the shipper is created. The two numbers following Enabled indicate the number of ECS instances on which the shipper is installed and the total number of ECS instances on which you want to install the shipper. If the shipper is installed on all ECS instances, the two numbers are equal.Status of the effective shipper
  8. View running ECS instances.
    After the shipper is created, you can view running ECS instances to check whether the shipper installation succeeds and handle exceptions as prompted.
    1. In the Manage Shippers section, find the shipper that you installed and click View Instances in the Actions column.
    2. In the View Instances pane, check whether the shipper installation on the ECS instance succeeds.
      The Installed Shippers column provides the values Heartbeat Normal, Heartbeat Abnormal, and Installation Failed to indicate whether the shipper installation on an ECS instance succeeds.View running ECS instances
      Notice If the value of Installed Shippers is Heartbeat Abnormal or Installation Failed, you can remove the instance or retry the installation on the instance. If the installation fails again, troubleshoot the issue. For more information, see Installation failures of Beats shippers.
    3. Click Add Instance to add ECS instances on which you want to install a shipper with the same configuration and type as the created shipper.
  9. Optional:View monitoring and dashboard information.
    If you select Enable Kibana Monitoring or Enable Kibana Dashboard in the Configure Shipper step, you can view the monitoring information or dashboard graphs in the Kibana console of your Elasticsearch cluster after Beats is started.
    1. In the Manage Shippers section, find the shipper that you installed and choose More > Dashboard in the Actions column.
    2. On the logon page of the Kibana console, enter the username and password, and click Log in.
    3. In the left-side navigation pane, click Dashboard and click a metric whose dashboard you want to view. Then, you can view the dashboard of the metric.
      View a dashboard
    4. In the left-side navigation pane, click Monitoring and select a monitoring item whose information you want to view. Then, you can view the monitoring information of the monitoring item.
      View monitoring information