Collect the logs of an ECS instance

Procedure

1. Log on to the Elasticsearch console.
2. In the top navigation bar, select a region. In the left-side navigation pane, click Beats Data Shippers.
3. Optional:If this is the first time you go to the Beats Data Shippers page, click Confirm in the Confirm Service Authorization message to authorize the system to create a service-linked role for your account.
   

   Notice When Beats collects data from various data sources, it depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Overview of the Elasticsearch service-linked role.
4. Configure and enable Filebeat to collect the logs of the ECS instance.
   1. In the Create Shipper section, move the pointer over Filebeat and click ECS Logs.
      

      Note For other shippers on this page, click the shipper name. For example, to create a Metricbeat shipper, click Metricbeat.
   2. In the Configure Shipper step, configure the following parameters. Then, click Next.
      

      Parameter	Description
      Shipper Name	The name of the shipper. Enter a name for the shipper. The name must be 1 to 30 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.
      Version	Set Version to 6.8.5, which is the only version supported by Filebeat.
      Output	The destination for the data collected by Filebeat. The system provides Elasticsearch and Logstash clusters for you to select. If you select Elasticsearch for Output, the protocol must be the same as that of the selected Elasticsearch cluster.
      Username/Password	If you select Elasticsearch for Output, enter the username and password used to access the Elasticsearch cluster. This way, Filebeat can write data to the cluster. The default username is elastic. The password is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting the password, see Reset the access password for an Elasticsearch cluster.
      Enable Kibana Monitoring	Determine whether to monitor the metrics of Filebeat. If you select Elasticsearch for Output, the Kibana monitor uses the same Alibaba Cloud Elasticsearch cluster as Output. If you select Logstash for Output, you must configure a monitor in the configuration file.
      Enable Kibana Dashboard	Determine whether to enable the default Kibana dashboard. Alibaba Cloud Kibana is configured in a VPC. You must enable the Private Network Access feature for Kibana on the Kibana configuration page. For more information, see Configure an IP address whitelist for access to the Kibana console over the Internet or an internal network.
      Filebeat Log File Path	This parameter is specific to Filebeat. Alibaba Cloud deploys Beats with Docker. You must map the directory from which logs are collected to Docker. We recommend that you specify a directory that is consistent with input.path in filebeat.yml.
      Shipper YML Configuration	Prepare configuration files for the shipper. You can modify the YML configuration files based on your business requirements. For more information, see Prepare the YML configuration files for a shipper.

      Notice If you already specify Output, you do not need to specify it again in Shipper YML Configuration. Otherwise, the system reports a shipper installation error.
   3. In the Install Shipper step, select the ECS instance on which you want to install a shipper.
      

      Notice The system displays all the ECS instances within your account that reside in the same VPC as the Elasticsearch or Logstash cluster selected for Output. A shipper can be installed only on an ECS instance on which Cloud Assistant and Docker are installed.
   4. Click Enable.
   5. In the Enable Shipper dialog box, click Back to Beats Shippers. In the Manage Shippers section of the Beats Data Shippers page, view the newly created shipper.
      After the Status of the shipper changes to Enabled, the shipper is created. The two numbers following Enabled indicate the number of ECS instances on which the shipper is installed and the total number of ECS instances on which you want to install the shipper. If the shipper is installed on all ECS instances, the two numbers are equal.
5. View the running ECS instance.
   After the shipper is created, you can view the running ECS instance to check whether the shipper installation succeeds and handle exceptions as prompted.
   1. In the Manage Shippers section, find the shipper that you created and click View Instances in the Actions column.
   2. In the View Instances panel, check whether the shipper installation on the ECS instance succeeds.
      The Installed Shippers column provides the values Heartbeat Normal, Heartbeat Abnormal, and Installation Failed to indicate whether the shipper installation on an ECS instance succeeds. If the value of Installed Shippers is Heartbeat Abnormal or Installation Failed, you can remove the instance or retry the installation on the instance. If the retry fails, you can troubleshoot the issue based on the instructions provided in Installation failures of Beats shippers.
   3. Click Add Instance to add the ECS instances on which you want to install a shipper with the same configuration and type as the created shipper.
6. Optional:View monitoring and dashboard information.
   If you select Enable Kibana Monitoring or Enable Kibana Dashboard in the Configure Shipper step, you can view the monitoring information or dashboard graphs in the Kibana console of your Elasticsearch cluster after Beats is started.
   1. In the Manage Shippers section, find the shipper that you created and choose More > Dashboard in the Actions column.
   2. On the logon page of the Kibana console, enter the username and password, and click Log in.
   3. In the left-side navigation pane, click Dashboard and click a metric whose dashboard you want to view. Then, you can view the dashboard of the metric.
      
   4. In the left-side navigation pane, click Monitoring and select a monitoring item whose information you want to view. Then, you can view the information of the monitoring item.