Elasticsearch:Collect ECS service logs using Filebeat

Step 1: Open the Beats Data Shippers page

1. Log on to the Alibaba Cloud Elasticsearch console.

2. In the top navigation bar, select a region.

3. In the left-side navigation pane, click Beats Data Shippers.

4. (Optional) If this is your first visit, read the message that appears and click OK to authorize the system to create a service-linked role for your account.

Beats depends on this service-linked role to access your data sources. Do not delete it. For details, see Elasticsearch service-linked roles.

Step 2: Configure the Filebeat shipper

1. In the Create Shipper section, hover over Filebeat and click ECS Logs.

   For other shipper types, click the shipper name directly — for example, click Metricbeat to create a Metricbeat shipper.

   

2. In the Configure Shipper step, fill in the parameters described in the following table. A minimal Shipper YML Configuration example:

   Important

   • Do not configure the output destination in Shipper YML Configuration — you already selected it in the Output field. Duplicating it causes a shipper installation error.

   • Index names are fixed and cannot be customized. The system creates indexes in the format filebeat-%{[agent.version]}-%{+yyyy.MM.dd} — for example, filebeat-6.8.5-2021.12.11.

   Parameter	Description
   Shipper Name	1–30 characters; letters, digits, underscores (_), and hyphens (-); must start with a letter.
   Version	Set to 6.8.5 — the only version supported for Filebeat.
   Output	The destination cluster. Select from your existing Elasticsearch or Logstash clusters. The protocol must match the selected Elasticsearch cluster. Elasticsearch V8.X clusters are not supported.
   Username and Password	If you select Elasticsearch for Output, enter the credentials for the destination Elasticsearch cluster. The default username is elastic. The password was set when you created the cluster; reset it if needed. See Reset the access password for an Elasticsearch cluster.
   Enable Kibana Monitoring	Monitors shipper metrics in Kibana. If Output is Elasticsearch, the same cluster is used for monitoring. If Output is Logstash, configure a monitor in the Logstash cluster's configuration file.
   Enable Kibana Dashboard	Enables the default Kibana dashboard. Kibana is deployed in a VPC — enable the Private Network Access feature for Kibana beforehand. See Configure a public or private IP address whitelist for Kibana.
   Filebeat File Path	The directory on the ECS instance from which logs are collected. This path is mounted into the Docker container and must match the paths value in Shipper YML Configuration. For paths containing wildcards (*), specify only the non-wildcard parent directory — for example, to collect from /var/log/*/*.log, enter /var/ or /var/log/. To collect from multiple directories, click Add and enter each path, then add all paths to the paths parameter in Shipper YML Configuration.
   Shipper YML Configuration	The Filebeat configuration. Log collection must be enabled and paths must match Filebeat File Path. See Prepare a YML configuration file for a shipper for additional options.

   filebeat.inputs:
   - type: log
     enabled: true
     paths:
     - /var/log/*.log

   

3. Click Next.

Step 3: Authorize and install the shipper

1. (Optional) If this is your first time on the Install Shipper step, click Authorize Now. On the Cloud Resource Access Authorization page, click Agree to Authorization. This creates two system roles via Resource Access Management (RAM): Do not delete these roles or policies while Beats is in use. If they are accidentally deleted, go to the Cloud Resource Access Authorization page to re-authorize. Your RAM user also needs the following custom policy — see Create a custom policy:

   • AliyunElasticsearchAccessingOOSRole (policy: AliyunElasticsearchAccessingOOSRolePolicy)

   • AliyunOOSAccessingECS4ESRole (policy: AliyunOOSAccessingECS4ESRolePolicy)

   {
       "Version": "1",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ram:CreateRole",
                   "ram:AttachPolicyToRole",
                   "ram:GetRole"
               ],
               "Resource": "acs:ram:*:*:role/*",
               "Condition": {
                   "ForAllValues:StringEquals": {
                       "ram:TrustedPrincipalTypes": "Service",
                       "ram:ServiceNames": [
                           "elasticsearch.aliyuncs.com",
                           "oos.aliyuncs.com"
                       ]
                   }
               }
           },
           {
               "Effect": "Allow",
               "Action": "ram:AttachPolicyToRole",
               "Resource": "acs:ram:*:*:policy/*"
           },
           {
               "Action": "ram:PassRole",
               "Resource": "*",
               "Effect": "Allow",
               "Condition": {
                   "StringEquals": {
                       "acs:Service": "elasticsearch.aliyuncs.com"
                   }
               }
           }
       ]
   }

   

2. Select the ECS instance on which you want to install the shipper.

   The list shows all ECS instances in your account that share the same VPC as the Output cluster. Only instances with Cloud Assistant and Docker installed can run a shipper.

3. Click Start.

4. In the Start Shipper dialog box, click Back to Beats Shippers.

Step 4: Verify the shipper is running

In the Manage Shippers section, find your newly created shipper and check its Status column.

• Enabled — the shipper was created. The two numbers that follow (for example, 2/3) show how many ECS instances have the shipper installed versus the total requested. When both numbers match, installation is complete on all instances.

To check per-instance status, click View Instances in the Actions column. The Installed Shippers column shows one of three values:

Step 5: View monitoring and dashboards (optional)

If you enabled Kibana Monitoring or Kibana Dashboard during configuration, access them from the Kibana console after the shipper starts.

1. In the Manage Shippers section, hover over in the Actions column for your shipper, and select View Dashboards.

2. Log in to the Kibana console with your username and password.

3. To view dashboards: click Dashboard in the left-side navigation pane, then select a metric.

4. To view monitoring data: click Monitoring in the left-side navigation pane, then select a monitoring item.

FAQ

How do I collect logs from multiple directories?

Click Add when configuring the Filebeat File Path parameter, enter each additional directory, and add all paths to the paths parameter in Shipper YML Configuration.

What do I do if Beats shipper installation fails on an ECS instance?

Follow the troubleshooting steps in Installation failures of Beats shippers.

API reference

To create a shipper programmatically, use the CreateCollector API operation.