Elasticsearch:Collect ECS service logs using Filebeat

Procedure

1. Log on to the Alibaba Cloud Elasticsearch console.

2. Navigate to the Beats Data Shippers page.

   1. In the top navigation bar, select a region.

   2. In the left-side navigation pane, click Beats Data Shippers.

   3. Optional: If this is the first time you go to the Beats Data Shippers page, view the information displayed in the message that appears and click OK to authorize the system to create a service-linked role for your account.

      Note

      When Beats collects data from various data sources, Beats depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Alibaba Cloud ES service-linked roles.

3. Configure and start collecting ECS service logs.

   Note

   The following steps use Filebeat as an example.

   1. In the Create Shipper section, hover over Filebeat and click ECS Logs.

      

      Note

      For other types of shippers, such as Metricbeat, you can click the shipper name directly.

   2. In the Configure Shipper wizard, enter or select the shipper information.

      

      Parameter	Description
      Shipper Name	Enter a custom name for the shipper. The name must be 1–30 characters, start with a letter, and contain only letters, digits, underscores (_), and hyphens (-).
      Version	Only version 6.8.5 is supported for Filebeat.
      Output	The destination for data from Filebeat. Select an existing Alibaba Cloud Elasticsearch or Logstash instance. The access protocol must match the protocol of the selected Elasticsearch instance. Important Elasticsearch instances of version 8.x or later are not supported.
      Username and Password	If you select Output as the Elasticsearch, you must provide the corresponding username and password to allow Filebeat to write data to the Elasticsearch instance. The default username is elastic. You set the password when you created the instance. If you have forgotten the password, you can reset it. For more information, see Reset the access password of an instance.
      Enable Kibana Monitoring	Enables monitoring for Filebeat metrics. If you select Output as the Elasticsearch, monitoring uses the same Alibaba Cloud Elasticsearch instance as the Output by default. If you select Output as the Logstash, you must add configuration to the configuration file.
      Enable Kibana Dashboard	Configures the default Kibana dashboard. Because Alibaba Cloud Kibana is deployed in a VPC, you must enable private network access for Kibana on its configuration page. For more information, see Log on to the Kibana console.
      Filebeat File Path	Since Alibaba Cloud uses Docker to deploy Beats, you must map the host directories for log collection into the Docker container. The path you enter here must be a parent directory of the paths specified in the filebeat.yml file's input.path parameter. Wildcards (*) are not allowed in this field. For example, if input.path: /var/log/*/*.log is set, the Filebeat file directory can be /var/ or /var/log/. Important The path can be a directory or a full path. You can also use glob patterns to match specific types of files within a path. If you need to collect logs from multiple directories, click Add to enter another path. You must also add the new path to the paths parameter in the Shipper YML Configuration.
      Shipper YML Configuration	The shipper configuration file. Use the following sample to ensure that log collection is enabled and that the directories in paths are consistent with the directories you entered for Filebeat file directories. filebeat.inputs: - type: log enabled: true paths: - /var/log/*.log For other parameters, modify the configuration based on your business requirements. For more information, see Shipper YML configuration.

      Important

      • After you specify the Output, you do not need to configure it again in the Shipper YML Configuration. Doing so will cause an installation error.

      • Alibaba Cloud Filebeat does not support custom index names. By default, it creates an index with the name filebeat-%{[agent.version]}-%{+yyyy.MM.dd}, for example, filebeat-6.8.5-2021.12.11.

   3. Click Next.

   4. Optional: If this is your first time in the Install Shipper wizard, click Authorize Now. On the Cloud Resource Access Authorization page, click Authorize to grant the Elasticsearch instance permissions to access ECS.

      

      • The authorization service is provided by Resource Access Management (RAM). After you grant authorization, the system automatically creates two service-linked roles: AliyunElasticsearchAccessingOOSRole and AliyunOOSAccessingECS4ESRole. The default system policies attached are AliyunElasticsearchAccessingOOSRolePolicy and AliyunOOSAccessingECS4ESRolePolicy, respectively. Do not delete these system roles or policies.

      • If you delete a default policy or system role from the RAM console, you can re-grant the permissions by using Cloud Resource Access Authorization. You must also grant the RAM user the following custom permissions to avoid permission errors. For instructions on how to create a custom policy, see Create a custom policy.

        {
            "Version": "1",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "ram:CreateRole",
                        "ram:AttachPolicyToRole",
                        "ram:GetRole"
                    ],
                    "Resource": "acs:ram:*:*:role/*",
                    "Condition": {
                        "ForAllValues:StringEquals": {
                            "ram:TrustedPrincipalTypes": "Service",
                            "ram:ServiceNames": [
                                "elasticsearch.aliyuncs.com",
                                "oos.aliyuncs.com"
                            ]
                        }
                    }
                },
                {
                    "Effect": "Allow",
                    "Action": "ram:AttachPolicyToRole",
                    "Resource": "acs:ram:*:*:policy/*"
                },
                {
                    "Action": "ram:PassRole",
                    "Resource": "*",
                    "Effect": "Allow",
                    "Condition": {
                        "StringEquals": {
                            "acs:Service": "elasticsearch.aliyuncs.com"
                        }
                    }
                }
            ]
        }

   5. In the Install Shipper wizard, select the target ECS instances.

      Note

      The instance list displays all ECS instances under your account that are in the same VPC as the Elasticsearch or Logstash instance selected for the Output. You can install the shipper only on ECS instances that have both Cloud Assistant and Docker installed.

   6. Click Start.

   7. In the Start Shipper dialog box, click Back to Beats Shippers. View the newly created shipper in the shipper list.

      Wait for the Shipper Status to change to Enabled, which indicates that the shipper is ready. The two numbers after Enabled represent the number of successful ECS installations and the total number of target ECS instances. If the installation succeeds on all instances, the two numbers will match.

4. View running instances.

   After creating the shipper, you can view its running instances to check the installation status and troubleshoot exceptions.

   1. In the Manage Shippers section, find the shipper and click View Instances in the Actions column.

   2. On the View Instances page, check the Installed Shippers.

      The Installed Shippers can be one of the following: Normal Heartbeat, No Heartbeat, or Installation Failed. If the status is No Heartbeat or Installation Failed, you can remove the problematic node or retry the installation. If the retry fails, see Troubleshoot Beats installation failures.

5. Optional: View the monitoring page or dashboard.

   If you selected Enable Kibana Monitoring or Enable Kibana Dashboard when creating the shipper, you can view the monitoring information or dashboard in Kibana after the shipper starts.

   1. In the Manage Shippers section, find the shipper, click the > Actions > View Dashboards.

   2. On the Kibana logon page, enter your username and password, and click Log in.

   3. In the left-side navigation pane, click Dashboard. Then, click a metric to view its dashboard.

   4. In the left-side navigation pane, click Monitoring. Then, click a monitoring item to view its information.

Related documentation

API operation for creating a shipper (referred to as a collector in the API): CreateCollector

FAQ

• How do I use Beats to collect logs from multiple directories?

  When you configure the shipper, click Add, enter the path in the text box, and add the path to the paths parameter in the Shipper YML Configuration.

• What do I do if the Beats installation on an ECS instance fails?

  For troubleshooting information, see Troubleshoot Beats installation failures.