Alibaba Cloud Filebeat can be used to collect the logs of Container Service for Kubernetes (ACK) clusters and send the collected logs to Alibaba Cloud Elasticsearch for analysis and presentation. This topic describes how to configure Filebeat to collect the logs of an ACK cluster.

Prerequisites

  • An Alibaba Cloud Elasticsearch cluster is created.

    For more information, see Create a cluster.

  • A prefix is customized for the names of indexes created based on the Auto Indexing feature.
    To avoid a conflict between the alias of the index that is generated during a rollover and the index name, we recommend that you customize the filebeat-* prefix for index names. You can specify +.*,+filebeat-*,-* in the filebeat-* field. For more information, see Configure the YML file.Customize a prefix for index names
    Notice If you enable the Rolling Update feature when you configure lifecycle management for the index, you must disable the Auto Indexing feature to avoid a conflict between the alias of the index after the rolling update and the index name. If you do not enable the Rolling Update feature for the index, you must enable the Auto Indexing feature. We recommend that you customize a prefix for index names.
  • The permissions on Beats and ACK clusters are granted to a RAM user.

    For more information, see Create a custom policy and Assign RBAC roles to a RAM user.

  • An ACK cluster is created, and a pod is created in the cluster. In this topic, a NGINX container is used.

    For more information, see Create a managed Kubernetes cluster.

Procedure

  1. Log on to the Alibaba Cloud Elasticsearch console.
  2. In the top navigation bar, select the region where your cluster resides. In the left-side navigation pane, click Beats Data Shippers.
    Notice If this is the first time you go to the Beats Data Shippers page, confirm authorization as prompted.
  3. In the Create Shipper section, move the pointer over Filebeat and click ACK Logs.
  4. In the Select Destination Elasticsearch Cluster step, configure the following parameters. Then, click Next.
    Select Destination Elasticsearch Cluster
    Parameter Description
    Shipper Name The name of the shipper. Enter a name for the shipper. The name must be 1 to 30 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.
    Version Set Version to 6.8.13, which is the only version supported by Filebeat.
    Output The destination for the data collected by Filebeat. The destination is the Elasticsearch cluster you created. The protocol must be the same as that of the selected Elasticsearch cluster.
    Username/Password The username and password used to access the Elasticsearch cluster. The default username is elastic. The password is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting the password, see Reset the access password for an Elasticsearch cluster.
    Enable Kibana Monitoring Determine whether to monitor the metrics of Filebeat. If you select Elasticsearch for Output, the Kibana monitor uses the same Alibaba Cloud Elasticsearch cluster as Output.
    Enable Kibana Dashboard Determine whether to enable the default Kibana dashboard. Alibaba Cloud Kibana is configured in a VPC. You must enable the Private Network Access feature for Kibana on the Kibana configuration page. For more information, see Configure an IP address whitelist for access to the Kibana console over the Internet or an internal network.
  5. In the Configure Collection Object step, configure the collection object.
    1. Select Source ACK Cluster.
      Notice You must select a running ACK cluster that resides in the same VPC as the Elasticsearch cluster and is not a managed edge ACK cluster. For more information, see ACK@Edge overview.
    2. Optional:Click Install to install ES-operator on which Beats depends for the ACK cluster.
      If the Install button is not displayed, ES-operator has been installed. If the Install button disappears after you install ES-operator, the installation is successful.
    3. Click Create Collection Object in the lower-left corner to configure the collection object. You can configure multiple collection objects.
      Create Collection Object
      Parameter Description
      Object Name The name of the collection object. You can create multiple collection objects. The names of the collection objects must be unique.
      Namespace The namespace of the ACK cluster where the pod from which you want to collect logs is deployed. If you do not specify a namespace when you create the pod, the namespace default is used by default.
      Pod Label Add a label to the pod. If you add multiple labels to the pod, the labels have logical AND relations.
      Notice You can delete the labels that are added to the pod only if you add a minimum of two labels to it.
      Container Name The full name of the container. If this parameter is not specified, Filebeat collects logs from all containers in the namespace that comply with the labels added to the pod.
      Note For more information about how to obtain the configurations of the pod, such as the namespace, labels, and name, see View pods in ACK clusters.
    4. Click Next.
  6. In the Configure Log Collection step, click Add Log Collection Configuration in the lower-left corner to configure log collection information. You can specify multiple pods from which you want to collect logs. Click Next.Configure Log Collection
    Parameter Description
    Log Name You can specify multiple containers for each pod. Each container corresponds to a log name. Each log name must be unique. A log name can be a part of an index name and used for subsequent collection output.
    Configure Shipper A general template for collecting logs from a Docker container is used. The Autodiscover provider is integrated into the log collection configuration. The collection configuration supports Docker input.
    • type: the input type. If Filebeat collects logs from a container, the value of this parameter is docker. The value of this parameter varies based on the input type. For more information, see Configure inputs.
    • combine_partial: Enable partial message joining. For more information, see Docker input (combine_partial).
    • container ids: the IDs of the Docker containers from which you want to read logs. For more information, see Docker input (containers.ids).
    • fields.k8s_container_name: Add the k8s_container_name field to the output information of Filebeat to reference the variable ${data.kubernetes.container.name}.
      Note The Autodiscover provider is integrated into the Docker container configuration. You can reference the configuration of the Autodiscover provider to configure the Docker container.
    • fileds.k8s_node_name: Add the k8s_node_name field to the output information of Filebeat to reference the variable ${data.kubernetes.node.name}.
    • fields.k8s_pod: Add the k8s_pod field to the output information of Filebeat to reference the variable ${data.kubernetes.pod.name}.
    • fileds.k8s_pod_namespace: Add the k8s_pod_namespace field to the output information of Filebeat to reference the variable ${data.kubernetes.namespace}.
    • fields_under_root: If this parameter is set to true, the fields are stored as top-level fields in the output document. For more information, see Docker input (fields_under_root).
    Note
    • If a general template cannot meet your requirements, you can modify the configurations. For more information, see Filebeat Docker input.
    • Only one Docker container can be configured for each shipper. If you want to configure multiple Docker containers, click Add Log Collection Configuration to perform the operation.
  7. Optional:In the Manage Index Storage step, enable and configure the Index Storage Management for Collected Data feature based on your business requirements.
    After you enable the Index Storage Management for Collected Data feature, click Add Management Policy to create and configure the index management policy based on the following parameters. You can configure multiple index management policies.Manage Index Storage
    Parameter Description
    Policy Name You can specify multiple containers for each pod. Each container corresponds to a log file name. A log file name can be a part of an index name and used for subsequent collection output.
    Log Name The name of the log file that you want to collect. You must specify a minimum of one name. You can specify multiple log files for a management policy, but you can specify each log file for only one management policy.
    Maximum Storage Space If the storage space consumed by the index (including replica shards) reaches the value of this parameter, the system deletes old data to save storage space.
    Lifecycle Management Specifies whether to turn on Lifecycle Management for the index. After you turn on Lifecycle Management, the system separates hot data from warm data for data nodes and automatically deletes old data. For more information, see Use ILM to separate hot data from cold data and Managing the index lifecycle.
    Notice
    • If you turn on Rolling Update, Filebeat writes data to the index named <Log name>-<Date>-<Number>, such as log-web-2021.01.22-000001.
    • If you do not turn on Rolling Update, Filebeat writes data to the index named filebeat-<Log name>-<Date>.
  8. Click Enable.
    After the shipper is enabled, you can view it in the Manage Shippers section. You can also perform the following operations.
    Operation Description
    View Configuration View information such as the destination Elasticsearch cluster, source ACK cluster, and name of the log collection task of the shipper. You cannot modify the information.
    Modify Configuration Modify the collection object, log collection configuration, and index storage management policy.
    More Enable, disable, restart, or delete the log collection task. You can also view the information related to the task in the dashboard and helm charts.