You can use Alibaba Cloud Filebeat to collect the logs of Container Service for Kubernetes (ACK) clusters and send the collected logs to Alibaba Cloud Elasticsearch for analysis and presentation. This topic describes how to configure Filebeat to collect the logs of an ACK cluster. This topic also describes the containers of Filebeat.

Prerequisites

  • An Alibaba Cloud Elasticsearch cluster is created.

    For more information, see Create an Alibaba Cloud Elasticsearch cluster.

  • A prefix is customized for the names of indexes that will be automatically created based on the Auto Indexing feature.
    To avoid conflicts between the aliases of the indexes that are generated during a rollover and the names of the indexes, we recommend that you customize the filebeat- prefix for index names. You can enter +.*,+filebeat-*,-* in the Custom field. For more information, see Configure the YML file. Customize a prefix for index names
    Notice If you enable the Rolling Update feature when you configure lifecycle management for indexes in your Elasticsearch cluster, you must disable the Auto Indexing feature to avoid conflicts between the aliases of the indexes that are generated after rolling updates and the names of the indexes. If you do not enable the Rolling Update feature for indexes, you must enable the Auto Indexing feature. We recommend that you customize a prefix for index names.
  • The permissions on Beats and ACK clusters are granted to a RAM user.

    For more information, see Create a custom policy and Assign RBAC roles to RAM users.

  • An ACK cluster is created, and a pod is created in the cluster. In this example, an NGINX container is used.

    For more information, see Create a managed Kubernetes cluster.

Procedure

  1. Log on to the Elasticsearch console.
  2. In the top navigation bar, select a region. In the left-side navigation pane, click Beats Data Shippers.
  3. Optional:If this is the first time you go to the Beats Data Shippers page, click Confirm in the Confirm Service Authorization message to authorize the system to create a service-linked role for your account.
    Confirm Service Authorization
    Notice When Beats collects data from various data sources, it depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Overview of the Elasticsearch service-linked role.
  4. In the Create Shipper section, move the pointer over Filebeat and click ACK Logs.
  5. In the Select Destination Elasticsearch Cluster step, configure the parameters. Then, click Next.
    Select Destination Elasticsearch Cluster
    Parameter Description
    Shipper Name The name of the shipper. The name must be 1 to 30 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.
    Version Set this parameter to 6.8.13, which is the only version supported by Filebeat.
    Output The destination for the data collected by the shipper. The destination is the Elasticsearch cluster you created. The protocol must be the same as that of the selected Elasticsearch cluster.
    Username/Password The username and password that are used to access the Elasticsearch cluster. The default username is elastic. The password is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting the password, see Reset the access password for an Elasticsearch cluster.
    Enable Kibana Monitoring Specifies whether to monitor the metrics of the shipper. If you select Elasticsearch for Output, the Kibana monitor uses the Elasticsearch cluster that you configured for the Output parameter as the destination.
    Enable Kibana Dashboard Specifies whether to enable the default Kibana dashboard. Alibaba Cloud Kibana is deployed in a virtual private cloud (VPC). You must enable the Private Network Access feature for Kibana on the Kibana Configuration page. For more information, see Configure an IP address whitelist for access to the Kibana console over the Internet or an internal network.
  6. In the Configure Collection Object step, configure a collection object.
    1. Select an option from the Source ACK Cluster drop-down list.
      Notice You must select a running ACK cluster that resides in the same VPC as the Elasticsearch cluster and is not a managed edge ACK cluster. For more information, see ACK@Edge overview.
    2. Optional:Click Install to install ES-operator for the ACK cluster. Beats is dependent on ES-operator.
      If the Install button is not displayed, ES-operator is installed. If the Install button is not displayed after you install ES-operator, the installation is successful.
    3. Click Create Collection Object in the lower-left corner to configure a collection object. You can configure multiple collection objects.
      Create Collection Object
      Parameter Description
      Object Name The name of the collection object. You can create multiple collection objects. The name of each collection object must be unique.
      Namespace The namespace of the ACK cluster in which the pod from which you want to collect logs is deployed. By default, the namespace default is used if you do not specify a namespace when you create the pod.
      Pod Label The label of the pod. You can specify multiple labels. The specified labels have logical AND relations.
      Notice
      • You can delete the labels that you specified only if at least two labels are specified.
      • You can specify only the labels defined for the pod. If you specify the labels defined for other objects such as Deployments, the shipper fails to be created.
      Container Name The complete name of the container. If you leave this parameter empty, the shipper collects logs from all the containers in the namespace that comply with the labels added to the pod.
      Note For more information about how to obtain the configurations of the pod, such as the namespace, labels, and name, see Manage pods.
    4. Click Next.
  7. In the Configure Log Collection step, click Add Log Collection Configuration in the lower-left corner to configure log collection information. You can specify multiple containers from which you want to collect logs. Then, click Next. Configure Log Collection
    Parameter Description
    Log Name You can specify multiple containers for each pod. Each container corresponds to a log name. Each log name must be unique. A log name can be a part of an index name and used for subsequent collection output.
    Configure Shipper The general template for collecting logs from a Docker container is used. The Autodiscover provider is integrated into the log collection configuration. The collection configuration supports Docker inputs.
    • type: the input type. If you want to collect logs from containers, the value of this parameter is docker. The value of this parameter varies based on the input type. For more information, see Configure inputs.
    • combine_partial: specifies whether to enable partial message joining. For more information, see Docker input (combine_partial).
    • container ids: the IDs of the Docker containers from which you want to collect logs. For more information, see Docker input (containers.ids).
    • fields.k8s_container_name: Add the k8s_container_name field to the output information of the shipper to reference the variable ${data.kubernetes.container.name}.
      Note The Autodiscover provider is integrated into the Docker container configuration. You can reference the configuration of the Autodiscover provider to configure the Docker container.
    • fileds.k8s_node_name: Add the k8s_node_name field to the output information of the shipper to reference the variable ${data.kubernetes.node.name}.
    • fields.k8s_pod: Add the k8s_pod field to the output information of the shipper to reference the variable ${data.kubernetes.pod.name}.
    • fileds.k8s_pod_namespace: Add the k8s_pod_namespace field to the output information of the shipper to reference the variable ${data.kubernetes.namespace}.
    • fields_under_root: If you set this parameter to true, the fields are stored as top-level fields in the output document. For more information, see Docker input (fields_under_root).
    Note
    • If the general template does not meet your requirements, you can modify the configurations. For more information, see Filebeat Docker input.
    • You can configure only one Docker container in each collection configuration. If you want to configure multiple Docker containers, click Add Log Collection Configuration to perform the operation.
  8. Optional:In the Manage Index Storage step, enable and configure the Index Storage Management for Collected Data feature based on your business requirements.
    After you enable the Index Storage Management for Collected Data feature, click Add Management Policy to create and configure an index management policy. You can configure multiple index management policies. Manage Index Storage
    Parameter Description
    Policy Name The name of the policy. You can customize a name.
    Log Name The name of the log file that you want to associate with the policy. You must select at least one name. Multiple log files can be associated with the same policy, but each log file can be associated with only one policy.
    Maximum Storage Space If the disk space consumed by an index (including the replica shards of the index) reaches the value of this parameter, the system deletes old data to save disk space.
    Lifecycle Management Specifies whether to enable lifecycle management for the index. After you turn on Lifecycle Management, the system separates hot data from cold data for data nodes and automatically deletes old data. For more information, see Use ILM to separate hot data from cold data and Managing the index lifecycle.
    Notice
    • If you turn on Rolling Update, the shipper writes data to an index named <Log name>-<Date>-<Sequence number>, such as log-web-2021.01.22-000001.
    • If you do not turn on Rolling Update, the shipper writes data to an index named filebeat-<Log name>-<Date>.
  9. Click Enable.
    After the shipper is enabled, you can view the information about the shipper in the Manage Shippers section. You can also perform the operations provided in the following table.
    Note Resources for the shipper are deployed in the logging namespace of the ACK cluster.
    Operation Description
    View Configuration View information such as the destination Elasticsearch cluster, source ACK cluster, and name of the log collection task of the shipper. You cannot modify the information.
    Modify Configuration Modify collection objects, log collection configurations, and index storage management policies.
    More Enable, disable, restart, or delete the log collection task, and view the information about the task in the dashboard and helm charts.

View resources used by the shipper

Use kubectl to connect to the ACK cluster and view the resources that are used by the shipper in the logging namespace.

kubectl get pods  -n logging
fig01
Warning You are not allowed to perform operations on the resources that are deployed in the logging namespace. For example, you are not allowed to delete the resources. If you perform operations on the resources, the shipper cannot run as expected.
Pod name Description Example
Cluster name-binding-Serial number The container that is used to manage indexes, such as a container that is used to delete old data on a regular basis. ct-cn-ew8qx563gu4ng4ot6-binding-7e245-1617347400-c****
Cluster name-policy-Serial number The policy for rolling updates on indexes. ct-cn-ew8qx563gu4ng4ot6-policy-696b7-hot-rollover-1g-16173v****
Cluster name-Serial number The container for which the shipper is installed. ct-cn-ew8qx563gu4ng4ot6-q****
es-operator-Serial number The container for which ES-operator is installed. es-operator-cb63cc9a6302e4e90aeb2f79adf358b19-56fcd754db-b****