Elasticsearch:Collect the logs of an ACK cluster

Precautions

• Alibaba Cloud Filebeat shippers support only Docker containers. The shippers do not support containers such as containerd containers and sandboxed containers. If you use containers other than Docker containers, Filebeat shippers are stuck in the active state after the shippers are created.

• Alibaba Cloud Filebeat shippers can be installed only on ACK dedicated clusters or ACK managed clusters.

  Important

  Filebeat shippers can be installed on Kubernetes clusters only of 1.18 or 1.20.

Procedure

1. Log on to the Alibaba Cloud Elasticsearch console.
2. Navigate to the Beats Data Shippers page.
   1. In the top navigation bar, select a region.
   2. In the left-side navigation pane, click Beats Data Shippers.
   3. Optional:If this is the first time you go to the Beats Data Shippers page, view the information displayed in the message that appears and click OK to authorize the system to create a service-linked role for your account.
      Note When Beats collects data from various data sources, Beats depends on the service-linked role and the rules specified for the role. Do not delete the service-linked role. Otherwise, the use of Beats is affected. For more information, see Overview of the Elasticsearch service-linked role

3. In the Create Shipper section, move the pointer over Filebeat and click ACK Logs.

4. In the Select Destination Elasticsearch Cluster step, configure the parameters.

   

   Parameter	Description
   Shipper Name	The name of the shipper. The name must be 1 to 30 characters in length and can contain letters, digits, underscores (_), and hyphens (-). The name must start with a letter.
   Version	Set this parameter to 6.8.13, which is the only version supported by Filebeat.
   Output	The destination for the data collected by the shipper. The destination is the Elasticsearch cluster that you created. The protocol must be the same as that of the selected Elasticsearch cluster. Important Elasticsearch V8.X clusters are not supported.
   Username/Password	The username and password that are used to access the Elasticsearch cluster. The default username is elastic. The password is specified when you create the Elasticsearch cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting the password, see Reset the access password of an Elasticsearch cluster.
   Enable Kibana Monitoring	Specifies whether to monitor the metrics of the shipper. If you select Elasticsearch for Output, the Kibana monitor uses the Elasticsearch cluster that you configured for the Output parameter as the destination.
   Enable Kibana Dashboard	Specifies whether to enable the default Kibana dashboard. Alibaba Cloud Kibana is deployed in a virtual private cloud (VPC). You must enable the Private Network Access feature for Kibana on the Kibana Configuration page. For more information, see Configure a public or private IP address whitelist for Kibana.

5. Click Next. In the Configure Collection Object step, configure a collection object.

   1. Select the desired ACK cluster from the Source ACK Cluster drop-down list.

      Important

      You must select a running ACK cluster that resides in the same VPC as the Elasticsearch cluster and is not an ACK edge cluster. For more information, see What is ACK Edge?

   2. Click Install to install ES-operator for the ACK cluster. Beats is dependent on ES-operator.

      If the Install button is not displayed, ES-operator is installed. If the Install button is not displayed after you install ES-operator, the installation is successful.

   3. Click Create Collection Object in the lower-left corner to configure a collection object. You can configure multiple collection objects.

      

      Parameter	Description
      Object Name	The name of the collection object. You can create multiple collection objects. The name of each collection object must be unique.
      Namespace	The namespace of the ACK cluster in which the pod from which you want to collect logs is deployed. By default, the namespace default is used if you do not specify a namespace when you create the pod.
      Pod Label	The label of the pod. You can specify multiple labels. The specified labels have logical AND relations. Important You can delete the labels that you specified only if at least two labels are specified. You can specify only the labels defined for the pod. If you specify the labels defined for other objects such as Deployments, the shipper fails to be created.
      Container Name	The complete name of the container. If you leave this parameter empty, the shipper collects logs from all the containers in the namespace that comply with the labels added to the pod.

      Note

      For more information about how to obtain the configurations of the pod, such as the namespace, labels, and name, see Manage pods.

6. Click Next. In the Configure Log Collection step, click Add Log Collection Configuration in the lower-left corner to configure log collection information. You can specify multiple containers from which you want to collect logs.

   Parameter	Description
   Log Name	You can specify multiple containers for each pod. Each container corresponds to a log name. Each log name must be unique. A log name can be a part of an index name and used for subsequent collection output.
   Configure Shipper	The general template for collecting logs from a Docker container is used. The Autodiscover provider is integrated into the log collection configuration. The collection configuration supports Docker inputs. type: the input type. If you want to collect logs from containers, the value of this parameter is docker. The value of this parameter varies based on the input type. For more information, see Configure inputs. combine_partial: specifies whether to enable partial message joining. For more information, see Docker input (combine_partial). container ids: the IDs of the Docker containers from which you want to collect logs. For more information, see Docker input (containers.ids). fields.k8s_container_name: Add the k8s_container_name field to the output information of the shipper to reference the variable ${data.kubernetes.container.name}. Note The Autodiscover provider is integrated into the Docker container configuration. You can reference the configuration of the Autodiscover provider to configure the Docker container. fileds.k8s_node_name: Add the k8s_node_name field to the output information of the shipper to reference the variable ${data.kubernetes.node.name}. fileds.k8s_pod: Add the k8s_pod field to the output information of the shipper to reference the variable ${data.kubernetes.pod.name}. fileds.k8s_pod_namespace: Add the k8s_pod_namespace field to the output information of the shipper to reference the variable ${data.kubernetes.namespace}. fields_under_root: If you set this parameter to true, the fields are stored as top-level fields in the output document. For more information, see Docker input (fields_under_root). Note If the general template cannot meet your requirements, you can modify the configurations. For more information, see Filebeat Docker input. You can configure only one Docker container in each collection configuration. If you want to configure multiple Docker containers, click Add Log Collection Configuration to perform the operation.

7. (Optional) Click Next. In the Manage Index Storage step, enable and configure the Index Storage Management for Collected Data feature based on your business requirements.

   After you enable the Index Storage Management for Collected Data feature, click Add Management Policy to create and configure an index management policy. You can configure multiple index management policies. To create and configure an index management policy, configure the following parameters.

   Parameter	Description
   Policy Name	The name of the policy. You can customize a name.
   Log Name	The name of the log file that you want to associate with the policy. You must select at least one name. Multiple log files can be associated with the same policy, but each log file can be associated with only one policy.
   Maximum Storage Space	If the disk space consumed by an index (including the replica shards of the index) reaches the value of this parameter, the system deletes old data to save disk space.
   Lifecycle Management	Specifies whether to enable lifecycle management for the index. After you turn on Lifecycle Management, the system separates hot data from cold data for data nodes and automatically deletes old data. Important If you turn on Rolling Update, the shipper writes data to an index whose name is in the <Log name>-<Date>-<Serial number> format, such as log-web-2021.01.22-000001. If you do not turn on Rolling Update, the shipper writes data to an index whose name is in the filebeat-<Log name>-<Date> format.

8. Click Enable.

   After the shipper is enabled, you can view the information about the shipper in the Manage Shippers section. You can also perform the operations provided in the following table.

   Note

   By default, resources for the shipper are deployed in the logging namespace of the ACK cluster. After the resources are deployed, you can access the ACK cluster to view the resources, such as containers for index management and the rolling update policy for index lifecycles. For more information, see View the resources deployed in the ACK cluster for the shipper.

   Operation	Description
   View Configuration	View information such as the destination Elasticsearch cluster, source ACK cluster, and name of the log collection task of the shipper. You cannot modify the information.
   Modify Configuration	Modify collection objects, log collection configurations, and index storage management policies.
   More	Enable, disable, restart, or delete the log collection task, and view the information about the task in the dashboard and helm charts.

View the resources deployed in the ACK cluster for the shipper

Run the following command, use kubectl to connect to the ACK cluster, and view the resources for the shipper in the logging namespace. For more information, see Obtain the kubeconfig file of a cluster and use kubectl to connect to the cluster.

kubectl get pods -n logging