If you want to use a separate Logtail process to collect logs from all containers on a Kubernetes node, you can install Logtail in a Kubernetes cluster in DaemonSet mode. This topic describes the implementation, limits, prerequisites, and procedure of collecting container text logs in DaemonSet mode.
Implementation
DaemonSet mode
In DaemonSet mode, only one Logtail container runs on a node in a Kubernetes cluster. You can use Logtail to collect logs from all containers on a node.
When a node is added to a Kubernetes cluster, the cluster automatically creates a Logtail container on the new node. When a node is removed from a Kubernetes cluster, the cluster automatically destroys the Logtail container on the current node. DaemonSets and custom identifier-based machine groups eliminate the need to manually manage Logtail processes.
Container discovery
Before a Logtail container can collect logs from other containers, the Logtail container must identify and determine which containers are running. This process is called container discovery.
In the container discovery phase, the Logtail container does not communicate with the kube-apiserver component of the Kubernetes cluster. Instead, the Logtail container communicates with the container runtime daemon of the node on which the Logtail container runs to obtain information about all containers on the node. This prevents the container discovery process from generating pressure on the kube-apiserver component. kube-apiserver is the central management component of a Kubernetes cluster and is responsible for providing API services. For more information, see kube-apiserver.
When you use Logtail to collect logs, you can specify conditions such as namespaces, pod names, pod labels, and container environment variables to determine containers from which Logtail collects or does not collect logs.
File path mapping for containers
Pods in a Kubernetes cluster are isolated. As a result, the Logtail container in a pod cannot directly access the files of containers in a different pod. The file system of a container is created by mounting the file system of the container host to the container. A Logtail container can access any file on the container host only after the file system that includes the root directory of the container host is mounted to the Logtail container. This way, the Logtail container can collect logs from the files in the file system. The relationship between file paths in a container and file paths on the container host is called file path mapping.
For example, a file path in a container is /log/app.log
. After file path mapping, the file path on the container host is /var/lib/docker/containers/<container-id>/log/app.log
. By default, the file system that includes the root directory of the container host is mounted to the /logtail_host
directory of the Logtail container. Therefore, the Logtail container collects logs from /logtail_host/var/lib/docker/containers/<container-id>/log/app.log
.
Methods to manage Logtail configurations
Use the Simple Log Service console
The procedure is simple. You do not need to connect to your Kubernetes cluster or use an Alibaba Cloud account for authorization.
(Recommended) Use a CRD
Custom resource definitions (CRDs) allow you to create custom Kubernetes resources. Simple Log Service defines a CRD named AliyunLogConfig
, which is equivalent to a Logtail configuration. The CRD allows you to manage your Logtail configuration in cloud-native mode. You can submit a YAML configuration file to the kube-apiserver component to create an AliyunLogConfig
CRD. Then, you can create a Logtail configuration. In this case, you must connect to your Kubernetes cluster and complete Kubernetes-related authorization.
If you use a CRD to create a Logtail configuration and modify the Logtail configuration in the Simple Log Service console, the modification is not synchronized to the CRD. If you want to modify a Logtail configuration that is created by using a CRD, you must modify the CRD. If you modify the configuration in the Simple Log Service console, Logtail configuration inconsistency may occur.
Limits
Container runtime: Logtail can collect data only from the containers that use the Docker engine or containerd engine. For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.
Volume mounting: If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot install Logtail in DaemonSet mode. In this case, we recommend that you install Logtail in Sidecar or Deployment mode. Then, Logtail can collect logs. For more information, see Collect text logs from Kubernetes containers in Sidecar mode and Share a PVC between an application container and a Logtail container to collect logs.
Log file path:
The symbolic links of file paths in containers are not supported. You must specify an actual path as the collection directory.
If a volume is mounted to the data directory of an application container, you must specify the data directory or its subdirectory as the collection directory. For example, if a volume is mounted to the
/var/log/service
directory and you set the collection directory to/var/log
, Logtail cannot collect logs from the /var/log directory. You must specify/var/log/service
or its subdirectory as the collection directory.
Log collection stopping:
Docker: When a container is stopped, Logtail immediately releases the current container file handles. Then, the container can exit as expected. If collection latency exists due to issues such as network latency or resource overconsumption before the container is stopped, some logs that are collected before the container is stopped may be lost.
containerd: When a container is stopped, Logtail does not release the current container file handles until all logs are sent. In this period, the files keep open. If collection latency exists due to issues such as network latency or resource overconsumption, the container may fail to be destroyed in a timely manner.
Prerequisites
The Logtail components are installed. For more information, see Install Logtail components in an ACK cluster.
A Logstore is created in the project that you use to install the Logtail components. For more information, see Create a Logstore.
Ports 80 and 443 are enabled for the server on which Logtail is installed. If the server is an Elastic Computing Service (ECS) instance, you can reconfigure the related security group rules to enable the ports. For more information about how to configure a security group rule, see Add a security group rule.
Logs are continuously generated in the container from which you want to collect logs. Logtail collects only incremental logs. If a log file on your server is not updated after a Logtail configuration is delivered and applied to the server, Logtail does not collect logs from the file. For more information, see Read log files.
The required UNIX domain socket exists, and Logtail has access permissions on the UNIX domain socket. The UNIX domain socket varies based on the container engine.
Docker:
/run/docker.sock
containerd:
/run/containerd/containerd.sock
Create a Logtail configuration
If you use a CRD to create a Logtail configuration and modify the Logtail configuration in the Simple Log Service console, the modification is not synchronized to the CRD. If you want to modify a Logtail configuration that is created by using a CRD, you must modify the CRD. If you modify the configuration in the Simple Log Service console, Logtail configuration inconsistency may occur.
Console
Log on to the Simple Log Service console.
On the right side of the page that appears, click Quick Data Import. In the Import Data dialog box, click the Kubernetes - File card.
Select the required project and Logstore. Then, click Next. In this example, select the project that you use to install the Logtail components and the Logstore that you create.
In the Machine Group Configurations step, perform the following operations:
Use one of the following settings based on your business requirements:
- Important
Subsequent settings vary based on the preceding settings.
Confirm that the required machine groups are added to the Applied Server Groups section. Then, click Next. After you install Logtail components in a Container Service for Kubernetes (ACK) cluster, Simple Log Service automatically creates a machine group named
k8s-group-${your_k8s_cluster_id}
. You can directly use this machine group.ImportantIf you want to create a machine group, click Create Machine Group. In the panel that appears, configure the parameters to create a machine group. For more information, see Use the wizard in the ACK console to create an application and configure Simple Log Service.
If the heartbeat status of a machine group is FAIL, click Automatic Retry. If the issue persists, see How do I troubleshoot an error that is related to a Logtail machine group in a host environment?
Create a Logtail configuration and click Next. Simple Log Service starts to collect logs after the Logtail configuration is created.
NoteA Logtail configuration requires up to 3 minutes to take effect.
Create indexes and preview data. Then, click Next. By default, full-text indexing is enabled in Simple Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Simple Log Service automatically creates field indexes. For more information, see Create indexes.
ImportantIf you want to query all fields in logs, we recommend that you use full-text indexes. If you want to query only specific fields, we recommend that you use field indexes. This helps reduce index traffic. If you want to analyze fields, you must create field indexes. You must include a SELECT statement in your query statement for analysis.
Click Query Log. Then, you are redirected to the query and analysis page of your Logstore.
You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.
CRD
Create a Logtail configuration
To create a Logtail configuration, you need to only create an AliyunLogConfig CRD. After the Logtail configuration is created, the system automatically applies the Logtail configuration. If you want to delete the Logtail configuration, you need to only delete the CRD.
Log on to your Kubernetes cluster.
Run the following command to create a YAML file.
In the following command,
cube.yaml
is a sample file name. You can specify a different file name based on your business requirements.vim cube.yaml
Enter the following script in the YAML file and configure the parameters based on your business requirements.
ImportantThe value of the
configName
parameter must be unique in the Simple Log Service project that you use to install the Logtail components.If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify one of the CRDs. After a CRD is deleted or modified, the status of other associated CRDs becomes inconsistent with the status of the Logtail configuration in Simple Log Service.
apiVersion: log.alibabacloud.com/v1alpha1 # The default value is used. You do not need to modify this parameter. kind: AliyunLogConfig # The default value is used. You do not need to modify this parameter. metadata: name: simple-stdout-example # The name of the resource. The name must be unique in the current Kubernetes cluster. spec: project: k8s-my-project # Optional. The name of the project. The default value is the name of the project that you use to install the Logtail components. logstore: k8s-stdout # The name of the Logstore. If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore. logstoreMode: standard # Optional. The type of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. shardCount: 2 # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. lifeCycle: 90 # The data retention period of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. Valid values: 1 to 3650. Default value: 90. Unit: days. The value 3650 specifies that data is permanently stored in the Logstore. logtailConfig: # The Logtail configuration. inputType: plugin # The type of the data source. Valid values: file and plugin. The value file specifies text logs. The value plugin specifies stdout and stderr. configName: simple-stdout-example # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name. inputDetail: # The detailed settings of the Logtail configuration. For more information, see the following configuration examples. ...
Run the following command to apply the Logtail configuration. After the Logtail configuration is applied, Logtail collects stdout and stderr or text logs from each container, and then sends the collected logs to Simple Log Service.
In the following command,
cube.yaml
is a sample file name. You can specify a different file name based on your business requirements.kubectl apply -f cube.yaml
ImportantAfter logs are collected, you must create indexes. Then, you can query and analyze the logs in the Logstore. For more information, see Create indexes.
View Logtail configurations
Console
Log on to the Simple Log Service console.
In the Projects section, click the target project.
Choose . Click the > icon of the target Logtail configurations. Choose .
Click the target Logtail configurations to view the details of the configurations.
CRD
View all Logtail configurations for the current Kubernetes cluster
View the details and status of a Logtail configuration
Query and analyze the collected logs
In the Projects section of the Simple Log Service console, click the project that you want to manage to go to the details page of the project.
Find the Logstore that you want to manage, move the pointer over the Logstore, click the
icon, and then choose Search & Analysis to view the logs of your Kubernetes cluster.
CRD configuration examples
If you want to collect container text logs, set the inputType
parameter to file
and specify the required information for the inputDetail
parameter. For more information about parameter settings, see Create a Logtail configuration.
Filter containers by using Kubernetes information
Default fields in container text logs
The following table describes the fields that are included by default in each container text log.
Field name | Description |
__tag__:__hostname__ | The name of the container host. |
__tag__:__path__ | The log file path in the container. |
__tag__:_container_ip_ | The IP address of the container. |
__tag__:_image_name_ | The name of the image that is used by the container. |
__tag__:_pod_name_ | The name of the pod. |
__tag__:_namespace_ | The namespace to which the pod belongs. |
__tag__:_pod_uid_ | The unique identifier (UID) of the pod. |
Troubleshooting
If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:
What do I do if errors occur when I collect logs from containers?