Use the Log Service console to collect container text logs in DaemonSet mode

Features

Limits

• If Logtail detects the die event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost.
• For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.

  If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode.

• Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
• If a volume is mounted to the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.

  For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.

• By default, Kubernetes mounts the root directory of the host to the /logtail_host directory of the Logtail container. If you want to collect text logs from the host, you must specify /logtail_host as the prefix of the log file path.

  For example, if you want to collect logs from the /home/logs/app_log/ directory of the host, you must specify /logtail_host/home/logs/app_log/ as the log file path.

• Logtail collects data from containers that use the Docker engine or containerd engine.
  • Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
  • containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

Create a Logtail configuration

1. Log on to the Log Service console.
2. In the Import Data section, click Kubernetes - Object.
3. Select a project and a Logstore. Then, click Next.
   In this example, select the project that you use to install the Logtail component and the Logstore that you create.
4. Click Use Existing Machine Groups.
   After you install the Logtail component, Log Service automatically creates a machine group named k8s-group-${your_k8s_cluster_id}. You can select this machine group.
5. Select the k8s-group-${your_k8s_cluster_id} machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
   Notice If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
6. Configure the parameters for the Logtail configuration and click Next.
   1. Configure the basic settings, such as the name, log path, and mode. For more information, see Collect text logs.
   2. Turn on Docker File.
   3. Optional:Specify conditions to filter containers.
      • For versions earlier than Logtail V1.0.29, containers can be filtered only by using environment variables and container labels.

        A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the LabelKey parameter for a namespace is io.kubernetes.pod.namespace. The value of the LabelKey parameter for a container name is io.kubernetes.container.name. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specify io.kubernetes.pod.namespace : backend-prod or io.kubernetes.container.name : worker-server in the container label whitelist.

        Notice

        • Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels.
        • Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
        • Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values for the LabelKey parameter, only one of the values takes effect.

        Parameter	Description
        Label Whitelist	The container label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If you leave the LabelValue parameter empty, containers whose container labels contain the keys specified by LabelKey are matched. If you specify a value for the LabelValue parameter, containers whose container labels consist of the key-value pair specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container to which the container label belongs is matched.
        Label Blacklist	The container label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If you leave the LabelValue parameter empty, containers whose container labels contain the keys specified by LabelKey are filtered out. If you specify a value for the LabelValue parameter, containers whose container labels consist of the key-value pair specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched. Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container to which the container label belongs is filtered out.
        Environment Variable Whitelist	The environment variable whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional. If you leave the EnvValue parameter empty, containers whose environment variables contain the keys specified by EnvKey are matched. If you specify a value for the EnvValue parameter, containers whose environment variables consist of the key-value pair specified by EnvKey and EnvValue are matched. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container to which the environment variable belongs is matched.
        Environment Variable Blacklist	The environment variable blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional. If you leave the EnvValue parameter empty, containers whose environment variables contain the keys specified by EnvKey are filtered out. If you specify a value for the EnvValue parameter, containers whose environment variables consist of the key-value pair specified by EnvKey and EnvValue are filtered out. By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched. Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container to which the environment variable belongs is filtered out.

      • For Logtail V1.0.29 or later, we recommend that you use different levels of Kubernetes information, such as pod names, namespaces, container names, and labels to filter containers.

        Turn on Deployed in K8s and configure the following parameters to filter containers.

        Note If you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods.

        Parameter	Description
        K8s Pod Name Regular Matching	The pod name. The pod name specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(nginx-log-demo.*)$, all containers in the pod whose name starts with nginx-log-demo are matched.
        K8s Namespace Regular Matching	The namespace. The namespace specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(default|nginx)$, all containers in the nginx and default namespaces are matched.
        K8s Container Name Regular Matching	The container name. The container name specifies the containers from which text logs are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify ^(container-test)$, all containers whose name is container-test are matched.
        K8s Label Whitelist	The Kubernetes label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional. If you leave the LabelValue parameter empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched. If you specify a value for the LabelValue parameter, containers whose Kubernetes labels consist of the key-value pair specified by LabelKey and LabelValue are matched. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container to which the Kubernetes label belongs is matched.
        K8s Label Blacklist	The Kubernetes label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional. If you leave the LabelValue parameter empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out. If you specify a value for the LabelValue parameter, containers whose Kubernetes labels consist of the key-value pair specified by LabelKey and LabelValue are filtered out. By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched. Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container to which the Kubernetes label belongs is filtered out.

   4. Optional:Specify log labels.

      For Logtail V1.0.29 or later, we recommend that you specify environment variables and Kubernetes labels for logs as log labels.

      Parameter	Description
      Environment Variable Log Tag	After you specify environment variables as log labels, Log Service adds environment variable-related fields to logs. For example, if you set the EnvKey parameter to VERSION and set the EnvValue parameter to env_version, Log Service adds the __tag__:__env_version__: v1.0.0 field to logs if the environment variable configurations of a container include VERSION=v1.0.0.
      K8s Label Log Tag	After you specify Kubernetes labels as log labels, Log Service adds Kubernetes label-related fields to logs. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to k8s_label_app, Log Service adds the __tag__:__k8s_label_app__: serviceA field to logs if the label configurations of a Kubernetes cluster include app=serviceA.

7. Preview data, configure indexes, and then click Next.
   By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual or automatic mode. For more information, see Configure indexes.

Configuration examples

Example 1: Filter containers based on the environment variable whitelist and the environment variable blacklist

Collect text logs from the containers whose environment variable configurations include NGINX_SERVICE_PORT=80 but exclude POD_NAMESPACE=kube-system. The log file path is /var/log/nginx/access.log. The logs are parsed in simple mode.

1. Obtain environment variables.

   To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.

   
2. Create a Logtail configuration.
   The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Example 2: Filter containers based on the container label whitelist and the container label blacklist

Collect text logs from the containers whose container label is io.kubernetes.container.name=nginx. The log file path is /var/log/nginx/access.log. The logs are parsed in simple mode.

1. Obtain container labels.

   To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.

   
2. Create a Logtail configuration.
   The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Example 3: Filter containers by using Kubernetes namespaces, pod names, and container names

Collect text logs from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.

1. Obtain different levels of Kubernetes information.
   • Obtain information about pods.
   • Obtain information about namespaces.
2. Create a Logtail configuration.

   The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

   

Example 4: Filter containers by using Kubernetes labels

Collect text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.

1. Obtain Kubernetes labels.
2. Create a Logtail configuration.
   The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

Default fields