This topic describes how to create a Logtail configuration in the Log Service console and use the Logtail configuration to collect container text logs in DaemonSet mode.

Prerequisites

The Logtail component is installed. For more information, see Install the Logtail component.

Features

Logtail can collect container text logs, and then upload the text logs together with container metadata to Log Service. Logtail supports the following features:
  • Allows you to specify a log file path in a container. You do not need to manually map the log file path to a path on the host.
  • Uses the container label whitelist to specify containers from which text logs are collected.
  • Uses the container label blacklist to specify containers from which text logs are not collected.
  • Uses the environment variable whitelist to specify containers from which text logs are collected.
  • Uses the environment variable blacklist to specify containers from which text logs are not collected.
  • Collects multi-line logs. For example, Logtail can collect Java stack logs.
  • Automatically associates container metadata that needs to be uploaded together with the collected container text logs. The metadata includes container names, image names, pod names, namespaces, and environment variables.
  • If a container runs in a Kubernetes cluster, Logtail also supports the following features:
    • Uses Kubernetes namespaces, pod names, and container names to specify containers from which text logs are collected.
    • Uses the Kubernetes label whitelist to specify containers from which text logs are collected.
    • Uses the Kubernetes label blacklist to specify containers from which text logs are not collected.
    • Automatically associates Kubernetes labels that need to be uploaded together with the collected container text logs.

Limits

  • If Logtail detects the die event on a container that is stopped, Logtail no longer collects text logs from the container. If collection latency exists, some text logs that are collected before the container is stopped may be lost.
  • For Docker containers, only overlay and overlay2 storage drivers are supported. If other storage drivers are used, you must mount a volume to the directory of logs. Then, a temporary directory is generated.

    If an Apsara File Storage NAS (NAS) file system is mounted to the directory of logs by using a PersistentVolumeClaim (PVC), you cannot collect logs in DaemonSet mode. In this case, we recommend that you collect logs in Sidecar mode.

  • Logtail cannot access the symbolic link of a container. You must specify an actual path as the collection directory.
  • If a volume is mounted to the data directory of a container, Logtail cannot collect data from the parent directory of the data directory. You must specify the complete path of the data directory as the collection directory.

    For example, if a volume is mounted to the /var/log/service directory and you set the collection directory to /var/log, Logtail cannot collect logs from the /var/log directory. You must specify /var/log/service as the collection directory.

  • By default, Kubernetes mounts the root directory of the host to the /logtail_host directory of the Logtail container. If you want to collect text logs from the host, you must specify /logtail_host as the prefix of the log file path.

    For example, if you want to collect logs from the /home/logs/app_log/ directory of the host, you must specify /logtail_host/home/logs/app_log/ as the log file path.

  • Logtail collects data from containers that use the Docker engine or containerd engine.
    • Docker: Logtail accesses the Docker engine in the /run/docker.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.
    • containerd: Logtail accesses the containerd engine in the /run/containerd/containerd.sock directory. Make sure that the directory exists and Logtail has the permissions to access the directory.

Create a Logtail configuration

  1. Log on to the Log Service console.
  2. In the Import Data section, click Kubernetes - Object.
  3. Select a project and a Logstore. Then, click Next.
    In this example, select the project that you use to install the Logtail component and the Logstore that you create.
  4. Click Use Existing Machine Groups.
    After you install the Logtail component, Log Service automatically creates a machine group named k8s-group-${your_k8s_cluster_id}. You can select this machine group.
  5. Select the k8s-group-${your_k8s_cluster_id} machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.
    Notice If the heartbeat status of the machine group is FAIL, you can click Automatic Retry. If the issue persists, see What do I do if no heartbeat connections are detected on Logtail?
  6. Configure the parameters for the Logtail configuration and click Next.
    1. Configure the basic settings, such as the name, log path, and mode. For more information, see Collect text logs.
    2. Turn on Docker File.
    3. Optional:Specify conditions to filter containers.
      • For versions earlier than Logtail V1.0.29, containers can be filtered only by using environment variables and container labels.

        A namespace of a Kubernetes cluster and the name of a container in a Kubernetes cluster can be mapped to container labels. The value of the LabelKey parameter for a namespace is io.kubernetes.pod.namespace. The value of the LabelKey parameter for a container name is io.kubernetes.container.name. We recommend that you use the two container labels to filter containers. If the container labels do not meet your business requirements, you can use the environment variable whitelist or the environment variable blacklist to filter containers. For example, the namespace of a pod is backend-prod, and the name of a container in the pod is worker-server. If you want the logs of the worker-server container to be collected, you can specify io.kubernetes.pod.namespace : backend-prod or io.kubernetes.container.name : worker-server in the container label whitelist.

        Notice
        • Container labels are retrieved by running the docker inspect command. Container labels are different from Kubernetes labels. For more information, see Obtain container labels.
        • Environment variables are the same as the environment variables that are configured to start containers. For more information, see Obtain environment variables.
        • Do not specify duplicate values for the LabelKey parameter. If you specify duplicate values for the LabelKey parameter, only one of the values takes effect.
        Parameter Description
        Label Whitelist The container label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the container label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
        • If you leave the LabelValue parameter empty, containers whose container labels contain the keys specified by LabelKey are matched.
        • If you specify a value for the LabelValue parameter, containers whose container labels consist of the key-value pair specified by LabelKey and LabelValue are matched.

          By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.

        Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container to which the container label belongs is matched.

        Label Blacklist The container label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the container label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
        • If you leave the LabelValue parameter empty, containers whose container labels contain the keys specified by LabelKey are filtered out.
        • If you specify a value for the LabelValue parameter, containers whose container labels consist of the key-value pair specified by LabelKey and LabelValue are filtered out.

          By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the container labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to io.kubernetes.container.name and set the LabelValue parameter to ^(nginx|cube)$, a container named nginx and a container named cube are matched.

        Key-value pairs are connected by using the OR operator. If a container label consists of one of the specified key-value pairs, the container to which the container label belongs is filtered out.

        Environment Variable Whitelist The environment variable whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the environment variable whitelist, the EnvKey parameter is required, and the EnvValue parameter is optional.
        • If you leave the EnvValue parameter empty, containers whose environment variables contain the keys specified by EnvKey are matched.
        • If you specify a value for the EnvValue parameter, containers whose environment variables consist of the key-value pair specified by EnvKey and EnvValue are matched.

          By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.

        Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container to which the environment variable belongs is matched.

        Environment Variable Blacklist The environment variable blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the environment variable blacklist, the EnvKey parameter is required, and the EnvValue parameter is optional.
        • If you leave the EnvValue parameter empty, containers whose environment variables contain the keys specified by EnvKey are filtered out.
        • If you specify a value for the EnvValue parameter, containers whose environment variables consist of the key-value pair specified by EnvKey and EnvValue are filtered out.

          By default, string matching is performed for the values of the EnvValue parameter. Containers are matched only if the values of the environment variables are the same as the values of the EnvValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the EnvValue parameter, regular expression matching is performed. For example, if you set the EnvKey parameter to NGINX_SERVICE_PORT and set the EnvValue parameter to ^(80|6379)$, containers whose port number is 80 and containers whose port number is 6379 are matched.

        Key-value pairs are connected by using the OR operator. If an environment variable consists of one of the specified key-value pairs, the container to which the environment variable belongs is filtered out.

      • For Logtail V1.0.29 or later, we recommend that you use different levels of Kubernetes information, such as pod names, namespaces, container names, and labels to filter containers.

        Turn on Deployed in K8s and configure the following parameters to filter containers.

        Note If you change Kubernetes labels when Kubernetes control resources, such as Deployments, are running, the operational pod is not restarted. Therefore, the pod cannot detect the change. This may cause a matching rule to become invalid. When you specify the Kubernetes label whitelist and the Kubernetes label blacklist, we recommend that you use the Kubernetes labels of pods.
        Parameter Description
        K8s Pod Name Regular Matching The pod name. The pod name specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(nginx-log-demo.*)$, all containers in the pod whose name starts with nginx-log-demo are matched.
        K8s Namespace Regular Matching The namespace. The namespace specifies the containers from which text logs are collected. Regular expression matching is supported. For example, if you specify ^(default|nginx)$, all containers in the nginx and default namespaces are matched.
        K8s Container Name Regular Matching The container name. The container name specifies the containers from which text logs are collected. Regular expression matching is supported. Kubernetes container names are defined in spec.containers. For example, if you specify ^(container-test)$, all containers whose name is container-test are matched.
        K8s Label Whitelist The Kubernetes label whitelist. The whitelist specifies the containers from which text logs are collected. When you configure the Kubernetes label whitelist, the LabelKey parameter is required, and the LabelValue parameter is optional.
        • If you leave the LabelValue parameter empty, containers whose Kubernetes labels contain the keys specified by LabelKey are matched.
        • If you specify a value for the LabelValue parameter, containers whose Kubernetes labels consist of the key-value pair specified by LabelKey and LabelValue are matched.

          By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.

        Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container to which the Kubernetes label belongs is matched.

        K8s Label Blacklist The Kubernetes label blacklist. The blacklist specifies the containers from which text logs are not collected. When you configure the Kubernetes label blacklist, the LabelKey parameter is required, and the LabelValue parameter is optional.
        • If you leave the LabelValue parameter empty, containers whose Kubernetes labels contain the keys specified by LabelKey are filtered out.
        • If you specify a value for the LabelValue parameter, containers whose Kubernetes labels consist of the key-value pair specified by LabelKey and LabelValue are filtered out.

          By default, string matching is performed for the values of the LabelValue parameter. Containers are matched only if the values of the Kubernetes labels are the same as the values of the LabelValue parameter. If you specify a value that starts with a caret (^) and ends with a dollar sign ($) for the LabelValue parameter, regular expression matching is performed. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to ^(test1|test2)$, containers whose Kubernetes labels consist of app:test1 or app:test2 are matched.

        Key-value pairs are connected by using the OR operator. If a Kubernetes label consists of one of the specified key-value pairs, the container to which the Kubernetes label belongs is filtered out.

    4. Optional:Specify log labels.

      For Logtail V1.0.29 or later, we recommend that you specify environment variables and Kubernetes labels for logs as log labels.

      Parameter Description
      Environment Variable Log Tag After you specify environment variables as log labels, Log Service adds environment variable-related fields to logs. For example, if you set the EnvKey parameter to VERSION and set the EnvValue parameter to env_version, Log Service adds the __tag__:__env_version__: v1.0.0 field to logs if the environment variable configurations of a container include VERSION=v1.0.0.
      K8s Label Log Tag After you specify Kubernetes labels as log labels, Log Service adds Kubernetes label-related fields to logs. For example, if you set the LabelKey parameter to app and set the LabelValue parameter to k8s_label_app, Log Service adds the __tag__:__k8s_label_app__: serviceA field to logs if the label configurations of a Kubernetes cluster include app=serviceA.
  7. Preview data, configure indexes, and then click Next.
    By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual or automatic mode. For more information, see Configure indexes.

Configuration examples

Example 1: Filter containers based on the environment variable whitelist and the environment variable blacklist

Collect text logs from the containers whose environment variable configurations include NGINX_SERVICE_PORT=80 but exclude POD_NAMESPACE=kube-system. The log file path is /var/log/nginx/access.log. The logs are parsed in simple mode.

  1. Obtain environment variables.

    To view the environment variables of a container, you can log on to the host on which the container resides. For more information, see Obtain environment variables.

    Configuration example of environment variables
  2. Create a Logtail configuration.
    The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode. Configuration example of a data source

Example 2: Filter containers based on the container label whitelist and the container label blacklist

Collect text logs from the containers whose container label is io.kubernetes.container.name=nginx. The log file path is /var/log/nginx/access.log. The logs are parsed in simple mode.

  1. Obtain container labels.

    To view the container labels of a container, you can log on to the host on which the container resides. For more information, see Obtain container labels.

    Configuration example of labels
  2. Create a Logtail configuration.
    The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode. Configuration example of a data source

Example 3: Filter containers by using Kubernetes namespaces, pod names, and container names

Collect text logs from the nginx-log-demo-0 container in pods whose name starts with nginx-log-demo in the default namespace.

  1. Obtain different levels of Kubernetes information.
    • Obtain information about pods.Kubernetes resources
    • Obtain information about namespaces.Kubernetes resources
  2. Create a Logtail configuration.

    The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode.

    Logtail configuration

Example 4: Filter containers by using Kubernetes labels

Collect text logs from containers whose Kubernetes labels contain the job-name key and a specific value. The value starts with nginx-log-demo.

  1. Obtain Kubernetes labels.Kubernetes resources
  2. Create a Logtail configuration.
    The following figure shows an example of a Logtail configuration. For more information about how to create a Logtail configuration that is used to collect logs in simple mode, see Collect logs in simple mode. Logtail configurations

Default fields

The following table describes the fields that are included by default in each container text log.
Log field Description
_image_name_ The name of the image.
_container_name_ The name of the container.
_pod_name_ The name of the pod.
_namespace_ The namespace of the pod.
_pod_uid_ The unique identifier of the pod.
_container_ip_ The IP address of the pod.