All Products
Search
Document Center

Simple Log Service:Use CRDs to collect container text logs in Sidecar mode

Last Updated:Dec 26, 2023

This topic describes how to collect container text logs in Sidecar mode by using a custom resource definition (CRD). To do this, you must install Sidecar and then create a Logtail configuration.

Prerequisites

  • The Logtail component is installed. For more information, see Install Logtail components in a Kubernetes cluster.

  • The container from which you want to collect logs continuously generates logs.

    Important

    Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.

Background information

In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to the log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.

Step 1: Install Sidecar

  1. Log on to your Kubernetes cluster.

  2. Create a YAML file.

    In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.

    vim sidecar.yaml
  3. Enter the following script in the YAML file and configure the parameters based on your business scenario.

    Important

    Make sure that the time zone you specify for the TZ field in the env parameter is valid. If the time zones in raw logs and processed logs in a Simple Log Service project are inconsistent, the time that is recorded for the collected logs may be a point in time in the past or in the future. For example, if the Simple Log Service project resides in greater China, you can set the time zone to Asia/Shanghai.

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: nginx-log-sidecar-demo
      namespace: default
    spec:
      template:
        metadata:
          name: nginx-log-sidecar-demo
        spec:
          restartPolicy: Never
          containers:
            # The main container.
            - name: nginx-log-demo
              image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
              command: ["/bin/sh", "-c"]
              # To prevent the loss of the first several logs, the main business process does not start until Logtail starts to collect logs. Use /tasksite/cornerstone to receive notifications. 
              # After the main business process exits, Logtail is notified to exit. Use /tasksite/tombstone to send notifications. 
              args:
                - until [[ -f /tasksite/cornerstone ]]; do sleep 1; done;
                  /bin/mock_log --log-type=nginx --stdout=false --stderr=true --path=/var/log/nginx/access.log --total-count=100 --logs-per-sec=100;
                  retcode=$?;
                  touch /tasksite/tombstone;
                  exit $retcode
            resources:
              limits:
                cpu: 500m
                memory: 512Mi
              requests:
                cpu: 10m
                memory: 30Mi
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
            # The Logtail sidecar container.
            - name: logtail
              # The new image version. For more information, see https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/images. 
              # Change the address based on your business requirements. 
              image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:v1.5.1.0-aliyun
              command: ["/bin/sh", "-c"]
              # The first sleep 10 indicates that the system waits for 10 seconds after the Logtail process is started. Before Logtail can collect logs, the Logtail client must pull the corresponding Logtail configuration from the server. 
              # The second "sleep 10" indicates that the system waits for 10 seconds after Logtail completes log collection. This prevents the loss of the last several logs. Before Logtail completes collection, the Logtail client must send all data to the server. 
              args:
                - /etc/init.d/ilogtaild start;
                  sleep 10;
                  touch /tasksite/cornerstone;
                  until [[ -f /tasksite/tombstone ]]; do sleep 1; done;
                  sleep 10;
                  /etc/init.d/ilogtaild stop;
              livenessProbe:
                exec:
                  command:
                    - /etc/init.d/ilogtaild
                    - status
                initialDelaySeconds: 30
                periodSeconds: 30
              resources:
                limits:
                  cpu: 500m
                  memory: 512Mi
                requests:
                  cpu: 10m
                  memory: 30Mi
              env:
                ##### base config
                # user id
                - name: ALIYUN_LOGTAIL_USER_ID
                  value: "${your_aliyun_user_id}"
                # user defined id
                - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
                  value: ${your_machine_group_user_defined_id}
                # config file path in logtail's container
                - name: ALIYUN_LOGTAIL_CONFIG
                  value: /etc/ilogtail/conf/${your_region_config}/ilogtail_config.json
                ##### env tags config
                - name: "ALIYUN_LOG_ENV_TAGS"
                  value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
                - name: "_pod_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: "_pod_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
                - name: "_namespace_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: "_node_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: "_node_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.hostIP
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
          ##### share this volume
          volumes:
            - name: nginx-log
              emptyDir: {}
            - name: tasksite
              emptyDir:
                medium: Memory
                            
    1. Configure the basic variables in the configuration script. The following table describes the variables.

      ##### base config
                # user id
                - name: "ALIYUN_LOGTAIL_USER_ID"
                  value: "${your_aliyun_user_id}"
                # user defined id
                - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
                  value: "${your_machine_group_user_defined_id}"
                # config file path in logtail's container
                - name: "ALIYUN_LOGTAIL_CONFIG"
                  value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"

      Variable

      Description

      ${your_aliyun_user_id}

      The ID of your Alibaba Cloud account. For more information, see Step 1: Obtain the ID of the Alibaba Cloud account for which Log Service is activated.

      ${your_machine_group_user_defined_id}

      The custom identifier of your machine group. Example: nginx-log-sidecar.

      Important

      The identifier must be unique in the region where your project resides.

      ${your_region_config}

      The ID of the region where your project resides and the type of the network that your project uses. For more information about regions, see Install Logtail on a Linux server.

      • If logs are collected to your project over the Internet, specify the value in the region-internet format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou-internet.

      • If logs are collected to your project over an internal network of Alibaba Cloud, specify the value in the region format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou.

    2. Specify the mount path in the configuration script.

      Note

      We recommend that you mount containers on a volume of the emptyDir type.

      volumeMounts:
      - name: nginx-log
        mountPath: /var/log/nginx
        ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

      Parameter

      Description

      name

      The name of the volume. You can specify a name based on your business requirements.

      Important

      The value of the name parameter in the volumeMounts node and the value of the name parameter in the volumes node must be the same. This ensures that the Logtail container and the application container are mounted on the same volume.

      mountPath

      The mount path. You can enter the path of files in which container text logs are recorded.

    3. Specify a waiting period for the Logtail container in the configuration script.

      In most cases, the waiting period is 10 seconds. This value specifies that the Logtail container exits 10 seconds after the container receives a stop command. This setting helps prevent incomplete data collection.

      command:        
      - sh        
      - -c        
      - /usr/local/ilogtail/run_logtail.sh 10
  4. Run the following command to apply the configurations in the sidecar.yaml file.

    In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.

    kubectl apply -f sidecar.yaml

Step 2: Create a Logtail configuration

To create a Logtail configuration, you need to only create an AliyunLogConfig CRD. After the Logtail configuration is created, the system automatically applies the Logtail configuration. If you want to delete the Logtail configuration, you need to only delete the CRD.

  1. Log on to your Kubernetes cluster.

  2. Run the following command to create a YAML file.

    In this command, the file name is cube.yaml. Replace the file name with an actual file name.

    vim cube.yaml
  3. Enter the following script in the YAML file and configure the parameters based on your business scenario.

    Important
    • The value of the configName parameter must be unique in the Simple Log Service project that you use.

    • If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify one of the CRDs. After a CRD is deleted or modified, the status of the other associated CRDs becomes inconsistent with the status of the Logtail configuration in Simple Log Service.

    • In Sidecar mode, only text logs can be collected. You must set the dockerFile parameter to false.

    apiVersion: log.alibabacloud.com/v1alpha1      # The default value is used. You do not need to modify this parameter. 
    kind: AliyunLogConfig                          # The default value is used. You do not need to modify this parameter. 
    metadata:
      name: simple-stdout-example                  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
    spec:
      project: k8s-my-project                      # Optional. The name of the project. The default value is the name of the project that you use to install Logtail components. 
      logstore: k8s-stdout                         # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. 
      logstoreMode: standard                       # Optional. The type of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. 
      machineGroups:                               # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. This machine group is used to associate Sidecar with the CRD. 
      - nginx-log-sidecar
      shardCount: 2                                # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
      lifeCycle: 90                                # Optional. The data retention period of the Logstore. Valid values: 1 to 3650. Default value: 90. The value 3650 specifies that log data is permanently stored in the Logstore. 
      logtailConfig:                               # The Logtail configuration. 
        inputType: file                            # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
        configName: simple-stdout-example          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. 
        inputDetail:                               # The detailed settings of the Logtail configuration. For more information, see the following configuration examples. 
        ...

    Parameter

    Type

    Required

    Description

    project

    string

    No

    The name of the project. The default value is the name of the project that you use to install Logtail components.

    logstore

    string

    Yes

    The name of the Logstore.

    If the specified Logstore does not exist, Simple Log Service automatically creates a Logstore.

    logstoreMode

    string

    No

    The type of the Logstore. For more information, see Manage a Logstore. Valid values:

    • query: Query Logstore

    • standard: Standard Logstore

    Important
    • The value of this parameter takes effect only if you configure the parameter when you create the Logstore. If you change the value of this parameter for an existing Logstore that is specified by the logstore parameter, the new value does not take effect.

    • The value of this parameter takes effect only in alibaba-log-controller V0.3.3 and later.

    shardCount

    int

    No

    The number of shards. Valid values: 1 to 10. Default value: 2.

    lifeCycle

    int

    No

    The data retention period of the Logstore. Valid values: 1 to 3650. Default value: 90. The value 3650 specifies that log data is permanently stored in the Logstore.

    machineGroups

    array

    Yes

    The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. Example: nginx-log-sidecar. For more information, see Step 1: Install Sidecar.

    Simple Log Service creates a machine group to associate Sidecar with the CRD based on the name that you specify.

    Important

    You must specify a custom identifier for the machine group in the following format:

      machineGroups:      
      - nginx-log-sidecar

    logtailConfig

    object

    Yes

    The detailed settings of the Logtail configuration. In most cases, you need to configure only the inputType, configName, and inputDetail parameters. For more information about the parameters, see Logtail configurations.

    For more information about configuration examples, see Configuration examples and Single directory.

  4. Run the following command to apply the Logtail configuration.

    In this command, the file name is cube.yaml. Replace the file name with an actual file name.

    kubectl apply -f cube.yaml

    After you create the Logtail configuration, you can view the Logtail configuration in the Simple Log Service console or by using a CRD. For more information, see View Logtail configurations.

    Important

    After logs are collected, you must create indexes. Then, you can query and analyze the logs in the Logstore. For more information, see Create indexes.

Configuration examples

Single directory

This section provides an example on how to use a CRD to collect text logs from the nginx-log-demo container in Sidecar mode. The container belongs to a self-managed Kubernetes cluster in a data center. The text logs include NGINX access logs and NGINX error logs and are stored in a single directory. The following list describes the basic information:

  • The Simple Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.

  • The name of the volume to be mounted is nginx-log and the volume is of the emptyDir type. The nginx-log volume is mounted on the /var/log/nginx directory of the nginx-log-demo and Logtail containers.

  • The path to NGINX access logs is /var/log/nginx/access.log. The name of the Logstore that is used to store the NGINX access logs is nginx-access.

  • The path to NGINX error logs is /var/log/nginx/error.log. The name of the Logstore that is used to store the NGINX error logs is nginx-error.

Examples:

  • Sidecar configuration example

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: nginx-log-sidecar-demo
      namespace: default
    spec:
      template:
        metadata:
          name: nginx-log-sidecar-demo
        spec:
          restartPolicy: Never
          containers:
            # The main container.
            - name: nginx-log-demo
              image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
              command: ["/bin/sh", "-c"]
              # To prevent the loss of the first several logs, the main business process does not start until Logtail starts to collect logs. Use /tasksite/cornerstone to receive notifications. 
              # After the main business process exits, Logtail is notified to exit. Use /tasksite/tombstone to send notifications. 
              args:
                - until [[ -f /tasksite/cornerstone ]]; do sleep 1; done;
                  /bin/mock_log --log-type=nginx --stdout=false --stderr=true --path=/var/log/nginx/access.log --total-count=100 --logs-per-sec=100;
                  retcode=$?;
                  touch /tasksite/tombstone;
                  exit $retcode
            resources:
              limits:
                cpu: 500m
                memory: 512Mi
              requests:
                cpu: 10m
                memory: 30Mi
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
            # The Logtail sidecar container.
            - name: logtail
              # The new image version. For more information, see https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/images. 
              # Change the address based on your business requirements. 
              image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:v1.5.1.0-aliyun
              command: ["/bin/sh", "-c"]
              # The first sleep 10 indicates that the system waits for 10 seconds after the Logtail process is started. Before Logtail can collect logs, the Logtail client must pull the corresponding Logtail configuration from the server. 
              # The second sleep 10 indicates that the system waits for 10 seconds after Logtail completes log collection. This prevents the loss of the last several logs. Before Logtail completes collection, the Logtail client must send all data to the server. 
              args:
                - /etc/init.d/ilogtaild start;
                  sleep 10;
                  touch /tasksite/cornerstone;
                  until [[ -f /tasksite/tombstone ]]; do sleep 1; done;
                  sleep 10;
                  /etc/init.d/ilogtaild stop;
              livenessProbe:
                exec:
                  command:
                    - /etc/init.d/ilogtaild
                    - status
                initialDelaySeconds: 30
                periodSeconds: 30
              resources:
                limits:
                  cpu: 500m
                  memory: 512Mi
                requests:
                  cpu: 10m
                  memory: 30Mi
              env:
                ##### base config
                # user id
                - name: ALIYUN_LOGTAIL_USER_ID
                  value: "1023****3423"
                # user defined id
                - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
                  value: nginx-log-sidecar
                # config file path in logtail's container
                - name: ALIYUN_LOGTAIL_CONFIG
                  value: /etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json
                ##### env tags config
                - name: "ALIYUN_LOG_ENV_TAGS"
                  value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
                - name: "_pod_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: "_pod_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
                - name: "_namespace_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: "_node_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: "_node_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.hostIP
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
          ##### share this volume
          volumes:
            - name: nginx-log
              emptyDir: {}
            - name: tasksite
              emptyDir:
                medium: Memory
  • CRD configuration example

    Create two Logtail configurations to collect NGINX access logs and NGINX error logs.

    • Collect NGINX access logs

      Important

      In Sidecar mode, you must set the dockerFile parameter to false.

      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # The name of the resource. The name must be unique in your Kubernetes cluster. 
        name: nginx-log-access-example
      spec:
        # The name of the project. The default value is the name of the project that you use to install Logtail. 
        project: k8s-nginx-sidecar-demo
        # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. 
        logstore: nginx-access
        # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. 
        machineGroups:
        - nginx-log-sidecar
        # Configure the Logtail configuration. 
        logtailConfig:
          # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
          inputType: file
          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. 
          configName: nginx-log-access-example
          inputDetail:
            # The settings that allow Logtail to collect text logs in full regex mode. 
            logType: common_reg_log
            # The log file path. 
            logPath: /var/log/nginx
            # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
            filePattern: access.log
            # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
            dockerFile: false
            # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. 
            logBeginRegex: '.*'
            # The regular expression that is used to extract log content. Configure this parameter based on your business scenario. 
            regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
            # The new fields that you want to extract from logs. 
            key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
    • Collect NGINX error logs

      Important

      In Sidecar mode, you must set the dockerFile parameter to false.

      # config for error log
      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # The name of the resource. The name must be unique in the current Kubernetes cluster. 
        name: nginx-log-error-example
      spec:
        # The name of the project. The default value is the name of the project that you use to install Logtail. 
        project: k8s-nginx-sidecar-demo
        # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. 
        logstore: nginx-error
        # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. 
        machineGroups:
        - nginx-log-sidecar
        # Configure the Logtail configuration. 
        logtailConfig:
          # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
          inputType: file
          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. 
          configName: nginx-log-error-example
          inputDetail:
            # The settings that allow Logtail to collect text logs in full regex mode. 
            logType: common_reg_log
            # The log file path. 
            logPath: /var/log/nginx
            # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
            filePattern: error.log
            # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
            dockerFile: false

Multiple directories

This section provides an example on how to use a CRD to collect text logs from the nginx-log-demo container in Sidecar mode. The container belongs to a self-managed Kubernetes cluster in a data center. The text logs include NGINX access logs and are stored in different directories. The following list describes the basic information:

  • The Simple Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.

  • The names of the volumes to be mounted are nginx-log and nginx-logs and the volumes are of the emptyDir type. The nginx-log volume is mounted on the /var/log/nginx directory of the nginx-log-demo and Logtail containers. The nginx-logs volume is mounted on the /var/log/nginxs directory of the nginx-log-demo and Logtail containers.

  • One log file path is /var/log/nginx/access.log and the other log file path is /var/log/nginxs/access.log.

  • The name of the Logstore that is used to store NGINX access logs is nginx-access.

Examples:

  • Sidecar configuration example

    apiVersion: batch/v1
    kind: Job
    metadata:
      name: nginx-log-sidecar-demo
      namespace: default
    spec:
      template:
        metadata:
          name: nginx-log-sidecar-demo
        spec:
          restartPolicy: Never
          containers:
            # The main container.
            - name: nginx-log-demo
              image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
              command: ["/bin/sh", "-c"]
              # To prevent the loss of the first several logs, the main business process does not start until Logtail starts to collect logs. Use /tasksite/cornerstone to receive notifications. 
              # After the main business process exits, Logtail is notified to exit. Use /tasksite/tombstone to send notifications. 
              args:
                - until [[ -f /tasksite/cornerstone ]]; do sleep 1; done;
                  /bin/mock_log --log-type=nginx --stdout=false --stderr=true --path=/var/log/nginx/access.log --total-count=100 --logs-per-sec=100;
                  retcode=$?;
                  touch /tasksite/tombstone;
                  exit $retcode
              limits:
                cpu: 500m
                memory: 512Mi
              requests:
                cpu: 10m
                memory: 30Mi
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
            # The Logtail sidecar container.
            - name: logtail
              # The new image version. For more information, see https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/images. 
              # Change the address based on your business requirements. 
              image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:v1.5.1.0-aliyun
              command: ["/bin/sh", "-c"]
              # The first "sleep 10" indicates that the system waits for 10 seconds after the Logtail process is started. Before Logtail can collect logs, the Logtail client must pull the corresponding Logtail configuration from the server. 
              # The second sleep 10 indicates that the system waits for 10 seconds after Logtail completes log collection. This prevents the loss of the last several logs. Before Logtail completes collection, the Logtail client must send all data to the server. 
              args:
                - /etc/init.d/ilogtaild start;
                  sleep 10;
                  touch /tasksite/cornerstone;
                  until [[ -f /tasksite/tombstone ]]; do sleep 1; done;
                  sleep 10;
                  /etc/init.d/ilogtaild stop;
              livenessProbe:
                exec:
                  command:
                    - /etc/init.d/ilogtaild
                    - status
                initialDelaySeconds: 30
                periodSeconds: 30
              resources:
                limits:
                  cpu: 500m
                  memory: 512Mi
                requests:
                  cpu: 10m
                  memory: 30Mi
              env:
                ##### base config
                # user id
                - name: ALIYUN_LOGTAIL_USER_ID
                  value: "1023****3423"
                # user defined id
                - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
                  value: nginx-log-sidecar
                # config file path in logtail's container
                - name: ALIYUN_LOGTAIL_CONFIG
                  value: /etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json
                ##### env tags config
                - name: "ALIYUN_LOG_ENV_TAGS"
                  value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
                - name: "_pod_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.name
                - name: "_pod_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.podIP
                - name: "_namespace_"
                  valueFrom:
                    fieldRef:
                      fieldPath: metadata.namespace
                - name: "_node_name_"
                  valueFrom:
                    fieldRef:
                      fieldPath: spec.nodeName
                - name: "_node_ip_"
                  valueFrom:
                    fieldRef:
                      fieldPath: status.hostIP
              volumeMounts:
                - name: nginx-log
                  mountPath: /var/log/nginx
                - mountPath: /tasksite
                  name: tasksite
          ##### share this volume
          volumes:
            - name: nginx-log
              emptyDir: {}
            - name: tasksite
              emptyDir:
                medium: Memory
  • CRD configuration example

    Create two Logtail configurations to collect NGINX access logs from different directories.

    • Collect NGINX access logs from the /var/log/nginx/access.log directory.

      Important

      In Sidecar mode, you must set the dockerFile parameter to false.

      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # The name of the resource. The name must be unique in the current Kubernetes cluster. 
        name: nginx-log-access-example
      spec:
        # The name of the project. The default value is the name of the project that you use to install Logtail. 
        project: k8s-nginx-sidecar-demo
        # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. 
        logstore: nginx-access
        # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. 
        machineGroups:
        - nginx-log-sidecar
        # Configure the Logtail configuration. 
        logtailConfig:
          # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
          inputType: file
          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. 
          configName: nginx-log-access-example
          inputDetail:
            # The settings that allow Logtail to collect text logs in full regex mode. 
            logType: common_reg_log
            # The log file path. 
            logPath: /var/log/nginx
            # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
            filePattern: access.log
            # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
            dockerFile: false
            # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. 
            logBeginRegex: '.*'
            # The regular expression that is used to extract log content. 
            regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
            # The new fields that you want to extract from logs. 
            key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
    • Collect NGINX access logs from the /var/log/nginxs/access.log directory.

      Important

      In Sidecar mode, you must set the dockerFile parameter to false.

      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        # The name of the resource. The name must be unique in the current Kubernetes cluster. 
        name: nginxs-log-access-example
      spec:
        # The name of the project. The default value is the name of the project that you use to install Logtail. 
        project: k8s-nginx-sidecar-demo
        # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. 
        logstore: nginxs-access
        # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. 
        machineGroups:
        - nginx-log-sidecar
        # Configure the Logtail configuration. 
        logtailConfig:
          # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
          inputType: file
          # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. 
          configName: nginxs-log-access-example
          inputDetail:
            # The settings that allow Logtail to collect text logs in full regex mode. 
            logType: common_reg_log
            # The log file path. 
            logPath: /var/log/nginxs
            # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. 
            filePattern: access.log
            # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
            dockerFile: false
            # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. 
            logBeginRegex: '.*'
            # The regular expression that is used to extract log content. 
            regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
            # The new fields that you want to extract from logs. 
            key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]

Troubleshooting

If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:

What do I do if an error occurs when I use Logtail to collect logs from containers?