Use CRDs to collect Kubernetes container logs in Sidecar mode - Data Collection| Alibaba Cloud Documentation Center

Background information

In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. The Logtail container monitors the changes of log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.

Step 1: Install Sidecar

The following sample code provides an example on how to install Sidecar.

Notice Make sure that the time zone below env in the configuration file is correctly set. If the time zones are inconsistent between raw logs and processed logs in a Log Service project, the time recorded for the collected logs may be a point in time in the past or in the future. For example, if the Log Service project resides in mainland China, you can set the time zone to Asia/Shanghai.

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

Log on to your Kubernetes cluster.

Configure the following basic parameters.

##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"


Parameter	Description
`${your_region_config}`	Specify this parameter based on the ID of the region where your Log Service project resides and the type of the network for your project. For more information about regions, see Table 1. If your project is accessible over the Internet, set the value in the `region-internet` format. For example, if your project resides in the China (Hangzhou) region, set the value to cn-hangzhou-Internet. If your project is accessible over an internal network of Alibaba Cloud, set the value in the `region` format. For example, if your project resides in the China (Hangzhou) region, set the value to cn-hangzhou.
`${your_aliyun_user_id}`	Enter the ID of your Alibaba Cloud account. For more information, see Obtain the account ID.
`${your_machine_group_user_defined_id}`	Enter the custom identifier of your machine group. The identifier must be unique in the region where your project resides. For more information, see Create a custom ID-based machine group.

Configure the mount path of logs.
```
volumeMounts:
- name: nginx-log
mountPath: /var/log/nginx
```
- The Logtail container and the application container must share the same directory.
- We recommend that you mount a volume of the emptyDir type in the directory.
Configure a waiting period for the Logtail container.
In most cases, the waiting period is 10 seconds. This value indicates that the Logtail container exits 10 seconds after it receives a stop command. This setting prevents incomplete data collection.
```
command:        
- sh        
- -c        
- /usr/local/ilogtail/run_logtail.sh 10
```

Step 2: Create a Logtail configuration

After Sidecar is installed, you can configure an AliyunLogConfig CRD to create a Logtail configuration. The following script provides an example on how to configure a CRD. If you want to delete a Logtail configuration in this situation, you need only to delete the CRD.

apiVersion: log.alibabacloud.com/v1alpha1      ## The default value. You do not need to modify this parameter. 
kind: AliyunLogConfig                          ## The default value. You do not need to modify this parameter. 
metadata:
  name: simple-stdout-example                  ## The name of the resource. The name must be unique in the cluster. 
spec:
  project: k8s-my-project                      ## The name of the project. If you do not specify this parameter, logs are collected to the project that is configured when you install Sidecar. 
  logstore: k8s-stdout                         ## The name of the Logstore. If the specified Logstore does not exist, a Logstore is automatically created. 
  machineGroups:                               ## The name of the machine group. This machine group must be the same as that specified by ${your_machine_group_user_defined_id} in Sidecar. This machine group is used to associate Sidecar and the CRD. 
  - nginx-log-sidecar
  shardCount: 2                                ## Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
  lifeCycle: 90                                ## Optional. The retention period during which log data is stored in the Logstore. Unit: days. Valid values: 1 to 7300. Default value: 90. The value 7300 indicates that log data is permanently stored in the Logstore. 
  logtailConfig:                               ## The details of the Logtail configuration. For more information, see Logtail configurations. 
    inputType: file                            ## The type of the data source. Valid values: file and plugin. The stdout and stderr logs of containers are not supported. 
    configName: simple-stdout-example          ## The name of the Logtail configuration. The name must be the same as the resource name specified by the metadata.name field. 
    inputDetail:                               ## The detailed settings of the Logtail configuration. For more information, see Configuration example for a single directory. 
    ...

After the Logtail configuration is created, the Logtail container automatically collects and uploads logs to Log Service based on the Logtail configuration. You can log on to the Log Service console to view the logs.

Notice

The value of the configName field in the CRD must be unique in the specified Log Service Logstore. If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify an associated CRD. In this case, the status of the other CRDs that are associated with the Logtail configuration becomes inconsistent with the status of the Logtail configuration on the server side.
In Sidecar mode, only text logs can be collected. You must set the dockerFile field to false.

Configuration example for a single directory

The following procedure provides an example on how to use a CRD to collect logs in Sidecar mode from a self-managed Kubernetes cluster in a data center. The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet. The volume that needs to be mounted is nginx-log and is of the emptyDir type. The volume will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers.

Install Sidecar.

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "xxxxxxxxxx"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "nginx-log-sidecar"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

Create a Logtail configuration.

Set the path of access logs to /var/log/nginx/access.log and the Logstore to nginx-access.

Note In Sidecar mode, only text logs can be collected. You must set the dockerFile field to false.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the Logtail configuration. The name must be unique in your Kubernetes cluster. 
  name: nginx-log-access-example
spec:
  # The name of the project. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. 
  logstore: nginx-access
  # The machine group to which the Logtail configuration is applied. This machine group must be the same as that specified by [ALIYUN_LOGTAIL_USER_DEFINED_ID] in Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The details of the Logtail configuration. 
  logtailConfig:
    # The type of the data source. Set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name specified by the metadata.name field. 
    configName: nginx-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path of logs. 
      logPath: /var/log/nginx
      # Specify the name of the log file from which you want to collect data. You can include wildcards in the name. Example: log_*.log. 
      filePattern: access.log
      # Set dockerFile to false in Sidecar mode. 
      dockerFile: false
      # Specify a regular expression to match the start position in the first line of a log. For single-line logs, set the value to.*. 
      logBeginRegex: '.*'
      # Specify a regular expression to match logs. You can specify this parameter based on the actual situation. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # Specify the keys that can be extracted from logs. 
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
# config for error log

Set the path of error logs to /var/log/nginx/error.log and the Logstore to nginx-error.

# config for error log
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # your config name, must be unique in you k8s cluster
  name: nginx-log-error-example
spec:
  # The name of the project.
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore.
  logstore: nginx-error
  # The machine group to which the Logtail configuration is applied. This machine group must be the same as that specified by [ALIYUN_LOGTAIL_USER_DEFINED_ID] in Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The details of the Logtail configuration. 
  logtailConfig:
    # The type of the data source. Set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name specified by the metadata.name field. 
    configName: nginx-log-error-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path of logs. 
      logPath: /var/log/nginx
      # Specify the name of the log file from which you want to collect data. You can include wildcards in the name. Example: log_*.log. 
      filePattern: error.log
      # Set dockerFile to false in Sidecar mode. 
      dockerFile: false

Configuration example for different directories

The following procedure provides an example on how to use a CRD to collect logs in Sidecar mode from a self-managed Kubernetes cluster in a data center. The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet. The volumes that need to be mounted are nginx-log and nginx-logs. Both are of the emptyDir type. nginx-log will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers. nginx-logs will be mounted to the /var/log/nginxs directory of the nginx-log-demo and Logtail containers.

Install Sidecar.

apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        lifecycle:
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
        - name: nginx-logs
          mountPath: /var/log/nginxs
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        lifecycle:
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: ""
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: ""
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
        - name: nginx-logs
          mountPath: /var/log/nginxs
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
      - name: nginx-logs
        emptyDir: {}

Create a Logtail configuration.

Set the path of access logs to /var/log/nginx/access.log and the Logstore to nginx-access.

Note In Sidecar mode, only text logs can be collected. You must set the dockerFile field to false.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the Logtail configuration. The name must be unique in your Kubernetes cluster. 
  name: nginx-log-access-example
spec:
  # The name of the project. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. 
  logstore: nginx-access
  # The machine group to which the Logtail configuration is applied. This machine group must be the same as that specified by [ALIYUN_LOGTAIL_USER_DEFINED_ID] in Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The details of the Logtail configuration. 
  logtailConfig:
    # The type of the data source. Set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name specified by the metadata.name field. 
    configName: nginx-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path of logs. 
      logPath: /var/log/nginx
      # Specify the name of the log file from which you want to collect data. You can include wildcards in the name. Example: log_*.log. 
      filePattern: access.log
      # Set dockerFile to false in Sidecar mode. 
      dockerFile: false
      # Specify a regular expression to match the start position in the first line of a log. For single-line logs, set the value to.*. 
      logBeginRegex: '.*'
      # Specify a regular expression to match logs. You can specify this parameter based on the actual situation. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # Specify the keys that can be extracted from logs. 
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
# config for error log

Set the path of access logs to /var/log/nginxs/access.log and the Logstore to nginxs-access.

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the Logtail configuration. The name must be unique in your Kubernetes cluster. 
  name: nginxs-log-access-example
spec:
  # The name of the project. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. 
  logstore: nginxs-access
  # The machine group to which the Logtail configuration is applied. This machine group must be the same as that specified by [ALIYUN_LOGTAIL_USER_DEFINED_ID] in Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The details of the Logtail configuration. 
  logtailConfig:
    # The type of the data source. Set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name specified by the metadata.name field. 
    configName: nginxs-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path of logs. 
      logPath: /var/log/nginxs
      # Specify the name of the log file from which you want to collect data. You can include wildcards in the name. Example: log_*.log. 
      filePattern: access.log
      # Set dockerFile to false in Sidecar mode. 
      dockerFile: false
      # Specify a regular expression to match the start position in the first line of a log. For single-line logs, set the value to.*. 
      logBeginRegex: '.*'
      # Specify a regular expression to match logs. You can specify this parameter based on the actual situation. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # Specify the keys that can be extracted from logs. 
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
# config for error log

Prerequisites

Background information

Step 1: Install Sidecar

Step 2: Create a Logtail configuration

Configuration example for a single directory

Configuration example for different directories