This topic describes how to install Sidecar. This topic also describes how to use a custom resource definition (CRD) to create a Logtail configuration that is used to collect container text logs in Sidecar mode.
Prerequisites
Background information
In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.
Step 1: Install Sidecar
Step 2: Create a Logtail configuration
To create a Logtail configuration, you need only to configure an AliyunLogConfig CRD. After you create a Logtail configuration, Logtail automatically collects logs based on the Logtail configuration. If you want to delete the Logtail configuration, you need only to delete the CRD.
Configuration example for a single directory
- The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
- The volume that needs to be mounted is nginx-log and is of the emptyDir type. The nginx-log volume will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers.
- The path to NGINX access logs is /var/log/nginx/access.log. The Logstore that is used to store the NGINX access logs is nginx-access.
- The path to NGINX error logs is /var/log/nginx/error.log. The Logstore that is used to store the NGINX error logs is nginx-error.
- Sidecar configuration example
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/mock_log"] args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"] volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### logtail sidecar container - name: logtail # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail # this images is released for every region image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest # when recevie sigterm, logtail will delay 10 seconds and then stop command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10 livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 env: ##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "1023****3423" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "nginx-log-sidecar" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json" ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### share this volume volumes: - name: nginx-log emptyDir: {}
- CRD configuration example
Create a Logtail configuration to collect NGINX access logs and another Logtail configuration to collect NGINX error logs.
- Collect NGINX access logs
Notice In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in your Kubernetes cluster. name: nginx-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail components. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. logstore: nginx-access # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. machineGroups: - nginx-log-sidecar # The Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. configName: nginx-log-access-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the path to the log file. logPath: /var/log/nginx # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. You can configure this parameter based on your business requirements. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The keys that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol",ayloa "latency", "pd", "status", "response-size",user-agent"]
- Collect NGINX error logs
Notice In Sidecar mode, you must set the dockerFile parameter to false.
# config for error log apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in your Kubernetes cluster. name: nginx-log-error-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail components. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. logstore: nginx-error # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. machineGroups: - nginx-log-sidecar # The Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. configName: nginx-log-error-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the path to the log file. logPath: /var/log/nginx # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. filePattern: error.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false
- Collect NGINX access logs
Configuration example for different directories
The following procedure shows how to use a CRD to collect text logs from the nginx-log-demo container in Sidecar mode. The container belongs to a self-managed Kubernetes cluster in a data center. The text logs include NGINX access logs and are stored in different directories. The following list describes basic information:
- The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
- The volumes that need to be mounted are nginx-log and nginx-logs and are of the emptyDir type. The nginx-log volume will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers. The nginx-logs volume will be mounted to the /var/log/nginxs directory of the nginx-log-demo and Logtail containers.
- One log file path is /var/log/nginx/access.log and the other log file path is /var/log/nginxs/access.log.
- The Logstore that is used to store NGINX access logs is nginx-access.
- Sidecar configuration example
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/mock_log"] args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"] lifecycle: volumeMounts: - name: nginx-log mountPath: /var/log/nginx - name: nginx-logs mountPath: /var/log/nginxs ##### logtail sidecar container - name: logtail # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail # this images is released for every region image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest # when recevie sigterm, logtail will delay 10 seconds and then stop lifecycle: command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10 livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 resources: limits: memory: 512Mi requests: cpu: 10m memory: 30Mi env: ##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "1023****3423" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "nginx-log-sidecar" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json" ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx - name: nginx-logs mountPath: /var/log/nginxs ##### share this volume volumes: - name: nginx-log emptyDir: {} - name: nginx-logs emptyDir: {}
- CRD configuration example
Create two Logtail configurations to collect NGINX access logs from different directories.
- Collect NGINX access logs from /var/log/nginx/access.logNotice In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in your Kubernetes cluster. name: nginx-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail components. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. logstore: nginx-access # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. machineGroups: - nginx-log-sidecar # The Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. configName: nginx-log-access-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the path to the log file. logPath: /var/log/nginx # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The keys that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
- Collect NGINX access logs from /var/log/nginxs/access.logNotice In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in your Kubernetes cluster. name: nginxs-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail components. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. logstore: nginxs-access # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. machineGroups: - nginx-log-sidecar # The Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. configName: nginxs-log-access-example inputDetail: # Set logType to common_reg_log. logType: common_reg_log # Specify the path to the log file. logPath: /var/log/nginxs # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The keys that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"] # config for error log
- Collect NGINX access logs from /var/log/nginx/access.log