This topic describes how to collect container text logs in Sidecar mode by using a custom resource definition (CRD). To do this, you must install Sidecar and then create a Logtail configuration.
Prerequisites
- The Logtail component is installed. For more information, see Install Logtail components in a Kubernetes cluster.
- The container from which you want to collect logs continuously generates logs. Important Logtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.
Background information
In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to the log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.
Step 1: Install Sidecar
- Log on to your Kubernetes cluster.
- Create a YAML file.
In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.
vim sidecar.yaml
- Enter the following script in the YAML file and configure the parameters based on your business scenario. Important Make sure that the time zone you specify for the TZ field in the env parameter is valid. If the time zones in raw logs and processed logs in a Log Service project are inconsistent, the time that is recorded for the collected logs may be a point in time in the past or in the future. For example, if the Log Service project resides in greater China, you can set the time zone to Asia/Shanghai.
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/mock_log"] args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"] volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### logtail sidecar container - name: logtail # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail # this images is released for every region image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest # when recevie sigterm, logtail will delay 10 seconds and then stop command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10 livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 resources: limits: memory: 512Mi requests: cpu: 10m memory: 30Mi env: ##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "${your_aliyun_user_id}" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "${your_machine_group_user_defined_id}" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json" ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### share this volume volumes: - name: nginx-log emptyDir: {}
- Run the following command to apply the configurations in the sidecar.yaml file.
In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.
kubectl apply -f sidecar.yaml
Step 2: Create a Logtail configuration
To create a Logtail configuration, you need to only create an AliyunLogConfig CRD. After the Logtail configuration is created, the system automatically applies the Logtail configuration. If you want to delete the Logtail configuration, you need to only delete the CRD.
- Log on to your Kubernetes cluster.
- Run the following command to create a YAML file.
In this command, the file name is cube.yaml. Replace the file name with an actual file name.
vim cube.yaml
- Enter the following script in the YAML file and configure the parameters based on your business scenario. Important
- The value of the configName parameter must be unique in the Log Service project that you use.
- If multiple CRDs are associated with the same Logtail configuration, the Logtail configuration is affected when you delete or modify one of the CRDs. After a CRD is deleted or modified, the status of the other associated CRDs becomes inconsistent with the status of the Logtail configuration in Log Service.
- In Sidecar mode, only text logs can be collected. You must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 # The default value is used. You do not need to modify this parameter. kind: AliyunLogConfig # The default value is used. You do not need to modify this parameter. metadata: name: simple-stdout-example # The name of the resource. The name must be unique in the current Kubernetes cluster. spec: project: k8s-my-project # Optional. The name of the project. The default value is the name of the project that you use to install Logtail components. logstore: k8s-stdout # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. logstoreMode: standard # Optional. The type of the Logstore. The value of this parameter takes effect only if you configure the parameter when you create the Logstore. machineGroups: # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. This machine group is used to associate Sidecar with the CRD. - nginx-log-sidecar shardCount: 2 # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. lifeCycle: 90 # Optional. The data retention period of the Logstore. Valid values: 1 to 3650. Default value: 90. The value 3650 specifies that log data is permanently stored in the Logstore. logtailConfig: # The Logtail configuration. inputType: file # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. configName: simple-stdout-example # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. inputDetail: # The detailed settings of the Logtail configuration. For more information, see the following configuration examples. ...
Parameter Type Required Description project string No The name of the project. The default value is the name of the project that you use to install Logtail components. logstore string Yes The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore.
logstoreMode string No The type of the Logstore. For more information, see Manage a Logstore. Valid values: - query: Query Logstore
- standard: Standard Logstore
Important- The value of this parameter takes effect only if you configure the parameter when you create the Logstore. If you change the value of this parameter for an existing Logstore that is specified by the logstore parameter, the new value does not take effect.
- The value of this parameter takes effect only in alibaba-log-controller V0.3.3 and later.
shardCount int No The number of shards. Valid values: 1 to 10. Default value: 2. lifeCycle int No The data retention period of the Logstore. Valid values: 1 to 3650. Default value: 90. The value 3650 specifies that log data is permanently stored in the Logstore. machineGroups array Yes The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. Example: nginx-log-sidecar. For more information, see Step 1: Install Sidecar. Log Service creates a machine group to associate Sidecar with the CRD based on the name that you specify.ImportantYou must specify a custom identifier for the machine group in the following format:
machineGroups: - nginx-log-sidecar
logtailConfig object Yes The detailed settings of the Logtail configuration. In most cases, you need to configure only the inputType, configName, and inputDetail parameters. For more information about the parameters, see Logtail configurations. For more information about configuration examples, see Configuration examples and Single directory.
- Run the following command to apply the Logtail configuration.
In this command, the file name is cube.yaml. Replace the file name with an actual file name.
kubectl apply -f cube.yaml
After you create the Logtail configuration, you can view the Logtail configuration in the Log Service console or by using a CRD. For more information, see View Logtail configurations.Important After logs are collected, you must create indexes. Then, you can query and analyze the logs in the Logstore. For more information, see Create indexes.
Configuration examples
Single directory
- The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
- The name of the volume to be mounted is nginx-log and the volume is of the emptyDir type. The nginx-log volume is mounted on the /var/log/nginx directory of the nginx-log-demo and Logtail containers.
- The path to NGINX access logs is /var/log/nginx/access.log. The name of the Logstore that is used to store the NGINX access logs is nginx-access.
- The path to NGINX error logs is /var/log/nginx/error.log. The name of the Logstore that is used to store the NGINX error logs is nginx-error.
Examples:
- Sidecar configuration example
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/mock_log"] args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"] volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### logtail sidecar container - name: logtail # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail # this images is released for every region image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest # when recevie sigterm, logtail will delay 10 seconds and then stop command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10 livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 env: ##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "1023****3423" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "nginx-log-sidecar" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json" ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### share this volume volumes: - name: nginx-log emptyDir: {}
- CRD configuration example
Create two Logtail configurations to collect NGINX access logs and NGINX error logs.
- Collect NGINX access logsImportant In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in your Kubernetes cluster. name: nginx-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. logstore: nginx-access # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. machineGroups: - nginx-log-sidecar # Configure the Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. configName: nginx-log-access-example inputDetail: # The settings that allow Logtail to collect text logs in full regex mode. logType: common_reg_log # The log file path. logPath: /var/log/nginx # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. Configure this parameter based on your business scenario. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The new fields that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
- Collect NGINX error logsImportant In Sidecar mode, you must set the dockerFile parameter to false.
# config for error log apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in the current Kubernetes cluster. name: nginx-log-error-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. logstore: nginx-error # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. machineGroups: - nginx-log-sidecar # Configure the Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. configName: nginx-log-error-example inputDetail: # The settings that allow Logtail to collect text logs in full regex mode. logType: common_reg_log # The log file path. logPath: /var/log/nginx # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. filePattern: error.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false
- Collect NGINX access logs
Multiple directories
- The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
- The names of the volumes to be mounted are nginx-log and nginx-logs and the volumes are of the emptyDir type. The nginx-log volume is mounted on the /var/log/nginx directory of the nginx-log-demo and Logtail containers. The nginx-logs volume is mounted on the /var/log/nginxs directory of the nginx-log-demo and Logtail containers.
- One log file path is /var/log/nginx/access.log and the other log file path is /var/log/nginxs/access.log.
- The name of the Logstore that is used to store NGINX access logs is nginx-access.
Examples:
- Sidecar configuration example
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/mock_log"] args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"] lifecycle: volumeMounts: - name: nginx-log mountPath: /var/log/nginx - name: nginx-logs mountPath: /var/log/nginxs ##### logtail sidecar container - name: logtail # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail # this images is released for every region image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest # when recevie sigterm, logtail will delay 10 seconds and then stop lifecycle: command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10 livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 resources: limits: memory: 512Mi requests: cpu: 10m memory: 30Mi env: ##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "1023****3423" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "nginx-log-sidecar" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json" ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx - name: nginx-logs mountPath: /var/log/nginxs ##### share this volume volumes: - name: nginx-log emptyDir: {} - name: nginx-logs emptyDir: {}
- CRD configuration example
Create two Logtail configurations to collect NGINX access logs from different directories.
- Collect NGINX access logs from the /var/log/nginx/access.log directory.Important In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in the current Kubernetes cluster. name: nginx-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. logstore: nginx-access # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. machineGroups: - nginx-log-sidecar # Configure the Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. configName: nginx-log-access-example inputDetail: # The settings that allow Logtail to collect text logs in full regex mode. logType: common_reg_log # The log file path. logPath: /var/log/nginx # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The new fields that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
- Collect NGINX access logs from the /var/log/nginxs/access.log directory.Important In Sidecar mode, you must set the dockerFile parameter to false.
apiVersion: log.alibabacloud.com/v1alpha1 kind: AliyunLogConfig metadata: # The name of the resource. The name must be unique in the current Kubernetes cluster. name: nginxs-log-access-example spec: # The name of the project. The default value is the name of the project that you use to install Logtail. project: k8s-nginx-sidecar-demo # The name of the Logstore. If the specified Logstore does not exist, Log Service automatically creates a Logstore. logstore: nginxs-access # The name of the machine group. The name must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. machineGroups: - nginx-log-sidecar # Configure the Logtail configuration. logtailConfig: # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. inputType: file # The name of the Logtail configuration. The name must be the same as the resource name that is specified in the metadata.name field. configName: nginxs-log-access-example inputDetail: # The settings that allow Logtail to collect text logs in full regex mode. logType: common_reg_log # The log file path. logPath: /var/log/nginxs # The log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log. filePattern: access.log # Set the dockerFile parameter to false. This setting is required in Sidecar mode. dockerFile: false # The regular expression that is used to match the beginning of the first line of a log. If you want to collect single-line logs, set the value to '.*'. logBeginRegex: '.*' # The regular expression that is used to extract log content. regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*' # The new fields that you want to extract from logs. key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
- Collect NGINX access logs from the /var/log/nginx/access.log directory.
Troubleshooting
If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:
What do I do if an error occurs when I use Logtail to collect logs from containers?