This topic describes how to collect container text logs in Sidecar mode in the Simple Log Service console. To collect container text logs, you must install Sidecar, create a machine group, and then create a Logtail configuration.
Prerequisites
A project and a Logstore are created. For more information, see Create a project and Create a Logstore.
The container from which you want to collect logs continuously generates logs.
ImportantLogtail collects only incremental logs. If a log file on a server is not updated after the applied Logtail configuration is delivered to the server, Logtail does not collect logs from the file. For more information, see Read log files.
Background information
In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to the log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.
Step 1: Install Sidecar
Log on to your Kubernetes cluster.
Create a YAML file.
In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.
vim sidecar.yaml
Enter the following script in the YAML file and configure the parameters based on your business scenario.
ImportantMake sure that the time zone you specify for the TZ field in the env parameter is valid. If the time zones in raw logs and processed logs in a Simple Log Service project are inconsistent, the time that is recorded for the collected logs may be a point in time in the past or in the future. For example, if the Simple Log Service project resides in greater China, you can set the time zone to Asia/Shanghai.
apiVersion: batch/v1 kind: Job metadata: name: nginx-log-sidecar-demo namespace: default spec: template: metadata: name: nginx-log-sidecar-demo spec: restartPolicy: Never containers: # The main container. - name: nginx-log-demo image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest command: ["/bin/sh", "-c"] # To prevent the loss of the first several logs, the main business process does not start until Logtail starts to collect logs. Use /tasksite/cornerstone to receive notifications. # After the main business process exits, Logtail is notified to exit. Use /tasksite/tombstone to send notifications. args: - until [[ -f /tasksite/cornerstone ]]; do sleep 1; done; /bin/mock_log --log-type=nginx --stdout=false --stderr=true --path=/var/log/nginx/access.log --total-count=100 --logs-per-sec=100; retcode=$?; touch /tasksite/tombstone; exit $retcode resources: limits: cpu: 500m memory: 512Mi requests: cpu: 10m memory: 30Mi volumeMounts: - name: nginx-log mountPath: /var/log/nginx - mountPath: /tasksite name: tasksite # The Logtail sidecar container. - name: logtail # The new image version. For more information, see https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/images. # Change the address based on your business requirements. image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:v1.5.1.0-aliyun command: ["/bin/sh", "-c"] # The first sleep 10 indicates that the system waits for 10 seconds after the Logtail process is started. Before Logtail can collect logs, the Logtail client must pull the corresponding Logtail configuration from the server. # The second "sleep 10" indicates that the system waits for 10 seconds after Logtail completes log collection. This prevents the loss of the last several logs. Before Logtail completes collection, the Logtail client must send all data to the server. args: - /etc/init.d/ilogtaild start; sleep 10; touch /tasksite/cornerstone; until [[ -f /tasksite/tombstone ]]; do sleep 1; done; sleep 10; /etc/init.d/ilogtaild stop; livenessProbe: exec: command: - /etc/init.d/ilogtaild - status initialDelaySeconds: 30 periodSeconds: 30 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 10m memory: 30Mi env: ##### base config # user id - name: ALIYUN_LOGTAIL_USER_ID value: "${your_aliyun_user_id}" # user defined id - name: ALIYUN_LOGTAIL_USER_DEFINED_ID value: ${your_machine_group_user_defined_id} # config file path in logtail's container - name: ALIYUN_LOGTAIL_CONFIG value: /etc/ilogtail/conf/${your_region_config}/ilogtail_config.json ##### env tags config - name: "ALIYUN_LOG_ENV_TAGS" value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_" - name: "_pod_name_" valueFrom: fieldRef: fieldPath: metadata.name - name: "_pod_ip_" valueFrom: fieldRef: fieldPath: status.podIP - name: "_namespace_" valueFrom: fieldRef: fieldPath: metadata.namespace - name: "_node_name_" valueFrom: fieldRef: fieldPath: spec.nodeName - name: "_node_ip_" valueFrom: fieldRef: fieldPath: status.hostIP volumeMounts: - name: nginx-log mountPath: /var/log/nginx - mountPath: /tasksite name: tasksite ##### share this volume volumes: - name: nginx-log emptyDir: {} - name: tasksite emptyDir: medium: Memory
Configure the basic variables in the configuration script. The following table describes the variables.
##### base config # user id - name: "ALIYUN_LOGTAIL_USER_ID" value: "${your_aliyun_user_id}" # user defined id - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID" value: "${your_machine_group_user_defined_id}" # config file path in logtail's container - name: "ALIYUN_LOGTAIL_CONFIG" value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
Variable
Description
${your_aliyun_user_id}
The ID of your Alibaba Cloud account. For more information, see Step 1: Obtain the ID of the Alibaba Cloud account for which Log Service is activated.
${your_machine_group_user_defined_id}
The custom identifier of your machine group. Example: nginx-log-sidecar.
ImportantThe identifier must be unique in the region where your project resides.
${your_region_config}
The ID of the region where your project resides and the type of the network that your project uses. For more information about regions, see Region names for Logtail installation.
If logs are collected to your project over the Internet, specify the value in the
region-internet
format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou-internet.If logs are collected to your project over an internal network of Alibaba Cloud, specify the value in the
region
format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou.
Specify the mount path in the configuration script.
NoteWe recommend that you mount containers on a volume of the emptyDir type.
volumeMounts: - name: nginx-log mountPath: /var/log/nginx ##### share this volume volumes: - name: nginx-log emptyDir: {}
Parameter
Description
name
The name of the volume. You can specify a name based on your business requirements.
ImportantThe value of the name parameter in the volumeMounts node and the value of the name parameter in the volumes node must be the same. This ensures that the Logtail container and the application container are mounted on the same volume.
mountPath
The mount path. You can enter the path of files in which container text logs are recorded.
Specify a waiting period for the Logtail container in the configuration script.
In most cases, the waiting period is 10 seconds. This value specifies that the Logtail container exits 10 seconds after the container receives a stop command. This setting helps prevent incomplete data collection.
command: - sh - -c - /usr/local/ilogtail/run_logtail.sh 10
Run the following command to apply the configurations in the sidecar.yaml file.
In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.
kubectl apply -f sidecar.yaml
Step 2: Create a machine group
Log on to the Log Service console.
In the Projects section, click the project.
In the left-side navigation pane, choose .
In the Machine Groups list, choose .
In the Create Machine Group panel, configure the parameters and click OK. The following table describes the parameters.
Parameter
Description
Name
The name of the machine group.
ImportantAfter you create a machine group, you cannot change the name of the machine group. Proceed with caution.
Identifier
The identifier of the machine group. Select Custom ID.
Topic
The topic of the machine group. The topic is used to differentiate the logs that are generated by different servers. For more information, see Log topics.
Custom Identifier
The custom identifier of the machine group. The identifier must be the same as the value of the ${your_machine_group_user_defined_id} parameter that you configured when you installed Sidecar. Example: nginx-log-sidecar. For more information, see Step 1: Install Sidecar.
Step 3: Create a Logtail configuration
Log on to the Log Service console.
In the Import Data section, click Delimiter Mode - Text Log.
In this example, the delimiter mode is used. For information about other collection modes, see Collection modes.
Select a project and a Logstore. Then, click Next.
Click Use Existing Machine Groups.
Select a machine group from the Source Server Groups section and move the machine group to the Applied Server Groups section. Then, click Next.
Select the machine group that you created in Step 2: Create a machine group.
ImportantIf you enable a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?
Create a Logtail configuration and click Next.
You can collect logs in simple mode, NGINX mode, delimiter mode, JSON mode, or full regex mode. For more information, see Collect text logs.
ImportantIn Sidecar mode, you must turn off Docker File.
- Preview data, configure indexes, and then click Next. By default, full-text indexing is enabled for Log Service. You can also configure field indexes based on collected logs in manual mode or automatic mode. To configure field indexes in automatic mode, click Automatic Index Generation. This way, Log Service automatically creates field indexes. For more information, see Create indexes.Important If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable both full-text indexing and field indexing, the system uses only field indexes.
- Click Log Query. You are redirected to the query and analysis page of your Logstore. You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.
Troubleshooting
If an exception occurs when you use Logtail to collect logs from containers, such as standard containers and Kubernetes containers, you can troubleshoot the issue based on the following topic:
What do I do if an error occurs when I use Logtail to collect logs from containers?