Simple Log Service:Use the Simple Log Service console to collect container text logs in Sidecar mode

Background information

In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to the log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.

Step 1: Install Sidecar

1. Log on to your Kubernetes cluster.

2. Create a YAML file.

   In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.

   vim sidecar.yaml

3. Enter the following script in the YAML file and configure the parameters based on your business scenario.

   Important

   Make sure that the time zone you specify for the TZ field in the env parameter is valid. If the time zones in raw logs and processed logs in a Simple Log Service project are inconsistent, the time that is recorded for the collected logs may be a point in time in the past or in the future. For example, if the Simple Log Service project resides in greater China, you can set the time zone to Asia/Shanghai.

   apiVersion: batch/v1
   kind: Job
   metadata:
     name: nginx-log-sidecar-demo
     namespace: default
   spec:
     template:
       metadata:
         name: nginx-log-sidecar-demo
       spec:
         restartPolicy: Never
         containers:
           # The main container.
           - name: nginx-log-demo
             image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
             command: ["/bin/sh", "-c"]
             # To prevent the loss of the first several logs, the main business process does not start until Logtail starts to collect logs. Use /tasksite/cornerstone to receive notifications. 
             # After the main business process exits, Logtail is notified to exit. Use /tasksite/tombstone to send notifications. 
             args:
               - until [[ -f /tasksite/cornerstone ]]; do sleep 1; done;
                 /bin/mock_log --log-type=nginx --stdout=false --stderr=true --path=/var/log/nginx/access.log --total-count=100 --logs-per-sec=100;
                 retcode=$?;
                 touch /tasksite/tombstone;
                 exit $retcode
             resources:
               limits:
                 cpu: 500m
                 memory: 512Mi
               requests:
                 cpu: 10m
                 memory: 30Mi
             volumeMounts:
               - name: nginx-log
                 mountPath: /var/log/nginx
               - mountPath: /tasksite
                 name: tasksite
           # The Logtail sidecar container.
           - name: logtail
             # The new image version. For more information, see https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/images. 
             # Change the address based on your business requirements. 
             image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:v1.5.1.0-aliyun
             command: ["/bin/sh", "-c"]
             # The first sleep 10 indicates that the system waits for 10 seconds after the Logtail process is started. Before Logtail can collect logs, the Logtail client must pull the corresponding Logtail configuration from the server. 
             # The second sleep 10 indicates that the system waits for 10 seconds after Logtail completes log collection. This prevents the loss of the last several logs. Before Logtail completes collection, the Logtail client must send all data to the server. 
             args:
               - /etc/init.d/ilogtaild start;
                 sleep 10;
                 touch /tasksite/cornerstone;
                 until [[ -f /tasksite/tombstone ]]; do sleep 1; done;
                 sleep 10;
                 /etc/init.d/ilogtaild stop;
             livenessProbe:
               exec:
                 command:
                   - /etc/init.d/ilogtaild
                   - status
               initialDelaySeconds: 30
               periodSeconds: 30
             resources:
               limits:
                 cpu: 500m
                 memory: 512Mi
               requests:
                 cpu: 10m
                 memory: 30Mi
             env:
               ##### base config
               # user id
               - name: ALIYUN_LOGTAIL_USER_ID
                 value: "${your_aliyun_user_id}"
               # user defined id
               - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
                 value: ${your_machine_group_user_defined_id}
               # config file path in logtail's container
               - name: ALIYUN_LOGTAIL_CONFIG
                 value: /etc/ilogtail/conf/${your_region_config}/ilogtail_config.json
               ##### env tags config
               - name: "ALIYUN_LOG_ENV_TAGS"
                 value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
               - name: "_pod_name_"
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.name
               - name: "_pod_ip_"
                 valueFrom:
                   fieldRef:
                     fieldPath: status.podIP
               - name: "_namespace_"
                 valueFrom:
                   fieldRef:
                     fieldPath: metadata.namespace
               - name: "_node_name_"
                 valueFrom:
                   fieldRef:
                     fieldPath: spec.nodeName
               - name: "_node_ip_"
                 valueFrom:
                   fieldRef:
                     fieldPath: status.hostIP
             volumeMounts:
               - name: nginx-log
                 mountPath: /var/log/nginx
               - mountPath: /tasksite
                 name: tasksite
         ##### share this volume
         volumes:
           - name: nginx-log
             emptyDir: {}
           - name: tasksite
             emptyDir:
               medium: Memory

   1. Configure the basic variables in the configuration script. The following table describes the variables.

      ##### base config
                # user id
                - name: "ALIYUN_LOGTAIL_USER_ID"
                  value: "${your_aliyun_user_id}"
                # user defined id
                - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
                  value: "${your_machine_group_user_defined_id}"
                # config file path in logtail's container
                - name: "ALIYUN_LOGTAIL_CONFIG"
                  value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"

      Variable	Description
      ${your_aliyun_user_id}	The ID of your Alibaba Cloud account. For more information, see Step 1: Obtain the ID of the Alibaba Cloud account for which Simple Log Service is activated.
      ${your_machine_group_user_defined_id}	The custom identifier of your machine group. Example: nginx-log-sidecar. Important The identifier must be unique in the region where your project resides.
      ${your_region_config}	The region ID and network type of the project. For more information about regions, see Install Logtail on a Linux server. If logs are collected to your project over the Internet, specify the value in the region-internet format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou-internet. If logs are collected to your project over an internal network of Alibaba Cloud, specify the value in the region format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou.

   2. Specify the mount path in the configuration script.

      Note

      We recommend that you mount containers on a volume of the emptyDir type.

      volumeMounts:
      - name: nginx-log
        mountPath: /var/log/nginx
        ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}

      Parameter	Description
      name	The name of the volume. You can specify a name based on your business requirements. Important The value of the name parameter in the volumeMounts node and the value of the name parameter in the volumes node must be the same. This ensures that the Logtail container and the application container are mounted on the same volume.
      mountPath	The mount path. You can enter the path of files in which container text logs are recorded.

   3. Specify a waiting period for the Logtail container in the configuration script.

      In most cases, the waiting period is 10 seconds. This value specifies that the Logtail container exits 10 seconds after the container receives a stop command. This setting helps prevent incomplete data collection.

      command:        
      - sh        
      - -c        
      - /usr/local/ilogtail/run_logtail.sh 10

4. Run the following command to apply the configurations in the sidecar.yaml file.

   In this command, the file name is sidecar.yaml. Replace the file name with an actual file name.

   kubectl apply -f sidecar.yaml

Step 2: Create a machine group

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you used to install Logtail components.

3. In the left-side navigation pane, choose Resources > Machine Groups.

4. In the Machine Groups list, choose > Create Machine Group.

5. In the Create Machine Group panel, configure the parameters and click OK. The following table describes the parameters.

   Parameter	Description
   Name	The name of the machine group. After you create a machine group, you cannot change the name of the machine group.
   Identifier	Select Custom ID.
   Topic	The topic of the machine group. The topic is used to identify logs that are generated by different servers. For more information, see Log topics.
   Custom Identifier	The custom identifier of the machine group. The identifier must be the same as the value of the ${your_machine_group_user_defined_id} variable that you configured in Step 1: Install Sidecar.

Step 3: Create a Logtail configuration

1. Log on to the Simple Log Service console.

2. In the Import Data section, find and click Kubernetes - Object.

3. Select a project and a Logstore. Then, click Next.

   Select the project that you used to install Logtail components and the Logstore that you created.

4. On the Kubernetes Clusters > Sidecar tab, click Use Existing Machine Groups.

5. Select a machine group from Source Server Groups and move the machine group to Applied Server Groups. Then, click Next.

   Select the machine group that you created in Step 2: Create a machine group.

   Important

   If you enable a machine group immediately after you create the machine group, the heartbeat status of the machine group may be FAIL. This issue occurs because the machine group is not connected to Simple Log Service. To resolve this issue, you can click Automatic Retry. If the issue persists, see What do I do if a Logtail machine group has no heartbeats?

6. Create a Logtail configuration and click Next.

   Global Configurations

   Parameter	Description
   Config Name	Enter a name for the Logtail configuration. The name must be unique in a project. After you create the Logtail configuration, you cannot change the name of the Logtail configuration.
   Log Sample	Enter a sample log that is collected from an actual scenario. You can use the sample log to configure parameters that are related to log processing with ease. Examples: [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened at TestPrintStackTrace.f(TestPrintStackTrace.java:3) at TestPrintStackTrace.g(TestPrintStackTrace.java:7) at TestPrintStackTrace.main(TestPrintStackTrace.java:16) You can enter multiple sample logs. The total length of the logs cannot exceed 1,500 characters.
   Log Topic Type	Select the topic generation mode. For more information, see Log topics. Valid values: Topic: In this mode, topics are configured at the machine group level. If you want to distinguish the logs that are generated by different servers, select this mode. File Path Extraction: In this mode, you must specify a custom regular expression. The part of a log path that matches the regular expression is used as the topic. If you want to distinguish the logs that are generated by different users or instances, select this mode. Custom: In this mode, you must specify a custom log topic.

   Input Configurations

   Parameter	Description
   Logtail Deployment Mode	The deployment mode of Logtail. Select Sidecar.
   Log Path	The directory and name of log files. Configure this parameter based on the location of the logs that you want to collect. If the container runs on a Linux server, the log path must start with a forward slash (/). Example: /apsara/nuwa/**/app.Log. If the container runs on a Windows server, the log path must start with a drive letter. Example: C:\Program Files\Intel\**\*.Log. You can specify an exact directory and an exact name. You can also use wildcard characters to specify the directory and name. For more information, see Wildcard matching. When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcard characters. Simple Log Service scans all levels of the specified directory for the log files that match specified conditions. Examples: If you specify /apsara/nuwa/**/*.log, Simple Log Service collects logs from the log files whose names are suffixed by .log in the /apsara/nuwa directory and the recursive subdirectories of the directory. If you specify /var/logs/app_*/**/*.log, Simple Log Service collects logs from the log files that meet the following conditions: The file name is suffixed by .log. The file is stored in a subdirectory under the /var/logs directory or in a recursive subdirectory of the subdirectory. The name of the subdirectory matches the app_* pattern. If you specify /var/log/nginx/**/access*, Simple Log Service collects logs from the log files whose names start with access in the /var/log/nginx directory and the recursive subdirectories of the directory.
   Maximum Directory Monitoring Depth	Specify the maximum number of levels of subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of subdirectories that can be matched for the wildcard characters ** included in the value of File Path. A value of 0 specifies that only the log file directory that you specify is monitored.
   Multi-line Mode	If you want to collect multi-line logs, turn on Multi-line Mode.
   Regex to Match First Line	Configure a regular expression to match the beginning of the first line of a log. Logtail uses the regular expression to match the beginning of the first line of a log and considers the content that does not match the regular expression as part of the log. Simple Log Service can automatically generate a regular expression or use the regular expression that you manually specify. Automatic generation Click Auto Generate. Simple Log Service automatically generates a regular expression based on your sample log to match the beginning of the first line of a log. Manual configuration Click Manual and manually specify a regular expression to match the beginning of the first line of a log. Then, click Validate to check whether the regular expression is valid. For more information, see How do I test a regular expression?
   File Encoding	The format in which the log files are encoded.
   First Collection Size	Specify the size of data that Logtail can collect from a log file the first time Logtail collects logs from the file. The default value of First Collection Size is 1024. Unit: KB. If the file size is less than 1,024 KB, Logtail collects data from the beginning of the file. If the file size is greater than 1,024 KB, Logtail collects the last 1,024 KB of data in the file. You can specify First Collection Size based on your business requirements. Valid values: 0 to 10485760. Unit: KB.
   Collection Blacklist	If you turn on Collection Blacklist, you must configure a blacklist to specify the directories or files that you want Simple Log Service to skip when it collects logs. You can specify exact directories and file names. You can also use wildcard characters to specify directories and file names. When you configure this parameter, you can use only asterisks (*) or question marks (?) as wildcard characters. Important If you use wildcard characters to configure File Path and you want to skip some directories in the specified directory, you must configure Collection Blacklist and enter a complete directory. For example, if you set File Path to /home/admin/app*/log/*.log and you want to skip all subdirectories in the /home/admin/app1* directory, you must select Directory Blacklist and enter /home/admin/app1*/** in the Directory Name field. If you enter /home/admin/app1*, the blacklist does not take effect. When a blacklist is in use, computational overhead is generated. We recommend that you add up to 10 entries to the blacklist. You cannot specify a directory path that ends with a forward slash (/). For example, if you set the path to /home/admin/dir1/, the directory blacklist does not take effect. The following types of blacklists are supported: File Path Blacklist, File Blacklist, and Directory Blacklist. File Path Blacklist If you select File Path Blacklist and enter /home/admin/private*.log in the File Path Name field, all files whose names are prefixed by private and suffixed by .log in the /home/admin/ directory are skipped. If you select File Path Blacklist and enter /home/admin/private*/*_inner.log in the File Path Name field, all files whose names are suffixed by _inner.log in the subdirectories whose names are prefixed by private in the /home/admin/ directory are skipped. For example, the /home/admin/private/app_inner.log file is skipped, but the /home/admin/private/app.log file is not skipped. File Blacklist If you select File Blacklist and enter app_inner.log in the File Name field, all files whose names are app_inner.log are skipped. Directory Blacklist If you select Directory Blacklist and enter /home/admin/dir1 in the Directory Name field, all files in the /home/admin/dir1 directory are skipped. If you select Directory Blacklist and enter /home/admin/dir* in the Directory Name field, the files in all subdirectories whose names are prefixed by dir in the /home/admin/ directory are skipped. If you select Directory Blacklist and enter /home/admin/*/dir in the Directory Name field, all files in the dir subdirectory in each second-level subdirectory of the /home/admin/ directory are skipped. For example, the files in the /home/admin/a/dir directory are skipped, but the files in the /home/admin/a/b/dir directory are not skipped.
   Allow File to Be Collected for Multiple Times	By default, you can use only one Logtail configuration to collect logs from a log file. To use multiple Logtail configurations to collect logs from a log file, turn on Allow File to Be Collected for Multiple Times.

   Processor Configurations

   Each Logtail plug-in defines a processing method. After you add Logtail plug-ins, Logtail calls the processing methods in sequence and parses logs into structured data. You can use the native Logtail plug-ins to perform the following operations. For more information, see Limits.

   • Parsing in regex mode

   • Parsing in JSON mode

   • Parsing in delimiter mode

   • Parsing in NGINX mode

   • Parsing in Apache mode

   • Parsing in IIS mode

   • Time parsing

   • Data filtering

   • Data masking

   If the structure of your business logs is complex or not fixed and native plug-ins cannot meet your log parsing requirements, you can use extended Logtail plug-ins. For more information, see Overview.

   Click Next. Then, the Logtail configuration is created, and Simple Log Service starts to collect logs.

   Note

   A Logtail configuration requires up to 3 minutes to take effect.

7. Preview data, configure indexes, and then click Next.

   By default, full-text indexing is enabled in Simple Log Service. You can configure field indexes in manual or automatic mode based on collected logs . To configure field indexes in automatic mode, click Automatic Index Generation. Simple Log Service automatically creates field indexes. For more information, see Create indexes.

   Important

   If you want to query and analyze logs, you must enable full-text indexing or field indexing. If you enable full-text indexing and field indexing, the system uses only field indexes.

8. Click Log Query. Then, you are redirected to the query and analysis page of your Logstore.

   You must wait approximately 1 minute for the indexes to take effect. Then, you can view the collected logs on the Raw Logs tab. For more information, see Query and analyze logs.

Troubleshooting

What do I do if an error occurs when I use Logtail to collect logs from containers?