Container Service for Kubernetes:Collect container logs (standard output and files)

1. Global and input configuration

In this step, define the name, log source, and scope for your collection configuration.

View collected logs

1. Confirm that the log file has new content: LoongCollector only collects incremental logs. Run tail -f /path/to/your/log/file and trigger business operations to ensure that new logs are being written.

2. Search for logs: Go to the Search & Analyze page of the target Logstore. Click Search & Analyze. The default time range is the last 15 minutes. Check whether new logs are ingested. Each collected container text log includes the following fields by default:

   Parameter	Description
   __tag__:__hostname__	The name of the container host.
   __tag__:__path__	The path of the log file within the container.
   __tag__:_container_ip_	The IP address of the container.
   __tag__:_image_name_	The name of the image used by the container.
   __tag__:_pod_name_	The name of the Pod.
   __tag__:_namespace_	The namespace of the Pod.
   __tag__:_pod_uid_	The unique identifier (UID) of the Pod.

Troubleshoot common issues

Manage multi-destination distribution configurations

Since multi-destination distribution configurations apply to multiple logstores, manage them from the Project-level management page:

1. Log on to the Simple Log Service console and click the name of the target Project.

2. On the Project page, in the left-side navigation pane, click Resource Group > Configurations.

   Note

   This page manages all collection configurations within the Project, including configurations for logstores that were accidentally deleted.

Send ACK logs to a cross-account Project

You can send container logs to a Simple Log Service (SLS) Project in another Alibaba Cloud account by manually installing the LoongCollector (Logtail) component in the ACK cluster and configuring it with the target account's Alibaba Cloud account ID or access credential (AccessKey).

Scenario description: If you need to collect log data into another account's SLS Project for organizational, permission, or monitoring purposes, manually install LoongCollector (Logtail) to enable cross-account collection.

Procedure: Follow these steps to manually install LoongCollector. For information about how to install Logtail, see Install and configure Logtail.

1. Connect to the Kubernetes cluster and run the appropriate command for your region:

   China regions

   wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

   Regions outside China

   wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

2. Go to the loongcollector-custom-k8s-package directory and modify the ./loongcollector/values.yaml configuration file.

   # ===================== Required Information =====================
   # The name of the Project to which logs from this cluster are sent. Example: k8s-log-custom-sd89ehdq
   projectName: ""
   # The region where the Project is located. Example: cn-shanghai
   region: ""
   # The UID of the Alibaba Cloud account that owns the Project. Enclose the UID in quotation marks (""). Example: "123456789"
   aliUid: ""
   # The network type. Valid values: Internet and Intranet. Default value: Internet.
   net: Internet
   # The AccessKey ID and AccessKey secret of the Alibaba Cloud account or RAM user. The account or RAM user must have the AliyunLogFullAccess system policy permission.
   accessKeyID: ""
   accessKeySecret: ""
   # A custom cluster ID. The ID can contain only uppercase letters, lowercase letters, digits, and hyphens (-).
   clusterID: ""

3. In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependencies:

   bash k8s-custom-install.sh install

4. After the installation is complete, verify that the components are running.

   If a pod fails to start, verify the configurations in the values.yaml file and ensure the required images have been pulled.

   # Check the pod status
   kubectl get po -n kube-system | grep loongcollector-ds

   Simple Log Service also automatically creates the following resources. You can view them in the Simple Log Service console.

   Resource type	Resource name	Description
   Project	The value of projectName that you specified in the values.yaml file	Isolates logs from different services. For flexible log management, Create a Project.
   Machine group	k8s-group-${cluster_id}	A collection of log collection nodes. Kubernetes node IP addresses change dynamically as the cluster scales in or out, so identifying a machine group by IP address can cause heartbeat failures. Therefore, machine groups in Kubernetes scenarios use a custom identifier. For more information, see Guide to associating machine groups with collection configurations.

   Important

   The LoongCollector component does not create a logstore named config-operation-log. If this logstore already exists, LoongCollector does not write logs to it.

Use multiple configurations for the same source

By default, to prevent data duplication, Simple Log Service (SLS) allows only one collection configuration to collect logs from each source:

• A text log file can match only one Logtail collection configuration.

• A container's standard output (stdout):

  • If you use the new version of the standard output template, stdout can be collected by only one standard output collection configuration by default.

  • If you use the old version of the standard output template, no extra configuration is required, and multiple collections are supported by default.

1. Log on to the Simple Log Service console and go to the target Project.

2. In the left-side navigation pane, choose Logstores and find the target logstore.

3. Click the icon next to its name to expand the logstore.

4. Click Logtail Configuration. In the list of configurations, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit and scroll to the Input Configurations section:

   • To collect text file logs, enable Allow File to Be Collected for Multiple Times.

   • To collect container standard output, enable Allow Collection by Different Logtail Configurations.

Dependency error when uninstalling loongcollector

Problem: When you try to uninstall the loongcollector (logtail-ds) component in Container Service for Kubernetes (ACK), you receive an error that the component's dependencies are not met.

Dependencies of addons are not met: terway-eniip depends on logtail-ds(>0.0) whose version is v3.x.x.x-aliyun or will be v3.x.x.x-aliyun.

Cause: The terway-eniip network plugin has log collection enabled, which creates a dependency on the loongcollector (logtail-ds) component. Therefore, ACK does not allow you to uninstall loongcollector (logtail-ds) until you remove this dependency.

Solution: Follow these steps to remove the dependency and then uninstall the component:

1. Log on to the ACK console.

2. In the cluster list, click the name of the target cluster to go to its details page.

3. In the left-side navigation pane, click Add-ons.

4. In the list of add-ons, search for the terway-eniip component and click Disable Logging.

5. In the dialog box that appears, click OK.

6. After the configuration takes effect, try to uninstall the loongcollector (logtail-ds) component again.

Delayed or truncated last log entry

Cause: Log truncation typically occurs when a log file is missing a final line feed, or when a multiline log entry, such as an exception stack, has not been fully written. Because the collector cannot determine if the log entry is complete, the last log entry may be split or delayed. How this is handled depends on your LoongCollector (Logtail) version:

• Versions earlier than 1.8:
  If the last line of a log lacks a line feed (carriage return), or if a multiline log block is incomplete, the collector waits for the next write operation to trigger an output. This can significantly delay the last log entry until a new log is written.
  
  
  
  

• Version 1.8 and later:
  These versions introduce a timeout flush mechanism to prevent logs from getting stuck. When an incomplete log line is detected, the system starts a timer. After the timeout period expires, the current content is automatically submitted to ensure the log is eventually collected.
  
  
  
  

  • Default timeout: 60 seconds (ensures data integrity in most scenarios)

  • You can adjust this value based on your requirements. However, we recommend that you do not set it to 0, as this may cause log truncation or partial data loss.

Solution:

You can extend the waiting time to ensure that a complete log entry is written before it is collected:

1. Log on to the Simple Log Service console and go to the target Project.

2. In the left-side navigation pane, choose Logstores and find the target logstore.

3. Click the icon next to its name to expand the logstore.

4. Click Logtail Configuration. In the list of configurations, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit.

   • Go to Input Configurations > Other Input Configurations > Advanced Parameters and add the following JSON configuration to customize the timeout period.

     {
       "FlushTimeoutSecs": 1
     }

     • Default value: Determined by the default_reader_flush_timeout startup parameter, which is typically a few seconds.

     • Unit: Seconds.

     • Recommended value: ≥ 1 second. We recommend that you do not set it to 0, as this may cause log truncation or partial data loss.

6. After you complete the configuration, click OK.

LoongCollector network failover and recovery

If LoongCollector (Logtail) detects an internal network connectivity issue, such as a failure or timeout, it automatically switches to the public network. This failover mechanism ensures reliable log collection and prevents backlogs or data loss.

• LoongCollector: Automatically switches back to the internal network after the network recovers.

• Logtail: Does not automatically switch back. You must restart it to restore internal network communication.

On the Logtail Configuration page, in the Processor Configurations area, you can add processing plug-ins to structure raw logs. To add a processing plug-in to an existing collection configuration, you can follow these steps:

1. In the navigation pane on the left, choose Logstores and find the destination Logstore.

2. Click to the left of its name to expand the Logstore.

3. Click Logtail Configuration. In the configuration list, find the destination Logtail configuration and click Manage Logtail Configuration in the Actions column.

4. On the Logtail configuration page, click Edit.

This section introduces only commonly used processing plug-ins that cover common log processing scenarios. For more features, see Extension processing plug-ins.

Delimiter parsing

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Standard JSON parsing

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Nested JSON parsing

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

JSON array parsing

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Apache log parsing

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Data masking

Mask sensitive data in your logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and select Native Processor > Data Masking:

• Original Field: The source field containing the content to be masked.

• Data Masking Method:

  • const: Replaces sensitive content with a specified string.

  • md5: Replaces sensitive content with its MD5 hash value.

• Replacement String: The string to use for masking. This is required when the Data Masking Method is set to const.

• Content Expression that Precedes Replaced Content: A regular expression in RE2 syntax to locate the content immediately before the sensitive data.

• Content Expression to Match Replaced Content: A regular expression in RE2 syntax to match the sensitive data that you want to replace.

1. Named group syntax differences

Go uses the (?P<name>...) syntax to define a named group and does not support the (?<name>...) syntax from PCRE.

• Correct: (?P<year>\d{4})

• Incorrect: (?<year>\d{4})

2. Unsupported regular expression features

The RE2 engine does not support the following common but complex regular expression features. Avoid using them:

• Assertions: (?=...), (?!...), (?<=...), and (?<!...)

• Conditional expressions: (?(condition)true|false)

• Recursive matching: (?R) and (?0)

• Subprogram references: (?&name) and (?P>name)

• Atomic groups: (?>...)

3. Recommendations

Use a tool such as Regex101 to debug your regular expressions. To ensure compatibility, select the Golang (RE2) mode for validation. If you use any unsupported syntax, the plugin will fail to parse or match the expression.