Container Service for Kubernetes:Collect container logs (stdout/stderr or files) from Kubernetes clusters using the console

1. Global and input configuration

Before configuring, ensure you have selected a data ingestion template and bound a machine group. This step defines the collection configuration name, log sources, and collection scope.

View reported logs

1. Confirm that new content is added to the log file: LoongCollector collects only incremental logs. Run the tail -f /path/to/your/log/file command and trigger a business operation to ensure that new logs are being written.

2. Query logs: Go to the query and analysis page of the destination LogStore. Click Search & Analyze. The default time range is the last 15 minutes. Check whether new logs are ingested. By default, each collected container text log contains the following fields:

   Field name	Description
   __tag__:__hostname__	The name of the container's host.
   __tag__:__path__	The path of the log file in the container.
   __tag__:_container_ip_	The IP address of the container.
   __tag__:_image_name_	The name of the image that the container uses.
   __tag__:_pod_name_	The name of the pod.
   __tag__:_namespace_	The namespace to which the pod belongs.
   __tag__:_pod_uid_	The unique identifier (UID) of the pod.

Troubleshoot common issues

Manage Multi-Target Distribution Configurations

Because multi-target distribution configurations are associated with multiple Logstores, manage these configurations on the Project-level management page:

1. Log on to the Simple Log Service console. Then, click the name of the target Project.

2. On the target Project page, click Resource Group > Configurations in the navigation pane on the left.

   Note

   This page centrally manages all collection configurations under the Project, including those that remain because Logstores were accidentally deleted.

Transfer ACK Cluster Logs to a Project in Another Alibaba Cloud Account

Manually install the Simple Log Service LoongCollector (Logtail) component in the ACK cluster. Then, configure the root account ID or access credential (AccessKey) of the destination account. This sends container logs to a Simple Log Service Project in another Alibaba Cloud account.

Scenario: When you need to collect log data from an ACK cluster into an independent Simple Log Service Project in another Alibaba Cloud account due to organizational structure, permission isolation, or unified monitoring requirements, manually install LoongCollector (Logtail) for cross-account configuration.

Procedure: This procedure uses manual installation of LoongCollector as an example. For information about how to install Logtail, see Logtail Installation and Configuration.

1. Connect to your Kubernetes cluster and run the corresponding command based on the region:

   Regions in China

   wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

   Regions outside China

   wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

2. Go to the loongcollector-custom-k8s-package directory and modify the configuration file ./loongcollector/values.yaml.

   # ===================== Required fields =====================
   # Project name for this cluster, for example, k8s-log-custom-sd89ehdq
   projectName: ""
   # Region of the project, for example, cn-shanghai for Shanghai
   region: ""
   # Alibaba Cloud account ID of the project owner. Enclose it in quotes, for example, "123456789"
   aliUid: ""
   # Network type. Valid values: Internet (public network) and Intranet (private network). Default value: Internet.
   net: Internet
   # AccessKey ID and AccessKey secret of the Alibaba Cloud account or RAM user. The account must have the AliyunLogFullAccess system policy.
   accessKeyID: ""
   accessKeySecret: ""
   # Custom cluster ID. Only letters, digits, and hyphens (-) are allowed.
   clusterID: ""

3. In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependencies:

   bash k8s-custom-install.sh install

4. After installation, check the component status.

   If the pod fails to start, verify that the values.yaml configuration is correct and that the related images are pulled successfully.

   # Check pod status
   kubectl get po -n kube-system | grep loongcollector-ds

   Simple Log Service also automatically creates the following resources. You can view them in the Simple Log Service console.

   Resource type	Resource name	Purpose
   Project	Value of projectName in the values.yaml file	Resource management unit that isolates logs from different businesses. To create your own project for more flexible log resource management, see Create a project.
   Machine group	k8s-group-${cluster_id}	Collection of log collection nodes.
   Logstore	config-operation-log Important Do not delete this Logstore.	Stores logs from the loongcollector-operator component. It uses the same billing method as a standard Logstore. For details, see Billing items for the pay-by-ingested-data metering method. We recommend that you do not create collection configurations in this Logstore.

Allow Multiple Collection Configurations to Collect the Same Log File or Container Standard Output

By default, to prevent data duplication, Simple Log Service restricts each log source to be collected by only one collection configuration:

• A text log file can match only one Logtail collection configuration.

• A container's standard output (stdout):

  • If you use the new standard output template, only one standard output collection configuration can collect it by default.

  • If you use the old standard output template, no additional configuration is required. It supports collecting multiple copies by default.

1. Log on to the Simple Log Service console. Go to the target Project.

2. In the navigation pane on the left, choose Logstores. Find the target Logstore.

3. Click before its name to expand the Logstore.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration. Then, click Manage Logtail Configuration in the Actions column.

5. On the Logtail Configuration page, click Edit. Scroll down to the Input Configurations section:

   • To collect text file logs, enable Allow File to Be Collected for Multiple Times.

   • To collect container standard output, enable Allow Collection by Different Logtail Configurations.

Dependency Error When Uninstalling the loongcollector (logtail-ds) Component in ACK

Problem Description: When you try to delete the loongcollector (logtail-ds) log collection component in Container Service for Kubernetes (ACK), the system reports an error: The dependencies of this component are not met.

Dependencies of addons are not met: terway-eniip depends on logtail-ds(>0.0) whose version is v3.x.x.x-aliyun or will be v3.x.x.x-aliyun.

Cause: The terway-eniip network plug-in enables the log collection feature. It depends on the loongcollector (logtail-ds) component. Therefore, ACK does not allow you to directly uninstall loongcollector (logtail-ds) before you remove this dependency.

Solution: Follow these steps to remove the dependency and then uninstall the component:

1. Log on to the Container Service for Kubernetes console.

2. In the cluster list, click the name of the target cluster. Go to the cluster details page.

3. In the navigation pane on the left, click Add-ons.

4. In the component list, search for and find the terway-eniip component. Click Disable Logging.

5. In the dialog box that appears, click OK.

6. After the configuration takes effect, try to uninstall the loongcollector (logtail-ds) component again.

Why is the Last Log Segment Reported with a Long Delay and Sometimes Truncated?

Cause Analysis: Log truncation usually occurs when a log file lacks a line feed at the end or when multi-line logs (such as exception stacks) are not fully written. Because the data collector cannot determine if the log has ended, the last segment of content may be chunked prematurely or reported with a delay. Different versions of LoongCollector (Logtail) have different processing mechanisms:

• Versions earlier than 1.8:
  If the last log line does not have a line feed (carriage return) or a multi-line log segment is not complete, the data collector waits for the next write to trigger output. This may cause the last log entry to remain unsent for a long time until new logs are written.
  
  
  
  
  
  
  
  

• Versions 1.8 and later:
  A timeout refresh mechanism is introduced to prevent logs from getting stuck. When an incomplete log line is detected, the system starts a timer. After the timeout, it automatically submits the current content to ensure that logs are eventually collected.
  
  
  
  
  
  
  
  

  • Default timeout: 60 seconds (ensures completeness in most scenarios)

  • Adjust this value as needed. However, do not set it to 0. Otherwise, logs may be truncated or some content may be lost.

Solution:

You can extend the wait time to ensure that a complete log is written before it is collected:

1. Log on to the Simple Log Service console. Go to the target Project.

2. In the navigation pane on the left, choose Logstores. Find the target Logstore.

3. Click before its name to expand the Logstore.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration. Then, click Manage Logtail Configuration in the Actions column.

5. On the Logtail Configuration page, click Edit:

   • Choose Input Configurations > Other Input Configurations > Advanced Parameters. Add the following JSON configuration to customize the timeout:

     {
       "FlushTimeoutSecs": 1
     }

     • Default value: Determined by the startup parameter default_reader_flush_timeout (usually a few seconds).

     • Unit: seconds.

     • Recommended value: ≥1 second. Do not set it to 0. Otherwise, logs may be truncated or some content may be lost.

6. After the configuration is complete, click OK.

Why Does LoongCollector (Logtail) Switch from a Private Network Domain Name to a Public Network During Runtime? Can It Automatically Switch Back?

During LoongCollector (Logtail) runtime, if it detects abnormal private network domain name communication (such as network unavailability or connection timeouts), the system automatically switches to the public network domain name to send data. This ensures the continuity and reliability of log collection and prevents log accumulation or loss.

• LoongCollector: After the private network recovers, it automatically switches back to the private network.

• Logtail: It does not automatically switch back. You must manually restart it to restore private network communication.

In the Processor Configurations section of the Logtail Configuration page, add processors to structure raw logs. To add a processing plugin to an existing collection configuration, follow these steps:

1. In the navigation pane on the left, choose Logstores and find the target logstore.

2. Click the icon before its name to expand the logstore.

3. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

4. On the Logtail configuration page, click Edit.

This section introduces only commonly used processing plugins that cover common log processing use cases. For more features, see Extended processors.

Delimiter Parsing

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Standard JSON Parsing

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Nested JSON Parsing

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

JSON Array Parsing

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Apache Log Parsing

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Data Masking

Mask sensitive data in logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Configuration steps: In the Logtail Configuration area on the Processor Configurations page, click Add Processor, and select Native Processor > Data Masking:

• Original Field: The source field that stores log content before parsing.

• Data Masking Method:

  • const: Replaces sensitive content with the modified string.

  • md5: Replaces sensitive content with its MD5 hash.

• Replacement String: When you select Data Masking Method as const, enter the string to replace sensitive content.

• Content Expression that Precedes Replaced Content: Used to find sensitive content. Configure using RE2 syntax.

• Content Expression to Match Replaced Content: The expression for sensitive content. Configure using RE2 syntax.

1. Named group syntax differences

Go uses the (?P<name>...) syntax to define named groups. It does not support the (?<name>...) syntax used in PCRE.

• Correct example: (?P<year>\d{4})

• Incorrect example: (?<year>\d{4})

2. Unsupported regular expression features

The following common but complex regular expression features are not available in RE2. Avoid using them:

• Assertions: (?=...), (?!...), (?<=...), （?<!...)

• Conditional expressions: (?(condition)true|false)

• Recursive matching: (?R), (?0)

• Subroutine references: (?&name), (?P>name)

• Atomic groups: (?>...)

3. Usage recommendations

When debugging regular expressions using tools such as Regex101, select the Golang (RE2) mode for validation to ensure compatibility. If the unsupported syntax mentioned above is used, the plugin cannot correctly parse or match.