Simple Log Service:Collect container logs (standard output and files) from a cluster using the console

1. Global and input configurations

Before you begin, make sure that you have selected a data import template and applied it to a machine group. In this step, you define the name of the collection configuration, the log source, and the collection scope.

View reported logs

1. Confirm that the log file has new content: LoongCollector collects only incremental logs. Run the tail -f /path/to/your/log/file command and trigger a service operation to ensure that new logs are being written.

2. Query logs: Go to the query and analysis page of the target logstore and click Search & Analyze. The default time range is the last 15 minutes. Check whether new logs are flowing in. Each collected container text log contains the following default field information:

   Field	Description
   __tag__:__hostname__	The name of the container's host.
   __tag__:__path__	The path of the log file within the container.
   __tag__:_container_ip_	The IP address of the container.
   __tag__:_image_name_	The name of the image used by the container.
   __tag__:_pod_name_	The name of the pod.
   __tag__:_namespace_	The namespace to which the pod belongs.
   __tag__:_pod_uid_	The unique identifier (UID) of the pod.

Common troubleshooting issues

How do I send ACK cluster logs to a project in another Alibaba Cloud account?

Manually install the SLS LoongCollector (Logtail) component in the ACK cluster and configure it with the Alibaba Cloud account ID or access credential (AccessKey) of the target account. This lets you send container logs to an SLS project that belongs to another Alibaba Cloud account.

Use case: You want to collect log data from an ACK cluster and send it to an SLS project that belongs to a different Alibaba Cloud account for reasons such as organizational structure, permission isolation, or unified monitoring. Manually install LoongCollector (Logtail) for cross-account configuration.

Procedure: The following procedure describes how to manually install LoongCollector. For information about how to install Logtail, see Install and configure Logtail.

1. Connect to the Kubernetes cluster and run the command for your cluster's region:

   China regions

   wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

   Regions outside China

   wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

2. Go to the loongcollector-custom-k8s-package directory and modify the ./loongcollector/values.yaml configuration file.

   # ===================== Required Information =====================
   # The name of the project where logs from this cluster will be collected, for example, k8s-log-custom-sd89ehdq
   projectName: ""
   # The region of the project, for example, Shanghai: cn-shanghai
   region: ""
   # The UID of the Alibaba Cloud account that owns the project. Enclose it in quotation marks, for example, "123456789"
   aliUid: ""
   # The network to use. Optional parameters: Internet, Intranet. The default is Internet.
   net: Internet
   # The AccessKey ID and AccessKey secret of the Alibaba Cloud account or RAM user. The account must have the AliyunLogFullAccess system policy permission.
   accessKeyID: ""
   accessKeySecret: ""
   # A custom cluster ID. The name can contain only uppercase letters, lowercase letters, digits, and hyphens (-).
   clusterID: ""

3. In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependent components:

   bash k8s-custom-install.sh install

4. After the installation is complete, check the running status of the components.

   If a pod fails to start, check whether the `values.yaml` configuration is correct and whether the relevant images were pulled successfully.

   # Check the pod status
   kubectl get po -n kube-system | grep loongcollector-ds

   SLS also automatically creates the following resources. Log on to the Simple Log Service console to view them.

   Resource type	Resource name	Function
   Project	The value of projectName defined in the values.yaml file	A resource management unit that isolates logs of different services. To create a Project for more flexible log resource management, see Create a Project.
   Machine group	k8s-group-${cluster_id}	A collection of log collection nodes.
   Logstore	config-operation-log Important Do not delete this logstore.	Stores logs of the loongcollector-operator component. Its billing method is the same as that of a regular logstore. For more information, see Pay-by-data-written billable items. We recommend that you do not create collection configurations in this logstore.

How can I collect logs from the same file or container standard output using multiple collection configurations?

By default, to prevent data duplication, SLS allows each log source to be collected by only one collection configuration:

• A text log file can match only one Logtail collection configuration.

• A container's standard output (stdout):

  • If you use the new version of the standard output template, the standard output can be collected by only one standard output collection configuration by default.

  • If you use the old version of the standard output template, no extra configuration is required. It supports multiple collections by default.

1. Log on to the Simple Log Service console and go to the target project.

2. In the navigation pane on the left, choose Logstores and find the target logstore.

3. Click the icon before its name to expand the logstore.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit and scroll down to the Input Configurations section:

   • To collect text file logs: Enable Allow File To Be Collected For Multiple Times.

   • To collect container standard output: Enable Allow Collection By Different Logtail Configurations.

Why do I get a dependency error when uninstalling the loongcollector(logtail-ds) component in ACK?

Problem: When you try to delete the loongcollector (logtail-ds) log collection component in Container Service for Kubernetes (ACK), the system reports an error: `The dependencies of this component are not met`.

Dependencies of addons are not met: terway-eniip depends on logtail-ds(>0.0) whose version is v3.x.x.x-aliyun or will be v3.x.x.x-aliyun.

Cause: The terway-eniip network plugin has the log collection feature enabled, which depends on the loongcollector (logtail-ds) component. Therefore, ACK does not allow you to directly uninstall loongcollector (logtail-ds) before you remove this dependency.

Solution: Follow these steps to remove the dependency and then uninstall the component:

1. Log on to the ACK console.

2. In the cluster list, click the name of the target cluster.

3. In the navigation pane on the left, click Add-ons.

4. In the component list, search for and find the terway-eniip component. Click > Disable Logging.

5. In the dialog box that appears, click OK.

6. After the configuration takes effect, try to uninstall the loongcollector (logtail-ds) component again.

Why is the last log entry reported after a long delay? Why is it sometimes truncated?

Cause: Logs are usually truncated if a log file is missing a line feed at the end, or if a multiline log, such as an exception stack, has not been fully written. Because the collector cannot determine whether the log has ended, the last part of the content might be split prematurely or reported with a delay. The handling mechanism differs based on the version of LoongCollector (Logtail):

• Versions earlier than 1.8:
  If the last line of a log does not have a line feed (carriage return), or if a multiline log paragraph is not finished, the collector waits for the next write to trigger an output. This can cause the last log entry to be held for a long time without being sent until a new log is written.

• Version 1.8 and later:
  A timeout refresh mechanism was introduced to prevent logs from getting stuck. When an incomplete log line is detected, the system starts a timer. After the timeout period ends, the current content is automatically submitted, which ensures that the log is eventually collected.

  • Default timeout: 60 seconds. This ensures integrity in most use cases.

  • Adjust this value as needed, but do not set it to 0. This might cause logs to be truncated or partially lost.

Solution:

Extend the waiting time to ensure that the complete log is written before it is collected:

1. Log on to the Simple Log Service console and go to the target project.

2. In the navigation pane on the left, choose Logstores and find the target logstore.

3. Click the icon before its name to expand the logstore.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the Logtail configuration page, click Edit:

   • Choose Input Configurations > Other Input Configurations > Advanced Parameters. Add the following JSON configuration to customize the timeout period.

     {
       "FlushTimeoutSecs": 1
     }

     • Default value: Determined by the startup parameter default_reader_flush_timeout, which is usually a few seconds.

     • Unit: Seconds.

     • Recommended value: ≥1 second. Do not set it to 0. This might cause logs to be truncated or partially lost.

6. After configuration, click Save.

Why does LoongCollector (Logtail) switch from an internal network domain to the public network during operation? Can it switch back automatically?

If LoongCollector (Logtail) detects an anomaly in internal network domain communication, such as a network failure or connection timeout, the system automatically switches to the public domain name for data transmission. This ensures the continuity and reliability of log collection and prevents log accumulation or loss.

• LoongCollector: Automatically switches back to the internal network after the network is restored.

• Logtail: Does not automatically switch back. You must manually restart it to resume internal network communication.

In the Processor Configurations section of the Logtail Configuration page, add processors to structure raw logs. To add a processing plugin to an existing collection configuration, follow these steps:

1. In the navigation pane on the left, choose Logstores and find the target logstore.

2. Click the icon before its name to expand the logstore.

3. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

4. On the Logtail configuration page, click Edit.

This section introduces only commonly used processing plugins that cover common log processing use cases. For more features, see Extended processors.

Delimiter-based parsing

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Standard JSON parsing

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Nested JSON parsing

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

JSON array parsing

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Apache log parsing

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Data masking

Mask sensitive data in logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Procedure: In the Processor Configurations section of the Logtail Configuration page, click Add Processor and choose Native Processor > Data Masking:

• Original Field: The field used to store log content before parsing.

• Data Masking Method:

  • const: Replaces the sensitive content with a specified string.

  • md5: Replaces the sensitive content with its corresponding MD5 hash.

• Replacement String: If you set Data Masking Method to const, you must enter a string to replace the sensitive content.

• Content Expression that Precedes Replaced Content: Used to find sensitive content, configured using RE2 syntax.

• Content Expression to Match Replaced Content: An expression for sensitive content, configured using RE2 syntax.

1. Named group syntax differences

Go uses the (?P<name>...) syntax to define named groups and does not support the (?<name>...) syntax from PCRE.

• Correct: (?P<year>\d{4})

• Incorrect: (?<year>\d{4})

2. Unsupported regular expression features

The following common but complex regular expression features are not supported by RE2. Avoid using them:

• Assertions: (?=...), (?!...), (?<=...), or (?<!...)

• Conditional expressions: (?(condition)true|false)

• Recursive matching: (?R) or (?0)

• Subprogram references: (?&name) or (?P>name)

• Atomic groups: (?>...)

3. Recommendations

Use a tool such as Regex101 to debug regular expressions. To validate compatibility, select the Golang (RE2) mode. The plugin cannot parse or match expressions that contain unsupported syntax.