Simple Log Service:Collect cluster container logs via the console

1. Global and input configuration

Before you start, ensure you have selected a data collection template and applied it to a machine group. In this step, you define the collection configuration name, log source, and scope.

View collected logs

1. Confirm that new logs are being written to the log file: LoongCollector collects only incremental logs. Run the tail -f /path/to/your/log/file command and trigger a service operation to generate new logs.

2. Query logs: Go to the query and analysis page of the target Logstore and click Search & Analyze. The default time range is the last 15 minutes. Check for new logs. By default, each collected container text log includes the following fields:

   Field name	Description
   __tag__:__hostname__	The name of the container host.
   __tag__:__path__	The path of the log file within the container.
   __tag__:_container_ip_	The IP address of the container.
   __tag__:_image_name_	The name of the image used by the container.
   __tag__:_pod_name_	The name of the Pod.
   __tag__:_namespace_	The Pod's namespace.
   __tag__:_pod_uid_	The unique identifier (UID) of the Pod.

Troubleshooting common issues

Manage multi-destination distribution configurations

Because multi-destination distribution configurations are associated with multiple Logstores, they must be managed on the Project-level management page:

1. Log on to the Simple Log Service Console and click the name of the target Project.

2. On the Project page, click Resource Group > Configurations in the left-side navigation pane.

   Note

   This page centrally manages all collection configurations in the Project, including remaining configurations from Logstores that were accidentally deleted.

Transfer ACK cluster logs to another account

You can send container logs to a Simple Log Service Project in another Alibaba Cloud account by manually installing the LoongCollector (Logtail) component in the ACK cluster and configuring it with the target account's Alibaba Cloud account ID or access credential (AccessKey).

Scenario: To collect log data from an ACK cluster into a Simple Log Service Project in a different Alibaba Cloud account for reasons such as organizational structure, permission isolation, or centralized monitoring, manually install LoongCollector (Logtail) to enable cross-account collection.

Procedure: Follow these steps to manually install LoongCollector. For more information, see Install and configure Logtail.

1. Connect to your Kubernetes cluster and run the command that corresponds to your cluster's region:

   Chinese mainland regions

   wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

   Regions outside China

   wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

2. Change to the loongcollector-custom-k8s-package directory and modify the ./loongcollector/values.yaml configuration file.

   # ===================== Required information =====================
   # The Project where logs from this cluster are collected. Example: k8s-log-custom-sd89ehdq
   projectName: ""
   # The region where the Project is located. Example: cn-shanghai
   region: ""
   # The ID of the Alibaba Cloud account that owns the Project. Enclose the ID in quotation marks (""). Example: "123456789"
   aliUid: ""
   # The network type. Valid values: Internet and Intranet. Default value: Internet.
   net: Internet
   # The AccessKey ID and AccessKey secret of the Alibaba Cloud account or RAM user. The AliyunLogFullAccess policy is required.
   accessKeyID: ""
   accessKeySecret: ""
   # A custom cluster ID. The ID can contain only uppercase letters, lowercase letters, digits, and hyphens (-).
   clusterID: ""

3. In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependencies:

   bash k8s-custom-install.sh install

4. After the installation is complete, check the status of the component.

   If a pod fails to start, verify the configuration in the values.yaml file and ensure the required images have been pulled.

   # Check the pod status
   kubectl get po -n kube-system | grep loongcollector-ds

   Simple Log Service also automatically creates the following resources, which you can view on the Simple Log Service console.

   Type	Parameter	Description
   Project	The value of projectName defined in the values.yaml file	A resource management unit that isolates logs from different services. To create a Project for more flexible log resource management, see Create a Project.
   machine group	k8s-group-${cluster_id}	A set of log collection nodes.

   Important

   The LoongCollector component does not create a logstore named config-operation-log. If this logstore already exists, LoongCollector will not write new logs to it.

Multiple configurations for a single source

By default, Simple Log Service uses only one collection configuration per log source to prevent data duplication:

• Each text log file can be matched with only one Logtail collection configuration.

• For a container's standard output (stdout):

  • If you use the new standard output template, only one collection configuration can collect from stdout by default.

  • The legacy standard output template supports multiple collections by default without extra configuration.

1. Log on to the Simple Log Service Console and go to the target Project.

2. In the left-side navigation pane, click Logstores and find the target Logstore.

3. Click the icon next to the Logstore name to expand its details.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the configuration page, click Edit and scroll down to the Input Configurations section.

   • For text file logs, enable Allow File to Be Collected for Multiple Times.

   • For container standard output, enable Allow Collection by Different Logtail Configurations.

Dependency error during uninstallation

Symptom: When you try to uninstall the loongcollector (logtail-ds) log collection component in Container Service for Kubernetes (ACK), the system reports the following error:

Dependencies of addons are not met: terway-eniip depends on logtail-ds(>0.0) whose version is v3.x.x.x-aliyun or will be v3.x.x.x-aliyun

Cause: This error occurs because the terway-eniip network plugin's log collection feature creates a dependency on the loongcollector (logtail-ds) component. Therefore, ACK prevents uninstallation until this dependency is removed.

Solution: Follow these steps to remove the dependency before uninstalling the component:

1. Log on to the ACK Console.

2. In the cluster list, click the name of the target cluster.

3. In the left-side navigation pane, click Add-ons.

4. On the Add-ons page, find the terway-eniip component and click Disable Logging.

5. In the confirmation dialog box that appears, click OK.

6. After the configuration is applied, try to uninstall the loongcollector (logtail-ds) component again.

Delayed or truncated last log entry

Analysis: Log truncation often occurs when the last line in a log file lacks a line feed character, or when a multiline log entry (such as an exception stack) is incomplete. This happens because the collector cannot determine if a log entry is complete, which may cause the final segment to be split prematurely or reported after a delay. The handling mechanism varies by the LoongCollector (Logtail) version:

• Versions earlier than 1.8:
  If the last line of a log lacks a line feed (or carriage return), or if a multiline log is incomplete, the collector waits for the next write operation to trigger the output. This can cause the last log entry to be held for an extended period until a new log is written.
  
  
  
  
  
  
  
  

• Version 1.8 and later:
  These versions introduce a timeout flush mechanism to prevent logs from being held indefinitely. When the collector detects an incomplete log line, a timer starts. After the timeout period, the current buffer is automatically submitted, ensuring the log is eventually collected.
  
  
  
  
  
  
  
  

  • Default timeout: 60 seconds (ensures log integrity in most scenarios).

  • You can adjust this value based on your needs, but do not set it to 0. Setting the value to 0 may cause log truncation or partial data loss.

Solution:

You can extend the timeout period to ensure logs are fully written before being collected:

1. Log on to the Simple Log Service Console and go to the target Project.

2. In the left-side navigation pane, click Logstores and find the target Logstore.

3. Click the icon next to the Logstore name to expand its details.

4. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

5. On the configuration page, click Edit.

   • Navigate to Input Configurations > Other Input Configurations > Advanced Parameters and add the following JSON configuration to customize the timeout period.

     {
       "FlushTimeoutSecs": 1
     }

     • Default value: Determined by the default_reader_flush_timeout startup parameter, which is typically a few seconds.

     • Unit: Seconds.

     • Recommended value: ≥1. Do not set this to 0, as it may cause log truncation or partial data loss.

6. After you complete the configuration, click OK.

Network endpoint failover

If LoongCollector (Logtail) detects a communication anomaly on the internal network, such as a network failure or connection timeout, it automatically fails over to the public network endpoint for data transmission. This ensures continuous and reliable log collection, preventing log backlogs or data loss.

• LoongCollector: Automatically switches back to the internal network once connectivity is restored.

• Logtail: Does not automatically switch back. You must manually restart the component to resume communication over the internal network.

In the Processor Configurations section of the Logtail Configuration page, add processors to structure raw logs. To add a processing plugin to an existing configuration:

1. In the navigation pane on the left, choose Logstores and find the target logstore.

2. Click the icon before its name to expand the logstore.

3. Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.

4. On the Logtail configuration page, click Edit.

This section introduces only commonly used processing plugins that cover common log processing use cases. For more features, see Extended processors.

Data parsing (delimiter mode)

05/May/2025:13:30:28,10.10.*.*,"POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1",200,18204,aliyun-sdk-java

ip:10.10.*.*
request:POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=****************&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=******************************** HTTP/1.1
size:18204
status:200
time:05/May/2025:13:30:28
user_agent:aliyun-sdk-java

Data parsing (JSON mode)

{"url": "POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek********&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1", "ip": "10.200.98.220", "user-agent": "aliyun-sdk-java", "request": {"status": "200", "latency": "18204"}, "time": "05/Jan/2025:13:30:28"}

ip: 10.200.98.220
request: {"status": "200", "latency" : "18204" }
time: 05/Jan/2025:13:30:28
url: POST /PutData?Category=YunOsAccountOpLog&AccessKeyId=U0Ujpek******&Date=Fri%2C%2028%20Jun%202013%2006%3A53%3A30%20GMT&Topic=raw&Signature=pD12XYLmGxKQ%2Bmkd6x7hAgQ7b1c%3D HTTP/1.1
user-agent:aliyun-sdk-java

Expand JSON field

{"s_key":{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}}

0_s_key_k1_k2_k3_k41:41
0_s_key_k1_k2_k3_k4_k51:51
0_s_key_k1_k2_k3_k4_k52:52

1_s_key:{"k1":{"k2":{"k3":{"k4":{"k51":"51","k52":"52"},"k41":"41"}}}}

JSON array parsing

[{"key1":"value1"},{"key2":"value2"}]

json1:{"key1":"value1"}
json2:{"key2":"value2"}

* | extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')

Data parsing (Apache mode)

1 192.168.1.10 - - [08/May/2024:15:30:28 +0800] "GET /index.html HTTP/1.1" 200 1234 "https://www.example.com/referrer" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36"

http_referer:https://www.example.com/referrer
http_user_agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.X.X Safari/537.36
remote_addr:192.168.1.10
remote_ident:-
remote_user:-
request_method:GET
request_protocol:HTTP/1.1
request_uri:/index.html
response_size_bytes:1234
status:200
time_local:[08/May/2024:15:30:28 +0800]

Data masking

Mask sensitive data in your logs.

Example:

[{'account':'1812213231432969','password':'04a23f38'}, {'account':'1812213685634','password':'123a'}]

[{'account':'1812213231432969','password':'********'}, {'account':'1812213685634','password':'********'}]

Procedure: On the Logtail Configuration page, in the Processor Configurations section, click Add Processor, and then select Native Processor > Data Masking:

• Original Field: The field that contains the content to be masked.

• Data Masking Method:

  • const: Replaces sensitive content with a specified string.

  • md5: Replaces sensitive content with its MD5 hash value.

• Replacement String: If you set Data Masking Method to const, you must enter the string to replace the sensitive content.

• Content Expression that Precedes Replaced Content: A regular expression that matches the content immediately before the sensitive data. This expression uses the RE2 syntax.

• Content Expression to Match Replaced Content: A regular expression that matches the sensitive content. This expression uses the RE2 syntax.

1. Named group syntax differences

Go uses the (?P<name>...) syntax for named groups and does not support the (?<name>...) syntax used in PCRE.

• Correct: (?P<year>\d{4})

• Incorrect: (?<year>\d{4})

2. Unsupported regular expression features

RE2 does not support the following common but complex regular expression features. Avoid using them:

• Assertion: (?=...), (?!...), (?<=...), or (?<!...)

• Conditional expression: (?(condition)true|false)

• Recursive matching: (?R) or (?0)

• Subprogram reference: (?&name) or (?P>name)

• Atomic group: (?>...)

3. Recommendations

To debug your regular expressions, use a tool like Regex101. To ensure compatibility, select the Golang (RE2) mode for validation. The plugin will fail to parse or find matches with expressions that use unsupported syntax.