Collect container logs with the AliyunPipelineConfig CRD - Simple Log Service

Define log collection configurations as Kubernetes CRDs for version control, environment-specific deployments, and automated releases with kubectl or CI/CD pipelines across ACK and self-managed clusters. LoongCollector hot-reloads configuration changes without restart.

The legacy AliyunLogConfig CRD is deprecated. Use AliyunPipelineConfig instead. See CRD types for a comparison.

Important

CRD-based collection configurations can only be modified by updating the CRD. Changes made in the SLS console are not synchronized and do not take effect.

Applicability

Operating environment
- ACK clusters (managed and dedicated editions) and self-managed Kubernetes clusters.
- The Kubernetes version must be 1.16.0 or later and support Mount propagation: HostToContainer.
- container runtime (Docker and Containerd only)
  - Docker:
    - Requires access to docker.sock.
    - Standard output collection supports only the JSON-file log driver.
    - Only the overlay and overlay2 storage drivers are supported. For other types of storage drivers, you must manually mount the log directories.
  - Containerd: Requires access to containerd.sock.
Resource requirements: LoongCollector (Logtail) runs with the 'system-cluster-critical' priority class. Do not deploy if the cluster has insufficient resources — it may evict Pods on nodes.
- CPU: Reserve at least 0.1 Core.
- Memory: Reserve at least 150 MB for the collection component and at least 100 MB for the controller component.
- Actual usage depends on collection rate, monitored directories and files, and send congestion. Keep usage below 80% of configured limits.
Permission requirements: To deploy, the Alibaba Cloud account or RAM user must have the AliyunLogFullAccess permission.
To create a custom policy, refer to the AliyunCSManagedLogRolePolicy system policy. Copy the permissions from this policy and grant them to the target RAM user or role to configure fine-grained permissions.

Collection configuration workflow

Install LoongCollector: Deploy LoongCollector as a DaemonSet to run a collection container on each cluster node. This configuration centralizes log collection for all containers on the node.
Create a LogStore: A LogStore is a storage unit for log data. Multiple LogStores can be created within a Project.

Create a collection configuration YAML file: Use kubectl to connect to the cluster. You can create the collection configuration file in one of two ways:

Method 1: Use the collection configuration generator
Use the collection configuration generator in the Log Service console to visually configure parameters and automatically generate a standard YAML file.
Method 2: Manually write the YAML file
Manually write a YAML file based on the examples in this document. Build progressively: start with a minimal configuration, add processing logic, then enable advanced features.
For complex scenarios not covered in this document or fields that require deep customization, see AliyunPipelineConfig parameter reference for a complete list of fields, value rules, and plugin capability details.

A complete collection configuration typically includes the following parts:

Minimal configuration (Required): This configuration builds the data pipeline from the cluster to Log Service and consists of two parts:
- Inputs (inputs): Defines the log source. For container logs, sources include the following two types. To collect other types of logs, such as MySQL query results, see Input plugins.
  - Container standard output (stdout and stderr): Logs that the container application prints to the console.
  - Text log files: Log files written to a specified path inside the container.
- Outputs (flushers): Defines the log destination. Collected logs are sent to the specified LogStore.
  If the target Project or LogStore does not exist, the system automatically creates it. You can also manually create a Project and a LogStore in advance.
Common processing configurations (Optional): Define the processors field to perform structured parsing (such as regular expression or delimiter parsing), data masking, or filtering on raw logs.
This document describes only native processing plugins that cover common log processing scenarios. For more features, see Extended processing plugins.
Other advanced configurations (Optional): Enables advanced features such as multi-line log collection and log tag enrichment to meet more specific collection requirements.

Structure example:

apiVersion: telemetry.alibabacloud.com/v1alpha1 # Use the default value. Do not modify.
kind: ClusterAliyunPipelineConfig               # Use the default value. Do not modify.
metadata:
  name: test-config                             # Set the resource name, which must be unique within the current Kubernetes cluster.
spec:
  project:                                      # Set the name of the target Project.
    name: k8s-your-project                      
  config:                                       # Set the Logtail collection configuration.
    inputs:                                     # Set the input plugins for the Logtail collection configuration.
      ...
    processors:                                 # Set the processing plugins for the Logtail collection configuration.
      ...
    flushers:                                   # Set the output plugins for the Logtail collection configuration.
      ...

Apply the configuration
```
kubectl apply -f <your_yaml>
```

Install LoongCollector (Logtail)

LoongCollector is the next-generation log collection agent from SLS, an upgraded version of Logtail. The two cannot coexist. To install Logtail instead, see Install and configure Logtail.

Follow these steps for basic LoongCollector installation. For detailed parameters, see Installation and configuration. If LoongCollector or Logtail is already installed, skip to logstore creation.

ACK cluster

You can install LoongCollector from the Container Service for Kubernetes (ACK) console. By default, logs are sent to a Simple Log Service Project under the current Alibaba Cloud account.

Log on to the ACK console. In the left navigation pane, click Clusters.
On the Clusters page, click the name of the target cluster to open its details page.
In the left-side navigation pane, click Add-ons.

On the Logs and Monitoring tab, find loongcollector, and click Install.

Note

When you create a cluster, on the Component Configurations page, select the Enable Log Service checkbox. You can then choose to Create Project or Select Project.

After the installation is complete, Simple Log Service automatically creates related resources in your ACK cluster's region. View these resources in the Simple Log Service console.

Resource type

Resource name

Description

Project

k8s-log-${cluster_id}

Isolates logs from different services.

To create a Project for more flexible log resource management, see Create a Project.

machine group

k8s-group-${cluster_id}

A group of log collection nodes.

Important

LoongCollector does not create a logstore named config-operation-log. If a logstore with this name already exists, LoongCollector stops writing logs to it.

Self-managed cluster

Connect to your Kubernetes cluster. Then, run the appropriate command for your region to download LoongCollector and its dependencies:

For regions in the Chinese mainland:

wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

For regions outside the Chinese mainland:

wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

Go to the loongcollector-custom-k8s-package directory and modify the ./loongcollector/values.yaml configuration file.

# ===================== Required parameters =====================
# The Project where logs are sent. Example: k8s-log-custom-sd89ehdq.
projectName: ""
# The Project's region. Example: cn-shanghai.
region: ""
# The UID of the Alibaba Cloud account that owns the Project. Enclose the value in quotation marks. Example: "123456789"
aliUid: ""
# The network type. Valid values: Internet (public network) and Intranet (internal network). Default: Internet.
net: Internet
# The AccessKey ID and secret for an Alibaba Cloud account or RAM user that has the AliyunLogFullAccess policy.
accessKeyID: ""
accessKeySecret: ""
# The custom cluster ID. The ID can contain only uppercase letters, lowercase letters, digits, and hyphens (-).
clusterID: ""

In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependencies:
```
bash k8s-custom-install.sh install
```

After the installation is complete, check the status of the components.

If the pods fail to start, verify the values.yaml configuration and ensure the required images were pulled.

# Check pod status
kubectl get po -n kube-system | grep loongcollector-ds

Simple Log Service also automatically creates the following resources. View these resources in the Simple Log Service console.

Resource type	Resource name	Description
Project	The value of `projectName` that you specified in the values.yaml file	Isolates logs from different services.
machine group	`k8s-group-${cluster_id}`	A group of log collection nodes.

Important

LoongCollector does not create a logstore named config-operation-log. If a logstore with this name already exists, LoongCollector stops writing logs to it.

Create a Logstore

If you have already created a logstore, skip this step and go to collection configuration.

Log on to the Log Service console and click the target project.
In the left-side navigation pane, choose Log Storage and click +.
On the Create Logstore page, configure the following core parameters:
- Logstore Name: Enter a name that is unique within the project. This name cannot be changed after creation.
- Logstore Type: Choose Standard or Query based on their specifications.
- Billing Mode:
  - Pay-by-feature: You are billed independently for each resource, such as storage, indexing, and read/write operations. Suitable for small-scale use cases or when feature usage is uncertain.
  - Pay-by-ingested-data: You are billed only for the amount of raw data ingested. This mode provides a 30-day free storage period and free features such as data transformation and delivery. This simple cost model is ideal for cases where the storage period is close to 30 days or the data processing pipeline is complex.
- Data Retention Period: Set the number of days to retain logs. The value ranges from 1 to 3650 days. A value of 3650 means permanent storage. The default is 30 days.
Leave the other settings at their defaults and click OK. Manage logstores.

Minimal configuration

In spec.config, you configure input and output plugins to define the log source and destination.

Container standard output

Purpose: Collects container standard output logs (stdout/stderr) that are printed directly to the console.

inputs (input plugin)

Defines the log source. Currently, you can configure only one input plugin.

Type String (Required)
Set to input_container_stdio.
IgnoringStderr boolean (Optional)
Whether to ignore the standard error stream (stderr). Default: false.
- true: Does not collect stderr.
- false: Collects stderr.
IgnoringStdout boolean (Optional)
Whether to ignore the standard output stream (stdout). Default: false.
- true: Does not collect stdout.
- false: Collects stdout.

Example

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  # Set a unique resource name within the Kubernetes cluster. 
  # This name is also used for the created Logtail collection configuration.
  name: new-stdio-config
spec:
  project:
    name: test-not-exist
  logstores:
    - name: new-stdio-logstore

  # Define the collection and processing configuration for LoongCollector (Logtail).
  config:
    # --- input plugin: Defines the log source ---
    inputs:
      # Use the input_container_stdio plugin to collect container standard output logs.
      - Type: input_container_stdio
        IgnoringStderr: false
        IgnoringStdout: false

    # --- processing plugin (Optional): Defines how to parse and process logs ---
    processors: []

    # --- output plugin: Defines the log destination ---
    flushers: # Configure one or more flushers to define log destinations. This example sends logs to two Logstores.
      - Type: flusher_sls    # Specify the SLS output plugin.
        Logstore: new-stdio-logstore1
      - Type: flusher_sls    # Specify the SLS output plugin.
        Logstore: new-stdio-logstore2

flushers (output plugin)

Sends collected logs to a specified Logstore in a Project. You can configure up to five output plugins.

Type String (Required)
Set to flusher_sls.
Logstore String (Required)
The name of the target Logstore, which determines where the logs are stored.
Note
- The specified Logstore must exist or be declared in spec.logstores.
- When you configure multiple output targets, the configuration is no longer displayed in the Logstore's configuration list. To manage the multi-target configuration, see How do I manage a multi-target configuration?.

Container text files

Purpose: Collects logs written to a specific file path inside a container, such as access.log or app.log.

inputs (input plugin)

Defines the log source. Currently, you can configure only one input plugin.

Type String (Required)
Set to input_file.
FilePaths String (Required)
A list of paths to the log files.
- Currently, you can configure only one path.
- You can use the following wildcard characters:
  - *: Matches file names in a single-level directory.
  - **: Recursively matches multi-level subdirectories. It can be used only once and must be placed before the filename.
MaxDirSearchDepth integer (Optional)
Maximum directory depth for recursive search when the path contains **. Default: 0. Valid values: 0 to 1,000.
FileEncoding String (Optional)
The file encoding. Default: utf8. Valid values:
- utf8
- gbk
EnableContainerDiscovery boolean (Optional)
Whether to enable container discovery. Default: true.
Note
This parameter takes effect only when LoongCollector (Logtail) runs in DaemonSet mode and the specified file path is inside the container.

Example

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: easy-row-config
spec:
  # Specify the target Project to which logs are sent.
  project:
    name: test-not-exist
  logstores:
    - name: easy-row-logstore

  # Define the collection and processing configuration for LoongCollector (Logtail).
  config:
    # Log sample (optional)
    sample: ''
    # --- input plugin: Defines the log source ---
    inputs:
      # Use the input_file plugin to collect text log files from containers.
      - Type: input_file         
        # ... Specific configuration for the input plugin ...
        # The file path inside the container.
        FilePaths:
          - /var/log/text1.log
        # The maximum directory monitoring depth.
        MaxDirSearchDepth: 0
        FileEncoding: utf8  
        # Enable the container discovery feature.
        EnableContainerDiscovery: true
        

    # --- processing plugin (Optional): Defines how to parse and process logs ---
    processors: []    

    # --- output plugin: Defines the log destination ---
    flushers: # Configure one or more flushers to define log destinations. This example sends logs to two Logstores.
      - Type: flusher_sls        # Specify the SLS output plugin.
        Logstore: easy-row-logstore1
      - Type: flusher_sls        # Specify the SLS output plugin.
        Logstore: easy-row-logstore2

flushers (output plugin)

Sends collected logs to a specified Logstore in a Project. You can configure up to five output plugins.

Type String (Required)
Set to flusher_sls.
Logstore String (Required)
The name of the target Logstore, which determines where the logs are stored.
Note
- The specified Logstore must exist or be declared in spec.logstores.
- When you configure multiple output targets, the configuration is no longer displayed in the Logstore's configuration list. To manage the multi-target configuration, see How do I manage a multi-target configuration?.

Common configuration

After completing the minimal configuration, add processor plugins to parse, mask, or filter raw logs.

Core configuration: To configure processing plugins, add processors to spec.config, which allows you to enable multiple plugins simultaneously.

This topic covers only native processing plugins that handle common log processing use cases. For more features, see extended processing plugins.

Important

For Logtail 2.0+ and LoongCollector, follow these plugin ordering guidelines:

Prioritize native plugins.
If native plugins do not meet your needs, configure extension plugins to run after them.
Native plugins can only run before extension plugins.

Structured configuration

Regex parsing

Extracts log fields using a regular expression and parses the logs into key-value pairs.

Parameter

Example

Type String (Required)

Fixed to processor_parse_regex_native.

 # ... under spec.config ...
 processors:
  # Use the regular expression parsing processor to parse log content.
  - Type: processor_parse_regex_native
    # Specify the source field, which is usually content.
    SourceKey: content

    # A regular expression to match and extract log fields.
    Regex: >-
      (\S+)\s-\s(\S+)\s$$([^]]+)$$\s"
      (\w+)\s(\S+)\s([^"]+)"
      \s(\d+)\s(\d+)\s"
      ([^"]+)"\s"
      ([^"]+).*

    # A list of extracted fields, corresponding to the regex groups in order.
    Keys:
      - remote_addr
      - remote_user
      - time_local
      - request_method
      - request_uri
      - request_protocol
      - status
      - body_bytes_sent
      - http_referer
      - http_user_agent

    # Whether to keep the source field if parsing fails.
    KeepingSourceWhenParseFail: true

    # Whether to keep the source field if parsing succeeds.
    KeepingSourceWhenParseSucceed: true

    # If the source field is kept, you can specify a new name for it.
    RenamedSourceKey: fail

SourceKey String (Required)

The name of the source field.

Regex String (Required)

The regular expression for parsing the log.

Keys String (Required)

A list of keys for the extracted fields.

KeepingSourceWhenParseFail boolean (Optional)

Whether to retain the source field when parsing fails. Default: false.

KeepingSourceWhenParseSucceed boolean (Optional)

Whether to retain the source field on successful parsing. Default: false.

RenamedSourceKey String (Optional)

The new name for the source field if it is kept. If omitted, the source field is not renamed.

Delimiter parsing

Parses logs into key-value pairs using a single- or multi-character delimiter.

Parameter

Example

Type String (Required)

Fixed to processor_parse_delimiter_native.

# ... under spec.config ...
processors:
  # Delimiter parsing processor configuration
  - Type: processor_parse_delimiter_native
    # The source field, usually content
    SourceKey: content

    Separator: ','

    Quote: '"'

    # Define the names for the extracted fields in order.
    Keys:
      - time
      - ip
      - request
      - status
      - size
      - user_agent

SourceKey String (Required)

The name of the source field.

Separator String (Required)

The field delimiter. For example, CSV files use a comma (,).

Keys [String] (Required)

A list of keys for the extracted fields.

Quote String (Optional)

The quote character used to wrap fields that contain special characters, such as the delimiter.

AllowingShortenedFields boolean (Optional)

Whether to allow fewer extracted fields than Keys. Default: true. If not allowed, the operation is treated as a parsing failure.

OverflowedFieldsTreatment String (Optional)

Behavior when more fields are extracted than Keys. Default: extend. Valid values:

extend: Retains excess fields and adds them to the log as separate fields. The fields are named _column$i_, where $i is the index of the excess field, starting from 0.
keep: Retains the excess fields, but adds the excess content to the log as a single field named _column0_.
discard: Discards extra fields.

KeepingSourceWhenParseFail boolean (Optional)

Whether to retain the source field if parsing fails. Default: false.

KeepingSourceWhenParseSucceed boolean (Optional)

Whether to retain the source field on successful parsing. Default: false.

RenamedSourceKey String (Optional)

The new name for the source field if it is kept. If omitted, the source field is not renamed.

Standard JSON parsing

Parses a JSON object from a log field into key-value pairs.

Parameter

Example

Type String (Required)

Fixed to processor_parse_json_native.

# ... under spec.config ...
processors:
  # JSON parsing processor configuration
  - Type: processor_parse_json_native
    # The source field of the raw log
    SourceKey: content
    KeepingSourceWhenParseFail: true
    RenamedSourceKey: raw_log

SourceKey String (Required)

The name of the source field.

KeepingSourceWhenParseFail boolean (Optional)

Whether to retain the source field when parsing fails. Default: false.

KeepingSourceWhenParseSucceed boolean (Optional)

Whether to retain the source field on successful parsing. Default: false.

RenamedSourceKey String (Optional)

The new name for the source field if it is kept. If omitted, the source field is not renamed.

Note

Parameter names in the CRD YAML use camelCase, such as RenamedSourceKey. Names in snake_case, such as renamed_source_key, are not valid.

Nested JSON parsing

Flattens a nested JSON object into key-value pairs and allows you to specify the expansion depth.

Parameter

Example

Type String (Required)

Fixed to processor_json.

# ... under spec.config ...
processors:
  # Configure the JSON field expansion processor.
  - Type: processor_json
    # Specify the source field to parse.
    SourceKey: content
    
    ExpandDepth: 0

    ExpandConnector: '_'

    Prefix: expand

    IgnoreFirstConnector: false

    # Whether to expand array elements into separate fields.
    ExpandArray: false

    # Whether to keep the source field content.
    KeepSource: true

    # Whether to report an error if the source field is missing.
    NoKeyError: true

    # Whether to use the source field name as a prefix for expanded field names.
    UseSourceKeyAsPrefix: false

    # Whether to keep the source log data if JSON parsing fails.
    KeepSourceIfParseError: true

SourceKey String (Required)

The name of the source field.

ExpandDepth integer (Optional)

The maximum expansion depth for the nested JSON object. Default: 0.

0: Expands the object to the deepest parsable level.
1: Expands only the top level of the object, and so on.

ExpandConnector String (Optional)

The connector used between keys when flattening a nested JSON object. Default: an underscore (_).

Prefix String (Optional)

The prefix to add to all expanded field names.

IgnoreFirstConnector String (Optional)

Whether to omit the connector before the top-level field. Default: false.

ExpandArray boolean (Optional)

Whether to expand array types. Default: false.

false (default): Does not expand the array.
true: The array is expanded. For example, {"k":["1","2"]} is expanded to {"k[0]":"1","k[1]":"2"}.

Note

This parameter is supported in Logtail 1.8.0 and later.

KeepSource boolean (Optional)

Whether to retain the original field in the parsed log. Default: true.

true: Keep
false: Discard

NoKeyError boolean (Optional)

Whether to report an error if the specified source field is not found. Default: true.

true: Report an error.
false: Do not report an error.

UseSourceKeyAsPrefix boolean (Optional)

If set to true, uses the source field name as a prefix for all expanded field names.

KeepSourceIfParseError boolean (Optional)

Whether to retain raw logs when parsing fails. Default: true.

true: Keep
false: Discard

JSON array parsing

Use the json_extract function to extract a JSON object from a JSON array. JSON Functions.

Parameter	Example
Type `String` (Required) The plugin type. The SPL plugin type is `processor_spl`.	`# ... under spec.config ... processors: # Use an SPL script to process log fields. - Type: processor_spl # Script timeout in milliseconds. TimeoutMilliSeconds: 1000 # The SPL script, used to extract elements from the JSON array in the content field. Script: >- * \| extend json1 = json_extract(content, '$[0]'), json2 = json_extract(content, '$[1]')`
Script `String` (Required) The SPL script used to extract elements from a JSON array in the content field.
TimeoutMilliSeconds `integer` (Optional) The script execution timeout in milliseconds. The value must be between 0 and 10,000. Default value: 1,000.

NGINX log parsing

Parses NGINX logs into key-value pairs based on your log_format definition. If the default format does not meet your requirements, you can use a custom format.

Parameter

Example

Type String (Required)

Fixed to processor_parse_regex_native.

# ... under spec.config ...
processors:
  # NGINX log parsing processor configuration
  - Type: processor_parse_regex_native
    # The source field of the raw log
    SourceKey: content
    
    # Regular expression parsing rule
    Regex: >-
      (\S*)\s*-\s*(\S*)\s*\[
      (\d+/\S+/\d+:\d+:\d+:\d+)\s+\S+\]
      \s*"(\S+)\s+(\S+)\s+\S+"
      \s*(\S*)\s*(\S*)\s*(\S*)\s*(\S*)
      \s*"([^"]*)"\s*"([^"]*)".*
    
    # Map of extracted fields
    Keys:
      - remote_addr
      - remote_user
      - time_local
      - request_method
      - request_uri
      - request_time
      - request_length
      - status
      - body_bytes_sent
      - http_referer
      - http_user_agent
    
    # NGINX-specific configuration
    Extra:
      Format: >-
        log_format main  '$remote_addr - $remote_user [$time_local]
        "$request" ''$request_time $request_length ''$status
        $body_bytes_sent "$http_referer" ''"$http_user_agent"';
      LogType: NGINX

SourceKey String (Required)

The name of the source field.

Regex String (Required)

The regular expression for parsing the NGINX log.

Keys String (Required)

A list of keys for the extracted fields.

Extra

Format String (Required)
The log_format directive from your NGINX configuration file.
In a production environment, this log_format must be consistent with the definition in the Nginx configuration file (typically located at /etc/nginx/nginx.conf).
LogType String (Required)
Fixed to NGINX.

KeepingSourceWhenParseFail boolean (Optional)

Whether to keep the original field if parsing fails. Default: false.

KeepingSourceWhenParseSucceed boolean (Optional)

Whether to retain the original field on successful parsing. Default: false.

RenamedSourceKey String (Optional)

The new name for the source field if it is kept. If omitted, the source field is not renamed.

Apache log parsing

Parses Apache logs into key-value pairs based on the format defined in your Apache configuration file.

Parameter

Example

Type String (Required)

Fixed to processor_parse_regex_native.

# ... under spec.config ...
processors:
  # Configure the Apache Combined log parsing processor (based on a regular expression).
  - Type: processor_parse_regex_native
    # The source field, usually content.
    SourceKey: content

    # A regular expression to match and extract logs in Apache combined format.
    Regex: >-
      ([0-9.-]+)\s                          # remote_addr
      ([\w.-]+)\s                           # remote_ident
      ([\w.-]+)\s                           # remote_user
      (\[[^\[\]]+\]|-)\s                    # time_local
      "((?:[^"]|\")+)"\s                     # request_method + request_uri + request_protocol
      "((?:[^"]|\")+)"\s                     # request_uri (duplicate capture? check the logic)
      "((?:[^"]|\")+)"\s                     # request_protocol
      (\d{3}|-)\s                           # status
      (\d+|-)\s                             # response_size_bytes
      "((?:[^"]|\")+)"\s                     # http_referer
      "((?:[^"]|\"|')+)"                     # http_user_agent

    # A list of extracted fields, corresponding to the regex groups in order.
    Keys:
      - remote_addr
      - remote_ident
      - remote_user
      - time_local
      - request_method
      - request_uri
      - request_protocol
      - status
      - response_size_bytes
      - http_referer
      - http_user_agent

    # Extra plugin information (optional, used to describe the log format).
    Extra:
      Format: >-
        LogFormat "%h %l %u %t \"%r\" %>s %b
        \"%{Referer}i\" \"%{User-Agent}i\"" combined
      LogType: Apache
      SubType: combined

SourceKey String (Required)

The name of the source field.

Regex String (Required)

The regular expression for parsing the Apache log.

Keys String (Required)

A list of keys for the extracted fields.

Extra

Format String (Required)
The LogFormat directive from your Apache configuration file.
LogType String (Required)
Fixed to Apache.
SubType String (Required)
The log format.
- common
- combined
- custom

KeepingSourceWhenParseFail boolean (Optional)

Whether to keep the original field if parsing fails. Default: false.

KeepingSourceWhenParseSucceed boolean (Optional)

Whether to keep the original field on successful parsing. Default: false.

RenamedSourceKey String (Optional)

The new name for the source field if it is kept. If omitted, the source field is not renamed.

Data masking

Use the processor_desensitize_native plugin to mask sensitive data in your logs.

Parameters

Example

Type String (Required)

Set to processor_desensitize_native.

# ... under spec.config ...
processors:
  # Configure the native log masking plugin
  - Type: processor_desensitize_native

    # The name of the source field
    SourceKey: content

    # Masking method, such as 'const' which replaces sensitive data with a fixed string
    Method: const

    # The string to replace sensitive data with
    ReplacingString: '********'

    # A regular expression for content preceding the sensitive data
    ContentPatternBeforeReplacedString: 'password'':'''

    # A regular expression for the sensitive data to be replaced
    ReplacedContentPattern: '[^'']*'

    # Specifies whether to replace all matches. Default is true.
    ReplacingAll: true

SourceKey String (Required)

The name of the source field.

Method String (Required)

The masking method. Supported values:

const: Replaces sensitive data with a constant string.
md5: Replaces sensitive data with its MD5 hash.

ReplacingString String (Optional)

The constant string that replaces sensitive data. This parameter is required when Method is set to const.

ContentPatternBeforeReplacedString String (Required)

A regular expression for the content that precedes the sensitive data.

ReplacedContentPattern String (Required)

A regular expression that matches the sensitive data.

ReplacingAll boolean (Optional)

Whether to replace all matches. Default: true.

Content filtering

Configure the processor_filter_regex_native plugin to match log field values against a regular expression and retain only matching logs.

Parameters	Example
Type `String` (required) The plugin type. This value must be `processor_filter_regex_native`.	`# ...under spec.config... processors: # Configure the regular expression filtering plugin (for log masking or sensitive word filtering). - Type: processor_filter_regex_native # Define a list of regular expressions to match log field content. FilterRegex: # Example: Matches log field values that contain "WARNING" or "ERROR". - WARNING\|ERROR # Specify the names of the log fields to filter. This example filters the 'level' field. FilterKey: - level`
FilterRegex `String` (required) A list of regular expressions for matching log field values.
FilterKey `String` (required) A list of log field names to apply the FilterRegex patterns to.

Time parsing

Parse the time field in a log and use the result as the log's __time__ field.

Key fields

Example

Type String (Required)

The plugin type. Set toprocessor_parse_timestamp_native.

# ...under spec.config...
processors:
  # Configure the native time parsing plugin.
  - Type: processor_parse_timestamp_native
    # The source field containing the timestamp, usually 'content'.
    SourceKey: content

    # The format of the source timestamp. Must match the log's format exactly.
    SourceFormat: '%Y-%m-%d %H:%M:%S'
    
    SourceTimezone: 'GMT+00:00'

SourceKey String (Required)

The source field name.

SourceFormat String (Required)

The time format. This must exactly match the time format in the log.

SourceTimezone String (Optional)

The time zone of the source timestamp. Default: the time zone of the machine running LoongCollector.

Format:

GMT+HH:MM: Time zones east of GMT
GMT-HH:MM: Time zones west of GMT

Advanced configurations

After completing the minimal configuration, use these advanced settings for more granular collection:

Configure multiline log collection: Enable multiline mode with a start-pattern regex to collect entries spanning multiple lines (such as stack traces) as single log records.
Configure log topic types: Assign topics to log streams for easier organization and retrieval.
Specify containers for collection (filtering and blacklisting): Use whitelists and blacklists to collect logs from specific containers and paths only.
Enrich logs with tags: Add metadata from environment variables and pod labels to your logs as tags.

Configure multiline log collection

By default, SLS splits logs line by line, breaking multiline entries like stack traces into separate records.

Enable multiline mode and configure a start-pattern regex to group multiline entries into single log records.

Key configuration: Add the Multiline parameter to the spec.config.inputs configuration.

Field details

Example

Multiline

Enables multiline log collection.

Mode
The collection mode. The default value is custom.
- custom: Use a custom regular expression to match the start of a line.
- JSON: Multiline JSON format.
StartPattern
The regular expression for the start of a line. This parameter is required when Mode is set to custom.

# ...under spec.config...
inputs:
  - Type: input_file
    # Enable multiline log collection.
    Multiline:
      # Mode selection: custom indicates a custom regular expression to match the start of a line.
      Mode: custom
      # The regular expression matches the start of each log entry, which marks the beginning of a new log.
      StartPattern: '\d+-\d+-\d+\s\d+:\d+:\d+'

Log topic types

Key configuration: Add the global parameter to spec.config to set the log topic.

Field details

Example

TopicType

The topic type. Valid values:

machine_group_topic: Uses the topic from the machine group where this configuration is applied. This helps distinguish logs from different machine groups.
filepath: Extracts a topic from the file path. This helps distinguish log data from different users or applications.
custom: Uses a custom, static log topic.

Machine group topic

spec: 
  config:
    global: 
    # Use the topic of the machine group to which this configuration is applied as the log topic.
      TopicType: machine_group_topic

File path extraction

spec:  
  config:
    global: 
      TopicType: filepath
    # The topic format. Required when TopicType is filepath or custom.
    # The extraction results are __topic__: userA, __topic__: userB, and __topic__: userC.
      TopicFormat: \/data\/logs\/(.*)\/serviceA\/.*

Custom

spec:  
  config:
    global: 
      TopicType: custom
    # The topic format. Required when TopicType is filepath or custom.
      TopicFormat: customized://

TopicFormat

The topic format. This parameter is required when TopicType is set to filepath or custom.

Container filtering and blacklisting

Filtering

Collect logs only from containers matching specified conditions. Multiple conditions combine with logical AND. Empty conditions are ignored. All conditions support regular expressions.

Key configuration: Configure container filtering parameters under ContainerFilters in the spec.config.inputs section.

Field details

Example

ContainerFilters

Container filtering settings.

Pod label whitelist/blacklist
- IncludeK8sLabel
  K8s pod label whitelist. Collects logs from containers with the specified pod labels.
- ExcludeK8sLabel
  K8s pod label blacklist. Excludes logs from containers with the specified pod labels.
Environment variable whitelist/blacklist
- IncludeEnv
  Environment variable whitelist.
- ExcludeEnv
  Environment variable blacklist.
Regex matching for pod/namespace/container names
- K8sNamespaceRegex
  Matches namespace names with a regular expression.
- K8sPodRegex
  Matches pod names with a regular expression.
- K8sContainerRegex
  Matches container names with a regular expression.

All regular expression matching is based on the Go language's RE2 engine, which has certain limitations compared to engines like PCRE. Write your regular expressions according to the guidelines in Appendix: Regular expression limits for container filtering.

# ...under spec.config...
inputs:
  - Type: input_file # or input_container_stdio
    # When the input plugin type is input_file, you must set EnableContainerDiscovery to true.
    EnableContainerDiscovery: true
    # Container filtering
    ContainerFilters:
      # K8s pod label whitelist: Specifies the containers from which to collect logs.
      IncludeK8sLabel:
        # Example: Match all pods that have the app label with a value of nginx or redis.
        app: ^(nginx|redis)$

      # K8s pod label blacklist: Excludes log collection from containers that meet specific conditions.
      ExcludeK8sLabel:
        # Example: Exclude all pods with the app:test label.
        app: test
      
      # Environment variable whitelist.
      IncludeEnv:
        # Match all containers with NGINX_SERVICE_PORT=80 or NGINX_SERVICE_PORT=6379.
        NGINX_SERVICE_PORT: ^(80|6379)$

      # Environment variable blacklist.
      ExcludeEnv:
        # Exclude all containers with ENVIRONMENT=test.
        ENVIRONMENT: test
      
      # Match namespace names. Example: Match all containers in the default and nginx namespaces.
      K8sNamespaceRegex: ^(default|nginx)$
      # Match pod names. Example: Match containers in all pods whose names start with nginx-log-demo.
      K8sPodRegex: ^(nginx-log-demo.*)$
      # Match container names. Example: Match all containers named container-test.
      K8sContainerRegex: ^(container-test)$

Blacklisting

Exclude files that match specified criteria. To do this, add the following parameters under config.inputs in your YAML configuration as needed:

Field details

Example

# ...under spec.config...
inputs:
  - Type: input_file
    # File path blacklist. Excludes files based on the specified conditions. The path must be an absolute path and supports the * wildcard character.
    ExcludeFilePaths:
      - /var/log/*.log

    # File name blacklist. Excludes files based on the specified conditions. Supports the * wildcard character.
    ExcludeFiles:
      - test

    # Directory blacklist. Excludes directories based on the specified conditions. The path must be an absolute path and supports the * wildcard character.
    ExcludeDirs:
      - /var/log/backup*

ExcludeFilePaths

File path blacklist. Excludes files that match the specified absolute paths. Paths can include the * wildcard character.

ExcludeFiles

File name blacklist. Excludes files with matching names. The names can include the * wildcard character.

ExcludeDirs

Directory blacklist. Excludes directories based on the specified paths. Paths must be absolute and can include the * wildcard character.

Log enrichment with tags

Key configuration: In spec.config.inputs, configure ExternalEnvTag and ExternalK8sLabelTag to map container environment variables and pod labels to log tags.

Field details

Example

ExternalEnvTag

Maps the value of a specified environment variable to a log tag. The format is <environment_variable_name>: <tag_name>.

# ...under spec.config...
inputs:
  - Type: input_file # or input_container_stdio
    ExternalEnvTag:
      <environment_variable_name>: <tag_name>
    
    ExternalK8sLabelTag:
      <pod_label_name>: <tag_name>

ExternalK8sLabelTag

Maps the value of a Kubernetes pod label to a log tag. The format is <pod_label_name>: <tag_name>.

Configuration examples

Use case 1: Parse Nginx access logs

This configuration parses an Nginx log and structures the content into multiple key-value pairs based on the log_format definition.

Complete YAML example

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: nginx-config
spec:
  config:
    aggregators: []
    global: {}
    inputs:
      - Type: input_file
        FilePaths:
          - /root/log/text1.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        EnableContainerDiscovery: true
    processors:
      - Type: processor_parse_regex_native
        SourceKey: content
        Regex: >-
          (\S*)\s*-\s*(\S*)\s*\[(\d+/\S+/\d+:\d+:\d+:\d+)\s+\S+\]\s*"(\S+)\s+(\S+)\s+\S+"\s*(\S*)\s*(\S*)\s*(\S*)\s*(\S*)\s*"([^"]*)"\s*"([^"]*)".*
        Keys:
          - remote_addr
          - remote_user
          - time_local
          - request_method
          - request_uri
          - request_time
          - request_length
          - status
          - body_bytes_sent
          - http_referer
          - http_user_agent
        Extra:
          Format: >-
            log_format main  '$remote_addr - $remote_user [$time_local]
            "$request" ''$request_time $request_length ''$status
            $body_bytes_sent "$http_referer" ''"$http_user_agent"';
          LogType: NGINX
    flushers:
      - Type: flusher_sls
        Logstore: my-log-logstore
    sample: >-
      192.168.*.* - - [15/Apr/2025:16:40:00 +0800] "GET /nginx-logo.png
      HTTP/1.1" 0.000 514 200 368 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64)
      AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.*.* Safari/537.36"
  project:
    name: my-log-project
  logstores:
    - name: my-log-logstore

Use case 2: Process multiline logs

By default, SLS treats each line as a separate log entry, splitting multiline logs like stack traces into multiple records.

Enable multiline mode with a start-pattern regex to group all lines of a multiline log into a single entry.

Complete YAML example

apiVersion: telemetry.alibabacloud.com/v1alpha1
kind: ClusterAliyunPipelineConfig
metadata:
  name: multiline-config
spec:
  config:
    aggregators: []
    global: {}
    inputs:
      - Type: input_file
        FilePaths:
          - /root/log/text1.log
        MaxDirSearchDepth: 0
        FileEncoding: utf8
        Multiline:
          StartPattern: '\[\d+-\d+-\w+:\d+:\d+,\d+]\s\[\w+]\s.*'
          Mode: custom
          UnmatchedContentTreatment: single_line
        EnableContainerDiscovery: true
    processors: []
    flushers:
      - Type: flusher_sls
        Logstore: my-log-logstore
    sample: |-
      [2023-10-01T10:30:01,000] [INFO] java.lang.Exception: exception happened
          at TestPrintStackTrace.f(TestPrintStackTrace.java:3)
          at TestPrintStackTrace.g(TestPrintStackTrace.java:7)
          at TestPrintStackTrace.main(TestPrintStackTrace.java:16)
  project:
    name: my-log-project
  logstores:
    - name: my-log-logstore

FAQ

Managing multi-target distribution

Multi-target distribution configurations are associated with multiple Logstores and must be managed on the project-level management page:

Log on to the Simple Log Service console and click the name of the target project.
On the project page, in the left-side navigation pane, click Resources > Configurations.
Note
This page lists all collection configurations in the project, including residual configurations from accidentally deleted Logstores.

Send ACK logs to a cross-account project

Install LoongCollector (Logtail) manually in an ACK cluster and configure it with the target account's credentials to send logs to a project in another Alibaba Cloud account.

Use case: Collect logs from an ACK cluster into a project under a separate account for organizational structure, permission isolation, or unified monitoring.

Procedure: This procedure shows how to manually install LoongCollector. To learn how to install Logtail, see Install and configure Logtail.

Connect to your Kubernetes cluster. Then, run the appropriate command for your region to download LoongCollector and its dependencies:

For regions in the Chinese mainland:

wget https://aliyun-observability-release-cn-shanghai.oss-cn-shanghai.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

For regions outside the Chinese mainland:

wget https://aliyun-observability-release-ap-southeast-1.oss-ap-southeast-1.aliyuncs.com/loongcollector/k8s-custom-pkg/3.0.12/loongcollector-custom-k8s-package.tgz; tar xvf loongcollector-custom-k8s-package.tgz; chmod 744 ./loongcollector-custom-k8s-package/k8s-custom-install.sh

Go to the loongcollector-custom-k8s-package directory and modify the ./loongcollector/values.yaml configuration file.

# ===================== Required parameters =====================
# The Project where logs are sent. Example: k8s-log-custom-sd89ehdq.
projectName: ""
# The Project's region. Example: cn-shanghai.
region: ""
# The UID of the Alibaba Cloud account that owns the Project. Enclose the value in quotation marks. Example: "123456789"
aliUid: ""
# The network type. Valid values: Internet (public network) and Intranet (internal network). Default: Internet.
net: Internet
# The AccessKey ID and secret for an Alibaba Cloud account or RAM user that has the AliyunLogFullAccess policy.
accessKeyID: ""
accessKeySecret: ""
# The custom cluster ID. The ID can contain only uppercase letters, lowercase letters, digits, and hyphens (-).
clusterID: ""

In the loongcollector-custom-k8s-package directory, run the following command to install LoongCollector and its dependencies:
```
bash k8s-custom-install.sh install
```

After the installation is complete, check the status of the components.

If the pods fail to start, verify the values.yaml configuration and ensure the required images were pulled.

# Check pod status
kubectl get po -n kube-system | grep loongcollector-ds

Simple Log Service also automatically creates the following resources. View these resources in the Simple Log Service console.

Resource type	Resource name	Description
Project	The value of `projectName` that you specified in the values.yaml file	Isolates logs from different services.
machine group	`k8s-group-${cluster_id}`	A group of log collection nodes.

Important

LoongCollector does not create a logstore named config-operation-log. If a logstore with this name already exists, LoongCollector stops writing logs to it.

Multiple collections for a single source

By default, SLS restricts each log source to a single collection configuration to prevent duplication:

A text log file can match only one Logtail collection configuration.
A container's standard output (stdout) can match only one standard output collection configuration.

Log on to the Simple Log Service console and go to the target project.
In the left-side navigation pane, select Logstores and find the target logstore.
Click the icon next to its name to expand the logstore.
Click Logtail Configuration. In the configuration list, find the target Logtail configuration and click Manage Logtail Configuration in the Actions column.
On the Logtail Configurations page, click Edit and scroll down to the Input Configurations section:
- To collect text file logs, enable Allow File to Be Collected for Multiple Times.
- To collect container standard output, enable Allow Collection by Different Logtail Configurations.

Appendix: Regex limits (container filtering)

Regular expressions for container filtering use the Go RE2 engine, which has syntax limitations compared to PCRE:

1. Differences in named group syntax

Go uses the (?P<name>...) syntax for named groups. It does not support the (?<name>...) syntax used in PCRE.

Correct example: (?P<year>\d{4})
Incorrect syntax: (?<year>\d{4})

2. Unsupported regular expression features

RE2 does not support the following common but complex regular expression features. Avoid using them:

Assertions: (?=...), (?!...), (?<=...), (?<!...)
Conditional expressions: (?(condition)true|false)
Recursive matching: (?R), (?0)
Subprogram references: (?&name), (?P>name)
Atomic groups: (?>...)

3. Recommendations

When debugging regex with a tool like Regex101, select Golang (RE2) mode. Unsupported syntax prevents the plugin from parsing or matching correctly.