Simple Log Service:Collect standard output from self-managed cluster - new version (DaemonSet)

CRD - AliyunPipelineConfig (recommended)

To create a Logtail configuration, you need to only create a Custom Resource (CR) from the AliyunPipelineConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to the ACK console.

2. In the left navigation bar, select Clusters.

3. On the Clusters page, click More in the Actions column of the cluster that you want to manage, and then click Manage Cluster.

4. Create a file named example-k8s-stdout.yaml.

   Note

   You can use the configuration generator to generate a YAML script for your scenario. This tool helps you quickly complete the configuration and reduces manual operations.

   The following example YAML file collects standard output in multiline mode from pods that have the app: ^(.*test.*)$ label in the default namespace, and sends the collected logs to the k8s-stdout Logstore (automatically created) in the k8s-log-test project. You need to modify the following parameters in the YAML file based on your business requirements:

   1. project, for example, k8s-log-test.

      Log on to the Simple Log Service console and check the name of the project that is generated by the installed Logtail. The name is typically in the k8s-log-<YOUR_CLUSTER_ID> format.

   2. IncludeK8sLabel, for example, app: ^(.*test.*)$. This parameter is used to filter target pods. In this example, pods that have the app label whose value contains test are collected.

   For more information about the config item in the YAML file, including the supported input, output, and processing plug-ins and container filtering methods, see Kubernetes standard output (new version). For more information about the complete YAML parameters, see CR parameters.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   # Create a CR from the ClusterAliyunPipelineConfig CRD.
   kind: ClusterAliyunPipelineConfig
   metadata:
     # The name of the resource. The name must be unique in your Kubernetes cluster. This name is also the name of the created Logtail configuration. If the name is duplicated, the Logtail configuration does not take effect.
     name: example-k8s-stdout
   spec:
     # Define the Logtail configuration
     config:
       aggregators: [ ]
       global: {}
       # Configure the Logtail input plug-ins.
       inputs:
         # Use the service_docker_stdout plug-in to collect text logs from containers
         - Type: input_container_stdio
           # Collect stdout.
           IgnoringStderr: false
           # Collect stderr
           IgnoringStdout: false
           # Enable container metadata preview
           CollectingContainersMeta: true
           # Container filtering
           ContainerFilters:
             IncludeK8sLabel:
               app: ^(.*test*.)$
           # Configure multiline collection
           Multiline:
             # Specify the regular expression that is used to match the beginning of the first line of a log.
             StartPattern: \d+-\d+-\d+.*
             Mode: custom
             UnmatchedContentTreatment: single_line
           # Use processing plug-ins to parse logs
       processors:
           # Parse logs in regular expression mode
         - Type: processor_parse_regex_native
           SourceKey: content
           # Regular expression
           Regex: (\d+-\d+-\d+\s\S+)(.*)
           # Field index
           Keys:
             - time
             - detail
       # Configure the Logtail output plug-ins.
       flushers:
         - Type: flusher_sls
           Logstore: k8s-stdout
       sample: |-
         2025-04-02 16:00:03
         1
         2
         3
         4
         5
         6
         7
         8
         9
     project:
       name: k8s-log-test
     logstores:
       - name: k8s-stdout

5. Run kubectl apply -f example-k8s-stdout.yaml, where example-k8s-stdout.yaml is the name of the YAML file that you created. Logtail starts to collect standard output from containers and sends the collected logs to Simple Log Service.

Simple Log Service console

1. Log on to the Simple Log Service console.

2. In the Projects section, select the project that you used when you installed the Logtail component, such as k8s-log-<your_cluster_id>. On the Project page, click the Logtail configuration of the target Logstore, add a Logtail configuration, and then click Access Now in the Kubernetes-Standard Output-New Version section.

   

3. Because you have installed the Logtail component for the ACK cluster in the previous step, click Use Existing Machine Group.

4. On the Machine Group Configuration page, select the k8s-group-${your_k8s_cluster_id} machine group in the ACK DaemonSet Mode section of the K8s Scenario section, click > to add the machine group to the Selected Machine Groups section, and then click Next.

5. Create a Logtail configuration. Configure the required parameters as described in the following section and click Next. Approximately 1 minute is required to create a Logtail configuration.

   This section describes only the required parameters. For more information about the parameters, see Kubernetes standard output (new version).

   In the Global Configuration section, enter a configuration name.

   

6. Create Index and Preview Data: Simple Log Service enables full-text indexing by default. In this case, all fields in logs are indexed for queries. You can also manually create field indexes based on the collected logs, or click Auto-Generate Index. Simple Log Service generates field indexes. You can use field indexes to perform term queries on specific fields. This reduces indexing fees and improves query efficiency. For more information, see Create indexes.