Simple Log Service:Collect standard output from self-managed clusters - old version (DaemonSet)

CRD - AliyunPipelineConfig (recommended)

To create a Logtail configuration, you need to only create a Custom Resource (CR) from the AliyunPipelineConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Modify the parameters in the following YAML example as needed, copy and paste it into the template, then click Create.

   Note

   You can use the Logtail configuration generator to create a target scenario YAML script. This tool helps you quickly complete the configuration and reduces manual operations.

   The example YAML file below captures standard output from Pods labeled with app: ^(.*test.*)$ within the default namespace, using multi-line text mode, and forwards it to a logstore called k8s-stdout, which is automatically created within a project named k8s-log-<YOUR_CLUSTER_ID>. Adjust the parameters in the YAML as needed:

   • project: Log on to the Simple Log Service Console, and identify the project name created by the Logtail you installed, typically in the format k8s-log-<YOUR_CLUSTER_ID>.

   • IncludeK8sLabel: Used to filter the labels of the target pod. For example, app: ^(.*test.*)$ indicates that the label key is app, and it will collect pods with values that include test.

   • Endpoint and Region: For example, ap-southeast-1.log.aliyuncs.com and ap-southeast-1.

   For more information on config in the YAML file, such as supported inputs, outputs, processing plug-in types, and container filtering methods, see PipelineConfig. For a comprehensive list of YAML parameters, see CR parameters.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   # Create a ClusterAliyunPipelineConfig.
   kind: ClusterAliyunPipelineConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. This name is also the name of the Logtail configuration created. 
     name: example-k8s-stdout
   spec:
     # Specify the target project.
     project:
       name: k8s-log-<YOUR_CLUSTER_ID>
     # Create a logstore for storing logs.
     logstores:
       - name: k8s-stdout
     # Define the Logtail configuration.
     config:
       # Sample log (optional)
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Define input plug-ins.
       inputs:
             # Use the service_docker_stdout plug-in to collect text logs inside the container.
         - Type: service_docker_stdout
           Stdout: true
           Stderr: true
           # Configure container information filter conditions. Multiple options are in an "and" relationship.
           # Specify the namespace to which the pod containing the container to be collected belongs. Supports regular expression matching.
           K8sNamespaceRegex: "^(default)$"
           # Enable container metadata preview.
           CollectContainersFlag: true
           # Collect containers that meet the Pod label conditions. Multiple entries are in an "or" relationship.
           IncludeK8sLabel:
             app: ^(.*test.*)$
           # Configure multi-line chunk configuration. Invalid configuration for single-line log collection.
           # Configure the regular expression for the beginning of the line.
           BeginLineRegex: \d+-\d+-\d+.*
       # Define output plug-ins
       flushers:
         # Use the flusher_sls plug-in to send logs to the specified logstore.
         - Type: flusher_sls
           # Make sure that the logstore exists.
           Logstore: k8s-stdout
           # Make sure that the endpoint is valid.
           Endpoint: ap-southeast-1.log.aliyuncs.com
           Region: ap-southeast-1
           TelemetryType: logs

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you created. Logtail starts to collect standard output from containers and sends the collected logs to Simple Log Service.

CRD-AliyunLogConfig

To create a Logtail configuration, you need to only create a CR from the AliyunLogConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   The YAML script creates a Logtail configuration named simple-stdout-example. The Logtail configuration collects standard output in multi-line mode from all containers whose names start with app in the cluster. The collected logs are sent to the k8s-stdout Logstore in the k8s-log-<your_cluster_id> project.

   For more information about the logtailConfig field in the YAML file, including the supported input, output, and processing plug-ins and container filtering methods, see AliyunLogConfigDetail. For more information about the parameters in the YAML file, see CR parameters.

   # Standard output configuration
   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in your Kubernetes cluster.
     name: simple-stdout-example
   spec:
     # Specify the name of the project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # The name of the Logstore. If the Logstore that you specify does not exist, Simple Log Service automatically creates a Logstore.
     logstore: k8s-stdout
     # Configure the Logtail configuration.
     logtailConfig:
       # The type of the data source. If you want to collect stdout logs, you must set the value to plugin.
       inputType: plugin
       # The name of the Logtail configuration. The name must be the same as the resource name that is specified in metadata.name.
       configName: simple-stdout-example
       inputDetail:
         plugin:
           inputs:
             - type: service_docker_stdout
               detail:
                 # The settings that allow Logtail to collect both stdout and stderr logs.
                 Stdout: true
                 Stderr: true
                 # Specify the namespace of the pods to which the required containers belong. Regular expression matching is supported.
                 K8sNamespaceRegex: "^(default)$"
                 # Specify the name of the required containers. Regular expression matching is supported.
                 K8sContainerRegex: "^(app.*)$"
                 # Configure settings for multi-line log collection.
                 # Specify the regular expression that is used to match the beginning of the first line of a log.
                 BeginLineRegex: \d+-\d+-\d+.*
                 

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you created. Logtail starts to collect standard output from containers and sends the collected logs to Simple Log Service.

Simple Log Service console

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you used when you installed the Logtail component, for example, k8s-log-<your_cluster_id>. On the project details page, click the Logtail Configuration tab next to the target Logstore. Add a Logtail configuration and click Kubernetes-Standard Output-Old Version.

3. Because you have installed the Logtail component for the ACK cluster in the previous step, click Use Existing Machine Group.

4. On the Machine Group Configuration page, select the k8s-group-${your_k8s_cluster_id} machine group in the ACK DaemonSet Mode section of the Kubernetes scenario, click > to add the machine group to the Selected Machine Groups section, and then click Next.

5. Create a Logtail configuration. Configure the required parameters and click Next. Approximately 1 minute is required to create a Logtail configuration.

   This section describes only the required parameters. For more information about the parameters, see Logtail configuration.

   • Global Configuration

     In the Global Configuration section, enter a configuration name.

     

6. Create Index and Preview Data: Simple Log Service enables full-text indexing by default. In this case, all fields in logs are indexed for queries. You can also manually create field indexes based on the collected logs, or click Auto-generate Index. Simple Log Service generates field indexes that you can use to perform term queries on specific fields. This reduces indexing fees and improves query efficiency. For more information, see Create indexes.