Simple Log Service:Collect text logs from a self-managed cluster in DaemonSet mode

CRD - AliyunPipelineConfig (recommended)

To create a Logtail configuration, you need to only create a Custom Resource (CR) from the AliyunPipelineConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   You can use the configuration generator to generate a YAML script for your scenario, or manually write a YAML script based on the following example.

   The following example YAML file collects the content of the test.LOG file in the /data/logs/app_1 directory of pods in the default namespace. The pods have the app: ^(.*test.*)$ label. The logs are collected in multi-line text mode and sent to the k8s-file Logstore (automatically created) in the k8s-log-test project. You need to modify the following parameters in the YAML file based on your business requirements:

   1. project. Example: k8s-log-test.

      Log on to the Simple Log Service console to check the name of the project that is created when you install the Logtail component. The name is in the format of k8s-log-<YOUR_CLUSTER_ID>.

   2. IncludeK8sLabel. Example: app: ^(.*test.*)$. This parameter specifies the label that is used to filter pods. In this example, the system collects logs from pods whose app label value contains test.

      Note

      If you want to collect logs from all pods whose names contain test in the cluster, you can replace IncludeK8sLabel with K8sContainerRegex and use a wildcard character to specify the value. Example: K8sContainerRegex: ^(.test.)$.

   3. FilePaths. Example: /data/logs/app_1/**/test.LOG. For more information, see Container file path mapping.

   4. Endpoint and Region. Example: cn-hangzhou.log.aliyuncs.com and cn-hangzhou.

   For information about the config field in the YAML file, including the supported input, output, and processing plug-ins and container filtering methods, see PipelineConfig. For information about all parameters in the YAML file, see CR parameters.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   kind: ClusterAliyunPipelineConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster. This name is also used as the name of the Logtail configuration. If a Logtail configuration with the same name already exists, the Logtail configuration does not take effect.
     name: example-k8s-file
   spec:
     # Specify the destination project
     project:
       name: k8s-log-test
     logstores:
       # Create a Logstore named k8s-file
       - name: k8s-file
     # Define the Logtail configuration
     config:
       # Specify the sample log. You can leave this parameter empty.
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Configure the input plug-ins
       inputs:
         # Use the input_file plug-in to collect multi-line text logs from containers
         - Type: input_file
           # The log file path in the containers
           FilePaths:
             - /data/logs/app_1/**/test.LOG
           # Enable the container discovery feature.
           EnableContainerDiscovery: true
           # Add conditions to filter containers. Multiple conditions are evaluated by using a logical AND.
           CollectingContainersMeta: true
           ContainerFilters:
             # Specify the namespace of the pods to which the required containers belong. Regular expression matching is supported.
             K8sNamespaceRegex: default
             # Specify the name of the required containers. Regular expression matching is supported.
             IncludeK8sLabel:
               app: ^(.*app.*)$
           # Enable multi-line log collection. Delete this configuration for single-line log collection
           Multiline:
             # Specify the custom mode to match the beginning of the first line of a log based on a regular expression.
             Mode: custom
             # Specify the regular expression that is used to match the beginning of the first line of a log.
             StartPattern: '\d+-\d+-\d+\s\d+:\d+:\d+'
       # Configure the Logtail processing plug-ins
       processors:
         # Use the processor_parse_regex_native plug-in to parse logs based on the specified regular expression.
         - Type: processor_parse_regex_native
           # Specify the name of the input field.
           SourceKey: content
           # Specify the regular expression that is used for the parsing. Use capturing groups to extract fields.
           Regex: (\d+-\d+-\d+\s\S+)(.*)
           # Specify the fields that you want to extract.
           Keys: ["time", "detail"]
       # Configure the output plug-ins
       flushers:
         # Use the flusher_sls plug-in to send logs to a specific Logstore.
         - Type: flusher_sls
           # Make sure that the Logstore exists.
           Logstore: k8s-file
           # Make sure that the endpoint is valid.
           Endpoint: cn-beijing.log.aliyuncs.com
           Region: cn-beijing
           TelemetryType: logs

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container text logs to Simple Log Service.

CRD - AliyunLogConfig

To create a Logtail configuration, you need to only create a CR from the AliyunLogConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   This YAML script creates a Logtail configuration named example-k8s-file. The Logtail configuration collects the content of the test.LOG file in the /data/logs/app_1 directory of all containers whose names start with app in the cluster. The logs are collected in simple text mode and sent to the k8s-file Logstore (automatically created) in the k8s-log-<YOUR_CLUSTER_ID> project.

   You need to modify the file path in the container based on your business requirements. For more information, see Container file path mapping.

   • logPath: the path from which logs are collected. Example: /data/logs/app_1.

   • filePattern: the name of the log file from which logs are collected. Example: test.LOG.

   For information about the logtailConfig field in the YAML file, including the supported input, output, and processing plug-ins and container filtering methods, see AliyunLogConfigDetail. For information about all parameters in the YAML file, see CR parameters.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # The name of the resource. The name must be unique in the current Kubernetes cluster.
     name: example-k8s-file
     # The namespace of the resource.
     namespace: kube-system
   spec:
     # Specify the name of the project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # The name of the Logstore. If the Logstore that you specify does not exist, Simple Log Service automatically creates a Logstore.
     logstore: k8s-file
     # Configure the Logtail configuration.
     logtailConfig:
       # The type of the data source. If you want to collect text logs, set the value to file.
       inputType: file
       # The name of the Logtail configuration. The name must be the same as the value of metadata.name.
       configName: example-k8s-file
       inputDetail:
         # Configure Logtail to collect text logs in simple mode.
         logType: common_reg_log
         # The path of the log file.
         logPath: /data/logs/app_1
         # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log.
         filePattern: test.LOG
         # If you want to collect container text logs, you must set dockerFile to true.
         dockerFile: true
         # Enable multi-line log collection. Delete this configuration for single-line log collection
         # The regular expression that is used to match the beginning of the first line of a log
         logBeginRegex: \d+-\d+-\d+.*
         #The conditions that are used to filter containers.
         advanced:
           k8s:
             K8sPodRegex: '^(app.*)$'

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container text logs to Simple Log Service.

Simple Log Service console

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you use to install Logtail components. Example: k8s-log-<your_cluster_id>. On the page that appears, click the Logstore that you want to manage and then click Logtail Configurations. On the Logtail Configuration page, click Add Logtail Configuration. In the Quick Data Import dialog box, find the Kubernetes - File card and click Integrate Now.

3. In the Machine Group Configurations step of the Import Data wizard, set the Scenario parameter to Kubernetes Clusters and the Deployment Method parameter to ACK Daemonset, select the k8s-group-${your_k8s_cluster_id} machine group and click the > icon to move the machine group from the Source Machine Group section to the Applied Server Groups section, and then click Next.

4. Create a Logtail configuration. In the Logtail Configuration step of the Import Data wizard, configure the required parameters and click Next. Approximately 1 minute is required to create a Logtail configuration.

   The following list describes the main parameter settings. For more information, see Create a Logtail configuration.

   • Global Configurations

     In the Global Configurations section, configure the Configuration Name parameter.

     

   • Input Configurations

     • Logtail Deployment Mode: The Logtail deployment mode. Select DaemonSet.

     • File Path Type: The type of the file path that you want to use to collect logs. Valid values: Path in Container and Host Path. If a hostPath volume is mounted to a container and you want to collect logs from files based on the mapped file path on the container host, set this parameter to Host Path. In other scenarios, set this parameter to Path in Container.

     • File Path: The directory used to store the logs that you want to collect. The file path must start with a forward slash (/). In this example, set the File Path parameter to /data/wwwlogs/main/**/*.Log, which indicates that logs are collected from files suffixed with .Log in the /data/wwwlogs/main directory. You can configure the Maximum Directory Monitoring Depth parameter to specify the maximum number of levels of the subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of the subdirectories that the ** wildcard characters can match in the value of the File Path parameter. The value 0 specifies that only the specified log file directory is monitored.

5. Create indexes and preview data. By default, full-text indexing is enabled for Simple Log Service. In this case, full-text indexes are created. You can query all fields in logs based on the indexes. You can also manually create indexes for fields based on the collected logs. Alternatively, you can click Automatic Index Generation. Then, Simple Log Service generates indexes for fields. You can query data in an accurate manner based on field indexes. This reduces indexing costs and improves query efficiency. For more information, see Create indexes.