Container Service for Kubernetes:Collect text logs from a self-managed cluster in DaemonSet mode

CRD - AliyunPipelineConfig (recommended)

To create a Logtail configuration, you need to only create a Custom Resource (CR) from the AliyunPipelineConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   You can use the Logtail configuration generator to generate a YAML script used to create a Logtail configuration for your scenario, or manually write a YAML script based on the following example.

   The following code provides an example of a YAML file used to collect text logs from the test.LOG file in the /data/logs/app_1 directory of pods labeled with app: ^(.*test.*)$ in the default namespace in multi-line mode to the automatically created k8s-file logstore in the k8s-log-test project. Modify the following parameters in the YAML file as needed:

   1. project: Example: k8s-log-test.

      Log on to the Simple Log Service console. Check the name of the project generated after Logtail is installed. In most cases, the project name is in the k8s-log-<YOUR_CLUSTER_ID> format.

   2. IncludeK8sLabel: The label used to filter pods. Example: app: ^(.*test.*)$. In this example, logs in the pods whose label key is app and label value contains test are collected.

      Note

      To collect logs in pods whose names contain test in your cluster, replace the IncludeK8sLabel parameter with the K8sContainerRegex parameter and use wildcard characters to specify a value for the K8sContainerRegex parameter. Example: K8sContainerRegex: ^(.test.)$.

   3. FilePaths: Example: /data/logs/app_1/**/test.LOG. For more information, see File path mapping for containers.

   4. Endpoint and Region: Example for the Endpoint parameter: ap-southeast-1.log.aliyuncs.com. Example for the Region parameter: ap-southeast-1.

   The value of the config parameter includes the types of input, output, and processing plug-ins and container filtering methods. For more information, see PipelineConfig. For more information about the complete parameters in the YAML file, see CR parameters.

   apiVersion: telemetry.alibabacloud.com/v1alpha1
   kind: ClusterAliyunPipelineConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster. The name is the same as the name of the created Logtail configuration. If the resource name already exists, the name does not take effect.
     name: example-k8s-file
   spec:
     # Specify the name of the project.
     project:
       name: k8s-log-test
     logstores:
       # Create a logstore named k8s-file.
       - name: k8s-file
     # Create a Logtail configuration.
     config:
       # Enter a sample log. You can leave this parameter empty.
       sample: |
         2024-06-19 16:35:00 INFO test log
         line-1
         line-2
         end
       # Specify the input plug-in.
       inputs:
         # Use the input_file plug-in to collect multi-line text logs from containers.
         - Type: input_file
           # Specify the file path in the containers.
           FilePaths:
             - /data/logs/app_1/**/test.LOG
           # Enable the container discovery feature.
           EnableContainerDiscovery: true
           # Add conditions to filter containers. Multiple conditions are evaluated by using a logical AND.
           CollectingContainersMeta: true
           ContainerFilters:
             # Specify the namespace of the pods to which the required containers belong. Regular expression matching is supported.
             K8sNamespaceRegex: default
             # Specify the name of the required containers. Regular expression matching is supported.
             IncludeK8sLabel:
               app: ^(.*app.*)$
           # Enable multi-line log collection. If you want to collect single-line logs, delete this parameter.
           Multiline:
             # Specify the custom mode to match the beginning of the first line of a log based on a regular expression.
             Mode: custom
             # Specify the regular expression that is used to match the beginning of the first line of a log.
             StartPattern: '\d+-\d+-\d+\s\d+:\d+:\d+'
       # Specify the processing plug-in.
       processors:
         # Use the processor_parse_regex_native plug-in to parse logs based on the specified regular expression.
         - Type: processor_parse_regex_native
           # Specify the name of the original field.
           SourceKey: content
           # Specify the regular expression that is used for parsing. Use capturing groups to extract fields.
           Regex: (\d+-\d+-\d+\s\S+)(.*)
           # Specify the fields that you want to extract.
           Keys: ["time", "detail"]
       # Specify the output plug-in.
       flushers:
         # Use the flusher_sls plug-in to deliver logs to a specific logstore.
         - Type: flusher_sls
           # Make sure that the logstore exists.
           Logstore: k8s-file
           # Make sure that the endpoint is valid.
           Endpoint: ap-southeast-1.log.aliyuncs.com
           Region: ap-southeast-1
           TelemetryType: logs

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container text logs to Simple Log Service.

CRD - AliyunLogConfig

To create a Logtail configuration, you need to only create a CR from the AliyunLogConfig CRD. After the CR is created, the Logtail configuration takes effect.

1. Log on to your Kubernetes cluster.

2. Create a file named example-k8s-file.yaml.

   The following code provides an example of a YAML file used to create a Logtail configuration named example-k8s-file. You can use the Logtail configuration to collect text logs from the test.LOG file in the /data/logs/app_1 directory of the containers whose names start with app in your cluster in simple mode to the automatically created k8s-file logstore in the k8s-log-<YOUR_CLUSTER_ID> project.

   You can modify the log file path in the example based on your business requirements. For more information, see File path mapping for containers.

   • logPath: The log file path. Example: /data/logs/app_1.

   • filePattern: The name of the file from which you want to collect logs. Example: test.LOG.

   The logtailConfig parameter specifies the Logtail details, which include the types of input, output, and processing plug-ins and container filtering methods. For more information, see AliyunLogConfigDetail. For more information about the complete parameters in the YAML file, see CR parameters.

   apiVersion: log.alibabacloud.com/v1alpha1
   kind: AliyunLogConfig
   metadata:
     # Specify the name of the resource. The name must be unique in the current Kubernetes cluster.
     name: example-k8s-file
     # Specify the namespace to which the resource belongs.
     namespace: kube-system
   spec:
     # Specify the name of the project. If you leave this parameter empty, the project named k8s-log-<your_cluster_id> is used.
     # project: k8s-log-test
     # Specify the name of the logstore. If the specified logstore does not exist, Simple Log Service automatically creates a logstore.
     logstore: k8s-file
     # Create a Logtail configuration.
     logtailConfig:
       # Specify the type of the data source. If you want to collect text logs, set the value to file.
       inputType: file
       # Specify the name of the Logtail configuration. The name must be the same as the resource name that is specified by metadata.name.
       configName: example-k8s-file
       inputDetail:
         # Specify the settings that allow Logtail to collect text logs in simple mode.
         logType: common_reg_log
         # Specify the log file path.
         logPath: /data/logs/app_1
         # Specify the log file name. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the log file name. Example: log_*.log.
         filePattern: test.LOG
         # If you want to collect text logs from containers, set the value to true.
         dockerFile: true
         # Enable multi-line log collection. If you want to collect single-line logs, delete this parameter.
         # Specify the regular expression to match the beginning of the first line of a log.
         logBeginRegex: \d+-\d+-\d+.*
         # Specify the conditions to filter containers.
         advanced:
           k8s:
             K8sPodRegex: '^(app.*)$'

3. Run the kubectl apply -f example.yaml command. Replace example.yaml with the name of the YAML file that you create. Then, Logtail starts to collect container text logs to Simple Log Service.

Simple Log Service console

1. Log on to the Simple Log Service console.

2. In the Projects section, click the project that you use to install Logtail components. Example: k8s-log-<your_cluster_id>. On the page that appears, click the Logstore that you want to manage and then click Logtail Configurations. On the Logtail Configuration page, click Add Logtail Configuration. In the Quick Data Import dialog box, find the Kubernetes - File card and click Integrate Now.

3. In the Machine Group Configurations step of the Import Data wizard, set the Scenario parameter to Kubernetes Clusters and the Deployment Method parameter to ACK Daemonset, select the k8s-group-${your_k8s_cluster_id} machine group and click the > icon to move the machine group from the Source Machine Group section to the Applied Server Groups section, and then click Next.

4. Create a Logtail configuration. In the Logtail Configuration step of the Import Data wizard, configure the required parameters and click Next. Approximately 1 minute is required to create a Logtail configuration.

   The following list describes the main parameter settings. For more information, see Create a Logtail configuration.

   • Global Configurations

     In the Global Configurations section, configure the Configuration Name parameter.

     

   • Input Configurations

     • Logtail Deployment Mode: The Logtail deployment mode. Select DaemonSet.

     • File Path Type: The type of the file path that you want to use to collect logs. Valid values: Path in Container and Host Path. If a hostPath volume is mounted to a container and you want to collect logs from files based on the mapped file path on the container host, set this parameter to Host Path. In other scenarios, set this parameter to Path in Container.

     • File Path: The directory used to store the logs that you want to collect. The file path must start with a forward slash (/). In this example, set the File Path parameter to /data/wwwlogs/main/**/*.Log, which indicates that logs are collected from files suffixed with .Log in the /data/wwwlogs/main directory. You can configure the Maximum Directory Monitoring Depth parameter to specify the maximum number of levels of the subdirectories that you want to monitor. The subdirectories are in the log file directory that you specify. This parameter specifies the levels of the subdirectories that the ** wildcard characters can match in the value of the File Path parameter. The value 0 specifies that only the specified log file directory is monitored.

5. Create indexes and preview data. By default, full-text indexing is enabled for Simple Log Service. In this case, full-text indexes are created. You can query all fields in logs based on the indexes. You can also manually create indexes for fields based on the collected logs. Alternatively, you can click Automatic Index Generation. Then, Simple Log Service generates indexes for fields. You can query data in an accurate manner based on field indexes. This reduces indexing costs and improves query efficiency. For more information, see Create indexes.