Container Service for Kubernetes:Collect Logs from Knative Services

CRD - AliyunPipelineConfig (recommended)

Create a Custom Resource (CR) from the AliyunPipelineConfig Custom Resource Definition (CRD). The configuration is applied automatically when the CR is created. To modify the configuration, update the CR.

Key constraints:

• The configName value must be unique within the SLS project.

• Create one CR per Logtail configuration. If multiple CRs reference the same configuration, only the first CR takes effect.

Steps:

1. Get the kubeconfig file and connect to your cluster using kubectl.

2. Create a YAML file:

   vim cube.yaml

3. Add the configuration to the YAML file. See the examples below.

4. Apply the configuration:

   kubectl apply -f cube.yaml

   Logtail starts collecting logs from the specified containers and sending them to SLS.

apiVersion: telemetry.alibabacloud.com/v1alpha1
# Create a CR from the ClusterAliyunPipelineConfig CRD.
kind: ClusterAliyunPipelineConfig
metadata:
  # The name must be unique in the Kubernetes cluster. It is also used as the Logtail configuration name.
  name: example-k8s-file
spec:
  # Specify the project to collect logs into.
  project:
    name: k8s-log-test
  # Create a Logstore to store logs.
  logstores:
    - name: k8s-file
  config:
    inputs:
      # Use input_file to collect text logs from containers.
      - Type: input_file
        # Log file path inside the containers.
        FilePaths:
          - /data/logs/app_1/**/test.LOG
        # Enable container discovery.
        EnableContainerDiscovery: true
        # Filter containers. Multiple conditions use logical AND.
        ContainerFilters:
          # Match containers in the default namespace.
          K8sNamespaceRegex: default
          # Match containers whose name contains "app".
          K8sContainerRegex: ^(.*app.*)$
    flushers:
      # Use flusher_sls to send logs to a Logstore.
      - Type: flusher_sls
        Logstore: k8s-file
        # For valid endpoint and region values, see https://www.alibabacloud.com/help/en/sls/developer-reference/service-entrance
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

apiVersion: telemetry.alibabacloud.com/v1alpha1
# Create a CR from the ClusterAliyunPipelineConfig CRD.
kind: ClusterAliyunPipelineConfig
metadata:
  name: example-k8s-file
spec:
  project:
    name: k8s-log-test
  logstores:
    - name: k8s-file
  config:
    # Optional: add a sample log for reference.
    sample: |
      2024-06-19 16:35:00 INFO test log
      line-1
      line-2
      end
    inputs:
      - Type: input_file
        FilePaths:
          - /data/logs/app_1/**/test.LOG
        EnableContainerDiscovery: true
        # Enable multi-line log collection.
        Multiline:
          # Use a custom regex to identify the first line of each log entry.
          Mode: custom
          StartPattern: \d+-\d+-\d+.*
    processors:
      # Parse logs using a regular expression. Capturing groups define extracted fields.
      - Type: processor_parse_regex_native
        SourceKey: content
        Regex: (\d+-\d+-\d+\s*\d+:\d+:\d+)\s*(\S+)\s*(.*)
        Keys: ["time", "level", "msg"]
    flushers:
      - Type: flusher_sls
        Logstore: k8s-file
        Endpoint: cn-hangzhou.log.aliyuncs.com
        Region: cn-hangzhou
        TelemetryType: logs

CRD - AliyunLogConfig

Create a CR from the AliyunLogConfig CRD. The configuration is applied automatically. To modify it, update the CR.

Steps:

1. Get the kubeconfig file and connect to your cluster using kubectl.

2. Create a YAML file:

   vim cube.yaml

3. Add the configuration. See the example below. The configName value must be unique in the SLS project.

4. Apply the configuration:

   kubectl apply -f cube.yaml

apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  name: example-k8s-file
  namespace: kube-system
spec:
  # If left blank, the project named k8s-log-<your_cluster_id> is used.
  project: k8s-log-test
  # If the specified Logstore does not exist, SLS creates it automatically.
  logstore: k8s-file
  logtailConfig:
    # Set to "file" for text logs.
    inputType: file
    configName: example-k8s-file
    inputDetail:
      # Simple mode for text logs.
      logType: common_reg_log
      logPath: /data/logs/app_1
      # Supports wildcard characters (* and ?). Example: log_*.log
      filePattern: test.LOG
      # Required for container log collection.
      dockerFile: true
      advanced:
        k8s:
          K8sPodRegex: '^(app.*)$'

SLS console

1. Log on to the Simple Log Service console.

2. In the Quick Data Import section, click Import Data. In the Import Data dialog box, click the Kubernetes - File card.

   

3. Select the project and Logstore to use, then click Next. Select the project created when you installed the Logtail components.

4. In the Machine Group Configurations step, configure the following:

   1. Select a machine group option based on your setup:

      • Kubernetes Clusters > ACK Daemonset

      • Kubernetes Clusters > Self-managed Cluster in DaemonSet Mode

      Important

      Subsequent settings vary based on which option you select.

   2. Confirm the required machine groups appear in the Applied Server Groups section, then click Next. After installing Logtail in an ACK cluster, SLS automatically creates a machine group named k8s-group-${your_k8s_cluster_id} that you can use directly.

      Important

      • To create a new machine group, click Create Machine Group and configure the parameters. See Collect container logs from ACK clusters.

      • If a machine group's heartbeat status is FAIL, click Automatic Retry. If the issue persists, see How do I troubleshoot an error related to a Logtail machine group?

5. Create a Logtail configuration and click Next. SLS starts collecting logs after the configuration is created. Container filtering options The following filtering options are available for Logtail 1.0.34 and later. For earlier versions, use only environment variables and container labels.

   • K8s Pod Name Regular Matching: Match containers by pod name using a regular expression. Example: ^(nginx-log-demo.*)$ matches all containers in pods whose name starts with nginx-log-demo.

   • K8s Namespace Regular Matching: Match containers by namespace. Example: ^(default|nginx)$ matches containers in the nginx and default namespaces.

   • K8s Container Name Regular Matching: Match containers by container name (as defined in spec.containers). Example: ^(container-test)$ matches containers named container-test.

   • Container Label Whitelist / Blacklist: Filter containers by container labels (key-value pairs). Kubernetes namespaces and container names map to the labels io.kubernetes.pod.namespace and io.kubernetes.container.name. Key-value pairs are evaluated with OR logic. String matching is the default; prefix with ^ and suffix with $ for regex matching.

   • Environment Variable Whitelist / Blacklist: Filter containers by environment variable key-value pairs. OR logic applies. Example: set Environment Variable Name to NGINX_SERVICE_PORT and Environment Variable Value to ^(80|6379)$ to match containers on port 80 or 6379.

   • Kubernetes Pod Label Whitelist / Blacklist: Filter containers by Kubernetes pod labels (distinct from container labels). OR logic applies. Example: set Label Name to environment and Label Value to ^(dev|pre)$ to match containers with pod labels environment:dev or environment:pre.

   Note A Logtail configuration takes up to 3 minutes to take effect.
   Important

   Container labels (retrieved via docker inspect) differ from Kubernetes labels. See Obtain labels and Obtain environment variables.

   Global configurations

   Parameter	Description
   Configuration name	Enter a unique name for the configuration within the project. The name cannot be changed later.
   Log topic type	Choose how log topics are generated. Options: Machine Group Topic (uses machine group topics to distinguish logs from different groups), File Path Extraction (uses a regex to extract part of the file path as the topic), Custom (specify a custom topic). See Log topics.
   Advanced parameters	Optional. Configure advanced global parameters. See CreateLogtailPipelineConfig.

   Input configurations

   Parameter	Description
   Logtail deployment mode	Select Daemonset for this use case.
   File path type	Select Path in Container in most cases. Select Host Path only if a hostPath volume is mounted and you want to collect logs based on the mapped path on the container host.
   File path	Specify the log file path. Linux paths start with / (example: /apsara/nuwa//app.Log`). Windows paths start with a drive letter (example: `C:\Program Files\Intel\\*.Log). Use * and ? as wildcard characters only. Examples: /apsara/nuwa/**/*.log collects all .log files under /apsara/nuwa recursively. /var/logs/app_*/**/*.log collects .log files in subdirectories matching app_* under /var/logs. /var/log/nginx/**/access* collects files starting with access under /var/log/nginx recursively.
   Maximum directory monitoring depth	Set the maximum number of subdirectory levels to monitor for the ** wildcard in the file path. A value of 0 monitors only the specified directory. Keep this value as low as your needs allow — a large value increases monitoring resource consumption and may cause collection latency.
   Enable container metadata preview	When enabled, shows matched container information and full container metadata after the configuration is created.
   Container filtering	Filter which containers to collect logs from. Filtering behavior depends on Logtail version: if Logtail is earlier than 1.0.34, use environment variables and container labels only. If Logtail is 1.0.34 or later, use Kubernetes-level filtering (K8s Pod Name, Namespace, Container Name, and Pod Label). See container filtering options below.
   Log tag enrichment	Add log tags from environment variables and pod labels.
   File encoding	Select the encoding format of the log files.
   First collection size	Size of data Logtail collects from a file on first collection. Default: 1,024 KB. Range: 0–10,485,760 KB. If the file is smaller than 1,024 KB, Logtail starts from the beginning. If larger, Logtail collects the last 1,024 KB.
   Collection blacklist	Exclude specific directories or files from collection using exact paths or wildcards (* and ?). Maximum 10 entries recommended. Paths cannot end with /. Supports File Path Blacklist, File Blacklist, and Directory Blacklist.
   Allow file to be collected multiple times	By default, only one Logtail configuration can collect logs from a given file. Enable this to allow multiple configurations to collect from the same file.
   Advanced parameters	Configure additional parameters manually. See Create a Logtail pipeline configuration.

   Processor configurations

   Parameter	Description
   Log sample	Add sample logs from real scenarios to help configure processing. Maximum total length: 1,500 characters.
   Multi-line mode	Identify multi-line log entries. Options: Custom (use Regex to Match First Line to identify log boundaries), Multi-line JSON (each JSON object spans multiple lines). Configure Processing Method If Splitting Fails: Discard (discard unmatched logs) or Retain Single Line (keep each line as a separate log entry).
   Processing method	Add processors for data processing. Logtail V2.0 allows combining native and extended processors freely (extended processors must follow native ones). Earlier versions cannot mix native and extended processors.

6. Create indexes and preview data, then click Next. Full-text indexing is enabled by default. To create field indexes automatically, click Automatic Index Generation. For more information, see Create indexes.

   Important

   Use full-text indexes to query all fields. Use field indexes to query specific fields and reduce index traffic. Field indexes are required for log analysis (queries must include a SELECT statement).

7. Click Query Log to go to the query and analysis page of your Logstore. Wait approximately 1 minute for indexes to take effect, then view collected logs on the Raw Logs tab. See Guide to log query and analysis.

Environment variables

Use environment variables to configure log collection directly in your Knative Service YAML. All environment variable names for log collection must use the aliyun_logs_ prefix. The {key} part can contain only lowercase letters, digits, and hyphens (-).

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: helloworld-go-log
spec:
  template:
    spec:
      containers:
      - name: my-demo-app
        image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
        env:
        # Collect stdout to Logstore "log-stdout".
        - name: aliyun_logs_log-stdout
          value: stdout
        # Collect /var/demo/*.log files to Logstore "log-varlog".
        - name: aliyun_logs_log-varlog
          value: /var/demo/*.log
        # Add a custom tag to all logs collected by this container.
        - name: aliyun_logs_mytag1_tags
          value: tag1=v1
        volumeMounts:
        - name: volumn-sls-mydemo
          mountPath: /var/demo
        # Optional: add a sleep command if the pod restarts repeatedly.
        command: ["sh", "-c"]
        args: ["sleep 3600"]
      volumes:
      - name: volumn-sls-mydemo
        emptyDir: {}