Container Service for Kubernetes:Use a Simple Log Service CRD to collect application logs

Precautions

The log collection feature that is enabled by using Simple Log Service CRDs is valid only for the Elastic Container Instance pods that are created after the CRD is created. If you want to collect logs of existing pods, you must perform a rolling release for the existing pods.

Create an AliyunLogConfig CRD

After you deploy the alibaba-log-controller component in a cluster, you can define an AliyunLogConfig CRD (CRD for log collection configurations) to create a Logtail configuration.

1. Log on to the Container Service for Kubernetes (ACK) console.

2. On the Clusters page, find the cluster that you want to manage and click the cluster name. The details page of the cluster appears.

3. Install alibaba-log-controller in the cluster.

   1. In the left-side navigation pane of the details page, choose Operations > Add-ons.

   2. Click the Logs and Monitoring tab. Find alibaba-log-controller and click Install.

   3. In the message that appears, click OK.

      After alibaba-log-controller is installed, the characters "Installed" are displayed in the upper-right corner of the card of alibaba-log-controller.

4. Create an AliyunLogConfig CRD.

   After you deploy the alibaba-log-controller component in a cluster, you can define an AliyunLogConfig CRD to create a Logtail configuration.

   Note

   After an AliyunLogConfig CRD is defined, you can view the generated Logstore and logtail in the Simple Log Service console. You can modify AliyunLogConfig CRD to update log collection configuration. The system automatically synchronizes the updated configuration to Simple Log Service.

   1. Connect to the cluster.

   2. Write a YAML file for an AliyunLogConfig CRD by referring to the following examples.

      The logs that are collected to Simple Log Service can be categorized into stdouts (including stderrs) logs and text logs. For more information about relevant parameters, see Logtail configurations.

      Example: YAML file for a CRD that is used to collect stdout logs.

      Create a file named log-stdout.yaml and copy the following template into the file.

      apiVersion: log.alibabacloud.com/v1alpha1      
      kind: AliyunLogConfig                         
      metadata:
        name: test-stdout               # The name of the CRD, which is unique in the current Kubernetes cluster. 
      spec:
        project: k8s-log-c326bc86****   # Optional. The name of the project. You can specify a name for the project. We recommend that you use the [k8s-log- <ID of the cluster>] format.
        logstore: test-stdout                 # Required. The name of the Logstore. If the specified Logstore does not exist, the system automatically creates a Logstore.                  
        shardCount: 2     # Optional. The number of shards. Default value: 2. Valid values: 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10.                         
        lifeCycle: 90                   # Optional. The period for which logs in the Logstore can be retained. Unit: days. The parameter is valid only when you create a Logstore. Default value: 90. Valid values: 1 to 3650. The value of 3650 indicates that logs can be permanently reserved.
        logtailConfig:                  # The Logtail configuration.
          inputType: plugin             # The type of the data source. The value of file indicates text logs, and the value of plugin indicates stdout logs.                                
          configName: test-stdout       # The name of the Logtail configuration. The name must be the same as the CRD name specified in the metadata.name parameter. 
          inputDetail:                  # The details of the Logtail configuration.
            plugin:
              inputs:
                - type: service_docker_stdout
                  detail:
                    Stdout: true
                    Stderr: true
      #              IncludeEnv:
      #                aliyun_logs_test-stdout: "stdout"

      Example: YAML file for a CRD that is used to collect text logs.

      Create a file named log-file.yaml and copy the following template into the file.

      apiVersion: log.alibabacloud.com/v1alpha1
      kind: AliyunLogConfig
      metadata:
        name: test-file               # The name of the CRD, which must be unique in the current Kubernetes cluster. 
      spec:
        project: k8s-log-c326bc86****    # Optional. The name of the project. You can specify a name for the project. We recommend that you use the [k8s-log- <ID of the cluster>] format.
        logstore: test-file                 # Required. The name of the Logstore. If the specified Logstore does not exist, the system automatically creates a Logstore.   
        logtailConfig:                   # The Logtail configuration.
          inputType: file             # The type of the data source. The value of file indicates text logs, and the value of plugin indicates stdout logs.
          configName: test-file       # The name of the Logtail configuration. The name must be the same as the CRD name specified in the metadata.name parameter. 
          inputDetail:                  # The details of the Logtail configuration.
            logType: common_reg_log        # The type of the logs that you want to collect. For logs that are parsed by using delimiters, you can set the logType parameter to json_log.
            logPath: /log/    # The folder in which logs are stored.
            filePattern: "*.log"           # The name of the log file. The name supports wildcards. Example: log_*.log.
            dockerFile: true             # To collect text logs in the container, set the dockerFile parameter to true.
            # The key that is used to parse time.
            #timeKey: 'time'
            # The format of time parsing.
            #timeFormat: '%Y-%m-%dT%H:%M:%S'
            # Prevents conflicts caused by the same collection directory that is specified in different collection configurations.
            #dockerIncludeEnv:
            #  aliyun_logs_test-file: "/log/*.log"

   3. Run the following commands to create an AliyunLogConfig CRD:

      kubectl create -f log-stdout.yaml
      kubectl create -f log-file.yaml

Create an application to test log collection

After you create an AliyunLogConfig CRD, Simple Log Service automatically collects the logs of the pods that are created later.

1. Connect to the cluster.

2. Create an application.

   The following sample YAML file shows how to create a Deployment. In the example, the system runs relevant commands after the container is started. stdout logs and text logs of the container are continuously displayed.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: eci-sls-demo
     labels:
       app: sls
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: sls
     template:
       metadata:
         name: sls-test
         labels:
           app: sls
           alibabacloud.com/eci: "true" 
       spec:
         containers:
         - args:
           - -c
           - mkdir -p /log;while true; do echo hello world; date; echo hello sls >> /log/busy.log; sleep 1;done
           command:
           - /bin/sh
           image: registry-vpc.cn-beijing.aliyuncs.com/eci_open/busybox:1.30
           imagePullPolicy: Always
           name: busybox

   Create a file named test-sls-crd.yaml and copy the preceding YAML file template into the file. Run the following commands to create an application:

   kubectl create -f test-sls-crd.yaml

3. Check the status of the application.

   kubetcl get pod

   Expected output:

   NAME                            READY   STATUS    RESTARTS   AGE
   eci-sls-demo-7bf8849b9f-cgpbn   1/1     Running   0          2m14s

4. View logs.

   1. Log on to the Simple Simple Log Service console.

   2. Click the name of the project.

   3. Find the Logstore in which the logs of your containers are stored. Click the name of the Logstore to view the logs.

      • Collection of stdout logs

        

      • Collection of text logs

        

Disable log collection

After you create an AliyunLogConfig CRD, the system automatically collects logs of all pods that meet the conditions. If you do not want to collect logs of some pods, you can add the k8s.aliyun.com/eci-sls-enable: "false" annotation to the metadata section of the pods to disable log collection. This prevents waste of resources caused by the system automatically creating Logtail.

Sample configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: eci-sls-demo2
  labels:
    app: sls
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sls
  template:
    metadata:
      name: sls-test
      labels:
        app: sls
        alibabacloud.com/eci: "true"
      annotations:
        k8s.aliyun.com/eci-sls-enable: "false" # Disables log collection.
    spec:
      containers:
      - args:
        - -c
        - mkdir -p /log;while true; do echo hello world; date; echo hello sls >> /log/busy.log; sleep 1;
          done
        command:
        - /bin/sh
        image: registry.cn-shanghai.aliyuncs.com/eci_open/busybox:1.30
        imagePullPolicy: Always
        name: busybox