Container Service for Kubernetes:Collect logs using a sidecar

Prerequisites

• An ACK Serverless cluster has been created. For more information, see Create a cluster.

• Simple Log Service has been activated.

  When you log on to the Simple Log Service console, if the service is not activated, follow the on-screen instructions to activate it.

Background information

Alibaba Cloud SLS supports collecting ECI logs using a sidecar. This means that in each ECI instance, a sidecar container runs as a log agent alongside the application container to collect logs generated by the application container.

The collected logs can be divided into two types:

• Standard output

  Collection of standard output relies on the ECI `stdlog` volume. When you create a pod, mount this volume to the sidecar container. This allows the sidecar to access the standard output logs as files. These logs are collected by ECI base components.

• Text files

  Text files can be collected using a shared volume within the pod. The same volume can be mounted to multiple containers in the pod. This allows the sidecar to directly collect text files that the application container writes to the volume.

Step 1: Deploy the sidecar container

1. Create a deployment that includes a sidecar container.

   The following is a sample YAML file. Replace the placeholder variables with your actual values.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: nginx-log-sidecar-demo
     name: nginx-log-sidecar-demo
   spec:
     replicas: 2
     selector:
       matchLabels:
         app: nginx-log-sidecar-demo
     template:
       metadata:
         labels:
           app: nginx-log-sidecar-demo
       spec:
         containers:
           - name: nginx-log-demo
             image: registry-vpc.${RegionId}.aliyuncs.com/log-service/docker-log-test:latest
             command:
               - /bin/mock_log
             args:
               - '--log-type=nginx'
               - '--stdout=false'
               - '--stderr=true'
               - '--path=/var/log/nginx/access.log'
               - '--total-count=100000000'
               - '--logs-per-sec=100'
             imagePullPolicy: Always
             volumeMounts:
               - mountPath: /var/log/nginx
                 name: nginx-log
           - name: logtail
             image: registry-vpc.${RegionId}.aliyuncs.com/log-service/logtail:latest
             env:
               - name: ALIYUN_LOGTAIL_USER_ID
                 value: "${Aliuid}"
               - name: ALIYUN_LOGTAIL_USER_DEFINED_ID
                 value: nginx-log-sidecar
               - name: ALIYUN_LOGTAIL_CONFIG
                 value: /etc/ilogtail/conf/${RegionId}/ilogtail_config.json
               - name: aliyun_logs_machinegroup
                 value: k8s-group-app-alpine
             imagePullPolicy: Always
             volumeMounts:
               - mountPath: /var/log/nginx
                 name: nginx-log
               - mountPath: /stdlog
                 name: stdlog
         volumes:
           - emptyDir: {}        # Store text file logs in an emptyDir volume
             name: nginx-log
           - name: stdlog        # Store standard output logs in a stdlog volume
             flexVolume:
               driver: alicloud/pod-stdlog

2. Retrieve the pod information.

   kubectl get pods -l app=nginx-log-sidecar-demo

   The expected output is as follows:

   NAME                                      READY   STATUS    RESTARTS   AGE
   nginx-log-sidecar-demo-84587d9796-krn5z   2/2     Running   0          32m
   nginx-log-sidecar-demo-84587d9796-vhnld   2/2     Running   0          32m

3. View the logs.

   • View logs using kubectl commands

     

   • View logs in the Elastic Container Instance console

     

Step 2: Configure Logtail to collect logs

After you deploy the sidecar container, create a Logtail configuration in the Simple Log Service console to collect logs.

1. Log on to the Simple Log Service console.

2. On the right side of the console, click the Quick Data Import card. On the Import Data dialog box, click Regular Expression - Text Logs.

3. Select a Project and a Logstore, and then click Next.

   If you do not have a Project or Logstore, click Create Now to create them.

   Note

   The system automatically creates a Project named k8s-log-{K8s-Cluster-ID} for each Kubernetes cluster.

4. Configure the machine group and click Next.

   • Scenario: Select Kubernetes Clusters.

   • Deployment Method: Select Sidecar.

   • Select Machine Group: In the Source Machine Group area, select the target machine group and move it to the Applied Server Groups area. If a machine group is not available, click Create Machine Group.

5. Configure Logtail and click Next.

   Logtail supports collecting text logs in modes such as simple single-line, regex, separator, and JSON. For more information, see Collect logs from hosts.

   The following are configuration examples:

   • Standard output example

     When you collect standard output, the log path is the mount path of the stdlog volume in the container. The following table describes the sample parameter settings.

     Configuration section	Parameter	Example
     Global Configurations	Configuration Name	stdout
     Input Configurations	Logtail Deployment Mode	Text Log Collection
     	File Path	/stdlog/**/*.log
     Processor Configurations	Processing Method	None Note To add a processing plug-in, see Process data during collection (processing plug-ins).

   • Text file example

     When you collect text files, the log path is the mount path of the shared volume in the container. The following table describes the sample parameter settings.

     Configuration section	Parameter	Example
     Global Configurations	Configuration Name	file
     Input Configurations	Logtail Deployment Mode	Text Log Collection
     	File Path	/var/log/nginx/**/*.log
     Processor Configurations	Processing Method	None Note To add a processing plug-in, see Process data during collection (processing plug-ins).

6. Configure query and analysis settings.

   By default, indexes are configured. You can also reconfigure the indexes as needed. For more information, see Configure indexes.

7. View the collected ECI logs.

   After you complete the preceding steps, SLS starts to collect ECI logs. The following figure shows an example of standard output logs in an SLS Logstore.