This topic describes how to install Sidecar. This topic also describes how to use a custom resource definition (CRD) to create a Logtail configuration that is used to collect container text logs in Sidecar mode.

Prerequisites

The Logtail component is installed. For more information, see Install the Logtail component.

Background information

In Sidecar mode, the Logtail container shares a log directory with an application container. The application container writes logs to the shared directory. Logtail monitors changes to log files in the shared directory and collects logs. For more information, see Sidecar container with a logging agent and How Pods manage multiple containers.

Step 1: Install Sidecar

  1. Log on to your Kubernetes cluster.
  2. Create a YAML file.

    In this example, sidecar.yaml is used as the file name. You can specify a file name based on your business requirements. 

    vim sidecar.yaml
  3. Enter the following script in the YAML file and configure the parameters based on your business requirements.
    Notice Make sure that the time zone below env in the configuration file is correctly set. If the time zones are inconsistent between raw logs and processed logs in a Log Service project, the time recorded for the collected logs may be a point in time in the past or in the future. For example, if the Log Service project resides in greater China, you can set the time zone to Asia/Shanghai. 
    apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
    1. Configure the following basic variables in the configuration script. 
      ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "${your_aliyun_user_id}"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "${your_machine_group_user_defined_id}"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/${your_region_config}/ilogtail_config.json"
      Variable Description
      ${your_aliyun_user_id} Enter the ID of your Alibaba Cloud account. For more information, see Step 1: Obtain the ID of the Alibaba Cloud account for which Log Service is activated.
      ${your_machine_group_user_defined_id} Enter the custom identifier of your machine group. The identifier must be unique in the region where your project resides. Example: nginx-log-sidecar. For more information, see Create a custom ID-based machine group.
      ${your_region_config} Specify a value based on the ID of the region where your project resides and the type of the network for your project. For more information about regions, see Table 1.
      • If logs are collected to your project over the Internet, specify the value in the region-internet format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou-internet.
      • If logs are collected to your project over an internal network of Alibaba Cloud, specify the value in the region format. For example, if your project resides in the China (Hangzhou) region, specify cn-hangzhou.
    2. Specify the mount path in the configuration script.
      Note We recommend that you mount a volume of the emptyDir type. 
              volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
      Parameter Description
      name The name of the volume. You can specify a name based on your business requirements.
      Notice The value of the name parameter in the volumeMounts node and the value of the name parameter in the volumes node must be the same. This ensures that the same volume is mounted for both the Logtail container and the application container.
      mountPath The mount path. You can enter the path of files in which container text logs are recorded.
    3. Specify a waiting period for the Logtail container in the configuration script.
      In most cases, the waiting period is 10 seconds. This value specifies that the Logtail container exits 10 seconds after the container receives a stop command. This setting helps prevent incomplete data collection. 
      command:        
- sh        
- -c        
- /usr/local/ilogtail/run_logtail.sh 10
  4. Run the following command to apply the configurations in the sidecar.yaml file.

    In this example, sidecar.yaml is used as the file name. You can specify a file name based on your business requirements. 

    kubectl apply -f sidecar.yaml

Step 2: Create a Logtail configuration

To create a Logtail configuration, you need only to configure an AliyunLogConfig CRD. After you create a Logtail configuration, Logtail automatically collects logs based on the Logtail configuration. If you want to delete the Logtail configuration, you need only to delete the CRD.

  1. Log on to your Kubernetes cluster.
  2. Run the following command to create a YAML file.

    In this example, cube.yaml is used as the file name. You can specify a file name based on your business requirements. 

    vim cube.yaml
  3. Enter the following script in the YAML file and configure the parameters based on your business requirements.
    Notice
    • The value of the configName parameter must be unique in the Log Service project that you use.
    • If multiple CRDs are associated with the same Logtail configuration, and you delete or modify one of the CRDs, the Logtail configuration is affected. The status of the other CRDs that are associated with the Logtail configuration becomes inconsistent with the status of the Logtail configuration in Log Service.
    • In Sidecar mode, you can collect only text logs. Therefore, you must set the dockerFile parameter to false. 
    apiVersion: log.alibabacloud.com/v1alpha1      # The default value is used. You do not need to modify this parameter. 
kind: AliyunLogConfig                          # The default value is used. You do not need to modify this parameter. 
metadata:
  name: simple-stdout-example                  # The name of the resource. The name must be unique in the current Kubernetes cluster. 
spec:
  project: k8s-my-project                      # Optional. The name of the project. The default value is the name of the project that you use to install Logtail components. 
  logstore: k8s-stdout                         # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  machineGroups:                               # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. This machine group is used to associate Sidecar with the CRD. 
  - nginx-log-sidecar
  shardCount: 2                                # Optional. The number of shards. Valid values: 1 to 10. Default value: 2. 
  lifeCycle: 90                                # Optional. The period during which log data is stored in the Logstore. Valid values: 1 to 3650. Default value: 90. A value of 3650 specifies that data is permanently stored in the Logstore. 
  logtailConfig:                               # The Logtail configuration. 
    inputType: file                            # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
    configName: simple-stdout-example          # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. 
    inputDetail:                               # The detailed settings of the Logtail configuration. For more information, see the following example. 
    ...
    Parameter Data type Required Description
    project string No The name of the project. The default value is the name of the project that you use to install Logtail components.
    logstore string Yes The name of the Logstore.

    If the Logstore that you specify does not exist, Log Service automatically creates a Logstore.

    shardCount int No The number of shards. Valid values: 1 to 10. Default value: 2.
    lifeCycle int No The period during which log data is stored in the Logstore. Valid values: 1 to 3650. Default value: 90. A value of 3650 specifies that data is permanently stored in the Logstore.
    machineGroups array Yes The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. Example: nginx-log-sidecar.
    Log Service creates a machine group based on the name that you specify to associate Sidecar with the CRD.
    Notice

    You must specify a custom identifier for the machine group in the following format: 

      machineGroups:      
  - nginx-log-sidecar
    logtailConfig object Yes The detailed settings of the Logtail configuration. In most cases, you need only to configure the inputType, configName, and inputDetail parameters. For more information about the parameters, see Logtail configurations.

    For more information about configuration examples, see Configuration example for a single directory and Configuration example for different directories.

  4. Run the following command to apply the Logtail configuration.

    In this example, cube.yaml is used as the file name. You can specify a file name based on your business requirements. 

    kubectl apply -f cube.yaml
    After you create the Logtail configuration, you can view the Logtail configuration in the Log Service console or by using a CRD. For more information, see View Logtail configurations.

Configuration example for a single directory

The following procedure shows how to use a CRD to collect text logs from the nginx-log-demo container in Sidecar mode. The container belongs to a self-managed Kubernetes cluster in a data center. The text logs include NGINX access logs and NGINX error logs and are stored in a single directory. The following list describes basic information:
  • The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
  • The volume that needs to be mounted is nginx-log and is of the emptyDir type. The nginx-log volume will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers.
  • The path to NGINX access logs is /var/log/nginx/access.log. The Logstore that is used to store the NGINX access logs is nginx-access.
  • The path to NGINX error logs is /var/log/nginx/error.log. The Logstore that is used to store the NGINX error logs is nginx-error.
  • Sidecar configuration example
    apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "1023****3423"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "nginx-log-sidecar"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
  • CRD configuration example

    Create a Logtail configuration to collect NGINX access logs and another Logtail configuration to collect NGINX error logs.

    • Collect NGINX access logs
      Notice In Sidecar mode, you must set the dockerFile parameter to false. 
      apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in your Kubernetes cluster. 
  name: nginx-log-access-example
spec:
  # The name of the project. The default value is the name of the project that you use to install Logtail components. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: nginx-access
  # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. 
    configName: nginx-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path to the log file. 
      logPath: /var/log/nginx
      # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. 
      filePattern: access.log
      # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
      dockerFile: false
      # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. 
      logBeginRegex: '.*'
      # The regular expression that is used to extract log content. You can configure this parameter based on your business requirements. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # The keys that you want to extract from logs. 
      key : ["time", "ip", "method", "url", "protocol",ayloa "latency", "pd", "status", "response-size",user-agent"]
    • Collect NGINX error logs
      Notice In Sidecar mode, you must set the dockerFile parameter to false. 
      # config for error log
apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in your Kubernetes cluster. 
  name: nginx-log-error-example
spec:
  # The name of the project. The default value is the name of the project that you use to install Logtail components. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: nginx-error
  # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. 
    configName: nginx-log-error-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path to the log file. 
      logPath: /var/log/nginx
      # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. 
      filePattern: error.log
      # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
      dockerFile: false

Configuration example for different directories

The following procedure shows how to use a CRD to collect text logs from the nginx-log-demo container in Sidecar mode. The container belongs to a self-managed Kubernetes cluster in a data center. The text logs include NGINX access logs and are stored in different directories. The following list describes basic information:

  • The Log Service project for log collection resides in the China (Hangzhou) region. Logs are collected over the Internet.
  • The volumes that need to be mounted are nginx-log and nginx-logs and are of the emptyDir type. The nginx-log volume will be mounted to the /var/log/nginx directory of the nginx-log-demo and Logtail containers. The nginx-logs volume will be mounted to the /var/log/nginxs directory of the nginx-log-demo and Logtail containers.
  • One log file path is /var/log/nginx/access.log and the other log file path is /var/log/nginxs/access.log.
  • The Logstore that is used to store NGINX access logs is nginx-access.
  • Sidecar configuration example
    apiVersion: batch/v1
kind: Job
metadata:
  name: nginx-log-sidecar-demo
  namespace: default
spec:
  template:
    metadata:
      name: nginx-log-sidecar-demo
    spec:
      restartPolicy: Never
      containers:
      - name: nginx-log-demo
        image: registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest
        command: ["/bin/mock_log"]
        args: ["--log-type=nginx", "--stdout=false", "--stderr=true", "--path=/var/log/nginx/access.log", "--total-count=1000000000", "--logs-per-sec=100"]
        lifecycle:
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
        - name: nginx-logs
          mountPath: /var/log/nginxs
      ##### logtail sidecar container
      - name: logtail
        # more info: https://cr.console.aliyun.com/repository/cn-hangzhou/log-service/logtail/detail
        # this images is released for every region
        image: registry.cn-hangzhou.aliyuncs.com/log-service/logtail:latest
        # when recevie sigterm, logtail will delay 10 seconds and then stop
        lifecycle:
        command:
        - sh
        - -c
        - /usr/local/ilogtail/run_logtail.sh 10
        livenessProbe:
          exec:
            command:
            - /etc/init.d/ilogtaild
            - status
          initialDelaySeconds: 30
          periodSeconds: 30
        resources:
          limits:
            memory: 512Mi
          requests:
            cpu: 10m
            memory: 30Mi
        env:
          ##### base config
          # user id
          - name: "ALIYUN_LOGTAIL_USER_ID"
            value: "1023****3423"
          # user defined id
          - name: "ALIYUN_LOGTAIL_USER_DEFINED_ID"
            value: "nginx-log-sidecar"
          # config file path in logtail's container
          - name: "ALIYUN_LOGTAIL_CONFIG"
            value: "/etc/ilogtail/conf/cn-hangzhou-internet/ilogtail_config.json"
          ##### env tags config
          - name: "ALIYUN_LOG_ENV_TAGS"
            value: "_pod_name_|_pod_ip_|_namespace_|_node_name_|_node_ip_"
          - name: "_pod_name_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: "_pod_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: "_namespace_"
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: "_node_name_"
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          - name: "_node_ip_"
            valueFrom:
              fieldRef:
                fieldPath: status.hostIP
        volumeMounts:
        - name: nginx-log
          mountPath: /var/log/nginx
        - name: nginx-logs
          mountPath: /var/log/nginxs
      ##### share this volume
      volumes:
      - name: nginx-log
        emptyDir: {}
      - name: nginx-logs
        emptyDir: {}
  • CRD configuration example

    Create two Logtail configurations to collect NGINX access logs from different directories.

    • Collect NGINX access logs from /var/log/nginx/access.log
      Notice In Sidecar mode, you must set the dockerFile parameter to false. 
      apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in your Kubernetes cluster. 
  name: nginx-log-access-example
spec:
  # The name of the project. The default value is the name of the project that you use to install Logtail components. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: nginx-access
  # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. 
    configName: nginx-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path to the log file. 
      logPath: /var/log/nginx
      # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. 
      filePattern: access.log
      # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
      dockerFile: false
      # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. 
      logBeginRegex: '.*'
      # The regular expression that is used to extract log content. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # The keys that you want to extract from logs. 
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
    • Collect NGINX access logs from /var/log/nginxs/access.log
      Notice In Sidecar mode, you must set the dockerFile parameter to false. 
      apiVersion: log.alibabacloud.com/v1alpha1
kind: AliyunLogConfig
metadata:
  # The name of the resource. The name must be unique in your Kubernetes cluster. 
  name: nginxs-log-access-example
spec:
  # The name of the project. The default value is the name of the project that you use to install Logtail components. 
  project: k8s-nginx-sidecar-demo
  # The name of the Logstore. If the Logstore that you specify does not exist, Log Service automatically creates a Logstore. 
  logstore: nginxs-access
  # The name of the machine group. Set the value to the value of ${your_machine_group_user_defined_id} when you install Sidecar. 
  machineGroups:
  - nginx-log-sidecar
  # The Logtail configuration. 
  logtailConfig:
    # The type of the data source. In Sidecar mode, you can use CRDs to collect only text logs. Therefore, you must set the value to file. 
    inputType: file
    # The name of the Logtail configuration. The name must be the same as the resource name that is specified by the metadata.name parameter. 
    configName: nginxs-log-access-example
    inputDetail:
      # Set logType to common_reg_log. 
      logType: common_reg_log
      # Specify the path to the log file. 
      logPath: /var/log/nginxs
      # The name of the log file. You can use wildcard characters such as asterisks (*) and question marks (?) when you specify the name of the log file. Example: log_*.log. 
      filePattern: access.log
      # Set the dockerFile parameter to false. This setting is required in Sidecar mode. 
      dockerFile: false
      # The regular expression that is used to match the beginning of the first line of a log. If you collect single-line logs, set the value to '.*'. 
      logBeginRegex: '.*'
      # The regular expression that is used to extract log content. 
      regex: '(\S+)\s(\S+)\s\S+\s\S+\s"(\S+)\s(\S+)\s+([^"]+)"\s+(\S+)\s(\S+)\s(\d+)\s(\d+)\s(\S+)\s"([^"]+)"\s.*'
      # The keys that you want to extract from logs. 
      key : ["time", "ip", "method", "url", "protocol", "latency", "payload", "status", "response-size",user-agent"]
# config for error log