All Products
Search
Document Center

Container Service for Kubernetes:Collect application logs by using pod environment variables

Last Updated:Apr 07, 2024

Simple Log Service is a one-stop log service of Alibaba Cloud. It allows you to directly collect, consume, deliver, query, and analyze log data in the console. Simple Log Service is hosted in ACK Serverless clusters. This topic describes how to configure Simple Log Service environment variables in an ACK Serverless cluster to allow Simple Log Service to collect stdout and log files from application pods in the cluster.

Prerequisites

An ACK Serverless cluster is created and alibaba-log-controller is installed in the cluster.

  • To install alibaba-log-controller when you create the cluster, select Enable Log Service on the Component Configurations wizard page in the ACK console. For more information, see Create an ACK Serverless cluster.

  • To install alibaba-log-controller after you create the cluster, go to the cluster details page, choose Operations > Add-ons in the left-side navigation pane, and then click Install in the alibaba-log-controller card in the Logs and Monitoring section. In the message that appears, click OK.

Step 1: Create an application and configure Simple Log Service to collect application logs

You can configure Simple Log Service to collect container logs when you create an application. You can create an application from an image or a YAML template.

Create an application from an image

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Deployments in the left-side navigation pane.

  3. In the upper part of the Deployments page, select the namespace where you want to create an application from the Namespace drop-down list. Then, click Create from Image in the upper-right corner of the page.

    1. On the Basic Information wizard page, set the Name, Replicas, and Type parameters. Then, click Next.

    2. On the Container wizard page, set the parameters in the Log Service section.

      Note

      The following table describes only the parameters related to Simple Log Service. For more information about other application parameters, see Create a Deployment.

      Example:采集配置

      Parameter

      Description

      Collection Configuration

      Click Collection Configuration. Set Logstore and Log Path in Container (Can be set to stdout).

      • Logstore: the name of the Logstore that you want to use. The name can contain only lowercase letters, digits, and hyphens (-).

        You can use this parameter to specify the Logstore that stores the collected log data. If the specified Logstore does not exist, the system automatically creates a Logstore in the Simple Log Service project that is associated with the cluster.

      • Log Path in Container (Can be set to stdout): the path from which you want to collect log data. A value of /usr/local/tomcat/logs/catalina.*.log indicates that the log files of a Tomcat application are collected.

        Note

        If you set the value to stdout, stdout and stderr are collected.

      All settings are added as configuration entries to the corresponding Logstore. By default, logs are collected in simple mode (by row). If you want to use other methods to collect log data, log on to the Simple Log Service console and modify the log collection configurations of the project and Logstore. By default, the project uses the k8s-log prefix.

      Custom Tag

      Click Custom Tag. Set Tag Key and Tag Value.

      Each tag is a key-value pair that is appended to the collected log data. You can use custom tags to mark log data. For example, you can use a tag to denote the application version.

    3. After you configure the parameters, click Next to configure advanced settings. For more information about the subsequent steps, see Create a Deployment.

Create an application from a YAML template

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Workloads > Deployments in the left-side navigation pane.

  3. In the upper part of the Deployments page, select the namespace where you want to create an application from the Namespace drop-down list. Then, click Create from YAML in the upper-right corner of the page.

  4. On the Create page, select a template from the Sample Template drop-down list and modify the template content in the Template section based on your requirements. Then, click Create.

    YAML templates comply with the Kubernetes syntax. You can use env to define log collection configurations and custom tags. You must also set the volumeMounts and volumes parameters. Example:

    apiVersion: v1
    kind: Pod
    metadata:
      name: my-demo
    spec:
      containers:
      - name: my-demo-app
        image: 'registry.cn-hangzhou.aliyuncs.com/log-service/docker-log-test:latest'
        env:
        ######### Specify environment variables ###########
        - name: aliyun_logs_log-stdout
          value: stdout
        - name: aliyun_logs_log-varlog
          value: /var/log/*.log
        - name: aliyun_logs_mytag1_tags
          value: tag1=v1
        ######### Specify volume mounting parameters. ###########
        volumeMounts:
        - name: volumn-sls-mydemo
          mountPath: /var/log
      volumes:
      - name: volumn-sls-mydemo
        emptyDir: {}

    You can add the following configurations based on your requirements:

    Note

    If you have other requirements on log collection, see Step 2: Configure advanced settings in the env field.

    1. Add log collection configurations and custom tags by using environment variables. All environment variables related to log collection must use aliyun_logs_ as the prefix.

      1. Add log collection configurations in the following format:

        - name: aliyun_logs_log-stdout
          value: stdout
        - name: aliyun_logs_log-varlog
          value: /var/log/*.log

        In the preceding example, two environment variables in the following format are added to the log collection configuration: aliyun_logs_{key}. The {keys} of the environment variables are log-stdout and log-varlog.

        • The aliyun_logs_log-stdout environment variable indicates that a Logstore named log-stdout is created to store the stdout collected from containers. The name of the collection configuration is log-stdout. This way, the stdout of containers is collected to the Logstore named log-stdout.

        • The aliyun_logs_log-varlog environment variable indicates that a Logstore named log-varlog is created to store the collected log files from the /var/log/*.log path. The name of the collection configuration is log-varlog. This way, the log files in the /var/log/*.log path are collected to the Logstore named log-varlog.

      2. Add custom tags in the following format:

        - name: aliyun_logs_mytag1_tags
          value: tag1=v1

        After a tag is added, the tag is automatically appended to the log data that is collected from the container. The value of mytag1 can contain only lowercase letters, digits, and hyphens (-).

    2. If you specify a log path to collect log files other than stdout, you must set the volumeMounts parameter. In the preceding YAML template, the mountPath field in volumeMounts is set to /var/log. This allows Logtail to collect log data from the /var/log/*.log file.

Step 2: Configure advanced settings in the env field

You can configure container environment variables to customize log collection. You can configure advanced settings to meet your log collection requirement.

Important

You cannot use environment variables to configure log collection in edge computing scenarios.

Variable

Description

Example

Usage notes

aliyun_logs_{key}

  • Required. {key} can contain only lowercase letters, digits, and hyphens (-).

  • If the specified aliyun_logs_{key}_logstore does not exist, a Logstore named {key} is created.

  • To collect the stdout of a container, set the value to stdout. You can also set the value to a path inside the container to collect the log files.

  • - name: aliyun_logs_catalina
    
       stdout
  • - name: aliyun_logs_access-log
    
       /var/log/nginx/access.log
  • By default, the Simple Log Service component collects log files in simple mode. In this case, the collected log data is not parsed. If you want to parse the log data, we recommend that you change the collection mode in the Simple Log Service console.

  • The value of {key} must be unique in the cluster.

aliyun_logs_{key}_tags

Optional. This variable is used to add tags to log data. The value must be in the following format: {tag-key}={tag-value}.

- name: aliyun_logs_catalina_tags

   app=catalina

-

aliyun_logs_{key}_project

Optional. The variable specifies a project in Simple Log Service. By default, the project that you specified when you created the cluster is used.

- name: aliyun_logs_catalina_project

   my-k8s-project

The project must be deployed in the same region as Logtail.

aliyun_logs_{key}_logstore

Optional. The variable specifies a Logstore in Simple Log Service. By default, the Logstore is named {key}.

- name: aliyun_logs_catalina_logstore

   my-logstore

-

aliyun_logs_{key}_shard

Optional. The variable specifies the number of shards of the Logstore. Valid values: 1 to 10. Default value: 2.

- name: aliyun_logs_catalina_shard

   4

-

aliyun_logs_{key}_ttl

Optional. The variable specifies the log retention period. Valid values: 1 to 3650.

  • To retain log data permanently, set the value to 3650.

  • The default retention period is 90 days.

- name: aliyun_logs_catalina_ttl

   3650

-

aliyun_logs_{key}_machinegroup

Optional. The variable specifies the node group in which the application is deployed. The default node group is the one in which Logtail is deployed.

- name: aliyun_logs_catalina_machinegroup

   my-machine-group

-

  • Scenario 1: Collect log data from multiple applications and store them in the same Logstore

    In this scenario, set the aliyun_logs_{key}_logstore variable. The following example shows how to collect stdout from two applications and store the output in stdout-logstore.

    The {key} of Application 1 is set to app1-stdout. The {key} of Application 2 is set to app2-stdout.

    Configure the following environment variables for Application 1:

    ######### Specify environment variables. ###########
        - name: aliyun_logs_app1-stdout
          value: stdout
        - name: aliyun_logs_app1-stdout_logstore
          value: stdout-logstore

    Configure the following environment variables for Application 2:

    ######### Specify environment variables. ###########
        - name: aliyun_logs_app2-stdout
          value: stdout
        - name: aliyun_logs_app2-stdout_logstore
          value: stdout-logstore
  • Scenario 2: Collect log data from different applications and store them in different projects

    In this scenario, perform the following steps:

    1. Create a machine group in each project and set the machine group ID in the following format: k8s-group-{cluster-id}, where {cluster-id} is the ID of the cluster. You can customize the machine group name.

    2. Specify the project, Logstore, and the machine group in the environment variables for each application. The name of the machine group is the same as that of the machine group that you created in the previous step.

      In the following example, the {key} of Application 1 is set to app1-stdout. The {key} of Application 2 is set to app2-stdout. If the two applications are deployed in the same ACK cluster, you can use the same machine group for the applications.

      Configure the following environment variables for Application 1:

      ######### Specify environment variables. ###########
          - name: aliyun_logs_app1-stdout
            value: stdout
          - name: aliyun_logs_app1-stdout_project
            value: app1-project
          - name: aliyun_logs_app1-stdout_logstore
            value: app1-logstore
          - name: aliyun_logs_app1-stdout_machinegroup
            value: app1-machine-group

      Configure the following environment variables for Application 2:

      ######### Specify environment variables for Application 2.###########
          - name: aliyun_logs_app2-stdout
            value: stdout
          - name: aliyun_logs_app2-stdout_project
            value: app2-project
          - name: aliyun_logs_app2-stdout_logstore
            value: app2-logstore
          - name: aliyun_logs_app2-stdout_machinegroup
            value: app1-machine-group

Step 3: View log data

To view the collected log data in the ACK console, perform the following steps:

  1. Log on to the ACK console. In the left-side navigation pane, click Clusters.

  2. On the Clusters page, click the name of the cluster that you want to manage and choose Operations > Log Center in the left-side navigation pane.

  3. On the Log Center page, click the Application Logs tab and select a Logstore to view the logs of containers.